Benchmarking Methodology

timezonefinder publishes performance numbers, appends them to a trend chart on every push to master and compares every pull request against its own baseline. This page describes how those measurements are taken and, more importantly, what they can and cannot tell you.

The short version: these numbers are noisy for reasons that have nothing to do with this package’s code, the measurement design exists to work around that, and every threshold is derived from measured noise rather than picked.

This page is the why. The operational side - which make target to run and what to check when a report looks wrong - lives in Benchmarking and performance validation.

The Workload

Each benchmark times one pass over a fixed batch of inputs rather than a single call, so every round performs identical work and the spread between rounds is measurement noise rather than a difference in what was measured. The batch size is BATCH_SIZE in benchmarks/conftest.py, currently 2,500 - large enough that a single round is well above timer resolution, and bounded above by the committed fixtures it draws from (_load_batch needs BATCH_SIZE points per fixture, pip_inputs_by_stratum needs BATCH_SIZE per stratum, the binding ceiling). Changing it invalidates the historical trend data, because a data point is only comparable to another data point that did the same amount of work.

The inputs themselves are deterministic committed fixtures (tests/fixtures/benchmarks/, generated by scripts/generate_benchmark_fixtures.py) rather than freshly drawn random data, so two runs of the same commit execute the exact same workload. They are pinned to the DATA_VERSION and FIXTURE_VERSION they were generated against, and the loader refuses a mismatch instead of silently benchmarking a workload the checkout no longer describes.

Two samplers, on purpose

Benchmark query points are drawn uniformly per unit of surface area (get_rnd_query_pt_area_weighted). The rest of the test suite uses get_rnd_query_pt, which is uniform in latitude and therefore oversamples the poles by roughly 2.5x.

That is not an inconsistency to fix. Correctness and fuzz tests want the polar bias - more edge cases per draw. A benchmark must instead represent real query load, and a pole-biased sampler inflates the share of ambiguous-shortcut queries, which are by far the most expensive class (see below). Using the wrong sampler would make the headline number describe a workload nobody has.

The point-in-polygon fixture is stratified by polygon vertex count for the same reason: the cost of the largest polygons would otherwise disappear behind an unweighted average.

`ubuntu-latest` does not pin the CPU

This is the single most important thing to know about the CI numbers.

runs-on: ubuntu-latest guarantees a runner image, not hardware. This project’s runs have landed on AMD EPYC 9V74, AMD EPYC 7763 and Intel Xeon Platinum 8573C parts between 2.30 and 3.69 GHz, and the clock varies run to run even within one model. Measured across eleven recorded runs whose lookup path was unchanged, the tracked min spread 134-158 % - larger than most changes worth reviewing.

The consequence is not subtle. A merged change that was a genuine 1.5x improvement once appeared on the trend chart as a 21 % regression, purely because of which machine each run drew. Any methodology that compares two arbitrary CI runs to each other is measuring the runner pool.

Consequences for the measurement design

Same-runner, merge-base comparison

A pull request is measured against its own merge base, in the same job, on the same runner - never against a stored master baseline. The measuring job checks out the merge base alongside the head, installs and measures both, and renders the base/head ratio table (scripts/compare_benchmark_runs.py).

That table verifies rather than assumes that both sides ran on one machine, and warns if the batch size, fixture set, boundary data or acceleration path differ between them - a comparison across any of those is meaningless and should say so rather than print a plausible-looking ratio.

The job holds no write permissions and no secrets, so it behaves identically for branch PRs and fork PRs, and a fork PR never fails for want of a token. The comparison comment is posted by a separate, privileged workflow triggered via workflow_run.

The base is measured twice

The base is measured once before and once after the head, sandwiching it, and the two passes are reduced by min. A runner that drifts over the job’s lifetime then shows up in the base’s own spread instead of looking like a code change. This roughly doubles the measuring job (~2-3 min), almost all of it the second checkout, uv sync and C extension build - the measurement itself is seconds.

Every run names its machine

scripts/describe_benchmark_machine.py prints the CPU, the acceleration path and the workload provenance to the job summary, and both chart exports write the same label into every stored point. Hovering a data point therefore attributes it to a CPU long after the artifact has expired - which is the first thing to check when the chart shows a step change.

The published report pages are also measured in CI, but by an explicit full-suite job rather than the per-pull-request job. That job uploads commit-bound pages for the release workflow to consume; it never commits to master. Every page prints the runner CPU from its measurement JSON, so an absolute figure remains attributable after the artifact expires.

The tracked estimator: `min`, not `mean`

Since every round performs an identical fixed batch of work, the fastest round is the one least perturbed by whatever else the shared, virtualised runner happened to be doing. The tracked value is therefore pytest-benchmark’s min.

Getting that past the tooling takes a deliberate step, because every reader of a pytest-benchmark JSON reaches for the mean by default. scripts/normalize_benchmark_json.py rewrites stats.ops/stats.mean from the min and records which estimator it used, so the stored report and every consumer of it - the pull request comparison, the job summary, both chart exports - carry the estimator this project chose rather than the one the tooling defaults to.

What the trend chart plots: lookups/sec

benchmark-action/github-action-benchmark picks its number with a per-tool extractor, and the pytest one is hard-wired to stats.ops under the label iter/sec. One “iteration” here is a whole batch, so that chart read 183 iter/sec for test_timezone_at[random-in_memory]: batches per second, a unit nothing else in this project quotes, under a name that says nothing about the workload.

Both suites are therefore exported explicitly instead. scripts/export_timing_chart_json.py divides the tracked duration by the batch size the measuring run recorded and stores lookups/sec - the same quantity the published reports state as Time/Query, though not the same number, since the chart tracks the core subset’s min per commit and that column is a fixed-round full-suite mean from another job - under the same human-readable labels those reports use (TimezoneFinder.timezone_at() - random points, in-memory). tool: customBiggerIsBetter takes that shape; scripts/export_memory_chart_json.py is its mirror for the footprint, where a rise is the regression.

The unit is a batch throughput, not a per-call latency: it says nothing about the tail, for exactly the reasons the next section gives.

What the batch form cannot say, and why a distribution is published beside it

Everything above is built on the batch: one round is one pass over 2,500 points, and the tracked value is the fastest such pass. That design buys comparability on a noisy runner, and it pays for it by discarding the shape of the distribution before any estimator sees it. A batch mean cannot distinguish 2,500 queries that each cost 2 µs from 2,475 that cost 1 µs and 25 that cost 100 µs.

For this package that distinction is the interesting one. A query whose H3 cell holds a single zone reads no geometry at all and costs ~1 µs; a query that falls in a very large boundary polygon is answered by one ray cast across that whole ring and costs tens of microseconds. Query time correlates 0.92 with vertices tested and only 0.76 with the number of candidate polygons: the tail is one expensive test, not many cheap ones. A change that halves that tail while leaving the median alone is a large improvement for anyone with a latency budget, and the batch estimators report it as a modest shift in one number.

So the per-query distribution is measured separately - scripts/measure_query_latency.py, run by make latency - and published as a section of Timezone Finding Performance Benchmark. It reports p50/p90/p99/p99.9 per point class, in the default memory-mapped mode on the same acceleration path the batch suite asserts, and it times a single timezone_at call per sample. Each query keeps its fastest observation across repeated passes, which is the same argument min rests on applied per query rather than per round.

It is published beside the batch tables rather than instead of them, because the two answer different questions and the batch form remains the right instrument for comparing two commits: it is the one whose noise behaviour is characterised, whose thresholds are derived, and which CI can run on a runner it does not control. The distribution is not tracked on the trend chart for exactly that reason - a p99 taken on one arbitrary machine of a heterogeneous pool would be noise on top of noise.

What CI measures

The core subset

The nine -m benchmark_core cases comprise three scalar in_memory benchmarks and six batch benchmarks in the default file-based mode. The batch cases measure timezone_ids_at and timezone_names_at across the same random, unique-shortcut and ambiguous-shortcut strata, so CI can detect regressions confined to vectorised validation, scaling, or result conversion. The rest of the suite is for the docs and is not run per PR.

test_timezone_at[random-in_memory] is the headline. Uniformly random points are the only globally representative workload: they contain unique- and ambiguous-shortcut queries in their real ratio (~11 % ambiguous), so a change is weighted by how much real query load it actually helps.

unique_shortcut-in_memory and ambiguous_shortcut-in_memory are tracked alongside it as diagnostics, because the headline alone cannot attribute a change to a code path. On the tracked configuration an ambiguous lookup costs ~14x a unique one (~7x with Numba), so ambiguous work takes ~62 % of the wall clock despite being ~11 % of the queries. A win confined to the unique path therefore moves the headline by only ~0.38x its true size - enough dilution to sink a small win below the noise floor. The per-class benchmarks show it undiluted.

The tracked configuration

Only the no-Numba / clang C extension path, because that is what a plain pip install timezonefinder gives you and what constrained containers actually run.

timezonefinder/utils.py selects the point-in-polygon implementation at import time, so Numba and clang are completely different code paths whose numbers must never share a benchmark name. The workflow asserts the active path (scripts/assert_acceleration_path.py) rather than assuming it, and so does every make target that measures - the memory ones included, because importing Numba costs resident memory as well as changing the timings. A Numba install sneaking into the environment would otherwise silently corrupt the entire trend history rather than fail, and locally it is the normal state: make install syncs every dependency group.

For the same reason, local numbers are not comparable to CI numbers - different CPU, different memory bandwidth, different background load, and often a different acceleration path. The published pages and the trend chart both use the clang path, but are still separate experiments on independently drawn runners: the pages lead with mean over the full, fixed-round suite, while the chart records min over benchmark_core. Each published page states which of its rows, if any, belongs to that tracked subset. Compare local-to-local and paired CI-to-CI only.

Thresholds derived from measured noise

The trend chart alert: 180 %

ALERT_THRESHOLD is derived from measurement, not chosen. Across eleven recorded scalar-core runs whose lookup path was identical, the tracked min spread 134-158 % (unique 134.3 %, random 145.9 %, ambiguous 158.4 %) purely because of the hardware each run drew. After the six batch cases joined the core, five fresh runners measured a worst scalar spread of 138.9 % and a worst batch spread of 137.9 %. That sample drew AMD EPYC 9V74 and 7763 machines but not the Intel Xeon class in the earlier study, so it demonstrates that the batch cases do not widen the known bound; it does not erase the wider scalar observation. Worst known spread plus 20 % headroom rounds to the shipped 180 %.

Being honest about what that buys: at 180 % the chart catches only a catastrophic regression and is blind to the 10-30 % changes actually worth reviewing. That is not a gap to close by tightening the number - a cross-machine chart cannot resolve better than the machines it spans. The same-runner pull request comparison is the real gate; the trend alert is a deliberately weak backstop for master, which nothing else watches. Alerts are non-blocking and stay that way: master must never be blocked on which machine a run drew.

It is re-derived whenever the runner pool or the core set changes, by repeating the measurement on unchanged code (scripts/benchmark_noise.py). Note what that job characterises: each repetition runs on a different machine, so it measures the runner pool’s spread, not a single runner’s jitter.

The pull request flag: 110 %

Same-runner measurement removes the machine-to-machine term but not the runner’s own jitter, so a few percent either way is still noise. Rows in the comparison table are flagged at REGRESSION_THRESHOLD_PCT (110 %).

That number comes from the closest thing there is to a same-runner measurement: a five-run study that spread only 106.8 %, against a pool that spreads up to 158 % across machines - so those five must have drawn near-identical hardware. It was originally mistaken for a cross-runner bound when it set the trend threshold; as a stand-in for single-machine jitter it is defensible, and an upper bound on it either way.

Two estimators, and the rows they disagree about

Every row of the comparison also carries the median change beside the tracked min one, and is marked unresolved when the two land on different sides of the flag. This is the same rule the candidate comparisons below are held to, applied to the committed base/head table: where a difference is real the two estimators move together, and where they disagree the honest reading is that there is no demonstrable difference - an answer a single estimator has no way to give. The flag itself stays on min, because that is the number the trend chart and --fail-on-regression use. median is the corroborating statistic because scripts/normalize_benchmark_json.py overwrites mean with the tracked value, which leaves it the only untouched estimator in a stored report.

Reporting only

The comparison is reporting only. --fail-on-regression exists but is not passed, and stays off until a single-runner noise study has said what the residual floor actually is. Until then a gate would fire on noise, and a gate everyone learns to ignore is worse than no gate.

The trend chart is likewise not used to judge a pull request. It is cross-machine by construction, and the comment workflow deliberately does not compare against it - a constraint tests/test_benchmark_workflows.py enforces rather than leaves to convention.

Comparing two implementations of one stage

The suites above compare one implementation across commits or machines. Deciding between two candidate implementations of a single stage - a data structure, an accessor, a dispatch - is a different measurement, and three designs for it have produced wrong answers in this repository. All three flattered the newer candidate.

Do not microbenchmark the stage in isolation. Two candidates rarely divide the work at the same place, so a boundary drawn around “the lookup” charges one of them for something the other pays a moment later. Replacing the shortcut dict was measured this way and the dict came out ~100 ns ahead, which would have been ~10 % of a unique-zone query. It was an artefact: the shipped code answers dict.get with a match value: case int(zone_id) that the flat structure needs no equivalent of, and which costs 84 -> 188 ns. Measure the whole public call and let the boundary fall where it falls.

Alternate the order of a paired comparison. Running A then B inside each round lets A warm everything the two share - coordinate validation, the H3 call, the zone-name lookup, the branch predictors - and hands B the benefit for free. The same shortcut comparison read as 13.3 % faster in a fixed order and 0.3 % once the order alternated round by round.

Report two estimators and believe them only when they agree. The ratio of the two best rounds is the least noise-sensitive estimator; the count of rounds where the candidate won assumes nothing about the noise distribution. Where a difference is real they move together. Where they disagree - one saying +0.5 % and the other 26 of 61 rounds - there is no effect to find, and that disagreement is a more useful output than either number alone.

Sample the inputs at random. Iterating a dict’s keys in its own order walks its table front to back and hands it a cache-friendly access pattern no real query stream has. Worth 77 ns against 108 ns on the same lookup - a third of the quantity being compared.

Two processes cannot hold a pair. Three of the four properties above are properties of one loop holding both candidates: the order cannot alternate across processes, the two cannot be handed the same draw, and the round-level win count is not even defined. What survives is the best-round ratio alone - the single estimator this list exists to stop anyone believing on its own. So where a comparison genuinely spans two environments, as comparing the three point-in-polygon acceleration paths does (Numba and pure Python are one source decorated or not, so no process holds both), each pair is still measured inside one process against the implementation both environments hold, and the ratio that would have to cross the boundary is not computed at all. The two runs’ shared baseline is published beside the results as the reader’s own comparability check - see Point-in-Polygon Acceleration Paths, where it shows that installing Numba moves the coordinate validators too, not only the kernel.

All four are held by benchmarks/candidate_comparison.py rather than re-derived per attempt. compare_candidates takes two named callables that each perform the whole public call for one input, draws a fresh random sample per round and hands the same draw to both, alternates which of them runs first, and reports the best-round ratio beside the round-level win count. Where the two estimators agree it says faster, slower or no difference; where they disagree it says unresolved, which is the answer neither estimator can give alone. The default 61 rounds is odd so the sign count cannot tie, and the default 3 % threshold is the bottom of a single machine’s own jitter - an effect below it is not demonstrable by any number of rounds. Like every other measurement here it is reporting only: nothing in it fails a build.

Memory is measured the same way

Memory has its own harness (scripts/measure_memory.py) rather than a benchmarks/ suite: pytest-benchmark measures wall clock, and running tracemalloc across its calibration rounds would distort the very timings those suites exist to produce.

Each configuration is measured in a fresh subprocess. import timezonefinder costs ~95 MiB of NumPy and H3 before any timezone data is touched, so only a delta against a post-import baseline is meaningful - and a second finder built in the same process would inherit the first one’s warmed page cache and freed-but-unreturned arenas.

Two metrics per checkpoint, and the gap between them is the signal. *_heap is what tracemalloc accounts for (Python and NumPy allocations); *_rss is the resident set, which additionally counts memory-mapped pages. With in_memory=False the coordinate data is mapped rather than read, so its heap stays small while its RSS grows across the workload as lookups fault pages in - which is why there is both an init and a steady checkpoint. See TimezoneFinder Memory Footprint for the measured figures.

Only the heap metrics are charted. RSS residency is decided by machine-wide memory pressure, so tracking it would alert on the runner’s mood: repeated measurement puts the heap metrics at a 100.0 % spread against 102-111 % for RSS. The alert threshold for memory is correspondingly tight (110 %), because tracemalloc is near-deterministic and a change there is signal rather than jitter.

In the shared CI job, every memory step runs after every timing step. The harness reads the whole boundary dataset and warms the OS file cache, which is exactly what benchmarks/test_initialization.py disables pytest-benchmark’s warmup to avoid.

Names are join keys

The chart labels are the join key of the timing trend chart, and memory metric names are the join key of the memory one. Renaming either does not move a metric’s history - it silently starts a new, empty one alongside the orphaned old chart. Because the timing labels are the rendered ones, a docs-facing wording change in scripts/render_benchmark_reports.py is a history-facing change; node ids remain the join key of every stored report and every pull request comparison.

All three sets are therefore pinned by tests (tests/test_benchmark_names.py, tests/test_memory_metric_names.py), so a rename is a deliberate act with a visible cost rather than an invisible reset. When one is intended, scripts/migrate_benchmark_chart_history.py restates the stored points under the new names so the history follows the rename instead of being orphaned by it. This is also why benchmarks must always pass explicit ids=/pytest.param(..., id=...) instead of relying on pytest’s autogenerated parametrize ids, which change when an unrelated parameter is added.

Comparing against another package

Everything above is about comparing this package to itself. benchmarks/test_comparison.py compares it to tzfpy, and the rules that make a cross-machine comparison meaningless apply with more force across packages, not less: the published figures for two libraries come from two machines, two acceleration paths and two query workloads, and the difference between them is not a difference between the libraries.

So the comparison is run the only way that answers anything. Both packages answer the same committed query fixtures, in the same process, on the same machine, and each is called through its own API inside its own loop - timezone_at(lng=, lat=) and get_tz(lng, lat) - so neither pays for an adapter frame the other does not. At per-query times of a few hundred nanoseconds, one extra Python call frame is worth tens of percent, which is the same order as the thing being measured. Comparison against tzfpy carries the result.

Three things this deliberately does not do:

It is not tracked. None of these benchmarks carry benchmark_core, so none reach the trend chart. The number is a ratio between two packages and one of them releases on a schedule this project does not control, so a step in that chart would as often mean “they shipped” as “we changed” - and nothing on the chart could say which. The measured version is stamped into the report instead.
It does not decide accuracy. The two packages disagree on a small fraction of points. Counting disagreements says nothing about which answer is right; that needs ground truth neither package carries. Alternatives states the design difference instead of scoring it.
It does not measure the other package’s memory. scripts/measure_memory.py constructs finders from this repository and has no configuration for anything else.

The startup half of that suite is worth its own note, because the obvious measurement is the wrong one. Timing construction would score tzfpy as free: it imports in about a millisecond and then deserialises its index inside the first query, where this package imports NumPy and H3 and reads its index when a finder is built. What a caller actually waits for is the first answer, so that is what is measured - a fresh python -c per round, with a bare-interpreter row alongside it so the interpreter’s own cost can be subtracted.

tzfpy lives in the compare dependency group, which only make benchmarks and tox -e benchmarks install. The CI-tracked environment is deliberately uv sync --group test - what a plain pip install timezonefinder gives you - and must not gain a package for the sake of a report.