Benchmarking Methodology

timezonefinder publishes performance numbers, appends them to a trend chart on every push to master and compares every pull request against its own baseline. This page describes how those measurements are taken and, more importantly, what they can and cannot tell you.

The short version: these numbers are noisy for reasons that have nothing to do with this package’s code, the measurement design exists to work around that, and every threshold is derived from measured noise rather than picked.

This page is the why. The operational side - which make target to run and what to check when a report looks wrong - lives in Benchmarking and performance validation.

The Workload

Each benchmark times one pass over a fixed batch of inputs rather than a single call, so every round performs identical work and the spread between rounds is measurement noise rather than a difference in what was measured. The batch size is BATCH_SIZE in benchmarks/conftest.py, currently 2,500 - large enough that a single round is well above timer resolution, and bounded above by the committed fixtures it draws from (_load_batch needs BATCH_SIZE points per fixture, pip_inputs_by_stratum needs BATCH_SIZE per stratum, the binding ceiling). Changing it invalidates the historical trend data, because a data point is only comparable to another data point that did the same amount of work.

The inputs themselves are deterministic committed fixtures (tests/fixtures/benchmarks/, generated by scripts/generate_benchmark_fixtures.py) rather than freshly drawn random data, so two runs of the same commit execute the exact same workload. They are pinned to the DATA_VERSION and FIXTURE_VERSION they were generated against, and the loader refuses a mismatch instead of silently benchmarking a workload the checkout no longer describes.

Two samplers, on purpose

Benchmark query points are drawn uniformly per unit of surface area (get_rnd_query_pt_area_weighted). The rest of the test suite uses get_rnd_query_pt, which is uniform in latitude and therefore oversamples the poles by roughly 2.5x.

That is not an inconsistency to fix. Correctness and fuzz tests want the polar bias - more edge cases per draw. A benchmark must instead represent real query load, and a pole-biased sampler inflates the share of ambiguous-shortcut queries, which are by far the most expensive class (see below). Using the wrong sampler would make the headline number describe a workload nobody has.

The point-in-polygon fixture is stratified by polygon vertex count for the same reason: the cost of the largest polygons would otherwise disappear behind an unweighted average.

`ubuntu-latest` does not pin the CPU

This is the single most important thing to know about the CI numbers.

runs-on: ubuntu-latest guarantees a runner image, not hardware. This project’s runs have landed on AMD EPYC 9V74, AMD EPYC 7763 and Intel Xeon Platinum 8573C parts between 2.30 and 3.69 GHz, and the clock varies run to run even within one model. Measured across eleven recorded runs whose lookup path was unchanged, the tracked min spread 134-158 % - larger than most changes worth reviewing.

The consequence is not subtle. A merged change that was a genuine 1.5x improvement once appeared on the trend chart as a 21 % regression, purely because of which machine each run drew. Any methodology that compares two arbitrary CI runs to each other is measuring the runner pool.

Consequences for the measurement design

Same-runner, merge-base comparison

A pull request is measured against its own merge base, in the same job, on the same runner - never against a stored master baseline. The measuring job checks out the merge base alongside the head, installs and measures both, and renders the base/head ratio table (scripts/compare_benchmark_runs.py).

That table verifies rather than assumes that both sides ran on one machine, and warns if the batch size, fixture set, boundary data or acceleration path differ between them - a comparison across any of those is meaningless and should say so rather than print a plausible-looking ratio.

The job holds no write permissions and no secrets, so it behaves identically for branch PRs and fork PRs, and a fork PR never fails for want of a token. The comparison comment is posted by a separate, privileged workflow triggered via workflow_run.

The base is measured twice

The base is measured once before and once after the head, sandwiching it, and the two passes are reduced by min. A runner that drifts over the job’s lifetime then shows up in the base’s own spread instead of looking like a code change. This roughly doubles the measuring job (~2-3 min), almost all of it the second checkout, uv sync and C extension build - the measurement itself is seconds.

Every run names its machine

scripts/describe_benchmark_machine.py prints the CPU, the acceleration path and the workload provenance to the job summary, and scripts/normalize_benchmark_json.py stamps the same label into the one field that survives into the trend chart. Hovering a data point therefore attributes it to a CPU long after the artifact has expired - which is the first thing to check when the chart shows a step change.

The tracked estimator: `min`, not `mean`

Since every round performs an identical fixed batch of work, the fastest round is the one least perturbed by whatever else the shared, virtualised runner happened to be doing. The tracked value is therefore pytest-benchmark’s min.

Getting that past the tooling takes a deliberate step: benchmark-action/github-action-benchmark’s pytest extractor reads only stats.ops (= 1 / stats.mean). scripts/normalize_benchmark_json.py rewrites ops/mean from the min before handing the report over, so the chart tracks the estimator this project chose rather than the one the extractor defaults to.

What CI measures

The core subset

Only three benchmarks (-m benchmark_core), all in_memory. The full suite is for the docs, on demand; it is not run per PR.

test_timezone_at[random-in_memory] is the headline. Uniformly random points are the only globally representative workload: they contain unique- and ambiguous-shortcut queries in their real ratio (~11 % ambiguous), so a change is weighted by how much real query load it actually helps.

unique_shortcut-in_memory and ambiguous_shortcut-in_memory are tracked alongside it as diagnostics, because the headline alone cannot attribute a change to a code path. On the tracked configuration an ambiguous lookup costs ~14x a unique one (~7x with Numba), so ambiguous work takes ~62 % of the wall clock despite being ~11 % of the queries. A win confined to the unique path therefore moves the headline by only ~0.38x its true size - enough dilution to sink a small win below the noise floor. The per-class benchmarks show it undiluted.

The tracked configuration

Only the no-Numba / clang C extension path, because that is what a plain pip install timezonefinder gives you and what constrained containers actually run.

timezonefinder/utils.py selects the point-in-polygon implementation at import time, so Numba and clang are completely different code paths whose numbers must never share a benchmark name. The workflow asserts the active path (scripts/assert_acceleration_path.py) rather than assuming it: a Numba install sneaking into the environment would otherwise silently corrupt the entire trend history rather than fail.

For the same reason, local numbers are not comparable to CI numbers - different CPU, different memory bandwidth, different background load, and a deliberately different acceleration path. Compare local-to-local and CI-to-CI only.

Thresholds derived from measured noise

The trend chart alert: 180 %

ALERT_THRESHOLD is derived from a measurement, not chosen. Across eleven recorded runs whose lookup path was identical, the tracked min spread 134-158 % (unique 134.3 %, random 145.9 %, ambiguous 158.4 %) purely because of the hardware each run drew. Worst spread plus 20 % headroom rounds to the shipped 180 %.

Being honest about what that buys: at 180 % the chart catches only a catastrophic regression and is blind to the 10-30 % changes actually worth reviewing. That is not a gap to close by tightening the number - a cross-machine chart cannot resolve better than the machines it spans. The same-runner pull request comparison is the real gate; the trend alert is a deliberately weak backstop for master, which nothing else watches. Alerts are non-blocking and stay that way: master must never be blocked on which machine a run drew.

It is re-derived whenever the runner pool or the core set changes, by repeating the measurement on unchanged code (scripts/benchmark_noise.py). Note what that job characterises: each repetition runs on a different machine, so it measures the runner pool’s spread, not a single runner’s jitter.

The pull request flag: 110 %

Same-runner measurement removes the machine-to-machine term but not the runner’s own jitter, so a few percent either way is still noise. Rows in the comparison table are flagged at REGRESSION_THRESHOLD_PCT (110 %).

That number comes from the closest thing there is to a same-runner measurement: a five-run study that spread only 106.8 %, against a pool that spreads up to 158 % across machines - so those five must have drawn near-identical hardware. It was originally mistaken for a cross-runner bound when it set the trend threshold; as a stand-in for single-machine jitter it is defensible, and an upper bound on it either way.

Reporting only

The comparison is reporting only. --fail-on-regression exists but is not passed, and stays off until a single-runner noise study has said what the residual floor actually is. Until then a gate would fire on noise, and a gate everyone learns to ignore is worse than no gate.

The trend chart is likewise not used to judge a pull request. It is cross-machine by construction, and the comment workflow deliberately does not compare against it - a constraint tests/test_benchmark_workflows.py enforces rather than leaves to convention.

Comparing two implementations of one stage

The suites above compare one implementation across commits or machines. Deciding between two candidate implementations of a single stage - a data structure, an accessor, a dispatch - is a different measurement, and three designs for it have produced wrong answers in this repository. All three flattered the newer candidate.

Do not microbenchmark the stage in isolation. Two candidates rarely divide the work at the same place, so a boundary drawn around “the lookup” charges one of them for something the other pays a moment later. Replacing the shortcut dict was measured this way and the dict came out ~100 ns ahead, which would have been ~10 % of a unique-zone query. It was an artefact: the shipped code answers dict.get with a match value: case int(zone_id) that the flat structure needs no equivalent of, and which costs 84 -> 188 ns. Measure the whole public call and let the boundary fall where it falls.

Alternate the order of a paired comparison. Running A then B inside each round lets A warm everything the two share - coordinate validation, the H3 call, the zone-name lookup, the branch predictors - and hands B the benefit for free. The same shortcut comparison read as 13.3 % faster in a fixed order and 0.3 % once the order alternated round by round.

Report two estimators and believe them only when they agree. The ratio of the two best rounds is the least noise-sensitive estimator; the count of rounds where the candidate won assumes nothing about the noise distribution. Where a difference is real they move together. Where they disagree - one saying +0.5 % and the other 26 of 61 rounds - there is no effect to find, and that disagreement is a more useful output than either number alone.

Sample the inputs at random. Iterating a dict’s keys in its own order walks its table front to back and hands it a cache-friendly access pattern no real query stream has. Worth 77 ns against 108 ns on the same lookup - a third of the quantity being compared.

Memory is measured the same way

Memory has its own harness (scripts/measure_memory.py) rather than a benchmarks/ suite: pytest-benchmark measures wall clock, and running tracemalloc across its calibration rounds would distort the very timings those suites exist to produce.

Each configuration is measured in a fresh subprocess. import timezonefinder costs ~95 MiB of NumPy and H3 before any timezone data is touched, so only a delta against a post-import baseline is meaningful - and a second finder built in the same process would inherit the first one’s warmed page cache and freed-but-unreturned arenas.

Two metrics per checkpoint, and the gap between them is the signal. *_heap is what tracemalloc accounts for (Python and NumPy allocations); *_rss is the resident set, which additionally counts memory-mapped pages. With in_memory=False the coordinate data is mapped rather than read, so its heap stays small while its RSS grows across the workload as lookups fault pages in - which is why there is both an init and a steady checkpoint. See TimezoneFinder Memory Footprint for the measured figures.

Only the heap metrics are charted. RSS residency is decided by machine-wide memory pressure, so tracking it would alert on the runner’s mood: repeated measurement puts the heap metrics at a 100.0 % spread against 102-111 % for RSS. The alert threshold for memory is correspondingly tight (110 %), because tracemalloc is near-deterministic and a change there is signal rather than jitter.

In the shared CI job, every memory step runs after every timing step. The harness reads the whole boundary dataset and warms the OS file cache, which is exactly what benchmarks/test_initialization.py disables pytest-benchmark’s warmup to avoid.

Names are join keys

Benchmark node ids are the join key of the timing trend chart, and memory metric names are the join key of the memory one. Renaming either does not move a metric’s history - it silently starts a new, empty one alongside the orphaned old chart.

Both sets are therefore pinned by tests (tests/test_benchmark_names.py, tests/test_memory_metric_names.py), so a rename is a deliberate act with a visible cost rather than an invisible reset. This is also why benchmarks must always pass explicit ids=/pytest.param(..., id=...) instead of relying on pytest’s autogenerated parametrize ids, which change when an unrelated parameter is added.

Comparing against another package

Everything above is about comparing this package to itself. benchmarks/test_comparison.py compares it to tzfpy, and the rules that make a cross-machine comparison meaningless apply with more force across packages, not less: the published figures for two libraries come from two machines, two acceleration paths and two query workloads, and the difference between them is not a difference between the libraries.

So the comparison is run the only way that answers anything. Both packages answer the same committed query fixtures, in the same process, on the same machine, and each is called through its own API inside its own loop - timezone_at(lng=, lat=) and get_tz(lng, lat) - so neither pays for an adapter frame the other does not. At per-query times of a few hundred nanoseconds, one extra Python call frame is worth tens of percent, which is the same order as the thing being measured. Comparison against tzfpy carries the result.

Three things this deliberately does not do:

It is not tracked. None of these benchmarks carry benchmark_core, so none reach the trend chart. The number is a ratio between two packages and one of them releases on a schedule this project does not control, so a step in that chart would as often mean “they shipped” as “we changed” - and nothing on the chart could say which. The measured version is stamped into the report instead.
It does not decide accuracy. The two packages disagree on a small fraction of points. Counting disagreements says nothing about which answer is right; that needs ground truth neither package carries. Alternatives states the design difference instead of scoring it.
It does not measure the other package’s memory. scripts/measure_memory.py constructs finders from this repository and has no configuration for anything else.

The startup half of that suite is worth its own note, because the obvious measurement is the wrong one. Timing construction would score tzfpy as free: it imports in about a millisecond and then deserialises its index inside the first query, where this package imports NumPy and H3 and reads its index when a finder is built. What a caller actually waits for is the first answer, so that is what is measured - a fresh python -c per round, with a bare-interpreter row alongside it so the interpreter’s own cost can be subtracted.

tzfpy lives in the compare dependency group, which only make benchmarks and tox -e benchmarks install. The CI-tracked environment is deliberately uv sync --group test - what a plain pip install timezonefinder gives you - and must not gain a package for the sake of a report.