Benchmarking Methodology

timezonefinder publishes performance numbers, appends them to a trend chart on every push to master and compares every pull request against its own baseline. This page describes how those measurements are taken and, more importantly, what they can and cannot tell you.

The short version: these numbers are noisy for reasons that have nothing to do with this package’s code, the measurement design exists to work around that, and every threshold is derived from measured noise rather than picked.

This page is the why. The operational side - which make target to run, what to check when a report looks wrong - lives in CONTRIBUTING.md.

The Workload

Each benchmark times one pass over a fixed batch of inputs rather than a single call, so every round performs identical work and the spread between rounds is measurement noise rather than a difference in what was measured. The batch size is BATCH_SIZE in benchmarks/conftest.py, currently 2,500 - large enough that a single round is well above timer resolution, and bounded above by the committed fixtures it draws from (_load_batch needs BATCH_SIZE points per fixture, pip_inputs_by_stratum needs BATCH_SIZE per stratum, the binding ceiling). Changing it invalidates the historical trend data, because a data point is only comparable to another data point that did the same amount of work.

The inputs themselves are deterministic committed fixtures (tests/fixtures/benchmarks/, generated by scripts/generate_benchmark_fixtures.py) rather than freshly drawn random data, so two runs of the same commit execute the exact same workload. They are pinned to the DATA_VERSION and FIXTURE_VERSION they were generated against, and the loader refuses a mismatch instead of silently benchmarking a workload the checkout no longer describes.

Two samplers, on purpose

Benchmark query points are drawn uniformly per unit of surface area (get_rnd_query_pt_area_weighted). The rest of the test suite uses get_rnd_query_pt, which is uniform in latitude and therefore oversamples the poles by roughly 2.5x.

That is not an inconsistency to fix. Correctness and fuzz tests want the polar bias - more edge cases per draw. A benchmark must instead represent real query load, and a pole-biased sampler inflates the share of ambiguous-shortcut queries, which are by far the most expensive class (see below). Using the wrong sampler would make the headline number describe a workload nobody has.

The point-in-polygon fixture is stratified by polygon vertex count for the same reason: the cost of the largest polygons would otherwise disappear behind an unweighted average.

ubuntu-latest does not pin the CPU

This is the single most important thing to know about the CI numbers.

runs-on: ubuntu-latest guarantees a runner image, not hardware. This project’s runs have landed on AMD EPYC 9V74, AMD EPYC 7763 and Intel Xeon Platinum 8573C parts between 2.30 and 3.69 GHz, and the clock varies run to run even within one model. Measured across eleven recorded runs whose lookup path was unchanged, the tracked min spread 134-158 % - larger than most changes worth reviewing.

The consequence is not subtle. A merged change that was a genuine 1.5x improvement once appeared on the trend chart as a 21 % regression, purely because of which machine each run drew. Any methodology that compares two arbitrary CI runs to each other is measuring the runner pool.

Consequences for the measurement design

Same-runner, merge-base comparison

A pull request is measured against its own merge base, in the same job, on the same runner - never against a stored master baseline. The measuring job checks out the merge base alongside the head, installs and measures both, and renders the base/head ratio table (scripts/compare_benchmark_runs.py).

That table verifies rather than assumes that both sides ran on one machine, and warns if the batch size, fixture set, boundary data or acceleration path differ between them - a comparison across any of those is meaningless and should say so rather than print a plausible-looking ratio.

The job holds no write permissions and no secrets, so it behaves identically for branch PRs and fork PRs, and a fork PR never fails for want of a token. The comparison comment is posted by a separate, privileged workflow triggered via workflow_run.

The base is measured twice

The base is measured once before and once after the head, sandwiching it, and the two passes are reduced by min. A runner that drifts over the job’s lifetime then shows up in the base’s own spread instead of looking like a code change. This roughly doubles the measuring job (~2-3 min), almost all of it the second checkout, uv sync and C extension build - the measurement itself is seconds.

Every run names its machine

scripts/describe_benchmark_machine.py prints the CPU, the acceleration path and the workload provenance to the job summary, and scripts/normalize_benchmark_json.py stamps the same label into the one field that survives into the trend chart. Hovering a data point therefore attributes it to a CPU long after the artifact has expired - which is the first thing to check when the chart shows a step change.

The tracked estimator: min, not mean

Since every round performs an identical fixed batch of work, the fastest round is the one least perturbed by whatever else the shared, virtualised runner happened to be doing. The tracked value is therefore pytest-benchmark’s min.

Getting that past the tooling takes a deliberate step: benchmark-action/github-action-benchmark’s pytest extractor reads only stats.ops (= 1 / stats.mean). scripts/normalize_benchmark_json.py rewrites ops/mean from the min before handing the report over, so the chart tracks the estimator this project chose rather than the one the extractor defaults to.

What CI measures

The core subset

Only three benchmarks (-m benchmark_core), all in_memory. The full suite is for the docs, on demand; it is not run per PR.

test_timezone_at[random-in_memory] is the headline. Uniformly random points are the only globally representative workload: they contain unique- and ambiguous-shortcut queries in their real ratio (~25 % ambiguous), so a change is weighted by how much real query load it actually helps.

unique_shortcut-in_memory and ambiguous_shortcut-in_memory are tracked alongside it as diagnostics, because the headline alone cannot attribute a change to a code path. On the tracked configuration an ambiguous lookup costs ~14x a unique one (~7x with Numba), so ambiguous work takes ~83 % of the wall clock despite being ~25 % of the queries. A win confined to the unique path therefore moves the headline by only ~0.17x its true size - invisible against the noise floor. The per-class benchmarks show it undiluted.

The tracked configuration

Only the no-Numba / clang C extension path, because that is what a plain pip install timezonefinder gives you and what constrained containers actually run.

timezonefinder/utils.py selects the point-in-polygon implementation at import time, so Numba and clang are completely different code paths whose numbers must never share a benchmark name. The workflow asserts the active path (scripts/assert_acceleration_path.py) rather than assuming it: a Numba install sneaking into the environment would otherwise silently corrupt the entire trend history rather than fail.

For the same reason, local numbers are not comparable to CI numbers - different CPU, different memory bandwidth, different background load, and a deliberately different acceleration path. Compare local-to-local and CI-to-CI only.

Thresholds derived from measured noise

The trend chart alert: 180 %

ALERT_THRESHOLD is derived from a measurement, not chosen. Across eleven recorded runs whose lookup path was identical, the tracked min spread 134-158 % (unique 134.3 %, random 145.9 %, ambiguous 158.4 %) purely because of the hardware each run drew. Worst spread plus 20 % headroom rounds to the shipped 180 %.

Being honest about what that buys: at 180 % the chart catches only a catastrophic regression and is blind to the 10-30 % changes actually worth reviewing. That is not a gap to close by tightening the number - a cross-machine chart cannot resolve better than the machines it spans. The same-runner pull request comparison is the real gate; the trend alert is a deliberately weak backstop for master, which nothing else watches. Alerts are non-blocking and stay that way: master must never be blocked on which machine a run drew.

It is re-derived whenever the runner pool or the core set changes, by repeating the measurement on unchanged code (scripts/benchmark_noise.py). Note what that job characterises: each repetition runs on a different machine, so it measures the runner pool’s spread, not a single runner’s jitter.

The pull request flag: 110 %

Same-runner measurement removes the machine-to-machine term but not the runner’s own jitter, so a few percent either way is still noise. Rows in the comparison table are flagged at REGRESSION_THRESHOLD_PCT (110 %).

That number comes from the closest thing there is to a same-runner measurement: a five-run study that spread only 106.8 %, against a pool that spreads up to 158 % across machines - so those five must have drawn near-identical hardware. It was originally mistaken for a cross-runner bound when it set the trend threshold; as a stand-in for single-machine jitter it is defensible, and an upper bound on it either way.

Reporting only

The comparison is reporting only. --fail-on-regression exists but is not passed, and stays off until a single-runner noise study has said what the residual floor actually is. Until then a gate would fire on noise, and a gate everyone learns to ignore is worse than no gate.

The trend chart is likewise not used to judge a pull request. It is cross-machine by construction, and the comment workflow deliberately does not compare against it - a constraint tests/test_benchmark_workflows.py enforces rather than leaves to convention.

Memory is measured the same way

Memory has its own harness (scripts/measure_memory.py) rather than a benchmarks/ suite: pytest-benchmark measures wall clock, and running tracemalloc across its calibration rounds would distort the very timings those suites exist to produce.

Each configuration is measured in a fresh subprocess. import timezonefinder costs ~95 MiB of NumPy and H3 before any timezone data is touched, so only a delta against a post-import baseline is meaningful - and a second finder built in the same process would inherit the first one’s warmed page cache and freed-but-unreturned arenas.

Two metrics per checkpoint, and the gap between them is the signal. *_heap is what tracemalloc accounts for (Python and NumPy allocations); *_rss is the resident set, which additionally counts memory-mapped pages. With in_memory=False the coordinate data is mapped rather than read, so its heap stays small while its RSS grows across the workload as lookups fault pages in - which is why there is both an init and a steady checkpoint. See TimezoneFinder Memory Footprint for the measured figures.

Only the heap metrics are charted. RSS residency is decided by machine-wide memory pressure, so tracking it would alert on the runner’s mood: repeated measurement puts the heap metrics at a 100.0 % spread against 102-111 % for RSS. The alert threshold for memory is correspondingly tight (110 %), because tracemalloc is near-deterministic and a change there is signal rather than jitter.

In the shared CI job, every memory step runs after every timing step. The harness reads the whole boundary dataset and warms the OS file cache, which is exactly what benchmarks/test_initialization.py disables pytest-benchmark’s warmup to avoid.

Names are join keys

Benchmark node ids are the join key of the timing trend chart, and memory metric names are the join key of the memory one. Renaming either does not move a metric’s history - it silently starts a new, empty one alongside the orphaned old chart.

Both sets are therefore pinned by tests (tests/test_benchmark_names.py, tests/test_memory_metric_names.py), so a rename is a deliberate act with a visible cost rather than an invisible reset. This is also why benchmarks must always pass explicit ids=/pytest.param(..., id=...) instead of relying on pytest’s autogenerated parametrize ids, which change when an unrelated parameter is added.