.. _benchmarking_methodology: ========================= Benchmarking Methodology ========================= ``timezonefinder`` publishes performance numbers, appends them to a `trend chart `__ on every push to ``master`` and compares every pull request against its own baseline. This page describes how those measurements are taken and, more importantly, what they can and cannot tell you. The short version: these numbers are noisy for reasons that have nothing to do with this package's code, the measurement design exists to work around that, and every threshold is derived from measured noise rather than picked. This page is the *why*. The operational side - which ``make`` target to run, what to check when a report looks wrong - lives in `CONTRIBUTING.md `__. The Workload ------------ Each benchmark times **one pass over a fixed batch** of inputs rather than a single call, so every round performs identical work and the spread between rounds is measurement noise rather than a difference in what was measured. The batch size is ``BATCH_SIZE`` in ``benchmarks/conftest.py``, currently 2,500 - large enough that a single round is well above timer resolution, and bounded above by the committed fixtures it draws from (``_load_batch`` needs ``BATCH_SIZE`` points per fixture, ``pip_inputs_by_stratum`` needs ``BATCH_SIZE`` *per stratum*, the binding ceiling). Changing it invalidates the historical trend data, because a data point is only comparable to another data point that did the same amount of work. The inputs themselves are **deterministic committed fixtures** (``tests/fixtures/benchmarks/``, generated by ``scripts/generate_benchmark_fixtures.py``) rather than freshly drawn random data, so two runs of the same commit execute the exact same workload. They are pinned to the ``DATA_VERSION`` and ``FIXTURE_VERSION`` they were generated against, and the loader refuses a mismatch instead of silently benchmarking a workload the checkout no longer describes. Two samplers, on purpose ~~~~~~~~~~~~~~~~~~~~~~~~ Benchmark query points are drawn **uniformly per unit of surface area** (``get_rnd_query_pt_area_weighted``). The rest of the test suite uses ``get_rnd_query_pt``, which is uniform in *latitude* and therefore oversamples the poles by roughly 2.5x. That is not an inconsistency to fix. Correctness and fuzz tests *want* the polar bias - more edge cases per draw. A benchmark must instead represent real query load, and a pole-biased sampler inflates the share of ambiguous-shortcut queries, which are by far the most expensive class (see below). Using the wrong sampler would make the headline number describe a workload nobody has. The point-in-polygon fixture is stratified by polygon vertex count for the same reason: the cost of the largest polygons would otherwise disappear behind an unweighted average. ``ubuntu-latest`` does not pin the CPU -------------------------------------- This is the single most important thing to know about the CI numbers. ``runs-on: ubuntu-latest`` guarantees a runner *image*, not hardware. This project's runs have landed on AMD EPYC 9V74, AMD EPYC 7763 and Intel Xeon Platinum 8573C parts between 2.30 and 3.69 GHz, and the clock varies run to run even within one model. Measured across eleven recorded runs whose lookup path was **unchanged**, the tracked ``min`` spread 134-158 % - larger than most changes worth reviewing. The consequence is not subtle. A merged change that was a genuine 1.5x improvement once appeared on the trend chart as a 21 % regression, purely because of which machine each run drew. Any methodology that compares two arbitrary CI runs to each other is measuring the runner pool. Consequences for the measurement design --------------------------------------- Same-runner, merge-base comparison ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A pull request is measured **against its own merge base, in the same job, on the same runner** - never against a stored ``master`` baseline. The measuring job checks out the merge base alongside the head, installs and measures both, and renders the base/head ratio table (``scripts/compare_benchmark_runs.py``). That table *verifies* rather than assumes that both sides ran on one machine, and warns if the batch size, fixture set, boundary data or acceleration path differ between them - a comparison across any of those is meaningless and should say so rather than print a plausible-looking ratio. The job holds no write permissions and no secrets, so it behaves identically for branch PRs and fork PRs, and a fork PR never fails for want of a token. The comparison comment is posted by a separate, privileged workflow triggered via ``workflow_run``. The base is measured twice ~~~~~~~~~~~~~~~~~~~~~~~~~~ The base is measured once before and once after the head, sandwiching it, and the two passes are reduced by ``min``. A runner that drifts over the job's lifetime then shows up in the base's *own* spread instead of looking like a code change. This roughly doubles the measuring job (~2-3 min), almost all of it the second checkout, ``uv sync`` and C extension build - the measurement itself is seconds. Every run names its machine ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``scripts/describe_benchmark_machine.py`` prints the CPU, the acceleration path and the workload provenance to the job summary, and ``scripts/normalize_benchmark_json.py`` stamps the same label into the one field that survives into the trend chart. Hovering a data point therefore attributes it to a CPU long after the artifact has expired - which is the first thing to check when the chart shows a step change. The tracked estimator: ``min``, not ``mean`` -------------------------------------------- Since every round performs an identical fixed batch of work, the fastest round is the one least perturbed by whatever else the shared, virtualised runner happened to be doing. The tracked value is therefore pytest-benchmark's ``min``. Getting that past the tooling takes a deliberate step: ``benchmark-action/github-action-benchmark``'s pytest extractor reads only ``stats.ops`` (``= 1 / stats.mean``). ``scripts/normalize_benchmark_json.py`` rewrites ``ops``/``mean`` from the min before handing the report over, so the chart tracks the estimator this project chose rather than the one the extractor defaults to. What CI measures ---------------- The core subset ~~~~~~~~~~~~~~~ Only three benchmarks (``-m benchmark_core``), all ``in_memory``. The full suite is for the docs, on demand; it is not run per PR. ``test_timezone_at[random-in_memory]`` is the headline. Uniformly random points are the only globally representative workload: they contain unique- and ambiguous-shortcut queries in their real ratio (~25 % ambiguous), so a change is weighted by how much real query load it actually helps. ``unique_shortcut-in_memory`` and ``ambiguous_shortcut-in_memory`` are tracked alongside it as diagnostics, because the headline alone cannot attribute a change to a code path. On the tracked configuration an ambiguous lookup costs ~14x a unique one (~7x with Numba), so ambiguous work takes ~83 % of the wall clock despite being ~25 % of the queries. A win confined to the unique path therefore moves the headline by only ~0.17x its true size - invisible against the noise floor. The per-class benchmarks show it undiluted. The tracked configuration ~~~~~~~~~~~~~~~~~~~~~~~~~ Only the **no-Numba / clang C extension** path, because that is what a plain ``pip install timezonefinder`` gives you and what constrained containers actually run. ``timezonefinder/utils.py`` selects the point-in-polygon implementation **at import time**, so Numba and clang are completely different code paths whose numbers must never share a benchmark name. The workflow *asserts* the active path (``scripts/assert_acceleration_path.py``) rather than assuming it: a Numba install sneaking into the environment would otherwise silently corrupt the entire trend history rather than fail. For the same reason, **local numbers are not comparable to CI numbers** - different CPU, different memory bandwidth, different background load, and a deliberately different acceleration path. Compare local-to-local and CI-to-CI only. Thresholds derived from measured noise -------------------------------------- The trend chart alert: 180 % ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``ALERT_THRESHOLD`` is derived from a measurement, not chosen. Across eleven recorded runs whose lookup path was identical, the tracked ``min`` spread 134-158 % (unique 134.3 %, random 145.9 %, ambiguous 158.4 %) purely because of the hardware each run drew. Worst spread plus 20 % headroom rounds to the shipped **180 %**. Being honest about what that buys: at 180 % the chart catches only a catastrophic regression and is blind to the 10-30 % changes actually worth reviewing. That is not a gap to close by tightening the number - a cross-machine chart cannot resolve better than the machines it spans. The same-runner pull request comparison is the real gate; the trend alert is a deliberately weak backstop for ``master``, which nothing else watches. Alerts are non-blocking and stay that way: ``master`` must never be blocked on which machine a run drew. It is re-derived whenever the runner pool or the core set changes, by repeating the measurement on unchanged code (``scripts/benchmark_noise.py``). Note what that job characterises: each repetition runs on a *different* machine, so it measures the runner pool's spread, not a single runner's jitter. The pull request flag: 110 % ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Same-runner measurement removes the machine-to-machine term but not the runner's own jitter, so a few percent either way is still noise. Rows in the comparison table are flagged at ``REGRESSION_THRESHOLD_PCT`` (110 %). That number comes from the closest thing there is to a same-runner measurement: a five-run study that spread only 106.8 %, against a pool that spreads up to 158 % across machines - so those five must have drawn near-identical hardware. It was originally mistaken for a cross-runner bound when it set the trend threshold; as a stand-in for single-machine jitter it is defensible, and an upper bound on it either way. Reporting only ~~~~~~~~~~~~~~ The comparison is **reporting only**. ``--fail-on-regression`` exists but is not passed, and stays off until a single-runner noise study has said what the residual floor actually is. Until then a gate would fire on noise, and a gate everyone learns to ignore is worse than no gate. The trend chart is likewise **not** used to judge a pull request. It is cross-machine by construction, and the comment workflow deliberately does not compare against it - a constraint ``tests/test_benchmark_workflows.py`` enforces rather than leaves to convention. Memory is measured the same way ------------------------------- Memory has its own harness (``scripts/measure_memory.py``) rather than a ``benchmarks/`` suite: pytest-benchmark measures wall clock, and running ``tracemalloc`` across its calibration rounds would distort the very timings those suites exist to produce. Each configuration is measured in a **fresh subprocess**. ``import timezonefinder`` costs ~95 MiB of NumPy and H3 before any timezone data is touched, so only a delta against a post-import baseline is meaningful - and a second finder built in the same process would inherit the first one's warmed page cache and freed-but-unreturned arenas. Two metrics per checkpoint, and the gap between them is the signal. ``*_heap`` is what ``tracemalloc`` accounts for (Python and NumPy allocations); ``*_rss`` is the resident set, which additionally counts memory-mapped pages. With ``in_memory=False`` the coordinate data is mapped rather than read, so its heap stays small while its RSS grows across the workload as lookups fault pages in - which is why there is both an ``init`` and a ``steady`` checkpoint. See :doc:`benchmark_results_memory` for the measured figures. **Only the heap metrics are charted.** RSS residency is decided by machine-wide memory pressure, so tracking it would alert on the runner's mood: repeated measurement puts the heap metrics at a 100.0 % spread against 102-111 % for RSS. The alert threshold for memory is correspondingly tight (110 %), because ``tracemalloc`` is near-deterministic and a change there is signal rather than jitter. In the shared CI job, **every memory step runs after every timing step**. The harness reads the whole boundary dataset and warms the OS file cache, which is exactly what ``benchmarks/test_initialization.py`` disables pytest-benchmark's warmup to avoid. Names are join keys ------------------- Benchmark node ids are the join key of the timing trend chart, and memory metric names are the join key of the memory one. Renaming either does not move a metric's history - it silently starts a new, empty one alongside the orphaned old chart. Both sets are therefore pinned by tests (``tests/test_benchmark_names.py``, ``tests/test_memory_metric_names.py``), so a rename is a deliberate act with a visible cost rather than an invisible reset. This is also why benchmarks must always pass explicit ``ids=``/``pytest.param(..., id=...)`` instead of relying on pytest's autogenerated parametrize ids, which change when an unrelated parameter is added.