Benchmarking

Prepare the Environment

Assuming you have forked the repository and cloned it locally.

Use cd JAnim to enter the project folder, then install the environment with dependencies required for benchmarking:

pip install -e .[bench]

Run Benchmarks

The project provides a convenient script entry point to run benchmarks. The command format is:

python scripts bench [options]

Examples of available options:

  • --untested_tags: test all untested tags since v2.1.0.

  • --tags <tag> [<tag> ...]: run tests for one or more tags, for example --tags v2.1.0 v2.2.0.

  • --hashes <hash> [<hash> ...]: run tests by commit hash (multiple values allowed).

  • -o / --open: automatically open the preview in the browser.

Usage Examples

  1. Test all untested tags (after v2.1.0):

python scripts bench --untested_tags

  1. Test specified tags:

python scripts bench --tags v2.1.0 v2.2.0

  1. Test with specified hashes and automatically open the preview:

python scripts bench --hashes 78b7c20 2c1fcfd -o

  • You can also directly use asv preview -b to preview published results locally.

Script Behavior

  • When --untested_tags is used, the script queries hashes for tags since v2.1.0 in the repository (the internal implementation is in scripts/bench/hashes.py), compares them against already tested hashes, and runs benchmarks only for untested commits.

  • If --tags or --hashes are also provided, they will be added to the hash list to be tested as well.

  • Before execution starts, the script writes the hash list to be tested into .asv/_tag_hashes.txt, runs benchmark cases via asv run HASHFILE:.asv/_tag_hashes.txt, then publishes results locally with asv publish, and finally (optionally) opens a browser preview with asv preview -b.