Build-time test discovery

Build-time test discovery is an experimental, opt-in mode that discovers your Dart tests when you build the app — instead of at runtime, when the app first launches on the device — and generates a real, statically-declared native test method for each Dart test.

The payoff: every Dart test becomes an individually-addressable native test, which unlocks per-test sharding, running a single test without rebuilding, cleaner test reports, and compatibility with external test-distribution tools.

How it works

Normally, Patrol discovers your tests at runtime: the app is launched once, and the native runner asks the Dart side for the list of tests, then registers them dynamically (class_addMethod on iOS, a parameterized JUnit runner on Android). Nothing about your tests exists in the compiled test binary until the app runs.

With build-time discovery, patrol build / patrol test:

1. Runs a host flutter test in a special discovery mode (--dart-define PATROL_TEST_DISCOVERY=true) that walks your test tree without executing the bodies, and serializes it to build/patrol/patrol_test_manifest.json.
2. Generates static native test classes from that manifest, one class per Dart test file:
   • iOS: ios/RunnerUITests/PatrolGeneratedTests.inc - one PatrolGeneratedTests_<file> : RunnerUITests subclass per test file, each with one - (void)test_<name> method per Dart test.
   • Android: one PatrolGeneratedTests_<file>.java per test file in your androidTest source set, each with one @Test public void test_<name>() per Dart test, plus a PatrolGeneratedTests marker class holding no tests.
3. Compiles those classes into the test binary, so the native test framework (XCTest / JUnit) discovers them natively.

Class and method names are sanitized (Dart names contain spaces and punctuation), so they are not the Dart names verbatim, and they are identical on both platforms. The file lives in the class name and the method carries only the test, which gives reports a real hierarchy (file, then test) and makes a whole file selectable with a single native selector. What stays byte-identical is the Dart test name embedded in each method body and handed to Patrol, so the tests run exactly the same as under runtime discovery. Use the generated class and method names when building native selectors, and the Dart names (or a test file path) when selecting tests through Patrol (e.g. --only).

Setup

patrol:
  test_directory: patrol_test

patrol:
  emit_test_manifest: true

@import XCTest;
@import patrol;
@import ObjectiveC.runtime;

PATROL_INTEGRATION_TEST_IOS_RUNNER_STATIC_BASE(RunnerUITests)
#include "PatrolGeneratedTests.inc"

patrol build ios --emit-test-manifest    # or: patrol build android

✓ Discovered Dart tests → build/patrol/patrol_test_manifest.json (9.0s)
Discovered 3 Dart test(s) (1 skipped) in 1 file(s):

  patrol_test/example_test.dart  (3 test(s), 1 skipped)
    • example_test counter state is the same after going to Home and switching apps
    • example_test short test with two tags  [skip]
    • example_test short test with tag

Generated 3 static XCTest method(s) → ios/RunnerUITests/PatrolGeneratedTests.inc

Platform-dependent tests

Your tests are registered twice now: on your machine, to build the manifest, and on the device. So anything that decides whether a test exists or is skipped has to give the same answer in both runs — and defaultTargetPlatform doesn't. Under flutter test Flutter reports android no matter what you're building for, and dart:io's Platform describes your computer. Use patrolTargetPlatform:

patrolTest(
  'shares the invoice',
  ($) async { ... },
  skip: patrolTargetPlatform == PatrolTargetPlatform.iOS,
);

Patrol tells it which platform the build targets, so discovery and the device agree. Inside a test body nothing changes — there you're on the device, and any platform check works.

Running tests without rebuilding

Once you've built with build-time discovery, you can run the already-built tests without rebuilding — reusing the artifacts from the previous patrol build — with patrol test-without-building:

# Run all discovered tests, no rebuild.
patrol test-without-building

# Run a single test by its exact Dart name (as printed during discovery).
patrol test-without-building --only "example_test tap counter increments"

# Run every test from one file, as a single native selector.
patrol test-without-building --only patrol_test/example_test.dart

--only takes either form and is repeatable, so files and single tests can be mixed. A file wins over its own tests, so passing both runs it once.

It's a separate command rather than a flag on patrol test because nothing is bundled, built or reinstalled here — so the options that shape a build (--target, --tags, --dart-define, …) don't apply and aren't accepted. It needs a prior patrol build with emit_test_manifest, since selecting individual tests natively is only possible thanks to the generated static tests. Under the hood it runs xcodebuild test-without-building -only-testing … on iOS and adb shell am instrument -e class <fqcn>#<method> … on Android — no compilation step. This is ideal for a fast edit-nothing / re-run-one-test loop and for debugging a single failing test.

Sharding & external tools

Because each Dart test is now a real, statically-declared native test, it can be targeted individually:

• iOS: -only-testing RunnerUITests/PatrolGeneratedTests_<file>/test_<name> for one test, or -only-testing RunnerUITests/PatrolGeneratedTests_<file> for a whole file.
• Android: -e class <fqcn>#<method> for one test, -e class <fqcn> for a whole file (or a balanced, per-test split).

This is what enables per-test sharding across machines (e.g. SauceLabs testListFile, or any tool that enumerates the compiled test bundle), balanced Android sharding, and clean per-test names in reports.

iOS sharding on device farms that need a test list

XCUITest has no built-in sharding. Device farms shard it by taking an explicit list of test identifiers (Bundle/Class/method) and splitting it across parallel runners — for example SauceLabs' testListFile (with shard: concurrency) or BrowserStack's XCUITest sharding.

With stock Patrol this is impossible: tests are registered at runtime (class_addMethod) only after the app launches, so no static list of test identifiers exists before the run — there's nothing to hand to the farm.

Build-time discovery fixes exactly this. Because every Dart test is compiled into a real RunnerUITests/PatrolGeneratedTests_<file>/test_<name> method, you can extract the full list of selectors straight from the built test bundle (or from the generated PatrolGeneratedTests.inc) and feed it to the farm's test-list sharding. Each shard then runs its slice with -only-testing, in parallel, on a separate machine — which also sidesteps the fixed-port limitation, since each shard gets its own host.

Deriving the list from the generated .inc keeps it byte-identical to what XCTest discovers. For example, to produce a SauceLabs testListFile:

awk '/^@implementation /{cls=$2} /^- \(void\)test_/{m=$2; sub(/^\(void\)/,"",m); \
  print "RunnerUITests/" cls "/" m}' ios/RunnerUITests/PatrolGeneratedTests.inc \
  > .sauce/ios_testlist.txt

Then point SauceLabs at it with shard: concurrency + testListFile. See dev/e2e_app/scripts/generate_ios_testlist.sh for a ready-made version (SAUCE_DEVICE=real|simulator).

Trade-offs

Benefits

• Re-run without rebuilding. patrol test-without-building re-runs the whole suite against the already-built artifacts, skipping compilation entirely.
• Run a single test without rebuilding. patrol test-without-building --only "<dart name>" runs just one test from the built set — a fast loop for debugging a single failing test.
• Per-test selection → sharding across machines and device farms.
• Per-file selection: one native selector runs a whole test file, and reports group by file on both platforms under the same class and method names.
• Cleaner reports: each test appears under its own name (Android no longer groups everything under a parameterized runDartTest[...] wrapper).
• Tests show up in the Xcode test navigator and in test plans.

Limitations & caveats

• Host-vs-device discovery. Discovery runs on the host (flutter test), not on the device. Tests registered conditionally on the runtime environment — a description built from a --dart-define, say — may be discovered differently than they would on-device, and such a test can end up missing from the manifest. For the platform, use patrolTargetPlatform; for everything else, keep your test registration environment-independent.
• Test location. Discovery needs your tests to run as plain widget tests on the host. Tests kept under integration_test/ are treated by Flutter as integration tests (they require a device and a direct integration_test dependency), so discovery won't run for them. Keep your Patrol tests under patrol_test/ (the default) or another non-integration_test/ directory.
• Parallel simulators on one host. Supported on iOS: the native server picks a free port pair per run and the app reads it at launch, so shards on one machine don't collide. See the Marathon integration guide. On Android, shard across separate hosts/VMs (one emulator each).
• patrol develop regenerates for its own target. A develop session bundles only the single --target test file, so the tests it generates (and the manifest) cover just that file. Run patrol build again before patrol test-without-building.
• Slower initial build. Discovery adds a host flutter test pass to every patrol build, so the first build takes a bit longer. This cost is paid once — subsequent patrol test-without-building runs skip building altogether.
• test-without-building reuses the previous build. It runs exactly what was last built, so re-run it only when the app and test code are unchanged (change Dart or native code → rebuild). On Android it targets the instrumentation resolved from the device (pm list instrumentation); a custom testApplicationId or runner is picked up when installed, but an exotic setup that can't be resolved falls back to the default <applicationId>.test / PatrolJUnitRunner, and a flavor applicationIdSuffix that isn't reflected in patrol.android.package_name may not match. For those setups, prefer a normal patrol test.
• Experimental: the setup and flags may change.