CI - Platforms
In this document, we'll outline a few ways to run Patrol UI tests of Flutter apps.
Generally, the solutions for running UI tests of mobile apps can be divided into 2 groups:
-
Device labs - platforms that provide access to mobile devices in the cloud. You upload an app binary with tests to the device lab, which runs the tests and reports the results back to you.
-
Traditional – containers or VMs, either managed or self-hosted. In this approach, you get access to the shell, so everything is possible. You manually script what you want to achieve, which is usually: installing the Android/iOS SDK tools, creating a virtual device, running tests, and collecting results.
There are quite a few solutions in each of these groups, and each is unique, but generally, device labs trade flexibility for ease of use. They're a good fit for most apps but make certain more complicated scenarios impossible.
Device labs
Firebase Test Lab
Firebase Test Lab is one of the most popular device labs. It is a good choice for most projects.
You upload the app main app, the test app, select devices to run on, and after a while, test results along with a video recording are available.
Firebase Test Lab has a large pool of physical and virtual devices.
See also:
emulator.wtf
emulator.wtf is a fairly new solution created by Madis Pink and Tauno Talimaa. It claims to provide a 2-5x speedup compared to Firebase Test Lab, and 4-20x speedup compared to spawning emulators on CI machines. It works similarly to Firebase Test Lab - you upload your main apk, test apk, select emulators to run on, and the rest is up to emulator.wtf - it runs the tests and outputs results.
The emulators are indeed rock stable. Emulator.wtf automatically records videos from test runs, and it presents the test results nicely.
It's a solid choice if you can accept that your tests will run only on Android emulator.
Reports are available in JUnit.
See also:
Xcode Cloud
Xcode Cloud is a CI/CD platform built into Xcode and designed expressly for Apple developers. It doesn't support testing on Android.
Since integration tests written with Patrol are also native XCTests, it should
be possible to run Patrol on Xcode Cloud. We plan to research it soon and share
our findings here.
Other
Another popular device lab is AWS Device Farm.
If your use-case is highly specific, you might want to build an in-house device farm. A project that helps with this is Simple Test Farm.
Limitations
We mentioned above that device labs make certain scenarios impossible to accomplish.
An example of such a scenario scanning a QR code. One of the apps we worked on had this feature, and we wanted to test it because it was a critical part of the user flow. When you have access to the shell filesystem (which you do have in the "manual" approach, and don't have in the "device lab" approach), you can easily replace the scene that is visible in the camera's viewfinder.
This is not possible on device labs.
Traditional
GitHub Actions
GitHub Actions is a very popular CI/CD platform, especially among open-source projects thanks to unlimited minutes.
Unfortunately, running Flutter integration tests on GitHub Actions is not a pleasant experience.
Android
We used the ReactiveCircus/android-emulator-runner GitHub Action to run Android emulator on GitHub Actions. Our takeaway is this: Running an Android emulator on the default GitHub Actions runner is a bad idea. It is slow to start and unstable (apps crash randomly) and very slow. Really, really slow. We tried to mitigate its instability by using Test Butler, but it comes with its own restrictions, most notably, it doesn't allow for Google Play Services.
iOS
We use the futureware-tech/simulator-action GitHub Action to run iOS simulator on GitHub Actions is stable. But given that the iOS simulator is just that – a simulator, not an emulator – the range of cases it can be used for is reduced. For example, there's no easy way to disable an internet connection, which makes it very hard to test the behavior of an app when offline.
Bear in mind that to run an iOS simulator on GitHub Actions, you have to use a macOS runner. 1 minute on macos-latest counts as 10 minutes on ubuntu-latest. You can also use a custom runner – more on that below.
Custom Runners Workflows on GitHub Actions can run on external runners, in addition to default runners such as ubuntu-latest and macos-latest.
One example of such a custom runner provider is BuildJet. We tried running Android emulator on it, hoping that the performance benefits it brings would help with the abysmal stability, but we've found that, even though the emulator works faster and is more stable, it sometimes just crashes with no actionable error message.
Other
There are many more CI/CD platforms. Some of the most popular include CircleCI, CirrusCI, and GitLab CI/CD. There are also CI providers that are focused specifically on mobile apps, for example Bitrise and Codemagic. If you used these platforms, we (and other Patrol users) will be happy to hear about your experiences!