Tips and Tricks for Building Test Automation and Continuous Integration on Mobile
Intro While great minds are arguing with each other about whether it is better to disable flaky tests or to retry them, I’d like to show you the way how you can achieve the latter. This is for the brave, so do it at your own risk! 🤠 Implementing retry strategy Let’s create two classes: Retry - this one will be able to retry any specific test RetryRule - this one will be responsible for the whole…
Intro For a long time, setting up test coverage in Android projects has been a quite complex and frustrating process. Developers often found themselves wading through a rabbit hole of conflicting information, scrolling the docs, surfing the net, scouring the forums and blog posts in search of elusive solutions. What worked seamlessly for one dev might fail spectacularly for another, leading to…
Intro Today I’d like to talk about test orchestration on Android. And about parsing test names in particular, as one of the main pitfalls along the way. Imagine you have hundreds or thousands of tests that take more than an hour in a raw. You better start thinking how to speed them up or run them in parallel. There are multiple options to parallel the tests: Parallelization at the CI level: when…
Intro Today I’d like to show you how to do a proper size analysis of an App or an SDK on iOS. I’ll try to be as short as possible, so fasten the seat belts 🤠 Building the app Let’s create a new empty app and integrate some third-party SDK via SPM . For example StreamChatSwiftUI . 1. Open Xcode 2. Create New Project... -> iOS App -> Product name: `Size` ; Interface: `SwiftUI` -> Next -> Create 3.…
Intro The proof of concept we’ll discuss today goes a bit beyond the boundaries of what XCTest Performance offers out of the box. And that’s only because the standard tooling has limitations, which we’ll try to detail shortly. Although if you have a self-hosted runner and a farm of physical devices that you gonna use for Performance testing then all good and you’re covered, just follow the…
Introduction There came a day when we needed to run Unit or E2E tests on older iOS versions on CI. Sounds like a piece of cake. What could go wrong? Let’s see what options we have. xcode-install a.k.a xcversion ⛔️ Deprecated now. But back then it was a no-brainer. $ gem install xcode-install $ xcversion simulators --install = 'iOS 13.7' ############################### 82%…
Credentials, what can be easier, right? Just get them from .env file or whatever and you’re good to go. Well, not so fast… not on iOS . To be honest, it was a bit challenging to make it work from Xcode and from the Terminal at the same time. Passing credentials through the command-line only is not a problem at all, but as soon as you want to run these tests from Xcode it takes a ridiculous turn.…
Let’s say you have a code generation tool that adds and deletes files from the project’s sources. In addition to physically doing so, Xcode requires updating the project file to reflect these changes, i.e. adding and deleting file references. Here is where the tricky part starts because this project.pbxproj file is an endless old-style ASCII property list that is not meant to be edited manually.…
There is no secret that the UI tests are pretty limited in terms of accessing app internals. That’s why sometimes you have to craft some hackish workarounds to bypass these limitations. The app logs are one of those things that would be quite useful to have access to from UI tests to observe any events that happen in the app or whatnot. Even though there is nothing out of the box that could help…
Page Object Model ( POM ) is a design pattern, used in test automation that creates Object Repository for UI elements. Even though the word «Page» comes from the web world, the pattern applies to any UI, including mobile and desktop apps. Maybe think of it as a «Screen Object Model» . Sounds a bit more applicable to mobile, doesn’t it? Anyways, the main idea is to create an object for each screen…
Offline support is one of the most complex features not only to implement but also to test. In this post, I’ll show you how to disable the network on iOS Simulator from XCTest . Fasten your seat belts, it can open up a whole new world for your automated test cases. Options How can we test «Offline support»? There are a few options: To modify the source code so you can control the «network» from…
What’s this? The command line is quite a powerful tool. It can be used to automate a bunch of things here and there. Obviously, it’s not only about Shell scripts, you can run any other tool from it as well. But using it directly is not always possible. Recently, I showed how to gain access to the command line from XCTest . This time it’s Maestro’s turn, as it also works inside a sandbox (more…
What is accessibility value? It’s a localized string that contains the current value of an element. For example, the value of a slider might be 42% and the value of a text field is the text it contains. When to use it? Usually, it’s used only when an accessibility element can have a value that is not represented by its label. Why to use it? For example, a volume slider’s label might be «Volume»…
Do you like BDD ? To be honest, I don’t. I do like the idea behind it, but the implementation usually way too complicates things for no reason and makes it super painful to develop and support the automated testing frameworks. People usually say it’s there to allow non-technical folks to read and write the automated tests on their own. Really? Okay. If something breaks under the hood, these folks…
So, today, February 24th , Ruby turns 30 years old! It was first born in Japan back in 1993 , thanks to Yukihiro «Matz» Matsumoto , and it’s been swiftly taking over the world of ours ever since! It was designed with the idea that programming should be fun for programmers. It emphasizes the necessity for software to be understood by humans first and computers second. Even though right now there…
XCTest is an excellent tool for testing your apps on iOS. It’s fast, reliable, and easy to use. But the default test reports aren’t as pretty as they could be, especially for UI tests. In this post, I’ll show you how to make them look much better. Quick intro Usually, tests’ result representation has two «heads»: trace in logs — basically test names with ✅ or ❌ prefixes json/xml/xcresult reports —…
What’s this? The command line is quite a powerful tool. It can be used to automate a bunch of things here and there. Obviously, it’s not only about Shell scripts, you can run any other tool from it as well. But using it directly is not always possible. XCTest , for instance, works inside a sandbox, which means you literally can’t even stick your nose out of the box. Well, almost. Let’s bypass this…
Monkey testing There are three things you can watch forever: fire burning, water falling, and monkey testing 📱🐒 In my humble opinion, automated monkey testing is the most mesmerizing testing technique. No, really, random or pseudo-random clicks and swipes, and yet it’s so breathtaking, isn’t it? Also, it’s such a fun way to stress-test apps that you’re developing, in a random yet repeatable…
Many apps track users’ location. Some of them use it for navigation, some — for analytics. Is this for our good? Emm… But in order to autotest related features, we need to mock some things. Today, I’ll show you how to do it in XCTest . Precondition Install any local web server (e.g.: Sinatra ): gem install sinatra Configure the server First of all, we need to create an endpoint that will receive a…
It’s no secret that tests can fail with or without a reason, especially those flaky UI ones on CI. Would be great to know exactly why they failed, wouldn’t it? XCTest runs in the sandbox, which makes it hard to get anything out of it, but let’s hack it and record the failed tests! Precondition The easiest way to record a video is to run simctl command from the Terminal : xcrun simctl io booted…
Sometimes it can be useful to have the UDID of the iOS simulator handy for automation purposes. Let’s learn how to extract it within XCTest. Right off the boat 🤿 import XCTest class SampleTest : XCTestCase { func testSimulatorUDID () { let env = ProcessInfo . processInfo . environment let udid = env [ "SIMULATOR_UDID" ] ?? "" XCTAssertNotEqual ( udid , "" ) } } Open an Xcode project Create a test…
Xcode 11.4 introduced a handy feature that allowed us to test push notifications on Simulator ( xcrun simctl push ). Unfortunately, it’s still not possible to take advantage of it within the XCTest framework. So, let’s build our own bike! Tooling I will use the following tools: XCTest - a native framework to write Unit and UI tests on iOS Sinatra - an awesome DSL for quickly creating web…
Dealing with multiple Xcode versions locally as well as on CI has never been easier than it is today. Installing Xcode command line tools, creating symlinks, switching between different versions, setting up the default one - from now on it’s just a matter of a couple of mouse clicks. Less words, more code snippets and examples. GUI For me, using Xcodes is by far the easiest way to manage multiple…
As with everything, there are at least two ways to run tests in parallel on multiple iOS versions: Parallelization at the CI level Parallelization at the test framework level In our case, the CI will be GitHub Actions and the test framework will be Fastlane - the wrapper on top of XCTest . Just to be clear, we are going to use GitHub Actions as well as Fastlane in both scenarios, the only…
Introduction An automated testing can be a tricky thing, especially when you’re dealing with unstable or constantly changing backend. What if we could get rid of the real backend and use a mock server in our tests instead? Perhaps that would solve a couple of our problems, eh? Check out this repository to dive into the source code I’ll be playing with in this note. Precondition I’ll use Swifter…
Useful links https://docs.mitmproxy.org/stable/concepts-certificates https://docs.mitmproxy.org/stable/api/mitmproxy/flow.html Precondition Let’s put this code at the top of our mitmproxy script file: from mitmproxy import http import json import time import os server_folder = "mitmproxy_data" def parse_json ( content : bytes ): try : return json . loads ( content . decode ( "utf-8" )) except :…
Introduction There is a chance that one day you’ll be asked to enter a two-factor authentication ( 2FA ) code. There is even a possibility that you’ll be asked to enter a code every time you log in. Such a good area for automation, isn’t it? My notes are usually dancing around mobile automation, so let’s imagine we have an idea to bypass 2FA on Play Store as part of our automation script. Action…
Introduction Have you ever dealt with a long-long Fastfile or a bunch of fastlane actions or even any build scripts that hang out side-by-side with a source code of your project and turned to be completely unreadable? Then I have a modern architectural idea to make such things less complicated and much more readable, reliable and extendable. I suggest to craft your own private fastlane plugin that…
Introduction Have you ever come across a GitHub Actions workflow that stumbled into level ground failed without good reason? If so I believe you’re well familiar with this ruthless button: Yeah, the thing is that it restarts absolutely all the jobs ( green , red , whatever ), even if just the only one has failed. You can imagine how annoying this can be. While GitHub Actions doesn’t provide a way…
Introduction Today I’ll briefly explain how to provide the Tor anonymity guarantee to almost any scripts and CLI tools that you have in your pocket. The most complicated question would be: «For what?», and, to be honest, I can’t come up with a strong answer. Maybe you are a hacker, or you just enjoy being hidden and safe, or you simply need to test your app from Madagascar, who knows? Installation…
Introduction So, right off the bat, - leaks are one of the most dangerous types of vulnerabilities for companies, one of the yummiest things for hackers, and one of the most well-paid targets for bug hunters. This note is full of referrals on multiple great third-party researches, docs, articles, and whatnot, so sorry in advance. Preparation Would be great to first of all pick up as many target…
Definition What does impact analysis actually mean in terms of CI ? in brief : It’s about time-saving thing — we define what was impacted and skip unrelated actions (e.g.: building or testing ). in practice : We did a change to production code. We do analysis of what was affected. We will make a decision about what should be built and which tests should be executed depending on that analysis.…
I once wrote a note about poking around Android apps. The other day I realised I still haven’t done it for iOS . Let’s fix the gap (: Nice to have Hopper Disassembler (free version is enough for now) class-dump ( binary selection depends on your app) passionfruit (for those who like UI ) – as it’s kind of archived I’ll update the note right after depreciation of the tool objection (for those who…
Requirements macOS Xcode usbmuxd Jailbroken iDevice connected via USB (e.g.: checkra1n ) Installation Extracting debugserver (via hdiutil ) cd /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/DeviceSupport/<iOS version> hdiutil attach DeveloperDiskImage.dmg cp /Volumes/DeveloperDiskImage/usr/bin/debugserver ~/Desktop Signing debugserver (via codesign ) touch…
What is «Waiting»? Lifeguard from flaky tests. What variety do we have? Hmm, let me think… 🤔 We knew about Idling Resources Pros: It’s the most honest, fastest, and technically perfect way to wait; It doesn’t eat up the memory. Cons: Sometimes it’s not a bit easier than writing production code; Sometimes we’ll have to touch production code; Most cases will probably require the own Idling…
While doing pretty and flexible things with Espresso , have you ever faced with AmbiguousViewMatcherException ? If so, go on reading, if no, you are lucky guy, just keep the article (: When does this exception appear? Imagine, you have a RecyclerView with a lot of elements and some of them have the same resource-id as the neighbours have. Do you wanna count the specific element? Do you wanna tap…
Set secure web proxy on Wi-Fi networksetup -setsecurewebproxy Wi-Fi "127.0.0.1" "1234" Set secure web proxy on Ethernet networksetup -setsecurewebproxy Ethernet "127.0.0.1" "1234" Turn secure web proxy on networksetup -setsecurewebproxystate Ethernet 1 Turn secure web proxy off networksetup -setsecurewebproxystate Ethernet 0 To get more interesting things about network automation read this note .
Introduction Today I want to write about sniffing network traffic on Android . Let me show you what we had before and what we have after Android Nougat : In other words, Google removed the feature to sniff a network traffic out of the box in Android Nougat . Furthermore, it means, that you won’t be able to sniff a network traffic in apps from Play Market . Thereby, we should install network…
Get an url of required app (e.g. YouTube) https://apps.apple.com/us/app/youtube-watch-listen-stream/id544007664 Take an app identifier and set it in a new request https://itunes.apple.com/lookup?id=544007664 Enjoy the result version , rating , release date and much more, just look at the sample response : { "results" : [ { "primaryGenreName" : "Photo & Video" , "artworkUrl100" : "https: \/\/…
Swiping, swiping, swiping. From dusk till dawn. iOS extension XCUIApplication { func openNotifications () { let up = self . coordinate ( withNormalizedOffset : CGVector ( dx : 0.1 , dy : 0.001 )) let down = self . coordinate ( withNormalizedOffset : CGVector ( dx : 0.1 , dy : 0.8 )) up . press ( forDuration : 0.1 , thenDragTo : down ) } } extension XCUIApplication { func openControlCenter () { let…
Git Hooks are really useful guys that help us to control interactions with Git . I would like to show you couple of samples of their usage. Precondition First of all we should: create .githooks folder near the .git folder create a Makefile in the project’s root that will set githooks path and activate them fill the Makefile like in the sample below: init: git config core.hooksPath .githooks git…
Jenkins - free, flexible, popular and gorgeous continuous integration server. Jenkins Pipeline - plugin to build continuous delivery pipelines as a code (further Pipeline ). Jenkins Job Builder - tool to make jobs from templates in a human readable text format and keep them in a version control system (further JJB ). Purpose minimize or exclude the relationships with Jenkins UI provide CI which…
Today I would like to consider one awesome thing — OCR ( optical character recognition or optical character reader ) and try to apply it in mobile automation. Last time I told about snapshot testing in Android and iOS , but this is completely different feature. Purpose I’ll show you how to read text from a mobile screen. I know we could do it with XCTest or Espresso (or even UIAutomator ) in our…
Would you like to look under the hood of any Android app? You are welcome! (: Precondition dex2jar — tool for converting .dex file to .class files brew install dex2jar jd-gui — tool for decompiling and analysing Java byte code brew cask install jd-gui Let’s go First of all we should get the .apk . If you haven’t one, you are able to download it from store (for more info read this note . Then we…
Sometimes, when you’d like to install a third-party iOS app outside of the App Store on your device you might come across with «Application Verification Failed» error. It can be a rare case, because it’s really a concatenation of circumstances and means, that you received the app without your team’s provision profile . Anyways, we are able to solve this problem. Plan Get own provisioning profile…
The best mobile automation content delivered straight to your inbox. 1,true,6,Contact Email,2 1,false,1,First Name,2 1,false,1,Last Name,2
The best mobile automation content delivered straight to your inbox. 1,true,6,Contact Email,2 1,false,1,First Name,2 1,false,1,Last Name,2
Introduction First of all, I would like to link wiki MITM definition. If you are interested in my simple explanation, MITM is a hacker attack, in which there is a client, a server, and that man from the title in the middle. Man in the middle changes the original connection of the user and receives full access to his traffic with the ability to add or remove any stuff in requests and responses.…
All below is worth to read, but if you request my humble opinion today on which exactly tool to use, I’d suggest going with paparazzi . Introduction At last time I talked about snapshot testing on iOS . Now I would like to turn to the second, no less popular mobile platform, — Android . Here we are very limited in the choice of tools for the snapshot verifying using native frameworks. The popular…
All below is worth to read, but if you request my humble opinion today on which exactly tool to use, I’d suggest going with pointfreeco . Introduction «It’s not a bug, it’s a feature». Almost every member of the team has already learned this phrase. And yes, curiously enough, it can be said about some amount of defects. But not about badly displayed layout. This is especially important for mobile…
