At Stitch Fix, we’re all about being bright, kind and goal-oriented. As an engineering team, one of our goals is to have seamless continuous integration (CI) and continuous deployment (CD) processes. Most of our engineers focus on web development with Ruby on Rails. I’m jealous of them – the Rails community has testing tools and processes that are well-established. Testing is a bit newer for the iOS community, and there are still lots of nascent tools out there to support iOS testing. So, our iOS team has had to do a bit of experimenting to find the tools that can best support our goal of a seamless CI and CD process.
In this blog post, I’ll talk about:
- What do we test and why?
- How do we test our iOS app? How has it changed over time?
- How do we do CI? (It’s evolved since we wrote about it last year!)
- How do we do CD?
We could not have the luxury of continuous deployment if we didn’t have the discipline of a strong testing process. Tests help us ensure that each new version of the app works well, without having to go through an extensive manual QA process. Tests help us continuously deploy with confidence. Thus, the first 2 sections of this post focus primarily on testing.
What do we test? And why?
This is the easy question! We test everything. Why? To ensure that we continue to deliver a delightful experience to our clients! Also, to enable our continuous deployment to be continuously wonderful.
So we test everything. But, “everything” means very different things to various people. For some people, “everything” means “everything but view controllers”. For others, it means “only the networking layer.” For still others, it means “only things that have broken in the past.”
For us, “everything” is literal. We take it seriously. One great heuristic for deciding whether to test something is to ask yourself: would it be bad for the business, or frustrating for the user if this line of code broke? If yes, write a test! If no – is that line of code really necessary?
Let’s dig into an example – a user navigating through our sign-up form.
- When user taps the help button, we show an action sheet with options to visit our help center or contact us.
- When user taps any text field, the keyboard becomes active.
- When user taps the “next” or “previous” buttons in the keyboard accessory, the appropriate adjacent text field becomes active.
- When the user taps the big red “NEXT” button and they have not filled out any fields, we show client-side validation errors.
- When the user taps the big red “NEXT” button and they have filled out all the fields, we want to sign them up for Stitch Fix! We want to show a spinner while doing so. A few things can happen from there:
- Sign up succeeds - in this case, we want to show them the onboarding flow.
- Sign up fails because this email address already has an account - in this case, we want to show an alert that reminds the user of their account.
- Sign up fails because of server-side validation errors - in this case, we want to display those validation errors
- Sign up fails for some other reason - we want to display the appropriate error.
- Regardless of whether sign up succeeds or fails, we want to stop the spinner.
As you can see, that is a lot of behavior. If any of it broke, users would be frustrated, and the business could lose a potential client. So, we test it all. However, it’s not all in a view controller unit test (that would be a very long file), nor is it all in a UI test (that would be a very slow test).
In fact, the view controller for sign-up is less than 150 lines. It delegates most of its behavior to other objects. Each of those objects have their own unit tests, and their behavior is entirely decoupled from the view controller. This view controller’s dependencies include:
- An error presenter (responsible for presenting alerts)
- A view validator (responsible for validating input and displaying validation errors)
- A help presenter (responsible for presenting the action sheet that links users to our help center)
- A sign-up service (responsible for sending the sign up request). This has quite a few of its own dependencies:
- A request model (models the actual HTTP request to send to the server)
- An HTTP client (responsible for the actual network request)
- A HTTP response deserializer (deserializes errors from any failed network request)
- A sign-up response deserializer (deserializes the successful sign-up JSON response into the appropriate model)
How do we test all these components? How has this changed over time?
We test each dependency’s happy path, and all of its sad paths. It’s a lot of unit tests, but they’re great - they allow us to move quickly and not break things. They allow us to re-use components (i.e the error presenter & help presenter) throughout the app, and be confident that they’ll work correctly, without sign-up specific behavior.
We use Quick. We love it! It allows us to write descriptive unit tests - all of our tests explain what we expect, and many of them also explain why.
We test every internal method on our objects. (Because we’re not building a library, we have not marked any methods as “public”). If a method is not called by other objects, we mark it as private. It’s that simple. Our favorite objects have 1 or 2 internal methods at most. Generally, each method will have 1 or more success modes, and 1 or more failure modes – we test every single one of those.
Additionally, we ensure that every class we write conforms to a protocol. (We’re big fans of the “Crusty” talk from WWDC 2015.) This makes it easier to write unit tests, because we can create fake versions of all of an object’s dependencies. When we’re testing an object (like the sign up service), we inject a fake version of all its’ dependencies. It gets a fake HTTP client & fake deserializers. This makes it easy to test ONLY the behavior of the sign-up service, and not the behavior of its dependencies.
Some might call us crazy – we also write unit tests for our view controllers. They’re slightly different than the unit tests for other classes, but the basic idea is the same – we want to make sure that they have the correct behavior when you interact with their public interfaces. (If you’re interested in learning more about testing view controllers, you can check out this talk that I gave at a local meetup last year.)
Our focus is primarily on unit tests. They’re easy to write and quick to run. They frequently guide the structure of our code towards small, reusable, understandable components. However, these are not the only tests we write!
We also use KIF to test some happy-path flows between different screens. These are slow to run, and occasionally flakey - we use them sparingly. We’re in the process of migrating our KIF tests to use Apple’s UI Testing framework.
How do we do CI?
For us, CI is all about ensuring that our app continues to build and our tests continue to pass.
Last year, we wrote a series of blog posts about how we used a combination of Xcode Server and CircleCI to build and deploy the app. We should have paused when it took 3 blog posts to describe our tool – it was not helping us towards our goal of a seamless CI/CD process. It was slow and clunky, with intermittent flakey failure. Keeping it green was a full-time job. We had trouble finding resources online to help us troubleshoot.
So, like any good agile team, we switched tools! We’re now using CircleCI for our 100% of our CI/CD flow. We’re running a slightly modified version of their default iOS build command to run our tests. All 3600 of our unit tests run in 4 minutes. They consistently pass without false negatives, and CircleCI notifies us when there are legitimate failures.
How do we do CD?
When we merge to our main development branch, CircleCI executes a Fastlane “lane” (for you Ruby developers – this is basically a Rake task) to build the app and distribute it to internal stakeholders. We use Fastlane Match to manage our provisioning profiles and signing certs. We use Fastlane Gym to build the app and Fastlane Deliver to upload it.
On our development branch, Fastlane builds a scheme that specifically runs the iOS app against our staging servers. This allows us to QA our builds without affecting production data. On a good day, we’ll send out 2-3 new internal builds. (Thus achieving continuous delivery internally!)
When we merge to our master branch, CircleCI executes a Fastlane task to build the app for the app store and upload it to Apple’s iTunesConnect platform. We don’t quite have continuous delivery for our customer-facing builds. We deploy to the App Store 1-3 times per month, depending on the scope of our new features.
We updated our build process slightly each time we updated any of our build dependencies – Xcode version, Swift version or Fastlane version. Outside of these events though, we rarely have issues. We rarely think about build distributions or fight against upload issues.
When our iOS team started almost 2 years ago, writing tests was a given. Everything else about our build process was up for discovery and debate. We iterated a fair bit, but are happy with our current process – it achieves the goal of a seamless CI/CD process.
