Continuous Deployment at Facebook

Continuous deployment is the practice of releasing each change to production as soon as it passes automated testing, with the decision to deploy left to developers rather than a separate test or release team. Facebook’s studies of its own data show it can scale without a loss of quality, and where mobile apps force a slower variant [1], [2].

1. Web: deploy every day

Savor and colleagues compared Facebook with OANDA, a smaller currency-trading company, using seven years of Facebook data [1]:

Measure Facebook OANDA
Updates released per developer per week 3.5 about 1
Median update size 33 lines 57 lines
Engineers working on deployment and test tooling about 5% 15%

Over the period, Facebook’s engineering team grew 20-fold and its code base 50-fold, yet productivity per developer did not drop, and the number of high-priority production failures grew far more slowly than the number of deployments (a trend-line slope of 0.0004). Neither company used a separate testing team: developers wrote the tests and were responsible for their code in production. Management mattered: at OANDA, the average number of deployments fell 23% after a new management team took over, although the engineers were largely the same.

2. Mobile: a release train

Mobile apps cannot be updated silently: users must install each version, app stores review releases, a fix cannot be rolled back easily, and the code ships as one binary to hundreds of device types. Facebook therefore runs a release train for its apps [2]:

  1. Developers keep pushing changes to the main branch.
  2. A release branch is cut once a week for Android and every two weeks for iOS.
  3. For five days the branch is stabilized; only fixes are cherry-picked in, and builds are installed on employees’ phones (“dogfooding”) three times a day.
  4. The code is frozen for a three-day soak, accepting only launch-blocking fixes.
  5. The release goes to the app store for review; for iOS this happens on the second Monday after the cut, and users get it later that week.

Example. For iOS, the branch is cut on a Sunday evening every two weeks. Five days of stabilization and three of soak bring it to the second Monday after the cut, eight days later, when it goes to Apple for review; users get it later that week. A change committed just before a cut therefore reaches users in about one and a half weeks; one committed just after a cut first waits two weeks for the next cut, so about three and a half. A web change at Facebook could reach production the same day. The difference is the cost of a bad release: on the web it can be replaced in hours; in an app it stays on phones until users update.

3. Automated testing at diff time

Facebook’s Sapienz generates test sequences for Android apps with a multi-objective search algorithm and reports crashes as comments on the developer’s change (a “diff”) in code review, before it lands. Developers fixed at least an estimated 75% of the crashes it reported [3]. Like the static analyzer Infer before it, Sapienz succeeded by delivering its findings where developers already work.

When to use it — and when not

Continuous deployment fits services under the team’s control, with automated tests, monitoring and fast rollback. A release train fits products that users install. Neither replaces testing; both depend on it, and Facebook and OANDA invested 5–15% of their engineers in the tooling.

Limitations

  • Companies studying themselves. All three papers are written by or with the companies’ engineers.
  • Context. Facebook’s results come with its scale, tooling budget and culture; OANDA shows a smaller company can follow, at a higher relative tooling cost.

How solid is this?

  • Where it comes from. Studies of continuous deployment at Facebook and OANDA, of Facebook’s mobile release process, and of Sapienz’s deployment [1], [2], [3].
  • What is contested. Whether to keep a separate testing team; the authors call it one of the more controversial questions.

References

  1. T. Savor, M. Douglas, M. Gentili, L. Williams, K. Beck, and M. Stumm,
    “Continuous Deployment at Facebook and OANDA,” in ICSE Companion 2016, ACM, 2016, pp. 21–30. doi: 10.1145/2889160.2889223.
  2. C. Rossi, E. Shibley, S. Su, K. Beck, T. Savor, and M. Stumm,
    “Continuous Deployment of Mobile Software at Facebook,” in FSE 2016, ACM, 2016, pp. 12–23. doi: 10.1145/2950290.2994157.
  3. N. Alshahwan et al.,
    “Deploying Search Based Software Engineering with Sapienz at Facebook,” in SSBSE 2018, in LNCS, vol. 11036. Springer, 2018, pp. 3–45. doi: 10.1007/978-3-319-99241-9_1.

Disclaimer: AI is used for text summarization, polishing and explaining. Authors have verified all facts and claims. In case of an error, feel free to file an issue.

This site uses Just the Docs, a documentation theme for Jekyll.