SN L00: Course Introduction and CI
The opening lecture asks two questions: why should a software engineer care about quality at all, and what does checking quality look like in a modern team? Its answer to the first is the cost of failure; its answer to the second is continuous integration, where every change is built and tested automatically within minutes. The revision questions test this lecture.
1. Software failures are expensive, and each one breaks a nameable quality
The lecture opens with failures students may remember. In July 2024 a faulty content update to CrowdStrike’s Falcon security software crashed about 8.5 million Windows machines, under 1% of all Windows machines, and still stopped airlines, hospitals and banks [1]. In 2012 Knight Capital lost more than US$460 million in 45 minutes because a deployment left old code active on one of its servers [2]. In December 2021 a critical remote-code-execution flaw in Log4j, a widely used Java logging library, was already being exploited when the fix was released [3].
The useful habit is to name the quality that failed: availability for CrowdStrike, maintainability for Knight Capital (dead code nobody removed), security for Log4j. A named quality can become a requirement, and a requirement can be checked.
→ Why Software Quality Matters
2. Poor quality has a price for a whole economy, but the estimate is rough
Single incidents make the news; the total is larger. A report for the Consortium for Information and Software Quality estimated the cost of poor software quality in the United States in 2022 at at least US$2.41 trillion, spread over four overlapping areas: operational failures (the largest, mostly cybercrime), legacy-system problems, unsuccessful projects, and finding and fixing defects; technical debt is estimated separately [4]. Treat the figure as an order of magnitude. It is extrapolated from earlier estimates, its parts overlap, and its rise from US$2.08 trillion in 2020 (16%) is almost exactly the 15% inflation of those two years.
For a team, the more useful number is its own: how many hours went into finding and fixing defects this month. Whether a defect costs more the later it is found is a separate question, with a less dramatic answer than the often-quoted claim that a fix in production costs 100 times more than one made during requirements: measured ratios run from about 5:1 on small systems to over 100:1 on large aircraft programmes, and one study of 171 software projects found no such effect [5], [6], [7].
→ Cost of Poor Software Quality · Cost of Late Fixes
3. Released software still carries defects: a few per thousand lines
How many defects reach users? A study that collected published figures for 109 projects found a median of 4.3 post-release defects per thousand lines of code (KLOC), and a mean of 7.47 pulled up by a few very defective projects [8]. Open-source projects, Java projects (compared with C) and very large projects had lower densities in that sample. These are rough benchmarks: the projects counted lines and defects in different ways.
For example, to compare your own project, divide post-release defects by size: 30 defects reported in the first year of a 12,000-line service is 30 / 12 = 2.5 defects per KLOC.
4. The course moves from defining quality to checking it
The course follows the order in which a team meets the problem. First it defines quality: quality models say which characteristics matter, and measurement puts numbers on them. Then it checks quality: verification methods, coverage criteria, input-domain, random and exploratory testing. Finally it covers qualities in practice: reliability, performance and security. Setting up quality gates in a CI pipeline runs through all of it as the main practical skill: each later technique becomes one more automated check.
→ Quality
5. Continuous integration checks every change, automatically
Continuous integration (CI) means that every developer merges changes into the shared main line at least daily, and every merge is verified by an automated build that includes the tests, so that integration errors appear within minutes [9]. Without it, developers work on separate branches for weeks and then spend days making them fit together. A broken build is fixed first; in Kent Beck’s words, quoted by Fowler, “nobody has a higher priority task than fixing the build”. The quickest fix is often to revert the faulty commit and repair it afterwards; while the build is red, nobody can tell whether a newly pushed change is good, and new failures hide behind the old one [9].
Two extensions share the abbreviation CD. Continuous delivery keeps every change that passes the pipeline ready to release; continuous deployment releases it automatically. Both depend on a trusted green build.
The evidence is encouraging but not proof. In a study of open-source projects on GitHub, projects using CI released about twice as often (0.54 against 0.24 releases a month), yet the same projects had already released 0.34 times a month before adopting CI, more than projects that never did [10].
6. A pipeline is a chain of quality gates
A quality gate is a checkpoint a change must pass before it moves on [11]. The lecture’s pipeline has four: before the merge (code review, linter, unit tests), the build (compilation, and static analysis: tools that inspect the code without running it), the full test run (unit, integration, end-to-end) and the release (checks on a staging system, then a canary that sends a small share of real traffic to the new version). A gate that is automated runs on every change; a gate that waits for a person runs only when someone has time.
For example, in a small Python project the first gates take one short GitHub Actions workflow: on every push, check out the code, install the dependencies, run the linter (ruff), then the tests (pytest). The first two steps only set the job up; the linter and the tests are the checks, and any failing step turns the build red. The author then sees a red cross next to the commit or pull request, gets a notification and can open the log of the failing step; if the repository requires a green build, the pull request cannot be merged. A linter is a light form of static analysis: it checks style and simple error patterns without running the code.
7. Most tests should be small and fast
A pipeline that takes an hour defeats its purpose, so the test suite needs a shape: many narrow unit tests, fewer integration tests, a few end-to-end tests. Software Engineering at Google recommends about 80% unit, 15% integration and 5% end-to-end tests by count [11]. The 70/20/10 split on the lecture slide comes from an earlier Google testing-blog post; both are rules of thumb, not measured optima. Small tests are fast and point at the faulty code; end-to-end tests are slow and fail for reasons unrelated to the change.
8. At Google’s scale, CI has to choose which tests to run
Google’s Test Automation Platform integrates and tests more than 13,000 projects a day, with 800,000 builds and 150 million test runs, while developers commit about once a second [12]. That is too fast to test every commit on its own, so commits are batched, and developers sometimes waited hours for results. A test target is one buildable test suite, run again for every change that affects it, so the daily runs far outnumber the targets. In one month, of 5.5 million test targets affected by changes, only about 63,000 ever failed. Most tests almost never fail, which is why test selection, running first the tests most likely to fail, pays off; Google’s researchers propose it for exactly this reason.
9. Coverage shows what no test touched, not that the code is right
Statement coverage is the share of statements that at least one test executes. The lecture shows a coverage report at 37%: 63% of the statements never ran in any test. Either that code is unused and should go, or nobody knows whether it works. Full coverage proves less than it seems: a test can run every line and assert nothing, and coverage cannot reveal a feature that is missing altogether. Read an uncovered line as a clue about a missing test, not as a number to push up.
10. Automate the repeatable checks, and do not tolerate flaky tests
Regression tests, builds, static analysis and coverage reports run on every change because a machine repeats them faster and more reliably than a person. Judging usability, exploring for unexpected behaviour and reviewing privacy still need people.
A flaky test passes and fails on the same code, for example because it depends on network timing or on the order in which tests run. In one company’s million-line Java project, about one test-suite run in twenty had at least one flaky failure, and handling flaky tests took at least 2.5% of developers’ productive time [13]. The bigger damage is to trust: when a red build is often a false alarm, people stop reacting to red builds.
11. Quality does not stop at deployment
The lecture closes with a preview of site reliability engineering (SRE), Google’s way of running production services [14]. A service level objective (SLO) sets a measured reliability target, such as 99.9% of requests succeeding over 30 days; the error budget is the remaining 0.1%, the failure the team is allowed to spend on change. The mindset is the same as in CI: measure, automate, improve.
→ Site Reliability Engineering
Where this goes next
The next lecture asks what “quality” means precisely: SN L01: Quality Models turns vague praise like “reliable” or “secure” into named characteristics that can be specified and measured, the step from section 1’s incidents to requirements. The first lab is practical: set up a Python project and a GitHub Actions workflow that installs it, runs a linter and the tests on every push, and blocks a merge when the build is red.
References
- D. Weston,“Helping Our Customers Through the CrowdStrike Outage.” Official Microsoft Blog, 20 July 2024, 2024.
- U.S. Securities and Exchange Commission,“In the Matter of Knight Capital Americas LLC: Order Instituting Administrative and Cease-and-Desist Proceedings, Release No. 70694.” Administrative Proceeding File No. 3-15570, 16 October 2013, 2013.
- CISA,“Apache Releases Log4j Version 2.15.0 to Address Critical RCE Vulnerability Under Exploitation.” Dec. 2021. [Online]. Available: https://www.cisa.gov/news-events/alerts/2021/12/10/apache-releases-log4j-version-2150-address-critical-rce-vulnerability-under-exploitation
- H. Krasner,“The Cost of Poor Software Quality in the US: A 2022 Report,” Consortium for Information and Software Quality (CISQ), 2022. [Online]. Available: https://www.it-cisq.org/the-cost-of-poor-quality-software-in-the-us-a-2022-report/
- B. Boehm and V. R. Basili,“Software Defect Reduction Top 10 List,” Computer, vol. 34, no. 1, pp. 135–137, 2001, doi: 10.1109/2.962984.
- B. Haskins, J. Stecklein, B. Dick, G. Moroney, R. Lovell, and J. Dabney,“Error Cost Escalation Through the Project Life Cycle,” in INCOSE International Symposium, 2004, pp. 1723–1737. doi: 10.1002/j.2334-5837.2004.tb00608.x.
- T. Menzies, W. Nichols, F. Shull, and L. Layman,“Are Delayed Issues Harder to Resolve? Revisiting Cost-to-Fix of Defects Throughout the Lifecycle,” Empirical Software Engineering, vol. 22, no. 4, pp. 1903–1935, 2017, doi: 10.1007/s10664-016-9469-x.
- S. M. A. Shah, M. Morisio, and M. Torchiano,“An Overview of Software Defect Density: A Scoping Study,” in Proceedings of the 19th Asia-Pacific Software Engineering Conference (APSEC), 2012, pp. 406–415. doi: 10.1109/APSEC.2012.93.
- M. Fowler,“Continuous Integration.” 2024. [Online]. Available: https://martinfowler.com/articles/continuousIntegration.html
- M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig,“Usage, Costs, and Benefits of Continuous Integration in Open-Source Projects,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), ACM, 2016, pp. 426–437. doi: 10.1145/2970276.2970358.
- T. Winters, T. Manshreck, and H. Wright,Software Engineering at Google: Lessons Learned from Programming Over Time. O’Reilly Media, 2020.
- A. Memon et al.,“Taming Google-Scale Continuous Testing,” in ICSE-SEIP 2017, IEEE, 2017, pp. 233–242. doi: 10.1109/ICSE-SEIP.2017.16.
- F. Leinen, D. Elsner, A. Pretschner, and others,“Cost of Flaky Tests in Continuous Integration: An Industrial Case Study,” in ICST 2024, IEEE, 2024. doi: 10.1109/ICST60714.2024.00037.
- B. Beyer, C. Jones, J. Petoff, and N. R. Murphy,Site Reliability Engineering: How Google Runs Production Systems. O’Reilly Media, 2016.
Disclaimer: AI is used for text summarization, polishing and explaining. Authors have verified all facts and claims. In case of an error, feel free to file an issue.