Smoke testing versus sanity testing — what's the difference, and which one gates a new CI build?
- Smoke is shallow and wide — a small tagged set of critical happy paths, run on every new build to check if it's even worth testing further.
- Sanity is narrow and deep — no separate tag, you filter by the feature markers already on your tests, scoped to whatever just changed.
- Smoke is the CI build gate. Sanity is the targeted post-fix check.
- Same test pool for both — smoke, sanity, and regression are reasons for running a slice, not different sets of tests.
The long answer
Smoke, sanity, and regression are not three different populations of test cases — there is one pool of tests, tagged by feature and sub-feature at write-time (e.g. login, forgot_password, checkout). The three words describe why you’re running a slice of that pool on a given day, not which tests exist.
Smoke testing: a small, fixed, hand-picked set of the most critical happy-path tests across the whole application, tagged directly as smoke at write-time. Runs on every new build, before anything else. If smoke fails, the build is rejected outright — no point running deeper tests on a broken build. This is the CI build gate.
Sanity testing: not a separate tag. When a specific area is fixed or changed (e.g. forget-password), you filter using the existing feature/sub-feature marker (pytest -m forgot_password) and run everything under that tag. If the tag returns a small number of tests (say 8), you run all of them. If it returns a large number (hundreds), you’d narrow further to the happy path plus the most critical negative case. Sanity confirms the fix works and didn’t break its immediate neighborhood.
Regression testing: running the whole pool, or a large curated slice of it, typically before a release. Every test case counts as a regression test — regression isn’t a special tag, it’s the default population.
The contrast to say in an interview: smoke is shallow-wide, sanity is narrow-deep. Smoke asks “is this build stable enough to test at all?” Sanity asks “did this specific fix work, without breaking what’s around it?”
First move when each one fails. If smoke fails: stop immediately, reject the build, don’t run anything deeper — the failure could be anywhere, and diagnosing it on an unstable build wastes time. If sanity fails: this is local information, not a build-wide red flag — the specific fix didn’t work as expected, so the first move is going back to whoever made the change with the exact failing case, not launching the wider regression suite.
Saying "we write separate smoke, sanity, and regression test cases" — you don't. You tag one pool of tests and choose which slice to run.
