Testing strategy

A test strategy defines what evidence the team needs before changing or releasing software. The goal is not the largest test count. It is fast, trustworthy feedback about the failures that matter.

Use the test pyramid as a cost model

The classic pyramid has:

1. many fast unit tests;
2. fewer integration and contract tests;
3. a small set of end-to-end tests.

The shape is guidance, not a quota. A SQL-heavy application may need more database integration tests than pure unit tests. A thin UI over stable APIs may need only a few end-to-end journeys.

Evaluate each level by:

• defect detection value;
• execution time;
• determinism;
• diagnostic quality;
• maintenance cost;
• resemblance to production.

Unit tests protect domain rules

Unit tests run one coherent unit without real network, database, clock, or filesystem dependencies. They are ideal for:

• calculations and validation;
• state transitions;
• parsing already-loaded data;
• permission decisions;
• retry decision policy;
• orchestration with injected collaborators.

from dataclasses import dataclass
from datetime import UTC, datetime
from typing import Protocol


class Clock(Protocol):
    def now(self) -> datetime: ...


@dataclass(frozen=True)
class Subscription:
    expires_at: datetime

    def is_active(self, clock: Clock) -> bool:
        return clock.now() < self.expires_at

Inject the clock rather than freezing global time. A fixed fake gives deterministic boundary tests.

Integration tests verify boundaries

Integration tests use real implementations of important adjacent components:

• repository code against the supported database engine;
• HTTP serialization through the ASGI application;
• migrations from a previous schema;
• CSV or Parquet round trips;
• HTTP adapters against a controlled local server;
• package installation and console entry points.

Example repository test:

import sqlite3
from collections.abc import Iterator

import pytest


@pytest.fixture
def database() -> Iterator[sqlite3.Connection]:
    connection = sqlite3.connect(":memory:")
    connection.execute("PRAGMA foreign_keys = ON")
    initialize(connection)
    try:
        yield connection
    finally:
        connection.close()


@pytest.mark.integration
def test_creating_duplicate_project_rolls_back(
    database: sqlite3.Connection,
) -> None:
    repository = SqliteProjectRepository(database)
    repository.create("alpha")

    with pytest.raises(DuplicateProjectError):
        repository.create("alpha")

    assert repository.list_names() == ["alpha"]

SQLite in memory is valid only if production behavior being tested is SQLite. It is not a faithful substitute for PostgreSQL concurrency, SQL dialect, extensions, or isolation. Use disposable production-engine instances in CI when those differences matter.

Contract tests check assumptions

Your code can satisfy a mock that does not resemble the real dependency. Contract tests verify the assumptions at an integration boundary:

• required request headers and body shape;
• accepted status codes;
• response schema and optional fields;
• provider error format;
• version compatibility.

Consumer-driven contracts can help when teams release services independently. They do not replace provider integration tests or a few end-to-end flows.

End-to-end tests protect critical journeys

E2E tests exercise the deployed or production-like system through public interfaces. Keep them focused:

• create a record through the API and retrieve it;
• run the installed CLI against a disposable database;
• authenticate and complete one critical browser workflow;
• verify a migration plus application startup.

Do not encode every field validation through a browser. Lower levels produce faster, clearer failures.

Organize and select tests

[tool.pytest.ini_options]
testpaths = ["tests"]
markers = [
  "integration: real database, filesystem, or service boundary",
  "contract: verifies an external interface assumption",
  "e2e: exercises a complete user journey",
]

# Fast development loop
uv run pytest tests/unit

# Boundaries
uv run pytest -m "integration or contract"

# Full pre-release suite
uv run pytest

Directory and marker names should communicate cost and dependencies. A marker is not isolation: integration tests still need deterministic setup and teardown.

Design test data for meaning

Prefer builders with safe defaults:

from dataclasses import replace


def an_item(**changes: object) -> Item:
    base = Item(id=1, name="Keyboard", quantity=3)
    return replace(base, **changes)

Use the smallest dataset that demonstrates the rule. Large opaque fixtures hide which values matter. For database tests, explicitly create the rows each behavior depends on.

Prevent flaky tests

Common flake sources and remedies:

• Real clock: inject time or use bounded eventual assertions.
• Shared database: isolate by transaction, schema, or disposable instance.
• Randomness: inject and report the seed.
• Network internet: use a controlled local fake; keep provider smoke tests separate.
• Order dependence: reset global state and randomize order periodically.
• Fixed sleeps: poll for a condition with a strict deadline and useful failure details.
• Port collisions: ask the operating system for an available port where possible.

Do not solve flakes with unconditional retries. Retries can collect evidence temporarily, but the test remains untrustworthy until the cause is fixed.

Test failure behavior

Happy paths are usually easy. Prioritize:

• malformed and missing inputs;
• boundary values;
• duplicate writes;
• timeouts and partial responses;
• database constraint violations;
• rollback after the second step fails;
• authorization denial;
• cancellation and shutdown;
• retry exhaustion;
• corrupted files and incompatible schema versions.

Failures at boundaries often carry more operational risk than another happy-path permutation.

CI stages

A balanced pipeline might run:

1. Ruff formatting and linting;
2. mypy;
3. unit tests with branch coverage;
4. integration tests with disposable services;
5. package build and wheel smoke test;
6. selected E2E tests;
7. deployment smoke checks.

Parallelize independent stages, but preserve logs, coverage, and failure artifacts. Run the same commands locally and in CI through project scripts or documented uv commands.

Decide what not to test

Avoid tests that merely prove:

• Python assigns an attribute;
• Pydantic performs already-covered basic validation;
• a third-party library's internals work;
• private helper call order remains unchanged;
• generated property accessors return stored values.

Test your configuration and use of a framework where mistakes are plausible. For example, test that the API returns the intended public fields, not that FastAPI can serialize any Pydantic model.

A risk-based planning exercise

For each feature, write a small matrix:

Prioritize impact and likelihood, then select the cheapest test that provides credible evidence.

Common pitfalls

• Counting tests instead of evaluating risk coverage.
• Mocking every dependency and testing an imaginary system.
• Using production-like E2E tests for all validation.
• Sharing state between tests to save setup time.
• Marking flaky tests as skipped indefinitely.
• Requiring 100% coverage without branch or assertion quality.
• Running SQLite tests while claiming PostgreSQL compatibility.

Practice

1. Classify ten existing tests by level and identify misplaced ones.
2. Replace one E2E validation case with a parametrized unit test.
3. Add a real database integration test for rollback behavior.
4. Write a contract test for one external JSON response.
5. Create a risk matrix for the capstone and choose one test at each useful level.