test-engineering

Generated at build time from skills/test-engineering/SKILL.md. Do not edit this page directly.

Testing exists to reduce uncertainty about meaningful behavior.

R → Graph → Proof<P, B, O>
│              │      │  │  │
│              │      │  │  └─ observable outcome / invariant (§6)
│              │      │  └──── faithful test boundary           (§3)
│              │      └─────── property that must be proven     (§2)
│              │
│              └─ nodes = setup, stimulus, system boundary, observations
│
└─ behavior / invariant / failure / regression risk

§1  Gate         testing itself must require design
§2  P            property, invariant, or regression to prove
§3  B            smallest faithful boundary where correctness lives
§4  Setup        deterministic state and controlled dependencies
§5  Exercise     stimulate the real behavior, including failures/races
§6  O            observe outcome, state, side effect, or invariant
§7  Doubles      isolate boundaries; never fake the semantics under proof
§8  Regression   reproduce the bug at the lowest useful faithful boundary
§9  Suite        reliability, speed, brittleness, and architecture
§10 Handoff      tests are executable proof; no testing ceremony artifact

Read the behavior or risk. Decide exactly what must be proven. Find where that correctness actually lives. Use the smallest test that still exercises those semantics faithfully.

If a test does not materially increase confidence in a behavior, invariant, boundary, or failure mode, do not create it.

1. Applicability gate

test-engineering is a specialist capability, not a mandatory stage.

Do not invoke it when the verification is obvious:

simple deterministic business rule
→ write a unit test directly

simple bug with an obvious reproduction
→ write the regression test directly

existing local test pattern clearly fits
→ follow it directly

mechanical/refactor-only change
→ run targeted existing verification

Invoke test-engineering when testing itself has a design question, for example:

Which boundary should prove this behavior?

Unit, integration, contract, concurrency, or E2E?

How do we reproduce this race?

How do we prove transaction atomicity?

How do we verify retry / idempotency?

How do we test async or queue semantics?

How do we verify backward-compatible migration?

How do we inject a realistic dependency failure?

How do we prove backup/restore or recovery behavior?

Why is this suite flaky, slow, brittle, or expensive to maintain?

Routing rule:

obvious verification
→ implement directly

verification itself is complex or high-risk
→ test-engineering

Testing is continuous feedback during implementation, not a phase after “dev complete.”

2. Define P: what exactly must be proven?

Start from behavior and risk, not from files or functions.

Ask:

What can be wrong?
What property would distinguish correct from incorrect?
What observable evidence would prove it?

Good proof targets include:

business rule
calculation
parser/validator semantics
state transition
permission decision
invariant
public/API contract
transaction atomicity
concurrency outcome
idempotency
ordering
retry semantics
migration compatibility
failure/recovery behavior
critical user journey
bug regression

Examples:

P1:
two overlapping active bookings for the same room/time
must never both commit

P2:
a retried payment command must not create a second charge

P3:
records written by schema version N+1
must still be readable by version N during rollout

Do not define the proof as:

"method X was called"
"mock Y received argument Z"
"line 42 executed"
"coverage increased"

unless that call/line is itself part of a real external contract.

Test behavior, not implementation structure.

3. Choose B: the smallest faithful boundary

Ask:

Where does correctness actually live?

Then choose the smallest boundary that still contains those semantics.

Unit

Use when correctness is local and does not depend on real infrastructure semantics.

Strong candidates:

business rules
calculations
parsers
validators
state machines
permission/policy decisions
error mapping
retry decisions
pure transformations
algorithmic logic

Example:

requested slot + existing slots
        ↓
overlap policy
        ↓
allowed / denied

A unit test is valuable because the rule itself is local.

Integration

Use when correctness depends on a real boundary or infrastructure semantics:

SQL constraint
transaction
locking/isolation
ORM mapping
filesystem behavior
serialization
request pipeline
real dependency protocol
database migration

Do not replace the semantics under proof with mocks.

If the risk is:

"only one concurrent booking may commit"

the faithful boundary may require:

real concurrent execution
+
real transaction/storage boundary

A mocked transaction() call does not prove atomicity.

Contract

Use when the primary risk is compatibility between independently evolving producer/consumer boundaries.

Examples:

API request/response shape
event schema
consumer/provider expectation
backward/forward compatibility

Contract tests do not replace integration tests when runtime infrastructure semantics are the thing at risk.

Concurrency

Use when the property only exists under overlapping execution.

Model the race explicitly:

start A
start B
synchronize at critical point
release both
        ↓
observe committed results
        ↓
assert invariant

Avoid “concurrency tests” that merely call the same function twice sequentially.

E2E / Critical User Journey

Use when the property depends on the whole user-visible journey:

signup → verify → login
booking → pay → confirmation
edit → save → refresh → persisted state

Keep E2E count small and focused on critical journeys and failures that cannot be proven faithfully lower in the stack.

Specialized

Use specialized tests only when the risk demands them:

load / stress
fault injection
property-based
fuzz
compatibility
restore/recovery
browser/network lifecycle

Do not force them onto ordinary feature work.

Boundary rule

Choose the smallest test
that faithfully proves
the actual risk.

“Smallest” without “faithful” creates false confidence.

“Faithful” without “smallest” creates slow, brittle suites.

4. Build deterministic setup

A test must control the preconditions needed for the proof.

Define:

initial state
test data
clock/time
randomness
identity
configuration
dependency behavior
resource lifecycle
concurrency coordination

Control only what affects the property.

Prefer:

ephemeral data
isolated namespaces/databases
explicit clocks
seeded randomness
deterministic synchronization
known dependency fixtures

Avoid:

shared mutable test state
arbitrary sleeps
dependency on test execution order
production data assumptions
uncontrolled wall clock
network timing as synchronization

If the test cannot reproduce its own starting state, it is not reliable evidence.

5. Exercise the real behavior

Stimulate the system through the boundary that owns the property.

Examples:

domain function
API
database transaction
event consumer
worker
browser
restore procedure

Do not reach into private helpers merely because they are easy to call.

For failure testing, inject the failure at the real dependency boundary:

payment provider timeout
database serialization failure
lost response after successful commit
duplicate message delivery
worker restart
partial migration state

Do not simulate a failure in a way that bypasses the code path being validated.

For async systems, define completion from the behavior, not a sleep:

wait until observable state reaches X
or
until bounded deadline expires

6. Observe O: assert the property, not the choreography

Assert on the smallest set of outputs that proves P.

Possible evidence:

return value
public response
persisted state
event emitted
event not duplicated
state transition
invariant
external side effect
durable work
visible user outcome
recovered data

Example:

two concurrent booking attempts
        ↓
assert:
- exactly one succeeds
- exactly one is rejected/retried as designed
- one valid booking exists
- overlap invariant still holds

Avoid assertions such as:

internal helper called once
exact private method ordering
every mock interaction
incidental JSON field ordering
exact UI text when wording is irrelevant

A refactor that preserves behavior should not break the test.

7. Use doubles without invalidating the proof

Rule:

Mock a boundary to isolate behavior.

Do not mock the thing whose semantics
you are trying to prove.

Good:

testing payment failure handling
→ payment provider double can produce timeout/decline

Bad:

testing database transaction atomicity
→ mock database transaction

Bad:

testing serialization compatibility
→ bypass the real serializer with hand-built objects

Choose among:

fake
stub
mock
real dependency
ephemeral dependency

based on the semantic fidelity required by P.

Prefer state/result assertions over interaction assertions.

Use mocks for interaction contracts only when the interaction itself is the behavior.

8. Regression rule

For a real bug:

Bug
 ↓
Can it be reproduced automatically
at a useful faithful boundary?
 ↓
yes
→ create the smallest regression proof
→ then fix / verify the fix

The test should fail for the original defect and pass for the corrected behavior.

Choose the boundary where the bug actually exists.

Example:

server commits completion
but response is lost
        ↓
client refresh shows completed state
        ↓
UI must not report false failure

If that behavior can be proven at service/integration level, do not require browser E2E.

If the defect only emerges from browser/network lifecycle, use E2E.

Do not write a regression test that merely repeats the implementation logic and therefore could share the same bug.

9. Special proof patterns

Transactions and concurrency

When correctness depends on isolation, locking, constraints, or atomicity:

real storage semantics
+
overlapping execution
+
final invariant

Do not infer race safety from unit tests.

Test expected conflict/retry outcomes when the storage model can legitimately abort concurrent work.

Idempotency / retries

Model at least:

first delivery
duplicate delivery
retry after known failure
retry after uncertain outcome

Assert on the final external effect:

one charge
one booking
one durable job

not merely on handler return values.

Async / queue systems

Consider only semantics that matter:

at-least-once delivery
duplicate message
ordering
poison message
retry/dead-letter behavior
crash between effect and acknowledgement
backlog recovery

Test the property promised by the system; do not attempt to prove delivery guarantees the underlying broker does not provide.

Migrations / compatibility

When old and new versions may coexist, test the actual compatibility window:

old schema/data
  ↓ migrate/expand
old reader + new writer?
new reader + old writer?
  ↓
contract still valid
  ↓ contract/cleanup

Only test coexistence directions that the release strategy actually requires.

Use real migration scripts against representative schema/data where migration semantics are the risk.

Failure / recovery

When the system claims it can recover:

establish valid state
→ induce relevant failure
→ execute recovery path
→ assert state/invariants
→ re-run critical behavior

A recovery command exiting 0 is not proof of recovery.

Security properties

security-review owns which attack/trust property matters.

test-engineering owns the executable proof when testing that property is non-trivial.

Example:

security-review:
tenant A must never read tenant B data

test-engineering:
design the faithful API/DB proof for that isolation property

Do not independently invent a full security test matrix.

10. No coverage theater

Coverage is execution evidence, not correctness evidence.

Do not do:

coverage = 74%
→ generate tests until 90%

Use coverage only as a diagnostic clue:

important behavior appears unexercised
        ↓
inspect whether a meaningful proof is missing

Do not optimize line, branch, or mutation score without tying the added test back to behavior/risk.

A high-coverage suite can still fail to prove the important thing.

A low-coverage subsystem can still be adequately protected if most code is wiring and the meaningful behaviors are proven.

11. Treat flaky and brittle tests as defects

A useful automated test should provide a trustworthy signal.

If the same code can randomly produce pass/fail, the test is not reliable evidence.

When diagnosing flakiness, inspect:

test setup/data
ordering dependence
shared state
clock/time
randomness
async synchronization
resource cleanup
test runner/framework
SUT nondeterminism
OS/network/external dependency

Do not normalize:

"rerun until green"
"known flaky"
"ignore intermittent failure"

Short-term quarantine may protect the signal of the main suite, but it must not become permanent acceptance.

Brittleness is different from flakiness:

flaky
→ same behavior, inconsistent result

brittle
→ harmless implementation change breaks the test

For brittle tests, reduce knowledge of implementation details and assert on stable behavior.

12. Keep feedback proportional and fast

Fast feedback matters, but test fidelity comes first.

Structure the suite so the most relevant cheap proofs run earliest:

local targeted test
    ↓
broader integration / contract
    ↓
critical E2E / specialist suite

Do not run every expensive suite for every local edit when dependency analysis or CI stages can select them safely.

Do not split tests merely to hit a timing quota.

As a practical suite-health signal, automated developer/CI feedback should normally remain in minutes, not become a long stabilization phase. If the default suite becomes slow enough that developers avoid running it, its architecture needs work.

Separate expensive specialist suites when necessary:

load
long-running compatibility
chaos/fault
large E2E
recovery drills

but ensure they run at the release/risk point where their evidence is needed.

13. Improve test architecture only when it pays

Use test-engineering for suite-level problems when they create real engineering cost:

flakiness
slow feedback
duplicated setup
brittle selectors/contracts
unrepresentative mocks
hard-to-create test data
unclear test ownership
expensive environment boot
tests that cannot reproduce production bugs

Prefer fixing the test boundary or system testability over adding more helpers around a bad design.

Examples:

hard-to-test time logic
→ explicit clock boundary

massive mock graph
→ component owns too many dependencies

every UI change breaks E2E
→ assertions/selectors coupled to presentation details

DB bug only tested with mocks
→ add focused real-DB integration boundary

Do not introduce a testing framework/abstraction unless it removes repeated complexity or enables a proof that was previously unreliable.

14. Relationship to other skills

design-thinking

design-thinking owns obvious verification while implementing:

design
→ implement
→ obvious targeted proof

Hand off to test-engineering only when verification itself requires non-trivial boundary, environment, concurrency, failure, or suite design.

test-engineering may return a test design that design-thinking implements with the production change.

code-review

code-review asks:

Does the implementation have meaningful risk?
Is the existing verification faithful and sufficient?

If the missing/incorrect verification is non-trivial:

code-review
→ test-engineering

test-engineering does not duplicate structural review.

engineering-design

Engineering Design owns system guarantees and invariants.

Test Engineering turns those guarantees into executable proof when the test strategy is non-obvious.

If a guarantee cannot be tested without violating boundaries or massive mocking, that can reveal an engineering-design/testability problem.

security-review

Security Review identifies security properties and attack paths.

Test Engineering designs reliable executable proof for those properties when needed.

production-ops

Production Ops owns what operational recovery/health must be proven.

Test Engineering can design complex recovery tests:

backup
→ isolated restore
→ integrity checks
→ application smoke

only when that proof is non-trivial.

release-engineering

Release Engineering decides which already-defined proofs become release gates and where they run.

Test Engineering designs the proof itself.

Do not put CI/CD orchestration in this skill.

15. Output and persistence

The primary output is executable tests and the minimum supporting fixtures/helpers needed to make those tests reliable.

Do not create:

.agents/tests.md
TEST_PLAN.md
QA_PLAN.md
coverage-goal.md

by default.

The tests are the durable executable proof.

If a test strategy introduces a consequential architecture decision, such as a new system boundary solely to make critical behavior testable, feed that back to engineering-design rather than hiding the decision in test code.

For an analysis-only request, output:

Property
→ risk to prove

Boundary
→ why this is the smallest faithful test level

Setup
→ required deterministic state/dependencies

Exercise
→ action/failure/concurrency schedule

Assertions
→ observable evidence

Gaps
→ only material limitations

For an implementation task:

design proof
→ implement test
→ run it
→ confirm it can fail for the intended defect/property
→ run relevant surrounding tests

Do not report a test as proof merely because it passes once.

---

Proportionality

Depth follows testing complexity and risk.

tiny deterministic logic
→ direct unit/regression test; no specialist skill

normal feature
→ targeted unit/integration tests using existing patterns

boundary/state/failure-sensitive feature
→ test-engineering on affected proof graph

concurrency/migration/async/recovery/complex regression
→ specialist test design + faithful environment

large or unhealthy suite
→ test architecture analysis only where feedback/reliability cost justifies it

Do not maximize test count.

Maximize:

confidence gained
────────────────
cost + brittleness + runtime

subject to the proof still being faithful.

---

The Pipeline

BEHAVIOR / RISK
  → "What exactly can be wrong?"
      → define P: property/invariant/regression

  → "Where does that correctness actually live?"
      → choose B: smallest faithful boundary

  → "What state and dependencies must be controlled?"
      → deterministic setup

  → "How do we exercise the real behavior?"
      → stimulate success/failure/race/compatibility path

  → "What observable outcome proves correctness?"
      → define O: result/state/invariant/side effect

  → "Am I mocking the semantics I am trying to prove?"
      → replace with real/ephemeral dependency when necessary

  → "Can this fail deterministically for the bug/property?"
      → validate test efficacy

  → "Does it remain useful after harmless refactors?"
      → remove implementation-detail coupling

  → EXECUTABLE PROOF
      → run at the cheapest lifecycle point where it preserves confidence

Choose the smallest test that faithfully proves the actual risk.