08 — Testing & Quality Strategy

Layers

Layer	Tool	Scope	Runs
Unit	Vitest	core pure logic: splitter, analyzer, buildSelect/buildDml, type handlers, cache, SecretRef resolution	every commit
Driver conformance	Vitest + docker (PG · SQLite · Mongo · Redis)	the shared kit from 03 §6, per driver, per server version, through each driver's EngineFixture	every commit
Integration	Vitest + docker: PG 13/15/17 · Redis 6.2/7 · Mongo 7 and an 8 replica set · MySQL 8 · MariaDB 11 · SQL Server 2022	connect, introspect, query, cancel, stream, transaction, edit-apply-rollback	every commit
Webview component	Vitest + Testing Library	grid virtualization, cell editors, filter row, staged edits	every commit
E2E	@vscode/test-electron	real VS Code: connect → browse → query → edit → export	PR + nightly
Performance	custom bench in CI	budgets below	PR (fail on regression)
Security	targeted suites	every S* requirement in 05-SECURITY.md	every commit

Fixture schema

test/fixtures/schema.sql — deliberately nasty:

• Every PG type incl. arrays, nested composites, ranges, domains, enums, bytea, interval, timestamptz, numeric(38,10), int8 at max, uuid, jsonb with deep nesting, vector(1536) when pgvector is present.
• Tables with: no PK · composite PK · generated columns · identity · partitions · inheritance · RLS policies · circular FKs · 200 columns · a name needing quoting ("weird Name", unicode, embedded ").
• 1M-row table for perf runs, generated by generate_series.
• A view, a matview, a foreign table (postgres_fdw self-loop), functions in plpgsql + sql.

Fixture data, non-relational [MM]

test/fixtures/mongo-seed.js — documents with disjoint key sets; every BSON type including Decimal128 at full precision and Int64 at max; deep nesting; arrays of subdocuments; a field that is a string in half the documents and a number in the other half; a field present in 1% of documents; keys containing dots, $, and unicode; a 16 MB document; a 100k-document collection for paging and a 1M-document one for the benchmark; a view; a collection with a $jsonSchema validator; a TTL index; a capped collection.

test/fixtures/redis-seed.txt — a key of every type; a binary, non-UTF-8 value; a 5 MB value; a hash with 100 000 fields; a zset with float scores that must not lose precision; keys with separators at several depths, keys with none, keys with unicode and spaces; TTLs at second and millisecond precision; a stream with consumer groups and pending entries; 1M keys for the tree benchmark.

Correctness invariants, non-relational [MM]

7. Projection stability: the sampled column union is deterministic for a given sample and does not reshuffle columns while paging.
8. Missing survives: a field absent from a document round-trips as absent, never as null, and $unset is generated for removal rather than $set: null.
9. Unknown fails closed: an unclassifiable command against a prod profile is refused without the typed confirmation.
10. Partial-apply honesty: with supportsAtomicEdits: false, a batch whose third edit fails reports exactly the first two as applied.
11. No KEYS, ever: the Redis suite asserts, against a 1M-key instance, that no code path issues KEYS, and that no tree request returns more than pitta.explorer.treePageSize keys.
12. No evaluation: a .mongodb.js file containing process.exit(), require(...), or a function call outside the allowlist produces a diagnostic and executes nothing.

Performance budgets (CI-enforced from Phase 8)

Metric	Budget
Extension activation	< 200 ms
Tree expand, 10 000 objects	< 1 s
First result page visible	< 500 ms
Stream 1M × 20 cols	ext-host RSS < 600 MB
Grid scroll	≥ 50 fps
Completion p95, 5 000 tables	< 50 ms
Export 1M rows to CSV	RSS flat, no full materialization

Every figure above is the target on a developer machine. A time budget measured where CI is set is scaled by timeBudgetMs in test/bench/harness.ts, because a shared runner is roughly half the speed and varies run to run: activation measures 174 ms locally and between 230 ms and 280 ms on GitHub's runners, which failed the build on commits that changed nothing about activation. The measured number is printed on every run whatever ceiling it was compared against, so a creeping regression stays visible - and a real one is a multiple, not a third.

Correctness invariants (property tests)

1. Round-trip: for every type, write(v) → read() → equal. numeric compared as string.
2. No lossy numbers: no code path passes numeric/int8 through Number. Enforced by an eslint rule and a runtime assertion in dev builds.
3. Params only: every generated statement's user-supplied values appear in the params array, never the SQL text. Property test over random filter/edit sets.
4. Transaction atomicity: an edit batch with a deliberately failing final statement leaves the DB byte-identical to before.
5. Cancel really cancels: after cancel(), the backend PID has no active query within 1 s.
6. Identifier safety: objects named with quotes, dots, unicode, and reserved words are browsable, queryable, and editable.

Manual QA checklist (per release)

macOS / Linux / Windows × local · Remote-SSH · WSL · devcontainer · Codespaces (expect: works, with the remote-localhost caveat) · Cursor + VSCodium via Open VSX · light/dark/high-contrast themes · screen-reader pass on the grid.