Testing Workflow

Use ./fake.sh build -t Dev for the normal local check. It restores tools and packages, builds the solution, and runs the default non-visual test projects:

FAKE-backed test and validation commands share repository .fake state and are not safe to run concurrently. Run multiple FAKE-backed tests or targets sequentially in a deterministic order, even when they look independent. Non-FAKE checks may still be parallelized when they do not invoke FAKE or depend on .fake.

Sample contract checks are executable smoke contracts. They must run through the Expecto executable, not the VSTest adapter:

dotnet run --project tests/Smoke.Tests/Smoke.Tests.fsproj --no-restore

Smoke tests that launch samples or child processes must keep explicit per-process timeouts and include captured stdout/stderr in timeout diagnostics. dotnet test remains the runner for package, semantic, and governance suites that do not shell out to sample applications.

The full testing and evidence target set is Dev, Verify, Ci, VerifyPreflight, CiPreflight, PackLocal, RefreshSurfaceBaselines, PackageSurfaceCheck, FsiTranscripts, SampleContractSmoke, TemplateCheck, CapabilityCheck, SkillCheck, GeneratedProductCheck, DependencyReport, GeneratedGuidanceCheck, TemplateDrift, TargetMetadata, TargetMetadataDrift, StaleBoundaryScan, EvidenceGraph, EvidenceAudit, and FinalReadiness.

Governance tests include section-aware generated guidance checks, semantic template drift fixtures, public record invariant inventory coverage, and build workflow command-contract checks. The process reliability contract tests cover process-health snapshots, bootstrap validation, broad verification verdicts, focused-gate direct invocation, stale build/restore diagnostics, scanner fixture helpers, and pure BuildModel/BuildMsg/BuildEffect transition assertions. Layout tests include recoverable Yoga execution fallback diagnostics through the existing LayoutDiagnostic surface.

Generated app public-contract smoke tests should use app-owned names: Product.Program.view returns FS.Skia.UI.Scene.Scene, Product.Program.generatedHost is the host value passed to the viewer launch API, and reducer checks call Product.Program.update.

PackageSurfaceCheck runs tests/Package.Tests/Package.Tests.fsproj against the stable package surface baselines in readiness/surface-baselines/*.txt. The target verifies that the current exported public names still satisfy the reviewed baseline.

Screenshot Evidence Validation

Screenshot evidence is validated through FS.Skia.UI.Testing.EvidenceReports. Accepted records must include status=ok, evidence-kind=screenshot, capture-source=live-viewer-window, proves-screenshot=true, command, app/sample identity, host facts, capture mode, artifact path, positive decoded dimensions, pixel-content-validation=non-blank, blocked stage, classification, category, message, and timestamp.

validateScreenshotArtifact decodes the PNG artifact and rejects missing, unreadable, out-of-readiness, dimension-mismatched, or blank images. Structural layout reports, deterministic scene hashes, launch logs, manual descriptions, fallback diagnostics, and metadata-only records do not satisfy screenshot evidence requirements.

FsiTranscripts runs:

SampleContractSmoke runs every sample that exposes --contract-smoke and writes one log per sample under specs/006-template-framework-governance/readiness/sample-smoke/.

V3 Template Matrix

TemplateCheck validates eight rows:

CapabilityCheck, SkillCheck, GeneratedProductCheck, DependencyReport, GeneratedGuidanceCheck, TemplateDrift, TargetMetadataDrift, EvidenceGraph, and EvidenceAudit are focused governance targets and can be run independently while developing those surfaces.

Process Health And Focused Gate Checks

Run broad preflight checks directly when diagnosing runner health:

./fake.sh build -t VerifyPreflight
./fake.sh build -t CiPreflight

These targets write process-health.md, bootstrap-runner.md, and verification-verdicts.md. Malformed threshold overrides, missing runner dependencies, CoreCLR startup failures, process pressure, and unsupported signals are classified as environment evidence.

Focused gates should be run directly when isolating a failure:

1. ./fake.sh build -t PackageSurfaceCheck
2. ./fake.sh build -t ControlsCatalogCheck
3. ./fake.sh build -t ControlsInteractionCheck
4. ./fake.sh build -t ControlsRenderingCheck
5. ./fake.sh build -t CapabilityCheck
6. ./fake.sh build -t SkillCheck
7. ./fake.sh build -t GeneratedProductCheck
8. ./fake.sh build -t DependencyReport
9. ./fake.sh build -t GeneratedGuidanceCheck
10. ./fake.sh build -t TemplateDrift
11. ./fake.sh build -t TargetMetadataDrift
12. ./fake.sh build -t EvidenceGraph
13. ./fake.sh build -t EvidenceAudit

If the failure looks race-like or the concurrent FAKE context is unknown, record that classification and rerun the affected FAKE-backed commands in the same sequential order before product debugging.

If a focused gate relies on restored or built artifacts, stale assumptions must produce a diagnostic naming the affected gate and the remediation command. The command-contract tests fail if a focused gate is recoupled to Verify, Ci, or undocumented broad work.

Template Exclusions

Full visual evidence, release validation, an external template repository split, and distribution automation are deferred. They must not become pass/fail requirements for Dev, Verify, or Ci until a later roadmap phase adds explicit targets.

Package consumer smoke is intentionally opt-in through PackageSmoke, which sets FS_SKIA_RUN_PACKAGE_CONSUMER_SMOKE=1 for the package test project.