Skip to main content
Version: main 🚧

Screen, API and Detailed Case Coverage

Use this inventory with the feature matrix. A feature row can contain multiple assertions: its ID is a stable group, not evidence of a single completed E2E test. Expand groups into maintained assertion records and report each applicable runtime/persona/provider/context separately.

Source inventory gate​

node scripts/validate-regression-inventory.mjs
node --test scripts/validate-regression-inventory.test.mjs

CI compares the tables below with current ui/src/app/**/page.tsx routes and ui/src/app/api/**/route.ts top-level families. It rejects missing/stale entries, duplicate IDs/routes and references to undefined feature cases. This is a plan completeness check, not execution evidence. It does not inspect every API method, catch-all destination, backend-only endpoint or installed app's own repository.

For every release, enumerate those children from the candidate source and API contracts, map each changed behavior to assertions, and record unresolved mappings. A new handler inside an existing family still requires review even if this gate passes. Record actual enabled profiles, inherited settings, roles and app versions. Optional controls absent by design receive an explicit NOT_APPLICABLE reason; missing prerequisites for enabled controls are BLOCKED.

Required personas and expansion​

  • Fixture owner/admin; independent shared-team ordinary user; outside-team user.
  • Anonymous and invalid/expired session; scoped machine identity where enabled.
  • Additional organization only for isolated, authorized cross-organization tests.
  • Declared supported browsers and desktop/narrow viewport; use the user's chosen real browser sessions for live acceptance and label automated engines accurately.
  • Each runtime, model/provider, source/document type and bot destination/context required by the feature matrix. One member cannot substitute for an outsider.

Page route inventory​

Keep existing SCR IDs when paths persist. Verify navigation, direct URL, reload, back/forward, loading/empty/error states, keyboard focus and direct API enforcement for every applicable row. Each row maps to feature cases plus the dimensions below.

IDRouteFeature casesPositive assertionNegative assertion
SCR-001/SET-01, SET-03, DEP-01Correct enabled home cards and navigationAnonymous and disabled capabilities disclose no privileged data
SCR-002/adminRBAC-01, SET-02, OBS-01Permitted category and default destination loadOrdinary users cannot invoke privileged mutation APIs
SCR-003/agent-builderAGT-05Create, validate, run, cancel and save an owned definitionInvalid schema or foreign agent/tool cannot execute
SCR-004/agent-builder/historyAGT-05Owned history and rerun retain inputsForeign/missing run cannot disclose inputs
SCR-006/appsAPP-01, APP-03Installed permitted applications launchHidden/disabled/uninstalled entries follow direct-access policy
SCR-011/apps/embed/[appId]APP-01, APP-05Frame context, navigation and scoped requests workForeign app tokens and unsafe targets are denied
SCR-012/chatCHAT-01, SET-01Accessible default agent and new/resumed chat workMissing default and removed agent recover safely
SCR-013/chat/[uuid]CHAT-01, CHAT-02, CHAT-03, CHAT-04, CHAT-05, CHAT-06, CHAT-07Owner and shared reader/writer receive intended capabilitiesRevoked/foreign readers cannot replay or mutate
SCR-014/credentialsCRED-01, CRED-02, CRED-03, CRED-04Owned secrets and connections have correct lifecycleForeign credentials and callback forgery disclose no secret
SCR-015/dynamic-agentsAGT-01, AGT-02, AGT-03, AGT-04Owned agent lifecycle and real execution workViewer/outsider and invalid bindings cannot mutate or execute
SCR-016/insightsOBS-03Permitted aggregates and ranges match fixture eventsRaw foreign activity and privileged aggregates remain private
SCR-017/knowledge-basesKB-01, KB-04Redirect to the first permitted enabled knowledge viewNo-access and disabled features do not leak gated navigation
SCR-018/knowledge-bases/graphKB-04Query, expand, reset and deep-link allowed entitiesDisabled graph, foreign entities and malformed IDs fail safely
SCR-019/knowledge-bases/ingestKB-01, KB-03, KB-05Create, monitor, retry and delete a marker sourceSearch-only users, invalid credentials and duplicates cannot bypass policy
SCR-020/knowledge-bases/mcp-toolsKB-02Create and actually call an allowed search toolRevoked or foreign tool calls are denied
SCR-021/knowledge-bases/searchKB-06Scoped search, citations and source opening workRevoked/deleted sources and malformed filters reveal no content
SCR-022/loginSEC-01, SEC-03Real SSO and safe callback return to requested routeExternal callback, stale state and forged identity cannot authenticate
SCR-023/logoutSEC-01Session clears across UI, API and stale tabsBack navigation and replay cannot restore revoked access
SCR-024/projectsAPP-06Project compatibility entry resolves through installed app contractUnavailable app and foreign project fail safely
SCR-029/schedulesWF-04, AUTO-03Create, validate, observe one invocation, disable and deleteForeign owner, invalid timezone and revoked identity cannot execute
SCR-030/skillsSKL-01, SKL-02Discover, filter, preview and install allowed skillPrivate/foreign and flagged packages cannot install
SCR-031/skills/editorSKL-01, SKL-03Legacy editor resolves to owned workspaceCrafted query and foreign ID cannot edit
SCR-032/skills/gatewaySKL-03Configured source discovery and permitted installs workUnsafe/disallowed sources and non-admin changes are denied
SCR-033/skills/scan-historySKL-03Owned scan status/findings/report links agreeForeign scans and scanner errors do not leak content
SCR-034/skills/workspace/[id]SKL-01, SKL-02, SKL-03Files, tools, versions, scan and unsaved edits workBuilt-in/foreign edits, traversal and stale saves fail safely
SCR-035/unauthorizedSEC-01, RBAC-01Stable denial and safe support navigationNo privileged details; denied APIs remain denied
SCR-036/workflowsWF-01, WF-02, WF-03, WF-05Owned definition and sharing lifecycle persistViewer/outsider and invalid steps cannot mutate/run
SCR-037/workflows/run/[id]WF-01, WF-02, WF-03, WF-05Actual inputs, step handoffs, history and cancellation workForeign/disabled workflow and stale permissions cannot execute
SCR-038/admin/[...path]RBAC-01, SET-02, IDN-01, IDN-02, SVC-01, POL-01Every registered admin destination and subview deep-linksUnknown destinations and privileged direct API requests are denied
SCR-039/agentic-app-routes/[appId]/[...path]APP-03, APP-05, APP-06Owned host route resolves to installed app with query stateRoute collision, disabled app and traversal cannot bypass gateway
SCR-040/apps/[appId]/[[...path]]APP-01, APP-02, APP-06Each installed app virtual route survives refresh/backUnauthorized nested route and foreign resource ID fail safely
SCR-041/autonomousAUTO-01, AUTO-02, AUTO-03, AUTO-04Owned tasks/triggers/runs and pause/delete workForeign task, invalid webhook and disabled automation cannot run
SCR-042/chat/webhooks/[taskId]AUTO-02, AUTO-04Authorized webhook conversation preserves task identityMissing/foreign task or run does not disclose history
SCR-043/credentials/[section]CRED-04, IDN-03Supported connection sections deep-link and preserve stateUnknown section, expired state and foreign identity cannot connect
SCR-044/knowledge-bases/collectionsKB-04, KB-06, POL-02Permitted collections, details and publication state workForeign/private collection and stale approval cannot publish/search
SCR-045/knowledge-bases/ingestion-sourcesKB-03, KB-05Per-source status, incremental ingest and retry workForeign source, invalid connector and partial failure stay visible
SCR-046/native-extensions/[extensionId]/[[...path]]APP-04, APP-05, APP-06Trusted native entry receives correct host props/routesMissing/mismatched module and disabled installation fail safely
SCR-047/settingsSET-01, SET-02Personal defaults and accessible platform sections loadForeign preferences and non-admin platform changes are denied
SCR-048/settings/[section]SET-01, SET-02, NOTIF-01Each registered section deep-links and autosaves correctlyUnknown section, failed save and View as cannot persist unauthorized state

Historical SCR-005, SCR-007 through SCR-010 and SCR-025 through SCR-028 referred to former host-owned app/project routes. Their IDs are reserved; do not reuse them. Verify their candidate-supported redirects or documented removal through APP-06 and the app-owned cases, rather than pretending removed pages still exist.

API-family inventory​

For each family, inventory every current handler and exported method during run planning, including internal/test/admin endpoints. Exercise allowed requests and anonymous, foreign-resource, revoked-access and invalid-input denial as applicable; verify persisted state, audit, pagination, retry and cleanup. Do not treat all methods as equivalent because they share a URL. Internal/test routes must meet their deployment exposure policy, including being unavailable when disabled.

IDAPI familyFeature cases
APIF-001/api/[...path]APP-05, APP-06, SEC-03
APIF-002/api/_testSEC-03, RBAC-01
APIF-003/api/accessFGA-01, RBAC-01
APIF-004/api/adminRBAC-01, IDN-01, IDN-02, SET-02, SVC-01, POL-01, OBS-02
APIF-005/api/agentic-appsAPP-01, APP-03, APP-05
APIF-006/api/aiAGT-05, MODEL-01
APIF-007/api/auditOBS-02
APIF-008/api/authSEC-01, SEC-03, IDN-03
APIF-009/api/authzFGA-01, RBAC-01
APIF-010/api/autonomousAUTO-01, AUTO-02, AUTO-03, AUTO-04
APIF-011/api/catalog-api-keysSKL-04
APIF-012/api/changelogSET-02, QUAL-01
APIF-013/api/chatCHAT-01, CHAT-02, CHAT-03, CHAT-04, CHAT-05, CHAT-06, CHAT-07, RUN-01, RUN-02, RUN-03, RUN-04
APIF-014/api/configSET-01, SET-02
APIF-015/api/credentialsCRED-01, CRED-02, CRED-03, CRED-04
APIF-016/api/dynamic-agentsAGT-01, AGT-02, AGT-03, AGT-04, PROFILE-01
APIF-017/api/feedbackOBS-03
APIF-018/api/filesSEC-04, CHAT-06
APIF-019/api/harness-engineRUN-01, RUN-02, RUN-03, RUN-04, RUN-05, RUN-06, ROUTE-01
APIF-020/api/healthDEP-01, OBS-01
APIF-021/api/integrationsINT-01, SLACK-01, WEBEX-01
APIF-022/api/internalSEC-03, SVC-02, APP-05
APIF-023/api/llm-modelsMODEL-01
APIF-024/api/mcp-serversMCP-01, MCP-02, MCP-03, MCP-04, MCP-05
APIF-025/api/native-extensionsAPP-04, APP-05
APIF-026/api/notificationsNOTIF-01, NOTIF-02, NOTIF-03
APIF-027/api/onboardingSET-03
APIF-028/api/platformSET-02, OBS-01
APIF-029/api/policiesPOL-01, POL-03
APIF-030/api/publication-requestsPOL-02
APIF-031/api/ragKB-01, KB-02, KB-03, KB-04, KB-05, KB-06
APIF-032/api/rbacFGA-01, RBAC-01
APIF-033/api/release-notesSET-02, QUAL-01
APIF-034/api/remote-agentsREMOTE-01, PROFILE-01, RUN-01
APIF-035/api/review-configsPOL-01
APIF-036/api/schedulesWF-04, AUTO-03
APIF-037/api/settingsSET-01, SET-02
APIF-038/api/skill-hubsSKL-03
APIF-039/api/skill-templatesSKL-01, SKL-03
APIF-040/api/skillsSKL-01, SKL-02, SKL-03, RUN-05
APIF-041/api/ticketsINT-01, POL-02
APIF-042/api/userSET-01, IDN-03
APIF-043/api/usersIDN-01, RBAC-01
APIF-044/api/v1SVC-02, MCP-05, RUN-01
APIF-045/api/versionDEP-01, QUAL-01
APIF-046/api/workflow-configsWF-01, WF-02, WF-05
APIF-047/api/workflow-runsWF-02, WF-03, WF-05

Catch-all and application-owned surfaces​

  • Enumerate admin destinations from ui/src/components/admin/workspace/admin-routes.ts, and personal sections from ui/src/components/settings/settings-routes.ts. Test every destination, gate, direct URL and backing method with admin and ordinary identities.
  • Admin coverage includes users/teams/sync, defaults/setup/tour/announcements, agent/runtime/MCP/RAG/skill catalogs, credentials/service accounts, bots, health/metrics/statistics/feedback, access-before-sign-in, AI Review/audit, publication approvals and access operations. Expand nested diagnostics, relationship/tuple viewers and migration controls from the actual candidate.
  • Settings: expand appearance, chat/default agents, notifications, features, developer/account/access views and each permitted platform setting. Verify autosave rollback, independent web/bot preferences and read-only View as.
  • Apps: enumerate installed manifests, selected native/iframe profile, owned routes/API mounts/slots and public/hidden/disabled/service-only modes. Include each app's own test plan and source boundary. Inventory the installed SDLC app's repository/epic/metrics/settings flows only when that app is actually present.
  • Knowledge: collections, ingestion sources, graph, MCP tools and search need their own role gates and direct API checks; a redirect is not a passed search.
  • Skills: Overview, Files, Tools, Versions and Scan include unsaved edits, conflict, built-in read-only behavior, hub/template provenance and scanner decisions.
  • Application-owned feature routes, data models and source-provider flows are inventoried and tested in the application's own repository. This public plan verifies only the host/app contract. Discover app suites through protected run configuration when the requested scope includes external applications.

Cross-cutting assertion dimensions​

Expand applicable dimensions with stable child IDs, for example SCR-013/DIM-02/shared-reader. Record explicit exclusions; do not silently omit negative cases because the positive flow succeeded.

IDDimensionRequired checks
DIM-01AuthenticationReal login, renewal, logout, stale/revoked/invalid sessions, safe callback and cross-origin policy
DIM-02AuthorizationOwner/member/outsider/machine allow/deny; direct URL/API, ID substitution, cross-team/org and revoked cache boundaries
DIM-03Visibility lifecyclePrivate → Team → Global → Private, downgrade/revoke/delete where supported; exact persisted grants, stale-grant absence and effective decisions
DIM-04Input boundariesValid minimum/maximum/Unicode/multiline; required/oversize/duplicate/malformed values, unsafe markup/path/URL and unsupported files
DIM-05ConcurrencyDouble-submit, stale version, update/delete and share/revoke races, idempotent retry, duplicate callback/event/run admission
DIM-06Async recoveryLoading/progress/completion, timeout, partial result, cancellation, 409/422/429/5xx and unavailable dependency
DIM-07NavigationMenu/direct link, query/filters, refresh/back/forward, unknown/disabled destination and unsaved-change escape
DIM-08Data safetyScoped lists/redaction, private metadata, attachment/download ownership, safe links and secret-free evidence
DIM-09AccessibilityKeyboard-only, visible/returned focus, labels/announcements, reduced motion, dialog escape and usable validation
DIM-10Responsive layoutDesktop/narrow viewport, long labels, scrolling and overflow; dialogs/actions/errors remain reachable
DIM-11ObservabilityRequest/run correlation, actor/object, attributable success/failure audit and bounded secret-free logs
DIM-12CleanupManifest-scoped deletion, dependent bindings, grants/checkpoints/jobs, zero residue; preserve failure after manual repair

Detailed numbered case register​

A new or changed assertion must have a permanent ID and enough information to repeat it. Keep the feature group ID, stable child ID, prerequisites, steps, expected result, adapter/location and required evidence. Reuse an existing test when its actual contract matches; record whether it is live, mocked or unit-only. These initial detailed cases anchor security and the previously under-tested runtime/workflow/skill/bot flows. Other groups must be expanded before execution; they remain planning work until their required assertions and adapters are named. Do not renumber existing children when inserting a new assertion.

Assertion IDPrerequisites and stepsExpected result / required evidence
REL-04.01Declared plan with missing/partial results, scoped evidence, cleanup failure, excluded assertions with successful cleanup, and an optional accepted risk: render twiceDeterministic artifacts and recomputed counts/verdicts agree; missing coverage and failed cleanup remain visible; successful cleanup leaves excluded cases NOT_APPLICABLE; see renderer tests
REL-04.02Open locally generated HTML at desktop/narrow viewport; search/filter, expand assertions and download JSONNo external layout dependencies or script errors; readable numbered dispositions, controls and embedded records agree; filtered-out rows stay hidden at both viewport widths
REL-04.03Try private-suite/private-evidence public export, unsafe evidence URLs, untrusted text, inconsistent PASS records, and planned assertions containing result fieldsInvalid export/records reject; text is escaped; planned fields cannot replace validated execution results or mark missing results PASS; no automatic upload/redaction; release summary preserves actual scope
REL-05.01Run release-policy and finalization tests with first/subsequent RC tags, unrelated branch tags, and mocked release APIsGenerated RC delta uses the previous reachable same-series RC or preceding stable; full stable comparison and Helm/image commands use the exact candidate version; retain unit/CI evidence from .github/actions/tests/test_release_policy.py
REL-05.02Exercise required-build success, failure, missing/running/skipped runs, wrong commit, PR/fork completion, existing draft and repeated publicationOnly successful builds for the tag's exact commit permit publication with prerelease enabled and Latest disabled; retries preserve published notes; retain mocked/CI evidence from release-policy and tests/test_release_finalization.py tests
REL-05.03Run stable changelog, stable release-note baseline and image-channel regression tests for stable, RC, dev, hotfix and canary versionsStable history and candidate rollups are preserved; stable notes start at the preceding stable tag even after RC publication; candidates never move image Latest or require a stable blog; retain unit/CI evidence from release-policy and Harness publication tests
REL-05.04Exercise manual dispatch from main/tag/feature refs, hotfix merge-back detection, unavailable Git remotes, padded/large suffix histories and each publishing metadata stepNon-main manual dispatches reject before release credentials; hotfix merge-backs do not prepare or auto-promote stable versions; remote errors stop validation; suffixes compare numerically without overflow; every image metadata step independently protects Latest; retain unit/CI evidence from release-policy tests
REL-05.05Execute the pinned Docker metadata action with synthetic stable, RC, dev, hotfix, canary, manual and PR contexts; check every build, manifest and prebuild metadata configurationAutomatic Latest is disabled in every invocation; RC/dev/hotfix/canary/PR contexts never emit Latest, including variant aliases; stable versions retain their explicit Latest rule; removing the flavor setting reproduces the RC leak; adapter: .github/actions/tests/verify_metadata_action.py, invoked by the Verify actual Docker tags for release channels step in .github/workflows/release-policy-tests.yml after checking out the pinned action bundle; retain offline action-execution and CI evidence
SEC-01.01Real approved ordinary-user session: sign in and open a permitted protected page/APIDisplayed identity and returned subject agree; screenshot and redacted API evidence
SEC-01.02Same session: exercise its actual renewal boundary and revisit a protected APISession renews per deployed contract without changed identity or widened grants; redacted before/after results
SEC-01.03Owned session: log out, revisit protected URL and call protected APIServer and UI enforce logout; no authenticated data reappears through back navigation; API/status and screenshot
SEC-01.04Two approved tabs: revoke/expire fixture session, then replay/read/send from stale tabEvery protected operation denies according to declared policy; stale UI cannot authorize a write; request/stream evidence
SEC-01.05Anonymous/invalid fixture credentials: read and mutate representative protected endpointsContract-appropriate denial or safe login redirect; no persisted mutation or protected payload; API/state evidence
SEC-01.06Approved fixture operations: inspect responses, errors, audit and generated artifactsTokens/secrets remain redacted; record checks and sanitized evidence without copying secret values
RBAC-01.01Disposable subject: CAS grant manage on system_config:platform_settings before and after login promotion, then repeat demotionIndependent grant survives; explicit revoke followed by demotion never restores it. Adapter: ui/src/lib/rbac/__tests__/admin-grant-ownership.test.ts; source/service evidence is distinct from browser acceptance
RBAC-01.02Same subject: CAS revoke while a valid login reason remains, then remove the login reason twiceLogin access remains until demotion; both ownership reasons and effective access end revoked. Same ownership test adapter; retain tuple/ledger and decision evidence
RBAC-01.03Inject existence-read failure, lost write response before/after commit, and ownership-confirmation failure for a new independent grantRows retaining independent_pending do not activate the new reason and block later reconciliation. A lost confirmation response may already have committed ownership: verify persisted state and report pending without assuming rollback; both grant APIs return AUTHZ_WRITE_PENDING, HTTP 503, no automatic retry; audit uses PDP_WRITE_PENDING. Adapters: ownership suite, ui/src/lib/authz/__tests__/{index,write-pending-http}.test.ts; these are fault-injection tests, not live SSO acceptance
RBAC-01.04Approved disposable authenticated identity: grant/revoke across fresh SSO, role promotion/demotion and requests routed to each UI replicaActual login and protected HTTP operations agree with intended ownership without test-driven cache invalidation; capture per-replica decisions. Adapter gap: requires approved real identity/session; module probes do not pass this case
RBAC-01.05Isolated upgrade fixture: include a revoked ownership record whose tuple is absent and an unattributed existing tuple; follow the upgrade audit in the RBAC workflows guideConfirmed stale independent reason is revoked through CAS and cannot return on login/demotion; valid login reasons and unattributed grants remain preserved. Operator adapter: docs/docs/security/rbac/workflows.md; retain sanitized before/after tuple, ledger and audit evidence
RUN-01.01Each enabled runtime with owned agent and allowed model: send a unique marker promptReal selected worker/model completes; runtime binding and persisted response prove execution
RUN-01.02Same conversation: ask for the earlier marker without repeating itCorrect marker follows from retained context; both turn records and stream evidence
RUN-01.03Same runtime with approved harmless tool: require an actual tool callSuccessful tool request/result and caller authorization agree; narrative claims alone fail
RUN-01.04Completed owned chat: reload, navigate away/back and reopen deep linkActual completed turns, tool lineage and immutable runtime binding persist
WF-02.01Owned disposable definition: create two harmless dependent steps with marker inputSave/reload preserves ordered steps, variable references and allowed agent/tool bindings
WF-02.02Saved fixture workflow: execute with marker inputBoth real step outputs and handoff agree with run history; catalog rendering is insufficient
WF-02.03Independent ordinary identity: attempt unauthorized edit/run using fixture IDDeclared owner/team policy enforced by direct APIs; no unauthorized run or changed definition
WF-02.04After workflow run: clean only manifest IDsNo owned definition/run/job residue or stale grants; automatic failures remain visible
RUN-05.01Each enabled runtime with a safe owned marker skill: attach and execute itOutput and runtime trace demonstrate skill instructions actually applied
RUN-05.02Saved baseline skill and agent: restart/reload within approved isolated scope and run againSaved attachment, instructions and tool scope survive upgrade/reconnect without widened access
SLACK-01.01Approved test identity/destination: send labelled DM, then same-thread follow-upReal replies and host history preserve marker/context with one admitted run per turn
SLACK-01.02Approved fixture channel mapping: mention bot, then follow up in its threadCorrect scoped route/identity, delivered reply and persisted conversation binding; record provider message IDs
WEBEX-01.01Approved test identity/destination: send labelled 1:1, then contextual follow-upDelivered replies and host history preserve marker/context without duplicate runs
WEBEX-01.02Approved fixture space mapping: mention bot, then follow up in its threadCorrect scoped route/identity and actual delivery agree with stored host conversation

Record adapter gaps explicitly. This register and inventory do not restore the old Playwright runners, synthetic-cookie helpers or monolithic app endpoints. Port only compatible assertions to current owning suites; track every remaining gap in the run report and feature plan.