deer-flow/contracts/run_event_stream_contract.json
Zeren Wang a58ab484a6
feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash (#5443)
* feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash

Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md
(issue #5160, tracker #5129) in the slice order of the spec's §16.

Slices:
- A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY
  admission pinning (both server-owned sets + worker hoist); latest-only
  request-scoped <project> block via DynamicContextMiddleware
  wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID
  prefix + marker + provenance, never persisted); journal audit
  fingerprints; Instructions tab.
- B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with
  locked check-and-set; hash-qualified immutable shelf storage with
  Paths helpers; upload/list/content/delete-to-trash routes; project
  delete trashes the shelf in-transaction; request-scoped bounded
  <documents> index with honest count/shown + actionable overflow note;
  list_project_documents/read_project_document tools registered only on
  pinned runs; PAT allowlist + drift guards; blocking-IO anchors.
- C: shared thread-upload ingestion service (uploads router refactored to
  parity); POST from-thread with provenance; attach-to-thread with
  lock-staged copy (archived source allowed); read-only thread-files
  view with per-group truncation reporting.
- D: restore (restored/merged/not_found/no_target/content_missing; no
  file moves), purge (continuous row lock across unlink/delete/commit,
  retryable on FS errors), retention sweep (lazy + startup, 24h orphan
  guard, row-side reconciliation never deletes).
- E: Documents tab (shelf + conversation-files browser, provenance,
  archived banner, content-missing rows), /workspace/trash route,
  sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks
  + specs.

Review hardening folded in (10 rounds, all with tests):
- force active shelf content (HTML/XML family) to download; nosniff on
  artifact + content responses; unified unsandboxed-iframe PDF preview
  (fixes the pre-existing Chromium sandbox blank in the artifact viewer)
- scope document trash to the URL project under the document lock
- atomic no-overwrite filename reservation for ALL ingestion (seeded
  claims + os.link commit with suffix retry; same-name re-upload now
  unique-names instead of replacing); hidden staging only, no visible
  placeholders; lease cleanup on setup failure
- serialize conversion under the document lock with post-lock active
  revalidation; drain locked filesystem work on cancellation; preserve
  bytes when an insert's commit state is uncertain (including trashed
  rows)
- original-integrity checks before serving text or cached conversions;
  content_missing surfaced in list responses (UI reads the flag, no
  409-probe); downloads always serve original bytes
- bounded streaming document reads with cached char counts; shelf limits
  declared in middleware release identity
- thread-root confinement for from-thread sources; config fallback
  rejects fractional/infinite values; composer counts staged
  attachments; pending attachments persist until submission or removal;
  in-flight instruction/rename edits survive save refetches; shelf and
  trash pagination; conversation-file and thread-files pages stay
  subscribed to refetches

Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md
contracts, config.example.yaml projects block.

Review follow-ups (head b4807477 → this revision):
- The trash retention sweep is split so repeated lazy triggers stay
  bounded: the indexed expiry purge still runs on every trigger
  (GET /api/trash/documents, POST /api/trash/purge) while the
  O(all rows + all files) reconciliation is throttled to one run per
  user per 15 minutes (process-local, per-user window). The startup
  sweep now runs as a background task instead of blocking gateway
  readiness, and shutdown awaits it (bounded).
- The export scrub (stripInternalMarkers) is fence- and indentation-aware
  like the render path, so a pasted, fenced <project>/<documents> snippet
  survives markdown export while real injected blocks (never fenced) are
  still removed. Fence regexes moved to a dependency-free leaf module to
  avoid the messages↔streamdown import cycle.
- The artifact viewer's PDF iframe no longer carries an added title
  attribute (the upstream e2e contract locates it via :not([title])), and
  the upstream artifact-preview spec now pins the new contract: PDFs
  render unsandboxed, images keep sandbox="".

* fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title

- Shutdown cancelled only the shield around the background startup sweep,
  so an all-users reconciliation that outlived the 5s budget kept walking
  rows and files while the document repo and DB engine were disposed
  underneath it. The wait now lives in `_shutdown_startup_trash_sweep`,
  which cancels the task and drains it before worker exit: the shield
  keeps the wait bounded, the cancel makes it final (CancelledError lands
  at the sweep's next await, and `_run_startup_trash_sweep` only catches
  `Exception`, so nothing swallows it).
- The browser-preview iframe lost `title={getFileName(filepath)}` in the
  previous fix round, leaving the PDF frame without an accessible name
  while its siblings keep theirs. Restore it (WCAG frame titles), assert
  it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]`
  instead of `iframe:not([title])`.

* fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel

`Task.cancel()` returns False when the sweep already finished inside the
window between the deadline firing and the cancel, so the shutdown log
claimed a cancellation that never happened. Branch on that outcome: the
warning stays for a real cancel, a late finish is logged at info, and both
paths still reap the task before worker exit.

* fix(projects): round-4 review — make Empty trash delete what it confirms

`POST /api/trash/purge` only ran the retention sweep, and the sweep's
candidate selection is age-gated, so a freshly trashed document survived
"Empty trash" even though the confirmation promises that every listed
document is permanently deleted. With one trashed row the route answered
`{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps
expired rows before listing, so the visible rows were normally ineligible
for the action by construction.

Empty trash now drives `purge_all_trashed`: the caller's trashed rows
(`list_all_trashed`, no age filter) each go through the same guarded,
row-locked `purge` as the single-document delete — bytes first, then the
row, in one transaction — so a row restored mid-flight is skipped instead of
force-deleted, and an unlink failure rolls that row back and answers 500 with
a retryable message. Retention expiry stays where it was: the sweep's
`purge_candidates` is now the only age-gated selection, and the lazy
retention sweep still runs on the listing and at startup.

Tests: the router suite replaces the retention-gated expectation with the
reviewer's repro (fresh row purged, bytes unlinked, shelf and other users'
trash untouched, a failing unlink stays retryable and 500); a blocking-I/O
anchor drives the new entry point through the offload; the mocked e2e covers
the action end to end; a new real-backend spec performs it against the real
gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE
and the phase-2 design docs (en+zh) state the age-independent contract.
2026-09-16 18:46:18 +08:00

458 lines
18 KiB
JSON

{
"version": 1,
"schema_dialect": "https://json-schema.org/draft/2020-12/schema",
"description": "The current DeerFlow run event stream contract. It freezes existing names and categories, describes producer payloads, and records compatibility rules without changing runtime behavior.",
"compatibility": {
"current_event_names": "frozen",
"consumer_rule": "Consumers must ignore unknown event types, unknown envelope fields, and unknown optional payload or metadata fields.",
"additive_changes": [
"add_event_type",
"add_optional_payload_field",
"add_optional_metadata_field",
"add_envelope_field"
],
"breaking_changes": [
"remove_event_type",
"rename_event_type",
"change_event_category",
"remove_required_field",
"change_required_field_type"
]
},
"legacy_event_aliases": [
{
"event_type": "ai_message",
"canonical_event_type": "llm.ai.response",
"status": "read-only compatibility",
"produced_by_current_runtime": false,
"compatibility_scope": "Category-based message projections and last-visible-AI store queries, including the /messages/page endpoint, recognize this historical name.",
"known_limitations": "The legacy /messages endpoint returns category=message rows but only enriches feedback for llm.ai.response.",
"notes": "New producers must use llm.ai.response."
}
],
"record_schema": {
"type": "object",
"required": [
"thread_id",
"run_id",
"seq",
"event_type",
"category",
"content",
"metadata",
"created_at"
],
"properties": {
"thread_id": {"type": "string"},
"run_id": {"type": "string"},
"seq": {"type": "integer", "minimum": 1},
"event_type": {"type": "string", "minLength": 1, "maxLength": 32},
"category": {"type": "string", "minLength": 1, "maxLength": 16},
"content": {"type": ["string", "object", "array", "number", "boolean", "null"]},
"metadata": {"type": "object"},
"created_at": {"type": "string", "format": "date-time"},
"user_id": {"type": ["string", "null"]}
},
"additionalProperties": true
},
"sequence": {
"scope": "thread_id",
"guarantee": "strictly increasing within a thread",
"jsonl_multi_process_limit": "JsonlRunEventStore only guarantees monotonic assignment within one process; use DbRunEventStore for shared multi-process writes."
},
"categories": {
"trace": "Execution evidence that is excluded from message projections.",
"message": "A candidate message projection. Server and frontend visibility filters still apply.",
"outputs": "Root graph completion output; not an authoritative run lifecycle status.",
"error": "Callback-observed run or chain failure evidence.",
"middleware": "Middleware state-change audit evidence.",
"context": "Identity-only evidence about effective hidden context.",
"subagent": "Subagent lifecycle and persisted step evidence.",
"workspace": "Run-scoped workspace and output file-change evidence."
},
"events": [
{
"event_type": "run.start",
"category": "trace",
"producer": "RunJournal.on_chain_start(parent_run_id=None)",
"content_schema": {
"type": "object",
"required": ["chain"],
"properties": {"chain": {"type": ["string", "null"]}},
"additionalProperties": true
},
"metadata_schema": {
"type": "object",
"required": ["caller"],
"properties": {"caller": {"type": "string"}},
"additionalProperties": true
}
},
{
"event_type": "run.end",
"category": "outputs",
"producer": "RunJournal.on_chain_end(parent_run_id=None)",
"content_schema": true,
"content_description": "Opaque root graph outputs as supplied by LangGraph. Values nested inside the output may have backend-dependent representations when they are not directly JSON serializable.",
"storage_semantics": {
"memory": "Retains the original Python container and nested values.",
"jsonl": "Persists JSON with json.dumps(default=str), so nested non-JSON values are restored as strings.",
"database": "Persists the top-level payload as JSON text with json.dumps(default=str), then restores the JSON container on read; nested non-JSON values remain strings."
},
"metadata_schema": {
"type": "object",
"required": ["status"],
"properties": {"status": {"const": "success"}},
"additionalProperties": true
}
},
{
"event_type": "run.error",
"category": "error",
"producer": "RunJournal.on_chain_error()",
"content_schema": {"type": "string"},
"metadata_schema": {
"type": "object",
"required": ["error_type"],
"properties": {"error_type": {"type": "string"}},
"additionalProperties": true
}
},
{
"event_type": "llm.human.input",
"category": "message",
"producer": "RunJournal.on_chat_model_start() for the first persisted lead-agent HumanMessage",
"content_schema": {
"type": "object",
"required": ["type", "content"],
"properties": {
"type": {"const": "human"},
"content": true,
"additional_kwargs": {"type": "object"}
},
"additionalProperties": true
},
"metadata_schema": {
"type": "object",
"required": ["caller"],
"properties": {"caller": {"const": "lead_agent"}},
"additionalProperties": true
}
},
{
"event_type": "llm.ai.response",
"category": "message",
"producer": "RunJournal.on_llm_end()",
"content_schema": {
"type": "object",
"required": ["type", "content", "tool_calls"],
"properties": {
"type": {"const": "ai"},
"content": true,
"tool_calls": {"type": "array"},
"additional_kwargs": {"type": "object"},
"response_metadata": {"type": "object"}
},
"additionalProperties": true
},
"metadata_schema": {
"type": "object",
"required": ["caller", "usage", "latency_ms", "llm_call_index"],
"properties": {
"caller": {"type": "string"},
"usage": {"type": "object"},
"latency_ms": {"type": ["integer", "null"], "minimum": 0},
"llm_call_index": {"type": "integer", "minimum": 1}
},
"additionalProperties": true
}
},
{
"event_type": "llm.tool.result",
"category": "message",
"producer": "RunJournal.on_tool_end() for ToolMessage or Command(update.messages[])",
"content_schema": {
"type": "object",
"required": ["type", "content"],
"properties": {
"type": {"type": "string"},
"content": true,
"tool_call_id": {"type": ["string", "null"]}
},
"additionalProperties": true
},
"metadata_schema": {"type": "object", "additionalProperties": true}
},
{
"event_type": "llm.error",
"category": "trace",
"producer": "RunJournal.on_llm_error()",
"content_schema": {"type": "string"},
"metadata_schema": {"type": "object", "additionalProperties": true}
},
{
"event_type": "context:memory",
"category": "context",
"producer": "RunJournal.record_memory_context() from DynamicContextMiddleware",
"content_schema": {
"type": "object",
"required": ["content_sha256"],
"properties": {
"content_sha256": {"type": ["string", "null"], "pattern": "^[0-9a-f]{64}$"},
"project_context_revision": {"type": ["string", "null"], "pattern": "^[0-9a-f]{64}$"},
"project_shelf_revision": {"type": ["string", "null"], "pattern": "^[0-9a-f]{64}$"}
},
"additionalProperties": true
},
"metadata_schema": {"type": "object", "additionalProperties": true}
},
{
"event_type": "subagent.start",
"category": "subagent",
"producer": "subagent_run_event(task_started)",
"content_schema": {
"type": "object",
"required": ["task_id", "description"],
"properties": {
"task_id": {"type": "string", "minLength": 1},
"description": {"type": ["string", "null"]}
},
"additionalProperties": true
},
"metadata_schema": {
"type": "object",
"required": ["task_id"],
"properties": {"task_id": {"type": "string", "minLength": 1}},
"additionalProperties": true
}
},
{
"event_type": "subagent.step",
"category": "subagent",
"producer": "subagent_run_event(task_running)",
"content_schema": {
"type": "object",
"required": ["task_id", "message_index", "kind", "text", "truncated"],
"properties": {
"task_id": {"type": "string", "minLength": 1},
"message_index": {"type": "integer", "minimum": 0},
"kind": {"enum": ["ai", "tool"]},
"text": {"type": "string"},
"truncated": {"type": "boolean"},
"tool_calls": {
"type": "array",
"items": {
"type": "object",
"required": ["name", "args"],
"properties": {
"name": {"type": ["string", "null"]},
"args": true,
"args_truncated": {"type": "boolean"}
},
"additionalProperties": true
}
},
"tool_name": {"type": ["string", "null"]}
},
"allOf": [
{
"if": {"properties": {"kind": {"const": "ai"}}},
"then": {"required": ["tool_calls"]}
},
{
"if": {"properties": {"kind": {"const": "tool"}}},
"then": {"required": ["tool_name"]}
}
],
"additionalProperties": true
},
"metadata_schema": {
"type": "object",
"required": ["task_id", "message_index"],
"properties": {
"task_id": {"type": "string", "minLength": 1},
"message_index": {"type": "integer", "minimum": 0}
},
"additionalProperties": true
}
},
{
"event_type": "subagent.end",
"category": "subagent",
"producer": "subagent_run_event(task_completed|task_failed|task_cancelled|task_timed_out)",
"content_schema": {
"type": "object",
"required": ["task_id", "status"],
"properties": {
"task_id": {"type": "string", "minLength": 1},
"status": {"enum": ["completed", "failed", "cancelled", "timed_out"]},
"model_name": {"type": "string"},
"usage": {"type": "object"},
"result": {"type": "string"},
"result_truncated": {"type": "boolean"},
"error": {"type": "string"},
"error_truncated": {"type": "boolean"}
},
"additionalProperties": true
},
"metadata_schema": {
"type": "object",
"required": ["task_id"],
"properties": {"task_id": {"type": "string", "minLength": 1}},
"additionalProperties": true
}
},
{
"event_type": "workspace_changes",
"category": "workspace",
"producer": "workspace_changes.record_workspace_changes()",
"content_schema": {"type": "string"},
"metadata_schema": {
"type": "object",
"required": ["workspace_changes"],
"properties": {
"workspace_changes": {
"type": "object",
"required": ["version", "summary", "files", "limits"],
"properties": {
"version": {"const": 1},
"summary": {
"type": "object",
"required": ["created", "modified", "deleted", "symlink_created", "additions", "deletions", "truncated"],
"properties": {
"created": {"type": "integer", "minimum": 0},
"modified": {"type": "integer", "minimum": 0},
"deleted": {"type": "integer", "minimum": 0},
"symlink_created": {"type": "integer", "minimum": 0},
"additions": {"type": "integer", "minimum": 0},
"deletions": {"type": "integer", "minimum": 0},
"truncated": {"type": "boolean"}
},
"additionalProperties": true
},
"files": {
"type": "array",
"items": {
"type": "object",
"required": [
"path",
"root",
"status",
"binary",
"sensitive",
"size_before",
"size_after",
"sha256_before",
"sha256_after",
"diff",
"diff_truncated",
"diff_unavailable_reason",
"additions",
"deletions",
"symlink",
"symlink_target_before",
"symlink_target_after"
],
"properties": {
"path": {"type": "string"},
"root": {"type": "string"},
"status": {"enum": ["created", "modified", "deleted", "symlink_created"]},
"binary": {"type": "boolean"},
"sensitive": {"type": "boolean"},
"size_before": {"type": ["integer", "null"], "minimum": 0},
"size_after": {"type": ["integer", "null"], "minimum": 0},
"sha256_before": {"type": ["string", "null"]},
"sha256_after": {"type": ["string", "null"]},
"diff": {"type": "string"},
"diff_truncated": {"type": "boolean"},
"diff_unavailable_reason": {"type": ["string", "null"]},
"additions": {"type": "integer", "minimum": 0},
"deletions": {"type": "integer", "minimum": 0},
"symlink": {"type": "boolean"},
"symlink_target_before": {"type": ["string", "null"]},
"symlink_target_after": {"type": ["string", "null"]}
},
"additionalProperties": true
}
},
"limits": {
"type": "object",
"required": ["max_files", "max_scanned_files", "max_file_bytes_for_diff", "max_total_diff_bytes"],
"properties": {
"max_files": {"type": "integer", "minimum": 0},
"max_scanned_files": {"type": "integer", "minimum": 0},
"max_file_bytes_for_diff": {"type": "integer", "minimum": 0},
"max_total_diff_bytes": {"type": "integer", "minimum": 0}
},
"additionalProperties": true
}
},
"additionalProperties": true
}
},
"additionalProperties": true
}
}
],
"dynamic_event_patterns": [
{
"pattern": "middleware:{tag}",
"category": "middleware",
"producer": "RunJournal.record_middleware(tag, ...)",
"known_tags": ["guardrail", "loop_detection", "safety_termination", "skill_activation", "skill_secrets", "tool_promotion", "tool_progress"],
"event_type_schema": {
"type": "string",
"pattern": "^middleware:",
"minLength": 12,
"maxLength": 32
},
"tag_schema": {"type": "string", "minLength": 1, "maxLength": 21},
"content_schema": {
"type": "object",
"required": ["name", "hook", "action", "changes"],
"properties": {
"name": {"type": "string"},
"hook": {"type": "string"},
"action": {"type": "string"},
"changes": {"type": "object"}
},
"additionalProperties": true
},
"metadata_schema": {"type": "object", "additionalProperties": true}
}
],
"known_gaps": [
{
"id": "mixed-event-name-separators",
"status": "frozen for compatibility",
"notes": "Current dot, colon, and bare-word event names remain unchanged. Any future normalization requires a versioned migration or dual-write period."
},
{
"id": "tool-call-intent",
"status": "not first-class",
"notes": "Tool call requests are embedded in llm.ai.response content.tool_calls. llm.tool.result records returned messages, and a missing or timed-out result may leave no dedicated outcome event."
},
{
"id": "terminal-run-status",
"status": "split source of truth",
"notes": "run.end is only a root graph completion marker and always says success. RunRow.status is authoritative for success, error, interrupted, and timeout; worker loss may leave no terminal run event."
},
{
"id": "run-end-backend-serialization",
"status": "backend-dependent opaque payload",
"affected_event_types": ["run.end"],
"notes": "Memory retains nested Python values in root graph outputs, while JSONL and database persistence stringify nested values that are not directly JSON serializable. Consumers must not rely on backend-identical nested run.end content."
},
{
"id": "middleware-coverage",
"status": "partial",
"notes": "Loop-detection, deferred-tool promotion, and tool-progress events cover lead-agent and ordinary task-tool subagent runs. Durable batch subagents have no parent run journal and do not currently emit these events."
},
{
"id": "run-scoped-observation-context",
"status": "manual wiring",
"notes": "Journal attribution, token accounting, and external tracing metadata are still attached manually at several LLM call sites."
}
]
}