deer-flow/backend/docs/RUN_EVENT_STREAM.md

# Run Event Stream

The run event stream is DeerFlow's append-only record of what happened during
an agent run. Producers write through `RunEventStore`; history, debug, subtask,
memory-audit, and workspace-review consumers read projections of the same rows.

The machine-readable contract is
`contracts/run_event_stream_contract.json`. Canonical event names and
categories live in `deerflow.runtime.events.catalog`; conformance tests require
the runtime catalog and JSON contract to match exactly.

## Record Envelope

Every persisted event has these required fields:

| Field | Meaning |
| --- | --- |
| `thread_id` | Thread that owns the event. |
| `run_id` | Run that produced the event. |
| `seq` | Store-assigned sequence, strictly increasing within a thread. |
| `event_type` | Fixed event name or documented dynamic pattern. |
| `category` | Consumer-routing bucket. |
| `content` | Event payload, normally a string or JSON object. |
| `metadata` | Filterable or audit metadata. |
| `created_at` | Timezone-aware ISO-8601 timestamp. |

Backends may return additional fields. `DbRunEventStore`, for example, returns
`user_id` and may add serialization markers such as `content_is_json` to
metadata. Consumers must ignore unknown envelope and metadata fields.

`event_type` is limited to 32 characters and `category` to 16 characters by the
database schema. Catalog-backed definitions enforce the same limits before
writing so they cannot emit values that only the memory or JSONL store accepts.

`seq` is thread-global, not run-local. Memory and database stores assign it
monotonically for their supported deployment modes. JSONL only provides this
guarantee within one process; shared multi-process deployments must use the
database store.

The database store additionally keeps a per-thread high-water mark
(`run_event_thread_seq`) that is independent of the event rows. Deleting a run
therefore never lowers the next allocated `seq`: values are not reused within a
thread, and a consumer whose `after_seq` cursor already advanced past a deleted
run does not miss later events. Thread deletion drops the mark only when no
events remain for the thread across every owner and category; surviving rows
retain the allocation floor.

## Categories

`category="message"` means an event is eligible for a message projection; it
does not guarantee that the row is visible in the UI. Thread-history APIs also
filter middleware model calls, subagent AI responses, and superseded regenerate
runs, and the frontend honors message-level visibility markers such as
`hide_from_ui`. Subagent events remain available through the run-events
endpoint; parent `task` ToolMessages remain in thread history so subtask cards
can restore their terminal status after reload.

All other categories are excluded from message projections and are available
through run-event or specialized APIs:

| Category | Purpose |
| --- | --- |
| `trace` | Execution evidence. |
| `outputs` | Root graph completion output. |
| `error` | Callback-observed failure evidence. |
| `middleware` | Middleware state-change audit evidence. |
| `context` | Effective hidden-context identity. |
| `subagent` | Subagent lifecycle and step history. |
| `workspace` | Workspace/output file-change evidence. |

## Producers

`RunJournal` emits callback-derived events:

| Event type | Category | Producer |
| --- | --- | --- |
| `run.start` | `trace` | Root `on_chain_start()` |
| `run.end` | `outputs` | Root `on_chain_end()` |
| `run.error` | `error` | `on_chain_error()` |
| `llm.human.input` | `message` | First persisted lead-agent human input |
| `llm.ai.response` | `message` | `on_llm_end()` |
| `llm.tool.result` | `message` | `on_tool_end()` |
| `llm.error` | `trace` | `on_llm_error()` |
| `context:memory` | `context` | `record_memory_context()` |
| `middleware:{tag}` | `middleware` | `record_middleware()` |

Current middleware tags are `guardrail`, `loop_detection`,
`safety_termination`, `skill_activation`, `skill_secrets`, `tool_promotion`,
`tool_progress`, and `summarize`. The pattern is intentionally open so new middleware tags
are additive. Because the full event type is limited to 32 characters and
`middleware:` uses 11, a tag must contain 1-21 characters.

`middleware:loop_detection` records transitions into the warned state (first
per call hash or per tool-frequency burst) and each hard stop produced by
`LoopDetectionMiddleware` in lead-agent and ordinary task-tool subagent runs.
Task-tool subagents forward the append to the parent loop because `RunJournal`
and its event store must not cross the isolated-loop boundary. Durable batch
subagents have no parent run journal and do not emit these events. The event's
`action` is `warn` or
`hard_stop`. The `changes` object identifies the detection layer, affected tool
names, observed count, effective threshold, whether the producer was a
subagent, and its agent id when applicable. Tool arguments, prompts, message
content, tool results, and argument-derived hashes are not persisted in this
event.

`middleware:tool_progress` records effective result-quality state transitions
from `ToolProgressMiddleware`: `warn` for ACTIVE → WARNED, `block` for a
transition to BLOCKED, `recover` for WARNED → ACTIVE after a useful result,
and `reset` when a later agent invocation (for example, goal continuation)
deliberately clears a prior WARNED/BLOCKED phase.
The bounded `changes` object contains the tool name, source and destination
phases, consecutive-problem count, normalized result status, error
classification and recovery advice, effective count threshold (null when no
count threshold fired), and
subagent attribution. Tool
arguments, prompts, message content, tool results, and content-derived hashes
are never copied into this event. Producer-supplied metadata is projected onto
fixed error/action vocabularies and a strict boolean-or-null before persistence,
so a custom tool cannot smuggle arbitrary values through its result stamp.
Recorder calls happen after the state lock is released, so a slow custom
recorder cannot stall unrelated tool-state updates. Recorder failures are
fail-open and do not change guard behavior. These events are emitted only when
`tool_progress.enabled` is true (the default is false).

`middleware:tool_promotion` records deferred MCP schemas newly promoted for
the active catalog. `changes.source` distinguishes automatic `routing_hint`
promotion from an explicit `tool_search`; the remaining fields contain sorted
new tool names, their count, `is_subagent`, and the optional subagent
`agent_id`. Explicit-search events observe the final `Command` after the active
skill policy has removed denied schemas. Automatic promotion records the graph
state transition, but promotion never bypasses the independent skill or
authorization execution policies. Repeated model passes or searches that add
no names emit no event. Queries, routing keywords, catalog hashes, tool schemas
and descriptions, arguments, and results are not copied into this middleware
event. Other event types retain their existing payload contracts.

`middleware:summarize` records each successful summary result from
`SummarizationMiddleware` with `action` `summary_result`, including results
reused from the no-op cache. The `changes` object carries the no-op and
cache-skip flags, summary and previous-summary lengths, the number of
summarized messages, and the middleware's cumulative counters; see
[summarization.md](summarization.md). Summary text and message content are
never copied into this event.

Ordinary task-tool subagents forward loop-detection, tool-promotion, and
tool-progress appends to the parent run loop through dedicated recorder context
keys. The loop-bound `RunJournal` itself never enters the isolated subagent
loop.

### Opaque Run Outputs

`run.end.content` is the root graph output and is intentionally opaque. Its
nested representation is not currently identical across storage backends:

- `MemoryRunEventStore` retains the original Python container and nested
  values.
- `JsonlRunEventStore` and `DbRunEventStore` serialize through
  `json.dumps(default=str)`, so nested values that are not directly JSON
  serializable are read back as strings.

Consumers may use `run.end` as completion evidence, but must not depend on
backend-identical nested output values. Normalizing those values would be a
separate runtime compatibility change rather than part of this current-state
contract.

`subagents/step_events.py::subagent_run_event()` maps streamed `task_*` chunks
to persisted events. The worker batches them through `put_batch()`:

| Event type | Source chunk | Required content |
| --- | --- | --- |
| `subagent.start` | `task_started` | `task_id`, `description` |
| `subagent.step` | `task_running` | `task_id`, `message_index`, `kind`, `text`, `truncated`; AI steps add `tool_calls`, tool steps add `tool_name` |
| `subagent.end` | terminal `task_*` | `task_id`, `status`; optional model, usage, result/error, and truncation fields |

Terminal subagent status is one of `completed`, `failed`, `cancelled`, or
`timed_out`.

Malformed lifecycle chunks are not persisted. Every chunk requires a non-empty
string `task_id`; `task_running` additionally requires a non-negative integer
`message_index` and a message object.

`workspace_changes.record_workspace_changes()` writes `workspace_changes` in
category `workspace` when a run changed files. Its string content is a summary;
the structured versioned summary, file list, and limits live in
`metadata.workspace_changes`.

The JSON contract defines required and optional payload fields using JSON
Schema. It is the authoritative field-level reference.

## Consumers

| Consumer | Read path and behavior |
| --- | --- |
| Frontend thread history | `GET /api/threads/{thread_id}/messages/page` scans `list_messages()`, removes middleware rows, subagent AI responses, and superseded regenerate runs, then applies frontend message visibility rules. |
| Per-run message clients | Thread-scoped and stateless run message endpoints call `list_messages_by_run()`. |
| Run debug/audit | `GET /api/threads/{thread_id}/runs/{run_id}/events` calls `list_events()` and supports `event_types`, `task_id`, `limit`, and `after_seq`. |
| Historical subtask cards | Fetch `subagent.step` through the run-events endpoint, filtered and paginated by `task_id`. |
| Memory audit | Filters run events to `context:memory` and compares `content_sha256`; full memory text is not duplicated into the event store. |
| Workspace review | `GET /api/threads/{thread_id}/runs/{run_id}/workspace-changes` projects the latest `workspace_changes` payload. |

Token and cost summaries are not reconstructed by reading event rows.
`RunJournal` accumulates usage while callbacks fire, and the worker writes the
aggregates to `RunRow`.

External Langfuse/LangSmith tracing is a parallel callback pipeline, not a
`RunEventStore` consumer. It is correlated through trace metadata rather than
being derived from these rows.

Evaluation consumers discussed in #4243 are planned rather than present in
this tree. They should read evidence through `list_events()` and treat the
compatibility and terminal-state limits below as part of that integration.

## Compatibility

The existing mixture of dot-separated, colon-separated, and bare-word names is
frozen. This contract documents current behavior; it does not normalize names.
A rename, removal, category change, required-field removal, or required-field
type change is breaking and needs an explicit versioned migration or dual-write
period.

Adding a new event type or optional field is additive. Consumers must ignore
unknown event types and unknown optional fields. Producers must add a catalog
entry, update the JSON contract and this document, and extend the conformance
tests in the same change.

`ai_message` is a read-only legacy alias for `llm.ai.response`. Current
producers never emit it. Category-based message projections and store queries
for the last visible AI message recognize previously persisted alias rows, so
the `/messages/page` endpoint also attaches feedback correctly. The legacy
`/messages` endpoint still returns those rows but only enriches feedback for the
canonical name. Legacy aliases live outside the canonical catalog and must not
be used by new producers.

## Known Gaps

- Tool-call intent is embedded in `llm.ai.response.content.tool_calls`; it is
  not a first-class event. A missing or timed-out result may have no dedicated
  outcome event.
- `run.end.metadata.status` is only a root graph completion marker and is
  always `success`. `RunRow.status` remains authoritative for lifecycle state,
  and worker loss may leave no terminal event.
- Nested non-JSON values in `run.end.content` have backend-dependent
  representations: memory retains Python values, while JSONL and database
  stores read them back as strings.
- Durable batch subagent loop detection, deferred-tool promotion, and
  tool-progress transitions do not emit middleware events because those runs
  have no parent run journal.
- Journal attribution, token accounting, and external tracing metadata still
  depend on manual instrumentation at several LLM call sites.