mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-29 07:56:18 +00:00
cd35363a05
* fix(history): stop dropping user messages that fall outside the loaded page window Two independent paths made a user's own message disappear from a long thread (#4666, #4508, #4363). Both are reproduced by a real two-round run: once the thread passes the 50-row `/messages/page` window AND context compaction fires, the two sources of truth stop overlapping at the head. 1. Middleware-answered tool results never reached the event store. A middleware that short-circuits a tool call (e.g. ReadBeforeWriteMiddleware's blocked write) returns a user-visible ToolMessage, but LangChain never emits `on_tool_end`, so RunJournal never persisted it — the user saw it during the run and it vanished on reload. RunJournal already reconciles final-output tool messages, but only for an `ask_clarification` allowlist. The allowlist is removed; scope stays bounded by the three conditions that actually matter (visible, this run's lead agent, not already persisted), so subagent results still stay in their own step feed. 2. mergeMessages discarded the checkpoint prefix before the first shared anchor. #4065 correctly established that a summarization-rescued early message must not be appended to the tail, and suppressed it instead. That suppression is what deletes the message when the first history page no longer reaches back to it. It is now woven in before the first shared anchor — the one position both the checkpoint and seq-sorted history agree on — so #4065's invariant (never the tail) still holds. A collapsed unloaded gap is recoverable by paging; a dropped message is not. Verified against real captured payloads from the reproducing run: the first user message returns to the transcript. Its exact position is still approximate — after compaction the live window carries too few anchors to place it precisely, which only seq-based ordering can close. Backend: 10809 passed (baseline 10808; same 15 pre-existing failures in browser/crawler community tools). Frontend: 986 passed, typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(events): look up a persisted message's seq by identity Groundwork for placing checkpoint messages in the seq-ordered thread feed (#4666). A checkpoint carries no seq of its own and loses messages to summarization, so once the feed's 50-row page window no longer reaches back to a surviving old message, a client has nothing to place it by. The seq already exists in run_events keyed by the message id — this exposes it without paging the whole feed. `message_identity` is the backend half of the identity rule the frontend applies in `hooks.ts::messageIdentity`: a ToolMessage is keyed by `tool_call_id`, and DynamicContextMiddleware's `X` / `X__user` human copies collapse to one identity. The two halves must stay in sync — a mismatch is silent, degrading placement rather than raising. `get_message_seqs` is implemented for all three stores. Misses are absent from the result rather than an error, so callers degrade to their own placement rule; the earliest seq wins when one identity resolves to several rows, so a re-persisted message keeps the position it first occupied. The DB store decodes rows in Python because `content` is a TEXT column holding a JSON string, not a JSON column — the identity fields cannot be projected in SQL. Nothing consumes this yet; no behavior change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(runtime): carry each persisted message's feed seq on values frames Attaches `additional_kwargs.deerflow_seq` to messages in a root `values` frame that the thread feed already holds, so a client can place a message the checkpoint kept but its loaded history page window no longer reaches (#4666). Nothing is written back to the checkpoint: the seq is added when the frame is serialized and belongs to that frame only. Cost is bounded to frames introducing identities the run has not resolved yet. Messages this run produces are not in the feed while streaming, so they are looked up once, recorded as misses, and never retried — in a real run the only frame that pays for a query is the one where compaction brings older messages back into view. Measured on a reproducing two-round run: 1 lookup across 25 values frames. The stamper is built once per run rather than per `_stream_once`, or a goal continuation would discard the resolved seqs. Subgraph frames are not stamped: a subagent's snapshot is not part of this thread's feed ordering. A lookup failure logs and leaves the frame unstamped rather than failing it — placement is an enhancement and clients fall back to their own ordering rule. Frontend does not read the field yet; no behavior change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(gateway): strip the server-owned message seq from untrusted input `deerflow_seq` is display metadata the Gateway attaches when it serializes a values frame. A client replaying messages (regenerate / edit-and-rerun) would otherwise write it into the checkpoint, where it becomes wrong the moment the thread is forked — a branch re-seeds its feed and reassigns seq (#4380). Joins the existing server-owned key set, so it follows the same trusted-internal rule as the dynamic-context and view-image markers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(frontend): place a checkpoint message by its feed seq, not its nearest anchor Completes #4666. Weaving a compaction-rescued message before the first shared anchor keeps it in the transcript, but not in the right place: after compaction the live window carries too few anchors, and the nearest one can sit deep inside the loaded page window — measured at row 25 of 50 on a reproducing run, which is why the first user turn rendered mid-transcript instead of at the head. Both sides now carry the backend's thread-global seq. `buildVisibleHistoryMessages` copies each row's `seq` onto the message (same shape as the existing `run_id`), and the Gateway stamps it onto `values` frame messages it has already persisted. A live message whose seq is below the loaded window's lower bound is placed ahead of everything on screen rather than before the nearest anchor. A message with no seq — still streaming, so not in the feed yet — keeps the weaving path, since the tail is already its correct position. Verified against the captured payloads of the reproducing run: the first user message goes from absent, to #13 (behind the second question), to #0. Frontend: 988 passed, typecheck + eslint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(frontend): place a pre-window checkpoint message even when no anchor is shared Also #4666. Placing a compaction-rescued message by its feed seq was gated on reaching a shared anchor, because the split ran inside the anchor walk. When the loaded page and the live checkpoint share no identity at all, that walk never runs and the message fell through to `[...canonical, ...live]` — appended after the entire window, the one arrangement #4065 proved wrong, with its seq known the whole time. That is not a corner case. Open an old, already-summarized conversation and send a message: the page on screen is the newest rows from before that turn, while the checkpoint holds the rescued first user turn plus steps of the new run that are not in the feed yet. On a reproducing run the two sides shared zero anchors and the user's own first question rendered at row 50 of 50 — the reported "first message jumps to the bottom". Split `beforeWindow` out of `live` before walking anchors, walk `liveInWindow`, and use it for the no-anchor branch as well, so a message routed ahead of the window is not re-appended at the tail by dedup. Measured on captured payloads of a reproducing run (real gateway, real compaction), first user message position: no shared anchor: row 50 -> row 0, seq order monotonic again shared anchors: row 0 -> row 0 (unchanged) paged to the top: row 0 -> row 0 (unchanged) Regression test verified red-green: reverting the fix fails it with the message rendered after the window. Frontend: 989 passed, eslint + tsc clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(gateway): stamp the message feed seq on checkpoint reads, not only on stream frames Completes #4666. `_MessageSeqStamper` sits on the streaming publish path, so a client that joins a live run learns where a summarization-rescued turn belongs while a client that merely opens the conversation does not — and opening is the common case. `GET /threads/{id}/state` and `POST /threads/{id}/history` returned the checkpoint with no seq at all, so the merge fell back to the nearest shared anchor, which after summarization sits deep inside the loaded page. Reproduced in a browser against a real gateway, on a thread that had already compacted: the user's first question rendered at row 320 of 389, behind the newest question instead of at the head. Both reads showed 0 of 13 messages carrying a seq. That is the reported symptom, still present after the streaming fix. Add `stamp_messages_with_seq`, the request-scoped counterpart of the stamper: everything a checkpoint still holds is already persisted, so one batched lookup resolves the whole list and there is nothing to retry later. Resolve the store through `_optional_run_event_store` rather than `get_run_event_store`, because seq is placement metadata — a deployment without a feed must still be able to read a thread. After the fix, on the same thread in the same browser: 13 of 13 messages carry a seq and the first question renders at the head, ahead of the newest one. Backend: ruff clean, 326 passed across the touched suites. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(harness): move the injected-user-id suffix helpers to utils.messages to break an import cycle message_identity imported strip_injected_user_message_id_suffix from the dynamic-context middleware, closing a cycle (middleware -> deerflow.runtime -> worker -> events -> middleware) that only stayed hidden while an earlier import happened to break it. Define INJECTED_USER_MESSAGE_ID_SUFFIX and the strip helper in deerflow.utils.messages and re-export them from the middleware so existing importers keep working. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(docs): improve formatting and clarity in AGENTS.md and message-merge.test.ts * perf(events): stop the seq scan once every wanted identity is resolved Rows past the last wanted seq can only be re-persisted copies that already lose the earliest-seq-wins tiebreak, so all three stores now break out of the scan (and the db store out of its per-row JSON decoding) once found covers wanted. Matters most for /state and /history reads of long threads, where this lookup runs with no run cache and a typically tiny wanted set. Raised by review on #4696. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(events): share the seq-stamping expression between the two stampers The walrus-plus-merge expression was duplicated verbatim between stamp_messages_with_seq and _MessageSeqStamper.stamp — two counterparts of one rule where silent divergence is the likely failure mode if only one side is edited. Both now call attach_message_seq next to MESSAGE_SEQ_KEY in message_identity.py. The trailing isinstance(message, Mapping) guard was unreachable (a non-Mapping entry already got identity = None) and is gone with the extraction. Raised by review on #4696. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(events): seq stamping survives launch paths without user context The db store's get_message_seqs defaults to user_id=AUTO, which raises when no user is in the contextvar — the first strict-AUTO read ever called from the worker context. On a launch path that never inherits the auth context (e.g. a null-owner scheduled task), stamp()'s except clause swallowed that into a per-frame warning and silently disabled seq stamping for exactly the background runs that need it. The stamper now soft-resolves the user id once at build time — the same rule as the worker's write paths beside it (unset -> no filter) — and passes it explicitly. jsonl/memory stores gain the same user_id kwarg the base list_messages contract already carries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf(events): SQL-prefilter the message seq lookup's candidate rows get_message_seqs scanned and JSON-decoded every message row of the thread: the early exit never fires when a wanted identity is absent from the feed (a message still streaming, or checkpoint-only), and /state / /history reads want the newest messages, so the ascending scan traversed essentially the whole feed — with the content column carrying full tool outputs, that is heavy I/O plus N JSON parses on exactly the long threads this lookup exists for. A LIKE prefilter now keeps that cost in SQL: only rows containing a wanted raw id as a substring are fetched and decoded. False positives are re-checked by message_identity; LIKE wildcards are escaped; an id json.dumps would escape (breaking the verbatim-substring guarantee) falls the whole set back to the full scan rather than silently missing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(agents): sink runtime mechanism docs below the gateway guidance budget Merging main pushed backend/app/gateway/AGENTS.md past its 40KB soft budget (main had left 81 bytes of headroom). Per the nearest-file rule, move the mechanism detail of the message-seq stamping and run-delivery receipt sections — both owned by runtime/ code — into packages/harness/deerflow/runtime/AGENTS.md, leaving the gateway file the REST-surface summary and a pointer. The seq section also documents the stamper's build-time soft user-id resolution and the db store's SQL prefilter from the review follow-ups. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(agents): sink durable-MCP task detail below the backend guidance budget Merging main pushed backend/AGENTS.md past its 24KB module soft budget (main itself is at 24762 after #4848 — this branch adds zero net bytes to the file). Per the nearest-file rule, move the two durable-MCP task runtime bullets' mechanism detail into packages/harness/deerflow/mcp/AGENTS.md, leaving summaries and pointers; this also restores ~2KB of headroom so the next merge does not trip the same wire. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(events): re-ask a message-seq miss once the feed advances The run-scoped stamper cached lookup misses for the whole run. A message this run produces reaches a values frame before RunJournal flushes it, so its first lookup legitimately misses — and the journal persists it moments later, giving it a feed seq the stamper never asks for again. A long run that afterwards rolls past the history page and compacts then carries that message unstamped, back to the approximate anchor placement this stamper exists to replace (#4666). A transient store error had the same permanent effect, since the except clause degrades to an empty result. A miss is now provisional while a hit stays final: RunJournal counts its successful event-store writes as `feed_generation`, and the stamper re-asks a missed identity only once that counter moves. Retrying is therefore bounded by feed writes rather than by frames — the per-frame query the run-scoped cache was built to avoid — and a failed lookup costs one generation instead of the run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
160 lines
38 KiB
Markdown
160 lines
38 KiB
Markdown
### Gateway API (`app/gateway/`)
|
||
|
||
FastAPI listens on port 8001; health: `GET /health`. Set `GATEWAY_ENABLE_DOCS=false` to disable the default `/docs`, `/redoc`, and `/openapi.json` endpoints.
|
||
|
||
Durable MCP notifications use internal Agent runs. Keep their trusted delivery instruction outside the user-input boundary, and frame serialized remote events as untrusted before model invocation. Strict thread existence/ownership admission dead-letters events whose task outlives its deleted chat instead of recreating the thread.
|
||
|
||
CORS is same-origin by default when requests enter through nginx on port 2026. Split-origin or port-forwarded browser clients must opt in with `GATEWAY_CORS_ORIGINS` (exact origins); Gateway `CORSMiddleware` and `CSRFMiddleware` both read that variable so browser CORS and auth-origin checks stay aligned. Those clients also need `CORS_EXPOSED_HEADERS` (`csrf_middleware.py`): run-creating routes return the run's id in `Content-Location`, which is not CORS-safelisted, so JS cannot read it unless it is exposed — and the LangGraph SDK resolves run metadata from that header alone, so withholding it breaks `useStream`'s `onCreated` and thread-gated actions.
|
||
|
||
Browser auth sessions are owned by `app.gateway.auth.session_cookie`. Login accepts a `remember_me` form flag, but the Gateway never stores passwords. `SessionCookiePolicy` persists the `HttpOnly access_token` cookie only for HTTPS/trusted-forwarded HTTPS, direct-host localhost HTTP, or explicit operator opt-in for insecure persistence; public HTTP sandbox URLs degrade to session cookies. Session-creating handlers stamp the final `max_age` on `request.state`; CSRF cookie creation mirrors it so the double-submit pair expires together, including re-issue after password changes and OIDC callbacks. A small `HttpOnly` preference cookie preserves the remember choice across re-issues. Logout clears all auth cookies and suppresses CSRF re-issue on the logout response.
|
||
|
||
Personal Access Tokens (`app.gateway.auth.pat`, `Authorization: Bearer dfp_...`) run as their owning user: an invalid Bearer is a hard 401 with no cookie fallback, which keeps `CSRFMiddleware`'s Bearer skip safe (origin checks still run). Scopes narrow within the allowlisted threads/runs routes; every other authenticated route 403s PAT callers (admin included). PAT management and `/change-password` require session auth; only SHA-256 digests are stored (`0017`).
|
||
|
||
Localhost persistence deliberately reads the direct request `Host` and ignores `Forwarded` / `X-Forwarded-Host`. Scheme and auth-origin reconstruction still consume forwarding headers. The bundled nginx sets `X-Forwarded-Proto`, but preserves an upstream HTTPS value and does not overwrite every forwarded header, so the outer trusted proxy must replace or strip client-supplied forwarding headers before traffic reaches DeerFlow.
|
||
|
||
Standalone local LangGraph Studio is recognized only through the upstream
|
||
`Auth.types.StudioUser` principal type, never by its reusable identity string.
|
||
The type is resolved once at import; an older SDK without it degrades to normal
|
||
owner scoping.
|
||
For that principal's assistant reads/searches, `langgraph_auth.add_owner_filter`
|
||
selects genuine server-registered assistants plus assistants owned by Studio;
|
||
all other resources remain owner-scoped. Assistant create/update handlers make
|
||
both `user_id` and `created_by=user` server-owned, because LangGraph gives
|
||
`created_by=system` privileged ownership semantics during run creation. The
|
||
custom application module in `langgraph_studio.py` is imported before the
|
||
locked in-memory runtime lifespan. At that pre-runtime boundary it derives
|
||
genuine system assistant IDs from the CLI-provided graph registry, removes
|
||
their persisted active/version rows so graph registration recreates them, and
|
||
demotes every other legacy `created_by=system` marker in both active assistants
|
||
and version history. This must happen before runtime 0.30.0 loads and purges
|
||
system-marked rows; a user application lifespan is too late. LangGraph executes
|
||
this file-backed custom app without first registering its module in
|
||
`sys.modules`; keep its annotations eager so dataclass processing remains
|
||
compatible with that loader, and preserve the direct file-loader regression
|
||
test. An empty graph registry or absent persistence file is a no-op, while
|
||
persistence parse/write errors fail startup closed. The harness requires
|
||
in-memory runtime 0.30.0 or newer, and a persisted store containing no
|
||
expected registered assistant row emits a drift warning so changes to
|
||
LangGraph's internal persistence contract are observable. With current
|
||
create/update writes and all legacy versions sanitized, ordinary
|
||
owner-scoped assistant version selection remains enabled.
|
||
|
||
**Routers**:
|
||
|
||
| Router | Endpoints |
|
||
|--------|-----------|
|
||
| **Models** (`/api/models`) | `GET /` - list models; `GET /{name}` - model details |
|
||
| **Features** (`/api/features`) | `GET /` - UI capabilities: hot-reloaded agents, guarded browser, startup MCP tasks, and separate batch repository/worker states so history stays readable without a worker |
|
||
| **Console** (`/api/console`) | Read-only cross-thread observability for the current user (the data layer for an operations dashboard or external monitoring): `GET /stats` - headline counters (runs/threads/agents/tokens/cost); `GET /runs` - paginated run history joined with thread titles (per-run cost); `GET /usage` - zero-filled daily token series + per-model breakdown with spend. Queries `runs`/`threads_meta` directly as a reporting layer (no new `RunStore` methods); requires a SQL database backend — returns 503 on `database.backend: memory`. Real-cost estimation reads optional `models[*].pricing` (`currency`, `input_per_million`, `output_per_million`, `input_cache_hit_per_million`; `ModelConfig` is `extra="allow"`, so no schema change) and prices each run from its `token_usage_by_model` input/output split. Pricing is **cache-aware**: `RunJournal` accumulates prompt-cache hits from `usage_metadata.input_token_details.cache_read` into a sparse `cache_read_tokens` bucket key (also threaded through `SubagentTokenCollector` → `record_external_llm_usage_records`), and cache-hit input tokens are billed at `input_cache_hit_per_million` (omitted → billed at the miss price, a conservative upper bound). All priced models must use one currency; mixed currencies disable cost reporting and leave cost/currency fields null instead of producing invalid aggregates. Legacy rows fall back to run-level totals at `model_name`; unpriced models yield `cost: null` and cost fields are null when no pricing is configured |
|
||
| **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - replace the full config with whole-payload stdio validation; `PATCH /config` - toggle one server while preserving the raw extensions config and validating only an enabled target; both writes reload config and reset the process-local MCP cache |
|
||
| **MCP Tasks** (`/api/threads/{id}/mcp-tasks`) | `GET /` - current user's durable tasks for one owned thread; `GET /{task_id}` - bounded result/input/status-error/cancellation-error detail, including cancellation attempt count, without remote task IDs or driver configuration |
|
||
| **Skills** (`/api/skills`) | `GET /` - list; `GET /{name}` - inspect; `PUT /{name}` - toggle; `POST /install` - install a thread-local .skill archive; `POST /install/upload` - admin-only multipart, authorized before parsing and capped at a 100 MiB file plus 1 MiB framing; `POST /reload` - invalidate process-local cache after trusted filesystem changes |
|
||
| **Subagents** (`/api/subagents`) | Admin managed-worker CRUD and listing. |
|
||
| **Integrations** (`/api/integrations`) | `GET /lark/status` - inspect managed Lark/Feishu CLI integration state, including `sandbox_runtime_mode` / `sandbox_runtime_ready` (whether `lark-cli` will actually be present in the sandbox at chat time); `POST /lark/install` - admin-only install of the official `lark-*` managed skill pack; `POST /lark/config/start` and `/lark/config/complete` - internal first-time Lark connection setup; `POST /lark/config/credentials` - atomically switch the caller's per-user Lark app after validating the new `app_id`/`app_secret` through the official CLI's live tenant-token probe, revoke/remove the previous OAuth tokens, and restore the prior credential tree if the switch fails; `POST /lark/auth/start` and `/lark/auth/complete` - browser device-flow user authorization without terminal access, with optional `domains` / exact `scope` for incremental permission grants. Config and auth flows carry a server-issued, per-user generation persisted under the credential lock; a rejected direct switch leaves the current generation unchanged, stale completions return 409, and browser re-registration uses the same token-clearing/revocation transaction as direct credential switches. |
|
||
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
|
||
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete |
|
||
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; `POST /branches` - branch a completed assistant turn with a replay checkpoint; inherited titles take next-free displayed sibling suffixes, including explicit/renamed ones, while explicit titles stay unchanged. Durable `branch` admission rejects races. Workspace files are not checkpointed, so the branch only best-effort copies the current workspace when branching from the **latest** turn (`workspace_clone_mode="current_thread_best_effort"`); branching from an older/historical turn skips the copy (`workspace_clone_mode="skipped_historical_turn"`) so the branch never inherits files that only exist in a later timeline. Thread-scoped runtime channels (`sandbox`, `thread_data`) are not copied onto the branch: the parent's `sandbox_id` binds path mappings and the release lifecycle to the parent's workspace, so the branch lazily acquires its own sandbox instead. Branch creation also seeds the new thread's run-event feed from the branch checkpoint's visible messages (`history_seed_mode` in the response): the thread feed reads run_events, not checkpoints, so without the seed the inherited history disappears from the UI after the branch's first run (#4380). Seeded rows are grouped into one synthetic run per inherited turn (`branch-seed-{thread_id}-{n}`, a new turn opening at every persisted human message, including an allowlisted hidden `ask_clarification` reply) because `run_id` is a turn identity to the feed's consumers, not a provenance tag: regenerating an inherited answer supersedes that row's whole `run_id` in `GET /messages/page`, so one shared id for the entire seed deleted the complete inherited history on a branch's first regenerate (#4458); `GET /goal`, `PUT /goal`, `DELETE /goal` - read, set, and clear the active thread goal; `POST /compact` - manually summarize older active context into `summary_text` and retain the recent message window, blocked while a run is in flight; unexpected failures are logged server-side and return a generic 500 detail |
|
||
| **Artifacts** (`/api/threads/{id}/artifacts`) | `GET /{path}` - stream regular text and binary artifacts with `FileResponse`, including byte-`Range` 206/416 behavior used by bounded text previews and media seeking; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types. `PUT /{path}` atomically replaces an existing UTF-8 text file under `/mnt/user-data/outputs` when its expected SHA-256 still matches; active runs conflict, and non-mounted sandbox providers receive the same update explicitly. Atomic replacement applies the existing POSIX permission handling when descriptor-based APIs are available and otherwise keeps the platform-native temporary-file permissions (Windows). |
|
||
| **Suggestions** (`/api/suggestions`) | `GET /config` - returns global suggestions config boolean; `POST /threads/{id}/suggestions` - generate follow-up questions; rich list/block model content is normalized and inline reasoning (`<think>...</think>`, including unclosed/truncated blocks from reasoning models like MiniMax-M3) is stripped before JSON parsing |
|
||
| **Input Polish** (`/api/input-polish`) | `POST /` - rewrite a composer draft before it is sent. This is a short authenticated `runs:create` LLM request using `input_polish` config; it does not create a LangGraph run, persist a message, or modify thread state. Shares the non-graph one-shot LLM path (`deerflow.utils.oneshot_llm.run_oneshot_llm`) with the suggestions route so model build + Langfuse metadata + invoke stay in one place; validates the same stripped view of the draft it sends to the model, and preserves literal `<think>` substrings in the rewrite (`strip_think_blocks(truncate_unclosed=False)`) |
|
||
| **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block. Before the first journaled run, seed an empty feed from a checkpoint so legacy checkpoint-only history keeps its order and visibility; skip absent checkpoints or populated feeds. `POST /regenerate/prepare` - prepare clean input + checkpoint metadata for regenerating the latest completed or interrupted assistant answer, carrying the latest non-empty thread title in graph input so resuming an older checkpoint cannot roll back a later manual rename (#4457); `POST /edit-regenerate/prepare` - prepare a checkpoint replay from the latest editable human turn with a replacement user message and edit replay metadata; it carries the current thread title the same way, but only when the replay base already has one — an untitled base belongs to a thread the title middleware has not named yet, so pinning the current title there would keep a name generated from the prompt the edit just replaced; `GET /` - list runs; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/stream` hides action/wait; GET action 405 pre-owner; POST needs `runs:cancel`; `GET /{rid}/messages` - paginated per-run messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /{rid}/workspace-changes` - workspace/output file change summary and optional diffs; `GET/POST /{rid}/artifacts/archive` - receipt manifest / bounded ZIP; `GET /../messages` - legacy thread message array; `GET /../messages/page` - backward thread-global `seq` history page with middleware/subagent-AI/successful-regenerate/edit-replay filtering and page-run-scoped feedback enrichment; subagent AI callbacks remain available through run events while parent `task` ToolMessages stay visible for card restoration; `GET /../token-usage` - aggregate tokens plus an optional `context_usage` percentage. Context usage approximately counts messages from the latest materialized thread state through `build_thread_checkpoint_state_accessor`, so full and delta checkpoint modes expose the same input. The percentage uses the latest run's model and its `context_window`. |
|
||
| **Feedback** (`/api/threads/{id}/runs/{rid}/feedback`) | `PUT /` - upsert feedback; `DELETE /` - delete user feedback; `POST /` - create feedback; `GET /` - list feedback; `GET /stats` - aggregate stats; `DELETE /{fid}` - delete specific |
|
||
| **Runs** (`/api/runs`) | `POST /stream`, `/wait` - stateless runs requiring `runs:create`; optional body `thread_id` is owner-checked. Scheduled-task create/update/resume/trigger also require `threads:write` plus `runs:create`. `GET /{rid}/messages`, `/feedback` - run messages/feedback |
|
||
| **GitHub Webhooks** (`/api/webhooks/github`) | `POST /` - receive GitHub App / repo webhook deliveries. Verifies `X-Hub-Signature-256` against `GITHUB_WEBHOOK_SECRET`; exempt from auth + CSRF because authenticity is enforced by HMAC. The route is fail-closed: mounted only when `GITHUB_WEBHOOK_SECRET` is set, or when explicit dev opt-in `DEER_FLOW_ALLOW_UNVERIFIED_GITHUB_WEBHOOKS=1` is set. Recognized events include `ping`, `issues`, `issue_comment`, `pull_request`, `pull_request_review`, and `pull_request_review_comment`; unknown events return 200 with `handled=false`. Fan-out runtime failures return 503, keeping the delivery recorded as failed for manual/API/scripted redelivery (GitHub does not automatically retry any failed delivery, 5xx included); permanent/non-retryable conditions such as `channels.github.enabled: false`, unknown events, malformed payloads, or unavailable channel service return 200 with a skipped/handled response. |
|
||
| **GitHub Event-Driven Agents** | Custom agents can declare a `github:` block in their `config.yaml` to bind to repos and event triggers. Webhook fan-out publishes one `InboundMessage` per matching binding to the channel bus; `GitHubChannel` routes those messages through `ChannelManager`. The response `dispatch` summarizes matched/fired/skipped agents. |
|
||
|
||
Thread identifiers use the shared `deerflow.utils.thread_id` contract
|
||
`^[A-Za-z0-9_-]{1,64}$`. Caller-provided opaque IDs remain supported; UUIDs
|
||
are generated only for `None`, while explicit empty strings fail validation.
|
||
Gateway creation and state-producing request boundaries, embedded-client
|
||
entry points, filesystem/upload/event-store consumers, scheduled launches,
|
||
and the standalone Provisioner enforce the same contract before persistence
|
||
or workspace initialization. Route-addressable legacy IDs remain accepted by
|
||
pure reads and cleanup/control endpoints; deleting one best-effort removes
|
||
metadata and checkpoints but skips local filesystem cleanup, so the raw value
|
||
is never interpolated into a host path. New runs, workspace/sandbox
|
||
operations, and other state-producing mutations remain blocked.
|
||
|
||
**Message feed seq** (#4666): streaming `values` frames, `GET
|
||
/threads/{id}/state`, and `POST /threads/{id}/history` stamp serialized
|
||
messages with `additional_kwargs.deerflow_seq` so clients can place
|
||
checkpoint-kept messages against the paged feed; the REST reads resolve the
|
||
store via `threads.py::_optional_run_event_store` (a feed-less deployment
|
||
still reads threads), and `services.py::normalize_input` strips the
|
||
server-owned key from client input (#4380). Mechanism and identity rule:
|
||
`packages/harness/deerflow/runtime/AGENTS.md`.
|
||
|
||
**Workspace change review**: `packages/harness/deerflow/workspace_changes/`
|
||
captures a pre-run and post-run snapshot of the thread-owned `workspace` and
|
||
`outputs` directories. `runtime/runs/worker.py` performs the filesystem scan via
|
||
`asyncio.to_thread` and writes a `workspace_changes` event with category
|
||
`workspace` when changes exist. Uploads are intentionally excluded. Text diffs
|
||
are size-limited; binary, large, and sensitive-looking paths are persisted as
|
||
metadata only. Internal process-feedback directories never count as changes:
|
||
the scanner's `EXCLUDED_DIR_NAMES` drops `BROWSER_FRAMES_DIRNAME` (transient
|
||
browser screenshots) and `TOOL_RESULTS_DIRNAME` (the tool-output budget
|
||
middleware's default externalization subdir, `constants.py` is the shared
|
||
source of truth for both writers and the scanner), and the worker threads the
|
||
configured `tool_output.storage_subdir` through the snapshot capture as an
|
||
extra excluded dir name so custom storage locations stay excluded too.
|
||
|
||
**Run delivery receipts**: the worker derives delivery requirements from the
|
||
run's workspace snapshots rather than a client request option (files
|
||
created/modified under `/mnt/user-data/outputs`, minus internal
|
||
process-feedback exclusions) and idempotently persists a run-scoped
|
||
`run.delivery` receipt before the terminal run status; missing or
|
||
unverifiable `present_files` coverage downgrades the run to error, while runs
|
||
without changed outputs keep ordinary chat behavior. Journal mechanics
|
||
(callback attribution, receipt idempotency and retries, orphan recovery):
|
||
`packages/harness/deerflow/runtime/AGENTS.md`. Multi-worker deployments
|
||
require `run_events.backend: db` for shared, ordered delivery events; the
|
||
startup gate rejects process-local memory and JSONL event stores when
|
||
`GATEWAY_WORKERS > 1`.
|
||
|
||
**RunManager / RunStore contract**:
|
||
- LangGraph-compatible run requests validate their supported subset before creating a run. `runtime/stream_modes.py` is the shared backend contract for public stream modes and the worker's `graph.astream` mapping; the public `messages-tuple` mode maps to LangGraph's internal `messages` mode, while public `messages`, `events`, and other unsupported modes are rejected instead of being dropped or replaced with `values`. `app/gateway/run_models.py::RunCreateRequest` is shared by HTTP and internal scheduled launch paths, retains only truthful compatibility defaults for unimplemented options (`if_not_exists="create"` plus `None` placeholders), returns 422 for unsupported values including `on_completion="complete"`, `on_completion="continue"`, and `multitask_strategy="enqueue"`, and forbids undeclared SDK options so fields such as `checkpoint_during` and `durability` cannot be silently discarded. A placeholder must still accept the stock SDK's own default: `langgraph_sdk` drops only `None` from its run payload, so `stream_resumable=False` reaches every request and means "non-resumable", which is what DeerFlow serves — rejecting it 422'd every IM channel run (#4466). `tests/test_run_request_validation.py::test_gateway_accepts_langgraph_sdk_default_payload` pins the real SDK payload against this boundary; channel tests mock the SDK client and cannot catch this class of drift.
|
||
- `RunManager.get()` is async; direct callers must `await` it.
|
||
- The history batch helpers `list_successful_regenerate_sources()`, `list_edit_regenerate_runs()`, and `get_many_by_thread()` default to `user_id=AUTO`: they resolve the request user and fail closed when no user context exists. Migration/admin callers that intentionally need an unscoped read must pass `user_id=None` explicitly.
|
||
- Edit-and-rerun visibility is derived from edit replay runs (`metadata.replay_kind="edit"` plus `regenerate_from_run_id`) by `RunManager.list_edit_replay_visibility()`: the newest attempt for each source run is authoritative. Pending/running/success attempts hide the original source run; failed, timed-out, or interrupted attempts hide only the failed attempt so the original conversation reappears.
|
||
- When a persistent `RunStore` is configured, `get()` and `list_by_thread()` hydrate historical runs from the store. In-memory records win for the same `run_id` so task, abort, and stream-control state stays attached to active local runs.
|
||
- Thread metadata status switches to `running` only after `RunManager.try_start()` succeeds. Pending-cancelled runs therefore skip the old `running` projection, while clients may observe the prior thread status during the short worker-startup window.
|
||
- `cancel()` returns a :class:`~deerflow.runtime.CancelOutcome` enum: `cancelled` (local cancel), `requested` (the non-owning worker durably recorded the first cancellation action for the live owner), `taken_over` (non-owning worker claimed the run because the owner's lease expired — marks it as `error`), `lease_valid_elsewhere` (legacy/custom store lacks the durable request primitive — caller retains the safe 409 + `Retry-After` fallback), `not_active_locally` (heartbeat disabled, preserving the old 409 path), `not_cancellable` (terminal state), or `unknown` (not found in memory or store). `create_or_reject(..., multitask_strategy="interrupt"|"rollback")` persists interrupted status through `RunStore.update_status()`, matching normal `set_status()` transitions.
|
||
- Interrupt/rollback admission registers the replacement before its best-effort persistence of locally interrupted predecessors. If the admitting caller is cancelled during that post-registration await, `RunManager` drains a shielded replacement cleanup before propagating `CancelledError`, including across repeated cancellation. The cleanup normally persists `interrupted`; if that best-effort transition fails, it retries the active-to-`interrupted` store transition strictly and verifies the result with the replacement's captured owner identity. A concurrent peer terminal transition wins and is synchronized back into the local record rather than being overwritten or deleted.
|
||
- Store-only hydrated runs are readable history. In multi-worker mode with heartbeat enabled, cancel on a store-only run records `runs.cancel_action` / `cancel_requested_at` while the owner's lease is live; the first action wins even if a retry later lands on the owner. `RunStore.request_cancel()` and owner completion through `finalize_if_not_cancelled()` are competing active-row CAS operations, so an accepted cancel cannot be overwritten by a later success. `RunStore.renew_lease()` renews and observes the request atomically in the SQL implementation. The owner then executes the normal process-local interrupt/rollback and terminal stream path without transferring the lease. An expired owner is still taken over and marked `error`. `wait=true` and cancel-then-stream use the shared bridge to observe owner finalization; a non-standard process-local bridge returns accepted 202 instead of subscribing to an unreachable stream. In single-worker mode (heartbeat off), store-only runs still return 409.
|
||
- A local worker's `RunRecord.lease_expires_at` is the last durably confirmed ownership deadline. `_renew_leases()` bounds each renewal attempt by that deadline: transient store exceptions remain retryable while it is valid, but an exception or blocked call that reaches expiry sets the process-local `ownership_lost` fence, raises `abort_event`, and cancels the run task. Successful renewals collect durable cancellation actions; after all local renewals have been attempted, heartbeat only signals the corresponding process-local tasks, leaving status writes and rollback cleanup to the worker finalization path. Fenced workers do not perform subsequent journal/delivery-receipt, progress/completion/status, checkpoint/thread-metadata, or `on_run_completed` writes; the peer recovery path owns the terminal receipt. `RunStore.update_run_completion()` also refuses to replace a different terminal status, closing the peer-takeover/late-finalization race. `grace_seconds` delays peer reclamation for clock skew but is not extra execution time for an owner that can no longer confirm its lease. Already-committed remote tool side effects remain outside this local cancellation boundary.
|
||
- Startup/orphan reconciliation must claim stale active rows with `RunStore.claim_for_takeover()`, not a plain `update_status()`. The final claim re-checks `status` and lease expiry atomically, so a heartbeat renewal between the candidate scan and the recovery write keeps the run active.
|
||
- Run admission and independent writes are first-class thread operations. `runs.operation_kind` distinguishes `run` from `checkpoint_write`, `artifact_write`, `artifact_archive`, `branch`, and `delete`; every active kind shares the durable active-thread uniqueness constraint. New operation kinds must go through `RunStore.create_thread_operation_atomic()` and `RunManager.reserve_thread_operation()` rather than adding another lock or metadata marker. Live and lease-less reservations are non-interruptible; an expired leased reservation can be reclaimed immediately by interrupt/rollback admission without waiting for orphan reconciliation. Lease-less rows stay fail-closed because the store cannot distinguish a stale row from a live writer in another heartbeat-disabled worker; a rare failed delete therefore requires startup reconciliation, and heartbeat-disabled multi-worker deployment remains unsupported. Reservation bodies are attached to their caller task so loss detected by lease renewal cancels the writer before it can continue after takeover; the context manager translates that lease-loss cancellation to `ConflictError` after cleanup so Gateway mutation routes return a retryable 409 instead of dropping the HTTP request. The cleanup scope begins immediately after durable admission, including the await that attaches the caller task, so cancellation cannot strand a locally renewed pending reservation. A failed renewal is revalidated under the manager lock before cancellation; if the reservation completed and unregistered while the store update was in flight, its request task must not be cancelled after the write. Reservations are excluded from run history/reporting and from run-only helpers such as `list_by_thread()` and `has_inflight()`, release uses the captured owner rather than ambient user context, and local cleanup still runs when the best-effort store delete fails. `RunStore.create_run_atomic()` remains a deprecated compatibility shim for external stores that only admit normal runs; new stores must implement `create_thread_operation_atomic()` to support internal operation kinds.
|
||
- Gateway checkpoint mutations outside run execution must use `services.reserve_checkpoint_write()`, which composes the process-local thread lock with the durable `checkpoint_write` reservation. Manual compaction, `POST /threads/{id}/state`, and both goal mutation routes (`PUT` / `DELETE /threads/{id}/goal`, including creation of a missing goal checkpoint) use this boundary, so an existing run blocks the write and the reservation blocks new reject/interrupt/rollback runs across workers.
|
||
- `POST /wait` (both thread-scoped and `/api/runs/wait`) drains the stream bridge via `wait_for_run_completion()` instead of bare `await record.task`, so it honours the run's `on_disconnect` setting and cancels the background run on real client disconnect rather than returning a stale checkpoint (issue #3265).
|
||
- Memory and Redis `StreamBridge` implementations retain only `stream_bridge.queue_maxsize` data events. A syntactically valid `Last-Event-ID` older than the retained watermark, or a live subscriber that falls behind it, yields `StreamGap` before any partial replay. `sse_consumer` maps that control item to an id-less SSE `gap` payload (`stream_replay_gap`) and intentionally leaves the run active; internal `/wait` consumers resume from its latest retained ID because they only need terminal completion. Redis checks bounds plus the non-blocking read in one transaction, using blocking `XREAD` only as a wake-up before repeating the atomic snapshot. For a no-cursor subscriber that established a wait on an empty stream, the first wake response remains provisional until that next snapshot verifies its tail is still retained; this closes the pre-first-delivery trimming window without changing malformed-cursor live tailing. The correctness tradeoff is one three-command snapshot pipeline per poll plus the blocking wake round trip while idle. Malformed cursor behavior remains backend-specific. Memory treats a syntactically numeric cursor below its watermark conservatively as a gap even when the evicted timestamp can no longer be verified; unknown ids at or above the watermark retain the legacy replay-from-earliest policy.
|
||
- Redis `StreamBridge` keys use a rolling retained-buffer TTL (`stream_bridge.stream_ttl_seconds`, refreshed on `publish()` / `publish_end()`) as a leak safety net, not as a run timeout. Startup and lease-driven periodic orphan recovery share one Gateway stream-terminalization path: after `RunManager` durably marks a run `error` with `stop_reason=orphan_recovered`, Gateway publishes `END_SENTINEL` and schedules stream cleanup. The periodic store scan, per-row status writes, and Gateway callback run as one supervised single-flight task, so a slow pass is skipped at the next interval instead of piling up or pausing the sole lease-renewal loop. Store retries have bounded attempts/backoff; an individual operation still relies on the database driver/pool timeout. `RunManager.shutdown()` gives active user runs priority within its shared deadline, then drains or cancels orphan recovery. Gateway tracks delayed recovered-stream cleanups and converts unfinished delays to immediate deletes before closing the bridge; the Redis TTL remains the outage safety net. Only startup recovery, before the runtime yields to requests, projects the latest affected thread to `error`; periodic recovery deliberately avoids that non-atomic projection because `ThreadMetaStore` has no `latest_run_id` conditional-update contract. Store-only SSE and `/wait` consumers wait for the bridge's real END marker after an ordinary durable terminal status, because status persistence can precede tail events. The explicit `orphan_recovered` signal is the only heartbeat fallback: its publisher is known to be gone, so it supplies the liveness boundary if END publication fails or the retained key expires. Malformed `Last-Event-ID` reconnect values live-tail new Redis events rather than replaying the retained buffer. Keep cross-component recovery orchestration in Gateway through the generic `RunManager.on_orphans_recovered` callback; do not introduce a harness-to-app dependency. Callback failure warnings include every recovered `run_id` so operators can identify rows whose Gateway-side terminalization needs inspection.
|
||
- Thread-scoped run creation accepts `checkpoint` / `checkpoint_id`; Gateway validates the checkpoint belongs to the request thread before writing `checkpoint_id` / `checkpoint_ns` into `config.configurable` for LangGraph branching. In `delta` checkpoint mode the worker rewrites that fork into a linear head write before the graph starts (see "A delta-mode run cannot fork" under Checkpoint Channel Modes), because delta state for a fork replays the abandoned sibling's writes.
|
||
- Thread-scoped Gateway runs evaluate an active `ThreadState.goal` after the visible turn completes. `runtime/goal.py` asks a non-thinking evaluator model to judge only visible conversation evidence and return a typed blocker; the evaluator model is created once per run and reused across hidden continuation checks. The evaluator runs after the graph root's tracing scope has already closed, so `create_goal_evaluator_model`/`evaluate_goal_completion` attach their own model-level tracing callbacks (`attach_tracing=True`) and inject Langfuse trace metadata (`thread_id`/`user_id`/`deerflow_trace_id`) directly onto the `ainvoke` call — the same standalone-caller pattern as `oneshot_llm.run_oneshot_llm` and `MemoryUpdater` (see Tracing System below). Satisfied goals are cleared; every non-satisfied evaluation — continuable or stand-down — is persisted with `last_evaluation` (the blocker, reason, and evidence summary; outcomes that stop the loop additionally record a `stand_down_reason` for observability), but only `goal_not_met_yet` evaluations are streamed as hidden `HumanMessage` continuations, and only when a durable assistant end-of-turn checkpoint exists, the run has not been aborted, the thread did not change during evaluation, and the no-progress breaker has not fired. The continuation cap is 8 — a hard maximum in the `0`–`8` range; callers requesting more are clamped (`set_goal`/TUI) or rejected with 422 (`PUT /goal`). The no-progress breaker keys on the latest visible assistant evidence (not the evaluator's free-text reason, which an LLM rewords every turn), so two consecutive continuations that add no new visible assistant output stop the loop after 2 attempts. Model-response cleanup helpers such as think-block stripping and code-fence stripping live in `deerflow.utils.llm_text` so `runtime/goal.py` and Gateway suggestion parsing share the same JSON-prep behavior.
|
||
- Run event stream changes must keep producer code, `deerflow/constants.py`, `runtime/events/catalog.py`, `contracts/run_event_stream_contract.json`, `backend/docs/RUN_EVENT_STREAM.md`, and `tests/test_run_event_stream_contract.py` in sync. The dependency-free constants module owns the persisted envelope limits (`event_type` 32 characters, `category` 16) and cross-layer workspace event identity; the catalog owns validated runtime definitions and categories. Dynamic middleware tags are limited to 21 characters after the `middleware:` prefix. The JSON contract owns payload schemas, backend-specific storage semantics, legacy aliases, and compatibility rules; conformance tests require both views and all producer groups to agree. `run.end.content` remains opaque and may retain nested Python values in memory while JSONL/database stores stringify non-JSON nested values, so consumers must not assume backend-identical nested output representations.
|
||
|
||
Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runtime, all other `/api/*` → Gateway REST APIs.
|
||
|
||
**Branch/regenerate checkpoint invariant**: `app/gateway/checkpoint_lineage.py`
|
||
walks `parent_config` rather than globally ordered checkpoint history so replay
|
||
anchors stay on the selected lineage after regenerations create sibling branches.
|
||
New conversation branches persist the pre-user replay anchor before their visible
|
||
head through the state mutation graph, which preserves materialized state in both
|
||
full and delta checkpoint modes. Only an explicitly absent legacy parent link may
|
||
use chronological compatibility lookup; cycles, dangling links, and depth-limit
|
||
exhaustion fail closed. Existing single-checkpoint branches are never repaired by
|
||
copying a raw checkpoint because delta state is not self-contained in one tuple.
|
||
Both lookups additionally require the replay base to be a **settled** checkpoint
|
||
(`has_pending_tasks` — no scheduled `next` tasks). A checkpoint with pending tasks
|
||
is a mid-run snapshot: resuming from it replays the writes of the node that was
|
||
about to run. Message ids alone cannot exclude those, because middleware may
|
||
rewrite a message's id inside the run that produced it — `DynamicContextMiddleware`
|
||
moves the first user turn to `{id}__user` and gives `{id}` to the injected
|
||
reminder, so every checkpoint written before it holds the same prompt under an
|
||
unmatched id. Selecting one of those re-added the original prompt *after* the
|
||
edited one, and the model answered the question the edit was replacing (#4531).
|
||
`next` is not derivable on the degraded raw-checkpoint read path, which reports no
|
||
tasks; absence of evidence stays permissive there rather than failing closed.
|
||
Edit replay resolves its base through the same lineage-first path as regenerate;
|
||
it must pass `head_checkpoint` or it silently degrades to the chronological scan
|
||
that cannot tell sibling branches apart.
|