deer-flow/backend/app/gateway/AGENTS.md

Gateway API (app/gateway/)

Reject external sandbox, thread_data, viewed_images, goal_outcome. Non-null goals: valid text, int counters/limits >=0, no Overwrite; keep sparse data.

Studio retains sanitized creation metadata.

HTTP and run metadata creation share goal_thread_lock. Lock in detached _ensure_thread_metadata; preserve await-free attachment and cancel/drain ownership. Its 5s budget covers admission and I/O: timeout fails strict starts; others omit incarnation (MCP fails closed). Tests: test_thread_create_run_metadata_race.py.

Capability Center's business adapter validates bundled credentials and creates an MCP connection. The MCP API permits only its isolated launcher and credential environment keys; reject arbitrary commands and manifest-policy bypasses. Keep credentials out of tool schemas and preserve admin checks, masked atomic edits, and MCP cache reloads.

Memory shutdown resolves hot-reloaded config/backend, flushes and closes in one await_drained operation. Resolve config off-loop inside the best-effort handler so malformed edits cannot abort teardown. Cancellation waits for owned workers. The host sets no close timeout: close() must be quick or internally bounded; budget it, flush timeout and other hooks within pod grace. Drain extension stop_services() across cancellation before later resources unwind; each service stop() has a 30-second asyncio.timeout.

Routers with persistent writes drain through app.gateway.persistent_writes.run_drained_write.

lifespan first holds hold_sandbox_shutdown_ownership(). After langgraph_runtime drains runs and extensions, close MCP (bounded, isolated), then drain shutdown_sandbox_provider() off-loop in finally, even on cancel. Never allocate for cleanup; log failures, stay retryable.

Retention holds durable thread admission and the checkpoint lock before checkpoint reads. Inspect root head/parent and leaf writes only; protect selectors, require equal channel versions and invalidate caches before deletion. No scans, stats or blob GC; failures stay non-fatal.

conversation_access.py grants the opt-in read-only tool only for explicit conversation_references with effective runs:read; never infer grants from messages or checkpoints. Pass the callback through RunContext, not serialized config, and let the worker inject and clear it. conversation_reader.py shares HTTP transcript visibility/pagination without a Request; callers own authorization. Exclude non-text/internal content. RunCreateRequest lifts SDK-compatible context.conversation_references into its top-level field before validation, then removes it from context; supplying both is 422. This shares bounds/error locations and keeps merge_run_context_overrides clear. Admission and /api/features use conversation_references_enabled() as the sole configured predicate.

FastAPI listens on 8001. GET /health is liveness; GET /health/ready concurrently probes startup snapshots of ORM, checkpointer/Store and stream bridge (StreamBridge.ping()) under one deadline and per-kind gates. Unreachable or unresolved gating backends return 503; memory reports not_configured; provisioner /health is report-only. See health.py. GATEWAY_ENABLE_DOCS=false disables /docs, /redoc and /openapi.json.

build_run_config() resolves the default LangGraph super-step budget from the hot-reloaded top-level recursion_limit setting. A valid request-level value takes precedence; invalid values fall back to that default, and max_recursion_limit caps both sources.

Durable MCP notifications use internal Agent runs. Keep their trusted delivery instruction outside the user-input boundary, and frame serialized remote events as untrusted before model invocation. Strict thread existence/ownership admission dead-letters events whose task outlives its deleted chat instead of recreating the thread. start_run marks run-manager admission conflicts with BusyThreadConflict; the notification launcher retries only that typed 409 and treats other 409 responses as permanent.

CORS defaults to same-origin behind nginx on 2026. Split-origin and port-forwarded browser clients must set exact GATEWAY_CORS_ORIGINS, shared by CORSMiddleware and CSRFMiddleware. Expose Content-Location through CORS_EXPOSED_HEADERS (csrf_middleware.py): run-creating routes put the run ID there, and the LangGraph SDK uses only that header for run metadata. Without exposure, JS cannot read it, breaking useStream.onCreated and thread-gated actions.

Browser sessions belong to app.gateway.auth.session_cookie. Login's remember_me flag never stores passwords. SessionCookiePolicy persists HttpOnly access_token only for HTTPS/trusted-forwarded HTTPS, direct-host localhost HTTP, or explicit operator opt-in; public HTTP otherwise uses session cookies. Session creators set final max_age on request.state; CSRF mirrors it across login, password changes and OIDC callbacks. A small HttpOnly preference cookie preserves the choice. Logout clears auth cookies and suppresses CSRF re-issue.

OIDC redirects retain existing endpoint query bytes (RFC 6749 section 3.1). Malformed IdP bodies raise OIDCError (502/sso_failed). tests/test_oidc_auth.py pins query, state-cookie, nonce and PKCE agreement.

PATs (app.gateway.auth.pat, Authorization: Bearer dfp_...) act as their owner. Invalid Bearers return 401 without cookie fallback, keeping CSRFMiddleware's Bearer skip safe; origin checks still run. Scopes narrow allowlisted thread/run/ project routes, including POST /api/threads/{id}/move; other authenticated routes reject PATs with 403, even for admins. PAT management and /change-password require session auth. Store only SHA-256 digests (0017).

Thread→project membership is written only by thread creation (POST /api/threads with validated project_id), branch creation (inherit the source project; archived/deleted projects become unassigned rather than failing), and explicit POST /api/threads/{id}/move. Run admission pins resolved project context read-only and never changes membership. Server-reserved deerflow_project_id metadata exposes threads_meta.project_id read-only and is stripped from client writes.

Localhost persistence trusts direct Host, not Forwarded/X-Forwarded-Host. Scheme/auth-origin reconstruction still uses forwarding headers. Nginx sets X-Forwarded-Proto but preserves upstream HTTPS and some other forwarded headers; the outer trusted proxy must replace/strip client copies before DeerFlow.

Recognize standalone Studio only by upstream Auth.types.StudioUser, never its reusable identity string. Resolve the type once at import; older SDKs without it retain normal owner scoping. For its assistant reads/searches, langgraph_auth.add_owner_filter admits genuine server-registered assistants and Studio-owned rows; other resources stay owner-scoped. Assistant create/update forces server-owned user_id and created_by=user: LangGraph treats created_by=system as privileged run ownership. The custom application module in langgraph_studio.py is imported before the locked in-memory runtime lifespan. At that pre-runtime boundary it derives genuine system assistant IDs from the CLI-provided graph registry, removes their persisted active/version rows so graph registration recreates them, and demotes every other legacy created_by=system marker in both active assistants and version history. This must happen before runtime 0.30.0 loads and purges system-marked rows; a user application lifespan is too late. LangGraph executes this file-backed custom app without first registering its module in sys.modules; keep its annotations eager so dataclass processing remains compatible with that loader, and preserve the direct file-loader regression test. An empty graph registry or absent persistence file is a no-op, while persistence parse/write errors fail startup closed. The harness requires in-memory runtime 0.30.0 or newer, and a persisted store containing no expected registered assistant row emits a drift warning so changes to LangGraph's internal persistence contract are observable. With current create/update writes and all legacy versions sanitized, ordinary owner-scoped assistant version selection remains enabled.

Routers:

Router	Endpoints
Managed Models (/api/managed-models)	Admin-only GET/PUT with revisions; streaming POST /test (DeepSeek thinking off). Clients/storage off-loop; YAML names reserved; keys private.
Models (/api/models)	GET / - list models; GET /{name} - model details
Features (/api/features)	GET / - UI capabilities: hot-reloaded agents, guarded browser, startup MCP tasks, separate batch repository/worker states so history stays readable without a worker, conversation_references (read_conversation configured, per-run reference cap), knowledge scope selection, scheduled_tasks, and thread_activity
Thread Activity (/api/thread-activity)	GET /?cursor= - per-user feed over runs.change_seq: no cursor seeds; lists threads of the caller's server-originated runs, passes interactive rows; read_version. POST /api/threads/{id}/read and search unread use thread_read_markers. SQL only
Knowledge (/api/knowledge)	Authenticated, capability-driven read-only retrieval catalog and per-assistant /capabilities; provider adapters own resource authorization and mapping. Unsupported filters fail admission/execution without widening scope; management stays outside core
Console (/api/console)	Read-only cross-thread observability for the current user: GET /stats - headline counters (runs/threads/agents/tokens/cost); GET /runs - paginated run history joined with thread titles (per-run cost); GET /usage - zero-filled daily token series + per-model breakdown with spend. Queries runs/threads_meta directly (no new RunStore methods); SQL only (503 on database.backend: memory). Real-cost estimation reads optional models[*].pricing (currency, input_per_million, output_per_million, input_cache_hit_per_million; ModelConfig is extra="allow", so no schema change) and prices each run from its token_usage_by_model input/output split. Pricing is cache-aware: RunJournal accumulates prompt-cache hits from usage_metadata.input_token_details.cache_read into a sparse cache_read_tokens bucket key (also threaded through SubagentTokenCollector → record_external_llm_usage_records), and cache-hit input tokens are billed at input_cache_hit_per_million (omitted → billed at the miss price, a conservative upper bound). Mixed currencies null cost/currency fields. Legacy rows fall back to run-level totals at model_name; unpriced models yield cost: null and cost fields are null when no pricing is configured
MCP (/api/mcp)	GET /config - raw/masked; PUT /config - bulk; PATCH /config - toggle; POST /config/servers - add; PUT /config/server - replace; DELETE /config/servers/{server_name:path} - bodyless. POST /cache/reset. Validate expanded, save raw; reload/reset; invalid -> 400.
MCP Tasks (/api/threads/{id}/mcp-tasks)	GET / - current user's durable tasks for one owned thread; GET /{task_id} - bounded result/input/status-error/cancellation-error detail, including cancellation attempt count, without remote task IDs or driver configuration
Skills (/api/skills)	GET / - list; GET /{name} - inspect; PUT /{name} - toggle; POST /install - install a thread-local .skill archive; POST /install/upload - admin-only multipart, authorized before parsing and capped at a 100 MiB file plus 1 MiB framing; POST /reload - refresh, then publish the shared skills reset marker
Subagents (/api/subagents)	Admin managed-worker CRUD and listing.
Integrations (/api/integrations)	GET /lark/status - inspect managed Lark/Feishu CLI integration state, including sandbox_runtime_mode / sandbox_runtime_probed / sandbox_runtime_ready (whether lark-cli will be present in the sandbox at chat time); POST /lark/install - admin-only install of the official lark-* managed skill pack; POST /lark/config/start and /lark/config/complete - internal first-time Lark connection setup; POST /lark/config/credentials - atomically switch the caller's per-user Lark app after validating the new app_id/app_secret through the official CLI's live tenant-token probe, revoke/remove the previous OAuth tokens, and restore the prior credential tree if the switch fails; POST /lark/auth/start and /lark/auth/complete - browser device-flow user authorization without terminal access, with optional domains / exact scope for incremental permission grants. Config and auth flows carry a server-issued, per-user generation persisted under the credential lock; a rejected direct switch leaves the current generation unchanged, stale completions return 409, and browser re-registration uses the same token-clearing/revocation transaction as direct credential switches.
Memory (/api/memory)	GET / - memory data; POST /reload - force reload; GET /config - config; GET /status - config + data
Uploads (/api/threads/{id}/uploads)	POST / - upload/convert PDF/PPT/Excel/Word; upload/shelf names share normalize_filename before staging; non-mounted sync lease ends when its last holder exits; GET /list; DELETE /{filename} - regular files, symlinks 404, keeps converted .md (#5672)
Threads (/api/threads/{id})	DELETE / - remove DeerFlow-managed local thread data after LangGraph thread deletion; POST /branches - branch a completed assistant turn with a replay checkpoint; inherited titles take next-free displayed sibling suffixes, including explicit/renamed ones, while explicit titles stay unchanged. Durable branch admission rejects races. Workspace files are not checkpointed: only a branch from the latest turn best-effort copies the workspace (workspace_clone_mode="current_thread_best_effort"); older turns skip it ("skipped_historical_turn"), so no later-timeline file leaks in. sandbox/thread_data channels are not copied (parent sandbox_id binds paths/release); the branch lazily acquires its own sandbox. Branch creation also seeds the new thread's run-event feed from the branch checkpoint's visible messages (history_seed_mode in the response): the thread feed reads run_events, not checkpoints, so without the seed the inherited history disappears from the UI after the branch's first run (#4380). Seeded rows get one synthetic run per inherited turn (branch-seed-{thread_id}-{n}, a new turn at every persisted human message, including an allowlisted hidden ask_clarification reply): run_id is a turn identity, and regenerating an inherited answer supersedes its whole run_id in GET /messages/page, so one shared seed id deleted all inherited history (#4458); GET /goal, PUT /goal, DELETE /goal - read, set, and clear the active thread goal; POST /compact - summarize older active context, deriving memory policy and bucket from the state-producing checkpoint rather than request agent_name, and block while a run is in flight; 409 unless idle; unexpected failures return a generic 500 detail
Artifacts (/api/threads/{id}/artifacts)	GET streams byte ranges; HTML/XML/XSL/+xml (also .skill members) and ?download=true force attachments. PUT atomically replaces existing UTF-8 outputs on matching SHA-256; active runs conflict. Non-mounted sync uses a request lease and drains remote/local commit-or-rollback across cancellation before releasing reservations/leases. resolve_outputs_confined_path normalizes .. and confines host paths against traversal/symlinks. Preserve POSIX ownership/mode where supported; Windows keeps temp-file permissions. Digest cache: path/dev/ino/ctime_ns/mtime_ns/size; identity invalidates same-size/mtime replacements. Large regular files use stat-prefixed ETags; If-Range requires matching ETags. Windows ctime is creation time. Keep cache reuse, size limits and worker hashing.
Suggestions (/api/suggestions)	GET /config - returns global suggestions config boolean; POST /threads/{id}/suggestions - generate follow-up questions; rich list/block model content is normalized and inline reasoning (<think>...</think>, including unclosed/truncated blocks from reasoning models like MiniMax-M3) is stripped before JSON parsing
Input Polish (/api/input-polish)	POST / - rewrite a composer draft before it is sent. This is a short authenticated runs:create LLM request using input_polish config; it does not create a LangGraph run, persist a message, or modify thread state. Shares the non-graph one-shot LLM path (deerflow.utils.oneshot_llm.run_oneshot_llm) with the suggestions route so model build + Langfuse metadata + invoke stay in one place; validates the same stripped view of the draft it sends to the model, and preserves literal <think> substrings in the rewrite (strip_think_blocks(truncate_unclosed=False))
Thread Runs (/api/threads/{id}/runs)	POST / - create background run; POST /stream - create + SSE stream; POST /wait - create + block. Before the first journaled run, seed an empty feed from a checkpoint so legacy checkpoint-only history keeps its order and visibility; skip absent checkpoints or populated feeds. POST /regenerate/prepare - prepare clean input + checkpoint metadata for regenerating the latest completed or interrupted assistant answer, treating a protocol-valid hidden human_input_response as confirmed replay input while rejecting other hidden/control humans, preserving that response's hidden/correlation metadata, and carrying the latest non-empty thread title in graph input so resuming an older checkpoint cannot roll back a later manual rename (#4457); POST /edit-regenerate/prepare - prepare a checkpoint replay from the latest editable human turn with a replacement user message and edit replay metadata; it carries the current thread title the same way, but only when the replay base already has one — an untitled base belongs to a thread the title middleware has not named yet, so pinning the current title there would keep a name generated from the prompt the edit just replaced; GET / - newest 100 runs as an array; GET /page - keyset history page {data, has_more, next_before_created_at, next_before_run_id}; GET /{rid} - run details; POST /{rid}/cancel - cancel; GET /{rid}/join - join SSE; GET /{rid}/stream hides action/wait; GET action 405 pre-owner; POST needs runs:cancel; GET /{rid}/messages - paginated per-run messages {data, has_more}; GET /{rid}/events - full event stream; GET /{rid}/workspace-changes - workspace/output file change summary and optional diffs; GET/POST /{rid}/artifacts/archive - receipt manifest / bounded ZIP; GET /../messages - legacy thread message array; GET /../messages/page - backward thread-global seq history page with middleware/subagent-AI/successful-regenerate/edit-replay filtering and page-run-scoped feedback enrichment; subagent AI callbacks remain available through run events while parent task ToolMessages stay visible for card restoration; GET /../token-usage - aggregate tokens plus an optional context_usage percentage. Context usage approximately counts messages from the latest materialized thread state through build_thread_checkpoint_state_accessor, so full and delta checkpoint modes expose the same input. The percentage uses the latest run's model and its context_window.
Feedback (/api/threads/{id}/runs/{rid}/feedback)	PUT / - upsert feedback; DELETE / - delete user feedback; POST / - create feedback; GET / - list feedback; GET /stats - aggregate stats; DELETE /{fid} - delete specific
Runs (/api/runs)	POST /stream, /wait - stateless runs requiring runs:create; optional body thread_id is owner-checked. Scheduled-task create/update/resume/trigger also require threads:write plus runs:create. GET /{rid}/messages, /feedback - run messages/feedback
GitHub Webhooks (/api/webhooks/github)	POST / - receive GitHub App / repo webhook deliveries. Verifies X-Hub-Signature-256 against GITHUB_WEBHOOK_SECRET; exempt from auth + CSRF because authenticity is enforced by HMAC. The route is fail-closed: mounted only when GITHUB_WEBHOOK_SECRET is set, or when explicit dev opt-in DEER_FLOW_ALLOW_UNVERIFIED_GITHUB_WEBHOOKS=1 is set. Recognized events include ping, issues, issue_comment, pull_request, pull_request_review, and pull_request_review_comment; unknown events return 200 with handled=false. Fan-out runtime failures return 503, keeping the delivery recorded as failed for manual/API/scripted redelivery (GitHub does not automatically retry any failed delivery, 5xx included); permanent/non-retryable conditions such as channels.github.enabled: false, unknown events, malformed payloads, or unavailable channel service return 200 with a skipped/handled response.
GitHub Event-Driven Agents	Custom agents can declare a github: block in their config.yaml to bind to repos and event triggers. Webhook fan-out publishes one InboundMessage per matching binding to the channel bus; GitHubChannel routes those messages through ChannelManager. The response dispatch summarizes matched/fired/skipped agents.

Edit replay and messages-input clarification may replace knowledge scope with a validated client snapshot. Regenerate, null-input and Command resumes recover scope from the genuine source user turn, skip hidden server context, and ignore client scope. Treat input.messages: null as empty to allow promptless conversation-reference runs.

Thread identifiers use the shared deerflow.utils.thread_id contract ^[A-Za-z0-9_-]{1,64}$. Caller-provided opaque IDs remain supported; UUIDs are generated only for None, while explicit empty strings fail validation. Gateway creation and state-producing request boundaries, embedded-client entry points, filesystem/upload/event-store consumers, scheduled launches, and the standalone Provisioner enforce the same contract before persistence or workspace initialization. Route-addressable legacy IDs remain accepted by pure reads and cleanup/control endpoints; deleting one best-effort removes metadata and checkpoints but skips local filesystem cleanup, so the raw value is never interpolated into a host path. New runs, workspace/sandbox operations, and other state-producing mutations remain blocked.

Message feed seq (#4666): streaming values frames, GET /threads/{id}/state, and POST /threads/{id}/history stamp serialized messages with additional_kwargs.deerflow_seq so clients can place checkpoint-kept messages against the paged feed; the REST reads resolve the store via threads.py::_optional_run_event_store (a feed-less deployment still reads threads), and services.py::normalize_input strips the server-owned key from client input (#4380). Mechanism and identity rule: packages/harness/deerflow/runtime/AGENTS.md.

Workspace change review: packages/harness/deerflow/workspace_changes/ captures a pre-run and post-run snapshot of the thread-owned workspace and outputs directories. runtime/runs/worker.py performs the filesystem scan via asyncio.to_thread and writes a workspace_changes event with category workspace when changes exist. Uploads are intentionally excluded. Text diffs are size-limited; binary, large, and sensitive-looking paths are persisted as metadata only. Internal process-feedback directories never count as changes: the scanner's EXCLUDED_DIR_NAMES drops BROWSER_FRAMES_DIRNAME (transient browser screenshots) and TOOL_RESULTS_DIRNAME (the tool-output budget middleware's default externalization subdir, constants.py is the shared source of truth for both writers and the scanner), and the worker threads the configured tool_output.storage_subdir through the snapshot capture as an extra excluded dir name so custom storage locations stay excluded too.

Run delivery receipts: the worker derives delivery requirements from the run's workspace snapshots rather than a client request option (files created/modified under /mnt/user-data/outputs, minus internal process-feedback exclusions) and idempotently persists a run-scoped run.delivery receipt before the terminal run status; missing or unverifiable present_files coverage downgrades the run to error, while runs without changed outputs keep ordinary chat behavior. Journal mechanics (callback attribution, receipt idempotency and retries, orphan recovery): packages/harness/deerflow/runtime/AGENTS.md. Multi-worker deployments require run_events.backend: db for shared, ordered delivery events; the startup gate rejects memory/JSONL event stores when GATEWAY_WORKERS > 1 or deployment.multi_instance: true.

RunManager / RunStore contract:

• LangGraph-compatible run requests validate their supported subset before creating a run. runtime/stream_modes.py is the shared backend contract for public stream modes and the worker's graph.astream mapping; the public messages-tuple mode maps to LangGraph's internal messages mode, while public messages, events, and other unsupported modes are rejected instead of being dropped or replaced with values. app/gateway/run_models.py::RunCreateRequest is shared by HTTP and internal scheduled launch paths, retains only truthful compatibility defaults for unimplemented options (if_not_exists="create" plus None placeholders), returns 422 for unsupported values including on_completion="complete", on_completion="continue", and multitask_strategy="enqueue", and forbids undeclared SDK options so fields such as checkpoint_during and durability cannot be silently discarded. A placeholder must still accept the stock SDK's own default: langgraph_sdk drops only None from its run payload, so stream_resumable=False reaches every request and means "non-resumable", which is what DeerFlow serves — rejecting it 422'd every IM channel run (#4466). test_run_request_validation.py::test_gateway_accepts_langgraph_sdk_default_payload pins the real SDK payload; channel tests mock the SDK and miss this drift.
• RunManager.get() is async; direct callers must await it.
• The history batch helpers list_successful_regenerate_sources(), list_edit_regenerate_runs(), and get_many_by_thread() default to user_id=AUTO: they resolve the request user and fail closed when no user context exists. Migration/admin callers that intentionally need an unscoped read must pass user_id=None explicitly.
• Edit-and-rerun visibility is derived from edit replay runs (metadata.replay_kind="edit" plus regenerate_from_run_id) by RunManager.list_edit_replay_visibility(): the newest attempt for each source run is authoritative. Pending/running/success attempts hide the original source run; failed, timed-out, or interrupted attempts hide only the failed attempt so the original conversation reappears.
• When a persistent RunStore is configured, get() and list_by_thread() hydrate historical runs from the store. In-memory records win for the same run_id so task, abort, and stream-control state stays attached to active local runs.
• Thread metadata status switches to running only after RunManager.try_start() succeeds. Pending-cancelled runs therefore skip the old running projection, while clients may observe the prior thread status during the short worker-startup window.
• cancel() returns a :class:~deerflow.runtime.CancelOutcome enum: cancelled (local cancel), requested (the non-owning worker durably recorded the first cancellation action for the live owner), taken_over (non-owning worker claimed the run because the owner's lease expired — marks it as error), lease_valid_elsewhere (legacy/custom store lacks the durable request primitive — caller retains the safe 409 + Retry-After fallback), not_active_locally (heartbeat disabled, preserving the old 409 path), not_cancellable (terminal state), or unknown (not found in memory or store). create_or_reject(..., multitask_strategy="interrupt"|"rollback") persists interrupted status through RunStore.update_status(), matching normal set_status() transitions.
• Interrupt/rollback admission registers the replacement before its best-effort persistence of locally interrupted predecessors. If the admitting caller is cancelled during that post-registration await, RunManager drains a shielded replacement cleanup before propagating CancelledError, including across repeated cancellation. The cleanup normally persists interrupted; if that best-effort transition fails, it retries the active-to-interrupted store transition strictly and verifies the result with the replacement's captured owner identity. A concurrent peer terminal transition wins and is synchronized back into the local record rather than being overwritten or deleted.
• Store-only hydrated runs are readable history. In multi-worker mode with heartbeat enabled, cancel on a store-only run records runs.cancel_action / cancel_requested_at while the owner's lease is live; the first action wins even if a retry later lands on the owner. RunStore.request_cancel() and owner completion through finalize_if_not_cancelled() are competing active-row CAS operations, so an accepted cancel cannot be overwritten by a later success. RunStore.renew_lease() renews and observes the request atomically in the SQL implementation. The owner then executes the normal process-local interrupt/rollback and terminal stream path without transferring the lease. An expired owner is taken over with the orphan_recovered stop reason so cross-worker consumers can synthesize END. wait=true and cancel-then-stream use the shared bridge to observe owner finalization; a non-standard process-local bridge returns accepted 202 instead of subscribing to an unreachable stream. In single-worker mode (heartbeat off), store-only runs still return 409.
• A local worker's RunRecord.lease_expires_at is the last durably confirmed ownership deadline. _renew_leases() bounds each renewal attempt by that deadline: transient store exceptions remain retryable while it is valid, but an exception or blocked call that reaches expiry sets the process-local ownership_lost fence, raises abort_event, and cancels the run task. Successful renewals collect durable cancellation actions; after all local renewals have been attempted, heartbeat only signals the corresponding process-local tasks, leaving status writes and rollback cleanup to the worker finalization path. Fenced workers do not perform subsequent journal/delivery-receipt, progress/completion/status, checkpoint/thread-metadata, or on_run_completed writes; the peer recovery path owns the terminal receipt. RunStore.update_run_completion() also refuses to replace a different terminal status, closing the peer-takeover/late-finalization race. grace_seconds delays peer reclamation for clock skew but is not extra execution time for an owner that can no longer confirm its lease. Already-committed remote tool side effects remain outside this local cancellation boundary.
• Startup/orphan reconciliation must claim stale active rows with RunStore.claim_for_takeover(), not a plain update_status(). The final claim re-checks status and lease expiry atomically, so a heartbeat renewal between the candidate scan and the recovery write keeps the run active.
• Run admission and independent writes are first-class thread operations. runs.operation_kind distinguishes run from checkpoint_write, artifact_write, artifact_archive, branch, and delete; every active kind shares the durable active-thread uniqueness constraint. New operation kinds must go through RunStore.create_thread_operation_atomic() and RunManager.reserve_thread_operation() rather than adding another lock or metadata marker. Live and lease-less reservations are non-interruptible; an expired leased reservation can be reclaimed immediately by interrupt/rollback admission without waiting for orphan reconciliation. Lease-less rows stay fail-closed because the store cannot distinguish a stale row from a live writer in another heartbeat-disabled worker; a rare failed delete therefore requires startup reconciliation, and heartbeat-disabled multi-worker deployment remains unsupported. Reservation bodies are attached to their caller task so loss detected by lease renewal cancels the writer before it can continue after takeover; the context manager translates that lease-loss cancellation to ConflictError after cleanup so Gateway mutation routes return a retryable 409 instead of dropping the HTTP request. The cleanup scope begins immediately after durable admission, including the await that attaches the caller task, so cancellation cannot strand a locally renewed pending reservation. A failed renewal is revalidated under the manager lock before cancellation; if the reservation completed and unregistered while the store update was in flight, its request task must not be cancelled after the write. Reservations are excluded from run history/reporting and from run-only helpers such as list_by_thread() and has_inflight(), release uses the captured owner rather than ambient user context, and local cleanup still runs when the best-effort store delete fails. RunStore.create_run_atomic() remains a deprecated compatibility shim for external stores that only admit normal runs; new stores must implement create_thread_operation_atomic() to support internal operation kinds.
• Gateway checkpoint mutations outside run execution must use services.reserve_checkpoint_write(), which composes the process-local thread lock with the durable checkpoint_write reservation. Manual compaction, POST /threads/{id}/state, and both goal mutation routes (PUT / DELETE /threads/{id}/goal, including creation of a missing goal checkpoint) use this boundary, so an existing run blocks the write and the reservation blocks new reject/interrupt/rollback runs across workers.
• Branch/state-update checkpoints copy only the source checkpoint's persisted deerflow_agent_name, never request metadata; missing or malformed bindings stay unbound so compaction fails closed.
• DELETE /api/threads/{id} holds a durable delete reservation for the whole cleanup and removes thread filesystem data (off-loop, drained on cancel), checkpoints, owner-scoped historical runs, run events, feedback and thread metadata best-effort. Historical-run cleanup removes only operation_kind="run" rows, so internal thread-operation rows — including the reservation protecting that request — stay durable until RunManager.reserve_thread_operation() exits. This cleans persisted rows; preventing an already-admitted write from re-creating state for a deleted thread is a separate thread-incarnation contract, not part of this cleanup.
• Creator SSE and /wait apply on_disconnect only to confirmed request disconnect or cancellation; bridge failures propagate without cancelling the run. /wait consumes bridge END; errors return SDK __error__, never stale state.
• Memory and Redis StreamBridge implementations retain only stream_bridge.queue_maxsize data events. A syntactically valid Last-Event-ID older than the retained watermark, or a live subscriber that falls behind it, yields StreamGap before any partial replay. sse_consumer maps that control item to an id-less SSE gap payload (stream_replay_gap) and intentionally leaves the run active; internal /wait consumers resume from its latest retained ID because they only need terminal completion. Redis checks bounds plus the non-blocking read in one transaction, using blocking XREAD only as a wake-up before repeating the atomic snapshot. For a no-cursor subscriber that established a wait on an empty stream, the first wake response remains provisional until that next snapshot verifies its tail is still retained; this closes the pre-first-delivery trimming window without changing malformed-cursor live tailing. The correctness tradeoff is one three-command snapshot pipeline per poll plus the blocking wake round trip while idle. Malformed cursor behavior remains backend-specific. Memory treats a syntactically numeric cursor below its watermark conservatively as a gap even when the evicted timestamp can no longer be verified; unknown ids at or above the watermark retain the legacy replay-from-earliest policy.
• Redis StreamBridge keys use a rolling retained-buffer TTL (stream_bridge.stream_ttl_seconds, refreshed on data and first-END writes) as a leak safety net, not as a run timeout. Startup and lease-driven periodic orphan recovery share one Gateway stream-terminalization path: after RunManager durably marks a run error with stop_reason=orphan_recovered, Gateway publishes END_SENTINEL and schedules stream cleanup. The periodic store scan, per-row status writes, and Gateway callback run as one supervised single-flight task, so a slow pass is skipped at the next interval instead of piling up or pausing the sole lease-renewal loop. Store retries have bounded attempts/backoff; an individual operation still relies on the database driver/pool timeout. RunManager.shutdown() gives active user runs priority within its shared deadline, then drains or cancels orphan recovery. Gateway tracks delayed recovered-stream cleanups and converts unfinished delays to immediate deletes before closing the bridge; the Redis TTL remains the outage safety net. Only startup recovery, before the runtime yields to requests, projects the latest affected thread to error; periodic recovery deliberately avoids that non-atomic projection because ThreadMetaStore has no latest_run_id conditional-update contract. Store-only SSE and /wait consumers wait for the bridge's real END marker after an ordinary durable terminal status, because status persistence can precede tail events. Terminal orphan_recovered and scheduled_task_orphan_recovered rows permit heartbeat fallback because their producer is gone; ordinary terminal rows still await END. Malformed Last-Event-ID reconnect values live-tail new Redis events rather than replaying the retained buffer. Keep cross-component recovery orchestration in Gateway through the generic RunManager.on_orphans_recovered callback; do not introduce a harness-to-app dependency. Callback failure warnings include every recovered run_id so operators can identify rows whose Gateway-side terminalization needs inspection.
• Thread-scoped create, stream and wait share an optional owner/thread-scoped Idempotency-Key (internal callers: owner-only, cross-thread reuse 409); never pass the raw external key to RunManager's process-wide index. Stores resolve an existing key before interrupt/rollback, so a retry reuses rather than interrupts its run. Reuse requires matching input, assistant and conversation references; keyed resume stores a versioned strict typed-JSON SHA-256 (true, 1 and 1.0 differ) in the private runs.idempotency_request_json column, never public kwargs; ordinary input needs no extra identity. New readers accept the exact old-writer resume identity shape from kwargs, every response strips it; identity-less legacy resume fails closed. Validate strict resume JSON with or without a key; persist the identity only for keyed resume admission. Missing headers stay non-idempotent; stateless /api/runs/* is outside this contract (it creates a thread before admission).
• For a rolling upgrade, route every keyed resume submission/retry on create/stream/wait only to upgraded workers; otherwise pause that traffic or replace old workers first. Old workers ignore the private identity and can reuse approve for deny with null input. Old-reader schema compatibility does not protect admission. Upgraded workers reject identity-less legacy resumes with 409. Rollout details: README.md, backend/docs/API.md.
• Reused /wait returns durable status/error, never the latest thread checkpoint; task is None is not completion, and store-backed fields must refresh after waiting. Capture reuse before awaiting because the cached record's idempotency_reused flag is sticky. A creating /runs/stream retry whose terminal stream is gone opts into gap/stream_replay_gap via emit_gap_on_missing_stream, while observer joins and default/stateless consumers emit end; never infer this from disconnect policy or the sticky flag. A reused running store_only record on a process-local bridge returns 409 without Retry-After.
• Thread-scoped run creation accepts checkpoint / checkpoint_id; Gateway validates the checkpoint belongs to the request thread before writing checkpoint_id / checkpoint_ns into config.configurable for LangGraph branching. In delta checkpoint mode the worker rewrites that fork into a linear head write before the graph starts (see "A delta-mode run cannot fork" under Checkpoint Channel Modes), because delta state for a fork replays the abandoned sibling's writes.
• Thread-scoped Gateway runs evaluate an active ThreadState.goal after the visible turn completes. runtime/goal.py asks a non-thinking evaluator model to judge only visible conversation evidence and return a typed blocker; the evaluator model is created once per run and reused across hidden continuation checks. The evaluator runs after the graph root's tracing scope has already closed, so create_goal_evaluator_model/evaluate_goal_completion attach their own model-level tracing callbacks (attach_tracing=True) and inject Langfuse trace metadata (thread_id/user_id/deerflow_trace_id) directly onto the ainvoke call — the same standalone-caller pattern as oneshot_llm.run_oneshot_llm and MemoryUpdater (see Tracing System below). Satisfied goals are cleared; every non-satisfied evaluation — continuable or stand-down — is persisted with last_evaluation (the blocker, reason, and evidence summary; outcomes that stop the loop additionally record a stand_down_reason for observability), but only goal_not_met_yet evaluations are streamed as hidden HumanMessage continuations, and only when the turn did not end on an unanswered Human Input Card (ask_clarification or the sandbox network prompt; the question is still open, so the goal stands down with blocker: "needs_user_input" without calling the evaluator), a durable assistant end-of-turn checkpoint exists, the run has not been aborted, the run has not hit its token-budget hard stop (stop_reason == "token_capped"; the goal stands down with stand_down_reason: "token_capped"), the thread did not change during evaluation, and the no-progress breaker has not fired. The continuation cap is 8 — a hard maximum in the 0–8 range; callers requesting more are clamped (set_goal/TUI) or rejected with 422 (PUT /goal). The no-progress breaker keys on the latest visible assistant evidence (not the evaluator's free-text reason, which an LLM rewords every turn), so two consecutive continuations that add no new visible assistant output stop the loop after 2 attempts. Model-response cleanup helpers such as think-block stripping and code-fence stripping live in deerflow.utils.llm_text so runtime/goal.py and Gateway suggestion parsing share the same JSON-prep behavior.
• Run event stream changes must keep producer code, deerflow/constants.py, runtime/events/catalog.py, contracts/run_event_stream_contract.json, backend/docs/RUN_EVENT_STREAM.md, and tests/test_run_event_stream_contract.py in sync. The dependency-free constants module owns the persisted envelope limits (event_type 32 characters, category 16) and cross-layer workspace event identity; the catalog owns validated runtime definitions and categories. Dynamic middleware tags are limited to 21 characters after the middleware: prefix. The JSON contract owns payload schemas, backend-specific storage semantics, legacy aliases, and compatibility rules; conformance tests require both views and all producer groups to agree. run.end.content remains opaque and may retain nested Python values in memory while JSONL/database stores stringify non-JSON nested values, so consumers must not assume backend-identical nested output representations.

Scheduled-task PATCH: omitted assistant_id preserves the pin; explicit null resets to lead_agent.

Two origin keys; never read one for the other. deerflow_scheduled_origin marks the scheduled prompt message (scheduled_goal_notes_contract.json); deerflow_origin says who started a run/created a thread (thread_origin_contract.json), set only via request.state.run_origin or internal body.metadata.

Thread lifecycle: Before changing branching, regeneration, edit replay, or archive/search behavior, read Thread lifecycle invariants. It owns lineage and settled-checkpoint rules, legacy fallback boundaries, archive filtering before pagination, owner isolation, and activity-time preservation.

MCP create/replace/state writes require unique installation IDs. DELETE validates schema but permits residual legacy ID collisions. Explicit selections exclude ambiguous IDs until repaired.

Capability skill discovery (/api/capabilities/installations/skills) reuses routers/skills.py::_filter_visible_skills after its off-thread storage read. It must preserve the ordinary skill listing's caller visibility and provider failure policy rather than exposing the unfiltered user-scoped catalog.

Cap edit reads at MAX_EDITABLE_ARTIFACT_BYTES + 1 despite stale lstat.

Route and skill authorization

authz.authorize_model_use shares the model:use decision between model details and follow-up suggestions, including the factory's first configured model when the caller omits model_name. Suggestions check admission before their best-effort LLM error handler: explicit denies return 403 without invoking the model, while provider failures honor authorization.fail_closed.

authz.py::resolve_route_permissions() is shared by HTTP, decorator-only auth, and Live Browser WebSocket admission. It evaluates full resource:action targets; provider errors follow authorization.fail_closed, while disabled authorization preserves legacy permissions. Owner/admin checks stay separate. Live requires threads:write with is_internal=False after login/Origin checks; denial closes 4403 and setup errors close 4501. Admission is connection-scoped. On disconnect, Live drains owned stop_screencast() cleanup before releasing the session lease. Tests: test_authorization_route_permissions.py, test_browser_readonly_security.py, test_auth.py, test_auth_middleware.py.

Skill listing authorization mirrors the models pattern: routers/skills.py routes resolve (provider, principal) through authz.py::resolve_skill_authorization() — a thin sibling of resolve_model_authorization, both delegating to the shared _resolve_route_scoped_authorization core — and the shared _filter_visible_skills helper filters the user-scoped catalog (public + caller's custom skills) through filter_resources(principal, "skill", ...) by name. Three user-facing surfaces apply it: GET /api/skills (the frontend skill list / slash-command autocomplete), GET /api/skills/custom, and GET /api/skills/{name} — the detail endpoint returns the standard 404 for an invisible skill so it cannot become an existence oracle the filtered list closed (get_model's 403 is an execution decision via authorize("model", "use"), which has no skills equivalent in this layer). Anonymous callers are unfiltered (mirroring list_models), and provider resolution/decision errors follow authorization.fail_closed (fail-closed → empty listing / 404, fail-open → full listing). Skill management endpoints (install/upload/reload/custom-skill CRUD) stay require_admin_user-gated, and runtime activation authorization is a separate layer. The built-in RBAC provider maps this to the per-role skills policy key. Tests: tests/test_skills_listing_authorization.py.

Batch workers pin app.state.extensions; never persist snapshots.

Personal MCP is owner-only; platform MCP shared.

Web model provider presets

Managed model routes expose provider metadata; routing stays server-owned. Align validation, probing, storage, keys, and revisions.