github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-09-29 07:56:18 +00:00
Code Issues Packages Projects Releases Wiki Activity
deer-flow/backend/packages/harness/deerflow/AGENTS.md
Hyeonsang Cho 9146bfa03d
feature(gateway): issue request trace ids unconditionally (#5119) 
* refactor(gateway): issue request trace ids unconditionally

The request trace id was gated behind logging.enhance.enabled at every
entry point, so downstream code had to keep asking whether one existed:
a header-provenance flag in its own ContextVar, a precedence resolver,
and three-level carrier fallbacks at each consumer.

Bind one unconditionally instead. TraceMiddleware covers Gateway HTTP;
ensure_trace_context covers the entry points that never touch ASGI --
scheduled occurrences, MCP task notification runs, IM channel messages,
and the embedded client -- each scoped to one unit of work so a
long-lived worker task cannot leak one occurrence's id into the next.
The ContextVar becomes the only source; the response header, runtime
context, run metadata and log records are derived outputs.

Consumers now use ensure_trace_id() or resolve_trace_id(*carriers) and
drop their presence guards. Removed: resolve_deerflow_trace_id, the
header-provenance flag and its three helpers, set/reset_current_trace_id,
is_trace_correlation_enabled and its gateway alias.

BREAKING CHANGE: every Gateway HTTP response now carries X-Trace-Id and
it cannot be turned off; logging.enhance.enabled controls log output
only. Installations on the default enabled: false will start seeing the
header. No config keys were added or removed.

* fix(gateway): stop persisting a caller-supplied trace id on the run record

body.metadata forks two ways: through build_run_config into the live run
config, which the run worker restamps, and through create_or_reject into
the run record that the runs API echoes verbatim. Only the first was
covered, so a client sending metadata.deerflow_trace_id made the most
durable and most visible surface of a run disagree with the X-Trace-Id
and the log lines the same request produced -- a correlation id that
does not match the logs is worse than none.

Stamp the server-issued id once at the trust boundary so both forks
receive it, preserving the caller's own metadata keys. Close the same
gap on config.context, which reaches the runtime context by a separate
path: _build_runtime_context no longer merges server-owned keys from the
caller, and _install_runtime_context assigns rather than setdefaults.

A thread's metadata is no longer seeded with the run-scoped id of
whichever run created it -- one thread spans many runs and as many
trace ids.

Found by driving a real run through the Gateway and reading the run back
from the runs API; every unit test built its metadata by hand and so
could not see it.

* fix(gateway): expose X-Trace-Id to split-origin browser clients

X-Trace-Id is not on the CORS safelist, so a browser client served from
a separate origin could not read it -- and those are exactly the clients
that cannot read the Gateway's logs either, leaving them with nothing to
quote in a bug report. Same-origin nginx deployments were unaffected,
which is why this stayed hidden.

Add it to CORS_EXPOSED_HEADERS beside Content-Location, referencing
TRACE_ID_HEADER rather than repeating the literal.

* fix(gateway): keep X-Trace-Id on unhandled-exception 500s

Starlette's ServerErrorMiddleware sits outside every user middleware and
emits unhandled-exception 500s through the raw send, so those responses
never pass TraceMiddleware's header-writing wrapper. The 500 for a server
bug is exactly the response a user most needs to correlate with a log line,
and it was the one response that shipped without the id.

TraceMiddleware now tracks whether http.response.start has been sent. On an
exception with no response started it emits its own plain 500 carrying the
header, then re-raises: the outer ServerErrorMiddleware sees the response
already started and only re-raises too, so the server's exception logging is
untouched. An exception mid-stream keeps propagating unchanged — a second
response start cannot be sent, and the already-written header stands.

The trace id is printable ASCII by construction (normalize_trace_id /
generate_trace_id), which is what makes the raw latin-1 header encoding
safe.

* fix(gateway): strip the forged trace id from the persisted request echo

The run-record fix stopped a forged metadata.deerflow_trace_id on the
authoritative metadata surface, but the raw request echo still carried one:
create_or_reject persists body.config verbatim as runs.kwargs_json, which
the runs API serves back. A client posting config.context.deerflow_trace_id
therefore still got its forged value stored and echoed on one API surface
while the header, logs, run metadata, and checkpoint all carried the real
id — the id is ignored as input there, so echoing it back only manufactures
disagreement.

Two changes close it. redact_config_secrets — already the shared scrub for
that echo, applied at admission and again at serve time, so historical
records are covered too — now also drops deerflow_trace_id from
config.metadata and config.context. And build_run_config now merges run
metadata onto a copy of the caller's config["metadata"] instead of updating
it in place: the nested values of the request config are reference copies,
so the in-place merge was writing the server-stamped key through into
body.config, contaminating the "what the client sent" record before it was
persisted (and incidentally masking the forged-value echo on the metadata
container).

The regression test posts a forged id through body.metadata,
config.metadata, and config.context at once and reads the kwargs echo back
off the run record, failing if either leak returns.

* docs(harness): record the trace-echo scrub, 500 fallback, and accepted retry divergence

The trace section of the harness AGENTS.md now covers the two fixes that
close the derived-output rule (the kwargs-echo scrub in
redact_config_secrets plus build_run_config's copy merge, and
TraceMiddleware's own 500 for unhandled exceptions), and CHANGELOG gains
their Fixed entries.

It also writes down the one accepted divergence: a crash-recovered
scheduled launch reuses the durable run through its idempotency key, and
start_run returns early on idempotency_reused without restamping — so the
run record keeps the first attempt's deerflow_trace_id while the retry's
own log lines carry the freshly minted id of its ensure_trace_context
binding. The divergence is confined to the crash-recovery window and is
accepted rather than fixed: restamping on reuse would rewrite a persisted
record for a run that already exists, which is worse than two ids that each
correlate their own attempt's logs. Written down so the next reader of the
scheduler recovery path does not diagnose it as a bug.

* docs(config): align the logging.enhance schema note with the unconditional trace id

The config-module AGENTS.md still described logging.enhance as the gate for
the Gateway X-Trace-Id header and Langfuse deerflow_trace_id. That model is
gone: ids are issued unconditionally and this block decides log output only.
Left as-is, the stale wording invites an agent to "restore" a header gate it
believes was lost. Reworded to match the sibling AGENTS.md files and
config.example.yaml, with a pointer to the Request Trace Context section
that owns the full model.

* docs(changelog): link the trace entries to #5119

The five new entries pointed at the ([#XXXX]) placeholder with no reference
definition, rendering as literal text instead of a link — and RELEASING.md
step 2 relies on those references when the section becomes release notes.
All five now point at #5119, with the definition appended to the reference
block.

* refactor(harness): rename _stream_without_trace_context to _stream_turn

The name asserted the opposite of what the method now does. It was accurate
while logging.enhance.enabled could route stream() around the trace scope;
with the gate gone it is the only stream implementation left, and it binds
the id itself via ensure_trace_id(). Private, so the rename touches only the
definition and the one stream() call site.

* docs(harness): fit the trace-context guidance inside the AGENTS.md chain budget

The expanded Request Trace Context section pushed the effective AGENTS.md
chain for agents/middlewares to 99,815 bytes, past the 98,304 hard limit
scripts/check_agent_guidance.py enforces in CI (AG002). Compressed the
section from 7,359 to 4592 bytes with no facts removed: the entry-point
table, the derived-output rule and its enforcement points, the accepted
scheduled-retry divergence, the two resolution helpers, the stream()
binding rationale, the log-output-only gate, the CORS listing, the 500
fallback, and the test map all remain.

Sized against the merge, not just the branch: current main grew the same
chain by ~724 bytes, so the check was verified on the merged tree as well
(97,772 bytes; branch tree 97,048).

* fix(gateway): declare content-length on the fallback 500

The pre-response 500 declared content-type but no content-length, leaving
the framing to the ASGI server: chunked on HTTP/1.1, close-delimited on
HTTP/1.0 — the one wire difference from the ServerErrorMiddleware response
it replaces, which sends content-length: 21. The explicit header keeps the
fallback byte-identical to what clients saw before.

* docs(readme): drop the trace-correlation condition from the translations

The zh/ja/fr/ru Langfuse sections still said metadata.deerflow_trace_id
matches X-Trace-Id "when request trace correlation is enabled". The id now
always matches and that condition no longer exists, so each bullet states
the unconditional match and that logging.enhance.enabled only controls
whether the id is printed into logs — the one piece of the feature a user
can still configure.

* test(gateway): pin TraceMiddleware wiring through create_app()

Every X-Trace-Id test exercised a hand-built four-route app, so the real
stack's add_middleware(TraceMiddleware) line was pinned by nothing: deleting
it — or short-circuiting above it — passed CI while silently dropping both
the response header and the ambient id the run-record stamp and enhanced log
records derive from. One case now drives /health through create_app() and
asserts the inbound id round-trips; mutation-checked by removing the wiring
line, which fails exactly this test.

* docs(gateway): note the fallback 500 is CORS-opaque

The pre-response 500 is emitted outside CORSMiddleware — the exception has
already unwound past it — so it carries no Access-Control-Allow-Origin and
a split-origin browser client cannot read the id on this one response,
unchanged from the ServerErrorMiddleware 500 it replaces. Documented on the
class and in the CHANGELOG entry rather than fixed: replicating the origin
allowlist outside CORSMiddleware would let the two policies drift.

* fix(harness): keep abandoned-stream cleanup inside the trace binding

stream() binds the turn's id around each next(inner) and resets it before
yielding, but the finally's inner.close() ran after that binding was gone.
Abandoning the stream therefore drove the inner LangGraph generator's
GeneratorExit/finally path with no trace id — or an unrelated ambient one
from whichever context ran the close — so cancellation and finalization
logs and callbacks did not correlate with the turn they belong to.

inner.close() is now wrapped in a local bind/reset of the same turn id. The
token is set and reset in the same frame, never across a yield, so the
per-step cross-context safety is preserved even when GC closes the
generator from another Context — pinned by the existing copy_context close
test, which now exercises this path. The regression test records the id
from the inner generator's finally and fails without the binding.

* test(harness): teach the worker-trace fake about RunManager.cleanup

Upstream #5112 (bound gateway memory after terminal runs) added a
run_manager.cleanup(run_id) call to run_agent's finalization, so the
merge-commit CI run failed all five worker-trace-binding tests with
AttributeError on this PR's _FakeRunManager. The fake gains the same no-op
shape as its other methods.

* docs(gateway): bring the gateway AGENTS.md back under its soft budget

Upstream #5092 grew backend/app/gateway/AGENTS.md to 40,966 bytes, 6 over
the 40,960 soft budget that
test_agent_guidance_check.py::test_repository_guidance_stays_below_soft_budgets_and_avoids_doc_indexes
enforces — its Unit Tests run on main was cancelled by push concurrency, so
main is currently red on that test and every PR merge-run inherits the
failure. Two whitespace/wording trims in the row #5092 touched (a doubled
space, and "its configured `context_window`" → "its `context_window`")
bring the file to 40,953 with no content change.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-01 16:49:39 +08:00

124 lines
11 KiB
Markdown
Raw Blame History

### Request Trace Context (`packages/harness/deerflow/trace_context.py`)
DeerFlow's request-level correlation id — the `X-Trace-Id` header and the `deerflow_trace_id` key. Not Langfuse's trace id, not `run_id`, not the short subagent `trace_id` log label.
**The ContextVar is the only source.** Every path that reaches a run binds one first; downstream treats the id as a plain `str`, no `if trace_id:` guards.
Entry points and binders: Gateway HTTP — `TraceMiddleware`; scheduled occurrence — `ScheduledTaskService._attempt_queued_run` → `launch_scheduled_thread_run`; MCP task notification — `launch_mcp_task_notification_run`; IM inbound — `ChannelManager._worker_loop`; embedded / TUI / CLI turn — `DeerFlowClient.stream()`.
Only the first is HTTP; the rest run outside ASGI, so the binding cannot live in middleware alone. Each scopes **one unit of work**, never a poller loop — a leaked binding on a reused worker task would tag later occurrences with the first id. `ensure_trace_context` inherits, keeping layered scheduled bindings and a manual trigger inside a Gateway request on one trace.
**Every other carrier is a derived output, never read back as an input.** `worker._bind_trace_id` stamps the runtime context and `config["metadata"]`; `services.start_run` stamps the run record; a caller-sent `deerflow_trace_id` (`body.metadata`, `body.config.context`) is replaced — honouring it would let the persisted run disagree with the header and the logs. `_SERVER_OWNED_RUNTIME_CONTEXT_KEYS` covers the embedded path, `redact_config_secrets` scrubs the kwargs echo (`runs.kwargs_json`), and `build_run_config` merges metadata onto a copy so the stamp cannot reach `body.config`. Callers pin an id with `X-Trace-Id`.
Accepted divergence: a crash-recovered scheduled launch reuses its run via the idempotency key without restamping — the record keeps the first attempt's id, the retry's logs a fresh one; restamping would rewrite an existing record. Not a bug. Thread metadata omits the key entirely — a thread spans many runs.
**Do not open-code fallback chains.** Two helpers own the resolution order:
- `resolve_trace_id(*carriers)` — first usable carrier, else ambient. For ids travelling as data in `runtime.context`; ContextVars do not survive a bare thread hop.
- `ensure_trace_context(trace_id)` — reuse the surrounding scope, else start a self-contained one. For boundary crossings (`SubagentExecutor._aexecute`, the memory `trace_context_manager` hook) and non-HTTP entry points; no argument mints a scoped id.
`request_trace_context` (HTTP) deliberately does **not** inherit: a crafted header must not fall back to the previous request's id.
`get_current_trace_id()` stays nullable only for the logging filter (pre-entry-point records render as `trace_id=-`); everything else uses `ensure_trace_id()`/`resolve_trace_id()`.
`DeerFlowClient.stream()` binds per `next()` step and around `inner.close()`, never across a `yield`: a sync generator shares the caller's context, so a scope held across yields would leak the id and break on cross-context GC finalization.
`logging.enhance.enabled` gates **log output only** (`trace_id` field presence and format) — not the id, the header, or the run metadata — so `TraceMiddleware` reads no `AppConfig`; `logging` stays restart-required (`STARTUP_ONLY_FIELDS["logging"]`). `X-Trace-Id` is in `CORS_EXPOSED_HEADERS` (not safelisted). Unhandled-exception 500s keep the header — `TraceMiddleware` sends its own plain 500 (CORS-opaque, see its docstring) before re-raising; mid-stream failures propagate unchanged.
Tests: the `tests/test_trace_*` and `tests/test_worker_trace_binding.py` suites, `test_gateway_services.py`, `test_run_metadata_secret_safety.py`, plus the Langfuse suites in `tracing/AGENTS.md`.
### Managed Lark CLI credentials (`integrations/lark_cli.py`)
App registration and direct app switching replace the per-user Lark credential
tree transactionally. Clear the old OAuth data before running `lark-cli config
init`: on Linux that command writes the new app secret into the file-backed
keychain under the data directory, so clearing the directory afterward would
leave `config.json` with a dangling keychain reference. The transaction snapshot
still supplies the previous OAuth data for logout and restores the complete old
tree if any switch step fails.
### Browser Progress Screenshots (`community/browser_automation/`)
Hidden per-action browser progress frames use JPEG at quality 80 to keep their
storage and transfer cost bounded relative to lossless PNG. The explicit
`browser_screenshot` tool remains PNG because it creates a user-requested
artifact. New automatic capture entry points must reuse the shared progress
encoding definition in `tools.py` so the byte encoding and `.jpg` suffix cannot
drift.
### Embedded Client (`packages/harness/deerflow/client.py`)
`DeerFlowClient` provides direct in-process access to all DeerFlow capabilities without HTTP services. All return types align with the Gateway API response schemas, so consumer code works identically in HTTP and embedded modes.
**Architecture**: Imports the same `deerflow` modules that Gateway API uses. Shares the same config files and data directories. No FastAPI dependency.
**Agent Conversation**:
- `chat(message, thread_id)` — synchronous, accumulates streaming deltas per message-id and returns the final AI text
- `stream(message, thread_id)` — subscribes to LangGraph `stream_mode=["values", "messages", "custom"]` and yields `StreamEvent`:
- `"values"` — full state snapshot (title, messages, artifacts); AI text already delivered via `messages` mode is **not** re-synthesized here to avoid duplicate deliveries; serialized `ToolMessage` entries preserve a non-`None` native `artifact`
- `"messages-tuple"` — per-chunk update: for AI text this is a **delta** (concat per `id` to rebuild the full message); tool calls and tool results are emitted once each, and tool results preserve a non-`None` native `artifact`
- `"custom"` — forwarded from `StreamWriter`; DeerFlow-built-in custom events are dual-emitted through `deerflow.utils.custom_events`, so `astream_events(version="v2")` consumers also receive one `on_custom_event` with `name=payload["type"]` and the unchanged payload as `data`
- `"end"` — stream finished (carries cumulative `usage` counted once per message id)
- **Custom-event invariant** — production DeerFlow emitters must use `emit_custom_event` / `aemit_custom_event`, not call `StreamWriter` alone. Every built-in payload must carry a non-empty string `type`; typeless payloads remain writer-only and are intentionally absent from `astream_events`. The writer runs first and remains authoritative for Gateway, Web UI, and embedded-client compatibility; callback dispatch is best-effort and must not break that path. Async graph hooks must await the async helper rather than invoking synchronous dispatch on a running event loop.
- Agent created lazily via `create_agent()` + `build_middlewares()`, same as `make_lead_agent`
- Supports `checkpointer` parameter for state persistence across turns
- `reset_agent()` forces agent recreation (e.g. after memory or skill changes)
- See [docs/STREAMING.md](../../../docs/STREAMING.md) for the full design: why Gateway and DeerFlowClient are parallel paths, LangGraph's `stream_mode` semantics, the per-id dedup invariants, and regression testing strategy
**Gateway Equivalent Methods** (replaces Gateway API):
| Category | Methods | Return format |
|----------|---------|---------------|
| Models | `list_models()`, `get_model(name)` | `{"models": [...]}`, `{name, display_name, ...}` |
| MCP | `get_mcp_config()`, `update_mcp_config(servers)` | `{"mcp_servers": {...}}` |
| Skills | `list_skills()`, `get_skill(name)`, `update_skill(name, enabled)`, `install_skill(path)` | `{"skills": [...]}` |
| Goals | `get_goal(thread_id)`, `set_goal(thread_id, objective, max_continuations=8)`, `clear_goal(thread_id)` | `{"goal": {...}}` or `{"goal": None}` |
| Memory | `get_memory()`, `reload_memory()`, `get_memory_config()`, `get_memory_status()` | dict |
| Uploads | `upload_files(thread_id, files)`, `list_uploads(thread_id)`, `delete_upload(thread_id, filename)` | `{"success": true, "files": [...]}`, `{"files": [...], "count": N}` |
| Artifacts | `get_artifact(thread_id, path)` → `(bytes, mime_type)` | tuple |
**Key difference from Gateway**: Upload accepts local `Path` objects instead of HTTP `UploadFile`, rejects directory paths before copying, and reuses a single worker when document conversion must run inside an active event loop. Artifact returns `(bytes, mime_type)` instead of HTTP Response. The new Gateway-only thread cleanup route deletes `.deer-flow/threads/{thread_id}` after LangGraph thread deletion; there is no matching `DeerFlowClient` method yet. `update_mcp_config()` and `update_skill()` automatically invalidate the cached agent.
**Tests**: `tests/test_client.py` (offline unit tests including
`TestGatewayConformance`), `tests/test_client_live.py` (live integration tests,
requires a root `config.yaml`, valid API credentials, and explicit opt-in via
`make test-live` or `DEER_FLOW_RUN_LIVE_TESTS=1`). The live suite calls real
external APIs and may incur API costs or create local sandboxes, artifacts, and
files. It is marked `live`, excluded from `make test`, and skipped in default
CI.
**Gateway Conformance Tests** (`TestGatewayConformance`): Validate that every dict-returning client method conforms to the corresponding Gateway Pydantic response model. Each test parses the client output through the Gateway model — if Gateway adds a required field that the client doesn't provide, Pydantic raises `ValidationError` and CI catches the drift. Covers: `ModelsListResponse`, `ModelResponse`, `SkillsListResponse`, `SkillResponse`, `SkillInstallResponse`, `McpConfigResponse`, `UploadResponse`, `MemoryConfigResponse`, `MemoryStatusResponse`.
### E2B Mount Uploads
The E2B provider uploads host mounts during sandbox creation. It passes binary file objects to the E2B SDK.
Each mount has these fixed limits:
- 100 MiB for one file.
- 512 MiB for all files.
- 2,000 files.
The full sandbox creation pass also allows 512 MiB and 2,000 files. Skill
projections and configured mounts share this budget.
The pass has a cooperative deadline controlled by
``mount_upload_deadline_seconds`` (default: 120 seconds). The provider checks it before
each mount, during directory preflight, and before each SDK write. The deadline
does not interrupt active filesystem or E2B SDK calls.
The provider checks mount limits before upload. It rechecks each opened file descriptor against its preflight size before SDK upload.
For policy-scoped turns, clearing the four managed remote skill categories and
uploading their prepared projection is one per-user/thread/skills-root critical
section, shared with acquire and release. The provider snapshots that canonical
root at startup and carries it through warm-pool identity and E2B metadata; a VM
from another root is never adopted. A second policy sync cannot reset the remote
tree until the first upload pass has completed.
An invalid mount does not block later mounts.
Each successful upload logs its source, destination, file count, byte count, and elapsed time.
A stopped pass logs its limit reason and elapsed time. It reports attempted and completed upload totals separately.
Reference in New Issue View Git Blame Copy Permalink