deer-flow/backend/packages/harness/deerflow/AGENTS.md

Request Trace Context (packages/harness/deerflow/trace_context.py)

Request trace correlation is controlled by logging.enhance.enabled at both entry points, gated through the shared helper deerflow.config.app_config.is_trace_correlation_enabled so the Gateway and embedded paths cannot drift:

• Gateway HTTP: app.gateway.trace_middleware.TraceMiddleware binds one request-level trace id per HTTP request, inheriting inbound X-Trace-Id when present or generating a new id otherwise. A valid inbound header also marks the request so runtime/runs/worker.py prefers that id over config.metadata.deerflow_trace_id, keeping logs, response headers, Langfuse, and runtime context aligned when callers send both. The middleware writes the final value to every HTTP response at http.response.start, which covers SSE / streaming responses without consuming the body.
• Embedded / TUI / CLI: DeerFlowClient.stream() mints (or inherits) a request-level trace id per turn only when the flag is on. When it is off, no fresh id is minted — a caller that explicitly wraps stream() in request_trace_context(...) still opts in, because the downstream get_current_trace_id() read propagates that value into Langfuse metadata regardless of the flag. Because stream() is a sync generator (which shares the caller's context), the id binding is set/reset around each next() step rather than around yield from: this keeps LangGraph node execution and its log records inside the binding, while returning control to the caller with the ContextVar restored — avoids cross-request leak between yields and ValueError: <Token> was created in a different Context on GC-driven close of an abandoned generator (regression pinned by tests/test_client_langfuse_metadata.py::test_stream_does_not_leak_trace_id_to_caller_context_between_yields and ::test_stream_abandoned_generator_close_does_not_raise_cross_context).

The same ContextVar value is injected into enhanced log records as trace_id and into Langfuse metadata as deerflow_trace_id.

logging is registered as a restart-required field (STARTUP_ONLY_FIELDS["logging"]): configure_logging() installs the trace-context filter and enhanced formatter on root handlers only during app.py lifespan startup, and TraceMiddleware captures logging.enhance.enabled once when the FastAPI app is constructed (via resolve_trace_enabled(get_app_config()) in create_app(), itself a thin alias for is_trace_correlation_enabled). This keeps the response X-Trace-Id header, log trace_id fields, and Langfuse deerflow_trace_id coherent — a runtime config.yaml edit to logging.enhance.* needs a Gateway restart to take effect. The deerflow_trace_id chain inherits this guarantee transitively because every injection point ultimately reads the same trace_context ContextVar that the middleware alone populates. DeerFlowClient reads its own self._app_config snapshot (captured at __init__) through the same helper for the embedded gate.

deerflow_trace_id is a DeerFlow correlation metadata key, not Langfuse's native trace id and not a DeerFlow run_id. Keep the existing subagent trace_id field separate: that short id is still only for subagent execution logs/status.

Browser Progress Screenshots (community/browser_automation/)

Hidden per-action browser progress frames use JPEG at quality 80 to keep their storage and transfer cost bounded relative to lossless PNG. The explicit browser_screenshot tool remains PNG because it creates a user-requested artifact. New automatic capture entry points must reuse the shared progress encoding definition in tools.py so the byte encoding and .jpg suffix cannot drift.

Embedded Client (packages/harness/deerflow/client.py)

DeerFlowClient provides direct in-process access to all DeerFlow capabilities without HTTP services. All return types align with the Gateway API response schemas, so consumer code works identically in HTTP and embedded modes.

Architecture: Imports the same deerflow modules that Gateway API uses. Shares the same config files and data directories. No FastAPI dependency.

Agent Conversation:

• chat(message, thread_id) — synchronous, accumulates streaming deltas per message-id and returns the final AI text
• stream(message, thread_id) — subscribes to LangGraph stream_mode=["values", "messages", "custom"] and yields StreamEvent:
  • "values" — full state snapshot (title, messages, artifacts); AI text already delivered via messages mode is not re-synthesized here to avoid duplicate deliveries; serialized ToolMessage entries preserve a non-None native artifact
  • "messages-tuple" — per-chunk update: for AI text this is a delta (concat per id to rebuild the full message); tool calls and tool results are emitted once each, and tool results preserve a non-None native artifact
  • "custom" — forwarded from StreamWriter; DeerFlow-built-in custom events are dual-emitted through deerflow.utils.custom_events, so astream_events(version="v2") consumers also receive one on_custom_event with name=payload["type"] and the unchanged payload as data
  • "end" — stream finished (carries cumulative usage counted once per message id)
• Custom-event invariant — production DeerFlow emitters must use emit_custom_event / aemit_custom_event, not call StreamWriter alone. Every built-in payload must carry a non-empty string type; typeless payloads remain writer-only and are intentionally absent from astream_events. The writer runs first and remains authoritative for Gateway, Web UI, and embedded-client compatibility; callback dispatch is best-effort and must not break that path. Async graph hooks must await the async helper rather than invoking synchronous dispatch on a running event loop.
• Agent created lazily via create_agent() + build_middlewares(), same as make_lead_agent
• Supports checkpointer parameter for state persistence across turns
• reset_agent() forces agent recreation (e.g. after memory or skill changes)
• See docs/STREAMING.md for the full design: why Gateway and DeerFlowClient are parallel paths, LangGraph's stream_mode semantics, the per-id dedup invariants, and regression testing strategy

Gateway Equivalent Methods (replaces Gateway API):

Category	Methods	Return format
Models	list_models(), get_model(name)	{"models": [...]}, {name, display_name, ...}
MCP	get_mcp_config(), update_mcp_config(servers)	{"mcp_servers": {...}}
Skills	list_skills(), get_skill(name), update_skill(name, enabled), install_skill(path)	{"skills": [...]}
Goals	get_goal(thread_id), set_goal(thread_id, objective, max_continuations=8), clear_goal(thread_id)	{"goal": {...}} or {"goal": None}
Memory	get_memory(), reload_memory(), get_memory_config(), get_memory_status()	dict
Uploads	upload_files(thread_id, files), list_uploads(thread_id), delete_upload(thread_id, filename)	{"success": true, "files": [...]}, {"files": [...], "count": N}
Artifacts	get_artifact(thread_id, path) → (bytes, mime_type)	tuple

Key difference from Gateway: Upload accepts local Path objects instead of HTTP UploadFile, rejects directory paths before copying, and reuses a single worker when document conversion must run inside an active event loop. Artifact returns (bytes, mime_type) instead of HTTP Response. The new Gateway-only thread cleanup route deletes .deer-flow/threads/{thread_id} after LangGraph thread deletion; there is no matching DeerFlowClient method yet. update_mcp_config() and update_skill() automatically invalidate the cached agent.

Tests: tests/test_client.py (offline unit tests including TestGatewayConformance), tests/test_client_live.py (live integration tests, requires a root config.yaml, valid API credentials, and explicit opt-in via make test-live or DEER_FLOW_RUN_LIVE_TESTS=1). The live suite calls real external APIs and may incur API costs or create local sandboxes, artifacts, and files. It is marked live, excluded from make test, and skipped in default CI.

Gateway Conformance Tests (TestGatewayConformance): Validate that every dict-returning client method conforms to the corresponding Gateway Pydantic response model. Each test parses the client output through the Gateway model — if Gateway adds a required field that the client doesn't provide, Pydantic raises ValidationError and CI catches the drift. Covers: ModelsListResponse, ModelResponse, SkillsListResponse, SkillResponse, SkillInstallResponse, McpConfigResponse, UploadResponse, MemoryConfigResponse, MemoryStatusResponse.

E2B Mount Uploads

The E2B provider uploads host mounts during sandbox creation. It passes binary file objects to the E2B SDK.

Each mount has these fixed limits:

• 100 MiB for one file.
• 512 MiB for all files.
• 2,000 files.

The full sandbox creation pass also allows 512 MiB and 2,000 files. Skill projections and configured mounts share this budget.

The pass has a cooperative deadline controlled by mount_upload_deadline_seconds (default: 120 seconds). The provider checks it before each mount, during directory preflight, and before each SDK write. The deadline does not interrupt active filesystem or E2B SDK calls.

The provider checks mount limits before upload. It rechecks each opened file descriptor against its preflight size before SDK upload.

An invalid mount does not block later mounts.

Each successful upload logs its source, destination, file count, byte count, and elapsed time.

A stopped pass logs its limit reason and elapsed time. It reports attempted and completed upload totals separately.