* feat(community): offload image search to a thread via asyncio.to_thread
Make image_search_tool async and run the blocking ddgs call in a worker
thread with asyncio.to_thread, continuing the TODO.md planned feature
('Replace sync requests with httpx.AsyncClient in community tools').
The ddgs SDK exposes no async API, so instead of swapping the HTTP
client the blocking DDGS.images() call is offloaded to a thread — the
same pattern the infoquest/jina_ai tools use for readability
extraction. Callers see no change in behavior.
Tests invoke the async tool via ainvoke under @pytest.mark.anyio.
* fix(image-search): offload config lookup with DDGS search
* test(image-search): pin DDGS search off the event loop
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(logging): redact credential-bearing header values in urllib3's header-parse dump
* fix(logging): bind redacted before the header-dump pass runs
* fix(logging): collapse the first field of a header dump, which shares its line with the url argument
* fix(logging): collapse every header-dump field, in the message and in the exc_info traceback
* test(logging): give the traceback message copy real CRLF
The message half of the record built its payload with literal backslash-r
text while the exc_info copy used real CRLF, so the two halves rendered the
dump in different shapes and the message only exercised the pass through a
fragment match instead of the escape the real record produces.
* fix(logging): redact obs-fold continuations and Proxy-Authentication-Info
A malformed response can fold a header value onto the next line, and this
warning exists precisely to dump malformed upstream bytes, so a folded
credential cannot be assumed absent: the pass replaced only the matched
segment and left the continuation - `CookieSecret; Path=/` - in the log
whole. A collapsed field now also swallows the SP/HTAB-prefixed segments
that follow it, while a non-sensitive field keeps its own continuation.
Proxy-Authentication-Info is the proxy-side twin of Authentication-Info,
which was already on the list; nextnonce reaches the payload and was
logged verbatim in both the message and the exception copy.
* fix(logging): redact a credential folded under an unlisted field, past repr escapes, and behind an empty first segment
Addresses the three P1 findings on the header-parse dump pass:
- A credential that exists only in an obs-fold continuation is now collapsed.
``_CREDENTIAL_FIELD_RE`` no longer carries the ``^``: the anchor is applied
past a segment's fold marker instead, because under RFC 7230 unfolding a
bearer token folded under a field this list does not name is still a bearer
token.
- ``_OBS_FOLD_RE`` now also recognises the repr-escaped ``\t`` spelling. The
payload reaches the record through ``!r``, so a real tab is the two
characters backslash-t and never matched a literal-class marker.
- The continuation walk runs even when the matched value is empty, so
``Set-Cookie:\r\n Secret; Path=/`` collapses instead of leaving the secret in
the segment that follows.
Evidence: 3 new regression tests, red on the previous head (3 failed / 23
passed) and green on this commit (26 passed) against urllib3 2.7.0; ``ruff
check`` and ``ruff format --check`` clean on both files.
* test(logging): move the header-parse tests out of the EOF hunk
Upstream `22b5b59` appended a redaction test at the end of this file, the same
position this branch appends its own block to, so the PR reported a conflict.
Relocating this branch's block to the preceding top-level boundary frees the
EOF hunk; the three-way merge against current `main` is then clean.
Content is unchanged: base->head is a single pure insertion of the same 312
lines (0 deletions, same line multiset as before this commit), and the merge
result is byte-identical in aggregate to resolving the conflict by hand.
* fix(logging): treat an escaped bare CR as a header-dump line break
urllib3's header-parse warning carries the unparsed block through repr(), so
its separators are escaped text. The splitter recognised escaped CRLF and
escaped LF but not escaped bare CR, which http.client also accepts as a line
terminator: a payload separated on bare CR alone stayed in one segment, no
field name landed at a segment start, so the credential anchor never fired
and the Set-Cookie and signed Location values were logged verbatim in both
the message and the repeated exc_text. Add the escaped \r alternative and an
end-to-end case driven through the real parse_headers/assert_header_parsing
pair on the pinned urllib3 2.7.0.
Co-Authored-By: sxh313 <sxh313@users.noreply.github.com>
* test(logging): pin the bare-CR header dump on the JSON path too
The Gateway runs with logging.enhance.format=json, and JsonTraceFormatter
renders the exception itself, so the text-format case alone left the deployed
emission unguarded for an escaped bare-CR payload. Record the measured cost of
reordering the break alternation, which obs-fold continuations depend on.
---------
Co-authored-by: sxh313 <sxh313@users.noreply.github.com>
* fix(projects): label untitled threads in the conversation-files view
GET /api/projects/{id}/threads/files returns each member thread's
display_name as `str | None`: the thread meta row is created with NULL and
only filled once title generation has run after the first reply, or never
if that step fails. The frontend's ProjectThreadFileGroup declared the field
as a required string and ConversationFileGroup rendered it verbatim, so a
thread whose files were uploaded before its first reply showed them under
an empty heading. The sibling ProjectThread type already models the field
as nullable and every consumer of it falls back to t.projects.untitled.
Make the group type nullable to match the wire, and render the heading
through a shared threadTitle() helper that both the thread-name map and the
file group now use, so the "Untitled" rule lives in one place.
The DOM test covers null, missing and whitespace-only titles.
* docs(changelog): reference #5879 in the conversation-files entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): unlink a symlinked custom skill package on delete
The storage contract (_is_external_skill_directory_symlink) accepts a
one-level symlink under custom/ that points at an operator-managed skill
package, and every read path honours it: the skill loads, lists, validates
and can be enabled. delete_custom_skill, however, removed the package with
shutil.rmtree, which refuses symlinks. The delete raised OSError after the
history record had already been appended, skill_projection_mutation cleared
the user's projected custom view on the way out (rebuilt on the next
acquire), and DELETE /api/skills/custom/{name} answered 500 — for both
LocalSkillStorage and UserScopedSkillStorage, which inherits the method.
Unlink the entry when it is a symlink and keep rmtree for real directories.
The external tree is not ours to delete: removing the skill means removing
the link. Checking is_symlink() first also covers a dangling link, which
exists() would report as absent.
Tests cover both storages: the link is gone, the skill no longer exists,
the external package (SKILL.md and a support file) is untouched, the
history record was written, and the sibling skill's projected view
survives the delete.
* docs(changelog): reference #5881 in the symlinked-skill delete entries
* feat(community): migrate tavily tools to AsyncTavilyClient
Replace the synchronous TavilyClient with AsyncTavilyClient in the tavily
community tools, as tracked in backend/docs/TODO.md planned features
('Replace sync requests with httpx.AsyncClient in community tools').
web_search_tool and web_fetch_tool become async functions and await
client.search()/client.extract(). External behavior (return values,
error strings, credentials handling) is unchanged; the client is still
constructed with api_key=None when the tool config has no key, letting
the SDK read TAVILY_API_KEY from the environment.
Tests are updated to invoke the async tools via ainvoke under
@pytest.mark.anyio, and the credential assertions now verify the client
is constructed with the right api_key (AsyncTavilyClient keeps the key
inside its httpx client and exposes no public api_key attribute).
* fix(tavily): close async clients after each tool request
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(dev): keep whole worktree paths when serve.sh builds DEERFLOW_ROOTS
serve.sh collects every deer-flow worktree root so `make stop` / `make dev`
can reclaim the shared dev ports (8001/3000/2026) from a sibling checkout.
It parsed `git worktree list --porcelain` with `awk '{print $2}'`, but that
output does not quote paths, so a worktree at `.../deer flow two` was
recorded as `.../deer`. `_is_deerflow_pid` matches open files against
"<root>/", which the truncated root never satisfies, so a Gateway or
frontend started from that worktree was treated as a foreign process and
the start aborted with "port already in use".
Take the remainder of the line after the `worktree ` key instead. The
dedupe and longest-first ordering that follow are unchanged.
The new test builds a scratch repo with a spaced linked worktree and runs
the DEERFLOW_ROOTS block from serve.sh against it, from both the main
checkout and the spaced one.
* docs(changelog): reference #5856 in the worktree-roots entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): never let a malformed files[*].size abort the run
UploadsMiddleware._files_from_kwargs validates every field of a
client-supplied file entry fail-soft — non-dict entries, non-basename or
staging filenames, and files missing from the uploads dir are all skipped —
except `size`, which went straight to int(). A value int() cannot convert
("abc", a list, inf) raised ValueError/TypeError/OverflowError out of
before_agent (and the executor-wrapped abefore_agent), failing the whole run
before the model was called. The Gateway strips only server-owned metadata
keys and edit/regenerate deep-copies `files` verbatim, so the same entry
failed again on every retry of that message.
The size only feeds the human-readable "(N KB)" line in <current_uploads>,
so route it through _coerce_file_size: a non-negative finite number (or a
numeric string such as "2048") is kept as int; None, bools, negatives,
non-finite floats and anything unconvertible fall back to 0, the value a
missing size already gets.
Tests pin the fallback for each unusable shape, the kept values, and that
before_agent still injects an existing upload whose size is malformed.
* docs(changelog): reference #5855 in the uploads size entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
bind_task_tool and _bind_batch_tool copied their process-wide singletons
with only the coroutine rebound. Once get_available_tools had wrapped a
singleton in place with a sync func, the copy inherited that wrapper
around the unbound coroutine, so synchronous invocation bypassed the
explicit SDK submitter/capacity and fell through to the process-global
one -- or raised NotImplementedError in a fresh process where the
singleton was never wrapped.
Rebind the copy's func to a sync wrapper around the bound coroutine so
both invocation paths use the runtime's own submitter and capacity.
* fix(sandbox): list the skills root when only category mounts exist
With the default (non-policy-scoped) LocalSandboxProvider layout, the
four skill categories (/mnt/skills/public, custom, legacy, integrations)
are mounted at separate host directories and there is no mapping for the
aggregate /mnt/skills root itself. LocalSandbox.list_dir resolved the
root to the unmapped host path, and the host list_dir raised
FileNotFoundError before the virtual sub-directory overlay -- which
exists exactly so the agent can discover the categories via
"ls /mnt/skills" -- could run, so the ls tool reported "Directory not
found: /mnt/skills" even with skills installed.
Treat a FileNotFoundError from the host scan as an empty listing when
mappings are mounted inside the requested container path, so the overlay
can surface those children. Paths with nothing mounted inside them keep
raising FileNotFoundError as before.
* fix(sandbox): preserve missing virtual parent errors
* test(screening): align PII fixture with keyed placeholders
* fix(sandbox): preserve non-directory errors for virtual parents
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): migrate infoquest client to httpx.AsyncClient
Replace blocking sync requests.post calls with httpx.AsyncClient in the
InfoQuest community tool, as tracked in backend/docs/TODO.md planned
features. fetch/web_search/image_search and their raw-results helpers are
now async, keeping the 30s transport timeout and external behavior
(return values, error strings) unchanged.
The langchain tool wrappers in infoquest/tools.py become async functions;
web_fetch offloads readability extraction to a thread (same pattern as
the already-migrated jina_ai tool). Tests are updated to mock
httpx.AsyncClient instead of requests.post.
* test(community): use AsyncMock for infoquest fetch in relative-links test
The infoquest web_fetch_tool now awaits client.fetch, so the test stub
must return an awaitable instead of a sync lambda.
* fix(community): follow InfoQuest API redirects like the sync client did
Address the review note on #5782: requests.post follows redirects by
default while httpx.AsyncClient defaults to follow_redirects=False, so
a 3xx answer from the reader/search endpoints would have surfaced as
'Error: fetch API returned status 30x' (or fallen into the generic
error path after raise_for_status) instead of the final content.
All three AsyncClient call sites now pass follow_redirects=True. Adds
regression tests asserting each client is constructed with
follow_redirects=True, and documents the parity in the test contract.
* test(community): fix InfoQuest mock endpoint typo
* fix(release): refresh backend/uv.lock when bumping the version
`scripts/bump_version.sh` rewrote backend/pyproject.toml,
frontend/package.json and the Helm chart's `version`/`appVersion`, but
`backend/uv.lock` also records the root package version (uv keeps its PEP 440
form, so `2.1.0-rc0` is stored as `2.1.0rc0`). The documented release step left
the lock pinning the previous version, and that is not cosmetic drift:
`uv lock --check` (lint CI) fails on a stale lock and `uv sync --locked`
(e2e / plugin workflows) refuses the tree, so the release commit could not pass
CI. RELEASING.md only mentioned the manual `cd backend && uv lock` step for
release candidates.
The script now refreshes the lock via `uv lock` and refuses to start without
`uv` — it exits before editing anything rather than leaving a half-bumped tree
whose lock CI rejects. `uv lock` stays the authority for the normalized
prerelease form; the lock diff for a bump is a single line. RELEASING.md and
AGENTS.md now describe the lock as part of the bump.
Tests: backend/tests/test_bump_version_lockfile_sync.py (red on main, green
here) pins both halves — the lock moves with the sources, and a missing uv is an
error instead of a silent no-op.
* docs(changelog): record the release bump lockfile fix (#5859)
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(memory): redact queued memory payloads (pii_redaction follow-up)
Closes the remaining vector from #3190 (memory storage): request-scoped
redaction leaves originals in thread state, and MemoryMiddleware queued
that raw conversation into the durable extraction buffer — so the
extraction model's input and the persisted facts carried raw PII even
with pii_redaction.enabled.
- MemoryMiddleware gains pii_redaction_config (same pattern as
summarization / durable-context / title); the payload returned by
_resolve_add_args() is redacted at the enqueue boundary before
add()/aadd(), covering both after_agent and aafter_agent
- Reuses the shared redact_text() seam from #5527 via
active_pii_detectors/_Redactor/_redact_content; the buffered payload
itself becomes clean, not just the extraction model's view
- Wired at the lead-agent assembly site
- Tests: queue-payload redaction enabled/disabled, detector toggles,
message immutability, token stability across turns
* fix(memory): reserve existing placeholder indices across the queued batch
Review round 1 on #5577: thread state mixes raw user turns with
already-redacted tool results and summaries, so a token in a later
message must not collide with a new value in an earlier one. Pre-scan
the whole queued batch with redactor.reserve() before allocating.
Also extend the two MemoryMiddleware stubs in
test_lead_agent_model_resolution.py to accept the new
pii_redaction_config kwarg (CI backend-unit-tests red).
* fix(memory): redact the compaction-triggered memory_flush_hook queue too
Review round 2 on #5577: with memory + summarization enabled,
memory_flush_hook() queues list(event.messages_to_summarize) directly
into MemoryManager.add_nowait(), bypassing MemoryMiddleware entirely —
and compaction removes those messages from state right after, so the
after-agent redaction can never repair a raw batch queued there.
Extract the batch redaction (reserve pre-scan + rebuild) into a shared
redact_queued_messages() used by both MemoryMiddleware's enqueue
boundary and the hook; the hook gains an optional pii_redaction_config
wired via functools.partial at the summarization factory (subagents use
skip_memory_flush=True and are unaffected). Regression covers the hook
path enabled/disabled/absent.
* test(summarization): unwrap the partial in the factory hook identity check
edd916e wrapped memory_flush_hook in functools.partial to carry the
pii_redaction config, but missed committing the matching update to
test_factory_attaches_memory_flush_hook_by_default — a functools.partial
is not equal to its wrapped function, so the identity assertion failed
deterministically (caught by static review on #5577).
* fix(memory): cover tool-call arguments and lenient text blocks in queued payloads
Review round 4 on #5577 — two reproduced gaps still allowed raw PII into
memory payloads with pii_redaction.enabled=true:
- OpenViking retains the full message object, so parsed tool_calls args
and provider-format arguments in additional_kwargs bypassed the
content-only redaction; queued payloads now redact tool-call args
recursively (and reserve them in the pre-scan).
- DeerMem's format_conversation_for_update reads p.get("text") of any
dict block, not only type=="text" — _redact_content now redacts every
string-valued text field so a lenient consumer cannot bypass it.
Both repro shapes are pinned as regressions.
* fix(pii): value-derived placeholders for stable capture identity
Review round 5 on #5577: sequential allocation is order-dependent — the
same unchanged message re-redacted in a later batch can get a different
token, which flips OpenViking's content-signature and replays already
recorded turns into persistent memory sessions. Placeholders are now a
pure function of the matched value (sha256-6 per category): identities
stay stable across batches, enqueues, seams, and backends with no shared
state and no stored mapping.
- _Redactor becomes stateless; the reserve machinery (index pre-scans,
_PLACEHOLDER_PATTERN) is removed along with its call sites
- redact_texts keeps its signature, now a simple per-text redaction
- reserve-semantics tests are superseded by cross-enqueue identity tests
(two successive captures where the second batch carries higher existing
placeholders); the compiled-graph echo regex matches the new format
* fix(pii): 128-bit placeholder digests; redact tool-argument keys
Review round 6 on #5577:
- [P2] The 24-bit truncation collided distinct identities (reproduced:
contact3513@ / contact3727@example.com both rendered [EMAIL_575dbb]).
Retain 128 bits of the digest so separate people stay separate at any
realistic volume; the compiled-graph echo regex and the AGENTS.md entry
follow the new format.
- [P2] _redact_strings redacted values but preserved mapping keys, so
{"contacts": {"alice@example.com": "manager"}} kept the raw email
as a key. Redact string keys along with values; deterministic 128-bit
tokens make distinct-key collisions a hash-collision non-event, and a
redacted key equal to another key merges the entries. Regression pins
the email-keyed arguments shape.
* docs(agents): trim PiiRedactionMiddleware entry; drop stale reserve wording
The chain grew past the 98304-byte AG002 hard limit again (main growth
plus the round-6 format example). Trim the entry to fit, and replace the
stale reserve-indices wording with the value-derived semantics that
replaced it.
* docs(agents): shrink PiiRedactionMiddleware entry below the chain budget
main moved (#5609) and the base comparison flipped the nearly-over chain
back to an AG002 error (98441 > 98304). Trim the entry to the facts that
are not documented elsewhere: seams, allowlist, order, enqueue boundary.
* fix(pii): letters-only value-derived tokens; wire the factory SDK path
Review round 7 on #5577:
- [P2] Hex digests carry digit runs that later digit-anchored detectors
re-scan, corrupting minted tokens into nested placeholders (reproduced:
user280@example.com -> [EMAIL_…[PHONE_…]…], ~1/140 emails). The 128-bit
digest is now encoded over a-z, so a minted token survives the full
pinned detector order.
- [P2] create_deerflow_agent (the SDK path) constructed MemoryMiddleware
and DurableContextMiddleware without pii_redaction_config; the knob is
now an explicit optional parameter threaded through
_assemble_from_features (the factory reads no config files by design,
so SDK callers pass it explicitly).
- [Nit] hoist the Command rebuild comprehension out of the per-message
loop in _redact_result.
- [Nit] drop the dead _iter_tool_calls helper and Iterator import.
* test(factory): pin pii_redaction_config wiring; document the parameter
Review round 8 on #5577:
- [P3] test_create_deerflow_agent.py had zero coverage of the
pii_redaction_config parameter; two regressions now assert that
create_deerflow_agent(pii_redaction_config=cfg) builds a
MemoryMiddleware and a DurableContextMiddleware carrying cfg, and that
the default is None (redaction off).
- [Nit] document the parameter in the create_deerflow_agent docstring:
None leaves redaction off; a middleware= takeover bypasses the
assembly that would apply it.
* fix(pii): HMAC-scoped placeholder tokens; install model-bound redaction in the SDK chain
Review round 10 on #5577:
- [P2] sha256(category||value) is an unkeyed, globally deterministic
fingerprint: a provider or memory backend can confirm candidate values
offline and link identical values across users and threads, contradicting
the documented no-re-identification guarantee. Placeholders are now
HMAC-derived over a deployment-scoped token_secret (new config field);
the docstring documents the linkability scope for both modes.
- [P2] _assemble_from_features() never appended PiiRedactionMiddleware, so
the SDK knob enabled only the downstream seams while the primary model
still received raw content. The middleware is now appended in the same
relative position as the lead-agent base chain when the gate is on,
with a model-call regression through the assembled SDK chain.
* docs(pii): document token_secret; update the placeholder format examples
Follow-up to the round-10 review on #5577:
- config.example.yaml documents the new token_secret knob: a
deployment-scoped secret makes placeholders HMAC-derived and only
linkable within one deployment; leaving it unset keeps publicly
computable fingerprints (documented trade).
- the PiiRedactionConfig class docstring's placeholder example now
matches the shipped format (128-bit HMAC digests over base-26
letters).
* docs(pii): update the module docstring's placeholder format example
The module docstring still showed the sequential [EMAIL_1] example;
align it with the shipped value-derived format (128-bit HMAC digests
over base-26 letters).
* fix(pii): require a non-empty token_secret when redaction is enabled
Review round 11 on #5577: with the secret optional, the default enabled
configuration minted empty-key HMAC digests — publicly computable, globally
linkable fingerprints of the raw values, confirmable by offline guessing for
low-entropy identifiers such as phone numbers.
- PiiRedactionConfig now rejects enabled=true without a non-empty
token_secret (minimum 16 characters); the unkeyed mode is gone entirely.
- The request-scoped redactor in _process_request now carries the configured
key: it built its redactor without it, so user-message placeholders
silently diverged from every other seam for the same value. The empty-key
guard in _placeholder_token is the backstop that keeps every path keyed.
- Module/config docstrings and config.example.yaml no longer describe the
placeholders as irreversible or the secret as optional; the stale
reserve-semantics wording and the "memory path not covered" note are
corrected.
- Regressions: validation errors for missing/blank/short secrets, and the
user-message boundary minting the same keyed token as redact_text.
* fix(memory): redact original_user_content on queued memory payloads
Review round 12 on #5577: UploadsMiddleware preserves the raw user turn in
additional_kwargs.original_user_content, but redact_queued_messages rewrote
only content, tool_calls, and provider-format tool_calls — the queued copy
still carried the raw value, so the memory queue and backend received raw PII
with redaction enabled.
The shared helper now redacts the provenance string field on the queued copy
(same batch redactor, merged with the existing additional_kwargs rewrite,
rebuilt via model_copy — thread-state messages stay untouched), covering both
the enqueue boundary and the compaction flush hook. Regressions pin the
redacted provenance field through both paths and the untouched original.
* fix(memory): redact human_input_response on queued memory payloads
Review round 13 on #5577: clarification replies keep the raw user answer in
additional_kwargs.human_input_response.value (DeerMem reads that mapping via
its read_human_input_response mirror), but redact_queued_messages rewrote
only content, tool_calls, and original_user_content — the queued copy still
carried the raw value into the memory queue and backend.
The shared helper now rebuilds that mapping with value redacted (same batch
redactor) while preserving the protocol fields (version/kind/source/
request_id/option_id/response_kind), merged with the existing
additional_kwargs rewrite and rebuilt via model_copy so thread-state
messages stay untouched. Regressions pin the redacted value, preserved
protocol fields, the untouched original, and the flush-hook path.
* chore(config): bump config_version to 47; generate token_secret on upgrade
Review round 13 on #5577: existing deployments can legitimately sit at
config_version 46 with pii_redaction.enabled: true and no token_secret; the
mandatory-secret validator would fail their startup while config.example.yaml
still said 46, so the upgrade checker considered them current and emitted no
migration warning.
- Bump config_version to 47 so the outdated-config warning fires for v46.
- Add a v47 migration to scripts/config-upgrade.sh: when pii_redaction is
enabled and token_secret is missing/blank, generate a random deployment-
scoped secret (secrets.token_urlsafe) and persist it, so the upgraded
deployment starts under the mandatory validation.
- Regressions: a v46 enabled config is flagged outdated against the real
example; the upgrade run generates a >= 16-char secret and the result
passes AppConfig validation; a disabled config gets no secret.
- Also switch the two v41 script tests from a hardcoded "bash" to
SCRIPT_BASH/skipif, matching the documented Windows constraint and the
neighboring tests (hardcoded bash resolves to the WSL stub there).
* chore(helm): sync chart config_version to 47
Review round 14 on #5577: check_config_version.sh fails whenever the chart's
embedded config_version lags config.example.yaml, and the chart workflow is
triggered by config.example.yaml — so the v47 bump left this head failing
that check. Update the values.yaml config block and the README example to
47; scripts/check_config_version.sh now passes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Follow-up to #5833 after an independent review of the merged code.
- A malformed answer (deeply nested JSON raising RecursionError, or a
huge noul integer raising OverflowError outside the try) escaped the
fail-open path, and in the per-message task group it cancelled the
other messages' requests and dropped every flag of that tool call.
Parsing now happens inside the try, which also catches those errors.
- The 16 KiB response cap counted decoded bytes, so a compressed body
could expand far past it first. Requests now ask for identity encoding
and drop encoded responses.
- A plain-HTTP loopback endpoint followed HTTP_PROXY/ALL_PROXY, sending
the bearer key and excerpt to the proxy in cleartext. Such endpoints
now ignore proxy environment variables; HTTPS endpoints keep them.
Tests cover each case, including a real local server behind an
unreachable proxy, and pin previously untested behavior: independent
per-request deadlines, the sync hook without a task store, the >=
threshold and all four named remote tools, warnings for messages that
share a tool-call ID, consumed flags, and scanning past host messages.
The root README no longer says the excerpt is always PII-redacted, and
the example README documents the new limits and that the sync path's
deadline does not cover a stalled DNS lookup.
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* chore(version):update the version to 2.2.0-dev
* chore(doc):Updated the CHANGLOG with the 2.1.0 release version
* chore(doc): Updated the CHANGLOG for the latest changes
* chore(doc): Added the release SOP document
* feat(extensions): grant bounded host model invocation
* fix(extensions): retain model budgets and isolate schema validation
* fix(extensions): normalize provider-origin model cancellation
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): strip section anchors from code-span resource refs
Markdown-link targets already drop their #fragment before the
reference is resolved, but code-span refs did not: a prose reference
like `references/faq.md#pricing` kept the anchor and was resolved as
a literal path, producing a resource.missing finding for a file that
exists and is referenced. The bare-path token scanner stops at '#',
so only the code-span branch leaked fragments.
Split the fragment off code-span tokens the same way the link branch
does, then strip trailing sentence punctuation. Real dotted filenames
keep their dots: the split happens at the first '#', and '.' is only
stripped from the very end.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): prefer exact hash-bearing filenames over fragment stripping
Package paths permit '#', so a code-span reference to a real file like
references/C#.md was truncated to references/C by the fragment split,
producing both a false resource.missing and an orphan report (the bare
path token stopped at '#' the same way). Keep the whole reference in
every extraction branch (link target, code span, bare path token now
includes '#' in its character class) and resolve the fragment
existence-aware: prefer the exact token when it names a real file and
treat the suffix after the first '#' as a section fragment only when
it does not.
Red/green: a new test pins references/C#.md resolving to the real file
while references/faq.md#pricing still strips to faq.md — it fails on
the reviewed head and passes with the change; full
test_skill_review_core.py is 36 passed.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): strip the fragment before normalizing the fallback path
The exact-path preference checked the normalized reference, so a token
like references/faq.md#/../other.md normalized to references/other.md;
when that file existed the check succeeded and the fragment-removal
branch was skipped, recording the wrong edge and orphaning faq.md.
Check the literal token against the package files instead, and drop
the fragment before the remaining path is normalized — normalizing a
hash-bearing segment must never retarget an edge.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): canonicalize leading segments in the hash-filename lookup
The literal-file check compared the verbatim joined path, whose leading
relative segments are uncanonicalized: from references/sub/guide.md, a
reference to ../C#.md joins to references/sub/../C#.md, matches no
snapshot key, and falls through to fragment stripping — truncating the
real references/C#.md into a false missing and an orphan. Compare
against the normpath-canonicalized whole token instead, and treat the
post-'#' text as a fragment whenever it is path-like (carries / or ..)
so it can still never participate in normalization and retarget an
edge.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): check the canonical whole path before stripping a fragment
The path-like-fragment heuristic dropped any post-'#' text carrying '/':
a real package file such as references/C#/readme.md (a '#' inside a
directory name) was truncated to references/C, producing a false
resource.missing and orphaning readme.md. Only a '..' segment in the
post-'#' text forces the strip-first fallback — that is the one case
where fragment text participates in normalization and can retarget an
edge (faq.md#/../other.md). Every other fragment first gets the
canonicalized whole-token lookup, which already canonicalizes leading
relative segments for nested sources.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): always strip markdown link fragments and blind the bare-path pass to links
Markdown link targets have fixed fragment semantics: the text after '#'
is a URL fragment, never part of the filename, so a link to a file
literally named references/faq.md#pricing would have to percent-encode
it. The exact-file branch applied the hash-filename preference to link
targets too, recording a valid edge and suppressing resource.missing
for a broken link when a literal same-named file existed. Links strip
the fragment again, and their full construct is blanked out of the
residual text so the code-span and bare-path passes — literal-path
contexts where '#' may be part of a real filename — never see
link-internal text.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(browserless): read reject_resource_types / reject_request_pattern from config
web_fetch_tool declared both parameters, initialised them to None, and passed
them to the client -- but never read them from tool config, so they were always
None on the wire. BrowserlessClient implements them (and is tested for it at
`test_fetch_html_with_reject_params`), so the provider's resource blocking was
unreachable without patching the source. Skipping images/media/fonts is the
main lever for render latency and bandwidth against a self-hosted Browserless,
and it could not be turned on from config.yaml.
Read both keys through a new `_as_str_list` helper that accepts a YAML list or
a comma-separated string and yields None for anything unusable, so an unparseable
value omits the parameter instead of reaching the server. Both keys are now
documented on the Browserless web_fetch entry in config.example.yaml.
Default behaviour is unchanged: an unset key is still None, so the payload is
identical to before for every existing config.
Adds four regression tests covering the list form, the comma-separated form,
the unset case, and unusable values.
* fix(browserless): validate reject list entries
---------
Co-authored-by: RXQ6 <RXQ6@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(config): sign the bytes the config cache parsed, not a second read
_load_and_cache_app_config parsed config.yaml through AppConfig.from_file and
then called get_config_signature, which opens and hashes the file again. A
write landing between those two reads left the cache holding the older
content under the newer content's signature. get_app_config() compares the
on-disk signature against the recorded one, so from then on it saw
"unchanged" and the edit only surfaced when yet another edit changed the
signature — silently stale until then, with no self-healing.
Read the file once through a new shared helper,
file_signature.read_config_with_signature, which returns the bytes together
with the (mtime, size, sha256) of exactly those bytes, and parse them through
AppConfig._from_yaml_text (from_file's body, split out; from_file itself is
unchanged). The recorded signature now always describes the parsed content:
a write that races the load can only produce a signature mismatch on the
next call, i.e. one extra reload. Signing before parsing but re-reading for
the parse would not be enough — an edit that lands between the two reads and
is then reverted would be served forever — so the tests pin that the parsed
bytes are the signed bytes.
The stale-metadata reload test now routes the loader's read through the same
stale-stat simulation it already applies to the probe, since the loader no
longer records the probe's result.
* docs(changelog): reference #5848 in the config cache entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(authz): add plugin resource targets, provenance, action and management enforcement
PR1 of the extension-tool-authorization series
(docs/plans/2026-09-24-extension-tool-authorization-spec.md): plugin
authorization plumbing, no new author-facing declaration.
- `authz/plugin_targets.py` is the single encoder/validator for composite
targets (`namespace/part`), with one validating composer behind three
kind-checked constructors; invalid input raises instead of degrading.
- `authz/plugin_authz.py` is the harness decision layer (`plugin_action`,
`plugin_management`, plus the batch filters the page slice will consume):
disabled is a no-op, an explicit deny raises, and provider errors follow
`fail_closed` like the sandbox gate.
- `POST /api/plugins/{ns}/actions/{name}` decides `plugin_action:invoke`
after the action resolves and before the body is streamed.
- `deerflow_extension_api.auth` publishes `require_plugin_management` /
`arequire_plugin_management` with two `app.state` resolver keys; the
Gateway installs both. Contract version 0.2.3 -> 0.2.4.
- `tools/tool_provenance.py` records host-written plugin provenance for
display (assembly descriptor, Layer-2 context later); no decision reads a
label.
- Plugin providers use a `(config signature, loop key)` cache so a
loop-affine provider is never handed to another loop or to a sync caller.
- RBAC gains `plugin_actions` / `plugin_management` aliases;
`config_version` 47 -> 48 across the three mirrors.
Tests: new plugin target/provenance/action/management suites plus a
blocking-IO anchor whose offloads were reverted to prove red->green.
* fix(authz): decide plugin resources from the request's config snapshot
Review round 1 on PR #5842 (willem-bd).
P1 — the decision layer no longer reloads the configuration. The six
`authz/plugin_authz.py` functions take a required `app_config` snapshot (the
same request-scoped config the Gateway used to resolve the provider), and
`resolve_/aresolve_plugin_authorization` return it as a third element, so
`enabled`/`fail_closed` can never come from a second read that drifted from
the one that produced the provider. A snapshot the caller could not read at
all is *unavailable*, not disabled: it fails closed as
`authz.config_unavailable`. The Gateway's `_plugin_app_config` now separates
an absent `config.yaml` (no gate, today's behavior) from a config that exists
but cannot be read or validated (deny) instead of swallowing both through
`safe_app_config`.
P2 — `_validated_decision` requires `type(decision.allow) is bool` on both
the sync `authorize` and async `aauthorize` paths, so a provider returning
`AuthzDecision(allow="false")` is a malformed verdict that follows
`fail_closed` instead of being read as an allow.
Both fixes are covered by regression tests that were verified red against the
pre-fix behavior (P1: 200 vs 403 and resolver True vs False; P2: 12 cases
across fail_closed/fail_open and sync/async), plus tests pinning that an
absent config and `authorization.enabled: false` keep today's behavior.
* fix(authz): treat a request-time config absence as unavailable
Review round 2 on PR #5842 (willem-bd).
P1 — `_plugin_app_config[_async]` no longer maps a missing `config.yaml` to
`None` ("no gate"). A Gateway cannot have started without a readable config
(`app/gateway/app.py` reads it through this same accessor in `lifespan`), so
absence *during a request* means the configuration this process was running on
became unavailable — a hot reload, an atomic replace, a ConfigMap remount. It is
unavailable, not disabled: the error propagates,
`resolve_/aresolve_plugin_authorization` raise
`_PluginAuthorizationUnavailable(fail_closed=True)`, the action dispatcher
returns 403 and the management resolvers answer `False`. The dead
`(None, None, None)` allow path is gone, the resolvers return a non-optional
`AppConfig` as their third element, and callers key "authorization disabled" on
`provider is None` alone. This matches the rule the decision layer already
documented (`authz.config_unavailable`) and the public docs.
P2 — `_deny_reason_code` is total: a `reasons` that is not iterable degrades the
code to `authz.denied` instead of raising `TypeError` outside the guarded
provider path, which turned a denial into an unhandled error. It is deliberately
not folded into `_validated_decision`: `allow` was a real `bool`, so the verdict
is a valid denial, and treating it as malformed would flip it to an allow under
`fail_closed: false`.
Both fixes are covered by regression tests verified red against the pre-fix
behavior (P1: resolver True vs False, async `PermissionError` not raised, route
200 vs 403; P2: `TypeError: 'NoneType' object is not iterable` at
`plugin_authz.py:113`). The action-route tests in
`tests/test_plugin_contributions.py` now install a readable
`authorization.enabled: false` config instead of relying on a config-less
process, and `docs/full-stack-plugins.md` plus the PR1 decision log record the
new rule.
* fix(authz): apply the running policy when the plugin config is unreadable
Follow-up to f03d2ea9 (review round 2 on PR #5842, willem-bd).
The first attempt read the P1 finding as "absence is never disabled" and failed
every config read closed unconditionally. That is too broad for this repo: a
config-less host is deliberately treated as having no policy to apply
(`_get_route_authorization_config` falls back to a disabled config,
`sandbox_authz.safe_app_config` documents the same no-op for "CI runners and
direct-call tests"), and the rule turned the config-less e2e plugin host
(`extension_test_fixtures.bookmark_plugin_gateway`) into a 403 — the three
`bookmark-plugin.spec.ts` cases lost the save action. Reproduced locally:
`POST /api/plugins/community.bookmarks/actions/save` against that fixture host
returned 403 before, 200 after this change.
The rule is now "has this process loaded a configuration?", not "was the last
read successful?":
- No loaded config (`peek_loaded_app_config()`, new read-only accessor on
`deerflow.config.app_config`) and no `config.yaml`: no policy to apply, so the
gate stays a no-op — unchanged for CI runners, direct-call tests and hosts
that mount only the plugins router.
- A loaded config whose file disappears or cannot be read: *unavailable*, never
*disabled*. The resolvers raise `_PluginAuthorizationUnavailable` carrying the
failure flag of the configuration this process runs on (authorization
disabled there -> no gate; enabled -> that policy's `fail_closed`, `True` by
default), so an enabled Gateway answers 403/False in the window instead of
letting protected plugin actions and management routes through.
- A config file that exists but cannot be read or validated right now: fail
closed when nothing is loaded (round 1's rule), otherwise the loaded policy
decides.
`peek_loaded_app_config()` is preferred over a Gateway-side startup snapshot: it
reflects the last successfully loaded config, so a hot-reloaded policy is what
applies during the window, and there is no lifecycle hook to keep in sync.
Evidence: the three no-config/lost-config branches in
`tests/test_plugin_management_guard.py` and
`tests/test_plugin_action_authorization.py`, round 1's unreadable-config tests
unchanged, `test_peek_loaded_app_config_survives_a_missing_file` in
`tests/test_app_config_reload.py`, and the fixture-host probe above. Docs
(`docs/full-stack-plugins.md`, `reference.mdx` en/zh) and the PR1 decision log
state the final rule; the log records the superseded variant.
* fix(authz): keep a denial whose reasons raise while being read
* fix(authz): intersect plugin batch filters with the candidates
The plugin_page/plugin_management batch helpers returned the provider's
`filter_resources` answer verbatim, so a custom provider could hand the
projections a target that was never asked about ("*", another namespace,
a phantom page). Every other filter call site re-projects from its own
candidate list; do the same here, state the intersection on the provider
contract, and pin it with a regression test that fails on the old
pass-through.
* docs(plugins): map a management denial to 403 in the route example
The contributed-route example called `arequire_plugin_management` and let
the documented `PermissionError` escape. The Gateway installs no
`PermissionError` handler, so a route copied from the example answered 500
on a normal policy denial instead of 403. Both example routes now translate
the denial, and the surrounding bullets say why the translation is the
route's job.
* docs(extensions): point the 0.2.4 contract row at the PR that ships it
Every other row in the version-history table links the pull request that
shipped the version; 0.2.4 linked the authorization tracking issue instead.
Both the English and Chinese tables now link this PR.
* feat(examples): add opt-in fetched-content screening extension
* fix(examples): screen sync and async results through supported middleware hooks
* fix(examples): preserve empty message IDs when applying screening warnings
* fix(examples): include screening policy in assembly identity
* feat(examples): load fetched-content screening through the extension entry
Package the screening example as a plugins: extension instead of an
operator-configured middleware. install() validates the private config
and, once enabled, contributes one TOOL_VISIBLE middleware for lead
agents and subagents. The package imports only deerflow_extension_api.
Extension tool wrappers are observational, so detection and the warning
use two supported contracts: the tool wrapper classifies the visible
result and records flagged tool-call IDs in the run's task store, and
before_model replaces the matching messages of the latest tool step by
ID. The excerpt sent to Jev is now the redacted, sanitized text the
model sees. Runs without a task store send nothing. Unexpected local
errors surface as isolation diagnostics; provider errors fail open.
Tests load the package through load_extensions() and the host isolation
wrapper and run real lead/subagent graphs with a task store bound as the
Gateway worker binds it. The README records what the example exercised
in the extension API and the gaps it found.
* fix(examples): classify each tool message of a screened result on its own
A Command result was classified as one concatenated excerpt and the flag
landed on its first text message, so a benign first result could take the
warning while an injected later result reached the model unmarked.
Each text message now gets its own bounded excerpt and Jev request, for at
most eight messages per tool call. The requests share one client and run
concurrently in a task group, each under its own deadline, so one slow or
failed request cannot hide another message's flag. Every flagged message's
tool-call ID is recorded. The message cap joins the declared policy, and the
README describes the per-message requests and data sharing.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(uploads): keep document-conversion stat and write-back off the event loop
convert_file_to_markdown offloads the conversion itself for files above
1 MB (documented in its docstring), but the two bookend filesystem calls
still ran on the event loop: the size stat before the threshold check and
the write of the converted markdown afterwards — the latter a potentially
multi-megabyte write in the per-upload ingestion path. Route both through
run_file_io (the dedicated file IO pool that also keeps the caller's
context), matching the off-the-loop fixes already applied to the sibling
upload/document paths.
Add a tests/blocking_io anchor: with the converter patched to a 2 MB
payload it fails on the previous head (Blockbuster catches the blocking
os.stat first; the broad except swallows it into a None return) and
passes with this change.
* fix(uploads): address review — drain conversion write-back, make run_file_io re-entrant
Review follow-up on the off-loop conversion stat/write PR:
1. The write-back now goes through await_drained(): a plain run_file_io
await cancels immediately while the worker keeps writing, so the
ingestion scope could remove its private directory over an in-flight
write and leave a large hidden partial file. The drain delivers the
cancellation only after the worker settled, and the handler removes
the partial output before re-raising.
2. run_file_io is now re-entrant: a nested offload issued from code
already running inside a file-io worker (project-document conversion
runs asyncio.run(convert_file_to_markdown(...)) inside a run_file_io
worker) executes inline instead of submitting to the same bounded
executor, which deadlocked with a single configured worker.
Both fixes carry deterministic regressions: the cancellation test pins
the drain ordering (cancel issued before the worker is released) and the
partial-file cleanup; the re-entrancy test pins inline execution by
thread identity. Both fail on the previous head. Full blocking_io +
project-document + conversion suites show an identical failure set to
the previous head (pre-existing Windows platform failures only).
* fix(uploads): keep cancellation authoritative when conversion cleanup fails
If the post-cancellation unlink of the staging output raised, the new
exception escaped the CancelledError handler, was swallowed by the
broad except as an ordinary conversion failure (None return), and left
both the caller's cancellation and the partial file behind. Catch and
log cleanup failures inside the handler and always re-raise the
captured cancellation; a second cancellation still passes through
untouched (CancelledError is not an Exception).
Red/green: tests/blocking_io/test_file_conversion_offloop.py gains a
regression where unlink fails while the write-back is in flight — on
the reviewed head the task settled on None (DID NOT RAISE); with the
change the CancelledError propagates and the failure is logged.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* docs(backend): drop references to APIs that do not exist
PATH_EXAMPLES.md's upload example imports THREAD_DATA_BASE_DIR from
thread_data_middleware; the module exists but exports no such name, so the
snippet fails with ImportError as written. Resolve the same directory through
the accessor the middleware itself uses, get_paths().sandbox_uploads_dir(),
which yields the layout the file's own path table documents.
CONFIGURATION.md's Docker sandbox block lists `auto_start: true`. SandboxConfig
declares use/port/container_prefix/mounts/network but not auto_start, and the
provider is constructed as cls(**kwargs) from the caller of
get_sandbox_provider() rather than from config extras, so the key is accepted by
extra="allow" and then silently dropped. Remove the line from the example.
* docs(backend): resolve the uploads example through the owner-scoped bucket
get_paths().sandbox_uploads_dir(thread_id) without a user_id selects the legacy
threads/<id>/ bucket, but Gateway uploads land in
users/<user_id>/threads/<id>/user-data/uploads/, so the example imported cleanly
yet still raised FileNotFoundError for a normally uploaded file. Use
deerflow.uploads.manager.get_uploads_dir(), which resolves the effective user
and validates the thread id.
* docs(backend): show the real uploads path layout in every PATH_EXAMPLES demo
The remaining sections still documented the pre-isolation
threads/<thread_id>/ bucket and called `path` a backend-relative string, while
the upload and list APIs return an absolute path under
users/<user_id>/threads/<thread_id>/user-data/uploads/. Reproduced with
deerflow.uploads.manager.get_uploads_dir() and
uploads._list_uploaded_files_for_thread() against DEER_FLOW_HOME: the listed
path is /<home>/users/default/threads/abc123/user-data/uploads/report.pdf.
* docs(backend): clarify runtime home precedence
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(config): honor $VAR substitution for request_admission integer fields
AppConfig.resolve_env_variables replaces every `$VAR` reference with the raw
environment string. RequestAdmissionConfig.requests_per_minute and
max_queue_size are declared `strict=True` so a bool or float cannot slip in,
but strict mode also refuses the string "60", so `requests_per_minute: $RPM`
failed the whole config load with "Input should be a valid integer" even when
RPM=60 — contradicting config.example.yaml's promise that environment
variables work for every field. They were the only strict fields in the
config schema; every other integer coerces in lax mode.
Wrap both fields in a BeforeValidator that converts a decimal integer literal
arriving as a string and leaves everything else untouched, so the strict check
behind it still rejects bools, floats, "60.0" and non-numeric strings.
Tests pin the accepted literals, the rejections that must survive, and the
end-to-end path from a config.yaml `$VAR` reference through AppConfig.from_file.
* docs(changelog): reference #5838 in the request_admission entries
* fix(mcp): preserve backslash paths during bare-filename rewriting
* test(mcp): cover the trailing backslash boundary in bare-filename rewrite
Review noted that treating `\` as a separator also suppresses a rewrite when a
single trailing backslash precedes a newline, which is more often a Markdown
hard break, and that neither the tests nor AGENTS.md covered that position.
Lock the behavior in through the production result converter and document the
tradeoff: a backslash is a legal filename character on POSIX, so forgoing that
one rewrite is preferable to risking a corrupted path reference. Runtime
behavior is unchanged.
---------
Co-authored-by: lakers-abing <236384465+lakers-abing@users.noreply.github.com>
* fix(mcp): run lazy MCP discovery once when initialization itself raises
get_cached_mcp_tools() wrapped the whole lazy-initialization call in the
`except RuntimeError` that was only meant to catch asyncio.get_event_loop()
raising when no loop is set. run_until_complete() and Future.result() re-raise
the coroutine's own exception, so a RuntimeError from discovery — e.g. the
deliberately re-raised McpTaskConfigurationError — took the same fallback and
ran asyncio.run(initialize_mcp_tools()) a second time, respawning every stdio
server and re-fetching OAuth tokens before giving up. Inside a running loop
that fallback fails with "asyncio.run() cannot be called from a running event
loop", so the logged traceback hid the real cause.
Resolve the loop first, in its own try/except, and drive initialization exactly
once: asyncio.run when no loop is set or the current loop is closed, a worker
thread when the loop is running, run_until_complete otherwise. Any failure is
logged once and returns [] as before.
Regression tests pin one discovery call and the real logged cause with and
without a running loop, and keep both fallbacks (no loop set, closed loop)
covered.
* test(mcp): surface worker-thread failures in the no-loop fallback test
Thread.join() does not propagate exceptions, so a regression of the
asyncio.run fallback failed this test with a bare KeyError: 'result' instead
of the real traceback. Capture sys.exc_info() in the worker and re-raise it
on the main thread before the assertions.
* docs(checkpoints): add a bilingual checkpoint storage manual
Add the missing docs-site coverage for dual-mode checkpoint storage (`database.checkpoint_channel_mode: full | delta`) as a ten-chapter manual under `harness/checkpoints/` in both languages: concepts, quick start, channel modes, snapshot cadence, history cache, resume and rollback, operating checkpoints, observability, troubleshooting by symptom, and a reference of every configuration key, error message, HTTP status mapping, public symbol, load-bearing test, and pinned upstream defect.
The index page is `asIndexPage`, so `/docs/harness/checkpoints` is the section itself and its `_meta.ts` lists only the nine child chapters. Discovery: `harness/_meta.ts` registers the section, the harness configuration module table gains a `database` row, and the thread persistence section of `agents-and-threads` links to the manual.
Content is grounded in the code. Only the three measured values recorded in `backend/docs/runtime-guidance-details.md` are quoted; the upstream-derived 41x storage figure in the untracked delta-channel RFC is deliberately not cited. Pinned upstream defects are documented with their issue and PR numbers and the versions they are pinned at (`langgraph 1.2.9`, `langgraph-checkpoint 4.2.0`, `langgraph-checkpoint-postgres 3.1.2`).
* docs(checkpoints): correct the history-cache restart requirement
The history cache is built with the checkpointer at Gateway startup (`langgraph_runtime()` -> `make_checkpointer()` -> `make_checkpoint_cache()`), so editing `database.checkpoint_cache.*` does not change a running process. The manual previously marked the whole section "not restart-required", which conflated "not frozen for correctness" with "hot-reloadable" and could mislead an operator changing capacity, credentials, or retention.
Distinguish the two properties everywhere the section is described: history-cache.mdx marks every field as needing a restart and adds a callout explaining the difference, reference.mdx redefines the Restart column and fixes the five rows and the effect list (adding the `asyncprov` evidence alias), troubleshooting.mdx replaces the "two keys need no restart" claim and gains a dedicated symptom entry, and observability.mdx notes that changed counters need a restart. The Chinese text is corrected in parallel.
Verified: the docs link test and the docs unit suite pass, frontmatter parses for all 20 pages, and every in-page anchor resolves on the rendered English and Chinese history-cache, troubleshooting, reference, and observability pages.
* docs(checkpoints): account for periodic snapshots in the storage-growth claim
Every periodic snapshot stores the entire message list as of that step (`_DeltaSnapshot(ch.get())`, persisted as a checkpoint blob), so with a fixed snapshot frequency K the retained total after N turns is O(N²/K + N), not O(N). The manual claimed unconditional linear growth, which understated capacity planning — the more so because nothing in DeerFlow prunes those snapshots yet.
Distinguish per-step delta writes from total retained storage in index.mdx (prose and table), modes.mdx (step-cost row), and operations.mdx; note in cadence.mdx that K is also the capacity knob that scales the snapshot bytes. The English and Chinese chapters are corrected in parallel. The reference chapter keeps the released #4292 wording as a verbatim quotation and now names the retained-storage bound beside it.
Verified: `_DeltaSnapshot(ch.get())` and its blob serialization in the pinned langgraph (channels/delta.py:196-198, checkpoint/serde/types.py:19-25) confirm the premise; the docs link test and docs unit suite pass; all ten changed pages render with resolving anchors and no stale linear-growth wording.
* docs(checkpoints): correct the run-error surfaces and repair two evidence pointers
Review follow-ups on the checkpoint manual, both languages.
Error-surface corrections:
- Run creation is not always asynchronous: when a thread's run-event feed is
empty but a checkpoint head exists, ensure_checkpoint_history_seeded reads
that head inside start_run, where only ConflictError (409) and
UnsupportedStrategyError (501) are mapped, so a mode mismatch returns a
plain 500 before admission. Troubleshooting, observability and the
reference route tables now scope the "does not pre-gate" claim.
- The 503 symptom is retargeted. Only Gateway startup, make_lead_agent and
DeerFlowClient.__init__ call the freeze helpers, so a config.yaml edit
cannot make a state/history/thread route answer 503. A mode edit is
injected over by the run worker and ignored until a restart; a cadence edit
fails the next run with the restart-required message. The router's 503
mapping is marked defensive in modes and the reference.
- The regenerate section title no longer groups the state endpoints, which
map a mismatch to 409.
- operations: the Textual TUI catches the mismatch through stream_actions();
only the headless --print/--json runners print a traceback.
Evidence-pointer repairs:
- reference: this branch's CHANGELOG insertion shifted the #5734 entry from
3032-3039 to 3040-3048; the citation is re-pointed.
- reference: the #8548 row cited an untracked worktree plan file; replaced by
the in-repo test anchor, and "mirror issue #8551" reworded to "related".
- reference: new `agent` alias for the agent-factory freeze sites.
- history-cache: the bare `deps:`/`asyncprov:` aliases are expanded to full
paths; the alias legend lives only on the reference page.
* docs(checkpoints): say a delta fork is accepted, not refused
"Delta mode cannot fork" overstated the constraint. Nothing in the stack
rejects the fork: LangGraph stores the second child, no saver checks for an
existing sibling, and ensure_checkpoint_mode_compatible returns immediately
when the process mode is delta. What breaks is the read — the walk replays
the abandoned sibling's writes stored under the shared parent.
Reproduced on the pinned stack (langgraph 1.2.9 / langgraph-checkpoint
4.2.0, AsyncSqliteSaver) by running this repo's own #4458 fixture without
the linearization call: the forked resume returns
["h1", "a1", "h2", "a2", "a2-new"] instead of ["h1", "a1", "h2", "a2-new"],
i.e. the superseded answer comes back.
- resume-and-rollback: title/description/heading now say "cannot fork
correctly"; new "Forks are accepted, not refused" subsection covering the
unguarded write side.
- troubleshooting: the "old answer reappears" cause states the same split.
- reference: the POST /threads/{id}/state row notes that a client-supplied
checkpoint_id older than the head is applied as a delta fork with no
linearization — the write shape upstream tracks as #8551 (open), which the
run-path linearization does not cover.
* docs(checkpoints): scope the unguarded state-update fork to what the probe showed
The previous commit said a mutation against an older checkpoint stores its
writes under the shared parent and implied the sibling branch suffers for it.
Probing the pinned stack (AsyncSqliteSaver, delta messages channel) shows:
- the write does land under the shared parent and joins the write set the
parent's other children walk (raw walk for a sibling includes the abandoned
head's own write);
- but reducer channels are written with Overwrite, which replaces the
accumulated value, so the forked branch reads clean and the abandoned
branch is unchanged in that shape — #8551 did not reproduce;
state the masking mechanism and the upstream tracking issue, not a corruption
claim.
* docs(checkpoints): fix a frontmatter description that broke the docs build
`resume-and-rollback.mdx` shipped `description: Delta mode cannot fork a
thread correctly: a forked delta read ...` in the previous commit. The
unquoted ": " turns the value into a nested mapping, so Nextra's YAML
frontmatter parse failed and the e2e lane could not start its web server
(YAMLParseError at line 2, column 14) — the exact failure class this manual
warns docs authors about, introduced by me two commits ago.
- Replace the colon with an em dash in the English description (the Chinese
one uses a full-width colon and was unaffected).
- Add tests/unit/content/docs-frontmatter.test.ts: every `src/content/**/*.mdx`
frontmatter block must be YAML-parseable, implemented as the one rule that
actually breaks it (an unquoted scalar containing ": "), because the unit
suite never compiles MDX and only a build or a dev render catches it.
Verified the guard fails on the broken line and passes on the fix.
Verified with a clean `BETTER_AUTH_SECRET=local-dev-secret pnpm build` on this
tree: all pages prerender, resume-and-rollback EN/ZH included.
* fix(sandbox): clamp a negative start_line in provider read_file
LocalSandbox.read_file clamps the start line with max(start_line or 1, 1),
but e2b, OpenSandbox and BoxLite used start_line or 1, so a negative value
reached Python's negative-index slicing and read from the end of the file
instead of the first line. Clamp all three the same way, and pin the
boundary cases in their existing range tests: a past-EOF start still returns
an empty string, a negative start reads from line 1.
Surfaced by the review of #5616 (e2b has the same gap, consistency nit).
* fix(sandbox): clamp negative end_line across providers
Complete the range-bound consistency fix for e2b, OpenSandbox and BoxLite:
clamp a negative end_line to zero like LocalSandbox instead of letting
Python's negative-index slicing silently drop trailing lines. Add the
symmetric boundary assertion to each provider's existing range test.
Follows #5801 for the remaining numeric env reads that break on an empty
value (`-e VAR=`, or `${VAR}` with VAR unset):
- The WeChat QR worker guard read a blank GATEWAY_WORKERS as an invalid
count and refused QR login, although the compose command starts one
worker for it (`--workers ${GATEWAY_WORKERS:-1}`) and the other worker
checks treat it as 1. A blank value now falls through to WEB_CONCURRENCY
and then 1, so the guard still fails closed for real multi-worker setups.
- The Lark broker aborted startup on int("") for a blank
DEERFLOW_LARK_BROKER_PORT or DEERFLOW_LARK_BROKER_TIMEOUT; it now uses the
defaults.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(prompts): add operator overlays for core system prompts
* test(prompts): isolate SOUL forwarding test from operator config
* fix(prompts): wrap complete subagent system message with overlay
* fix(subagents): serialize prompt overlays in durable batches
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(exa): coerce configured result limits before calling the SDK
`$VAR` references in config.yaml resolve to strings, and exa-py rejects a
string `num_results` ("Invalid value for option 'num_results': 10. Expected
one of [<class 'int'>]"), so `max_results: $EXA_MAX_RESULTS` made every Exa
web search return an error. `contents_max_characters` reached the API as a
string the same way.
Coerce both like the DDG and SearXNG providers do: parse an integer, and warn
and keep the default for a blank, non-numeric, zero or negative value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(exa): reject fractional and boolean limits instead of truncating them
Review follow-up: int() turned an unquoted YAML `3.5` into 3 and `true`
into 1, so a misconfigured limit was used with a different value. Accept
only non-boolean integers and integer-form strings; everything else warns
and keeps the default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(feishu): stabilize file batch timing
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
`deerflow.agents.memory.storage` does not exist (memory/__init__.py says
DeerMem-private symbols are imported from
deerflow.agents.memory.backends.deermem.deermem.core.*), the documented class
shape cannot be built by create_storage() (it instantiates cls(config)) or
called by the memory updater (load/reload/save all pass user_id=, save also
expected_revision=), and storage_class is split on the last dot, so the
`module:Class` value the guide shows never resolves. memory.mdx also promised a
fallback that the code explicitly refuses.
* fix(slack): log Socket Mode connect failures instead of dropping them
SlackChannel.start() ran SocketModeClient.connect() through
loop.run_in_executor() and discarded the returned future. When the
connect raised (rejected app token, network error, TLS failure) the
exception was stored on a future nothing awaited, so the only trace was
asyncio's "Future exception was never retrieved" at garbage collection.
Meanwhile the channel had already logged "Slack channel started" and
reported itself running.
Route the executor call through _connect_socket_mode(), which logs the
failure with logger.exception() the way the Telegram and Discord channel
threads already do. The regression test injects a failing connect and
asserts the error reaches the channel logger.
* fix(slack): bind queued connect to its client
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): strip trailing sentence punctuation from extracted resource refs
The resource-graph extractor's _PATH_TOKEN_RE includes '.' in its path
character class, so a bare package-relative path at the end of a sentence
(e.g. "Read references/setup.md.") is extracted with the trailing period
still attached. build_resource_graph then emits a false-positive
resource.missing finding for "references/setup.md." and, because the
real file is never matched, reports the actual references/setup.md as an
unreferenced orphan.
Other sentence punctuation (",;:!?()") already falls outside the regex
character class and is correctly dropped; only '.' is ambiguous because it
is both a valid path character and prose punctuation.
Strip trailing '.', '?' and '!' from every extracted reference in
_extract_references. Real dotted filenames are unaffected: rstrip only
removes punctuation at the very end, so references/config.yaml and a link
target such as references/v1.0.md keep their dots.
Adds two regression tests covering prose-terminated bare paths and real
dotted filenames.
* style(tests): format skill review tests
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(models): add declarative reasoning capability contract (#5073)
Introduce an optional per-model `reasoning:` block beside the legacy
`supports_thinking` / `supports_reasoning_effort` booleans. It declares
thinking availability (unsupported | optional | required), the accepted
effort values with aliases, default, and serialization path, the payload
dialect, and the reasoning-history requirement. When present the booleans
are derived from it and contradictory profiles fail at config load.
`deerflow.models.reasoning` normalizes legacy and declared profiles into
one immutable contract and resolves every caller's generic request against
it. `create_chat_model` is the single enforcement point: required-thinking
models never receive a synthesized disable payload, unsupported effort
values never reach the provider, and the dialect synthesizes the on/off
payload when no template exists. Legacy profiles keep the historical path
byte-for-byte. The lead agent and subagent descriptor resolve the same
policy first so run metadata reports effective values.
`/api/models` and `DeerFlowClient` project the contract as `reasoning`;
the frontend derives mode clamping and effort choices from it through
`core/models/reasoning.ts`, and user preferences accept provider-specific
effort tokens. The setup wizard's Z.AI GLM-5.3-Flash profile migrates to
the contract, restoring its low/high/max effort control.
* docs(models): move the reasoning contract guidance into models/AGENTS.md
backend/AGENTS.md sits within 46 bytes of the 96 KiB effective-chain hard
limit for the middlewares, runtime, sandbox, and subagents guides, so the
new section pushed scripts/check_agent_guidance.py over it. The models
guide is the nearest file for the factory and the contract module, and
its chain has ample room.
* fix(models): address review on the reasoning contract
- vLLM dialect: the enable path now mirrors whichever chat_template_kwargs
switch the template declares, like the disable path already did, so a
template using the older `thinking` key no longer also grows
`enable_thinking`.
- Validate operator-supplied effort values at `effort.path` inside
`when_thinking_enabled`, `when_thinking_disabled`, and the `thinking`
shortcut against the contract, not only the profile-level value; they are
forwarded when the caller chooses nothing, so the "unknown values never
reach the provider" guarantee now holds for them too.
- Constrain `effort.path` to dotted identifiers and reject single-segment
paths that would replace a whole mapping (`extra_body`, `thinking`,
`model_kwargs`, `default_headers`, `default_query`).
- Z.AI GLM-5.3-Flash profile: `default: high` instead of `max`, because the
default also governs summarization, title generation, and subagent calls
that never choose an effort; `max` stays selectable. Documented in the
config example, the configuration manual, and the design note.
* fix(models): close reasoning contract review gaps
* fix(models): preserve native Ollama reasoning config
* fix(models): keep native reasoning strings and project reasoning into the descriptor
- `ModelConfig.reasoning` also accepts the level string ChatOllama takes
for gpt-oss style models (`low` / `medium` / `high`); the factory forwards
both the boolean and the string on the legacy path, so profiles that
passed `reasoning: high` through before the contract field existed keep
loading and keep sending it.
- The assembly descriptor no longer excludes `reasoning` from
`model_parameters`: the native ChatOllama value and a declared contract's
dialect/history/effort change the request payload, so they now move the
assembly fingerprint.
- e2e: the sidecar spec pinned the pre-contract body where a model without
effort control still received `reasoning_effort: minimal`; the preset is
now dropped for such models (the Gateway discarded it anyway), so the
assertion expects the key to be absent.
* fix(sandbox): scope viewed image host reads to the current thread
* docs(gateway): trim agent guidance to fit CI limit
* fix(images): honor thread data paths for saved host copies
* test(images): align authorization fixture with thread paths
---------
Co-authored-by: YxinMiracle <“939157765@qq.com”>
* fix(memory): reject a Honcho base_url that can never resolve
HonchoConfig only did str(...).rstrip('/') on backend_config.base_url, so a
scheme-less or non-http value was accepted at Gateway startup and then made
every memory read return empty and every write drop silently, with only a
per-operation WARNING. httpx turns 'localhost:8000' into the bogus scheme
'localhost:'. The same check already exists in the mem0 and OpenViking
backends, and honcho/README.md plus HonchoMemoryManager.from_config both
promise a bad URL fails fast at startup.
* fix(memory): cover the scheme-case and host-less variants of the guard
- drive the api_key-over-plain-HTTP check off urlsplit(...).scheme: urlsplit
lowercases the scheme while startswith("http://") does not, so
"HTTP://internal:8000" reached httpx over plain HTTP with the key set and
allow_insecure_http still False
- read hostname instead of netloc in the base_url guard, so an authority
without a host ("http://", "http://:8000") is rejected at config time like
the scheme-less shape this PR already removes
- test direct HonchoConfig(base_url=...) construction, the path the guard's
placement in __post_init__ exists to protect
The docs site pointed at POST /api/extensions/{skills,mcp}/{name}/enable
and /disable, and backend/docs/API.md at POST /api/skills/{name}/enable
and /disable. None of these routes exist. The gateway toggles a skill
with PUT /api/skills/{skill_name} and an MCP server with
PATCH /api/mcp/config, both admin-only, which is also what the
frontend calls.
