* fix(deploy): honor BETTER_AUTH_SECRET and DEER_FLOW_INTERNAL_AUTH_TOKEN from .env
deploy.sh only checked the shell before reloading a persisted secret or
generating a fresh one and exporting it. Compose interpolation lets shell
variables outrank --env-file, so a secret written to the repo-root .env --
the surface every deployment doc points at -- was silently replaced by the
generated one: sessions were signed with a secret the operator never chose
and Gateway workers outside the stack got 401 with the configured token.
Leave a .env-provided secret unexported so Compose parses it itself
(shell -> .env -> persisted file -> generated), and keep generating when
the shell exports an empty value, since Compose would otherwise pass the
empty string through.
* docs(changelog): reference #5928 in the deploy.sh dotenv secrets entries
* fix(deploy): decide whether .env provides a secret with Compose's own parser
read_dotenv_value only matches KEY=VALUE and returns the literal text, but
Compose also accepts `KEY: VALUE` lines and interpolates `${VAR}` inside
values. So `TOKEN: abc` looked absent (and got shadowed by a generated
token) while `TOKEN=${UNSET}` looked present (and suppressed generation,
leaving the stack with an empty token).
Ask Compose for its resolved interpolation environment instead
(`docker compose config --environment`, same --env-file, stub project on
stdin, project directory docker/) and treat a resolved-empty value as
missing. Clients older than Compose 2.28 lack the flag and fall back to
the plain reader. Tests cover the canned Compose answers, the fallback,
and every dotenv spelling against the real Compose client when installed.
* fix(deploy): resolve .env secrets through Compose on every supported client
`config --environment` needs Compose 2.28, so 2.24–2.27 clients took the
plain KEY=VALUE fallback and kept both defects: `TOKEN: abc` was missed
and shadowed, `TOKEN=${UNSET}` counted as present and left the stack with
an empty token.
Drop the fallback. Render a stub project whose only environment entry is
`${KEY}` through `docker compose config` — same --env-file and project
directory as the real command — and read the interpolated value back;
`""` means empty or unset. This is plain interpolation, available on every
Compose v2, and verified identical on 2.24.0, 2.27.0 and 2.32. A failing
probe stops the script instead of guessing. The real-Compose tests now run
against the docker CLI and any standalone binaries listed in
DEER_FLOW_TEST_COMPOSE_BINARIES.
* feat: add optional role filtering to task history search
* docs: scope history role guidance to task continuity
* test: register task continuity guidance in scoped inventory
* docs: separate English guidance from Chinese README translation
* fix(skills): bail out of the markdown-link scan when no link construct exists (#5714)
* fix(skills): scan markdown links linearly and skip failed target runs (#5714)
* fix(skills): make link scan exactly match finditer, blank spans by position
Address review feedback: drive the scan by openers (the regex engine's
own start-position scan) so it reproduces finditer exactly, blank
matched spans by position instead of str.replace, pin the equivalence
with tests, and add CHANGELOG entries.
* fix: resolve skill review test lint errors
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(models): keep Claude prompt-cache markers out of thread messages
ClaudeChatModel._apply_prompt_caching set cache_control on payload blocks in
place. langchain-anthropic forwards Claude-native blocks (image or document
blocks with a source, search results) by reference, so the marker landed in
the thread's own message and was checkpointed with it. Each later request
carried those stale markers on top of its own four, and from the third turn
of a thread with such blocks the API rejected every request (400, "A maximum
of 4 blocks with cache_control may be provided").
Breakpoints are now placed on copies, and markers already in the request are
dropped first, so threads whose checkpoints already carry them recover
without a migration. The OAuth path's _strip_cache_control popped markers
from the same shared objects; it now copies too, and prompt caching reuses it.
* fix(models): drop stored Claude cache markers with caching off too
The strip only ran inside _apply_prompt_caching, so with
enable_prompt_caching: false a thread whose checkpoint already carried five
or more markers kept failing: exactly the switch an operator would flip in
response to these 400s. _get_request_payload now strips on the non-caching
path as well, so recovery no longer depends on the flag.
The stored-checkpoint recovery is now pinned through _get_request_payload
with real messages, for caching on and off. The docstring and the AGENTS.md
note now say which objects are really shared (blocks, list-form system
blocks, a reused binding's tool dicts), that the message-dict and list
copies are defensive, and that _apply_prompt_caching must strip first.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(docker): strip the IPv6 listen in the production nginx launcher too
docker/nginx/nginx.conf listens on 2026 and [::]:2026. On a host booted
with ipv6.disable=1 (or a kernel without IPv6) nginx refuses to start —
socket() [::]:2026 failed (97: Address family not supported by protocol) —
so the container restart-loops under `restart: unless-stopped` and
`make up` waits on a readiness probe that never turns healthy. #2027 added
a launcher guard that deletes the IPv6 listen when /proc/net/if_inet6 is
absent, but only to docker-compose-dev.yaml; the Helm chart mirrors it;
docker-compose.yaml — the file behind `make up` and scripts/deploy.sh —
still ran the bare copy-and-start command.
Give the production nginx service the same launcher as the dev file:
`set -e`, copy the template, guard, and `exec nginx` so nginx is PID 1 and
receives docker stop's signal directly.
The new test pins the guard in all three launchers (prod, dev, Helm),
requires the three sed patterns to be identical so they cannot drift again,
checks the guard edits the file nginx actually loads, and applies the
pattern to the shipped compose and Helm configs to confirm it removes only
the IPv6 listen. Verified on nginx:alpine: BusyBox sed strips exactly that
line and `nginx -t` accepts the result; the rendered production launcher
runs unchanged when IPv6 is present.
* docs(changelog): reference #5900 in the nginx IPv6 guard entries
* fix(skills): let remove_file/write_file handle binary support files and reject directories
A .skill archive may carry binary support files such as assets/logo.png:
the installer only rejects executable binaries by magic bytes. But both
skill_manage file actions read the previous content as UTF-8 text before
touching the file — purely to fill the history record's prev_content — so
a binary member raised UnicodeDecodeError out of
SkillStorage.remove_custom_skill_file (and out of the tool's write_file
prelude) and was never removed or overwritten.
A bare support directory ("assets") also slipped through
ensure_safe_support_path, because it resolves to its own allowed root and
passes the containment check, and then raised IsADirectoryError from the
same read.
Add read_text_or_none for the history reads: UTF-8 text is returned as
before, undecodable content yields None (the record simply has no previous
text), and IO errors still propagate. Require a filename below the support
subdirectory in ensure_safe_support_path, and reject an existing directory
with ValueError in both remove_custom_skill_file and the tool's write_file,
so the model gets an actionable validation error instead of a crash.
prev_content is informational for support files — rollback only restores
SKILL.md — so recording None loses nothing that could be used.
* docs(changelog): reference #5893 in the support-file entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): offload image search to a thread via asyncio.to_thread
Make image_search_tool async and run the blocking ddgs call in a worker
thread with asyncio.to_thread, continuing the TODO.md planned feature
('Replace sync requests with httpx.AsyncClient in community tools').
The ddgs SDK exposes no async API, so instead of swapping the HTTP
client the blocking DDGS.images() call is offloaded to a thread — the
same pattern the infoquest/jina_ai tools use for readability
extraction. Callers see no change in behavior.
Tests invoke the async tool via ainvoke under @pytest.mark.anyio.
* fix(image-search): offload config lookup with DDGS search
* test(image-search): pin DDGS search off the event loop
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(logging): redact credential-bearing header values in urllib3's header-parse dump
* fix(logging): bind redacted before the header-dump pass runs
* fix(logging): collapse the first field of a header dump, which shares its line with the url argument
* fix(logging): collapse every header-dump field, in the message and in the exc_info traceback
* test(logging): give the traceback message copy real CRLF
The message half of the record built its payload with literal backslash-r
text while the exc_info copy used real CRLF, so the two halves rendered the
dump in different shapes and the message only exercised the pass through a
fragment match instead of the escape the real record produces.
* fix(logging): redact obs-fold continuations and Proxy-Authentication-Info
A malformed response can fold a header value onto the next line, and this
warning exists precisely to dump malformed upstream bytes, so a folded
credential cannot be assumed absent: the pass replaced only the matched
segment and left the continuation - `CookieSecret; Path=/` - in the log
whole. A collapsed field now also swallows the SP/HTAB-prefixed segments
that follow it, while a non-sensitive field keeps its own continuation.
Proxy-Authentication-Info is the proxy-side twin of Authentication-Info,
which was already on the list; nextnonce reaches the payload and was
logged verbatim in both the message and the exception copy.
* fix(logging): redact a credential folded under an unlisted field, past repr escapes, and behind an empty first segment
Addresses the three P1 findings on the header-parse dump pass:
- A credential that exists only in an obs-fold continuation is now collapsed.
``_CREDENTIAL_FIELD_RE`` no longer carries the ``^``: the anchor is applied
past a segment's fold marker instead, because under RFC 7230 unfolding a
bearer token folded under a field this list does not name is still a bearer
token.
- ``_OBS_FOLD_RE`` now also recognises the repr-escaped ``\t`` spelling. The
payload reaches the record through ``!r``, so a real tab is the two
characters backslash-t and never matched a literal-class marker.
- The continuation walk runs even when the matched value is empty, so
``Set-Cookie:\r\n Secret; Path=/`` collapses instead of leaving the secret in
the segment that follows.
Evidence: 3 new regression tests, red on the previous head (3 failed / 23
passed) and green on this commit (26 passed) against urllib3 2.7.0; ``ruff
check`` and ``ruff format --check`` clean on both files.
* test(logging): move the header-parse tests out of the EOF hunk
Upstream `22b5b59` appended a redaction test at the end of this file, the same
position this branch appends its own block to, so the PR reported a conflict.
Relocating this branch's block to the preceding top-level boundary frees the
EOF hunk; the three-way merge against current `main` is then clean.
Content is unchanged: base->head is a single pure insertion of the same 312
lines (0 deletions, same line multiset as before this commit), and the merge
result is byte-identical in aggregate to resolving the conflict by hand.
* fix(logging): treat an escaped bare CR as a header-dump line break
urllib3's header-parse warning carries the unparsed block through repr(), so
its separators are escaped text. The splitter recognised escaped CRLF and
escaped LF but not escaped bare CR, which http.client also accepts as a line
terminator: a payload separated on bare CR alone stayed in one segment, no
field name landed at a segment start, so the credential anchor never fired
and the Set-Cookie and signed Location values were logged verbatim in both
the message and the repeated exc_text. Add the escaped \r alternative and an
end-to-end case driven through the real parse_headers/assert_header_parsing
pair on the pinned urllib3 2.7.0.
Co-Authored-By: sxh313 <sxh313@users.noreply.github.com>
* test(logging): pin the bare-CR header dump on the JSON path too
The Gateway runs with logging.enhance.format=json, and JsonTraceFormatter
renders the exception itself, so the text-format case alone left the deployed
emission unguarded for an escaped bare-CR payload. Record the measured cost of
reordering the break alternation, which obs-fold continuations depend on.
---------
Co-authored-by: sxh313 <sxh313@users.noreply.github.com>
* fix(projects): label untitled threads in the conversation-files view
GET /api/projects/{id}/threads/files returns each member thread's
display_name as `str | None`: the thread meta row is created with NULL and
only filled once title generation has run after the first reply, or never
if that step fails. The frontend's ProjectThreadFileGroup declared the field
as a required string and ConversationFileGroup rendered it verbatim, so a
thread whose files were uploaded before its first reply showed them under
an empty heading. The sibling ProjectThread type already models the field
as nullable and every consumer of it falls back to t.projects.untitled.
Make the group type nullable to match the wire, and render the heading
through a shared threadTitle() helper that both the thread-name map and the
file group now use, so the "Untitled" rule lives in one place.
The DOM test covers null, missing and whitespace-only titles.
* docs(changelog): reference #5879 in the conversation-files entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): unlink a symlinked custom skill package on delete
The storage contract (_is_external_skill_directory_symlink) accepts a
one-level symlink under custom/ that points at an operator-managed skill
package, and every read path honours it: the skill loads, lists, validates
and can be enabled. delete_custom_skill, however, removed the package with
shutil.rmtree, which refuses symlinks. The delete raised OSError after the
history record had already been appended, skill_projection_mutation cleared
the user's projected custom view on the way out (rebuilt on the next
acquire), and DELETE /api/skills/custom/{name} answered 500 — for both
LocalSkillStorage and UserScopedSkillStorage, which inherits the method.
Unlink the entry when it is a symlink and keep rmtree for real directories.
The external tree is not ours to delete: removing the skill means removing
the link. Checking is_symlink() first also covers a dangling link, which
exists() would report as absent.
Tests cover both storages: the link is gone, the skill no longer exists,
the external package (SKILL.md and a support file) is untouched, the
history record was written, and the sibling skill's projected view
survives the delete.
* docs(changelog): reference #5881 in the symlinked-skill delete entries
* feat(community): migrate tavily tools to AsyncTavilyClient
Replace the synchronous TavilyClient with AsyncTavilyClient in the tavily
community tools, as tracked in backend/docs/TODO.md planned features
('Replace sync requests with httpx.AsyncClient in community tools').
web_search_tool and web_fetch_tool become async functions and await
client.search()/client.extract(). External behavior (return values,
error strings, credentials handling) is unchanged; the client is still
constructed with api_key=None when the tool config has no key, letting
the SDK read TAVILY_API_KEY from the environment.
Tests are updated to invoke the async tools via ainvoke under
@pytest.mark.anyio, and the credential assertions now verify the client
is constructed with the right api_key (AsyncTavilyClient keeps the key
inside its httpx client and exposes no public api_key attribute).
* fix(tavily): close async clients after each tool request
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(dev): keep whole worktree paths when serve.sh builds DEERFLOW_ROOTS
serve.sh collects every deer-flow worktree root so `make stop` / `make dev`
can reclaim the shared dev ports (8001/3000/2026) from a sibling checkout.
It parsed `git worktree list --porcelain` with `awk '{print $2}'`, but that
output does not quote paths, so a worktree at `.../deer flow two` was
recorded as `.../deer`. `_is_deerflow_pid` matches open files against
"<root>/", which the truncated root never satisfies, so a Gateway or
frontend started from that worktree was treated as a foreign process and
the start aborted with "port already in use".
Take the remainder of the line after the `worktree ` key instead. The
dedupe and longest-first ordering that follow are unchanged.
The new test builds a scratch repo with a spaced linked worktree and runs
the DEERFLOW_ROOTS block from serve.sh against it, from both the main
checkout and the spaced one.
* docs(changelog): reference #5856 in the worktree-roots entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): never let a malformed files[*].size abort the run
UploadsMiddleware._files_from_kwargs validates every field of a
client-supplied file entry fail-soft — non-dict entries, non-basename or
staging filenames, and files missing from the uploads dir are all skipped —
except `size`, which went straight to int(). A value int() cannot convert
("abc", a list, inf) raised ValueError/TypeError/OverflowError out of
before_agent (and the executor-wrapped abefore_agent), failing the whole run
before the model was called. The Gateway strips only server-owned metadata
keys and edit/regenerate deep-copies `files` verbatim, so the same entry
failed again on every retry of that message.
The size only feeds the human-readable "(N KB)" line in <current_uploads>,
so route it through _coerce_file_size: a non-negative finite number (or a
numeric string such as "2048") is kept as int; None, bools, negatives,
non-finite floats and anything unconvertible fall back to 0, the value a
missing size already gets.
Tests pin the fallback for each unusable shape, the kept values, and that
before_agent still injects an existing upload whose size is malformed.
* docs(changelog): reference #5855 in the uploads size entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
bind_task_tool and _bind_batch_tool copied their process-wide singletons
with only the coroutine rebound. Once get_available_tools had wrapped a
singleton in place with a sync func, the copy inherited that wrapper
around the unbound coroutine, so synchronous invocation bypassed the
explicit SDK submitter/capacity and fell through to the process-global
one -- or raised NotImplementedError in a fresh process where the
singleton was never wrapped.
Rebind the copy's func to a sync wrapper around the bound coroutine so
both invocation paths use the runtime's own submitter and capacity.
* fix(sandbox): list the skills root when only category mounts exist
With the default (non-policy-scoped) LocalSandboxProvider layout, the
four skill categories (/mnt/skills/public, custom, legacy, integrations)
are mounted at separate host directories and there is no mapping for the
aggregate /mnt/skills root itself. LocalSandbox.list_dir resolved the
root to the unmapped host path, and the host list_dir raised
FileNotFoundError before the virtual sub-directory overlay -- which
exists exactly so the agent can discover the categories via
"ls /mnt/skills" -- could run, so the ls tool reported "Directory not
found: /mnt/skills" even with skills installed.
Treat a FileNotFoundError from the host scan as an empty listing when
mappings are mounted inside the requested container path, so the overlay
can surface those children. Paths with nothing mounted inside them keep
raising FileNotFoundError as before.
* fix(sandbox): preserve missing virtual parent errors
* test(screening): align PII fixture with keyed placeholders
* fix(sandbox): preserve non-directory errors for virtual parents
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): migrate infoquest client to httpx.AsyncClient
Replace blocking sync requests.post calls with httpx.AsyncClient in the
InfoQuest community tool, as tracked in backend/docs/TODO.md planned
features. fetch/web_search/image_search and their raw-results helpers are
now async, keeping the 30s transport timeout and external behavior
(return values, error strings) unchanged.
The langchain tool wrappers in infoquest/tools.py become async functions;
web_fetch offloads readability extraction to a thread (same pattern as
the already-migrated jina_ai tool). Tests are updated to mock
httpx.AsyncClient instead of requests.post.
* test(community): use AsyncMock for infoquest fetch in relative-links test
The infoquest web_fetch_tool now awaits client.fetch, so the test stub
must return an awaitable instead of a sync lambda.
* fix(community): follow InfoQuest API redirects like the sync client did
Address the review note on #5782: requests.post follows redirects by
default while httpx.AsyncClient defaults to follow_redirects=False, so
a 3xx answer from the reader/search endpoints would have surfaced as
'Error: fetch API returned status 30x' (or fallen into the generic
error path after raise_for_status) instead of the final content.
All three AsyncClient call sites now pass follow_redirects=True. Adds
regression tests asserting each client is constructed with
follow_redirects=True, and documents the parity in the test contract.
* test(community): fix InfoQuest mock endpoint typo
* fix(release): refresh backend/uv.lock when bumping the version
`scripts/bump_version.sh` rewrote backend/pyproject.toml,
frontend/package.json and the Helm chart's `version`/`appVersion`, but
`backend/uv.lock` also records the root package version (uv keeps its PEP 440
form, so `2.1.0-rc0` is stored as `2.1.0rc0`). The documented release step left
the lock pinning the previous version, and that is not cosmetic drift:
`uv lock --check` (lint CI) fails on a stale lock and `uv sync --locked`
(e2e / plugin workflows) refuses the tree, so the release commit could not pass
CI. RELEASING.md only mentioned the manual `cd backend && uv lock` step for
release candidates.
The script now refreshes the lock via `uv lock` and refuses to start without
`uv` — it exits before editing anything rather than leaving a half-bumped tree
whose lock CI rejects. `uv lock` stays the authority for the normalized
prerelease form; the lock diff for a bump is a single line. RELEASING.md and
AGENTS.md now describe the lock as part of the bump.
Tests: backend/tests/test_bump_version_lockfile_sync.py (red on main, green
here) pins both halves — the lock moves with the sources, and a missing uv is an
error instead of a silent no-op.
* docs(changelog): record the release bump lockfile fix (#5859)
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(memory): redact queued memory payloads (pii_redaction follow-up)
Closes the remaining vector from #3190 (memory storage): request-scoped
redaction leaves originals in thread state, and MemoryMiddleware queued
that raw conversation into the durable extraction buffer — so the
extraction model's input and the persisted facts carried raw PII even
with pii_redaction.enabled.
- MemoryMiddleware gains pii_redaction_config (same pattern as
summarization / durable-context / title); the payload returned by
_resolve_add_args() is redacted at the enqueue boundary before
add()/aadd(), covering both after_agent and aafter_agent
- Reuses the shared redact_text() seam from #5527 via
active_pii_detectors/_Redactor/_redact_content; the buffered payload
itself becomes clean, not just the extraction model's view
- Wired at the lead-agent assembly site
- Tests: queue-payload redaction enabled/disabled, detector toggles,
message immutability, token stability across turns
* fix(memory): reserve existing placeholder indices across the queued batch
Review round 1 on #5577: thread state mixes raw user turns with
already-redacted tool results and summaries, so a token in a later
message must not collide with a new value in an earlier one. Pre-scan
the whole queued batch with redactor.reserve() before allocating.
Also extend the two MemoryMiddleware stubs in
test_lead_agent_model_resolution.py to accept the new
pii_redaction_config kwarg (CI backend-unit-tests red).
* fix(memory): redact the compaction-triggered memory_flush_hook queue too
Review round 2 on #5577: with memory + summarization enabled,
memory_flush_hook() queues list(event.messages_to_summarize) directly
into MemoryManager.add_nowait(), bypassing MemoryMiddleware entirely —
and compaction removes those messages from state right after, so the
after-agent redaction can never repair a raw batch queued there.
Extract the batch redaction (reserve pre-scan + rebuild) into a shared
redact_queued_messages() used by both MemoryMiddleware's enqueue
boundary and the hook; the hook gains an optional pii_redaction_config
wired via functools.partial at the summarization factory (subagents use
skip_memory_flush=True and are unaffected). Regression covers the hook
path enabled/disabled/absent.
* test(summarization): unwrap the partial in the factory hook identity check
edd916e wrapped memory_flush_hook in functools.partial to carry the
pii_redaction config, but missed committing the matching update to
test_factory_attaches_memory_flush_hook_by_default — a functools.partial
is not equal to its wrapped function, so the identity assertion failed
deterministically (caught by static review on #5577).
* fix(memory): cover tool-call arguments and lenient text blocks in queued payloads
Review round 4 on #5577 — two reproduced gaps still allowed raw PII into
memory payloads with pii_redaction.enabled=true:
- OpenViking retains the full message object, so parsed tool_calls args
and provider-format arguments in additional_kwargs bypassed the
content-only redaction; queued payloads now redact tool-call args
recursively (and reserve them in the pre-scan).
- DeerMem's format_conversation_for_update reads p.get("text") of any
dict block, not only type=="text" — _redact_content now redacts every
string-valued text field so a lenient consumer cannot bypass it.
Both repro shapes are pinned as regressions.
* fix(pii): value-derived placeholders for stable capture identity
Review round 5 on #5577: sequential allocation is order-dependent — the
same unchanged message re-redacted in a later batch can get a different
token, which flips OpenViking's content-signature and replays already
recorded turns into persistent memory sessions. Placeholders are now a
pure function of the matched value (sha256-6 per category): identities
stay stable across batches, enqueues, seams, and backends with no shared
state and no stored mapping.
- _Redactor becomes stateless; the reserve machinery (index pre-scans,
_PLACEHOLDER_PATTERN) is removed along with its call sites
- redact_texts keeps its signature, now a simple per-text redaction
- reserve-semantics tests are superseded by cross-enqueue identity tests
(two successive captures where the second batch carries higher existing
placeholders); the compiled-graph echo regex matches the new format
* fix(pii): 128-bit placeholder digests; redact tool-argument keys
Review round 6 on #5577:
- [P2] The 24-bit truncation collided distinct identities (reproduced:
contact3513@ / contact3727@example.com both rendered [EMAIL_575dbb]).
Retain 128 bits of the digest so separate people stay separate at any
realistic volume; the compiled-graph echo regex and the AGENTS.md entry
follow the new format.
- [P2] _redact_strings redacted values but preserved mapping keys, so
{"contacts": {"alice@example.com": "manager"}} kept the raw email
as a key. Redact string keys along with values; deterministic 128-bit
tokens make distinct-key collisions a hash-collision non-event, and a
redacted key equal to another key merges the entries. Regression pins
the email-keyed arguments shape.
* docs(agents): trim PiiRedactionMiddleware entry; drop stale reserve wording
The chain grew past the 98304-byte AG002 hard limit again (main growth
plus the round-6 format example). Trim the entry to fit, and replace the
stale reserve-indices wording with the value-derived semantics that
replaced it.
* docs(agents): shrink PiiRedactionMiddleware entry below the chain budget
main moved (#5609) and the base comparison flipped the nearly-over chain
back to an AG002 error (98441 > 98304). Trim the entry to the facts that
are not documented elsewhere: seams, allowlist, order, enqueue boundary.
* fix(pii): letters-only value-derived tokens; wire the factory SDK path
Review round 7 on #5577:
- [P2] Hex digests carry digit runs that later digit-anchored detectors
re-scan, corrupting minted tokens into nested placeholders (reproduced:
user280@example.com -> [EMAIL_…[PHONE_…]…], ~1/140 emails). The 128-bit
digest is now encoded over a-z, so a minted token survives the full
pinned detector order.
- [P2] create_deerflow_agent (the SDK path) constructed MemoryMiddleware
and DurableContextMiddleware without pii_redaction_config; the knob is
now an explicit optional parameter threaded through
_assemble_from_features (the factory reads no config files by design,
so SDK callers pass it explicitly).
- [Nit] hoist the Command rebuild comprehension out of the per-message
loop in _redact_result.
- [Nit] drop the dead _iter_tool_calls helper and Iterator import.
* test(factory): pin pii_redaction_config wiring; document the parameter
Review round 8 on #5577:
- [P3] test_create_deerflow_agent.py had zero coverage of the
pii_redaction_config parameter; two regressions now assert that
create_deerflow_agent(pii_redaction_config=cfg) builds a
MemoryMiddleware and a DurableContextMiddleware carrying cfg, and that
the default is None (redaction off).
- [Nit] document the parameter in the create_deerflow_agent docstring:
None leaves redaction off; a middleware= takeover bypasses the
assembly that would apply it.
* fix(pii): HMAC-scoped placeholder tokens; install model-bound redaction in the SDK chain
Review round 10 on #5577:
- [P2] sha256(category||value) is an unkeyed, globally deterministic
fingerprint: a provider or memory backend can confirm candidate values
offline and link identical values across users and threads, contradicting
the documented no-re-identification guarantee. Placeholders are now
HMAC-derived over a deployment-scoped token_secret (new config field);
the docstring documents the linkability scope for both modes.
- [P2] _assemble_from_features() never appended PiiRedactionMiddleware, so
the SDK knob enabled only the downstream seams while the primary model
still received raw content. The middleware is now appended in the same
relative position as the lead-agent base chain when the gate is on,
with a model-call regression through the assembled SDK chain.
* docs(pii): document token_secret; update the placeholder format examples
Follow-up to the round-10 review on #5577:
- config.example.yaml documents the new token_secret knob: a
deployment-scoped secret makes placeholders HMAC-derived and only
linkable within one deployment; leaving it unset keeps publicly
computable fingerprints (documented trade).
- the PiiRedactionConfig class docstring's placeholder example now
matches the shipped format (128-bit HMAC digests over base-26
letters).
* docs(pii): update the module docstring's placeholder format example
The module docstring still showed the sequential [EMAIL_1] example;
align it with the shipped value-derived format (128-bit HMAC digests
over base-26 letters).
* fix(pii): require a non-empty token_secret when redaction is enabled
Review round 11 on #5577: with the secret optional, the default enabled
configuration minted empty-key HMAC digests — publicly computable, globally
linkable fingerprints of the raw values, confirmable by offline guessing for
low-entropy identifiers such as phone numbers.
- PiiRedactionConfig now rejects enabled=true without a non-empty
token_secret (minimum 16 characters); the unkeyed mode is gone entirely.
- The request-scoped redactor in _process_request now carries the configured
key: it built its redactor without it, so user-message placeholders
silently diverged from every other seam for the same value. The empty-key
guard in _placeholder_token is the backstop that keeps every path keyed.
- Module/config docstrings and config.example.yaml no longer describe the
placeholders as irreversible or the secret as optional; the stale
reserve-semantics wording and the "memory path not covered" note are
corrected.
- Regressions: validation errors for missing/blank/short secrets, and the
user-message boundary minting the same keyed token as redact_text.
* fix(memory): redact original_user_content on queued memory payloads
Review round 12 on #5577: UploadsMiddleware preserves the raw user turn in
additional_kwargs.original_user_content, but redact_queued_messages rewrote
only content, tool_calls, and provider-format tool_calls — the queued copy
still carried the raw value, so the memory queue and backend received raw PII
with redaction enabled.
The shared helper now redacts the provenance string field on the queued copy
(same batch redactor, merged with the existing additional_kwargs rewrite,
rebuilt via model_copy — thread-state messages stay untouched), covering both
the enqueue boundary and the compaction flush hook. Regressions pin the
redacted provenance field through both paths and the untouched original.
* fix(memory): redact human_input_response on queued memory payloads
Review round 13 on #5577: clarification replies keep the raw user answer in
additional_kwargs.human_input_response.value (DeerMem reads that mapping via
its read_human_input_response mirror), but redact_queued_messages rewrote
only content, tool_calls, and original_user_content — the queued copy still
carried the raw value into the memory queue and backend.
The shared helper now rebuilds that mapping with value redacted (same batch
redactor) while preserving the protocol fields (version/kind/source/
request_id/option_id/response_kind), merged with the existing
additional_kwargs rewrite and rebuilt via model_copy so thread-state
messages stay untouched. Regressions pin the redacted value, preserved
protocol fields, the untouched original, and the flush-hook path.
* chore(config): bump config_version to 47; generate token_secret on upgrade
Review round 13 on #5577: existing deployments can legitimately sit at
config_version 46 with pii_redaction.enabled: true and no token_secret; the
mandatory-secret validator would fail their startup while config.example.yaml
still said 46, so the upgrade checker considered them current and emitted no
migration warning.
- Bump config_version to 47 so the outdated-config warning fires for v46.
- Add a v47 migration to scripts/config-upgrade.sh: when pii_redaction is
enabled and token_secret is missing/blank, generate a random deployment-
scoped secret (secrets.token_urlsafe) and persist it, so the upgraded
deployment starts under the mandatory validation.
- Regressions: a v46 enabled config is flagged outdated against the real
example; the upgrade run generates a >= 16-char secret and the result
passes AppConfig validation; a disabled config gets no secret.
- Also switch the two v41 script tests from a hardcoded "bash" to
SCRIPT_BASH/skipif, matching the documented Windows constraint and the
neighboring tests (hardcoded bash resolves to the WSL stub there).
* chore(helm): sync chart config_version to 47
Review round 14 on #5577: check_config_version.sh fails whenever the chart's
embedded config_version lags config.example.yaml, and the chart workflow is
triggered by config.example.yaml — so the v47 bump left this head failing
that check. Update the values.yaml config block and the README example to
47; scripts/check_config_version.sh now passes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Follow-up to #5833 after an independent review of the merged code.
- A malformed answer (deeply nested JSON raising RecursionError, or a
huge noul integer raising OverflowError outside the try) escaped the
fail-open path, and in the per-message task group it cancelled the
other messages' requests and dropped every flag of that tool call.
Parsing now happens inside the try, which also catches those errors.
- The 16 KiB response cap counted decoded bytes, so a compressed body
could expand far past it first. Requests now ask for identity encoding
and drop encoded responses.
- A plain-HTTP loopback endpoint followed HTTP_PROXY/ALL_PROXY, sending
the bearer key and excerpt to the proxy in cleartext. Such endpoints
now ignore proxy environment variables; HTTPS endpoints keep them.
Tests cover each case, including a real local server behind an
unreachable proxy, and pin previously untested behavior: independent
per-request deadlines, the sync hook without a task store, the >=
threshold and all four named remote tools, warnings for messages that
share a tool-call ID, consumed flags, and scanning past host messages.
The root README no longer says the excerpt is always PII-redacted, and
the example README documents the new limits and that the sync path's
deadline does not cover a stalled DNS lookup.
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* chore(version):update the version to 2.2.0-dev
* chore(doc):Updated the CHANGLOG with the 2.1.0 release version
* chore(doc): Updated the CHANGLOG for the latest changes
* chore(doc): Added the release SOP document
* feat(extensions): grant bounded host model invocation
* fix(extensions): retain model budgets and isolate schema validation
* fix(extensions): normalize provider-origin model cancellation
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): strip section anchors from code-span resource refs
Markdown-link targets already drop their #fragment before the
reference is resolved, but code-span refs did not: a prose reference
like `references/faq.md#pricing` kept the anchor and was resolved as
a literal path, producing a resource.missing finding for a file that
exists and is referenced. The bare-path token scanner stops at '#',
so only the code-span branch leaked fragments.
Split the fragment off code-span tokens the same way the link branch
does, then strip trailing sentence punctuation. Real dotted filenames
keep their dots: the split happens at the first '#', and '.' is only
stripped from the very end.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): prefer exact hash-bearing filenames over fragment stripping
Package paths permit '#', so a code-span reference to a real file like
references/C#.md was truncated to references/C by the fragment split,
producing both a false resource.missing and an orphan report (the bare
path token stopped at '#' the same way). Keep the whole reference in
every extraction branch (link target, code span, bare path token now
includes '#' in its character class) and resolve the fragment
existence-aware: prefer the exact token when it names a real file and
treat the suffix after the first '#' as a section fragment only when
it does not.
Red/green: a new test pins references/C#.md resolving to the real file
while references/faq.md#pricing still strips to faq.md — it fails on
the reviewed head and passes with the change; full
test_skill_review_core.py is 36 passed.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): strip the fragment before normalizing the fallback path
The exact-path preference checked the normalized reference, so a token
like references/faq.md#/../other.md normalized to references/other.md;
when that file existed the check succeeded and the fragment-removal
branch was skipped, recording the wrong edge and orphaning faq.md.
Check the literal token against the package files instead, and drop
the fragment before the remaining path is normalized — normalizing a
hash-bearing segment must never retarget an edge.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): canonicalize leading segments in the hash-filename lookup
The literal-file check compared the verbatim joined path, whose leading
relative segments are uncanonicalized: from references/sub/guide.md, a
reference to ../C#.md joins to references/sub/../C#.md, matches no
snapshot key, and falls through to fragment stripping — truncating the
real references/C#.md into a false missing and an orphan. Compare
against the normpath-canonicalized whole token instead, and treat the
post-'#' text as a fragment whenever it is path-like (carries / or ..)
so it can still never participate in normalization and retarget an
edge.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): check the canonical whole path before stripping a fragment
The path-like-fragment heuristic dropped any post-'#' text carrying '/':
a real package file such as references/C#/readme.md (a '#' inside a
directory name) was truncated to references/C, producing a false
resource.missing and orphaning readme.md. Only a '..' segment in the
post-'#' text forces the strip-first fallback — that is the one case
where fragment text participates in normalization and can retarget an
edge (faq.md#/../other.md). Every other fragment first gets the
canonicalized whole-token lookup, which already canonicalizes leading
relative segments for nested sources.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): always strip markdown link fragments and blind the bare-path pass to links
Markdown link targets have fixed fragment semantics: the text after '#'
is a URL fragment, never part of the filename, so a link to a file
literally named references/faq.md#pricing would have to percent-encode
it. The exact-file branch applied the hash-filename preference to link
targets too, recording a valid edge and suppressing resource.missing
for a broken link when a literal same-named file existed. Links strip
the fragment again, and their full construct is blanked out of the
residual text so the code-span and bare-path passes — literal-path
contexts where '#' may be part of a real filename — never see
link-internal text.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(browserless): read reject_resource_types / reject_request_pattern from config
web_fetch_tool declared both parameters, initialised them to None, and passed
them to the client -- but never read them from tool config, so they were always
None on the wire. BrowserlessClient implements them (and is tested for it at
`test_fetch_html_with_reject_params`), so the provider's resource blocking was
unreachable without patching the source. Skipping images/media/fonts is the
main lever for render latency and bandwidth against a self-hosted Browserless,
and it could not be turned on from config.yaml.
Read both keys through a new `_as_str_list` helper that accepts a YAML list or
a comma-separated string and yields None for anything unusable, so an unparseable
value omits the parameter instead of reaching the server. Both keys are now
documented on the Browserless web_fetch entry in config.example.yaml.
Default behaviour is unchanged: an unset key is still None, so the payload is
identical to before for every existing config.
Adds four regression tests covering the list form, the comma-separated form,
the unset case, and unusable values.
* fix(browserless): validate reject list entries
---------
Co-authored-by: RXQ6 <RXQ6@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(config): sign the bytes the config cache parsed, not a second read
_load_and_cache_app_config parsed config.yaml through AppConfig.from_file and
then called get_config_signature, which opens and hashes the file again. A
write landing between those two reads left the cache holding the older
content under the newer content's signature. get_app_config() compares the
on-disk signature against the recorded one, so from then on it saw
"unchanged" and the edit only surfaced when yet another edit changed the
signature — silently stale until then, with no self-healing.
Read the file once through a new shared helper,
file_signature.read_config_with_signature, which returns the bytes together
with the (mtime, size, sha256) of exactly those bytes, and parse them through
AppConfig._from_yaml_text (from_file's body, split out; from_file itself is
unchanged). The recorded signature now always describes the parsed content:
a write that races the load can only produce a signature mismatch on the
next call, i.e. one extra reload. Signing before parsing but re-reading for
the parse would not be enough — an edit that lands between the two reads and
is then reverted would be served forever — so the tests pin that the parsed
bytes are the signed bytes.
The stale-metadata reload test now routes the loader's read through the same
stale-stat simulation it already applies to the probe, since the loader no
longer records the probe's result.
* docs(changelog): reference #5848 in the config cache entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(authz): add plugin resource targets, provenance, action and management enforcement
PR1 of the extension-tool-authorization series
(docs/plans/2026-09-24-extension-tool-authorization-spec.md): plugin
authorization plumbing, no new author-facing declaration.
- `authz/plugin_targets.py` is the single encoder/validator for composite
targets (`namespace/part`), with one validating composer behind three
kind-checked constructors; invalid input raises instead of degrading.
- `authz/plugin_authz.py` is the harness decision layer (`plugin_action`,
`plugin_management`, plus the batch filters the page slice will consume):
disabled is a no-op, an explicit deny raises, and provider errors follow
`fail_closed` like the sandbox gate.
- `POST /api/plugins/{ns}/actions/{name}` decides `plugin_action:invoke`
after the action resolves and before the body is streamed.
- `deerflow_extension_api.auth` publishes `require_plugin_management` /
`arequire_plugin_management` with two `app.state` resolver keys; the
Gateway installs both. Contract version 0.2.3 -> 0.2.4.
- `tools/tool_provenance.py` records host-written plugin provenance for
display (assembly descriptor, Layer-2 context later); no decision reads a
label.
- Plugin providers use a `(config signature, loop key)` cache so a
loop-affine provider is never handed to another loop or to a sync caller.
- RBAC gains `plugin_actions` / `plugin_management` aliases;
`config_version` 47 -> 48 across the three mirrors.
Tests: new plugin target/provenance/action/management suites plus a
blocking-IO anchor whose offloads were reverted to prove red->green.
* fix(authz): decide plugin resources from the request's config snapshot
Review round 1 on PR #5842 (willem-bd).
P1 — the decision layer no longer reloads the configuration. The six
`authz/plugin_authz.py` functions take a required `app_config` snapshot (the
same request-scoped config the Gateway used to resolve the provider), and
`resolve_/aresolve_plugin_authorization` return it as a third element, so
`enabled`/`fail_closed` can never come from a second read that drifted from
the one that produced the provider. A snapshot the caller could not read at
all is *unavailable*, not disabled: it fails closed as
`authz.config_unavailable`. The Gateway's `_plugin_app_config` now separates
an absent `config.yaml` (no gate, today's behavior) from a config that exists
but cannot be read or validated (deny) instead of swallowing both through
`safe_app_config`.
P2 — `_validated_decision` requires `type(decision.allow) is bool` on both
the sync `authorize` and async `aauthorize` paths, so a provider returning
`AuthzDecision(allow="false")` is a malformed verdict that follows
`fail_closed` instead of being read as an allow.
Both fixes are covered by regression tests that were verified red against the
pre-fix behavior (P1: 200 vs 403 and resolver True vs False; P2: 12 cases
across fail_closed/fail_open and sync/async), plus tests pinning that an
absent config and `authorization.enabled: false` keep today's behavior.
* fix(authz): treat a request-time config absence as unavailable
Review round 2 on PR #5842 (willem-bd).
P1 — `_plugin_app_config[_async]` no longer maps a missing `config.yaml` to
`None` ("no gate"). A Gateway cannot have started without a readable config
(`app/gateway/app.py` reads it through this same accessor in `lifespan`), so
absence *during a request* means the configuration this process was running on
became unavailable — a hot reload, an atomic replace, a ConfigMap remount. It is
unavailable, not disabled: the error propagates,
`resolve_/aresolve_plugin_authorization` raise
`_PluginAuthorizationUnavailable(fail_closed=True)`, the action dispatcher
returns 403 and the management resolvers answer `False`. The dead
`(None, None, None)` allow path is gone, the resolvers return a non-optional
`AppConfig` as their third element, and callers key "authorization disabled" on
`provider is None` alone. This matches the rule the decision layer already
documented (`authz.config_unavailable`) and the public docs.
P2 — `_deny_reason_code` is total: a `reasons` that is not iterable degrades the
code to `authz.denied` instead of raising `TypeError` outside the guarded
provider path, which turned a denial into an unhandled error. It is deliberately
not folded into `_validated_decision`: `allow` was a real `bool`, so the verdict
is a valid denial, and treating it as malformed would flip it to an allow under
`fail_closed: false`.
Both fixes are covered by regression tests verified red against the pre-fix
behavior (P1: resolver True vs False, async `PermissionError` not raised, route
200 vs 403; P2: `TypeError: 'NoneType' object is not iterable` at
`plugin_authz.py:113`). The action-route tests in
`tests/test_plugin_contributions.py` now install a readable
`authorization.enabled: false` config instead of relying on a config-less
process, and `docs/full-stack-plugins.md` plus the PR1 decision log record the
new rule.
* fix(authz): apply the running policy when the plugin config is unreadable
Follow-up to f03d2ea9 (review round 2 on PR #5842, willem-bd).
The first attempt read the P1 finding as "absence is never disabled" and failed
every config read closed unconditionally. That is too broad for this repo: a
config-less host is deliberately treated as having no policy to apply
(`_get_route_authorization_config` falls back to a disabled config,
`sandbox_authz.safe_app_config` documents the same no-op for "CI runners and
direct-call tests"), and the rule turned the config-less e2e plugin host
(`extension_test_fixtures.bookmark_plugin_gateway`) into a 403 — the three
`bookmark-plugin.spec.ts` cases lost the save action. Reproduced locally:
`POST /api/plugins/community.bookmarks/actions/save` against that fixture host
returned 403 before, 200 after this change.
The rule is now "has this process loaded a configuration?", not "was the last
read successful?":
- No loaded config (`peek_loaded_app_config()`, new read-only accessor on
`deerflow.config.app_config`) and no `config.yaml`: no policy to apply, so the
gate stays a no-op — unchanged for CI runners, direct-call tests and hosts
that mount only the plugins router.
- A loaded config whose file disappears or cannot be read: *unavailable*, never
*disabled*. The resolvers raise `_PluginAuthorizationUnavailable` carrying the
failure flag of the configuration this process runs on (authorization
disabled there -> no gate; enabled -> that policy's `fail_closed`, `True` by
default), so an enabled Gateway answers 403/False in the window instead of
letting protected plugin actions and management routes through.
- A config file that exists but cannot be read or validated right now: fail
closed when nothing is loaded (round 1's rule), otherwise the loaded policy
decides.
`peek_loaded_app_config()` is preferred over a Gateway-side startup snapshot: it
reflects the last successfully loaded config, so a hot-reloaded policy is what
applies during the window, and there is no lifecycle hook to keep in sync.
Evidence: the three no-config/lost-config branches in
`tests/test_plugin_management_guard.py` and
`tests/test_plugin_action_authorization.py`, round 1's unreadable-config tests
unchanged, `test_peek_loaded_app_config_survives_a_missing_file` in
`tests/test_app_config_reload.py`, and the fixture-host probe above. Docs
(`docs/full-stack-plugins.md`, `reference.mdx` en/zh) and the PR1 decision log
state the final rule; the log records the superseded variant.
* fix(authz): keep a denial whose reasons raise while being read
* fix(authz): intersect plugin batch filters with the candidates
The plugin_page/plugin_management batch helpers returned the provider's
`filter_resources` answer verbatim, so a custom provider could hand the
projections a target that was never asked about ("*", another namespace,
a phantom page). Every other filter call site re-projects from its own
candidate list; do the same here, state the intersection on the provider
contract, and pin it with a regression test that fails on the old
pass-through.
* docs(plugins): map a management denial to 403 in the route example
The contributed-route example called `arequire_plugin_management` and let
the documented `PermissionError` escape. The Gateway installs no
`PermissionError` handler, so a route copied from the example answered 500
on a normal policy denial instead of 403. Both example routes now translate
the denial, and the surrounding bullets say why the translation is the
route's job.
* docs(extensions): point the 0.2.4 contract row at the PR that ships it
Every other row in the version-history table links the pull request that
shipped the version; 0.2.4 linked the authorization tracking issue instead.
Both the English and Chinese tables now link this PR.
* feat(examples): add opt-in fetched-content screening extension
* fix(examples): screen sync and async results through supported middleware hooks
* fix(examples): preserve empty message IDs when applying screening warnings
* fix(examples): include screening policy in assembly identity
* feat(examples): load fetched-content screening through the extension entry
Package the screening example as a plugins: extension instead of an
operator-configured middleware. install() validates the private config
and, once enabled, contributes one TOOL_VISIBLE middleware for lead
agents and subagents. The package imports only deerflow_extension_api.
Extension tool wrappers are observational, so detection and the warning
use two supported contracts: the tool wrapper classifies the visible
result and records flagged tool-call IDs in the run's task store, and
before_model replaces the matching messages of the latest tool step by
ID. The excerpt sent to Jev is now the redacted, sanitized text the
model sees. Runs without a task store send nothing. Unexpected local
errors surface as isolation diagnostics; provider errors fail open.
Tests load the package through load_extensions() and the host isolation
wrapper and run real lead/subagent graphs with a task store bound as the
Gateway worker binds it. The README records what the example exercised
in the extension API and the gaps it found.
* fix(examples): classify each tool message of a screened result on its own
A Command result was classified as one concatenated excerpt and the flag
landed on its first text message, so a benign first result could take the
warning while an injected later result reached the model unmarked.
Each text message now gets its own bounded excerpt and Jev request, for at
most eight messages per tool call. The requests share one client and run
concurrently in a task group, each under its own deadline, so one slow or
failed request cannot hide another message's flag. Every flagged message's
tool-call ID is recorded. The message cap joins the declared policy, and the
README describes the per-message requests and data sharing.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(uploads): keep document-conversion stat and write-back off the event loop
convert_file_to_markdown offloads the conversion itself for files above
1 MB (documented in its docstring), but the two bookend filesystem calls
still ran on the event loop: the size stat before the threshold check and
the write of the converted markdown afterwards — the latter a potentially
multi-megabyte write in the per-upload ingestion path. Route both through
run_file_io (the dedicated file IO pool that also keeps the caller's
context), matching the off-the-loop fixes already applied to the sibling
upload/document paths.
Add a tests/blocking_io anchor: with the converter patched to a 2 MB
payload it fails on the previous head (Blockbuster catches the blocking
os.stat first; the broad except swallows it into a None return) and
passes with this change.
* fix(uploads): address review — drain conversion write-back, make run_file_io re-entrant
Review follow-up on the off-loop conversion stat/write PR:
1. The write-back now goes through await_drained(): a plain run_file_io
await cancels immediately while the worker keeps writing, so the
ingestion scope could remove its private directory over an in-flight
write and leave a large hidden partial file. The drain delivers the
cancellation only after the worker settled, and the handler removes
the partial output before re-raising.
2. run_file_io is now re-entrant: a nested offload issued from code
already running inside a file-io worker (project-document conversion
runs asyncio.run(convert_file_to_markdown(...)) inside a run_file_io
worker) executes inline instead of submitting to the same bounded
executor, which deadlocked with a single configured worker.
Both fixes carry deterministic regressions: the cancellation test pins
the drain ordering (cancel issued before the worker is released) and the
partial-file cleanup; the re-entrancy test pins inline execution by
thread identity. Both fail on the previous head. Full blocking_io +
project-document + conversion suites show an identical failure set to
the previous head (pre-existing Windows platform failures only).
* fix(uploads): keep cancellation authoritative when conversion cleanup fails
If the post-cancellation unlink of the staging output raised, the new
exception escaped the CancelledError handler, was swallowed by the
broad except as an ordinary conversion failure (None return), and left
both the caller's cancellation and the partial file behind. Catch and
log cleanup failures inside the handler and always re-raise the
captured cancellation; a second cancellation still passes through
untouched (CancelledError is not an Exception).
Red/green: tests/blocking_io/test_file_conversion_offloop.py gains a
regression where unlink fails while the write-back is in flight — on
the reviewed head the task settled on None (DID NOT RAISE); with the
change the CancelledError propagates and the failure is logged.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* docs(backend): drop references to APIs that do not exist
PATH_EXAMPLES.md's upload example imports THREAD_DATA_BASE_DIR from
thread_data_middleware; the module exists but exports no such name, so the
snippet fails with ImportError as written. Resolve the same directory through
the accessor the middleware itself uses, get_paths().sandbox_uploads_dir(),
which yields the layout the file's own path table documents.
CONFIGURATION.md's Docker sandbox block lists `auto_start: true`. SandboxConfig
declares use/port/container_prefix/mounts/network but not auto_start, and the
provider is constructed as cls(**kwargs) from the caller of
get_sandbox_provider() rather than from config extras, so the key is accepted by
extra="allow" and then silently dropped. Remove the line from the example.
* docs(backend): resolve the uploads example through the owner-scoped bucket
get_paths().sandbox_uploads_dir(thread_id) without a user_id selects the legacy
threads/<id>/ bucket, but Gateway uploads land in
users/<user_id>/threads/<id>/user-data/uploads/, so the example imported cleanly
yet still raised FileNotFoundError for a normally uploaded file. Use
deerflow.uploads.manager.get_uploads_dir(), which resolves the effective user
and validates the thread id.
* docs(backend): show the real uploads path layout in every PATH_EXAMPLES demo
The remaining sections still documented the pre-isolation
threads/<thread_id>/ bucket and called `path` a backend-relative string, while
the upload and list APIs return an absolute path under
users/<user_id>/threads/<thread_id>/user-data/uploads/. Reproduced with
deerflow.uploads.manager.get_uploads_dir() and
uploads._list_uploaded_files_for_thread() against DEER_FLOW_HOME: the listed
path is /<home>/users/default/threads/abc123/user-data/uploads/report.pdf.
* docs(backend): clarify runtime home precedence
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(config): honor $VAR substitution for request_admission integer fields
AppConfig.resolve_env_variables replaces every `$VAR` reference with the raw
environment string. RequestAdmissionConfig.requests_per_minute and
max_queue_size are declared `strict=True` so a bool or float cannot slip in,
but strict mode also refuses the string "60", so `requests_per_minute: $RPM`
failed the whole config load with "Input should be a valid integer" even when
RPM=60 — contradicting config.example.yaml's promise that environment
variables work for every field. They were the only strict fields in the
config schema; every other integer coerces in lax mode.
Wrap both fields in a BeforeValidator that converts a decimal integer literal
arriving as a string and leaves everything else untouched, so the strict check
behind it still rejects bools, floats, "60.0" and non-numeric strings.
Tests pin the accepted literals, the rejections that must survive, and the
end-to-end path from a config.yaml `$VAR` reference through AppConfig.from_file.
* docs(changelog): reference #5838 in the request_admission entries
* fix(mcp): preserve backslash paths during bare-filename rewriting
* test(mcp): cover the trailing backslash boundary in bare-filename rewrite
Review noted that treating `\` as a separator also suppresses a rewrite when a
single trailing backslash precedes a newline, which is more often a Markdown
hard break, and that neither the tests nor AGENTS.md covered that position.
Lock the behavior in through the production result converter and document the
tradeoff: a backslash is a legal filename character on POSIX, so forgoing that
one rewrite is preferable to risking a corrupted path reference. Runtime
behavior is unchanged.
---------
Co-authored-by: lakers-abing <236384465+lakers-abing@users.noreply.github.com>
* fix(mcp): run lazy MCP discovery once when initialization itself raises
get_cached_mcp_tools() wrapped the whole lazy-initialization call in the
`except RuntimeError` that was only meant to catch asyncio.get_event_loop()
raising when no loop is set. run_until_complete() and Future.result() re-raise
the coroutine's own exception, so a RuntimeError from discovery — e.g. the
deliberately re-raised McpTaskConfigurationError — took the same fallback and
ran asyncio.run(initialize_mcp_tools()) a second time, respawning every stdio
server and re-fetching OAuth tokens before giving up. Inside a running loop
that fallback fails with "asyncio.run() cannot be called from a running event
loop", so the logged traceback hid the real cause.
Resolve the loop first, in its own try/except, and drive initialization exactly
once: asyncio.run when no loop is set or the current loop is closed, a worker
thread when the loop is running, run_until_complete otherwise. Any failure is
logged once and returns [] as before.
Regression tests pin one discovery call and the real logged cause with and
without a running loop, and keep both fallbacks (no loop set, closed loop)
covered.
* test(mcp): surface worker-thread failures in the no-loop fallback test
Thread.join() does not propagate exceptions, so a regression of the
asyncio.run fallback failed this test with a bare KeyError: 'result' instead
of the real traceback. Capture sys.exc_info() in the worker and re-raise it
on the main thread before the assertions.
