* fix(agents): key generic tool calls on all non-narration args in loop detection
Layer 1 keyed every tool without a dedicated rule on its first salient
field (path/url/query/command/...) and dropped all other arguments, so
paging a URL or a search (start_index, page, cursor) and rewriting one
skill file collapsed onto one key and hard-stopped on the fifth distinct
call. Key on every argument except the narration field (description),
which keeps the #1905 bypass closed.
Fixes#5871
* fix(agents): drop description only for sandbox UI narration in loop keys
update_agent/setup_agent persist description and MCP tools such as
create_issue send it, so description-only updates must stay distinct.
Drop it only for bash/ls/glob/grep, where it is the UI explanation of
the call and was already outside the key before this PR.
---------
Co-authored-by: destire-mio <248462155+destire-mio@users.noreply.github.com>
* fix(agents): preserve response_metadata in dynamic_context user copy
The re-id'd user message was rebuilt field-by-field, dropping
response_metadata (and any other caller-attached field) from what the
run actually sees. model_copy keeps the documented {id}__user swap
while preserving the rest, matching the pattern already used by the
pii_redaction, thread_data and input_sanitization middlewares.
Fixes#5976
* fix(agents): preserve message metadata during upload injection
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add Unbrowse web fetch provider
Add a community web_fetch provider backed by Unbrowse
(https://unbrowse.ai), a hosted service that returns a page as markdown,
over plain HTTP when that is enough and through a cloud browser when the
page needs JavaScript. The tool sends one JSON-RPC tools/call POST
(unbrowse.scrape) to Unbrowse's MCP endpoint, so it needs no extra Python
package and no MCP client (uses httpx, already a dependency).
Unbrowse is not a general web search engine, so only web_fetch is added.
Changes:
- backend/packages/harness/deerflow/community/unbrowse/{__init__,tools}.py
web_fetch_tool with the same output shape as the Sofya, Exa and
Firecrawl fetch tools (# <title> plus markdown, 4KB cap). API key is read
from the config.yaml api_key field or the UNBROWSE_API_KEY env var.
Optional render: auto|never|always; unknown values log a warning and fall
back to auto. JSON-RPC errors, isError tool results, HTTP errors and
malformed payloads come back as "Error: ..." strings.
- backend/tests/test_unbrowse_tools.py: key resolution, render handling,
request shape, structuredContent vs text payloads, error paths.
- scripts/doctor.py, scripts/wizard/providers.py: register the key and the
web_fetch provider; backend/tests/test_doctor.py covers both doctor paths.
- config.example.yaml, .env.example, backend/docs/CONFIGURATION.md,
tools/AGENTS.md, EN/ZH harness/tools.mdx: document the provider.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* docs(changelog): add Unbrowse web_fetch provider entry
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(tools): make the search coercion guards catch OverflowError
The community search tools coerce max_results (and Sofya's content
limit) from YAML config or tool parameters through guards that catch
only TypeError and ValueError. `int(float('inf'))` raises OverflowError
instead, so a YAML `.inf` (or a JSON `Infinity`) value still crashes
the coercion - SearXNG's search even fails wholesale with
'cannot convert float infinity to integer'.
Catch OverflowError alongside them so an infinite value falls back to
the default like every other invalid input. NaN is already covered by
ValueError.
Signed-off-by: yetuge <2219677952@qq.com>
* test(tools): move the GroundRoute inf test to module scope
The regression test was appended inside TestWebSearchTool, ending the
class early and turning its last two methods into uncollected nested
functions. Move it after TestWebFetchTool so pytest collects all 20
tests again.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(scheduler): coerce serialized timestamps in ScheduledTaskRepository.update()
Editing an interval task's title or prompt returned 500. The edit dialog
always sends schedule_spec beside the changed field, and when the cadence
is unchanged the PATCH route keeps existing["next_run_at"] — a value the
repository had handed back serialized as an ISO string. update() assigned
it to the DateTime column untouched, so SQLite raised StatementError
("only accepts Python datetime") and Postgres a DataError; only a cadence
change worked, because that path computes a fresh datetime.
Coerce every serialized timestamp update() accepts through the existing
_coerce_datetime, sharing one _TIMESTAMP_KEYS definition with the row
serializer, as update_after_launch() already does (#4785). The router test
now drives the real SQLite repository for the same-cadence rename and the
cadence change; the in-memory stand-in the other router tests use never
saw the type mismatch.
* docs(changelog): reference #5964 in the scheduled-task PATCH entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(middleware): preserve message fields when PII redaction rebuilds a user message
PiiRedactionMiddleware._process_request rebuilt the redacted HumanMessage
with a hand-constructed constructor call copying only content, id, name and
additional_kwargs, silently dropping response_metadata and every other field
of the original. That is the third and last site of the family already fixed
for ThreadDataMiddleware (#5823) and InputSanitizationMiddleware (#5875);
the review on #5823 scoped the follow-up to the two input-sanitization sites
and left this one out.
The middleware was also inconsistent with itself: _redact_tool_message in the
same file already rebuilds with model_copy and carries the comment explaining
that a hand-built constructor call silently drops response_metadata.
The user message now rebuilds with msg.model_copy(update={...}). The content
is deep-copied because _redact_content appends untouched content blocks by
reference: without the copy, mutating a non-text block on the redacted
message would write through to the message retained in thread state.
Red/green: test_redacted_user_message_preserves_other_fields asserts
response_metadata/id/name/additional_kwargs survive the rewrite — red on main
(response_metadata == {}), green here. A second test guards the aliasing
behaviour above and goes red if the deep copy is dropped. Both suites 258
passed; ruff 0.15.12 check + format clean repo-wide.
Fixes#5967
Signed-off-by: yetuge <2219677952@qq.com>
* fix(middleware): keep the PII rewrite on copy failure and detach metadata
Addresses the two copy-boundary findings on the previous commit.
1. A content block may carry a value whose copy raises. deepcopy(content) then
raised out of _process_request, _try_process fell back to the original
request, and the handler received raw PII unredacted. Copying now falls back
to a shallow per-container copy, so a copy failure still produces a redacted
message and never bypasses redaction.
2. msg.model_copy(update=...) is shallow, so the rebuilt message and the
original shared one response_metadata dict: mutating metadata on the
model-facing message wrote through to the request message retained in thread
state. response_metadata is now copied independently.
Red/green: test_preserved_metadata_is_not_shared_with_the_original and
test_redaction_survives_a_content_block_that_cannot_be_copied both fail on the
reviewed commit (the second with the raw address reaching the handler) and pass
here. Suites: pii 77 passed, and pii + input_sanitization + uploads 327 passed.
ruff check + format --check clean repo-wide (1694 files).
Also drops the "third and last site" wording from the previous commit message's
claim: UploadsMiddleware is a further site of the same pattern (#5972).
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(gateway): load skills off the event loop in the listing and detail routes
GET /api/skills, GET /api/skills/custom and GET /api/skills/{name} called
load_skills() inline. Resolving the caller's storage builds absolute
paths, and load_skills walks every public and custom skill directory and
parses each SKILL.md, so the work grows with the number of installed
skills. #5747 moved the same call off the loop for get_custom_skill and
documented why; these three routes were missed, and the strict
Blockbuster gate raises BlockingError on each of them.
Offload the load through one shared _load_user_skills helper (also used
by update_skill's reload), and add a gate module covering the three
routes including get_skill's 404 branch.
* docs(changelog): reference #5945 in the skills listing blocking I/O entries
* fix(gateway): close pooled MCP sessions on shutdown
The Gateway lifespan closes the browser session manager on the way out but
never closed MCPSessionPool, so every pooled MCP session was abandoned when
the worker exited: the owner task's transport __aexit__ never ran and a stdio
MCP server or SSE/HTTP connection had no orderly teardown.
MCPSessionPool.close_all() already existed for exactly this and had no
production caller -- only the config-retirement path used close_all_sync().
The async form is the right one here: close_all_sync()'s own docstring directs
callers running inside a live event loop to await close_all() instead of
signalling and returning.
Wired into lifespan shutdown with the same bounded, best-effort shape as the
neighbouring browser-session hook.
* fix(gateway): close MCP sessions after run drain
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
dsn_with_search_path rebuilt the query with parse_qsl + urlencode. parse_qsl
decodes '+' as a space, an HTML-form rule libpq does not apply, so any existing
parameter carrying a literal plus came back as %20 and the server received a
different value: application_name=web+app became 'web app', and
options=-c%20timezone%3DUTC+8 became '-c timezone=UTC 8'. The function's own
docstring already states the %XX-only rule and that existing parameters are
preserved.
Rebuild the query from the raw pairs instead, passing every non-options
parameter through untouched and decoding options with unquote.
update_status guarded expected_lease_owner (and the protect_terminal
backfill) on a plain session.get. On SQLite a concurrent requeue plus
re-claim can commit between that read and this session's write, so the
stale guard passed and the late write clobbered the fresh claim —
clearing the new owner's lease fencing and overwriting its status. The
guard now re-reads under the parent's writer lock (the same task -> run
lock order as fail_launching_run / expire_queued_runs), with
populate_existing so the identity map does not serve the pre-lock row
back untouched. This is the run-row counterpart of the staleness #5777
fixed for the task-level lease release.
Red/green: tests/test_scheduled_task_dispatch_race.py gains a parking
session proxy that commits a requeue + re-claim by another scheduler
between update_status's read and its write — on main the late write
returned True and clobbered the claim; with the change it is fenced
(False) and the re-claim survives.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(models): deep-merge when_thinking_enabled on the legacy path
The legacy thinking path (profiles without a `reasoning:` block, i.e. every
existing config.yaml) applied `when_thinking_enabled` with a shallow
`dict.update`, so a template such as `extra_body.thinking.type=enabled`
replaced the profile's whole `extra_body` and silently dropped sibling keys
like GLM's `extra_body.tool_stream` — but only while thinking was on. The
disable branch of the same function already deep-merged into `extra_body`,
and the contract path added in #5780 deep-merges through `_merge_settings`
for exactly this reason (its docstring names `tool_stream`). The on/off
asymmetry cannot be intended: the same profile sent `tool_stream` with
thinking off and lost it with thinking on.
Route the enable template through `_merge_settings` as well. Precedence is
unchanged (template values win on conflicts); the only observable
difference is that keys present only in the profile now survive a
thinking-on request. Everything else on the legacy path — the synthesized
`reasoning_effort=minimal`, the disable payloads, the supports_thinking
guard — is untouched.
Tests pin both directions for a base extra_body beside a thinking template,
recursive merging of nested chat_template_kwargs, and template precedence.
* docs(changelog): reference #5894 in the when_thinking_enabled entries
* fix(models): keep the template's vLLM switch spelling authoritative when merging
Review follow-up. Deep-merging the thinking template into a profile keeps
both spellings of vLLM's toggle when they differ — base
`chat_template_kwargs.enable_thinking: false` plus a template using the
legacy `thinking: true` alias — and VllmChatModel's normalizer maps the
alias onto `enable_thinking` only when the latter is absent, so the
profile's `false` won and thinking stayed off despite thinking_enabled=True.
The previous shallow replacement never hit this because it discarded the
base switch. The contract path had the same trap since it already
deep-merged, and both disable branches had the mirror image (a base
`enable_thinking: true` outlived a template `thinking: false`).
Route every thinking on/off merge through _merge_thinking_payload: it drops
the profile's *other* spelling of the switch before the deep merge, so the
payload's spelling decides in both directions on both paths, and unrelated
chat_template_kwargs still survive.
Tests cover both spelling combinations, both directions, and both paths,
asserting the switch as the provider will send it after normalization.
* fix(models): deep-merge the legacy disable template and mirror the vLLM switch instead of pruning
Review of the previous head found two regressions. The legacy
when_thinking_disabled branch was still dict.update-ed, so a profile that
declares both templates (the shape of most extra_body examples) kept
tool_stream while thinking was on and lost it while off. And the merge
pruned the profile's other vLLM switch spelling for every model class,
although only VllmChatModel maps `thinking` onto `enable_thinking`: on a
plain ChatOpenAI profile the prune removed the only key vLLM reads, so
thinking-off could no longer switch reasoning off.
Route when_thinking_disabled through _merge_thinking_payload too, and
replace the prune with mirroring: after the deep merge, the value the
payload declares for its switch spelling is copied onto the profile's
other spelling when the profile carries it. VllmChatModel collapses the
pair, plain OpenAI-compatible classes send both, and either way the
server reads the template's intent. No spelling is invented, a payload
that spells both is left alone, and non-mapping template values are
forwarded unchanged instead of raising. _VLLM_THINKING_SWITCHES is now
the single definition used by the disable, toggle and dialect helpers.
Docs state the merge semantics (keys are never removed; a template can
only add or override) and drop the "byte-for-byte" wording; tests cover
the disable template on/off, nested inheritance, non-mapping values, the
raw payload for a non-normalizing class, and the no-invention rule.
* fix(models): copy thinking templates on merge and document the never-remove rule
Review follow-up. Templates merged into the constructor settings were
stored by reference wherever the profile lacked the key, so the kwargs
handed to the model class aliased the cached ModelConfig; the mirroring
step now writes into that mapping, so make _merge_settings and
_deep_merge_dicts deep-copy every template mapping they store. Pinned at
every depth by a test that also checks the profile is unchanged.
Deep-merging the legacy when_thinking_disabled template means it can no
longer clear a key the profile's base extra_body sets: a base
extra_body.thinking {type: enabled, budget_tokens} reaches the provider
as {type: disabled, budget_tokens}, which Anthropic-style APIs reject.
The synthesized disable payloads and the contract path already behaved
this way. State the rule and the migration (enable-only keys belong in
when_thinking_enabled) in the changelog, models/AGENTS.md and
configuration.mdx, and pin it with a test for both shapes.
Route the legacy synthesized OpenAI disable through
_merge_thinking_payload too (identical output), narrow the docstring and
guidance wording about the native-Anthropic disable, and make the two
vLLM switch suites resolve the real VllmChatModel / ChatOpenAI classes
so a class-gated merge would fail the case it is named after.
* feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client
- refactor(typesafe): extract a shared Jev (TypeSafe) client and unify its
configuration: one connection-precedence rule, credential fingerprint, wire-size
counting and error taxonomy, shared by the guardrail gate and the memory hooks.
- feat(memory): wire `memory.prescreen` and `memory.signal_classification` into
DeerMem through one injected judge hook (both off by default, `enforce` gated by
the shadow evaluation).
- config: add the top-level `typesafe:` block and the two judging slots to
`config.example.yaml` and the Helm values, bump `config_version` to 50, and sync
the Helm README example with it.
- test(memory): add the shadow-evaluation script (`eval_memory_prescreen.py`) with
per-stratum gates, censoring and the no-network baseline.
Review follow-ups folded in:
- `mutations_accepted` counts only facts that survive capacity enforcement and the
upload-event scrub that precedes persistence, only removals/reinforcements of ids
that existed, and persisted near-duplicate merges (matched by id, or by identity
for legacy id-less facts); it is published only after persistence succeeds.
- The cached manager hot-reloads its judge when a memory slot or the shared
`typesafe` block changes (including a credential rotation); publication is
serialized and revalidates the config generation, so a build started before an
`enforce` -> `off` rollback cannot land after it.
- Cache provenance is per answer: the `cached` flag and the reported model follow
the answers a side actually consumed, across partial merges and answerless retries.
- An oversized noul integer stays a per-question error instead of raising.
- Incremental call savings use the shared scored denominator.
- The sensitive-stratum gate requires evidence for each of identity, preference
and correction.
* fix(typesafe): resolve one credential per layer and refuse unusable connection values
Review follow-ups on the shared TypeSafe client (#5906):
- A consumer's `api_key_env` is no longer overridden by a literal `api_key` in the
shared `typesafe:` block: the first layer that sets either credential setting
decides it, and inside one layer the literal key wins (the standalone
semantics). A consumer whose endpoint does not accept the block's key no longer
sends it on every call, and the missing-credential message keeps naming the
variable that layer configured.
- `base_url` is validated where it is configured: `http(s)` only, with no query,
fragment, embedded credentials, whitespace or control characters, and a
malformed port fails at construction instead of on every request (httpx raises
`UnsupportedProtocol` / `InvalidURL` per call, and the query would have been
swallowed into the appended endpoint path).
- A credential that cannot travel as a header value — surrounding whitespace from
a secret mounted from a file, a non-printable character — is refused when the
connection resolves. It used to construct fine and then put the whole
`Bearer <key>` value into h11's `LocalProtocolError` message, which the
guardrail middleware's `logger.exception` prints.
- The client raises transport failures without chaining the original exception,
so that message can no longer carry the credential through the cause chain; the
exception type stays in the message for diagnosis.
- `TypeSafeConfig` validates in strict mode, agreeing with the consumer `config`
path: `max_attempts: true` was silently read as `1`. An integer stays a usable
float, so `timeout: 5` keeps working.
Tests: 20 cases across `tests/test_typesafe_config.py` and
`tests/test_typesafe_client.py`, each red on the previous revision and green now.
367 focused typesafe/memory/config tests, `make test-blocking-io` (167) and the
eval-script suites (50) pass; `ruff check` and `ruff format --check` are clean.
Docs synced: `backend/docs/GUARDRAILS.md`, `typesafe/AGENTS.md`,
`config.example.yaml`.
* fix(typesafe,memory): bound the reply, record request failures, gate enforce on the evidence
Second review round on #5906: the remaining items from the review thread and from
the follow-up pass.
Typesafe client (the reply is treated as hostile input too):
- Send `Accept-Encoding: identity`, refuse a body that carries a non-identity
`Content-Encoding`, and read through a bounded `iter_bytes` that stops past
`MAX_RESPONSE_BYTES` (64 KiB). A gzip body that expands to 128 MiB used to be
decoded in full and, on the gate path, allowed the tool call unexamined; it is
now `invalid_response` (a denial with `fail_closed: true`).
- Map `RecursionError` from `json.loads` to `invalid_response`: deep nesting
escaped the module's taxonomy, so the memory consumers lost the round's record
instead of recording a fallback.
Memory judge:
- A request-level failure is recorded as `request_failed`, never `no_verdict`
(`REASON_REQUEST_FAILED` was defined and never emitted, so an unreachable
endpoint was indistinguishable from a provider with no usable answer). The
adapters' `decide()` now propagates `TypeSafeError` for the coordinator to record
and log; `no_verdict` stays for a question-level failure, and an ineligible side
keeps its own reason even when the round's request failed.
Shadow evaluation (the enablement gate reads the evidence, not one number):
- The miss-rate gate requires the exact one-sided Clopper-Pearson upper bound to
meet the 1% target; a sample that only meets the point estimate is
`INSUFFICIENT` (0/250 was reported as a pass with a 1.19% bound).
- New `baseline_incremental_savings` gate: a baseline that saves as many calls as
the model — the shipped all-trivial sample — fails instead of enabling.
- Negative counts (`mutations_accepted`, message / batch / token counts) are
treated as absent rather than scored: `-3` read as "nothing lost", adding a
phantom saving to the numerator and the denominator.
Docs: the design documents cited but absent from the repository are replaced with
in-repo pointers (`MEMORY_IMPROVEMENTS.md`, `agents/memory/AGENTS.md`,
`GUARDRAILS.md`), and the gate and limit wording now matches what the code
enforces.
Tests: 31 new or updated cases, each red on the previous revision (compressed body
refused even when it decodes to a valid envelope, body over the cap refused, body
at the cap answered, deep nesting mapped, `request_failed` on each request-level
failure, the bound and baseline gates, the negative counter). 454 focused tests,
`make test-blocking-io` (167) and the eval suites pass; `ruff check` and
`ruff format --check` are clean.
* fix(memory-eval): pair the baseline comparison on the same records
Review follow-up on the incremental-savings gate added in ed930478 (P2): both
sides of the comparison are now the records the no-network baseline can score
(`trivial_only` plus a readable signals list).
Subtracting that subset's heuristic savings from model savings over *all* scorable
skips credited the model with batches the baseline never saw: 300 reviewed skips
with a single annotated record computed 300-1=299 "incremental", passed the gate
and set `enable_recommended`, while the paired comparison was model=1 against
heuristic=1 (incremental 0).
- `Evidence` carries the paired numbers (`baseline_paired_skips`,
`baseline_model_saved_calls`, `baseline_heuristic_saved_calls`) instead of one
heuristic count, filled by `build_report` from the baseline summary.
- The gate reports `INSUFFICIENT` when no scored skip carries the annotation and
when the annotation covers only a subset of the scored skips (the message carries
the paired counts and the `paired of scored` coverage), `FAIL` on a non-positive
paired gain, and `PASS` only on a positive gain over the full paired set.
- Docs state the pairing/coverage requirement in
`backend/docs/MEMORY_IMPROVEMENTS.md` and in the script docstring.
Tests: the reviewer's counterexample as a report-level regression (it failed with
`assert 'PASS' == 'INSUFFICIENT'` on ed930478 while `enable_recommended` was true),
a unit case for the paired-subset `INSUFFICIENT`, and the renamed paired fields.
344 focused tests, `make test-blocking-io` (167) and the eval suites pass;
`ruff check` / `ruff format --check` clean.
* fix(typesafe): bound the sync body read by the deadline
`deadline_seconds` only covered the sync path's headers, the completed body and
the parse, so an endpoint that drips bytes just under the read timeout held the
caller for one read per piece — the whole of `MAX_RESPONSE_BYTES` past the budget.
The read loop now checks the budget before every transport read, which bounds the
overshoot to one read. The pieces are the transport's own reads, not re-chunked
output: `iter_bytes(chunk_size)` pulls many reads inside a single yielded chunk,
so a check at that granularity would never see a drip.
`tests/test_typesafe_client.py` pins it with a valid envelope delivered as 24
pieces of 50 ms against a 0.2 s deadline: red before (24 pieces pulled), green now
(~5), on both the sync and async paths. Docstring and `typesafe/AGENTS.md` track
the new granularity.
* fix(memory): read the YAML off spelling, and veto a skip over the whole batch
Two review findings on the memory judging path.
`memory.prescreen.mode` / `memory.signal_classification.mode` rejected the
`mode: off` spelling `config.example.yaml` ships and a rollback types: YAML 1.1
parses the unquoted value as a boolean, so both fields saw `False`. The failed
reload left the previous judge running, so both fields now normalize `False` to
`off` before the `Literal` runs. `True` stays a validation error (no mode it
could mean). The examples quote the value, and the docs record why.
The pre-screen's L3 veto read `detect_signals`' 6-message window while the batch
it answers for is the whole post-watermark feed, so an explicit correction in an
earlier turn of an eight-message batch did not stop the skip that then advanced
the watermark past it. `detect_signals` gains a `window` parameter (default
unchanged) and the updater scans the full feed for the veto; the extraction hint
and the reinforcement gate keep the window.
Both regressions are red on the previous head: the config test reproduced the
`input_value=False` rejection through `AppConfig.from_file`, and the batch test
reproduced the prescreen judging (and skipping) a batch whose first turn carries
the correction.
* fix(models): skip credential files whose access token is not a string
* fix(models): normalize the file-route access token and guard the refresh token
* fix(gateway): keep /api/user-profile filesystem work off the event loop
GET and PUT /api/user-profile resolved the per-user USER.md path (which
builds absolute paths on every call), stat'ed, read, created the user
bucket and wrote the file inline on the event loop, while every other
handler in the custom-agent router offloads that work with
asyncio.to_thread. The strict Blockbuster gate raises BlockingError on
both; in production a slow disk stalls every other request on the worker.
Offload the whole resolve-stat-read and resolve-mkdir-write sequences,
capturing the effective user id on the loop first like the sibling
handlers. The blocking-IO gate test for this router now covers both
profile routes (missing profile, write, read back).
* docs(changelog): reference #5935 in the user-profile blocking I/O entries
Running the suite on native Windows blocked forever inside
test_mcp_task_service.py, and the whole shard with it. Root cause chain:
1. The tests patch _CANCELLATION_DRAIN_TIMEOUT_SECONDS to 0.01s. On
Windows + CPython 3.12, asyncio.wait_for over an already-created task
with a 10ms timeout raises TimeoutError immediately (measured 30/30 on
3.12.13, 0/30 on 3.11.9, same host; 50ms works on 3.12) - the claim
never gets its first scheduling slice inside the drain window.
2. _claim_with_cancellation_release therefore abandons the instant claim
to the background handoff; when it completes, the handoff calls
repo.release_poll_claim_after_cancellation - which
OrdinaryReleaseBatchRepository (the test fake) does not implement,
so the release dies with AttributeError.
3. The test then waits unbounded on repo.release_started.wait(), which
can now never fire: the whole suite hangs.
- add the missing release_poll_claim_after_cancellation to the fake so
it matches the repository contract the service drives
- raise the patched drain 0.01 -> 0.25 and the outer caller wait 0.2 ->
0.5 so the race window is above Windows timer granularity and the
handoff-vs-outer-timeout ordering the assertions expect is preserved
- bound release_started.wait() so a contract/timing regression fails in
seconds instead of hanging the run
The file now completes on Windows in ~12s (previously hung forever);
Linux/CI semantics are unchanged - the claims still complete well inside
the drain there.
* fix(goal): show the active goal to the agent on every model call
Nothing put ThreadState.goal into model requests. The agent only knew the
goal from messages: the objective the Web UI and IM channels send, and the
hidden <goal_continuation> message. Goals set through PUT /goal, the
embedded client or the TUI never reached it, and compaction could remove
the messages that stated the others.
DurableContextMiddleware now renders the active objective first in its
hidden durable-context data message, escaped, bounded, PII-redacted and
treated as data like the summary. Only the objective is rendered, so the
block changes only when the goal is set or cleared.
* fix(goal): let the agent pursue the active goal at user priority
The durable-context contract told the model never to follow instructions in
its data message, and that message now carried the goal. While a goal is
rendered, the contract ends with a static exception for the <active_goal>
element that opens the data message: work toward it as a request in a user
message, with no system or developer authority, while every other field
value, and any other text that calls itself a goal, stays data.
The element replaces the heading the goal used, which a summary could
reproduce. Other field values in the data message are HTML-escaped, and
active_goal joins the sanitizer denylist, so user input and remote tool
results cannot carry a raw one either.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* feat(agents): persist tool artifacts as resolvable handles (#4676)
MCP and sandbox tools return file paths, URLs, and task ids that are
lost when context compaction summarizes the conversation. This adds a
per-thread artifact handle registry so the model can reference
artifacts by short deterministic handles (art_xxxxxxxx) that survive
compaction and resolve to real references at the tool-call boundary.
- ThreadState.tool_artifacts: reducer channel capped at 100 entries,
same-handle latest-wins, survives summarization compaction
- artifact_registry: deterministic handle derivation from
(thread_id, tool_call_id, call_index); extraction precedence
structured_content -> file/image blocks -> conservative text scan;
HTML-escaped model-context rendering with char budget
- ArtifactCaptureMiddleware: before_model hook captures lightweight
metadata (never content) and tracks handle consumption
- ArtifactResolutionMiddleware: rewrites handles in tool-call args to
real references before execution; unknown handles untouched
- DurableContextMiddleware: projects the registry into the hidden
durable-context data block ([available]/[consumed] status)
- config: new tool_artifacts section (all switches default on)
- frontend: artifact badges on tool-call steps keyed by tool_call_id
Fixes#4676
* fix(agents): harden artifact registry after review (#4676)
Review findings on #4929:
- capture/consumption updates emitted in the same before_model call are
now concatenated instead of dict-merged; dict-merging clobbered the
shared tool_artifacts key and permanently dropped fresh captures
- handles embed a per-result ordinal so multiple references inside one
tool result never collide (two paths in one text block previously
collapsed into one handle)
- structured_content extraction pulls concrete refs from known keys
(file/path/url/task_id/..., string or list-of-string values) instead
of storing a truncated JSON preview, falls back to an untruncated
whole-object data entry, and no longer skips file/image content blocks
- plain-string tool results (e.g. bash/write_file output) are scanned
for sandbox paths / remote URLs too, gated by detect_refs_in_text
- wire max_entries through configure_tool_artifact_max_entries at
assembly (clamped 10..1000) and gate detect_refs_in_text scans
- resolution requires both enabled and resolve_handles_in_args,
matching its docs
- add real ToolCallRequest.override coverage and the spec 14.4
integration cycle (capture -> inject -> resolve -> consumption)
* fix(agents): address second-round artifact registry review (#4676)
Follow-up findings on #4929:
- drop generic `output`/`outputs` from structured ref keys: prose under
those keys no longer becomes a bogus file entry (falls back to the
whole-object data entry; path-shaped refs are still caught by the
free-text scan)
- strip closing quotes, brackets, and backticks from detected refs so
markdown- and JSON-wrapped paths resolve to clean real references
- remove the process-wide cap setter: the reducer enforces only an
absolute ceiling (1000) and ArtifactCaptureMiddleware applies the
configured max_entries per agent when emitting updates, so agents
with different configs in one process never share or flap a cap
- skip extraction for tool messages whose tool_call_id already has
registry entries and memoize known-empty results (bounded FIFO), so
steady-state capture cost drops to the new message tail instead of a
full-history rescan per model call
- correct the inverted ordering claim: capture is position-independent
in the wrap chain because it is a before_model hook reading state
messages; fixed the assembly comment, AGENTS.md item 15, and the
ordering test docstring
CI parity verified locally: ruff check + format --check, full offline
suite, blocking-io suite, collect-only, uv lock --check,
check_agent_guidance, frontend format/lint/typecheck/build/tests.
* fix(agents): sliding-window cap and URI gates for artifact registry (#4676)
Third-round review findings on #4929:
- max_entries is now a sliding retention window: fresh captures are
always registered, and when the projection exceeds the configured
cap the middleware appends a trailing {"op": "trim_to", "keep": N}
directive that the reducer applies by evicting the oldest entries.
The previous remaining-budget emission froze the registry at the cap,
permanently dropping post-compaction artifacts — the exact failure
mode #4676 targets — and silently disabled the round-2 rescan
optimizations at the point threads grew longest
- _track_consumption memoizes settled tool-call args in a bounded FIFO
alongside the existing empty-result memo, so historical AIMessage
args are not regex-rescanned every model call
- file/image content-block URLs must be http(s) or path-shaped;
data:/blob: URIs (MCP embedded resources carrying base64 payloads)
can no longer enter thread state, tool args, or crowd out the render
budget; protocol-relative //host forms are rejected as well
- structured-content ref values go through the same URI-shape gate as
content blocks, closing the embedded-payload hole via known keys such
as url/urls
- tool_artifacts.enabled now gates the whole middleware including
consumption tracking via an early return in before_model
Docs updated to describe the sliding window, URI gates, enabled scope,
and both bounded memos.
* fix(agents): final evictions and pending-aware consumption for artifact registry (#4676)
Fourth-round review findings on #4929:
- the sliding window interacted with the seen_call_ids skip to
resurrect evicted entries every round: once an entry was trimmed, its
tool_call_id left the seen set while its ToolMessage stayed in
context, so the next model call re-extracted it, re-registered it as
newest, and evicted something else — a never-converging rotation.
The empty-result memo is now a unified _processed_results memo that
records every extraction regardless of outcome, making evictions
final; steady state over a saturated thread is quiescent (pinned by
a multi-round test asserting no updates and zero extractions)
- the consumption scan built its handle map from pre-capture state, so
a call referencing a handle captured in the same before_model call
missed, got settled as quiet, and its consumption mark was lost
forever. _track_consumption now receives the pending capture entries
and resolves against state + pending; a call settles as quiet only
after every referenced handle resolved, so transient misses retry
next round instead of permanently suppressing the mark
AGENTS.md entry 15 updated to describe both memos and the
eviction-is-final semantics.
* fix(artifacts): address occurrence identity, PII and URL review findings
* fix(artifacts): enforce resolved policy checks and bounded registry handling
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(docker): give project-document uploads their own nginx location
POST /api/projects/{id}/documents is a multipart upload that Gateway
accepts up to uploads.max_file_size (50 MiB by default), but no nginx
location matched it. It fell through to the /api/ catch-all, which has no
client_max_body_size, so nginx's 1m default answered a 2 MB PDF with a
bare 413 before Gateway saw the request. Reproduced on nginx:alpine with
the shipped config; the same body to /api/threads/{id}/uploads passed.
Add a `~ ^/api/projects/[^/]+/documents` location to the Docker, make dev
and Helm configs with the thread-uploads settings (100M, streamed body)
and the 600s read timeout the catch-all already granted these routes. The
catch-all keeps nginx's defaults. The shared location-block helpers move
to tests/support/nginx_conf.py so the new test and the existing body-size
test use one implementation.
* docs(changelog): reference #5934 in the project-documents nginx entries
22 backend tests assert facilities that only exist on a POSIX host and
hard-fail on Windows (CI is ubuntu-latest, so upstream never sees them):
- test_extension_dependency_sync.py: 11 tests spawn bare 'make -n';
hosts without GNU make get WinError 2 (make is not part of the
documented Windows setup).
- test_lark_broker.py: 10 tests execute the '#!' fake lark-cli / shim /
launcher scripts; CreateProcess cannot run shebang scripts
(WinError 193).
- test_bench_sandbox_provider.py: the boxlite shim workaround test
asserts POSIX exec-bit semantics after os.chmod; Windows has no
execute bit and boxlite sandboxes are Linux-only.
Each now skips with an explicit reason when its facility is absent;
Linux/CI coverage is unchanged. On a Windows host the three files go
from 22 failed to 0 failed / 22 skipped.
Fixes#5930
* fix(deploy): honor BETTER_AUTH_SECRET and DEER_FLOW_INTERNAL_AUTH_TOKEN from .env
deploy.sh only checked the shell before reloading a persisted secret or
generating a fresh one and exporting it. Compose interpolation lets shell
variables outrank --env-file, so a secret written to the repo-root .env --
the surface every deployment doc points at -- was silently replaced by the
generated one: sessions were signed with a secret the operator never chose
and Gateway workers outside the stack got 401 with the configured token.
Leave a .env-provided secret unexported so Compose parses it itself
(shell -> .env -> persisted file -> generated), and keep generating when
the shell exports an empty value, since Compose would otherwise pass the
empty string through.
* docs(changelog): reference #5928 in the deploy.sh dotenv secrets entries
* fix(deploy): decide whether .env provides a secret with Compose's own parser
read_dotenv_value only matches KEY=VALUE and returns the literal text, but
Compose also accepts `KEY: VALUE` lines and interpolates `${VAR}` inside
values. So `TOKEN: abc` looked absent (and got shadowed by a generated
token) while `TOKEN=${UNSET}` looked present (and suppressed generation,
leaving the stack with an empty token).
Ask Compose for its resolved interpolation environment instead
(`docker compose config --environment`, same --env-file, stub project on
stdin, project directory docker/) and treat a resolved-empty value as
missing. Clients older than Compose 2.28 lack the flag and fall back to
the plain reader. Tests cover the canned Compose answers, the fallback,
and every dotenv spelling against the real Compose client when installed.
* fix(deploy): resolve .env secrets through Compose on every supported client
`config --environment` needs Compose 2.28, so 2.24–2.27 clients took the
plain KEY=VALUE fallback and kept both defects: `TOKEN: abc` was missed
and shadowed, `TOKEN=${UNSET}` counted as present and left the stack with
an empty token.
Drop the fallback. Render a stub project whose only environment entry is
`${KEY}` through `docker compose config` — same --env-file and project
directory as the real command — and read the interpolated value back;
`""` means empty or unset. This is plain interpolation, available on every
Compose v2, and verified identical on 2.24.0, 2.27.0 and 2.32. A failing
probe stops the script instead of guessing. The real-Compose tests now run
against the docker CLI and any standalone binaries listed in
DEER_FLOW_TEST_COMPOSE_BINARIES.
* feat: add optional role filtering to task history search
* docs: scope history role guidance to task continuity
* test: register task continuity guidance in scoped inventory
* docs: separate English guidance from Chinese README translation
* fix(skills): bail out of the markdown-link scan when no link construct exists (#5714)
* fix(skills): scan markdown links linearly and skip failed target runs (#5714)
* fix(skills): make link scan exactly match finditer, blank spans by position
Address review feedback: drive the scan by openers (the regex engine's
own start-position scan) so it reproduces finditer exactly, blank
matched spans by position instead of str.replace, pin the equivalence
with tests, and add CHANGELOG entries.
* fix: resolve skill review test lint errors
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(models): keep Claude prompt-cache markers out of thread messages
ClaudeChatModel._apply_prompt_caching set cache_control on payload blocks in
place. langchain-anthropic forwards Claude-native blocks (image or document
blocks with a source, search results) by reference, so the marker landed in
the thread's own message and was checkpointed with it. Each later request
carried those stale markers on top of its own four, and from the third turn
of a thread with such blocks the API rejected every request (400, "A maximum
of 4 blocks with cache_control may be provided").
Breakpoints are now placed on copies, and markers already in the request are
dropped first, so threads whose checkpoints already carry them recover
without a migration. The OAuth path's _strip_cache_control popped markers
from the same shared objects; it now copies too, and prompt caching reuses it.
* fix(models): drop stored Claude cache markers with caching off too
The strip only ran inside _apply_prompt_caching, so with
enable_prompt_caching: false a thread whose checkpoint already carried five
or more markers kept failing: exactly the switch an operator would flip in
response to these 400s. _get_request_payload now strips on the non-caching
path as well, so recovery no longer depends on the flag.
The stored-checkpoint recovery is now pinned through _get_request_payload
with real messages, for caching on and off. The docstring and the AGENTS.md
note now say which objects are really shared (blocks, list-form system
blocks, a reused binding's tool dicts), that the message-dict and list
copies are defensive, and that _apply_prompt_caching must strip first.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(docker): strip the IPv6 listen in the production nginx launcher too
docker/nginx/nginx.conf listens on 2026 and [::]:2026. On a host booted
with ipv6.disable=1 (or a kernel without IPv6) nginx refuses to start —
socket() [::]:2026 failed (97: Address family not supported by protocol) —
so the container restart-loops under `restart: unless-stopped` and
`make up` waits on a readiness probe that never turns healthy. #2027 added
a launcher guard that deletes the IPv6 listen when /proc/net/if_inet6 is
absent, but only to docker-compose-dev.yaml; the Helm chart mirrors it;
docker-compose.yaml — the file behind `make up` and scripts/deploy.sh —
still ran the bare copy-and-start command.
Give the production nginx service the same launcher as the dev file:
`set -e`, copy the template, guard, and `exec nginx` so nginx is PID 1 and
receives docker stop's signal directly.
The new test pins the guard in all three launchers (prod, dev, Helm),
requires the three sed patterns to be identical so they cannot drift again,
checks the guard edits the file nginx actually loads, and applies the
pattern to the shipped compose and Helm configs to confirm it removes only
the IPv6 listen. Verified on nginx:alpine: BusyBox sed strips exactly that
line and `nginx -t` accepts the result; the rendered production launcher
runs unchanged when IPv6 is present.
* docs(changelog): reference #5900 in the nginx IPv6 guard entries
* fix(skills): let remove_file/write_file handle binary support files and reject directories
A .skill archive may carry binary support files such as assets/logo.png:
the installer only rejects executable binaries by magic bytes. But both
skill_manage file actions read the previous content as UTF-8 text before
touching the file — purely to fill the history record's prev_content — so
a binary member raised UnicodeDecodeError out of
SkillStorage.remove_custom_skill_file (and out of the tool's write_file
prelude) and was never removed or overwritten.
A bare support directory ("assets") also slipped through
ensure_safe_support_path, because it resolves to its own allowed root and
passes the containment check, and then raised IsADirectoryError from the
same read.
Add read_text_or_none for the history reads: UTF-8 text is returned as
before, undecodable content yields None (the record simply has no previous
text), and IO errors still propagate. Require a filename below the support
subdirectory in ensure_safe_support_path, and reject an existing directory
with ValueError in both remove_custom_skill_file and the tool's write_file,
so the model gets an actionable validation error instead of a crash.
prev_content is informational for support files — rollback only restores
SKILL.md — so recording None loses nothing that could be used.
* docs(changelog): reference #5893 in the support-file entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): offload image search to a thread via asyncio.to_thread
Make image_search_tool async and run the blocking ddgs call in a worker
thread with asyncio.to_thread, continuing the TODO.md planned feature
('Replace sync requests with httpx.AsyncClient in community tools').
The ddgs SDK exposes no async API, so instead of swapping the HTTP
client the blocking DDGS.images() call is offloaded to a thread — the
same pattern the infoquest/jina_ai tools use for readability
extraction. Callers see no change in behavior.
Tests invoke the async tool via ainvoke under @pytest.mark.anyio.
* fix(image-search): offload config lookup with DDGS search
* test(image-search): pin DDGS search off the event loop
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
