* fix(sandbox): expose DEERFLOW_USER_ID to bash commands
A skill's SKILL.md can trigger scripts, but those scripts run as bash
subprocesses inside the sandbox and cannot reach the Gateway auth context,
so get_effective_user_id() is unavailable to them and user-scoped skills
are impossible to write. The maintainer settled the direction on #3919:
inject the id as DEERFLOW_USER_ID.
Every bash command now begins with 'export DEERFLOW_USER_ID=<resolved>; '
(or 'unset' when the resolved id is unusable), mirroring the existing
DEERFLOW_CHANNEL_USER_ID channel-identity prefix (#3914). The id rides the
command string rather than execute_command(env=...), because a non-empty env
switches AioSandbox to the bash.exec API reserved for request-scoped secrets.
The earlier attempt (#4029) was closed for making every command carry the
new export without updating the exact-command assertions in
tests/test_channel_user_id_env.py. Those assertions are updated here, and the
composition of the two prefixes is pinned by a dedicated test.
* fix(sandbox): fix the DEERFLOW_USER_ID unit test and cover review findings
Follow-up to the review on this PR.
Unit test (CI `backend-unit-tests (2)`): the change moved the command
contract but left the exact-command assertion in
tests/test_skill_request_scoped_secrets.py::test_local_bash_forwards_env_and_timeout
expecting the bare `echo hi`. The local validation run used a `-k` filter that
did not select that file, so the breakage was invisible there. The assertion
now pins the composed command, and `_is_windows` is patched explicitly so it no
longer depends on the host OS.
[P2] Windows local sandbox: `export` is invalid under the PowerShell/cmd.exe
fallback, but skipping the prefix outright left the promise unmet on Windows.
The id is published through `execute_command(env=...)` there instead:
LocalSandbox layers `env` into the per-process environment and, unlike
AioSandbox, has no persistent session that could carry a stale value. It is
deliberately kept out of `injected_env` so the identifier does not enter the
secret-redaction set.
Docstring: `_user_identity_prefix` now carries the same "informational, not
authenticated identity" caveat the channel variable has in
backend/app/channels/AGENTS.md. The value is authorization-grade at the Gateway
boundary, but any bash command can overwrite its own environment, so skills
must not treat the shell variable as proof of identity.
Docs: the en/zh sandbox.mdx identity sections and the channels/AGENTS.md
identity paragraph now describe the second, outermost prefix, its ordering
against the channel prefix, and the Windows behaviour.
* fix(goal): record a failed evaluator call as a stand-down
When the goal evaluator call raised (a provider error, or an answer that
was not valid JSON), _prepare_goal_continuation_input() logged a warning
and returned without writing anything. The goal stayed active with no
last_evaluation, or with an earlier run's verdict and run id, unlike the
other paths that stop the goal loop.
Record it like the no_durable_end_of_turn stand-down: blocker run_failed,
stand_down_reason evaluator_failed, the current run id, and a reason that
names only the exception type. Nothing is recorded if the run was aborted
meanwhile. A failure before the evaluator is called keeps its behavior and
now logs "Could not prepare goal evaluation".
* docs(gateway): fit the evaluator-failure note in the AGENTS.md budget
Main grew backend/app/gateway/AGENTS.md to 49,006 bytes, and the longer
note pushed the merged file past the 49,152-byte hard limit that
tests/test_agent_guidance_check.py enforces. Say it in one clause.
* docs(gateway): keep main's AGENTS.md line after the merge
The merge from main kept this branch's version of the goal paragraph,
which restored the Human Input Card wording #6068 replaced and put the
file at 49,194 bytes, over the 49,152-byte limit the agent-guidance
check enforces. Take main's line. It already says every non-satisfied
evaluation is persisted with last_evaluation, which this fix makes true
for a failed evaluator call.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(subagents): retain completed model usage on cancellation
* docs(subagents): fit guidance within chain budget
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
---------
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(goal): show the goal evaluator the tool calls and results
The evaluator read only the text of user and assistant messages and was told
to fail closed without explicit evidence, so it stood down with
missing_evidence on most completed file tasks: the tool calls and tool
results that the web UI shows never reached it.
Include the assistant's tool calls and the tools' results inside the
existing message window, shortened (200 characters per argument value, 600
per step, line breaks kept), and tell the evaluator to treat tool results as
data, that a successful tool call alone does not meet a goal, and that a
guessed input means needs_user_input. The no-progress and final-clear
signatures are unchanged.
* fix(goal): tighten the evaluator's tool evidence after review
- Pair each tool result with the latest call of its id before it. Some
providers reuse ids such as call_0 across turns, and the window-wide
map labelled an earlier read with a later write's name.
- Keep model-supplied tool names on one line in calls and result labels,
and escape U+0085, U+2028 and U+2029 in arguments and results, which
json.dumps(ensure_ascii=False) leaves raw, so no tool step can start a
new evidence line.
- Leave out the results of a hidden assistant message's calls. The web UI
renders a result only under its call, so neither shows there either.
A clarification prompt stays: the UI shows it as its own card.
- Over the 12,000-char cap, keep whole lines from the end instead of
slicing mid-line. Keep the latest user message that the cut drops at
the top, whole when it and the lines after it fit, else shortened to
2,000 chars. If the next assistant message back does not fit whole,
keep its end in the room left. A marker counts the lines left out.
- Docs: the Gateway AGENTS.md and a worker docstring no longer say the
evaluator never sees tool messages.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(events): read a non-filename run ID as an unknown run on JSONL
The run-scoped routes (events, messages, workspace-changes) pass the URL's
run_id straight to the event store. With run_events.backend: jsonl, an ID
that cannot be a filename (run.1, a%20b, %2e%2e) raised ValueError and
surfaced as a 500, while the memory and database stores return an empty
result.
JSONL reads and deletes now treat such an ID as an unknown run through one
helper, _existing_run_file(), which returns before building any path.
Writes still reject it.
* docs(changelog): reference #6070 for the JSONL run ID fix
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(persistence): reject SQLite file: URIs instead of opening them as filenames
resolve_sqlite_conn_str passed a file: connection string through
unchanged, but LangGraph's SQLite checkpointer and Store factories
(SqliteSaver, SqliteStore, AsyncSqliteSaver, AsyncSqliteStore) connect
without uri=True, so SQLite treated the whole URI as a literal filename:
file:checkpoints.db?mode=rwc created a file with that exact name in the
working directory, file::memory:?cache=shared persisted to disk, and a
file:/// URI failed to open. The readiness probe did open URIs with
uri=True, so it checked a different file than the runtime used and
reported in-memory URIs as not_configured.
Reject file: URIs in resolve_sqlite_conn_str, the one helper all four
factories go through, with an error that names
checkpointer.connection_string. The probe drops its URI parsing and
reports a rejected connection string as unreachable.
* docs(changelog): reference #6069
* test(persistence): pin the case-sensitive file: URI check
* fix(config): upgrade the config.yaml the Gateway loads
scripts/config-upgrade.sh resolved DEER_FLOW_CONFIG_PATH, then
backend/config.yaml, then <checkout>/config.yaml. make dev exports
DEER_FLOW_PROJECT_ROOT=<checkout>, so the Gateway reads the checkout copy
first: with both files present the script upgraded a file nothing loads
and left the live config outdated while reporting success. It also
ignored DEER_FLOW_PROJECT_ROOT and a DEER_FLOW_CONFIG_PATH set in .env,
and fell back to another file when DEER_FLOW_CONFIG_PATH named a
missing one.
Ask the harness resolver (AppConfig.resolve_config_path) instead of
copying its order, defaulting DEER_FLOW_PROJECT_ROOT to the checkout as
serve.sh does. A missing DEER_FLOW_CONFIG_PATH or invalid project root
now fails with the Gateway's error; no config anywhere still creates
<checkout>/config.yaml from the example.
* docs(changelog): reference #5991
* test(config): pin config-upgrade seeding a missing config from the example
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
path_matches backs the glob tool and grep's glob= filter for every sandbox provider. It relied on PurePosixPath.match, which matches from the right and treats ** as a single-segment * on Python 3.12+, so only a leading **/ was recursive: src/**/*.py missed src/top.py and anything two or more levels deep, and docs/*.md also matched vendor/docs/b.md.
Match patterns that contain a / segment by segment from the search root, with ** spanning zero or more directories. Patterns without / keep matching the basename at any depth.
* fix(community): close the per-call Firecrawl client's async HTTP pool
The firecrawl tools build a fresh AsyncFirecrawlApp for every web_search
and web_fetch call. The migration to the async client (#5786) kept the
per-call construction, but AsyncFirecrawlApp eagerly creates an
httpx.AsyncClient-backed pool in its constructor and exposes no public
teardown, so each tool call abandons one unclosed pool. The sibling
providers either scope their httpx clients with 'async with' or close
them in a finally (tavily); firecrawl now does the same via a
best-effort teardown that tolerates SDK shapes without the v2 client.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(firecrawl): make per-call pool teardown resilient
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
* test(firecrawl): skip the real-SDK teardown test when the private layout differs
The real-SDK test hard-asserted the locked firecrawl-py 4.23.0 private
layout (_v2_client.async_http_client._client.is_closed), so an
environment resolving another version in the declared >=1.15.0 range
died with AttributeError instead of skipping — while the helper (and
test_teardown_survives_a_client_without_v2_attributes) deliberately
treat a missing or reshaped layout as a no-op. Guard on the same
attributes and skip when they are absent.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
Signed-off-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): unwrap Overwrite-wrapped sandbox state in tools and middlewares
* fix(changelog): restore database entry and add #4380 reference link, simplify task_tool unwrap
* fix(subagents): preserve Overwrite wrapper when task_tool forwards sandbox state to executor
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(integrations): keep Lark runtime status consistent
Probe sandbox runtime readiness for every Lark mutation response outside the credential lock. Expose explicit sandbox_runtime_probed state so the frontend can distinguish conservative fallbacks without inferring semantics from runtime detail text, while cancelling stale reads before updating the cache.
* style(frontend): format Lark cache test
* fix(integrations): snapshot Lark credential status under the lock
Why this change is needed:
PR #5994 moved the post-mutation status read outside the per-user
credential lock so the new sandbox runtime probe would not extend the
critical section. As flagged in review, that lets a concurrent app
switch or auth flow land between the committed mutation and the status
read; in remote-provisioner mode the runtime probe can spend seconds
before probe_lark_auth runs, so the raced switch can clear the new
tokens and make a completed login report success=false, or mix an old
app_id with a newer auth state.
What changed:
- add _LarkCredentialSnapshot (app config + auth probe) and capture it
in the same _lark_credential_lock section that commits the mutation
- thread the snapshot through _get_lark_mutation_status into
get_lark_integration_status(credential_snapshot=...); only the
credential-independent runtime probe still runs after the lock
- apply to complete_lark_auth, complete_lark_config, and
set_lark_app_credentials; install keeps the one-shot status build
Problem solved:
Mutation responses stay tied to the just-committed credential
generation: a raced app switch can no longer flip a completed
device-code login to success=false or blend a stale app_id with newer
auth state, while the runtime probe stays off the lock.
* fix(integrations): address Lark status review follow-ups
- install_lark_integration: snapshot credential-derived status fields
under _lark_credential_lock so the install response cannot mix newer
credential state read across the unlocked runtime probe
- useInstallLarkIntegration: route the cache write through
cacheLarkMutationStatus so install cancels in-flight reads and keeps
probed runtime fields under frontend/backend version skew
- README: soften the UI claim - the status API exposes
sandbox_runtime_probed and the mutation cache honors it
* test(integrations): tighten Lark status fixtures and guidance docs
- cache.test: use a distinct mutation runtime mode in the unprobed-merge
case so the assertions prove the cached runtime fields win
- e2e mock: include broker in the sandbox_runtime_mode union cast so the
fixture cannot drift from LarkSandboxRuntimeMode
- AGENTS.md (gateway + skills): mention sandbox_runtime_probed alongside
the existing runtime readiness fields, keeping the gateway guidance
file under its byte budget
* fix(integrations): harden Lark mutation cache against skew and races
- cacheLarkMutationStatus: await cancelQueries before writing so an
in-flight GET whose fetch already resolved cannot clobber the fresher
mutation status, matching the existing threads/archive pattern
- self-heal version skew: an unprobed (pre-field backend) mutation
response now triggers a status refetch instead of risking the
conservative runtime fallback lingering until the next focus/remount
- api: normalize a missing sandbox_runtime_probed to false at the
boundary so the declared-non-optional type matches skewed runtimes
- tests: pin the skew contract (missing-field cases for cache and api)
and add a useInstallLarkIntegration regression test proving the
install cache write goes through cacheLarkMutationStatus
- README: describe the probed flag as cross-version semantics instead of
claiming the same-version API surfaces both states
- lark_cli: note that install's credential snapshot runs the auth probe
subprocess while holding the credential lock
* refactor(integrations): derive sandbox_runtime_probed inside the resolver
_resolve_sandbox_runtime_readiness now returns a fourth probed element
next to the branch that produces the unevaluated remote fallback, so the
'was readiness actually evaluated' invariant can no longer drift from the
fallback logic it mirrors. Callers and resolver-level tests updated to
pin probed per branch.
* fix(doctor): check the config file the Gateway actually loads
make doctor always inspected <checkout>/config.yaml and ignored
DEER_FLOW_CONFIG_PATH and DEER_FLOW_PROJECT_ROOT. A missing override
that stops the Gateway from starting still reported "config.yaml found"
and "config.yaml loadable", and a valid override pointing elsewhere got
the wrong file checked.
Doctor now resolves the path through the harness's own resolver,
AppConfig.resolve_config_path, after defaulting DEER_FLOW_PROJECT_ROOT
to the checkout as scripts/serve.sh does for make dev. An override the
Gateway would reject fails "config.yaml found" with the Gateway's error
and the config checks skip; an unimportable harness is reported rather
than raised.
* docs(changelog): reference #5987
* fix(doctor): hand the config-location variables over like serve.sh
Review follow-up. serve.sh sources the checkout .env over the shell and
then replaces an unset or empty DEER_FLOW_PROJECT_ROOT with the
checkout. Doctor loaded .env shell-first and only defaulted an unset
root, so a shell DEER_FLOW_CONFIG_PATH / DEER_FLOW_PROJECT_ROOT that
differed from .env, or an empty root, made it check a different file
than the Gateway loads.
For those two variables the .env value now overrides the shell (other
keys keep doctor's shell-first loading), and an empty root becomes the
checkout, before the harness resolver runs.
* fix(doctor): report any harness import failure and expand ~ like source
Review follow-up.
- The harness-import guard caught only ImportError, so a harness that is
present but broken differently (TypeError/SyntaxError while the module
body runs) still crashed doctor. Catch Exception, as
check_config_loadable does, and name the exception type in the report.
- serve.sh's `source .env` expands an unquoted leading `~` in
DEER_FLOW_CONFIG_PATH / DEER_FLOW_PROJECT_ROOT and keeps a quoted one
literal; dotenv_values expands neither. Expand the unquoted form so a
`~/...` path in .env no longer fails a setup that starts, and keep the
quoted form literal so doctor does not pass one the Gateway rejects.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): close thread-scoped MCP sessions on thread deletion
Deleting a thread removed its filesystem data, checkpoints, runs, events,
feedback, metadata and browser session, but never closed its persistent MCP
sessions. MCPSessionPool.close_scope() had no production caller at all, so a
recreated thread id under the same user could reuse stale stateful MCP server
state, and the leaked owner task plus subprocess kept holding memory and file
descriptors.
Add MCPSessionPool.close_thread_scope(user_id, thread_id) and call it from the
reserved deletion path under the same best-effort policy as the neighbouring
cleanups. It matches every incarnation of the thread -- the legacy user:thread
scope and any v2:[user, thread, incarnation] scope -- because a deletion cannot
reliably read the current incarnation: the record may predate incarnation
tracking, and a new incarnation can be minted concurrently. close_scope() is
unchanged for callers that already hold one exact key.
Add mcp_scope_belongs_to_thread() to own the matching rule, and hoist the v2:
prefix into a shared constant so the encoder and matcher cannot drift.
Refs #5188
* test(mcp): bind owner identity in thread cleanup checks
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
_coerce_line_number() caught only (TypeError, ValueError), but a
model-emitted JSON number such as 1e999 parses to float('inf') and
int(inf) raises OverflowError, which escaped _track_and_check() through
after_model() and crashed the run. Treat the bound as unusable (None),
matching the OverflowError guards already merged for the search-tool
coercers.
`health_check_skip_seconds` was the only float field in SandboxConfig
without `allow_inf_nan=False`, and `ge=0` alone does not stop inf
(`inf >= 0` is true). A config.yaml value of `1e999` loads as the string
"1e999" and pydantic's float coercion turns it into inf, so the field
accepted it and the BoxLite warm-reuse site then skipped the health
check permanently: `skip_seconds > 0 and (time.time() - released_at) <
skip_seconds` holds forever for inf, so every box this instance
recently released is promoted without validation instead of failing
fast at config load.
Add `allow_inf_nan=False`, matching the sibling sandbox float fields
(`bash_command_timeout`, `renewal_interval_seconds`, `ttl_multiplier`)
and the ownership lease-TTL model validator.
Red/green: `test_sandbox_config_rejects_non_finite_health_check_skip_seconds`
parametrizes inf and "1e999"; both were accepted before the fix (RED)
and are rejected after. Full boxlite suite: 54 passed / 15 skipped.
Regression: the sandbox/config/ownership subset fails 17 tests on this
Windows host both with and without the change, with a byte-identical
failure-name set (environment-type: process-spawn and file-stat
behavior), so no regressions are introduced.
Signed-off-by: yetuge <2219677952@qq.com>
Replace the synchronous FirecrawlApp with AsyncFirecrawlApp in the
firecrawl community tools, continuing the TODO.md planned feature
('Replace sync requests with httpx.AsyncClient in community tools').
web_search_tool and web_fetch_tool become async functions and await
client.search()/client.scrape(). Return values (SearchData.web,
Document.markdown/metadata) are identical between the sync and async
clients, so callers see no change; error handling and the api_key /
api_url construction path are unchanged.
Tests now invoke the async tools via ainvoke under @pytest.mark.anyio,
and the client search/scrape methods are mocked with AsyncMock.
* fix(tui): report headless --print/--json failures instead of leaking tracebacks
The headless --print and --json runners dispatched straight into the
embedded client with no error boundary, so session-creation, thread-
resolution, chat, and stream failures escaped main() and surfaced as
unhandled Python tracebacks from the console script - for example the
documented CheckpointModeMismatchError case, which the checkpoint
operations/troubleshooting docs recorded as a raw traceback.
Both runners now catch Exception (KeyboardInterrupt/SystemExit still
propagate), mirroring the TUI's stream_actions() boundary:
- --print writes one concise 'Error: ...' line to stderr and returns 1
- --json appends a terminal {"type": "error"} record to the NDJSON
stream (diagnostics stay on stderr) and returns 1; every stdout line
remains independently valid JSON, including failures mid-stream
The checkpoint operations/troubleshooting docs (en + zh) no longer
describe the headless runners as printing an uncaught traceback.
Fixes#5962
* fix(tui): close the broken-pipe edge and document the --json error contract
Review follow-ups on #5963 (both non-blocking):
- print(answer) sat outside the headless boundary, so a consumer closing
the pipe early (deerflow --print ... | head) still got a
BrokenPipeError traceback plus the interpreter's shutdown-flush noise.
The final --print write and the --json terminal error-record write now
catch BrokenPipeError, point the stdout descriptor at devnull (the
documented SIGPIPE-adjacent recipe) and exit 1 without a traceback.
- The terminal {"type": "error"} record is a real contract for --json
scripters but was only documented in the checkpoint pages; the
backend/docs/TUI.md command table and the README cheat line now state
the error record and the exit-1 semantics where CLI consumers look.
* fix(mcp): isolate personal connections from deployment configuration
* fix(mcp): preserve personal task ownership and repair plugin E2E fixtures
* fix(mcp): handle corrupt personal config and cache owned calls
* fix(mcp): pin personal endpoint addresses and handle invalid listings
* fix(mcp): recheck current authority for privileged personal connections
config.example.yaml documents web_fetch timeout/fetch_time/navigation_timeout and
the two search_time_range options as seconds, and a $VAR reference resolves
verbatim from the environment, so any of them can reach InfoQuestClient as a
string. The client compares each with 0, which raises TypeError for a string,
None or a mapping and takes the whole web_fetch/web_search/image_search call
down. Reuse the provider-local coercion the sibling tools already have: accept
an int, integral float or integer-form string, otherwise warn and keep the
documented -1 (no filter).
* fix(agents): key generic tool calls on all non-narration args in loop detection
Layer 1 keyed every tool without a dedicated rule on its first salient
field (path/url/query/command/...) and dropped all other arguments, so
paging a URL or a search (start_index, page, cursor) and rewriting one
skill file collapsed onto one key and hard-stopped on the fifth distinct
call. Key on every argument except the narration field (description),
which keeps the #1905 bypass closed.
Fixes#5871
* fix(agents): drop description only for sandbox UI narration in loop keys
update_agent/setup_agent persist description and MCP tools such as
create_issue send it, so description-only updates must stay distinct.
Drop it only for bash/ls/glob/grep, where it is the UI explanation of
the call and was already outside the key before this PR.
---------
Co-authored-by: destire-mio <248462155+destire-mio@users.noreply.github.com>
* fix(agents): preserve response_metadata in dynamic_context user copy
The re-id'd user message was rebuilt field-by-field, dropping
response_metadata (and any other caller-attached field) from what the
run actually sees. model_copy keeps the documented {id}__user swap
while preserving the rest, matching the pattern already used by the
pii_redaction, thread_data and input_sanitization middlewares.
Fixes#5976
* fix(agents): preserve message metadata during upload injection
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add Unbrowse web fetch provider
Add a community web_fetch provider backed by Unbrowse
(https://unbrowse.ai), a hosted service that returns a page as markdown,
over plain HTTP when that is enough and through a cloud browser when the
page needs JavaScript. The tool sends one JSON-RPC tools/call POST
(unbrowse.scrape) to Unbrowse's MCP endpoint, so it needs no extra Python
package and no MCP client (uses httpx, already a dependency).
Unbrowse is not a general web search engine, so only web_fetch is added.
Changes:
- backend/packages/harness/deerflow/community/unbrowse/{__init__,tools}.py
web_fetch_tool with the same output shape as the Sofya, Exa and
Firecrawl fetch tools (# <title> plus markdown, 4KB cap). API key is read
from the config.yaml api_key field or the UNBROWSE_API_KEY env var.
Optional render: auto|never|always; unknown values log a warning and fall
back to auto. JSON-RPC errors, isError tool results, HTTP errors and
malformed payloads come back as "Error: ..." strings.
- backend/tests/test_unbrowse_tools.py: key resolution, render handling,
request shape, structuredContent vs text payloads, error paths.
- scripts/doctor.py, scripts/wizard/providers.py: register the key and the
web_fetch provider; backend/tests/test_doctor.py covers both doctor paths.
- config.example.yaml, .env.example, backend/docs/CONFIGURATION.md,
tools/AGENTS.md, EN/ZH harness/tools.mdx: document the provider.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* docs(changelog): add Unbrowse web_fetch provider entry
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(tools): make the search coercion guards catch OverflowError
The community search tools coerce max_results (and Sofya's content
limit) from YAML config or tool parameters through guards that catch
only TypeError and ValueError. `int(float('inf'))` raises OverflowError
instead, so a YAML `.inf` (or a JSON `Infinity`) value still crashes
the coercion - SearXNG's search even fails wholesale with
'cannot convert float infinity to integer'.
Catch OverflowError alongside them so an infinite value falls back to
the default like every other invalid input. NaN is already covered by
ValueError.
Signed-off-by: yetuge <2219677952@qq.com>
* test(tools): move the GroundRoute inf test to module scope
The regression test was appended inside TestWebSearchTool, ending the
class early and turning its last two methods into uncollected nested
functions. Move it after TestWebFetchTool so pytest collects all 20
tests again.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(scheduler): coerce serialized timestamps in ScheduledTaskRepository.update()
Editing an interval task's title or prompt returned 500. The edit dialog
always sends schedule_spec beside the changed field, and when the cadence
is unchanged the PATCH route keeps existing["next_run_at"] — a value the
repository had handed back serialized as an ISO string. update() assigned
it to the DateTime column untouched, so SQLite raised StatementError
("only accepts Python datetime") and Postgres a DataError; only a cadence
change worked, because that path computes a fresh datetime.
Coerce every serialized timestamp update() accepts through the existing
_coerce_datetime, sharing one _TIMESTAMP_KEYS definition with the row
serializer, as update_after_launch() already does (#4785). The router test
now drives the real SQLite repository for the same-cadence rename and the
cadence change; the in-memory stand-in the other router tests use never
saw the type mismatch.
* docs(changelog): reference #5964 in the scheduled-task PATCH entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(middleware): preserve message fields when PII redaction rebuilds a user message
PiiRedactionMiddleware._process_request rebuilt the redacted HumanMessage
with a hand-constructed constructor call copying only content, id, name and
additional_kwargs, silently dropping response_metadata and every other field
of the original. That is the third and last site of the family already fixed
for ThreadDataMiddleware (#5823) and InputSanitizationMiddleware (#5875);
the review on #5823 scoped the follow-up to the two input-sanitization sites
and left this one out.
The middleware was also inconsistent with itself: _redact_tool_message in the
same file already rebuilds with model_copy and carries the comment explaining
that a hand-built constructor call silently drops response_metadata.
The user message now rebuilds with msg.model_copy(update={...}). The content
is deep-copied because _redact_content appends untouched content blocks by
reference: without the copy, mutating a non-text block on the redacted
message would write through to the message retained in thread state.
Red/green: test_redacted_user_message_preserves_other_fields asserts
response_metadata/id/name/additional_kwargs survive the rewrite — red on main
(response_metadata == {}), green here. A second test guards the aliasing
behaviour above and goes red if the deep copy is dropped. Both suites 258
passed; ruff 0.15.12 check + format clean repo-wide.
Fixes#5967
Signed-off-by: yetuge <2219677952@qq.com>
* fix(middleware): keep the PII rewrite on copy failure and detach metadata
Addresses the two copy-boundary findings on the previous commit.
1. A content block may carry a value whose copy raises. deepcopy(content) then
raised out of _process_request, _try_process fell back to the original
request, and the handler received raw PII unredacted. Copying now falls back
to a shallow per-container copy, so a copy failure still produces a redacted
message and never bypasses redaction.
2. msg.model_copy(update=...) is shallow, so the rebuilt message and the
original shared one response_metadata dict: mutating metadata on the
model-facing message wrote through to the request message retained in thread
state. response_metadata is now copied independently.
Red/green: test_preserved_metadata_is_not_shared_with_the_original and
test_redaction_survives_a_content_block_that_cannot_be_copied both fail on the
reviewed commit (the second with the raw address reaching the handler) and pass
here. Suites: pii 77 passed, and pii + input_sanitization + uploads 327 passed.
ruff check + format --check clean repo-wide (1694 files).
Also drops the "third and last site" wording from the previous commit message's
claim: UploadsMiddleware is a further site of the same pattern (#5972).
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
dsn_with_search_path rebuilt the query with parse_qsl + urlencode. parse_qsl
decodes '+' as a space, an HTML-form rule libpq does not apply, so any existing
parameter carrying a literal plus came back as %20 and the server received a
different value: application_name=web+app became 'web app', and
options=-c%20timezone%3DUTC+8 became '-c timezone=UTC 8'. The function's own
docstring already states the %XX-only rule and that existing parameters are
preserved.
Rebuild the query from the raw pairs instead, passing every non-options
parameter through untouched and decoding options with unquote.
update_status guarded expected_lease_owner (and the protect_terminal
backfill) on a plain session.get. On SQLite a concurrent requeue plus
re-claim can commit between that read and this session's write, so the
stale guard passed and the late write clobbered the fresh claim —
clearing the new owner's lease fencing and overwriting its status. The
guard now re-reads under the parent's writer lock (the same task -> run
lock order as fail_launching_run / expire_queued_runs), with
populate_existing so the identity map does not serve the pre-lock row
back untouched. This is the run-row counterpart of the staleness #5777
fixed for the task-level lease release.
Red/green: tests/test_scheduled_task_dispatch_race.py gains a parking
session proxy that commits a requeue + re-claim by another scheduler
between update_status's read and its write — on main the late write
returned True and clobbered the claim; with the change it is fenced
(False) and the re-claim survives.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(models): deep-merge when_thinking_enabled on the legacy path
The legacy thinking path (profiles without a `reasoning:` block, i.e. every
existing config.yaml) applied `when_thinking_enabled` with a shallow
`dict.update`, so a template such as `extra_body.thinking.type=enabled`
replaced the profile's whole `extra_body` and silently dropped sibling keys
like GLM's `extra_body.tool_stream` — but only while thinking was on. The
disable branch of the same function already deep-merged into `extra_body`,
and the contract path added in #5780 deep-merges through `_merge_settings`
for exactly this reason (its docstring names `tool_stream`). The on/off
asymmetry cannot be intended: the same profile sent `tool_stream` with
thinking off and lost it with thinking on.
Route the enable template through `_merge_settings` as well. Precedence is
unchanged (template values win on conflicts); the only observable
difference is that keys present only in the profile now survive a
thinking-on request. Everything else on the legacy path — the synthesized
`reasoning_effort=minimal`, the disable payloads, the supports_thinking
guard — is untouched.
Tests pin both directions for a base extra_body beside a thinking template,
recursive merging of nested chat_template_kwargs, and template precedence.
* docs(changelog): reference #5894 in the when_thinking_enabled entries
* fix(models): keep the template's vLLM switch spelling authoritative when merging
Review follow-up. Deep-merging the thinking template into a profile keeps
both spellings of vLLM's toggle when they differ — base
`chat_template_kwargs.enable_thinking: false` plus a template using the
legacy `thinking: true` alias — and VllmChatModel's normalizer maps the
alias onto `enable_thinking` only when the latter is absent, so the
profile's `false` won and thinking stayed off despite thinking_enabled=True.
The previous shallow replacement never hit this because it discarded the
base switch. The contract path had the same trap since it already
deep-merged, and both disable branches had the mirror image (a base
`enable_thinking: true` outlived a template `thinking: false`).
Route every thinking on/off merge through _merge_thinking_payload: it drops
the profile's *other* spelling of the switch before the deep merge, so the
payload's spelling decides in both directions on both paths, and unrelated
chat_template_kwargs still survive.
Tests cover both spelling combinations, both directions, and both paths,
asserting the switch as the provider will send it after normalization.
* fix(models): deep-merge the legacy disable template and mirror the vLLM switch instead of pruning
Review of the previous head found two regressions. The legacy
when_thinking_disabled branch was still dict.update-ed, so a profile that
declares both templates (the shape of most extra_body examples) kept
tool_stream while thinking was on and lost it while off. And the merge
pruned the profile's other vLLM switch spelling for every model class,
although only VllmChatModel maps `thinking` onto `enable_thinking`: on a
plain ChatOpenAI profile the prune removed the only key vLLM reads, so
thinking-off could no longer switch reasoning off.
Route when_thinking_disabled through _merge_thinking_payload too, and
replace the prune with mirroring: after the deep merge, the value the
payload declares for its switch spelling is copied onto the profile's
other spelling when the profile carries it. VllmChatModel collapses the
pair, plain OpenAI-compatible classes send both, and either way the
server reads the template's intent. No spelling is invented, a payload
that spells both is left alone, and non-mapping template values are
forwarded unchanged instead of raising. _VLLM_THINKING_SWITCHES is now
the single definition used by the disable, toggle and dialect helpers.
Docs state the merge semantics (keys are never removed; a template can
only add or override) and drop the "byte-for-byte" wording; tests cover
the disable template on/off, nested inheritance, non-mapping values, the
raw payload for a non-normalizing class, and the no-invention rule.
* fix(models): copy thinking templates on merge and document the never-remove rule
Review follow-up. Templates merged into the constructor settings were
stored by reference wherever the profile lacked the key, so the kwargs
handed to the model class aliased the cached ModelConfig; the mirroring
step now writes into that mapping, so make _merge_settings and
_deep_merge_dicts deep-copy every template mapping they store. Pinned at
every depth by a test that also checks the profile is unchanged.
Deep-merging the legacy when_thinking_disabled template means it can no
longer clear a key the profile's base extra_body sets: a base
extra_body.thinking {type: enabled, budget_tokens} reaches the provider
as {type: disabled, budget_tokens}, which Anthropic-style APIs reject.
The synthesized disable payloads and the contract path already behaved
this way. State the rule and the migration (enable-only keys belong in
when_thinking_enabled) in the changelog, models/AGENTS.md and
configuration.mdx, and pin it with a test for both shapes.
Route the legacy synthesized OpenAI disable through
_merge_thinking_payload too (identical output), narrow the docstring and
guidance wording about the native-Anthropic disable, and make the two
vLLM switch suites resolve the real VllmChatModel / ChatOpenAI classes
so a class-gated merge would fail the case it is named after.
* feat(memory): add Jev pre-screening and signal classification over a shared TypeSafe client
- refactor(typesafe): extract a shared Jev (TypeSafe) client and unify its
configuration: one connection-precedence rule, credential fingerprint, wire-size
counting and error taxonomy, shared by the guardrail gate and the memory hooks.
- feat(memory): wire `memory.prescreen` and `memory.signal_classification` into
DeerMem through one injected judge hook (both off by default, `enforce` gated by
the shadow evaluation).
- config: add the top-level `typesafe:` block and the two judging slots to
`config.example.yaml` and the Helm values, bump `config_version` to 50, and sync
the Helm README example with it.
- test(memory): add the shadow-evaluation script (`eval_memory_prescreen.py`) with
per-stratum gates, censoring and the no-network baseline.
Review follow-ups folded in:
- `mutations_accepted` counts only facts that survive capacity enforcement and the
upload-event scrub that precedes persistence, only removals/reinforcements of ids
that existed, and persisted near-duplicate merges (matched by id, or by identity
for legacy id-less facts); it is published only after persistence succeeds.
- The cached manager hot-reloads its judge when a memory slot or the shared
`typesafe` block changes (including a credential rotation); publication is
serialized and revalidates the config generation, so a build started before an
`enforce` -> `off` rollback cannot land after it.
- Cache provenance is per answer: the `cached` flag and the reported model follow
the answers a side actually consumed, across partial merges and answerless retries.
- An oversized noul integer stays a per-question error instead of raising.
- Incremental call savings use the shared scored denominator.
- The sensitive-stratum gate requires evidence for each of identity, preference
and correction.
* fix(typesafe): resolve one credential per layer and refuse unusable connection values
Review follow-ups on the shared TypeSafe client (#5906):
- A consumer's `api_key_env` is no longer overridden by a literal `api_key` in the
shared `typesafe:` block: the first layer that sets either credential setting
decides it, and inside one layer the literal key wins (the standalone
semantics). A consumer whose endpoint does not accept the block's key no longer
sends it on every call, and the missing-credential message keeps naming the
variable that layer configured.
- `base_url` is validated where it is configured: `http(s)` only, with no query,
fragment, embedded credentials, whitespace or control characters, and a
malformed port fails at construction instead of on every request (httpx raises
`UnsupportedProtocol` / `InvalidURL` per call, and the query would have been
swallowed into the appended endpoint path).
- A credential that cannot travel as a header value — surrounding whitespace from
a secret mounted from a file, a non-printable character — is refused when the
connection resolves. It used to construct fine and then put the whole
`Bearer <key>` value into h11's `LocalProtocolError` message, which the
guardrail middleware's `logger.exception` prints.
- The client raises transport failures without chaining the original exception,
so that message can no longer carry the credential through the cause chain; the
exception type stays in the message for diagnosis.
- `TypeSafeConfig` validates in strict mode, agreeing with the consumer `config`
path: `max_attempts: true` was silently read as `1`. An integer stays a usable
float, so `timeout: 5` keeps working.
Tests: 20 cases across `tests/test_typesafe_config.py` and
`tests/test_typesafe_client.py`, each red on the previous revision and green now.
367 focused typesafe/memory/config tests, `make test-blocking-io` (167) and the
eval-script suites (50) pass; `ruff check` and `ruff format --check` are clean.
Docs synced: `backend/docs/GUARDRAILS.md`, `typesafe/AGENTS.md`,
`config.example.yaml`.
* fix(typesafe,memory): bound the reply, record request failures, gate enforce on the evidence
Second review round on #5906: the remaining items from the review thread and from
the follow-up pass.
Typesafe client (the reply is treated as hostile input too):
- Send `Accept-Encoding: identity`, refuse a body that carries a non-identity
`Content-Encoding`, and read through a bounded `iter_bytes` that stops past
`MAX_RESPONSE_BYTES` (64 KiB). A gzip body that expands to 128 MiB used to be
decoded in full and, on the gate path, allowed the tool call unexamined; it is
now `invalid_response` (a denial with `fail_closed: true`).
- Map `RecursionError` from `json.loads` to `invalid_response`: deep nesting
escaped the module's taxonomy, so the memory consumers lost the round's record
instead of recording a fallback.
Memory judge:
- A request-level failure is recorded as `request_failed`, never `no_verdict`
(`REASON_REQUEST_FAILED` was defined and never emitted, so an unreachable
endpoint was indistinguishable from a provider with no usable answer). The
adapters' `decide()` now propagates `TypeSafeError` for the coordinator to record
and log; `no_verdict` stays for a question-level failure, and an ineligible side
keeps its own reason even when the round's request failed.
Shadow evaluation (the enablement gate reads the evidence, not one number):
- The miss-rate gate requires the exact one-sided Clopper-Pearson upper bound to
meet the 1% target; a sample that only meets the point estimate is
`INSUFFICIENT` (0/250 was reported as a pass with a 1.19% bound).
- New `baseline_incremental_savings` gate: a baseline that saves as many calls as
the model — the shipped all-trivial sample — fails instead of enabling.
- Negative counts (`mutations_accepted`, message / batch / token counts) are
treated as absent rather than scored: `-3` read as "nothing lost", adding a
phantom saving to the numerator and the denominator.
Docs: the design documents cited but absent from the repository are replaced with
in-repo pointers (`MEMORY_IMPROVEMENTS.md`, `agents/memory/AGENTS.md`,
`GUARDRAILS.md`), and the gate and limit wording now matches what the code
enforces.
Tests: 31 new or updated cases, each red on the previous revision (compressed body
refused even when it decodes to a valid envelope, body over the cap refused, body
at the cap answered, deep nesting mapped, `request_failed` on each request-level
failure, the bound and baseline gates, the negative counter). 454 focused tests,
`make test-blocking-io` (167) and the eval suites pass; `ruff check` and
`ruff format --check` are clean.
* fix(memory-eval): pair the baseline comparison on the same records
Review follow-up on the incremental-savings gate added in ed930478 (P2): both
sides of the comparison are now the records the no-network baseline can score
(`trivial_only` plus a readable signals list).
Subtracting that subset's heuristic savings from model savings over *all* scorable
skips credited the model with batches the baseline never saw: 300 reviewed skips
with a single annotated record computed 300-1=299 "incremental", passed the gate
and set `enable_recommended`, while the paired comparison was model=1 against
heuristic=1 (incremental 0).
- `Evidence` carries the paired numbers (`baseline_paired_skips`,
`baseline_model_saved_calls`, `baseline_heuristic_saved_calls`) instead of one
heuristic count, filled by `build_report` from the baseline summary.
- The gate reports `INSUFFICIENT` when no scored skip carries the annotation and
when the annotation covers only a subset of the scored skips (the message carries
the paired counts and the `paired of scored` coverage), `FAIL` on a non-positive
paired gain, and `PASS` only on a positive gain over the full paired set.
- Docs state the pairing/coverage requirement in
`backend/docs/MEMORY_IMPROVEMENTS.md` and in the script docstring.
Tests: the reviewer's counterexample as a report-level regression (it failed with
`assert 'PASS' == 'INSUFFICIENT'` on ed930478 while `enable_recommended` was true),
a unit case for the paired-subset `INSUFFICIENT`, and the renamed paired fields.
344 focused tests, `make test-blocking-io` (167) and the eval suites pass;
`ruff check` / `ruff format --check` clean.
* fix(typesafe): bound the sync body read by the deadline
`deadline_seconds` only covered the sync path's headers, the completed body and
the parse, so an endpoint that drips bytes just under the read timeout held the
caller for one read per piece — the whole of `MAX_RESPONSE_BYTES` past the budget.
The read loop now checks the budget before every transport read, which bounds the
overshoot to one read. The pieces are the transport's own reads, not re-chunked
output: `iter_bytes(chunk_size)` pulls many reads inside a single yielded chunk,
so a check at that granularity would never see a drip.
`tests/test_typesafe_client.py` pins it with a valid envelope delivered as 24
pieces of 50 ms against a 0.2 s deadline: red before (24 pieces pulled), green now
(~5), on both the sync and async paths. Docstring and `typesafe/AGENTS.md` track
the new granularity.
* fix(memory): read the YAML off spelling, and veto a skip over the whole batch
Two review findings on the memory judging path.
`memory.prescreen.mode` / `memory.signal_classification.mode` rejected the
`mode: off` spelling `config.example.yaml` ships and a rollback types: YAML 1.1
parses the unquoted value as a boolean, so both fields saw `False`. The failed
reload left the previous judge running, so both fields now normalize `False` to
`off` before the `Literal` runs. `True` stays a validation error (no mode it
could mean). The examples quote the value, and the docs record why.
The pre-screen's L3 veto read `detect_signals`' 6-message window while the batch
it answers for is the whole post-watermark feed, so an explicit correction in an
earlier turn of an eight-message batch did not stop the skip that then advanced
the watermark past it. `detect_signals` gains a `window` parameter (default
unchanged) and the updater scans the full feed for the veto; the extraction hint
and the reinforcement gate keep the window.
Both regressions are red on the previous head: the config test reproduced the
`input_value=False` rejection through `AppConfig.from_file`, and the batch test
reproduced the prescreen judging (and skipping) a batch whose first turn carries
the correction.
* fix(models): skip credential files whose access token is not a string
* fix(models): normalize the file-route access token and guard the refresh token
* fix(goal): show the active goal to the agent on every model call
Nothing put ThreadState.goal into model requests. The agent only knew the
goal from messages: the objective the Web UI and IM channels send, and the
hidden <goal_continuation> message. Goals set through PUT /goal, the
embedded client or the TUI never reached it, and compaction could remove
the messages that stated the others.
DurableContextMiddleware now renders the active objective first in its
hidden durable-context data message, escaped, bounded, PII-redacted and
treated as data like the summary. Only the objective is rendered, so the
block changes only when the goal is set or cleared.
* fix(goal): let the agent pursue the active goal at user priority
The durable-context contract told the model never to follow instructions in
its data message, and that message now carried the goal. While a goal is
rendered, the contract ends with a static exception for the <active_goal>
element that opens the data message: work toward it as a request in a user
message, with no system or developer authority, while every other field
value, and any other text that calls itself a goal, stays data.
The element replaces the heading the goal used, which a summary could
reproduce. Other field values in the data message are HTML-escaped, and
active_goal joins the sanitizer denylist, so user input and remote tool
results cannot carry a raw one either.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
