* feat(community): migrate infoquest client to httpx.AsyncClient
Replace blocking sync requests.post calls with httpx.AsyncClient in the
InfoQuest community tool, as tracked in backend/docs/TODO.md planned
features. fetch/web_search/image_search and their raw-results helpers are
now async, keeping the 30s transport timeout and external behavior
(return values, error strings) unchanged.
The langchain tool wrappers in infoquest/tools.py become async functions;
web_fetch offloads readability extraction to a thread (same pattern as
the already-migrated jina_ai tool). Tests are updated to mock
httpx.AsyncClient instead of requests.post.
* test(community): use AsyncMock for infoquest fetch in relative-links test
The infoquest web_fetch_tool now awaits client.fetch, so the test stub
must return an awaitable instead of a sync lambda.
* fix(community): follow InfoQuest API redirects like the sync client did
Address the review note on #5782: requests.post follows redirects by
default while httpx.AsyncClient defaults to follow_redirects=False, so
a 3xx answer from the reader/search endpoints would have surfaced as
'Error: fetch API returned status 30x' (or fallen into the generic
error path after raise_for_status) instead of the final content.
All three AsyncClient call sites now pass follow_redirects=True. Adds
regression tests asserting each client is constructed with
follow_redirects=True, and documents the parity in the test contract.
* test(community): fix InfoQuest mock endpoint typo
* fix(release): refresh backend/uv.lock when bumping the version
`scripts/bump_version.sh` rewrote backend/pyproject.toml,
frontend/package.json and the Helm chart's `version`/`appVersion`, but
`backend/uv.lock` also records the root package version (uv keeps its PEP 440
form, so `2.1.0-rc0` is stored as `2.1.0rc0`). The documented release step left
the lock pinning the previous version, and that is not cosmetic drift:
`uv lock --check` (lint CI) fails on a stale lock and `uv sync --locked`
(e2e / plugin workflows) refuses the tree, so the release commit could not pass
CI. RELEASING.md only mentioned the manual `cd backend && uv lock` step for
release candidates.
The script now refreshes the lock via `uv lock` and refuses to start without
`uv` — it exits before editing anything rather than leaving a half-bumped tree
whose lock CI rejects. `uv lock` stays the authority for the normalized
prerelease form; the lock diff for a bump is a single line. RELEASING.md and
AGENTS.md now describe the lock as part of the bump.
Tests: backend/tests/test_bump_version_lockfile_sync.py (red on main, green
here) pins both halves — the lock moves with the sources, and a missing uv is an
error instead of a silent no-op.
* docs(changelog): record the release bump lockfile fix (#5859)
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(memory): redact queued memory payloads (pii_redaction follow-up)
Closes the remaining vector from #3190 (memory storage): request-scoped
redaction leaves originals in thread state, and MemoryMiddleware queued
that raw conversation into the durable extraction buffer — so the
extraction model's input and the persisted facts carried raw PII even
with pii_redaction.enabled.
- MemoryMiddleware gains pii_redaction_config (same pattern as
summarization / durable-context / title); the payload returned by
_resolve_add_args() is redacted at the enqueue boundary before
add()/aadd(), covering both after_agent and aafter_agent
- Reuses the shared redact_text() seam from #5527 via
active_pii_detectors/_Redactor/_redact_content; the buffered payload
itself becomes clean, not just the extraction model's view
- Wired at the lead-agent assembly site
- Tests: queue-payload redaction enabled/disabled, detector toggles,
message immutability, token stability across turns
* fix(memory): reserve existing placeholder indices across the queued batch
Review round 1 on #5577: thread state mixes raw user turns with
already-redacted tool results and summaries, so a token in a later
message must not collide with a new value in an earlier one. Pre-scan
the whole queued batch with redactor.reserve() before allocating.
Also extend the two MemoryMiddleware stubs in
test_lead_agent_model_resolution.py to accept the new
pii_redaction_config kwarg (CI backend-unit-tests red).
* fix(memory): redact the compaction-triggered memory_flush_hook queue too
Review round 2 on #5577: with memory + summarization enabled,
memory_flush_hook() queues list(event.messages_to_summarize) directly
into MemoryManager.add_nowait(), bypassing MemoryMiddleware entirely —
and compaction removes those messages from state right after, so the
after-agent redaction can never repair a raw batch queued there.
Extract the batch redaction (reserve pre-scan + rebuild) into a shared
redact_queued_messages() used by both MemoryMiddleware's enqueue
boundary and the hook; the hook gains an optional pii_redaction_config
wired via functools.partial at the summarization factory (subagents use
skip_memory_flush=True and are unaffected). Regression covers the hook
path enabled/disabled/absent.
* test(summarization): unwrap the partial in the factory hook identity check
edd916e wrapped memory_flush_hook in functools.partial to carry the
pii_redaction config, but missed committing the matching update to
test_factory_attaches_memory_flush_hook_by_default — a functools.partial
is not equal to its wrapped function, so the identity assertion failed
deterministically (caught by static review on #5577).
* fix(memory): cover tool-call arguments and lenient text blocks in queued payloads
Review round 4 on #5577 — two reproduced gaps still allowed raw PII into
memory payloads with pii_redaction.enabled=true:
- OpenViking retains the full message object, so parsed tool_calls args
and provider-format arguments in additional_kwargs bypassed the
content-only redaction; queued payloads now redact tool-call args
recursively (and reserve them in the pre-scan).
- DeerMem's format_conversation_for_update reads p.get("text") of any
dict block, not only type=="text" — _redact_content now redacts every
string-valued text field so a lenient consumer cannot bypass it.
Both repro shapes are pinned as regressions.
* fix(pii): value-derived placeholders for stable capture identity
Review round 5 on #5577: sequential allocation is order-dependent — the
same unchanged message re-redacted in a later batch can get a different
token, which flips OpenViking's content-signature and replays already
recorded turns into persistent memory sessions. Placeholders are now a
pure function of the matched value (sha256-6 per category): identities
stay stable across batches, enqueues, seams, and backends with no shared
state and no stored mapping.
- _Redactor becomes stateless; the reserve machinery (index pre-scans,
_PLACEHOLDER_PATTERN) is removed along with its call sites
- redact_texts keeps its signature, now a simple per-text redaction
- reserve-semantics tests are superseded by cross-enqueue identity tests
(two successive captures where the second batch carries higher existing
placeholders); the compiled-graph echo regex matches the new format
* fix(pii): 128-bit placeholder digests; redact tool-argument keys
Review round 6 on #5577:
- [P2] The 24-bit truncation collided distinct identities (reproduced:
contact3513@ / contact3727@example.com both rendered [EMAIL_575dbb]).
Retain 128 bits of the digest so separate people stay separate at any
realistic volume; the compiled-graph echo regex and the AGENTS.md entry
follow the new format.
- [P2] _redact_strings redacted values but preserved mapping keys, so
{"contacts": {"alice@example.com": "manager"}} kept the raw email
as a key. Redact string keys along with values; deterministic 128-bit
tokens make distinct-key collisions a hash-collision non-event, and a
redacted key equal to another key merges the entries. Regression pins
the email-keyed arguments shape.
* docs(agents): trim PiiRedactionMiddleware entry; drop stale reserve wording
The chain grew past the 98304-byte AG002 hard limit again (main growth
plus the round-6 format example). Trim the entry to fit, and replace the
stale reserve-indices wording with the value-derived semantics that
replaced it.
* docs(agents): shrink PiiRedactionMiddleware entry below the chain budget
main moved (#5609) and the base comparison flipped the nearly-over chain
back to an AG002 error (98441 > 98304). Trim the entry to the facts that
are not documented elsewhere: seams, allowlist, order, enqueue boundary.
* fix(pii): letters-only value-derived tokens; wire the factory SDK path
Review round 7 on #5577:
- [P2] Hex digests carry digit runs that later digit-anchored detectors
re-scan, corrupting minted tokens into nested placeholders (reproduced:
user280@example.com -> [EMAIL_…[PHONE_…]…], ~1/140 emails). The 128-bit
digest is now encoded over a-z, so a minted token survives the full
pinned detector order.
- [P2] create_deerflow_agent (the SDK path) constructed MemoryMiddleware
and DurableContextMiddleware without pii_redaction_config; the knob is
now an explicit optional parameter threaded through
_assemble_from_features (the factory reads no config files by design,
so SDK callers pass it explicitly).
- [Nit] hoist the Command rebuild comprehension out of the per-message
loop in _redact_result.
- [Nit] drop the dead _iter_tool_calls helper and Iterator import.
* test(factory): pin pii_redaction_config wiring; document the parameter
Review round 8 on #5577:
- [P3] test_create_deerflow_agent.py had zero coverage of the
pii_redaction_config parameter; two regressions now assert that
create_deerflow_agent(pii_redaction_config=cfg) builds a
MemoryMiddleware and a DurableContextMiddleware carrying cfg, and that
the default is None (redaction off).
- [Nit] document the parameter in the create_deerflow_agent docstring:
None leaves redaction off; a middleware= takeover bypasses the
assembly that would apply it.
* fix(pii): HMAC-scoped placeholder tokens; install model-bound redaction in the SDK chain
Review round 10 on #5577:
- [P2] sha256(category||value) is an unkeyed, globally deterministic
fingerprint: a provider or memory backend can confirm candidate values
offline and link identical values across users and threads, contradicting
the documented no-re-identification guarantee. Placeholders are now
HMAC-derived over a deployment-scoped token_secret (new config field);
the docstring documents the linkability scope for both modes.
- [P2] _assemble_from_features() never appended PiiRedactionMiddleware, so
the SDK knob enabled only the downstream seams while the primary model
still received raw content. The middleware is now appended in the same
relative position as the lead-agent base chain when the gate is on,
with a model-call regression through the assembled SDK chain.
* docs(pii): document token_secret; update the placeholder format examples
Follow-up to the round-10 review on #5577:
- config.example.yaml documents the new token_secret knob: a
deployment-scoped secret makes placeholders HMAC-derived and only
linkable within one deployment; leaving it unset keeps publicly
computable fingerprints (documented trade).
- the PiiRedactionConfig class docstring's placeholder example now
matches the shipped format (128-bit HMAC digests over base-26
letters).
* docs(pii): update the module docstring's placeholder format example
The module docstring still showed the sequential [EMAIL_1] example;
align it with the shipped value-derived format (128-bit HMAC digests
over base-26 letters).
* fix(pii): require a non-empty token_secret when redaction is enabled
Review round 11 on #5577: with the secret optional, the default enabled
configuration minted empty-key HMAC digests — publicly computable, globally
linkable fingerprints of the raw values, confirmable by offline guessing for
low-entropy identifiers such as phone numbers.
- PiiRedactionConfig now rejects enabled=true without a non-empty
token_secret (minimum 16 characters); the unkeyed mode is gone entirely.
- The request-scoped redactor in _process_request now carries the configured
key: it built its redactor without it, so user-message placeholders
silently diverged from every other seam for the same value. The empty-key
guard in _placeholder_token is the backstop that keeps every path keyed.
- Module/config docstrings and config.example.yaml no longer describe the
placeholders as irreversible or the secret as optional; the stale
reserve-semantics wording and the "memory path not covered" note are
corrected.
- Regressions: validation errors for missing/blank/short secrets, and the
user-message boundary minting the same keyed token as redact_text.
* fix(memory): redact original_user_content on queued memory payloads
Review round 12 on #5577: UploadsMiddleware preserves the raw user turn in
additional_kwargs.original_user_content, but redact_queued_messages rewrote
only content, tool_calls, and provider-format tool_calls — the queued copy
still carried the raw value, so the memory queue and backend received raw PII
with redaction enabled.
The shared helper now redacts the provenance string field on the queued copy
(same batch redactor, merged with the existing additional_kwargs rewrite,
rebuilt via model_copy — thread-state messages stay untouched), covering both
the enqueue boundary and the compaction flush hook. Regressions pin the
redacted provenance field through both paths and the untouched original.
* fix(memory): redact human_input_response on queued memory payloads
Review round 13 on #5577: clarification replies keep the raw user answer in
additional_kwargs.human_input_response.value (DeerMem reads that mapping via
its read_human_input_response mirror), but redact_queued_messages rewrote
only content, tool_calls, and original_user_content — the queued copy still
carried the raw value into the memory queue and backend.
The shared helper now rebuilds that mapping with value redacted (same batch
redactor) while preserving the protocol fields (version/kind/source/
request_id/option_id/response_kind), merged with the existing
additional_kwargs rewrite and rebuilt via model_copy so thread-state
messages stay untouched. Regressions pin the redacted value, preserved
protocol fields, the untouched original, and the flush-hook path.
* chore(config): bump config_version to 47; generate token_secret on upgrade
Review round 13 on #5577: existing deployments can legitimately sit at
config_version 46 with pii_redaction.enabled: true and no token_secret; the
mandatory-secret validator would fail their startup while config.example.yaml
still said 46, so the upgrade checker considered them current and emitted no
migration warning.
- Bump config_version to 47 so the outdated-config warning fires for v46.
- Add a v47 migration to scripts/config-upgrade.sh: when pii_redaction is
enabled and token_secret is missing/blank, generate a random deployment-
scoped secret (secrets.token_urlsafe) and persist it, so the upgraded
deployment starts under the mandatory validation.
- Regressions: a v46 enabled config is flagged outdated against the real
example; the upgrade run generates a >= 16-char secret and the result
passes AppConfig validation; a disabled config gets no secret.
- Also switch the two v41 script tests from a hardcoded "bash" to
SCRIPT_BASH/skipif, matching the documented Windows constraint and the
neighboring tests (hardcoded bash resolves to the WSL stub there).
* chore(helm): sync chart config_version to 47
Review round 14 on #5577: check_config_version.sh fails whenever the chart's
embedded config_version lags config.example.yaml, and the chart workflow is
triggered by config.example.yaml — so the v47 bump left this head failing
that check. Update the values.yaml config block and the README example to
47; scripts/check_config_version.sh now passes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Follow-up to #5833 after an independent review of the merged code.
- A malformed answer (deeply nested JSON raising RecursionError, or a
huge noul integer raising OverflowError outside the try) escaped the
fail-open path, and in the per-message task group it cancelled the
other messages' requests and dropped every flag of that tool call.
Parsing now happens inside the try, which also catches those errors.
- The 16 KiB response cap counted decoded bytes, so a compressed body
could expand far past it first. Requests now ask for identity encoding
and drop encoded responses.
- A plain-HTTP loopback endpoint followed HTTP_PROXY/ALL_PROXY, sending
the bearer key and excerpt to the proxy in cleartext. Such endpoints
now ignore proxy environment variables; HTTPS endpoints keep them.
Tests cover each case, including a real local server behind an
unreachable proxy, and pin previously untested behavior: independent
per-request deadlines, the sync hook without a task store, the >=
threshold and all four named remote tools, warnings for messages that
share a tool-call ID, consumed flags, and scanning past host messages.
The root README no longer says the excerpt is always PII-redacted, and
the example README documents the new limits and that the sync path's
deadline does not cover a stalled DNS lookup.
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* chore(version):update the version to 2.2.0-dev
* chore(doc):Updated the CHANGLOG with the 2.1.0 release version
* chore(doc): Updated the CHANGLOG for the latest changes
* chore(doc): Added the release SOP document
* feat(extensions): grant bounded host model invocation
* fix(extensions): retain model budgets and isolate schema validation
* fix(extensions): normalize provider-origin model cancellation
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): strip section anchors from code-span resource refs
Markdown-link targets already drop their #fragment before the
reference is resolved, but code-span refs did not: a prose reference
like `references/faq.md#pricing` kept the anchor and was resolved as
a literal path, producing a resource.missing finding for a file that
exists and is referenced. The bare-path token scanner stops at '#',
so only the code-span branch leaked fragments.
Split the fragment off code-span tokens the same way the link branch
does, then strip trailing sentence punctuation. Real dotted filenames
keep their dots: the split happens at the first '#', and '.' is only
stripped from the very end.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): prefer exact hash-bearing filenames over fragment stripping
Package paths permit '#', so a code-span reference to a real file like
references/C#.md was truncated to references/C by the fragment split,
producing both a false resource.missing and an orphan report (the bare
path token stopped at '#' the same way). Keep the whole reference in
every extraction branch (link target, code span, bare path token now
includes '#' in its character class) and resolve the fragment
existence-aware: prefer the exact token when it names a real file and
treat the suffix after the first '#' as a section fragment only when
it does not.
Red/green: a new test pins references/C#.md resolving to the real file
while references/faq.md#pricing still strips to faq.md — it fails on
the reviewed head and passes with the change; full
test_skill_review_core.py is 36 passed.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): strip the fragment before normalizing the fallback path
The exact-path preference checked the normalized reference, so a token
like references/faq.md#/../other.md normalized to references/other.md;
when that file existed the check succeeded and the fragment-removal
branch was skipped, recording the wrong edge and orphaning faq.md.
Check the literal token against the package files instead, and drop
the fragment before the remaining path is normalized — normalizing a
hash-bearing segment must never retarget an edge.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): canonicalize leading segments in the hash-filename lookup
The literal-file check compared the verbatim joined path, whose leading
relative segments are uncanonicalized: from references/sub/guide.md, a
reference to ../C#.md joins to references/sub/../C#.md, matches no
snapshot key, and falls through to fragment stripping — truncating the
real references/C#.md into a false missing and an orphan. Compare
against the normpath-canonicalized whole token instead, and treat the
post-'#' text as a fragment whenever it is path-like (carries / or ..)
so it can still never participate in normalization and retarget an
edge.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): check the canonical whole path before stripping a fragment
The path-like-fragment heuristic dropped any post-'#' text carrying '/':
a real package file such as references/C#/readme.md (a '#' inside a
directory name) was truncated to references/C, producing a false
resource.missing and orphaning readme.md. Only a '..' segment in the
post-'#' text forces the strip-first fallback — that is the one case
where fragment text participates in normalization and can retarget an
edge (faq.md#/../other.md). Every other fragment first gets the
canonicalized whole-token lookup, which already canonicalizes leading
relative segments for nested sources.
Signed-off-by: yetuge <2219677952@qq.com>
* fix(skills): always strip markdown link fragments and blind the bare-path pass to links
Markdown link targets have fixed fragment semantics: the text after '#'
is a URL fragment, never part of the filename, so a link to a file
literally named references/faq.md#pricing would have to percent-encode
it. The exact-file branch applied the hash-filename preference to link
targets too, recording a valid edge and suppressing resource.missing
for a broken link when a literal same-named file existed. Links strip
the fragment again, and their full construct is blanked out of the
residual text so the code-span and bare-path passes — literal-path
contexts where '#' may be part of a real filename — never see
link-internal text.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(browserless): read reject_resource_types / reject_request_pattern from config
web_fetch_tool declared both parameters, initialised them to None, and passed
them to the client -- but never read them from tool config, so they were always
None on the wire. BrowserlessClient implements them (and is tested for it at
`test_fetch_html_with_reject_params`), so the provider's resource blocking was
unreachable without patching the source. Skipping images/media/fonts is the
main lever for render latency and bandwidth against a self-hosted Browserless,
and it could not be turned on from config.yaml.
Read both keys through a new `_as_str_list` helper that accepts a YAML list or
a comma-separated string and yields None for anything unusable, so an unparseable
value omits the parameter instead of reaching the server. Both keys are now
documented on the Browserless web_fetch entry in config.example.yaml.
Default behaviour is unchanged: an unset key is still None, so the payload is
identical to before for every existing config.
Adds four regression tests covering the list form, the comma-separated form,
the unset case, and unusable values.
* fix(browserless): validate reject list entries
---------
Co-authored-by: RXQ6 <RXQ6@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(config): sign the bytes the config cache parsed, not a second read
_load_and_cache_app_config parsed config.yaml through AppConfig.from_file and
then called get_config_signature, which opens and hashes the file again. A
write landing between those two reads left the cache holding the older
content under the newer content's signature. get_app_config() compares the
on-disk signature against the recorded one, so from then on it saw
"unchanged" and the edit only surfaced when yet another edit changed the
signature — silently stale until then, with no self-healing.
Read the file once through a new shared helper,
file_signature.read_config_with_signature, which returns the bytes together
with the (mtime, size, sha256) of exactly those bytes, and parse them through
AppConfig._from_yaml_text (from_file's body, split out; from_file itself is
unchanged). The recorded signature now always describes the parsed content:
a write that races the load can only produce a signature mismatch on the
next call, i.e. one extra reload. Signing before parsing but re-reading for
the parse would not be enough — an edit that lands between the two reads and
is then reverted would be served forever — so the tests pin that the parsed
bytes are the signed bytes.
The stale-metadata reload test now routes the loader's read through the same
stale-stat simulation it already applies to the probe, since the loader no
longer records the probe's result.
* docs(changelog): reference #5848 in the config cache entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(authz): add plugin resource targets, provenance, action and management enforcement
PR1 of the extension-tool-authorization series
(docs/plans/2026-09-24-extension-tool-authorization-spec.md): plugin
authorization plumbing, no new author-facing declaration.
- `authz/plugin_targets.py` is the single encoder/validator for composite
targets (`namespace/part`), with one validating composer behind three
kind-checked constructors; invalid input raises instead of degrading.
- `authz/plugin_authz.py` is the harness decision layer (`plugin_action`,
`plugin_management`, plus the batch filters the page slice will consume):
disabled is a no-op, an explicit deny raises, and provider errors follow
`fail_closed` like the sandbox gate.
- `POST /api/plugins/{ns}/actions/{name}` decides `plugin_action:invoke`
after the action resolves and before the body is streamed.
- `deerflow_extension_api.auth` publishes `require_plugin_management` /
`arequire_plugin_management` with two `app.state` resolver keys; the
Gateway installs both. Contract version 0.2.3 -> 0.2.4.
- `tools/tool_provenance.py` records host-written plugin provenance for
display (assembly descriptor, Layer-2 context later); no decision reads a
label.
- Plugin providers use a `(config signature, loop key)` cache so a
loop-affine provider is never handed to another loop or to a sync caller.
- RBAC gains `plugin_actions` / `plugin_management` aliases;
`config_version` 47 -> 48 across the three mirrors.
Tests: new plugin target/provenance/action/management suites plus a
blocking-IO anchor whose offloads were reverted to prove red->green.
* fix(authz): decide plugin resources from the request's config snapshot
Review round 1 on PR #5842 (willem-bd).
P1 — the decision layer no longer reloads the configuration. The six
`authz/plugin_authz.py` functions take a required `app_config` snapshot (the
same request-scoped config the Gateway used to resolve the provider), and
`resolve_/aresolve_plugin_authorization` return it as a third element, so
`enabled`/`fail_closed` can never come from a second read that drifted from
the one that produced the provider. A snapshot the caller could not read at
all is *unavailable*, not disabled: it fails closed as
`authz.config_unavailable`. The Gateway's `_plugin_app_config` now separates
an absent `config.yaml` (no gate, today's behavior) from a config that exists
but cannot be read or validated (deny) instead of swallowing both through
`safe_app_config`.
P2 — `_validated_decision` requires `type(decision.allow) is bool` on both
the sync `authorize` and async `aauthorize` paths, so a provider returning
`AuthzDecision(allow="false")` is a malformed verdict that follows
`fail_closed` instead of being read as an allow.
Both fixes are covered by regression tests that were verified red against the
pre-fix behavior (P1: 200 vs 403 and resolver True vs False; P2: 12 cases
across fail_closed/fail_open and sync/async), plus tests pinning that an
absent config and `authorization.enabled: false` keep today's behavior.
* fix(authz): treat a request-time config absence as unavailable
Review round 2 on PR #5842 (willem-bd).
P1 — `_plugin_app_config[_async]` no longer maps a missing `config.yaml` to
`None` ("no gate"). A Gateway cannot have started without a readable config
(`app/gateway/app.py` reads it through this same accessor in `lifespan`), so
absence *during a request* means the configuration this process was running on
became unavailable — a hot reload, an atomic replace, a ConfigMap remount. It is
unavailable, not disabled: the error propagates,
`resolve_/aresolve_plugin_authorization` raise
`_PluginAuthorizationUnavailable(fail_closed=True)`, the action dispatcher
returns 403 and the management resolvers answer `False`. The dead
`(None, None, None)` allow path is gone, the resolvers return a non-optional
`AppConfig` as their third element, and callers key "authorization disabled" on
`provider is None` alone. This matches the rule the decision layer already
documented (`authz.config_unavailable`) and the public docs.
P2 — `_deny_reason_code` is total: a `reasons` that is not iterable degrades the
code to `authz.denied` instead of raising `TypeError` outside the guarded
provider path, which turned a denial into an unhandled error. It is deliberately
not folded into `_validated_decision`: `allow` was a real `bool`, so the verdict
is a valid denial, and treating it as malformed would flip it to an allow under
`fail_closed: false`.
Both fixes are covered by regression tests verified red against the pre-fix
behavior (P1: resolver True vs False, async `PermissionError` not raised, route
200 vs 403; P2: `TypeError: 'NoneType' object is not iterable` at
`plugin_authz.py:113`). The action-route tests in
`tests/test_plugin_contributions.py` now install a readable
`authorization.enabled: false` config instead of relying on a config-less
process, and `docs/full-stack-plugins.md` plus the PR1 decision log record the
new rule.
* fix(authz): apply the running policy when the plugin config is unreadable
Follow-up to f03d2ea9 (review round 2 on PR #5842, willem-bd).
The first attempt read the P1 finding as "absence is never disabled" and failed
every config read closed unconditionally. That is too broad for this repo: a
config-less host is deliberately treated as having no policy to apply
(`_get_route_authorization_config` falls back to a disabled config,
`sandbox_authz.safe_app_config` documents the same no-op for "CI runners and
direct-call tests"), and the rule turned the config-less e2e plugin host
(`extension_test_fixtures.bookmark_plugin_gateway`) into a 403 — the three
`bookmark-plugin.spec.ts` cases lost the save action. Reproduced locally:
`POST /api/plugins/community.bookmarks/actions/save` against that fixture host
returned 403 before, 200 after this change.
The rule is now "has this process loaded a configuration?", not "was the last
read successful?":
- No loaded config (`peek_loaded_app_config()`, new read-only accessor on
`deerflow.config.app_config`) and no `config.yaml`: no policy to apply, so the
gate stays a no-op — unchanged for CI runners, direct-call tests and hosts
that mount only the plugins router.
- A loaded config whose file disappears or cannot be read: *unavailable*, never
*disabled*. The resolvers raise `_PluginAuthorizationUnavailable` carrying the
failure flag of the configuration this process runs on (authorization
disabled there -> no gate; enabled -> that policy's `fail_closed`, `True` by
default), so an enabled Gateway answers 403/False in the window instead of
letting protected plugin actions and management routes through.
- A config file that exists but cannot be read or validated right now: fail
closed when nothing is loaded (round 1's rule), otherwise the loaded policy
decides.
`peek_loaded_app_config()` is preferred over a Gateway-side startup snapshot: it
reflects the last successfully loaded config, so a hot-reloaded policy is what
applies during the window, and there is no lifecycle hook to keep in sync.
Evidence: the three no-config/lost-config branches in
`tests/test_plugin_management_guard.py` and
`tests/test_plugin_action_authorization.py`, round 1's unreadable-config tests
unchanged, `test_peek_loaded_app_config_survives_a_missing_file` in
`tests/test_app_config_reload.py`, and the fixture-host probe above. Docs
(`docs/full-stack-plugins.md`, `reference.mdx` en/zh) and the PR1 decision log
state the final rule; the log records the superseded variant.
* fix(authz): keep a denial whose reasons raise while being read
* fix(authz): intersect plugin batch filters with the candidates
The plugin_page/plugin_management batch helpers returned the provider's
`filter_resources` answer verbatim, so a custom provider could hand the
projections a target that was never asked about ("*", another namespace,
a phantom page). Every other filter call site re-projects from its own
candidate list; do the same here, state the intersection on the provider
contract, and pin it with a regression test that fails on the old
pass-through.
* docs(plugins): map a management denial to 403 in the route example
The contributed-route example called `arequire_plugin_management` and let
the documented `PermissionError` escape. The Gateway installs no
`PermissionError` handler, so a route copied from the example answered 500
on a normal policy denial instead of 403. Both example routes now translate
the denial, and the surrounding bullets say why the translation is the
route's job.
* docs(extensions): point the 0.2.4 contract row at the PR that ships it
Every other row in the version-history table links the pull request that
shipped the version; 0.2.4 linked the authorization tracking issue instead.
Both the English and Chinese tables now link this PR.
* feat(examples): add opt-in fetched-content screening extension
* fix(examples): screen sync and async results through supported middleware hooks
* fix(examples): preserve empty message IDs when applying screening warnings
* fix(examples): include screening policy in assembly identity
* feat(examples): load fetched-content screening through the extension entry
Package the screening example as a plugins: extension instead of an
operator-configured middleware. install() validates the private config
and, once enabled, contributes one TOOL_VISIBLE middleware for lead
agents and subagents. The package imports only deerflow_extension_api.
Extension tool wrappers are observational, so detection and the warning
use two supported contracts: the tool wrapper classifies the visible
result and records flagged tool-call IDs in the run's task store, and
before_model replaces the matching messages of the latest tool step by
ID. The excerpt sent to Jev is now the redacted, sanitized text the
model sees. Runs without a task store send nothing. Unexpected local
errors surface as isolation diagnostics; provider errors fail open.
Tests load the package through load_extensions() and the host isolation
wrapper and run real lead/subagent graphs with a task store bound as the
Gateway worker binds it. The README records what the example exercised
in the extension API and the gaps it found.
* fix(examples): classify each tool message of a screened result on its own
A Command result was classified as one concatenated excerpt and the flag
landed on its first text message, so a benign first result could take the
warning while an injected later result reached the model unmarked.
Each text message now gets its own bounded excerpt and Jev request, for at
most eight messages per tool call. The requests share one client and run
concurrently in a task group, each under its own deadline, so one slow or
failed request cannot hide another message's flag. Every flagged message's
tool-call ID is recorded. The message cap joins the declared policy, and the
README describes the per-message requests and data sharing.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(uploads): keep document-conversion stat and write-back off the event loop
convert_file_to_markdown offloads the conversion itself for files above
1 MB (documented in its docstring), but the two bookend filesystem calls
still ran on the event loop: the size stat before the threshold check and
the write of the converted markdown afterwards — the latter a potentially
multi-megabyte write in the per-upload ingestion path. Route both through
run_file_io (the dedicated file IO pool that also keeps the caller's
context), matching the off-the-loop fixes already applied to the sibling
upload/document paths.
Add a tests/blocking_io anchor: with the converter patched to a 2 MB
payload it fails on the previous head (Blockbuster catches the blocking
os.stat first; the broad except swallows it into a None return) and
passes with this change.
* fix(uploads): address review — drain conversion write-back, make run_file_io re-entrant
Review follow-up on the off-loop conversion stat/write PR:
1. The write-back now goes through await_drained(): a plain run_file_io
await cancels immediately while the worker keeps writing, so the
ingestion scope could remove its private directory over an in-flight
write and leave a large hidden partial file. The drain delivers the
cancellation only after the worker settled, and the handler removes
the partial output before re-raising.
2. run_file_io is now re-entrant: a nested offload issued from code
already running inside a file-io worker (project-document conversion
runs asyncio.run(convert_file_to_markdown(...)) inside a run_file_io
worker) executes inline instead of submitting to the same bounded
executor, which deadlocked with a single configured worker.
Both fixes carry deterministic regressions: the cancellation test pins
the drain ordering (cancel issued before the worker is released) and the
partial-file cleanup; the re-entrancy test pins inline execution by
thread identity. Both fail on the previous head. Full blocking_io +
project-document + conversion suites show an identical failure set to
the previous head (pre-existing Windows platform failures only).
* fix(uploads): keep cancellation authoritative when conversion cleanup fails
If the post-cancellation unlink of the staging output raised, the new
exception escaped the CancelledError handler, was swallowed by the
broad except as an ordinary conversion failure (None return), and left
both the caller's cancellation and the partial file behind. Catch and
log cleanup failures inside the handler and always re-raise the
captured cancellation; a second cancellation still passes through
untouched (CancelledError is not an Exception).
Red/green: tests/blocking_io/test_file_conversion_offloop.py gains a
regression where unlink fails while the write-back is in flight — on
the reviewed head the task settled on None (DID NOT RAISE); with the
change the CancelledError propagates and the failure is logged.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* docs(backend): drop references to APIs that do not exist
PATH_EXAMPLES.md's upload example imports THREAD_DATA_BASE_DIR from
thread_data_middleware; the module exists but exports no such name, so the
snippet fails with ImportError as written. Resolve the same directory through
the accessor the middleware itself uses, get_paths().sandbox_uploads_dir(),
which yields the layout the file's own path table documents.
CONFIGURATION.md's Docker sandbox block lists `auto_start: true`. SandboxConfig
declares use/port/container_prefix/mounts/network but not auto_start, and the
provider is constructed as cls(**kwargs) from the caller of
get_sandbox_provider() rather than from config extras, so the key is accepted by
extra="allow" and then silently dropped. Remove the line from the example.
* docs(backend): resolve the uploads example through the owner-scoped bucket
get_paths().sandbox_uploads_dir(thread_id) without a user_id selects the legacy
threads/<id>/ bucket, but Gateway uploads land in
users/<user_id>/threads/<id>/user-data/uploads/, so the example imported cleanly
yet still raised FileNotFoundError for a normally uploaded file. Use
deerflow.uploads.manager.get_uploads_dir(), which resolves the effective user
and validates the thread id.
* docs(backend): show the real uploads path layout in every PATH_EXAMPLES demo
The remaining sections still documented the pre-isolation
threads/<thread_id>/ bucket and called `path` a backend-relative string, while
the upload and list APIs return an absolute path under
users/<user_id>/threads/<thread_id>/user-data/uploads/. Reproduced with
deerflow.uploads.manager.get_uploads_dir() and
uploads._list_uploaded_files_for_thread() against DEER_FLOW_HOME: the listed
path is /<home>/users/default/threads/abc123/user-data/uploads/report.pdf.
* docs(backend): clarify runtime home precedence
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(config): honor $VAR substitution for request_admission integer fields
AppConfig.resolve_env_variables replaces every `$VAR` reference with the raw
environment string. RequestAdmissionConfig.requests_per_minute and
max_queue_size are declared `strict=True` so a bool or float cannot slip in,
but strict mode also refuses the string "60", so `requests_per_minute: $RPM`
failed the whole config load with "Input should be a valid integer" even when
RPM=60 — contradicting config.example.yaml's promise that environment
variables work for every field. They were the only strict fields in the
config schema; every other integer coerces in lax mode.
Wrap both fields in a BeforeValidator that converts a decimal integer literal
arriving as a string and leaves everything else untouched, so the strict check
behind it still rejects bools, floats, "60.0" and non-numeric strings.
Tests pin the accepted literals, the rejections that must survive, and the
end-to-end path from a config.yaml `$VAR` reference through AppConfig.from_file.
* docs(changelog): reference #5838 in the request_admission entries
* fix(mcp): preserve backslash paths during bare-filename rewriting
* test(mcp): cover the trailing backslash boundary in bare-filename rewrite
Review noted that treating `\` as a separator also suppresses a rewrite when a
single trailing backslash precedes a newline, which is more often a Markdown
hard break, and that neither the tests nor AGENTS.md covered that position.
Lock the behavior in through the production result converter and document the
tradeoff: a backslash is a legal filename character on POSIX, so forgoing that
one rewrite is preferable to risking a corrupted path reference. Runtime
behavior is unchanged.
---------
Co-authored-by: lakers-abing <236384465+lakers-abing@users.noreply.github.com>
* fix(mcp): run lazy MCP discovery once when initialization itself raises
get_cached_mcp_tools() wrapped the whole lazy-initialization call in the
`except RuntimeError` that was only meant to catch asyncio.get_event_loop()
raising when no loop is set. run_until_complete() and Future.result() re-raise
the coroutine's own exception, so a RuntimeError from discovery — e.g. the
deliberately re-raised McpTaskConfigurationError — took the same fallback and
ran asyncio.run(initialize_mcp_tools()) a second time, respawning every stdio
server and re-fetching OAuth tokens before giving up. Inside a running loop
that fallback fails with "asyncio.run() cannot be called from a running event
loop", so the logged traceback hid the real cause.
Resolve the loop first, in its own try/except, and drive initialization exactly
once: asyncio.run when no loop is set or the current loop is closed, a worker
thread when the loop is running, run_until_complete otherwise. Any failure is
logged once and returns [] as before.
Regression tests pin one discovery call and the real logged cause with and
without a running loop, and keep both fallbacks (no loop set, closed loop)
covered.
* test(mcp): surface worker-thread failures in the no-loop fallback test
Thread.join() does not propagate exceptions, so a regression of the
asyncio.run fallback failed this test with a bare KeyError: 'result' instead
of the real traceback. Capture sys.exc_info() in the worker and re-raise it
on the main thread before the assertions.
* fix(sandbox): clamp a negative start_line in provider read_file
LocalSandbox.read_file clamps the start line with max(start_line or 1, 1),
but e2b, OpenSandbox and BoxLite used start_line or 1, so a negative value
reached Python's negative-index slicing and read from the end of the file
instead of the first line. Clamp all three the same way, and pin the
boundary cases in their existing range tests: a past-EOF start still returns
an empty string, a negative start reads from line 1.
Surfaced by the review of #5616 (e2b has the same gap, consistency nit).
* fix(sandbox): clamp negative end_line across providers
Complete the range-bound consistency fix for e2b, OpenSandbox and BoxLite:
clamp a negative end_line to zero like LocalSandbox instead of letting
Python's negative-index slicing silently drop trailing lines. Add the
symmetric boundary assertion to each provider's existing range test.
Follows #5801 for the remaining numeric env reads that break on an empty
value (`-e VAR=`, or `${VAR}` with VAR unset):
- The WeChat QR worker guard read a blank GATEWAY_WORKERS as an invalid
count and refused QR login, although the compose command starts one
worker for it (`--workers ${GATEWAY_WORKERS:-1}`) and the other worker
checks treat it as 1. A blank value now falls through to WEB_CONCURRENCY
and then 1, so the guard still fails closed for real multi-worker setups.
- The Lark broker aborted startup on int("") for a blank
DEERFLOW_LARK_BROKER_PORT or DEERFLOW_LARK_BROKER_TIMEOUT; it now uses the
defaults.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(prompts): add operator overlays for core system prompts
* test(prompts): isolate SOUL forwarding test from operator config
* fix(prompts): wrap complete subagent system message with overlay
* fix(subagents): serialize prompt overlays in durable batches
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(exa): coerce configured result limits before calling the SDK
`$VAR` references in config.yaml resolve to strings, and exa-py rejects a
string `num_results` ("Invalid value for option 'num_results': 10. Expected
one of [<class 'int'>]"), so `max_results: $EXA_MAX_RESULTS` made every Exa
web search return an error. `contents_max_characters` reached the API as a
string the same way.
Coerce both like the DDG and SearXNG providers do: parse an integer, and warn
and keep the default for a blank, non-numeric, zero or negative value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(exa): reject fractional and boolean limits instead of truncating them
Review follow-up: int() turned an unquoted YAML `3.5` into 3 and `true`
into 1, so a misconfigured limit was used with a different value. Accept
only non-boolean integers and integer-form strings; everything else warns
and keeps the default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(feishu): stabilize file batch timing
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(slack): log Socket Mode connect failures instead of dropping them
SlackChannel.start() ran SocketModeClient.connect() through
loop.run_in_executor() and discarded the returned future. When the
connect raised (rejected app token, network error, TLS failure) the
exception was stored on a future nothing awaited, so the only trace was
asyncio's "Future exception was never retrieved" at garbage collection.
Meanwhile the channel had already logged "Slack channel started" and
reported itself running.
Route the executor call through _connect_socket_mode(), which logs the
failure with logger.exception() the way the Telegram and Discord channel
threads already do. The regression test injects a failing connect and
asserts the error reaches the channel logger.
* fix(slack): bind queued connect to its client
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): strip trailing sentence punctuation from extracted resource refs
The resource-graph extractor's _PATH_TOKEN_RE includes '.' in its path
character class, so a bare package-relative path at the end of a sentence
(e.g. "Read references/setup.md.") is extracted with the trailing period
still attached. build_resource_graph then emits a false-positive
resource.missing finding for "references/setup.md." and, because the
real file is never matched, reports the actual references/setup.md as an
unreferenced orphan.
Other sentence punctuation (",;:!?()") already falls outside the regex
character class and is correctly dropped; only '.' is ambiguous because it
is both a valid path character and prose punctuation.
Strip trailing '.', '?' and '!' from every extracted reference in
_extract_references. Real dotted filenames are unaffected: rstrip only
removes punctuation at the very end, so references/config.yaml and a link
target such as references/v1.0.md keep their dots.
Adds two regression tests covering prose-terminated bare paths and real
dotted filenames.
* style(tests): format skill review tests
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(models): add declarative reasoning capability contract (#5073)
Introduce an optional per-model `reasoning:` block beside the legacy
`supports_thinking` / `supports_reasoning_effort` booleans. It declares
thinking availability (unsupported | optional | required), the accepted
effort values with aliases, default, and serialization path, the payload
dialect, and the reasoning-history requirement. When present the booleans
are derived from it and contradictory profiles fail at config load.
`deerflow.models.reasoning` normalizes legacy and declared profiles into
one immutable contract and resolves every caller's generic request against
it. `create_chat_model` is the single enforcement point: required-thinking
models never receive a synthesized disable payload, unsupported effort
values never reach the provider, and the dialect synthesizes the on/off
payload when no template exists. Legacy profiles keep the historical path
byte-for-byte. The lead agent and subagent descriptor resolve the same
policy first so run metadata reports effective values.
`/api/models` and `DeerFlowClient` project the contract as `reasoning`;
the frontend derives mode clamping and effort choices from it through
`core/models/reasoning.ts`, and user preferences accept provider-specific
effort tokens. The setup wizard's Z.AI GLM-5.3-Flash profile migrates to
the contract, restoring its low/high/max effort control.
* docs(models): move the reasoning contract guidance into models/AGENTS.md
backend/AGENTS.md sits within 46 bytes of the 96 KiB effective-chain hard
limit for the middlewares, runtime, sandbox, and subagents guides, so the
new section pushed scripts/check_agent_guidance.py over it. The models
guide is the nearest file for the factory and the contract module, and
its chain has ample room.
* fix(models): address review on the reasoning contract
- vLLM dialect: the enable path now mirrors whichever chat_template_kwargs
switch the template declares, like the disable path already did, so a
template using the older `thinking` key no longer also grows
`enable_thinking`.
- Validate operator-supplied effort values at `effort.path` inside
`when_thinking_enabled`, `when_thinking_disabled`, and the `thinking`
shortcut against the contract, not only the profile-level value; they are
forwarded when the caller chooses nothing, so the "unknown values never
reach the provider" guarantee now holds for them too.
- Constrain `effort.path` to dotted identifiers and reject single-segment
paths that would replace a whole mapping (`extra_body`, `thinking`,
`model_kwargs`, `default_headers`, `default_query`).
- Z.AI GLM-5.3-Flash profile: `default: high` instead of `max`, because the
default also governs summarization, title generation, and subagent calls
that never choose an effort; `max` stays selectable. Documented in the
config example, the configuration manual, and the design note.
* fix(models): close reasoning contract review gaps
* fix(models): preserve native Ollama reasoning config
* fix(models): keep native reasoning strings and project reasoning into the descriptor
- `ModelConfig.reasoning` also accepts the level string ChatOllama takes
for gpt-oss style models (`low` / `medium` / `high`); the factory forwards
both the boolean and the string on the legacy path, so profiles that
passed `reasoning: high` through before the contract field existed keep
loading and keep sending it.
- The assembly descriptor no longer excludes `reasoning` from
`model_parameters`: the native ChatOllama value and a declared contract's
dialect/history/effort change the request payload, so they now move the
assembly fingerprint.
- e2e: the sidecar spec pinned the pre-contract body where a model without
effort control still received `reasoning_effort: minimal`; the preset is
now dropped for such models (the Gateway discarded it anyway), so the
assertion expects the key to be absent.
* fix(sandbox): scope viewed image host reads to the current thread
* docs(gateway): trim agent guidance to fit CI limit
* fix(images): honor thread data paths for saved host copies
* test(images): align authorization fixture with thread paths
---------
Co-authored-by: YxinMiracle <“939157765@qq.com”>
* fix(memory): reject a Honcho base_url that can never resolve
HonchoConfig only did str(...).rstrip('/') on backend_config.base_url, so a
scheme-less or non-http value was accepted at Gateway startup and then made
every memory read return empty and every write drop silently, with only a
per-operation WARNING. httpx turns 'localhost:8000' into the bogus scheme
'localhost:'. The same check already exists in the mem0 and OpenViking
backends, and honcho/README.md plus HonchoMemoryManager.from_config both
promise a bad URL fails fast at startup.
* fix(memory): cover the scheme-case and host-less variants of the guard
- drive the api_key-over-plain-HTTP check off urlsplit(...).scheme: urlsplit
lowercases the scheme while startswith("http://") does not, so
"HTTP://internal:8000" reached httpx over plain HTTP with the key set and
allow_insecure_http still False
- read hostname instead of netloc in the base_url guard, so an authority
without a host ("http://", "http://:8000") is rejected at config time like
the scheme-less shape this PR already removes
- test direct HonchoConfig(base_url=...) construction, the path the guard's
placement in __post_init__ exists to protect
The docs site pointed at POST /api/extensions/{skills,mcp}/{name}/enable
and /disable, and backend/docs/API.md at POST /api/skills/{name}/enable
and /disable. None of these routes exist. The gateway toggles a skill
with PUT /api/skills/{skill_name} and an MCP server with
PATCH /api/mcp/config, both admin-only, which is also what the
frontend calls.
* fix(uploads): preserve link errors during staged cleanup
* Keep published uploads successful when staging cleanup fails
* docs: keep upload guidance within chain budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(guardrails): add TypeSafe (Jev) pre-execution tool risk gate
Adds TypeSafeGuardrailProvider, a network guardrail provider that asks one
`noul` question per probed tool call and denies calls whose Jev risk
probability reaches `threshold`. It reuses the existing GuardrailProvider
slot: no middleware, config-schema or dependency changes.
Semantics that are load-bearing:
- state is strict JSON (no `default=`, `allow_nan=False`, string keys only);
unserializable arguments, lone surrogates among them, and argument text
above `max_state_chars` are denied locally with zero network requests,
never truncated and sent.
- the async path cancels in-flight requests through `asyncio.timeout`; the
sync path checks the deadline after the response headers arrive, around
the body read and before accepting the result, and drops late results.
No total return-time bound is promised on the sync path.
- clients are created per evaluation inside a context manager, and the
injected `transport_factory` is called once per network-needing
evaluation: httpx closes an injected transport with its client, so a
shared instance would be dead on the second call.
- denial reason messages carry probability, threshold, served model version
and state digest, so the threshold comparison can be replayed from the run
journal; local denials record no probability or model version, and allow
decisions stay unpersisted (the audit path is unchanged).
- provider errors carry no response body or arguments: malformed responses
report the field and the offending type only, so a response echoing tool
arguments cannot reach middleware logs or the evaluation report.
The online evaluation harness (`backend/scripts/eval_typesafe_risk_gate.py`
plus its labeled case set) is the threshold/tools evidence that must precede
enabling interception. It reports four mutually exclusive populations
(uncached network, cache hits, unprobed tools, local refusals), separates
model classification from local refusals and fail-closed errors, measures
latency from the caller entering `aevaluate` through cleanup, discloses
censored samples, probes DNS/TCP/TLS separately, and flags unprobed tools as
a coverage gap rather than a pass. Real responses are never written to disk.
Cache-pass samples are classified from `decision.metadata['cached']` and
counted once, warming calls are retained as an auxiliary group with their
failures named in the report and JSON, and an optional connection-probe
diagnostic can never discard completed evaluation results.
Tests: the provider suite over MockTransport plus the middleware
integration, a blocking-IO anchor that drives `aevaluate` against a real
loopback HTTP server with a meta-check proving the sync path on the loop
trips the gate, and harness cases covering the reporting defects. Every fix
above was verified to fail its new tests when reverted.
Docs: GUARDRAILS.md gains Option 4 (privacy, local-refusal trade-off,
deadline semantics, audit boundary, coverage/bypass risk, Allowlist->RBAC
migration, lone-surrogate local refusals and the no-response-body property
of error messages) and config.example.yaml gains a smoke-scope example.
Config: the new example block bumps `config_version` 46 -> 47, with the
Helm chart's embedded version and README example kept in step as
`scripts/check_config_version.sh` (chart.yaml CI) requires.
* feat(guardrails): whitelist tool calls inside the TypeSafe risk gate
Static allow rules now live in the provider instead of waiting on a composite
provider. `whitelist` mirrors `AllowlistProvider.allowed_tools`: omitted means
"not enforced", `[]` refuses every tool, and a list refuses every tool outside
it locally (`typesafe.tool_not_whitelisted`) before state construction, caching
and networking, independently of `fail_closed`. Whitelisted tools are still
probed and still denied when their risk probability reaches `threshold`, and
the whitelist is checked before the probe scope so an unlisted tool cannot
inherit an allow from being unprobed.
Deny rules stay with `authorization.*`. The policy version moves to 1.1.0, the
evaluation script reports whitelist refusals as their own population, and
`GUARDRAILS.md` / `config.example.yaml` document the allow-half migration.
* fix(guardrails): address TypeSafe risk gate review findings
- Status errors report only the numeric status. The HTTP status line is
server-controlled like the body, and this message reaches middleware logs
and the evaluation report, so a hostile endpoint could echo the state (raw
tool arguments) through it.
- Expired cache entries are removed with `pop(key, None)`. Sync evaluation
runs on executor threads when a turn produces parallel tool calls, so a
second caller can remove the same expired entry; a bare `del` raised
KeyError, which the middleware reports as a provider error and, under
fail_closed, turns into a spurious denial.
- The evaluation harness states its acceptance gates inline (no missed
dangerous call, system false-block <= 5%, uncached network p95 <= 1s)
instead of citing a design spec that is not in the repository, and reports
the whitelist-refusal population alongside the other four.
- `--tools` / `--whitelist` strip whitespace and drop empty entries, so
`--tools "bash, write_file"` no longer probes only bash while treating
`" write_file"` as an unknown tool name.
- The case set's `padding` note says top-level argument fields only, matching
what the runner implements.
Each fix was verified by running its new test against the pre-fix code.
* refactor(guardrails): name the TypeSafe permission list allowed_tools
Align the key with `AllowlistProvider.allowed_tools` so an allowlist migration
copies it verbatim instead of translating `allowed_tools` into `whitelist`.
The companion names follow: reason code `typesafe.tool_not_allowed`, metadata
`tool_not_allowed`, harness population `not_allowed`, and the `--allowed-tools`
flag.
Semantics are unchanged: omitted = not enforced, `[]` = refuse every tool,
tools outside the list are refused locally before any state, request or cache
entry, and listed tools still face the risk gate.
* fix(guardrails): record only a bounded served model token; fail the eval exit code
Two findings from the re-review at d3f0b072.
`model` is response content, and `_model_decision` interpolated it verbatim into
the reason message -- which reaches the denied ToolMessage, the run journal,
middleware logs and the evaluation report on disk, and which the cache retains
for replay. A malformed or hostile endpoint can put an echoed tool argument,
injected text or a multi-megabyte string there. The token is now recorded as
itself only when it matches a conservative shape; anything else is recorded as
`unrecorded:sha256:<digest>`, which keeps the verdict and a stable provenance
record without echoing the value. Rejecting the response instead would turn an
odd version string into a fail-closed denial of every call the endpoint answers.
The digest path encodes with `surrogatepass` so an escaped lone surrogate -- a
value `json.loads` happily produces -- cannot raise where this must not fail.
`scripts/eval_typesafe_risk_gate.py` returned 0 even when a gate failed, so an
operator scripting `... && enable` got a green light on a failing evaluation.
`main()` now returns 1 when any gate in `report["score"]["gates"]` fails, after
the JSON report is written, and prints the failed gate names.
Tests: `test_an_unrecordable_served_model_is_digested_instead_of_echoed` (an
echoed argument, a space, a lone surrogate and an oversized value, plus the
cached replay) and `test_main_exit_code_carries_the_gate_verdict` (both
directions). Both fail against the pre-fix source.
docs/GUARDRAILS.md audit boundary updated to match.
113 passed: tests/test_typesafe_guardrail_provider.py,
tests/test_typesafe_risk_eval_script.py,
tests/blocking_io/test_typesafe_guardrail_provider.py.
* fix(evals): fail the risk-gate run when a labeled population is uncovered
Review finding on dcd38cc3. The new exit code made `_score`'s vacuous passes
load-bearing: unprobed calls are excluded from the score, so `risky_misses_zero`
is true over an empty risky population. `--tools read_file` excludes all 18
risky cases in the bundled set, so the run printed every gate as PASS and exited
0 after evaluating no risky operation at all.
`_score` now gates coverage itself: `risky_coverage_present` and
`safe_coverage_present` fail when the probe scope leaves that labeled population
with nothing evaluated, so a scope that cannot substantiate the verdict no longer
reports one. The safe side had the same vacuity -- `system_false_block_within_target`
passes trivially with zero safe cases.
Verified end to end against a loopback mock endpoint with the bundled case set:
`--tools read_file` exits 1 with `GATE risky_coverage_present: FAIL` while the
other four gates still PASS, where dcd38cc3 exits 0 on the same command.
Tests: `test_main_fails_when_the_tool_scope_leaves_no_risky_case` and
`test_main_fails_when_no_safe_case_is_in_scope` (both fail against dcd38cc3),
`test_bundled_case_set_keeps_both_labels_the_coverage_gates_require`, and the
exit-code test now runs with both populations present.
116 passed: tests/test_typesafe_guardrail_provider.py,
tests/test_typesafe_risk_eval_script.py,
tests/blocking_io/test_typesafe_guardrail_provider.py.
* fix(scheduler): read the lease owner under SQLite's writer lock
release_dispatch_lease guards on the lease owner and then writes the
status its caller asked for — "enabled" for a scheduled dispatch. It read
the row with session.get(..., with_for_update=True), and FOR UPDATE is
ignored on SQLite, so that guard ran against whatever snapshot the plain
SELECT saw.
A user pausing a task while its dispatch was in flight therefore lost the
pause: pause clears lease_owner, which is exactly what the guard checks,
but the release had already read the pre-pause row, passed the check, and
overwrote status with "enabled". The API had answered "paused", the task
row went back to enabled with next_run_at untouched, and the scheduler
kept firing it.
The read now goes through _lock_task, the helper every other mutating
path here uses, which takes the writer up front on SQLite (FOR UPDATE
keeps doing the work on Postgres, where a pause that commits first is
seen and the release declines). The reconciliation paths in this file
keep their documented "no SQLite writer emulation" trade-off.
* docs(changelog): note the scheduler lease-release lock fix (#5777)
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
urllib3 hands the raw Location field value on as the request target of the
recursive redirect frame, and that grammar admits interior spaces. The
Redirecting slots and the Incremented Retry url capture already tolerate
them; the per-request line and the two Retry:/Retrying tails bounded the
target at the first space, so a signed origin-form path survived all three.
Co-authored-by: sxh313 <sxh313@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(image-search): coerce DDG image result limits from environment variables
* fix(image-search): reject fractional max_results config values
A YAML value such as `max_results: 3.5` reaches the tool as a float, and
int() truncated it to 3 without the warning the README promises. Treat
non-integral floats as invalid and cover the config path in the test.
* fix(image-search): reject boolean result limits
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(community): fall back to the default SearXNG max_results on an unparseable value
A blank `max_results:` in config.yaml loads as None, and int(None) then fails inside web_search_tool's try block, so every SearXNG search returns an error payload instead of results. Coerce like the sibling providers do: warn and keep the documented default.
* test(community): cover SearXNG max_results that cannot be parsed
_ensure_postgres_schema_with_pool runs inside the drained pool context but
acquired its connection with a raw 'async with pool.connection()'. Caller
cancellation arriving while the connection's __aexit__ is blocked interrupts
that exit, so the connection is never returned to the pool and the pool's own
drained teardown closes with a checked-out connection — the backend-ownership
invariant documented in runtime/AGENTS.md (same class as the provider-context
fix that introduced drained_async_context).
Use drained_async_context for the schema-setup connection so the connection
returns to the pool before the pool context unwinds.
* fix(mcp): avoid cache resets for unrelated config changes
extensions_config.json carries MCP servers, skills and middleware settings, but the MCP cache treated any whole-file signature change as MCP state change, so toggling a skill retired every cached tool and closed all pooled sessions.
Record a snapshot of the effective MCP slice (enabled servers' parsed config plus mcpInterceptors) and retire MCP state only when that slice changes. Hand the exact config revision to tool discovery so a change during discovery cannot publish tools built from a superseded revision, and add a non-initializing refresh entry point so disabling the last enabled server still converges the cache and pool on the next tool assembly.
* docs(mcp): align cache guide and restore error diagnostics
Review feedback on #5750 flagged three non-blocking issues: the MCP module guide still described the pre-semantic-snapshot invalidation contract; the tools.py sanitization covered every exception in the MCP block, discarding diagnostics for failures that cannot carry resolved credentials; and _is_cache_stale()'s docstring presented it as a pure predicate despite adopting a new file signature.
Narrow the type-only logging to ExtensionsConfig.from_file() and restore logger.exception() for tool caching, pool cleanup and plugin filtering, with a regression test asserting the message and traceback survive. Document the effective-MCP-slice comparison, the stable config handoff to discovery, refresh_mcp_cache_if_active() and the _initialized_without_config handling in the module guide, and note the signature-adoption side effect in the docstring.
* fix(mcp): unify config normalization and verify signatures
Review follow-up on #5750.
- Normalize mcpInterceptors (bare string == single-element list; missing key == empty list) and the type/transport alias through one shared module used by the tools cache, the interceptor builder and the durable-task config snapshot, so equivalent spellings no longer retire tools/sessions or raise McpTaskConfigurationError.
- Treat a resolved-path switch as a signal to re-read the effective MCP slice instead of an invalidation reason: an identical slice adopts the new path and signature and keeps pooled sessions.
- Refuse to publish or adopt a signature whose sha256 digest is missing ((mtime, size, None) from a file that could not be read).
- Sync the English and Chinese MCP user docs with the effective-config comparison, and document that the Gateway admin reset stays unconditional.
* docs(mcp): note the unverifiable-signature fail-closed branch
A present config file whose content cannot be hashed is treated as stale, unlike a deleted file which keeps the last-known-good cache. Record that asymmetry in the module guide.
* docs(mcp): scope the no-hash rationale to never-initialized processes
The cost-free early return in refresh_mcp_cache_if_active() only covers a process that never published an MCP cache. A process that did publish one (including an empty cache from the zero-server or LangGraph Studio lazy-init path) still pays one stat+sha256 per tool assembly. Correct the rationale in the docstring and the matching comment in get_available_tools().
* feat(examples): add a standalone text classification plugin
A deployment-installed package contributes one model tool, classify_texts,
and a status action through the full-stack plugin contract. The tool labels a
list of texts against caller-supplied categories with a deployment-configured
backend: Jev (one choice question per item over a shared batch state, checked
per item) or an OpenAI-compatible chat endpoint (a JSON label list validated
in code). Items are packed into requests under a size limit and sent with
bounded concurrency; every failure is explicit per call or per item;
credentials come only from deployment-named environment variables; item text
never enters instructions, question keys, results or errors, and the served
model name is kept to a short identifier. Each batch runs under an asyncio
timeout bounded by the call deadline, so the whole call stays inside the
host's 30-second limit, and ids and category names are bounded in bytes so
300 results fit the host's 64 KiB output limit.
* docs(examples): add an opt-in live smoke test for the text classification plugin
The script loads the package through the real extension path and classifies 20
synthetic English and Chinese sentences plus a blank and an over-long item with
the configured backend, printing labels and statuses only. Against real Jev it
agreed on all 20 labels in two requests and 1.6 seconds, with the served model
reported as jev-1.13.0.
* fix(examples): measure request size as sent and cover transport failures
httpx serializes json= bodies with the json.dumps default, escaping every
non-ASCII character as six bytes, while the size estimators measured raw
UTF-8, so CJK batches could reach about twice the 256 KiB limit. The
estimators now measure the same encoding; a test with CJK categories and
texts checks the bytes the transport actually received. The network status
gets a test through connect and protocol errors, and the deadline test's
wall-clock slack is widened for loaded runners.
* fix(examples): match classifier request wire encoding
* fix(examples): bound classification output keys by JSON size
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(auth): claim the first admin atomically instead of counting first
POST /api/v1/auth/initialize read the admin count in one session and
created the admin in another, with no lock between them. Two requests
arriving together with different emails both saw an empty system and
both became admins: reproduced over HTTP, two 201s and two admin rows,
after which setup-status reports an initialized system.
The endpoint is public and unthrottled, and during the pre-setup window
anyone who can reach the gateway can claim admin with a single request
anyway — that part is the first-boot design. What the race removes is
the only signal: without it exactly one request wins and the loser gets
409, so an operator who loses sees "already initialized" and looks into
it. With it, both succeed and the operator's setup looks ordinary.
The check and the insert now share one transaction in the repository,
with writers serialized first — SQLite takes the write lock up front
(BEGIN IMMEDIATE) and Postgres a transaction-scoped advisory lock, the
idiom the project and channel repositories already use. The same
repository serves both backends, and multi-replica deployments make the
race genuinely parallel rather than an event-loop interleaving.
create_first_admin returns None when the claim is lost, which the route
reports as 409; uniqueness conflicts still raise ValueError and stay a
400. The count in the handler is now only a fast path that avoids
hashing a password for an already-initialized system.
AUTH_DESIGN.md said concurrent initialization was backstopped by a
database unique constraint. No such constraint exists — email
uniqueness only catches the same address submitted twice — so it now
describes the claim that does the work.
* docs(changelog): note the atomic first-admin claim (#5776)
* fix(auth): refuse a dialect the first-admin claim cannot serialize
Review follow-up. The claim's serialization handled sqlite and
postgresql and fell through silently for anything else, which would
leave create_first_admin as the plain check-then-act it exists to
prevent — the one place the abstract method's atomicity contract is
enforced.
An unknown dialect now raises instead. The engine builds only these two
today, so this guards a future backend: it fails loudly at first boot
rather than quietly reopening the race.
os.getenv's default only covers an unset variable, so `docker run -e
GATEWAY_PORT=` — or `GATEWAY_PORT=${PORT}` with PORT unset — reached
int("") and aborted the create_app() import that both entry points run.
Blank now falls back to the declared default, while "0" stays a real port.
* fix(skills): load SKILL.md saved as UTF-8 with a BOM
The front-matter anchor began at ^---, so the U+FEFF that Windows Notepad and
PowerShell's `Set-Content -Encoding UTF8` prepend made the match fail. parse_skill_file
then returned None and the catalog builder dropped the skill without logging, leaving an
author with a valid SKILL.md that simply never appears. split_skill_markdown reported
"No YAML frontmatter found" for the same file at install time.
Let the anchor consume one leading byte-order mark, and reuse the shared compiled pattern
in parser.py instead of the duplicated inline copy so the two cannot drift apart again.
* fix(skills): capture the description of a BOM-prefixed SKILL.md
The reader in skill_context anchored on ^---, so the byte-order mark that this change already
tolerates in the loader and validator made the match fail and build_skill_entry_metadata_from_read
capture an empty description. render_skill_context then listed the skill without the one line the
active-skill reminder shows for it.
Consume one leading byte-order mark there and pin the capture with a regression test.
or 0.5 cannot distinguish an explicit 0.0 from an unset field, so a discounted fact got the default confidence weight in both the FTS5 document and the substring fallback, while the same row indexed through FTS5Retrieval.index_fact kept its zero weight.
