* fix(browserless): reject timeouts httpx cannot use
`_coerce_timeout` forwarded any int, float or numeric string to
`httpx.AsyncClient(timeout=...)` unchanged, so a configured `timeout_s: 0`
or a negative value failed every fetch and capture with `ConnectTimeout`
against a healthy backend, `nan` raised `ValueError: deadline must not be
NaN`, and `inf` left the request without a deadline so it never completed.
Each of those silently disabled the provider.
Fall back to the documented 30s default with a warning when the number is
not a usable deadline. This matches `crawl4ai._coerce_timeout` after #7481;
`jina_ai._coerce_timeout` still forwards non-positive values.
* fix(browserless): address timeout review and format tests
Explain how HTTPX handles NaN and positive infinity. Document that
Browserless accepts positive finite timeouts without an upper limit,
and pin that behavior with a 1e308 test case.
Crawl4AI and Jina timeout validation are tracked in #7481 and #7823.
Remove the Browserless docstring's claim that Crawl4AI shares its
coercion behavior. Format the test file for the backend lint check.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
In long-running agent workflows with repeated context compaction, empirical gateway ablations across 132 real sessions revealed that standard boundary compliance remains robust, while semantic drift can occur when model-generated summaries rephrase literal approval phrases and action conditions.
This change converges PR #8114 into a clean built-in summary prompt enhancement:
- Replaces regex extraction, tag elevation, and heuristic revocation with an extended built-in DeerFlow summary prompt (_DEERFLOW_SUMMARY_PROMPT).
- Instructs the summarizer to faithfully record active user requirements, literal approval phrases, permitted actions, and later explicit revisions without paraphrasing.
- Treats summary text strictly as historical data, fully preserving the security boundary and control/data plane separation of DurableContextMiddleware.
- Unifies automatic and manual compaction factory paths to select the built-in template when unconfigured, while preserving explicit summary_prompt precedence.
- Adds comprehensive sync/async unit tests covering built-in and custom templates.
Co-authored-by: Totoro <Totoro-qaq@users.noreply.github.com>
Co-authored-by: Totoro <msy626836554@gmail.com>
* docs: turn README into an index and move reference detail to docs/guide
The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md.
* docs: send user docs to topic guides and budget the root README
The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003).
* chore: remove unreferenced screenshots and finished plan notes
pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them.
* docs: route always-loaded AGENTS.md files to scoped guides
* docs(agents): cut scoped guides to scope, verify, traps, decisions
Each AGENTS.md keeps four kinds of content: the scope of the directory, the
verification commands, traps a local test run does not show, and decisions the
code cannot state. Values, field lists, function inventories, test names, issue
numbers and history are deleted, because the code, the tests and git state them.
Search and fetch providers share one community guide. Four guides with no
remaining content are removed. The size check now allows 6 KiB per file (hard
limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB).
* style: format frontend guide table and guidance test imports
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(knowledge): add optional ingestion and status center
* docs(knowledge): record upstream CI budget fix and regression results
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor(skills): keep allowed-tools as compatibility metadata
* docs(skills): align current guides with agent tool authority
* test(skills): simplify authority regressions and sync Korean docs
* feat: return history search excerpts near keyword matches
* docs: use English for history search guidance and comments
* docs: sync Chinese history search excerpt contract
* docs: clarify history excerpt fallback for oversized matches
* fix(history): skip oversized occurrences when selecting excerpts
Continue scanning casefolded substring matches until an occurrence fits the 600-character excerpt, including overlaps. Cover later fitting hits through synchronous and asynchronous active/archive search and read tools, and update the documented contract.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): bind local make dev and make start to loopback
serve.sh and backend/Makefile started the Gateway with --host 0.0.0.0,
nginx.local.conf listened on every interface, and Next.js kept its
all-interfaces default outside Windows, so a local stack on a LAN or VPN
exposed ports 2026, 8001 and 3000, including /setup before the first
admin existed. The Docker stack and the README already document a
loopback-only default.
Bind the Gateway and frontend to 127.0.0.1 and keep nginx on loopback
unless BIND_HOST is set, the variable the Docker stack already honors.
nginx-local-conf.sh renders a copy listening on BIND_HOST, and serve.sh
resolves it before stopping anything so an invalid value cannot tear
down a running stack.
* docs(changelog): reference #6587
* docs(changelog): separate the loopback entry from the next bullet
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(release): reject a stale backend/uv.lock in the version gate
verify_versions.sh compared only Chart.yaml, pyproject.toml and
package.json, so bumping those three by hand passed the v* gate while
backend/uv.lock still recorded the previous root package version. The
backend image installs with uv sync --locked, so the chart and the other
images published while the backend image failed to build.
Run uv lock --check in backend/ as part of the gate, letting uv own the
PEP 440 normalization, and fail when uv is missing. The reusable
verify-versions workflow installs the uv version the backend image pins.
* docs(changelog): reference #6588
* fix(release): explain an unparsed lock version and stop blaming the lock for every uv failure
Print a fallback when the lock's root entry is not found, so the job log
does not show an empty field, and word the uv lock --check annotation so a
network error or uv crash is not reported as a stale lock.
* feat(authz): construction-enforced async decision batches (ActivationDecisions)
The follow-up promised in the #4541 review threads. Four review rounds
traced four instances of one failure family: an async execution path
whose decision-map miss silently fell back to the synchronous
authorize() from a worker thread — the wrong API for loop-affine
providers, and under fail_closed:false a provider error there flips a
denial into an allow. The carrier was a plain dict|None held together by
caller discipline.
ActivationDecisions (deerflow/authz/activation_decisions.py) makes the
contract construction-enforced:
- a covered name returns the batched decision (unchanged semantics);
- a MISS resolves per the configured provider-error policy (fail-closed
denies, fail-open allows) with a loud WARNING — a miss is a
prepass/consumer divergence, a construction bug, not a policy
question — and never touches a provider;
- None remains the sync-chain signal (synchronous authorize() is correct
there);
- an empty candidate set still yields an empty BATCH, not None, so an
unexpected consumer name fails per policy instead of silently taking
the sync path;
- get_or_none lets publishers distinguish 'uncovered' from False;
- fail_closed travels with the batch from the resolved authorization
context.
Both middlewares (_activation_allowed in SkillActivationMiddleware and
SkillToolPolicyMiddleware) migrate to the type; their collectors return
batches; the render-decision publisher adapts. Tests 65->70+5: type unit
tests (covered/miss both directions with loud-log assertions/
get_or_none/container), middleware-level miss semantics (fail-open
allows with provider.sync_calls == [], production-default fail-closed
denies, None sync chain still synchronous), and mutation M18 (restoring
the miss's sync fallback) verified red. 252 tests green on the affected
surface.
* docs(authz): sync every decision-batch contract description to the new miss semantics
Review round 1 on #6508 — all documentation-level: the docstrings still
described the miss-as-synchronous-fallback contract this PR removes.
- _activation_allowed (both middlewares): a covered name returns the
batched decision; a miss resolves per the carried provider-error policy
with a WARNING and never touches a provider; None (sync chain /
disabled) is the only synchronous case.
- _collect_activation_decisions / _resolve_policy_registry /
_active_skills_for_paths: the reason not to diverge is now a
policy-resolved miss (denied under the production fail-closed default;
allowed without any provider consult under fail-open; logged loudly),
not a silent sync fallback.
- The two _REGISTRY_LOAD_FAILED sentinel comments, same rewording.
- Drop unused tmp_path/monkeypatch fixtures from the two new miss tests.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): add host-bound Agent run control
* fix(extensions): preserve internal launches and scope run control
* docs(extensions): use a plugin-local run operation key
* feat(frontend): support extension providers in composer mentions
* fix(mentions): retain settled plugin suggestions during search
* fix(extensions): validate identifiers and skip empty mention searches
* test: stabilize history pagination after latest page refresh
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(authz): extension tool authorization PR2 — middleware-declared tool path
Layer 1 now covers tools declared by middlewares, which LangChain merges
into the bound tool set after the host's explicit-list filter:
- new agents/middlewares/tool_declarations.py: record the ordinary pass's
verdict, collect declarations from the built stack, decide only the
delta seeded by the first pass (a name denied for this build stays
denied), narrow the stack build-locally on independent state-preserving
copies (direct __dict__ write, refused up front for tools data
descriptors, never mutating caller-owned instances), and verify the
exact bound stack after the final normalization copy.
- IsolatedMiddleware ships a state-preserving __copy__ so narrowing
survives full/delta normalization and copies of copies.
- TodoMiddleware degrades when write_todos is denied: no system-prompt
injection, no completion/context-loss reminders, no jump_to.
- Layer 2 forwards host-resolved tool provenance into AuthzRequest.context
and binds the infrastructure exemption to the concrete host-created
tool object instead of a name set (tool=None goes to the provider).
- Wired into all four assembly paths (lead x2, native subagent with the
decision offloaded via asyncio.to_thread, DeerFlowClient).
Tests: new test_middleware_declared_tools.py (35) and
test_layer2_tool_provenance.py (7); blocking-IO anchor pinning the
subagent offload (teeth verified red->green); placement-guarantee and
enforcement suites extended; executor test doubles converted for the
async _create_agent. Behavior change called out in CHANGELOG: with
authorization enabled and a tools policy that excludes write_todos,
plan-mode builds no longer bind it (built-in RBAC default unchanged).
* fix(authz): close residual gaps from review — narrow non-collectable declarations, dedupe candidates, warn on skipped pass
- apply_declared_tool_view now removes non-BaseTool declarations from the
bound view when authorization is enabled (they would otherwise bind via
LangChain's create_tool auto-conversion with no provider decision);
verify_declared_tool_view fails loudly if one survives normalization
- decide_declared_tools dedupes never-submitted names so the provider sees
each candidate once
- subagent executor logs a warning when the provider is set but the seeded
Layer-1 state is missing, instead of skipping the declaration pass silently
- docs: middleware-declared tools must be BaseTool instances with usable names
* fix(authz): fail loudly on non-list/tuple middleware tools declarations
_iter_declared warned and returned () for a 'tools' attribute that is not
a list/tuple, but LangChain's factory iterates the attribute as-is — so a
set/generator/dict_values declaration bound with no Layer-1 decision.
Raise DeclaredToolViewError instead, matching the module's fail-loud
standard for every other un-narrowable shape. Disabled path remains a
strict no-op. Docs updated to match.
* feat(authz): enforce skill authorization at assembly and slash-activation (#4063 Phase 3)
Phase 3 / Skills — the second of three resource-type PRs (Models, Skills,
Sandbox). The RBAC provider already maps "skill" → config key "skills"
(rbac.py _RESOURCE_POLICY_KEYS), so no schema change is needed.
Layer 1 (assembly-time, mirrors Phase 1B):
- filter_available_skills_by_authorization() in skill_filter.py filters the
skill-name allowlist (set[str] | None) by the provider's "skill" policy.
In lead agent, runs after _available_skill_names() so the catalog
(describe_skill) and SkillActivationMiddleware share one filtered set.
In subagent executor _load_skills(), filters config.skills before disk load.
- filter_resources_by_authorization() in enforcement.py — generic version of
filter_tools_by_authorization for any resource with a name attribute.
Layer 2 (slash-activation):
- No new middleware code needed. SkillActivationMiddleware._resolve_activation
already checks "reference.name not in self._available_skills". Since Layer 1
filters the allowlist, denied skills cannot be slash-activated.
authorization.enabled: false is a complete no-op. available_skills=None (no
agent allowlist) + enabled: resolves all skill names from config and filters.
12 new tests + 243 existing authz/skills tests pass.
* test(subagents): wait on the chained future instead of polling done() in caller-loop teardown test
* feat(authz): enforce action-scoped skill:activate at runtime and reuse the catalog loader for filter candidates
* feat(authz): arm skill:activate on the subagent chain, unify the client skill-surface user, and record the Layer 2 reversal in the design notes
* chore(authz): align the subagent filter's fallback user with the chain's bucket convention
* feat(authz): gate autonomous skill loading on skill:activate, close the bootstrap assembly leak, and reuse one provider per subagent assembly
Review round 4 (willem-bd, CHANGES_REQUESTED):
- [P1] Autonomous load path: describe_skill filters its match set on the
action-scoped skill:activate decision (all-denied is indistinguishable
from "no match"), and ToolErrorHandlingMiddleware stamps skill_context
entries only when the action allows it — a denied SKILL.md read gets a
skill_context_denied marker instead, so durable context, allowed-tools
policy, and autonomous secret bindings never activate a denied skill.
File visibility itself stays Layer 1 (sandbox-projection semantics).
- [P1] Bootstrap assembly: build_middlewares/apply_prompt_template now
receive the authorization-filtered set instead of the raw
_BOOTSTRAP_SKILL_NAMES, and an empty skill_names index is preserved
when deferred discovery is on rather than round-tripping through None
into the legacy full-metadata prompt that re-advertises a denied
skill (deferred-off keeps the legacy rendering; the filtered empty
allowlist blanks it on denial).
- [P2] SubagentExecutor resolves ResolvedSkillAuthorization once per
assembly (_resolve_skill_authorization, memoized per executor) and
shares it across the Layer 1 filter (authorization=), the activation
middleware, and both new gates — uncached provider factories can no
longer hand the layers different instances/policy snapshots.
All gates share one helper (skill_filter.skill_activation_allowed) so
slash, describe, and skill-file-load cannot drift; provider errors keep
the configured fail-closed/fail-open policy. skill_authorization is
threaded through _build_runtime_middlewares into both runtime builders.
Tests: describe gating (deny/fail-closed/fail-open/disabled), read
stamping (denied marker vs entry) + extract_skills skip without warning,
bootstrap three-way param (allow/deny x deferred on/off), lead + subagent
chain wiring for the stamp gate, and the distinct-instance provider
factory repro. Each fix is mutation-verified (reverting it fails its
test). Full A/B suite run: no new failures vs HEAD.
* feat(authz): re-authorize persisted skill_context before reuse and route async paths through aauthorize
Review round 5 (willem-bd):
- [P1] Persisted entries: skill_context survives across runs, so both
consumers now re-check the action-scoped skill:activate decision before
reuse. SkillToolPolicyMiddleware._active_skills_for_paths skips a
denied skill (same skip as a disabled/unallowlisted one; all-denied
keeps the existing fail-closed-to-builtins semantics) and
_in_context_secret_sources binds no secrets for a denied entry.
skill_authorization is threaded into SkillToolPolicyMiddleware on both
chains.
- [P2] Async provider API: skill_activation_allowed_async mirrors the sync
helper through await provider.aauthorize() (same pattern as
authorize_sandbox_execution_async). Async hooks precompute an
activation_decisions map on the event loop (slash target + persisted
entry names, one batch of awaits) and hand it to the worker-thread
handler, which consults the map instead of the sync API:
SkillActivationMiddleware.awrap_model_call (activation + secret
binding), ToolErrorHandlingMiddleware.awrap_tool_call (_amaybe_stamp
awaits the decision before stamping), and SkillToolPolicyMiddleware's
awrap_model_call/awrap_tool_call (entry names + slash source name).
describe_skill becomes a StructuredTool with both a sync function
(invoke -> authorize) and a coroutine (ainvoke -> aauthorize), so async
agent execution never routes a loop-affine provider through the sync
API while sync callers keep their semantics.
Tests: allow-read-then-deny-next-run regressions for secret binding and
allowed-tools (partial deny does not poison, all-denied fails closed to
builtins), and an _AsyncOnlyProvider (sync authorize always fails,
aauthorize works) driving four async-path assertions (activation, stamp
allow/deny, describe, tool policy). All six fixes are mutation-verified;
full A/B suite run against HEAD shows no new failures.
* fix(authz): authorize skill activation by declared Skill.name, not directory basename
ShenAC-SAC's P2: bundled skills may declare a name that differs from their
directory (skills/public/vercel-deploy-claimable declares name: vercel-deploy),
but the runtime gates derived authorization targets from the container path.
- New deerflow/skills/container_registry.py: build_container_path_registry()
+ canonical_skill_name() — the one path-to-Skill.name resolution shared by
the activation, tool-policy, and read-stamping middlewares.
- ToolErrorHandlingMiddleware resolves the read path through the (user-scoped,
via new user_id kwarg threaded from both runtime builders) registry before
the skill:activate decision, sync and async (registry lookup in a worker
thread, aauthorize() on the loop). RBAC allow by the declared name now
activates; allow by the directory name does not.
- SkillToolPolicyMiddleware async hooks canonicalize the policy paths first
(three-phase: thread registry -> loop aauthorize -> thread filter reusing
the registry; the cached per-model-step decision still short-circuits),
so the decision map is keyed by the names _active_skills_for_paths checks —
no more map miss falling back to the sync provider API from the thread.
_entry_names removed: persisted entry["name"] is path-derived by
construction; entries are resolved by path instead.
- SkillActivationMiddleware collects slash names (already canonical) plus
persisted entry *paths*, canonicalized off-loop via
_canonical_names_for_paths().
Regressions (test_skills_authorization.py 45->50): sync stamp target, RBAC
allow declared-name/deny directory-name both directions, async stamp via
_AsyncOnlyProvider, composed async slash -> tool-policy keeps the skill's
tools with zero sync fallback, persisted-entry secret binding survives the
name mismatch. All three mutations (stamp target, policy map key, entry
canonicalization) fail their tests.
* fix(authz): close the sync-fallback class on skill activation paths
willem-bd's two P2s on the canonical-name fix, plus a fourth instance of
the same class found while sweeping the membership:
- SkillToolPolicyMiddleware: a transient prepass registry failure is now
preserved (_REGISTRY_LOAD_FAILED marker) — the worker-side filter applies
its own fail-closed treatment instead of silently retrying storage,
which recovered skills absent from the decision map and fell back to
the synchronous provider API from the thread (fail-open retaining a
denied skill's allowed-tools).
- SkillActivationMiddleware: an empty entry-path list no longer pays a
full skill-tree scan on ordinary authorization-enabled async model
steps (previously a passive path with zero registry I/O).
- 4th instance (reproduced with the PR fixtures, then fixed): the entry
prepass failure on the secret-binding path let the consumer-side
_load_skill_registry_by_path silently recover and resolve entry names
outside the decision map — the same sync fallback, fail-open binding
the aauthorize-denied skill's declared secret. The fix goes one step
further than failure propagation: the prepass hands down the registry
*snapshot* the decision map was keyed by, and entry sources resolve
against that snapshot, so a map miss is impossible by construction
(this also closes the narrower window where both loads succeed but
disagree because storage changed mid-step). The slash source is
deliberately unaffected: it does not consult the decision map
(validated once at activation) and keeps its fresh per-call load.
- Class membership swept: skill_activation_allowed() sync call sites are
exactly four (stamping sync path and describe's sync implementation are
legitimately sync); the only two async-reachable fallback points (the
_activation_allowed helpers) are both construction-guarded now.
Structural hardening (typed async-batch decision object where a map
miss is fail-closed by construction) recorded in the plan notes as a
follow-up, anchored at those two helpers.
- _RegistryArg type alias replaces the degenerate dict|object|None union
in the policy middleware signatures.
Tests (test_skills_authorization.py 50->54): transient prepass failure ->
builtins-only with exactly one storage attempt and zero provider calls;
ordinary async step -> zero registry scans; two-call scenario -> entry
binding zeroed while the slash binding survives; entries resolve against
the prepass snapshot under a mid-step rename. Mutations verified red:
marker reverted to None, over-broad slash kill, snapshot bypass.
* fix(authz): correct secret-binding reference points and gate the rendered skill reminder
Review round 9 (willem-bd) plus construction-audit follow-ups on the same
class — every async divergence between two loads of the same data now has
an explicit, pinned reference point:
- Slash secret source resolves from a FRESH post-activation registry: the
prepass snapshot is taken before the aauthorize awaits, so binding the
slash source from it would activate NEW_KEY content while injecting
OLD_KEY when a skill's declared secrets change in that window. The
'one scan per step' bonus that motivated sharing the snapshot is
withdrawn — consistency wins. The two secret sources resolve
independently: a transient fresh-load failure zeroes only the slash
binding; entries keep binding from their own snapshot.
- Same-skill dominance: when a skill is both the slash source (fresh) and
a persisted entry (snapshot), only the activation-era declaration binds
— unioning both eras injected a secret the skill no longer declares.
Aligns with the tool-policy rule (slash activation dominates for the
run); different-skill entries still stack additively.
- The rendered 'Active skills' reminder (DurableContextMiddleware) no
longer advertises skills the provider denies — surface consistency with
describe_skill. Publication/consumption split: the activation
middleware publishes per-step path-keyed skill:activate decisions into
the run context (async: reused from the prepass snapshot, zero extra
I/O; sync: one scan + authorize() only when entries exist — passive
steps stay scan-free), the renderer consumes them and filters its
render copy (state untouched; unpublished = permissive, matching the
other run-context carriers). New carrier key listed in
REDACTED_CONTEXT_KEYS with a redaction regression.
- Wiring regressions on both chains for everything added above (user_id
to the stamp gate, skill_authorization + token pairing + activation-
before-durable ordering to the durable renderer): each mutation
verified red (M4-M13 across this change set).
- Tests 54->65: slash post-activation lookup, source independence,
dominance, sync-chain sync-API path, render filter (async/sync/
disabled with load-count assertions), chain wiring x2, redaction
listing. Plan notes record rounds 9-2..6 including the reference-point
post-mortem and the audit axes exhausted.
* fix(authz): anchor same-skill dominance on the slash identity, not binding results
Review round 10 (willem-bd) plus a self-audit extension the new checklist
items caught in the first fix itself:
- R10 (P2): the same-skill exclusion was derived from successfully BOUND
slash sources. When the prepass snapshots required-secrets [OLD_KEY]
and the operator strips the declaration during the await window, the
slash source binds nothing (required_secrets gate) — the exclusion set
stayed empty and the stale entry view kept injecting OLD_KEY for a
skill that now declares no secrets. The exclusion now derives from the
AUTHENTICATED slash activation identity (run-context source path,
owner-token checked): the identity's declared name resolves from the
fresh registry first, then the entry snapshot, and joins the exclusion
set regardless of binding success.
- Self-audit (value->gate->downstream on the fix itself): the declared
name can also change mid-run (same path, foo->bar). A name-anchored
exclusion then holds only the new name while the snapshot entry
resolves the same path to the old one — both eras bound again. The
PATH is the stable identity shared by the activation and every earlier
read: entries at the slash identity's path are now excluded before
registry resolution (exclude_paths), independent of what either
version calls the skill; the name-based set remains for cross-path
same-name shadowing.
Tests 65->67: no-secrets-era dominance (ACTIVE secrets None under the
OLD_KEY-to-empty interleaving), path-anchored dominance across a mid-run
rename (only the activation-era key binds). Mutations verified red:
identity-derived exclusion disabled (M14), path exclusion disabled
(M16).
* test(authz): pass explicit config to lead middleware wiring test
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(mcp): isolate personal connections from deployment configuration
* fix(mcp): preserve personal task ownership and repair plugin E2E fixtures
* fix(mcp): handle corrupt personal config and cache owned calls
* fix(mcp): pin personal endpoint addresses and handle invalid listings
* fix(mcp): recheck current authority for privileged personal connections
* fix(models): deep-merge when_thinking_enabled on the legacy path
The legacy thinking path (profiles without a `reasoning:` block, i.e. every
existing config.yaml) applied `when_thinking_enabled` with a shallow
`dict.update`, so a template such as `extra_body.thinking.type=enabled`
replaced the profile's whole `extra_body` and silently dropped sibling keys
like GLM's `extra_body.tool_stream` — but only while thinking was on. The
disable branch of the same function already deep-merged into `extra_body`,
and the contract path added in #5780 deep-merges through `_merge_settings`
for exactly this reason (its docstring names `tool_stream`). The on/off
asymmetry cannot be intended: the same profile sent `tool_stream` with
thinking off and lost it with thinking on.
Route the enable template through `_merge_settings` as well. Precedence is
unchanged (template values win on conflicts); the only observable
difference is that keys present only in the profile now survive a
thinking-on request. Everything else on the legacy path — the synthesized
`reasoning_effort=minimal`, the disable payloads, the supports_thinking
guard — is untouched.
Tests pin both directions for a base extra_body beside a thinking template,
recursive merging of nested chat_template_kwargs, and template precedence.
* docs(changelog): reference #5894 in the when_thinking_enabled entries
* fix(models): keep the template's vLLM switch spelling authoritative when merging
Review follow-up. Deep-merging the thinking template into a profile keeps
both spellings of vLLM's toggle when they differ — base
`chat_template_kwargs.enable_thinking: false` plus a template using the
legacy `thinking: true` alias — and VllmChatModel's normalizer maps the
alias onto `enable_thinking` only when the latter is absent, so the
profile's `false` won and thinking stayed off despite thinking_enabled=True.
The previous shallow replacement never hit this because it discarded the
base switch. The contract path had the same trap since it already
deep-merged, and both disable branches had the mirror image (a base
`enable_thinking: true` outlived a template `thinking: false`).
Route every thinking on/off merge through _merge_thinking_payload: it drops
the profile's *other* spelling of the switch before the deep merge, so the
payload's spelling decides in both directions on both paths, and unrelated
chat_template_kwargs still survive.
Tests cover both spelling combinations, both directions, and both paths,
asserting the switch as the provider will send it after normalization.
* fix(models): deep-merge the legacy disable template and mirror the vLLM switch instead of pruning
Review of the previous head found two regressions. The legacy
when_thinking_disabled branch was still dict.update-ed, so a profile that
declares both templates (the shape of most extra_body examples) kept
tool_stream while thinking was on and lost it while off. And the merge
pruned the profile's other vLLM switch spelling for every model class,
although only VllmChatModel maps `thinking` onto `enable_thinking`: on a
plain ChatOpenAI profile the prune removed the only key vLLM reads, so
thinking-off could no longer switch reasoning off.
Route when_thinking_disabled through _merge_thinking_payload too, and
replace the prune with mirroring: after the deep merge, the value the
payload declares for its switch spelling is copied onto the profile's
other spelling when the profile carries it. VllmChatModel collapses the
pair, plain OpenAI-compatible classes send both, and either way the
server reads the template's intent. No spelling is invented, a payload
that spells both is left alone, and non-mapping template values are
forwarded unchanged instead of raising. _VLLM_THINKING_SWITCHES is now
the single definition used by the disable, toggle and dialect helpers.
Docs state the merge semantics (keys are never removed; a template can
only add or override) and drop the "byte-for-byte" wording; tests cover
the disable template on/off, nested inheritance, non-mapping values, the
raw payload for a non-normalizing class, and the no-invention rule.
* fix(models): copy thinking templates on merge and document the never-remove rule
Review follow-up. Templates merged into the constructor settings were
stored by reference wherever the profile lacked the key, so the kwargs
handed to the model class aliased the cached ModelConfig; the mirroring
step now writes into that mapping, so make _merge_settings and
_deep_merge_dicts deep-copy every template mapping they store. Pinned at
every depth by a test that also checks the profile is unchanged.
Deep-merging the legacy when_thinking_disabled template means it can no
longer clear a key the profile's base extra_body sets: a base
extra_body.thinking {type: enabled, budget_tokens} reaches the provider
as {type: disabled, budget_tokens}, which Anthropic-style APIs reject.
The synthesized disable payloads and the contract path already behaved
this way. State the rule and the migration (enable-only keys belong in
when_thinking_enabled) in the changelog, models/AGENTS.md and
configuration.mdx, and pin it with a test for both shapes.
Route the legacy synthesized OpenAI disable through
_merge_thinking_payload too (identical output), narrow the docstring and
guidance wording about the native-Anthropic disable, and make the two
vLLM switch suites resolve the real VllmChatModel / ChatOpenAI classes
so a class-gated merge would fail the case it is named after.
* feat(agents): persist tool artifacts as resolvable handles (#4676)
MCP and sandbox tools return file paths, URLs, and task ids that are
lost when context compaction summarizes the conversation. This adds a
per-thread artifact handle registry so the model can reference
artifacts by short deterministic handles (art_xxxxxxxx) that survive
compaction and resolve to real references at the tool-call boundary.
- ThreadState.tool_artifacts: reducer channel capped at 100 entries,
same-handle latest-wins, survives summarization compaction
- artifact_registry: deterministic handle derivation from
(thread_id, tool_call_id, call_index); extraction precedence
structured_content -> file/image blocks -> conservative text scan;
HTML-escaped model-context rendering with char budget
- ArtifactCaptureMiddleware: before_model hook captures lightweight
metadata (never content) and tracks handle consumption
- ArtifactResolutionMiddleware: rewrites handles in tool-call args to
real references before execution; unknown handles untouched
- DurableContextMiddleware: projects the registry into the hidden
durable-context data block ([available]/[consumed] status)
- config: new tool_artifacts section (all switches default on)
- frontend: artifact badges on tool-call steps keyed by tool_call_id
Fixes#4676
* fix(agents): harden artifact registry after review (#4676)
Review findings on #4929:
- capture/consumption updates emitted in the same before_model call are
now concatenated instead of dict-merged; dict-merging clobbered the
shared tool_artifacts key and permanently dropped fresh captures
- handles embed a per-result ordinal so multiple references inside one
tool result never collide (two paths in one text block previously
collapsed into one handle)
- structured_content extraction pulls concrete refs from known keys
(file/path/url/task_id/..., string or list-of-string values) instead
of storing a truncated JSON preview, falls back to an untruncated
whole-object data entry, and no longer skips file/image content blocks
- plain-string tool results (e.g. bash/write_file output) are scanned
for sandbox paths / remote URLs too, gated by detect_refs_in_text
- wire max_entries through configure_tool_artifact_max_entries at
assembly (clamped 10..1000) and gate detect_refs_in_text scans
- resolution requires both enabled and resolve_handles_in_args,
matching its docs
- add real ToolCallRequest.override coverage and the spec 14.4
integration cycle (capture -> inject -> resolve -> consumption)
* fix(agents): address second-round artifact registry review (#4676)
Follow-up findings on #4929:
- drop generic `output`/`outputs` from structured ref keys: prose under
those keys no longer becomes a bogus file entry (falls back to the
whole-object data entry; path-shaped refs are still caught by the
free-text scan)
- strip closing quotes, brackets, and backticks from detected refs so
markdown- and JSON-wrapped paths resolve to clean real references
- remove the process-wide cap setter: the reducer enforces only an
absolute ceiling (1000) and ArtifactCaptureMiddleware applies the
configured max_entries per agent when emitting updates, so agents
with different configs in one process never share or flap a cap
- skip extraction for tool messages whose tool_call_id already has
registry entries and memoize known-empty results (bounded FIFO), so
steady-state capture cost drops to the new message tail instead of a
full-history rescan per model call
- correct the inverted ordering claim: capture is position-independent
in the wrap chain because it is a before_model hook reading state
messages; fixed the assembly comment, AGENTS.md item 15, and the
ordering test docstring
CI parity verified locally: ruff check + format --check, full offline
suite, blocking-io suite, collect-only, uv lock --check,
check_agent_guidance, frontend format/lint/typecheck/build/tests.
* fix(agents): sliding-window cap and URI gates for artifact registry (#4676)
Third-round review findings on #4929:
- max_entries is now a sliding retention window: fresh captures are
always registered, and when the projection exceeds the configured
cap the middleware appends a trailing {"op": "trim_to", "keep": N}
directive that the reducer applies by evicting the oldest entries.
The previous remaining-budget emission froze the registry at the cap,
permanently dropping post-compaction artifacts — the exact failure
mode #4676 targets — and silently disabled the round-2 rescan
optimizations at the point threads grew longest
- _track_consumption memoizes settled tool-call args in a bounded FIFO
alongside the existing empty-result memo, so historical AIMessage
args are not regex-rescanned every model call
- file/image content-block URLs must be http(s) or path-shaped;
data:/blob: URIs (MCP embedded resources carrying base64 payloads)
can no longer enter thread state, tool args, or crowd out the render
budget; protocol-relative //host forms are rejected as well
- structured-content ref values go through the same URI-shape gate as
content blocks, closing the embedded-payload hole via known keys such
as url/urls
- tool_artifacts.enabled now gates the whole middleware including
consumption tracking via an early return in before_model
Docs updated to describe the sliding window, URI gates, enabled scope,
and both bounded memos.
* fix(agents): final evictions and pending-aware consumption for artifact registry (#4676)
Fourth-round review findings on #4929:
- the sliding window interacted with the seen_call_ids skip to
resurrect evicted entries every round: once an entry was trimmed, its
tool_call_id left the seen set while its ToolMessage stayed in
context, so the next model call re-extracted it, re-registered it as
newest, and evicted something else — a never-converging rotation.
The empty-result memo is now a unified _processed_results memo that
records every extraction regardless of outcome, making evictions
final; steady state over a saturated thread is quiescent (pinned by
a multi-round test asserting no updates and zero extractions)
- the consumption scan built its handle map from pre-capture state, so
a call referencing a handle captured in the same before_model call
missed, got settled as quiet, and its consumption mark was lost
forever. _track_consumption now receives the pending capture entries
and resolves against state + pending; a call settles as quiet only
after every referenced handle resolved, so transient misses retry
next round instead of permanently suppressing the mark
AGENTS.md entry 15 updated to describe both memos and the
eviction-is-final semantics.
* fix(artifacts): address occurrence identity, PII and URL review findings
* fix(artifacts): enforce resolved policy checks and bounded registry handling
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: add optional role filtering to task history search
* docs: scope history role guidance to task continuity
* test: register task continuity guidance in scoped inventory
* docs: separate English guidance from Chinese README translation
* chore(version):update the version to 2.2.0-dev
* chore(doc):Updated the CHANGLOG with the 2.1.0 release version
* chore(doc): Updated the CHANGLOG for the latest changes
* chore(doc): Added the release SOP document
* feat(authz): add plugin resource targets, provenance, action and management enforcement
PR1 of the extension-tool-authorization series
(docs/plans/2026-09-24-extension-tool-authorization-spec.md): plugin
authorization plumbing, no new author-facing declaration.
- `authz/plugin_targets.py` is the single encoder/validator for composite
targets (`namespace/part`), with one validating composer behind three
kind-checked constructors; invalid input raises instead of degrading.
- `authz/plugin_authz.py` is the harness decision layer (`plugin_action`,
`plugin_management`, plus the batch filters the page slice will consume):
disabled is a no-op, an explicit deny raises, and provider errors follow
`fail_closed` like the sandbox gate.
- `POST /api/plugins/{ns}/actions/{name}` decides `plugin_action:invoke`
after the action resolves and before the body is streamed.
- `deerflow_extension_api.auth` publishes `require_plugin_management` /
`arequire_plugin_management` with two `app.state` resolver keys; the
Gateway installs both. Contract version 0.2.3 -> 0.2.4.
- `tools/tool_provenance.py` records host-written plugin provenance for
display (assembly descriptor, Layer-2 context later); no decision reads a
label.
- Plugin providers use a `(config signature, loop key)` cache so a
loop-affine provider is never handed to another loop or to a sync caller.
- RBAC gains `plugin_actions` / `plugin_management` aliases;
`config_version` 47 -> 48 across the three mirrors.
Tests: new plugin target/provenance/action/management suites plus a
blocking-IO anchor whose offloads were reverted to prove red->green.
* fix(authz): decide plugin resources from the request's config snapshot
Review round 1 on PR #5842 (willem-bd).
P1 — the decision layer no longer reloads the configuration. The six
`authz/plugin_authz.py` functions take a required `app_config` snapshot (the
same request-scoped config the Gateway used to resolve the provider), and
`resolve_/aresolve_plugin_authorization` return it as a third element, so
`enabled`/`fail_closed` can never come from a second read that drifted from
the one that produced the provider. A snapshot the caller could not read at
all is *unavailable*, not disabled: it fails closed as
`authz.config_unavailable`. The Gateway's `_plugin_app_config` now separates
an absent `config.yaml` (no gate, today's behavior) from a config that exists
but cannot be read or validated (deny) instead of swallowing both through
`safe_app_config`.
P2 — `_validated_decision` requires `type(decision.allow) is bool` on both
the sync `authorize` and async `aauthorize` paths, so a provider returning
`AuthzDecision(allow="false")` is a malformed verdict that follows
`fail_closed` instead of being read as an allow.
Both fixes are covered by regression tests that were verified red against the
pre-fix behavior (P1: 200 vs 403 and resolver True vs False; P2: 12 cases
across fail_closed/fail_open and sync/async), plus tests pinning that an
absent config and `authorization.enabled: false` keep today's behavior.
* fix(authz): treat a request-time config absence as unavailable
Review round 2 on PR #5842 (willem-bd).
P1 — `_plugin_app_config[_async]` no longer maps a missing `config.yaml` to
`None` ("no gate"). A Gateway cannot have started without a readable config
(`app/gateway/app.py` reads it through this same accessor in `lifespan`), so
absence *during a request* means the configuration this process was running on
became unavailable — a hot reload, an atomic replace, a ConfigMap remount. It is
unavailable, not disabled: the error propagates,
`resolve_/aresolve_plugin_authorization` raise
`_PluginAuthorizationUnavailable(fail_closed=True)`, the action dispatcher
returns 403 and the management resolvers answer `False`. The dead
`(None, None, None)` allow path is gone, the resolvers return a non-optional
`AppConfig` as their third element, and callers key "authorization disabled" on
`provider is None` alone. This matches the rule the decision layer already
documented (`authz.config_unavailable`) and the public docs.
P2 — `_deny_reason_code` is total: a `reasons` that is not iterable degrades the
code to `authz.denied` instead of raising `TypeError` outside the guarded
provider path, which turned a denial into an unhandled error. It is deliberately
not folded into `_validated_decision`: `allow` was a real `bool`, so the verdict
is a valid denial, and treating it as malformed would flip it to an allow under
`fail_closed: false`.
Both fixes are covered by regression tests verified red against the pre-fix
behavior (P1: resolver True vs False, async `PermissionError` not raised, route
200 vs 403; P2: `TypeError: 'NoneType' object is not iterable` at
`plugin_authz.py:113`). The action-route tests in
`tests/test_plugin_contributions.py` now install a readable
`authorization.enabled: false` config instead of relying on a config-less
process, and `docs/full-stack-plugins.md` plus the PR1 decision log record the
new rule.
* fix(authz): apply the running policy when the plugin config is unreadable
Follow-up to f03d2ea9 (review round 2 on PR #5842, willem-bd).
The first attempt read the P1 finding as "absence is never disabled" and failed
every config read closed unconditionally. That is too broad for this repo: a
config-less host is deliberately treated as having no policy to apply
(`_get_route_authorization_config` falls back to a disabled config,
`sandbox_authz.safe_app_config` documents the same no-op for "CI runners and
direct-call tests"), and the rule turned the config-less e2e plugin host
(`extension_test_fixtures.bookmark_plugin_gateway`) into a 403 — the three
`bookmark-plugin.spec.ts` cases lost the save action. Reproduced locally:
`POST /api/plugins/community.bookmarks/actions/save` against that fixture host
returned 403 before, 200 after this change.
The rule is now "has this process loaded a configuration?", not "was the last
read successful?":
- No loaded config (`peek_loaded_app_config()`, new read-only accessor on
`deerflow.config.app_config`) and no `config.yaml`: no policy to apply, so the
gate stays a no-op — unchanged for CI runners, direct-call tests and hosts
that mount only the plugins router.
- A loaded config whose file disappears or cannot be read: *unavailable*, never
*disabled*. The resolvers raise `_PluginAuthorizationUnavailable` carrying the
failure flag of the configuration this process runs on (authorization
disabled there -> no gate; enabled -> that policy's `fail_closed`, `True` by
default), so an enabled Gateway answers 403/False in the window instead of
letting protected plugin actions and management routes through.
- A config file that exists but cannot be read or validated right now: fail
closed when nothing is loaded (round 1's rule), otherwise the loaded policy
decides.
`peek_loaded_app_config()` is preferred over a Gateway-side startup snapshot: it
reflects the last successfully loaded config, so a hot-reloaded policy is what
applies during the window, and there is no lifecycle hook to keep in sync.
Evidence: the three no-config/lost-config branches in
`tests/test_plugin_management_guard.py` and
`tests/test_plugin_action_authorization.py`, round 1's unreadable-config tests
unchanged, `test_peek_loaded_app_config_survives_a_missing_file` in
`tests/test_app_config_reload.py`, and the fixture-host probe above. Docs
(`docs/full-stack-plugins.md`, `reference.mdx` en/zh) and the PR1 decision log
state the final rule; the log records the superseded variant.
* fix(authz): keep a denial whose reasons raise while being read
* fix(authz): intersect plugin batch filters with the candidates
The plugin_page/plugin_management batch helpers returned the provider's
`filter_resources` answer verbatim, so a custom provider could hand the
projections a target that was never asked about ("*", another namespace,
a phantom page). Every other filter call site re-projects from its own
candidate list; do the same here, state the intersection on the provider
contract, and pin it with a regression test that fails on the old
pass-through.
* docs(plugins): map a management denial to 403 in the route example
The contributed-route example called `arequire_plugin_management` and let
the documented `PermissionError` escape. The Gateway installs no
`PermissionError` handler, so a route copied from the example answered 500
on a normal policy denial instead of 403. Both example routes now translate
the denial, and the surrounding bullets say why the translation is the
route's job.
* docs(extensions): point the 0.2.4 contract row at the PR that ships it
Every other row in the version-history table links the pull request that
shipped the version; 0.2.4 linked the authorization tracking issue instead.
Both the English and Chinese tables now link this PR.
* feat(models): add declarative reasoning capability contract (#5073)
Introduce an optional per-model `reasoning:` block beside the legacy
`supports_thinking` / `supports_reasoning_effort` booleans. It declares
thinking availability (unsupported | optional | required), the accepted
effort values with aliases, default, and serialization path, the payload
dialect, and the reasoning-history requirement. When present the booleans
are derived from it and contradictory profiles fail at config load.
`deerflow.models.reasoning` normalizes legacy and declared profiles into
one immutable contract and resolves every caller's generic request against
it. `create_chat_model` is the single enforcement point: required-thinking
models never receive a synthesized disable payload, unsupported effort
values never reach the provider, and the dialect synthesizes the on/off
payload when no template exists. Legacy profiles keep the historical path
byte-for-byte. The lead agent and subagent descriptor resolve the same
policy first so run metadata reports effective values.
`/api/models` and `DeerFlowClient` project the contract as `reasoning`;
the frontend derives mode clamping and effort choices from it through
`core/models/reasoning.ts`, and user preferences accept provider-specific
effort tokens. The setup wizard's Z.AI GLM-5.3-Flash profile migrates to
the contract, restoring its low/high/max effort control.
* docs(models): move the reasoning contract guidance into models/AGENTS.md
backend/AGENTS.md sits within 46 bytes of the 96 KiB effective-chain hard
limit for the middlewares, runtime, sandbox, and subagents guides, so the
new section pushed scripts/check_agent_guidance.py over it. The models
guide is the nearest file for the factory and the contract module, and
its chain has ample room.
* fix(models): address review on the reasoning contract
- vLLM dialect: the enable path now mirrors whichever chat_template_kwargs
switch the template declares, like the disable path already did, so a
template using the older `thinking` key no longer also grows
`enable_thinking`.
- Validate operator-supplied effort values at `effort.path` inside
`when_thinking_enabled`, `when_thinking_disabled`, and the `thinking`
shortcut against the contract, not only the profile-level value; they are
forwarded when the caller chooses nothing, so the "unknown values never
reach the provider" guarantee now holds for them too.
- Constrain `effort.path` to dotted identifiers and reject single-segment
paths that would replace a whole mapping (`extra_body`, `thinking`,
`model_kwargs`, `default_headers`, `default_query`).
- Z.AI GLM-5.3-Flash profile: `default: high` instead of `max`, because the
default also governs summarization, title generation, and subagent calls
that never choose an effort; `max` stays selectable. Documented in the
config example, the configuration manual, and the design note.
* fix(models): close reasoning contract review gaps
* fix(models): preserve native Ollama reasoning config
* fix(models): keep native reasoning strings and project reasoning into the descriptor
- `ModelConfig.reasoning` also accepts the level string ChatOllama takes
for gpt-oss style models (`low` / `medium` / `high`); the factory forwards
both the boolean and the string on the legacy path, so profiles that
passed `reasoning: high` through before the contract field existed keep
loading and keep sending it.
- The assembly descriptor no longer excludes `reasoning` from
`model_parameters`: the native ChatOllama value and a declared contract's
dialect/history/effort change the request payload, so they now move the
assembly fingerprint.
- e2e: the sidecar spec pinned the pre-contract body where a model without
effort control still received `reasoning_effort: minimal`; the preset is
now dropped for such models (the Gateway discarded it anyway), so the
assertion expects the key to be absent.
* feat(examples): add a standalone text classification plugin
A deployment-installed package contributes one model tool, classify_texts,
and a status action through the full-stack plugin contract. The tool labels a
list of texts against caller-supplied categories with a deployment-configured
backend: Jev (one choice question per item over a shared batch state, checked
per item) or an OpenAI-compatible chat endpoint (a JSON label list validated
in code). Items are packed into requests under a size limit and sent with
bounded concurrency; every failure is explicit per call or per item;
credentials come only from deployment-named environment variables; item text
never enters instructions, question keys, results or errors, and the served
model name is kept to a short identifier. Each batch runs under an asyncio
timeout bounded by the call deadline, so the whole call stays inside the
host's 30-second limit, and ids and category names are bounded in bytes so
300 results fit the host's 64 KiB output limit.
* docs(examples): add an opt-in live smoke test for the text classification plugin
The script loads the package through the real extension path and classifies 20
synthetic English and Chinese sentences plus a blank and an over-long item with
the configured backend, printing labels and statuses only. Against real Jev it
agreed on all 20 labels in two requests and 1.6 seconds, with the served model
reported as jev-1.13.0.
* fix(examples): measure request size as sent and cover transport failures
httpx serializes json= bodies with the json.dumps default, escaping every
non-ASCII character as six bytes, while the size estimators measured raw
UTF-8, so CJK batches could reach about twice the 256 KiB limit. The
estimators now measure the same encoding; a test with CJK categories and
texts checks the bytes the transport actually received. The network status
gets a test through connect and protocol errors, and the deadline test's
wall-clock slack is widened for loaded runners.
* fix(examples): match classifier request wire encoding
* fix(examples): bound classification output keys by JSON size
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* docs(extensions): add an eleven-page extension developer manual
Add a harness/extensions/ section in English and Chinese documenting the
deerflow-extension-api 0.2.3 contract:
index when to write an extension vs a tool, MCP server or
skill; contribution kinds; loading; failure and
trust model; versioning
quick-start build, test, install and remove a working extension
runtime load sequence, API version rules, scopes and
ExtensionData, fail-open and cancellation, diagnostics
middleware placements and where each lands, scope, ordering,
observe-only wrap hooks, failure isolation
observers task lifecycle, system model calls, agent assembly
and fingerprint, context compaction
services-and-routes service lifecycle, router rejection rules, auth,
principals
run-evidence service and request-scoped readers (#5727), cursor
semantics, deletions, redaction, store differences
plugins experimental full-stack plugins, inline modules and
manifest-based browser assets (#5685)
operations extension manager CLI, sources, upgrade, rollback,
Docker, Helm, recovery
troubleshooting indexed by the exact log and error strings
reference every public name and the contract version history
Also fix stale descriptions: the contribution kinds listed in AGENTS.md
and deerflow/extensions/AGENTS.md (now eight, including plugins), the
run-evidence redaction claim (only auth_token is removed), and the plugin
mount return value in docs/full-stack-plugins.md ({ dispose }, not a
function). Changelog entries added in both languages.
* docs(agents): trim root AGENTS.md to keep inherited chains under the size limit
* docs(extensions): scope the manual to the 0.2.1 contract
Plugins and request-scoped run evidence are not in 2.1.x-dev yet; they
move to a stacked follow-up so this change can be cherry-picked.
* docs(extensions): cover full-stack plugins and request-scoped run evidence
Brings the extension manual up to deerflow-extension-api 0.2.3. Stacked on
the 0.2.1 manual so that part can be cherry-picked to 2.1.x-dev alone.
* docs(extensions): clarify PAT rejection on contributed routes
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(knowledge): add verifiable RAGFlow source citations
* docs(knowledge): scope RAGFlow guidance to its own directory
* fix(knowledge): preserve citations through rendering and budgets
* feat(authz): filter per-caller skill visibility on the listing surfaces (#4063 Phase 4)
GET /api/skills, GET /api/skills/custom, and GET /api/skills/{name} now
filter the user-scoped catalog through filter_resources(principal,
"skill", ...) — mirroring list_models. Anonymous callers are unfiltered;
provider errors follow authorization.fail_closed (fail-closed -> empty
listing / 404, fail-open -> full listing). An invisible skill on the
detail surface returns the standard 404 so the endpoint cannot become an
existence oracle the filtered list closed. Management endpoints stay
require_admin_user-gated; runtime activation is #4541's layer.
resolve_skill_authorization joins resolve_model_authorization as a thin
sibling over a shared _resolve_route_scoped_authorization core.
* docs(authz): reflect per-caller skill visibility in OpenAPI metadata and implementation notes (#5489)
Address the two non-blocking review findings on #5489:
- The three user-facing GET routes (/skills, /skills/custom,
/skills/{name}) now say in their /docs-visible descriptions that
authorization filters the response (hidden skills 404 on detail).
- Add the dated Phase 4 decision-log entry to the authorization
implementation notes, per the convention of every prior merged
authz PR: listing-visibility semantics, the 404-vs-403
existence-oracle rationale, anonymous-caller behavior, and the
#4541 rebase reconciliation points (config.example.yaml roles
comment + this file's decision log).
* docs(authz): move route guidance into Gateway module guide
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash
Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md
(issue #5160, tracker #5129) in the slice order of the spec's §16.
Slices:
- A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY
admission pinning (both server-owned sets + worker hoist); latest-only
request-scoped <project> block via DynamicContextMiddleware
wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID
prefix + marker + provenance, never persisted); journal audit
fingerprints; Instructions tab.
- B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with
locked check-and-set; hash-qualified immutable shelf storage with
Paths helpers; upload/list/content/delete-to-trash routes; project
delete trashes the shelf in-transaction; request-scoped bounded
<documents> index with honest count/shown + actionable overflow note;
list_project_documents/read_project_document tools registered only on
pinned runs; PAT allowlist + drift guards; blocking-IO anchors.
- C: shared thread-upload ingestion service (uploads router refactored to
parity); POST from-thread with provenance; attach-to-thread with
lock-staged copy (archived source allowed); read-only thread-files
view with per-group truncation reporting.
- D: restore (restored/merged/not_found/no_target/content_missing; no
file moves), purge (continuous row lock across unlink/delete/commit,
retryable on FS errors), retention sweep (lazy + startup, 24h orphan
guard, row-side reconciliation never deletes).
- E: Documents tab (shelf + conversation-files browser, provenance,
archived banner, content-missing rows), /workspace/trash route,
sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks
+ specs.
Review hardening folded in (10 rounds, all with tests):
- force active shelf content (HTML/XML family) to download; nosniff on
artifact + content responses; unified unsandboxed-iframe PDF preview
(fixes the pre-existing Chromium sandbox blank in the artifact viewer)
- scope document trash to the URL project under the document lock
- atomic no-overwrite filename reservation for ALL ingestion (seeded
claims + os.link commit with suffix retry; same-name re-upload now
unique-names instead of replacing); hidden staging only, no visible
placeholders; lease cleanup on setup failure
- serialize conversion under the document lock with post-lock active
revalidation; drain locked filesystem work on cancellation; preserve
bytes when an insert's commit state is uncertain (including trashed
rows)
- original-integrity checks before serving text or cached conversions;
content_missing surfaced in list responses (UI reads the flag, no
409-probe); downloads always serve original bytes
- bounded streaming document reads with cached char counts; shelf limits
declared in middleware release identity
- thread-root confinement for from-thread sources; config fallback
rejects fractional/infinite values; composer counts staged
attachments; pending attachments persist until submission or removal;
in-flight instruction/rename edits survive save refetches; shelf and
trash pagination; conversation-file and thread-files pages stay
subscribed to refetches
Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md
contracts, config.example.yaml projects block.
Review follow-ups (head b4807477 → this revision):
- The trash retention sweep is split so repeated lazy triggers stay
bounded: the indexed expiry purge still runs on every trigger
(GET /api/trash/documents, POST /api/trash/purge) while the
O(all rows + all files) reconciliation is throttled to one run per
user per 15 minutes (process-local, per-user window). The startup
sweep now runs as a background task instead of blocking gateway
readiness, and shutdown awaits it (bounded).
- The export scrub (stripInternalMarkers) is fence- and indentation-aware
like the render path, so a pasted, fenced <project>/<documents> snippet
survives markdown export while real injected blocks (never fenced) are
still removed. Fence regexes moved to a dependency-free leaf module to
avoid the messages↔streamdown import cycle.
- The artifact viewer's PDF iframe no longer carries an added title
attribute (the upstream e2e contract locates it via :not([title])), and
the upstream artifact-preview spec now pins the new contract: PDFs
render unsandboxed, images keep sandbox="".
* fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title
- Shutdown cancelled only the shield around the background startup sweep,
so an all-users reconciliation that outlived the 5s budget kept walking
rows and files while the document repo and DB engine were disposed
underneath it. The wait now lives in `_shutdown_startup_trash_sweep`,
which cancels the task and drains it before worker exit: the shield
keeps the wait bounded, the cancel makes it final (CancelledError lands
at the sweep's next await, and `_run_startup_trash_sweep` only catches
`Exception`, so nothing swallows it).
- The browser-preview iframe lost `title={getFileName(filepath)}` in the
previous fix round, leaving the PDF frame without an accessible name
while its siblings keep theirs. Restore it (WCAG frame titles), assert
it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]`
instead of `iframe:not([title])`.
* fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel
`Task.cancel()` returns False when the sweep already finished inside the
window between the deadline firing and the cancel, so the shutdown log
claimed a cancellation that never happened. Branch on that outcome: the
warning stays for a real cancel, a late finish is logged at info, and both
paths still reap the task before worker exit.
* fix(projects): round-4 review — make Empty trash delete what it confirms
`POST /api/trash/purge` only ran the retention sweep, and the sweep's
candidate selection is age-gated, so a freshly trashed document survived
"Empty trash" even though the confirmation promises that every listed
document is permanently deleted. With one trashed row the route answered
`{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps
expired rows before listing, so the visible rows were normally ineligible
for the action by construction.
Empty trash now drives `purge_all_trashed`: the caller's trashed rows
(`list_all_trashed`, no age filter) each go through the same guarded,
row-locked `purge` as the single-document delete — bytes first, then the
row, in one transaction — so a row restored mid-flight is skipped instead of
force-deleted, and an unlink failure rolls that row back and answers 500 with
a retryable message. Retention expiry stays where it was: the sweep's
`purge_candidates` is now the only age-gated selection, and the lazy
retention sweep still runs on the listing and at startup.
Tests: the router suite replaces the retention-gated expectation with the
reviewer's repro (fresh row purged, bytes unlinked, shelf and other users'
trash untouched, a failing unlink stays retryable and 500); a blocking-I/O
anchor drives the new entry point through the offload; the mocked e2e covers
the action end to end; a new real-backend spec performs it against the real
gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE
and the phase-2 design docs (en+zh) state the age-independent contract.
* feat: add opt-in task notes and compacted history recall
* fix: validate task continuity state and preserve user answers
Honor explicit opt-out, preserve clarification replies and capture failure statuses, validate notebook writes, and clear branch archive references. Update the config version and audit optional LLM credentials, with regression and integration evidence.
* fix: align Helm config version with task continuity schema
* fix: preserve mixed task history and declare continuity policies
* fix: recover malformed history and evict archives atomically
* fix(gateway): gate github_token and disable_clarification on internal callers
`non_interactive` is honored only for internally-authenticated callers
because it strips `ask_clarification` from the lead-agent toolset. The two
sibling run-context keys reproduced that effect without the gate.
`merge_run_context_overrides` forwarded `_CONTEXT_RUNTIME_ONLY_KEYS`
regardless of `internal`, and `strip_internal_context_keys` scrubbed only
`_CONTEXT_INTERNAL_CALLER_KEYS` -- so any session or PAT caller could set
`disable_clarification` through `body.context`, or through the free-form
`body.config` that `build_run_config` copies verbatim. That is not a milder
flag than `non_interactive`: ClarificationMiddleware answers every
clarification -- `risk_confirmation` included -- with "proceed without
asking" instead of interrupting, and SandboxMiddleware reads the two keys as
the same non-interactive signal. `github_token` rode the same path into
`runtime.context`, where the bash tool exports it as `GH_TOKEN`/`GITHUB_TOKEN`,
and a copy smuggled through `body.config['configurable']` reached the
checkpoint store the context-only rule exists to avoid.
Both keys are produced server-side by the channel run policies, which reach
the Gateway over the internally-authenticated request channel, so gate them
the same way: forward them only when `internal=True`, and scrub the union
`_INTERNAL_ONLY_CONTEXT_KEYS` from both config sections for every other
caller. Destination stays an orthogonal axis -- `_CONTEXT_RUNTIME_ONLY_KEYS`
still land in `context` alone, never in checkpoint-persisted `configurable`.
Regression coverage in tests/test_gateway_services.py pins both smuggling
surfaces and replays the real start_run assembly order for a session caller
and for an internal one, so the GitHub channel keeps carrying its minted
token.
* docs(changelog): record the internal-only run-context key gate (#5338)
* docs(agents): keep the run-context note inside the AGENTS.md budgets
The AG002 inherited-chain check failed at this head. The new backend section
and the root scheduled-task sentence added 993 B to the root and backend
guidance both the sandbox and middlewares chains inherit, pushing sandbox 6 B
over the 98304 B hard limit and growing the middlewares chain, which main
already exceeds by 155 B. An already-over chain is only tolerated while it does
not grow, so the shared ancestors had to come back to their base size.
Condensed the new material and removed prose the root file was duplicating:
- The trust-boundary section keeps both gated surfaces, both helpers, the
trust-vs-destination split, and the disable_clarification note in half the
space.
- The root scheduled-task bullet names all three internal-only keys and both
smuggling surfaces while staying under its previous size.
- Dropped the root `scheduler.recursion_limit` bullet, which restated
backend/AGENTS.md:18 almost verbatim; its one unique fact (a YAML edit needs
no Gateway restart) moved to that bullet.
- Deduplicated the nginx routing sentence, which already deferred to the
backend routing table, and tightened the waiver note's sequencing tail.
Root and backend guidance now sit 50 B under their combined base size, so the
sandbox chain returns to 97310 B and the middlewares chain no longer grows.
Every file stays under its AG001 soft budget.
