* feat(channels): share encrypted runtime channel config across Gateway instances
IM bot credentials entered in the web UI lived in each host's plaintext
channels/runtime-config.json, and the router cached the merged config on
app.state for the life of the process. With several Gateway instances a
save or disconnect served by an instance that does not hold the
provider's leader lease never reached the transport holder: it kept the
old credentials until it next started the channel, and a disconnect
could leave the old bot connected.
- New shared table channel_runtime_configs (migration 0044): one row per
provider with the config block encrypted under DEER_FLOW_CREDENTIALS_KEY,
a plaintext disconnect tombstone, and a version bumped by every save.
- SqlChannelRuntimeConfigStore replaces the JSON file for sqlite/postgres
(memory keeps the file); it is synchronous on app_sync_sqlalchemy_url
because its callers already run in worker threads.
- The router re-reads the store on every request instead of caching the
merged config, and refuses (503) to save credentials without a key.
- The leader-lease supervisor compares the versions on each renewal tick
and restarts or stops the holder's transport when a provider's config
changed; a takeover reads a disconnect instead of reviving cached
credentials.
- The first start imports an existing runtime-config.json (rows already
in the table win) and deletes the plaintext file.
* docs(changelog): reference #6988 for the shared runtime channel config
* ci: rerun checks after #6999 fixed the frontend guidance budget on main
* fix(channels): start a held transport that is down when its runtime config changes
When the lease holder's transport was down (a start that failed on the
old credentials), applying a peer-saved config only updated the cached
config, and the transport started on the next lease round: two renewal
ticks instead of the one the docs promise. The apply step now starts it
on the held lease right away, with the lease round's backoff bookkeeping.
* feat(channels): generalize IM group targets and add task/goal run notifications
- Generalize proactive group notification capability (proactive_group_notifications), binding validation, and scheduled task group selection across Feishu, QQ, and WeCom while keeping WeChat personal-only.
- Add QQ group active notification delivery via POST /v2/groups/{group_openid}/messages (msg_type=2) and WeCom group active notification delivery via WS Client.sendMessage(chatid, body) with bot_id:group:chatid workspace isolation.
- Generalize notification outbox sources via migration 0043_runtime_im_notices (source_kind, source_key, run_id, nullable task_id/task_run_id) with backward-compatible SQLite/PostgreSQL upgrades.
- Add opt-in ordinary task & goal run IM notifications (run_completed, run_failed, human_input_requested) on the shared outbox, delivery worker, renderer, and settings UI.
* fix(notifications): address review feedback and group/runtime notification edge cases
* test(migrations): update CURRENT_HEAD to 0043_runtime_im_notices across migration chain tests
* fix(channels): preserve group binding names and legacy authorization
Carry owner-chosen group names through existing connect state, keep private bindings out of legacy group fallback, and restore completion metadata retries. Cover real binding and worker finalization paths and synchronize notification docs.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): keep IM runs alive when the creating request disconnects
The channel manager holds a `runs.wait` / `runs.stream` request open for
the whole run, and that request dies with the Gateway instance that
received the IM message: a leader hand-off, a rolling restart, a crash.
The Gateway cancels a run whose creating request disconnects unless the
request asked for `on_disconnect: continue`, so every IM run in flight
was cancelled on failover.
Channel runs now ask for `continue`, like scheduled-task and MCP-task
runs already do. The run finishes on the Gateway executing it and its
thread state is persisted, so the answer stays readable in the web UI
even though the IM reply is lost with the old transport. The key is set
after the fire-and-forget branch because the SDK's `runs.create` has no
`on_disconnect` parameter.
Docs: CONFIGURATION.md and the Helm/deployment guides no longer list
"failover cancels IM runs" as a known limit; IM_CHANNEL_CONNECTIONS.md
and the channels AGENTS.md describe the policy.
* docs(changelog): link #6876
* fix(channels): keep the inbound dedupe key once a run may be executing
With `on_disconnect: continue` a run outlives the manager worker that
requested it, but the worker loop released the inbound dedupe key on
cancellation, so a provider redelivery on the replacement instance
admitted the same event again and executed the turn twice (review on
#6876).
Each inbound message now carries an admission record: `requested`
flips before the run request leaves the manager, `confirmed` once the
Gateway proved it admitted the run (a run id, a stream frame, a
completed response). The generic release paths (worker cancellation,
worker/handler exceptions, a broken stream) keep the key when the run
may continue: after confirmation, on cancellation after the request
was sent, and on post-request transport failures (read timeout, read
error, protocol error). Pre-request failures, Gateway rejections
(busy thread, stale pin, status errors, connection failures) and the
stream's terminal `error` frame still release it so the redelivery
can retry.
* feat(channels): arbitrate IM transports with a per-provider leader lease
Every Gateway instance connected every enabled IM platform with the same
bot credentials, so channels could not run on more than one instance
(Telegram 409 storms, Discord answering twice, scattered per-process
state). channels.leader_lease was reserved by #6746; this implements it.
- channel_leases table (migration 0041): one row per provider. Acquire is
one dialect-native upsert decided against the database clock and bumps
a fencing epoch; renew extends only a live lease at the holder's epoch;
release sets the expiry to 0 so a peer takes over on its next tick.
- app/channels/leadership.py: the holder's local monotonic deadline
(send time of the last confirmed renewal + lease_seconds); holds() turns
false one renewal interval early, so the transport stops at least
grace_seconds before a peer may connect. DB calls are bounded.
- ChannelService: start fence in _start_channel (no DB call) behind every
start path; fixed-rate supervisor renews/acquires and runs transport
starts/stops as per-provider tasks; takeover reloads the provider's
saved config; failed starts back off; shutdown releases only leases
whose transport stopped. get_status reports leader/leader_here and the
holder's transport state on every instance.
- Notification worker leaves rows of providers another instance holds.
- Gate: the enabled=false refusal now recommends auto/true; an explicit
true with database.backend memory is refused when a channel is enabled.
- Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated.
* docs(changelog): link the channel leader lease entry to #6843
* fix(channels): see peer-held leases for providers absent from local config
Review on #6843: a provider connected from the web UI on a peer never
enters this instance's channel config, so the notification worker here
still claimed its rows (and parked them for 900s) and GET /api/channels/
reported no leader for it. Every supervision round, and startup, now also
reads the whole lease table once (bounded), and notification exclusions
and status use those observations: a provider whose live lease a peer
holds is left to it even without local config.
Running that read concurrently with renewals exposed that, through
SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on
another connection can hand back one of its own bind values instead of
the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do
not show it). A lease must never take such a row for a confirmed
renewal, so acquire/renew/release now decide by affected-row count and
acquire reads its epoch back in the same transaction.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): control IM notifications and add WeChat/QQ delivery
* fix(scheduler): skip notification preference locks without recipients
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(artifacts): preserve account-scoped drafts across navigation
* docs(artifacts): keep draft lifecycle guidance within its budget
* fix(artifacts): retain drafts through unknown auth and isolate typing updates
* fix(artifacts): preserve drafts when visiting public showcases
* fix(artifacts): preserve edits after pending save
* fix(extensions): isolate team workers and rotate admissions
* fix(extensions): index and batch pending team snapshots
* fix(agent-teams): retain unfinished owner rotation rounds
Carry the remaining owner queue across planning passes, including exhausted owners and newcomers. Read legacy rows completely before updating their pending projection and distinguish dispatch reservations from sorted entries.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): add channels.leader_lease and gate it under multi-process signals
IM channels other than GitHub connect outbound to their platform. With more
than one Gateway instance each one opens that connection with the same bot
credentials: Telegram rejects the second poller with 409, Discord answers every
message twice and per-process ChannelManager state diverges. The multi-process
startup gate said nothing about channels; the only guard was a comment in the
Helm values.
Add the `channels.leader_lease` section (`enabled: auto|true|false`,
`lease_seconds`, `grace_seconds`; startup-only, registered in the reload
boundary) that the per-provider leader lease will read, validated by a Pydantic
model from the non-schema `channels` mapping. The gate now refuses
`enabled: false` under a multi-process signal while a non-GitHub provider is
enabled under `channels.*` or connectable through `channel_connections`, naming
the providers and the rollback; a malformed block is refused on every topology.
`auto` follows the multi-process signal, so a single instance is unchanged.
ChannelService strips the block before its per-provider loops and exposes it
as `leader_lease`; `ensure_ready_channels` only visits registered providers.
The two-Gateway harness overlay leaves the block alone. config_version 60 in
the example, the Helm chart and its README; docs replace the comment-only
"keep 1 replica" guidance with the gate.
* docs(changelog): link #6746
* fix(channels): stop recommending the unimplemented leader lease as a remedy
Review on #6746: the refusal told operators to switch channels.leader_lease
from false to auto/true "so one instance owns each provider's connection", but
the transport arbitration ships separately, so that change only bypassed the
clause and every worker still started the same bot.
The refusal now offers only the remedies that work today (a single instance, or
disabling the channels) and states that auto/true are reserved for the lease
that ships separately. A regression helper in the gate tests asserts the
remedies start with the rollback and that auto/true are never offered. The
config example, Helm values/README, CONFIGURATION.md, README, the deployment
guide (en/zh), the model docstrings and both changelogs describe auto/true as
reserved settings rather than an ownership guarantee.
* fix(channels): warn on an enabled unknown channel key; reword the changelog entry
Review on #6746: the registry guard in ensure_ready_channels skipped a
misspelled provider key (channels.telegam) silently, where it previously
reached _start_channel and logged 'Unknown channel type'. An enabled dict
block that is neither a registered provider nor leader_lease now logs a
warning that lists the registry.
The changelog entry claimed the chart and docs drop the "keep 1 replica"
guidance; they keep it (auto/true do not make several replicas safe yet) and
only stop relying on the comment alone.
* fix(dev): disable UI channel providers in multi-instance overlay
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): add example guide and packaging checks
* fix(extensions): align packaging checks with installer
Reject Windows drive-qualified wheel paths before extraction and reuse the installer entry-point validator. Cover unsafe paths, dotted attributes, and valid dotted modules in regression tests.
* fix(extensions): skip unused wheel extraction
Only extract wheels when browser manifests need validation, with a backend-only regression test. Link the three remaining example READMEs in the developer guide.
* docs(frontend): trim guidance intro below size budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(client): convert the captured upload source
* test(uploads): cover ACL sources and scope capture failures
* fix(uploads): preserve concurrent uploads from one captured source
Combine the embedded client's source capture with no-replace publication for
first-time filenames and Markdown companions. Hold one private capture across
collision retries, retain original descriptor metadata, and keep replacement
behavior for names present when the call starts.
Port concurrent-upload and staging-publication regressions from PR #6687.
Extend source-edit/removal and real XLSX cases to both publication paths, and
cover edits or removal during two filename collisions in ordinary and event-loop
calls. Document capture, retry, publication, and cleanup contracts.
Validation: related suites 469 passed, 24 skipped; full offline suite 28,788
passed, 488 skipped; blocking-I/O suite 245 passed, 1 skipped; Ruff, guidance,
and diff checks passed. Nine race/collision cases failed on the #6632 baseline.
Co-authored-by: haotianqi666 <250384991+haotianqi666@users.noreply.github.com>
* docs: keep agent guidance within CI budgets
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: haotianqi666 <250384991+haotianqi666@users.noreply.github.com>
* fix(frontend): keep the thread list offset past hidden side chats
Primary thread lists hide sidecar threads on the client, so a page of 50
visible chats can consume more server rows. fetchInfiniteThreadsPage kept
that server offset in a Symbol property on the page array, but TanStack
Query's structural sharing (replaceEqualDeep) rebuilds a changed array
without symbol keys, and the list cache writers (rename, unread, pin,
delete, upsert) rebuild pages with map/filter. After the first refetch or
cache write, getInfiniteThreadsNextPageParam fell back to the sum of the
visible page lengths, so the next page started at 50 instead of 53 and
re-fetched rows already loaded. The same fallback treated a full page
that lost a row to a local delete as the end of the list.
Each cached page is now { threads, hiddenCount, hasMore }. The next offset
is the sum of visible and hidden rows over the loaded pages, so local
deletes and inserts still move it with the server list, and the end of the
list comes from the server response rather than the page length.
useInfiniteThreads projects the pages back to thread arrays with select,
so list components are unchanged.
* test(frontend): pin the sidebar's next-page offset past side chats in E2E
Three side chats in the first 50 server rows push the sidebar's second
page to offset 53. After a rename through the sidebar menu, scrolling to
the sentinel must request offset 53, not 50. On main this test fails with
the request at offset 50.
* docs(changelog): reference #6702
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): static-website gate and deep-link fallback for user management (#6641 follow-ups)
Close the two leftovers from the #6641 review:
- user-settings-page now gates on isStaticWebsiteOnly() like model management
(the static demo user is nominally an admin, but there is no Gateway to
query), and the users adminOnly copy gains the same 'unavailable in demos'
sentence the models copy already carries.
- the settings dialog derives the active section from the nav actually offered
to the current user, so a ?settings=users deep link (or a stale dialog
store) for a non-admin falls back to the first section instead of rendering
an orphan panel with no highlighted entry.
Tests: static demo loads nothing (mutation-verified red), non-admin deep link
falls back to models, admin deep link lands on user management; full unit
suite green.
* test(frontend): pin the static-on x admin matrix cell at the dialog level
Dialog-level coverage for the real demo path (review follow-up on #6691):
admin + isStaticWebsiteOnly() + ?settings=users keeps the Users nav entry
(the splice keys on the role only) while the page-level gate shows the
admin-only note and issues no query. Mutation-verified: adding the mode
flag to the nav splice turns this test red and only this test.
* fix(frontend): show a date-only run_at at midnight in the task's zone
A chat-created one-time task can store run_at as a bare date
("2026-12-07"). The server reads it with datetime.fromisoformat as
midnight in the task's timezone. onceRunAtInstant instead returned it
unchanged: its offset check matched the trailing "-07" as a UTC offset,
and its local-time pattern needed a time anyway. new Date() then read it
as UTC midnight, so the schedule line, chat card and edit form showed
08:00 in Asia/Shanghai and the previous evening west of UTC.
A bare date is now matched first and converted as midnight in the task's
zone, the same way a time-bearing local run_at is.
* docs(changelog): reference #6692
* test(frontend): pin the clock in the date-only run_at E2E
The detail page formats the schedule line against the real clock: it
drops the year inside the current year and switches to relative labels
in the final week. The exact-text assertion therefore failed throughout
2027 ("Once, Dec 7 00:00 ...") and again as "Once, Tue 00:00 ..." in the
week before the run. Fix "now" at 2026-10-01, the same instant the unit
test uses, so the assertion stays deterministic.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): return to the opened workspace page after signing in
Opening a workspace page without a valid session sent the user to a bare
/login: the workspace layout called redirect("/login"), and a layout
cannot read the request URL, so the path and query were both lost. A
scheduled-task notification link (/workspace/scheduled-tasks?task_id=X)
opened after the session expired therefore landed on the default
workspace after sign-in.
For an unauthenticated verdict the layout now renders
WorkspaceLoginRedirect, which replaces the page with
/login?next=<path+query> from the browser. It renders nothing else, so
the workspace pages under it never mount.
AuthProvider.refreshUser and useRequireAuth kept the path but dropped
the query. They now share currentReturnPath() with the fetcher, which
already kept both.
* docs(changelog): reference #6681
* fix(frontend): keep the fragment in the sign-in return path
currentReturnPath() kept the path and query but not location.hash, so
an anchored deep link (/workspace/chats/x#section) lost its anchor across
the login round trip. It is now the single source for all four bounce
sites, so include the fragment. The server never receives the fragment,
so only the browser-side bounce can keep it. validateAuthNextPath, the
backend's validate_next_param and its quote() into /auth/callback all
carry a '#' through unchanged.
The models API test stubbed window.location without hash; a real
Location always has one, so the stub now carries it.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(authz): admin user-management UI and disabled-account copy (#4063 gap 3)
Frontend slice of the account lifecycle backend (#6566):
- Settings gains an admin-only Users page (core/auth/admin-users.ts ->
GET/PATCH /api/v1/admin/users): email, read-only role, disabled state,
and a suspend/restore toggle per row; the Gateway's rejection detail
(last-active-admin 409) renders verbatim. The nav entry is hidden for
non-admins and the page guards independently; ?settings=users joins the
deep-link whitelist (now an exhaustive Record over SettingsSection).
- A session that turns out disabled maps account_disabled to a login
redirect that states the reason on all three surfaces: the SSR
AuthResult tag (consumed exhaustively by the workspace/artifacts/auth
layouts), the shared fetcher's 401 interception, and
AuthProvider.refreshUser. No next target is preserved - a disabled
account cannot sign back in. Login failures surface the localized
account_disabled copy when the code is present.
- userSchema stays disabled-optional (absent = not disabled); zh/en
locales and docs updated in the same change.
* fix(authz): address review on admin user UI and disabled copy (#4063 gap 3)
Review follow-ups on PR #6641:
- Direct /login visits with a suspended session now surface the SSR
account_disabled verdict through AuthDisabledContext instead of
discarding it (willem-bd): the auth layout wraps pages with the
provider and the login form seeds its reason from it — no redirect,
so no loop. Regression tests cover both verdict states.
- admin-users.ts parses both endpoints against userSchema /
z.array(userSchema) (zhfeng): malformed payloads fail the load
instead of rendering phantom rows; toggle error copy is scoped to
GatewayApiError with a localized fallback for anything else.
- The localized account_disabled copy is login-only (zhfeng): sign-up
keeps the backend message verbatim so a suspended account's email
cannot be probed through the register surface.
- Settings page folds the load banner and toggle error into a single
alert region (zhfeng); adds coded-envelope 409, malformed-row, and
load-failure tests; fetcher.ts documents the 401 body-consumption
contract.
- prettier across touched files (CI format gate).
The admin-users query key keeps user?.id, matching the
model-settings-page precedent (converge later per review).
* feat(community): add You.com web search provider
Adds You.com as an opt-in community `web_search` engine, mirroring the
existing community engines (tavily, brave, exa, firecrawl, groundroute).
You.com runs its own web index and returns extracted page snippets per
hit rather than a single meta description, so a research loop gets usable
context without a follow-up fetch.
The provider picks its endpoint by whether a key is configured, so it
works out of the box and scales up when the operator adds one:
- no key -> /v1/agents/search, keyless free tier
- YDC_API_KEY -> /v1/search with X-API-Key, full Search API
Endpoint and headers are always chosen together; the keyless endpoint
rejects an auth header, so a stale key can never be sent there. Both
return the same body shape, so one normalizer covers both.
httpx only, no new dependency and no pyproject extra, so uv.lock is
untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(community/youcom): review follow-ups — drop stray .pr-body.md, guard OverflowError, add 429 hint
* fix(doctor): make the You.com free-provider detail true with or without YDC_API_KEY
The doctor detail returned before any key check, so operators who had
set YDC_API_KEY still saw 'set YDC_API_KEY for the full Search API'.
Reworded to match the sibling free-provider entries and stay accurate in
both states.
* fix(search): sweep youcom max_results onto the shared owner
The single-owner gate that landed with #6563 flags this branch's private
_coerce_max_results as a newly hand-rolled copy, so backend-unit-tests
now fails after merging main. Follow the coordination note from review:
delegate to deerflow.community.search_max_results.coerce_max_results,
which owns the rejection profile (booleans, non-integral floats,
OverflowError), and keep only the 1-100 mirror of You.com's server-side
count clamp on top — clamping policy stays per-provider.
The .inf regression test now goes through the public tool path and also
pins the bool and non-integral-float shapes of the shared bar.
* docs(tools): use the bare you.com platform link in config and docs
Match every other provider entry: no other provider in config.example.yaml
or the en/zh tools.mdx uses campaign-tagged URLs, so drop the utm params
from the three you.com/platform links. Integration attribution stays in
the identifying User-Agent sent to You.com hosts.
* test(youcom): cover shared max_results ownership and bounds
Register You.com among the shared-owner providers and assert default handling and the 100-result cap through outbound request parameters.
---------
Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: valueadd-lgtm <339269770+valueadd-lgtm@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(authz): account disable/enable across every authentication surface (#4063 gap 3)
Vertical slice 1 of the user-lifecycle work: an operator can suspend an
account and every authentication surface rejects it, while the row —
credentials, role, OAuth linkage — is retained so re-enabling restores
exactly what was suspended.
- users.disabled (NOT NULL, server default 0) via migration
0036_user_disabled; User/UserResponse carry and expose it.
- Enforcement at all four surfaces: password login (no credential
comparison happens), JWT session resolution (401 ACCOUNT_DISABLED —
sessions die at the next request), PAT resolution (the generic 401, no
oracle), OAuth provisioning (403 with a contact-admin message; the
link is kept so re-enable restores access).
- SQLiteUserRepository.set_disabled: single-column, serialized write
(same idiom as update_system_role) with the last-ACTIVE-admin guard —
disabled admins cannot authenticate, so they cannot restore management
access; re-enabling is always allowed. Field-scoping completed both
ways: the credential writer mirrors and never writes disabled, so
a stale password-change snapshot cannot re-enable a suspended account.
- PATCH /api/v1/admin/users/{id} becomes a partial account update
(system_role and/or disabled), each field routed to its field-scoped
writer; empty bodies 422.
Tests 11->18: route (disable/re-enable, last-active-admin 409, empty
422, combined update), real-repo (field isolation both directions,
disabled-admin-not-active, guard inside the serialized write, re-enable
always), login rejection with the CORRECT password (only the disabled
gate can reject) plus resolver-visible flag. Mutations verified red:
guard removed (M19), login gate removed (M20 — after hardening the test
with a real hash so the gate is load-bearing).
* fix(authz): portable boolean default, alternate-authenticator suspension, active-admin role guard
Review round 1 on #6566 — three P1s:
- [PG default] sa.text("0") is a raw integer expression PostgreSQL
rejects as a BOOLEAN default, breaking startup migration and fresh
create_all bootstraps alike. Both the migration and the ORM model now
use sa.false()/false() — dialect-rendered, portable.
- [alternate authenticators] WebSockets bypass AuthMiddleware, so a
disabled user's still-valid cookie could open a NEW browser/stream
connection and drive the retained session; the standalone LangGraph
authenticate callback accepted the same suspended JWT. Both now route
through a shared post-lookup validator (validate_resolved_session_user
in deps.py — token_version + disabled in one place) that the HTTP
dependency also uses, so a lifecycle change lands everywhere at once.
- [active-admin role guard] update_system_role counted every admin row,
so 'disable B, then demote self' — or a combined {system_role, disabled}
self-update — could strand the deployment with zero administrators able
to authenticate or re-enable anyone. The demotion guard now counts
ACTIVE admins only (disabled admins cannot authenticate), under the same
serialization boundary.
Tests 18->22: shared-validator verdicts, WS admission after suspension
(real token, real repo), LangGraph-callback admission after suspension,
disable-then-demote-self + combined self-update both rejected (fake-repo
route level and real-repo update_system_role level). Mutations verified
red: helper's disabled check removed (M21 — three admission tests fail),
active-admin predicate reverted to all-admins (M22 — repo-level strand
scenario fails). 366 tests green on the affected surface.
* fix(migrations): chain 0036_user_disabled after upstream 0037
The maintainer main-merge brought 0037_project_document_summaries,
which branched from 0035 independently of this PR's 0036 — alembic now
sees two heads and every engine bootstrap fails. Chain 0036 after 0037
(single head; revision numbers are labels, the graph is what matters).
* fix(migrations): renumber user_disabled to 0038 and follow the chain-test convention
Upstream landed its own 0036 (run_idempotency_request) with 0037 chained
after it while this PR was in review — my 0036_user_disabled collided on
the number and my interim chain-fix (0036 revises 0037) left the graph
in a state upstream's chain tests reject (heads == [0037] hardcoded).
- 0036_user_disabled -> 0038_user_disabled, revises
0037_project_document_summaries: single head 0038.
- Chain tests follow the repo convention (each migration PR appends the
new head): 0032/0035 gain the DOC_SUMMARIES link and CURRENT_HEAD =
0038; 0037's single-head assertion moves to CURRENT_HEAD with its
own-revision link check unchanged.
* fix(migrations): pin the remaining chain tests to the 0038_user_disabled head
The 0027/0028, 0029 and 0034 chain tests still pinned the single head to
0037_project_document_summaries (and 0034 asserted the head's parent was
0036_run_idempotency_request), so the renumbered lifecycle migration broke
two unit-test shards. Follow the chain-test convention: every pinned head
moves to 0038_user_disabled and 0034 walks the chain through 0037.
* fix(authz): enforce account suspension on the internal-auth dispatch surface
Round-3 review finding: the internal-auth principal is synthesized from
trusted headers without a users-row lookup, so a suspended user with an
active channel binding kept driving the agent, and scheduler-owned runs
(their fabricated in-process requests skip AuthMiddleware entirely) kept
launching. Two gates now enforce the lifecycle there:
- AuthMiddleware rejects owner-bound internal calls whose users row is
disabled (401 account_disabled) at the single gate every internal HTTP
call passes through. Auth-disabled mode is exempt (no suspendable
accounts, and its owner header carries the synthetic identity), and
owner ids without a users row stay allowed — a nonexistent account
cannot be suspended.
- start_run re-asserts it at run admission after the owner is resolved
for attribution: the scheduler records the rejection as the
occurrence's launch error and no run record is created.
Also addresses the remaining round-3 nits: drop the inline token_version
check duplicated by validate_resolved_session_user, correct the
local_provider comment (login never surfaces ACCOUNT_DISABLED), refresh
the stale 0038 docstring header, and restore the deps module caches in
the two suspension tests instead of leaking a closed engine.
* fix(auth): carry the disabled field on the frontend synthetic users
UserResponse now always includes disabled, so the auth-disabled contract
e2e (gateway /auth/me deep-equals the frontend synthetic user) failed
without the field on AUTH_DISABLED_USER. Add disabled to both synthetic
users and declare it optional on the zod schema — absent still parses as
not disabled for pre-gap-3 backends.
* fix(authz): log when the internal-owner suspension check is skipped
The middleware gate's users-lookup failure path silently degraded to
allow; keep the fail-open behavior but leave a WARNING with the
exception, so a store that cannot be read is observable in production
logs instead of invisible.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): select the model per conversation via /model
IM channels could list models with /models but had no way to choose one:
the model for a conversation came only from channel/session config or the
server default, so a user who needed a stronger or cheaper model for one
task had to reconfigure the whole channel or move to the web UI.
- /model <name> pins a model to the current conversation, validated
against the caller-visible /api/models list (list membership doubles
as the authorization check), persisted in thread metadata so it
survives a Gateway restart and applies from the next message in
place — unlike /agent use, which must start a new conversation.
- The pin is written to all three runtime carriers (run context,
configurable, context) after the session layers merge, so it wins
over channel/session configured models; like any explicitly
requested model it also takes precedence over a Custom Agent's
configured model.
- /model (no args) reports the effective model and its source through
one resolver that mirrors the run merge order (default -> channel ->
user, context carrier over configurable, both session-config
carriers scanned); /model default clears the pin; /models names the
pinned model in a footer.
- Pin lookups degrade instead of failing messages: the status query
reports "Current model: unknown" on a lookup failure (matching the
/models footer), and the reuse-thread preload leaves the in-process
cache untouched so the next turn retries a transient threads.get.
- Reserve "model" in the slash-skill reserved-name contract (backend,
contract fixture, frontend), register the Telegram handler, and
document the command in /help, README, and README_zh.
* fix(channels): fence model pin cache and mirror runtime model resolution
Review follow-up for /model:
- Serialize cold pin loads with persist+cache publication under a
per-manager lock, so a slow threads.get that captured pre-pin
metadata can never overwrite a concurrently committed selection
(commands are not run-serialized).
- _resolve_configured_model_name now replicates the run path's shallow
session merges and applies the real runtime precedence —
config.context (preserved by build_run_config, unbeatable by the
setdefault-filled body context) > top-level context > configurable —
instead of scanning per-layer carriers; a later layer replacing a
nested carrier wholesale no longer leaks a stale value into the
status reply.
- /api/models resolves the trusted bound owner for internal channel
calls and filters the list with the owner's role, so /model
selection gates on the same principal that run admission admits for
that account.
Tests: pin-write-vs-in-flight-load race, status compared against the
real merged runtime config (parametrized across carriers),
nested-carrier replacement, and owner-role filtering with the real
RBAC provider; each verified red by reverting exactly its guard.
* fix(channels): share model resolvers with the run path and harden the pin lifecycle
The /model status reply re-implemented the session-layer merge and the
Gateway carrier precedence, and it ignored the custom-agent model
fallback — on a thread pinned to a custom agent whose config sets
``model:`` it reported "server default" while the run actually used
the agent's model. Review also found the pin lock spanned a network
call and a stale pin bricked its conversation behind the generic
error fallback.
- Extract ``_select_carried_model_name`` as the single carrier-precedence
rule; the status reply reuses the run path's exact ``_merge_dicts``
layer shapes, so a future layering change has one place to touch.
- Extract ``_resolve_message_assistant_id`` / ``_resolve_assistant_id``
and share the assistant precedence chain between the run path and
the status reply.
- Fall back to the resolved custom agent's configured ``model`` when no
session layer carries one, mirroring the agent factory's
``_resolve_model_name(requested or agent_model_name)``; the reply now
distinguishes "(from channel configuration)" and "(from agent
configuration)".
- Run the cold ``threads.get`` in ``_load_thread_model`` outside
``_model_pin_lock``; only cache publication takes the lock, with a
post-GET re-check so an in-flight /model write still wins over the
older snapshot.
- Detect the run-admission allowlist rejection in ``_handle_message``
and recover: clear the stale pin (durable write plus cache under the
lock) and reply with the reset and the /model recovery instead of
the generic internal-error fallback.
- Document /model and the ``channel_model_name`` pin semantics in
backend/app/channels/AGENTS.md.
* fix(channels): fence stale-pin recovery and cover streaming admission failures
Review round 4 found the stale-pin recovery itself unsafe and
incomplete:
- Recovery cleared whatever pin was current when the delayed
admission rejection arrived, so a rejection for an old pin could
erase a replacement selection committed mid-flight (and tell the
user the new model was unavailable). ``_stale_pin_recovery`` now
takes the failed run's own selection (``expected_pin``, read from
its run context) and clears only while that selection is still
current under the lock; a replaced or already-reset selection gets
a non-destructive reply instead.
- Feishu/Telegram/DingTalk streaming turns never reached the recovery
hook because ``_handle_streaming_chat`` swallows start_run's 400
into its terminal-error reply. The recovery now runs in the
streaming terminal-error path too, delivered as the single final
outbound with the same dedupe cleanup; blocking, streaming, and
fire-and-forget run-creation paths all handle the rejection at the
site that knows the submitted selection.
* docs(channels): fit the /model entry within the agent-guidance size budget
The base file sits one byte under the AGENTS.md hard limit, so
documenting /model requires net-zero growth: keep the command-inventory
addition and a compact pin-semantics sentence, and losslessly tighten
the surrounding prose (the file was already at the limit).
* fix(channels): apply the review suggestions on /model replies
Three non-blocking findings from the latest review:
- The /models pin footer decorated the failure string too, advertising a
pin next to an error that lists nothing. The command now fetches the
listing through ``_fetch_gateway_json`` and appends the footer only on
a successful fetch.
- The /model status reply labeled every session-layer model "from
channel configuration", misreporting user- and default-layer models.
The label is now "session configuration" for every session layer.
- Stale-pin recovery replies read as if the turn went through, but the
run was rejected before starting and the triggering message never
reached the agent. All recovery variants now end with an explicit
"Your message was not processed — please send it again."
* fix(channels): recover stale model pins on the follow-up drain path
The drain is the fourth run-creation path and the only one without
stale-pin recovery: a /model pin that left the allowlist while follow-ups
were buffered made every drained runs.create fail admission, requeueing
the batch into the same 400 on every cycle with no user-facing recovery —
contradicting the documented "stale pins self-clear" invariant.
- Compute expected_pin from the drain's run context (mirroring
_handle_chat_on_thread, initialized None since _resolve_run_params is
itself inside the guarded try) and run _stale_pin_recovery on drain
failure before requeueing; the pin self-clears and the requeued batch
succeeds on the next cycle.
- Pass the caller-authoritative thread_id into _stale_pin_recovery:
every call site already holds it, so recovery never re-derives it from
the message — a store-mapping rewrite between dispatch and recovery
can never clear the wrong conversation's pin, and the lookup's
failure mode (and its guard) disappears entirely.
- Return StalePinRecovery(text, resolved) so requeueing callers can
tell a cleared pin from a repeat failure: the drain logs INFO when
the next cycle is free of this rejection and WARNING when recovery
could not remove it (persist failed, or the selection came from
session configuration, which still carries it) — no silent infinite
requeue loop. Drain logs carry structured facts only: the reply text
ends with a "please resend" note that would double-process buffered
messages retried automatically (GitHub, the only buffering channel,
sends log-only anyway).
- Reword the config-carried recovery reply: with no pin held, the
rejected selection came from session configuration — name that source
and the fix instead of claiming a default takeover that would not
happen.
- Correct the _load_thread_model lock docstring (writes deliberately
hold _model_pin_lock across threads.update) and read the agent config
for /model status off the event loop via asyncio.to_thread, with a
Blockbuster-gated blocking_io regression test.
- Pin the under-lock re-check in _stale_pin_recovery with a concurrency
test (a /model committed during the lock-wait window wins; recovery
persists nothing), mutation-verified — the drain is the only caller
running on a background task without the serialized-run lock, so it is
the path most exposed to that window.
* Revert "fix(channels): recover stale model pins on the follow-up drain path"
This reverts commit 0b4652c689afb56791e5e670901c97a4bf44a156. The drain
recovery work moves to a dedicated PR stacked on this branch, keeping #6534
scoped to the /model feature itself.
* fix(channels): recover stale model pins on the follow-up drain path
The drain is the fourth run-creation path and the only one without
stale-pin recovery: a /model pin that left the allowlist while follow-ups
were buffered made every drained runs.create fail admission, requeueing
the batch into the same 400 on every cycle with no user-facing recovery —
contradicting the documented "stale pins self-clear" invariant.
- Compute expected_pin from the drain's run context (mirroring
_handle_chat_on_thread, initialized None since _resolve_run_params is
itself inside the guarded try) and run _stale_pin_recovery on drain
failure before requeueing; the pin self-clears and the requeued batch
succeeds on the next cycle.
- Pass the caller-authoritative thread_id into _stale_pin_recovery:
every call site already holds it, so recovery never re-derives it from
the message — a store-mapping rewrite between dispatch and recovery
can never clear the wrong conversation's pin, and the lookup's
failure mode (and its guard) disappears entirely.
- Return StalePinRecovery(text, resolved) so requeueing callers can
tell a cleared pin from a repeat failure: the drain logs INFO when
the next cycle is free of this rejection and WARNING when recovery
could not remove it (persist failed, or the selection came from
session configuration, which still carries it) — no silent infinite
requeue loop. Drain logs carry structured facts only: the reply text
ends with a "please resend" note that would double-process buffered
messages retried automatically (GitHub, the only buffering channel,
sends log-only anyway).
- Reword the config-carried recovery reply: with no pin held, the
rejected selection came from session configuration — name that source
and the fix instead of claiming a default takeover that would not
happen.
- Correct the _load_thread_model lock docstring (writes deliberately
hold _model_pin_lock across threads.update) and read the agent config
for /model status off the event loop via asyncio.to_thread, with a
Blockbuster-gated blocking_io regression test.
- Pin the under-lock re-check in _stale_pin_recovery with a concurrency
test (a /model committed during the lock-wait window wins; recovery
persists nothing), mutation-verified — the drain is the only caller
running on a background task without the serialized-run lock, so it is
the path most exposed to that window.
(cherry picked from commit 0b4652c689afb56791e5e670901c97a4bf44a156)
* fix(channels): apply the third review round on /model command replies
- Delete the dead kind == "models" branch in _fetch_gateway: /models has
rendered the listing inline since the pin-footer fix, so the helper's
copy was unreachable and had already diverged (unguarded m["name"], no
pin footer). The internal-auth-header test now drives the surviving
/memory kind.
- Guard the inline /models listing comprehension like the
_handle_model_command sibling (isinstance(m, Mapping) and
isinstance(m.get("name"), str)), so a malformed payload entry can no
longer turn the command into the generic internal-error reply;
regression: test_models_command_skips_malformed_entries.
- Drop the unused user parameter from _resolve_bound_owner_for_model_list:
the resolved owner comes solely from the trusted header, and the
parameter implied the caller's own principal shaped the filter.
* fix(channels): apply the fourth review round on the /model lifecycle
- Preserve fresh model pins across cache eviction: a cold threads.get
racing a successful /model write could still publish its stale
snapshot after clear-on-pressure evicted the fresh entry — the
in-cache re-check cannot see an eviction. The model-pin cache now
carries a write generation; a load whose generation moved during the
GET refetches once instead of publishing over the committed
selection. Regression:
test_fresh_model_pin_survives_cache_eviction_during_cold_load
(blocked GET, write, eviction, then the refetch wins),
mutation-verified.
- Resolve context-carried agents before reporting their model: /model
status now mirrors the run path's effective-agent rule — explicit
message/thread choice overwrites every carrier, then the merged
carriers' agent_name (new _select_carried_agent_name, sharing the
model-name precedence via _select_carried_value), then the
session-layer assistant_id that only setdefaults into context. A
session with context.agent_name=coder now reports coder-model instead
of server default, and assistant_id=writer alongside
context.agent_name=coder reports coder-model instead of writer-model.
Regressions: context-carried, context-beats-assistant_id, and
thread-pin-beats-context status tests.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Opt-in per-document summary column rendered into the injected
<documents> index, list_project_documents JSON, and the documents API.
Pipeline (projects/summaries.py, behind the narrow enqueue_summary
interface):
- The two shelf write routes enqueue a job on created=True; dedup hits
never regenerate. Off by default (projects.summaries_enabled).
- Bounded worker pool (summary_concurrency + bounded summary_queue_size;
a full queue drops the job) fed by FIFO; shutdown drains queued jobs,
delivers one sentinel per worker via blocking put (queue smaller than
the worker pool cannot strand idle workers), and bounds only
genuinely in-flight jobs with the grace period.
- Jobs carry (document_id, project_id, user_id) explicitly; the job's
owner is bound into the user ContextVar around the model call so
tracing/usage attribution never falls back to 'default'.
- Text source reuses read_text_serving_path (convertible-first rule,
content validation); conversion has no deadline (await_drained), the
model call has a caller-side wait limit — the worker slot is retained
until the provider call actually finishes and late results are
discarded.
- Output contract: strip leading <think> reasoning -> unwrap fence/
quotes -> collapse to one line -> empty is NULL -> UTF-8 byte cap on
a character boundary; length-capped responses (provider-normalized
detectors) are failures.
- set_summary pins updated_at (literal self-assignment) so a landing
summary never reorders the shelf; owner-predicated UPDATE is a no-op
on rows purged mid-generation.
- Metrics: eligible/success/failed/skipped/queue_dropped/
uncapped_accepted.
Schema/config:
- project_documents.summary (migration 0036, nullable, no backfill;
forward-compat floor unchanged).
- Six projects.summary_* keys, all registered in STARTUP_ONLY_FIELDS
(captured once at lifespan startup); config_version 57 -> 58.
- run_oneshot_llm gains an optional model_overrides passthrough (passed
only when set, existing callers unchanged) and a detailed variant
exposing the raw response message for termination-signal detection.
Docs: API.md response field, ARCHITECTURE.md pipeline paragraph,
migrations AGENTS.md chain head, root AGENTS.md records the scoped
all-core exception with the lift-out path.
* feat(goal): record the met goal and keep goal state on the history head
- Add a goal_outcome LastValue channel (GoalOutcomeState). Only the
guarded clear of a satisfied ordinary goal writes it, in the same
checkpoint as the clear; the reply anchor is the latest visible AI
message with text, the reply the visible signature keys on. A goal
stored without created_at still clears and gets a record.
- write_thread_goal removes the record on every other goal write and
lists it in channel_versions, new_versions and metadata writes when
it changes. Threads that never had a record keep their checkpoint
shape. The sync-only SqliteSaver and CachedHistorySaver, which take
the #6448 sync fallback, store and drop it the same way.
- goal_outcome is server-owned: POST /state and non-internal run input
that carry it get 400. A POST /state goal write, or a goal in
external run input, clears it.
- A branch head written onto its replay base resets the channels the
head lacks, so a goal or record that write_thread_goal removed does
not come back. Branching from the met turn keeps the record without
the met goal.
- The POST /history head now carries an active goal and an achieved
goal_outcome, also before any message exists, so the post-run
history refetch keeps them. Older entries stay title/thread_data.
- contracts/thread_goal_contract.json pins the chat stand-down and
check-failure codes, the record shape and status, the head keys and
the host-written reasons.
* feat(frontend): show why a goal stopped and when it was met
- The goal bar derives its state from the active goal, the goal_outcome
record and the messages: set, in progress, continuing, stopped (limit,
auto-continue off, no progress, token budget, missing evidence, failed
run, unknown code), waiting for you, waiting, couldn't check the goal,
paused and goal met. An 8/8 stand-down now reads "Stopped" with the
next step instead of "Continuing 8/8", and a count-0 stand-down no
longer looks like a fresh goal.
- "Goal met" shows while its anchor reply is the latest visible message
(a visible human message, or a visible AI message with text, moves the
chat on), so a reload agrees with the live view. An active goal wins
over the record, and this tab's /goal set or clear hides it.
- useActiveGoal drops the optimistic goal when the record of that same
goal arrives, which fixes the bar that stayed after a goal met on the
first run (the clear frame has no goal key), also for a goal stored
without created_at. Server goal values pass through isGoalState; a
/goal status read uses the backend's own active check, so the bar, the
toast and the edit lock agree with the server.
- Row 2 gives the reason and what to do next; Details holds the full
goal, the checker's own note (never host-written text, a check failure
or a paused goal's older check), the assumption note and an unknown
code, and starts collapsed in each new state. The bar is a labelled
section with a live region that announces state changes only (another
goal's same state counts) and re-reads its text after a language switch.
- The edit pencil on the latest turn is shown disabled with "Editing is
off while a goal is set" instead of hidden; the message toolbar shows
on focus-within, so keyboard users can reach it.
- A /goal set or clear rejected with 409 keeps the draft and says a run
is still going; other failures keep the server's message and the draft.
A /goal request replaced by a newer one leaves the composer alone, so
a double submit no longer empties it before the second one fails.
- Copy lives in inputBox.goalBar (en-US, zh-CN); a contract test reads
contracts/thread_goal_contract.json, and the raw-identifier guard now
catches stand-down codes and goal field names.
* test(frontend): cover the goal bar on both chat pages
- tests/e2e/goal-status.spec.ts runs every case on the chat page and the
agent chat page, since each wires the goal bar and the edit lock itself:
a goal stopped at 8/8, one waiting for details and one waiting on an
open question card show their reason and next step, also after a
reload; the locked edit pencil gets keyboard focus, shows its toolbar
and tooltip, does not open the editor, and hides while a file uploads;
Goal met after one automatic follow-up (In progress, Continuing 1/8,
then "Auto-continued once") survives a reload and hides after the next
visible message; a goal met on the first run of a new chat replaces the
set bar and frees the pencil, and a new /goal replaces Goal met while
its own run goes on; a /goal set or clear refused with 409 keeps the
draft for the retry and shows its own busy toast.
- The mock POST /history head carries only the contract's
history_head_keys, with goal only while active and goal_outcome only
when achieved; thread state and run values omit an unset goal instead
of sending goal: null. PUT and DELETE /goal drop the met record,
setGoalWritesBusy returns the Gateway's 409, and a repeat create of a
requested thread id returns the thread as is, like the Gateway.
- The raw-identifier guard also catches checker blockers and the met
record's field names.
- frontend/src/AGENTS.md: the GoalStatus sentence now names goal,
goal_outcome and useActiveGoal, reworded in place.
* test(frontend): keep the draft through a failed upload and a busy /goal
#6412 fires onSent only once a send is dispatched, so a failed upload
keeps the text, files and stored draft. A /goal the Gateway refuses with
409 never reaches sendMessage, and the composer rejects it so PromptInput
keeps the text. Drive both through the real sendMessage in one composer:
a failed upload keeps "summarize it" stored, a busy /goal keeps its text
and stored draft with no run, and the retry saves the goal and starts
one run with the objective, not the "/goal" text. After that run the
composer is empty and no stored draft comes back.
* fix(frontend): drop the stop condition's closing period before the pause clause
A stored stop condition often ends with a period, so the schedule card
read "…remain)., pauses itself" (and "…。，满足后自动暂停" in Chinese).
Trim the condition's closing sentence punctuation before the clause that
follows it. The event line, where the condition ends the sentence, is
unchanged.
* fix(frontend): point "evidence missing" at the agent, not a user reply
"Evidence missing" asked the user to reply with what's left, which is
the push a goal is meant to save. It now asks the agent to show the
result or explain what's missing, and offers /goal clear only for a goal that
no longer applies: clearing drops the met record, so work that is right
would never show "Goal met".
zh-CN follows. Copy only: no key, state or mapping changes, and the
scheduled page's reason labels are untouched.
tests/unit/components/workspace/goal-status-next-steps.test.ts checks
the row in en-US and zh-CN, and that no stop row offers /goal clear
without that condition. goal-status.dom.test.tsx and
goal-stream.dom.test.tsx now expect the new copy.
* fix(frontend): say "Couldn't check the goal" once in the goal bar
After a goal check that could not run, the bar's second line borrowed
the scheduled page's whole reason, so it repeated the chip: "Couldn't
check the goal · Couldn't check the goal; the run itself may be fine.
It's checked again after your next message." Chinese had the same
repeat.
The line now gives only why the check failed ("The run itself may be
fine", "No final reply was saved" or "The chat changed during the
check"), then when it is checked again. The screen-reader announcement
follows. The chip and the scheduled page's reason labels are unchanged.
tests/unit/components/workspace/goal-status-unchecked.test.ts checks
every check-failure code in en-US and zh-CN: the label appears once,
and the line still says why the check failed.
* feat(goal): redact the evaluator input when pii_redaction is enabled
The goal evaluator calls its model directly, outside
PiiRedactionMiddleware, so it sent raw user lines, tool arguments, tool
results and Human Input Card answers even with pii_redaction enabled.
With pii_redaction enabled:
- the messages in the evidence window are redacted whole before the
evidence caps can cut an identifier in two, as TitleMiddleware does.
This reuses redact_queued_messages, which already covers content,
call arguments and card answers. Earlier messages are left alone: the
evaluator never sees them, and the pass is synchronous work on the
event loop (0.84 s for an 800-message, 4 MB thread, 0.06 s for its
window);
- the assembled input, the goal objective included, is redacted again.
A redaction error fails the check with evaluator_failed; raw text is
never sent instead. When redaction is off, the input is byte-identical.
Thread state keeps the raw text. README.md, config.example.yaml and the
middleware docstring list the evaluator with the other redacted inputs.
tests/test_goal_pii.py puts an address where the evidence caps cut a
tool argument, a tool result and a card answer, and checks the
objective, the unchanged input when redaction is off, that only the
evidence window reaches the redactor and gives the same evidence as
redacting the whole thread, that a redaction error stands down with
evaluator_failed before the model is called, and the worker path.
* fix(goal): redact the hidden continuation when pii_redaction is enabled
The hidden /goal continuation repeats the goal objective and the
evaluator's reason and evidence summary. PiiRedactionMiddleware skips
it, since it is a framework message and not a user message, so with
pii_redaction enabled each continuation turn sent them raw to the
agent's model, and the message stayed in the thread for later model
calls. The durable-context block and the evaluator input already redact
the objective.
With pii_redaction enabled, make_goal_continuation_message now redacts
the objective, the reason and the evidence summary with redact_text
before the message is built. The thread keeps the redacted message,
which the UI hides; the goal state keeps the raw objective.
The worker builds the message before it counts the continuation. A
redaction error fails the check the way an evaluator redaction error
does: run_failed with stand_down_reason evaluator_failed, the goal stays
active, nothing is sent and the continuation budget is not spent. Only
the exception type is stored. With redaction off, the message is
byte-identical.
README.md, config.example.yaml and the middleware docstring list the
continuation with the other redacted inputs.
tests/test_goal_continuation_pii.py checks the redacted objective,
reason and evidence summary against the tokens redact_text gives, the
unchanged message with redaction off or no config, the worker path
under the run config, and a redaction error.
* docs(changelog): link the goal outcome entries to #6556
Fill the four ([#PR]) placeholders in CHANGELOG.md and CHANGELOG_zh.md with this PR's number and add the matching link references.
* fix(frontend): keep an active goal stored without timestamps visible
POST /threads/{id}/state and run input can store an active goal without
created_at or updated_at. The backend still evaluates it and rejects an
edit with 409 (is_active_goal checks only status "active"), but
isGoalState required both timestamps, so history and live values
dropped it: both chat pages hid the goal bar and unlocked the last-turn
pencil, and only a /goal status read, which used a looser predicate,
brought them back.
isGoalState now accepts any record with a string objective and status
"active", and normalizeGoalStatusRead uses it, so thread values, live
updates and status reads share one rule. Malformed or inactive values
are still dropped.
Tests cover a sparse active goal from the history head and from a live
update before the post-run refetch (bar shown, hasGoal true), and a
malformed goal that stays hidden.
* test(goal): check the outcome write metadata the same way on every saver
PostgresSaver stores checkpoint metadata through
get_serializable_checkpoint_metadata, which drops "writes", so the
[postgres] case of test_goal_outcome_round_trips_through_the_sync_fallback
failed with KeyError: 'writes' when CI ran it against a live database.
The test now records the metadata write_thread_goal passes to put and
asserts the clear's and the next goal's "writes" there, for every saver.
The stored channel values are checked as before, and on sqlite and
cached-sqlite the stored "writes" must still equal what was passed.
* docs(agents): keep goal_outcome out of the agents guidance chain
The middlewares AGENTS.md chain is already over the 98,304-byte AG002
hard limit on main, and the PR-mode check fails any chain a PR touches,
so adding goal_outcome to the ThreadState field list in
deerflow/agents/AGENTS.md failed the agent-guidance job. goal_outcome
stays documented in runtime/AGENTS.md and backend/app/gateway/AGENTS.md,
which are outside that chain.
* refactor(goal): read the evaluator's evidence and redaction from one window
format_visible_conversation recomputed the window start that the redaction
pass takes from _evidence_window. Both now use _evidence_window, so a later
change to the cap or to what counts as visible moves both.
The window test now fills the window with tool-only replies and tool results.
Before, it caught a cap change in one copy but not a change to what counts
as visible.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The Tavily, Exa and Firecrawl tool pages and the AIO sandbox sections
tell readers to run `uv add 'deerflow-harness[tavily|exa|firecrawl|aio-sandbox]'`.
deerflow-harness has no such extras: tavily-python, exa-py, firecrawl-py
and agent-sandbox are core dependencies. uv warns about the unknown extra
and still writes it into backend/pyproject.toml.
Say that no extra dependency is required, matching the Brave and Serply
tabs and the E2B sandbox section.
* docs(events): document durable thread seq watermark
RUN_EVENT_STREAM.md described seq as monotonic but did not cover the durable per-thread high-water mark that keeps allocations monotonic after a run is deleted. Record that behavior in the Record Envelope section, including that thread deletion drops the mark only when no events remain.
* test(scheduler): pin clock for scheduled-run timezone assertions
---------
Co-authored-by: yu1128-L <296164533+yu1128-L@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): share skill cache invalidation across Gateway replicas
SkillStorage.load_skills() rescans disk and re-reads extensions_config.json
on every call, but the prompt layer in deerflow/agents/lead_agent/prompt.py
caches the enabled-skills list, the per-(app_config, user_id) entries and
the rendered <skill_system> section per process, and every skill mutation
in routers/skills.py only refreshed the process that handled it. With
several uvicorn workers or several Pods sharing one home volume, a skill
installed, edited, deleted, rolled back or toggled through replica A kept
being offered as the old skill set by replica B until B restarted, and
POST /api/skills/reload had to be sent to every Pod separately.
Reuse the MCP cache-reset mechanism. A new generic helper,
deerflow.config.shared_reset_marker, owns a hidden JSON marker beside the
resolved extensions config (.<config name>.<suffix>.json): publish()
replaces it atomically with a random generation under
extensions_config_write_lock + extensions_config_file_lock, chaining
previous_generation and an optional user_id; SharedResetMarkerTracker
polls it at most once per second (monotonic clock, never blocking behind a
concurrent poller), adopts the current state silently on its first poll,
and classifies a change as user-scoped only when exactly one publication
carrying that user happened since the last poll. deerflow/mcp/cache.py now
builds its .<config>.mcp-cache-reset.json marker through the same helper
with no behavior change.
prompt.py publishes .<config>.skills-cache-reset.json via
publish_skills_cache_reset(user_id=...) and checks the marker on every
cache lookup (global cache, per-config cache, prompt-section LRU, user
scoped entries), calling _invalidate_enabled_skills_cache() or
invalidate_user_skill_cache(). The publishing process remembers its own
generation so it does not retire the caches it just rebuilt. Every
mutation tail in routers/skills.py (install/upload, edit, delete,
rollback, enable/disable, reload) refreshes locally and then publishes the
marker off the event loop; /skills/reload answers scope=shared_config,
or scope=process when no extensions config path resolves, and a marker
publication failure is a 500 so a 200 never means only one process was
refreshed.
Docs: README, backend/docs/API.md, skills/config/mcp/gateway AGENTS.md,
and frontend skills.mdx (en/zh; also fixes the reference to a
non-existent skills/loader.py) describe the shared invalidation and drop
the per-Pod guidance. CHANGELOG entry under Unreleased / Fixed.
* docs(changelog): link the skills cache reset entry to #6495
* fix(skills): drain the reload tail and keep marker failures out of 404
Two review follow-ups on the shared skills cache reset (#6495).
POST /skills/reload awaited the local refresh and the marker publication
unshielded. CancelledError is a BaseException, so a caller that
disconnected between the two slipped past the `except Exception -> 500`
mapping, surfaced no error, and left every peer replica serving the old
skill set -- the one outcome the operator hook for external mount writes
exists to prevent. The refresh and the publication now run as one
`await_drained(_drain_skill_mutation("reload", ...))` tail, the same
discipline the install/edit/delete/rollback/toggle tails already use; the
exception -> 500 mapping and the scope response are unchanged.
Publication errors could surface as 404. resolve_shared_config_path()
only swallows the resolution-time FileNotFoundError; a config directory
that vanishes between resolution and the lock file / atomic replace still
raised FileNotFoundError from the marker write, and the install, edit,
delete and rollback handlers mapped that to HTTP 404 with the raw OS
error text for a skill that had in fact been applied. publish_skills_cache_reset()
now re-raises OSError (and a misconfigured project root's ValueError) as
SkillCacheResetPublishError, a RuntimeError that no handler maps to
404/400, so every handler reports it through its generic 500 branch; the
message names only the exception type, never a server path. The install
handler gains an explicit clause that pins that contract beside its 404
mapping.
Tests: cancelling reload after the refresh still publishes the marker;
a FileNotFoundError from the marker write yields 500 (not 404) for
install, edit, delete and rollback, 500 for the public toggle and the
generic reload error; the wrapper type is not an OSError.
