* Initial plan
* docs(gateway): trim AGENTS guidance to stay within hard budget
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>
* fix: rebuild todo reminders after context compaction
* chore: remove implementation plan from PR
* refactor: share todo reminder message name
* docs: keep agent guidance within CI size budgets
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): stop deleting a converted companion we cannot prove we wrote
Conversion names a document's Markdown companion after the document's
stem and falls back to a _N suffix when that name is already taken, so
the .md beside a document may belong to another document sharing the
stem, or to the user. Delete removed it anyway: uploading a.docx and
a.pdf produces a.md and a_1.md, and deleting a.pdf destroyed a.docx's
companion while orphaning a.pdf's own.
Delete now removes only the file it was asked to remove. The companion
stays listed and can be deleted on its own. Orphans are the cost of not
guessing; issue #5672 covers giving companions a provable owner, which
is what a safe cleanup needs, along with the two related readers that
still guess (the outline injected for a document and the agent's file
listing).
convertible_extensions loses its last use and is dropped from the
signature and both call sites. The gateway router keeps importing
CONVERTIBLE_EXTENSIONS for the ingestion bridge and now declares it in
__all__, where that module documents its re-exports.
* docs(changelog): note that delete keeps the converted markdown (#5673)
A Redirecting slot stayed verbatim unless it started with "/", but the
Location field-value grammar (RFC 3986 relative-part) also admits
slash-less relative references: 'Redirecting /private/x?token=Q ->
download?sign=LeakedSig' rendered the signed query verbatim, and neither
the slot rule nor the generic absolute-URL pass (which needs a scheme)
could see it. A slot is now kept only when a scheme matches at position
0 (left for the generic pass to rewrite); everything else collapses -
slash-less relative paths, query-only and fragment-only forms,
network-path references (collapsing any userinfo they carry), and
non-hierarchical schemes such as data: URIs. The ^Redirecting prefix
anchor is unchanged, so non-URL '-> /path' arrows (sandbox mount
mappings) keep passing through untouched.
Regression table covers all relative-reference forms plus the absolute
regression anchor; mutation-verified (reverting to startswith('/') goes
red). Aligned with the fix sketched in the #5225 round-15 review thread.
* fix(uploads): convert the bytes we wrote, not the name they landed under
Document conversion re-opened the upload by name after it was already
visible in the thread's uploads directory: the Gateway converted the
committed file_path, and DeerFlowClient converted the copy it had just
placed there. That directory is writable from local and AIO sandboxes,
so a process watching it can replace the name with a symlink in the
window between the upload landing and the converter opening it. The
converter then reads whatever host file the link points at and writes
that content back into the thread as the .md companion, which the
sandbox can read. Reproduced end to end on both paths with a real xlsx:
the companion came back holding the host file's rows.
The Gateway now duplicates the descriptor of the staged file before the
link-commit, copies those bytes into a private directory outside the
uploads tree, and converts there. A descriptor cannot be redirected by
replacing a name, so the conversion input is the content this request
wrote. The client converts the caller's own source file instead of the
copy in uploads; the source is the file the caller handed in, which the
sandbox cannot reach.
Both already wrote the companion without following a symlink, so only
the read side changes. The uploads directory still receives exactly the
same files.
* docs(changelog): note upload conversion source fix (#5611)
* fix(uploads): close the conversion descriptor when staging its copy fails
Review follow-up. The private directory for the conversion copy was
created before the try that owns the duplicated descriptor, so a failure
there — a full or unwritable temporary filesystem — propagated without
closing it. The upload's own cleanup only unlinks the committed name and
releases the sandbox lease, so the descriptor stayed open for the life of
the process and kept the unlinked staged bytes allocated with it; repeated
failures accumulated both.
Directory creation now happens inside that try, and the finally removes
the directory only once it exists.
* fix(uploads): keep the conversion descriptor owned across cancellation
Review follow-up. run_file_io cannot interrupt its worker, so cancelling
the await around os.dup only abandoned the result: the duplicate was
created moments later with nothing left to close it, and it pinned the
staged bytes of an upload whose name the cleanup had already unlinked.
Cancellation after the duplication was just as leaky, because the
commit-path handler caught Exception and CancelledError is not one.
The duplication now runs as its own task, shielded from the caller's
cancellation, and closes its own result when the caller is gone by the
time the worker finishes. The commit path catches BaseException, closing
the descriptor it already owns before re-raising.
Both windows are pinned: one test stalls the duplication worker after it
allocates and cancels ingestion, the other stalls the commit so the
cancellation lands while the descriptor is owned.
* fix(uploads): drain the conversion copy so its descriptor always closes
Review follow-up. The copy worker owns the duplicated descriptor and
closes it in its own finally, but a bare await let a cancellation cancel
the executor job while it was still queued: the worker never ran, so that
finally never ran either, and the enclosing scope had already handed
ownership away and saw None. Draining also keeps a late worker from
writing into a private directory this scope has since removed.
The copy now goes through await_drained, the shield-and-drain helper the
Gateway already uses for offloads that must not be abandoned mid-flight.
Pinned by a test that holds the copy job queued, cancels ingestion, then
releases it and requires the descriptor to come back closed.
* feat(mcp): re-scope to MCP task claim lifecycle only
Keep PR #4966 a small, closed MCP lease/cancellation state-machine change and
move RunJournal and Run lifecycle work into dedicated follow-ups. This branch
contains only the MCP task claim lifecycle:
- mcp task release/snapshot fencing by owner + per-claim lease token
- phase-level single-flight poll/cancel/notification owners with retained handoff
- routine cancellation no longer persisted as a task failure diagnostic
- bounded ordinary release ownership retention past the drain deadline
- 0018_mcp_task_lease_tokens migration + migration/bootstrap head assertions
- wait_for_task_until helper (MCP uses it); worker-specific capture helper moved
to the run-finalization follow-up
RunJournal (journal.py + test_run_journal.py) and run lifecycle
(manager/worker/store/run sql + run tests) are preserved on
backup/cancellation-safety-full and will be raised as separate follow-ups.
* fix(mcp): unblock claims after ambiguous handoff resolves
A phase-level single-flight owner only guards an ambiguous claim outcome. Once
the claim resolves, the phase owner is released immediately; the handoff may
continue releasing returned rows as bounded, service-owned background work
(transferred to _compensation_tasks on timeout). Per-claim token fencing rejects
a late release against a newer claim generation, so a stuck release no longer
locks the whole phase until process restart.
- README: drop the stale progress-snapshot sentence from the bounded ordinary
release description.
- service: pop the identity-checked phase owner as soon as the claim outcome is
known, then release returned rows with the bounded path; carry the release in
_compensation_tasks if it exceeds the drain deadline.
- mcp/AGENTS.md: document that only an unresolved claim outcome (not the handoff)
blocks later phase scans, and that returned-row releases may continue in the
background once the owner is released.
- tests: pin that the phase owner is released before a stuck release finishes
while the release stays service strong-owned.
* refactor(mcp): remove unused single-record claim wrappers
_poll_one, _cancel_one, and _notify_one are unreachable in production: the
worker always processes claimed records through _run_claimed_batch, so these
wrappers preserved a second, dead single-record lifecycle (state is None)
whose only observable behavior was a wrapper-specific cancellation release.
Remove the three wrappers and migrate the regressions that guarded their
cancel/release invariants to exercise the production _run_claimed_batch path
(operation=_*_one_claimed, release=_release_*_after_cancellation). The single
wrapper-only "state is None" contract (test_poll_release_hang_without_batch)
is deleted; all 11 remaining invariants (CancelledError preservation, repeated
cancellation, poll-only token-fenced lease release, notification claimed vs
dispatched phase release, hung compensation -> service ownership, and
background compensation exactly-once observation) are now covered through the
real batch lifecycle.
* fix(mcp): fence claim-owned mutations against stale generations
The per-claim token check in the ORM release/apply paths was only in the
SELECT; the final write went out by primary key. On SQLite (where
with_for_update() is a no-op) a mutation from an older claim generation
could therefore clear a claim that a newer generation had reclaimed after lease
expiry — the exact distributed lease-fencing failure the per-claim token was
meant to prevent.
Make every claim-owned mutation a single atomic conditional UPDATE with the
owner and per-claim token in the WHERE clause (rowcount 0 => stale, return
False, no mutation):
- release_claim: atomic fence; record the poll-failure event after the fence
wins (same transaction, holding the write lock).
- apply_snapshot / apply_cancel_snapshot: atomic fence; record the event after.
- finish_notification_run: atomic fence; use a CASE on event_version >>
dispatch_version to keep a newer event pending for redelivery instead of
swallowing it as delivered.
Add one regression per path: a stale generation's release/apply/finish after a
same-worker reclaim is rejected and never clears the newer claim.
* test(mcp): pin the migration chain head to the lease-token revision
0026_mcp_task_lease_tokens becomes the alembic head, so the chain-head pin in the 0025 repair test had to move on. Follow the 0023 precedent there (single head plus expected predecessor) instead of pinning a literal head, and give the new revision its own migration test, which owns the pin and covers the nullable claim-token columns on upgrade and their removal on downgrade.
* refactor(mcp): close cancellation cleanup leftovers
* fix(mcp): retain cancelled release diagnostics
* test(mcp): remove obsolete settled compensation case
* test(mcp): cover interleaved lease reclaim races
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): add WeChat QR login and binding recovery
* fix(channels): enforce single-worker WeChat QR login and preserve bot ID
Reject QR login endpoints when multiple Gateway workers are configured, while keeping manual token setup available.
Preserve the configured bot ID when the provider omits it or returns an empty value. Add regression coverage for worker guards and credential persistence.
* fix(channels): sync WeChat completion state on provider updates
Show the connected step when refreshed provider data confirms the binding, so the dialog no longer waits indefinitely after polling is cancelled.
Add regression tests for provider updates during pending poll and binding requests, including late responses and expiry.
* fix(channels): preserve WeChat pairing codes across waits and redirects
---------
Co-authored-by: YxinMiracle <“939157765@qq.com”>
* fix(uploads): delete the requested upload, not a symlink's target
delete_file_safe resolved the requested path before unlinking it. The
uploads directory is writable from local and AIO sandboxes, so a
symlink planted under an upload name was followed: deleting alias.pdf
removed the victim.pdf it pointed to, and the companion cleanup then
removed victim.md, while the link itself survived and the call reported
"Deleted alias.pdf". A link resolving outside the directory was already
refused by the traversal check, so the damage stayed inside the
thread's uploads.
The function now checks and unlinks the requested entry itself and
treats a symlink as not found, the same way list_files_in_dir already
hides it. unlink() never follows the final component, so a file swapped
for a link between the check and the unlink removes only the link.
Tests cover the helper, the Gateway DELETE route, and
DeerFlowClient.delete_upload.
* docs(changelog): note upload delete symlink fix (#5547)
* fix(events): serialize DB deletion with thread writers
* fix(runs): delete thread history without dropping reservations
* fix(feedback): support owner-scoped thread cleanup
* fix(threads): clean persisted records on deletion
* fix(threads): correct the feedback cleanup rationale
* test(runs): drop the wall-clock probe from the in-flight delete test
* docs: record the thread-delete and event-store fence contracts
* fix(threads): preserve legacy event-store delete compatibility
* feat(gateway): thread checkpoint retention service on the #4189 deletion contract
Implements exactly the two contract-proven deletion shapes (trailing
duration-only leaves, opt-in leaf sibling branches) with head-chain
protection, explicit id protection, a strict pending-writes guard, and
joint writes-row cleanup. Head resolution uses LangGraph's time-ordered
checkpoint ids; storage deletion mirrors the contract's per-backend data
model. Ships without a production trigger by design. Validated against
the contract suite (12 passed) plus 14 service scenarios across memory
and SQLite; Postgres paths are gated on TEST_POSTGRES_URI.
Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
* fix(gateway): survivor-reachability blob GC and memory blob stats in retention service
Aligns the deletion service with the review-hardened contract: blob rows
are garbage-collected in a whole-thread pass against surviving
checkpoints' channel_versions (a real duration-only leaf shares its
parent's versions, so per-checkpoint version deletion would corrupt the
surviving state), the memory branch of the stats helper counts
saver.blobs and returns the full normalized shape, and per-node channel
versions are collected during the graph pass that already exists.
Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
* fix(gateway): address review findings on checkpoint retention service
Resolves the review at a479cfe (willem-bd):
- Untested savers now fail fast: an explicit isinstance allowlist
(InMemorySaver / AsyncSqliteSaver / AsyncPostgresSaver) raises
NotImplementedError before any row is read or deleted, so a shallow or
third-party saver can never issue partial DELETEs.
- The chain walk ends (break) instead of raising KeyError when the head's
ancestor row is missing, matching the deletable loop's tolerance for
missing parents.
- enforce_thread_retention takes an optional per-thread lock and documents
the concurrency requirement: classification and deletion are two separate
passes, so callers must serialize per-thread mutation (runtime
_checkpoint_thread_lock) or guarantee quiescence.
- Dropped the dead mid-run guard: CheckpointTuple has no `next` field in
langgraph-checkpoint 4.1.1, and pending_writes is populated for committed
writes too (verified on the list path), so neither is a usable mid-run
signal; the caller-held thread lock is the actual protection.
- Removed the write-only _node_step/_Node.step and fixed the head-selection
docstring (newest by checkpoint id, not (step, checkpoint_id)).
- Documented the E1 leaf / history fast-path interaction in the contract doc
and module docstring: the wiring PR must sequence retention away from
history reads or adopt a policy that spares cache-carrying leaves.
- Added regression tests: unsupported saver, missing ancestor row, thread
lock parameter.
Validation: test_checkpoint_retention_service 18 passed / 8 postgres-gated
skipped; contract + lineage suites 18 passed / 6 skipped; ruff check and
format clean.
* fix(retention): count non-empty writes dicts on memory saver
- _checkpoint_ids_with_writes now requires a non-empty writes dict on
InMemorySaver: the empty phantom entry for checkpoints whose task wrote
nothing no longer counts as "owns writes rows", so the default E1 pruning
reaches the memory backend again (it was a silent no-op there).
- test_runtime_duration_leaf_pruned_by_default runs the shipping default
(strict_pending_write_guard=True) and proves E1 is reachable out of the
box on every backend; the stale override and its wrong SQLite premise
are dropped.
- document that _checkpoint_thread_lock is non-reentrant: a caller already
holding it must not pass it in, or retention self-deadlocks.
* test(checkpoint-retention): fix stray duplicated def token in test_duration_link_protected_after_next_run
The previous push left `async def def test_...` at line 244, which made the
module unimportable and failed collection of the whole suite (and ruff
format --check). Local copy was already correct; this commit re-pushes the
clean file. 18 passed / 8 postgres-skipped verified from a head worktree.
* fix(gateway): make retention correct on Postgres and fail closed on a bad cap
* validate max_delete_per_run before any store read: a negative cap used to
widen the batch (Python slicing) instead of being rejected;
* report identical before/after stats for an empty thread instead of returning
before stats_after is collected;
* protect each namespace's resume head and ancestor chain, so a persistent
subgraph's latest checkpoint is no longer treated as a sibling leaf;
* read Postgres columns through a row-factory-agnostic helper (the PG savers
open cursors with dict_row, where positional access raises KeyError: 0);
* classify the duration-only leaf without relying on metadata["writes"], which
the Postgres saver strips via get_serializable_checkpoint_metadata.
Verified locally on memory, SQLite and a real Postgres 16 instance (62 passed,
0 skipped): the E1 shape now fires on Postgres, which no backend test covered
before CI ran the Postgres lig.
Signed-off-by: zeng-bohan <zengbh1@gmail.com>
* test(gateway): pin the Postgres-shape duration classifier; report per-namespace heads
- Deterministic regression for _mark_duration_leaves_without_the_marker:
hand-put the Postgres round-trip shape (writes marker popped, source=
update + accumulated run_durations + channel_versions identical to the
parent) and assert the shipping default prunes it; a control that bumps
one channel version (the client update_state shape) with otherwise
identical metadata stays protected. Both legs run on memory and SQLite,
so the class cannot silently re-widen (a resumable head losing head
protection) or re-narrow (E1 never firing on Postgres) without a
locally-executing test failing.
- RetentionReport.protected_head_id -> protected_head_ids: heads are now
selected per namespace, so the report carries every namespace's head
(root key = what an unsaved aget_tuple resolves) instead of only the
global max - reshape it before the wiring PR starts consuming reports
for audit/aggregation.
---------
Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
Signed-off-by: zeng-bohan <zengbh1@gmail.com>
Co-authored-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): preserve owner isolation when thread metadata is missing
Follow-up to the #5448 review P1 (post-merge finding): owner_check=True
also authorizes threads whose meta row is missing (legacy compatibility)
or NULL-owner (shared/pre-auth data). _run_scope_user_id returned None
for every trusted internal caller, which dropped the only remaining
per-user filter on those threads and let an internal caller acting for
owner A read owner B's persisted runs.
_run_scope_user_id now takes the thread_id and consults the thread meta
store: when an existing meta row establishes ownership, the authorized
thread's runs are still read unfiltered (merged #5448 semantics,
including owner-header-less internal callers); when the meta row is
missing or NULL-owner, the filter falls back to the acting owner's raw
stamp (the exact value start_run writes) — or the synthetic "default"
identity without an owner header — so cross-user runs stay hidden.
Isolation coverage uses the real MemoryThreadMetaStore with no metadata
row (and a NULL-owner row) plus another user's persisted run: /runs and
/runs/page must be empty and /runs/{run_id} must 404 for internal
callers, while an established-ownership thread keeps the unfiltered
read.
* fix(gateway): gate run-scoped sub-resource reads for internal callers
Review follow-up on #5484: the P1 owner-isolation class remained
reachable through run-scoped sibling reads that apply no per-user filter
at all — /runs/{run_id}/messages, /events, /join, /stream and
/workspace-changes query by (thread_id, run_id) directly, so on
missing/NULL-owner threads an internal caller acting for owner A could
still read owner B's run content by id (verified 200 at the previous
head).
- Extract _thread_ownership_established (shared meta-row check) and add
_require_run_visible_to_scope: for internal callers on threads without
established ownership, the run's own user_id stamp must match the
acting owner's raw value (or the legacy "default" stamp) or the read
404s. Established-ownership threads and every non-internal caller keep
their existing thread-scoped semantics.
- Wire the gate into join, stream, messages, events and
workspace-changes; reword the now-stale messages comment to track the
new scoping semantics.
Regression tests: sub-resource reads 404 for a mismatched internal
owner while the matching owner reads them normally, and the owner-less
fallback branch (synthetic "default" filter on missing-meta threads) is
pinned. Red confirmed against the pre-gate head.
* fix(gateway): gate cancel and artifact archive for internal callers
Review follow-up on #5484 round 2: POST /cancel resolved runs unscoped
(require_existing=True only closes the missing-meta case — NULL-owner
meta rows still pass), so an internal caller acting for a different
owner could interrupt another owner's active run on a shared thread
while /join and /stream were already gated. The archive manifest and
download pair likewise leaked the other owner's delivered-file count
and a 200-vs-409 delivery oracle on NULL-owner threads (missing-meta
threads were already denied by require_existing=True).
All three routes now call _require_run_visible_to_scope; its docstring
records the extended coverage. NULL-owner-thread regression tests pin:
a mismatched internal owner gets 404 from cancel, manifest and archive
download, while the acting owner reaches the real conflict path (409 on
a terminal run) and reads the manifest (file_count 2).
* fix(gateway): tolerate state-less request stand-ins in the scope helpers
The new owner-isolation gate and _run_scope_user_id read request.state
directly, which crashed the FakeRequest-based unit suites for the run
events, workspace-changes and scope endpoints (backend-unit-tests shards
1/2/4 on #5484). Read the state object defensively first: a request
without state is simply not an internal caller, so those paths keep
their pre-gate semantics.
* fix(gateway): scope the thread token-usage aggregate by owner
Review follow-up on #5484 round 4: GET /{thread_id}/token-usage called
aggregate_tokens_by_thread(thread_id) with no user filter at all, so on
missing/NULL-owner threads an internal caller acting for owner A read
owner B's spend, model names, run count and (with include_active=true)
live activity; the NULL-owner variant reached browser sessions too.
build_context_usage's latest-model lookup was unfiltered as well.
aggregate_tokens_by_thread gains an optional user_id (mirroring
list_by_thread: explicit None = unfiltered, AUTO resolves the contextvar)
in the memory store, the SQL repository and the store base;
build_context_usage/_resolve_thread_model_name thread the scope through
the latest-run lookup; the token-usage endpoint passes
_run_scope_user_id's value. Established-ownership threads aggregate
unfiltered as before; shared/missing-meta threads narrow to the acting
identity. Stale helper-test comment reworded after the #5482 merge
adaptation.
* test(gateway): pin the unfiltered aggregate on established-ownership threads
Review follow-up on #5484 round 5: the established-ownership branch of
the token-usage scoping (store receives user_id=None) was the only
unpinned half of the contract — the round-4 call-assertions never set
app.state.thread_store, so their None came from the user-less stand-in
path. test_token_usage_unfiltered_on_established_ownership_for_
internal_callers seeds an established meta row plus runs stamped by two
different identities and asserts the totals fold (166 = 111 + 55);
together with the isolation tests it now catches both failure modes
(always-stamp narrowing and always-None leak).
* feat(authz): filter per-caller skill visibility on the listing surfaces (#4063 Phase 4)
GET /api/skills, GET /api/skills/custom, and GET /api/skills/{name} now
filter the user-scoped catalog through filter_resources(principal,
"skill", ...) — mirroring list_models. Anonymous callers are unfiltered;
provider errors follow authorization.fail_closed (fail-closed -> empty
listing / 404, fail-open -> full listing). An invisible skill on the
detail surface returns the standard 404 so the endpoint cannot become an
existence oracle the filtered list closed. Management endpoints stay
require_admin_user-gated; runtime activation is #4541's layer.
resolve_skill_authorization joins resolve_model_authorization as a thin
sibling over a shared _resolve_route_scoped_authorization core.
* docs(authz): reflect per-caller skill visibility in OpenAPI metadata and implementation notes (#5489)
Address the two non-blocking review findings on #5489:
- The three user-facing GET routes (/skills, /skills/custom,
/skills/{name}) now say in their /docs-visible descriptions that
authorization filters the response (hidden skills 404 on detail).
- Add the dated Phase 4 decision-log entry to the authorization
implementation notes, per the convention of every prior merged
authz PR: listing-visibility semantics, the 404-vs-403
existence-oracle rationale, anonymous-caller behavior, and the
#4541 rebase reconciliation points (config.example.yaml roles
comment + this file's decision log).
* docs(authz): move route guidance into Gateway module guide
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The message edit/regenerate prepare chain resolves source runs through
_resolve_run_id_for_message, _require_successful_source_run and
_find_interrupted_target_run_id, which filtered by the authorization
identity (get_current_user) — the same conflation fixed for the runs and
messages read endpoints in #5448, left out of scope there. For trusted
internal callers the authorization identity never matches the raw
owner-stamped run rows, so regenerate/edit-regenerate prepare fails with
409 on threads the caller is authorized on (#5482).
The three helpers now resolve their filter id through _run_scope_user_id
as well; both prepare endpoints keep their owner_check=True
authorization and browser/API sessions keep the per-user filter.
Regression tests extend test_thread_runs_internal_scope.py to the helper
fallback paths: internal callers resolve raw-owner-stamped runs
(including the interrupted-run and status-409 paths), browser sessions
keep the per-user filter and 409 on cross-user runs.
* fix(goal): wait for the user when a turn ends on an unanswered question
ask_clarification and the sandbox network prompt put their question in a
ToolMessage and end the graph. The goal evaluator only reads human and AI text,
so it never saw the question, judged the goal not met, and the worker queued a
hidden continuation telling the agent to keep going while the card was still
open. The agent could then act on a guess before the user answered.
Stand the goal down with blocker needs_user_input, without calling the
evaluator, when the trailing tool results include a human input request.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(goal): cover resuming after answered clarification
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): expose incremental run evidence reader
* fix(extensions): address run evidence review feedback
* docs(extensions): clarify run deletion reconciliation
* docs(migrations): align current head documentation
* fix(extensions): isolate run evidence event reads
* test: avoid pinning run change migration to latest head
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash
Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md
(issue #5160, tracker #5129) in the slice order of the spec's §16.
Slices:
- A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY
admission pinning (both server-owned sets + worker hoist); latest-only
request-scoped <project> block via DynamicContextMiddleware
wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID
prefix + marker + provenance, never persisted); journal audit
fingerprints; Instructions tab.
- B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with
locked check-and-set; hash-qualified immutable shelf storage with
Paths helpers; upload/list/content/delete-to-trash routes; project
delete trashes the shelf in-transaction; request-scoped bounded
<documents> index with honest count/shown + actionable overflow note;
list_project_documents/read_project_document tools registered only on
pinned runs; PAT allowlist + drift guards; blocking-IO anchors.
- C: shared thread-upload ingestion service (uploads router refactored to
parity); POST from-thread with provenance; attach-to-thread with
lock-staged copy (archived source allowed); read-only thread-files
view with per-group truncation reporting.
- D: restore (restored/merged/not_found/no_target/content_missing; no
file moves), purge (continuous row lock across unlink/delete/commit,
retryable on FS errors), retention sweep (lazy + startup, 24h orphan
guard, row-side reconciliation never deletes).
- E: Documents tab (shelf + conversation-files browser, provenance,
archived banner, content-missing rows), /workspace/trash route,
sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks
+ specs.
Review hardening folded in (10 rounds, all with tests):
- force active shelf content (HTML/XML family) to download; nosniff on
artifact + content responses; unified unsandboxed-iframe PDF preview
(fixes the pre-existing Chromium sandbox blank in the artifact viewer)
- scope document trash to the URL project under the document lock
- atomic no-overwrite filename reservation for ALL ingestion (seeded
claims + os.link commit with suffix retry; same-name re-upload now
unique-names instead of replacing); hidden staging only, no visible
placeholders; lease cleanup on setup failure
- serialize conversion under the document lock with post-lock active
revalidation; drain locked filesystem work on cancellation; preserve
bytes when an insert's commit state is uncertain (including trashed
rows)
- original-integrity checks before serving text or cached conversions;
content_missing surfaced in list responses (UI reads the flag, no
409-probe); downloads always serve original bytes
- bounded streaming document reads with cached char counts; shelf limits
declared in middleware release identity
- thread-root confinement for from-thread sources; config fallback
rejects fractional/infinite values; composer counts staged
attachments; pending attachments persist until submission or removal;
in-flight instruction/rename edits survive save refetches; shelf and
trash pagination; conversation-file and thread-files pages stay
subscribed to refetches
Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md
contracts, config.example.yaml projects block.
Review follow-ups (head b4807477 → this revision):
- The trash retention sweep is split so repeated lazy triggers stay
bounded: the indexed expiry purge still runs on every trigger
(GET /api/trash/documents, POST /api/trash/purge) while the
O(all rows + all files) reconciliation is throttled to one run per
user per 15 minutes (process-local, per-user window). The startup
sweep now runs as a background task instead of blocking gateway
readiness, and shutdown awaits it (bounded).
- The export scrub (stripInternalMarkers) is fence- and indentation-aware
like the render path, so a pasted, fenced <project>/<documents> snippet
survives markdown export while real injected blocks (never fenced) are
still removed. Fence regexes moved to a dependency-free leaf module to
avoid the messages↔streamdown import cycle.
- The artifact viewer's PDF iframe no longer carries an added title
attribute (the upstream e2e contract locates it via :not([title])), and
the upstream artifact-preview spec now pins the new contract: PDFs
render unsandboxed, images keep sandbox="".
* fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title
- Shutdown cancelled only the shield around the background startup sweep,
so an all-users reconciliation that outlived the 5s budget kept walking
rows and files while the document repo and DB engine were disposed
underneath it. The wait now lives in `_shutdown_startup_trash_sweep`,
which cancels the task and drains it before worker exit: the shield
keeps the wait bounded, the cancel makes it final (CancelledError lands
at the sweep's next await, and `_run_startup_trash_sweep` only catches
`Exception`, so nothing swallows it).
- The browser-preview iframe lost `title={getFileName(filepath)}` in the
previous fix round, leaving the PDF frame without an accessible name
while its siblings keep theirs. Restore it (WCAG frame titles), assert
it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]`
instead of `iframe:not([title])`.
* fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel
`Task.cancel()` returns False when the sweep already finished inside the
window between the deadline firing and the cancel, so the shutdown log
claimed a cancellation that never happened. Branch on that outcome: the
warning stays for a real cancel, a late finish is logged at info, and both
paths still reap the task before worker exit.
* fix(projects): round-4 review — make Empty trash delete what it confirms
`POST /api/trash/purge` only ran the retention sweep, and the sweep's
candidate selection is age-gated, so a freshly trashed document survived
"Empty trash" even though the confirmation promises that every listed
document is permanently deleted. With one trashed row the route answered
`{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps
expired rows before listing, so the visible rows were normally ineligible
for the action by construction.
Empty trash now drives `purge_all_trashed`: the caller's trashed rows
(`list_all_trashed`, no age filter) each go through the same guarded,
row-locked `purge` as the single-document delete — bytes first, then the
row, in one transaction — so a row restored mid-flight is skipped instead of
force-deleted, and an unlink failure rolls that row back and answers 500 with
a retryable message. Retention expiry stays where it was: the sweep's
`purge_candidates` is now the only age-gated selection, and the lazy
retention sweep still runs on the listing and at startup.
Tests: the router suite replaces the retention-gated expectation with the
reviewer's repro (fresh row purged, bytes unlinked, shelf and other users'
trash untouched, a failing unlink stays retryable and 500); a blocking-I/O
anchor drives the new entry point through the offload; the mocked e2e covers
the action end to end; a new real-backend spec performs it against the real
gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE
and the phase-2 design docs (en+zh) state the age-independent contract.
* fix(channels): stream-cap and validate WeChat/WeCom inbound media downloads, fixes#5223
* fix(channels): address WeCom APPID, decompression, and log-sanitization review findings (#5223)
Round-4 review follow-ups on the inbound-media download cap:
- The COS bucket numeric suffix is the owner's Tencent Cloud APPID and bucket
names are user-chosen, so any Tencent Cloud account could register a
matching ww-aibot-img-* bucket and pass the shape gate. The built-in rule
now admits only the APPID observed in Tencent's published aibot callback
examples (1258476243), across regions; any other account (including a
future WeCom rotation) goes through channels.wecom.allowed_media_hosts.
- aiter_bytes() transparently decodes Content-Encoding, and the decoder
allocates the full decompressed body before the byte cap sees a chunk (an
~8 KB gzip wire chunk decoding to 8 MiB reproduces it). Both URL readers
now send Accept-Encoding: identity, refuse a response with a residual
Content-Encoding before reading, and iterate aiter_raw().
- httpx.HTTPStatusError formats the signed URL (path + query credentials)
into its message, so _ingest_inbound_files' reader-failure branch logs a
sanitized summary (class + status) instead of logger.exception, and the
WeChat extract paths catch httpx.HTTPError so the polling loop's
per-message logger.exception can never render a media URL.
Every change ships with a red/green regression: the reviewer's 403
mock-transport repro asserted against caplog.text (fully formatted logs),
the reviewer's different-APPID bucket host, and gzip bombs driven through
real httpx mock transports in both readers. Docs (channels AGENTS.md,
README, config.example.yaml) updated for the APPID pinning and encoding
gate.
* docs(channels): document why the inbound-media cap is 50 MB, not WeCom's 100 MB ceiling
* fix(logging): redact URLs in httpx request logs down to scheme + host, fixes#5223
httpx emits 'HTTP Request: GET <full URL>' at the Gateway's INFO level
before any response handling runs, so even successful signed-media
downloads leaked their credentials. HttpxUrlQueryRedactionFilter
(installed by configure_logging) rewrites those records in place — path
and query become /<redacted>, method/status/duration observability is
preserved — which also keeps Telegram's token-bearing Bot API paths out
of the logs. Reader-level regression tests run at production INFO level
with a real MockTransport, success paths included.
* fix(logging): blank userinfo credentials in httpx request-log redaction
* fix(logging): redact authority-only URLs and cover urllib3 redirect logs
Two follow-ups from the review plus one extrapolation of the same class:
- rest is now optional in _URL_REDACT_RE, so an authority-only URL
(scheme://user:pass@host, no path) is rewritten too — userinfo had
nowhere else to hide and previously passed through verbatim. A bare
credential-free origin still passes through unchanged.
- Renamed to UrlRedactionFilter / install_url_log_redaction and attached
to the urllib3 logger as well: urllib3 logs 'Redirecting <url> -> <url>'
at INFO with full URLs on both sides, the same leak class on a different
library logger. No gateway path today both uses requests and redirects
a signed URL, but the class stays closed instead of dormant.
- Unit tests now build records with the real httpx 0.28.1 format string
('HTTP Request: %s %s "%s %d %s"', 5 args) and httpx.URL args, per
the nit, instead of a synthetic shape httpx never emits.
* fix(logging): install URL redaction at handler level so propagated records are covered
A logging.Filter on a logger only runs for records emitted through that
exact logger — child loggers neither inherit it nor trigger it on
propagation — so the previous attachment to the bare urllib3 logger was
dead code: urllib3 emits Redirecting via urllib3.poolmanager at INFO and
urllib3.connectionpool at DEBUG. The filter is now attached to every root
handler (mirroring _install_trace_filter, which already iterates root
handlers; handler-level filters see propagated records) in addition to
the httpx logger (httpx emits via the bare name, and emission-point
coverage survives handlers added later). The wiring is pinned by tests
that emit through the real urllib3 child loggers — a mutation removing
the handler-level install turns them red. Comments, docstrings, and
AGENTS.md now state the actual emitter names and levels.
* fix(logging): redact urllib3 DEBUG request lines, whose split shape evaded the URL regex
urllib3's per-request line (connectionpool.py:545 on 2.7.0) renders as
`scheme://host:port "METHOD /path?query HTTP/x.x" status len` — the
authority ends at a space so _URL_REDACT_RE's bare-origin early return
applies, and the quoted origin-form target has no scheme, so neither half
was rewritten. UrlRedactionFilter now runs a dedicated request-line shape
first (collapsing the target to /<redacted>, keeping scheme+host+method+
version), then the absolute-URL pass. Regressions pin the exact format
string both at unit level and through the real urllib3.connectionpool
DEBUG emit path; AGENTS.md wording now names both covered DEBUG shapes.
* fix(logging): redact urllib3 retry lines and linearize scheme scanning
Closes the two open review threads on the inbound-media log hardening:
Retry/redirect targets: urllib3 logs the request target with no scheme in
five shapes the generic absolute-URL pass cannot see - `Retry: <target>`
(connectionpool.py:954 DEBUG), `Incremented Retry for (url='<target>')`
(util/retry.py:545 DEBUG, absolute on the redirect path), `Retrying (...)
after connection broken by '<err>': <target>` (connectionpool.py:869
WARNING, above the INFO root), and origin-form halves of both Redirecting
emitters (poolmanager.py:500 INFO / connectionpool.py:922 DEBUG). Each
gets a rewrite anchored to the exact urllib3 format, collapsing the
target to /<redacted>; the generic pass's rest now stops at quote
characters so a quoted URL keeps its closing punctuation (previously the
absolute-form increment line was mangled), and the request-line method
class accepts any case. The emitter enumeration in channels AGENTS.md is
closed against the installed urllib3 2.7.0 source.
Quadratic scanning: both scheme-bearing patterns start with a character
class, so re.sub retried every suffix of a long token - 64K paths cost
~1.8s and URL-free 64K error bodies ~3.1s per record, synchronously in
every root handler. The two passes are now driven from literal "://"
occurrences: _scheme_starts walks back over the scheme charset to each
run's first letter and the pattern is attempted only there, reproducing
re.sub's leftmost-non-overlapping result in linear time (256K path:
5.6ms; worst adversarial shapes <= 28ms). Long-input regressions pin the
URL-bearing and URL-free cases with mutation-verified bounds, plus
nested-scheme and digit-headed-run equivalence cases.
Validation: tests/test_logging_config.py 12/12; scheme-pass equivalence
against the old re.sub pipeline verified by two independent 30k+ case
fuzz runs; full-suite A/B against HEAD shows zero tests that pass on HEAD
and fail with this diff.
* fix(logging): boundary-aware quote stops and whole-message Redirecting anchor
Two follow-ups on the urllib3 redaction shapes:
Embedded quotes: `rest` treated ANY quote as a closing mark, so a URL
with an apostrophe in the path kept everything after it verbatim
(`https://h/path'quoted'?token=Q` rendered the credential suffix in
full) while the class docstring claimed path/query/fragment are
replaced. A quote now closes `rest` only at a boundary - followed by
whitespace, a closing parenthesis, or end of string - so urllib3's
Incremented Retry (url='...') scaffolding keeps its ') closer while an
embedded quote stays consumed. The increment line's url capture gets
the same rule narrowed to its fixed ')' closer.
Redirecting anchoring: the origin-half pass matched `(?
<=-> )/path` as a substring, and an `-> /path` arrow is not
urllib3-owned shape - the sandbox provider's actionable mount error
(`sandbox.mounts entry <host> -> /mnt/knowledge ignored: ...`) had its
container path rewritten to /<redacted>, failing
test_setup_path_mappings_logs_actionable_error_for_missing_host_path on
CI (backend-unit-tests shard 3). The pass is now anchored to the whole
`Redirecting <t> -> <t>` message, which is exactly urllib3's record;
origin slots collapse, absolute slots stay for the generic pass.
Regression tests pin the embedded-quote shapes and the sandbox error's
byte-for-byte passthrough; both mutations verified red.
Validation: tests/test_logging_config.py 14/14; the CI-failing sandbox
test green locally; every test file asserting redaction/arrow log
content passes (attachments, support bundle, run metadata, skill
secrets, ragflow, skillscan, sandbox provider); full offline backend
suite 14084 passed / 164 failed with the failure set matching this
machine's documented Windows-environment baseline (NTFS chmod/symlink,
docker/lark/langfuse absences) - no failure involves redaction output.
* fix(logging): redact redirects with spaced locations
* fix(logging): grammar-complete Redirecting anchor; neutral WeChat guard labels
Round-13 P3 (Redirecting anchor strictness): the whole-message anchor kept
the ^Redirecting prefix (the urllib3-owned literal that stops the sandbox
false positive) but required BOTH slots whitespace-free, so a Location
header with an interior space voided the pass and leaked the origin-form
request target in the first slot - redirect_location is the raw header
string and interior spaces are legal field syntax. The tail is now loose
(\S.*$) and the first slot gets the same grammar treatment (\S.*?): the
recursive urlopen frame passes the previous raw Location as its url, so
t1 can carry interior spaces too, lazy-split at the first arrow the way
the line is constructed. A space-carrying slot collapses whole when it
starts with /; the sandbox mount error keeps passing through untouched.
Round-14 nit (None conflation): _download_cdn_bytes returns None for two
reasons (in-flight cap abort, Content-Encoding refusal) but both image and
file callers labeled it "exceeds size limit (N bytes)" - contradicting the
accurate encoding line right above it, and reporting the plaintext limit
for a ciphertext-cap decision. Callers now log a neutral
"skipped by download guard" line (the manager reader callers' shape);
the accurate reason stays inside the download function. The same sweep
also logs _stage_downloaded_file's silent None (no state dir configured),
which made an attachment vanish with no log line at all.
Also anchors the emitter-enumeration closure to its urllib3 version: the
closure reopens if an upgrade changes these format strings, so the comment
now says so explicitly.
Validation: logging 15/15 and attachments 60/62 (the two pre-existing
Windows symlink-privilege failures documented in the PR body); three
mutations verified red (old wording, strict t1, silent staging None);
ruff clean. Full offline suite run before push (per round-11 lesson).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): scope runs read endpoints by data identity, not authorization identity
Trusted internal callers are authorized as a synthetic internal user
(id "default", or the make_safe_user_id-normalized owner with an owner
header), while start_run stamps run rows with the raw trusted-owner value.
list_runs / get_run / list_runs_page filtered by the authorization identity,
so the store-side user filter never matched and internal callers always saw
an empty runs list (or 404) for threads they are authorized to read.
The three read endpoints now resolve their filter id through
_run_scope_user_id: internal-role callers skip the per-user filter (thread
visibility is already authorized by owner_check=True), and browser/API
sessions keep the existing per-user filter unchanged.
Regression tests cover both identities across the three endpoints (list,
keyset page, single get) with a MemoryRunStore seeded with mixed-owner rows;
without the fix the four internal-caller cases fail while the
browser-session isolation case passes.
* fix(gateway): route the message read endpoints through the same data-identity scoping
Review follow-up on #5448: list_thread_messages and list_thread_messages_page
resolved get_current_user and passed it as the data filter to the event-store
scan, hidden-run lookups, turn-duration injection and the feedback queries —
the same authorization-vs-data identity conflation fixed for the runs
endpoints, leaving the #5437 empty-read symptom in place for lossy owner
values. Both endpoints now resolve their filter id through
_run_scope_user_id as well.
Regression tests extend to the two message endpoints, asserting the resolved
filter identity at the runs-store and feedback-repo boundaries (None for
internal callers, the session user id for browser sessions).
* fix(feedback): deterministic per-run collapse for unfiltered feedback reads
Review follow-up on #5448: with _run_scope_user_id returning None for
internal callers, the feedback lookups now receive an explicit-None user id,
which skips the user_id WHERE in FeedbackRepository. On shared/NULL-owner
threads several browser users can hold feedback on the same run, and
list_by_thread_grouped / list_by_run_ids collapsed rows per run_id via a
dict comprehension over unordered results — the feedback attached to the
last AI message would be an arbitrary user's row.
Both methods now order by created_at ASC with feedback_id as the tie-break,
so the collapse deterministically keeps the most recently created feedback.
_run_scope_user_id's docstring now documents that the resolved id also
scopes feedback and event-store reads, not just run rows.
Regression test seeds multi-user feedback on one run and asserts the
collapse outcome is stable across repeated unfiltered reads.
* docs(feedback): the collapse keeps the most recently written feedback
created_at is refreshed on upsert, so the surviving row per run is the
most recently written (created or updated), not the most recently
created — align both docstrings with the ordering key's actual semantics.
* feat(gateway): accept conversation references in run context and report the capability
LangGraph SDK clients build a fixed run body and drop unknown top-level
fields, so they cannot send the conversation_references field from #5399.
RunCreateRequest now lifts context.conversation_references into the
top-level field before validation, so it keeps the same bounds and error
locations, and drops it from context, so it never reaches the merged run
context or the checkpointed configurable. Sending both is a 422.
GET /api/features reports conversation_references {enabled, max_references}
with the same "tool is configured" predicate as run admission, so a client
can hide an entry point on deployments without the tool.
Related to #5398.
* fix(gateway): report the field type error for a malformed top-level reference list
A malformed top-level conversation_references sent alongside a context
list now fails with the field's own type error instead of the conflict
message. The tool-configured predicate reads tool.use directly, and the
features test doubles carry that attribute like every real ToolConfig.
* fix(gateway): treat every list-like top-level reference value as a conflict
Pydantic's lax mode coerces tuples, sets, frozensets and deques into the
list[str] field, so a direct Python caller passing one of those together
with context.conversation_references now reports the conflict instead of
slipping both grants through. Unreachable over HTTP, where JSON has no
such types.
* fix(gateway): ask pydantic whether a top-level reference value is list-like
Enumerating list-like types cannot track pydantic's lax acceptance set
(generators, UserList, dict key views also coerce into list[str]). The
conflict guard now validates the top-level value with a TypeAdapter for
list[Any]: whatever pydantic would coerce reports the conflict when it is
non-empty, and whatever it rejects still surfaces the field's own type
error. Regression tests cover deque, UserList, dict keys and a generator,
plus rejected scalars.
* fix(gateway): probe top-level references with the field's own annotation
The conflict guard now validates the top-level value with the exact item
annotation the field uses, so its acceptance set is the field's rather than
a superset: an item the field rejects (an empty string, a non-string, the
ints of a range or dict view) surfaces the field's own item error instead
of a conflict. The annotation is shared through one alias so the two cannot
drift.
* fix(gateway): materialise a one-shot iterator before probing top-level references
The item-validating probe could consume a generator while collecting an
item error, after which the field re-validated the exhausted iterator,
coerced it to [] and let the request through with the key still in
context. Iterators are now read once into a list that both the probe and
the field validate, so a bad item is reported at its index and a valid
generator is kept.
* fix(gateway): materialise every once-walkable iterable before probing references
Pydantic coerces any iterable into the list field, and an object whose
__iter__ hands out a generator once is not an Iterator instance, so the
previous gate let it reach the probe and be consumed. The lift now reads
every iterable except lists, tuples and the shapes the field rejects as a
whole (str, bytes, dict) into a list first, so the probe and the field
always validate the same items.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(discord): flush thread mappings on shutdown to survive restart (#2897)
The Discord channel already persists channel->thread mappings, but a mapping created right before a hard shutdown (process killed between thread creation and its background persistence write) could still be lost. Flush in-memory mappings in stop() as a best-effort safety net.
Add regression tests covering the persist/load round-trip across a simulated restart and the stop() flush path.
Refs #2897
* fix(discord): gate stop() flush on _thread_store_loaded (#5461 review)
Address review feedback:
- Only flush thread mappings from stop() once _load_active_threads() has
run, so a stop() taken before the load (start() bailed on a missing
bot_token / discord import error) cannot overwrite the store with {}.
- Fix isort order in the test module (ruff I001).
- Make the restart test a plain sync test (it never awaits).
- Add a regression test proving stop() does not clobber the store before load.
* fix(discord): gate stop() flush on _thread_store_loaded (#5461 review)
Address review feedback:
- Only flush thread mappings from stop() once _load_active_threads() has
run, so a stop() taken before the load (start() bailed on a missing
bot_token / discord import error) cannot overwrite the store with {}.
- Fix isort order in the test module (ruff I001).
- Make the restart test a plain sync test (it never awaits).
- Add a regression test proving stop() does not clobber the store before load.
---------
Co-authored-by: wcy12378 <wcy12378@users.noreply.github.com>
* feat(conversation): continue reading a cut message by offset
A referenced message longer than 4,000 characters was cut, and its
suffix could not be read back. Cut messages now carry a continuation
(message_seq, offset). read_conversation(thread_id, message_seq, offset)
returns the next part of that one message, sized to the same
tool-output budget as pages. The read scans only the requested row
under the existing visibility rules and rechecks ownership. Offsets
follow the source's current text; an offset past the end is rejected.
Related to #5398.
* docs(conversation): say continuations ignore limit
A continuation always returns one part of one message, so limit does not apply there. The tool schema now says so instead of discarding it silently.
Related to #5398.
* fix(conversation): stop instead of looping when no text fits the budget
With a read_conversation tool-output budget below the envelope size, the fitted text was empty and the continuation repeated the requested offset, so an agent would repeat the identical call forever. Page and continuation reads now return output_budget_too_small with no continuation.
Related to #5398.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(goal): stand the goal down once the run has hit its token budget
Since #5410 goal continuations share the run's token budget, so a
continuation queued after the budget's hard stop only spends one more
model call before its tool calls are stripped. Pass the run's stop_reason
into the goal loop and stand the goal down with "token_capped" in that
case. The evaluator still runs first, so a goal the capped run satisfied
is cleared as before.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(goal): list the token-budget stop among goal-loop preconditions
Also pin that a satisfied goal is cleared, not stood down, when the run
hit its token budget.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Document that read permission expiry and source deletion do not erase
text already copied into the destination conversation, and that reads
follow the source's current visible history. Truncated results now tell
the agent to acknowledge the omission and ask for the missing material
before claiming every requirement is covered.
Pages were filled to 20,000 text characters by cutting the last message
that did not fit, and that suffix could never be paged back. They could
also exceed the default 12,000-character tool-output budget, which
externalized the page to a file. Pages are now sized by their serialized
length against the read_conversation tool-output budget; a message that
does not fit starts the next page intact, so only a message over 4,000
characters (or one whose escaped JSON alone exceeds the budget) is cut.
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* feat(scheduled-tasks): filter run history by occurrence status
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* fix(scheduled-tasks): share occurrence status contract
---------
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): reject forged framework-injection markers in run input
`is_genuine_user_message` treats `hide_from_ui` and a human `name="summary"`
as proof the framework authored a message, and `InputSanitizationMiddleware`
skips those — escaping a real reminder's blocks would corrupt trusted context.
Neither marker was server-owned, so an external caller could set either one and
place a raw `<system-reminder>` outside the user-input boundary markers, which
the lead-agent system prompt declares trusted internal framework data. The
`hide_from_ui` variant is also filtered out of the thread UI, so the forgery
was invisible where it landed.
Both markers are now stripped from untrusted input, on the run path and on the
thread-state mutation route that writes straight into a checkpoint. Framework
injection happens inside the graph and never crosses this boundary, so nothing
the framework does is affected, and `trusted_internal` callers (IM channels,
the MCP task-notification launcher) keep writing hidden messages.
HumanInputCard replies are the one legitimate external `hide_from_ui`: the
frontend sends it alongside a `human_input_response` payload, so a message
carrying a valid one keeps the marker. That buys no bypass — the predicate
already classifies those as genuine, so they stay sanitized.
The name check reuses the predicate's own `_SUMMARY_MESSAGE_NAME` rather than a
fourth copy of the literal, and matches by `isinstance` exactly as the predicate
does: `HumanMessageChunk` is a `HumanMessage` whose `type` is not `"human"`, so
a type-based check would leave that subclass's marker settable. `name` is only
reserved on human messages — on a ToolMessage it is the tool's own name.
Three tests in test_gateway_services.py and test_message_provenance.py asserted
that a caller-supplied `hide_from_ui` survives. That assumption was the bypass;
they now pin the opposite, with a genuinely caller-owned key kept alongside to
prove the stripper is surgical.
* fix(agents): sanitize every genuine user message, not only the newest
The input guardrail scanned backwards for the first genuine user message and
returned, so only the newest turn was ever sanitized. The transformation is
request-scoped (`wrap_model_call`, never written to state), so thread state
keeps the raw text: once a newer turn arrived, the previous turn's payload was
replayed to the model verbatim, outside the boundary markers the lead-agent
prompt declares trusted framework data. The guardrail therefore held for
exactly one model call.
Reaching it needed no forged metadata and no crafted request body — type the
payload in one turn, then send anything at all in the next. A single request
carrying two user messages did it in one shot, since every message but the last
was skipped.
`_process_request` now walks the whole list and `_sanitize_message` owns the
per-message work; every existing branch (the `original_user_content` split for
upload turns, the multimodal rfind fallback, the metadata repair) is unchanged.
Framework-injected messages stay excluded by `is_genuine_user_message`.
Unexpected errors now fail open per message rather than per request. Iterating
history widened the old blast radius: one unprocessable row would have dropped
sanitization for the whole request, handing an attacker the newest turn by
crafting an older one. `GraphBubbleUp` still propagates.
Side effect worth noting: each turn's rendering is now stable across model
calls. Previously a turn was wrapped on its own call and unwrapped on the next,
changing the prompt prefix behind the newest turn and defeating prompt caching.
test_only_processes_last_user_message pinned the old scope; it now pins that
every turn is processed, and keeps driving the `wrap_model_call` entry point.
* docs: record the message-metadata trust boundary and sanitization scope
`agents/middlewares/AGENTS.md` owns the depth for InputSanitizationMiddleware
and documented only the `original_user_content` half of its trust boundary. Left
alone it would teach an agent that `hide_from_ui` is caller-owned and that the
guardrail covers one turn — and the usual failure mode is an agent "restoring"
the behaviour it believes was lost. The entry now carries both markers, the
HumanInputCard exception, the whole-history scope, and the per-message fail-open
rule.
The note lives only there. The root and `backend/AGENTS.md` layers are
orientation that points at the module guides owning the depth, and
`backend/AGENTS.md` is inherited by every backend chain — prose added there
inflates more than twenty of them, and `scripts/check_agent_guidance.py` shows
the middlewares chain has about a kilobyte of room against its hard limit.
CHANGELOG.md and CHANGELOG_zh.md record it under Security, continuing the
existing prompt-injection lineage.
* fix(gateway): mark caller-hidden messages instead of stripping the marker
Review follow-up. Stripping a caller-supplied `hide_from_ui` closed the bypass
but broke three frontend senders that use the marker purely to keep a context
message out of the transcript: the quoted conversation context
(`buildHiddenConversationQuoteMessage`), the sidecar context prompt
(`buildHiddenSidecarContextMessage`), and the agent save command. None carries a
`human_input_response`, so the HumanInputCard carve-out did not cover them, and
nothing else hides them — `_is_branch_visible_message` and the frontend's
`isHiddenFromUIMessage` both key solely on `hide_from_ui`, and no backend reads
`conversation_quote_context` or `sidecar_context`. All three would have rendered
as user-visible chat bubbles.
The marker plays two roles and only one of them is a vulnerability. The security
requirement is that a caller-supplied marker cannot skip sanitization, not that
it cannot hide a message. So the roles are separated instead of the marker being
removed: the Gateway keeps it and stamps the server-owned `UNTRUSTED_INPUT_KEY`,
and the guardrail now asks `requires_input_sanitization` — the mark, else the
genuine-user test. Hidden stays hidden; untrusted content is sanitized either
way. The reserved `summary` name is handled the same way and no longer rewritten.
Marking rather than removing is also the safer shape in general: `hide_from_ui`
is read for presentation, journal persistence, memory filtering and IM outbound
as well, and this boundary should not silently change any of them.
`is_genuine_user_message` is deliberately left alone. `ToolReceiptMiddleware`
uses it for turn-boundary detection, where a caller's hidden context message must
keep counting as not user-authored; widening it there would move the ledger's
turn window. `requires_input_sanitization` sits beside it in `message_utils` so
the two questions can be compared.
The three tests that asserted a caller-supplied `hide_from_ui` is removed now
assert it survives and carries the mark — which restores the original intent of
the two provenance cases, whose comment already read "caller-owned keys must
survive".
* docs(gateway): rewrite normalize_input's docstring around the mark
Review follow-up. The paragraph still described the pre-4a3344f6 strip model and
contradicted both the implementation and the middlewares AGENTS.md paragraph
updated in that same commit: it called `hide_from_ui` server-owned, said
carrying it skips sanitization entirely, and repeated the premise this branch
disproved — that HumanInputCard replies are the only legitimate external use.
It now describes what the code does: the markers stay caller-owned and are
preserved because three frontend senders rely on `hide_from_ui` for hiding
alone, the message is stamped with `untrusted_input` instead, and
`requires_input_sanitization` sanitizes it anyway. `untrusted_input` joins the
server-owned inventory in the preceding paragraph, which is what makes the stamp
unforgeable and unclearable.
The three surrounding docstrings now also say that these functions mark as well
as strip; `_strip_external_message_metadata` had advertised only the removal,
leaving a reader no way to find the stamp from the name.
* fix(gateway): mark state writes whose message omits additional_kwargs
Review follow-up. The state-route half of the fix missed the most natural
request shape. `_strip_external_metadata_from_message_like` returned early when
`additional_kwargs` was absent or not a dict — there was nothing to strip — and
that early return also skipped the mark. A `POST /threads/{id}/state` body of
`{"values": {"messages": [{"role": "user", "name": "summary", "content":
"<system-reminder>…</system-reminder>"}]}}` therefore reached the checkpoint
unmarked. The messages reducer's `convert_to_messages` then supplies
`additional_kwargs={}`, so at model-call time `requires_input_sanitization` fell
back to `is_genuine_user_message`, which a `summary` name fails, and the forged
tag reached the model raw and outside the boundary markers.
A missing or non-dict `additional_kwargs` is now treated as empty for both the
strip and the mark. The identity return is kept for the case where nothing
changes, so an ordinary key-omitted message does not gain an empty dict just by
passing through. The run path was never affected: `normalize_input` coerces to
BaseMessage first, which always carries the dict.
Every existing state-write test supplied an `additional_kwargs` dict, which is
why this shape slipped through; the regression now covers it at the route and
end to end through the reducer into the guardrail.
While checking the neighbouring shapes, `_skips_input_guardrail` keyed off key
presence where `is_genuine_user_message` keys off truthiness, so
`hide_from_ui: False` — already covered without a mark — would have been
stamped. It now mirrors the predicate exactly, as its docstring claimed.
* docs(middlewares): compress the sanitization note to fit the guidance chain
main's growth left the middlewares AGENTS.md chain 84 bytes under its hard
limit, and the fuller wording did not fit. The load-bearing facts stay — the
markers are marked rather than stripped, and the scan covers every turn — since
those are the two an agent editing this middleware could otherwise get wrong.
The full model lives in the normalize_input, _mark_untrusted_framework_markers
and requires_input_sanitization docstrings.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(settings): persist account preferences across browsers
* docs(settings): scope preference guidance to user persistence
* fix(settings): preserve SSR and fence custom-agent defaults
* test: include user persistence in scoped guidance inventory
* fix(settings): sync explicit edits and preserve local tab updates
* fix(runtime): keep agent construction off event loop
* fix:
- offload checkpoint state accessor graph construction to a worker thread
- update test
* import AsyncKeyedLockTable
* update Agents.md
* fix: update test
* fix: preserve single-flight builds after cancellation
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
