* feat(frontend): reference conversations from the composer
Adds a "Reference a conversation" button next to the attachment button,
shown only while GET /api/features reports read_conversation enabled. It
opens a picker over the recent-conversation list (current thread excluded,
capped at max_references) and shows removable chips in the composer.
On send the thread IDs ride SendMessageOptions.conversationReferences into
run context.conversation_references, which the Gateway consumes at
admission; the LangGraph SDK drops unknown top-level body fields. A
display-only copy ({thread_id, title}) on the visible human message lets
the transcript render read-only chips linking to the source.
References are per message: not persisted with the draft and cleared on
send or thread switch; regenerating or editing a turn runs without them
unless they are attached again.
Related to #5398. Depends on #5463.
* fix(frontend): pin the run-context contract and finish the picker states
Both thread.submit paths now build their run context through one exported
buildRunContext helper, tested directly: attached references travel as a
plain string[] under context.conversation_references only when the caller
passed them, a stray key in local settings is dropped instead of forwarded,
and the regenerate/edit replay path never carries references.
The picker shows a loading row while the conversation list is still in
flight instead of claiming there are no conversations, and the transcript
chip group is labelled with the previously unused referencedConversations
translation.
* fix(frontend): route conversation-reference chips to custom-agent sources
The picker offered custom-agent conversations but kept only the thread ID
and title, so transcript chips always linked to /workspace/chats/{id} and
dropped the source's custom-agent context on navigation.
Preserve the agent identity end to end: the picker now attaches
agentNameOfThread() (context first, then metadata.agent_name, mirroring
pathOfThread) to the selection, the display-only additional_kwargs metadata
round-trips it as agent_name, and the transcript chip passes it to
pathOfThread so custom-agent sources resolve to
/workspace/agents/{agent}/chats/{id}.
Tests: agent_name metadata round-trip and malformed-entry tolerance, picker
toggle carrying the metadata agent with run context winning, and a
picker-to-transcript regression pinning the /workspace/agents/writer/chats/
source-1 href.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* feat(extensions): expose incremental run evidence reader
* fix(extensions): address run evidence review feedback
* docs(extensions): clarify run deletion reconciliation
* docs(migrations): align current head documentation
* fix(extensions): isolate run evidence event reads
* test: avoid pinning run change migration to latest head
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash
Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md
(issue #5160, tracker #5129) in the slice order of the spec's §16.
Slices:
- A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY
admission pinning (both server-owned sets + worker hoist); latest-only
request-scoped <project> block via DynamicContextMiddleware
wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID
prefix + marker + provenance, never persisted); journal audit
fingerprints; Instructions tab.
- B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with
locked check-and-set; hash-qualified immutable shelf storage with
Paths helpers; upload/list/content/delete-to-trash routes; project
delete trashes the shelf in-transaction; request-scoped bounded
<documents> index with honest count/shown + actionable overflow note;
list_project_documents/read_project_document tools registered only on
pinned runs; PAT allowlist + drift guards; blocking-IO anchors.
- C: shared thread-upload ingestion service (uploads router refactored to
parity); POST from-thread with provenance; attach-to-thread with
lock-staged copy (archived source allowed); read-only thread-files
view with per-group truncation reporting.
- D: restore (restored/merged/not_found/no_target/content_missing; no
file moves), purge (continuous row lock across unlink/delete/commit,
retryable on FS errors), retention sweep (lazy + startup, 24h orphan
guard, row-side reconciliation never deletes).
- E: Documents tab (shelf + conversation-files browser, provenance,
archived banner, content-missing rows), /workspace/trash route,
sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks
+ specs.
Review hardening folded in (10 rounds, all with tests):
- force active shelf content (HTML/XML family) to download; nosniff on
artifact + content responses; unified unsandboxed-iframe PDF preview
(fixes the pre-existing Chromium sandbox blank in the artifact viewer)
- scope document trash to the URL project under the document lock
- atomic no-overwrite filename reservation for ALL ingestion (seeded
claims + os.link commit with suffix retry; same-name re-upload now
unique-names instead of replacing); hidden staging only, no visible
placeholders; lease cleanup on setup failure
- serialize conversion under the document lock with post-lock active
revalidation; drain locked filesystem work on cancellation; preserve
bytes when an insert's commit state is uncertain (including trashed
rows)
- original-integrity checks before serving text or cached conversions;
content_missing surfaced in list responses (UI reads the flag, no
409-probe); downloads always serve original bytes
- bounded streaming document reads with cached char counts; shelf limits
declared in middleware release identity
- thread-root confinement for from-thread sources; config fallback
rejects fractional/infinite values; composer counts staged
attachments; pending attachments persist until submission or removal;
in-flight instruction/rename edits survive save refetches; shelf and
trash pagination; conversation-file and thread-files pages stay
subscribed to refetches
Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md
contracts, config.example.yaml projects block.
Review follow-ups (head b4807477 → this revision):
- The trash retention sweep is split so repeated lazy triggers stay
bounded: the indexed expiry purge still runs on every trigger
(GET /api/trash/documents, POST /api/trash/purge) while the
O(all rows + all files) reconciliation is throttled to one run per
user per 15 minutes (process-local, per-user window). The startup
sweep now runs as a background task instead of blocking gateway
readiness, and shutdown awaits it (bounded).
- The export scrub (stripInternalMarkers) is fence- and indentation-aware
like the render path, so a pasted, fenced <project>/<documents> snippet
survives markdown export while real injected blocks (never fenced) are
still removed. Fence regexes moved to a dependency-free leaf module to
avoid the messages↔streamdown import cycle.
- The artifact viewer's PDF iframe no longer carries an added title
attribute (the upstream e2e contract locates it via :not([title])), and
the upstream artifact-preview spec now pins the new contract: PDFs
render unsandboxed, images keep sandbox="".
* fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title
- Shutdown cancelled only the shield around the background startup sweep,
so an all-users reconciliation that outlived the 5s budget kept walking
rows and files while the document repo and DB engine were disposed
underneath it. The wait now lives in `_shutdown_startup_trash_sweep`,
which cancels the task and drains it before worker exit: the shield
keeps the wait bounded, the cancel makes it final (CancelledError lands
at the sweep's next await, and `_run_startup_trash_sweep` only catches
`Exception`, so nothing swallows it).
- The browser-preview iframe lost `title={getFileName(filepath)}` in the
previous fix round, leaving the PDF frame without an accessible name
while its siblings keep theirs. Restore it (WCAG frame titles), assert
it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]`
instead of `iframe:not([title])`.
* fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel
`Task.cancel()` returns False when the sweep already finished inside the
window between the deadline firing and the cancel, so the shutdown log
claimed a cancellation that never happened. Branch on that outcome: the
warning stays for a real cancel, a late finish is logged at info, and both
paths still reap the task before worker exit.
* fix(projects): round-4 review — make Empty trash delete what it confirms
`POST /api/trash/purge` only ran the retention sweep, and the sweep's
candidate selection is age-gated, so a freshly trashed document survived
"Empty trash" even though the confirmation promises that every listed
document is permanently deleted. With one trashed row the route answered
`{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps
expired rows before listing, so the visible rows were normally ineligible
for the action by construction.
Empty trash now drives `purge_all_trashed`: the caller's trashed rows
(`list_all_trashed`, no age filter) each go through the same guarded,
row-locked `purge` as the single-document delete — bytes first, then the
row, in one transaction — so a row restored mid-flight is skipped instead of
force-deleted, and an unlink failure rolls that row back and answers 500 with
a retryable message. Retention expiry stays where it was: the sweep's
`purge_candidates` is now the only age-gated selection, and the lazy
retention sweep still runs on the listing and at startup.
Tests: the router suite replaces the retention-gated expectation with the
reviewer's repro (fresh row purged, bytes unlinked, shelf and other users'
trash untouched, a failing unlink stays retryable and 500); a blocking-I/O
anchor drives the new entry point through the offload; the mocked e2e covers
the action end to end; a new real-backend spec performs it against the real
gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE
and the phase-2 design docs (en+zh) state the age-independent contract.
* fix(channels): stream-cap and validate WeChat/WeCom inbound media downloads, fixes#5223
* fix(channels): address WeCom APPID, decompression, and log-sanitization review findings (#5223)
Round-4 review follow-ups on the inbound-media download cap:
- The COS bucket numeric suffix is the owner's Tencent Cloud APPID and bucket
names are user-chosen, so any Tencent Cloud account could register a
matching ww-aibot-img-* bucket and pass the shape gate. The built-in rule
now admits only the APPID observed in Tencent's published aibot callback
examples (1258476243), across regions; any other account (including a
future WeCom rotation) goes through channels.wecom.allowed_media_hosts.
- aiter_bytes() transparently decodes Content-Encoding, and the decoder
allocates the full decompressed body before the byte cap sees a chunk (an
~8 KB gzip wire chunk decoding to 8 MiB reproduces it). Both URL readers
now send Accept-Encoding: identity, refuse a response with a residual
Content-Encoding before reading, and iterate aiter_raw().
- httpx.HTTPStatusError formats the signed URL (path + query credentials)
into its message, so _ingest_inbound_files' reader-failure branch logs a
sanitized summary (class + status) instead of logger.exception, and the
WeChat extract paths catch httpx.HTTPError so the polling loop's
per-message logger.exception can never render a media URL.
Every change ships with a red/green regression: the reviewer's 403
mock-transport repro asserted against caplog.text (fully formatted logs),
the reviewer's different-APPID bucket host, and gzip bombs driven through
real httpx mock transports in both readers. Docs (channels AGENTS.md,
README, config.example.yaml) updated for the APPID pinning and encoding
gate.
* docs(channels): document why the inbound-media cap is 50 MB, not WeCom's 100 MB ceiling
* fix(logging): redact URLs in httpx request logs down to scheme + host, fixes#5223
httpx emits 'HTTP Request: GET <full URL>' at the Gateway's INFO level
before any response handling runs, so even successful signed-media
downloads leaked their credentials. HttpxUrlQueryRedactionFilter
(installed by configure_logging) rewrites those records in place — path
and query become /<redacted>, method/status/duration observability is
preserved — which also keeps Telegram's token-bearing Bot API paths out
of the logs. Reader-level regression tests run at production INFO level
with a real MockTransport, success paths included.
* fix(logging): blank userinfo credentials in httpx request-log redaction
* fix(logging): redact authority-only URLs and cover urllib3 redirect logs
Two follow-ups from the review plus one extrapolation of the same class:
- rest is now optional in _URL_REDACT_RE, so an authority-only URL
(scheme://user:pass@host, no path) is rewritten too — userinfo had
nowhere else to hide and previously passed through verbatim. A bare
credential-free origin still passes through unchanged.
- Renamed to UrlRedactionFilter / install_url_log_redaction and attached
to the urllib3 logger as well: urllib3 logs 'Redirecting <url> -> <url>'
at INFO with full URLs on both sides, the same leak class on a different
library logger. No gateway path today both uses requests and redirects
a signed URL, but the class stays closed instead of dormant.
- Unit tests now build records with the real httpx 0.28.1 format string
('HTTP Request: %s %s "%s %d %s"', 5 args) and httpx.URL args, per
the nit, instead of a synthetic shape httpx never emits.
* fix(logging): install URL redaction at handler level so propagated records are covered
A logging.Filter on a logger only runs for records emitted through that
exact logger — child loggers neither inherit it nor trigger it on
propagation — so the previous attachment to the bare urllib3 logger was
dead code: urllib3 emits Redirecting via urllib3.poolmanager at INFO and
urllib3.connectionpool at DEBUG. The filter is now attached to every root
handler (mirroring _install_trace_filter, which already iterates root
handlers; handler-level filters see propagated records) in addition to
the httpx logger (httpx emits via the bare name, and emission-point
coverage survives handlers added later). The wiring is pinned by tests
that emit through the real urllib3 child loggers — a mutation removing
the handler-level install turns them red. Comments, docstrings, and
AGENTS.md now state the actual emitter names and levels.
* fix(logging): redact urllib3 DEBUG request lines, whose split shape evaded the URL regex
urllib3's per-request line (connectionpool.py:545 on 2.7.0) renders as
`scheme://host:port "METHOD /path?query HTTP/x.x" status len` — the
authority ends at a space so _URL_REDACT_RE's bare-origin early return
applies, and the quoted origin-form target has no scheme, so neither half
was rewritten. UrlRedactionFilter now runs a dedicated request-line shape
first (collapsing the target to /<redacted>, keeping scheme+host+method+
version), then the absolute-URL pass. Regressions pin the exact format
string both at unit level and through the real urllib3.connectionpool
DEBUG emit path; AGENTS.md wording now names both covered DEBUG shapes.
* fix(logging): redact urllib3 retry lines and linearize scheme scanning
Closes the two open review threads on the inbound-media log hardening:
Retry/redirect targets: urllib3 logs the request target with no scheme in
five shapes the generic absolute-URL pass cannot see - `Retry: <target>`
(connectionpool.py:954 DEBUG), `Incremented Retry for (url='<target>')`
(util/retry.py:545 DEBUG, absolute on the redirect path), `Retrying (...)
after connection broken by '<err>': <target>` (connectionpool.py:869
WARNING, above the INFO root), and origin-form halves of both Redirecting
emitters (poolmanager.py:500 INFO / connectionpool.py:922 DEBUG). Each
gets a rewrite anchored to the exact urllib3 format, collapsing the
target to /<redacted>; the generic pass's rest now stops at quote
characters so a quoted URL keeps its closing punctuation (previously the
absolute-form increment line was mangled), and the request-line method
class accepts any case. The emitter enumeration in channels AGENTS.md is
closed against the installed urllib3 2.7.0 source.
Quadratic scanning: both scheme-bearing patterns start with a character
class, so re.sub retried every suffix of a long token - 64K paths cost
~1.8s and URL-free 64K error bodies ~3.1s per record, synchronously in
every root handler. The two passes are now driven from literal "://"
occurrences: _scheme_starts walks back over the scheme charset to each
run's first letter and the pattern is attempted only there, reproducing
re.sub's leftmost-non-overlapping result in linear time (256K path:
5.6ms; worst adversarial shapes <= 28ms). Long-input regressions pin the
URL-bearing and URL-free cases with mutation-verified bounds, plus
nested-scheme and digit-headed-run equivalence cases.
Validation: tests/test_logging_config.py 12/12; scheme-pass equivalence
against the old re.sub pipeline verified by two independent 30k+ case
fuzz runs; full-suite A/B against HEAD shows zero tests that pass on HEAD
and fail with this diff.
* fix(logging): boundary-aware quote stops and whole-message Redirecting anchor
Two follow-ups on the urllib3 redaction shapes:
Embedded quotes: `rest` treated ANY quote as a closing mark, so a URL
with an apostrophe in the path kept everything after it verbatim
(`https://h/path'quoted'?token=Q` rendered the credential suffix in
full) while the class docstring claimed path/query/fragment are
replaced. A quote now closes `rest` only at a boundary - followed by
whitespace, a closing parenthesis, or end of string - so urllib3's
Incremented Retry (url='...') scaffolding keeps its ') closer while an
embedded quote stays consumed. The increment line's url capture gets
the same rule narrowed to its fixed ')' closer.
Redirecting anchoring: the origin-half pass matched `(?
<=-> )/path` as a substring, and an `-> /path` arrow is not
urllib3-owned shape - the sandbox provider's actionable mount error
(`sandbox.mounts entry <host> -> /mnt/knowledge ignored: ...`) had its
container path rewritten to /<redacted>, failing
test_setup_path_mappings_logs_actionable_error_for_missing_host_path on
CI (backend-unit-tests shard 3). The pass is now anchored to the whole
`Redirecting <t> -> <t>` message, which is exactly urllib3's record;
origin slots collapse, absolute slots stay for the generic pass.
Regression tests pin the embedded-quote shapes and the sandbox error's
byte-for-byte passthrough; both mutations verified red.
Validation: tests/test_logging_config.py 14/14; the CI-failing sandbox
test green locally; every test file asserting redaction/arrow log
content passes (attachments, support bundle, run metadata, skill
secrets, ragflow, skillscan, sandbox provider); full offline backend
suite 14084 passed / 164 failed with the failure set matching this
machine's documented Windows-environment baseline (NTFS chmod/symlink,
docker/lark/langfuse absences) - no failure involves redaction output.
* fix(logging): redact redirects with spaced locations
* fix(logging): grammar-complete Redirecting anchor; neutral WeChat guard labels
Round-13 P3 (Redirecting anchor strictness): the whole-message anchor kept
the ^Redirecting prefix (the urllib3-owned literal that stops the sandbox
false positive) but required BOTH slots whitespace-free, so a Location
header with an interior space voided the pass and leaked the origin-form
request target in the first slot - redirect_location is the raw header
string and interior spaces are legal field syntax. The tail is now loose
(\S.*$) and the first slot gets the same grammar treatment (\S.*?): the
recursive urlopen frame passes the previous raw Location as its url, so
t1 can carry interior spaces too, lazy-split at the first arrow the way
the line is constructed. A space-carrying slot collapses whole when it
starts with /; the sandbox mount error keeps passing through untouched.
Round-14 nit (None conflation): _download_cdn_bytes returns None for two
reasons (in-flight cap abort, Content-Encoding refusal) but both image and
file callers labeled it "exceeds size limit (N bytes)" - contradicting the
accurate encoding line right above it, and reporting the plaintext limit
for a ciphertext-cap decision. Callers now log a neutral
"skipped by download guard" line (the manager reader callers' shape);
the accurate reason stays inside the download function. The same sweep
also logs _stage_downloaded_file's silent None (no state dir configured),
which made an attachment vanish with no log line at all.
Also anchors the emitter-enumeration closure to its urllib3 version: the
closure reopens if an upgrade changes these format strings, so the comment
now says so explicitly.
Validation: logging 15/15 and attachments 60/62 (the two pre-existing
Windows symlink-privilege failures documented in the PR body); three
mutations verified red (old wording, strict t1, silent staging None);
ruff clean. Full offline suite run before push (per round-11 lesson).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(gateway): accept conversation references in run context and report the capability
LangGraph SDK clients build a fixed run body and drop unknown top-level
fields, so they cannot send the conversation_references field from #5399.
RunCreateRequest now lifts context.conversation_references into the
top-level field before validation, so it keeps the same bounds and error
locations, and drops it from context, so it never reaches the merged run
context or the checkpointed configurable. Sending both is a 422.
GET /api/features reports conversation_references {enabled, max_references}
with the same "tool is configured" predicate as run admission, so a client
can hide an entry point on deployments without the tool.
Related to #5398.
* fix(gateway): report the field type error for a malformed top-level reference list
A malformed top-level conversation_references sent alongside a context
list now fails with the field's own type error instead of the conflict
message. The tool-configured predicate reads tool.use directly, and the
features test doubles carry that attribute like every real ToolConfig.
* fix(gateway): treat every list-like top-level reference value as a conflict
Pydantic's lax mode coerces tuples, sets, frozensets and deques into the
list[str] field, so a direct Python caller passing one of those together
with context.conversation_references now reports the conflict instead of
slipping both grants through. Unreachable over HTTP, where JSON has no
such types.
* fix(gateway): ask pydantic whether a top-level reference value is list-like
Enumerating list-like types cannot track pydantic's lax acceptance set
(generators, UserList, dict key views also coerce into list[str]). The
conflict guard now validates the top-level value with a TypeAdapter for
list[Any]: whatever pydantic would coerce reports the conflict when it is
non-empty, and whatever it rejects still surfaces the field's own type
error. Regression tests cover deque, UserList, dict keys and a generator,
plus rejected scalars.
* fix(gateway): probe top-level references with the field's own annotation
The conflict guard now validates the top-level value with the exact item
annotation the field uses, so its acceptance set is the field's rather than
a superset: an item the field rejects (an empty string, a non-string, the
ints of a range or dict view) surfaces the field's own item error instead
of a conflict. The annotation is shared through one alias so the two cannot
drift.
* fix(gateway): materialise a one-shot iterator before probing top-level references
The item-validating probe could consume a generator while collecting an
item error, after which the field re-validated the exhausted iterator,
coerced it to [] and let the request through with the key still in
context. Iterators are now read once into a list that both the probe and
the field validate, so a bad item is reported at its index and a valid
generator is kept.
* fix(gateway): materialise every once-walkable iterable before probing references
Pydantic coerces any iterable into the list field, and an object whose
__iter__ hands out a generator once is not an Iterator instance, so the
previous gate let it reach the probe and be consumed. The lift now reads
every iterable except lists, tuples and the shapes the field rejects as a
whole (str, bytes, dict) into a list first, so the probe and the field
always validate the same items.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(subagents): keep the subagent system prompt through context compaction
A subagent carries its whole system prompt (role, report contract, skills,
deferred tools) as the leading SystemMessage in state, because its agent is
built with system_prompt=None. Summarization only rescued tagged reminders and
the latest user message, so the first compaction summarized the prompt away
and every later model call in that subagent ran without its instructions.
Rescue system messages as well. The lead agent's prompt is not in state, and
its in-state SystemMessages are the tagged reminders already rescued, so the
lead chain is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(compaction): clarify preservation contract and trim agent guidance
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(conversation): continue reading a cut message by offset
A referenced message longer than 4,000 characters was cut, and its
suffix could not be read back. Cut messages now carry a continuation
(message_seq, offset). read_conversation(thread_id, message_seq, offset)
returns the next part of that one message, sized to the same
tool-output budget as pages. The read scans only the requested row
under the existing visibility rules and rechecks ownership. Offsets
follow the source's current text; an offset past the end is rejected.
Related to #5398.
* docs(conversation): say continuations ignore limit
A continuation always returns one part of one message, so limit does not apply there. The tool schema now says so instead of discarding it silently.
Related to #5398.
* fix(conversation): stop instead of looping when no text fits the budget
With a read_conversation tool-output budget below the envelope size, the fitted text was empty and the continuation repeated the requested offset, so an agent would repeat the identical call forever. Page and continuation reads now return output_budget_too_small with no continuation.
Related to #5398.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(sandbox): force UTF-8 console for PowerShell so CJK output is not garbled
LocalSandbox captures PowerShell output through a UTF-8 pipe reader
(errors=replace), but Windows PowerShell 5.1 writes console output in
the legacy OEM codepage (GBK on zh-CN Windows) unless told otherwise,
so every CJK character in tool output arrives as mojibake and the
decode never raises. Prepend a UTF-8 preamble
([Console]::InputEncoding/[Console]::OutputEncoding/$OutputEncoding)
to the -Command payload so both directions of the console are UTF-8
before the user command runs.
* fix(sandbox): pair PowerShell UTF-8 capture and guard console setup
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): close SkillScan bypasses in the skill review gate
The public skill review gate re-materialized a package snapshot into a
temp directory for SkillScan, but copied only entries the reader had
decoded as text and skipped every file under any evals/fixtures/
directory. Executable binaries and nested archives never reached the
package rules, and a fixture-shaped path hid any script from the scan.
Readers now keep binary bytes as content_base64, the analyzer writes
every non-symlink file byte for byte, and only eval fixture SKILL.md
samples stay exempt. Files are created exclusively, so a duplicate
archive member or a case-folded name fails the scan closed instead of
overwriting an earlier file.
SkillScan itself skipped any file that was not NUL-free UTF-8. One
Latin-1 byte in a comment hid a reverse shell from the review gate, and
a NUL byte skipped static analysis at install. Code files that fail
strict decoding now raise package-undecodable-script (HIGH) and are
analyzed over a lossy decode, so CRITICAL matches keep blocking.
"Code file" and "executable magic" were defined separately in the
installer and SkillScan and had drifted: SkillScan missed 32-bit
little-endian and fat Mach-O variants the installer blocks. Both rules
now live in skills/package_files.py, shared by the installer, the export
guard, and SkillScan.
* docs(changelog): link the skill review gate fix to #5431
* fix(skills): fail closed on bytes-less snapshot entries and skip text rules for executables
The review analyzer skipped any snapshot entry it could not turn into
bytes. Readers only emit such entries for oversized files, and they also
mark the snapshot truncated, but content_base64 is optional in the
contract, so a reader regression or a hand-built snapshot would silently
drop a file from SkillScan. An entry without bytes now fails the scan
closed (not_assessed: skillscan) unless the snapshot is truncated, and a
text entry without content no longer materializes as an empty file.
A real executable under scripts/ is a code file, so SkillScan decoded it
lossily and ran the text rules over its string tables. An OpenSSH binary
produced a CRITICAL secret-private-key finding from the key-format
banner it embeds. An undecodable file with executable magic still
reports package-undecodable-script, and its CRITICAL
package-executable-binary finding already blocks it, so it now skips the
text rules. Decodable files keep full text analysis.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Document that read permission expiry and source deletion do not erase
text already copied into the destination conversation, and that reads
follow the source's current visible history. Truncated results now tell
the agent to acknowledge the omission and ask for the missing material
before claiming every requirement is covered.
Pages were filled to 20,000 text characters by cutting the last message
that did not fit, and that suffix could never be paged back. They could
also exceed the default 12,000-character tool-output budget, which
externalized the page to a file. Pages are now sized by their serialized
length against the read_conversation tool-output budget; a message that
does not fit starts the next page intact, so only a message over 4,000
characters (or one whose escaped JSON alone exceeds the budget) is cut.
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* feat(scheduled-tasks): filter run history by occurrence status
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* fix(scheduled-tasks): share occurrence status contract
---------
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(settings): persist account preferences across browsers
* docs(settings): scope preference guidance to user persistence
* fix(settings): preserve SSR and fence custom-agent defaults
* test: include user persistence in scoped guidance inventory
* fix(settings): sync explicit edits and preserve local tab updates
* feat(memory): add deterministic near-duplicate fact gate
Add an opt-in write-side gate for DeerMem (issue #5252): a proposed NEW
fact whose bounded token-Jaccard similarity to an existing fact in the
same user/agent scope AND category reaches
fact_dedup_similarity_threshold merges into that fact instead of being
appended — the existing id/content/createdAt are kept, confidence is
raised to the maximum, and the source is refreshed, with one
facts_merged_dedup metric increment recording the merge.
Defaults preserve the legacy behavior exactly; targeted updates by fact
id and the exact-content key check are untouched. Companion write-side
step to the read-side relevance/diversity work in #5251.
Refs #5252
Signed-off-by: pwd11 <fvdsrc@163.com>
* fix(memory): preserve corrections during fact deduplication
Signed-off-by: pwd11 <fvdsrc@163.com>
---------
Signed-off-by: pwd11 <fvdsrc@163.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): add in-place upgrade that keeps private config
Replace a managed local snapshot or re-pin an already-installed
requirement without going through remove, which dropped plugins[].config.
* fix(extensions): keep snapshot, enabled, and git re-pin on upgrade
Rollback keys off staging_root so a failed snapshot rename cannot
rmtree the live tree. Upgrade preserves plugins[].enabled. Re-pin
identification uses tool.uv.sources so git upgrades adopt the
existing plugin record instead of failing closed after uv already
switched the revision.
* test(extensions): cover requirement re-pin identification on upgrade
Re-pinning deerflow-extension-demo==2.0.0 to ==3.0.0 leaves added_names
empty, so identification must take the added_specs fallback. Assert
private config/required/enabled survive and the lock records 3.0.0.
* fix(extensions): reject upgrade of an uninstalled git source
Bare git+ URLs are not named Requirements, so the pre-uv-add installed
check was skipped and upgrade acted as install. Resolve them against
[tool.uv.sources] in the extensions group before uv add.
* fix(uploads): bound document outline and preview text
Bound each document outline title to 200 characters and each fallback preview to 2000 characters across its lines, including omission markers.
Refs #5322
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* test(uploads): cover exact preview budget and restore title guidance
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
---------
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* feat(extensions): allow constructor kwargs on config-declared middlewares
extensions.middlewares entries may be a class-path string or {class, kwargs}.
String entries keep the zero-argument constructor. Unknown fields and blank
class paths fail at config validation. Constructor errors still fail at
agent creation.
Fixes#5311
* fix(extensions): coerce middleware kwargs to JSON types
YAML timestamps became datetime objects while JSON kept strings, so
constructors and to_file_dict() json.dump saw different types. Validate
kwargs as JSON types at config load, stringify dates, reject NaN and
other non-JSON values, and cover the raw-dict loader branch.
* style(extensions): wrap middleware Field description for ruff E501
make lint failed: the middlewares description was 289 chars (limit 240).
Wrap it and run ruff format on the two files this PR last touched.
* docs: compact configured middleware guidance to satisfy size limit
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): serve XML artifacts as attachments to block same-origin script
GET /api/threads/{id}/artifacts/{path} forced only text/html,
application/xhtml+xml and image/svg+xml to download. Every other XML
document was served inline from the application origin: `.xml` guesses
to text/xml or application/xml depending on the host's mime.types, and
both fell through to the inline text branches. Browsers render any XML
MIME type as a document and run an XHTML-namespaced <script> inside it,
so a report.xml written by a prompt-injected agent and opened from a
chat link executed with the viewer's session: the HttpOnly access_token
rides same-origin fetches, and the double-submit csrf_token cookie is
JS-readable, so state-changing calls are reachable as well.
Treat HTML plus every WHATWG XML MIME type (text/xml, application/xml,
any +xml subtype) and text/xsl, which Blink also renders as XML, as
active content. A single helper owns the rule for both the regular-file
and the .skill-archive-member branches. The artifacts panel already
previews .xml as code through a ranged fetch, so preview and editing
keep working against the attachment response.
* docs(frontend): name XML among the artifacts the Gateway downloads
Review follow-up on #5353: resolveArtifactOpenURL's comment still named
only HTML/SVG as the active content the Gateway serves as a download.
XML documents now join that bucket, so the frontend note matches the
Gateway rule. Comment-only; no behavior change.
* fix(scheduler): reconcile stuck once tasks from committed run outcome
Restart recovery (cancel_stuck_once_tasks and the multi-instance
reconcile_stuck_once_tasks) blindly flipped every stuck once-task to
'cancelled'. When handle_run_completion crashed between its two
transactions, a once-task whose run had already committed 'success'
was permanently reported as cancelled.
Both reconciliation paths now read the latest scheduled_task_runs row
without a status filter and finalize the parent to match:
success -> completed (last_error cleared),
failed -> failed with the run's error,
interrupted -> cancelled with the run's error when present,
skipped -> cancelled (no work performed).
Active occurrences (queued/launching/running) are left untouched — a
concurrent completion or a later recovery pass will finalize them once
the run reaches a terminal state.
Tasks without a terminal run row keep the previous generic cancellation.
Review follow-ups (willem-bd / Huixin615):
- Extract _finalise_once_task_from_run() so both recovery paths share one
outcome mapping (no more drift between single- and multi-instance paths).
Returns bool (True = finalised, False = active/no-op) for explicit
counter management at call sites.
- Fix a no-op (`run_row.error or None` -> `run_row.error`) in the skipped
branch.
- Drop the unused `status` parameter from the test task helpers.
- Use TERMINAL_RUN_STATUSES / ACTIVE_RUN_STATUSES constants (local copies
to avoid circular import; kept in sync with scheduled_task_runs.sql).
- [P1] Read the latest run AFTER acquiring the parent task row lock, not from
a pre-lock batch snapshot. The latest-run lookup now runs per task under
the lock with populate_existing so a concurrently committed status is read
back fresh.
- [P2] Race tests now use monkeypatch to actually enter the race window:
_intercepted_fetch commits success in a separate session at the moment the
per-task fetch fires, so a reverted pre-lock batch implementation fails the
test, while the current post-lock implementation passes.
- [P1] Do not finalize parent for active occurrences. A non-terminal
scheduled occurrence means the run is still in progress — the parent must
be left untouched until the completion path or a later recovery pass
establishes a terminal outcome.
- [P2] Add cancel_stuck_once_tasks to the single-instance poll loop so
stuck once-tasks are not left permanently "running" when the startup sweep
fails (mirrors multi-instance _reconcile_active_state behavior).
- Fix stale docstrings in cancel_stuck_once_tasks and _fetch_latest_run.
Adds regression tests for multiple historical runs (older success +
newer skipped/active) on both paths, monkeypatch-based race tests that
prove a concurrent completion committing success is reflected as
completed, and active-run tests that verify the parent is left
unchanged. Documents the behavior in AGENTS.md.
Fixes#5034
* fix(scheduler): address review comments on completion-consistency fix
- _fetch_latest_run: drop arbitrary id DESC tie-break; order by
scheduled_for DESC (deterministic recency on schedule position)
- _finalise_once_task_from_run: annotate bool return type
- Centralize TERMINAL/ACTIVE_RUN_STATUSES in scheduled_tasks/model.py;
stop duplicating them in scheduled_tasks/sql.py and
scheduled_task_runs/sql.py (removes stale circular-import workaround)
- cancel_stuck_once_tasks: run unconditionally in single-instance poll
loop (remove try/except swallow)
- tests: pin created_at/scheduled_for in _create_run so recency ordering
is actually exercised; correct docstrings that described the
active-occurrence branch as 'generic cancel' instead of 'left unchanged'
* fix(scheduler): correct finalizer return annotation
* fix: order scheduled task runs by creation time
* fix(scheduler): stabilize latest run reconciliation ordering
* fix(scheduler): order latest runs by creation time
* test: update trace scheduler stub
* fix(scheduler): clarify reconciliation diagnostics
Signed-off-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(scheduler): fail closed on startup recovery
Keep single-instance parent reconciliation at startup so it cannot race manual admission. Propagate recovery failures through the Gateway lifespan before channel startup, preventing a half-started scheduler.
Tests cover both recovery failure stages and a queued occurrence that survives startup before the ordinary poll drain launches it.
Signed-off-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(scheduler): order occurrences and fence stale parent writes
Allocate per-task occurrence sequences under the parent lock and guard parent projection across launch, recovery, completion, and queue failure paths. Track launch accounting separately so stale occurrences are counted once without replacing newer results. Commit completion and accounting atomically, preserve legacy history, and cover migrations and reordered execution on SQLite and PostgreSQL.
* fix(scheduler): tighten completion projection and launch fencing diagnostics
Share the once-task outcome mapping between completion and both recovery paths, validate the terminal status before opening the completion transaction, leave cron parent status untouched on completion, log the fenced launch update when an occurrence does not belong to the launched run, and drop the README capability line.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(scheduler): compare caller time only against unsequenced occurrences
Among sequenced rows the parent-locked occurrence_seq is the only recency key. An unsequenced row can only be legacy history or an admission by a pre-upgrade Gateway writer, so recovery prefers it over the sequence winner only when its caller timestamp is later, which is the previous ordering for that pair. A rolling upgrade therefore degrades to the pre-sequence behaviour instead of ranking every pre-upgrade admission below every sequenced one. Document that boundary instead of requiring every Gateway writer to stop before the upgrade.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(scheduler): gate once-task recovery on the same projection rule
Recovery now finalises a once-task parent only from the occurrence that can_project() accepts: the highest sequenced occurrence whenever one exists, or the timestamp-latest row for a task whose history is entirely unsequenced. An unsequenced row admitted by a pre-upgrade writer can no longer cancel a parent whose sequenced occurrence is still live, nor stall finalisation of a parent whose sequenced occurrence already completed. Document that pre-upgrade instances project their own admissions during a rolling upgrade.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(scheduler): defer once-task recovery while any occurrence is live
uq_scheduled_task_run_active allows one non-terminal occurrence per task, so a live row is the task's newest admission whatever its caller clock and whether it carries a sequence. Both once-task recovery paths now probe for any active occurrence after the fresh latest-run read and leave the parent untouched while one exists; cancel_stuck_once_tasks also locks the parent row so admission cannot insert a queued occurrence between that probe and the commit. Once no occurrence is live, the sequence winner decides and a terminalised unsequenced row never overrides it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(persistence): follow the local head past canonical 0019
Main's forward-revision tests assumed 0019_thread_incarnations was the local chain head. With 0022_scheduled_occurrence_seq chained after it, seed the canonical-0019 shape explicitly, assert the real head where a database is upgraded, derive the 0020 rollback binary's revision set from the ancestors of its head, and step the PostgreSQL restart scenario back to canonical 0019 before the rollback binary restarts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs(migrations): describe the chain through 0022_scheduled_occurrence_seq
The rolling-forward section still ended the local chain at canonical 0019; it now names 0022_scheduled_occurrence_seq as the head and lists it among the revisions the 0020 rollback-floor binary does not know.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(scheduler): accept CI's sync Postgres URL in occurrence fixtures
CI hands over TEST_POSTGRES_URI as postgresql://...?sslmode=disable. The occurrence, ordering and 0022 migration fixtures built async engines from it directly, so SQLAlchemy chose psycopg2, which is not installed. Normalize the scheme to postgresql+asyncpg and drop libpq-only query keys, matching the existing 0019 migration tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(scheduler): keep the backend AGENTS.md chain within its budget
The middlewares guidance chain was already above the hard limit on main, so any added byte in backend/AGENTS.md fails the agent guidance check. Leave backend/AGENTS.md identical to main and record the recovery projection rule in the 0022 migration entry, which already describes the occurrence fields.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Signed-off-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(web-fetch): resolve relative URLs in extracted Markdown
Pass the request URL through Jina and Browserless extraction and resolve
link/image destinations before Readability removes document base tags.
Preserve the optional legacy API and fallback text behavior.
Fixes#5307
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* fix(web-fetch): address provider and base URL review feedback
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* fix(web-fetch): preserve HTML source when resolving destinations
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
---------
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* feat(scheduler): add interval schedule type
Allow scheduled tasks to fire every N seconds from last dispatch, not
only wall-clock cron or a single run_at. Cadence is UTC now+N with no
missed-beat catch-up, bounded by min_once_delay_seconds and 30 days.
* fix(scheduler): let interval tasks create, edit, and keep next run
Create/edit now keep every_seconds. Unchanged interval spec no longer
resets next_run_at, including timezone-only PATCH.
* fix(scheduler): keep non-minute intervals on edit
Stop rounding every_seconds to whole minutes in the form. Values that
are not whole minutes or hours now use a seconds unit so edit/duplicate
round-trips the stored cadence instead of rewriting it and resetting
next_run_at. Document that min_once_delay_seconds is also the interval
floor.
* fix(scheduler): clamp interval seconds to the default 60s floor
The new seconds unit allowed 1–59, which the API rejects under the
default min_once_delay_seconds. Clamp the form to >= 60 and show the
floor next to the preview. Also mention interval in the scheduler
field_doc, matching config.example.yaml.
* fix(scheduler): do not clamp interval amount while typing
Keystroke clamp made 90 become 9 -> 60, then 600, and backspace could
not leave 60. Keep the raw field text and apply the 60s floor on blur
and emit only.
* test(scheduler): cover interval input editing
* fix(frontend): preserve saved interval cadence until edited
* style(tests): format scheduled task router tests
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(scheduler): let scheduled tasks pin a custom agent
Create and update accept optional assistant_id, defaulting to lead_agent.
Custom names are normalized and must already exist for the task owner.
The workspace form exposes the same choice, and duplicate copies it.
Fixes#5286
* fix(scheduler): keep assistant-id PR free of interval tests
Drop the six interval tests that belonged to the interval schedule PR
and fail here because this tree still only accepts once/cron.
Treat lead_agent case-insensitively so LEAD_AGENT / lead-agent store
as the default. Omit unchanged assistant_id on edit so a deleted custom
agent does not 422 unrelated PATCH (rename, reschedule).
* fix(scheduler): format task page and browser tests
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* test(threads): add red R3/R4 merge ordering regressions
* fix(threads): preserve trusted seq positions through content merge (R3/R4)
Extract the message ordering/identity logic into a pure
core/threads/message-order.ts module. Each normalized identity now tracks
latest visible content and trusted position separately: content replacement
no longer drops deerflow_seq/run_id/turn_duration (R3), and a seq-carrying
live message is placed by the ascending seq skeleton instead of the next
shared identity anchor (R4: 1,3,2,5 -> 1,2,3,5).
buildVisibleHistoryMessages converges repeated identities to the earliest
visible feed row (mirroring backend get_message_seqs), and the
summarization transient bridge plus rendered ledger share the same
position priority: trusted seq outranks anchor weaving, bridge refreshes
keep known seqs, and hidden control copies never contribute a visible
position.
* test(e2e): add long-thread ordering regression with compaction and pagination
Add tests/e2e/thread-ordering.spec.ts: a deterministic 68-row, 33-turn
fixture with two hidden compaction summaries, a paginated /messages/page
mock, and a live compaction during submit (real SSE frame shapes). Asserts
DOM group order at stage barriers, outline/scroll navigation across the
virtualized list, tool-card association, and order stability across reload.
Also close three mock gaps in mockLangGraphAPI (token-usage, mcp-tasks,
workspace-changes): unmocked they fell through to the absent gateway and
the 401 redirected thread pages to /login, breaking every thread-page spec
in a gateway-less Playwright environment.
* test(threads): address review on ordering regression coverage
- e2e: actually expand the collapsed web_search step and assert the
intermediate result payload (realistic JSON array fixture); assert the
new turn's DOM relative order via compareDocumentPosition instead of
racing viewport coordinates; add a Custom Agent route regression sharing
the same paginated fixture.
- Add a unit test for the hidden-control-only seq fallback path.
- Keep isNonEmptyString in hooks.ts (message-order.ts does not use it).
- Document the seq-first position authority contract in frontend/src/AGENTS.md.
* test(threads): type run_id fixtures via getMessageRunId accessor
* test(e2e): exercise the real collapsed-steps region for the tool payload check
The previous toolStep.click() was a no-op: as the last tool call, the
web_search step rendered unconditionally. Add a second tool call to the
turn-30 fixture so web_search falls into the collapsed moreSteps region,
assert the intermediate result payload is hidden while collapsed, then
click the "1 more step" button and assert it becomes visible.
* docs(frontend): prettier-format AGENTS.md merge contract
* fix(frontend): anchor mixed-sequence message segments
* test(auth): include project permissions in me contracts
* fix(frontend): anchor trailing steps to positioned live results
* fix(frontend): preserve prefixes before rescued sequence anchors
