The two scan-limit warnings on the regenerate and edit-replay paths
logged the assistant and user message ids raw. Those ids come from
checkpoint state, and run input keeps a client's message id verbatim,
so a thread owner could start a forged line in the Gateway's text log.
Pass them through sanitize_log_param like the other warnings in the
same functions.
* fix(agents): run the calls that fit when a batch crosses a per-tool loop limit
One model response with 50 or more calls to the same tool lost every
call: the per-tool frequency check stopped the run before any of them
ran (#8214).
- A per-tool limit now drops only the calls that would reach it; the
calls within the limit run. A response in which no call fits still
stops the run, and the identical-call hard limit still rejects the
whole batch.
- The capped tool stays blocked for the rest of the execution, whatever
the frequency window counts. The cap is a record in the private
loop_cap state channel, so it survives an approval pause, the resume's
new run_id, a Gateway restart and context compaction. before_agent
expires a record another run left behind, which gives a later task a
fresh budget.
- The run is loop_capped from the first dropped call, in the run record
and in a subagent result. A resumed run is marked again from the
record.
- The model is told on its next call how many calls were skipped and how
many were passed on, followed by the wrap-up instructions. The notice
is read from the record on every model call and cleared once the model
has answered.
- The Gateway treats loop_cap as a server-owned state channel.
- middleware:loop_detection gains the action reject_excess.
* fix(agents): count after the window eviction and keep the cap record readable
Review follow-up for #8267:
- A call reaches a per-tool limit when the window would hold that many
calls to its tool after taking the call in and dropping its oldest
entry, as on main. Counting before the eviction rejected a call that
only replaced an older call to the same tool, so 49 reads, one other
call and one more read capped read_file where main admits 49 more.
- The cap record holds only what its reader accepts. A tool whose name
the reader would reject, or one past the record's size bound, is left
out instead of making the next read discard every cap in the record.
The record and its notice are still written, so the drop is explained
and the execution stays loop_capped.
- A dropped batch is audited once per tool that reached its limit in it,
with that tool's own count and threshold.
- Tests: a seeded comparison against a plain sliding window, the
interleaved sequence, unnameable and overflowing records, and the
per-tool audit rows.
* fix(agents): report a partial loop cap when the run ends, not mid-run
Review follow-up for #8267. runtime.context["stop_reason"] is read in
two ways: the worker and Goal continuation read it when the run is over,
while run-input admission, the todo middleware and end-of-graph input
handling read it mid-run as "the run is ending" and skip their checks.
A partial cap wrote it at the first dropped call, while the admitted
calls of that response still had to run, so those calls skipped the
checks:
- input that arrived during inference no longer cancelled the planned
calls, so obsolete writes ran after a steer;
- two parallel write_todos calls both ran, each replacing the plan;
- input that arrived as the run finished was dropped.
A response with no call left is still a terminal stop and is reported
at once. A partial cap is reported to the subagent channel at once and
to the run context by after_agent, which runs after the end-of-graph
input handling; another guard's earlier reason is kept. A run that
pauses for approval no longer reports a reason; the run that resumes it
reports loop_capped when it ends.
Tests drive the three cases through real compiled graphs in the lead
chain's middleware order.
* fix(agents): undo a discarded response's loop accounting and report the guard that stopped the run
Third review follow-up for #8267.
- A response whose admitted calls another middleware discards before any
of them runs (input admission on a steer, approval on calls it cannot
address) no longer leaves a cap or counts toward a limit. As the next
model call starts, the frequency window, warnings and cap record go
back to their state before that response; caps from earlier responses
stay. Before, a steer whose replacement needed the same tool had that
write dropped as already capped. The undo runs in wrap_model_call, not
a before_model hook, so no graph node is added and lead runs do not
spend an extra super-step per turn of their recursion limit.
- A partial cap reports to both channels only when the run ends, and
only when no other guard has given the reason it stopped. A subagent
that a later token-budget stop ended now reports token_capped, as its
run context does, instead of loop_capped.
Tests drive the steer cases through compiled graphs with real input
admission, including a replacement that uses the capped tool and an
earlier cap that must survive the undo, and the token-budget case sync
and async.
---------
Co-authored-by: AochenShen99 <sac1063467113@hotmail.com>
* feat(extensions): recover opt-in workflows unattended after restart
Workflow launches from an authenticated browser session may opt in to
unattended recovery. The launch captures the session's existing authority
ceiling into a bounded, revocable grant on the workflow record; after a
Gateway restart the single-Gateway SQLite store resumes recoverable
records holding a usable grant through the existing journal-replay path,
without a sign-in.
Host: restore_delegated_authority (sibling of restore_batch_authority)
plus capture_authority on bound handles; the per-operation chokepoint
runs an extension-supplied grant validator so expiry and revocation fail
closed on every operation, including artifact reads. Clustered stores
and uncertain admissions stay on the manual path.
* fix(extensions): address review findings on unattended workflow recovery
- grant_validator validates the current persisted grant during locked
admission, where the entry is not published yet, and switches to the
live entry after publication; the previous entries-only lookup 403'd
every unattended restore of a perfectly usable grant
- delegated restore waits for the AgentRuns host installation signal
instead of racing the rest of Gateway startup and failing the scan
- cancellation defers grant revocation to the supervisor's post-cleanup
save, keeping the delegated handle authorized to cancel children the
recovered workflow already admitted
- the per-operation delegated guard rechecks the current account
disablement, which token_version does not reflect
- restore stub revalidates per operation like the real host, with
regression tests for all of the above; format the new fixture and
condense the extension guidance within its budget
* fix(tests): cancel the recovered workflow on its own service
The fixture's action handlers are bound to the original service, which
stop() already made unavailable; the recovered record lives on the
replacement instance, so the cancel must go through replacement.cancel.
* fix(extensions): let a cancelled replayed child keep its cancelled outcome
The replay fence forced every errored re-dispatched step to
needs_reconciliation, including the deterministic case where the error
is the workflow stopping after its child was already cancelled and
observed. Fence only genuinely uncertain outcomes; a clean cancel of a
recovered workflow now ends cancelled like a fresh one.
change-password checks that the new email is free, hashes the new
password, then writes. A /register of the same address that commits in
between makes update_user's commit hit the users.email unique index,
and the IntegrityError reached the client as an unhandled 500.
update_user now reports an email index violation as ValueError, like
create_user does for the same race, and the route maps it to the
existing 400 email_already_exists. Other integrity errors propagate.
* fix(gateway): map checkpoint-mode errors in run-prepare routes
The regenerate and edit-regenerate prepare routes read the same thread
checkpoints as the threads router but built the accessor and read the
latest checkpoint under a generic handler, so a delta/full mismatch
surfaced as a 500 instead of the 409 that names the mode flip. Move the
mapper to a shared module, reuse it for the accessor, latest-checkpoint
and history-scan failures, and update the troubleshooting page now that
the routes behave like the other state surfaces.
* fix(gateway): preserve route-level HTTP errors in run-prepare
The checkpoint-read fallback now sits around the accessor build, so a
config-loading HTTPException from the accessor was being rewritten as a
generic 500. Re-raise HTTPException before the generic handler and cover
the 503 passthrough with a test.
* docs: confirm the mode mismatch before advising a restart
A null token usage or a /runs/wait error body can also come from a
configuration or token-counting failure, so the troubleshooting entry
should point at the 409 from the thread/state endpoints first.
* fix(gateway): remove duplicate checkpoint imports after merge
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(threads): reject a state update as_node LangGraph cannot compile
POST /threads/{id}/state passes the client's as_node to the state
mutation graph as its only node. LangGraph refuses __start__, __end__,
its other reserved names and names containing ':' or '|', and that
compile runs before the route's error handling, so those values ended in
an unhandled 500.
Check the name in the request model with the same rules, so it is a 422
on the as_node field before any graph is built or state is written. A
parity test keeps the check aligned with LangGraph's own compile.
* docs(runtime): note the LangGraph floor the reserved-name check was verified on
* docs(checkpoints): re-anchor checkpoint_state rows and list validate_state_mutation_node
The new function shifts the cstate line ranges the reference page cites
for CheckpointStateAccessor.bind and build_state_mutation_graph.
Console pricing, the agent model check, the browser tool switch and
session config, the projects instructions cap, the channel connection
configs, scheduled task writes, the memory config routes and artifact
updates read get_app_config() inline on the event loop. Each now goes
through asyncio.to_thread (safe_app_config_async for artifacts).
Several of these helpers catch Exception, which also hides the gate's
BlockingError, so the anchors record where each read ran.
* fix(gateway): serialize the readiness database probe
/health/ready is public, and the module promises that connection-opening
probes run behind a per-kind gate. The ORM database probe never did: it
checked a connection out of the repositories' own engine pool on every
request, so concurrent probe traffic could hold the whole pool (15
connections by default) and leave request handlers waiting.
Run it behind _probe_gate("database") like the checkpointer, stream
bridge and provisioner probes. The gate is per kind, so one request's
probes still run concurrently.
* test(gateway): assert separate probe gates by rendezvous, not wall clock
Each probe now waits until the other has entered its gate, so a shared
gate fails through the readiness deadline instead of a timing margin.
* fix(auth): ignore a UserInfo response without a sub
fetch_userinfo() compared the UserInfo sub with the ID token's only when
it was truthy, so a response with no sub (or an empty one) was accepted
and its email, email_verified and name replaced the signed ID token's.
OIDC Core 5.3.2 requires the two subjects to match before UserInfo is
used. A missing, empty or non-string sub now raises OIDCUserInfoMismatch,
which the callback already treats as a skipped UserInfo request.
* fix(auth): log a non-string UserInfo sub apart from a missing one
The callback continues silently on a rejected UserInfo response, so the
log line is the only way to tell a provider that omits sub from one that
sends a malformed value. A non-string sub now reports its type.
* test(auth): name the UserInfo sub cases for what they cover
The case list and the callback test also cover a present but non-string
sub, so "no sub" undersold them.
* fix(gateway): reject artifact archives where a file collides with an implicit directory
When a run outputs both a file and a deeper file path whose parent directory
matches the first file's normalized name (e.g. 'Report' and
'report/slides/lesson.pptx', or Unicode NFC/NFD variations), the ZIP archive
builder allowed both entries because it only checked for exact full path
equality. On case-insensitive filesystems, extracting such archives fails because
the file entry blocks directory creation or vice versa.
This change collects all implicit directory prefixes from normalized casefolded
entries and rejects the archive with HTTP 409 (ArtifactArchiveError) if any file
entry name collides with an implicit directory, regardless of receipt order.
Fixes#7977
* fix(gateway): address review feedback on test portability and loop readability
* fix(gateway): match conversation reference URLs on the addressed host
A conversation URL reference was accepted only when its (scheme,
netloc) equalled Gateway's request.url. Behind TLS that URL is often
http: Docker Compose and Helm run uvicorn without FORWARDED_ALLOW_IPS,
so nginx's X-Forwarded-Proto from another container is ignored, and
nginx.local.conf resends its own $scheme. The browser's https chat link
then got a 422, and so did an explicit :443 or a differently cased host.
Resolve the run request's origin with _request_origin(), as the CSRF
check does (Forwarded / X-Forwarded-Host / Host), normalize the
reference's origin the same way, and compare host and port only. The
scheme cannot tell two deployments on the same host apart, and URLs
remain owner-bound local selectors that are never fetched.
* test(gateway): cover forwarded port, IPv6 and fragment for conversation URLs
Name X-Forwarded-Port in the API.md reference contract: it is appended
when the forwarded host has no port, so a stray value causes a 422.
* test(runtime): synchronize PostgreSQL lease expiration with row lock
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): reject an unusable promoted record at the input boundary
* fix(gateway): refuse an Overwrite-marker promotion record
* test(gateway): pin the server-owned metadata strip on promoted
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): read the run and authorization config off the event loop
get_app_config() stats and hashes config.yaml and extensions_config.json
on every call and reparses them after an edit. Run admission read it
inline on the loop two to four times per start (get_run_context,
build_run_config, the model allowlist, the approval switch, conversation
references), and the model list, model details, skill list, suggestions
and sandbox gates read the authorization section inline once each.
start_run now builds its run context in a worker and reuses that
context's app_config for the rest of the start, so one run start sees
one snapshot. Scheduled dispatch resolves its recursion limit in a
worker, run inputs take only the AgentRuns capability, and the
route-scoped authorization helpers run through asyncio.to_thread, as
/api/models/resolve already did.
* docs(changelog): reference #8137 for the off-loop run and authorization config reads
* fix(gateway): validate the tool approval mode against the run's config snapshot
The approval-mode check added on main read get_app_config() inline in
start_run, so every run start revalidated config.yaml on the event loop
again. It now uses the snapshot the run context was built with, like the
model allowlist and the approval switch.
* test(gateway): seed the skill catalog the authorization anchor lists
The skill listing anchor relied on the checkout's skills/public tree
containing academic-paper-review. It now writes one allowed and one
hidden skill under its own skills.path, so it passes without the repo
catalog and still fails if the policy is not applied.
* feat(search): find conversations and jump to matching messages
* test(extensions): align example registry with search contribution
* test(search): scope bookmark search to its extension
* fix(search): address review correctness and scaling gaps
* fix(gateway): reject unusable tool_artifacts entries at the input boundary
merge_tool_artifacts keys every registry entry by handle, so a caller-supplied
entry with a missing or unhashable handle raised while the state update
merged; the thread-state route persists reducer values with Overwrite, so the
bad shape was stored and every later artifact capture failed on it. Validate
the list-of-objects shape with a non-empty string handle on both boundaries,
and skip such an entry in the reducer so a poisoned checkpoint heals.
* fix(gateway): keep the per-entry metadata strip for tool_artifacts
The dedicated registry branch replaced the generic list branch and silently
dropped its _strip_external_metadata_from_message_like pass. Registry entries
are rich dicts, so a caller-crafted entry could persist server-owned message
metadata into the checkpointed registry. Apply the strip to each validated
entry; it is a no-op for legitimate registry entries.
* fix(github): build the App JWT off the loop and reuse the parsed key
mint_installation_token() runs on the channel manager's event loop. On
every cold or expiring installation token it built the App JWT inline:
it read GITHUB_APP_PRIVATE_KEY_PATH and passed the PEM to jwt.encode,
which parses and validates the RSA key each time. For RSA-2048 the parse
is ~45 ms of the ~50 ms loop stall, and it holds the GIL, so a worker
thread alone does not help.
Reuse the parsed key while the PEM text is unchanged (still read on
every mint, so rotation needs no restart; parsed through PyJWT's own
RS256 prepare_key, so formats and errors are unchanged; failures are not
cached), and build the JWT in asyncio.to_thread so the file read and the
signature leave the loop. Later mints drop from ~48 ms to ~1.4 ms of
loop stall.
* test(github): pin the single parse under concurrent mints
Async mints call mint_app_jwt in worker threads, and the lock around the
parsed-key cache is what keeps a cold race from parsing the key twice.
Two threads now start mint_app_jwt behind a barrier while the first
parse is held open; the key must be parsed once.
The test-only reset also takes the lock, so a mint still parsing in a
worker thread finishes before the reset instead of storing its key after
it.
* feat(goal): add /goal from-chat for IM channels
First slice of #7050: add an explicit IM command that composes the active goal
from the recent conversation instead of taking the text literally.
Natural-language intent detection remains out of scope.
Gateway:
- POST /api/threads/{thread_id}/goal/from-chat synthesizes the objective
from the thread summary and recent agreement turns (visible user/assistant
text, clarification questions, and card answers), preserving agreed
constraints and the stopping condition.
- Reject a busy thread before the model call; the save reuses the existing
checkpoint-write reservation so a run that starts mid-synthesis still wins.
- Refuse to synthesize while a human-input request is unanswered, and return
structured no_context/unclear/synthesis_failed outcomes without echoing
model-generated reasons.
IM:
- /goal from-chat [hint] shows the saved objective, then starts exactly one
run through the ordinary chat path; a longer read timeout covers synthesis.
- Map busy/pending/unclear/no-context/timeout outcomes to localized replies;
a saved goal whose run cannot start is reported honestly.
- The TUI explains that from-chat is not supported yet; the web input box
still treats it literally (documented follow-up).
Docs: README, CHANGELOG, IM channel guide, TUI guide, AGENTS, and the
acceptance plan/execution record. Tests: new route, synthesis and manager
suites, plus parser and TUI coverage.
* fix(goal): keep the from-chat confirmation visible on every channel
The saved-objective reply was published as a non-final progress update.
QQ and DingTalk card mode drop non-final outbounds before a stream exists,
and Feishu/Telegram fold the update into the run's stream, so the run
output overwrote the objective. Publish it as the turn's final message.
Redact PII on whole lines and the summary before the context budget can
split an identifier; the composed prompt is still redacted once more as
defense in depth, matching the goal evaluator's direct call.
Also correct the gateway/IM docs (from-chat never creates a checkpoint,
the busy pre-check is worker-local, and synthesis uses the deployment
default model), fix the route test clock patch, and add regression tests
for the final-outbound and truncation-boundary cases.
* fix(goal): align from-chat with the goal-route admission gates
Reject a from-chat save while a tool approval is pending, matching
PUT/DELETE /goal, and suppress subagent-batch continuations queued under
the previous goal after the save. Both invariants landed upstream while
this branch was in review; the new route now shares them.
Add route tests for both behaviors.
* docs(changelog): remove the from-chat entry to avoid conflicts
The maintainer asked to drop the changelog entry because the section conflicts with upstream changes. The feature remains documented in the goals guide and README.
* fix(goal): close three review gaps in /goal from-chat
- Enforce model-use authorization for the deployment default model before
the synthesis call, matching the suggestions route, so a denied role
never sends conversation context to the model.
- Save with expected_checkpoint_id pinned to the head the synthesis read;
a write that landed during the model call is a 409 goal_changed instead
of silently clobbering the newer goal.
- Re-check the conversation binding before starting the run: a /new during
synthesis keeps the goal in the original thread and reports that the
task did not start instead of running it in the switched conversation.
* fix(goal): preserve from-chat handoff and run status
Pin the saved thread through run preparation and reject a changed binding.
Read configuration and model authorization off the event loop, and report
rejected starts separately from requests whose response was lost.
Cover the handoff races and streaming/wait admission outcomes through the
real channel manager and SDK transport, plus file-backed config under the
strict blocking-I/O detector.
Co-authored-by: Loveca <44801706+Loveca@users.noreply.github.com>
* fix(goal): preserve reasoning tags in synthesis JSON
Use the existing JSON-aware reasoning stripper so literal tag text in
objective and reason values survives parsing. Cover complete and unclosed
tags while still removing model reasoning before the JSON payload.
Co-authored-by: Loveca <44801706+Loveca@users.noreply.github.com>
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Totoro <msy626836554@gmail.com>
* fix(gateway): keep LangGraph checkpoint keys out of client thread metadata
POST /api/threads spread client metadata over the root checkpoint's
step/source/writes/parents. A non-mapping parents broke every history
read and a non-integer step broke every later turn, leaving deletion as
the only way out. Server values now come after the client's.
* docs(changelog): reference #8019 for thread-create checkpoint metadata
* fix(gateway): keep server-owned checkpoint keys out of client thread metadata
The root checkpoint still copied client keys the server reads back as
trusted. A non-mapping counters_since_delta_snapshot failed every state
update and turn in both checkpoint modes, and deerflow_agent_name was
copied into the next state rewrite as the server-authored agent binding.
Drop a named set of checkpoint-owned keys before the spread so the fix no
longer relies on dict ordering alone.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): name the agent problem when run admission rejects agent_name
Since #5238, run admission loads the named custom agent's config for
every run, with or without a knowledge scope, and folded every failure
into "knowledge_scope assistant configuration could not be resolved".
Messaging a chat whose custom agent was deleted, or passing a malformed
agent_name, pointed callers at the knowledge base instead of the agent.
The 422 status is unchanged; the detail now says which problem it is:
- Agent '<name>' not found
- Invalid agent_name '<value>': must contain only letters, digits, and
hyphens after normalization. (shared with custom_agent_name(), which
gains a field= argument; existing callers keep the default)
- Invalid agent_name: expected a string, got <type>.
- Agent '<name>' configuration could not be loaded
* docs(changelog): reference #8087 for the run admission agent_name detail
* fix(gateway): bound the agent name echoed in run admission errors
Address review on #8087: admission now echoes the caller's raw
agent_name, which start_run merges from config/context without a size
check, so an oversized value was reflected verbatim into the 422 detail
and any log line that records it. Cap the echoed name at 200 characters
in the invalid-name message (shared with assistant_id validation) and in
the not-found / unreadable-config messages.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* docs: turn README into an index and move reference detail to docs/guide
The README had grown to 393 KB because every change appended its notes there. The text moves verbatim into 18 topic guides under docs/guide/; only headings are re-leveled and relative links re-pointed. The README keeps the introduction, a quick start and a guide index. Contributor test notes move to CONTRIBUTING.md.
* docs: send user docs to topic guides and budget the root README
The rule 'update README.md for user-facing changes' is what grew the README. Guidance now points to docs/guide/, and check_agent_guidance.py fails when the root README exceeds 16 KiB (AG003).
* chore: remove unreferenced screenshots and finished plan notes
pr-build/ holds two PR screenshots (3 MB) from #1986. plans/ holds the plan for the merged #4049. docs/SKILL_NAME_CONFLICT_FIX.md is a February change note. Nothing references them.
* docs: route always-loaded AGENTS.md files to scoped guides
* docs(agents): cut scoped guides to scope, verify, traps, decisions
Each AGENTS.md keeps four kinds of content: the scope of the directory, the
verification commands, traps a local test run does not show, and decisions the
code cannot state. Values, field lists, function inventories, test names, issue
numbers and history are deleted, because the code, the tests and git state them.
Search and fetch providers share one community guide. Four guides with no
remaining content are removed. The size check now allows 6 KiB per file (hard
limit 8 KiB) and 24 KiB per chain (hard limit 32 KiB).
* style: format frontend guide table and guidance test imports
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): reject unusable skill_context entries at the input boundary
merge_skill_context indexes entry["path"] and calls entry.get(), so a
caller-supplied non-object entry or one without a usable path raised during
the state merge and failed the run with a 500. Validate the list shape at the
run-input and thread-state boundaries, mirroring the todos and delegations
channels.
* Normalize entry by ensuring 'name' is a string
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(feedback): return 409 instead of 500 when feedback is POSTed twice
POST /api/threads/{id}/runs/{rid}/feedback did a plain INSERT. A second
POST by the same user for the same run (a retry, or feedback on another
message of that run) hit uq_feedback_thread_run_user and the uncaught
IntegrityError surfaced as HTTP 500.
FeedbackRepository.create now rolls back and raises
FeedbackAlreadyExistsError on that collision, and the route maps it to
409 pointing callers at PUT, which already upserts.
* fix(feedback): only map the run/user unique key to 409; cover concurrent POSTs
Address review on #8064:
- create() now re-reads the (thread, run, user) row after an
IntegrityError and raises FeedbackAlreadyExistsError only when it
exists; any other violation (e.g. a feedback_id collision) propagates.
- Add a commit-barrier test where two first POSTs race: one 200, one
409, and the winner's row is kept.
- Add CHANGELOG.md / CHANGELOG_zh.md Fixed entries.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): return 404 on cross-owner thread_id instead of 500
When a caller-supplied thread_id already belongs to another user, the SQL-backed store rejects the duplicate primary key with IntegrityError. The route's recovery path resolved the surviving row through the requesting owner's scope, so a foreign row could not be read back and the route surfaced a 500 instead of the documented 404 ownership-conflict response ("Thread not found").
Read the row back bypassing the owner scope before collapsing to a 500 so the cross-owner case mirrors ThreadOwnershipConflictError.
Fix#8052
Signed-off-by: inchang-ing <197932532+inchang-ing@users.noreply.github.com>
* test(gateway): pin thread-ownership 404 and real-failure 500
Address review feedback on the cross-owner 404 recovery:
- Update test_insert_race_concurrent_trusted_claims_have_one_winner: the
losing request now resolves the surviving unscoped row as a 404 ownership
conflict instead of a 500, while the single-winner assertions and the
non-disclosing detail are preserved.
- Add test_create_thread_keeps_real_integrity_failure_as_500 to pin the
fall-through branch where a duplicate-key IntegrityError coincides with no
surviving row, which must still surface as a 500 rather than collapse into
the ownership-conflict 404.
- Document the user-facing error-surface change in the Unreleased Fixed
section.
Signed-off-by: inchang-ing <197932532+inchang-ing@users.noreply.github.com>
---------
Signed-off-by: inchang-ing <197932532+inchang-ing@users.noreply.github.com>
Co-authored-by: inchang-ing <197932532+inchang-ing@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(artifacts): bind ranged reads to validated generation
* fix(artifacts): drain pinned reads without Windows hard links
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(threads): evict in-memory runs when a thread is deleted
RunManager.get() and list_by_thread() answer from the process-local _runs
registry before consulting the RunStore. DELETE /api/threads/{id} only
removed the durable rows, so a worker that had admitted the thread's runs
kept serving them: after the delete returned 200 and the thread itself
answered 404, GET /api/threads/{id}/runs still listed the run and
GET /api/threads/{id}/runs/{run_id} still returned 200 with the original
prompt intact in kwargs. The residue disappeared only on restart, which
is what made it look like a cache rather than a leak.
Add RunManager.discard_thread_runs() to drop this worker's copies for the
thread and call it from the delete path, next to the durable run
deletion. Internal thread-operation rows are left alone: the reservation
protecting the deletion is one of them and its own release path owns it.
Historical runs, the thread row, run events and checkpoints are unchanged.
* fix(threads): keep a live finalize tail attached when evicting runs
discard_thread_runs() popped every record it found, but worker.py clears
finalizing before the tail that publishes the terminal frame, closes the
journal and releases the sandbox lease. Popping mid-tail turns the tail's
manager-mediated writes into warnings on a record that no longer exists.
Skip records whose task is still running, mirroring what cleanup() already
does through shield(), and pin the guard with a unit test that seeds both
a finished run and a delete reservation. Document that a DELETE served by
one Gateway bounds, rather than removes, a peer's copies: they survive
until that peer's deferred cleanup(run_id) fires.
* fix(threads): hide a deleted run from reads during its finalize tail
A DELETE that arrives after worker.py clears `finalizing` but before the
worker task finishes cannot drop the record: the tail still publishes the
terminal frame, closes the journal and releases the sandbox lease through it,
so popping turns those writes into `set_status called for unknown run`.
Leaving the record readable instead keeps the run and its prompt reachable
through `GET /threads/{id}/runs` until `cleanup()`'s 300s delay fires, which
is exactly the window #8034 reports.
Records that a deletion reaches mid-tail are now flagged instead of skipped.
`get()`, `list_by_thread()`, `get_many_by_thread()`, the regenerate-source and
edit-replay reads skip a flagged record (and never rebuild it from a store row,
since the durable rows were deleted first), and the tail's own completion
evicts it — no second delete request and no timer.
* fix(threads): keep the tail eviction task referenced until it settles
The done-callback eviction is a separate task because it cannot await the
registry lock inline. The loop holds only weak references to tasks, so an
unreferenced eviction could be collected before it drops the record and the
deleted run would come back within `cleanup()`'s 300s window. Track it in a
manager-owned set and discard on completion.
* fix(threads): keep hidden runs hidden on every read path
The deletion flag set by discard_thread_runs is authoritative for "this run
is deleted", but two reads still treated it as a hint.
get() narrowed its store-less guard to a missing record, so a flagged
record on a store-less RunManager fell through to self._store.get() on
None; the AttributeError was swallowed by the broad except below while
every read in the flag window logged a hydration warning with a full
traceback. The record now settles the read before any store call, which
also keeps the read side-effect free as the comment intends. aget()
shares the path.
get_many_by_thread() rebuilt flagged runs from store rows. list_by_thread
already refuses that, because the durable row is deleted before the flag
is set and anything left is a stale or partial read; the batch read now
applies the same guard so the invariant is uniform.
---------
Co-authored-by: 2SinceJuly <202121119+2SinceJuly@users.noreply.github.com>
* fix(threads): keep the owner row when deleting checkpoints fails
DELETE /api/threads/{id} treated the checkpoint delete as best-effort: a
failure was logged at DEBUG, the threads_meta row was removed anyway and
the request returned success. Read routes allow a thread without a meta
row as an untracked legacy thread, so the surviving checkpoints (title,
messages, goal) became readable by every signed-in user through
GET /threads/{id}, /state, /history and /goal, and the owner could not
retry because DELETE requires the row.
A failed checkpoint delete now logs the exception and returns a generic
500 before the meta row is touched, so the thread stays owned and the
owner can retry. The remaining cleanup steps stay best-effort; their
reads are owner-scoped.
* docs(changelog): reference #7992 for the thread delete checkpoint fix
* fix(threads): require checkpoint deletion before dropping ownership
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): reject Windows reserved device names in an agent memory scope
`_management_agent_name_or_501()` validated the `agent_name` query parameter
with `AGENT_NAME_PATTERN` (`^[A-Za-z0-9-]+$`) only, which admits `NUL`, `CON`,
`COM1` and the rest of the Windows reserved device names.
An agent scope is not only a label: DeerMem resolves it to
`{storage}/users/{uid}/agents/{name}/facts` and creates that directory on the
first write. On Windows the path never materialises, so `create_fact` raised
`FileNotFoundError`, which the endpoint's `OSError` branch turned into
`500 Failed to create memory fact`.
Run `windows_incompatible_segment()` at the same boundary -- the check agent
directories, thread IDs, uploads, sandbox writes and skill support files
already go through -- and answer `422`. The `for_creation` flag scopes the rule
to the two routes that create the scope, `POST /memory/facts` and
`POST /memory/import`; read and delete routes stay permissive so a scope that an
older release stored under a reserved name can still be exported and cleared.
Closes#7940
* fix(memory): address review feedback for Windows reserved agent names
Cite PR #7941 in CHANGELOG instead of issue #7940; pin the clear route's permissive behavior for a legacy reserved-name scope with a regression test; cross-reference the shared validate_agent_name() reserved-name check so the two copies cannot drift.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): keep Gateway calls off the environment proxy
* fix(channels): keep the aio_sandbox policy import lazy
Importing deerflow.community.aio_sandbox.backend at module scope executes
the package __init__, which eagerly imports AioSandbox, AioSandboxProvider
and both backends. That makes every ChannelManager import load the entire
sandbox provider stack even when a different sandbox is configured.
Mirror app/gateway/health.py::_make_provisioner_client: move the import into
a small _make_gateway_client helper and route both Gateway call sites through
it, so the URL policy stays lazy.
* fix(channels): keep the run client off the environment proxy
langgraph_sdk.get_client() builds httpx.AsyncHTTPTransport(retries=5) on an
httpx.AsyncClient that leaves trust_env at its default True and exposes no
transport/trust_env override, so ChannelManager._get_client() still sent
runs.wait submissions to a loopback Gateway through HTTP_PROXY. That is the hop
#7876 reports: the control-plane calls are fixed, the run submission is not.
Route the SDK client through _make_langgraph_client: loopback and private
Gateways get an explicit trust_env=False httpx.AsyncClient carrying the SDK's
own retries, timeout and User-Agent and handed to LangGraphClient, while
external Gateways keep the stock get_client() path.
Also move the URL policy out of the aio_sandbox package into
deerflow.utils.http_policy (control_plane_trust_env). Keeping the import lazy
did not help on its own: aio_sandbox/__init__ eagerly imports the provider
stack, so importing aio_sandbox.backend still loaded all of it. The neutral
module has no httpx or sandbox dependency, so manager.py and gateway/health.py
import it at module scope, and aio_sandbox.backend re-exports it as
sandbox_http_trust_env for the existing sandbox call sites.
* fix(channels): keep AGENTS.md under budget and align legacy CSRF test with new client
* style(sandbox): remove unused urlparse import
---------
Co-authored-by: jack12178 <jack12178@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor(agent): ground runtime decisions in structured evidence
Use producer status and structured errors for tool recovery, preserve memory context without lexical confirmation, and remove predetermined receipt and skill conclusions.
Require explicit MCP discovery and authenticated schedule trial actions. Preserve authorization, ownership, isolation, and bounded execution; update documentation and regression coverage.
* test(sandbox): expect read_file initialization error callback
* fix(agent): address evidence review follow-ups
* fix(agent): preserve retry budgets for OpenAI transport errors
* fix(agent): expose tool execution status in model requests
Project validated execution metadata and exact tool text into request-only JSON so status survives provider serialization. Preserve stored results and receipt hashes, cover SDK and graph boundaries, and migrate replay input hashes without changing recorded responses.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(subagents): reject Windows reserved device names in managed subagent creation
* Reject Windows reserved device names (CON, PRN, AUX, NUL, COM1-9, LPT1-9)
in normalize_managed_subagent_name when for_creation=True via
windows_incompatible_segment.
* Validate subagent names on creation in FileManagedSubagentStore.create
and /api/subagents POST gateway router before attempting disk writes.
* Keep read, update, and delete paths permissive for backward compatibility.
* Add unit tests for store validation and router endpoints.
* Update CHANGELOG.md under Fixed.
* fix(subagents): reject reserved names in SQL managed-subagent create
Apply the same for_creation check used by the file store so both
backends refuse Windows device names. Pin mixed-case Nul in tests and
import the shared normalizer at module level.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
GitHub owner and repository names are case-insensitive, but the agent
registry indexed and looked up bindings by the raw repo string. A
binding for acme/widget never matched webhooks for Acme/Widget, so the
event was dropped without an error. Lowercase the repo on both sides.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): keep batch cancellation running when the scheduler store fails
_run_completion_hook awaited scheduled_task_service.handle_run_completion
before batch_continuations.source_finished, so a scheduler store outage
aborted the hook and skipped batch cancellation entirely: a claimed
follow-up of a hard-stopped run stayed in the dispatching state instead
of being cancelled.
The scheduler notification is now best-effort (logged as non-fatal);
batch cancellation always runs.
Fixes#7969.
* fix(gateway): isolate batch cancellation failures and trace completion warnings
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(parsing): keep literal think tags inside JSON string values
Structured parsers ran strip_think_blocks over the whole response before
json.loads. A model answer that quotes the tag -- "Explain <think>example
</think>" -- lost the quoted text, and an unclosed literal tag inside a
string cut the JSON document mid-string, so suggestions returned None and
goal evaluation failed to parse.
Strip reasoning only outside JSON string literals: leading blocks are still
removed whole (a JSON example inside reasoning is never taken as the
answer), and blocks after the payload starts are removed only outside
quotes. Suggestions and the goal evaluator both use the shared helper.
* fix(parsing): keep think-tag stripping complete outside the JSON payload
The string-aware walker only covered the payload itself: a reasoning block
after prose and before the payload was copied verbatim, so brackets inside
it became the parsers' find/rfind targets, and an unclosed block in
trailing prose was kept where strip_think_blocks used to truncate.
Find the payload start while stepping over think blocks, strip the head with
strip_think_blocks, keep strip/truncate semantics after the top-level
payload closes, and truncate at an unclosed block outside a string. Adds a
CHANGELOG entry for the user-visible parsing change.