224 Commits

Author SHA1 Message Date
rayhpeng
cb49dd67dc refactor(feedback): move the secondary adapters to app/adapters
Group them by bounded context instead of by technology, one file per
port, and align the directory name with the AWS Prescriptive Guidance
layout (entrypoints / domain-with-ports / adapters).

  app/infra/persistence/feedback.py
    -> app/adapters/feedback/feedback_repository.py   owned persistence
    -> app/adapters/feedback/run_lookup.py            anti-corruption layer

`persistence/` promised a technology-first classification that its own
contents contradicted: RunStoreRunLookup lived there while its docstring
said "no new SQL". Splitting per port makes that distinction structural.

SqlFeedbackRepository and _tz_aware move unchanged -- verified by
comparing their AST against the original rather than by eye. run_lookup.py
additionally gains a RunStore annotation behind TYPE_CHECKING (the module
is imported lazily by the composition root, so this keeps the runtime
import cost at zero), a docstring stating that this context owns no table
and writes no SQL against it, and a TODO recording the condition under
which the body is replaced: when the run context publishes a contract of
its own, the RunLookup port itself does not move.

Each module docstring opens with a fixed marker so the two kinds of
secondary adapter stay greppable:

  grep -rl "anti-corruption layer" app/adapters/

Filenames deliberately carry no sql_ / acl_ prefix: a prefix encodes an
implementation property, so switching storage would force a rename even
though the port -- and therefore the import path -- has not changed. The
class name already carries it. A prefix earns its place once one port has
several production implementations, which is not yet the case here.

app/infra/ held nothing else and is removed.
2026-07-28 17:34:10 +08:00
rayhpeng
fd74553f91 Merge branch 'main' into rayhpeng/hexagonal-feedback-slice
Rebase the feedback migration onto main's new chain tip again: main added
0009_webhook_dedupe (also chained after 0008_thread_operation_kind), so
0009_feedback_tags becomes 0010_feedback_tags with
down_revision=0009_webhook_dedupe, keeping the chain linear.

Conflicts resolved:
- Five head-pin tests take main's version with the pin bumped to
  0010_feedback_tags.
- test_thread_messages_page.py keeps this branch's feedback_service
  wiring over main's feedback_repo wiring, but stubs both service
  methods so the thread-grouped path main added coverage for stays
  stubbed (latest_per_run_in_thread alongside latest_for_runs).
- test_thread_messages_feedback.py keeps both sides' imports; each is
  used (Feedback for the fixture, EditReplayVisibility for the run
  manager stub).

Verified: 79 tests across the conflicted files plus the migration and
bootstrap suites, and 54 feedback tests, all pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-28 07:13:25 +08:00
qin-chenghan
795af20a6b
feat(memory): built-in FTS5/BM25 retrieval adapter (#4360)
* feat(memory): integrate FTS5 retrieval adapter

* deps: add jieba as default dependency for Chinese tokenization

Without jieba, FTS5 unicode61 tokenizer treats entire Chinese sentences
as single tokens, making single-character or sub-phrase searches
impossible (e.g. '吃' or '油泼面' returns 0 hits against
'用户喜欢吃油泼面'). jieba segments Chinese text into meaningful tokens
before indexing.

* fix(memory): avoid treating hyphens as FTS5 operators

* feat(memory): make Chinese tokenization optional

* fix(memory): warm every requested retrieval scope

* fix(memory): close retrieval resources on shutdown

* fix(memory): close backend when shutdown flush fails

* fix(memory): recreate corrupt retrieval index

* fix(memory): tolerate partial retrieval rebuilds

* fix(memory): warm retrieval index in background

* fix(memory): preserve shutdown flush budget

* fix(memory): stop retrying partial lazy rebuilds

* fix(memory): close retrieval through storage

* refactor(memory): simplify retrieval scope limit

* docs(memory): clarify retrieval shutdown lifecycle

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-27 23:17:18 +08:00
Ryker_Feng
fcbf0609b0
feat(chat): edit and rerun latest user turn (#4377)
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-27 22:46:51 +08:00
Vanzeren
6f53fd5e99
feat(runtime): enforce artifact delivery from workspace snapshots (#4494) 2026-07-27 22:27:16 +08:00
hataa
6091ce7576
feat(authz): derive Gateway route permissions from AuthorizationProvider (#4439)
* feat(authz): derive Gateway route permissions from AuthorizationProvider (Phase 2A, #4063)

Phase 2A: replace legacy _ALL_PERMISSIONS with provider-derived route
permissions. When authorization.enabled, each threads:*/runs:* permission
is evaluated independently via provider.aauthorize(resource='route').
Disabled mode preserves legacy behavior. owner_check and require_admin_user
remain unchanged.

10 new tests: disabled/enabled/RBAC policy/fail-closed/fail-open/
authenticate integration/middleware integration.

* fix(authz): move AuthorizationConfig import to runtime for fallback in _get_route_authorization_config
2026-07-27 14:19:04 +08:00
Vanzeren
625c07b993
fix(runtime): resume original title when regenerating (#4480)
* fix(runtime): rusume original title when regenerating

* test(runtime): cover regenerated title sync
2026-07-27 11:32:10 +08:00
rayhpeng
c1ee40667d Merge branch 'main' into rayhpeng/hexagonal-feedback-slice
Rebase the feedback migration onto main's new chain tip: main added
0008_thread_operation_kind (also chained after 0007), so
0008_feedback_tags becomes 0009_feedback_tags with
down_revision=0008_thread_operation_kind, keeping the chain linear.
The five head-pin test conflicts resolve to 0009_feedback_tags on top
of main's versions (preserving the new operation_kind assertions).
Verified: 42 migration/bootstrap tests, 54 feedback tests, and the
full frontend suite (797) pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 10:15:43 +08:00
Huixin615
1cd5dea336
fix(streaming): signal replay history gaps (#4426)
* fix(streaming): signal replay history gaps

* fix(streaming): guard initial Redis replay window

* fix(frontend): align inactive gap recovery

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-27 07:13:06 +08:00
Vanzeren
1c7531242c
feat(runtime): record terminal artifact delivery receipts (slice 1 of #4272) (#4365)
* feat(runtime): record terminal artifact delivery receipts (#4272)

* fix(runtime): persist delivery receipts across recovery

* test(runtime): cover delivery receipt invariants

* fix(runtime): preserve terminal status on receipt outages
2026-07-26 21:45:47 +08:00
Ryker_Feng
68c0ffdac8
feat(frontend): pin recent chats (#4442)
* feat(frontend): pin recent chats

* fix(threads): address pin-chat review feedback

- Stop bumping updated_at on metadata-only PATCH (pin/unpin) via a new
  update_metadata(touch=False) path so unpinning no longer jumps a chat
  to the top of the updated_at-sorted recent list.
- Narrow patchThreadMetadata to a ThreadMetadataPatchResponse matching
  the Gateway's actual response (no values/context).
- Namespace the pinned metadata key as deerflow_pinned for consistency
  with deerflow_sidecar / deerflow_branch.
- Cover touch/touch=False behavior in repo + router tests; document the
  e2e mock's updated_at preservation now mirrors production.

* style(frontend): format thread utils test

* fix(threads): make pinned ordering server-side

* test(frontend): keep infinite-scroll fixture order stable

* test(frontend): stabilize lark reconnect e2e

* docs: clarify thread pin metadata contract
2026-07-26 20:47:58 +08:00
Aari
e646188ab4
fix(runtime): accept the SDK's default stream_resumable=false (#4468)
RunCreateRequest declared stream_resumable as None-only, but langgraph_sdk
defaults it to False (not None) and its payload filter only drops None, so
every SDK request carried "stream_resumable": false and was rejected with 422.
False means "no resumable stream", which is what DeerFlow already serves.

This broke every IM channel run (runs.stream and runs.create both send the
field; only runs.wait omits it) and any external langgraph_sdk client. The web
frontend was unaffected because its compatibility wrapper does not send it.

An explicit true still returns 422. The new regression test drives a real SDK
client to capture its default payload and posts that to the HTTP boundary, so a
future non-None SDK default cannot regress this silently.

Fixes #4466
2026-07-26 15:00:42 +08:00
Aari
d1aeea2c3e
fix(checkpoint): unwrap Overwrite first writes into empty channels (#4383)
* fix(gateway): stop persisting Overwrite wrappers into empty reducer channels on branch

Thread branching (and POST /state on a never-written channel) wraps copied
reducer values in Overwrite. Upstream BinaryOperatorAggregate.update seeds
an empty (MISSING) channel with values[0] verbatim without unwrapping, so
Union-typed channels (sandbox/goal/todos/promoted) stored the wrapper
literally and the next consumer crashed with TypeError: 'Overwrite' object
is not subscriptable (#4380). Patch the channel to unwrap the first write
(mirroring DeltaChannel semantics), and stop copying thread-scoped channels
(sandbox, thread_data) into branches: the parent's sandbox_id would bind
the branch to the parent's workspace and release lifecycle.

* refactor(checkpoint): drop private _get_overwrite import for a local Overwrite check

Importing langgraph's underscored _get_overwrite at module top level meant an
upstream refactor that drops it - plausibly the same release that fixes the
bug - would fail this module's import and crash startup before the probe can
stand the patch down. Replace it with a local helper on the public Overwrite
type, and fix two test docstring nits.

* refactor(checkpoint): write patch flags via their constants to avoid drift

Both saver patches read their "already patched" idempotence flag through a
module constant (_PATCH_FLAG / _BINOP_PATCH_FLAG) but wrote it as a hard-coded
attribute literal, so renaming the constant would silently break the guard and
double-apply the patch. Write via the same constant (setattr), dropping the
now-unneeded attr-defined ignores.

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-26 10:44:50 +08:00
Daoyuan Li
b5cc3a81c3
fix(auth): resolve email accounts case-insensitively (#4101)
* fix(auth): handle email addresses case-insensitively

Normalize new and changed addresses, use case-insensitive lookup, and keep legacy case-variant rows stable during unrelated updates.

* fix(auth): reject legacy case-variant registrations

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-26 09:52:16 +08:00
Ryker_Feng
7aa314b4c1
feat: add Lark CLI integration (#3971)
* feat: add lark cli integration

* fix: polish lark integration actions

* feat: support lark incremental permissions

* fix: detect lark authorization completion

* fix: harden lark integration install

* feat: expand lark auth scopes and reuse host auth in sandbox

Default lark auth to least-privilege (recommend=false, base sign-in only)
and expose the full set of lark-cli --domain business domains as native
--domain grants instead of a 4-domain read-only mapping. Resolve the
skill pack from the latest larksuite/cli GitHub release at install time
with content-hash integrity, and surface version/runtime drift in status.

Share the per-user lark-cli config/data profile between the Gateway
Settings auth flow and agent conversations by mounting the integration
dirs into the AIO sandbox and injecting the matching env for lark-cli
commands, with an allowlisted extra_mounts path in the provisioner/K8s
backend and traversal guards on integration paths.

* style: fix lint issues from ruff and prettier

Sort imports in the provisioner PVC test and re-wrap two long i18n
description strings to satisfy backend ruff and frontend prettier CI.

* fix(lark): address managed integration review feedback

* fix(frontend): stabilize integrations settings e2e

* test(sandbox): isolate remote backend legacy visibility check

* test: fix backend unit failures after merge

* Harden Lark integration review fixes

* Format Lark integration E2E test

* fix(lark): harden sandbox credential exposure and status disclosure

Address willem_bd's security review on PR #3971:

- Mount the per-user lark-cli config dir (long-lived appSecret) read-only
  into the AIO sandbox; only the refreshable-token data dir stays writable.
- Redact host filesystem paths (install_path, cli.path) from
  GET /lark/status and the config/auth complete responses for non-admin
  callers, fail-closed on any auth error.
- Document the npm postinstall trade-off (--ignore-scripts is not viable
  because @larksuite/cli fetches its platform binary in postinstall).
- Document the sandbox credential trust boundary in AGENTS.md and README,
  pointing at the sidecar-broker follow-up (#4338).

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-26 08:09:17 +08:00
Aari
68797c5759
fix(gateway): scope branch history seed run ids per inherited turn (#4459)
Branch creation seeds the new thread's run-event feed from its checkpoint
so inherited history survives the first run (#4380). Every seeded row
carried one shared run id, but run_id is a *turn* identity to the feed's
consumers, not a provenance tag: regenerating the inherited answer
resolves that row's run id as the superseded source, and
GET /messages/page then drops every row carrying it. One shared id for
the whole seed therefore deleted the complete inherited history on a
branch's first regenerate, leaving only the regenerated turn.

Group seeded rows into one synthetic run per inherited turn
(branch-seed-{thread_id}-{n}), a new turn opening at each persisted human
message — the same boundary a real run has, including the allowlisted
hidden ask_clarification reply, which resumes as its own run. Supersession
is then confined to the turn actually regenerated.

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-26 08:00:48 +08:00
Aari
37c343fe30
fix(summarization): summarize with the run model, fall back on summary-provider failure (#4361)
* fix(summarization): own the run model for compaction; bound failure

With summarization.model_name: null the summary model resolved to
config.models[0] while the executing model is selected per run; when they
differ and models[0]'s provider is broken (expired key, quota, outage)
compaction silently failed every triggered turn and context grew unbounded
until the main provider 400s the run (#3103's shape), even though the run's
own model was healthy.

Model ownership is now sourced from the builders, not re-derived at runtime:
- The lead, subagent, and manual /compact builders each pass the resolved run
  model into create_summarization_middleware(run_model_name=...). The middleware
  no longer reads runtime.context / get_config(), which do not carry a custom
  agent's or a subagent's resolved model, so a custom-agent lead run and a
  distinct-model subagent now summarize with their own model, not models[0] /
  the parent's. Runtime re-resolution and the per-name model cache are removed.
- model_name: null summarizes with the run's own model; an explicitly configured
  summary model generates and falls back to the run model on failure. The
  fallback is built lazily after the primary fails and its construction is
  guarded, so a broken fallback cannot skip a healthy primary or escape the
  automatic failure boundary.

Failure is bounded and side-effect-safe:
- An empty or whitespace-only response is treated as a generation failure, not a
  valid summary, so compaction never removes all history for an empty replacement.
- compact_state/acompact_state take raise_on_failure independent of force: the
  manual /compact path always surfaces a generation failure (even force=false)
  and routes it to the existing ContextCompactionFailed path (HTTP 500 ->
  frontend error toast) instead of an unconsumed response reason. The automatic
  path leaves compaction state unchanged.
- before_summarization hooks fire only after a replacement summary exists.

SummarizationConfig.model_name, config.example.yaml, and docs/summarization.md
document the final lead/subagent/manual ownership rules.

Part of RFC #4346 (section A). Evaluating fraction/triggers against the run
model's profile (profile ownership) is a separate follow-up.

* fix(summarization): manual /compact model ownership + fail-open construct/parse

Manual /compact carried only agent_name, so it derived the run model from the
custom-agent model or config.models[0] and missed the request-selected model the
run path uses (request -> custom-agent -> default). Carry model_name through
ThreadCompactRequest and the frontend compact call, resolve with the same
precedence, and move the custom-agent config read off the event loop (asyncio
.to_thread) with user_id so the strict blocking-IO gate is not bypassed by the
broad except.

Make one summary attempt own its full lifecycle so the fail-open boundary covers
construction and response parsing, not just invocation: build each candidate model
lazily and guarded (a raising constructor falls through to the healthy run model
instead of breaking agent construction), build the model_name:null primary from the
run model rather than config.models[0], and run response text extraction inside the
invocation try so a failing .text accessor falls back instead of escaping compaction.
Adds factory-level constructor-failure, response-extraction-failure (sync/async), and
route-path model-ownership tests.
2026-07-26 07:39:39 +08:00
MiaoRuidx
735f67a5b2
fix: guard pending run startup cancellation (#4450)
* fix: guard pending run startup cancellation

* fix(run): address startup review feedback

* fix(run): narrow start_run store contract

---------

Co-authored-by: MiaoRuidx <12540796+MiaoRuidx@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-25 23:50:21 +08:00
Vanzeren
3c8b82c594
fix(runtime): serialize checkpoint writes with active runs (#4437)
* fix(runtime): serialize checkpoint writes with active runs

* fix(runtime): address checkpoint reservation reviews

* fix(runtime): address reservation race reviews

* fix(runtime): refine reservation conflict semantics
2026-07-25 23:18:34 +08:00
rayhpeng
c9ee725c0b Merge branch 'main' into rayhpeng/hexagonal-feedback-slice
Renumber the feedback tags migration to 0008 on top of main's
0007_scheduled_run_active_index so the alembic chain keeps a single
head; bootstrap head pins follow.
2026-07-25 18:28:34 +08:00
Nan Gao
58befaf248
fix(thread-history): keep completed subtask cards stable after reload (#4432)
* fix(thread-history): hide subagent AI responses

* refactor(thread-runs): remove unused _is_middleware_message_row helper

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-25 09:29:11 +08:00
Daoyuan Li
d2b5f884e3
fix(channels): buffer GitHub follow-ups during busy runs (#4133)
Queue comments received while a run is active, then submit one deduplicated follow-up after it finishes. Failed drains are requeued and watcher tasks stop cleanly with the channel manager.
2026-07-24 22:41:07 +08:00
ly-wang19
25d9ac0a43
fix(skills): offload blocking filesystem IO in get_custom_skill_history (#3563)
* fix(skills): offload blocking filesystem IO in get_custom_skill_history

The GET /api/skills/custom/{name}/history handler ran its storage probes and the
per-skill .history read directly on the asyncio event loop:
get_or_new_skill_storage(), custom_skill_exists(), get_skill_history_file().exists()
and read_history() are all blocking filesystem IO. make detect-blocking-io flagged
the existence probe (routers/skills.py:224) as DIRECT_ASYNC.

Move the whole read into a nested sync function run via asyncio.to_thread; a None
return signals 404 (distinct from an empty history list). Behavior is unchanged.

Per the blocking-io-guard SOP:
- Candidate: get_custom_skill_history (FILE_METADATA, DIRECT_ASYNC) -> FIX+ANCHOR.
- Re-scan: the finding no longer appears for this handler.
- Anchor: tests/blocking_io/test_skills_router.py drives the real handler against a
  real on-disk skill + history; teeth verified red (pre-fix) -> green (post-fix)
  under make test-blocking-io.

Scoped to this self-contained read handler. rollback_custom_skill and update_skill
also touch blocking IO but interleave it with awaits (security scan / cache refresh)
and do a read-modify-write, so offloading them needs the asyncio.Lock serialization
treatment (cf. #3552) and is left as a separate fix unit.

* test: trim dead skills history setup

* fix(skills): use the user-scoped storage accessor in the offloaded history read

The merge with main left the offloaded reader calling get_or_new_skill_storage,
which is not defined in this module (ruff F821), so lint failed and the handler
would raise NameError at runtime. Use _get_user_skill_storage(config) — the same
accessor every other handler in this router uses.

Also update the regression test for the current route signature: the handler is
now admin-only and takes a Request, so the test supplies request.state.user
(mirroring tests/blocking_io/test_channel_runtime_config_store.py) and seeds the
history through the same user-scoped accessor.

---------

Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-24 22:33:16 +08:00
H Haidong
c7538cfb35
fix(runs): terminate orphaned streams after lease recovery (#4420)
* fix(runs): terminate orphaned streams after lease recovery

* fix(runs): include recovered ids in callback warnings

* fix(runs): harden orphan recovery lifecycle
2026-07-24 19:34:20 +08:00
ShitK
a4ede80deb
fix(runtime): reject unsupported run options and stream modes (#4430)
* fix(runtime): reject unsupported run options

* fix(runtime): align SDK run compatibility

* fix(frontend): avoid unsupported events stream mode

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-24 19:24:24 +08:00
rayhpeng
02a0a57609 Merge branch 'main' into rayhpeng/hexagonal-feedback-slice
Resolve the one import conflict in thread_runs.py: keep main's new
checkpoint_lineage imports (#4358 regenerate-in-branched-threads fix)
alongside this branch's get_feedback_service (feedback domain-service
refactor). All other files auto-merged. Verified: backend feedback/
regenerate/branch/lineage tests (112 passed) and the full frontend suite.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 09:55:36 +08:00
Huixin615
fbc1463809
fix(gateway): preserve regenerate state in branched threads (#4358)
* fix(gateway): preserve regenerate state in branched threads

* test(gateway): isolate branch regenerate regression config

* fix(gateway): preserve branching for legacy histories

* fix(gateway): harden branch regenerate lineage

* docs(gateway): clarify branch checkpoint behavior

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-24 08:57:48 +08:00
rayhpeng
70613f8996 refactor(gateway): route feedback through the domain service
The composition root wires FeedbackService with its SQL adapter; routers
become protocol translation only (domain errors map to 400/404/409).

BREAKING CHANGE: removes five feedback endpoints with no frontend
consumers (POST, GET list, GET stats, DELETE by id under /threads, and
GET /runs/{id}/feedback). PUT/DELETE now follow the mainstream chat UX:
idempotent upsert plus retract, echoed through the message-list payload.
2026-07-23 15:30:48 +08:00
Daoyuan Li
04659cc8dd
fix(gateway): stop implying 200 webhook deliveries are unrecoverable (#4307)
PR #4289 corrected the false claim that GitHub auto-retries 5xx webhook
deliveries, but its replacement wording overcorrected: it described a
mistaken 200 response as dropping the webhook "forever with no way to
recover it" / "permanently" - implying manual recovery is impossible,
not just unprompted. fancyboi999 flagged this in a CHANGES_REQUESTED
review on #4289 (submitted 23:21:23Z, referencing
github_webhooks.py:190-197,325-335 and
test_github_webhooks.py:548-559) that went unaddressed before the PR
was approved and merged roughly 40 minutes later.

Verified directly against GitHub's documentation before changing
anything: the manual "Redeliver" button and the REST/App redelivery
endpoints place no failed-status precondition on the delivery id - any
past delivery, success or failure, can be redelivered within GitHub's
~3-day window
(https://docs.github.com/en/webhooks/testing-and-troubleshooting-webhooks/redelivering-webhooks,
https://docs.github.com/en/rest/repos/webhooks#redeliver-a-delivery-for-a-repository-webhook).
The real problem with swallowing a transient failure into 200 is
discoverability, not recoverability: the delivery never shows up as
failed in Recent Deliveries, and GitHub's own recommended
scripted-recovery pattern filters on non-OK status by convention, not
because the platform blocks redelivering a success. A 200'd delivery
can still be redelivered by hand if an operator happens to look - they
just get no signal telling them to, unlike the 503 path, which stays
correctly flagged as failed and so is actually found.

- github_webhooks.py: reworded the route docstring and the inline
  fan-out comment to describe the 200-vs-503 difference as
  discoverability, not raw recoverability, and added the redelivery
  docs link alongside the existing failed-deliveries link.
- test_github_webhooks.py: reworded
  test_dispatch_failure_returns_503_not_200's docstring the same way.
  No assertions changed.

All 44 tests in test_github_webhooks.py pass, plus
test_github_dispatcher.py / test_github_channel.py /
test_github_registry.py / test_channels.py (322 total). ruff check and
ruff format --check are clean on both touched files.
2026-07-23 14:32:49 +08:00
Aari
70fb91654d
fix(gateway): seed branch run-events so inherited history survives forking (#4385)
* fix(gateway): seed branch run-events so inherited history survives (#4380)

The thread feed (GET /messages, /messages/page) reads the run-event store,
but branch creation only wrote checkpoint state - a fresh branch had no
message rows, so the parent history vanished from the UI as soon as the
branch's first run refreshed the feed. Seed the branch's run_events from
the same checkpoint snapshot the branch was created from, mirroring
RunJournal's message-event contract (event types, hidden-message rules,
original-user-text restoration). Best-effort: a seeding failure degrades
to the old behavior and is reported as history_seed_mode=failed.

* docs(gateway): correct branch-seed docstring on RunJournal divergences

The "consumers cannot tell a seeded row from a journaled one" claim was
overstated for AI rows: seeded rows omit run-scoped enrichment (usage /
latency_ms / llm_call_index) and stamp caller=lead_agent rather than the
message's original caller, neither recoverable from a checkpoint message.
Rewrite the docstring to state these divergences explicitly and note they
are display-invisible today (no consumer indexes those keys; per-message
caller drives no attribution). Also add a code comment marking the
hide_from_ui filter as intentionally stricter than the live paths.

* fix(gateway): seed dict-shaped checkpoint messages + persist hidden AI/tool rows

Two review-driven fixes to build_branch_history_seed_events:

1. Checkpoint messages can arrive as model_dump()-shaped dicts (the
   branch-matching helpers in threads.py already handle both BaseMessage
   and dict). The seed only handled BaseMessage, so a dict-backed
   checkpoint seeded nothing and the branch reported skipped_empty while
   history existed. Coerce dicts back to BaseMessage via messages_from_dict
   (faithful: tool_calls / tool_call_id / additional_kwargs survive);
   unparseable dicts are dropped best-effort.

2. RunJournal.on_llm_end and _persist_tool_result_message persist
   hide_from_ui AI/tool rows unconditionally (the frontend hides them
   client-side); the hide check only gates the reconciliation pass. The
   seed dropped them, so a hidden turn vanished from a forked feed and
   seeded rows diverged from journaled ones. Match RunJournal and write
   them, restoring true row-level parity.

Adds tests for dict deserialization, the unparseable-dict drop, and the
hidden AI/tool persistence contract.
2026-07-23 13:57:32 +08:00
Aari
0d4d0cb17d
feat(agents): database-backed storage for custom agent definitions (#4359)
* feat(agents): database-backed storage for custom agent definitions

Add an agent_storage.backend switch (default file, behaviour-unchanged) with a
db backend that stores each custom agent as a row in the shared SQL persistence
layer, so a multi-instance deployment sees the same agents on every node
(#4331, #4357). Introduces an AgentStore interface routing all read/write
surfaces, an agents table + migration 0006, startup validation, and a file->db
importer. Follows the thread_meta store / run_events backend-switch /
0003_scheduled_tasks migration patterns; no new dependency.

* fix(agents): make db storage path production-ready (review round 1)

Addresses review feedback on the db/sync agent-storage path:

- sql.py: mirror the async engine's per-connection SQLite PRAGMAs on the sync
  engine (busy_timeout=30000, synchronous=NORMAL, foreign_keys=ON, WAL) so both
  engines behave identically against the shared DB; guard the engine cache with
  a lock (double-checked) so concurrent first-touch cannot build duplicate
  engines or register the connect listener twice.
- routers/agents.py + routers/assistants_compat.py: offload the sync-store reads
  that ran on the event loop (list/get/check, update's pre-read + legacy guard +
  refresh, and assistants_compat's four list routes) via asyncio.to_thread — on
  db+postgres each was a network round trip stalling the loop. Writes were
  already offloaded.
- file.py: translate the create() mkdir(exist_ok=False) race FileExistsError
  into AgentExistsError (router 409, matching SqlAgentStore's IntegrityError
  path); correct the _write docstring — per-file atomic replace, two commits
  sequential not transactional.

Tests: sync-engine PRAGMA + engine-cache reuse assertions; file create-race ->
AgentExistsError; strict Blockbuster anchor over the read endpoints so a
regression back onto the loop fails CI.

* fix(agents): address round-2 review on the db store path

- update_agent tool: align the docstring/inline comment with FileAgentStore._write.
  Cross-field write atomicity is db-only; the file backend commits config then
  soul via two sequential os.replace (a crash between them can leave a fresh
  config.yaml beside a stale SOUL.md). The dropped partial-write *reporting* is
  an intentional tradeoff — the stage-then-replace safety is preserved
  (test_update_agent_soul_failure_does_not_replace_config still holds).
- SqlAgentStore.update(): true upsert. Catch IntegrityError on the
  insert-on-missing branch, re-fetch and apply, so two concurrent first-time
  writes (e.g. two setup_agent handshakes) converge instead of surfacing a raw
  UNIQUE(user_id, name) violation as a 500. Symmetric with create().
- get_agent_store(): document the graph-subprocess config-resolution invariant
  (the except->file fallback is a genuine no-config path, not a mask for a
  misconfigured graph process) and pin it with two tests driving the real
  get_app_config() file resolution: db resolves from an on-disk config.yaml,
  file fallback when config is unresolvable.

* test(agents): cover SqlAgentStore.update() write-race upsert recovery

Mandatory-TDD test for the round-2 fix in 0680340a: two concurrent first-time
update()s where the loser's insert hits UNIQUE(user_id, name). Deterministically
forces the IntegrityError recovery path by making the first _row probe miss the
committed winner, and asserts last-writer-wins instead of a surfaced 500.
2026-07-23 08:03:21 +08:00
lllyfff
01a89f2379
[feat] memory: pluggable MemoryManager interface for backend onboarding (#4326)
* refactor(memory): pluggable MemoryManager interface for backend onboarding

Optimize the MemoryManager interface layer so new backends (mem0/openviking)
onboard with less code and the contract stays stable as capabilities are
added. A minimal backend now implements only from_config + add + get_context
(verified by test_memory_manager_interface.py::_MinimalBackend onboarding via
the factory); the factory no longer knows a backend's private hooks.

- MemoryManager: ABC -> pydantic BaseModel; three-tier methods (tier-1
  add/get_context abstract; tier-2 management defaults; tier-3 optional hooks
  warm/reload/fact + on_pre_compress/on_turn_start). Dropped 3 self-serving
  hooks. 6 hasattr probe sites -> direct call + try/except NotImplementedError.
- from_config classmethod: factory thins to resolve + inject storage_path +
  collect host hooks + call from_config; DeerMem-specific hook consumption
  moved from factory to DeerMem.from_config.
- Invariants: @model_validator (mode='tool' requires search via supports_search
  ClassVar); DeerMemConfig storage_path-is-file check moved here from factory.
- Async: aadd/aget_context/asearch default to the sync path (speculative).
- Callbacks: MemoryCallbacks + LangfuseMemoryCallbacks; on_memory_llm_call
  subsumes tracing_callback (same signature/timing/mutation); deleted the
  tracing_callback field. DeerMem decoupled from langfuse (portability).
- noop keeps read-op empty overrides (avoids router 500s on the
  disable-memory-via-noop path); only delete/export inherit the base raise.

Behavior preserved: 661 passed / 13 skipped. Docs: backends/README.md rewritten
(three-tier + from_config + callbacks); samples README updated; removed stale
private doc paths.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): 501 on unsupported read/manage endpoints + accurate warm log

Review follow-up on the three-tier MemoryManager refactor.

- Read/manage endpoints (GET /memory, /memory/export, /memory/status,
  DELETE /memory, POST /memory/import) and the /memory/reload fallback now
  catch NotImplementedError -> 501, matching the fact-CRUD endpoints. The
  hasattr->try/except migration had skipped these: they were @abstractmethod
  before (every backend implemented them, so they never raised), so once they
  became tier-2 default-raise a minimal backend (only add + get_context) hit a
  raw 500 -- there is no global NotImplementedError handler. get_memory is
  shared via _get_memory_or_501 (covers /memory, export, status, reload
  fallback). noop is unchanged: its read-op empty overrides never raise.
- warm() base default returns None (tri-state: True=warmed, False=failed,
  None=nothing to warm) so the Gateway lifespan logs "skipping" for a
  non-DeerMem backend (e.g. noop) instead of the inaccurate "warmed
  successfully" it never earned. DeerMem.warm keeps True/False.
- Tests: 6 router 501 tests (read/manage + reload fallback) + 2 lifespan
  warm-log tests (None->skipping, False->warning); conformance/pluggable
  assert warm() is None.

705 passed / 13 skipped; lint clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(memory): review follow-ups - search-flag consistency, client reload, backend_config purity

Address review feedback on the three-tier MemoryManager refactor:

- [Medium] supports_search/search drift: the invariant now requires the
  supports_search ClassVar flag to MATCH whether search() is actually
  overridden (type(self).search is not MemoryManager.search), so the flag
  can't drift from the impl. Catches both directions at instantiation: a
  backend that overrides search() but forgets supports_search=True (was a
  misleading tool-mode rejection), and one that sets the flag without
  overriding (was a runtime NotImplementedError on the first memory_search).
  noop sets supports_search=True to match its search() override. Conformance
  adds drift + consistent-backend tests.
- [Low] client.reload_memory fallback: wrap the get_memory fallback so a
  minimal backend (only add + get_context) surfaces a clean NotImplementedError
  ("implements neither reload_memory nor get_memory") instead of an uncaught
  propagation -- mirrors the router's 501. Test added.
- [Low] backend_config purity: DeerMem.from_config restores backend_config to
  the pure data the host passed after model_post_init parses the injected hooks
  into DeerMemConfig (self._config, PrivateAttr); the field stays serializable
  (no callables/LLM) and matches the README ("host hooks NOT in backend_config").
  Test asserts purity + hooks wired.
- [Low] CHANGELOG: breaking-change note that mode='tool' + non-search backend
  now fails fast at startup (was silently empty) so operators recognize it on
  upgrade.
- [Nit] .gitignore: drop the env-specific .tmp-pytest/ entry (--basetemp is
  local-only, not make test/CI).

709 passed / 13 skipped; lint clean.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(changelog): correct memory tool-mode fail-fast note

The CHANGELOG entry said mode='tool' + a non-search backend "(e.g. noop)"
fails fast at startup, but noop overrides search() (returns []) and sets
supports_search=True (required by the consistency invariant), so noop IS
search-capable and noop+tool does NOT fail fast. The fail-fast only affects a
custom backend that onboards without overriding search(). Reworded to drop the
misleading noop example and state both shipping backends implement search().

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-22 14:40:57 +08:00
Ryker_Feng
20debf9cc7
feat(agents): per-agent model and generation settings (#4347)
* feat(agents): per-agent model and generation settings

Let each custom agent choose its own model and sampling settings
(temperature, max_tokens) plus thinking / reasoning_effort defaults,
so agents sharing a model profile are no longer stuck with one shared
temperature and output length (#4336).

AgentConfig gains optional model_settings / thinking_enabled /
reasoning_effort (None = inherit). create_chat_model applies per-caller
model_overrides on top of the profile before the thinking/Codex
transforms; the lead agent resolves each knob with precedence
request > agent config > profile/default. The /api/agents create/update
routes persist the fields and reject an unknown model. The default lead
agent path is unchanged (no agent config -> overrides None). The agent
chat composer also stops force-overriding an agent's configured default
model with models[0].

* fix(agents): tri-state thinking control and default-model capability gating

The model-settings dialog seeded the thinking switch to false, so opening
it to tweak temperature and saving silently disabled thinking (the runtime
default is on) with no way back to inherit. It also hid the thinking /
reasoning controls whenever the agent inherited the global default model,
since `__default__` never resolved through `models.find`.

Give thinking an explicit Inherit / On / Off tri-state so an untouched save
is a no-op, and resolve `__default__` to the effective default (models[0])
for the capability check. Logic lives in the tested helpers module.
2026-07-22 13:44:55 +08:00
March7
ce4a6d4c3d
fix(backend): remove transient image context after model calls (#4267)
* fix(backend): discard transient image context

* fix(backend): protect client image context ids

* docs(backend): clarify image checkpoint lifecycle
2026-07-22 08:41:50 +08:00
Vanzeren
42baed8c8c
feat(checkpoint): dual-mode checkpoint storage with LangGraph DeltaChannel (#4292)
* feat(checkpoint): dual-mode checkpoint storage with LangGraph DeltaChannel

Add a restart-required database.checkpoint_channel_mode ("full" default,
"delta") that stores the messages channel via LangGraph 1.2 DeltaChannel,
cutting checkpoint storage from O(n^2) to O(n) for append-only history.
Existing full checkpoints seed delta state transparently; no data migration.

- config: mode schema + freeze-on-first-use with
  CheckpointModeReconfigurationError; mode marker persisted in checkpoint
  metadata; unsafe delta->full downgrade rejected fail-closed with
  CheckpointModeMismatchError (run-level error, failed state read)
- state: delta message state schema; CheckpointStateAccessor centralizes
  materialized reads for all consumers (threads API, branches,
  regeneration, compaction, state updates, memory, goal workers)
- runtime: raw writers (run durations, interrupted title, thread goal)
  parent their checkpoints to the checkpoint they derive from, preserving
  delta ancestry; rollback forks the pre-run lineage through a state
  mutation graph with Overwrite restores; InMemorySaver delta-history
  override delegates to the base walk (fixes dropped first write after
  migration, also present upstream)
- tests: conformance suite over {memory, sqlite, postgres} covering
  migration replay, stable message IDs, storage shape and writer
  preservation; conftest fixture isolates the frozen mode between tests;
  stale config fakes refreshed
- ci: backend unit tests gain a postgres service

* fix(checkpoint): close materialization gaps in goal flow, guard public factory

- Route goal-continuation message reads through CheckpointStateAccessor:
  raw channel_values reads see the delta sentinel in delta mode, which
  disabled goal continuation (stand_down=no_durable_end_of_turn) after
  durable assistant turns. Raw tuples remain for tuple-only metadata
  (checkpoint id, pending_writes).
- Reject checkpoint_channel_mode='delta' + checkpointer in
  create_deerflow_agent at construction: factory-built persisted graphs
  bypass mode-marker injection and the fail-closed gate, reproducing
  silent mixed-mode state loss. Delta without persistence stays allowed.
- Import the postgres saver lazily (pytest.importorskip in the fixture)
  so the documented default install collects the suite; add a CI job
  running pytest --collect-only on uv sync --group dev without extras.
- Fix test_checkpointer fallback test to patch get_app_config at its
  use site (provider module), making it deterministic when a local
  config.yaml selects a persistent backend.

* fix(gateway): preserve extension-owned channels in state mutations, bump config version

- build_state_mutation_graph / build_checkpoint_state_mutation_accessor
  accept an explicit state_schema; branch and POST /state now compile the
  mutation graph from the thread's effective schema
  (graph_state_schema on the assistant graph). The base-ThreadState
  fallback silently discarded channels contributed by custom
  AgentMiddleware.state_schema on branch (data loss) and returned a
  false-success 200 on POST /state.
- POST /state validates values keys against the mutation graph's
  channels and rejects unknown fields with 422 instead of ignoring
  them; reducer detection covers extension channels
  (BinaryOperatorAggregate or DeltaChannel) so Overwrite replace
  semantics work for middleware reducers in both modes.
- Endpoint regression: custom AgentMiddleware.state_schema value
  survives branch, updates through POST /state, and an unknown field
  receives 422.
- config_version 26 -> 27 for the new database.checkpoint_channel_mode
  (example, Helm chart values + README, support-bundle fixture), so
  existing installs get the outdated-config warning and
  make config-upgrade merges the field; covered by a test driving the
  real example file and the real config-upgrade script.

* fix(gateway): resolve assistant schema via one boundary, copy branch reducer values with Overwrite

GET /threads/{id}/state now resolves the thread's assistant_id through a
single reusable boundary (thread metadata -> assistant_id -> effective
graph), so channels contributed by a custom AgentMiddleware.state_schema
are materialized instead of dropped by the default lead schema. POST
/state uses the same boundary instead of resolving the schema ad hoc.

Branch writes wrap every copied reducer channel in Overwrite (derived
from the effective mutation graph: BinaryOperatorAggregate + DeltaChannel),
not just messages, so already-aggregated values are never re-merged.

Regression tests use a real AgentMiddleware.state_schema with a
non-identity reducer in both full and delta modes: GET /state returns the
extension value, POST /state replaces it, branch preserves it
byte-for-byte; the unknown-field 422 is a separate assertion.

* refactor(checkpoint): collapse read-path round-trips and ship dual-mode parity tests

Address review round 4 on PR #4292:

- Push ahistory/history limit through Pregel into checkpointer.alist
  (SQL LIMIT) instead of materializing all rows and breaking in Python
- Fold the read-side mode-compat gate onto the returned snapshot's
  metadata; only writes keep the pre-write tuple fetch (fail-closed)
- Cache factory-built accessor graphs per (assistant_id, mode) with
  factory-identity revalidation; state reads no longer build a lead
  agent per request
- get_thread: one snapshot fetch + one raw pending_writes fetch on the
  resolved checkpoint (post-checkpoint __error__ writes never surface
  in snapshot.tasks; verified empirically)
- DeerFlowClient.get_thread: single checkpointer.list walk collects
  pending_writes per checkpoint instead of N get_tuple calls
- InMemorySaver delta-history patch: stand-down when the upstream
  override disappears, try/except guard, validated-version warning,
  guard tests
- make_lead_agent mode precedence: first freeze is owned by app_config
  (client-supplied configurable key ignored); once frozen, injected
  key/app_config must match or fail closed
- Rollback: lock in non-message channel restoration via fork
  inheritance with a dedicated reducer-channel test
- Add tests/test_threads_checkpoint_mode.py and
  tests/test_gateway_checkpoint_mode.py referenced by AGENTS.md and
  the PR validation section: lifecycle parity (memory + sqlite),
  per-step blob-count storage guard, gateway endpoint parity

Counted-saver tests pin checkpoint round-trips for aget/ahistory so
these regressions cannot silently return.

* fix(checkpoint): precise mode-mismatch HTTP mapping, gate E2E, and accessor resilience

- threads router: map CheckpointModeMismatchError to 409 (with cause and
  thread id) and CheckpointModeReconfigurationError to 503 across all state
  endpoints instead of swallowing both into a generic 500
- gate coverage: seed a real delta checkpoint into AsyncSqliteSaver and
  assert aget/aupdate/ahistory fail closed; assert 409 at the HTTP boundary
  through the real route stack
- rollback: compile the restore mutation graph with the thread's effective
  state schema per the build_state_mutation_graph contract
- inheritance contract locks: rollback and manual compaction preserve
  middleware-contributed channels via checkpoint fork cloning
- services: revalidate the accessor-graph cache against app_config identity
  so config.yaml hot-reloads never serve a stale compiled graph
- services: degrade full-mode state reads to raw checkpointer reads when the
  agent factory is unavailable (delta gate still applies; delta mode has no
  fallback)
- deps: override websockets==16.0 (langgraph-sdk 0.4.2's <16 pin silently
  downgraded 16.0 -> 15.0.1; pin is not grounded in any API incompatibility)
  and bump the langchain lower bound to what the lockfile actually resolves

* fix(checkpoint): include anchor checkpoint in degraded history walk + cover get_thread

- _RawCheckpointReadAccessor.ahistory: alist(before=...) is exclusive while
  pregel's get_state_history treats config.checkpoint_id as the inclusive
  start; fetch the anchor explicitly so both read paths paginate identically
- extend the degraded-path gateway test: GET /thread returns raw values, and
  POST /history with before starts at the anchor checkpoint

* fix(gateway): preserve degraded checkpoint timestamps

* fix(gateway): harden degraded checkpoint access

* fix(gateway): resolve assistants for checkpoint reads

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-22 08:33:29 +08:00
RongfuShuiping
4bf028d048
feat(memory): add incremental agent-scoped Markdown fact storage (#4279)
* feat(memory): add Markdown fact storage repository

* docs(memory): explain storage rewrite for beginners

* docs(memory): fix plan markdown formatting

* refactor(memory): separate global summaries from agent facts

* fix(memory): make Markdown fact updates incremental and safe

* Update STORAGE_REWRITE_CHANGES.md

* Delete docs/plans/STORAGE_REWRITE_PLAN.md

* Delete docs/plans/STORAGE_REWRITE_CHANGES.md

* fix(memory): address Markdown storage review feedback

* fix(memory): complete review follow-ups

* fix(memory): resolve storage review findings

* feat(memory): add proactive Markdown migration CLI

* fix(memory): harden Markdown storage concurrency

* fix(memory): harden markdown storage migration

* fix(memory): close migration review gaps

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: qin-chenghan <qinchenghan@Huawei.com>
2026-07-22 07:46:03 +08:00
Ryker_Feng
fa496c0c8d
feat(browser): add agentic browser control (#4187)
* feat(browser): add agentic browser control

* fix(frontend): format browser view changes

* fix(browser): keep browser optional and isolate sidecar layout

* fix(browser): address PR review security and IME findings

- Nginx: add a browser-stream WebSocket location before the generic
  /api/threads regex so Live upgrades instead of downgrading to HTTP
  (both nginx.conf and nginx.local.conf).
- Ownership: require an existing owned thread for the WS stream and REST
  navigate, and tear down the browser session on thread deletion so a
  later caller cannot reuse a retained page/cookies by guessing the id.
- SSRF: enforce the URL policy at the browser request boundary via a
  context-level route guard covering redirects, popups, iframes, and
  subresources (skipped for CDP-attached Chrome).
- IME: skip key forwarding while a composition is active so confirming a
  CJK candidate with Enter no longer submits the remote page form.

Adds regression tests for the request guard, session teardown on delete,
and the composing-Enter key decision.

* fix(frontend): smooth streaming in long tool threads

* Revert "fix(frontend): smooth streaming in long tool threads"

This reverts commit f0462516eabe77f138d4027ea1c714fb226683cf.

* fix(browser): address review security and lifecycle findings

- Reject cross-origin WebSocket upgrades on the live browser stream
  (Origin allow-list reuse of CORS/same-origin helpers) to close a
  WS-CSRF hole, and fail closed when the ownership store is absent.
- Warn when a CDP-attached session runs with the SSRF request guard
  off, and drop the unreachable CDP screencast teardown dead code.
- Read browser session launch config from a single canonical source
  (browser_navigate) so it is deterministic regardless of call order.
- Bound per-thread Chromium accumulation with idle-timeout eviction
  and an LRU max-sessions cap.
- Reset the Live reconnect counter on a successful open so the stream
  can't permanently stall after the cumulative attempt cap.

* fix(frontend): reduce long tool thread render stalls

Reuse stable historical message groups during streaming, defer heavy Markdown and browser previews, and lazy-decode message images.

* fix(browser): keep live control responsive during continuous input

Why: Manual browser control felt laggy — a physical click ran the remote
Playwright click three times and each non-move input synchronously awaited a
JPEG screenshot, so events queued behind capture (queue wait up to ~237ms).
The first async attempt used a trailing-edge debounce, which froze the visible
page until a wheel/keyboard gesture stopped ("scroll finishes, then it jumps").

What:
- Frontend forwards one `click` per physical click instead of also emitting
  `down`/`up`, so the remote page is not clicked twice per gesture.
- Backend detaches live-frame capture from input dispatch: non-move actions
  start a rate-limited background refresh loop (leading frame + bounded cadence)
  that keeps emitting frames while input continues and never blocks dispatch.
- Add regression tests: input dispatch no longer awaits the screenshot, rapid
  inputs coalesce, and continuous input keeps refreshing before it stops.

Scenarios: Verified in the live Browser panel — a single click completes in
~57ms (was blocked behind a 171ms capture), and a 1.14s sustained wheel gesture
renders ~7 frames throughout the scroll instead of one frame after it ends.

* fix(browser): harden worker and session lifecycle

* fix(browser): address latest review feedback

* fix(frontend): preserve optimistic new-chat message

* test(e2e): preserve mocked message run ids

* fix(browser): address capability review feedback

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-21 11:46:33 +08:00
ChenglongZ
ca16b64b26
feat(agent): support config-declared lead middlewares (#3964)
* feat(agent): support config-declared lead middlewares

* fix(agent): preserve configured extension middlewares

* fix(agent): address middleware extension review

* fix(agent): tighten middleware extension docs and tests
2026-07-21 09:39:44 +08:00
Aari
09e25b8a32
fix(auth): let deployments close local self-registration (#4311)
* fix(auth): let deployments close local self-registration

The OIDC provisioning policy (allowed_email_domains, require_verified_email,
auto_create_users) is enforced only in the SSO callback via
get_or_provision_oidc_user. POST /api/v1/auth/register creates a local account
without consulting any of it, and nothing can turn that path off, so a
deployment declaring an email-domain allowlist can still be joined by any
address through local registration.

Add auth.local.allow_registration (default true, so existing deployments are
unchanged) and gate /register on it before the account is created. Report the
flag from /setup-status so the login page stops offering a signup entry the
Gateway will reject.

/initialize is deliberately not gated: it is the bootstrap path, guarded by
admin_count == 0, and closing it would leave a fresh install unable to create
its first admin.

An unreadable config.yaml falls back to the pre-gate default (open) rather than
making these two endpoints a hard dependency on the file.

* docs(auth): align registration-gate fallback wording with the FileNotFoundError catch
2026-07-20 23:33:09 +08:00
Daoyuan Li
474a0fd6b7
fix(gateway): correct GitHub auto-retry claim in webhook route docs (#4289)
While investigating a review on PR #4274 (which corrected the same
misconception in the channel manager's dedupe-TTL comment), I found a
more elaborate version of the same factual error in this route,
predating that PR: the docstring claimed "GitHub retries 5xx deliveries
with exponential backoff (up to ~5 attempts over ~8 hours)". No such
behavior exists. Verified directly against GitHub's documentation:
failed deliveries (5xx, timeout, connection error alike) are simply
recorded as failed and never automatically redelivered, for both
repository and GitHub App webhooks
(https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries).
Redelivery is always explicit: the "Redeliver" button, the REST API, or
an operator's own recovery script (GitHub's own recommended pattern for
the latter runs on a 6-hour cron, illustrating this is opt-in operator
tooling, not a GitHub-side mechanism).

The 503-vs-200 status code choice itself was already correct and is
unchanged: 503 keeps a transient failure recorded as failed so it can be
recovered via one of the explicit paths above, while 200 correctly
marks permanent/non-retryable conditions (disabled channel, unknown
event, missing service) as done so no pointless redelivery is invited.
Only the rationale text describing *why* was wrong.

- github_webhooks.py: reworded the route docstring and four inline
  comments/log messages that repeated "GitHub retries" / "so GitHub
  retries" framing, citing the actual documented behavior.
- test_github_webhooks.py: renamed
  test_dispatch_failure_returns_503_so_github_retries to
  test_dispatch_failure_returns_503_not_200 and reworded its docstring
  plus two nearby docstrings/comments with the same framing. No
  assertions changed - same status codes and response bodies checked.
- AGENTS.md: corrected the GitHub Webhooks router table row to match.

All 44 tests in test_github_webhooks.py pass, plus the full
test_github_dispatcher.py / test_github_channel.py /
test_github_registry.py / test_channels.py suites (313 total). ruff
check and ruff format --check are clean on both touched Python files.
2026-07-20 08:01:18 +08:00
Andrew Chen
a9a57fb711
fix(gateway): handle null config.configurable in _resolve_thread_id (#4301)
* fix(gateway): handle null config.configurable in _resolve_thread_id

`_resolve_thread_id` read the thread id with
`(body.config or {}).get("configurable", {}).get("thread_id")`. The
`.get("configurable", {})` fallback only applies when the key is absent.
`RunCreateRequest.config` is typed `dict[str, Any] | None`, so a client
can send `{"config": {"configurable": null}}`; the key is then present
with value `None`, and `None.get("thread_id")` raises `AttributeError` —
an unhandled HTTP 500 on `POST /api/runs/stream` and `/api/runs/wait`.

Coalesce the inner value with `or {}` so a null `configurable` yields a
freshly generated thread id, matching the docstring ("or generate a new
one") and the behaviour of `config: null` / `configurable: {}`. The
sibling `_checkpoint_configurable` in thread_runs.py already guards the
same field defensively. Behaviour is byte-for-byte identical for every
input that worked before.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* style: ruff format the new test (lint-backend runs ruff format --check)

Collapse the multi-line assert ruff format flags; behaviour unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Also coerce null configurable in build_run_config, not just _resolve_thread_id

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 07:54:15 +08:00
Daoyuan Li
9a5d701355
fix(channels): document GitHub dedupe TTL, tighten redelivery tests (#4274)
* fix(channels): document GitHub dedupe TTL, tighten redelivery tests

Follow-up to PR #4104's review, whose non-blocking notes were left
unaddressed at merge time:

- Document the previously-undocumented inbound-dedupe TTL
  (INBOUND_DEDUPE_TTL_SECONDS, 10 min, in-memory only) directly above
  the constant in ChannelManager: what it is, why the window, and what
  happens at the boundary (a manual "Redeliver" clicked after the TTL
  has elapsed, or any redelivery after a Gateway restart, is not
  deduped, since _recent_inbound_events is never persisted to
  ChannelStore). Cross-reference it from the GitHub dispatcher's
  fan-out comment, which previously implied unconditional redelivery
  coverage.
- test_channels.py::test_github_redelivery_is_deduped_like_other_channels:
  the _gh helper stamped a 2-part delivery:agent id while production
  stamps a 3-part delivery:user:agent id. Align the helper with the
  real shape and add a cross-user assertion the old 2-part shape could
  not even express.
- test_github_dispatcher.py::test_missing_delivery_header_leaves_dedupe_open:
  previously only asserted the dispatcher-layer id was None. Add the
  manager-level _is_duplicate_inbound assertion across two header-less
  deliveries so a future regression that treats a missing id as a real
  key is caught here too, not only at the dispatcher layer.

All four affected test files (test_github_dispatcher.py,
test_channels.py, test_github_channel.py, test_github_registry.py)
pass; ruff check and ruff format --check are clean. Verified the two
tightened tests actually discriminate by temporarily reintroducing
each bug they target (2-part id shape; missing-id treated as a real
key) and confirming the new assertions fail, then reverting.

* fix(channels): correct GitHub redelivery claim in dedupe comments

fancyboi999's review on this PR flagged that the previous commit's new
comments (manager.py, dispatcher.py) justified the 10-minute dedupe TTL
by claiming GitHub automatically retries a failed delivery after its
10-second timeout. GitHub's own documentation says the opposite:
failed deliveries (non-2xx, timeout, connection error) are recorded as
failed and never automatically redelivered, for both repository and
GitHub App webhooks alike. Every redelivery is explicit: the "Redeliver"
button, the REST API, or an operator's own recovery script.

- manager.py: rework the INBOUND_DEDUPE_TTL_SECONDS comment to state
  GitHub's actual behavior (no auto-retry), cite the authoritative doc
  (docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries),
  and justify the 10-minute window as a bound on explicit near-term
  replays rather than an automatic-retry window. No longer asserts
  unverified retry behavior for the other channels either.
- dispatcher.py: same correction for the dedupe_message_id comment,
  which previously implied "retried after a timeout" as a distinct,
  automatic case alongside the manual "Redeliver" button.
- test_channels.py / test_github_dispatcher.py: reworded the two
  docstrings and one section comment that repeated the same
  "retry-on-timeout" framing, so the corrected mental model doesn't
  sit next to contradicting prose nearby. No assertions changed.

Verified directly against GitHub's current documentation (fetched, not
paraphrased from memory): "Handling failed webhook deliveries" states
plainly that GitHub does not auto-redeliver; "Automatically redelivering
failed deliveries for a GitHub App webhook" / "...for a repository
webhook" are both guides for a user-written recovery script (GitHub's
own example runs on a 6-hour cron), not a native GitHub feature. No
difference in this behavior between GitHub Apps and repository webhooks.

All four affected test files pass; the two `test_bot_login_whitespace_only`
/ `test_trigger_mention_login_whitespace_only` failures in
test_github_dispatcher.py are pre-existing on this branch's unmodified
HEAD (confirmed against a clean checkout of the same commit) and are
unrelated to this change. ruff check and ruff format --check are clean
on all four touched files.
2026-07-19 22:08:48 +08:00
Aari
bc9c027a54
fix(uploads): claim the converted markdown filename before writing it (#4288)
* fix(uploads): claim the converted markdown filename before writing it

* doc(changelog): record the converted-markdown filename collision fix

Covers both surfaces that report markdown_file: the gateway uploads route
and DeerFlowClient.upload_files.

* fix(uploads): release the claimed markdown name when conversion fails

Claiming the companion .md name before conversion means a conversion that
writes nothing leaves the name reserved for the rest of the request, so a
later same-stem upload is renamed against a name nothing occupies. Uploading
notes.docx (conversion fails) + notes.md stored the user's file as notes_1.md
with no notes.md on disk, where main kept notes.md.

Discard the claim when conversion returns None, at both call sites that
pre-claim it. Covers both victims: a later same-stem .md upload, and the next
convertible's companion.
2026-07-19 17:24:31 +08:00
yym36991
0f088033fe
fix(gateway): prefer X-Trace-Id over metadata.deerflow_trace_id when header is set (#4283)
When TraceMiddleware binds a valid inbound X-Trace-Id, worker resolution
prefers that id over config.metadata.deerflow_trace_id so logs, response
headers, Langfuse, and runtime context stay aligned.
Also add trace-context marker reset coverage for the inbound-header flag.
2026-07-19 09:13:51 +08:00
Daoyuan Li
75fa028e89
fix(artifacts): serve inline binary artifacts via FileResponse for Range support (#4281)
* fix(artifacts): serve inline binary artifacts via FileResponse for Range support

Audio/video/image artifacts that are previewed inline (not downloaded, not
active content) were served by reading the whole file into memory and
returning it through a plain Response. That response never sets
Accept-Ranges and ignores any Range header the browser sends, so seeking
an <audio>/<video> element backed by this endpoint always gets the full
200 response back instead of a 206 partial response for the requested byte
range -- which is why dragging the seek bar on an audio artifact restarts
playback from the beginning instead of jumping to the new position.

Route this case through FileResponse instead (as the active-content/
download branch already does), which handles Range/If-Range natively.
Verified with a real Range request against the endpoint: initial load now
reports Accept-Ranges: bytes, and a ranged GET returns 206 with the correct
Content-Range and only the requested slice of bytes.

Fixes #3240

* fix(artifacts): address review nits on the inline FileResponse branch

Two non-blocking nits from review:

- Drop the redundant filename= kwarg on the inline_file FileResponse
  call. Content-Disposition is already set explicitly on this branch,
  and FileResponse only uses filename to setdefault that same header
  -- which is a no-op once it's already present. Harmless today, but
  removes the latent risk that a future Starlette version turning that
  setdefault into a hard set would silently flip inline preview to
  attachment.
- Reword the blocking-IO regression-anchor docstring: it claimed
  awaiting get_artifact for a binary artifact does "zero filesystem
  IO", but _read_artifact_payload still runs exists/is_file/
  mimetypes.guess_type/is_text_file_by_content (an 8 KB read) for
  binary files too, just offloaded via asyncio.to_thread like the text
  branch. The gate has nothing to catch because that IO is off the
  event loop, not because there's none -- reworded to say no full-file
  read happens, matching the accurate framing already used one
  paragraph up.

tests/test_artifacts_router.py, tests/blocking_io/test_artifacts_router.py
(19 tests) are green, and ruff check/format are clean on both changed
files.
2026-07-19 07:31:29 +08:00
Ryker_Feng
a028dfd5fb
feat(auth): add keep me signed in login option (#4255)
* feat(auth): add keep me signed in login option

* fix(auth): address remember-login review feedback
2026-07-18 22:17:15 +08:00
Ryker_Feng
e89edb39b1
fix(gateway): reject non-positive read limits (#4284) 2026-07-18 22:12:57 +08:00
Huixin615
c9b6131f8f
fix(skills): reload mounted skills without restarting Gateway (#4264)
* fix(skills): add admin-only reload endpoint

* fix(skills): preserve cache when reload fails
2026-07-17 23:22:16 +08:00
hataa
10890e10a8
feat(authz): propagate trusted authorization principal context (#4203) 2026-07-17 14:49:51 +08:00
AochenShen99
94a34f382d
feat(context): record effective memory identity per run (#3556)
* feat(context): record effective memory identity per run

* fix(context): address memory identity review feedback

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-16 09:39:09 +08:00