* feat(checkpoint): make delta snapshot_frequency configurable
* fix(config): carry legacy checkpoint_delta_snapshot_frequency with warning
Addresses review on #4516: the rename from the flat
database.checkpoint_delta_snapshot_frequency key to nested
database.checkpoint_delta.snapshot_frequency silently dropped the old
value (pydantic extra="ignore"). Add a before-validator that maps the
legacy key onto the nested one with a deprecation warning (nested key
wins when both are set), plus a CHANGELOG breaking-change note covering
the rename and the 1000 -> 10 default change.
* fix(checkpoint): validate frozen snapshot frequency
* feat(frontend): allow chat replies during clarification
* fix(frontend): unlock input polish during clarification
Remove hasOpenHumanInputCard from inputPolishDisabled so the polish
button stays available when a clarification card is open, matching
the composer unlock behavior. Clean up the now-unused useMemo and
import.
* feat(memory): integrate FTS5 retrieval adapter
* deps: add jieba as default dependency for Chinese tokenization
Without jieba, FTS5 unicode61 tokenizer treats entire Chinese sentences
as single tokens, making single-character or sub-phrase searches
impossible (e.g. '吃' or '油泼面' returns 0 hits against
'用户喜欢吃油泼面'). jieba segments Chinese text into meaningful tokens
before indexing.
* fix(memory): avoid treating hyphens as FTS5 operators
* feat(memory): make Chinese tokenization optional
* fix(memory): warm every requested retrieval scope
* fix(memory): close retrieval resources on shutdown
* fix(memory): close backend when shutdown flush fails
* fix(memory): recreate corrupt retrieval index
* fix(memory): tolerate partial retrieval rebuilds
* fix(memory): warm retrieval index in background
* fix(memory): preserve shutdown flush budget
* fix(memory): stop retrying partial lazy rebuilds
* fix(memory): close retrieval through storage
* refactor(memory): simplify retrieval scope limit
* docs(memory): clarify retrieval shutdown lifecycle
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(checkpoint): production-shaped full/delta benchmark with configurable snapshot frequency
- Group benchmark scripts into per-family folders (checkpoint/, sandbox/)
- Extract shared benchmark infrastructure into checkpoint_bench_common.py
- Add checkpoint_delta_snapshot_frequency config (default 1000, process-frozen);
freeze it in make_lead_agent and DeerFlowClient; key the state-schema
adaptation cache by resolved frequency
- New bench_production.py: per-case child processes run N ainvoke turns through
the real lead-agent graph (scripted deterministic model, real AsyncSqliteSaver),
then measure GET /state + POST /history through the real Gateway route stack
in one event loop (httpx ASGITransport), cold/warm accessor-cache split,
cross-mode digest gates
- New summarize_production.py: delta/full ratios plus decision metrics
(snapshot_write_spike, cache_effect_ms, checkpoint_write_share,
auto-discovered history per-limit ratios)
* fix(checkpoint): address production benchmark review
* feat: add lark cli integration
* fix: polish lark integration actions
* feat: support lark incremental permissions
* fix: detect lark authorization completion
* fix: harden lark integration install
* feat: expand lark auth scopes and reuse host auth in sandbox
Default lark auth to least-privilege (recommend=false, base sign-in only)
and expose the full set of lark-cli --domain business domains as native
--domain grants instead of a 4-domain read-only mapping. Resolve the
skill pack from the latest larksuite/cli GitHub release at install time
with content-hash integrity, and surface version/runtime drift in status.
Share the per-user lark-cli config/data profile between the Gateway
Settings auth flow and agent conversations by mounting the integration
dirs into the AIO sandbox and injecting the matching env for lark-cli
commands, with an allowlisted extra_mounts path in the provisioner/K8s
backend and traversal guards on integration paths.
* style: fix lint issues from ruff and prettier
Sort imports in the provisioner PVC test and re-wrap two long i18n
description strings to satisfy backend ruff and frontend prettier CI.
* fix(lark): address managed integration review feedback
* fix(frontend): stabilize integrations settings e2e
* test(sandbox): isolate remote backend legacy visibility check
* test: fix backend unit failures after merge
* Harden Lark integration review fixes
* Format Lark integration E2E test
* fix(lark): harden sandbox credential exposure and status disclosure
Address willem_bd's security review on PR #3971:
- Mount the per-user lark-cli config dir (long-lived appSecret) read-only
into the AIO sandbox; only the refreshable-token data dir stays writable.
- Redact host filesystem paths (install_path, cli.path) from
GET /lark/status and the config/auth complete responses for non-admin
callers, fail-closed on any auth error.
- Document the npm postinstall trade-off (--ignore-scripts is not viable
because @larksuite/cli fetches its platform binary in postinstall).
- Document the sandbox credential trust boundary in AGENTS.md and README,
pointing at the sidecar-broker follow-up (#4338).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(browser): add agentic browser control
* fix(frontend): format browser view changes
* fix(browser): keep browser optional and isolate sidecar layout
* fix(browser): address PR review security and IME findings
- Nginx: add a browser-stream WebSocket location before the generic
/api/threads regex so Live upgrades instead of downgrading to HTTP
(both nginx.conf and nginx.local.conf).
- Ownership: require an existing owned thread for the WS stream and REST
navigate, and tear down the browser session on thread deletion so a
later caller cannot reuse a retained page/cookies by guessing the id.
- SSRF: enforce the URL policy at the browser request boundary via a
context-level route guard covering redirects, popups, iframes, and
subresources (skipped for CDP-attached Chrome).
- IME: skip key forwarding while a composition is active so confirming a
CJK candidate with Enter no longer submits the remote page form.
Adds regression tests for the request guard, session teardown on delete,
and the composing-Enter key decision.
* fix(frontend): smooth streaming in long tool threads
* Revert "fix(frontend): smooth streaming in long tool threads"
This reverts commit f0462516eabe77f138d4027ea1c714fb226683cf.
* fix(browser): address review security and lifecycle findings
- Reject cross-origin WebSocket upgrades on the live browser stream
(Origin allow-list reuse of CORS/same-origin helpers) to close a
WS-CSRF hole, and fail closed when the ownership store is absent.
- Warn when a CDP-attached session runs with the SSRF request guard
off, and drop the unreachable CDP screencast teardown dead code.
- Read browser session launch config from a single canonical source
(browser_navigate) so it is deterministic regardless of call order.
- Bound per-thread Chromium accumulation with idle-timeout eviction
and an LRU max-sessions cap.
- Reset the Live reconnect counter on a successful open so the stream
can't permanently stall after the cumulative attempt cap.
* fix(frontend): reduce long tool thread render stalls
Reuse stable historical message groups during streaming, defer heavy Markdown and browser previews, and lazy-decode message images.
* fix(browser): keep live control responsive during continuous input
Why: Manual browser control felt laggy — a physical click ran the remote
Playwright click three times and each non-move input synchronously awaited a
JPEG screenshot, so events queued behind capture (queue wait up to ~237ms).
The first async attempt used a trailing-edge debounce, which froze the visible
page until a wheel/keyboard gesture stopped ("scroll finishes, then it jumps").
What:
- Frontend forwards one `click` per physical click instead of also emitting
`down`/`up`, so the remote page is not clicked twice per gesture.
- Backend detaches live-frame capture from input dispatch: non-move actions
start a rate-limited background refresh loop (leading frame + bounded cadence)
that keeps emitting frames while input continues and never blocks dispatch.
- Add regression tests: input dispatch no longer awaits the screenshot, rapid
inputs coalesce, and continuous input keeps refreshing before it stops.
Scenarios: Verified in the live Browser panel — a single click completes in
~57ms (was blocked behind a 171ms capture), and a 1.14s sustained wheel gesture
renders ~7 frames throughout the scroll instead of one frame after it ends.
* fix(browser): harden worker and session lifecycle
* fix(browser): address latest review feedback
* fix(frontend): preserve optimistic new-chat message
* test(e2e): preserve mocked message run ids
* fix(browser): address capability review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(skillscan): detect exfil through instance/dataflow network clients
* fix(skillscan): resolve client handles with Python lexical scopes
_walk_client_nested_scope copied every live handle into a nested scope after
excluding only parameters. That is not Python's name resolution: a class
namespace is not a closure scope for its methods, a function-local binding
shadows an enclosing name across the whole body, and comprehensions bind their
targets in a scope of their own. Benign skills were therefore blocked as
python-env-dump-exfil (CRITICAL) even where the outbound-looking call provably
cannot run on the tracked client.
Derive each scope's bindings lexically instead:
- a class body reads its enclosing scope, but the names it binds are not passed
into the methods defined in it;
- comprehensions are their own scope, so only the outermost iterable is
evaluated outside and every for-target shadows first;
- a function-local prepass excludes names the body binds anywhere, with
global/nonlocal opting back out.
Detection is unchanged where the client really is reachable: a comprehension
calling an unshadowed handle, a method closing over an enclosing function's
handle, and a global-declared module handle all still block.
* fix(skillscan): apply target rebinding in Python evaluation order
The generic ast.iter_child_nodes() fallback yields fields in declaration order,
which for `for`/`async for`, `:=` and `+=` puts the binding target ahead of the
expression Python evaluates to produce it. Visiting the target first dropped the
handle before the call that actually runs on it, so `for s in s.post(host, ...)`
reported nothing even though Python calls post() on the client before binding the
loop target. Match captures bind through a plain `name` string rather than a Name
node, so the Store branch never saw them and a rebound name kept a stale handle.
Assignment expressions inside a comprehension were applied only to the
comprehension-local map, leaving the containing scope stale as well.
Give every bind-after-evaluate construct its own branch:
- `for`/`async for` walk the iterable against the pre-loop binding, then rebind
the target, then walk the body;
- `:=` walks its value first and binds into the containing scope too, since PEP
572 puts a comprehension's walrus target there;
- `+=` walks its value before dropping the target;
- match captures drop the handle, matching how a rebind under `if`/`try` that may
equally not execute is already treated.
The bypasses this closes are the reason detection widens here; the comprehension
walrus and match cases narrow it back where the client provably cannot be the
receiver.
* fix(skillscan): preserve scoped client handle semantics
* fix(skillscan): scan sinks inside assignment target expressions
The Assign/AnnAssign, AugAssign, For/AsyncFor, with, and comprehension
branches evaluated only the value and rebound the target, so a client
call placed in an attribute receiver or a subscript value/index -- both
evaluated at bind time -- was never scanned. os.environ could exfil
through a tracked client while python-env-dump-exfil reported nothing.
Add _walk_client_target_exprs to walk the executable parts of a binding
target (attribute receiver, subscript value+slice, recursing through
tuple/list/starred) in Python evaluation order, without treating Store
name leaves as reads. The AnnAssign annotation expression is walked too.
Name-leaf invalidation is unchanged.
* fix(skillscan): apply assignment targets and annotations in runtime order
The round of assignment-target scanning walked every target then rebound
the names in one batch, and always walked an AnnAssign annotation before
the target. Python instead binds chained and destructured targets left to
right (so `session = out[session.post(...)] = cfg` runs the subscript on
the already-rebound name), evaluates a variable annotation only in module
or class scope (never in a function, never under `from __future__ import
annotations`), and evaluates an executed annotation after the target.
The scanner therefore hard-blocked benign skills.
Bind each target left to right via _bind_client_targets (walk its
executable sub-expressions against the current bindings, then rebind
before the next target), and walk an annotation only for the nodes
_evaluated_annotation_nodes marks as actually evaluated, after the
target. Target-expression scanning and name-leaf invalidation are
otherwise unchanged.
* fix(skillscan): scan client sinks in evaluated function-signature annotations
A function's parameter and return annotations are evaluated at def time
in the enclosing scope, like its decorators and defaults, so a tracked
client sink placed in one is a real egress. _client_scope_prelude walked
the decorators and defaults but not the annotations, so os.environ could
exfil through `def f() -> session.post(host, json=dict(os.environ))`
while python-env-dump-exfil reported nothing.
Record function/async-function defs in _evaluated_annotation_nodes (their
signatures evaluate at def time unless from __future__ import annotations
postpones them) and, for those nodes, add the parameter and return
annotation expressions to the enclosing-scope prelude walk.
* fix(skillscan): match runtime evaluation order for annotations and except handlers
* fix(skillscan): scope try/match branches to their own selection state
* fix(skillscan): propagate branch bindings to everything that observes them
A branch's net effect has to be visible exactly where Python makes it visible.
The walker isolated `except`/`else`/`match` bodies into scope copies and then
discarded them, so a client created on the branch was invisible to `finally`,
to the code after the statement, and to anything defined inside the branch,
while a name the branch replaced stayed a sink receiver.
- `except*` clauses are sequential, not alternatives: thread one scope through
body, clause types and clause bodies in source order instead of reusing the
mutually exclusive copies ordinary `except` needs.
- Fold each `except`/`else`/`match` branch's net effect back into the scope that
`finally` and the following code read, and make the branch scope what nested
definitions close over.
- Keep a fallthrough scope across `match` guards, so a guard that returned false
still hands its side effects to the next case, while pattern captures stay
isolated to their own case.
- Keep an `as` target path-local: Python unbinds it only on the path that ran, so
dropping it for every path erased a live handle where no such handler executed.
Adds a 14-case runtime-oracle regression covering both directions at every
branch site; each of the ten guards was deleted on its own to confirm the test
that pins it goes red.
* fix(skillscan): join alternative branches instead of overwriting one with another
Only one of a statement's alternative branches runs, but the walker folded each
one into a single destructive binding map. Whichever branch was visited last
therefore decided the state: a handler that rebinds the name erased a sibling
that leaves the client in place (missing a client Python really calls), and a
handler that builds one was credited on paths where it never ran (inventing a
CRITICAL sink). Alias targets had the mirror problem, since only key presence
was compared, so replacing `import x as name` on a live branch was ignored.
- Join alternatives into a may-state: a name stays a sink receiver when any
feasible branch leaves it a client, and stops being one only when every
feasible branch replaced it. Alias targets join toward the target that can
still name a constructor.
- Treat each `except*` clause as optional rather than threading every clause
body unconditionally, so a clause whose type never matched cannot erase a
handle the next clause calls.
- Keep the fall-through state (no exception raised, no case matched) as one more
alternative, and drop it only where the source decides the outcome: a literal
always-raising body selecting one handler, a literal exception group choosing
`except*` clauses, a wildcard or literal-equal `match` case.
Also corrects an existing match-capture test that asserted the non-exhaustive
case is benign: the runtime oracle shows the original client still takes the
call on the path where nothing matched. Adds runtime-oracle regressions for both
directions at every alternative site; each of the nine guards was deleted on its
own to confirm the test pinning it goes red.
* fix(skillscan): model feasible conditional client flow
* fix(skillscan): preserve feasible control-flow outcomes
* fix(skillscan): model expression evaluation paths
* fix(skillscan): narrow instance-client detection to lexical statement order
The construction-to-use signal had grown into a path-aware interpreter:
exception selection, except* subgroup consumption, match capture timing,
finally override, comprehension laziness, annotation evaluation order, and
may-state joins over feasible branches. That is the heavyweight analysis
RFC #2634 rules out of Phase 5, and because the signal feeds a CRITICAL
rule it hard-blocks skill installation, so every ambiguity it resolved by
over-reporting cost a benign skill instead of a human review.
Replace it with ordinary statement order over a one-level handle map: a
known constructor bound to a simple name (including `with ... as`), a
direct outbound method call on that name in the same lexical scope,
rebinding invalidation, and name-to-name alias propagation so `s = session`
does not shed the handle. A sink is recorded at the call, so a rebind after
the call cannot retract it.
Compound statements are not interpreted. Every name an if/try/except*/loop/
match may bind is dropped before its bodies are walked, and each body is
walked from an isolated copy. Dropping before rather than after is what
keeps a `finally` that runs after a handler rebound the name, a later
except* clause, and a second loop iteration from reporting a client the
runtime never calls. Bodies are still walked, or wrapping any construction
in `if True:` would be a universal bypass.
Lexical scoping is unchanged: class namespaces are not closures for their
methods, comprehension targets and function-local bindings shadow, and
alias visibility stays per scope.
The cases this gives up are false negatives by construction and are
recorded in #4296, pinned by a test that asserts the runtime really calls
the client while the scanner stays silent.
Verified: 102 SkillScan tests; full suite 8 failed / 7851 passed, the same
network-dependent web-fetch tests that fail on clean main; per-clause red
check 12/12 guards red; 925 repo-owned files scanned branch vs main with 0
new and 0 lost CRITICAL findings; #4158 bypass BLOCKED and #4153 false
positive allowed with 0 findings through the real enforce_static_scan gate.
* test(skillscan): pin closure boundary
* refactor(skillscan): narrow client handle analysis
* fix(skillscan): close client handle correctness gaps
* fix(skillscan): require proven client imports
