Remove the legacy stack the hexagonal slice replaced, now that nothing
assembles it: app/scheduler/service.py (the old orchestration),
app/gateway/routers/scheduled_tasks.py (the old dict-returning router,
no longer mounted), the deerflow/scheduler package (its cron/timezone
rules live in ScheduleSpec), the dict-returning repositories in
persistence/scheduled_task*/sql.py, and the deps.py providers and
app.state wiring that served them. The ORM rows and the
uq_scheduled_task_run_active partial unique index stay -- the table
definitions live with the shared engine/alembic infrastructure and the
schedule adapters are their only readers and writers.
The legacy test suites go with the code they pinned; every scenario has
a counterpart in the new suites. The one suite that guarded semantics
rather than the old implementation -- the real-database dispatch-race
TOCTOU tests -- is migrated to the new stack as
test_schedule_dispatch_race.py, driving ScheduleService over the real
SQL adapters with the same barrier, natural-timing, and index-semantics
cases.
Docs and comments that named the old classes as the current wiring
(backend/AGENTS.md, reload_boundary.py, channel/service comments) now
name the composition-root wiring instead.
created_at was minted twice for one construction: the aggregate's field
default read the clock, then the adapter's add() read it again and stored
its own value -- so the aggregate a caller held disagreed with the stored
row by however long the insert took, and the model docstring pointed at a
write path that no longer exists.
ScheduledTask.create now stamps created_at and updated_at from the
explicit now= it already receives (one clock reading, the rule input),
and the adapter persists the aggregate's instants verbatim on insert --
matching the feedback reference implementation. updated_at's later life
stays deliberately with the adapter's write paths: the CAS methods
(record_launch / record_completion) write the row without ever holding an
aggregate, so only storage can stamp 'last written'. The model docstring
now states that ownership split instead of citing the deleted legacy
repository. Pinned by a domain test on the factory and a contract case
on add() across both implementations.
SqlScheduledTaskRepository._to_domain raised InvalidScheduleError for a
row whose stored schedule no longer parses -- the same error the
aggregate raises for a client-submitted schedule, which the router maps
to 422. A stored fault therefore told the client its perfectly fine
request was wrong, and made the row unrepairable over HTTP: PATCH reads
the task before writing, so the fix path 422'd too. Enum rebuild
failures were worse -- a raw ValueError crossed the boundary
untranslated.
The rebuild is now translated to a dedicated CorruptStoredScheduleError,
raised only by the persistence adapter and deliberately absent from the
router's status table, so it falls through to the unclassified-500
branch: a server-side fault reported as one. List reads keep skipping
and logging the bad row. Pinned by tests/test_schedule_corrupt_rows.py
against a real sqlite database.
Add domain/schedule/commands.py with one frozen dataclass per HTTP-driven
write use case -- CreateScheduledTask, UpdateScheduledTask, PauseTask,
ResumeTask, DeleteTask, TriggerTask -- and make the service methods their
handlers, keeping the naming chain aligned across all three spellings
(command / handler / <Command>Request). The clock stays an explicit now=
handler parameter: it is a rule input, not part of the client's intent.
UpdateScheduledTask expresses partial updates with an UNSET sentinel, so
absence is unambiguous; the wire keeps its historical None-means-omitted
convention and the request model's to_command owns the translation. The
former ContextChange moves from service.py into commands.py unchanged.
The request models are renamed to <Command>Request and own transformation
① (to_command): identity is injected server-side and pinned by a test to
never appear on the wire models. Clock- and callback-driven writes
(run_once, dispatch_task, handle_run_completion, reconcile_on_startup)
deliberately stay plain methods -- those drivers have no wire shape to
translate, their inputs are already domain vocabulary.
The context package now also exports the commands and the service, which
completes its public API and retires the stale 'service not landed' note.
Move model/errors.py up one level to domain/schedule/exceptions.py, a
sibling of the model, matching the AWS domain layout the spec mandates
(exceptions/ is its own member of the domain folder, not part of the
model) and the feedback reference implementation. Class names keep the
PEP 8 Error suffix. Pure move -- the nine classes are AST-identical to
the originals; imports across the domain, adapters, router, and tests
now take errors from deerflow.domain.schedule.exceptions.
Resolves the alembic head fork: main introduced 0010_run_cancel_request
on 0009 while this branch carried 0010_feedback_tags -> 0011_..._drop.
The unmerged branch revisions are renumbered and rechained after main's
published one (0010_run_cancel_request -> 0011_feedback_tags ->
0012_feedback_drop_message_id), and every test head pin moves to 0012.
- command-ify the write use cases (RateRun / RetractRunRating; queries
keep plain parameters, commands stay dumb data)
- split the domain errors into exceptions.py, a peer of model.py
(PEP 8 Error suffixes, AWS-style module name)
- unify the aggregate->row mapping as _apply(row, feedback) so one
explicit field list serves both the insert and the update path
- drop the unused feedback.message_id column (migration 0011): feedback
is bound to a run, nothing ever wrote or read the field
- pin remove_for_run's equality semantics for user_id=None in the
contract suite and fix the port docstring that contradicted both
implementations
* fix(sandbox): unwrap Overwrite-wrapped state in ensure_sandbox_initialized
The same fork-restored wrapper that crashed after_agent also reaches the sandbox init path, where sandbox_state.get() on the Overwrite object raises AttributeError. Share the unwrap helper from #4381's follow-up module deerflow/sandbox/overwrite.py and apply it at both init sites.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(sandbox): note why discarding fork_restored at the reuse sites is safe
* fix(sandbox): unify the Overwrite unwrap helper and pin the fall-through
- middleware.py now imports unwrap_sandbox from overwrite.py instead of
keeping a second local copy whose docstring had already drifted; the
shared helper covers both crash forms (subscript TypeError and the
.get()-form AttributeError)
- test the acquire fall-through: when the fork-restored id is gone from
the provider, a fresh sandbox is acquired and the stale wrapped state
is replaced by the plain acquired dict
- the reuse-path test now also asserts runtime.state["sandbox"] stays
wrapped, pinning the don't-treat-as-owned contract after_agent relies on
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(sandbox): unwrap Overwrite state in the sibling sandbox readers
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(checkpoint): make delta snapshot_frequency configurable
* fix(config): carry legacy checkpoint_delta_snapshot_frequency with warning
Addresses review on #4516: the rename from the flat
database.checkpoint_delta_snapshot_frequency key to nested
database.checkpoint_delta.snapshot_frequency silently dropped the old
value (pydantic extra="ignore"). Add a before-validator that maps the
legacy key onto the nested one with a deprecation warning (nested key
wins when both are set), plus a CHANGELOG breaking-change note covering
the rename and the 1000 -> 10 default change.
* fix(checkpoint): validate frozen snapshot frequency
* feat(lark): sidecar credential broker for sandbox lark-cli (Pattern B)
Removes the plaintext Lark credential mounts (appSecret + OAuth tokens)
from the sandbox container. A long-running broker sidecar owns lark-cli
and the per-user config/data dirs and serves the command surface over
Pod loopback; the sandbox gets only a forwarding shim on PATH, so the
raw credential files never exist in the sandbox filesystem.
- lark_broker.py: stdlib-only loopback broker (argv passthrough with
shell=False, server-injected credential env, bounded I/O) + shim
script constant + install-shim mode.
- docker/lark-cli-broker: init(install-shim) + serve image.
- provisioner: LARK_CLI_BROKER_IMAGE + provision_lark_cli_broker →
shim init container + lark-cli-broker sidecar (config/data mounted
sidecar-only); credentials dropped from the sandbox container;
/api/capabilities reports lark_cli_broker_image. Broker supersedes
the Pattern A init-container binary when both are configured.
- gateway: lark_cli_env_overlay(broker=True) omits config/data env;
sandbox_lark_broker_active() TTL-cached mode resolver; broker added
to sandbox_runtime_mode / readiness and the settings UI.
Opt-in and off by default (empty LARK_CLI_BROKER_IMAGE ⇒ no change).
Closes#4338
* fix(lark): address Pattern B broker review findings (#4501)
Follow-up to the sidecar credential broker addressing the PR #4501 review:
- shim: split the on-PATH lark-cli into a /bin/sh launcher + Python shim body
so broker mode fails loudly (exit 127, actionable message) instead of ENOEXEC
when the sandbox image ships no python3; interpreter pinnable via
DEERFLOW_LARK_BROKER_PYTHON. Launcher bakes in the shim's absolute path since
$0 is the bare command name when run off PATH.
- broker: drop the dead cwd payload field (broker can't see the sandbox FS) and
document the command-surface-only / no-file-IO limitation.
- broker: return a structured 500 JSON on unexpected exec errors so the shim
gets a meaningful message, not an opaque transport failure; set a handler
socket timeout to bound slow/stuck connections.
- broker: add an opt-in DEERFLOW_LARK_BROKER_DENY_SUBCOMMANDS denylist that
refuses secret-dumping subcommands before spawning the binary, forwarded from
the provisioner sidecar.
- gateway: tighten the per-bash-call broker probe timeout (1.5s) and cache
negatives longer (300s) so non-broker remote-provisioner users don't pay a
latency hit; guard the mode cache with a lock; drop the dead
_probe_provisioner_lark_cli_init_image wrapper.
- docs: remove the broken design-doc link from the broker README.
Adds tests for launcher python resolution, cwd omission, denylist enforcement,
500-on-error, hot-path probe timeout + negative caching, and provisioner
denylist-env wiring.
Editing the only turn of a thread reran the original prompt: the model
answered the question the edit was replacing while the UI showed the
edited text, and the edit vanished on reload.
The replay-base lookup decided whether a checkpoint predates the target
user message by message id alone. DynamicContextMiddleware re-keys the
first user turn to `{id}__user` mid-run, so every checkpoint written
before it holds the same prompt under an id the lookup cannot match. The
scan walked past those and anchored inside the run that produced the
turn — a checkpoint that still contains the original prompt and owns the
injection node's pending writes, which the replay then re-added after the
edited message.
Require the replay base to be a settled checkpoint (no pending tasks) in
both the lineage walk and the chronological fallback. That rule is
middleware agnostic: the first turn now anchors on the thread's empty
initial checkpoint and later turns on the previous run's tail, which also
drops the existing reliance on LangGraph discarding a stale `__start__`
write.
Edit replay additionally passes `head_checkpoint` so it resolves its base
lineage-first like regenerate does, and a replayed user message is
restored to its pre-swap id: replaying `{id}__user` into a state that has
no reminder yet makes the middleware treat the turn as already injected
and silently drops its date and memory block.
Frontend: a prepared replay masks the turn it supersedes, so the
optimistic-message baseline is taken from the post-mask human count. The
pre-mask count can never be exceeded when the replay puts exactly one
human message back, and on the first turn the runtime re-keys the
replacement message so identity comparison cannot stand in for the count.
Fixes#4531
`spec_column.py` and `spec_wire.py` were the same 60 lines twice, kept in
sync by a parity suite. Both are gone; what they did is now split along the
line that actually separates the two boundaries.
Why two files existed
---------------------
They were one module until the slice split it, and the reason given for the
split holds: a primary adapter must not import a secondary one, and the two
shapes are equal only by coincidence. But that argument only requires the two
*shapes* to be independent -- it does not require the parsing *rule* to be
written twice, and writing it twice is what needed `test_schedule_spec_parity`
to assert the two agreed, down to identical error text.
Splitting it properly
---------------------
`ScheduleSpec.from_primitives(schedule_type, *, cron, run_at, timezone)` takes
four strings, not a `Mapping[str, Any]` -- the mapping was the thing that kept
this out of the domain, and four strings carry no transport or storage format
with them. It owns the whole rule: unknown type, missing or non-string field,
unparseable `run_at`, and (via __post_init__, unchanged) 5-field cron and
resolvable timezone. Values are checked rather than trusted, since both
callers read data a client can influence.
Each adapter keeps only what is genuinely its own -- which two keys its format
uses -- as private methods on the class that owns the boundary, matching how
`SqlFeedbackRepository` and AWS's own ports-and-adapters sample put the
conversion inside the adapter rather than beside it:
SqlScheduledTaskRepository._spec_from_row / _spec_to_column
models.ScheduledTaskCreateRequest.to_schedule / models._spec_to_wire
The emit direction stays duplicated, deliberately: it is three lines per side
with no rule in it, and the two are *allowed* to diverge -- one is an HTTP
contract, the other a storage format. Asserting they stay byte-identical was
a constraint neither side asked for, so that suite is not replaced.
The router stops building value objects
---------------------------------------
`create` passes `body.to_schedule()`. `update` passes
`body.to_schedule(current.schedule)`, replacing eight lines that re-emitted
the current spec to the wire shape purely to read defaults back out of it;
omitted parts now come off the value object directly.
Tests
-----
`test_schedule_spec_parity.py` is deleted (162 lines). Its structural and
value cases moved to `TestFromPrimitives` in the domain suite -- stated once
now instead of parametrized over two implementations. Emit coverage was
already elsewhere: wire in `test_schedule_response_models`, column via the
repository round-trip in `test_schedule_fakes`.
One gap found while removing it: the `once` half of the update fallback had
no coverage on either side (both existing cases use cron), and it is exactly
the branch this commit rewrites -- `spec_to_wire(current)` round-trip to
`current.run_at.isoformat()`. Added
`test_a_timezone_change_on_a_once_task_keeps_the_same_instant`, and confirmed
by mutation that it is the only case that catches that branch breaking.
Behaviour is unchanged: same wire shapes, same normalizations (whitespace in
cron, trailing Z re-emitted as +00:00), same error messages, so the 422
details clients see do not move.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A cron task could become permanently unschedulable after a run that failed
fast. Reproduced against the real SQL adapters, and against the legacy path
for contrast -- the legacy code does not have this bug, so the hexagonal
slice introduced it.
The window
----------
`dispatch_task` already documents it: "a fast-failing run can reach
handle_run_completion before this write lands". The hook therefore reads the
task while the dispatch path has not yet written its bookkeeping, and the
snapshot it holds still carries the elapsed `next_run_at` the claim was made
on. `handle_run_completion` then wrote that whole snapshot back through
`save()`, so whichever landed second undid the other.
For a cron task the result is terminal in the worst way. `record_launch`
writes the fresh fire time, bumps `run_count` and clears the claim; the
completion's whole-aggregate write restores the elapsed fire time, rolls
`run_count` back and leaves `status='running'` with `lease_expires_at IS
NULL`. Neither `claim_due` branch matches that shape (one needs `enabled`,
the other needs an expired claim), and `cancel_stuck_once_tasks` only sweeps
`once` rows -- so nothing can ever reach the task again.
The legacy `app/scheduler/service.py` wrote the same outcome field by field
(`update(..., updates={"last_error": ...})`) and never touched scheduling
state, which is why it survives the same interleaving.
The fix
-------
`record_completion` joins `record_launch` as a second deliberately narrow
port method, for the same stated reason `record_launch` is not expressed as
`save(task)`: the two race, so neither may write through the aggregate. They
now own disjoint fields -- the launch owns the schedule, the completion owns
the verdict (terminal status plus `last_error`, with `None` meaning "do not
move the status", i.e. every cron task).
`save()` keeps its documented purpose, the user-initiated whole-aggregate
updates (`update_task` / `pause` / `resume`). `handle_run_completion` still
reads the task first, but only to ask `status_after_completion`, which reads
nothing but the schedule type -- immutable, so that read carries no
time-of-check risk.
Tests
-----
The contract suite missed this because it groups cases by port method:
`record_launch` appears only among its own, never interleaved with `save`,
and `test_protect_terminal_keeps_a_concurrently_finalized_verdict` pins the
mirror-image direction only. The in-memory double replaces the whole row too,
so both implementations were faithfully wrong -- the defect was in the
contract, not either adapter.
Added: `TestRecordCompletion` (5 cases, both implementations), including the
interleaving itself; and `test_a_cron_task_survives_a_launch_write_landing_mid_completion`,
which drives a real `ScheduleService` over a repo double that commits
`record_launch` between the hook's read and its write. Both were watched
failing first -- the service case failing with the actual symptom, a rolled
back `next_run_at`, not a missing method.
Verified end to end against file-backed sqlite with the real service: the
task keeps `enabled`, the concurrent fire time and run count survive, the
verdict is recorded, and the next poll still claims it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adopts the layout feedback landed in cb49dd67: secondary adapters live
under `app/adapters/<context>/`, one file per port, the file named after
the port in snake_case with the technology carried by the class name.
`app/infra/` is now gone entirely.
The rename also separates two meanings of "run" that shared one filename
space: `run_sql.py` held `ScheduledRun` (an execution record), while the
`run_launcher.py` still to come deals in Gateway runs.
task_sql.py -> scheduled_task_repository.py
run_sql.py -> scheduled_run_repository.py
spec_mapping.py -> spec_mapping.py (implements no port, so no rename)
Both SQL adapters now carry the `Secondary adapter (owned persistence)`
docstring marker -- this context owns both tables and writes its own
queries. `spec_mapping` says instead that it is a boundary mapping and
names its two callers. The package `__init__.py` is empty, so imports go
through the full path and a class's home file stays unambiguous.
Pure move: every top-level symbol was compared against its pre-move
original by AST dump (docstrings excluded), plus a separate per-class
method-name comparison, since a whole-class dump reports a docstring edit
and a renamed method the same way.
Also repoints three docstrings and one diagram that still named the
deleted `app/infra/` path.
One conflict, in frontend/src/core/threads/types.ts: main's #4513
(preserve message order during long runs) made RunMessage.seq required,
while this branch had added the optional feedback field next to it. Both
changes are independent and both kept — seq is now required per main,
feedback stays optional.
No migration renumber this round; main added no new revision, so
0010_feedback_tags still chains cleanly after 0009_webhook_dedupe.
Verified: frontend typecheck + lint clean and 844 tests pass (the
required seq propagates through the test helpers main updated), plus
74 backend feedback/thread-message tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A review pass over domain/schedule found three genuinely untested
branches, all of which now have cases:
- ensure_launchable with a naive `now`. next_after already had this
covered; the delay floor did not, so a caller handing over a naive
clock reading could have had it shifted by the local offset unnoticed.
- update_task changing the prompt. Only the title path was exercised.
- _save finding the row gone. get_task saw it and save no longer does,
which is a concurrent delete; the caller must get the same not-found it
would have got a moment earlier rather than a None leaking out.
That takes service.py and every model module to 100%. The one remaining
uncovered line is croniter's naive-return guard, carried over verbatim
from schedules.py and unreachable with an aware input -- it now says so
instead of looking like an untested branch.
Also fills in the documentation the migration skipped: TaskStatus,
ContextMode and RunStatus arrived from the original draft without
docstrings while their newer siblings had them, and ScheduleService plus
four of its use cases were undocumented. Each now records the reasoning a
reader would otherwise have to reconstruct -- why RUNNING is not "the
agent is executing", why SKIPPED never passes through QUEUED, why
INTERRUPTED is not FAILED.
CRON_FIELD_COUNT stops being exported: it has no consumer outside the
module that defines it.
Four changes, all from the same observation -- the service was expressing
domain concepts in language the domain does not use.
DispatchOutcome replaces the Literal. The caller branches on all four
values and the SKIPPED/CONFLICT distinction is itself a business rule, so
it is domain vocabulary like every other enum in this context; a bare
string was the odd one out.
lease_owner leaves the domain entirely. It was a pure pass-through: the
service held it only to hand it to claim_due, which never reads it back.
Which process claimed a task is an identity, not a rule -- an adapter can
record one for diagnostics without the domain carrying it. lease_seconds
stays, because how long a claim survives genuinely changes recovery
behaviour.
The _UNSET sentinel is gone. It existed for one field: thread_id, the only
update parameter with a meaningful None. But thread_id and context_mode
always move together, so packaging them as ContextChange removes the
ambiguity and lets every other field use plain None for "not supplied" --
which also matches what the HTTP layer already does with exclude_none.
The previous `title: str = _UNSET` annotation was simply untrue.
pause/resume no longer route through a _transition helper taking an
unbound method. Two direct bodies plus a _save that raises on a missing
row read better, and _save documents why this stays a read-modify-write:
pushing "not while running" into a storage predicate would put the rule
beyond a zero-IO test and give it a second home. Closing that window
properly needs optimistic locking, which needs a schema change.
Also cleans three lint findings surfaced by a broader rule set than the
project enables: a regex metacharacter in a pytest match=, an unused
override parameter, and two deliberately-naive datetimes that now say so
with a noqa rather than looking accidental.
The input port of the context: every scheduled-task use case, orchestrated
over the four output ports. It holds no business rules -- each decision is
delegated to the aggregate -- and `test_schedule_service.py` runs the
complete lifecycle with no HTTP, no database and no run runtime, which is
the acceptance criterion this migration was for.
`dispatch_task` mirrors the pre-migration structure line for line,
including its comments, and makes exactly three substitutions: bare dicts
become domain objects, the HTTPException-409 sniffing becomes `except
ThreadBusyError`, and the status-derivation static methods become
aggregate methods. Its four exits and their two conflict paths are
unchanged, and the tests pin the collapse: the fast-path rejection and
the active-slot rejection must produce identical results, asserted over
every field of DispatchResult.
One behavioural narrowing, deliberate. The old code wrapped the launch
*and* its follow-up writes in `except Exception`, so a failing bookkeeping
write was recorded as a failed launch -- marking an execution that had
actually started as failed. The port contract admits exactly two escapes
from `launch`, so only the launch is guarded now and a genuine write fault
propagates instead of being misreported.
SchedulePolicy gains max_concurrent_runs and lease_seconds. Both are
operator-tunable thresholds the domain needs but must not read, which is
what that value object is for; the claiming process's identity stays a
constructor argument since it is an identity, not a threshold.
Four Protocols the domain declares and the outer ring will implement,
plus the two DTOs that keep infrastructure types out of the inner ring:
- ScheduledTaskRepository / ScheduledRunRepository, exchanging domain
objects rather than the bare dicts the current repositories return
- RunLauncher, whose contract is that only ThreadBusyError or
LaunchFailedError may escape -- that translation is what keeps the run
runtime and the web framework out of the domain
- ThreadLookup, one method rather than the whole thread store
- LaunchedRun and RunOutcome, so the completion path stops taking a
runtime record the purity test would reject
Two deliberate departures from the earlier sketch. There is no Clock
port: `now` is already an explicit parameter throughout, so the domain
never reads a clock and the tests are already deterministic -- adding
one would only create a second source of truth for the same value. And
`record_launch` is not expressed as `save(task)`, because
`protect_terminal` makes it a compare-and-set against a concurrently
finalizing run; a read-modify-write through the aggregate would
reintroduce the race the flag exists to close.
The in-memory doubles model the active-slot rule rather than skipping
it: a double that never refuses a second active run would let the
service's conflict collapse go untested. Their semantics are pinned by
test_schedule_fakes.py, which becomes the contract suite once the SQL
adapters land and both tiers run the same cases.
Concurrency is out of scope for the doubles and says so in their module
docstring -- it stays covered against a real database in
test_scheduled_task_dispatch_race.py.
First step of the scheduled-task context's hexagonal migration: the
inner-ring model, with zero infrastructure dependencies.
- ScheduleSpec / SchedulePolicy value objects
- ScheduledTask aggregate root
- ScheduledRun aggregate
- 9 domain errors, 5 enums
Rules are migrated verbatim from their current homes, each method's
docstring citing the source line: timezone/cron/next-run calculation
from deerflow/scheduler/schedules.py, context-mode and re-arm rules from
routers/scheduled_tasks.py, and the four status-derivation rules from
app/scheduler/service.py.
Two things previously held by convention are now enforced by
construction. Validation and normalization live in __post_init__, so
building a ScheduleSpec field-by-field cannot bypass them. The skipped
tombstone is a separate factory, so it can never be written as the
transient queued row that would collide with uq_scheduled_task_run_active.
The domain does not serialize itself: mapping the stored schedule_spec
JSON in and out stays with the adapter layer, keeping Mapping[str, Any]
out of every domain signature.
Production code still runs through app/scheduler/service.py -- this
commit adds no call sites and changes no behavior.
`browserless` reads `cfg["timeout_s"]`, while its sibling web providers
`crawl4ai` and `jina_ai` read `cfg["timeout"]`. Tool configs allow extra
fields, so the unrecognised spelling is dropped without a diagnostic: someone
adapting one provider's config snippet for another silently gets the 30s
default instead of the timeout they set. (Observed in the other direction, on a
deployment whose crawl4ai entry carried `timeout_s`.)
Accept both keys, preferring the documented `timeout_s` when both are present.
While adding coverage, two pre-existing bugs in the same three lines surfaced,
both already guarded in crawl4ai/jina_ai but not here:
- `timeout_s: "30s"` (or any non-numeric string) raised ValueError out of
`float(raw)` during tool construction rather than falling back.
- `timeout_s: off` -- YAML parses that as `False`, and `float(False)` is
`0.0`, so every request timed out immediately against a healthy server.
`_coerce_timeout` now mirrors the sibling providers: booleans and unparsable
strings fall back to the default, with a warning for the string case.
Tests: five cases in tests/test_browserless_client.py covering both keys, the
precedence order, and both coercion bugs. Verified red before the fix (3 of 5
fail) and green after.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Rebase the feedback migration onto main's new chain tip again: main added
0009_webhook_dedupe (also chained after 0008_thread_operation_kind), so
0009_feedback_tags becomes 0010_feedback_tags with
down_revision=0009_webhook_dedupe, keeping the chain linear.
Conflicts resolved:
- Five head-pin tests take main's version with the pin bumped to
0010_feedback_tags.
- test_thread_messages_page.py keeps this branch's feedback_service
wiring over main's feedback_repo wiring, but stubs both service
methods so the thread-grouped path main added coverage for stays
stubbed (latest_per_run_in_thread alongside latest_for_runs).
- test_thread_messages_feedback.py keeps both sides' imports; each is
used (Feedback for the fixture, EditReplayVisibility for the run
manager stub).
Verified: 79 tests across the conflicted files plus the migration and
bootstrap suites, and 54 feedback tests, all pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(memory): integrate FTS5 retrieval adapter
* deps: add jieba as default dependency for Chinese tokenization
Without jieba, FTS5 unicode61 tokenizer treats entire Chinese sentences
as single tokens, making single-character or sub-phrase searches
impossible (e.g. '吃' or '油泼面' returns 0 hits against
'用户喜欢吃油泼面'). jieba segments Chinese text into meaningful tokens
before indexing.
* fix(memory): avoid treating hyphens as FTS5 operators
* feat(memory): make Chinese tokenization optional
* fix(memory): warm every requested retrieval scope
* fix(memory): close retrieval resources on shutdown
* fix(memory): close backend when shutdown flush fails
* fix(memory): recreate corrupt retrieval index
* fix(memory): tolerate partial retrieval rebuilds
* fix(memory): warm retrieval index in background
* fix(memory): preserve shutdown flush budget
* fix(memory): stop retrying partial lazy rebuilds
* fix(memory): close retrieval through storage
* refactor(memory): simplify retrieval scope limit
* docs(memory): clarify retrieval shutdown lifecycle
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): share inbound webhook dedupe across pods via Postgres (#4120)
* ci: run cross-pod inbound dedupe integration tests in CI
Expose the job Postgres service via DEDUPE_TEST_POSTGRES_URL so the integration tests (issue #4120) actually execute instead of silently skipping. Normalize the URL for asyncpg (postgresql:// -> +asyncpg, drop libpq-only sslmode) and await the now-async _is_duplicate_inbound in test_github_dispatcher.
* feat(sandbox): add Tenki cloud sandbox provider
Adds deerflow.community.tenki, a SandboxProvider backed by Tenki cloud
microVMs, alongside the existing e2b_sandbox / boxlite / aio_sandbox
backends. Selected via `sandbox.use: deerflow.community.tenki:TenkiSandboxProvider`
(resolved by class path, so the change is purely additive).
The full Sandbox contract is implemented — execute_command plus
read/write/update/download_file and list_dir/glob/grep — with file ops run
as busybox-portable shell commands (cat / find / grep / chunked base64),
reusing deerflow.sandbox.search, mirroring e2b_sandbox and boxlite. Tenki's
SDK is synchronous, so unlike boxlite there is no event-loop bridge.
Tenki sandboxes run as an unprivileged user with /mnt root-owned, so the
/mnt/user-data virtual prefix is remapped under the writable home dir (like
e2b_sandbox); the provider also best-effort sudo-symlinks /mnt/user-data to
that home dir so agent shell commands using the literal path still work.
Sandboxes are pooled per (user, thread) with warm reclaim, a replica cap,
and an idle reaper via the shared WarmPoolLifecycleMixin. Transient
transport blips get one bounded retry; terminal session errors evict and
recreate.
Only the stable Tenki surface is used (create/terminate + exec/shell/fs) —
no volumes, snapshots, or template builds — so any stock base image works.
The tenki-sandbox SDK is an optional extra (deerflow-harness[tenki]) and is
imported lazily, so a default install and every other provider are
unaffected.
Tested: unit suite runs in CI without tenki-sandbox installed; a live
integration test and full-surface e2e were verified against real Tenki
sandboxes.
* fix(sandbox): remove unsafe auto-retry from Tenki exec
Pre-merge review caught that the transient-transport retry sat at the
universal _exec layer, so it retried every operation — execute_command and
base64 file-write chunks included. gRPC has no exactly-once guarantee: a
"socket closed" ack-drop after the server already ran the op means the retry
runs it twice, double-firing command side effects and duplicating a write
chunk mid-file (silent binary corruption on multi-chunk writes).
exec is not idempotent, so it must not be auto-retried. Reverts to the
boxlite/e2b behavior: a transient error surfaces to the caller (returned as
text by execute_command, raised by the file ops); a terminal session error
still evicts the sandbox so the next acquire rebuilds it. Verified live
end-to-end across 31 edge cases (empty/binary/unicode/chunk-boundary files,
shell-metachar content, error paths, list/glob/grep, warm-pool reclaim,
concurrency).
* fix(sandbox): address Tenki provider review feedback
- Use Tenki's native sandbox.fs API for all file transport (read_text,
read_bytes, write_stream, mkdir) instead of cat/chunked-base64 over shell.
Uploads stream in 1 MiB frames; append is read-modify-write because the
write stream has no append mode (same approach as community/e2b_sandbox).
- download_file streams via fs.read_stream and enforces the 100 MB cap on
bytes actually received, closing the TOCTOU window between the old
wc -c size probe and the read.
- list_dir/glob/grep report paths back under /mnt/user-data instead of the
sandbox-internal home dir, so results feed straight into the file APIs.
- Create with wait=False and await wait_ready() here: create(wait=True)
raises with the session handle still inside the SDK, leaking a running
microVM this provider could never terminate.
- Configure the sandbox lifetime (max_duration, default 4h) and expose
sticky; without it Tenki reaps a reused thread's sandbox after ~30 min.
- close() terminates before marking the adapter closed and re-raises real
failures, so a failed termination stays retryable instead of silently
leaking a billed microVM; an already-gone session still counts as closed.
- Bump the optional extra to tenki-sandbox>=0.4.0 and commit backend/uv.lock.
* fix(sandbox): scope tenki grep() glob filter to its directory prefix
Mirrors #4168, which fixed the same defect in the E2B provider. The tenki
adapter reduced a directory-scoped pattern like "src/*.js" to its basename
before filtering, so the search silently broadened to every matching-extension
file in the tree. Post-filter grep's hits through path_matches() against the
path relative to the search root, the same way glob() already does, so both
agree on what a directory-scoped pattern means.
* fix(sandbox): address Tenki provider review — eviction, id width, write lock, grep -H
Four fixes from the upstream review:
download_file no longer swallows terminal transport errors. The broad
`except OSError: raise` re-raised ConnectionError/BrokenPipeError/EOFError
(all OSError subclasses that _is_terminal_failure treats as terminal) before
_note_failure ran, so a session that died mid-download was never evicted. Only
our own EFBIG size-cap now passes through without eviction.
Sandbox id widened from 32 to 64 bits (`[:8]` to `[:16]`), matching
community/e2b_sandbox. The warm pool is keyed by this id with no full-seed
fallback, so a collision could let one user reclaim another's parked sandbox on
a multi-tenant gateway.
_fs_op now holds the lock across the op, not just the fs lookup, so concurrent
calls on the same sandbox serialise over the SDK's shared connection. The
eviction callback runs after the lock is released to avoid a lock-order
deadlock with the provider. The append read-modify-write is serialised by a
dedicated _write_lock so two concurrent appends can't clobber each other.
grep passes -H so a search whose path resolves to a single file still prints
the filename; without it the file:line:text unpack dropped every match.
* fix(sandbox): address Tenki provider review round 2
- validate config `environment` at load time (_validate_extra_env) so a bad
key fails fast instead of surfacing as an SDK error mid-command
- document the deliberate lock decision in download_file: the instance lock is
dropped before streaming so a 100 MB download can't block every other tool;
terminal transport errors still evict via _note_failure
- tighten the terminal-error comment to note ConnectionError/BrokenPipeError/
EOFError are also treated terminal via isinstance
- document TenkiSandboxProvider in backend/AGENTS.md (provider detail, warm-pool
destroy hook, community provider list)
- add a commented Tenki block to config.example.yaml for parity with AIO/BoxLite
- tests: config env validation, grep -F/case-sensitive flags, glob include_dirs,
list_dir max_depth, bootstrap-failure warning branch
* fix(sandbox): make Tenki bootstrap non-interactive and time-bounded
The create-time bootstrap runs under the per-scope acquire lock, so a hang
would stall acquire for that scope indefinitely:
- use `sudo -n` so a password-requiring sudoers entry fails fast (swallowed by
the existing `|| true`) instead of blocking on a tty password prompt
- pass a timeout to the bootstrap `remote.exec` so any other stall drops to the
existing warning path rather than wedging acquire
Best-effort by design; the file APIs still work via the home remap on failure.
* test(sandbox): pin Tenki bootstrap timeout to its actual value
Assert bootstrap["timeout"] == _BOOTSTRAP_TIMEOUT instead of `is not None`, so
a regression to timeout=0 (treated as no timeout by some SDKs) or an unrelated
value is caught rather than passing a weaker non-None check.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(clarification): structured form fields for human-input cards
Add a request-side v2 `form` mode to the ask_clarification protocol so
business flows (e.g. expense reimbursement) can collect several values
in one card instead of sequential free-text questions:
- `ask_clarification` gains a restricted `fields` parameter (text /
textarea / number / select / multi_select / checkbox / date)
- ClarificationMiddleware validates and normalizes fields explicitly
(whitelisted types, unknown -> text, select-likes without options ->
text, duplicate/invalid entries dropped, all-invalid falls back to
the legacy modes) since the middleware short-circuits before tool
execution; the plain-text fallback lists fields for IM channels
- Form payloads carry `version: 2` so older frontends degrade to the
text fallback; replies stay on the v1 response protocol — the card
submits a readable summary as `response_kind: "text"`, so journal
persistence and answered-card recovery are unchanged
- Frontend renders typed field controls with required-field validation
and compact multi-select chips
Part of #4400 (scope narrowed per maintainer feedback: request-side
only, no new response kinds, no top-level multi_choice).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(clarification): harden form protocol per review feedback
Address the five review points on #4406:
- Reject field names colliding with JS Object.prototype members on both
sides; frontend reads form values via own-property access only, so
`constructor`/`toString`-style names can no longer leak inherited
members into required validation or the submitted summary
- Close open requests answered through the legacy text fallback: a
visible plain human reply (no response metadata) now marks every
previously-opened request as answered, so upgrading to a v2-aware
frontend cannot leave the composer locked on an already-answered card
- Give checkbox fields deterministic boolean semantics: values are
seeded to an explicit false ("no" in the summary) and `required` means
must-agree/consent; documented in the tool schema
- Make middleware field validation atomic: structurally broken entries
(bad/duplicate/reserved names, over-cap field/option counts or text
lengths) degrade the whole form instead of silently dropping fields;
options are trimmed/deduped with blanks removed so the backend never
emits payloads the frontend parser rejects
- Associate form labels/controls (htmlFor/id), aria-required,
aria-invalid, and error descriptions for accessibility
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(clarification): type the fields item schema via TypedDict
Replace `fields: list[dict[str, Any]]` with `list[ClarificationFormField]`
(a TypedDict with `name` required and the type whitelist as a Literal) so
the provider-facing tool schema documents the item shape instead of an
opaque object relying on the docstring. Runtime validation is unchanged
and stays in ClarificationMiddleware, which intercepts the call before
tool execution. Addresses the non-blocking review suggestion on #4406.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(frontend): drop unsupported aria-invalid from multi-select group
jsx-a11y: role=group does not support aria-invalid; the error linkage
stays via aria-describedby.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(clarification): coerce numeric required flags and normalize fields once
- `_normalize_bool` now coerces 1/0 (some providers serialize booleans
as integers), so `required: 1` no longer silently flips to optional
- `_handle_clarification` normalizes `fields` once and passes the result
to both the text fallback and the payload builder
Addresses the non-blocking review nits on #4406.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(clarification): harden form protocol per contract review round 2
Backend:
- Guard unhashable JSON in the intercept path: `type: []`/`{}` degrades
the field to text and `clarification_type: []` coerces to str instead
of raising TypeError (which, with return_direct, ended the turn with
an error and no card or fallback)
- Add a total budget over the serialized normalized fields (16KB UTF-8
bytes): per-item caps alone admitted forms whose IM text fallback
exceeded channel delivery limits (Slack 40k chars, Feishu ~30KB card),
silently truncating trailing fields; a boundary test proves any
accepted form's fallback stays deliverable
Frontend:
- Submission value now appends a JSON block keyed by stable field names
(readable summary alone is delimiter-ambiguous), with a collision
regression test
- Parser boundary tightened to match backend constraints: empty option
values (Radix SelectItem crash), duplicate option ids/values,
duplicate field names, and the form<->version-2 binding are rejected
- Keep the error node mounted while any field is still invalid so
aria-describedby never points at a removed element (happy-dom
interaction test)
- Required semantics are now accessible: native checkbox control (no
HTML required attribute — it would intercept the custom submit path),
visually-hidden localized "required" markers next to the aria-hidden
asterisks
- Legacy-fallback closure narrowed to the latest unanswered request:
nothing guarantees a single outstanding clarification across runs, and
closing all would silently swallow older decisions; an older request
left open becomes the active card again
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(frontend): keep clarification selects controlled
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(checkpoint): production-shaped full/delta benchmark with configurable snapshot frequency
- Group benchmark scripts into per-family folders (checkpoint/, sandbox/)
- Extract shared benchmark infrastructure into checkpoint_bench_common.py
- Add checkpoint_delta_snapshot_frequency config (default 1000, process-frozen);
freeze it in make_lead_agent and DeerFlowClient; key the state-schema
adaptation cache by resolved frequency
- New bench_production.py: per-case child processes run N ainvoke turns through
the real lead-agent graph (scripted deterministic model, real AsyncSqliteSaver),
then measure GET /state + POST /history through the real Gateway route stack
in one event loop (httpx ASGITransport), cold/warm accessor-cache split,
cross-mode digest gates
- New summarize_production.py: delta/full ratios plus decision metrics
(snapshot_write_spike, cache_effect_ms, checkpoint_write_share,
auto-discovered history per-limit ratios)
* fix(checkpoint): address production benchmark review
Four findings from @willem-bd on #4401:
- FeedbackDialog stays mounted across messages, so selected tags and the
comment survived an ESC/click-outside dismiss and pre-filled the next
thumbs-down. Reset on every close path via a wrapped onOpenChange.
- Neither the dialog's handleSubmit nor handleDialogSubmit caught a failed
enrichment PUT, so a rejection went unhandled and the user got no signal.
Catch in the dialog (where the rejection lands), toast, and keep the input
for a retry.
- rate_run awaited the RunLookup port before Feedback.create validated the
rating, contradicting its own "before any I/O happens" docstring: an
invalid rating on an unknown run surfaced as RunNotFoundError. Validate
first, restoring the legacy router's 400-before-404 ordering. The service
test now uses an unknown run id so it actually pins that order.
- InMemoryFeedbackRepository moved out of test_feedback.py into
tests/feedback_fakes.py (with FakeRunLookup) so two test modules share it
without one importing the other; conftest states the tests-dir sys.path
dependency explicitly, which also makes it work under
--import-mode=importlib.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rebase the feedback migration onto main's new chain tip: main added
0008_thread_operation_kind (also chained after 0007), so
0008_feedback_tags becomes 0009_feedback_tags with
down_revision=0008_thread_operation_kind, keeping the chain linear.
The five head-pin test conflicts resolve to 0009_feedback_tags on top
of main's versions (preserving the new operation_kind assertions).
Verified: 42 migration/bootstrap tests, 54 feedback tests, and the
full frontend suite (797) pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
