mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-15 07:41:20 +00:00
15 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Totoro
|
2dd2e5415c
|
feat(scheduled-tasks): finish the user-facing side of conversation schedules (RFC #6340, 1/2) (#6378)
* feat(scheduler): lifecycle rules, goal/limit parity and coded errors for scheduled tasks
- Migration 0031: nullable scheduled_tasks.unmet_streak_after_seq (internal
streak boundary) and stop_condition (the user's stop rule, own column).
- Stop rule is normalized (stop_rule.py) and appended only at launch; the
stored prompt never contains it. Wording adapts when scheduler.tool_enabled
is off.
- Willem 1: page-created tasks get goal evaluation, <standing_notes>, the
previous-run reference and own-schedule stop; request_stop and the
scheduled capability no longer require an origin conversation.
- Repository: single reactivation choke point raising
ScheduledTaskLimitsExhausted; streak boundary moves on goal/prompt/stop
condition/note changes (not on resume); goal-check failures neither count
nor reset the streak; pause of a finished task returns "finished"; trials
on a paused task keep the host pause marker.
- New repository reads: automatic_runs_used_for, active_run_status_for,
list_manageable_from_thread, thread_relation/thread_run on thread tasks,
task_ids_for_thread, run_number, and run_number/total_tokens/summary on
run history rows.
- REST: goal/max_runs/end_at/stop_condition on create and PATCH (null
clears, naive end_at = task timezone), resume recomputes from now with an
optional renewal body, once_time_passed, task_finished, limits_exhausted,
scheduler_not_running on create, trigger returns outcome/existing/thread_id,
task responses add automatic_runs_used and active_run_status.
- Coded errors {code, message, params} for routes, shared validation and the
chat capability (scheduled_task_errors.py), pinned by
contracts/scheduled_task_errors_contract.json; goal notes contract v2 with
check_failure_codes, scheduled_origin_key and host_run_errors (host_notes.py).
- /api/features adds scheduled_tasks {available, running, tool_enabled,
min_interval_seconds}; ScheduledTaskService.is_running.
- Docs: six README scheduler sections, API.md scheduled tasks section
(breaking error shape), CONFIGURATION.md, AGENTS notes, CHANGELOGs.
- Tests: migration 0031, stop rule, error contract, run listing, lifecycle
over HTTP, repository, service and capability cases; updated tests for the
new error shape, finished-task pause and head revision.
* feat(scheduler): conversation update/resume, run-conversation authority and readable launches
- schedule_task: update and resume actions, stop_condition and clear_fields
parameters, plain parameter types (value rules come back as coded results),
rewritten docstrings for schedule_task and stop_scheduled_task.
- Capability: update/resume share the REST validation and error codes;
interactive turns in a task's run conversation may manage that task;
scheduled runs keep only own-stop; grants require a running scheduler.
- Results: compact JSON-safe task views with local times, display card/text,
last run in list, timezone_source/browser_timezone, now_local; no ISO, user,
lease or origin fields; stop_instructions removed.
- Timezone: explicit > browser (context.client_timezone, validated in
start_run and never forwarded to the run config) > not needed for intervals
and offset run_at > timezone_required.
- Trial gate accepts natural direct requests after acknowledgements and
particles; trial results report thread_id and existing.
- Scheduled launches: stable message id scheduled-<task_run_id>, server-owned
deerflow_scheduled_origin metadata with the user-language parts, run thread
title for fresh-thread tasks.
- Worker records goal continuations in the verdict; lead prompt language rule
covers stored task text and scheduled runs.
- Docs: six READMEs, CONFIGURATION, API, runtime/tools AGENTS, CHANGELOGs.
* feat(frontend): scheduled-task core types, formatting, errors and glossary
- throwGatewayApiError now throws GatewayApiError (status, code, params,
rawMessage); message stays the string detail or detail.message
- scheduled-task types gain stop_condition, automatic_runs_used,
active_run_status, origin/notes, run_number/total_tokens/summary, thread
relation, trigger result, renewal and scheduled origin; isTaskBusy and
hasActiveRun helpers
- api: fetchScheduledTask, resume with optional renewal body, typed trigger
result and update payload with null clears
- hooks: polling at 3 s while a run is active (list, thread list, run
history), useScheduledTask with live gating and final 404, useInView,
localized error toasts; run history refreshes tasks when a run finishes
- new pure modules: errors.ts (contract code to localized copy), format.ts
(relative task times, viewer time, display zone), status.ts, actions.ts,
run-error.ts; describeTaskSchedule in cron.ts; goal-outcome adds unchecked
goal checks, describeTaskOutcome and parseStopRunId
- features: scheduled_tasks block with defaults when missing
- buildRunContext sends client_timezone
- titleOfThread takes a localized untitled label; breadcrumb names scheduled
tasks and agents; reference chips use the localized untitled label
- schedule input: shared browserTimeZone, labelled selects, aria-pressed
type toggles, interval minimum from the hint template
- i18n: all PR1 scheduled-task keys in en-US and zh-CN, glossary fixes,
recipe instructions moved into i18n
- e2e mocks follow the new task, trigger, resume, thread-task and features
shapes; expectNoRawIdentifiers helpers for e2e and DOM tests
- tests for errors, format, cron, status, actions, goal outcome, run errors,
hooks, run context, thread titles and features; scheduled e2e copy updated
* feat(frontend): list-first scheduled tasks page with outcomes and editable goals
- page.tsx is now a thin composition of components/workspace/scheduled-tasks:
status tabs (All/Active/Paused/Finished) and search over the task list,
?task_id= deep link with router.replace on selection, ?thread_id= chip
with Show all, empty state, scheduler-off notice and unavailable state
- task list rows show the schedule in words and one status line (next run,
running now, paused by agent, auto-paused, finished by its cap)
- task detail: status badge, actions from availableActions with reasons,
More actions menu (Duplicate, Copy task ID, Delete), outcome notice,
Runs / Stops when / Each run's goal / Does / Notes from chat / History,
origin chat link; success toasts for every action
- run rows: run number or Trial run, summary, goal and assumption badges,
continuations, tokens, Open chat link; raw errors and the evaluator
reason only behind Details
- create/edit/duplicate dialog with goal, stop condition, safety caps and
advanced agent/context options; PATCH sends only changed fields, null
clears a cap
- renew dialog raises or removes caps and resumes in one request; a
limits_exhausted Resume opens it
- removed i18n keys the old page used; added feedback.resumedNoTime
- recorded en-3 and zh-1 page fixtures; scheduled e2e specs rewritten for
the new page; DOM tests for the detail and the form dialog
- CHANGELOG entries and the frontend AGENTS.md scheduled-task note
* feat(frontend): live schedule card, run conversation provenance and header entry
- schedule_task results (create, update, pause, resume, trial, delete) become
an assistant:scheduled-task message group rendered as a live card: schedule,
next run, stop condition with the safety cap, goal, where results go, Run
once now, Pause/Resume and Open task; the card polls the task only while it
is on screen and shows deleted, scheduler-off and limits-exhausted states
- one card per task per assistant turn; the card group is not a turn boundary
for usage, branching, editing, outline or skill attribution
- a human message with deerflow_scheduled_origin stays a human group with
scheduledOrigin and renders as a "Scheduled run / Trial run" block with the
task instructions, stop condition and notes collapsed; the launched text is
never shown and the message is not editable
- readable step labels for schedule_task actions and stop_scheduled_task
- ThreadScheduledTasksButton replaces the header link: hidden without tasks,
shows the count, links to the task (or the chat filter), and a run's chat
links to its task
- recorded en-3 / zh-1 chat and run-thread fixtures; chat-card and run-thread
e2e specs; unit and DOM tests for parsing, grouping, turn boundaries, card,
run block, header button, step labels and reconnect hydration
- i18n: header.countLabelOne added, unused card.next removed; CHANGELOG and
frontend AGENTS.md notes
* fix(scheduler): address verification findings
Move the scheduled-task lifecycle guidance from backend/AGENTS.md into the persistence guide and leave a one-line pointer, so the effective AGENTS.md chains for agents/middlewares, sandbox and subagents stay under the 98304-byte hard limit enforced by scripts/check_agent_guidance.py in lint-check.
* fix(scheduler): address review findings
* fix(scheduled-tasks): polish from live acceptance
- zh schedule wording: an interval of one unit reads 每分钟 / 每小时 / 每秒,
and a saved interval of whole days reads 每天 / 每 2 天 (Every day / Every 2
days) instead of 每 1 分钟 or 每 24 小时; several units keep the number
- zh spacing: 已于{time} 满足 and 在{time} 的运行中 now leave a space between
the interpolated time and the Chinese after it; a copy test guards every
scheduled-task zh string against a placeholder glued to Chinese
- run summary: when the first line of the final reply only leads in to a list
(ends with ":" or "："), the list items follow it as plain text (bullets,
numbering, checkboxes, bold, code and links stripped), joined with "；" for
CJK text and "; " otherwise, within the 160-character cap
- a schedule_task result (create, update, pause, resume, trial, delete) that
arrives in a chat invalidates that chat's task list, the tasks list and the
named task, so the header button and cards update without waiting for the
idle poll; results already in the loaded history do not refetch
- e2e and unit fixtures are now the live acceptance recordings (DeepSeek
Flash, demo data, fictional names): live-weekday.json (create, trial, edit,
pause, resume in one zh chat) and live-minute.json (per-minute chat, run 1
and run 2, paused by the agent), fetched with GET from the demo Gateway and
trimmed; the hand-shaped en-3 / zh-1 fixtures are removed and the chat-card,
run-thread and tasks-page specs run in both locales over the recorded data
* docs(changelog): reference #6378 for the scheduled-tasks entries
* docs(agents): shorten the scheduled-task pointer to stay under the guidance limit
* test(scheduled-tasks): pin the live takes to their recorded time zone
The live fixtures were recorded in Asia/Shanghai, and an interval task
reads in the viewer's zone, so the 16:48 assertions failed on UTC CI
runners. Pin the browser zone in the two live-take e2e specs and the
zh live-take unit test.
* fix(scheduled-tasks): keep untouched end times and drop a chat filter the new task is not in
- The edit, duplicate and renew dialogs keep a stored end time's instant
while its field still shows it. A minute-precision wall value cannot
keep seconds or name the later 01:30 of a DST fall-back, so a
title-only edit moved the cap an hour earlier, and raising the run
limit re-sent the end time without its seconds.
- Creating or duplicating a task from a chat's task view drops the chat
filter when the new task does not belong to that chat, instead of
showing "This task isn't one of this chat's tasks".
* fix(scheduled-tasks): tie pause notices to the pause they describe
- Paused by agent links and times only the run named in last_error. When
that run is not loaded, the notice shows no link and the task's own time
instead of an earlier run that also asked to stop.
- The outcome notice reads the latest history page whichever page is
shown, so browsing older runs no longer swaps the auto-pause reason and
"Open latest run" to an earlier miss. It shares that page's request.
* fix(scheduled-tasks): three lifecycle gaps found while re-checking review classes
- The edit form compares a cron in the form its schedule input emits, so an
untouched "0 09 * * 0,6" is not re-sent as "0 9 * * 6,0" and a title-only
edit no longer re-arms or refuses a finished task.
- Chat resume asks for a timezone, as update does, when a naive end_at would
land in the UTC placeholder of a zone-free task.
- Re-arming a task clears a host pause marker, as resume does, so a later
manual pause no longer reads as "Paused by agent".
* fix(scheduled-tasks): keep the notice's latest-runs query from refreshing older pages
useLatestScheduledTaskRuns kept React Query's default refetch on mount,
focus and reconnect, so going offline and back while an older history page
was shown fetched the latest page again. Refreshing stays with the history,
which refreshes the latest page only while it is shown.
* test(scheduled-tasks): let the viewer's time carry its own day label
Between 16:00 and 24:00 UTC, UTC and Asia/Shanghai are on different days,
so the run prompt shows "Tomorrow 01:00 · Today 09:00 your time" and the
fixed "01:00 · 09:00 your time" assertion failed depending on the hour.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Totoro
|
35023bb908
|
feat(scheduler): create conversation schedules and support agent self-stop (#6229)
* feat(scheduler): create conversation schedules and support agent self-stop * fix(scheduler): validate goals and trial requests before dispatch --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> |
||
|
xiaodu55
|
c368a9f229
|
fix(backend): stop forcing WindowsSelectorEventLoopPolicy so asyncio subprocesses work on Windows (#5937)
* fix(backend): stop forcing WindowsSelectorEventLoopPolicy on Windows backend/sitecustomize.py switched every Python process with backend/ on sys.path to WindowsSelectorEventLoopPolicy (added by #3236 for 'make dev' on Windows). On Windows the selector loop cannot spawn asyncio subprocesses - asyncio.create_subprocess_exec raises a bare NotImplementedError - so ACP agents, stdio MCP servers, and local sandbox command execution are all broken on native Windows, and the swallowed exception surfaces as 'Error invoking ACP agent x: ' with no reason. The forcing has no beneficiary left on Windows: the async postgres driver is asyncpg (not psycopg, as #3236 states), and asyncpg 0.31.0 ships no Windows wheels, so the postgres backend is unavailable on native Windows regardless of the loop; the default SQLite backend (aiosqlite) runs on either loop. The make dev startup fix in #3236 came from the serve.sh / wait-for-port.sh changes, which are untouched. With the Python-default ProactorEventLoop restored, subprocess features work again on native Windows. * fix(benchmark): scope a selector loop to the postgres checkpoint benchmark on Windows Static review of the sitecustomize removal flagged the one caller that relied on the old startup policy: bench_channels.py postgres workers run bare asyncio.run() and enter AsyncPostgresSaver.from_conn_string(), which the lockfile-pinned psycopg 3.3.3 rejects on Windows under ProactorEventLoop before connecting. Instead of restoring the global selector policy, the postgres branch of the benchmark now scopes WindowsSelectorEventLoopPolicy to itself right before asyncio.run(); memory/sqlite cases and every other backend process keep the subprocess-capable Proactor default. |
||
|
PeaceMaker-best
|
e4ce26f3d2
|
feat(memory): add DeerMem scope-isolation benchmark (#5564)
* feat(memory): add DeerMem scope-isolation benchmark Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> * fix(benchmark): grade persisted summaries and retry failed extractions --------- Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Wenchao An
|
6d5d7bb1d5
|
feat(subagents): add opt-in parent context snapshots (#5367)
* feat(subagents): add opt-in parent context snapshots * test(subagents): package synthetic snapshot evaluation * fix(subagents): preserve output text and defer snapshot capture * docs(subagents): keep snapshot guidance within chain budget * fix(subagents): omit unpaired tool calls from snapshots * fix(subagents): safely omit unserializable snapshot media |
||
|
Ryker_Feng
|
f52818fe5e
|
feat(skills): export custom skill packages with revision-bound preview (#5332)
* feat(skills): export custom skill packages with revision preview * docs(gateway): keep export guidance within size budget * ci: retry checks after transient uv setup download failure * docs: focus skill export agent guidance on maintenance invariants * fix(skills): handle export disconnects and bound archive transfers * docs(gateway): remove redundant export guidance to fit merged budget * fix(skills): reset export idle deadline after transfer progress |
||
|
zeng-bohan
|
a2808e8292
|
test(checkpoint): retention deletion contract + growth baseline (#4189 item 3) (#5255)
* test(checkpoint): retention deletion contract + growth baseline Six contract scenarios x memory/sqlite/postgres pin what retention deletions must never break (branch ancestors, explicit resume targets, pending writes, duration-only chain links), prove the two safe shapes (leaf sibling branches, trailing duration leaves), record the full-vs-delta growth baseline in the normalized bench shape, and add an item 4 probe showing the default ToolOutputBudgetMiddleware already externalizes oversized tool results. Refs #4189 * test(checkpoint): make the retention contract load-bearing per review Review findings from willem-bd and Ricky-7-Yan: - scenario D pins its own row: before/after stats delta plus a serde round-trip of the stored write, instead of an always-true > 0 check - _delete_checkpoint now performs the joint delete the doc mandates (checkpoint row + writes rows + blobs unreachable from surviving checkpoints), so E1/E2 exercise the shape they prescribe - E1 builds the real runtime duration shape via persist_run_durations (parent dict clone, fresh id/ts, real metadata), which surfaces the shared-version case: the leaf's blobs are the surviving parent's rows - contract doc: blob reachability must be computed from surviving checkpoints in a whole-thread pass; shared-version/duration-only hazard called out explicitly; memory data model includes saver.blobs - _stats counts memory blob rows and returns the full normalized shape (logical byte totals included) - probe: drops the unused middleware/outputs_dir graph parameters and discloses the manual-harness scope limit in the module docstring - E1/E2 assert default head resolution (protected set item 5); unused graph_for helper and DURATION_ONLY_METADATA stand-in removed Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com> * fix(checkpoint): scope probe cleanup to owned dirs, key report by backend Second-round review findings on #5255: - [P1] bench_tool_result_probe.py removed the whole user-supplied --outputs-dir (and the shared .probe-tmp) in its finally block, so pre-existing files were deleted on success and failure alike. The run now writes into (and removes) a fresh owned probe-run-* child beneath the requested directory, and SQLite databases live in a unique mkdtemp'd temp directory that is removed with the run. Regression tests pin that unrelated pre-existing files survive both a successful and a simulated failing run. - [P2] the optional retention report keyed every backend's measurements under one shared name, so a multi-backend invocation kept only the last backend's numbers. _report() now takes the parameterized backend explicitly (saver_env.kind); regression pins that memory and sqlite entries coexist in one report file. Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com> --------- Signed-off-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com> Co-authored-by: zengbohan1 <310902929+zengbohan1@users.noreply.github.com> |
||
|
Otavio Rodrigues Santana
|
27cb73659d
|
fix(auth): correct OAuth conflict error message + validate multi-worker Postgres claim with real concurrency benchmark (#5026)
* fix(auth): correct OAuth uniqueness error and index parity on Postgres
create_user() caught any IntegrityError on commit and always reported it
as a duplicate email. The email pre-check already rules out a real email
collision in the common case, so any IntegrityError reaching that handler
is actually idx_users_oauth_identity firing instead -- confirmed against
both backends: SQLite reports "UNIQUE constraint failed:
users.oauth_provider, users.oauth_id", Postgres reports a
UniqueViolationError naming the same index. The caller saw "Email already
registered" for an OAuth account conflict, which is wrong and would send
API consumers debugging the wrong field.
Distinguish the two cases via a substring check on the driver error text
(both backends name the oauth columns) and raise an accurate message for
each.
Also add postgresql_where to the same index, alongside the existing
sqlite_where. This is not a correctness fix -- verified empirically that
Postgres already enforces the same practical uniqueness without it
(NULL is never equal to NULL in either backends unique index, so real
duplicate (provider, id) pairs are already rejected and NULL/NULL rows
are already unconstrained). postgresql_where makes the index genuinely
partial on Postgres too, matching the stated intent in the surrounding
comment and keeping the index smaller as the common case (plain-password
accounts, both columns NULL) accumulates.
* test(bench): add multi-process SQLite vs Postgres concurrency benchmark
CONFIGURATION.md documents that multi-worker deployments must use Postgres
because "SQLite silently ignores row-level locks", but nothing in the repo
exercised that claim against real separate worker processes -- the existing
checkpoint benchmarks (scripts/benchmark/checkpoint/) measure single-process
read/write latency, and the existing Postgres tests
(test_pg_schema_integration.py, test_multi_worker_postgres_gate.py) cover
schema placement and config validation, not throughput or lock behavior
under concurrent load.
run_concurrency_bench.py spawns N real OS processes (subprocess.Popen, not
asyncio tasks or threads within one process) against the shared users
table, mixing reads (get_user_by_email) and writes (create_user) at a
configurable ratio, and reports throughput, error counts by exception
type, and p50/p95/p99/max latency per run.
Measured locally (2/4/8/16 workers, 100 ops/worker, 70/30 read/write):
SQLite completed all operations with zero errors at every worker count
(busy_timeout absorbs contention rather than raising), but total
throughput stayed flat around 28-34 ops/s regardless of worker count, and
p99 latency grew from ~400ms at 2 workers to ~5.9s at 16, with a 22s max.
Postgres throughput scaled with worker count (41 to 66 ops/s) and p99
stayed under 500ms at every worker count tested. Raw JSON output from
both runs is available on request; exact numbers will vary by machine and
are not asserted in the test suite.
test_bench_concurrency.py unit-tests the pure aggregation logic
(percentile math, error grouping, crashed-worker handling) the same way
test_bench_checkpoint_channels.py does for the existing benchmarks --
fast, no DB required, not the full multi-process sweep in CI.
* fix(auth): inspect the driver exception for OAuth conflict detection
str(exc) embeds the full failed INSERT statement, whose column list
always names oauth_provider/oauth_id, so a substring check on it
misclassified every commit-time IntegrityError on the users table as
an OAuth conflict (reproduced on SQLite: a duplicate primary key with
a different email raised "OAuth account already linked: None/None").
_is_oauth_identity_violation now inspects exc.orig instead: constraint_name
on Postgres, both violated column names present (not a bare "oauth"
substring) on SQLite.
Also ships the alembic revision idx_users_oauth_identity's postgresql_where
predicate needed: 0001_baseline created it as a full index on Postgres,
and ORM metadata changes only affect fresh create_all databases, never an
already-versioned deployment.
Addresses review feedback from willem-bd.
* fix(bench): run the concurrency benchmark in an isolated schema and derive paths from the checkout
--pg-url accepted an arbitrary database URL while the code pinned
postgres_schema="public" and unconditionally ran DELETE FROM users --
against any non-disposable database that permanently destroyed every
auth account. Each run now generates a unique throwaway schema
(bench_<uuid>), points both the seeder and every worker subprocess at
it via postgres_schema, and drops only that schema (DROP SCHEMA ...
CASCADE) once the full worker-count sweep finishes.
Also stopped hard-coding /opt/deer-flow/backend as the checkout path
and .venv/bin/python3 as the interpreter: BACKEND_DIR is now derived
from Path(__file__), and workers are spawned with sys.executable (the
orchestrator's own interpreter) instead, so the documented
uv run python scripts/benchmark/concurrency/run_concurrency_bench.py
command works from any checkout.
Addresses review feedback from willem-bd.
* fix: shorten oauth-index revision id, repin migration-head assertions, fix bench read/write mix
- 0017_users_oauth_identity_partial_pg (36 chars) exceeded
alembic_version.version_num's VARCHAR(32) limit, which would fail
stamping/upgrading on both fresh and existing Postgres deployments.
Renamed to 0017_oauth_identity_pg_partial (30 chars).
- Repinned every test asserting 0016_subagent_batches as the migration
head (test_persistence_bootstrap[.py|_concurrency.py|_regression.py],
test_migration_0004/0007/0015) to the new 0017 revision id.
- worker.py's `(i % 100) < int(read_ratio * 100)` assumed n_ops >= 100;
at the documented default (50 ops/worker, 0.7 read ratio) it produced
either all-reads or all-writes, never the claimed mixed workload.
Replaced with read_count()/is_read_op(), which distribute an exact
round(n_ops * read_ratio) reads evenly across the sequence via modular
spacing, and added test_bench_worker.py covering the default values
plus small op counts.
* fix(bench): establish a real physical connection before timing ops
async with sf(): pass entered an empty AsyncSession without checking out
a physical connection -- SQLAlchemy stays lazy until the first statement
executes. That pushed connection-establishment cost onto each worker's
first timed operation instead of conn_time_s, and at 16 workers those 16
cold first-ops (1% of a 1600-op sample) could skew the reported p99.
Execute a real `SELECT 1` before starting the timer instead.
Verified with a real end-to-end run (uv sync + sqlite backend, 2
workers/10 ops, 0 errors) plus the full auth/bench/migration-bootstrap
suites (135 tests) and ruff check/format, all clean.
* fix(bench): synchronize workers before timing, fix percentile off-by-one
Two remaining measurement issues from review:
- run_workers() started the wall clock before spawning any worker, so
throughput/wall_time absorbed N processes' staggered Python-startup and
connection-establishment cost, and early workers could run ahead of ones
still starting. Workers now print READY right before their timed loop
and block on stdin for a GO signal; the orchestrator waits for every
READY, then starts the timer and releases all workers together.
- summarize()'s pct() used int(len(latencies) * p) directly as a
zero-based index -- a one-based-rank-as-index bug that put p95 and p99
at the same slot (the max) for any 20-or-fewer-sample run, and for the
documented 100-sample default. Now delegates to
checkpoint_bench_common.percentile(), the already-correct nearest-rank
implementation used elsewhere in the same benchmark family, instead of
a second, broken one.
Verified: 14/14 unit tests pass (2 new pinned-value regression tests for
the percentile bug, using the reviewer's own 20-sample repro), ruff
clean, and a real 2/4-worker SQLite multi-process smoke run completes
with distinct p95/p99/max latencies and no hang.
* fix(bench): absolute SQLite bench path, surface crash diagnostics, exit nonzero on failure; share OAuth index constant + cover Postgres branch
Three more findings from review at 5fd25a7:
- seed_baseline() cleaned an absolute .deer-flow/bench_data path but
handed DatabaseConfig a relative one, which resolves against the
CALLER's CWD -- not BACKEND_DIR. Invoking the documented command from
anywhere other than backend/ silently pointed the seeder and the
(cwd=BACKEND_DIR) workers at two different directories: workers crashed
with 'unable to open database file' while the run still printed a
well-formed summary and exited 0. Both seed_baseline() and worker.py's
make_session_factory() now use the same absolute path.
- Crashed workers' stderr was captured then discarded, and main() always
exited 0 -- an all-crashed sweep was indistinguishable from a real
(uneventful) measurement to anything checking the exit code or
--out. run_workers() now prints each crash immediately and tags it with
the real worker_id (previously always None); summarize() exposes
crashed_worker_errors alongside the existing crashed_workers count;
main() exits 1 via the new summary_indicates_failure() whenever any
sweep crashed or fell short of expected_total_ops.
- idx_users_oauth_identity was hardcoded separately in the ORM Index and
in _is_oauth_identity_violation's Postgres branch, with no test to
catch drift, and that branch had zero non-skipped coverage (its only
guard needs a live Postgres CI never configures). Exported
OAUTH_IDENTITY_INDEX_NAME from user/model.py as the shared source of
truth (migrations intentionally keep their own frozen literal, matching
every other revision in that package) and added stub-exception unit
tests pinning both the asyncpg constraint_name path and the sqlite
message-substring path, positive and negative.
Verified: 107 passed locally (auth + bench-unit suites), ruff clean, and
two real reproductions -- invoking run_concurrency_bench.py from a
scratch directory outside backend/ (the reviewer's exact repro) now
completes 8/8 ops with crashed_workers: 0 instead of crashing, and the
new crashed_worker_errors/exit-code logic is exercised directly by the
new unit tests against the real summarize()/summary_indicates_failure().
* fix(auth): attribute create_user IntegrityErrors to the right constraint
Two coupled review findings on the classification helpers:
P3 (fall-through) -- after ruling out the OAuth-identity index, create_user
raised "Email already registered: {email}" for every remaining
IntegrityError, including the duplicate-primary-key case the new
regression test exercises, whose address is not registered. Added
_is_email_violation() so the email message is used only for an actual
users.email collision that raced past the pre-check; anything else (in
practice a duplicate id) now raises a neutral
"User already exists (constraint: <name>)".
P2 (unreachable asyncpg branch) -- exc.orig is not the asyncpg error.
SQLAlchemy's asyncpg dialect re-raises its own DBAPI IntegrityError
(pgcode/sqlstate only) 'from' the real asyncpg error, so constraint_name
lives on exc.orig.__cause__. getattr(exc.orig, "constraint_name", None)
was always None on Postgres; the helpers only worked there by accident,
matching asyncpg's DETAIL line in the message fallback. Added
_driver_constraint_name() which walks orig then orig.__cause__, and the
stub tests now model that real shape (orig wrapper + __cause__) instead of
a constraint_name that no driver puts on orig directly.
Tests: 76 passed. New coverage for the email-race path, both new helpers
on each backend, the neutral fallback message, and the cause-chain walk.
* fix(bench): match app SQLite PRAGMAs in workers; fail a sweep on any op error
Two review follow-ups:
- worker.py opened its SQLite engine with only connect_args timeout=30.
synchronous and foreign_keys are per-connection PRAGMAs, so workers ran
at SQLite's synchronous=FULL / foreign_keys=OFF while a real Gateway
worker runs synchronous=NORMAL (persistence/engine.py::_enable_sqlite_wal)
-- an extra fsync per commit on the measured 30%-write path, overstating
SQLite's cost in the direction that flatters the "use Postgres"
conclusion. Added a connect listener applying the same four PRAGMAs, with
a test asserting synchronous/foreign_keys/journal_mode on a real worker
connection.
- summary_indicates_failure() only looked at crashes and the completed vs
expected op counts, so a sweep where every op completed but raised
(e.g. writes hitting OperationalError) passed as a clean measurement:
completed_ops == expected, 0 crashes. Added an "errors > 0" clause; the
error breakdown stays in the JSON, only the exit code changes. Test added.
test_bench_concurrency.py + test_bench_worker.py green (20), plus a real
2-worker sqlite smoke run (6/6 ops, 0 errors, exit 0).
* fix(auth): match the real email index name; only claim "exists" for uniqueness
Review follow-ups on the classification helpers:
- email is mapped_column(unique=True, index=True), which SQLAlchemy and
0001_baseline realise as a single UNIQUE INDEX (ix_users_email), not a
named UNIQUE constraint. _is_email_violation compared the driver
constraint name against "users_email_key", which Postgres never emits,
so that arm was dead on Postgres (SQLite matched via the message). Fixed
to ix_users_email.
- the residual IntegrityError fallback raised "User already exists" for
every remaining IntegrityError -- a NOT NULL / CHECK / foreign-key
violation is not a "user already exists" condition and is not part of
create_user's ValueError contract. Added _is_uniqueness_violation
(sqlstate 23505, or the SQLite "UNIQUE/PRIMARY KEY constraint failed"
message); only that raises the "already exists" ValueError, everything
else propagates unchanged.
- documented scripts/benchmark/concurrency/ in backend/AGENTS.md alongside
the other benchmark family.
Tests: 78 auth + 20 bench-unit pass, ruff clean. New coverage for
_is_uniqueness_violation on both backends and for a non-uniqueness
IntegrityError propagating out of create_user.
* fix(bench): don't pre-close worker stdin (breaks communicate); require --pg-url for postgres
* fix(bench): ruff format; time throughput on the op phase, not teardown
- lint-backend: ruff format the files touched in this PR.
- Throughput window (P2): the orchestrator sampled its wall clock after
every worker's communicate() returned, so it also covered each worker's
engine.dispose(), result serialization and stdout transfer. Each worker
now times just its operation phase (GO -> last op) and reports
ops_elapsed_s; summarize() uses max(ops_elapsed_s) over the workers -- all
released by the same GO -- as the throughput window (ops_window_s).
- Exercise migration 0018 (P2): test_user_oauth_partial_index.py goes
through bootstrap create_all(), which builds the partial index from ORM
metadata and never runs 0018.upgrade(). New Postgres-gated
test_migration_0018_oauth_identity_pg_partial.py alembic-upgrades to 0017
(full index), then 0018 (asserts the predicate appears), then downgrades
(asserts the full index is restored) and re-upgrades.
* docs(middlewares): tighten SandboxAudit and Clarification entries in AGENTS.md
PR #5134 grew agents/middlewares/AGENTS.md ~1.8 KB, pushing the effective
AGENTS.md chain for that directory over the 96 KiB hard limit once this
branch also documents scripts/benchmark/concurrency/ in backend/AGENTS.md.
Condense the two longest middleware entries (SandboxAuditMiddleware,
ClarificationMiddleware) without dropping any identifier, example, issue
reference, ordering constraint, or documented gap; chain back to ~96.8 KiB.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Ricky-7-Yan
|
c56c7293f8
|
test(checkpoint): measure postgres storage growth (#5051) | ||
|
qingbo1011
|
2eba65449f
|
eval(memory): add a reproducible hybrid eviction evaluation (#4810)
* eval(memory): scaffold reproducible eviction evaluation * refactor(eval): align with benchmark layout * eval(memory): add deterministic QA grading Implement the disclosed deterministic-overlap-v1 grader as a pure offline module. Grading is blind by construction: grade_answer() accepts only the prediction and reference strings, never a policy identity. The undisclosed stopword list is committed as a fixed part of this grader version; yes/no/not are deliberately excluded because negation can be the entire answer. Before freezing, the grader locally reproduced all 90 historical (prediction, grade) pairs disclosed in #4789 with zero mismatches and no post-hoc tuning. validate-contracts now rejects a config whose qa.grader_version does not match the committed grader. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * eval(memory): add environment-configured QA runner Add the exact answer-prompt renderer (retained facts sorted by ID, CURRENT DATE line omitted when absent), an OpenAI-compatible provider adapter configured only through the environment variable names pinned in the config, and a resumable run-qa command that calls both policies with identical versioned settings. Each row persists as its own file on success, so a partial paid run resumes without repeating completed calls; qa_run.json binds an output directory to one config identity. Row files and errors carry predictions and non-secret metadata only -- never questions, references, memory text, credentials, or response headers. All tests are offline via mocked transports; run-qa fails fast before touching the dataset when the provider environment is missing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * eval(memory): add blind QA grading report and paired statistics Add grade-qa: it recomputes the deterministic selector output, rejects any answer row whose kept facts, capacity, or policy disagree with it, grades every prediction through the policy-blind grade_answer(prediction, reference) call, and only then joins grades back through stable row IDs. Published artifacts are qa.rows.jsonl (graded rows with non-secret metadata), qa.summary.json (accuracy by source/scenario/policy; official and synthetic suites never folded together), and qa.stats.json (exact paired McNemar and seeded paired bootstrap difference for the official, synthetic, and overall suites using the pinned statistics parameters). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * eval(memory): pin the official DeepSeek model ID The historical protocol recorded the answer model with an aggregator-style namespace (deepseek/deepseek-v4-flash). The live run calls the same underlying model (DeepSeek-V4-Flash-0731, released before the historical run) directly through DeepSeek's official OpenAI-compatible API, whose canonical ID is deepseek-v4-flash. The served model is recorded from the provider response in every answer row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * eval(memory): publish paired eviction QA results Publish the equal-budget live QA artifacts for pr4789-reproduction-v1: provenance, 90 graded rows, per-scenario summary, and paired statistics. At capacity 7 with identical settings, confidence answers 24/45 and hybrid-v1 40/45 (official 23/40 vs 35/40, exact McNemar p=0.0042; overall p=0.0004). The noisy-signal control is the one scenario where hybrid-v1 scored below the baseline (10/10 vs 8/10) and is reported separately. The offline suite now verifies the published statistics are recomputable from the published rows and that the artifacts carry no dataset text or credentials. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(eval): decompose the noisy-signal QA cell Both policies retained the support fact in all ten noisy-signal cases, so the two rows hybrid-v1 lost are grader phrasing boundaries (verbose numeric answers rejected by the numeric-conflict rule), not eviction failures. Documented from the published rows; the grader stays frozen. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * eval(memory): address review hardening findings - ignore the responses/ directory the runner actually writes instead of the stale provider-responses/ entry - cover the official selection-rule recomputation with direct synthetic tests: matching manifests pass, rule-breaking IDs and missing eligible rows fail, and every published exclusion is load-bearing - align the report docstring and README with the statistics contract: the summary never folds sources; the explicitly labeled overall suite is reported alongside the separate official and synthetic suites - recompute the published bootstrap intervals (not only McNemar) in the published-results test - wire required_policy_version to the production EVICTION_POLICY_HYBRID_V1 constant so validate-contracts rejects policy drift Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * eval(memory): restore historical evidence rendering and harden resume identity Address both blocking findings from the #4789 artifact cross-check. The evidence renderer now emits the historical SESSION {id} AT {date} line instead of the divergent bracket format. The byte representation is protocol-critical: the witness record 35a27287 renders at 697 characters again, stays inside the 700-character distractor-bank bound, and 60d45044 leaves the bank, restoring row-level pool reproduction. Deterministic capacity-7 retention is unchanged at 27/45 vs 45/45. qa_run.json now binds a run directory to the SHA-256 of all five protocol inputs (config, both manifests, answer prompt, dataset) and names the changed artifact when it refuses to resume. Stored rows are reused only when row identity, policy, capacity, kept facts, and the request fingerprint recomputed from the current task all match; the disclosed probe (changed message under the same config) is now a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * eval(memory): republish QA results under the historical protocol Replace the published artifacts with the fresh equal-budget run executed at 497ff3d0 under the restored historical evidence rendering; the earlier run under the divergent rendering is discarded entirely. At capacity 7 with identical settings, confidence answers 24/45 and hybrid-v1 38/45 (official 23/40 vs 33/40, exact McNemar p=0.0129; overall p=0.0013). The confidence control is the one scenario below baseline for hybrid-v1 (8/10 vs 6/10); both lost rows retained the support fact and are grader phrasing/abstention boundaries, documented from the published rows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * eval(memory): adopt the historical fact IDs and prompt serialization Pool facts now carry the historical protocol IDs (gold_{case} for the support fact, d_{case}_{index}_{source} for distractors in bank-draw order), and the rendered STORED MEMORY joins fact blocks with a blank line. Sorting by these IDs reproduces the historical selection tie-break: witness case 41698283 at capacity 7 again keeps the 58bf7951 distractor and evicts 001be529 under both policies. Deterministic capacity-7 retention is unchanged at 27/45 vs 45/45. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * eval(memory): republish QA results under the historical serialization Replace the published artifacts with the fresh equal-budget run executed at 01f99d61 under the historical fact IDs and prompt serialization; earlier runs under divergent serializations are discarded entirely. At capacity 7 with identical settings, confidence answers 24/45 and hybrid-v1 40/45 (official 23/40 vs 35/40, exact McNemar p=0.0018; overall p=0.0001). The single row below baseline (1cea1afa, confidence-control) retained its support fact; the model abstained. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * eval(memory): self-certify the publish path and pin offset coverage grade-qa now verifies (read-only) that the run marker's five protocol artifact hashes match the current inputs, rebuilds every answer task, and rejects any stored row whose request fingerprint does not match the task recomputed from the current protocol — the staleness class that previously required an out-of-band cross-check to detect. Verified end-to-end against the published run: all 90 rows pass and regrade to byte-identical artifacts, while a tampered fingerprint is refused by row ID. The distractor offset derivation and wraparound selection are now pinned by unit tests with hardcoded indices, including a wrapping offset, so a digest-slice or modulus regression can no longer stay green offline. Closes both non-blocking suggestions from the re-review. * fix(bench): bind persisted answer rows to their expected case identity Grading derived the reference case from the stored row's embedded case_id, so reassigning a valid row to another valid case passed every integrity check while silently changing the published grade. The resume path had the same gap: _row_matches_task() never compared case_id, source, or scenario. The recomputed task is now authoritative in both paths: grade_answer_rows() resolves the reference case from the expected PolicyResult and rejects any mismatch in the persisted row_id/case_id/source/scenario, and _row_matches_task() checks the same identity fields so a reassigned row is re-run instead of reused. Regressions tamper each field individually and exercise both paths. * docs(bench): document where to download the pinned LongMemEval file The README named the dataset but never said it lives on Hugging Face or how to fetch the pinned revision, so a reviewer could not run the offline commands. Add the direct download URL, the expected SHA-256, and the mirror and huggingface-cli alternatives; the CLI still never downloads anything. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Vanzeren
|
c8cf1bf2fb
|
feat(checkpoint): checkpoint history cache (#4638)
* feat(checkpoint-cache): delta-mode checkpoint history cache with recursive compose
Read-only, invalidation-free cache for LangGraph delta-channel history
({writes, seed}) at the get_delta_channel_history choke point:
- database.checkpoint_cache config (memory|redis; max_entries 0=disabled;
redis bounded by TTL, Gateway/async only)
- memory LRU backend (copy-on-read, zero-serde hit path) and redis backend
(lazy import, degrades to all-miss on outage)
- CachedHistorySaver: recursive composition from the nearest warm ancestor
(depth budget 8), caching each level; depth-0 cold chains delegate one
inner fast-path walk. Entries keyed by immutable
(db, thread, ns, checkpoint_id, channel) — no invalidation, coherent
across workers
- provider wiring: wraps in delta mode only (async + sync), full mode
untouched; sync path is memory-only
- bench opt-in: DEERFLOW_CHECKPOINT_BENCH_HISTORY_CACHE=1
sqlite bench (500 updates, payload 2KB): write phase 2.28x at f=250,
1.32x at f=10; one delegated walk per thread cold start.
* chore(config): bump config_version to 32 for database.checkpoint_cache
The checkpoint history cache feature added the database.checkpoint_cache
section to config.example.yaml; bump the schema version so existing
deployments get the outdated-config warning and can run make config-upgrade.
* chore(helm): bump config_version to 32 in chart values and README
* fix(checkpoint-cache): purge thread history entries on delete paths
Addresses review on #4638: delete_thread/prune removed source-of-truth
checkpoints but left the thread's materialized history payloads in the
cache (memory: until LRU eviction; redis: until TTL, default 1 day) — a
data-lifecycle gap for tenant offboarding / GDPR-style erasure.
- Cache contract gains thread-scoped adelete_thread/delete_thread
(lifecycle purge, not invalidation; entries remain immutable)
- Memory backend: stem scan over the LRU map; redis: SCAN MATCH + UNLINK,
outage degrades to TTL-bounded retention without raising
- CachedHistorySaver purges on delete_thread/adelete_thread and
prune/aprune (prune rewrites chains, so pre-prune histories must go);
delete_for_runs stays delegation-only (run->thread mapping unavailable,
no in-tree callers), documented in code
- ttl_seconds description documents the residual-retention window
- Tests: thread-scoped purge on both backends, saver-level delete/prune
purge, prefix-safety (t1 vs t10), redis outage degradation, and the
pinned no-purge behavior of delete_for_runs
* fix(checkpoint-cache): stable db identity, prefix-aware sync singleton, explicit zero TTL
Addresses Copilot review on #4638:
- checkpoint_cache_db_hash now hashes the credential-free postgres
identity (host:port/database + schema): credential rotation no longer
changes the cache namespace (cold cache + orphaned keys until TTL).
Unparseable URLs fall back to the raw string.
- The sync-path memory cache singleton is also keyed by its key_prefix:
a namespace change (db identity change or operator override) recreates
the cache instead of leaving stale-prefix entries unreachable and
unpurgeable.
- ttl_seconds=0 is now an explicit, documented opt-out of redis expiry
(SET without EX; redis maxmemory policy only) instead of a silent
'ttl_seconds or None' coercion.
Tests: credential-rotation hash stability, unparseable-URL fallback,
prefix-change singleton recreation, same-prefix singleton reuse, and
zero-TTL wire behavior (ex=None).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Vanzeren
|
c48de5e70b
|
feat(checkpoint): make delta snapshot_frequency configurable (#4516)
* feat(checkpoint): make delta snapshot_frequency configurable * fix(config): carry legacy checkpoint_delta_snapshot_frequency with warning Addresses review on #4516: the rename from the flat database.checkpoint_delta_snapshot_frequency key to nested database.checkpoint_delta.snapshot_frequency silently dropped the old value (pydantic extra="ignore"). Add a before-validator that maps the legacy key onto the nested one with a deprecation warning (nested key wins when both are set), plus a CHANGELOG breaking-change note covering the rename and the 1000 -> 10 default change. * fix(checkpoint): validate frozen snapshot frequency |
||
|
Vanzeren
|
e01173d8b2
|
bench(checkpoint): production-shaped full/delta benchmark with configurable snapshot frequency (#4467)
* feat(checkpoint): production-shaped full/delta benchmark with configurable snapshot frequency - Group benchmark scripts into per-family folders (checkpoint/, sandbox/) - Extract shared benchmark infrastructure into checkpoint_bench_common.py - Add checkpoint_delta_snapshot_frequency config (default 1000, process-frozen); freeze it in make_lead_agent and DeerFlowClient; key the state-schema adaptation cache by resolved frequency - New bench_production.py: per-case child processes run N ainvoke turns through the real lead-agent graph (scripted deterministic model, real AsyncSqliteSaver), then measure GET /state + POST /history through the real Gateway route stack in one event loop (httpx ASGITransport), cold/warm accessor-cache split, cross-mode digest gates - New summarize_production.py: delta/full ratios plus decision metrics (snapshot_write_spike, cache_effect_ms, checkpoint_write_share, auto-discovered history per-limit ratios) * fix(checkpoint): address production benchmark review |
||
|
Vanzeren
|
62dd8d2b67
|
bench(checkpoint): add channel mode benchmark (#4395)
* bench(checkpoint): add channel mode benchmark * bench(checkpoint): harden benchmark reporting |
||
|
Vanzeren
|
dc38d2d003
|
perf(boxlite): add benchmark-driven warm-pool reclaim tuning (#4001)
* bench: add provider-agnostic sandbox benchmark with BoxLite warm pool results - scripts/bench_sandbox_provider.py: CLI for measuring acquire/run/release across providers, scenarios, workloads, and concurrency levels - scripts/summarize_bench.py: JSONL aggregation with p50/p95/p99 tables - bench_results.jsonl: 110 turns across 7 scenarios on real BoxLite 0.9.7 Key findings: cold acquire: ~860ms warm reclaim: ~14ms (60x speedup) release: ~0ms warm_hit_rate: 95% (warm_same_thread) * perf(boxlite): skip health check for recently-released warm pool boxes Boxes released within health_check_skip_seconds (default 5.0s) are promoted directly without the ~14ms echo-ok round-trip. A VM alive seconds ago is overwhelmingly likely to still be alive. Add sandbox.health_check_skip_seconds config option. Set to 0 to always health-check (old behaviour). Benchmark (warm_same_thread, noop, 20 iters): acquire p50: 14.9ms → 0.0ms total p50: 29.9ms → 14.0ms * chore: move benchmark scripts into backend/scripts/benchmark/ * fix: address BoxLite benchmark review findings * fix(boxlite): only skip warm reclaim checks for released boxes * fix(benchmark): keep BoxLite shim workaround off the event loop * fix(boxlite): invalidate dead boxes from command path * test(boxlite): cover skip window and invalidation edge cases * fix(boxlite): treat sandbox-has-been-closed as terminal in _exec * fix(boxlite): harden warm-pool reclaim and benchmark accounting * fix(boxlite): validate warm-pool reclaims by default * fix(config): expose boxlite health-check skip setting * fix(boxlite): tighten failure classification and benchmark workaround * Update config_version to 21 in values.yaml --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |