mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-05 18:20:33 +00:00
318 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
AochenShen99
|
5b2a1352cd
|
fix(agents): allow SQL updates with retained legacy files (#6747) | ||
|
Totoro
|
5e67d41c47
|
fix(goal): validate goal text and preserve the current goal on message replay (#6685)
* fix(goal): validate goal text and preserve current goal on message replay * fix(goal): reject malformed continuation counters before admission * fix(goal): reject external reducer overwrite markers --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> |
||
|
Hyeonsang Cho
|
5116434712
|
fix(tui): keep every word of an unquoted --print/--json message (#6636)
* fix(tui): keep every word of an unquoted --print/--json message --print and --json took a single optional value and never read the positional words, so `deerflow --print summarize this repo` sent only "summarize", `deerflow summarize this repo --print` exited 2 asking for a message on a terminal, and with piped stdin the typed words were ignored. Both flags now take every following word and fall back to words given before them, as --cli does; a typed message wins over piped stdin. A message split around the flag is refused with exit 2, because argparse loses the word order between the two parts. * docs(changelog): reference #6636 * fix(tui): refuse --print and --json together Only one headless mode runs, so the other flag's words were dropped: '--print hello --json world' ran with 'hello', and '--print --json hello' asked for a message. Put both flags in a mutually exclusive group so argparse exits 2 for every combination. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Value Add
|
4beb6c4ea4
|
feat(community): add You.com web search provider (#6515)
* feat(community): add You.com web search provider Adds You.com as an opt-in community `web_search` engine, mirroring the existing community engines (tavily, brave, exa, firecrawl, groundroute). You.com runs its own web index and returns extracted page snippets per hit rather than a single meta description, so a research loop gets usable context without a follow-up fetch. The provider picks its endpoint by whether a key is configured, so it works out of the box and scales up when the operator adds one: - no key -> /v1/agents/search, keyless free tier - YDC_API_KEY -> /v1/search with X-API-Key, full Search API Endpoint and headers are always chosen together; the keyless endpoint rejects an auth header, so a stale key can never be sent there. Both return the same body shape, so one normalizer covers both. httpx only, no new dependency and no pyproject extra, so uv.lock is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(community/youcom): review follow-ups — drop stray .pr-body.md, guard OverflowError, add 429 hint * fix(doctor): make the You.com free-provider detail true with or without YDC_API_KEY The doctor detail returned before any key check, so operators who had set YDC_API_KEY still saw 'set YDC_API_KEY for the full Search API'. Reworded to match the sibling free-provider entries and stay accurate in both states. * fix(search): sweep youcom max_results onto the shared owner The single-owner gate that landed with #6563 flags this branch's private _coerce_max_results as a newly hand-rolled copy, so backend-unit-tests now fails after merging main. Follow the coordination note from review: delegate to deerflow.community.search_max_results.coerce_max_results, which owns the rejection profile (booleans, non-integral floats, OverflowError), and keep only the 1-100 mirror of You.com's server-side count clamp on top — clamping policy stays per-provider. The .inf regression test now goes through the public tool path and also pins the bool and non-integral-float shapes of the shared bar. * docs(tools): use the bare you.com platform link in config and docs Match every other provider entry: no other provider in config.example.yaml or the en/zh tools.mdx uses campaign-tagged URLs, so drop the utm params from the three you.com/platform links. Integration attribution stays in the identifying User-Agent sent to You.com hosts. * test(youcom): cover shared max_results ownership and bounds Register You.com among the shared-owner providers and assert default handling and the 100-result cap through outbound request parameters. --------- Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: valueadd-lgtm <339269770+valueadd-lgtm@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
creed
|
3461c12896
|
feat(jina): add process-local request admission (#6489)
* feat(jina): add process-local request admission Signed-off-by: 97three <2212371308@qq.com> * test(jina): cover closed admission owner loop Signed-off-by: 97three <2212371308@qq.com> * test(jina): retain admission across API redirects Signed-off-by: 97three <2212371308@qq.com> * fix(jina): drain redirect response cleanup before releasing admission Signed-off-by: 97three <2212371308@qq.com> --------- Signed-off-by: 97three <2212371308@qq.com> |
||
|
Nan Gao
|
35f6a98cd9
|
feat(dev): add a two-Gateway multi-instance harness script (#6613)
* feat(dev): add a two-Gateway multi-instance harness script There was no way to run the deployment.multi_instance topology locally: scripts/serve.sh pins one Gateway to port 8001, the local nginx config has a single upstream, and getting two processes past the multi-instance startup gate needs Postgres, Redis, a shared DEER_FLOW_HOME, shared auth secrets and a per-process DeerMem retrieval index by hand. scripts/dev_multi_instance.sh (make dev-multi / dev-multi-check / dev-multi-down) starts throwaway Postgres and Redis containers on loopback, renders a config from the developer's config.yaml (or config.example.yaml), and runs Gateway A on 8001 and Gateway B on 8011 with one shared home, one copy of extensions_config.json and generated AUTH_JWT_SECRET, DEER_FLOW_INTERNAL_AUTH_TOKEN and DEER_FLOW_CREDENTIALS_KEY values, plus an optional round-robin nginx on 2027. B starts after A answers /health: Alembic migrations are advisory-locked, but the LangGraph checkpointer/store setup() that follows is not. scripts/dev_multi_instance.py holds the testable parts: the config overlay (neutralizing and reporting settings the gate refuses), the secrets file, the nginx config, and `check`, which automates the scriptable cross-instance acceptance checks: readiness with stream_bridge ok, A-issued sessions and the internal token accepted by B, thread/upload/artifact visibility, a skill toggle seen by B, nginx balancing, and SSE join on B with Last-Event-ID resume on A. Tests pin that the rendered config passes the real _enforce_postgres_for_multi_worker gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): link the two-Gateway harness entries to #6613 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dev): keep the multi-instance harness inside its own state Address review findings on the two-Gateway harness (#6613): - Containers carry a deerflow.harness.state-dir label; up/down remove only containers of their own state dir and refuse those of another checkout or DEERFLOW_MI_STATE_DIR instead of force-removing them. - Containers are removed with `docker rm -f -v`, so the anonymous volumes the postgres and redis images declare no longer leak on every up/down. - `up` installs an EXIT trap once it starts containers, so any later failure (including B's port being taken after preflight) tears down A and both containers; start_gateway returns failures instead of exiting the script, and the Gateway is disowned so stopping it prints no job notice. - The overlay removes any checkpointer section (a postgres one would write checkpoints and the Store into the developer's database), points an explicit sandbox.ownership and a Redis checkpoint_cache at the harness Redis (keeping ownership timing), drops DeerMem storage_path (both spellings) and the blob_storage root so data lands in the harness home, gives a local AIO sandbox a state-dir-specific container_prefix, and reports a non-DeerMem memory backend it keeps sharing. - nginx paths are rendered as quoted string literals ('$' and control characters are refused), so a state dir with spaces works. - `check` skips the run-based SSE check when model discovery did not succeed, unless --with-llm, and enforces the SSE time budget per raw line so heartbeat-only traffic cannot hold it open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(scripts): keep the harness guidance under the AGENTS.md soft limit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nan Gao
|
db7340d2fa
|
fix(security): add DEER_FLOW_CREDENTIALS_KEY and wire the channel credentials cipher (#6611)
* fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption
There was no at-rest encryption key. ChannelCredentialCipher had no
production caller: the channel service, the scheduled-task notification
outbox and the channel-connections router all built
ChannelConnectionRepository without a cipher, so store_credentials refused
(RuntimeError) and get_credentials always returned None.
- New harness module deerflow.config.credentials_key: env-only
DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts,
all decrypt via MultiFernet), CredentialsCipher with encrypt_text /
decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1:
values still readable, and get_credentials_cipher() as the process-wide
entry point. Malformed keys are refused without echoing key material.
Without the variable a single instance generates
{base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET.
- New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL
create (0600) plus read-back with a bounded settle window, shared by
.credentials_key, .jwt_secret (fixes two replicas cold-starting on a
shared volume keeping different JWT secrets) and the managed-model key.
- ChannelCredentialCipher now builds on CredentialsCipher (same
constructors); the Gateway loads the cipher off the event loop at
startup when channel_connections is enabled and passes it to every
ChannelConnectionRepository. Undecryptable values stay "missing + warn".
- Startup gate: a malformed key stops startup in any topology; a declared
multi-instance deployment with channel_connections enabled and no key
is refused. Plain multi-worker shares the generated file and is allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh
- Helm: the app Secret generates a Fernet key (randBytes 32 translated to
the urlsafe alphabet) and preserves it across upgrades via lookup like
AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef
so a user-managed existingAppSecret without it still boots (the Gateway
gate refuses only multi-instance deployments that store credentials).
README, NOTES and values document backup, rotation and existingAppSecret.
- scripts/deploy.sh resolves the key shell -> .env -> persisted file ->
generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600,
noclobber/O_EXCL with read-back): the same file the Gateway generates in
its runtime home, so either side's key is the one both keep.
- Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway.
- .env.example entry with a generation hint; README and CHANGELOG notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): make abandoned secret-file replacement single-winner
A rename is last-writer-wins, so replicas that restart together and all
find the same abandoned empty secret file could each replace it and keep
a different value. The process that exclusively creates
<name>.replacing now owns the replacement, re-checks the file is still
empty, and publishes by renaming the claim over it; peers wait for that
value, bounded so a crashed replacer cannot block startup forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): name the setting that turns off a credentials-key consumer
The multi-instance refusal ended with "or disable
channel_connections.enabled=true"; it now says "or set
channel_connections.enabled=false".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): never expose a secret file before its value is written
O_EXCL published the name before its content, so a creator that stalled
mid-write left an empty file a peer replaced after the settle window; the
resumed creator then returned a value nobody persisted, or deleted the
peer's valid file when its own write failed. Creators now write and sync
a private temporary file and hard-link it to the name, which fails if
the name exists; a failed creator only removes its own temporary file.
Filesystems without hard links publish through the same single-winner
<name>.replacing claim that replaces abandoned empty files, so creation
and replacement are serialized there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): recover an empty managed-model key when no catalog exists
An interrupted first save could leave an empty key file; every later save
then passed empty bytes to Fernet and the endpoint returned 503 until the
file was removed by hand. An empty key with no catalog encrypted nothing,
so it now goes through the secret-file recovery path; next to an existing
catalog it is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start
scripts/docker.sh runs Compose from docker/ without --env-file, so the dev
gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry
interpolated to empty and overrode the key the checkout .env supplied via
env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
yangfucheng123
|
3b8141aa43
|
fix(community): Fail closed on delegated fetch backends to close SSRF gap (#6531)
* Fail closed on delegated fetch backends to close SSRF gap * fix(community): resolve delegated-fetch credentials from one config snapshot Address review feedback on the fail-closed SSRF guard: - firecrawl/fastcrw client construction no longer re-reads get_app_config(), so the API key and endpoint come from the same revision the endpoint screen validated (prevents cross-revision credential misrouting). - fix the browserless test double to accept the new two-argument _get_browserless_client signature. - document the check-only (DNS rebinding) caveat on validate_delegated_backend_url. - add a CHANGELOG upgrade note for the default self-hosted backend break. * fix(doctor): consolidate delegated fetch backend checks and deployment guidance --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
6c4c74a504
|
fix(channels): share chat-to-thread bindings through the database (#6558)
* fix(channels): share chat-to-thread bindings through the database ChannelManager kept its IM chat-to-thread bindings (channel_name:chat_id[:topic_id] -> thread_id) in a per-process JSON file, channels/store.json, loaded once and rewritten whole on every change. With several Gateway processes a binding created on one replica was invisible to the others (the next message for the same chat opened a second thread) and concurrent writers clobbered each other's file. - New channel_thread_bindings table (migration 0036_channel_thread_bindings) and harness package deerflow.persistence.channel_thread_bindings: key PK (the legacy composite string), channel_name, chat_id, nullable topic_id, thread_id, user_id, epoch created_at/updated_at; index (channel_name, chat_id). Dialect upsert keeps created_at; prefix remove and batched ON CONFLICT DO NOTHING import. - app.channels.store.ChannelStore is an async protocol: SqlChannelStore for sqlite/postgres, JsonChannelStore (file I/O via asyncio.to_thread) for database.backend: memory. resolve_channel_store picks at ChannelService construction; ChannelService.start() imports an existing store.json once into an empty table and renames it store.json.migrated (a concurrently importing peer neither duplicates nor clobbers; a populated table leaves the file alone). - Callers updated: manager, connection_identity.lookup_thread_id, Feishu (send() awaits the mapping writer; the synchronous lark callback bridges its two store calls to the Gateway loop with a bounded wait), Discord (channels_dir). - Head pins in the 0027_0028/0029/0032/0034/0035 migration tests point at the new head; docs (channels and migrations AGENTS.md, CONFIGURATION.md, README) and CHANGELOG updated. * docs(changelog): link the channel thread bindings entry to #6558 * fix(channels): reject only the Gateway loop in the Feishu store bridge lark-oapi 1.5.5 invokes the message callback synchronously from its async frame handler, on the private event loop that FeishuChannel._run_ws starts for the SDK thread. The bridge's guard treated any running loop as unsafe, so every real inbound message skipped the stored-topic lookup and the pending-mapping restore: a reply whose parent card was mapped but whose root was not fell back to the unknown root, and could consume an unrelated pending clarification. Only a wait issued from the Gateway loop itself (the loop that executes the submitted coroutine) is refused now; the SDK's own loop may block for the bounded wait, as that thread blocked on the JSON rewrite before. Tests drive _on_message from a worker thread running its own loop, the production shape, for both the lookup and the write path, and pin the Gateway-loop refusal. * fix(channels): defer unresolved Feishu lookups and harden the store.json import Review follow-ups on the chat-to-thread binding store (#6558): - Import: a legacy entry whose key, channel_name, chat_id, topic_id, thread_id or user_id exceeds its column width is skipped and counted with the malformed ones instead of failing the whole batch on PostgreSQL (SQLite silently accepted the same row, so the backends diverged on one file). The corrupt-file warning now says what each caller does: the JSON store starts fresh, the import is skipped and the file kept. - Feishu bridge: store calls run on the Gateway loop under a 3s asyncio.wait_for (the SDK loop also services ws keepalives; the old 10s bound could stall it per message), and the lark-thread wait is that budget plus 1s. A deadline or store error is StoreLookupUnresolved, never a miss: _on_message no longer routes such a reply onto its unknown root (which created a second thread and later overwrote the parent alias). The message joins a per-chat FIFO drained on the Gateway loop with backoff for up to 30s, then is dropped with a warning. _on_message is split into parse / resolve / dispatch so the deferred path dispatches the same way; later messages for the chat queue behind the deferred one. * fix(tests): expect follow_redirects in the Jina client kwargs after #5808 #5808 made JinaClient pass follow_redirects=True to httpx.AsyncClient and updated test_jina_client.py, but test_jina_response_limit.py and test_jina_retry_after.py still pin the exact client kwargs to proxy and trust_env, so every backend-unit-tests shard that lands on them fails on main (the backend unit-test workflow only runs on pull requests, which is why the push to main did not surface it). Update the two expectations to include the new keyword. (cherry picked from commit d5aa23156ce19ce36d95df03953442f83ae6d050) * Revert "fix(tests): expect follow_redirects in the Jina client kwargs after #5808" This reverts commit 5bee00bfb09c65a30f77fb4df47a1f82c47ecef7. * fix(channels): bound deferred Feishu routing by the inbound queue capacity Review follow-up on #6558: a store outage let every subsequent message for a chat join the deferred queue with no capacity check (the bus reservation was only taken after routing succeeded), and each queued message started a fresh 30s retry budget when it reached the head, so a burst could be retained for N x 30s and grow without bound. - Intake is reserved BEFORE routing for every store-backed message (bus.reserve_inbound is thread-safe and already used from SDK threads), so deferred and live messages share the one configured bound, channels.inbound_queue_maxsize; a message that cannot be admitted is dropped at once with a warning, never queued or routed as a miss. - The reservation travels with the message into dispatch (committed by _prepare_inbound), is released on the batch path (the flush reserves at publish time), on drop after the budget, on a failed deferral, on dispatch failure and for every leftover when the drain task is cancelled by stop(). - The retry budget runs from the message's arrival: a queued message whose budget is spent when it reaches the head is dropped without a lookup. - The bounded resolver returns the unresolved outcome instead of raising it through the tracked task, which logged an ERROR for an expected, handled store timeout. - Upstream's new blocking-io test constructs the store by its old class name; it now uses JsonChannelStore. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
creed
|
be62fff070
|
feat(serper): add opt-in bounded transient retries (#6456)
* feat(serper): add opt-in bounded transient retries Signed-off-by: 97three <2212371308@qq.com> * fix(serper): validate HTTP retry dates precisely Signed-off-by: 97three <2212371308@qq.com> * fix(serper): address retry review feedback --------- Signed-off-by: 97three <2212371308@qq.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
d3887531e9
|
docs: list the summarize event tag and scope the extension example's claims (#6582)
* docs: list the summarize event tag and scope the extension example's claims RUN_EVENT_STREAM.md listed seven middleware tags, but the catalog and the run event stream contract carry eight; `summarize` is emitted by SummarizationMiddleware for every successful summary result. Add the tag, describe the event, and pin the documented list to MIDDLEWARE_EVENT_TAGS so the next tag cannot drift silently. summarization.md now lists all eight `changes` fields of that event. The extension example said it showed all five (README, pyproject, module docstrings, test name) or every contribution kind, while ExtensionRegistry has eight. Say it demonstrates five, name the three it does not, point to the observers guide and the bookmarks plugin, and assert in the example's test that the two observer kinds stay unregistered. * docs(changelog): reference #6582 * docs: clarify extension example package layout scope --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zeren Wang
|
553f85e788
|
feat(projects): best-effort LLM summaries for shelf documents (#6581)
Opt-in per-document summary column rendered into the injected <documents> index, list_project_documents JSON, and the documents API. Pipeline (projects/summaries.py, behind the narrow enqueue_summary interface): - The two shelf write routes enqueue a job on created=True; dedup hits never regenerate. Off by default (projects.summaries_enabled). - Bounded worker pool (summary_concurrency + bounded summary_queue_size; a full queue drops the job) fed by FIFO; shutdown drains queued jobs, delivers one sentinel per worker via blocking put (queue smaller than the worker pool cannot strand idle workers), and bounds only genuinely in-flight jobs with the grace period. - Jobs carry (document_id, project_id, user_id) explicitly; the job's owner is bound into the user ContextVar around the model call so tracing/usage attribution never falls back to 'default'. - Text source reuses read_text_serving_path (convertible-first rule, content validation); conversion has no deadline (await_drained), the model call has a caller-side wait limit — the worker slot is retained until the provider call actually finishes and late results are discarded. - Output contract: strip leading <think> reasoning -> unwrap fence/ quotes -> collapse to one line -> empty is NULL -> UTF-8 byte cap on a character boundary; length-capped responses (provider-normalized detectors) are failures. - set_summary pins updated_at (literal self-assignment) so a landing summary never reorders the shelf; owner-predicated UPDATE is a no-op on rows purged mid-generation. - Metrics: eligible/success/failed/skipped/queue_dropped/ uncapped_accepted. Schema/config: - project_documents.summary (migration 0036, nullable, no backfill; forward-compat floor unchanged). - Six projects.summary_* keys, all registered in STARTUP_ONLY_FIELDS (captured once at lifespan startup); config_version 57 -> 58. - run_oneshot_llm gains an optional model_overrides passthrough (passed only when set, existing callers unchanged) and a detailed variant exposing the raw response message for termination-signal detection. Docs: API.md response field, ARCHITECTURE.md pipeline paragraph, migrations AGENTS.md chain head, root AGENTS.md records the scoped all-core exception with the lift-out path. |
||
|
Totoro
|
0fe6e63a68
|
feat(goal): show why a chat goal stopped and when it was met, and redact the goal evaluator and continuation input (#6556)
* feat(goal): record the met goal and keep goal state on the history head - Add a goal_outcome LastValue channel (GoalOutcomeState). Only the guarded clear of a satisfied ordinary goal writes it, in the same checkpoint as the clear; the reply anchor is the latest visible AI message with text, the reply the visible signature keys on. A goal stored without created_at still clears and gets a record. - write_thread_goal removes the record on every other goal write and lists it in channel_versions, new_versions and metadata writes when it changes. Threads that never had a record keep their checkpoint shape. The sync-only SqliteSaver and CachedHistorySaver, which take the #6448 sync fallback, store and drop it the same way. - goal_outcome is server-owned: POST /state and non-internal run input that carry it get 400. A POST /state goal write, or a goal in external run input, clears it. - A branch head written onto its replay base resets the channels the head lacks, so a goal or record that write_thread_goal removed does not come back. Branching from the met turn keeps the record without the met goal. - The POST /history head now carries an active goal and an achieved goal_outcome, also before any message exists, so the post-run history refetch keeps them. Older entries stay title/thread_data. - contracts/thread_goal_contract.json pins the chat stand-down and check-failure codes, the record shape and status, the head keys and the host-written reasons. * feat(frontend): show why a goal stopped and when it was met - The goal bar derives its state from the active goal, the goal_outcome record and the messages: set, in progress, continuing, stopped (limit, auto-continue off, no progress, token budget, missing evidence, failed run, unknown code), waiting for you, waiting, couldn't check the goal, paused and goal met. An 8/8 stand-down now reads "Stopped" with the next step instead of "Continuing 8/8", and a count-0 stand-down no longer looks like a fresh goal. - "Goal met" shows while its anchor reply is the latest visible message (a visible human message, or a visible AI message with text, moves the chat on), so a reload agrees with the live view. An active goal wins over the record, and this tab's /goal set or clear hides it. - useActiveGoal drops the optimistic goal when the record of that same goal arrives, which fixes the bar that stayed after a goal met on the first run (the clear frame has no goal key), also for a goal stored without created_at. Server goal values pass through isGoalState; a /goal status read uses the backend's own active check, so the bar, the toast and the edit lock agree with the server. - Row 2 gives the reason and what to do next; Details holds the full goal, the checker's own note (never host-written text, a check failure or a paused goal's older check), the assumption note and an unknown code, and starts collapsed in each new state. The bar is a labelled section with a live region that announces state changes only (another goal's same state counts) and re-reads its text after a language switch. - The edit pencil on the latest turn is shown disabled with "Editing is off while a goal is set" instead of hidden; the message toolbar shows on focus-within, so keyboard users can reach it. - A /goal set or clear rejected with 409 keeps the draft and says a run is still going; other failures keep the server's message and the draft. A /goal request replaced by a newer one leaves the composer alone, so a double submit no longer empties it before the second one fails. - Copy lives in inputBox.goalBar (en-US, zh-CN); a contract test reads contracts/thread_goal_contract.json, and the raw-identifier guard now catches stand-down codes and goal field names. * test(frontend): cover the goal bar on both chat pages - tests/e2e/goal-status.spec.ts runs every case on the chat page and the agent chat page, since each wires the goal bar and the edit lock itself: a goal stopped at 8/8, one waiting for details and one waiting on an open question card show their reason and next step, also after a reload; the locked edit pencil gets keyboard focus, shows its toolbar and tooltip, does not open the editor, and hides while a file uploads; Goal met after one automatic follow-up (In progress, Continuing 1/8, then "Auto-continued once") survives a reload and hides after the next visible message; a goal met on the first run of a new chat replaces the set bar and frees the pencil, and a new /goal replaces Goal met while its own run goes on; a /goal set or clear refused with 409 keeps the draft for the retry and shows its own busy toast. - The mock POST /history head carries only the contract's history_head_keys, with goal only while active and goal_outcome only when achieved; thread state and run values omit an unset goal instead of sending goal: null. PUT and DELETE /goal drop the met record, setGoalWritesBusy returns the Gateway's 409, and a repeat create of a requested thread id returns the thread as is, like the Gateway. - The raw-identifier guard also catches checker blockers and the met record's field names. - frontend/src/AGENTS.md: the GoalStatus sentence now names goal, goal_outcome and useActiveGoal, reworded in place. * test(frontend): keep the draft through a failed upload and a busy /goal #6412 fires onSent only once a send is dispatched, so a failed upload keeps the text, files and stored draft. A /goal the Gateway refuses with 409 never reaches sendMessage, and the composer rejects it so PromptInput keeps the text. Drive both through the real sendMessage in one composer: a failed upload keeps "summarize it" stored, a busy /goal keeps its text and stored draft with no run, and the retry saves the goal and starts one run with the objective, not the "/goal" text. After that run the composer is empty and no stored draft comes back. * fix(frontend): drop the stop condition's closing period before the pause clause A stored stop condition often ends with a period, so the schedule card read "…remain)., pauses itself" (and "…。，满足后自动暂停" in Chinese). Trim the condition's closing sentence punctuation before the clause that follows it. The event line, where the condition ends the sentence, is unchanged. * fix(frontend): point "evidence missing" at the agent, not a user reply "Evidence missing" asked the user to reply with what's left, which is the push a goal is meant to save. It now asks the agent to show the result or explain what's missing, and offers /goal clear only for a goal that no longer applies: clearing drops the met record, so work that is right would never show "Goal met". zh-CN follows. Copy only: no key, state or mapping changes, and the scheduled page's reason labels are untouched. tests/unit/components/workspace/goal-status-next-steps.test.ts checks the row in en-US and zh-CN, and that no stop row offers /goal clear without that condition. goal-status.dom.test.tsx and goal-stream.dom.test.tsx now expect the new copy. * fix(frontend): say "Couldn't check the goal" once in the goal bar After a goal check that could not run, the bar's second line borrowed the scheduled page's whole reason, so it repeated the chip: "Couldn't check the goal · Couldn't check the goal; the run itself may be fine. It's checked again after your next message." Chinese had the same repeat. The line now gives only why the check failed ("The run itself may be fine", "No final reply was saved" or "The chat changed during the check"), then when it is checked again. The screen-reader announcement follows. The chip and the scheduled page's reason labels are unchanged. tests/unit/components/workspace/goal-status-unchecked.test.ts checks every check-failure code in en-US and zh-CN: the label appears once, and the line still says why the check failed. * feat(goal): redact the evaluator input when pii_redaction is enabled The goal evaluator calls its model directly, outside PiiRedactionMiddleware, so it sent raw user lines, tool arguments, tool results and Human Input Card answers even with pii_redaction enabled. With pii_redaction enabled: - the messages in the evidence window are redacted whole before the evidence caps can cut an identifier in two, as TitleMiddleware does. This reuses redact_queued_messages, which already covers content, call arguments and card answers. Earlier messages are left alone: the evaluator never sees them, and the pass is synchronous work on the event loop (0.84 s for an 800-message, 4 MB thread, 0.06 s for its window); - the assembled input, the goal objective included, is redacted again. A redaction error fails the check with evaluator_failed; raw text is never sent instead. When redaction is off, the input is byte-identical. Thread state keeps the raw text. README.md, config.example.yaml and the middleware docstring list the evaluator with the other redacted inputs. tests/test_goal_pii.py puts an address where the evidence caps cut a tool argument, a tool result and a card answer, and checks the objective, the unchanged input when redaction is off, that only the evidence window reaches the redactor and gives the same evidence as redacting the whole thread, that a redaction error stands down with evaluator_failed before the model is called, and the worker path. * fix(goal): redact the hidden continuation when pii_redaction is enabled The hidden /goal continuation repeats the goal objective and the evaluator's reason and evidence summary. PiiRedactionMiddleware skips it, since it is a framework message and not a user message, so with pii_redaction enabled each continuation turn sent them raw to the agent's model, and the message stayed in the thread for later model calls. The durable-context block and the evaluator input already redact the objective. With pii_redaction enabled, make_goal_continuation_message now redacts the objective, the reason and the evidence summary with redact_text before the message is built. The thread keeps the redacted message, which the UI hides; the goal state keeps the raw objective. The worker builds the message before it counts the continuation. A redaction error fails the check the way an evaluator redaction error does: run_failed with stand_down_reason evaluator_failed, the goal stays active, nothing is sent and the continuation budget is not spent. Only the exception type is stored. With redaction off, the message is byte-identical. README.md, config.example.yaml and the middleware docstring list the continuation with the other redacted inputs. tests/test_goal_continuation_pii.py checks the redacted objective, reason and evidence summary against the tokens redact_text gives, the unchanged message with redaction off or no config, the worker path under the run config, and a redaction error. * docs(changelog): link the goal outcome entries to #6556 Fill the four ([#PR]) placeholders in CHANGELOG.md and CHANGELOG_zh.md with this PR's number and add the matching link references. * fix(frontend): keep an active goal stored without timestamps visible POST /threads/{id}/state and run input can store an active goal without created_at or updated_at. The backend still evaluates it and rejects an edit with 409 (is_active_goal checks only status "active"), but isGoalState required both timestamps, so history and live values dropped it: both chat pages hid the goal bar and unlocked the last-turn pencil, and only a /goal status read, which used a looser predicate, brought them back. isGoalState now accepts any record with a string objective and status "active", and normalizeGoalStatusRead uses it, so thread values, live updates and status reads share one rule. Malformed or inactive values are still dropped. Tests cover a sparse active goal from the history head and from a live update before the post-run refetch (bar shown, hasGoal true), and a malformed goal that stays hidden. * test(goal): check the outcome write metadata the same way on every saver PostgresSaver stores checkpoint metadata through get_serializable_checkpoint_metadata, which drops "writes", so the [postgres] case of test_goal_outcome_round_trips_through_the_sync_fallback failed with KeyError: 'writes' when CI ran it against a live database. The test now records the metadata write_thread_goal passes to put and asserts the clear's and the next goal's "writes" there, for every saver. The stored channel values are checked as before, and on sqlite and cached-sqlite the stored "writes" must still equal what was passed. * docs(agents): keep goal_outcome out of the agents guidance chain The middlewares AGENTS.md chain is already over the 98,304-byte AG002 hard limit on main, and the PR-mode check fails any chain a PR touches, so adding goal_outcome to the ThreadState field list in deerflow/agents/AGENTS.md failed the agent-guidance job. goal_outcome stays documented in runtime/AGENTS.md and backend/app/gateway/AGENTS.md, which are outside that chain. * refactor(goal): read the evaluator's evidence and redaction from one window format_visible_conversation recomputed the window start that the redaction pass takes from _evidence_window. Both now use _evidence_window, so a later change to the cap or to what counts as visible moves both. The window test now fills the window with tool-only replies and tool results. Before, it caught a cap change in one copy but not a change to what counts as visible. --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
wgwfn3
|
ee0ba401a6
|
fix(events): preserve JSONL framing after interrupted appends (#6520)
* fix(events): preserve JSONL framing after interrupted appends * docs(changelog): link #6520 * docs(runtime): cite #6520 for JSONL tail recovery --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Daoyuan Li
|
215758818e
|
fix(models): keep Claude retries behind request admission (#6545)
* fix(models): keep Claude retries behind request admission * fix(models): keep Codex retries behind admission and log overrides * fix(models): retry paced Codex HTTP 529 errors through middleware * test(jina): allow independent HTTPX options in compatibility checks --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
9c159ef880
|
fix(auth): key the compose login lockout by client, not by nginx (#6519)
* fix(auth): key the compose login lockout by client, not by nginx Failed logins are counted per client IP, and the Gateway honors X-Real-IP only from a peer listed in AUTH_TRUSTED_PROXIES. The compose files never set it, and every browser request reaches the Gateway from the nginx container, so all logins shared nginx's address: five wrong passwords from anyone locked every user out for five minutes. AUTH_TRUSTED_PROXIES now also accepts hostnames, resolved off the event loop and cached for 10 seconds (failed lookups included, and only when X-Real-IP is present), and both compose files default it to the bundled nginx service, whose address Docker assigns. nginx overwrites X-Real-IP with $remote_addr on every Gateway route; a test pins that for the three maintained configs. * docs(changelog): reference #6519 * fix(auth): share one trusted-proxy lookup across concurrent cache misses A burst of setup-status / login requests on an expired hostname entry started one getaddrinfo per request while DNS was slow, occupying the default executor that password verification also uses. The cache check and an in-flight task registration now happen without a yield between them; later callers join that task through asyncio.shield, so a cancelled request cannot cancel the lookup the others joined, and the task caches its own result. The compose default keeps `:-` on purpose: every browser request arrives through nginx, so an empty AUTH_TRUSTED_PROXIES would only bring back the shared lockout. The compose comment and .env.example say so, and the rendered-compose test now covers the explicitly empty value. * fix(docker): keep a .env AUTH_TRUSTED_PROXIES under make docker-start docker.sh start runs Compose from docker/ without --env-file, so the dev compose file's ${AUTH_TRUSTED_PROXIES:-nginx} interpolated from the shell only, and the resolved environment entry outranked the operator's value loaded through env_file. Export it from .env alongside the proxy variables (a shell export still wins), and strip a CRLF before the quotes. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
RongJie G
|
8b8fdf4387
|
fix(gateway): make resume retries idempotent (#6499)
* fix(gateway): make resume idempotency command-aware Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com> * docs(gateway): document keyed resume rollout restrictions --------- Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
yu1128-L
|
999477b359
|
docs(api): document custom skill endpoints and correct skill examples (#6549)
* docs(api): document custom skill endpoints and correct skill examples * docs(api): clarify skill auth and edit requirements --------- Co-authored-by: yu1128-L <296164533+yu1128-L@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
yu1128-L
|
5ecc1c2ac2
|
docs(events): document durable thread seq watermark (#6532)
* docs(events): document durable thread seq watermark RUN_EVENT_STREAM.md described seq as monotonic but did not cover the durable per-thread high-water mark that keeps allocations monotonic after a run is deleted. Record that behavior in the Record Envelope section, including that thread deletion drops the mark only when no events remain. * test(scheduler): pin clock for scheduled-run timezone assertions --------- Co-authored-by: yu1128-L <296164533+yu1128-L@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
b99ac916b6
|
fix(auth): count login lockouts in the database across Gateway replicas (#6501)
* fix(auth): count login lockouts in the database across Gateway replicas POST /api/v1/auth/login/local throttled failed logins per client IP in a module-level dict, so with N Gateway replicas behind one load balancer an attacker got N x auth.local.max_login_attempts guesses and a lockout on one replica was invisible to the others. Move the counter behind a LoginThrottleStore (deerflow.persistence. login_throttle): MemoryLoginThrottleStore keeps the historical per-process semantics and 10k tracked-IP bound; SqlLoginThrottleStore shares one login_throttle row per IP through the application database, incrementing and starting the lock in a single dialect-native upsert, applying check decisions with compare-and-set predicates in a separate write transaction, and sweeping served locks plus 24h-idle counters in bounded batches. Both stores pin the existing contract: the lock starts when the count reaches max_login_attempts, the committed duration is honored when the policy shrinks mid-lock, a served sentence is never resurrected, tightening keeps the count, success resets the IP, and the 429 status and message are unchanged. New startup-only selector auth.local.throttle_storage (auto | memory | db, default auto = the database whenever database.backend is sqlite or postgres). The Gateway installs the store once at startup after the persistence engine; deps._validate_login_throttle_storage warns when a multi-process deployment keeps memory. Migration 0034_login_throttle creates the table; config_version 55 -> 56 in config.example.yaml and the Helm chart. Docs: README, CONFIGURATION.md, AUTH_DESIGN.md, config and migrations AGENTS.md, CHANGELOG. * docs(changelog): link the login throttle entry to #6501 * fix(auth): re-check expiry on the row the login-throttle sweep deletes The bounded cleanup in SqlLoginThrottleStore.record_failure deleted `WHERE ip IN (SELECT ip ... WHERE <served or stale> LIMIT 200)`. On PostgreSQL READ COMMITTED the expiry predicate applied only inside the snapshot-bound subquery: when the DELETE had to wait on a row lock held by a concurrent check() that was extending the same lock under a raised lockout_seconds, PostgreSQL re-evaluated only the outer WHERE on the updated tuple, IP membership still held for the old candidate, and the now-active lock was deleted; the next attempt from that IP got a clean record and reached password verification (review of #6501, P2). Repeat the served/stale predicate on the DELETE target as well (SqlLoginThrottleStore.sweep_statement), keeping the bounded candidate subquery, so the re-evaluation sees the updated row's expiry. The other writes already carry their full compare-and-set predicate on the target row: _discard and the check() UPDATE both use _matches (ip, fail_count, locked_at, lock_duration_seconds), so a tuple changed while they waited no longer matches and the stale decision is skipped. _discard keeps the equality compare-and-set rather than adding an expiry condition: its early-release branch deletes a lock whose committed sentence has not elapsed under the lowered live policy, which an expiry predicate would wrongly block, and the equality predicate already pins every input of the decision. reset stays unconditional by design. Tests: a dialect-compile test asserts the DELETE's own WHERE carries the expiry predicate for postgresql and sqlite (red on the previous HEAD), and a PostgreSQL-only test (skipped without TEST_POSTGRES_URI) reproduces the update-versus-sweep interleaving with two engines, waiting on pg_stat_activity for the sweep to block before the extender commits. Docs: sql.py concurrency contract and the migrations guide's 0034 bullet. * fix(auth): keep an active login lock's clock and fail closed on explicit db Address the "changes requested" review on #6501. Lock clock (confirmed defect): record_failure re-stamped locked_at = now for every failure on an already-locked row, so a sustained attacker's sentence slid forward with each attempt. Both stores now apply one rule in one atomic decision: new_count = fail_count + 1; below max_attempts the IP counts with no lock (a failure after the operator raised the threshold clears an existing lock); at or over it, a failure during an *active* lock keeps locked_at and the committed lock_duration unchanged, while a never-locked or served row starts anew with now / lockout_seconds (so the failure that reaches a raised threshold starts a fresh lock rather than keeping a stale locked_at). The SQL upsert expresses it as nested case() on the row's own values; the memory store mirrors it. The sweeps in both stores now leave the row being recorded to that decision instead of deleting a served lock or stale counter first, which split the contract (SQL would have produced a fresh (1, NULL, NULL) row). base.py states the rule. Explicit db downgrade: resolve_login_throttle_store raises SystemExit — the failure type of the other startup validations in app.gateway.deps — when throttle_storage='db' resolves to the shared table but no engine is initialised, so langgraph_runtime refuses to start instead of silently running every replica on per-process counters; the lazy bare-app path applies the same rule and installs nothing. 'auto' keeps warning and falling back. 'db' on a memory database still degrades with a warning (there is no shared table to insist on). Non-blocking items: _key() logs a warning when a client key is truncated to 255 characters (never the key itself); the sql.py docstring covers wall-clock locked_at across replicas (run NTP; skew shifts evaluation by that much) and why STALE_COUNTER_SECONDS=24h / SWEEP_BATCH_SIZE=200 were chosen; a store-level blocking-I/O anchor covers the SQL reset path; the _validate_login_throttle_storage warning was verified to render 'throttle_storage=memory' for the StrEnum (the gate test now passes the enum member and asserts that text). reset stays unconditional: a peer failure that commits after a reset leaves fail_count=1, one genuine failure after the cleared history, which a new test documents. Docs: CHANGELOG entry, config.example.yaml, CONFIGURATION.md, AUTH_DESIGN.md and the throttle_storage field description. * fix(auth): re-evaluate a login-throttle check whose compare-and-set missed SqlLoginThrottleStore.check applied its decision (commit the live duration, or clear a served / released lock) with a compare-and-set predicate on the snapshot it had read, but ignored the affected-row count. When a peer changed the row between the read and the write — an admitted wrong password incrementing fail_count on another replica, or a successful login clearing the row — the UPDATE or DELETE matched zero rows, yet check reported the decision as applied: it could return an extended 60s sentence while the row kept its 1s duration, so the next check cleared the lock and let password verification through; a live duration decrease could be dropped the same way; and a row a peer had just cleared was still reported as locked (#6501 review, P2). Every compare-and-set write in check now checks result.rowcount. On a miss the row is re-read and re-evaluated from the top, up to CHECK_CAS_ATTEMPTS (3) times, applying a fresh compare-and-set decision on the new snapshot and never clobbering a reset or replacement: a row that is gone is reported as allowed, a fresh peer lock is reported as locked, and an incremented count gets the extension or decrease committed on it. Exhausting the bound fails closed: nothing is written and the answer is the last snapshot's lock under the live policy, logged at debug. The read stays in its own short transaction and each write in a separate one. The evaluation is now one shared pure function, evaluate_check in base.py (CheckDecision), used by both stores so the contract cannot diverge. The memory store keeps a single pass: its compare-and-set cannot miss because there is no await between the read and the write on the loop thread, which its docstring now states. Tests (red first on df26c2acf) use a session proxy that runs the peer mutation on a second engine right after the snapshot read ends its transaction: the reviewer's raised-duration interleaving (59.5s reported and committed, the increment survives, a later check still locked), the mirrored decrease, early release on the fresh row, a reset landing between read and write (0.0, no row recreated), and the bound-exhausted fail-closed case. * docs(changelog): resolve merge markers left in the Unreleased section * test(migrations): pin 0035_login_throttle as the head in the 0034 watermark test Upstream #6477 added test_migration_0034_run_event_seq_watermark.py asserting 0034 is the single head; this branch chains 0035_login_throttle after it, so the test now pins the current head and walks the chain like its siblings. * perf(auth): one session per login-throttle check, one transaction per failure Two non-blocking performance suggestions from the #6501 review. Single session per check. The router probed the row with store.get() and then called store.check(), which re-read it: a recorded IP cost two connection-pool checkouts and two SELECTs per request on the unauthenticated login path, letting a lockout flood compete with legitimate traffic for Postgres connections. LoginThrottleStore.check now takes the policy as an async callable (LoginThrottlePolicy) and resolves it lazily inside the store: the probe, the policy resolution and the compare-and-set loop share one session, a clean IP returns 0.0 without ever calling the policy (the config.yaml read is still skipped for the common case), and a recorded IP resolves it exactly once, also across compare-and-set retries. The decision is made on a snapshot read after the resolution: the policy read is a yield point, and a decision that needs no write has no compare-and-set to catch a row a peer changed meanwhile — so one session, but still a read after the yield. The memory store takes the same signature and re-reads its record after the await for the same reason. The router passes its asyncio.to_thread policy read as the callable and drops the separate probe; get stays on the protocol for tests and anchors. No short-TTL verdict cache: it would mask a successful login's reset for the TTL window, and the single session already halves the hot path. Sweep in the upsert's transaction, plus an index. record_failure ran the housekeeping DELETE in its own transaction and the upsert in a second one, two commits (two WAL flushes on PostgreSQL) per failed login; the sweep now rides in the upsert's transaction with one commit, the keep exclusion already protecting the row being recorded. The new ix_login_throttle_updated_at (ORM __table_args__ and the unreleased 0035 migration, created after the table and dropped before it, each guarded separately so a table left by an interrupted attempt still gets its index) serves the sweep's stale-counter predicate; the served-lock predicate scans a table the sweep itself keeps small. Tests (red first on a137fbc47): a clean IP never resolves the policy and a recorded IP resolves it exactly once (both stores), the SQL store opens exactly one session per check and commits once per record_failure (counting wrappers), the migration test asserts the index after upgrade, after completing a partial upgrade that lacked it, and gone after downgrade while compare_metadata stays empty; the router-level "mutated during policy resolution" tests now inject their mutation through the policy callable. * docs(changelog): note config_version 57 in the login throttle entry |
||
|
Nan Gao
|
5ae953fc2f
|
fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers (#6494)
* fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers
DeerMem's derived SQLite FTS5 index for every user was one WAL database pinned
to {storage_path}/.retrieval. With several Gateway Pods sharing one home volume
each Pod opened that file over a network filesystem (unsupported by SQLite),
every Pod start emptied and refilled it under its peers, one Pod's corruption
recovery deleted files the others held open, and a Pod never learned that a
peer had written a user's facts, so memory_search served stale results.
- New backend-private key memory.backend_config.retrieval_index_path: empty
keeps {storage_path}/.retrieval, a relative path resolves against
storage_path, an absolute path is used as-is. The full rebuild and the
corruption recovery act only on that configured index.
- rebuild_index records each agent scope's manifest signature
(mtime_ns, size, revision) before reading its facts; search_facts rebuilds a
scope whose live signature differs because another process wrote the user's
memory. An own commit advances the recorded signatures of that user's scopes
only when they were in sync at the previous revision, so own writes never
rebuild while an interleaved peer write still does.
- Gateway startup warns, in the style of _validate_agent_storage, when a
declared multi-instance deployment keeps the DeerMem index inside
storage_path.
- config.example.yaml documents the key; config_version 55 -> 56, mirrored in
the Helm chart and its README.
- Helm mounts an emptyDir at /var/lib/deerflow/memory-index, points the key at
it, drops the legacy memory.storage_path: memory.json line, and documents the
memory layout next to the other multi-replica prerequisites.
* docs(changelog): link the memory retrieval index entry to #6494
* fix(memory): never advance the index signature on a no-op commit and resync whole scopes
Two review findings on the cross-replica retrieval re-sync:
- save() and apply_changes() inferred the replaced manifest revision as
"returned revision - 1". The commit helper returns the existing manifest
unchanged for a no-op (unchanged-value or empty change set), so after a peer
bumped the manifest an own no-op write advanced this process's stale index
signature to the peer's revision without carrying the peer's delta: a fact the
peer deleted stayed searchable until the manifest changed again. Both paths
now read the real pre-commit revision under the user lock and pass
committed=(pre_revision, signature) only when the commit produced a new
revision; a no-op passes None so the next search still rebuilds the scope.
- The scoped rebuild (signature mismatch and the pre-existing dirty-scope path)
listed facts through list_facts() with its default limit of 100 and then
replaced every row of the scope, so a scope with more than 100 facts lost its
tail on the first re-sync. It now reads the scope's complete fact list.
* fix(memory): fence index signature promotion against a concurrent scope rebuild
Review finding on the cross-replica re-sync: a successful adapter notification
does not prove its delta is still in the index when the signature is promoted.
The SQLite lock covers single adapter calls only, and neither the scope lock
nor DeerMem's warm-up lock spans "notify, then promote", so rebuild_index can
replace a scope's rows in between. After a peer summary-only commit (r+1), A's
search starts a scoped refresh that captures the r+1 signature and reads f1/f2;
A's own deletion of f2 commits r+2 and removes the row; the refresh then
installs its stale f1/f2 and marks the scope synced at r+1; the deletion
resumes and promoted that signature to r+2, so later searches saw the live
signature as equal and kept returning the resurrected f2 until another manifest
change.
Mechanism: a generation fence rather than a lock. rebuild_index bumps a
per-scope generation (scoped rebuild) or a storage-wide generation (full
rebuild) under _cache_lock whenever it publishes or forgets a scope's rows, in
both mark_synced and forget_synced. _dispatch_retrieval_notifications snapshots
the generations under _cache_lock before its first adapter call and, at
promotion time, skips every scope whose generation moved (and everything when
the storage-wide generation moved), leaving the rebuild's own captured
signature in place; that signature predates the commit, so the next search's
comparison rebuilds the scope. A lock would have had to serialize notification
plus promotion with bulk install plus publication while search_facts ->
rebuild_index -> load() already takes the scope, file and cache locks; the
fence adds no lock-order obligations and only costs a redundant rebuild in the
rare race. Every interleaving is covered: a rebuild that completes before the
snapshot is reflected in the captured generation and promotion proceeds on an
index that carries both the rebuild's rows and this write's delta; one that
publishes between snapshot and promotion is skipped; one that publishes after
the promotion records its stale captured signature, which the next search
catches.
Tests reproduce both orders deterministically with threading.Event ordering
(adapter wrapper that holds the bulk install and the remove notification):
test_promotion_is_fenced_against_a_refresh_that_installs_in_between (red before
the fix) and
test_stale_refresh_installing_after_a_promotion_is_caught_by_the_next_search
(regression guard for the mirrored order).
* fix(memory): publish a rebuild's signature under the lock that replaced its rows
Review finding on the cross-replica re-sync: two peer-triggered refreshes of
one scope could replace rows in one order and publish their captured
signatures in the other. On a warmed Pod A, refresh R1 captures r+1 with f1/f2
and waits before installing; a peer deletes f2 at r+2; refresh R2 captures
r+2, installs f1 only and pauses after bulk_rebuild returns, before
mark_synced; R1 installs its stale f1/f2 and publishes r+1; R2 then publishes
r+2 over R1's rows. Later searches saw the live signature as equal and kept
returning the deleted fact. The generation fence only guards own-write
promotion, and the adapter's SQLite lock ends when the install returns.
Mechanism: a per-storage _retrieval_publish_lock held across the row
replacement and the signature publication, for the bulk path (bulk_rebuild +
mark_synced / forget_synced) and the clear+upsert fallback, scoped and full
alike. Overlapping refreshes of a scope therefore serialize their
install+publish pairs: whichever installs last also publishes last, and a stale
captured signature published last differs from the live manifest, so the next
search's compare rebuilds the scope. Fact reading (load() / the glob) stays
outside the lock so a slow read never blocks refreshes of other scopes. The
startup full rebuild holds the lock for its single bulk install; scoped
refreshes wait for it, which is acceptable because before warm-up completes a
search would be rebuilding its scope anyway (DeerMem.warm_retrieval takes its
own _retrieval_lock only after rebuild() returns, so the two never nest).
Lock order: the publish lock is acquired with no storage lock held --
search_facts releases _cache_lock before calling rebuild_index, reload() leaves
its scope/file-lock block before rebuilding, and DeerMem._ensure_retrieval_scopes
holds only its outer RLock, which nothing inside rebuild_index acquires. Inside
the lock only the adapter's own lock and _cache_lock are taken, and both are
leaf locks that never call back into storage, so no cycle is possible.
Test: test_rebuild_publication_stays_ordered_with_row_replacement drives the
reviewer's interleaving with threading.Event ordering (red before the fix). The
pause harness now holds a refresh after its fact read, outside the publish
lock, instead of inside the adapter's rebuild, so the earlier fence tests keep
their semantics without holding the new lock while paused.
* fix(memory): read each manifest once per full rebuild and serve the local index on a compare read error
Two suggestion-level review items on the cross-replica retrieval re-sync.
- The full rebuild cached manifest signatures keyed by (manifest path, agent
name), but _scope_signature reads only the manifest file -- every agent
bucket of a user shares one memory.json -- so the startup scan stat+read+
parsed the same file once per bucket that has facts. The cache is now keyed
by the manifest path alone while synced signatures are still recorded per
(user, agent) scope: users manifest reads instead of users x agents.
- The per-search freshness compare put a shared-volume manifest read on every
search_facts call, and _load_memory_file raises MemoryStorageCorruption on
OSError and malformed JSON, so a transient network-filesystem read error or a
corrupt memory.json failed a search the Pod-local index could still answer.
The compare now catches OSError, ValueError and MemoryStorageCorruption: it
logs a warning naming the scope (traceback at debug), serves the current
local index for that scope and leaves the recorded synced signature
untouched so the next search compares again. It deliberately does not mark
the scope dirty: a scoped rebuild would hit the same failing read, forget the
scope's signature and fail the search path too.
* docs(changelog): resolve merge markers left in the Unreleased section
* fix(memory): forget a scope's index signature when an own delta cannot be proved compatible
Review finding on the cross-replica re-sync: skipping promotion left a newer
rebuild's signature trusted even after this dispatcher had overwritten its rows
with an older notification. On a warmed Pod A, an own PATCH of f2 commits r+1
and pauses after releasing the user lock, before its adapter upsert; Pod B
deletes f2 at r+2; a concurrent search on A detects the peer change, rebuilds
the scope without f2 and publishes r+2; the pending r+1 upsert then reinserts
the deleted fact, and the revision check merely skipped because the synced
revision was r+2 rather than the PATCH's previous r. The signature stayed equal
to the live manifest, so searches returned the deleted fact indefinitely. The
generation fence could not help: it only suppresses promotion, the stale
adapter mutation still happens.
Rule: after an own commit's notifications are applied, the mutated scope's
recorded signature may stay only when the delta is provably compatible with
the published snapshot -- the scope is synced at exactly previous_revision and
no rebuild published rows for it (or storage-wide) since the fence snapshot --
in which case it is promoted as before. In every other case where this
dispatcher mutated the adapter for the scope (synced at another revision,
newer or older; generation moved; not synced at all; or notifications without
a committed revision, i.e. migration and clear paths) the scope's signature is
forgotten so the next search's compare rebuilds it. The user's other scopes
were not mutated by this delta: they are promoted when provable and otherwise
left untouched. The decision runs under _cache_lock together with the
generation reads, so a rebuild publishing concurrently is either observed or
observes the forget; the failure path (dirty + forget) is unchanged.
Orders covered: the reviewer's (publish r+2, then the stale upsert) forgets and
the next search rebuilds without f2; the earlier fence order (publish between
adapter op and promotion) now forgets too, one redundant rebuild; no concurrent
rebuild promotes exactly as before.
Test: test_stale_own_delta_replayed_over_a_newer_snapshot_forgets_the_scope
holds the PATCH's adapter upsert after the user lock is released (new
pause_next_upsert hook on the test adapter), Event-ordered, red before the fix.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Hyeonsang Cho
|
fb0ed9c960
|
fix(browserless): send the web_fetch token as a query parameter (#6484)
* fix(browserless): send the web_fetch token as a query parameter fetch_html_with_status put the configured token in the /content JSON body. Browserless reads the token only from the token query parameter or the Authorization header, and checks it before reading the body, so with BROWSERLESS_TOKEN set every web_fetch failed with HTTP 401; against an instance started without TOKEN the unknown body key failed schema validation with HTTP 400. capture_screenshot already sent ?token=, so web_capture worked with the same token. Both endpoints now take the token from BrowserlessClient._auth_params(). The test that pinned the body token is rewritten, and a stand-in for the Browserless request gate (query/header auth, no extra body keys) drives both endpoints through the real httpx request path. * docs(changelog): reference #6484 --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
litescraper
|
e1458eefee
|
feat: support configurable Serper base URL (#6483)
* feat: support configurable Serper base URL * fix(serper): address configurable endpoint review feedback * fix(serper): address endpoint configuration review * fix(serper): pair provider settings and validate endpoints * test(serper): account for URL redaction in endpoint diagnostics --------- Co-authored-by: litescraper <322634161+litescraper@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
4bab280250
|
fix(skills): share skill cache invalidation across Gateway replicas (#6495)
* fix(skills): share skill cache invalidation across Gateway replicas SkillStorage.load_skills() rescans disk and re-reads extensions_config.json on every call, but the prompt layer in deerflow/agents/lead_agent/prompt.py caches the enabled-skills list, the per-(app_config, user_id) entries and the rendered <skill_system> section per process, and every skill mutation in routers/skills.py only refreshed the process that handled it. With several uvicorn workers or several Pods sharing one home volume, a skill installed, edited, deleted, rolled back or toggled through replica A kept being offered as the old skill set by replica B until B restarted, and POST /api/skills/reload had to be sent to every Pod separately. Reuse the MCP cache-reset mechanism. A new generic helper, deerflow.config.shared_reset_marker, owns a hidden JSON marker beside the resolved extensions config (.<config name>.<suffix>.json): publish() replaces it atomically with a random generation under extensions_config_write_lock + extensions_config_file_lock, chaining previous_generation and an optional user_id; SharedResetMarkerTracker polls it at most once per second (monotonic clock, never blocking behind a concurrent poller), adopts the current state silently on its first poll, and classifies a change as user-scoped only when exactly one publication carrying that user happened since the last poll. deerflow/mcp/cache.py now builds its .<config>.mcp-cache-reset.json marker through the same helper with no behavior change. prompt.py publishes .<config>.skills-cache-reset.json via publish_skills_cache_reset(user_id=...) and checks the marker on every cache lookup (global cache, per-config cache, prompt-section LRU, user scoped entries), calling _invalidate_enabled_skills_cache() or invalidate_user_skill_cache(). The publishing process remembers its own generation so it does not retire the caches it just rebuilt. Every mutation tail in routers/skills.py (install/upload, edit, delete, rollback, enable/disable, reload) refreshes locally and then publishes the marker off the event loop; /skills/reload answers scope=shared_config, or scope=process when no extensions config path resolves, and a marker publication failure is a 500 so a 200 never means only one process was refreshed. Docs: README, backend/docs/API.md, skills/config/mcp/gateway AGENTS.md, and frontend skills.mdx (en/zh; also fixes the reference to a non-existent skills/loader.py) describe the shared invalidation and drop the per-Pod guidance. CHANGELOG entry under Unreleased / Fixed. * docs(changelog): link the skills cache reset entry to #6495 * fix(skills): drain the reload tail and keep marker failures out of 404 Two review follow-ups on the shared skills cache reset (#6495). POST /skills/reload awaited the local refresh and the marker publication unshielded. CancelledError is a BaseException, so a caller that disconnected between the two slipped past the `except Exception -> 500` mapping, surfaced no error, and left every peer replica serving the old skill set -- the one outcome the operator hook for external mount writes exists to prevent. The refresh and the publication now run as one `await_drained(_drain_skill_mutation("reload", ...))` tail, the same discipline the install/edit/delete/rollback/toggle tails already use; the exception -> 500 mapping and the scope response are unchanged. Publication errors could surface as 404. resolve_shared_config_path() only swallows the resolution-time FileNotFoundError; a config directory that vanishes between resolution and the lock file / atomic replace still raised FileNotFoundError from the marker write, and the install, edit, delete and rollback handlers mapped that to HTTP 404 with the raw OS error text for a skill that had in fact been applied. publish_skills_cache_reset() now re-raises OSError (and a misconfigured project root's ValueError) as SkillCacheResetPublishError, a RuntimeError that no handler maps to 404/400, so every handler reports it through its generic 500 branch; the message names only the exception type, never a server path. The install handler gains an explicit clause that pins that contract beside its 404 mapping. Tests: cancelling reload after the refresh still publishes the marker; a FileNotFoundError from the marker write yields 500 (not 404) for install, edit, delete and rollback, 500 for the public toggle and the generic reload error; the wrapper type is not an OSError. |
||
|
Daoyuan Li
|
46eecf2950
|
fix(models): align MindIE synchronous streaming compatibility (#6497) | ||
|
Hyeonsang Cho
|
89c9f08714
|
fix(mcp): skip task servers whose discovery failed instead of dropping all tools (#6481)
* fix(mcp): skip task servers whose discovery failed instead of dropping all tools Per-server discovery replaced a failed or timed-out server with an empty tool list. For a server with task_toolsets, _configure_task_tools_for_server then reported submit/status/cancel as missing and raised McpTaskConfigurationError, which get_mcp_tools re-raises, so the tools of every healthy server were discarded and the cache was never published: each agent build rediscovered every server. load_server_tools now returns None for a failed discovery, distinct from a server that answered with zero tools, and the wrapping loop skips such servers before task-toolset validation. A server that answers without its configured raw tools still raises as a configuration error. * docs(changelog): reference #6481 * docs(mcp): name the cache-reset endpoint as a task-server recovery path --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
c15ecb688d
|
fix(migration): read thread owners from the configured database (#6450)
* fix(migration): read thread owners from the configured database
migrate_user_isolation.py looked for thread owners in
{base_dir}/deer-flow.db, a file DeerFlow never creates. The database is
{sqlite_dir}/deerflow.db or PostgreSQL, so the owner map was always empty
and every legacy thread was moved to users/default/.
Read threads_meta through the configured database's sync SQLAlchemy URL
(SQLite or PostgreSQL, honoring postgres_schema). When legacy threads
exist but owners cannot be read (no database file, a memory backend, a
missing driver, or a failed query), stop before migrating anything;
--allow-missing-thread-owners keeps the old assign-to-default behavior
for installs that never recorded owners.
* docs(changelog): reference #6450
* fix(migration): document recovery and report unusable database URLs
Installs that already ran the old script have every legacy thread under
users/default/threads/, and re-running finds no threads/ to migrate. The
provisioner README and AUTH_DESIGN now give the recovery: move the owned
threads back to threads/ and re-run; an existing owner directory is never
overwritten and the old copy goes to migration-conflicts/. A test pins
that re-run behavior.
create_engine errors other than ImportError (e.g. ArgumentError from an
empty postgres_url) now stop with the same "Nothing was migrated." message
instead of a traceback.
The memory AGENTS.md note said threads go to --user-id; they go to their
threads_meta owner.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
哈基米
|
3772f7c7c3
|
fix(models): keep escaped newlines inside fences in MindIE native stream (#6478)
The no-tool native stream applied _decode_escaped_newlines_outside_fences
to each streamed chunk independently, but a fenced code block almost never
opens and closes within one token-sized chunk, so the per-chunk fence regex
never saw a complete ``` pair and decoded every literal \n inside the fence.
Code like print("a\nb") reached the streamed answer and the persisted
message content with a real newline, while the non-streaming path and the
tool-mode simulated stream preserved it.
Decode the native stream with a fence-stateful decoder instead: it carries
whether the stream is inside a fence across chunks and holds back a
trailing partial ```/`\n` sequence, so the joined stream equals the one-shot
decode of the same reply. A fence left unclosed by a truncated reply keeps
its escapes, matching how an unterminated fence still renders as code.
Document native no-tool streaming in the MindIE decode contract.
|
||
|
Nan Gao
|
5e3c24fe94
|
fix(gateway): probe the Redis stream bridge and provisioner in /health/ready (#6447)
* fix(gateway): probe the Redis stream bridge and provisioner in /health/ready GET /health/ready only probed the ORM engine and the LangGraph checkpointer. The Redis stream bridge client connects lazily and nothing ever pinged it, so a Gateway whose Redis was down started, answered 200 ready to the Kubernetes readinessProbe and Compose healthcheck, and kept receiving traffic while every run's first publish failed. Since #6328 the Redis bridge is mandatory for multi-instance deployments, so such a replica is not serviceable at all. - StreamBridge gains `async ping() -> bool`; the base returns True (no external backend), RedisStreamBridge round-trips PING and answers False on RedisError. - /health/ready adds a `stream_bridge` probe (not_configured for the memory bridge; unreachable flips the endpoint to 503; a missing bridge fails closed) and a report-only `provisioner` probe of `{sandbox.provisioner_url}/health` that never changes the status code, since every replica shares one provisioner. The provisioner URL is snapshotted on app.state at startup. - A probe that overruns the endpoint deadline is now the only one marked unreachable; finished verdicts are kept. - The public probe gate is one lock per probe kind, so one request's probes still run concurrently while a thundering herd opens at most one probe connection per kind per process. - Tests: bridge ping cases, stream-bridge/provisioner probe cases, verdict and deadline cases, and the first route-level /health/ready tests via TestClient. - Docs: gateway AGENTS.md, CONFIGURATION.md multi-instance notes, Helm README and probe comment, Compose healthcheck comment. * docs(changelog): note the readiness probe fix (#6447) * fix(gateway): apply the sandbox proxy policy to the provisioner probe Review follow-up for #6447. - The provisioner readiness probe built its httpx client with the default trust_env=True. httpx 0.28 treats CIDR entries in NO_PROXY as host patterns rather than subnet bypasses, so with HTTP_PROXY set and NO_PROXY=10.0.0.0/8 a private provisioner that the requests-based sandbox calls reach directly would have been probed through the proxy and reported unreachable on every check. The probe now reuses `sandbox_http_trust_env`: loopback, private, link-local and cluster-local addresses bypass the proxy, external hosts keep the environment's proxy settings. - Pin the liveness/readiness separation with a route test: `GET /health` stays 200 while the stream bridge and database are unreachable and `/health/ready` is 503, so a Redis blip can never become a restart loop. |
||
|
Felix
|
f154cfd221
|
feat(wizard): add Opper as an LLM provider (#6399)
* feat(wizard): add Opper as an LLM provider Opper (https://opper.ai) is an EU-hosted AI gateway: 700+ models from 50+ providers behind one OpenAI-compatible API and one key. This adds it to the setup wizard's LLM_PROVIDERS next to OpenRouter, with the same shape: langchain_openai:ChatOpenAI pointed at https://api.opper.ai/v3/compat, env var OPPER_API_KEY. Models use Opper's bare model ids: claude-sonnet-4-6 (default), gpt-5.5, gemini-3.8-flash and gpt-5.4-mini. Because the GPT ids are bare, langchain-openai drops the 0.7 temperature for them the same way it does for direct OpenAI gpt-5 models. Checked: python -m py_compile scripts/wizard/providers.py, and each model id against Opper's public model list (GET https://api.opper.ai/v3/models). Not run locally: make lint and make test. Disclosure: Opper is my startup. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs: add Opper to the config example and READMEs Add an Opper model example next to the other OpenAI-compatible gateways, as requested in review: - config.example.yaml: commented opper-claude-sonnet-4-6 block after the other gateway examples (ChatOpenAI + base_url, OPPER_API_KEY) - README.md and the translated READMEs: the same entry in the manual model configuration examples, after OpenRouter - .env.example and backend/docs/CONFIGURATION.md: OPPER_API_KEY, in the shape of the Novita lines claude-sonnet-4-6 is a pooled id in Opper's public catalogue (GET https://api.opper.ai/v3/models). config.example.yaml still parses with yaml.safe_load; config_version is unchanged because the schema is. Disclosure: Opper is my startup. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
haochen
|
cde9602631
|
docs: run configuration and migration commands in the backend environment (#6425)
* docs: run configuration and migration commands in the backend environment * docs: preserve runtime paths and extras in backend commands --------- Co-authored-by: Zhangchenhao-111 <285911435+Zhangchenhao-111@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
cangen185-sudo
|
4918bea98b
|
fix(models): preserve escaped newlines in MindIE XML JSON arguments (#6334)
* fix(models): preserve escaped newlines in MindIE XML JSON arguments * fix(models): restore multiline raw MindIE tool arguments |
||
|
Totoro
|
35cdcab1c5
|
feat(scheduled-tasks): signals, honest notices and fair queueing for server-started runs (RFC #6340, 2/2) (#6393)
* feat(scheduler): lifecycle rules, goal/limit parity and coded errors for scheduled tasks
- Migration 0031: nullable scheduled_tasks.unmet_streak_after_seq (internal
streak boundary) and stop_condition (the user's stop rule, own column).
- Stop rule is normalized (stop_rule.py) and appended only at launch; the
stored prompt never contains it. Wording adapts when scheduler.tool_enabled
is off.
- Willem 1: page-created tasks get goal evaluation, <standing_notes>, the
previous-run reference and own-schedule stop; request_stop and the
scheduled capability no longer require an origin conversation.
- Repository: single reactivation choke point raising
ScheduledTaskLimitsExhausted; streak boundary moves on goal/prompt/stop
condition/note changes (not on resume); goal-check failures neither count
nor reset the streak; pause of a finished task returns "finished"; trials
on a paused task keep the host pause marker.
- New repository reads: automatic_runs_used_for, active_run_status_for,
list_manageable_from_thread, thread_relation/thread_run on thread tasks,
task_ids_for_thread, run_number, and run_number/total_tokens/summary on
run history rows.
- REST: goal/max_runs/end_at/stop_condition on create and PATCH (null
clears, naive end_at = task timezone), resume recomputes from now with an
optional renewal body, once_time_passed, task_finished, limits_exhausted,
scheduler_not_running on create, trigger returns outcome/existing/thread_id,
task responses add automatic_runs_used and active_run_status.
- Coded errors {code, message, params} for routes, shared validation and the
chat capability (scheduled_task_errors.py), pinned by
contracts/scheduled_task_errors_contract.json; goal notes contract v2 with
check_failure_codes, scheduled_origin_key and host_run_errors (host_notes.py).
- /api/features adds scheduled_tasks {available, running, tool_enabled,
min_interval_seconds}; ScheduledTaskService.is_running.
- Docs: six README scheduler sections, API.md scheduled tasks section
(breaking error shape), CONFIGURATION.md, AGENTS notes, CHANGELOGs.
- Tests: migration 0031, stop rule, error contract, run listing, lifecycle
over HTTP, repository, service and capability cases; updated tests for the
new error shape, finished-task pause and head revision.
* feat(scheduler): conversation update/resume, run-conversation authority and readable launches
- schedule_task: update and resume actions, stop_condition and clear_fields
parameters, plain parameter types (value rules come back as coded results),
rewritten docstrings for schedule_task and stop_scheduled_task.
- Capability: update/resume share the REST validation and error codes;
interactive turns in a task's run conversation may manage that task;
scheduled runs keep only own-stop; grants require a running scheduler.
- Results: compact JSON-safe task views with local times, display card/text,
last run in list, timezone_source/browser_timezone, now_local; no ISO, user,
lease or origin fields; stop_instructions removed.
- Timezone: explicit > browser (context.client_timezone, validated in
start_run and never forwarded to the run config) > not needed for intervals
and offset run_at > timezone_required.
- Trial gate accepts natural direct requests after acknowledgements and
particles; trial results report thread_id and existing.
- Scheduled launches: stable message id scheduled-<task_run_id>, server-owned
deerflow_scheduled_origin metadata with the user-language parts, run thread
title for fresh-thread tasks.
- Worker records goal continuations in the verdict; lead prompt language rule
covers stored task text and scheduled runs.
- Docs: six READMEs, CONFIGURATION, API, runtime/tools AGENTS, CHANGELOGs.
* feat(frontend): scheduled-task core types, formatting, errors and glossary
- throwGatewayApiError now throws GatewayApiError (status, code, params,
rawMessage); message stays the string detail or detail.message
- scheduled-task types gain stop_condition, automatic_runs_used,
active_run_status, origin/notes, run_number/total_tokens/summary, thread
relation, trigger result, renewal and scheduled origin; isTaskBusy and
hasActiveRun helpers
- api: fetchScheduledTask, resume with optional renewal body, typed trigger
result and update payload with null clears
- hooks: polling at 3 s while a run is active (list, thread list, run
history), useScheduledTask with live gating and final 404, useInView,
localized error toasts; run history refreshes tasks when a run finishes
- new pure modules: errors.ts (contract code to localized copy), format.ts
(relative task times, viewer time, display zone), status.ts, actions.ts,
run-error.ts; describeTaskSchedule in cron.ts; goal-outcome adds unchecked
goal checks, describeTaskOutcome and parseStopRunId
- features: scheduled_tasks block with defaults when missing
- buildRunContext sends client_timezone
- titleOfThread takes a localized untitled label; breadcrumb names scheduled
tasks and agents; reference chips use the localized untitled label
- schedule input: shared browserTimeZone, labelled selects, aria-pressed
type toggles, interval minimum from the hint template
- i18n: all PR1 scheduled-task keys in en-US and zh-CN, glossary fixes,
recipe instructions moved into i18n
- e2e mocks follow the new task, trigger, resume, thread-task and features
shapes; expectNoRawIdentifiers helpers for e2e and DOM tests
- tests for errors, format, cron, status, actions, goal outcome, run errors,
hooks, run context, thread titles and features; scheduled e2e copy updated
* feat(frontend): list-first scheduled tasks page with outcomes and editable goals
- page.tsx is now a thin composition of components/workspace/scheduled-tasks:
status tabs (All/Active/Paused/Finished) and search over the task list,
?task_id= deep link with router.replace on selection, ?thread_id= chip
with Show all, empty state, scheduler-off notice and unavailable state
- task list rows show the schedule in words and one status line (next run,
running now, paused by agent, auto-paused, finished by its cap)
- task detail: status badge, actions from availableActions with reasons,
More actions menu (Duplicate, Copy task ID, Delete), outcome notice,
Runs / Stops when / Each run's goal / Does / Notes from chat / History,
origin chat link; success toasts for every action
- run rows: run number or Trial run, summary, goal and assumption badges,
continuations, tokens, Open chat link; raw errors and the evaluator
reason only behind Details
- create/edit/duplicate dialog with goal, stop condition, safety caps and
advanced agent/context options; PATCH sends only changed fields, null
clears a cap
- renew dialog raises or removes caps and resumes in one request; a
limits_exhausted Resume opens it
- removed i18n keys the old page used; added feedback.resumedNoTime
- recorded en-3 and zh-1 page fixtures; scheduled e2e specs rewritten for
the new page; DOM tests for the detail and the form dialog
- CHANGELOG entries and the frontend AGENTS.md scheduled-task note
* feat(frontend): live schedule card, run conversation provenance and header entry
- schedule_task results (create, update, pause, resume, trial, delete) become
an assistant:scheduled-task message group rendered as a live card: schedule,
next run, stop condition with the safety cap, goal, where results go, Run
once now, Pause/Resume and Open task; the card polls the task only while it
is on screen and shows deleted, scheduler-off and limits-exhausted states
- one card per task per assistant turn; the card group is not a turn boundary
for usage, branching, editing, outline or skill attribution
- a human message with deerflow_scheduled_origin stays a human group with
scheduledOrigin and renders as a "Scheduled run / Trial run" block with the
task instructions, stop condition and notes collapsed; the launched text is
never shown and the message is not editable
- readable step labels for schedule_task actions and stop_scheduled_task
- ThreadScheduledTasksButton replaces the header link: hidden without tasks,
shows the count, links to the task (or the chat filter), and a run's chat
links to its task
- recorded en-3 / zh-1 chat and run-thread fixtures; chat-card and run-thread
e2e specs; unit and DOM tests for parsing, grouping, turn boundaries, card,
run block, header button, step labels and reconnect hydration
- i18n: header.countLabelOne added, unused card.next removed; CHANGELOG and
frontend AGENTS.md notes
* fix(scheduler): address verification findings
Move the scheduled-task lifecycle guidance from backend/AGENTS.md into the persistence guide and leave a one-line pointer, so the effective AGENTS.md chains for agents/middlewares, sandbox and subagents stay under the 98304-byte hard limit enforced by scripts/check_agent_guidance.py in lint-check.
* fix(scheduler): address review findings
* fix(gateway): no-transform on SSE responses
All four text/event-stream responses (thread run create stream, join,
existing-run GET/POST stream, stateless create stream) now take their
headers from app/gateway/sse_headers.py and send
Cache-Control: no-cache, no-transform, so compressing proxies such as
Next's rewrite proxy stop buffering streams. Content-Location and
X-Accel-Buffering are unchanged; scheduled and MCP-notification runs keep
stream_mode=None.
Tests pin the headers on every SSE route and fail on a literal no-cache
header left in routers. Docs: API.md SSE headers note, CHANGELOGs.
* feat(scheduler): per-owner run cap and fair queue selection
New scheduler.max_concurrent_runs_per_user (default 2, 0 = off, effective
min(cap, max_concurrent_runs)) caps launching/running scheduled runs per task
owner. claim_queued_run checks it right after the global count under the same
budget lock, so it is atomic across workers and Pods. list_queued_runs ranks
same-thread FIFO heads per owner with ROW_NUMBER() OVER (PARTITION BY owner)
and orders the batch by that rank, so the drain is owner-fair even with the
cap off; owners at their cap drop out before the batch limit. Orphan rows
(task deleted while queued) are never held back by the cap.
Gateway lifespan passes the startup value; the service constructor default
stays 0. Docs: config.example.yaml (config_version 54), CONFIGURATION.md,
the six READMEs, backend AGENTS.md, CHANGELOGs (behavior change flagged).
The live-task quota is untouched.
* feat(scheduler): durable lifecycle events for the originating chat
When a chat-created schedule is paused by the agent, auto-paused or finishes,
finalization now reports task_stopped / task_paused / task_finished, only on a
live -> paused/finished transition. The new finish_task_at_end_condition
replaces the three inline "mark completed" blocks (admission rejected as ended,
claim-time skip, complete_if_ended), so every site emits exactly once.
The scheduler's finalization observer is always installed. It writes
scheduled_task_events rows for the originating chat in the same transaction as
the state change, deduplicated on (task_id, anchor, event), with a title
snapshot, stop condition, run number, last run outcome and the chat's newest
run as placement anchor. Lookups fall back on bad stored data; only database
errors propagate. Manual trials drop only run_* events. The IM step keeps
PR1's selection until B4.
New GET /api/threads/{thread_id}/scheduled-task-events (owner check, caller's
rows, oldest first); rows survive task deletion and go with the chat.
Migration 0032_activity_and_task_events adds the events table plus the
run-origin column/index and read-state tables used later, each object guarded
on its own. Contract scheduled_goal_notes v3 pins the lifecycle vocabulary.
Docs: API.md, migrations chain, backend AGENTS.md, six READMEs, CHANGELOGs.
* feat(scheduler): honest, localized and self-contained IM notices
Every scheduled-task IM notice is now enqueued by the finalization observer
in the transaction that records the outcome, one message per occurrence
(task_stopped > task_paused > task_finished > run_failed > run_unmet >
run_completed; a once task sends only its run event). The merged auto-pause
replaces the separate unmet + pause messages, agent stops and finishes now
notify, and recovery-finalized runs notify exactly once. Idle finishes use a
deterministic idle: key derived from the lifecycle anchor. The old
non-transactional enqueue in handle_run_completion is removed.
New app/channels/capabilities.py holds CHANNEL_CAPABILITIES with a
proactive_notifications flag (adds the missing qq entry); only push-capable
providers (WeCom) get rows, /api/channels/providers returns the flag, and the
worker ends legacy rows for other providers as terminal failed.
New app/scheduler/notification_text.py renders en-US/zh-CN text: title, what
happened (plus the last run's outcome), the tasks page's one-line run summary
(redacted) when the agent replied, and a closing line. No IDs, no links, no
public_base_url. Locale is the owner's new `locale` preference, else the new
channel_connections.notification_locale (default en-US).
Docs: config.example.yaml (config_version 55), CONFIGURATION.md, API.md,
channels AGENTS.md, six READMEs, CHANGELOGs (behavior change flagged).
* feat(gateway): thread origin, activity feed and read markers for server-created threads
Runs started by the server for a user (scheduler, IM channels, GitHub
agents, extension handles, MCP notifications) now carry a server-owned
metadata.deerflow_origin {kind, provider?, namespace?} (new harness module
deerflow.runtime.run_origin, pinned by contracts/thread_origin_contract.json).
It is stamped only from request.state.run_origin or by the internal channel
caller; run admission drops client copies from both metadata and
config.metadata, and thread create/patch strip it. RunRepository derives
runs.origin_kind on put and on create_thread_operation_atomic (agent runs
only), and gains latest_change().
New ThreadReadRepository keeps a per-user, never-decreasing read position
(thread_read_markers) and a per-user read clock (thread_read_versions).
POST /api/threads/{id}/read marks a thread read, thread search items carry
unread, and thread delete removes the markers. GET /api/thread-activity pages
the caller's run changes over the run-change clock, returns only threads
changed by server-originated runs plus read_version, and 503s without SQL;
/api/features reports thread_activity.available.
Docs: API.md, gateway AGENTS.md (condensed to stay under its budget),
CHANGELOGs.
* fix(scheduler): adapt PR2 to PR1 follow-ups
PR1 moved the scheduled-task lifecycle rules out of backend/AGENTS.md into
the persistence guide and grew the sandbox AGENTS.md chain to within 267
bytes of its hard limit. The per-owner queue-fairness detail now lives in
the persistence guide next to the lifecycle rules (backend/AGENTS.md keeps
the cap and a link), so check_agent_guidance passes again.
PR1's goal-outcome unit test pinned scheduled_goal_notes_contract version 2;
PR2 bumps it to 3 (lifecycle vocabulary only, the v2 keys are unchanged).
* fix(scheduled-tasks): polish from live acceptance
- zh schedule wording: an interval of one unit reads 每分钟 / 每小时 / 每秒,
and a saved interval of whole days reads 每天 / 每 2 天 (Every day / Every 2
days) instead of 每 1 分钟 or 每 24 小时; several units keep the number
- zh spacing: 已于{time} 满足 and 在{time} 的运行中 now leave a space between
the interpolated time and the Chinese after it; a copy test guards every
scheduled-task zh string against a placeholder glued to Chinese
- run summary: when the first line of the final reply only leads in to a list
(ends with ":" or "："), the list items follow it as plain text (bullets,
numbering, checkboxes, bold, code and links stripped), joined with "；" for
CJK text and "; " otherwise, within the 160-character cap
- a schedule_task result (create, update, pause, resume, trial, delete) that
arrives in a chat invalidates that chat's task list, the tasks list and the
named task, so the header button and cards update without waiting for the
idle poll; results already in the loaded history do not refetch
- e2e and unit fixtures are now the live acceptance recordings (DeepSeek
Flash, demo data, fictional names): live-weekday.json (create, trial, edit,
pause, resume in one zh chat) and live-minute.json (per-minute chat, run 1
and run 2, paused by the agent), fetched with GET from the demo Gateway and
trimmed; the hand-shaped en-3 / zh-1 fixtures are removed and the chat-card,
run-thread and tasks-page specs run in both locales over the recorded data
* docs(changelog): reference #6378 for the scheduled-tasks entries
* docs(agents): shorten the scheduled-task pointer to stay under the guidance limit
* test(scheduled-tasks): pin the live takes to their recorded time zone
The live fixtures were recorded in Asia/Shanghai, and an interval task
reads in the viewer's zone, so the 16:48 assertions failed on UTC CI
runners. Pin the browser zone in the two live-take e2e specs and the
zh live-take unit test.
* feat(frontend): thread origin, activity polling, read markers and task events core
Frontend core for server-created threads and schedule lifecycle events
(PR2 F1); the sidebar markers (F2) and the chat event line (F3) build on it.
- core/threads/origin.ts: threadOriginOf reads metadata.deerflow_origin,
then channel_source (github -> GitHub origin), then a legacy
scheduled_task_id; key and kinds pinned to
contracts/thread_origin_contract.json. labelOfThreadOrigin and the now
exported labelOfChannelProvider(provider, t) use the new localized
threads.origin.providers names (飞书, 企业微信, ...), with the English map
(gaining GitHub) only as the fallback.
- core/threads/activity.ts: useThreadActivity polls /api/thread-activity
every 15 s (focus refetch, never in the background, off unless
features.thread_activity.available). The first response only seeds the
cursor; server-origin threads, a truncated page or a higher read_version
invalidate the thread lists, a scheduled thread also ["scheduled-tasks"];
an invalid cursor re-seeds. useMarkThreadRead debounces POST /read by 1 s,
skips a thread whose cached unread is false, patches the search and
infinite caches optimistically and stores the returned read_version so a
tab's own read does not refetch its lists. AgentThread gains unread.
- core/scheduled-tasks/events.ts: ScheduledTaskEvent, the events query
(["scheduled-tasks","thread-events",id], one fetch per saved chat, not
gated on the task list so a deleted task's lines stay, refetched when a
task of the chat changes status or finishes a run), describeTaskEvent
(en/zh copy, title and condition as segments, last-run suffix, See that
run / Open task) and placeTaskEvents (end of the after_run_id turn, else
the tail, ordered by created_at); lifecycle names pinned to contract v3.
- features: thread_activity (default unavailable); channels:
proactive_notifications on providers.
- Locale sync: the I18nProvider reports the UI language and a signed-in
preference session saves it as the new `locale` preference when the
account's differs (and on every switch), so IM notices follow the UI
language; nothing is written with auth disabled.
- i18n: threads.* (unread, origin labels, provider names),
scheduledTasks.events.*, history.waitingForSlot (now shown for queued run
rows), channels.scheduledUpdates.*.
- e2e mocks: thread activity feed and controller, POST /read, the events
route, providers with proactive_notifications, search unread, features
thread_activity (off by default). Preference e2e mocks return the UI
locale so they keep asserting only their own edits.
- Docs: frontend AGENTS.md, CHANGELOGs.
* fix(scheduled-tasks): keep untouched end times and drop a chat filter the new task is not in
- The edit, duplicate and renew dialogs keep a stored end time's instant
while its field still shows it. A minute-precision wall value cannot
keep seconds or name the later 01:30 of a DST fall-back, so a
title-only edit moved the cap an hour earlier, and raising the run
limit re-sent the end time without its seconds.
- Creating or duplicating a task from a chat's task view drops the chat
filter when the new task does not belong to that chat, instead of
showing "This task isn't one of this chat's tasks".
* feat(frontend): origin markers and unread state in the thread lists
Server-created threads (PR2 F2) now say who created them and whether they
are new, in the sidebar and on the Chats page, and reach an open sidebar
without a reload.
- ThreadOriginIcon (thread-channel-source.tsx): clock for a scheduled run,
the provider mark for IM threads, the GitHub mark (new in
ChannelProviderIcon) for GitHub threads, a puzzle piece for extensions;
the localized "Scheduled run" / "From Feishu" / "From GitHub" / "From an
extension" label is its accessible name and tooltip. ThreadChannelIcon
stays exported as a wrapper, and the provider chip tooltip is localized
(no more "Feishu channel").
- Unread dot after the title (aria-hidden dot plus sr-only "Unread"); the
row aria-label becomes "{title}, unread" / "{title}，未读". Never shown for
the open thread.
- WorkspaceSidebar mounts useThreadActivity once.
- useMarkOpenThreadRead (components/workspace/chats): the default and
custom-agent chat pages mark the thread read once its metadata has
loaded, when its cached unread flips to true while open, when the tab
becomes visible, and from onFinish (a send or a joined run ended). Each
trigger goes through useMarkThreadRead, so it is debounced and a no-op
while the cached unread is false.
- Tests: thread-origin-icon and recent-chat-list DOM tests (en/zh, active
row, no raw identifiers), use-mark-open-thread-read DOM tests,
e2e thread-activity.spec.ts (new scheduled run appears unread within one
fake-clock poll without a reload, opening it posts /read and clears the
dot, a read on another device clears it, IM and GitHub markers, zh
labels, the Chats page, no polling when unavailable);
thread-history.spec.ts now finds the localized origin icon.
- Docs: frontend AGENTS.md, CHANGELOGs.
* feat(frontend): schedule event line in the originating chat and channel notice support
The chat that created a schedule now says when the schedule paused or
ended, and the channel cards say which apps receive scheduled task updates
(PR2 F3).
- ScheduledTaskEventLine (mockup m2, marker 4): a dashed, muted row with
the describeTaskEvent text (title in <strong>, stop condition clamped to
one line with the full text in its tooltip, last-run suffix), the time
via formatTaskTime and a right-aligned "See that run" (run chat) or
"Open task" link. PauseCircle for paused, CircleCheck for finished and
once-done, CircleX for once-failed. role="note", aria-label "Scheduled
task update" / "定时任务通知", data-task-id and data-event-id only.
- MessageList takes scheduledTaskEvents and renders placeTaskEvents output
through a new VirtualMessageList.renderAfterGroup, so each line sits at
the end of its after_run_id turn inside the group's measured row (also
when the list virtualizes); lines with nowhere to go render at the tail.
- Both chat pages load useThreadScheduledTaskEvents (not gated on the task
list, so the lines stay after a task is deleted). The custom-agent page
passes its agent so "See that run" opens the run on that agent's route.
- ChannelScheduledUpdates: "Scheduled task updates: sent here" / "... not
available for this app yet" (定时任务通知：会发送到这里 / 此应用暂不支持)
on the Settings provider cards and under each sidebar channel row; nothing
when an older backend does not report proactive_notifications.
- e2e mock controller: setScheduledTasks.
- Tests: scheduled-task-event-line and channel-scheduled-updates DOM tests
(every kind in en and zh, hrefs, clamp, no raw identifiers, Chinese
title), MessageList placement DOM tests; e2e
scheduled-task-chat-events.spec.ts (the line appears with the card's
"Paused by agent" without a reload, See that run opens the run chat,
anchored vs tail placement, two events in time order, the line survives
task deletion and Open task shows the task is gone; en-3 and zh-1) and
channels.spec.ts (WeCom sent here, Feishu not yet; en and zh).
- Docs: frontend AGENTS.md, CHANGELOGs.
* fix(scheduled-tasks): tie pause notices to the pause they describe
- Paused by agent links and times only the run named in last_error. When
that run is not loaded, the notice shows no link and the task's own time
instead of an earlier run that also asked to stop.
- The outcome notice reads the latest history page whichever page is
shown, so browsing older runs no longer swaps the auto-pause reason and
"Open latest run" to an earlier miss. It shares that page's request.
* fix(scheduler): address PR2 review findings
- Event lines: hold an event whose anchor run is on an unloaded history
page instead of placing it at the bottom (hasMoreHistory); tail fallback
only for fully loaded (branched/pruned) history or no anchor.
- Idle-finish anchor follows its reason: end:<iso> only for an end_at
finish, seq:<n> otherwise, so a later end-time finish is not deduplicated
away by an earlier max_runs finish.
- Observer takes FOR SHARE on the originating threads_meta row, so a racing
thread delete cannot leave an orphan event line.
- One taskPagePath (core), re-exported by the card.
- Notification locales derived from NotificationLocale; /preferences uses it.
- IM notices drop generic reasons (goal_not_met_yet, unknown) instead of
repeating the sentence.
- Localized provider names on Settings cards, sidebar rows and setup dialog.
- zh event lines: no space after a Chinese title (Latin-aware gap).
- Unread row label keeps the origin marker; chats page keeps the time as a
description.
- Sidebar shows the scheduled-updates line only for connected apps;
Settings says "available after you connect" for unconnected push apps.
- Event line keeps a real space before the time.
* fix(scheduled-tasks): three lifecycle gaps found while re-checking review classes
- The edit form compares a cron in the form its schedule input emits, so an
untouched "0 09 * * 0,6" is not re-sent as "0 9 * * 6,0" and a title-only
edit no longer re-arms or refuses a finished task.
- Chat resume asks for a timezone, as update does, when a naive end_at would
land in the UTC placeholder of a zone-free task.
- Re-arming a task clears a host pause marker, as resume does, so a later
manual pause no longer reads as "Paused by agent".
* test(scheduler): consolidate PR2 backend test setup
- Fold the repeated task, origin-chat and observer setup of the
lifecycle-event repository tests into one lifecycle_task builder.
- Build the per-owner-cap services through one _capped_service builder
that keeps every cap explicit at the call site.
- Table the five SSE route header checks into one parametrized test with
the same routes, bodies and Content-Location expectations.
- Reuse bind_wecom and hoist repeated local imports in the event tests.
Every case and assertion is kept; collected cases are unchanged.
* test(frontend): consolidate PR2 thread activity and origin test setup
Extract shared builders and helpers in the frontend tests PR2 adds, with
every case, input and assertion kept:
- activity.test.ts: seededClient/apply helpers and one LIST_KEYS constant
replace the per-test seed, spy and apply boilerplate.
- mark-read.dom.test.tsx: newClient() and advance() replace the repeated
QueryClient wrapper and fake-timer act blocks.
- origin.test.ts: the English and Chinese label checks become tables; the
no-English-name check now runs on every Chinese row.
- thread-activity.spec.ts: a scheduledRun() builder replaces four inline
copies of the scheduled run thread.
- channels.spec.ts: the PR2 block uses setLocaleCookie and one card helper.
Collected cases are unchanged (unit 211, e2e 60); no test was removed.
* fix(scheduled-tasks): keep the notice's latest-runs query from refreshing older pages
useLatestScheduledTaskRuns kept React Query's default refetch on mount,
focus and reconnect, so going offline and back while an older history page
was shown fetched the latest page again. Refreshing stays with the history,
which refreshes the latest page only while it is shown.
* fix(scheduler): two gaps found in live acceptance
- A fresh scheduled run thread is named when it launches. The worker
copied the title to the thread list only when the run ended, so the
sidebar, which now shows the new run thread at once, listed it as
untitled while it ran.
- A task capped at one run finishes with "its one run is done" /
"唯一一次运行已完成" in the chat event line and the IM notice, instead
of "all 1 runs are done".
* test(scheduled-tasks): let the viewer's time carry its own day label
Between 16:00 and 24:00 UTC, UTC and Asia/Shanghai are on different days,
so the run prompt shows "Tomorrow 01:00 · Today 09:00 your time" and the
fixed "01:00 · 09:00 your time" assertion failed depending on the hour.
* test(scheduler): expect the one-run finish copy in service notices
* fix(gateway): create a scheduled run thread with its title
Pre-setting the name after start_run raced the thread row, which the run
worker creates asynchronously: live, one of two run threads still reached
the sidebar untitled. Run admission now creates a new thread with the title
its run input carries, which the worker would copy only at run end.
* docs(changelog): reference #6393 for the scheduled-task signal entries
* fix(scheduled-tasks): route event-line run links by the agent that ran them
The originating chat's agent was used for "See that run", but a task's
agent can change after the chat created it, so the link could open the run
chat on another agent's route. Lifecycle events now record run_agent_name,
the task's agent when the run finished (edits are refused while a run is
active), and the line routes by it.
* fix(scheduled-tasks): load up to the route's maximum of chat event lines
The chat loads its lifecycle events once, and the route's default of 50
would silently drop the oldest lines of a chat with many schedules. Ask
for the maximum, 200.
* fix(threads): keep activity reads and the first cursor in step with the lists
- Accepting the first activity cursor refreshes the thread lists once. The
lists load on their own, possibly before a run the seed already counts,
and later polls start after it, so such a thread never appeared.
- A local read whose returned read clock jumps past that one read also
covers reads on other devices; the lists are refreshed instead of the
clock silently skipping those threads' stale unread dots.
- An invalid cursor re-seeds through the same path, so it refreshes once.
* docs(agents): bring the middlewares guidance chain back under the hard limit
#6378 and #6363 each fit the budget, but together the middlewares AGENTS.md
chain on main is 98,425 bytes, 121 over the 98,304 hard limit, so Lint
Check fails on main. Move the scheduled-task recovery, launch-update and
queue-timeout rules from backend/AGENTS.md into the persistence guide's
lifecycle section, which backend/AGENTS.md already points to. The chain is
now 98,067 bytes.
* fix(helm): bump the chart's config_version to 55
config.example.yaml moved to 55 in this PR, so the Helm chart's
validate-chart drift check failed. The chart's embedded config needs no
field changes: the new scheduler and channel keys have defaults.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
creed
|
51a33c79b3
|
feat(jina): add opt-in response byte budget (#6373)
* feat(jina): add opt-in response byte budget Signed-off-by: 97three <2212371308@qq.com> * test(jina): clarify response limit assertions Signed-off-by: 97three <2212371308@qq.com> --------- Signed-off-by: 97three <2212371308@qq.com> |
||
|
Nan Gao
|
4f64ca2ae0
|
fix(config): revalidate the extensions config singleton by file signature (#6386)
* fix(config): revalidate the extensions config singleton by file signature `get_extensions_config()` parsed `extensions_config.json` once per process and only the process that handled a write ever reloaded it. With several uvicorn workers, or several Pods sharing one volume, every other process kept its startup copy: the MCP servers the agent could use and the local-bash absolute path allowlist derived from the filesystem MCP server differed between replicas until each one restarted. The singleton now records the resolved path and the file's (mtime, size, sha256) signature and reloads when either changes, like `get_app_config()` does for `config.yaml` and `mcp/cache.py` does for the tools cache. After a successful load, a revision that cannot be loaded (file gone, truncated, invalid) keeps the last-known-good configuration and is logged once with the exception type only; the first load still raises. `set_extensions_config()` pins the injected instance until a reload or reset. The cached AppConfig's `extensions` snapshot is built from the singleton instance and refreshes when that instance changes, so it inherits the same fail-soft policy instead of parsing the file again. * docs(changelog): link the extensions config freshness entry to #6386 * fix(config): follow an explicitly reloaded extensions config path `reload_extensions_config(config_path=X)` recorded X, but the freshness probe re-resolved the default path, so the next read either flipped the cache back to the default file or pinned a stale copy of X. The cache now remembers the explicit path and both the probe and later reloads follow it until a reset or an argument-less reload. Also shortens the pointer in backend/AGENTS.md so the nested guidance chains stay under the hard size limit enforced by the agent-guidance check. * docs: keep backend guidance within the inherited size budget * fix(config): reload the probed extensions config path --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Totoro
|
2dd2e5415c
|
feat(scheduled-tasks): finish the user-facing side of conversation schedules (RFC #6340, 1/2) (#6378)
* feat(scheduler): lifecycle rules, goal/limit parity and coded errors for scheduled tasks
- Migration 0031: nullable scheduled_tasks.unmet_streak_after_seq (internal
streak boundary) and stop_condition (the user's stop rule, own column).
- Stop rule is normalized (stop_rule.py) and appended only at launch; the
stored prompt never contains it. Wording adapts when scheduler.tool_enabled
is off.
- Willem 1: page-created tasks get goal evaluation, <standing_notes>, the
previous-run reference and own-schedule stop; request_stop and the
scheduled capability no longer require an origin conversation.
- Repository: single reactivation choke point raising
ScheduledTaskLimitsExhausted; streak boundary moves on goal/prompt/stop
condition/note changes (not on resume); goal-check failures neither count
nor reset the streak; pause of a finished task returns "finished"; trials
on a paused task keep the host pause marker.
- New repository reads: automatic_runs_used_for, active_run_status_for,
list_manageable_from_thread, thread_relation/thread_run on thread tasks,
task_ids_for_thread, run_number, and run_number/total_tokens/summary on
run history rows.
- REST: goal/max_runs/end_at/stop_condition on create and PATCH (null
clears, naive end_at = task timezone), resume recomputes from now with an
optional renewal body, once_time_passed, task_finished, limits_exhausted,
scheduler_not_running on create, trigger returns outcome/existing/thread_id,
task responses add automatic_runs_used and active_run_status.
- Coded errors {code, message, params} for routes, shared validation and the
chat capability (scheduled_task_errors.py), pinned by
contracts/scheduled_task_errors_contract.json; goal notes contract v2 with
check_failure_codes, scheduled_origin_key and host_run_errors (host_notes.py).
- /api/features adds scheduled_tasks {available, running, tool_enabled,
min_interval_seconds}; ScheduledTaskService.is_running.
- Docs: six README scheduler sections, API.md scheduled tasks section
(breaking error shape), CONFIGURATION.md, AGENTS notes, CHANGELOGs.
- Tests: migration 0031, stop rule, error contract, run listing, lifecycle
over HTTP, repository, service and capability cases; updated tests for the
new error shape, finished-task pause and head revision.
* feat(scheduler): conversation update/resume, run-conversation authority and readable launches
- schedule_task: update and resume actions, stop_condition and clear_fields
parameters, plain parameter types (value rules come back as coded results),
rewritten docstrings for schedule_task and stop_scheduled_task.
- Capability: update/resume share the REST validation and error codes;
interactive turns in a task's run conversation may manage that task;
scheduled runs keep only own-stop; grants require a running scheduler.
- Results: compact JSON-safe task views with local times, display card/text,
last run in list, timezone_source/browser_timezone, now_local; no ISO, user,
lease or origin fields; stop_instructions removed.
- Timezone: explicit > browser (context.client_timezone, validated in
start_run and never forwarded to the run config) > not needed for intervals
and offset run_at > timezone_required.
- Trial gate accepts natural direct requests after acknowledgements and
particles; trial results report thread_id and existing.
- Scheduled launches: stable message id scheduled-<task_run_id>, server-owned
deerflow_scheduled_origin metadata with the user-language parts, run thread
title for fresh-thread tasks.
- Worker records goal continuations in the verdict; lead prompt language rule
covers stored task text and scheduled runs.
- Docs: six READMEs, CONFIGURATION, API, runtime/tools AGENTS, CHANGELOGs.
* feat(frontend): scheduled-task core types, formatting, errors and glossary
- throwGatewayApiError now throws GatewayApiError (status, code, params,
rawMessage); message stays the string detail or detail.message
- scheduled-task types gain stop_condition, automatic_runs_used,
active_run_status, origin/notes, run_number/total_tokens/summary, thread
relation, trigger result, renewal and scheduled origin; isTaskBusy and
hasActiveRun helpers
- api: fetchScheduledTask, resume with optional renewal body, typed trigger
result and update payload with null clears
- hooks: polling at 3 s while a run is active (list, thread list, run
history), useScheduledTask with live gating and final 404, useInView,
localized error toasts; run history refreshes tasks when a run finishes
- new pure modules: errors.ts (contract code to localized copy), format.ts
(relative task times, viewer time, display zone), status.ts, actions.ts,
run-error.ts; describeTaskSchedule in cron.ts; goal-outcome adds unchecked
goal checks, describeTaskOutcome and parseStopRunId
- features: scheduled_tasks block with defaults when missing
- buildRunContext sends client_timezone
- titleOfThread takes a localized untitled label; breadcrumb names scheduled
tasks and agents; reference chips use the localized untitled label
- schedule input: shared browserTimeZone, labelled selects, aria-pressed
type toggles, interval minimum from the hint template
- i18n: all PR1 scheduled-task keys in en-US and zh-CN, glossary fixes,
recipe instructions moved into i18n
- e2e mocks follow the new task, trigger, resume, thread-task and features
shapes; expectNoRawIdentifiers helpers for e2e and DOM tests
- tests for errors, format, cron, status, actions, goal outcome, run errors,
hooks, run context, thread titles and features; scheduled e2e copy updated
* feat(frontend): list-first scheduled tasks page with outcomes and editable goals
- page.tsx is now a thin composition of components/workspace/scheduled-tasks:
status tabs (All/Active/Paused/Finished) and search over the task list,
?task_id= deep link with router.replace on selection, ?thread_id= chip
with Show all, empty state, scheduler-off notice and unavailable state
- task list rows show the schedule in words and one status line (next run,
running now, paused by agent, auto-paused, finished by its cap)
- task detail: status badge, actions from availableActions with reasons,
More actions menu (Duplicate, Copy task ID, Delete), outcome notice,
Runs / Stops when / Each run's goal / Does / Notes from chat / History,
origin chat link; success toasts for every action
- run rows: run number or Trial run, summary, goal and assumption badges,
continuations, tokens, Open chat link; raw errors and the evaluator
reason only behind Details
- create/edit/duplicate dialog with goal, stop condition, safety caps and
advanced agent/context options; PATCH sends only changed fields, null
clears a cap
- renew dialog raises or removes caps and resumes in one request; a
limits_exhausted Resume opens it
- removed i18n keys the old page used; added feedback.resumedNoTime
- recorded en-3 and zh-1 page fixtures; scheduled e2e specs rewritten for
the new page; DOM tests for the detail and the form dialog
- CHANGELOG entries and the frontend AGENTS.md scheduled-task note
* feat(frontend): live schedule card, run conversation provenance and header entry
- schedule_task results (create, update, pause, resume, trial, delete) become
an assistant:scheduled-task message group rendered as a live card: schedule,
next run, stop condition with the safety cap, goal, where results go, Run
once now, Pause/Resume and Open task; the card polls the task only while it
is on screen and shows deleted, scheduler-off and limits-exhausted states
- one card per task per assistant turn; the card group is not a turn boundary
for usage, branching, editing, outline or skill attribution
- a human message with deerflow_scheduled_origin stays a human group with
scheduledOrigin and renders as a "Scheduled run / Trial run" block with the
task instructions, stop condition and notes collapsed; the launched text is
never shown and the message is not editable
- readable step labels for schedule_task actions and stop_scheduled_task
- ThreadScheduledTasksButton replaces the header link: hidden without tasks,
shows the count, links to the task (or the chat filter), and a run's chat
links to its task
- recorded en-3 / zh-1 chat and run-thread fixtures; chat-card and run-thread
e2e specs; unit and DOM tests for parsing, grouping, turn boundaries, card,
run block, header button, step labels and reconnect hydration
- i18n: header.countLabelOne added, unused card.next removed; CHANGELOG and
frontend AGENTS.md notes
* fix(scheduler): address verification findings
Move the scheduled-task lifecycle guidance from backend/AGENTS.md into the persistence guide and leave a one-line pointer, so the effective AGENTS.md chains for agents/middlewares, sandbox and subagents stay under the 98304-byte hard limit enforced by scripts/check_agent_guidance.py in lint-check.
* fix(scheduler): address review findings
* fix(scheduled-tasks): polish from live acceptance
- zh schedule wording: an interval of one unit reads 每分钟 / 每小时 / 每秒,
and a saved interval of whole days reads 每天 / 每 2 天 (Every day / Every 2
days) instead of 每 1 分钟 or 每 24 小时; several units keep the number
- zh spacing: 已于{time} 满足 and 在{time} 的运行中 now leave a space between
the interpolated time and the Chinese after it; a copy test guards every
scheduled-task zh string against a placeholder glued to Chinese
- run summary: when the first line of the final reply only leads in to a list
(ends with ":" or "："), the list items follow it as plain text (bullets,
numbering, checkboxes, bold, code and links stripped), joined with "；" for
CJK text and "; " otherwise, within the 160-character cap
- a schedule_task result (create, update, pause, resume, trial, delete) that
arrives in a chat invalidates that chat's task list, the tasks list and the
named task, so the header button and cards update without waiting for the
idle poll; results already in the loaded history do not refetch
- e2e and unit fixtures are now the live acceptance recordings (DeepSeek
Flash, demo data, fictional names): live-weekday.json (create, trial, edit,
pause, resume in one zh chat) and live-minute.json (per-minute chat, run 1
and run 2, paused by the agent), fetched with GET from the demo Gateway and
trimmed; the hand-shaped en-3 / zh-1 fixtures are removed and the chat-card,
run-thread and tasks-page specs run in both locales over the recorded data
* docs(changelog): reference #6378 for the scheduled-tasks entries
* docs(agents): shorten the scheduled-task pointer to stay under the guidance limit
* test(scheduled-tasks): pin the live takes to their recorded time zone
The live fixtures were recorded in Asia/Shanghai, and an interval task
reads in the viewer's zone, so the 16:48 assertions failed on UTC CI
runners. Pin the browser zone in the two live-take e2e specs and the
zh live-take unit test.
* fix(scheduled-tasks): keep untouched end times and drop a chat filter the new task is not in
- The edit, duplicate and renew dialogs keep a stored end time's instant
while its field still shows it. A minute-precision wall value cannot
keep seconds or name the later 01:30 of a DST fall-back, so a
title-only edit moved the cap an hour earlier, and raising the run
limit re-sent the end time without its seconds.
- Creating or duplicating a task from a chat's task view drops the chat
filter when the new task does not belong to that chat, instead of
showing "This task isn't one of this chat's tasks".
* fix(scheduled-tasks): tie pause notices to the pause they describe
- Paused by agent links and times only the run named in last_error. When
that run is not loaded, the notice shows no link and the task's own time
instead of an earlier run that also asked to stop.
- The outcome notice reads the latest history page whichever page is
shown, so browsing older runs no longer swaps the auto-pause reason and
"Open latest run" to an earlier miss. It shares that page's request.
* fix(scheduled-tasks): three lifecycle gaps found while re-checking review classes
- The edit form compares a cron in the form its schedule input emits, so an
untouched "0 09 * * 0,6" is not re-sent as "0 9 * * 6,0" and a title-only
edit no longer re-arms or refuses a finished task.
- Chat resume asks for a timezone, as update does, when a naive end_at would
land in the UTC placeholder of a zone-free task.
- Re-arming a task clears a host pause marker, as resume does, so a later
manual pause no longer reads as "Paused by agent".
* fix(scheduled-tasks): keep the notice's latest-runs query from refreshing older pages
useLatestScheduledTaskRuns kept React Query's default refetch on mount,
focus and reconnect, so going offline and back while an older history page
was shown fetched the latest page again. Refreshing stays with the history,
which refreshes the latest page only while it is shown.
* test(scheduled-tasks): let the viewer's time carry its own day label
Between 16:00 and 24:00 UTC, UTC and Asia/Shanghai are on different days,
so the run prompt shows "Tomorrow 01:00 · Today 09:00 your time" and the
fixed "01:00 · 09:00 your time" assertion failed depending on the hour.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Hyeonsang Cho
|
2f833669bd
|
fix(make): name what make clean deletes and refuse under a live gateway (#6351)
* fix(make): name what make clean deletes and refuse under a live gateway `make help` described `make clean` as cleaning up "temporary files", but it deletes backend/.deer-flow: the local database, users, threads, uploads, memory, and secrets. Both Docker stacks mount that directory into the deer-flow-gateway container, and `make stop` only stops local services, so the data could be deleted under a running Gateway. Say what is deleted in the help text and before deleting, and refuse while the deer-flow-gateway container is running. The check runs before `make stop`, which would otherwise stop the running stack's sandboxes first; a missing or unreachable Docker passes. * docs(changelog): reference #6351 for the make clean guard * fix(make): share make clean's data list and pin stop before the delete The help line listed "database, threads, uploads, secrets" while the deletion notice also named users and memory. Define the list once in RUNTIME_DATA_CONTENTS and use it in both. Running sandbox containers left behind by `make down` are already stopped by `make stop` (serve.sh --stop reaps deer-flow-sandbox*), which runs after the guard and before the delete. Note that in the guard script, and pin the ordering: the stub stop now records that the data still exists. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
928ef24eac
|
fix(tui): wait for an interrupted run before sending to its thread (#6350)
* fix(tui): wait for an interrupted run before sending to its thread Textual cannot stop a thread worker, so after an interrupt the agent stream keeps running until its next event. A tool step already in progress still completes and checkpoints while a new prompt runs on the same thread; whichever run checkpoints last becomes the thread's state, so the new turn could drop out of the conversation history. Track when each worker actually starts and returns (Textual reports a cancelled thread worker finished at once) and reject sends to a thread whose interrupted worker is still running. Other threads stay usable, and a worker cancelled before it started holds nothing. * docs(changelog): reference #6350 for the interrupted TUI run fix --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
creed
|
2afd1211a1
|
feat(jina): honor bounded Retry-After recovery (#6352)
* feat(jina): honor bounded Retry-After recovery Closes #6237 Honor server-directed waits for 503 and validly hinted 429 responses within the existing attempt and time budgets. Preserve cancellation, default single-attempt behavior and the tool schema. Add deterministic protocol regressions and stabilize mock setup in the real deadline test. Signed-off-by: 97three <2212371308@qq.com> * test(jina): pin local backoff for hinted 502 and 504 Signed-off-by: 97three <2212371308@qq.com> --------- Signed-off-by: 97three <2212371308@qq.com> |
||
|
asts
|
b88d8db5fc
|
fix(mcp): share rotating OAuth state with durable task calls (#6342)
* fix(mcp): share rotating OAuth state with durable task calls * refactor(mcp): centralize task server selection --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
PeaceMaker-best
|
3606c4a098
|
feat(agents): add portable custom agent packages (#6127)
* feat(agents): add portable agent packages Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> * fix(agents): show readable package import validation errors --------- Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
5872d07e9a
|
fix(gateway): gate declared multi-instance deployments and require the redis stream bridge (#6328)
* fix(gateway): gate declared multi-instance deployments and require the redis stream bridge GATEWAY_WORKERS / WEB_CONCURRENCY only count the uvicorn workers of one process tree, so a Kubernetes Deployment with replicas > 1 (one worker per Pod) passed every multi-worker startup check while each Pod's startup orphan reconciliation wrote the other Pods' lease-less runs off as crashed on every rolling update. Add `deployment.multi_instance` as an explicit declaration that runs the same gate, plus a DEER_FLOW_MULTI_INSTANCE env override so deploy tooling can set the declaration from its replica count without editing config.yaml (wiring the Helm chart to export it is a separate change). Both the worker count and the declaration now also require a Redis stream bridge and refuse an explicit memory sandbox ownership store, since the memory bridge and ownership store are process-local. The agent-storage divergence warning, the inbound webhook dedupe warning and the WeChat QR-login guard honor the same declaration. * docs(changelog): link the multi-instance gate entry to #6328 * fix(ci): bump chart config_version to 53 and keep AGENTS.md under the size limits config.example.yaml moved to config_version 53 for the new deployment section, so the Helm chart's embedded config and its README example follow. The multi-instance notes in backend/AGENTS.md and backend/app/gateway/AGENTS.md shrink to one line each: main already sits within 55 bytes of the per-file hard limit for the gateway guide and within 163 bytes of the chain hard limit under backend/, and the full contract lives in backend/docs/CONFIGURATION.md. * fix(gateway): name every active knob in the multi-process rollback When GATEWAY_WORKERS > 1 and a multi-instance declaration are both active, the refusal offered only "Set GATEWAY_WORKERS=1", so an operator who followed it bounced through a second refusal on DEER_FLOW_MULTI_INSTANCE / deployment.multi_instance at the next start. The rollback now withdraws both knobs, and the browser-tools refusal takes the same path instead of the worker-count-only message. multi_instance_declaration() returns a frozen MultiInstanceDeclaration (source, value) with knob/rollback properties instead of a display string, so deps.py no longer branches on message prose. Tests pin the combined rollback text and that following it actually clears the gate. * docs(gateway): keep the gateway AGENTS.md under the size hard limit after merging main main grew backend/app/gateway/AGENTS.md by 30 bytes, which left it 25 bytes under the 49152-byte hard limit; the one-sentence multi-instance note from this branch pushed the merged file to 49164. Shorten that sentence by 18 bytes so scripts/check_agent_guidance.py passes on the merge result. * ci: re-run checks after test_stop_cancels_a_hung_driver_poll flaked on shard 4 tests/test_mcp_task_service.py::test_stop_cancels_a_hung_driver_poll timed out once on backend-unit-tests (4) for 2c16206ac. The MCP task service and its test are identical to main, main's run for cfe2023df passed, and 15 consecutive local runs pass; the test gives service.stop() a 1s budget while cancellation drain allows 5s, so a loaded runner can exceed it. Fork contributors cannot re-run a job, hence this empty commit. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
subhash7476
|
a558884c3a
|
perf(summarization): skip the summarizer LLM call for an input already proven no-op (#6318)
* perf(summarization): skip the summarizer LLM call for an input already proven no-op When the exact rendered summarizer prompt (plus candidate models) previously produced output byte-identical to the current summary_text, reuse that summary instead of calling the LLM. Compaction proceeds through the normal path, so messages, hooks and state transitions are unchanged. - Only proven no-op inputs are cached (bounded LRU, thread-safe); failed attempts are never cached and any cache error falls back to the LLM call. - The middleware:summarize event gains changes.llm_call_skipped. - Applies to both compact_state and acompact_state. Builds on the per-call telemetry in the P0 branch. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(summarization): keep fallback no-ops retryable --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Totoro
|
cdbcc0d9e9
|
feat(scheduled-tasks): show goal outcomes and end conditions in run history (#6326)
* docs(scheduler): document tool_enabled, goal notices and goal-evaluation change Follow-up to #6229: - CONFIGURATION.md: add scheduler.tool_enabled to the example and notes. - CONFIGURATION.md, README (en/zh/pt): unmet goal runs send a goal-unmet notice instead of a completion notice, and three unmet runs add an auto-pause notice; the docs still said only success/failed notify. - CHANGELOG (en/zh): the #6229 entry now states the default-on goal evaluation change for scheduled/webhook/autonomous goal runs and that evaluator usage counts toward run usage. * feat(scheduled-tasks): show goal outcomes and end conditions in run history Follow-up to #6229. The API already returns these fields; the tasks page did not show them, so a met goal read as plain "Success", an unmet reason code rendered as a red execution error, and the goal-unmet IM notice pointed to a workspace showing neither objective nor verdict. - Task detail shows the per-run goal, max automatic runs and end time when a conversation-created task has them. - Goal runs show "Goal met" (or met relying on stated assumptions); unmet runs show a localized reason plus the host code in neutral styling. The run whose agent asked to stop the schedule is marked. - Runs without a goal, and failed runs, render as before. Read-only and frontend-only: no API change. * docs(changelog): add #6326 scheduled-task goal outcome entry * fix(scheduled-tasks): show readable unmet reasons instead of raw codes Unmet runs now show only the localized reason; the host code moves to the tooltip so it can still be matched with the IM notice. Codes without a label are still shown verbatim. * fix(scheduler): show readable goal reasons instead of internal codes Task detail now localizes the host-written last_error values from goal and stop finalization (agent stop, automatic pause, unmet reason codes) under a matching label, and Chinese copy is tightened. Goal-unmet and auto-pause IM notices state the reason in words instead of printing the raw code; unknown codes are still not forwarded. * docs(changelog): describe readable goal reasons in #6326 entry * test(scheduler): pin host goal strings with a cross-language contract Address review on #6326: - Add contracts/scheduled_goal_notes_contract.json listing the task last_error strings and unmet reason codes the tasks page and IM notices translate. A backend test checks finalization constants, blocker and stand-down codes against it; a frontend test checks every listed value has a label. - Move the two last_error strings in finalization.py into constants. - Frontend reason lookup strips the blocked: prefix like the IM map and now covers goal_not_met_yet, with en/zh labels. --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
PeaceMaker-best
|
fcee85b822
|
feat(storage): persist viewed images as shared blobs (#6160)
Store validated image bytes in the configured content-addressed blob backend and resolve checkpointed refs before local compatibility paths. Refs: #4189 Signed-off-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> Co-authored-by: PeaceMaker-best <221849497+PeaceMaker-best@users.noreply.github.com> |
||
|
LingYi
|
3a2f768d4c
|
fix(tui): isolate callbacks from interrupted runs (#6302)
* fix(tui): isolate callbacks from interrupted runs * fix(tui): recover when agent worker creation fails --------- Co-authored-by: LingYi-Liang <270316896+LingYi-Liang@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
52046aa82e
|
fix(projects): convert shelf documents outside the SQLite write lock (#6305)
* fix(projects): convert shelf documents outside the SQLite write lock Lazy conversion ran pymupdf/markitdown inside the BEGIN IMMEDIATE transaction that serializes the derived/converted.md publish against trash and purge. On SQLite that lock is database-wide, so every unrelated writer (run status, thread metadata, the scheduler) waited for the whole conversion and failed with "database is locked" past busy_timeout (30s). Conversion now writes into .staging/ outside any transaction. The document row lock is taken only to revalidate the live row and atomically rename the output into place (publish_under_live_lock, renamed from convert_under_live_lock); a row trashed or purged meanwhile discards the staged output, so nothing is published after purge. A read-only probe skips converting rows that are already gone, a failed conversion is still revalidated so a purged row reports content_missing, and the conversion is drained on cancellation so its staging file is always removed. * docs(changelog): reference #6305 in the shelf conversion lock entry * fix(projects): share one conversion across concurrent first reads A first-read conversion holds a bounded file-IO pool thread for its whole duration. With the conversion moved out of the row lock, concurrent first reads of one document each converted on their own thread, so a burst of reads could starve unrelated offloaded file work. Route the conversion through a single-flight keyed by (user, document, project): the first reader leads and the rest wait on its result without holding a pool thread. The map holds concurrent.futures.Future values under a threading.Lock because the read tool also runs on the isolated subagent loop. Followers wait through asyncio.shield, so a cancelled follower cannot cancel the shared future, and a leader that is cancelled or raises hands the work to a follower instead of failing it. |
||
|
Hyeonsang Cho
|
56aa71c583
|
fix(runtime): keep renewing the lease while a staged terminal run finalizes (#6263)
* fix(runtime): keep renewing the lease while a staged terminal run finalizes With an event store, run_agent stages the terminal status in memory and commits it to the run store only after the journal flush, delivery receipt, workspace scan and duration checkpoint. _renew_leases selected runs by local status, so it stopped renewing as soon as the status was staged. A finalization longer than lease_seconds + grace_seconds let a peer, or this worker's own reconciler, claim the still-active row and record the successful run as an orphan error. The worker now marks RunRecord.terminal_commit_pending until the deferred commit is attempted, and lease renewal treats it like the existing scheduled-goal barrier: keep renewing, release on a confirmed own commit, and fence when a peer claims the row. A fenced run stops renewing. * docs(changelog): reference #6263 for the staged terminal lease renewal fix --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Daoyuan Li
|
4dad94dcc0
|
fix(tui): guard conversation switching and invalid resume references (#6249)
* fix(tui): guard conversation switching and invalid resume references * fix(tui): discard late stream actions after conversation switches --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Daoyuan Li
|
5c3329afad
|
fix(tui): preserve cursor and undo on unchanged history recalls (#6248)
* fix(tui): preserve composer on Up with empty history * fix(tui): preserve cursor and undo on identical history recalls |