mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 05:06:59 +00:00
feat(scheduled-tasks): signals, honest notices and fair queueing for server-started runs (RFC #6340, 2/2) (#6393)
* feat(scheduler): lifecycle rules, goal/limit parity and coded errors for scheduled tasks
- Migration 0031: nullable scheduled_tasks.unmet_streak_after_seq (internal
streak boundary) and stop_condition (the user's stop rule, own column).
- Stop rule is normalized (stop_rule.py) and appended only at launch; the
stored prompt never contains it. Wording adapts when scheduler.tool_enabled
is off.
- Willem 1: page-created tasks get goal evaluation, <standing_notes>, the
previous-run reference and own-schedule stop; request_stop and the
scheduled capability no longer require an origin conversation.
- Repository: single reactivation choke point raising
ScheduledTaskLimitsExhausted; streak boundary moves on goal/prompt/stop
condition/note changes (not on resume); goal-check failures neither count
nor reset the streak; pause of a finished task returns "finished"; trials
on a paused task keep the host pause marker.
- New repository reads: automatic_runs_used_for, active_run_status_for,
list_manageable_from_thread, thread_relation/thread_run on thread tasks,
task_ids_for_thread, run_number, and run_number/total_tokens/summary on
run history rows.
- REST: goal/max_runs/end_at/stop_condition on create and PATCH (null
clears, naive end_at = task timezone), resume recomputes from now with an
optional renewal body, once_time_passed, task_finished, limits_exhausted,
scheduler_not_running on create, trigger returns outcome/existing/thread_id,
task responses add automatic_runs_used and active_run_status.
- Coded errors {code, message, params} for routes, shared validation and the
chat capability (scheduled_task_errors.py), pinned by
contracts/scheduled_task_errors_contract.json; goal notes contract v2 with
check_failure_codes, scheduled_origin_key and host_run_errors (host_notes.py).
- /api/features adds scheduled_tasks {available, running, tool_enabled,
min_interval_seconds}; ScheduledTaskService.is_running.
- Docs: six README scheduler sections, API.md scheduled tasks section
(breaking error shape), CONFIGURATION.md, AGENTS notes, CHANGELOGs.
- Tests: migration 0031, stop rule, error contract, run listing, lifecycle
over HTTP, repository, service and capability cases; updated tests for the
new error shape, finished-task pause and head revision.
* feat(scheduler): conversation update/resume, run-conversation authority and readable launches
- schedule_task: update and resume actions, stop_condition and clear_fields
parameters, plain parameter types (value rules come back as coded results),
rewritten docstrings for schedule_task and stop_scheduled_task.
- Capability: update/resume share the REST validation and error codes;
interactive turns in a task's run conversation may manage that task;
scheduled runs keep only own-stop; grants require a running scheduler.
- Results: compact JSON-safe task views with local times, display card/text,
last run in list, timezone_source/browser_timezone, now_local; no ISO, user,
lease or origin fields; stop_instructions removed.
- Timezone: explicit > browser (context.client_timezone, validated in
start_run and never forwarded to the run config) > not needed for intervals
and offset run_at > timezone_required.
- Trial gate accepts natural direct requests after acknowledgements and
particles; trial results report thread_id and existing.
- Scheduled launches: stable message id scheduled-<task_run_id>, server-owned
deerflow_scheduled_origin metadata with the user-language parts, run thread
title for fresh-thread tasks.
- Worker records goal continuations in the verdict; lead prompt language rule
covers stored task text and scheduled runs.
- Docs: six READMEs, CONFIGURATION, API, runtime/tools AGENTS, CHANGELOGs.
* feat(frontend): scheduled-task core types, formatting, errors and glossary
- throwGatewayApiError now throws GatewayApiError (status, code, params,
rawMessage); message stays the string detail or detail.message
- scheduled-task types gain stop_condition, automatic_runs_used,
active_run_status, origin/notes, run_number/total_tokens/summary, thread
relation, trigger result, renewal and scheduled origin; isTaskBusy and
hasActiveRun helpers
- api: fetchScheduledTask, resume with optional renewal body, typed trigger
result and update payload with null clears
- hooks: polling at 3 s while a run is active (list, thread list, run
history), useScheduledTask with live gating and final 404, useInView,
localized error toasts; run history refreshes tasks when a run finishes
- new pure modules: errors.ts (contract code to localized copy), format.ts
(relative task times, viewer time, display zone), status.ts, actions.ts,
run-error.ts; describeTaskSchedule in cron.ts; goal-outcome adds unchecked
goal checks, describeTaskOutcome and parseStopRunId
- features: scheduled_tasks block with defaults when missing
- buildRunContext sends client_timezone
- titleOfThread takes a localized untitled label; breadcrumb names scheduled
tasks and agents; reference chips use the localized untitled label
- schedule input: shared browserTimeZone, labelled selects, aria-pressed
type toggles, interval minimum from the hint template
- i18n: all PR1 scheduled-task keys in en-US and zh-CN, glossary fixes,
recipe instructions moved into i18n
- e2e mocks follow the new task, trigger, resume, thread-task and features
shapes; expectNoRawIdentifiers helpers for e2e and DOM tests
- tests for errors, format, cron, status, actions, goal outcome, run errors,
hooks, run context, thread titles and features; scheduled e2e copy updated
* feat(frontend): list-first scheduled tasks page with outcomes and editable goals
- page.tsx is now a thin composition of components/workspace/scheduled-tasks:
status tabs (All/Active/Paused/Finished) and search over the task list,
?task_id= deep link with router.replace on selection, ?thread_id= chip
with Show all, empty state, scheduler-off notice and unavailable state
- task list rows show the schedule in words and one status line (next run,
running now, paused by agent, auto-paused, finished by its cap)
- task detail: status badge, actions from availableActions with reasons,
More actions menu (Duplicate, Copy task ID, Delete), outcome notice,
Runs / Stops when / Each run's goal / Does / Notes from chat / History,
origin chat link; success toasts for every action
- run rows: run number or Trial run, summary, goal and assumption badges,
continuations, tokens, Open chat link; raw errors and the evaluator
reason only behind Details
- create/edit/duplicate dialog with goal, stop condition, safety caps and
advanced agent/context options; PATCH sends only changed fields, null
clears a cap
- renew dialog raises or removes caps and resumes in one request; a
limits_exhausted Resume opens it
- removed i18n keys the old page used; added feedback.resumedNoTime
- recorded en-3 and zh-1 page fixtures; scheduled e2e specs rewritten for
the new page; DOM tests for the detail and the form dialog
- CHANGELOG entries and the frontend AGENTS.md scheduled-task note
* feat(frontend): live schedule card, run conversation provenance and header entry
- schedule_task results (create, update, pause, resume, trial, delete) become
an assistant:scheduled-task message group rendered as a live card: schedule,
next run, stop condition with the safety cap, goal, where results go, Run
once now, Pause/Resume and Open task; the card polls the task only while it
is on screen and shows deleted, scheduler-off and limits-exhausted states
- one card per task per assistant turn; the card group is not a turn boundary
for usage, branching, editing, outline or skill attribution
- a human message with deerflow_scheduled_origin stays a human group with
scheduledOrigin and renders as a "Scheduled run / Trial run" block with the
task instructions, stop condition and notes collapsed; the launched text is
never shown and the message is not editable
- readable step labels for schedule_task actions and stop_scheduled_task
- ThreadScheduledTasksButton replaces the header link: hidden without tasks,
shows the count, links to the task (or the chat filter), and a run's chat
links to its task
- recorded en-3 / zh-1 chat and run-thread fixtures; chat-card and run-thread
e2e specs; unit and DOM tests for parsing, grouping, turn boundaries, card,
run block, header button, step labels and reconnect hydration
- i18n: header.countLabelOne added, unused card.next removed; CHANGELOG and
frontend AGENTS.md notes
* fix(scheduler): address verification findings
Move the scheduled-task lifecycle guidance from backend/AGENTS.md into the persistence guide and leave a one-line pointer, so the effective AGENTS.md chains for agents/middlewares, sandbox and subagents stay under the 98304-byte hard limit enforced by scripts/check_agent_guidance.py in lint-check.
* fix(scheduler): address review findings
* fix(gateway): no-transform on SSE responses
All four text/event-stream responses (thread run create stream, join,
existing-run GET/POST stream, stateless create stream) now take their
headers from app/gateway/sse_headers.py and send
Cache-Control: no-cache, no-transform, so compressing proxies such as
Next's rewrite proxy stop buffering streams. Content-Location and
X-Accel-Buffering are unchanged; scheduled and MCP-notification runs keep
stream_mode=None.
Tests pin the headers on every SSE route and fail on a literal no-cache
header left in routers. Docs: API.md SSE headers note, CHANGELOGs.
* feat(scheduler): per-owner run cap and fair queue selection
New scheduler.max_concurrent_runs_per_user (default 2, 0 = off, effective
min(cap, max_concurrent_runs)) caps launching/running scheduled runs per task
owner. claim_queued_run checks it right after the global count under the same
budget lock, so it is atomic across workers and Pods. list_queued_runs ranks
same-thread FIFO heads per owner with ROW_NUMBER() OVER (PARTITION BY owner)
and orders the batch by that rank, so the drain is owner-fair even with the
cap off; owners at their cap drop out before the batch limit. Orphan rows
(task deleted while queued) are never held back by the cap.
Gateway lifespan passes the startup value; the service constructor default
stays 0. Docs: config.example.yaml (config_version 54), CONFIGURATION.md,
the six READMEs, backend AGENTS.md, CHANGELOGs (behavior change flagged).
The live-task quota is untouched.
* feat(scheduler): durable lifecycle events for the originating chat
When a chat-created schedule is paused by the agent, auto-paused or finishes,
finalization now reports task_stopped / task_paused / task_finished, only on a
live -> paused/finished transition. The new finish_task_at_end_condition
replaces the three inline "mark completed" blocks (admission rejected as ended,
claim-time skip, complete_if_ended), so every site emits exactly once.
The scheduler's finalization observer is always installed. It writes
scheduled_task_events rows for the originating chat in the same transaction as
the state change, deduplicated on (task_id, anchor, event), with a title
snapshot, stop condition, run number, last run outcome and the chat's newest
run as placement anchor. Lookups fall back on bad stored data; only database
errors propagate. Manual trials drop only run_* events. The IM step keeps
PR1's selection until B4.
New GET /api/threads/{thread_id}/scheduled-task-events (owner check, caller's
rows, oldest first); rows survive task deletion and go with the chat.
Migration 0032_activity_and_task_events adds the events table plus the
run-origin column/index and read-state tables used later, each object guarded
on its own. Contract scheduled_goal_notes v3 pins the lifecycle vocabulary.
Docs: API.md, migrations chain, backend AGENTS.md, six READMEs, CHANGELOGs.
* feat(scheduler): honest, localized and self-contained IM notices
Every scheduled-task IM notice is now enqueued by the finalization observer
in the transaction that records the outcome, one message per occurrence
(task_stopped > task_paused > task_finished > run_failed > run_unmet >
run_completed; a once task sends only its run event). The merged auto-pause
replaces the separate unmet + pause messages, agent stops and finishes now
notify, and recovery-finalized runs notify exactly once. Idle finishes use a
deterministic idle: key derived from the lifecycle anchor. The old
non-transactional enqueue in handle_run_completion is removed.
New app/channels/capabilities.py holds CHANNEL_CAPABILITIES with a
proactive_notifications flag (adds the missing qq entry); only push-capable
providers (WeCom) get rows, /api/channels/providers returns the flag, and the
worker ends legacy rows for other providers as terminal failed.
New app/scheduler/notification_text.py renders en-US/zh-CN text: title, what
happened (plus the last run's outcome), the tasks page's one-line run summary
(redacted) when the agent replied, and a closing line. No IDs, no links, no
public_base_url. Locale is the owner's new `locale` preference, else the new
channel_connections.notification_locale (default en-US).
Docs: config.example.yaml (config_version 55), CONFIGURATION.md, API.md,
channels AGENTS.md, six READMEs, CHANGELOGs (behavior change flagged).
* feat(gateway): thread origin, activity feed and read markers for server-created threads
Runs started by the server for a user (scheduler, IM channels, GitHub
agents, extension handles, MCP notifications) now carry a server-owned
metadata.deerflow_origin {kind, provider?, namespace?} (new harness module
deerflow.runtime.run_origin, pinned by contracts/thread_origin_contract.json).
It is stamped only from request.state.run_origin or by the internal channel
caller; run admission drops client copies from both metadata and
config.metadata, and thread create/patch strip it. RunRepository derives
runs.origin_kind on put and on create_thread_operation_atomic (agent runs
only), and gains latest_change().
New ThreadReadRepository keeps a per-user, never-decreasing read position
(thread_read_markers) and a per-user read clock (thread_read_versions).
POST /api/threads/{id}/read marks a thread read, thread search items carry
unread, and thread delete removes the markers. GET /api/thread-activity pages
the caller's run changes over the run-change clock, returns only threads
changed by server-originated runs plus read_version, and 503s without SQL;
/api/features reports thread_activity.available.
Docs: API.md, gateway AGENTS.md (condensed to stay under its budget),
CHANGELOGs.
* fix(scheduler): adapt PR2 to PR1 follow-ups
PR1 moved the scheduled-task lifecycle rules out of backend/AGENTS.md into
the persistence guide and grew the sandbox AGENTS.md chain to within 267
bytes of its hard limit. The per-owner queue-fairness detail now lives in
the persistence guide next to the lifecycle rules (backend/AGENTS.md keeps
the cap and a link), so check_agent_guidance passes again.
PR1's goal-outcome unit test pinned scheduled_goal_notes_contract version 2;
PR2 bumps it to 3 (lifecycle vocabulary only, the v2 keys are unchanged).
* fix(scheduled-tasks): polish from live acceptance
- zh schedule wording: an interval of one unit reads 每分钟 / 每小时 / 每秒,
and a saved interval of whole days reads 每天 / 每 2 天 (Every day / Every 2
days) instead of 每 1 分钟 or 每 24 小时; several units keep the number
- zh spacing: 已于{time} 满足 and 在{time} 的运行中 now leave a space between
the interpolated time and the Chinese after it; a copy test guards every
scheduled-task zh string against a placeholder glued to Chinese
- run summary: when the first line of the final reply only leads in to a list
(ends with ":" or ":"), the list items follow it as plain text (bullets,
numbering, checkboxes, bold, code and links stripped), joined with ";" for
CJK text and "; " otherwise, within the 160-character cap
- a schedule_task result (create, update, pause, resume, trial, delete) that
arrives in a chat invalidates that chat's task list, the tasks list and the
named task, so the header button and cards update without waiting for the
idle poll; results already in the loaded history do not refetch
- e2e and unit fixtures are now the live acceptance recordings (DeepSeek
Flash, demo data, fictional names): live-weekday.json (create, trial, edit,
pause, resume in one zh chat) and live-minute.json (per-minute chat, run 1
and run 2, paused by the agent), fetched with GET from the demo Gateway and
trimmed; the hand-shaped en-3 / zh-1 fixtures are removed and the chat-card,
run-thread and tasks-page specs run in both locales over the recorded data
* docs(changelog): reference #6378 for the scheduled-tasks entries
* docs(agents): shorten the scheduled-task pointer to stay under the guidance limit
* test(scheduled-tasks): pin the live takes to their recorded time zone
The live fixtures were recorded in Asia/Shanghai, and an interval task
reads in the viewer's zone, so the 16:48 assertions failed on UTC CI
runners. Pin the browser zone in the two live-take e2e specs and the
zh live-take unit test.
* feat(frontend): thread origin, activity polling, read markers and task events core
Frontend core for server-created threads and schedule lifecycle events
(PR2 F1); the sidebar markers (F2) and the chat event line (F3) build on it.
- core/threads/origin.ts: threadOriginOf reads metadata.deerflow_origin,
then channel_source (github -> GitHub origin), then a legacy
scheduled_task_id; key and kinds pinned to
contracts/thread_origin_contract.json. labelOfThreadOrigin and the now
exported labelOfChannelProvider(provider, t) use the new localized
threads.origin.providers names (飞书, 企业微信, ...), with the English map
(gaining GitHub) only as the fallback.
- core/threads/activity.ts: useThreadActivity polls /api/thread-activity
every 15 s (focus refetch, never in the background, off unless
features.thread_activity.available). The first response only seeds the
cursor; server-origin threads, a truncated page or a higher read_version
invalidate the thread lists, a scheduled thread also ["scheduled-tasks"];
an invalid cursor re-seeds. useMarkThreadRead debounces POST /read by 1 s,
skips a thread whose cached unread is false, patches the search and
infinite caches optimistically and stores the returned read_version so a
tab's own read does not refetch its lists. AgentThread gains unread.
- core/scheduled-tasks/events.ts: ScheduledTaskEvent, the events query
(["scheduled-tasks","thread-events",id], one fetch per saved chat, not
gated on the task list so a deleted task's lines stay, refetched when a
task of the chat changes status or finishes a run), describeTaskEvent
(en/zh copy, title and condition as segments, last-run suffix, See that
run / Open task) and placeTaskEvents (end of the after_run_id turn, else
the tail, ordered by created_at); lifecycle names pinned to contract v3.
- features: thread_activity (default unavailable); channels:
proactive_notifications on providers.
- Locale sync: the I18nProvider reports the UI language and a signed-in
preference session saves it as the new `locale` preference when the
account's differs (and on every switch), so IM notices follow the UI
language; nothing is written with auth disabled.
- i18n: threads.* (unread, origin labels, provider names),
scheduledTasks.events.*, history.waitingForSlot (now shown for queued run
rows), channels.scheduledUpdates.*.
- e2e mocks: thread activity feed and controller, POST /read, the events
route, providers with proactive_notifications, search unread, features
thread_activity (off by default). Preference e2e mocks return the UI
locale so they keep asserting only their own edits.
- Docs: frontend AGENTS.md, CHANGELOGs.
* fix(scheduled-tasks): keep untouched end times and drop a chat filter the new task is not in
- The edit, duplicate and renew dialogs keep a stored end time's instant
while its field still shows it. A minute-precision wall value cannot
keep seconds or name the later 01:30 of a DST fall-back, so a
title-only edit moved the cap an hour earlier, and raising the run
limit re-sent the end time without its seconds.
- Creating or duplicating a task from a chat's task view drops the chat
filter when the new task does not belong to that chat, instead of
showing "This task isn't one of this chat's tasks".
* feat(frontend): origin markers and unread state in the thread lists
Server-created threads (PR2 F2) now say who created them and whether they
are new, in the sidebar and on the Chats page, and reach an open sidebar
without a reload.
- ThreadOriginIcon (thread-channel-source.tsx): clock for a scheduled run,
the provider mark for IM threads, the GitHub mark (new in
ChannelProviderIcon) for GitHub threads, a puzzle piece for extensions;
the localized "Scheduled run" / "From Feishu" / "From GitHub" / "From an
extension" label is its accessible name and tooltip. ThreadChannelIcon
stays exported as a wrapper, and the provider chip tooltip is localized
(no more "Feishu channel").
- Unread dot after the title (aria-hidden dot plus sr-only "Unread"); the
row aria-label becomes "{title}, unread" / "{title},未读". Never shown for
the open thread.
- WorkspaceSidebar mounts useThreadActivity once.
- useMarkOpenThreadRead (components/workspace/chats): the default and
custom-agent chat pages mark the thread read once its metadata has
loaded, when its cached unread flips to true while open, when the tab
becomes visible, and from onFinish (a send or a joined run ended). Each
trigger goes through useMarkThreadRead, so it is debounced and a no-op
while the cached unread is false.
- Tests: thread-origin-icon and recent-chat-list DOM tests (en/zh, active
row, no raw identifiers), use-mark-open-thread-read DOM tests,
e2e thread-activity.spec.ts (new scheduled run appears unread within one
fake-clock poll without a reload, opening it posts /read and clears the
dot, a read on another device clears it, IM and GitHub markers, zh
labels, the Chats page, no polling when unavailable);
thread-history.spec.ts now finds the localized origin icon.
- Docs: frontend AGENTS.md, CHANGELOGs.
* feat(frontend): schedule event line in the originating chat and channel notice support
The chat that created a schedule now says when the schedule paused or
ended, and the channel cards say which apps receive scheduled task updates
(PR2 F3).
- ScheduledTaskEventLine (mockup m2, marker 4): a dashed, muted row with
the describeTaskEvent text (title in <strong>, stop condition clamped to
one line with the full text in its tooltip, last-run suffix), the time
via formatTaskTime and a right-aligned "See that run" (run chat) or
"Open task" link. PauseCircle for paused, CircleCheck for finished and
once-done, CircleX for once-failed. role="note", aria-label "Scheduled
task update" / "定时任务通知", data-task-id and data-event-id only.
- MessageList takes scheduledTaskEvents and renders placeTaskEvents output
through a new VirtualMessageList.renderAfterGroup, so each line sits at
the end of its after_run_id turn inside the group's measured row (also
when the list virtualizes); lines with nowhere to go render at the tail.
- Both chat pages load useThreadScheduledTaskEvents (not gated on the task
list, so the lines stay after a task is deleted). The custom-agent page
passes its agent so "See that run" opens the run on that agent's route.
- ChannelScheduledUpdates: "Scheduled task updates: sent here" / "... not
available for this app yet" (定时任务通知:会发送到这里 / 此应用暂不支持)
on the Settings provider cards and under each sidebar channel row; nothing
when an older backend does not report proactive_notifications.
- e2e mock controller: setScheduledTasks.
- Tests: scheduled-task-event-line and channel-scheduled-updates DOM tests
(every kind in en and zh, hrefs, clamp, no raw identifiers, Chinese
title), MessageList placement DOM tests; e2e
scheduled-task-chat-events.spec.ts (the line appears with the card's
"Paused by agent" without a reload, See that run opens the run chat,
anchored vs tail placement, two events in time order, the line survives
task deletion and Open task shows the task is gone; en-3 and zh-1) and
channels.spec.ts (WeCom sent here, Feishu not yet; en and zh).
- Docs: frontend AGENTS.md, CHANGELOGs.
* fix(scheduled-tasks): tie pause notices to the pause they describe
- Paused by agent links and times only the run named in last_error. When
that run is not loaded, the notice shows no link and the task's own time
instead of an earlier run that also asked to stop.
- The outcome notice reads the latest history page whichever page is
shown, so browsing older runs no longer swaps the auto-pause reason and
"Open latest run" to an earlier miss. It shares that page's request.
* fix(scheduler): address PR2 review findings
- Event lines: hold an event whose anchor run is on an unloaded history
page instead of placing it at the bottom (hasMoreHistory); tail fallback
only for fully loaded (branched/pruned) history or no anchor.
- Idle-finish anchor follows its reason: end:<iso> only for an end_at
finish, seq:<n> otherwise, so a later end-time finish is not deduplicated
away by an earlier max_runs finish.
- Observer takes FOR SHARE on the originating threads_meta row, so a racing
thread delete cannot leave an orphan event line.
- One taskPagePath (core), re-exported by the card.
- Notification locales derived from NotificationLocale; /preferences uses it.
- IM notices drop generic reasons (goal_not_met_yet, unknown) instead of
repeating the sentence.
- Localized provider names on Settings cards, sidebar rows and setup dialog.
- zh event lines: no space after a Chinese title (Latin-aware gap).
- Unread row label keeps the origin marker; chats page keeps the time as a
description.
- Sidebar shows the scheduled-updates line only for connected apps;
Settings says "available after you connect" for unconnected push apps.
- Event line keeps a real space before the time.
* fix(scheduled-tasks): three lifecycle gaps found while re-checking review classes
- The edit form compares a cron in the form its schedule input emits, so an
untouched "0 09 * * 0,6" is not re-sent as "0 9 * * 6,0" and a title-only
edit no longer re-arms or refuses a finished task.
- Chat resume asks for a timezone, as update does, when a naive end_at would
land in the UTC placeholder of a zone-free task.
- Re-arming a task clears a host pause marker, as resume does, so a later
manual pause no longer reads as "Paused by agent".
* test(scheduler): consolidate PR2 backend test setup
- Fold the repeated task, origin-chat and observer setup of the
lifecycle-event repository tests into one lifecycle_task builder.
- Build the per-owner-cap services through one _capped_service builder
that keeps every cap explicit at the call site.
- Table the five SSE route header checks into one parametrized test with
the same routes, bodies and Content-Location expectations.
- Reuse bind_wecom and hoist repeated local imports in the event tests.
Every case and assertion is kept; collected cases are unchanged.
* test(frontend): consolidate PR2 thread activity and origin test setup
Extract shared builders and helpers in the frontend tests PR2 adds, with
every case, input and assertion kept:
- activity.test.ts: seededClient/apply helpers and one LIST_KEYS constant
replace the per-test seed, spy and apply boilerplate.
- mark-read.dom.test.tsx: newClient() and advance() replace the repeated
QueryClient wrapper and fake-timer act blocks.
- origin.test.ts: the English and Chinese label checks become tables; the
no-English-name check now runs on every Chinese row.
- thread-activity.spec.ts: a scheduledRun() builder replaces four inline
copies of the scheduled run thread.
- channels.spec.ts: the PR2 block uses setLocaleCookie and one card helper.
Collected cases are unchanged (unit 211, e2e 60); no test was removed.
* fix(scheduled-tasks): keep the notice's latest-runs query from refreshing older pages
useLatestScheduledTaskRuns kept React Query's default refetch on mount,
focus and reconnect, so going offline and back while an older history page
was shown fetched the latest page again. Refreshing stays with the history,
which refreshes the latest page only while it is shown.
* fix(scheduler): two gaps found in live acceptance
- A fresh scheduled run thread is named when it launches. The worker
copied the title to the thread list only when the run ended, so the
sidebar, which now shows the new run thread at once, listed it as
untitled while it ran.
- A task capped at one run finishes with "its one run is done" /
"唯一一次运行已完成" in the chat event line and the IM notice, instead
of "all 1 runs are done".
* test(scheduled-tasks): let the viewer's time carry its own day label
Between 16:00 and 24:00 UTC, UTC and Asia/Shanghai are on different days,
so the run prompt shows "Tomorrow 01:00 · Today 09:00 your time" and the
fixed "01:00 · 09:00 your time" assertion failed depending on the hour.
* test(scheduler): expect the one-run finish copy in service notices
* fix(gateway): create a scheduled run thread with its title
Pre-setting the name after start_run raced the thread row, which the run
worker creates asynchronously: live, one of two run threads still reached
the sidebar untitled. Run admission now creates a new thread with the title
its run input carries, which the worker would copy only at run end.
* docs(changelog): reference #6393 for the scheduled-task signal entries
* fix(scheduled-tasks): route event-line run links by the agent that ran them
The originating chat's agent was used for "See that run", but a task's
agent can change after the chat created it, so the link could open the run
chat on another agent's route. Lifecycle events now record run_agent_name,
the task's agent when the run finished (edits are refused while a run is
active), and the line routes by it.
* fix(scheduled-tasks): load up to the route's maximum of chat event lines
The chat loads its lifecycle events once, and the route's default of 50
would silently drop the oldest lines of a chat with many schedules. Ask
for the maximum, 200.
* fix(threads): keep activity reads and the first cursor in step with the lists
- Accepting the first activity cursor refreshes the thread lists once. The
lists load on their own, possibly before a run the seed already counts,
and later polls start after it, so such a thread never appeared.
- A local read whose returned read clock jumps past that one read also
covers reads on other devices; the lists are refreshed instead of the
clock silently skipping those threads' stale unread dots.
- An invalid cursor re-seeds through the same path, so it refreshes once.
* docs(agents): bring the middlewares guidance chain back under the hard limit
#6378 and #6363 each fit the budget, but together the middlewares AGENTS.md
chain on main is 98,425 bytes, 121 over the 98,304 hard limit, so Lint
Check fails on main. Move the scheduled-task recovery, launch-update and
queue-timeout rules from backend/AGENTS.md into the persistence guide's
lifecycle section, which backend/AGENTS.md already points to. The chain is
now 98,067 bytes.
* fix(helm): bump the chart's config_version to 55
config.example.yaml moved to 55 in this PR, so the Helm chart's
validate-chart drift check failed. The chart's embedded config needs no
field changes: the new scheduler and channel keys have defaults.
---------
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
This commit is contained in:
parent
51a33c79b3
commit
35cdcab1c5
116
CHANGELOG.md
116
CHANGELOG.md
@ -99,6 +99,96 @@ This release closes that milestone with **439 merged pull requests**.
|
||||
智能体; no `lead_agent` in labels). Untitled chats and the Scheduled tasks and
|
||||
Agents breadcrumbs follow the interface language. ([#6378])
|
||||
|
||||
- **scheduler:** Scheduled runs now share execution slots fairly between task
|
||||
owners. A new `scheduler.max_concurrent_runs_per_user` (default `2`, `0` =
|
||||
off, never more than `max_concurrent_runs`) caps how many scheduled runs one
|
||||
owner has launching or running at a time, manual "run now" included; it is
|
||||
checked in the same atomic claim, under the same database lock, as the global
|
||||
cap. The waiting queue is drained owner by owner whatever the cap: owners at
|
||||
their cap drop out before the batch limit, so one owner's backlog no longer
|
||||
pushes another owner's run out of the drain batch. Same-thread FIFO and the
|
||||
order within one owner are unchanged. A run that waits past
|
||||
`scheduler.queue_timeout_seconds` is skipped and shows "Skipped: it waited
|
||||
too long for a free slot" in the run history. ([#6393])
|
||||
|
||||
**Behavior change:** one owner now runs at most 2 scheduled runs at a time by
|
||||
default (before: up to `max_concurrent_runs`, default 3).
|
||||
`scheduler.max_concurrent_runs_per_user: 0` restores the old behavior.
|
||||
|
||||
- **scheduler:** When a schedule created in a chat is paused by the agent
|
||||
(its stop condition was met), is paused automatically after three missed
|
||||
goals, or finishes (all `max_runs` done, `end_at` passed, or a one-time task
|
||||
ran), a durable lifecycle event is recorded for that chat in the same
|
||||
database transaction as the state change, and recovery after a crash or a
|
||||
lost lease never records it twice. New `GET
|
||||
/api/threads/{thread_id}/scheduled-task-events` lists them (title snapshot,
|
||||
stop condition, run number and the last run's outcome); the rows stay after
|
||||
the task is deleted and go with the chat. The event names are pinned in
|
||||
`contracts/scheduled_goal_notes_contract.json` (version 3). Migration
|
||||
`0032_activity_and_task_events` adds this table and the run-origin and
|
||||
read-state schema used by later changes; `runs.origin_kind` stays NULL for
|
||||
existing runs. ([#6393])
|
||||
|
||||
- **scheduler:** Scheduled-task IM notices are honest, localized and
|
||||
self-contained. Each occurrence sends at most one message, queued by the
|
||||
finalization observer in the same transaction that records the outcome: a
|
||||
pause by the agent (`task_stopped`, new), the automatic pause (now one
|
||||
merged message instead of a goal-missed notice plus a pause notice), the
|
||||
task finishing (`task_finished`, new: all `max_runs` done or `end_at`
|
||||
reached), or the run's own outcome; a pause or finish still says how the
|
||||
last run went. Only apps with proactive push (WeCom today) get notices: the
|
||||
new `app/channels/capabilities.py` declares `proactive_notifications` per
|
||||
provider (and adds the missing `qq` entry), `GET /api/channels/providers`
|
||||
returns it, and other providers get no outbox rows. The text has the task
|
||||
title, what happened, the tasks page's one-line run summary when the agent
|
||||
replied (redacted) and "Open DeerFlow → Scheduled tasks for details.", with
|
||||
no IDs and no links, in the owner's web UI language (new `locale` user
|
||||
preference, `en-US` or `zh-CN`) or else the new
|
||||
`channel_connections.notification_locale` (default `en-US`). No public base
|
||||
URL setting is added: most deployments run on localhost or a LAN, where a
|
||||
link would be dead on the phone that receives it. ([#6393])
|
||||
|
||||
**Behavior change:** runs finalized by crash or lease recovery now notify
|
||||
once (they used to be silent); one message per occurrence replaces separate
|
||||
goal-missed and pause messages; agent stops and finishes now notify; notice
|
||||
text changed and no longer shows the task or run ID. Rows already queued for
|
||||
providers without proactive push end once as `failed` without retries.
|
||||
|
||||
- **frontend:** Threads the server creates now show who created them and
|
||||
whether they are new. In the sidebar and on the Chats page, a scheduled run
|
||||
has a clock icon ("Scheduled run"), an IM thread its app's icon ("From
|
||||
Feishu"), a GitHub thread the GitHub mark and an extension thread a puzzle
|
||||
icon. An unread dot follows the title until the thread is opened (never on
|
||||
the open thread; announced as "{title}, unread"). New and changed
|
||||
server-created threads appear within 15 seconds without a reload, opening
|
||||
one clears its dot on every device, and the open chat stays read while a
|
||||
scheduled run in it updates. Nothing polls when the Gateway has no thread
|
||||
activity (memory persistence). ([#6393])
|
||||
|
||||
- **frontend:** The chat that created a schedule now shows one line when the
|
||||
schedule pauses or ends ("Release checklist was paused by the agent. Stop
|
||||
condition met: …", "… was paused automatically: 3 runs in a row missed the
|
||||
goal.", "… finished: all 5 runs are done. The last run failed.", "… has
|
||||
run."), at the end of the turn it followed, with the time and "See that run"
|
||||
or "Open task". It appears without a reload when the task's state changes,
|
||||
and stays after the task is deleted. Settings → Channels and the sidebar's
|
||||
channel list say for each app whether scheduled task updates are sent there
|
||||
("sent here" for WeCom, "not available for this app yet" for the others).
|
||||
([#6393])
|
||||
|
||||
- **scheduler:** The web app keeps the account's `locale` preference equal to
|
||||
its interface language (after sign-in with session auth and on every
|
||||
language switch), so scheduled-task IM notices arrive in the language the
|
||||
user reads DeerFlow in; with auth disabled nothing is written and
|
||||
`notification_locale` applies. A queued run in the run history now reads
|
||||
"Waiting for a free slot". The frontend core for server-created threads
|
||||
lands here too: origin markers from `deerflow_origin` (IM provider names
|
||||
localized, `channel_source` and legacy scheduled threads still recognized),
|
||||
activity polling that refetches the thread lists only when a
|
||||
server-originated thread changed or a thread was read on another device,
|
||||
debounced read marking, and the per-chat lifecycle events with their
|
||||
placement in the conversation. ([#6393])
|
||||
|
||||
- **scheduler:** The tasks page shows the per-run goal and end conditions of
|
||||
conversation-created tasks. Run history shows whether a goal was met,
|
||||
including when it relied on stated assumptions; an unmet run shows a readable
|
||||
@ -118,6 +208,23 @@ This release closes that milestone with **439 merged pull requests**.
|
||||
|
||||
#### Agents & runtime
|
||||
|
||||
- **gateway:** Threads the server creates for you can be noticed without a
|
||||
reload. Runs started by a schedule, an IM channel, a GitHub agent, an
|
||||
extension or an MCP notification now carry a server-owned
|
||||
`metadata.deerflow_origin` (`{kind, provider?, namespace?}`, pinned by the new
|
||||
`contracts/thread_origin_contract.json`) and a denormalized
|
||||
`runs.origin_kind`; a thread created for such a run keeps the same marker.
|
||||
Clients cannot set it: thread create/patch and run admission strip client
|
||||
copies. The new `GET /api/thread-activity` feed pages the caller's run
|
||||
changes over the existing run-change clock and returns only threads changed
|
||||
by server-originated runs, plus a per-user `read_version`; an idle poll is one
|
||||
index seek. `POST /api/threads/{thread_id}/read` stores a per-user,
|
||||
never-decreasing read position, and thread search items gain `unread`. Only
|
||||
the caller's own server-originated runs make a thread unread; interactive
|
||||
runs, other users' runs in shared threads and runs from before the upgrade
|
||||
never do. `GET /api/features` reports `thread_activity.available` (SQL
|
||||
persistence only). ([#6393])
|
||||
|
||||
- **uploads:** Add stable cursor pagination to the `list_uploaded_files`
|
||||
discovery tool. With more than 100 historical uploads matching the same
|
||||
filters the tool could only return the first page, giving an agent no
|
||||
@ -726,6 +833,14 @@ This release closes that milestone with **439 merged pull requests**.
|
||||
128-token budget before later query terms. Tokens without a letter or digit are
|
||||
now dropped, as the no-jieba fallback and the FTS5 query filter already did.
|
||||
([#6388])
|
||||
- **gateway:** Run streams now arrive incrementally behind compressing proxies.
|
||||
Every SSE response (`POST /api/threads/{id}/runs/stream`, `GET .../join`,
|
||||
`GET`/`POST .../runs/{run_id}/stream` and `POST /api/runs/stream`) sends
|
||||
`Cache-Control: no-cache, no-transform` from one shared helper instead of
|
||||
`no-cache`, so proxies that compress responses, such as the Next.js rewrite
|
||||
proxy used by `pnpm start` without nginx, no longer buffer the stream and
|
||||
deliver it in bursts. `X-Accel-Buffering: no` and `Content-Location` are
|
||||
unchanged, and nginx deployments behave as before. ([#6393])
|
||||
- **config:** Every Gateway process that shares one `extensions_config.json`
|
||||
now sees the MCP and skill changes made by another one. The parsed file was
|
||||
cached once per process and only the process that handled the write
|
||||
@ -8982,5 +9097,6 @@ with **180 merged pull requests** since the first 2.0 milestone tag.
|
||||
[#6378]: https://github.com/bytedance/deer-flow/pull/6378
|
||||
[#6386]: https://github.com/bytedance/deer-flow/pull/6386
|
||||
[#6388]: https://github.com/bytedance/deer-flow/pull/6388
|
||||
[#6393]: https://github.com/bytedance/deer-flow/pull/6393
|
||||
[#6400]: https://github.com/bytedance/deer-flow/pull/6400
|
||||
[#6401]: https://github.com/bytedance/deer-flow/pull/6401
|
||||
|
||||
@ -80,6 +80,72 @@
|
||||
中英文界面统一使用一套定时任务用语(定时任务、运行、智能体,标签中不再出现
|
||||
`lead_agent`)。未命名对话以及“定时任务”“智能体”面包屑也会跟随界面语言。([#6378])
|
||||
|
||||
- **调度器:** 定时运行的执行位置现在在任务所有者之间公平分配。新增配置
|
||||
`scheduler.max_concurrent_runs_per_user`(默认 `2`,设为 `0` 即关闭,且不会超过
|
||||
`max_concurrent_runs`),限制同一所有者同时处于启动中或运行中的定时运行数量,
|
||||
手动“立即运行”也计算在内;它与全局上限在同一次原子领取中、持有同一把数据库锁时
|
||||
检查。无论是否设置这一上限,等待队列都按所有者轮流取出:已达上限的所有者在批量
|
||||
上限生效前就被排除,因此一个人积压的运行不会再把别人的运行挤出本轮批次。同一
|
||||
对话内的先进先出顺序,以及同一所有者内部的顺序都保持不变。等待超过
|
||||
`scheduler.queue_timeout_seconds` 的运行会被跳过,运行记录显示“已跳过:等待空闲
|
||||
位置的时间过长”。([#6393])
|
||||
|
||||
**行为变更:** 默认情况下,同一所有者同时最多运行 2 个定时运行(此前最多为
|
||||
`max_concurrent_runs`,默认 3)。设置 `scheduler.max_concurrent_runs_per_user: 0`
|
||||
可恢复原来的行为。
|
||||
|
||||
- **调度器:** 在对话中创建的调度由智能体暂停(停止条件已满足)、在连续 3 次未达成
|
||||
目标后被自动暂停,或已结束(`max_runs` 次运行全部完成、已过 `end_at`,或单次任务
|
||||
已运行)时,会为该对话记录一条持久的生命周期事件;它与状态变更在同一个数据库事务中
|
||||
提交,进程崩溃或租约失效后的恢复也不会重复记录。新增
|
||||
`GET /api/threads/{thread_id}/scheduled-task-events` 列出这些事件(含任务标题快照、
|
||||
停止条件、运行序号和最后一次运行的结果);删除任务后事件仍保留,删除对话时一并删除。
|
||||
事件名称固定在 `contracts/scheduled_goal_notes_contract.json`(版本 3)中。迁移
|
||||
`0032_activity_and_task_events` 新增这张表,以及后续改动要用的运行来源和已读状态
|
||||
结构;已有运行的 `runs.origin_kind` 保持为 NULL。([#6393])
|
||||
|
||||
- **调度器:** 定时任务的 IM 通知改为如实、按用户语言、可单独读懂。每次执行最多发
|
||||
一条消息,由结束处理的观察者在记录执行结果的同一个事务中入队:由智能体暂停
|
||||
(新增 `task_stopped`)、自动暂停(合并为一条,不再分别发“目标未达成”和“已暂停”)、
|
||||
任务已结束(新增 `task_finished`:`max_runs` 次运行全部完成或已到 `end_at`),
|
||||
否则是这次运行本身的结果;暂停或结束的消息仍会说明最后一次运行的情况。只有支持
|
||||
主动推送的应用(目前是企业微信)会收到通知:新增的 `app/channels/capabilities.py`
|
||||
为每个提供方声明 `proactive_notifications`(并补上缺失的 `qq` 条目),
|
||||
`GET /api/channels/providers` 返回该字段,其它提供方不再写入 outbox。消息包含任务
|
||||
标题、发生了什么、智能体有回复时附上与任务页一致的一行运行摘要(已脱敏),最后是
|
||||
“在 DeerFlow 的定时任务页查看详情。”,不含 ID 和链接;语言取任务所有者在网页界面中
|
||||
选择的语言(新增用户偏好 `locale`,`en-US` 或 `zh-CN`),未设置时用新增的
|
||||
`channel_connections.notification_locale`(默认 `en-US`)。不新增公网访问地址配置:
|
||||
多数部署运行在 localhost 或内网,链接在收到消息的手机上打不开。([#6393])
|
||||
|
||||
**行为变更:** 进程崩溃或租约失效后恢复的执行现在会通知一次(以前不通知);每次执行
|
||||
只发一条消息,不再分别发送“目标未达成”和“已暂停”;智能体暂停和任务结束现在会通知;
|
||||
通知文案已更改,不再显示任务或运行 ID。已为不支持主动推送的提供方排队的旧消息会
|
||||
直接落为 `failed`,不再重试。
|
||||
|
||||
- **前端:** 服务端创建的对话现在会标明来源以及是否有新内容。在侧边栏和“对话”页中,
|
||||
定时运行显示时钟图标(“定时运行”),IM 对话显示对应应用的图标(“来自飞书”),GitHub
|
||||
对话显示 GitHub 标志,扩展创建的对话显示拼图图标。打开之前,标题后会有一个未读圆点
|
||||
(当前打开的对话不显示;读屏读作“{标题},未读”)。新建或有变化的服务端对话会在 15 秒内
|
||||
出现,无需刷新;打开后所有设备上的圆点都会消失,定时运行在当前打开的对话里更新时也会
|
||||
保持已读。Gateway 不提供对话活动(内存持久化)时不会轮询。([#6393])
|
||||
|
||||
- **前端:** 创建定时任务的对话里,任务暂停或结束时会多出一行说明(“发布清单 已由智能体
|
||||
暂停。停止条件已满足:……”“……已自动暂停:连续 3 次未达成目标。”“……已结束:5 次运行
|
||||
已全部完成。最后一次运行出错了。”“……已运行。”),位于它之前那轮对话的末尾,带有时间
|
||||
和“查看那次运行”或“查看任务”。任务状态变化后无需刷新即可出现,删除任务后仍会保留。
|
||||
“设置 → 渠道”和侧边栏的渠道列表会标明每个应用能否收到定时任务通知(企业微信“会发送
|
||||
到这里”,其他应用“此应用暂不支持”)。([#6393])
|
||||
|
||||
- **调度器:** 网页端会让账号的 `locale` 偏好始终与界面语言一致(使用会话认证登录后、
|
||||
以及每次切换语言时写入),定时任务的 IM 通知因此使用用户在 DeerFlow 中看到的语言;
|
||||
关闭认证时不写入,改用 `notification_locale`。执行记录里排队中的运行现在显示
|
||||
“正在等待空闲位置”。服务端创建的对话所需的前端基础能力也在此加入:根据
|
||||
`deerflow_origin` 判断来源(IM 提供方名称已本地化,仍识别 `channel_source` 和旧的定时
|
||||
运行对话);活动轮询只在服务端发起的对话有变化、或在其他设备上读过对话时才重新拉取
|
||||
对话列表;防抖的已读标记;以及每个对话的定时任务生命周期事件和它们在对话中的位置。
|
||||
([#6393])
|
||||
|
||||
- **调度器:** 任务页显示对话创建任务的每次执行目标和结束条件。执行记录显示目标
|
||||
是否达成(含依赖已声明假设的情况);未达成的执行以中性样式显示可读的原因,不再
|
||||
按执行错误标红显示原始代码;请求停止调度的那次执行会被标出。任务详情里由 Agent
|
||||
@ -95,6 +161,19 @@
|
||||
|
||||
#### 智能体与运行时
|
||||
|
||||
- **网关:** 服务端替你创建的对话无需刷新即可被发现。由定时任务、IM 渠道、GitHub
|
||||
智能体、扩展或 MCP 通知发起的运行,现在带有服务端维护的
|
||||
`metadata.deerflow_origin`(`{kind, provider?, namespace?}`,由新增的
|
||||
`contracts/thread_origin_contract.json` 固定),并冗余写入 `runs.origin_kind`;
|
||||
为这类运行新建的对话也带同样的标记。客户端无法设置它:创建、修改对话以及发起运行时
|
||||
都会去掉客户端传入的副本。新增的 `GET /api/thread-activity` 基于现有的运行变更时钟
|
||||
分页读取调用者的运行变化,只返回因服务端发起的运行而变化的对话,并附带每个用户的
|
||||
`read_version`;空闲轮询只需一次索引查找。`POST /api/threads/{thread_id}/read`
|
||||
按用户保存只增不减的已读位置,对话搜索结果新增 `unread`。只有调用者自己的服务端
|
||||
发起的运行才会让对话变为未读;交互式运行、共享对话中其他用户的运行以及升级前的
|
||||
运行都不会。`GET /api/features` 返回 `thread_activity.available`(仅 SQL 持久化
|
||||
可用)。([#6393])
|
||||
|
||||
- **上传:** 为 `list_uploaded_files` 发现工具新增稳定的光标分页。
|
||||
当同一批过滤条件匹配到超过 100 条历史上传时,
|
||||
该工具此前只能返回第一页,智能体没有任何续页途径。现在可选的
|
||||
@ -633,6 +712,13 @@
|
||||
切成独立 token,因此同样含逗号的查询与无关事实也会得到非零分,近似去重的相似度被抬高,
|
||||
标点还会在后续查询词之前耗尽 128 个 token 的预算。现在不含字母或数字的 token 会被丢弃,
|
||||
与无 jieba 的回退分词及 FTS5 查询过滤保持一致。([#6388])
|
||||
- **网关:** 经过会压缩响应的代理时,运行流现在会逐帧到达。所有 SSE 响应
|
||||
(`POST /api/threads/{id}/runs/stream`、`GET .../join`、
|
||||
`GET`/`POST .../runs/{run_id}/stream` 与 `POST /api/runs/stream`)改由同一个
|
||||
辅助函数发送 `Cache-Control: no-cache, no-transform`,不再只是 `no-cache`。
|
||||
因此会压缩响应的代理(例如不经 nginx、直接 `pnpm start` 时的 Next.js 改写代理)
|
||||
不再先缓冲整段流、再成批送出。`X-Accel-Buffering: no` 与 `Content-Location`
|
||||
保持不变,使用 nginx 的部署不受影响。([#6393])
|
||||
- **中间件:** 工具输出预算不再让子智能体证据丢失失败的 shell 退出码。长度介于
|
||||
`externalize_min_chars`(12,000)与沙箱上限(20,000)之间的 bash 结果会被替换为
|
||||
以 `Access:` 页脚结尾的预览,结尾的 `Exit Code: N` 不再位于最后,
|
||||
@ -7397,5 +7483,6 @@ DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写,核心包含子
|
||||
[#6370]: https://github.com/bytedance/deer-flow/pull/6370
|
||||
[#6378]: https://github.com/bytedance/deer-flow/pull/6378
|
||||
[#6388]: https://github.com/bytedance/deer-flow/pull/6388
|
||||
[#6393]: https://github.com/bytedance/deer-flow/pull/6393
|
||||
[#6400]: https://github.com/bytedance/deer-flow/pull/6400
|
||||
[#6401]: https://github.com/bytedance/deer-flow/pull/6401
|
||||
|
||||
17
README.md
17
README.md
@ -2470,11 +2470,12 @@ Current MVP capabilities:
|
||||
- Editing or duplicating an interval task preserves its saved cadence until the interval is explicitly changed, including sub-minute intervals allowed by the operator's scheduler configuration
|
||||
- Run background scheduled executions as non-interactive DeerFlow runs (`ask_clarification` is not exposed there)
|
||||
- Persist a due execution as `queued` when its reused thread or the global execution budget is busy, then launch it when capacity is available; queued occurrences survive Gateway restarts and fail after `scheduler.queue_timeout_seconds`
|
||||
- Share execution slots fairly between task owners: one owner may have at most `scheduler.max_concurrent_runs_per_user` scheduled runs starting or running at a time (default 2, never more than `max_concurrent_runs`; `0` turns the per-owner cap off), and the waiting queue is drained owner by owner, so one owner's backlog never holds back another owner's run. A run that waits longer than `scheduler.queue_timeout_seconds` is skipped, and its history reads "Skipped: it waited too long for a free slot"
|
||||
- Freeze a task's definition while an occurrence is `queued`, `launching`, or `running`, so a durable occurrence cannot silently pick up a different prompt, thread, or schedule; transitioning a task to paused or deleting it cancels an existing waiting occurrence, while `launching`/`running` work must finish before those mutations are retried and an explicit manual trigger may still wait and run without resuming a paused schedule
|
||||
- Pause, resume, trigger, inspect history, and delete tasks
|
||||
- Search task titles or prompts, combined with status/type filters and the current thread scope.
|
||||
- Execute scheduled work through the normal DeerFlow run lifecycle
|
||||
- When `channel_connections.enabled: true`, push a summary to the task owner's connected IM identities when a scheduled run finishes as success or failed (outbox + delivery worker). Goal tasks send a goal-unmet notice for an `unmet` occurrence instead, plus an auto-pause notice when three unmet occurrences pause the task. Manual "run now" and interrupts stay silent, and so do occurrences that end without a finished run (launch error, queue timeout, restart recovery). Channel/transport outages park deliveries without exhausting retries, for up to about a day; platform rejections retry for roughly 15 minutes before settling as `failed`. An identity you disconnect while a delivery is waiting is never pushed to: the row is dropped as `failed`. Proactive push is currently implemented for WeCom; other connected providers enqueue but fail until they grow a `send_notification` path.
|
||||
- When `channel_connections.enabled: true`, send scheduled task updates to the task owner's connected IM identities (outbox + delivery worker) on apps that support proactive push, which today is WeCom; Settings shows for each app whether updates are sent there, and other apps get none. Each occurrence sends at most one message: a finished, failed or goal-missed run, the automatic pause after three missed goals, a pause by the agent (its stop condition was met) or the task finishing (all `max_runs` done, `end_at` reached); when several apply, the pause or finish wins and still says how the last run went. A one-time task sends only its run's outcome. The message is queued in the same database transaction that records the outcome, so runs finalized after a crash or a lost lease notify exactly once. It reads on its own, in your web UI language (else `channel_connections.notification_locale`): the task title, what happened, a one-line result when the agent replied, and "Open DeerFlow → Scheduled tasks for details.", with no IDs and no links. Plain manual "run now" trials and interrupted runs stay silent, and so do occurrences that end without a finished run (launch error, queue timeout, interrupted by a restart). Channel/transport outages park deliveries without exhausting retries, for up to about a day; platform rejections retry for roughly 15 minutes before settling as `failed`. An identity you disconnect while a delivery is waiting is never pushed to: the row is dropped as `failed`.
|
||||
- Browse execution history in pages of 50; older pages pause automatic refresh, with an explicit return to the latest runs. Counts appear only after a successful read; loading and failed reads are not reported as zero runs.
|
||||
|
||||
**Filter execution history through the API**
|
||||
@ -2530,9 +2531,12 @@ schedule, the next run and the stop condition in plain text.
|
||||
keep the saved zone; the browser zone never changes an existing task.
|
||||
- **Where results appear.** Each run posts its result in a new chat of its own,
|
||||
titled “{task} · {local time}”, or in the originating chat when the task runs
|
||||
there. Nothing else is posted back to the originating conversation. A run chat
|
||||
shows the task instructions as one collapsed “Task instructions” block under
|
||||
the run's header instead of a long user message.
|
||||
there. When the schedule is paused by the agent, is paused automatically or
|
||||
finishes, the originating chat shows one line where the conversation stood,
|
||||
with a link to that run or to the task; nothing else is posted back to it.
|
||||
The line stays after the task is deleted. A run chat shows the task
|
||||
instructions as one collapsed “Task instructions” block under the run's
|
||||
header instead of a long user message.
|
||||
- **Language.** The agent writes the title, instructions and stop condition in
|
||||
your language, and scheduled runs answer in the language of the instructions.
|
||||
|
||||
@ -2550,8 +2554,9 @@ Three eligible automatic unmet occurrences pause a recurring task. Accepted
|
||||
success resets the streak, including a success relying on disclosed assumptions;
|
||||
manual trials, interruption, execution failure, external waiting and
|
||||
goal-check failures do not advance it. Resume retains the streak, so another eligible unmet occurrence can
|
||||
pause the task again. Existing notification bindings receive goal-unmet and
|
||||
auto-pause notices through the same durable outbox; manual trials stay silent.
|
||||
pause the task again. Connected IM apps with proactive push receive one notice
|
||||
per occurrence through the same durable outbox (goal missed, auto-paused, paused
|
||||
by the agent, finished); plain manual trials stay silent.
|
||||
|
||||
You can ask the agent in a conversation that manages the task to save an
|
||||
explicit note for future runs (at most 10 notes of 500 characters). Fresh
|
||||
|
||||
@ -742,6 +742,8 @@ Activez le polling en arrière-plan avec `config.yaml -> scheduler.enabled`. Le
|
||||
- Les échecs de vérification de l'objectif (évaluateur en échec, ou conversation modifiée pendant la vérification) ne comptent pas pour la pause automatique après trois échecs et ne remettent pas le compteur à zéro. Modifier l'objectif, les instructions ou la condition d'arrêt, ou ajouter une note, repart d'un nouveau compte ; la reprise le conserve.
|
||||
- Tant que le planificateur de ce processus Gateway ne tourne pas, la création d'une tâche (y compris la duplication) renvoie `409 scheduler_not_running`. `GET /api/features` expose `scheduled_tasks.available`, `running`, `tool_enabled` et `min_interval_seconds`.
|
||||
- Les erreurs de `/api/scheduled-tasks*` ont la forme `{"detail": {"code", "message", "params"}}` ; voir [`backend/docs/API.md`](backend/docs/API.md#scheduled-tasks) et `contracts/scheduled_task_errors_contract.json`.
|
||||
- Les créneaux d'exécution sont partagés équitablement entre les propriétaires des tâches : un même propriétaire peut avoir au plus `scheduler.max_concurrent_runs_per_user` exécutions planifiées en démarrage ou en cours à la fois (2 par défaut, jamais plus que `max_concurrent_runs` ; `0` désactive ce plafond par propriétaire), et la file d'attente est vidée propriétaire par propriétaire, si bien que l'arriéré de l'un ne retarde jamais l'exécution d'un autre. Une exécution qui attend plus longtemps que `scheduler.queue_timeout_seconds` est ignorée, et l'historique indique qu'elle a attendu trop longtemps un créneau libre.
|
||||
- Avec `channel_connections.enabled: true`, les mises à jour des tâches planifiées sont envoyées aux identités IM connectées du propriétaire, uniquement sur les applications qui gèrent l'envoi proactif (WeCom aujourd'hui) ; les paramètres indiquent pour chaque application si les mises à jour y sont envoyées. Chaque occurrence envoie au plus un message (exécution terminée, en échec ou objectif non atteint, pause automatique, pause par l'agent, fin de la tâche), mis en file dans la même transaction que le résultat, donc une seule fois même après une reprise sur panne. Le message se lit seul, dans la langue de votre interface web (sinon `channel_connections.notification_locale`), sans identifiants ni liens. Les essais manuels simples et les exécutions interrompues ne notifient pas.
|
||||
|
||||
### Créer des planifications dans une conversation
|
||||
|
||||
@ -750,7 +752,7 @@ Activez `scheduler.enabled: true` et `scheduler.tool_enabled: true`, puis redém
|
||||
- **Tâches gérées par une conversation.** Celles qui y ont été créées et, dans une conversation d'exécution (le chat où une exécution planifiée a publié son résultat), la tâche de cette exécution : « mets-la en pause » ou « passe-la à 10 h » y fonctionnent aussi. Cela ne vaut que pour les messages que vous envoyez ; une exécution planifiée ne peut que mettre en pause sa propre planification avec `stop_scheduled_task`.
|
||||
- **Une modification garde la tâche.** Changer l'horaire, les instructions, l'objectif, la condition d'arrêt ou le plafond de sécurité est un `update` de la même tâche : son identifiant et son historique sont conservés. `resume` relance une tâche en pause ou terminée, sans exécution de rattrapage. Si le plafond est atteint, l'agent demande comment renouveler la limite épuisée (relever `max_runs` ou le supprimer ; repousser `end_at` ou le supprimer), puis l'envoie avec la reprise.
|
||||
- **Fuseau horaire.** Un fuseau que vous nommez l'emporte. Sinon, une nouvelle tâche utilise le fuseau du navigateur que l'application web envoie avec chaque message (`context.client_timezone`, lu uniquement pour cela), et le résultat indique le fuseau retenu. Les intervalles et les heures ponctuelles avec décalage UTC n'en ont pas besoin ; pour un cron ou une heure locale ponctuelle sans fuseau connu (par exemple depuis une messagerie), l'agent pose la question. Une modification garde le fuseau enregistré ; le fuseau du navigateur ne change jamais une tâche existante.
|
||||
- **Où apparaissent les résultats.** Chaque exécution publie son résultat dans un nouveau chat intitulé « {tâche} · {heure locale} », ou dans le chat d'origine lorsque la tâche s'y exécute. Rien d'autre n'est renvoyé dans la conversation d'origine. Le chat d'exécution affiche les instructions de la tâche dans un bloc replié « Instructions de la tâche » sous l'en-tête de l'exécution.
|
||||
- **Où apparaissent les résultats.** Chaque exécution publie son résultat dans un nouveau chat intitulé « {tâche} · {heure locale} », ou dans le chat d'origine lorsque la tâche s'y exécute. Quand le planning est mis en pause par l'agent, mis en pause automatiquement ou se termine, le chat d'origine affiche une ligne à l'endroit où en était la conversation, avec un lien vers cette exécution ou vers la tâche ; rien d'autre n'y est renvoyé. La ligne reste après la suppression de la tâche. Le chat d'exécution affiche les instructions de la tâche dans un bloc replié « Instructions de la tâche » sous l'en-tête de l'exécution.
|
||||
- **Langue.** L'agent rédige le titre, les instructions et la condition d'arrêt dans votre langue, et les exécutions planifiées répondent dans la langue des instructions.
|
||||
- **Essai.** Demandez-le directement, par exemple « Run it now » ou « 先跑一次吧 » ; le bouton **Run once now** de la carte fait de même. Un simple « yes », une mention de la tâche ou une demande citée ou conditionnelle ne lance pas d'exécution payante. Un essai ne compte pas dans `max_runs`.
|
||||
|
||||
|
||||
@ -729,6 +729,8 @@ DeerFlowには現在、ワークスペース内でファーストクラスのス
|
||||
- 目標を確認できなかった場合(評価器の失敗、または確認中に会話が変わった場合)は、3回連続未達成による自動一時停止に数えられず、カウントもリセットされません。目標・指示・停止条件の変更やメモの追加でカウントは新しく始まり、再開ではそのまま保たれます。
|
||||
- このGatewayプロセスのスケジューラーが動いていない間、タスクの作成(複製を含む)は`409 scheduler_not_running`を返します。`GET /api/features`は`scheduled_tasks.available`、`running`、`tool_enabled`、`min_interval_seconds`を返します。
|
||||
- `/api/scheduled-tasks*`のエラーは`{"detail": {"code", "message", "params"}}`の形式です。[`backend/docs/API.md`](backend/docs/API.md#scheduled-tasks)と`contracts/scheduled_task_errors_contract.json`を参照してください。
|
||||
- 実行枠はタスクの所有者間で公平に分配されます。1人の所有者が同時に起動中または実行中にできるスケジュール実行は最大`scheduler.max_concurrent_runs_per_user`件です(既定値は2で、`max_concurrent_runs`を超えません。`0`にすると所有者ごとの上限は無効になります)。待機キューは所有者ごとに順番に取り出されるため、1人に溜まった実行が他の人の実行を妨げることはありません。`scheduler.queue_timeout_seconds`を超えて待った実行はスキップされ、履歴には空き枠を長く待ちすぎたと表示されます。
|
||||
- `channel_connections.enabled: true`のとき、スケジュールタスクの更新は、プロアクティブ送信に対応したアプリ(現在はWeComのみ)で接続済みの所有者のIMアカウントに送られます。設定画面では、アプリごとに更新が送られるかどうかが表示されます。1回の実行で送られるメッセージは最大1件(実行の完了・失敗・目標未達成、自動一時停止、エージェントによる一時停止、タスクの終了)で、結果を記録するのと同じトランザクションでキューに入るため、障害からの復旧後も1回だけ通知されます。メッセージは単体で読める内容で、Web画面の表示言語(未設定なら`channel_connections.notification_locale`)で書かれ、IDやリンクは含みません。単なる手動の試し実行と中断された実行は通知されません。
|
||||
|
||||
### 会話からスケジュールを作成する
|
||||
|
||||
@ -737,7 +739,7 @@ DeerFlowには現在、ワークスペース内でファーストクラスのス
|
||||
- **会話が管理できるタスク。** その会話で作成したタスクに加え、実行用の会話(スケジュール実行が結果を投稿した会話)では、その実行が属するタスクも管理できます。そこで「これを止めて」「10 時に変えて」と頼めます。これはあなたが送るメッセージにのみ適用され、スケジュール実行そのものは `stop_scheduled_task` で自分のスケジュールを一時停止することしかできません。
|
||||
- **変更してもタスクは同じ。** 時刻、指示、目標、停止条件、安全上限の変更は同じタスクへの `update` なので、タスク ID と実行履歴は保たれます。`resume` は一時停止中または終了済みのタスクを再開し、取りこぼした実行の追い上げはしません。上限を使い切っている場合、エージェントは使い切った方の上限をどうするか(`max_runs` を増やすか外す、`end_at` を延ばすか外す)を確認し、それを再開と一緒に送ります。
|
||||
- **タイムゾーン。** あなたが指定したタイムゾーンが優先されます。指定がなければ、新しいタスクには Web アプリが各メッセージと一緒に送るブラウザのタイムゾーン(`context.client_timezone`、この用途にのみ使用)が使われ、結果にどのゾーンを使ったかが示されます。間隔実行と UTC オフセット付きの単発時刻にはタイムゾーンは不要です。cron やオフセットなしの単発のローカル時刻でゾーンが分からない場合(IM など)は、エージェントが確認します。変更時は保存済みのゾーンを使い、ブラウザのゾーンで既存タスクが変わることはありません。
|
||||
- **結果の表示先。** 各実行は「{タスク名} · {現地時刻}」という題名の新しい会話に結果を投稿します。タスクが元の会話で実行される場合はその会話に投稿します。それ以外に元の会話へ投稿されるものはありません。実行用の会話では、タスクの指示が実行ヘッダーの下に折りたたまれた「タスクの指示」ブロックとして表示されます。
|
||||
- **結果の表示先。** 各実行は「{タスク名} · {現地時刻}」という題名の新しい会話に結果を投稿します。タスクが元の会話で実行される場合はその会話に投稿します。スケジュールがエージェントによって一時停止されたとき、自動で一時停止されたとき、または終了したときは、元の会話のその時点の位置に 1 行の通知が表示され、その実行またはタスクへのリンクが付きます。それ以外に元の会話へ投稿されるものはありません。タスクを削除してもこの行は残ります。実行用の会話では、タスクの指示が実行ヘッダーの下に折りたたまれた「タスクの指示」ブロックとして表示されます。
|
||||
- **言語。** エージェントはタイトル、指示、停止条件をあなたの言語で書き、スケジュール実行は指示の言語で返答します。
|
||||
- **試運転。** 「Run it now」や「先跑一次吧」のように直接頼んでください。カードの **Run once now** ボタンでも同じです。単なる「yes」、タスクへの言及、引用や条件付きの依頼では有料の実行は始まりません。試運転は `max_runs` に数えられません。
|
||||
|
||||
|
||||
@ -1502,11 +1502,12 @@ Capacidades atuais do MVP:
|
||||
- Editar ou duplicar uma tarefa de intervalo preserva a cadência salva até que o intervalo seja alterado explicitamente, incluindo intervalos abaixo de um minuto permitidos pela configuração do scheduler do operador
|
||||
- Rodar as execuções agendadas em background como runs não interativos do DeerFlow (`ask_clarification` não é exposto ali)
|
||||
- Persistir uma execução devida como `queued` quando a thread reutilizada ou o orçamento global de execução está ocupado e iniciá-la quando houver capacidade; as ocorrências na fila sobrevivem a reinícios do Gateway e falham depois de `scheduler.queue_timeout_seconds`
|
||||
- Dividir os slots de execução de forma justa entre os donos das tarefas: cada dono pode ter no máximo `scheduler.max_concurrent_runs_per_user` execuções agendadas iniciando ou em andamento ao mesmo tempo (padrão 2, nunca acima de `max_concurrent_runs`; `0` desativa o limite por dono), e a fila de espera é drenada dono a dono, então o acúmulo de um dono nunca segura a execução de outro. Uma execução que espera mais que `scheduler.queue_timeout_seconds` é pulada, e o histórico mostra que ela esperou tempo demais por um slot livre
|
||||
- Congelar a definição de uma tarefa enquanto uma ocorrência está `queued`, `launching` ou `running`, para que uma ocorrência durável não pegue em silêncio outro prompt, thread ou agendamento; passar uma tarefa para pausada ou excluí-la cancela uma ocorrência em espera, enquanto o trabalho em `launching`/`running` precisa terminar antes que essas mutações sejam tentadas de novo, e um disparo manual explícito ainda pode esperar e rodar sem retomar um agendamento pausado
|
||||
- Pausar, retomar, disparar, inspecionar o histórico e excluir tarefas
|
||||
- Buscar em títulos ou prompts de tarefas, em combinação com filtros de status/tipo e com o escopo da thread atual.
|
||||
- Executar o trabalho agendado pelo ciclo de vida normal de runs do DeerFlow
|
||||
- Com `channel_connections.enabled: true`, enviar um resumo às identidades de IM conectadas do dono da tarefa quando um run agendado termina como sucesso ou falha (outbox + worker de entrega). Tarefas com goal enviam, em vez disso, um aviso de goal não atingido para uma ocorrência `unmet`, além de um aviso de pausa automática quando três ocorrências não atingidas pausam a tarefa. O "run now" manual e as interrupções ficam em silêncio, assim como as ocorrências que terminam sem um run concluído (erro de lançamento, timeout de fila, recuperação após reinício). Indisponibilidades de canal/transporte deixam as entregas estacionadas sem esgotar as tentativas, por até cerca de um dia; rejeições da plataforma são tentadas de novo por aproximadamente 15 minutos antes de se fixarem como `failed`. Uma identidade que você desconecta enquanto uma entrega está em espera nunca recebe o envio: a linha é descartada como `failed`. O push proativo está implementado hoje para o WeCom; outros provedores conectados enfileiram, mas falham até ganharem um caminho `send_notification`.
|
||||
- Com `channel_connections.enabled: true`, enviar atualizações de tarefas agendadas às identidades de IM conectadas do dono da tarefa (outbox + worker de entrega) nos apps com push proativo, hoje apenas o WeCom; as Configurações mostram em cada app se as atualizações são enviadas para lá, e os demais apps não recebem nada. Cada ocorrência envia no máximo uma mensagem: um run concluído, com falha ou com goal não atingido, a pausa automática após três goals não atingidos, uma pausa feita pelo agente (a condição de parada foi atendida) ou o fim da tarefa (todos os `max_runs` feitos, `end_at` alcançado); quando vários se aplicam, vale a pausa ou o fim, que ainda informa como foi o último run. Uma tarefa única envia só o resultado do seu run. A mensagem entra na fila na mesma transação de banco que registra o resultado, então runs finalizados após um crash ou uma lease perdida notificam exatamente uma vez. Ela se entende sozinha, no idioma da sua interface web (senão `channel_connections.notification_locale`): o título da tarefa, o que aconteceu, uma linha de resultado quando o agente respondeu e "Open DeerFlow → Scheduled tasks for details." (ou o equivalente em chinês), sem IDs e sem links. Testes manuais simples ("run now") e runs interrompidos ficam em silêncio, assim como as ocorrências que terminam sem um run concluído (erro de lançamento, timeout de fila, interrupção por reinício). Indisponibilidades de canal/transporte deixam as entregas estacionadas sem esgotar as tentativas, por até cerca de um dia; rejeições da plataforma são tentadas de novo por aproximadamente 15 minutos antes de se fixarem como `failed`. Uma identidade que você desconecta enquanto uma entrega está em espera nunca recebe o envio: a linha é descartada como `failed`.
|
||||
- Navegar pelo histórico de execuções em páginas de 50; páginas mais antigas pausam a atualização automática, com um retorno explícito às execuções mais recentes. As contagens só aparecem depois de uma leitura bem-sucedida; carregamento e leituras com falha não são reportados como zero execuções.
|
||||
|
||||
**Filtrar o histórico de execuções pela API**
|
||||
@ -1540,12 +1541,12 @@ Defina `scheduler.enabled: true` e `scheduler.tool_enabled: true` e reinicie o G
|
||||
- **Quais tarefas uma conversa gerencia.** As tarefas criadas nela e, numa conversa de execução (o chat em que uma execução agendada publicou o resultado), a tarefa a que essa execução pertence: "pause isto" ou "mude para as 10:00" também funcionam ali. Isso vale apenas para os turnos que você envia; uma execução agendada só pode pausar o próprio agendamento com `stop_scheduled_task`.
|
||||
- **Alterações mantêm a tarefa.** Mudar o horário, as instruções, o goal, a condição de parada ou o limite de segurança é um `update` da mesma tarefa, então o ID e o histórico de execuções permanecem. `resume` retoma uma tarefa pausada ou encerrada sem execução de recuperação. Se o limite já foi usado, o agente pergunta como renovar o limite esgotado (aumentar ou remover `max_runs`; adiar ou remover `end_at`) e envia isso junto com a retomada.
|
||||
- **Fuso horário.** Um fuso que você indicar tem prioridade. Caso contrário, uma tarefa nova usa o fuso do navegador que o app web envia com cada mensagem (`context.client_timezone`, lido só para isso), e o resultado informa qual fuso foi usado. Intervalos e horários únicos com deslocamento UTC não precisam de fuso; para um cron ou um horário local único sem fuso conhecido (por exemplo, vindo do IM), o agente pergunta. Alterações mantêm o fuso salvo; o fuso do navegador nunca altera uma tarefa existente.
|
||||
- **Onde aparecem os resultados.** Cada execução publica o resultado em um chat novo próprio, com o título "{tarefa} · {hora local}", ou no chat de origem quando a tarefa roda nele. Nada mais é publicado de volta na conversa de origem. O chat da execução mostra as instruções da tarefa como um bloco recolhido "Instruções da tarefa" sob o cabeçalho da execução, em vez de uma longa mensagem do usuário.
|
||||
- **Onde aparecem os resultados.** Cada execução publica o resultado em um chat novo próprio, com o título "{tarefa} · {hora local}", ou no chat de origem quando a tarefa roda nele. Quando o agendamento é pausado pelo agente, pausado automaticamente ou termina, o chat de origem mostra uma linha no ponto em que a conversa estava, com um link para essa execução ou para a tarefa; nada mais é publicado de volta nela. A linha continua lá depois que a tarefa é excluída. O chat da execução mostra as instruções da tarefa como um bloco recolhido "Instruções da tarefa" sob o cabeçalho da execução, em vez de uma longa mensagem do usuário.
|
||||
- **Idioma.** O agente escreve o título, as instruções e a condição de parada no seu idioma, e as execuções agendadas respondem no idioma das instruções.
|
||||
|
||||
Novas tarefas usam por padrão uma conversa nova a cada ocorrência. Um `goal_objective` configurado vale só para aquela ocorrência: o sucesso não interrompe um agendamento recorrente. O agente em execução pode pedir `stop_scheduled_task` para o próprio agendamento quando a sua condição de parada for atingida; o pedido passa a valer durante a finalização terminal. `max_runs` conta apenas os lançamentos automáticos, e `end_at` define um prazo. Qualquer uma das condições de término tem precedência sobre um pedido de pausa. Agendamentos com frequência menor que uma hora criados pela ferramenta exigem uma condição de término; cada dono pode manter no máximo 20 tarefas vivas criadas pela ferramenta, incluindo as pausadas.
|
||||
|
||||
Uma ocorrência não atendida é registrada como `unmet`, diferente de uma falha de execução. Três ocorrências automáticas não atendidas e elegíveis pausam uma tarefa recorrente. Um sucesso aceito zera a sequência, inclusive um sucesso apoiado em suposições declaradas; testes manuais, interrupção, falha de execução, espera externa e falhas na verificação do goal não a fazem avançar. Retomar mantém a sequência, então outra ocorrência não atendida elegível pode pausar a tarefa de novo. Os vínculos de notificação existentes recebem os avisos de goal não atendido e de pausa automática pelo mesmo outbox durável; os testes manuais ficam em silêncio.
|
||||
Uma ocorrência não atendida é registrada como `unmet`, diferente de uma falha de execução. Três ocorrências automáticas não atendidas e elegíveis pausam uma tarefa recorrente. Um sucesso aceito zera a sequência, inclusive um sucesso apoiado em suposições declaradas; testes manuais, interrupção, falha de execução, espera externa e falhas na verificação do goal não a fazem avançar. Retomar mantém a sequência, então outra ocorrência não atendida elegível pode pausar a tarefa de novo. Os apps de IM conectados com push proativo recebem um aviso por ocorrência pelo mesmo outbox durável (goal não atingido, pausa automática, pausa pelo agente, fim da tarefa); os testes manuais simples ficam em silêncio.
|
||||
|
||||
Você pode pedir ao agente, numa conversa que gerencia a tarefa, que salve uma nota explícita para execuções futuras (no máximo 10 notas de 500 caracteres). Execuções recorrentes novas podem ler a ocorrência executada anterior pelo `read_conversation`, que é opt-in, com as mesmas verificações de dono e de permissão de leitura. Isso oferece uma referência à fonte, não um resumo automático nem uma postagem de volta no chat de origem.
|
||||
|
||||
|
||||
@ -666,6 +666,8 @@ client.clear_goal("thread-1")
|
||||
- Сбои проверки цели (ошибка оценщика или изменение диалога во время проверки) не засчитываются в автоматическую паузу после трёх промахов подряд и не сбрасывают счётчик. Изменение цели, инструкций или условия остановки, а также новая заметка начинают отсчёт заново; возобновление его сохраняет.
|
||||
- Пока планировщик этого процесса Gateway не работает, создание задачи (включая дублирование) возвращает `409 scheduler_not_running`. `GET /api/features` сообщает `scheduled_tasks.available`, `running`, `tool_enabled` и `min_interval_seconds`.
|
||||
- Ошибки `/api/scheduled-tasks*` имеют вид `{"detail": {"code", "message", "params"}}`; см. [`backend/docs/API.md`](backend/docs/API.md#scheduled-tasks) и `contracts/scheduled_task_errors_contract.json`.
|
||||
- Слоты выполнения честно делятся между владельцами задач: у одного владельца одновременно может запускаться или выполняться не более `scheduler.max_concurrent_runs_per_user` запланированных запусков (по умолчанию 2, но не больше `max_concurrent_runs`; `0` отключает лимит на владельца), а очередь ожидания разбирается по владельцам поочерёдно, поэтому накопившиеся запуски одного владельца никогда не задерживают запуск другого. Запуск, который ждёт дольше `scheduler.queue_timeout_seconds`, пропускается, и в истории видно, что он слишком долго ждал свободного слота.
|
||||
- При `channel_connections.enabled: true` обновления запланированных задач отправляются в подключённые IM-аккаунты владельца, но только в приложения с проактивной отправкой (сейчас это WeCom); в настройках для каждого приложения указано, приходят ли туда обновления. Каждый запуск даёт не больше одного сообщения (запуск завершён, завершился ошибкой или не достиг цели, автоматическая пауза, пауза по решению агента, задача завершена). Сообщение ставится в очередь в той же транзакции, что и результат, поэтому даже после восстановления после сбоя оно приходит один раз. Оно понятно без контекста, написано на языке веб-интерфейса (иначе `channel_connections.notification_locale`) и не содержит идентификаторов и ссылок. Простые ручные пробные запуски и прерванные запуски уведомлений не дают.
|
||||
|
||||
### Создание расписаний в диалоге
|
||||
|
||||
@ -674,7 +676,7 @@ client.clear_goal("thread-1")
|
||||
- **Какими задачами управляет диалог.** Задачами, созданными в нём, а в диалоге запуска (чате, куда запланированный запуск опубликовал результат) — ещё и задачей этого запуска: там тоже работают «приостанови это» или «перенеси на 10:00». Это относится только к вашим сообщениям; сам запланированный запуск может лишь приостановить своё расписание через `stop_scheduled_task`.
|
||||
- **Изменение сохраняет задачу.** Смена времени, инструкций, цели, условия остановки или страховочного лимита — это `update` той же задачи, поэтому её ID и история запусков сохраняются. `resume` возобновляет приостановленную или завершённую задачу без догоняющего запуска. Если лимит исчерпан, агент спрашивает, как обновить исчерпанный лимит (увеличить или снять `max_runs`; перенести или снять `end_at`), и передаёт это вместе с возобновлением.
|
||||
- **Часовой пояс.** Названный вами пояс имеет приоритет. Иначе новая задача использует часовой пояс браузера, который веб-приложение отправляет с каждым сообщением (`context.client_timezone`, читается только для этого), и в результате указано, какой пояс выбран. Интервалам и разовым моментам со смещением UTC пояс не нужен; для cron или локального разового времени без известного пояса (например, из мессенджера) агент спросит. При изменении сохраняется прежний пояс; пояс браузера никогда не меняет существующую задачу.
|
||||
- **Где появляются результаты.** Каждый запуск публикует результат в новом чате с названием «{задача} · {местное время}» или в исходном чате, если задача выполняется в нём. Больше ничего в исходный диалог не отправляется. Чат запуска показывает инструкции задачи свёрнутым блоком «Инструкции задачи» под заголовком запуска.
|
||||
- **Где появляются результаты.** Каждый запуск публикует результат в новом чате с названием «{задача} · {местное время}» или в исходном чате, если задача выполняется в нём. Когда расписание приостановлено агентом, приостановлено автоматически или завершилось, в исходном чате там, где остановился диалог, появляется одна строка со ссылкой на этот запуск или на задачу; больше ничего в исходный диалог не отправляется. Строка остаётся и после удаления задачи. Чат запуска показывает инструкции задачи свёрнутым блоком «Инструкции задачи» под заголовком запуска.
|
||||
- **Язык.** Агент пишет название, инструкции и условие остановки на вашем языке, а запланированные запуски отвечают на языке инструкций.
|
||||
- **Пробный запуск.** Попросите прямо, например «Run it now» или «先跑一次吧»; кнопка **Run once now** на карточке делает то же самое. Простое «yes», упоминание задачи, цитата или условная просьба не запускают платное выполнение. Пробный запуск не учитывается в `max_runs`.
|
||||
|
||||
|
||||
15
README_zh.md
15
README_zh.md
@ -922,10 +922,11 @@ DeerFlow 现在在 workspace 里内置了一个一等的定时任务(scheduled
|
||||
- 支持 `once`、`cron` 和 `interval` 三种调度方式
|
||||
- 后台定时执行以非交互式 DeerFlow run 运行(那里不会暴露 `ask_clarification`)
|
||||
- 当所复用的 thread 或全局执行配额正忙时,到期执行会持久化为 `queued`,并在可用后启动;队列项在 Gateway 重启后保留,超过 `scheduler.queue_timeout_seconds` 后标记为失败
|
||||
- 执行位置在任务所有者之间公平分配:同一所有者同时启动或运行的定时运行最多 `scheduler.max_concurrent_runs_per_user` 个(默认 2,不超过 `max_concurrent_runs`;设为 `0` 即关闭这一按所有者的上限)。等待队列按所有者轮流取出,一个人积压的运行不会挡住别人的运行。等待超过 `scheduler.queue_timeout_seconds` 的运行会被跳过,运行记录显示“已跳过:等待空闲位置的时间过长”
|
||||
- 当某次执行处于 `queued`、`launching` 或 `running` 时冻结任务定义,避免持久化的执行意外换用新的 prompt、thread 或调度;将任务切换为暂停或删除任务会取消已在等待的执行,而 `launching`/`running` 执行结束后才能重试这些变更;显式手动触发在调度已暂停时仍可等待并执行,且不会自动恢复调度
|
||||
- 支持暂停、恢复、手动触发、查看历史和删除任务
|
||||
- 定时任务通过正常的 DeerFlow run 生命周期执行
|
||||
- 当 `channel_connections.enabled: true` 时,定时运行以成功或失败结束时,会把摘要推送到任务所有者已绑定的 IM 身份(outbox + 投递 worker)。带目标的任务在某次执行为 `unmet` 时改发“目标未达成”通知;连续三次未达成导致任务自动暂停时,再发一条自动暂停通知。手动「立即运行」和中断不推送;没有跑完一次运行就结束的执行记录(启动失败、排队超时、重启恢复)也不推送。渠道/传输宕机时会停车且不耗尽重试,最长约一天;平台拒绝大约重试 15 分钟后落为 `failed`。等待期间被你解绑的身份不会再收到推送:该条投递直接落为 `failed`。目前主动推送由企业微信实现;其它已绑定渠道会入队,但在实现 `send_notification` 前会失败。
|
||||
- 当 `channel_connections.enabled: true` 时,定时任务通知会发送到任务所有者已绑定、且支持主动推送的 IM 应用(outbox + 投递 worker),目前只有企业微信支持;设置页会在每个应用上标明是否发送,其它应用不会收到通知。每次执行最多发一条消息,内容可能是:一次运行完成、出错或未达成目标,连续 3 次未达成目标后自动暂停,由智能体暂停(停止条件已满足),或任务已结束(`max_runs` 次运行全部完成、已到 `end_at`);同时发生多件事时,以暂停或结束为准,并在同一条消息里说明最后一次运行的结果。单次任务只发它那次运行的结果。消息与执行结果在同一个数据库事务中入队,因此崩溃或租约失效后恢复的执行也只通知一次。消息内容可以单独读懂,使用你在网页界面中选择的语言(未设置时用 `channel_connections.notification_locale`):任务标题、发生了什么、智能体有回复时附一行结果,最后是“在 DeerFlow 的定时任务页查看详情。”,不含任何 ID 或链接。单纯的手动「立即运行」试跑和被中断的运行不通知;没有跑完一次运行就结束的执行记录(启动失败、排队超时、因重启而中断)也不通知。渠道/传输宕机时会停车且不耗尽重试,最长约一天;平台拒绝大约重试 15 分钟后落为 `failed`。等待期间被你解绑的身份不会再收到推送:该条投递直接落为 `failed`。
|
||||
- 按每页 50 条浏览执行历史;历史页暂停自动刷新,可随时返回最新记录。 仅在读取成功后显示条数,加载中或失败不会误显示为零条。
|
||||
|
||||
**通过 API 筛选执行历史**
|
||||
@ -975,9 +976,11 @@ release-checklist.md,把没勾的项告诉我;全部勾完就停。”在 We
|
||||
单次本地时间,又不知道时区(例如来自 IM),智能体会先问你。修改任务时沿用
|
||||
已保存的时区,浏览器时区不会改动已有任务。
|
||||
- **结果发在哪里。** 每次运行都会把结果发在一个新的对话里,标题为“任务名 · 本地
|
||||
时间”;如果任务在来源对话中运行,就发在来源对话里。除此之外,不会再往来源
|
||||
对话发送任何内容。运行对话会把任务指令收起为运行信息下方的一行“任务指令”,
|
||||
不再显示成一长段用户消息。
|
||||
时间”;如果任务在来源对话中运行,就发在来源对话里。调度由智能体暂停、被自动
|
||||
暂停或已结束时,来源对话会在当时对话所在的位置显示一行说明,并附上查看那次运行
|
||||
或打开任务的链接;除此之外,不会再往来源对话发送任何内容。删除任务后,这行说明
|
||||
仍会保留。运行对话会把任务指令收起为运行信息下方的一行“任务指令”,不再显示成
|
||||
一长段用户消息。
|
||||
- **语言。** 智能体会用你的语言写标题、任务指令和停止条件;定时运行用任务指令
|
||||
的语言回复。
|
||||
|
||||
@ -990,8 +993,8 @@ release-checklist.md,把没勾的项告诉我;全部勾完就停。”在 We
|
||||
未达目标的执行记为 `unmet`,与执行故障区分。周期任务连续三次符合条件的自动
|
||||
执行未达目标后会暂停。成功会重置计数,包括依赖已声明假设的成功;手动试跑、
|
||||
中断、执行故障、等待外部条件和未能检查目标都不递增计数。恢复不重置计数,下一次符合条件的
|
||||
未达目标可能再次暂停。目标未达成及自动暂停通知复用现有绑定和持久化 outbox,
|
||||
手动试跑不通知。
|
||||
未达目标可能再次暂停。支持主动推送的已绑定 IM 应用会通过同一个持久化 outbox,
|
||||
每次执行收到一条通知(未达成目标、自动暂停、由智能体暂停、已结束);单纯的手动试跑不通知。
|
||||
|
||||
用户可以在能管理该任务的对话中明确要求保存今后执行的备注,最多 10 条、每条
|
||||
500 字符。新会话中的周期任务可以通过按需开启的 `read_conversation` 回查上一次
|
||||
|
||||
@ -21,8 +21,8 @@ The backend runs a LangGraph-based super agent with sandbox execution, persisten
|
||||
- The multi-process startup gate also fires on `deployment.multi_instance: true`; contract in `docs/CONFIGURATION.md`.
|
||||
- Long-running MCP work uses a separate durable task runtime (`McpTaskService` + `mcp_tasks`, lease-based recovery) rather than keeping remote task IDs or status polling inside the Agent loop; only submit remains Agent-visible, the database is the source of truth, and `ThreadState` receives only a bounded current-thread projection. Full contract (leases, cancellation fencing, delivery idempotency, management-tool exposure): [packages/harness/deerflow/mcp/AGENTS.md](packages/harness/deerflow/mcp/AGENTS.md).
|
||||
- MCP task notification retries, dead-lettering, and the cancel endpoint's worker-stopped 503 are part of that same contract — see [packages/harness/deerflow/mcp/AGENTS.md](packages/harness/deerflow/mcp/AGENTS.md).
|
||||
- Scheduled-task dispatch permits one active occurrence per task via `uq_scheduled_task_run_active` (`task_id WHERE status IN ('queued','launching','running')`). Durable `queued` rows survive restarts; only lease-fenced `launching` may call Gateway launch; `running` references the durable run. Stable admission idempotency keys reuse that run after recovery. Reused-thread `ConflictError` returns `launching` to `queued`; other launch errors become `failed`. Atomic queue claims enforce `max_concurrent_runs`, excluding waiting rows; the budget count and its UPDATE are separate statements, so writers must serialize before the count (Postgres advisory lock; SQLite `BEGIN IMMEDIATE`, whose deferred transaction otherwise reserves the writer only at the UPDATE) or claims on distinct rows overshoot the cap. Repeated triggers coalesce; same-thread FIFO blocks behind older active rows. Queue admission, PATCH/resume, pause and delete lock the parent before the occurrence, freezing active task definitions. Pause/delete atomically cancel `queued` work but reject `launching`/`running`; PATCH/resume reject all active states. Only queued conflicts offer pause cancellation. Manual triggers may queue/run while paused. Recovery locks task/run pairs in task-id/run-id order and restores `run_id`, `started_at` and live errors before releasing launch claims. Launch/failure/timeout updates use one parent-first transaction to prevent interleaved claims. Queue timeout fails the occurrence and advances scheduled work to prevent immediate requeue. Repository boundaries coerce serialized timestamps before SQL `DateTime` binding.
|
||||
- Scheduled-task lifecycle: [persistence guide](packages/harness/deerflow/persistence/AGENTS.md#scheduled-task-lifecycle).
|
||||
- Scheduled-task dispatch permits one active occurrence per task via `uq_scheduled_task_run_active` (`task_id WHERE status IN ('queued','launching','running')`). Durable `queued` rows survive restarts; only lease-fenced `launching` may call Gateway launch; `running` references the durable run. Stable admission idempotency keys reuse that run after recovery. Reused-thread `ConflictError` returns `launching` to `queued`; other launch errors become `failed`. Atomic queue claims enforce `max_concurrent_runs` and the per-owner cap, excluding waiting rows; the budget count and its UPDATE are separate statements, so writers must serialize before the count (Postgres advisory lock; SQLite `BEGIN IMMEDIATE`, whose deferred transaction otherwise reserves the writer only at the UPDATE) or claims on distinct rows overshoot the cap. Repeated triggers coalesce; same-thread FIFO blocks behind older active rows. Queue admission, PATCH/resume, pause and delete lock the parent before the occurrence, freezing active task definitions. Pause/delete atomically cancel `queued` work but reject `launching`/`running`; PATCH/resume reject all active states. Only queued conflicts offer pause cancellation. Manual triggers may queue/run while paused.
|
||||
- Scheduled-task lifecycle, recovery and queue timeout: [persistence guide](packages/harness/deerflow/persistence/AGENTS.md#scheduled-task-lifecycle).
|
||||
- `POST /api/scheduled-tasks/preview-cron` requires authenticated `threads:read`. Bounded cron previews call the shared scheduler calculator in `asyncio.to_thread`, preserving its DST semantics. Capture the optional aware reference once; return UTC and offset-bearing local occurrences without acquiring task/thread/run stores or dispatching work. This advisory API does not reserve execution.
|
||||
- Gateway MCP/skill updates write `extensions_config.json` at runtime. Production Compose mounts it read-write (`config.yaml` stays `:ro`); Helm seeds a writable home-volume copy. Hold both `extensions_config_write_lock` and the sidecar advisory `extensions_config_file_lock` during read-modify-write to prevent lost updates across workers. `atomic_write_extensions_config` uses temp-file replacement; only mount-point `EBUSY` permits a non-atomic in-place overwrite (crashes can truncate). Warn about this fallback once per target; propagate other errors. `get_extensions_config()` revalidates reads by file signature; see [config caching](packages/harness/deerflow/config/AGENTS.md). Tests: `tests/test_compose_extensions_config_writable.py`, `tests/test_extensions_config_atomic_write.py`, `tests/test_helm_extensions_config_writable.py`.
|
||||
- MCP cache reset scope and discovery fencing: [MCP guide](packages/harness/deerflow/mcp/AGENTS.md).
|
||||
|
||||
File diff suppressed because one or more lines are too long
@ -121,13 +121,29 @@ class Channel(ABC):
|
||||
platform identity (e.g. the ``external_account_id`` recorded by the
|
||||
bind flow). Used by the notification delivery worker to deliver
|
||||
scheduled-task outcomes. Channels without proactive push support keep
|
||||
the default, which raises so the delivery outbox records a failure
|
||||
instead of silently dropping the notification.
|
||||
the default, which raises so a stray delivery is recorded as a failure
|
||||
instead of being silently dropped.
|
||||
|
||||
Connectivity preconditions should raise :class:`ChannelUnavailable`
|
||||
(not a generic ``RuntimeError``) so the outbox can park the row without
|
||||
consuming its retry budget. Platform rejections of a well-formed send
|
||||
remain ordinary exceptions that count as attempts.
|
||||
|
||||
The scheduler never imports channel classes: it reads the static
|
||||
``proactive_notifications`` flag in ``app/channels/capabilities.py``,
|
||||
and providers without it get no outbox rows at all. To add proactive
|
||||
push to a provider:
|
||||
|
||||
1. override this method and return only after the platform ACKs the send;
|
||||
2. raise :class:`ChannelUnavailable` for transient transport failures
|
||||
(not connected, socket closed, timeout); any other exception counts
|
||||
against the retry budget;
|
||||
3. set ``proactive_notifications`` to ``True`` for that provider in
|
||||
``CHANNEL_CAPABILITIES`` (a test fails while the flag and this
|
||||
override disagree).
|
||||
|
||||
The outbox, deduplication, retries, localized text and the Settings
|
||||
label are shared and need no provider code.
|
||||
"""
|
||||
if not self.is_running:
|
||||
raise ChannelUnavailable(f"channel '{self.name}' is not running")
|
||||
|
||||
34
backend/app/channels/capabilities.py
Normal file
34
backend/app/channels/capabilities.py
Normal file
@ -0,0 +1,34 @@
|
||||
"""Static per-provider channel capabilities.
|
||||
|
||||
Dependency-free on purpose: the scheduler and the Settings API read it without
|
||||
importing any channel SDK. ``tests/test_channel_capabilities.py`` keeps it in
|
||||
sync with the channel registry and the channel classes.
|
||||
|
||||
- ``supports_streaming``: the fallback the channel manager uses when no live
|
||||
channel instance is running (a running channel answers for itself).
|
||||
- ``proactive_notifications``: the provider's channel class implements
|
||||
``Channel.send_notification`` (push with no inbound message to reply to).
|
||||
Only these providers get scheduled-task notification rows; to add one, see
|
||||
the ``Channel.send_notification`` docstring.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
CHANNEL_CAPABILITIES: dict[str, dict[str, bool]] = {
|
||||
"buzz": {"supports_streaming": True, "proactive_notifications": False},
|
||||
"dingtalk": {"supports_streaming": False, "proactive_notifications": False},
|
||||
"discord": {"supports_streaming": False, "proactive_notifications": False},
|
||||
"feishu": {"supports_streaming": True, "proactive_notifications": False},
|
||||
"github": {"supports_streaming": False, "proactive_notifications": False},
|
||||
"qq": {"supports_streaming": False, "proactive_notifications": False},
|
||||
"slack": {"supports_streaming": False, "proactive_notifications": False},
|
||||
"telegram": {"supports_streaming": True, "proactive_notifications": False},
|
||||
"wechat": {"supports_streaming": False, "proactive_notifications": False},
|
||||
"wecom": {"supports_streaming": True, "proactive_notifications": True},
|
||||
}
|
||||
|
||||
|
||||
def supports_proactive_notifications(provider: str) -> bool:
|
||||
"""Whether scheduled-task notices can be pushed to ``provider`` (unknown provider: False)."""
|
||||
entry = CHANNEL_CAPABILITIES.get(provider) if isinstance(provider, str) else None
|
||||
return bool(entry) and entry.get("proactive_notifications") is True
|
||||
@ -24,6 +24,7 @@ from langgraph_sdk.errors import ConflictError
|
||||
|
||||
from app.channels import buzz_run_policy as _buzz_run_policy # noqa: F401
|
||||
from app.channels import feishu_run_policy as _feishu_run_policy # noqa: F401
|
||||
from app.channels.capabilities import CHANNEL_CAPABILITIES
|
||||
from app.channels.commands import KNOWN_CHANNEL_COMMANDS
|
||||
from app.channels.connection_identity import lookup_thread_id
|
||||
from app.channels.dedupe_store import InboundDedupeStore, MemoryInboundDedupeStore
|
||||
@ -50,6 +51,7 @@ from deerflow.config.paths import make_safe_user_id
|
||||
from deerflow.runtime import END_SENTINEL, StreamBridge
|
||||
from deerflow.runtime.goal import parse_goal_command
|
||||
from deerflow.runtime.keyed_lock import AsyncKeyedLockTable
|
||||
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY, make_origin
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
from deerflow.skills.slash import parse_slash_skill_reference
|
||||
from deerflow.skills.storage import get_or_new_skill_storage
|
||||
@ -140,17 +142,6 @@ INBOUND_DEDUPE_METADATA_KEYS = ("event_id", "message_id", "msg_id")
|
||||
# Slack is intentionally excluded: its channel ids are not globally unique.
|
||||
CHAT_SCOPED_WORKSPACE_CHANNELS = frozenset({"telegram", "feishu", "wechat"})
|
||||
|
||||
CHANNEL_CAPABILITIES = {
|
||||
"buzz": {"supports_streaming": True},
|
||||
"dingtalk": {"supports_streaming": False},
|
||||
"discord": {"supports_streaming": False},
|
||||
"feishu": {"supports_streaming": True},
|
||||
"github": {"supports_streaming": False},
|
||||
"slack": {"supports_streaming": False},
|
||||
"telegram": {"supports_streaming": True},
|
||||
"wechat": {"supports_streaming": False},
|
||||
"wecom": {"supports_streaming": True},
|
||||
}
|
||||
|
||||
InboundFileReader = Callable[[dict[str, Any], httpx.AsyncClient], Awaitable[bytes | None]]
|
||||
|
||||
@ -946,6 +937,24 @@ def _apply_effective_owner(msg: InboundMessage) -> InboundMessage:
|
||||
return msg
|
||||
|
||||
|
||||
def _run_origin(msg: InboundMessage) -> dict[str, str] | None:
|
||||
"""Server-owned ``deerflow_origin`` for a run this manager starts.
|
||||
|
||||
The Gateway honours it only from internal callers (this manager's client
|
||||
carries the internal token); a provider name outside the origin shape is
|
||||
dropped rather than failing the message.
|
||||
"""
|
||||
try:
|
||||
return make_origin("github" if msg.channel_name == "github" else "im_channel", provider=msg.channel_name)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _apply_run_origin(run_kwargs: dict[str, Any], msg: InboundMessage) -> None:
|
||||
if (origin := _run_origin(msg)) is not None:
|
||||
run_kwargs["metadata"] = {DEERFLOW_ORIGIN_KEY: origin}
|
||||
|
||||
|
||||
def _owner_headers(msg: InboundMessage) -> dict[str, str] | None:
|
||||
owner_user_id = _effective_owner_user_id(msg)
|
||||
if not owner_user_id:
|
||||
@ -1593,6 +1602,7 @@ class ChannelManager:
|
||||
"context": run_context,
|
||||
"multitask_strategy": "reject",
|
||||
}
|
||||
_apply_run_origin(run_kwargs, carrier_msg)
|
||||
if owner_headers := _owner_headers(carrier_msg):
|
||||
run_kwargs["headers"] = owner_headers
|
||||
|
||||
@ -2492,6 +2502,7 @@ class ChannelManager:
|
||||
"context": run_context,
|
||||
"multitask_strategy": "reject",
|
||||
}
|
||||
_apply_run_origin(run_kwargs, msg)
|
||||
if owner_headers := _owner_headers(msg):
|
||||
run_kwargs["headers"] = owner_headers
|
||||
|
||||
@ -2619,6 +2630,7 @@ class ChannelManager:
|
||||
"stream_mode": list(STREAM_MODES),
|
||||
"multitask_strategy": "reject",
|
||||
}
|
||||
_apply_run_origin(stream_kwargs, msg)
|
||||
if owner_headers := _owner_headers(msg):
|
||||
stream_kwargs["headers"] = owner_headers
|
||||
|
||||
|
||||
@ -108,9 +108,10 @@ owner-scoped assistant version selection remains enabled.
|
||||
|--------|-----------|
|
||||
| **Managed Models** (`/api/managed-models`) | Admin-only GET/PUT with revisions; streaming POST `/test` (DeepSeek thinking off). Clients/storage off-loop; YAML names reserved; keys private. |
|
||||
| **Models** (`/api/models`) | `GET /` - list models; `GET /{name}` - model details |
|
||||
| **Features** (`/api/features`) | `GET /` - UI capabilities: hot-reloaded agents, guarded browser, startup MCP tasks, separate batch repository/worker states so history stays readable without a worker, `conversation_references` (`read_conversation` configured, per-run reference cap), knowledge scope selection, and `scheduled_tasks` |
|
||||
| **Features** (`/api/features`) | `GET /` - UI capabilities: hot-reloaded agents, guarded browser, startup MCP tasks, separate batch repository/worker states so history stays readable without a worker, `conversation_references` (`read_conversation` configured, per-run reference cap), knowledge scope selection, `scheduled_tasks`, and `thread_activity` |
|
||||
| **Thread Activity** (`/api/thread-activity`) | `GET /?cursor=` - per-user feed over `runs.change_seq`: no cursor seeds; lists threads of the caller's server-originated runs, passes interactive rows; `read_version`. `POST /api/threads/{id}/read` and search `unread` use `thread_read_markers`. SQL only |
|
||||
| **Knowledge** (`/api/knowledge/retrieval-catalog`) | Authenticated, allowlist-safe, read-only dataset/document catalog used only by main and custom-agent chat scope selection; knowledge management remains in RAGFlow |
|
||||
| **Console** (`/api/console`) | Read-only cross-thread observability for the current user (the data layer for an operations dashboard or external monitoring): `GET /stats` - headline counters (runs/threads/agents/tokens/cost); `GET /runs` - paginated run history joined with thread titles (per-run cost); `GET /usage` - zero-filled daily token series + per-model breakdown with spend. Queries `runs`/`threads_meta` directly as a reporting layer (no new `RunStore` methods); requires a SQL database backend — returns 503 on `database.backend: memory`. Real-cost estimation reads optional `models[*].pricing` (`currency`, `input_per_million`, `output_per_million`, `input_cache_hit_per_million`; `ModelConfig` is `extra="allow"`, so no schema change) and prices each run from its `token_usage_by_model` input/output split. Pricing is **cache-aware**: `RunJournal` accumulates prompt-cache hits from `usage_metadata.input_token_details.cache_read` into a sparse `cache_read_tokens` bucket key (also threaded through `SubagentTokenCollector` → `record_external_llm_usage_records`), and cache-hit input tokens are billed at `input_cache_hit_per_million` (omitted → billed at the miss price, a conservative upper bound). All priced models must use one currency; mixed currencies disable cost reporting and leave cost/currency fields null instead of producing invalid aggregates. Legacy rows fall back to run-level totals at `model_name`; unpriced models yield `cost: null` and cost fields are null when no pricing is configured |
|
||||
| **Console** (`/api/console`) | Read-only cross-thread observability for the current user: `GET /stats` - headline counters (runs/threads/agents/tokens/cost); `GET /runs` - paginated run history joined with thread titles (per-run cost); `GET /usage` - zero-filled daily token series + per-model breakdown with spend. Queries `runs`/`threads_meta` directly (no new `RunStore` methods); SQL only (503 on `database.backend: memory`). Real-cost estimation reads optional `models[*].pricing` (`currency`, `input_per_million`, `output_per_million`, `input_cache_hit_per_million`; `ModelConfig` is `extra="allow"`, so no schema change) and prices each run from its `token_usage_by_model` input/output split. Pricing is **cache-aware**: `RunJournal` accumulates prompt-cache hits from `usage_metadata.input_token_details.cache_read` into a sparse `cache_read_tokens` bucket key (also threaded through `SubagentTokenCollector` → `record_external_llm_usage_records`), and cache-hit input tokens are billed at `input_cache_hit_per_million` (omitted → billed at the miss price, a conservative upper bound). Mixed currencies null cost/currency fields. Legacy rows fall back to run-level totals at `model_name`; unpriced models yield `cost: null` and cost fields are null when no pricing is configured |
|
||||
| **MCP** (`/api/mcp`) | GET /config - raw/masked; PUT /config - bulk; PATCH /config - toggle; POST /config/servers - add; PUT /config/server - replace; DELETE /config/servers/{server_name:path} - bodyless. POST /cache/reset. Validate expanded, save raw; reload/reset; invalid -> 400. |
|
||||
| **MCP Tasks** (`/api/threads/{id}/mcp-tasks`) | `GET /` - current user's durable tasks for one owned thread; `GET /{task_id}` - bounded result/input/status-error/cancellation-error detail, including cancellation attempt count, without remote task IDs or driver configuration |
|
||||
| **Skills** (`/api/skills`) | `GET /` - list; `GET /{name}` - inspect; `PUT /{name}` - toggle; `POST /install` - install a thread-local .skill archive; `POST /install/upload` - admin-only multipart, authorized before parsing and capped at a 100 MiB file plus 1 MiB framing; `POST /reload` - invalidate process-local cache after trusted filesystem changes |
|
||||
@ -118,7 +119,7 @@ owner-scoped assistant version selection remains enabled.
|
||||
| **Integrations** (`/api/integrations`) | `GET /lark/status` - inspect managed Lark/Feishu CLI integration state, including `sandbox_runtime_mode` / `sandbox_runtime_probed` / `sandbox_runtime_ready` (whether `lark-cli` will be present in the sandbox at chat time); `POST /lark/install` - admin-only install of the official `lark-*` managed skill pack; `POST /lark/config/start` and `/lark/config/complete` - internal first-time Lark connection setup; `POST /lark/config/credentials` - atomically switch the caller's per-user Lark app after validating the new `app_id`/`app_secret` through the official CLI's live tenant-token probe, revoke/remove the previous OAuth tokens, and restore the prior credential tree if the switch fails; `POST /lark/auth/start` and `/lark/auth/complete` - browser device-flow user authorization without terminal access, with optional `domains` / exact `scope` for incremental permission grants. Config and auth flows carry a server-issued, per-user generation persisted under the credential lock; a rejected direct switch leaves the current generation unchanged, stale completions return 409, and browser re-registration uses the same token-clearing/revocation transaction as direct credential switches. |
|
||||
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
|
||||
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload/convert PDF/PPT/Excel/Word; upload/shelf names share `normalize_filename` before staging; non-mounted sync lease ends when its last holder exits; `GET /list`; `DELETE /{filename}` - regular files, symlinks 404, keeps converted `.md` (#5672) |
|
||||
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; `POST /branches` - branch a completed assistant turn with a replay checkpoint; inherited titles take next-free displayed sibling suffixes, including explicit/renamed ones, while explicit titles stay unchanged. Durable `branch` admission rejects races. Workspace files are not checkpointed, so the branch only best-effort copies the current workspace when branching from the **latest** turn (`workspace_clone_mode="current_thread_best_effort"`); branching from an older/historical turn skips the copy (`workspace_clone_mode="skipped_historical_turn"`) so the branch never inherits files that only exist in a later timeline. Thread-scoped runtime channels (`sandbox`, `thread_data`) are not copied onto the branch: the parent's `sandbox_id` binds path mappings and the release lifecycle to the parent's workspace, so the branch lazily acquires its own sandbox instead. Branch creation also seeds the new thread's run-event feed from the branch checkpoint's visible messages (`history_seed_mode` in the response): the thread feed reads run_events, not checkpoints, so without the seed the inherited history disappears from the UI after the branch's first run (#4380). Seeded rows are grouped into one synthetic run per inherited turn (`branch-seed-{thread_id}-{n}`, a new turn opening at every persisted human message, including an allowlisted hidden `ask_clarification` reply) because `run_id` is a turn identity to the feed's consumers, not a provenance tag: regenerating an inherited answer supersedes that row's whole `run_id` in `GET /messages/page`, so one shared id for the entire seed deleted the complete inherited history on a branch's first regenerate (#4458); `GET /goal`, `PUT /goal`, `DELETE /goal` - read, set, and clear the active thread goal; `POST /compact` - summarize older active context, deriving memory policy and bucket from the state-producing checkpoint rather than request `agent_name`, and block while a run is in flight; unexpected failures return a generic 500 detail |
|
||||
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; `POST /branches` - branch a completed assistant turn with a replay checkpoint; inherited titles take next-free displayed sibling suffixes, including explicit/renamed ones, while explicit titles stay unchanged. Durable `branch` admission rejects races. Workspace files are not checkpointed: only a branch from the **latest** turn best-effort copies the workspace (`workspace_clone_mode="current_thread_best_effort"`); older turns skip it (`"skipped_historical_turn"`), so no later-timeline file leaks in. `sandbox`/`thread_data` channels are not copied (parent `sandbox_id` binds paths/release); the branch lazily acquires its own sandbox. Branch creation also seeds the new thread's run-event feed from the branch checkpoint's visible messages (`history_seed_mode` in the response): the thread feed reads run_events, not checkpoints, so without the seed the inherited history disappears from the UI after the branch's first run (#4380). Seeded rows get one synthetic run per inherited turn (`branch-seed-{thread_id}-{n}`, a new turn at every persisted human message, including an allowlisted hidden `ask_clarification` reply): `run_id` is a turn identity, and regenerating an inherited answer supersedes its whole `run_id` in `GET /messages/page`, so one shared seed id deleted all inherited history (#4458); `GET /goal`, `PUT /goal`, `DELETE /goal` - read, set, and clear the active thread goal; `POST /compact` - summarize older active context, deriving memory policy and bucket from the state-producing checkpoint rather than request `agent_name`, and block while a run is in flight; unexpected failures return a generic 500 detail |
|
||||
| **Artifacts** (`/api/threads/{id}/artifacts`) | GET streams byte ranges; HTML/XML/XSL/`+xml` (also `.skill` members) and `?download=true` force attachments. PUT atomically replaces existing UTF-8 outputs on matching SHA-256; active runs conflict. Non-mounted sync uses a request lease and drains remote/local commit-or-rollback across cancellation before releasing reservations/leases. `resolve_outputs_confined_path` normalizes `..` and confines host paths against traversal/symlinks. Preserve POSIX ownership/mode where supported; Windows keeps temp-file permissions. Digest cache: path/dev/ino/ctime_ns/mtime_ns/size; identity invalidates same-size/mtime replacements. Large regular files use stat-prefixed ETags; If-Range requires matching ETags. Windows ctime is creation time. Keep cache reuse, size limits and worker hashing. |
|
||||
| **Suggestions** (`/api/suggestions`) | `GET /config` - returns global suggestions config boolean; `POST /threads/{id}/suggestions` - generate follow-up questions; rich list/block model content is normalized and inline reasoning (`<think>...</think>`, including unclosed/truncated blocks from reasoning models like MiniMax-M3) is stripped before JSON parsing |
|
||||
| **Input Polish** (`/api/input-polish`) | `POST /` - rewrite a composer draft before it is sent. This is a short authenticated `runs:create` LLM request using `input_polish` config; it does not create a LangGraph run, persist a message, or modify thread state. Shares the non-graph one-shot LLM path (`deerflow.utils.oneshot_llm.run_oneshot_llm`) with the suggestions route so model build + Langfuse metadata + invoke stay in one place; validates the same stripped view of the draft it sends to the model, and preserves literal `<think>` substrings in the rewrite (`strip_think_blocks(truncate_unclosed=False)`) |
|
||||
@ -184,7 +185,7 @@ startup gate rejects memory/JSONL event stores when `GATEWAY_WORKERS > 1`
|
||||
or `deployment.multi_instance: true`.
|
||||
|
||||
**RunManager / RunStore contract**:
|
||||
- LangGraph-compatible run requests validate their supported subset before creating a run. `runtime/stream_modes.py` is the shared backend contract for public stream modes and the worker's `graph.astream` mapping; the public `messages-tuple` mode maps to LangGraph's internal `messages` mode, while public `messages`, `events`, and other unsupported modes are rejected instead of being dropped or replaced with `values`. `app/gateway/run_models.py::RunCreateRequest` is shared by HTTP and internal scheduled launch paths, retains only truthful compatibility defaults for unimplemented options (`if_not_exists="create"` plus `None` placeholders), returns 422 for unsupported values including `on_completion="complete"`, `on_completion="continue"`, and `multitask_strategy="enqueue"`, and forbids undeclared SDK options so fields such as `checkpoint_during` and `durability` cannot be silently discarded. A placeholder must still accept the stock SDK's own default: `langgraph_sdk` drops only `None` from its run payload, so `stream_resumable=False` reaches every request and means "non-resumable", which is what DeerFlow serves — rejecting it 422'd every IM channel run (#4466). `tests/test_run_request_validation.py::test_gateway_accepts_langgraph_sdk_default_payload` pins the real SDK payload against this boundary; channel tests mock the SDK client and cannot catch this class of drift.
|
||||
- LangGraph-compatible run requests validate their supported subset before creating a run. `runtime/stream_modes.py` is the shared backend contract for public stream modes and the worker's `graph.astream` mapping; the public `messages-tuple` mode maps to LangGraph's internal `messages` mode, while public `messages`, `events`, and other unsupported modes are rejected instead of being dropped or replaced with `values`. `app/gateway/run_models.py::RunCreateRequest` is shared by HTTP and internal scheduled launch paths, retains only truthful compatibility defaults for unimplemented options (`if_not_exists="create"` plus `None` placeholders), returns 422 for unsupported values including `on_completion="complete"`, `on_completion="continue"`, and `multitask_strategy="enqueue"`, and forbids undeclared SDK options so fields such as `checkpoint_during` and `durability` cannot be silently discarded. A placeholder must still accept the stock SDK's own default: `langgraph_sdk` drops only `None` from its run payload, so `stream_resumable=False` reaches every request and means "non-resumable", which is what DeerFlow serves — rejecting it 422'd every IM channel run (#4466). `test_run_request_validation.py::test_gateway_accepts_langgraph_sdk_default_payload` pins the real SDK payload; channel tests mock the SDK and miss this drift.
|
||||
- `RunManager.get()` is async; direct callers must `await` it.
|
||||
- The history batch helpers `list_successful_regenerate_sources()`, `list_edit_regenerate_runs()`, and `get_many_by_thread()` default to `user_id=AUTO`: they resolve the request user and fail closed when no user context exists. Migration/admin callers that intentionally need an unscoped read must pass `user_id=None` explicitly.
|
||||
- Edit-and-rerun visibility is derived from edit replay runs (`metadata.replay_kind="edit"` plus `regenerate_from_run_id`) by `RunManager.list_edit_replay_visibility()`: the newest attempt for each source run is authoritative. Pending/running/success attempts hide the original source run; failed, timed-out, or interrupted attempts hide only the failed attempt so the original conversation reappears.
|
||||
@ -209,6 +210,8 @@ or `deployment.multi_instance: true`.
|
||||
|
||||
Scheduled-task PATCH: omitted `assistant_id` preserves the pin; explicit null resets to `lead_agent`.
|
||||
|
||||
Two origin keys; never read one for the other. `deerflow_scheduled_origin` marks the scheduled prompt message (`scheduled_goal_notes_contract.json`); `deerflow_origin` says who started a run/created a thread (`thread_origin_contract.json`), set only via `request.state.run_origin` or internal `body.metadata`.
|
||||
|
||||
**Thread lifecycle**: Before changing branching, regeneration, edit replay, or
|
||||
archive/search behavior, read [Thread lifecycle invariants](../../docs/THREAD_LIFECYCLE.md).
|
||||
It owns lineage and settled-checkpoint rules, legacy fallback boundaries, archive
|
||||
|
||||
@ -51,6 +51,7 @@ from app.gateway.routers import (
|
||||
subagent_batches,
|
||||
subagents,
|
||||
suggestions,
|
||||
thread_activity,
|
||||
thread_runs,
|
||||
threads,
|
||||
trash,
|
||||
@ -456,6 +457,8 @@ async def _start_scheduled_task_notification_delivery(app: FastAPI, startup_conf
|
||||
# connection repository is present whenever the outbox one is.
|
||||
resolve_connections=connection_repo.list_connections,
|
||||
poll_interval_seconds=startup_config.scheduler.poll_interval_seconds,
|
||||
# Language of notices whose owner has no UI language preference.
|
||||
default_locale=startup_config.channel_connections.notification_locale,
|
||||
)
|
||||
await worker.start()
|
||||
except Exception:
|
||||
@ -638,6 +641,7 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
poll_interval_seconds=startup_config.scheduler.poll_interval_seconds,
|
||||
lease_seconds=startup_config.scheduler.lease_seconds,
|
||||
max_concurrent_runs=startup_config.scheduler.max_concurrent_runs,
|
||||
max_concurrent_runs_per_user=startup_config.scheduler.max_concurrent_runs_per_user,
|
||||
queue_timeout_seconds=startup_config.scheduler.queue_timeout_seconds,
|
||||
multi_instance=startup_config.scheduler.multi_instance,
|
||||
run_lease_grace_seconds=startup_config.run_ownership.grace_seconds,
|
||||
@ -1199,6 +1203,9 @@ This gateway provides runtime endpoints for agent runs plus custom endpoints for
|
||||
# Thread cleanup API is mounted at /api/threads/{thread_id}
|
||||
app.include_router(threads.router)
|
||||
|
||||
# Per-user activity feed (own prefix, so /api/threads/{thread_id} never captures it)
|
||||
app.include_router(thread_activity.router)
|
||||
|
||||
# Scheduled tasks API is mounted at /api/scheduled-tasks
|
||||
app.include_router(scheduled_tasks.router)
|
||||
|
||||
|
||||
@ -669,11 +669,13 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
|
||||
if sf is not None:
|
||||
from deerflow.persistence.mcp_tasks import McpTaskRepository
|
||||
from deerflow.persistence.projects import ProjectDocumentRepository, ProjectRepository
|
||||
from deerflow.persistence.scheduled_task_events import ScheduledTaskEventRepository
|
||||
from deerflow.persistence.scheduled_task_runs import (
|
||||
ScheduledTaskRunRepository,
|
||||
)
|
||||
from deerflow.persistence.scheduled_tasks import ScheduledTaskRepository
|
||||
from deerflow.persistence.subagent_batches import SubagentBatchRepository
|
||||
from deerflow.persistence.thread_reads import ThreadReadRepository
|
||||
|
||||
app.state.project_repo = ProjectRepository(sf)
|
||||
app.state.project_document_repo = ProjectDocumentRepository(sf)
|
||||
@ -685,6 +687,9 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
|
||||
sf,
|
||||
run_repository=app.state.run_store,
|
||||
)
|
||||
app.state.scheduled_task_event_repo = ScheduledTaskEventRepository(sf)
|
||||
# Per-user unread state and the activity feed's read clock.
|
||||
app.state.thread_read_repo = ThreadReadRepository(sf)
|
||||
app.state.mcp_task_repo = McpTaskRepository(sf)
|
||||
app.state.subagent_batch_repo = SubagentBatchRepository(sf)
|
||||
else:
|
||||
@ -694,6 +699,8 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
|
||||
app.state.subagent_batch_repo = None
|
||||
app.state.scheduled_task_repo = None
|
||||
app.state.scheduled_task_run_repo = None
|
||||
app.state.scheduled_task_event_repo = None
|
||||
app.state.thread_read_repo = None
|
||||
|
||||
# RunManager with store backing for persistence
|
||||
run_ownership_config = getattr(config, "run_ownership", None)
|
||||
@ -841,6 +848,13 @@ def get_scheduled_task_run_repo(request: Request):
|
||||
return val
|
||||
|
||||
|
||||
def get_scheduled_task_event_repo(request: Request):
|
||||
val = getattr(request.app.state, "scheduled_task_event_repo", None)
|
||||
if val is None:
|
||||
raise HTTPException(status_code=503, detail="Scheduled task event repo not available")
|
||||
return val
|
||||
|
||||
|
||||
def get_scheduled_task_service(request: Request):
|
||||
val = getattr(request.app.state, "scheduled_task_service", None)
|
||||
if val is None:
|
||||
|
||||
@ -15,6 +15,7 @@ from pydantic import TypeAdapter, ValidationError
|
||||
|
||||
from app.gateway.auth_disabled import AUTH_SOURCE_AUTH_DISABLED, AUTH_SOURCE_SESSION, get_auth_disabled_user, is_auth_disabled
|
||||
from app.gateway.authz import AuthContext, resolve_route_permissions
|
||||
from deerflow.runtime.run_origin import make_origin
|
||||
from deerflow.runtime.user_context import reset_current_user, set_current_user
|
||||
from deerflow.utils.thread_id import ThreadId
|
||||
|
||||
@ -100,6 +101,9 @@ class _BoundAgentRuns:
|
||||
request.state.user = user
|
||||
request.state.auth_source = self._source
|
||||
request.state.auth = AuthContext(user, list(permissions))
|
||||
# Host-set, never HTTP input: threads and runs created through this
|
||||
# handle are marked as extension work (activity feed, unread state).
|
||||
request.state.run_origin = make_origin("extension", namespace=self._plugin_namespace)
|
||||
token = set_current_user(user)
|
||||
try:
|
||||
if thread_id is not None:
|
||||
|
||||
@ -12,6 +12,7 @@ from typing import Any, Literal
|
||||
from fastapi import APIRouter, HTTPException, Request, Response
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.channels.capabilities import supports_proactive_notifications
|
||||
from app.channels.runtime_config_store import (
|
||||
ChannelRuntimeConfigStore,
|
||||
apply_runtime_connection_config,
|
||||
@ -53,6 +54,9 @@ class ChannelProviderResponse(BaseModel):
|
||||
connection_status: str
|
||||
credential_fields: list[ChannelCredentialFieldResponse] = Field(default_factory=list)
|
||||
credential_values: dict[str, str] = Field(default_factory=dict)
|
||||
# Whether scheduled-task updates can be pushed to this app (the provider
|
||||
# implements proactive push); Settings labels each provider card with it.
|
||||
proactive_notifications: bool = False
|
||||
|
||||
|
||||
class ChannelProvidersResponse(BaseModel):
|
||||
@ -477,6 +481,7 @@ def _provider_response(
|
||||
connection_status=connection_status,
|
||||
credential_fields=_credential_fields(provider),
|
||||
credential_values=credential_values,
|
||||
proactive_notifications=supports_proactive_notifications(provider),
|
||||
)
|
||||
|
||||
|
||||
|
||||
@ -14,6 +14,7 @@ from app.gateway.browser_capability import browser_capability
|
||||
from app.gateway.conversation_access import conversation_references_enabled
|
||||
from app.gateway.deps import get_config
|
||||
from app.gateway.knowledge_scope_admission import RAGFLOW_KNOWLEDGE_SEARCH_PROVIDER
|
||||
from app.gateway.routers.thread_activity import thread_activity_available
|
||||
from app.gateway.run_models import MAX_CONVERSATION_REFERENCES
|
||||
from deerflow.config.app_config import AppConfig
|
||||
from deerflow.scheduler.runtime import scheduler_tools_enabled
|
||||
@ -74,6 +75,12 @@ class ScheduledTasksFeature(BaseModel):
|
||||
min_interval_seconds: int = Field(..., description="Shortest interval and earliest one-time delay, in seconds")
|
||||
|
||||
|
||||
class ThreadActivityFeature(BaseModel):
|
||||
"""Availability of the activity feed and per-user read markers."""
|
||||
|
||||
available: bool = Field(..., description="Whether GET /api/thread-activity, POST /api/threads/{thread_id}/read and the thread search `unread` field work (SQL persistence)")
|
||||
|
||||
|
||||
class FeaturesResponse(BaseModel):
|
||||
"""Frontend-facing feature availability flags."""
|
||||
|
||||
@ -84,6 +91,7 @@ class FeaturesResponse(BaseModel):
|
||||
conversation_references: ConversationReferencesFeature
|
||||
knowledge_base: KnowledgeBaseFeature
|
||||
scheduled_tasks: ScheduledTasksFeature
|
||||
thread_activity: ThreadActivityFeature
|
||||
|
||||
|
||||
@router.get(
|
||||
@ -123,6 +131,8 @@ async def list_features(request: Request, config: AppConfig = Depends(get_config
|
||||
scope_selection_enabled=_knowledge_scope_selection_enabled(config),
|
||||
),
|
||||
scheduled_tasks=_scheduled_tasks_feature(request, config),
|
||||
# Startup-scoped: the read repository exists only with SQL persistence.
|
||||
thread_activity=ThreadActivityFeature(available=thread_activity_available(request.app.state)),
|
||||
)
|
||||
|
||||
|
||||
|
||||
@ -23,6 +23,7 @@ from app.gateway.services import (
|
||||
start_run,
|
||||
wait_for_run_completion,
|
||||
)
|
||||
from app.gateway.sse_headers import sse_response_headers
|
||||
from deerflow.runtime import RunStatus, serialize_channel_values_for_api
|
||||
from deerflow.utils.thread_id import resolve_thread_id
|
||||
|
||||
@ -53,12 +54,7 @@ async def stateless_stream(body: RunCreateRequest, request: Request) -> Streamin
|
||||
return StreamingResponse(
|
||||
sse_consumer(bridge, record, request, run_mgr),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"Connection": "keep-alive",
|
||||
"X-Accel-Buffering": "no",
|
||||
"Content-Location": f"/api/threads/{thread_id}/runs/{record.run_id}",
|
||||
},
|
||||
headers=sse_response_headers(content_location=f"/api/threads/{thread_id}/runs/{record.run_id}"),
|
||||
)
|
||||
|
||||
|
||||
|
||||
@ -13,6 +13,7 @@ from app.gateway.authz import require_permission
|
||||
from app.gateway.deps import (
|
||||
get_config,
|
||||
get_optional_user_from_request,
|
||||
get_scheduled_task_event_repo,
|
||||
get_scheduled_task_repo,
|
||||
get_scheduled_task_run_repo,
|
||||
get_scheduled_task_service,
|
||||
@ -449,3 +450,19 @@ async def list_thread_scheduled_tasks(thread_id: ThreadId, request: Request):
|
||||
if user is None:
|
||||
raise _authentication_required()
|
||||
return await _with_run_state(repo, await repo.list_by_user_and_thread(str(user.id), thread_id))
|
||||
|
||||
|
||||
@router.get("/threads/{thread_id}/scheduled-task-events")
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def list_thread_scheduled_task_events(thread_id: ThreadId, request: Request, limit: int = Query(default=50, ge=1, le=200)):
|
||||
"""Lifecycle events of the caller's tasks for this originating chat, oldest first.
|
||||
|
||||
Each row is one line in the chat ("Paused by agent", "Auto-paused",
|
||||
"Finished"). Rows outlive their task (they carry a title snapshot), so
|
||||
this route does not look the task up. IDs are for links only.
|
||||
"""
|
||||
repo = get_scheduled_task_event_repo(request)
|
||||
user = await get_optional_user_from_request(request)
|
||||
if user is None:
|
||||
raise _authentication_required()
|
||||
return {"events": await repo.list_for_thread(user_id=str(user.id), thread_id=thread_id, limit=limit)}
|
||||
|
||||
120
backend/app/gateway/routers/thread_activity.py
Normal file
120
backend/app/gateway/routers/thread_activity.py
Normal file
@ -0,0 +1,120 @@
|
||||
"""Per-user activity feed over the run-change clock.
|
||||
|
||||
``GET /api/thread-activity`` tells an open sidebar which threads changed
|
||||
because the server ran something for the caller (a schedule, an IM channel, a
|
||||
GitHub agent, an extension, an MCP notification), so it can refetch its thread
|
||||
list without a reload. It lives on its own prefix so it can never be captured
|
||||
by ``GET /api/threads/{thread_id}``.
|
||||
|
||||
The cursor is the ``(change_seq, run_id)`` position of the last run change the
|
||||
caller has seen. The clock is database-global and monotonic
|
||||
(``runs.change_seq``), so the feed is exact across workers and independent of
|
||||
wall-clock time. An idle poll is one index seek on ``ix_runs_user_change_seq``
|
||||
plus one primary-key lookup of the caller's read clock.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.authz import require_permission
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
router = APIRouter(prefix="/api/thread-activity", tags=["threads"])
|
||||
|
||||
_CURSOR_RE = re.compile(r"(0|[1-9][0-9]{0,18}):([^\s:]{0,64})")
|
||||
# runs.change_seq is a BIGINT: a larger position would fail inside the database
|
||||
# (Postgres) instead of reading as a malformed cursor.
|
||||
_MAX_CHANGE_SEQ = 2**63 - 1
|
||||
_SEED_CURSOR = "0:"
|
||||
|
||||
|
||||
class ThreadActivityItem(BaseModel):
|
||||
"""One thread with a server-originated run change since the cursor."""
|
||||
|
||||
thread_id: str
|
||||
origin_kind: str | None = Field(description="Server-owned origin of the latest changed run (never null in this list)")
|
||||
status: str = Field(description="Status of the thread's latest changed run")
|
||||
|
||||
|
||||
class ThreadActivityResponse(BaseModel):
|
||||
"""A page of the caller's run changes, reduced to the threads that matter."""
|
||||
|
||||
cursor: str = Field(description="Opaque position to pass back on the next poll")
|
||||
threads: list[ThreadActivityItem] = Field(default_factory=list)
|
||||
truncated: bool = Field(default=False, description="More run changes than `limit` were pending; poll again soon")
|
||||
read_version: int = Field(default=0, description="Per-user read clock; it changes when a thread is marked read on any device")
|
||||
|
||||
|
||||
def thread_activity_available(state: Any) -> bool:
|
||||
"""SQL persistence only: the feed needs the run-change clock and the read tables."""
|
||||
run_store = getattr(state, "run_store", None)
|
||||
return getattr(state, "thread_read_repo", None) is not None and callable(getattr(run_store, "latest_change", None))
|
||||
|
||||
|
||||
def _parse_cursor(cursor: str) -> tuple[int, str]:
|
||||
match = _CURSOR_RE.fullmatch(cursor)
|
||||
if match is None or int(match.group(1)) > _MAX_CHANGE_SEQ:
|
||||
raise HTTPException(status_code=422, detail={"code": "invalid_cursor", "message": "Invalid activity cursor"})
|
||||
return int(match.group(1)), match.group(2)
|
||||
|
||||
|
||||
def _format_cursor(change_seq: int, run_id: str) -> str:
|
||||
return f"{change_seq}:{run_id}"
|
||||
|
||||
|
||||
@router.get("", response_model=ThreadActivityResponse)
|
||||
@require_permission("threads", "read")
|
||||
async def list_thread_activity(
|
||||
request: Request,
|
||||
# No max_length: an overlong cursor must read as ``invalid_cursor`` too,
|
||||
# and the pattern bounds what is accepted.
|
||||
cursor: str | None = Query(default=None),
|
||||
limit: int = Query(default=200, ge=1, le=500),
|
||||
) -> ThreadActivityResponse:
|
||||
"""Threads changed by server-originated runs of the caller since ``cursor``.
|
||||
|
||||
Without a cursor the call only seeds: it returns the caller's current head
|
||||
and no threads. The caller's own interactive runs (``origin_kind`` NULL)
|
||||
advance the cursor but are not returned.
|
||||
"""
|
||||
state = request.app.state
|
||||
if not thread_activity_available(state):
|
||||
raise HTTPException(status_code=503, detail="Thread activity is not available")
|
||||
run_store = state.run_store
|
||||
read_repo = state.thread_read_repo
|
||||
# Always an explicit owner id: None would disable the owner filter.
|
||||
user_id = str(get_effective_user_id())
|
||||
|
||||
if cursor is None:
|
||||
head = await run_store.latest_change(user_id=user_id)
|
||||
next_cursor = _format_cursor(*head) if head is not None else _SEED_CURSOR
|
||||
return ThreadActivityResponse(cursor=next_cursor, read_version=await read_repo.read_version(user_id=user_id))
|
||||
|
||||
after_change_seq, after_run_id = _parse_cursor(cursor)
|
||||
rows = await run_store.list_changed(after_change_seq=after_change_seq, after_run_id=after_run_id, user_id=user_id, limit=limit + 1)
|
||||
truncated = len(rows) > limit
|
||||
used = rows[:limit]
|
||||
# Advance to the last row used, never to the head: a truncated page
|
||||
# continues on the next poll without skipping changes.
|
||||
next_cursor = _format_cursor(int(used[-1]["change_seq"]), str(used[-1]["run_id"])) if used else _format_cursor(after_change_seq, after_run_id)
|
||||
|
||||
latest: dict[str, ThreadActivityItem] = {}
|
||||
for row in used:
|
||||
origin_kind = row.get("origin_kind")
|
||||
if origin_kind is None:
|
||||
continue
|
||||
thread_id = str(row["thread_id"])
|
||||
latest.pop(thread_id, None)
|
||||
latest[thread_id] = ThreadActivityItem(thread_id=thread_id, origin_kind=origin_kind, status=str(row.get("status") or ""))
|
||||
|
||||
return ThreadActivityResponse(
|
||||
cursor=next_cursor,
|
||||
threads=list(latest.values()),
|
||||
truncated=truncated,
|
||||
read_version=await read_repo.read_version(user_id=user_id),
|
||||
)
|
||||
@ -59,6 +59,7 @@ from app.gateway.services import (
|
||||
start_run,
|
||||
wait_for_run_completion,
|
||||
)
|
||||
from app.gateway.sse_headers import sse_response_headers
|
||||
from app.gateway.utils import sanitize_log_param
|
||||
from deerflow.agents.human_input import read_human_input_response
|
||||
from deerflow.agents.middlewares.dynamic_context_middleware import strip_injected_user_message_id_suffix
|
||||
@ -1010,15 +1011,10 @@ async def stream_run(
|
||||
emit_gap_on_missing_stream=record.idempotency_reused,
|
||||
),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"Connection": "keep-alive",
|
||||
"X-Accel-Buffering": "no",
|
||||
# LangGraph Platform includes run metadata in this header.
|
||||
# The SDK uses a greedy regex to extract the run id from this path,
|
||||
# so it must point at the canonical run resource without extra suffixes.
|
||||
"Content-Location": f"/api/threads/{thread_id}/runs/{record.run_id}",
|
||||
},
|
||||
# LangGraph Platform includes run metadata in Content-Location.
|
||||
# The SDK uses a greedy regex to extract the run id from this path,
|
||||
# so it must point at the canonical run resource without extra suffixes.
|
||||
headers=sse_response_headers(content_location=f"/api/threads/{thread_id}/runs/{record.run_id}"),
|
||||
)
|
||||
|
||||
|
||||
@ -1327,11 +1323,7 @@ async def join_run(thread_id: ThreadId, run_id: str, request: Request) -> Stream
|
||||
# policy must not fire because an observer closed their connection.
|
||||
sse_consumer(bridge, record, request, run_mgr, apply_on_disconnect=False),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"Connection": "keep-alive",
|
||||
"X-Accel-Buffering": "no",
|
||||
},
|
||||
headers=sse_response_headers(),
|
||||
)
|
||||
|
||||
|
||||
@ -1416,11 +1408,7 @@ async def _stream_existing_run(
|
||||
# must not fire because a joiner closed their connection.
|
||||
sse_consumer(bridge, record, request, run_mgr, apply_on_disconnect=False),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"Connection": "keep-alive",
|
||||
"X-Accel-Buffering": "no",
|
||||
},
|
||||
headers=sse_response_headers(),
|
||||
)
|
||||
|
||||
|
||||
|
||||
@ -68,6 +68,7 @@ from deerflow.runtime.goal import (
|
||||
write_thread_goal,
|
||||
)
|
||||
from deerflow.runtime.journal import build_branch_history_seed_events
|
||||
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY, admit_origin
|
||||
from deerflow.runtime.runs.manager import ConflictError
|
||||
from deerflow.runtime.runs.worker import RUN_MESSAGE_IDS_METADATA_KEY, valid_duration_entry, valid_run_message_id_entry
|
||||
from deerflow.runtime.secret_context import redact_metadata_secrets
|
||||
@ -114,7 +115,9 @@ def _checkpoint_mode_http_error(exc: Exception, thread_id: str) -> HTTPException
|
||||
# owner identity through the API surface. Defense-in-depth — the
|
||||
# row-level invariant is still ``threads_meta.user_id`` populated from
|
||||
# the auth contextvar; this list closes the metadata-blob echo gap.
|
||||
_SERVER_RESERVED_METADATA_KEYS: frozenset[str] = frozenset({"owner_id", "user_id", THREAD_PROJECT_METADATA_KEY})
|
||||
# ``deerflow_origin`` marks a thread the server created for a schedule or an
|
||||
# extension; only the server sets it (``create_thread`` below, run admission).
|
||||
_SERVER_RESERVED_METADATA_KEYS: frozenset[str] = frozenset({"owner_id", "user_id", THREAD_PROJECT_METADATA_KEY, DEERFLOW_ORIGIN_KEY})
|
||||
_SIDECAR_METADATA_KEY = "deerflow_sidecar"
|
||||
_BRANCH_METADATA_KEY = "deerflow_branch"
|
||||
_BRANCH_TITLE_SEQUENCE_METADATA_KEY = "branch_title_sequence"
|
||||
@ -456,6 +459,14 @@ class ThreadResponse(_MetadataRedactingResponse):
|
||||
metadata: dict[str, Any] = Field(default_factory=dict, description="Thread metadata")
|
||||
values: dict[str, Any] = Field(default_factory=dict, description="Current state channel values")
|
||||
interrupts: dict[str, Any] = Field(default_factory=dict, description="Pending interrupts")
|
||||
unread: bool | None = Field(default=None, description="Thread search only: a server-originated run of the caller changed since the caller last opened the thread; null when unknown")
|
||||
|
||||
|
||||
class ThreadReadResponse(BaseModel):
|
||||
"""Response of ``POST /api/threads/{thread_id}/read``."""
|
||||
|
||||
unread: bool = Field(default=False)
|
||||
read_version: int = Field(description="The caller's per-user read clock after this call")
|
||||
|
||||
|
||||
class ThreadCreateRequest(BaseModel):
|
||||
@ -834,6 +845,29 @@ async def _delete_thread_data_with_reservation(thread_id: str, request: Request)
|
||||
except Exception:
|
||||
logger.debug("Could not delete thread_meta for %s (not critical)", sanitize_log_param(thread_id))
|
||||
|
||||
# Remove the chat's scheduled-task lifecycle lines (best-effort). They are
|
||||
# display-only history of this chat; the tasks themselves are untouched.
|
||||
# This runs after the thread_meta delete: the finalization observer writes
|
||||
# a line only while that row exists, and holds a share lock on it (FOR
|
||||
# SHARE) until its transaction commits. A thread_meta delete racing a
|
||||
# finalization therefore commits after the line, and this step removes it,
|
||||
# so a task finishing during the delete cannot leave a line behind.
|
||||
try:
|
||||
task_event_repo = getattr(request.app.state, "scheduled_task_event_repo", None)
|
||||
if task_event_repo is not None:
|
||||
await task_event_repo.delete_by_thread(thread_id, user_id=user_id)
|
||||
except Exception:
|
||||
logger.debug("Could not delete scheduled task events for thread %s (not critical)", sanitize_log_param(thread_id))
|
||||
|
||||
# Remove every user's read marker for the thread (best-effort): the thread
|
||||
# is gone, so its unread state is meaningless for all of them.
|
||||
try:
|
||||
thread_read_repo = getattr(request.app.state, "thread_read_repo", None)
|
||||
if thread_read_repo is not None:
|
||||
await thread_read_repo.delete_by_thread(thread_id)
|
||||
except Exception:
|
||||
logger.debug("Could not delete read markers for thread %s (not critical)", sanitize_log_param(thread_id))
|
||||
|
||||
# Tear down any live browser session (best-effort). Sessions are keyed only
|
||||
# by thread_id, so leaving one alive after the owner deletes the thread lets
|
||||
# a later caller who guesses the id reuse the retained page/cookies.
|
||||
@ -914,6 +948,11 @@ async def create_thread(body: ThreadCreateRequest, request: Request) -> ThreadRe
|
||||
thread_owner_kwargs = {"user_id": thread_owner_user_id} if thread_owner_user_id else {}
|
||||
# ``body.metadata`` is already stripped of server-reserved keys by
|
||||
# ``ThreadCreateRequest._strip_reserved`` — see the model definition.
|
||||
# A server-side caller (an extension handle) marks the thread it creates;
|
||||
# request.state is never HTTP input.
|
||||
metadata = dict(body.metadata)
|
||||
if (origin := admit_origin(getattr(request.state, "run_origin", None))) is not None:
|
||||
metadata[DEERFLOW_ORIGIN_KEY] = origin
|
||||
|
||||
# Idempotency: return existing record when already present
|
||||
existing_record = await _resolve_existing_thread(thread_store, thread_id, thread_owner_user_id, thread_owner_kwargs)
|
||||
@ -928,7 +967,7 @@ async def create_thread(body: ThreadCreateRequest, request: Request) -> ThreadRe
|
||||
thread_id,
|
||||
assistant_id=getattr(body, "assistant_id", None),
|
||||
**thread_owner_kwargs,
|
||||
metadata=body.metadata,
|
||||
metadata=metadata,
|
||||
project_id=body.project_id,
|
||||
)
|
||||
except ProjectNotAssignableError:
|
||||
@ -966,7 +1005,7 @@ async def create_thread(body: ThreadCreateRequest, request: Request) -> ThreadRe
|
||||
"source": "input",
|
||||
"writes": None,
|
||||
"parents": {},
|
||||
**body.metadata,
|
||||
**metadata,
|
||||
"created_at": now,
|
||||
}
|
||||
await checkpointer.aput(config, empty_checkpoint(), ckpt_metadata, {})
|
||||
@ -1228,6 +1267,7 @@ async def search_threads(body: ThreadSearchRequest, request: Request) -> list[Th
|
||||
)
|
||||
except InvalidMetadataFilterError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
unread_ids = await _unread_thread_ids(request, [r["thread_id"] for r in rows])
|
||||
return [
|
||||
ThreadResponse(
|
||||
thread_id=r["thread_id"],
|
||||
@ -1240,11 +1280,46 @@ async def search_threads(body: ThreadSearchRequest, request: Request) -> list[Th
|
||||
metadata=r.get("metadata", {}),
|
||||
values={"title": r["display_name"]} if r.get("display_name") else {},
|
||||
interrupts={},
|
||||
unread=(r["thread_id"] in unread_ids) if unread_ids is not None else None,
|
||||
)
|
||||
for r in rows
|
||||
]
|
||||
|
||||
|
||||
async def _unread_thread_ids(request: Request, thread_ids: list[str]) -> set[str] | None:
|
||||
"""One query for the page's unread threads; ``None`` (unknown) without SQL read state.
|
||||
|
||||
The list must keep loading when the lookup fails, so a failure degrades to
|
||||
``None`` instead of failing the search.
|
||||
"""
|
||||
repo = getattr(request.app.state, "thread_read_repo", None)
|
||||
if repo is None:
|
||||
return None
|
||||
if not thread_ids:
|
||||
return set()
|
||||
try:
|
||||
return await repo.unread_thread_ids(user_id=str(get_effective_user_id()), thread_ids=thread_ids)
|
||||
except Exception:
|
||||
logger.warning("Could not resolve unread threads for a thread search", exc_info=True)
|
||||
return None
|
||||
|
||||
|
||||
@router.post("/{thread_id}/read", response_model=ThreadReadResponse)
|
||||
@require_permission("threads", "read", owner_check=True, require_existing=True)
|
||||
async def mark_thread_read(thread_id: ThreadId, request: Request) -> ThreadReadResponse:
|
||||
"""Mark the thread read for the caller (per user, monotonic across devices).
|
||||
|
||||
A thread already read up to its latest run writes nothing and leaves
|
||||
``read_version`` unchanged, so other tabs and devices do not refetch.
|
||||
"""
|
||||
repo = getattr(request.app.state, "thread_read_repo", None)
|
||||
if repo is None:
|
||||
raise HTTPException(status_code=503, detail="Thread read state is not available")
|
||||
user_id = str(get_effective_user_id())
|
||||
await repo.mark_read(user_id=user_id, thread_id=thread_id)
|
||||
return ThreadReadResponse(unread=False, read_version=await repo.read_version(user_id=user_id))
|
||||
|
||||
|
||||
@router.patch("/{thread_id}", response_model=ThreadResponse)
|
||||
@require_permission("threads", "write", owner_check=True, require_existing=True)
|
||||
async def patch_thread(thread_id: ThreadId, body: ThreadPatchRequest, request: Request) -> ThreadResponse:
|
||||
|
||||
@ -7,6 +7,7 @@ from pydantic import BaseModel, ConfigDict, StringConstraints, ValidationError
|
||||
|
||||
from app.gateway.auth_disabled import AUTH_SOURCE_SESSION
|
||||
from app.gateway.deps import get_current_user_from_request
|
||||
from deerflow.config.channel_connections_config import NotificationLocale
|
||||
from deerflow.persistence.engine import get_session_factory
|
||||
from deerflow.persistence.user.preferences import UserPreferencesRepository
|
||||
|
||||
@ -23,6 +24,10 @@ class Preferences(BaseModel):
|
||||
# remembered values against the selected model; the factory enforces
|
||||
# declared contracts while preserving legacy direct-request behavior.
|
||||
reasoning_effort: Annotated[str, StringConstraints(max_length=32, pattern=r"^[A-Za-z0-9_.-]+$")] | None = None
|
||||
# The web app's UI language, synced on sign-in and on every language
|
||||
# switch. Scheduled-task IM notices are written in it; without it they use
|
||||
# ``channel_connections.notification_locale``.
|
||||
locale: NotificationLocale | None = None
|
||||
|
||||
|
||||
async def _owner(request: Request, expected_user: str) -> str:
|
||||
|
||||
35
backend/app/gateway/run_origin.py
Normal file
35
backend/app/gateway/run_origin.py
Normal file
@ -0,0 +1,35 @@
|
||||
"""Resolve the server-owned ``deerflow_origin`` of a run request.
|
||||
|
||||
Only two sources are trusted:
|
||||
|
||||
1. a server-side launcher that sets ``request.state.run_origin`` on the
|
||||
request object it builds (the scheduler, MCP notifications, extension
|
||||
handles). HTTP input can never reach ``request.state``;
|
||||
2. an internal HTTP caller (the IM channel manager, authenticated with the
|
||||
internal token) that declares it in ``body.metadata``.
|
||||
|
||||
Every other caller gets ``None``, and run admission drops any client copy of
|
||||
the key (``services.start_run``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
from app.gateway.auth_disabled import AUTH_SOURCE_INTERNAL
|
||||
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY, admit_origin
|
||||
|
||||
|
||||
def resolve_request_origin(request: Any, body_metadata: Mapping | None) -> dict[str, str] | None:
|
||||
"""The origin to stamp on this run, or ``None`` for interactive callers."""
|
||||
state = getattr(request, "state", None)
|
||||
host = getattr(state, "run_origin", None)
|
||||
if host is not None:
|
||||
return admit_origin(host)
|
||||
if getattr(state, "auth_source", None) == AUTH_SOURCE_INTERNAL and isinstance(body_metadata, Mapping):
|
||||
return admit_origin(body_metadata.get(DEERFLOW_ORIGIN_KEY))
|
||||
return None
|
||||
|
||||
|
||||
__all__ = ["resolve_request_origin"]
|
||||
@ -37,6 +37,7 @@ from app.gateway.internal_auth import (
|
||||
)
|
||||
from app.gateway.knowledge_scope_admission import admit_message_knowledge_scope
|
||||
from app.gateway.run_models import RunCreateRequest
|
||||
from app.gateway.run_origin import resolve_request_origin
|
||||
from app.gateway.utils import sanitize_log_param
|
||||
from app.mcp_tasks.errors import PermanentNotificationError
|
||||
from deerflow.agents.human_input import read_human_input_response
|
||||
@ -88,6 +89,7 @@ from deerflow.runtime.events.message_identity import MESSAGE_SEQ_KEY
|
||||
from deerflow.runtime.goal import goal_thread_lock
|
||||
from deerflow.runtime.journal import build_checkpoint_history_seed_events
|
||||
from deerflow.runtime.keyed_lock import KeyedLockTable
|
||||
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY, make_origin
|
||||
from deerflow.runtime.runs.naming import resolve_root_run_name
|
||||
from deerflow.runtime.secret_context import (
|
||||
LegacyRunMetadataSecretError,
|
||||
@ -261,9 +263,15 @@ async def _ensure_thread_metadata(
|
||||
# /threads/{id}/move — so the key must not persist either.
|
||||
if key not in (DEERFLOW_TRACE_METADATA_KEY, THREAD_PROJECT_METADATA_KEY)
|
||||
}
|
||||
# A run that names its thread in the input (a scheduled run) creates it
|
||||
# named: the worker copies the title to the thread list only when the
|
||||
# run ends, and a server-created thread is listed as soon as it exists.
|
||||
run_input = (record.kwargs or {}).get("input")
|
||||
title = run_input.get("title") if isinstance(run_input, dict) else None
|
||||
existing = await thread_store.create(
|
||||
record.thread_id,
|
||||
assistant_id=record.assistant_id,
|
||||
display_name=title if isinstance(title, str) and title.strip() else None,
|
||||
metadata=metadata,
|
||||
)
|
||||
return existing
|
||||
@ -1866,12 +1874,24 @@ async def start_run(
|
||||
# without this the run record is the one surface that persists a forged
|
||||
# id, disagreeing with the response header, the logs, and the
|
||||
# checkpoint. The caller's own metadata keys are preserved.
|
||||
# deerflow_origin is server-owned as well: only a server-side launcher
|
||||
# (request.state.run_origin) or the internal channel caller may set it,
|
||||
# and both metadata forks drop any client copy. build_run_config merges
|
||||
# body.config["metadata"] into the live config, so the second pop below
|
||||
# is what keeps a forged value out of the live run config.
|
||||
origin = resolve_request_origin(request, body.metadata)
|
||||
run_metadata = _admit_scheduler_metadata(body.metadata, trusted=scheduled_task_runtime is not None)
|
||||
run_metadata.pop(DEERFLOW_ORIGIN_KEY, None)
|
||||
if origin is not None:
|
||||
run_metadata[DEERFLOW_ORIGIN_KEY] = origin
|
||||
run_metadata[DEERFLOW_TRACE_METADATA_KEY] = ensure_trace_id()
|
||||
|
||||
config = build_run_config(thread_id, body.config, run_metadata, assistant_id=body.assistant_id)
|
||||
if isinstance(config.get("metadata"), dict):
|
||||
config["metadata"] = _admit_scheduler_metadata(config["metadata"], trusted=scheduled_task_runtime is not None)
|
||||
config["metadata"].pop(DEERFLOW_ORIGIN_KEY, None)
|
||||
if origin is not None:
|
||||
config["metadata"][DEERFLOW_ORIGIN_KEY] = origin
|
||||
await apply_checkpoint_to_run_config(config, body=body, thread_id=thread_id, request=request)
|
||||
|
||||
# Merge DeerFlow-specific context overrides into both ``configurable`` and ``context``.
|
||||
@ -2287,6 +2307,8 @@ async def launch_scheduled_thread_run(
|
||||
state=SimpleNamespace(
|
||||
user=get_internal_user(),
|
||||
auth_source=AUTH_SOURCE_INTERNAL,
|
||||
# Every trigger (schedule or "Run now") is server-started.
|
||||
run_origin=make_origin("schedule"),
|
||||
),
|
||||
cookies={},
|
||||
)
|
||||
@ -2379,7 +2401,7 @@ async def launch_mcp_task_notification_run(
|
||||
request = SimpleNamespace(
|
||||
app=app,
|
||||
headers={INTERNAL_OWNER_USER_ID_HEADER_NAME: owner_user_id},
|
||||
state=SimpleNamespace(user=get_internal_user(), auth_source=AUTH_SOURCE_INTERNAL),
|
||||
state=SimpleNamespace(user=get_internal_user(), auth_source=AUTH_SOURCE_INTERNAL, run_origin=make_origin("mcp_notification")),
|
||||
cookies={},
|
||||
)
|
||||
body = RunCreateRequest(
|
||||
|
||||
28
backend/app/gateway/sse_headers.py
Normal file
28
backend/app/gateway/sse_headers.py
Normal file
@ -0,0 +1,28 @@
|
||||
"""Response headers shared by every Server-Sent Events endpoint.
|
||||
|
||||
``no-transform`` tells compressing proxies (for example the Next.js rewrite
|
||||
proxy used by ``pnpm start``) not to gzip the body. A compressing proxy has to
|
||||
buffer a stream before it can emit compressed blocks, so without it SSE frames
|
||||
reach the browser in bursts instead of as they are produced.
|
||||
``X-Accel-Buffering: no`` does the same for nginx.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
SSE_CACHE_CONTROL = "no-cache, no-transform"
|
||||
|
||||
|
||||
def sse_response_headers(*, content_location: str | None = None) -> dict[str, str]:
|
||||
"""Return the headers for an SSE streaming response.
|
||||
|
||||
``content_location`` is set only by the routes that create a run, so the
|
||||
LangGraph SDK can read the run id from it.
|
||||
"""
|
||||
headers = {
|
||||
"Cache-Control": SSE_CACHE_CONTROL,
|
||||
"Connection": "keep-alive",
|
||||
"X-Accel-Buffering": "no",
|
||||
}
|
||||
if content_location is not None:
|
||||
headers["Content-Location"] = content_location
|
||||
return headers
|
||||
@ -1,7 +1,8 @@
|
||||
"""Outbox delivery worker for scheduled-task IM notifications (issue #4254).
|
||||
|
||||
The completion hook (``ScheduledTaskService._enqueue_run_notifications``)
|
||||
only writes durable outbox rows; this worker owns the actual IM send.
|
||||
The scheduler's finalization observer (``ScheduledTaskService._on_finalization``)
|
||||
writes durable outbox rows in the transaction that records each outcome; this
|
||||
worker owns the actual IM send. Notice text lives in ``notification_text``.
|
||||
Execution state and delivery state stay separated: the worker moves outbox
|
||||
rows through ``pending -> sending -> sent|failed`` and never touches run or
|
||||
task rows. Retries with backoff live in the repository (``mark_failed``);
|
||||
@ -13,12 +14,15 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import inspect
|
||||
import logging
|
||||
import re
|
||||
from collections.abc import Callable
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from app.channels.base import ChannelUnavailable
|
||||
from app.channels.capabilities import supports_proactive_notifications
|
||||
from app.scheduler.notification_text import DEFAULT_NOTIFICATION_LOCALE, redact_egress_text, render_notification_text, wants_result_summary
|
||||
|
||||
__all__ = ["NotificationDeliveryWorker", "redact_egress_text", "render_notification_text"]
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@ -27,9 +31,10 @@ logger = logging.getLogger(__name__)
|
||||
# are both accepted (``ChannelService.get_channel`` is sync).
|
||||
ChannelResolver = Callable[[str], Any]
|
||||
|
||||
# Resolves the run's final answer for a completed run, keyed
|
||||
# ``(run_id, owner_user_id)``, or None when no summary is available. Sync or
|
||||
# async callables are both accepted; failures fall back to the skeleton text.
|
||||
# Resolves the run's final reply, keyed ``(run_id, owner_user_id)``, or None
|
||||
# when no reply is available. The renderer reduces it to the tasks page's
|
||||
# one-line run summary. Sync or async callables are both accepted; failures
|
||||
# send the notice without a result line.
|
||||
SummaryResolver = Callable[[str, str | None], Any]
|
||||
|
||||
# Lists an owner's channel connections (``ChannelConnectionRepository
|
||||
@ -38,9 +43,6 @@ SummaryResolver = Callable[[str, str | None], Any]
|
||||
# it to confirm, at send time, that the delivery's target is still bound.
|
||||
ConnectionResolver = Callable[[str], Any]
|
||||
|
||||
_ERROR_TEXT_LIMIT = 500
|
||||
_SUMMARY_TEXT_LIMIT = 1000
|
||||
|
||||
# A claimed row stuck in "sending" longer than this is considered orphaned
|
||||
# (process died between claim and the final status write) and flipped back
|
||||
# to pending. Generous on purpose: a healthy send plus status write finishes
|
||||
@ -53,97 +55,6 @@ _STALE_SENDING_TIMEOUT_SECONDS = 600
|
||||
_STOP_TIMEOUT_SECONDS = 5.0
|
||||
|
||||
|
||||
def _truncate(text: str, *, limit: int = _ERROR_TEXT_LIMIT) -> str:
|
||||
return text if len(text) <= limit else text[: limit - 1] + "\u2026"
|
||||
|
||||
|
||||
_EGRESS_REDACTION = "[redacted]"
|
||||
# Best-effort scrub before run content leaves for a third-party IM platform.
|
||||
# Patterns mirror the high-confidence secret detectors in skillscan.
|
||||
_EGRESS_SECRET_PATTERNS: tuple[re.Pattern[str], ...] = (
|
||||
re.compile(
|
||||
r"-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----.*?-----END [A-Z0-9 ]*PRIVATE KEY-----",
|
||||
re.DOTALL,
|
||||
),
|
||||
re.compile(r"\b(?:AKIA|ASIA)[A-Z0-9]{16}\b"),
|
||||
re.compile(r"\bgh[pousr]_[A-Za-z0-9_]{20,}\b"),
|
||||
re.compile(r"\bxox[baprs]-[A-Za-z0-9-]{20,}\b"),
|
||||
re.compile(r"\bsk-[A-Za-z0-9]{20,}\b"),
|
||||
re.compile(r"(?im)\b(token|password|passwd|api[_-]?key|secret|credential)s?\b\s*[:=]\s*[\"']?([^\"'\s#]+)"),
|
||||
)
|
||||
|
||||
|
||||
def redact_egress_text(text: str) -> str:
|
||||
"""Scrub likely secrets from text before it is pushed to external IM."""
|
||||
if not text:
|
||||
return text
|
||||
redacted = text
|
||||
for pattern in _EGRESS_SECRET_PATTERNS:
|
||||
if pattern.groups:
|
||||
redacted = pattern.sub(lambda match: f"{match.group(1)}={_EGRESS_REDACTION}", redacted)
|
||||
else:
|
||||
redacted = pattern.sub(_EGRESS_REDACTION, redacted)
|
||||
return redacted
|
||||
|
||||
|
||||
# Readable text for host-defined goal reason codes (``blocked:`` prefix
|
||||
# removed). Keys mirror the goal blockers and stand-down reasons.
|
||||
_GOAL_REASON_TEXT = {
|
||||
"missing_evidence": "the goal check found evidence missing",
|
||||
"needs_user_input": "it needs your input",
|
||||
"run_failed": "the goal check found the work unfinished",
|
||||
"external_wait": "it is waiting on something external",
|
||||
"goal_not_met_yet": "the goal is not met yet",
|
||||
"no_verdict": "no goal verdict was recorded",
|
||||
"consecutive_unmet": "3 scheduled runs in a row did not meet the goal",
|
||||
"evaluator_failed": "the goal check could not run",
|
||||
"max_continuations_reached": "the continuation limit was reached",
|
||||
"no_progress_detected": "no progress was made between turns",
|
||||
"token_capped": "the token budget was reached",
|
||||
"no_durable_end_of_turn": "no final reply was saved",
|
||||
"thread_changed_after_evaluation": "the conversation changed during the goal check",
|
||||
"thread_changed_before_continuation": "the conversation changed during the goal check",
|
||||
}
|
||||
|
||||
|
||||
def render_notification_text(delivery: dict[str, Any]) -> str:
|
||||
"""Render a bounded markdown summary of the run outcome for IM push."""
|
||||
payload = delivery.get("payload") or {}
|
||||
event = delivery.get("event") or ""
|
||||
task_label = payload.get("task_title") or payload.get("task_id") or delivery.get("task_id")
|
||||
if event in {"run_unmet", "task_paused"}:
|
||||
label = "Scheduled task goal was not met" if event == "run_unmet" else "Scheduled task automatically paused"
|
||||
lines = [f"**{label}**", f"Task: `{task_label}`"]
|
||||
# Only host-defined result codes are translated; model/provider error
|
||||
# text and unknown codes are never forwarded.
|
||||
reason = payload.get("reason_code")
|
||||
if isinstance(reason, str):
|
||||
reason = reason.removeprefix("blocked:")
|
||||
lines.append(f"Reason: {_GOAL_REASON_TEXT.get(reason, 'unknown') if isinstance(reason, str) else 'unknown'}.")
|
||||
lines.append("See the DeerFlow workspace for the result and next steps.")
|
||||
elif event == "run_failed":
|
||||
lines = ["**Scheduled task failed**", f"Task: `{task_label}`"]
|
||||
# Do not forward raw error text to external IM: scheduled runs can
|
||||
# surface hostnames, paths, and token fragments in tracebacks. Users
|
||||
# can open the workspace for the full detail.
|
||||
lines.append("See the DeerFlow workspace for error details.")
|
||||
else:
|
||||
heading = "Scheduled task completed"
|
||||
if payload.get("relied_on_assumption") is True:
|
||||
heading += " — met, relying on stated assumptions"
|
||||
lines = [f"**{heading}**", f"Task: `{task_label}`"]
|
||||
# The result summary belongs to a successful outcome only: on failed
|
||||
# runs a partial answer would be misleading, so the error line above
|
||||
# stays the sole detail.
|
||||
summary = payload.get("result_summary")
|
||||
if isinstance(summary, str) and summary.strip():
|
||||
safe_summary = redact_egress_text(summary.strip())
|
||||
lines.append(f"Result: {_truncate(safe_summary, limit=_SUMMARY_TEXT_LIMIT)}")
|
||||
if delivery.get("run_id"):
|
||||
lines.append(f"Run: `{delivery['run_id']}`")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
class NotificationDeliveryWorker:
|
||||
"""Polls the notification outbox and pushes due deliveries over IM."""
|
||||
|
||||
@ -158,6 +69,7 @@ class NotificationDeliveryWorker:
|
||||
resolve_connections: ConnectionResolver | None = None,
|
||||
stale_sending_timeout_seconds: int = _STALE_SENDING_TIMEOUT_SECONDS,
|
||||
stop_timeout_seconds: float = _STOP_TIMEOUT_SECONDS,
|
||||
default_locale: str = DEFAULT_NOTIFICATION_LOCALE,
|
||||
) -> None:
|
||||
self._delivery_repo = delivery_repo
|
||||
self._resolve_channel = resolve_channel
|
||||
@ -167,6 +79,9 @@ class NotificationDeliveryWorker:
|
||||
self._batch_size = batch_size
|
||||
self._stale_sending_timeout_seconds = stale_sending_timeout_seconds
|
||||
self._stop_timeout_seconds = stop_timeout_seconds
|
||||
# ``channel_connections.notification_locale``: the language of rows
|
||||
# whose owner has no UI language preference (and of legacy rows).
|
||||
self._default_locale = default_locale
|
||||
self._task: asyncio.Task | None = None
|
||||
self._stop = asyncio.Event()
|
||||
|
||||
@ -227,11 +142,11 @@ class NotificationDeliveryWorker:
|
||||
logger.warning("Failed to reset stale sending rows; retrying next poll", exc_info=True)
|
||||
|
||||
async def _resolve_summary(self, delivery: dict[str, Any]) -> str | None:
|
||||
"""Best-effort lookup of the run's final answer at delivery time.
|
||||
"""Best-effort lookup of the run's final reply at delivery time.
|
||||
|
||||
Read at delivery time, not enqueue time, so retried deliveries see
|
||||
the freshest value and the outbox payload stays skeleton-only. Any
|
||||
failure degrades to the skeleton notification instead of blocking it.
|
||||
the freshest value and the outbox payload stays small. Any failure
|
||||
sends the notice without its result line instead of blocking it.
|
||||
"""
|
||||
if self._resolve_run_summary is None:
|
||||
return None
|
||||
@ -298,6 +213,17 @@ class NotificationDeliveryWorker:
|
||||
async def _deliver(self, delivery: dict[str, Any]) -> None:
|
||||
delivery_id = delivery["id"]
|
||||
provider = delivery.get("provider") or ""
|
||||
if not supports_proactive_notifications(provider):
|
||||
# The scheduler enqueues only for providers with proactive push;
|
||||
# a row for any other provider predates that rule. It can never
|
||||
# be sent, so it ends now instead of using up its retries.
|
||||
await self._delivery_repo.mark_failed(
|
||||
delivery_id,
|
||||
claim_token=delivery.get("claim_token"),
|
||||
error="provider does not support proactive notifications",
|
||||
terminal=True,
|
||||
)
|
||||
return
|
||||
if self._resolve_connections is not None:
|
||||
# Re-check the binding at delivery time, before anything else: a
|
||||
# target the owner has disconnected since enqueue must never
|
||||
@ -331,7 +257,7 @@ class NotificationDeliveryWorker:
|
||||
await self._delivery_repo.mark_failed(delivery_id, claim_token=delivery.get("claim_token"), error=f"channel '{provider}' is not running", count_attempt=False)
|
||||
return
|
||||
enriched = delivery
|
||||
if delivery.get("event") == "run_completed":
|
||||
if wants_result_summary(delivery):
|
||||
summary = await self._resolve_summary(delivery)
|
||||
if summary is not None:
|
||||
# Copy before enriching: the claimed row must not be mutated
|
||||
@ -341,7 +267,7 @@ class NotificationDeliveryWorker:
|
||||
try:
|
||||
await channel.send_notification(
|
||||
target=delivery.get("target") or "",
|
||||
text_markdown=render_notification_text(enriched),
|
||||
text_markdown=render_notification_text(enriched, default_locale=self._default_locale),
|
||||
)
|
||||
except ChannelUnavailable as exc:
|
||||
await self._delivery_repo.mark_failed(delivery_id, claim_token=delivery.get("claim_token"), error=str(exc), count_attempt=False)
|
||||
|
||||
316
backend/app/scheduler/notification_text.py
Normal file
316
backend/app/scheduler/notification_text.py
Normal file
@ -0,0 +1,316 @@
|
||||
"""Text of scheduled-task IM notices.
|
||||
|
||||
A notice reads on its own, in the owner's language, with one item per line:
|
||||
|
||||
1. the task title (``Scheduled task “{title}”``);
|
||||
2. what happened, plus the last run's outcome where the main sentence would
|
||||
otherwise hide it;
|
||||
3. ``Result: {summary}``, the same one-line run summary the tasks page shows,
|
||||
only when the agent replied (a run that succeeded or missed its goal);
|
||||
4. where to look: ``Open DeerFlow → Scheduled tasks for details.``
|
||||
|
||||
It carries no task, run or thread IDs, no URLs (most deployments run on
|
||||
localhost or a LAN, so a link would be dead on the phone that receives it), no
|
||||
timestamps and no internal codes. Agent-written text (the stop condition, the
|
||||
reply) passes through :func:`redact_egress_text` before it leaves for a
|
||||
third-party platform.
|
||||
|
||||
Payload (``payload_version`` 2, written by the scheduler's finalization
|
||||
observer): ``task_title``, ``locale`` (``None`` = the configured default),
|
||||
``reason_code``, ``latest_reason_code``, ``run_status``, optional
|
||||
``relied_on_assumption`` (present only when a goal was met), ``stop_condition``
|
||||
(``task_stopped``) and ``max_runs`` (``task_finished``). Rows written before
|
||||
version 2 render with the same templates.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any, get_args
|
||||
|
||||
from deerflow.config.channel_connections_config import NotificationLocale
|
||||
from deerflow.persistence.scheduled_task_runs.sql import run_summary
|
||||
|
||||
# One source of truth: the config type that ``/preferences`` also accepts.
|
||||
NOTIFICATION_LOCALES: tuple[str, ...] = get_args(NotificationLocale)
|
||||
DEFAULT_NOTIFICATION_LOCALE = "en-US"
|
||||
|
||||
_CONDITION_LIMIT = 200
|
||||
_TITLE_LIMIT = 200
|
||||
|
||||
_EGRESS_REDACTION = "[redacted]"
|
||||
# Best-effort scrub before run content leaves for a third-party IM platform.
|
||||
# Patterns mirror the high-confidence secret detectors in skillscan.
|
||||
_EGRESS_SECRET_PATTERNS: tuple[re.Pattern[str], ...] = (
|
||||
re.compile(
|
||||
r"-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----.*?-----END [A-Z0-9 ]*PRIVATE KEY-----",
|
||||
re.DOTALL,
|
||||
),
|
||||
re.compile(r"\b(?:AKIA|ASIA)[A-Z0-9]{16}\b"),
|
||||
re.compile(r"\bgh[pousr]_[A-Za-z0-9_]{20,}\b"),
|
||||
re.compile(r"\bxox[baprs]-[A-Za-z0-9-]{20,}\b"),
|
||||
re.compile(r"\bsk-[A-Za-z0-9]{20,}\b"),
|
||||
re.compile(r"(?im)\b(token|password|passwd|api[_-]?key|secret|credential)s?\b\s*[:=]\s*[\"']?([^\"'\s#]+)"),
|
||||
)
|
||||
|
||||
|
||||
def redact_egress_text(text: str) -> str:
|
||||
"""Scrub likely secrets from text before it is pushed to external IM."""
|
||||
if not text:
|
||||
return text
|
||||
redacted = text
|
||||
for pattern in _EGRESS_SECRET_PATTERNS:
|
||||
if pattern.groups:
|
||||
redacted = pattern.sub(lambda match: f"{match.group(1)}={_EGRESS_REDACTION}", redacted)
|
||||
else:
|
||||
redacted = pattern.sub(_EGRESS_REDACTION, redacted)
|
||||
return redacted
|
||||
|
||||
|
||||
# Readable text for host-defined goal reason codes (``blocked:`` prefix
|
||||
# removed), per locale. Keys mirror the goal blockers and stand-down reasons;
|
||||
# model or provider error text is never forwarded.
|
||||
_GOAL_REASON_TEXT: dict[str, dict[str, str]] = {
|
||||
"en-US": {
|
||||
"missing_evidence": "the goal check found evidence missing",
|
||||
"needs_user_input": "it needs your input",
|
||||
"run_failed": "the goal check found the work unfinished",
|
||||
"external_wait": "it is waiting on something external",
|
||||
"goal_not_met_yet": "the goal is not met yet",
|
||||
"no_verdict": "no goal verdict was recorded",
|
||||
"consecutive_unmet": "3 scheduled runs in a row did not meet the goal",
|
||||
"evaluator_failed": "the goal check could not run",
|
||||
"max_continuations_reached": "the continuation limit was reached",
|
||||
"no_progress_detected": "no progress was made between turns",
|
||||
"token_capped": "the token budget was reached",
|
||||
"no_durable_end_of_turn": "no final reply was saved",
|
||||
"thread_changed_after_evaluation": "the conversation changed during the goal check",
|
||||
"thread_changed_before_continuation": "the conversation changed during the goal check",
|
||||
"unknown": "no reason was recorded",
|
||||
},
|
||||
"zh-CN": {
|
||||
"missing_evidence": "目标检查发现缺少依据",
|
||||
"needs_user_input": "需要你提供信息",
|
||||
"run_failed": "目标检查发现工作未完成",
|
||||
"external_wait": "正在等待外部条件",
|
||||
"goal_not_met_yet": "目标尚未达成",
|
||||
"no_verdict": "没有记录目标检查结果",
|
||||
"consecutive_unmet": "连续 3 次定时运行未达成目标",
|
||||
"evaluator_failed": "目标检查未能运行",
|
||||
"max_continuations_reached": "已达到继续处理的次数上限",
|
||||
"no_progress_detected": "多轮之间没有进展",
|
||||
"token_capped": "已达到 token 预算",
|
||||
"no_durable_end_of_turn": "没有保存最终回复",
|
||||
"thread_changed_after_evaluation": "目标检查期间对话发生了变化",
|
||||
"thread_changed_before_continuation": "目标检查期间对话发生了变化",
|
||||
"unknown": "没有记录原因",
|
||||
},
|
||||
}
|
||||
|
||||
_TEXT: dict[str, dict[str, str]] = {
|
||||
"en-US": {
|
||||
"title": "Scheduled task “{title}”",
|
||||
"untitled": "Untitled task",
|
||||
"run_completed": "Finished a run.",
|
||||
"goal_met": "Goal met.",
|
||||
"goal_met_with_assumption": "Goal met, with an assumption.",
|
||||
"run_failed": "A run failed.",
|
||||
"run_unmet": "Ran, but the goal wasn't met: {reason}.",
|
||||
"run_unmet_bare": "Ran, but the goal wasn't met.",
|
||||
"task_paused": "Paused after 3 runs in a row missed the goal: {reason}.",
|
||||
"task_paused_bare": "Paused after 3 runs in a row missed the goal.",
|
||||
"task_stopped_condition": "Paused by agent: its stop condition was met ({condition}).",
|
||||
"task_stopped": "Paused by agent: its stop condition was met.",
|
||||
"finished_max_runs": "Finished: all {max_runs} automatic runs are done.",
|
||||
"finished_one_run": "Finished: its one automatic run is done.",
|
||||
"finished_all_runs": "Finished: all automatic runs are done.",
|
||||
"finished_end_at": "Finished: its end time has been reached.",
|
||||
"finished": "Finished.",
|
||||
"update": "There is an update.",
|
||||
"last_failed": "The last run failed.",
|
||||
"last_unmet": "The last run didn't meet the goal: {reason}.",
|
||||
"last_unmet_bare": "The last run didn't meet the goal.",
|
||||
"last_interrupted": "The last run was interrupted.",
|
||||
"result": "Result: {summary}",
|
||||
"closing": "Open DeerFlow → Scheduled tasks for details.",
|
||||
"sentence_gap": " ",
|
||||
},
|
||||
"zh-CN": {
|
||||
"title": "定时任务“{title}”",
|
||||
"untitled": "未命名任务",
|
||||
"run_completed": "完成了一次运行。",
|
||||
"goal_met": "目标已达成。",
|
||||
"goal_met_with_assumption": "目标已达成(含假设)。",
|
||||
"run_failed": "一次运行出错了。",
|
||||
"run_unmet": "已运行,但目标未达成:{reason}。",
|
||||
"run_unmet_bare": "已运行,但目标未达成。",
|
||||
"task_paused": "连续 3 次未达成目标,已自动暂停。最近一次的原因:{reason}。",
|
||||
"task_paused_bare": "连续 3 次未达成目标,已自动暂停。",
|
||||
"task_stopped_condition": "已由智能体暂停:停止条件已满足({condition})。",
|
||||
"task_stopped": "已由智能体暂停:停止条件已满足。",
|
||||
"finished_max_runs": "已结束:{max_runs} 次自动运行已全部完成。",
|
||||
"finished_one_run": "已结束:唯一一次自动运行已完成。",
|
||||
"finished_all_runs": "已结束:自动运行已全部完成。",
|
||||
"finished_end_at": "已结束:已到结束时间。",
|
||||
"finished": "已结束。",
|
||||
"update": "有新的动态。",
|
||||
"last_failed": "最后一次运行出错了。",
|
||||
"last_unmet": "最后一次运行未达成目标:{reason}。",
|
||||
"last_unmet_bare": "最后一次运行未达成目标。",
|
||||
"last_interrupted": "最后一次运行被中断了。",
|
||||
"result": "结果:{summary}",
|
||||
"closing": "在 DeerFlow 的定时任务页查看详情。",
|
||||
"sentence_gap": "",
|
||||
},
|
||||
}
|
||||
|
||||
# Events whose notice may carry the run's one-line result, and the run
|
||||
# outcomes that have one (the agent replied; a failed or interrupted run's
|
||||
# partial reply is not a result).
|
||||
_SUMMARY_EVENTS = frozenset({"run_completed", "run_unmet", "task_paused", "task_stopped", "task_finished"})
|
||||
_SUMMARY_RUN_STATUSES = frozenset({"success", "unmet"})
|
||||
# Rows written before payload version 2 carry no ``run_status`` for these.
|
||||
_LEGACY_RUN_STATUS = {"run_completed": "success", "run_failed": "failed", "run_unmet": "unmet", "task_paused": "unmet"}
|
||||
|
||||
|
||||
def _payload(delivery: dict[str, Any]) -> dict[str, Any]:
|
||||
payload = delivery.get("payload")
|
||||
return payload if isinstance(payload, dict) else {}
|
||||
|
||||
|
||||
def _run_status(delivery: dict[str, Any]) -> str | None:
|
||||
status = _payload(delivery).get("run_status")
|
||||
if isinstance(status, str) and status:
|
||||
return status
|
||||
return _LEGACY_RUN_STATUS.get(delivery.get("event") or "")
|
||||
|
||||
|
||||
def wants_result_summary(delivery: dict[str, Any]) -> bool:
|
||||
"""Whether the notice carries a result line, so the worker resolves the run's reply."""
|
||||
return bool(delivery.get("run_id")) and delivery.get("event") in _SUMMARY_EVENTS and _run_status(delivery) in _SUMMARY_RUN_STATUSES
|
||||
|
||||
|
||||
def resolve_locale(delivery: dict[str, Any], default_locale: str = DEFAULT_NOTIFICATION_LOCALE) -> str:
|
||||
"""The row's locale (the owner's UI language), else the configured default."""
|
||||
locale = _payload(delivery).get("locale")
|
||||
if isinstance(locale, str) and locale in NOTIFICATION_LOCALES:
|
||||
return locale
|
||||
return default_locale if default_locale in NOTIFICATION_LOCALES else DEFAULT_NOTIFICATION_LOCALE
|
||||
|
||||
|
||||
def _one_line(text: str) -> str:
|
||||
return " ".join(text.split())
|
||||
|
||||
|
||||
def _truncate(text: str, *, limit: int) -> str:
|
||||
return text if len(text) <= limit else text[: limit - 1].rstrip() + "…"
|
||||
|
||||
|
||||
# Codes that only restate "the goal wasn't met" (or say nothing): the notice
|
||||
# uses its reason-less sentence instead of "…wasn't met: the goal is not met yet."
|
||||
_NO_EXTRA_REASON = frozenset({"goal_not_met_yet", "unknown"})
|
||||
|
||||
|
||||
def _reason_text(code: object, locale: str) -> str | None:
|
||||
"""Readable reason for a goal code, or None when it would add nothing."""
|
||||
texts = _GOAL_REASON_TEXT[locale]
|
||||
if isinstance(code, str):
|
||||
code = code.removeprefix("blocked:")
|
||||
if not isinstance(code, str) or code not in texts or code in _NO_EXTRA_REASON:
|
||||
return None
|
||||
return texts[code]
|
||||
|
||||
|
||||
def _with_reason(key: str, code: object, locale: str) -> str:
|
||||
"""The ``key`` sentence with the reason, or its ``{key}_bare`` form without one."""
|
||||
reason = _reason_text(code, locale)
|
||||
text = _TEXT[locale]
|
||||
return text[f"{key}_bare"] if reason is None else text[key].format(reason=reason)
|
||||
|
||||
|
||||
def _title(payload: dict[str, Any], locale: str) -> str:
|
||||
title = payload.get("task_title")
|
||||
title = _one_line(title) if isinstance(title, str) else ""
|
||||
return _truncate(title, limit=_TITLE_LIMIT) if title else _TEXT[locale]["untitled"]
|
||||
|
||||
|
||||
def _positive_int(value: object) -> int | None:
|
||||
if isinstance(value, bool) or not isinstance(value, int):
|
||||
return None
|
||||
return value if value > 0 else None
|
||||
|
||||
|
||||
def _last_run_suffix(event: str, run_status: str | None, payload: dict[str, Any], locale: str) -> str | None:
|
||||
"""The last run's outcome, appended where the main sentence would hide it."""
|
||||
text = _TEXT[locale]
|
||||
if run_status == "failed":
|
||||
return text["last_failed"]
|
||||
if run_status == "unmet":
|
||||
return _with_reason("last_unmet", payload.get("latest_reason_code"), locale)
|
||||
if run_status == "interrupted" and event == "task_stopped":
|
||||
return text["last_interrupted"]
|
||||
return None
|
||||
|
||||
|
||||
def _what_happened(delivery: dict[str, Any], locale: str) -> str:
|
||||
text = _TEXT[locale]
|
||||
payload = _payload(delivery)
|
||||
event = delivery.get("event") or ""
|
||||
run_status = _run_status(delivery)
|
||||
parts: list[str]
|
||||
if event == "run_completed":
|
||||
parts = [text["run_completed"]]
|
||||
if "relied_on_assumption" in payload:
|
||||
parts.append(text["goal_met_with_assumption"] if payload.get("relied_on_assumption") is True else text["goal_met"])
|
||||
elif event == "run_failed":
|
||||
# Raw error text never leaves: tracebacks can carry hosts, paths and tokens.
|
||||
parts = [text["run_failed"]]
|
||||
elif event == "run_unmet":
|
||||
parts = [_with_reason("run_unmet", payload.get("reason_code"), locale)]
|
||||
elif event == "task_paused":
|
||||
# The pause reason itself is fixed (3 misses); show the latest miss's reason.
|
||||
latest = payload.get("latest_reason_code")
|
||||
if latest is None and payload.get("reason_code") != "consecutive_unmet":
|
||||
latest = payload.get("reason_code")
|
||||
parts = [_with_reason("task_paused", latest, locale)]
|
||||
elif event == "task_stopped":
|
||||
condition = payload.get("stop_condition")
|
||||
condition = _truncate(_one_line(redact_egress_text(condition)), limit=_CONDITION_LIMIT) if isinstance(condition, str) else ""
|
||||
parts = [text["task_stopped_condition"].format(condition=condition) if condition else text["task_stopped"]]
|
||||
if (suffix := _last_run_suffix(event, run_status, payload, locale)) is not None:
|
||||
parts.append(suffix)
|
||||
elif event == "task_finished":
|
||||
reason = payload.get("reason_code")
|
||||
if reason == "end_at":
|
||||
parts = [text["finished_end_at"]]
|
||||
elif reason == "max_runs":
|
||||
max_runs = _positive_int(payload.get("max_runs"))
|
||||
parts = [text["finished_one_run"] if max_runs == 1 else text["finished_max_runs"].format(max_runs=max_runs) if max_runs is not None else text["finished_all_runs"]]
|
||||
else:
|
||||
parts = [text["finished"]]
|
||||
if (suffix := _last_run_suffix(event, run_status, payload, locale)) is not None:
|
||||
parts.append(suffix)
|
||||
else:
|
||||
parts = [text["update"]]
|
||||
return text["sentence_gap"].join(parts)
|
||||
|
||||
|
||||
def render_notification_text(delivery: dict[str, Any], *, default_locale: str = DEFAULT_NOTIFICATION_LOCALE) -> str:
|
||||
"""Render one self-contained notice: title, what happened, result, where to look.
|
||||
|
||||
``payload["result_summary"]`` is the run's final reply, attached by the
|
||||
delivery worker at send time; it is redacted as a whole (so a multi-line
|
||||
secret is gone before a line is taken) and reduced to the tasks page's
|
||||
one-line run summary.
|
||||
"""
|
||||
locale = resolve_locale(delivery, default_locale)
|
||||
text = _TEXT[locale]
|
||||
payload = _payload(delivery)
|
||||
lines = [text["title"].format(title=_title(payload, locale)), _what_happened(delivery, locale)]
|
||||
reply = payload.get("result_summary")
|
||||
if wants_result_summary(delivery) and isinstance(reply, str):
|
||||
summary = run_summary(redact_egress_text(reply))
|
||||
if summary:
|
||||
lines.append(text["result"].format(summary=summary))
|
||||
lines.append(text["closing"])
|
||||
return "\n".join(lines)
|
||||
@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import socket
|
||||
import uuid
|
||||
@ -11,8 +12,17 @@ from zoneinfo import ZoneInfo
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.channels.capabilities import supports_proactive_notifications
|
||||
from app.scheduler.notification_text import NOTIFICATION_LOCALES
|
||||
from deerflow.persistence.channel_connections.model import ChannelConnectionRow
|
||||
from deerflow.persistence.run.model import RunRow
|
||||
from deerflow.persistence.scheduled_task_events import ScheduledTaskEventRepository
|
||||
from deerflow.persistence.scheduled_task_runs import ActiveScheduledRunConflict, ScheduledTaskAdmissionRejected
|
||||
from deerflow.persistence.scheduled_task_runs.finalization import LIFECYCLE_EVENTS, RUN_EVENT_BY_STATUS, lifecycle_anchor, lifecycle_reason
|
||||
from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow
|
||||
from deerflow.persistence.scheduled_task_runs.sql import run_number_expression
|
||||
from deerflow.persistence.thread_meta.model import ThreadMetaRow
|
||||
from deerflow.persistence.user.model import UserPreferenceRow
|
||||
from deerflow.runtime import ConflictError, RunRecord
|
||||
from deerflow.scheduler.host_notes import RUN_ERROR_DELETED_WHILE_QUEUED, RUN_ERROR_INTERRUPTED, RUN_ERROR_LEASE_LOST, RUN_ERROR_PAUSED_WHILE_QUEUED, RUN_ERROR_QUEUE_TIMEOUT, RUN_ERROR_RESTARTED
|
||||
from deerflow.scheduler.schedules import next_run_at
|
||||
@ -53,6 +63,132 @@ def _as_utc(value: datetime | str | None) -> datetime | None:
|
||||
return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
|
||||
|
||||
|
||||
def _as_utc_or_none(value: object) -> datetime | None:
|
||||
"""A datetime as aware UTC; anything else (or an unusable value) is None."""
|
||||
if not isinstance(value, datetime):
|
||||
return None
|
||||
try:
|
||||
return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
|
||||
except (ValueError, OverflowError):
|
||||
return None
|
||||
|
||||
|
||||
def _int_or_none(value: object) -> int | None:
|
||||
try:
|
||||
return int(value) if value is not None else None
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _chat_event_payload(task, occurrence, event: str, *, run_number: int | None) -> dict[str, Any]:
|
||||
"""Display snapshot of a lifecycle event (survives task deletion and edits).
|
||||
|
||||
``run_thread_id`` is set only when the occurrence launched a run (a skipped
|
||||
occurrence's thread was never created). ``latest_reason_code`` is the
|
||||
unmet reason of that run. Never raises for bad stored values.
|
||||
"""
|
||||
title = task.title.strip() if isinstance(task.title, str) else ""
|
||||
stop_condition = getattr(task, "stop_condition", None)
|
||||
end_at = _as_utc_or_none(getattr(task, "end_at", None))
|
||||
run_status = getattr(occurrence, "status", None) if occurrence is not None else None
|
||||
launched = occurrence is not None and occurrence.run_id is not None
|
||||
# The agent that ran it (edits are refused while a run is active), so the
|
||||
# run link opens on that agent's route, not the originating chat's.
|
||||
agent = getattr(task, "assistant_id", None)
|
||||
return {
|
||||
"task_title": title or None,
|
||||
"stop_condition": stop_condition if event == "task_stopped" and isinstance(stop_condition, str) and stop_condition.strip() else None,
|
||||
"run_thread_id": occurrence.thread_id if launched else None,
|
||||
"run_agent_name": agent.strip() if launched and isinstance(agent, str) and agent.strip() else None,
|
||||
"run_number": run_number,
|
||||
"run_status": run_status,
|
||||
"latest_reason_code": occurrence.error if run_status == "unmet" and isinstance(occurrence.error, str) else None,
|
||||
"max_runs": _int_or_none(getattr(task, "max_runs", None)),
|
||||
"end_at": end_at.isoformat() if end_at is not None else None,
|
||||
"schedule_type": task.schedule_type if isinstance(task.schedule_type, str) else None,
|
||||
}
|
||||
|
||||
|
||||
def _origin_thread_share_lock(thread_id: str):
|
||||
"""``SELECT ... FOR SHARE`` on the originating chat's ``threads_meta`` row.
|
||||
|
||||
The share lock makes a concurrent thread delete wait until this
|
||||
finalization commits (Postgres; SQLite already serializes writers), so the
|
||||
delete's later ``delete_by_thread`` step removes the line written here. A
|
||||
delete that committed first leaves no row to lock, and no line is written.
|
||||
"""
|
||||
return select(ThreadMetaRow.thread_id).where(ThreadMetaRow.thread_id == thread_id).with_for_update(read=True)
|
||||
|
||||
|
||||
def _event_time(task, occurrence) -> datetime:
|
||||
"""When the reported transition happened (for the ``end_at`` / ``max_runs`` reason)."""
|
||||
return _as_utc_or_none(getattr(occurrence, "finished_at", None)) or _as_utc_or_none(getattr(task, "updated_at", None)) or datetime.now(UTC)
|
||||
|
||||
|
||||
# One IM message per occurrence (or idle finish): the first event of this
|
||||
# order that the transition produced. A once task's ``task_finished`` is never
|
||||
# the message (its run event says the same thing), see ``_notice_event``.
|
||||
_NOTICE_PRIORITY: tuple[str, ...] = ("task_stopped", "task_paused", "task_finished", "run_failed", "run_unmet", "run_completed")
|
||||
NOTIFICATION_PAYLOAD_VERSION = 2
|
||||
|
||||
|
||||
def _notice_event(task, occurrence, events: tuple[str, ...], *, now: datetime) -> tuple[str, str | None] | None:
|
||||
"""The single event an IM notice reports, with its lifecycle reason (None for run events)."""
|
||||
for event in _NOTICE_PRIORITY:
|
||||
if event not in events:
|
||||
continue
|
||||
if event == "task_finished":
|
||||
reason = lifecycle_reason(task, event, now=now, occurrence=occurrence)
|
||||
if reason.startswith("once_"):
|
||||
continue
|
||||
return event, reason
|
||||
if event in LIFECYCLE_EVENTS:
|
||||
return event, lifecycle_reason(task, event, now=now, occurrence=occurrence)
|
||||
return event, None
|
||||
return None
|
||||
|
||||
|
||||
def _idle_notice_key(task_id: str, anchor: str) -> str:
|
||||
"""Outbox ``task_run_id`` of a finish without an occurrence (fits the 64-char column).
|
||||
|
||||
Derived from the lifecycle anchor, so re-running ``complete_if_ended`` for
|
||||
the same end dedupes, and page-created tasks (no event row) get one too.
|
||||
"""
|
||||
digest = hashlib.sha256(f"{task_id}\x00{anchor}".encode()).hexdigest()
|
||||
return f"idle:{digest[:48]}"
|
||||
|
||||
|
||||
def _notice_payload(task, occurrence, event: str, reason: str | None, *, locale: str | None) -> dict[str, Any]:
|
||||
"""Payload v2 of an IM notice (rendered by ``app.scheduler.notification_text``).
|
||||
|
||||
No thread or run ids: notices carry no links. ``task_id`` stays for
|
||||
diagnostics only. Never raises for bad stored values.
|
||||
"""
|
||||
title = task.title.strip() if isinstance(task.title, str) else ""
|
||||
run_status = getattr(occurrence, "status", None) if occurrence is not None else None
|
||||
occurrence_error = getattr(occurrence, "error", None) if occurrence is not None else None
|
||||
unmet_reason = occurrence_error if run_status == "unmet" and isinstance(occurrence_error, str) else None
|
||||
payload: dict[str, Any] = {
|
||||
"payload_version": NOTIFICATION_PAYLOAD_VERSION,
|
||||
"task_id": task.id,
|
||||
"task_title": title or None,
|
||||
"locale": locale,
|
||||
"reason_code": unmet_reason if event == "run_unmet" else reason,
|
||||
"latest_reason_code": unmet_reason,
|
||||
"run_status": run_status if isinstance(run_status, str) else None,
|
||||
}
|
||||
verdict = getattr(occurrence, "goal_verdict", None) if occurrence is not None else None
|
||||
if run_status == "success" and getattr(occurrence, "goal_objective", None) is not None and isinstance(verdict, dict) and verdict.get("satisfied") is True:
|
||||
# Present only when a goal was met; True when it relied on an assumption.
|
||||
payload["relied_on_assumption"] = verdict.get("relied_on_assumption") is True
|
||||
stop_condition = getattr(task, "stop_condition", None)
|
||||
if event == "task_stopped" and isinstance(stop_condition, str) and stop_condition.strip():
|
||||
payload["stop_condition"] = stop_condition
|
||||
if event == "task_finished":
|
||||
payload["max_runs"] = _int_or_none(getattr(task, "max_runs", None))
|
||||
return payload
|
||||
|
||||
|
||||
# Shared so the active-row fast path and the atomic-admission conflict path
|
||||
# return byte-identical outcomes for the same active-occurrence condition.
|
||||
_ACTIVE_RUN_CONFLICT_ERROR = "task already has an active run"
|
||||
@ -74,6 +210,7 @@ class ScheduledTaskService:
|
||||
poll_interval_seconds: int,
|
||||
lease_seconds: int,
|
||||
max_concurrent_runs: int,
|
||||
max_concurrent_runs_per_user: int = 0,
|
||||
queue_timeout_seconds: int = 3600,
|
||||
multi_instance: bool = False,
|
||||
run_lease_grace_seconds: int = 10,
|
||||
@ -87,12 +224,17 @@ class ScheduledTaskService:
|
||||
self._poll_interval_seconds = poll_interval_seconds
|
||||
self._lease_seconds = lease_seconds
|
||||
self._max_concurrent_runs = max_concurrent_runs
|
||||
# Per-owner cap on launching/running rows (0 = off). It never exceeds
|
||||
# the global cap, so with the defaults (3 global, 2 per owner) one
|
||||
# owner always leaves a slot for everyone else.
|
||||
self._max_concurrent_runs_per_user = min(max_concurrent_runs_per_user, max_concurrent_runs) if max_concurrent_runs_per_user > 0 else 0
|
||||
self._queue_timeout_seconds = queue_timeout_seconds
|
||||
self._multi_instance = multi_instance
|
||||
self._run_lease_grace_seconds = run_lease_grace_seconds
|
||||
# Notification outbox wiring (issue #4254): both repos must be present
|
||||
# for the feature to be active; the completion hook only enqueues, the
|
||||
# delivery worker sends. Either being None keeps legacy behavior.
|
||||
# for IM notices. The finalization observer stages them in the outcome's
|
||||
# transaction (it reads bindings in that session); the delivery worker
|
||||
# sends. Either being None sends no IM notices.
|
||||
self._connection_repo = connection_repo
|
||||
self._notification_repo = notification_repo
|
||||
# Whether scheduled runs get stop_scheduled_task (scheduler.tool_enabled).
|
||||
@ -130,38 +272,122 @@ class ScheduledTaskService:
|
||||
return self._task is not None and not self._task.done()
|
||||
|
||||
def _install_finalization_observers(self) -> None:
|
||||
observer = self._enqueue_finalization_notices if self._connection_repo is not None and self._notification_repo is not None else None
|
||||
# Always installed: chat events need no outbox. The IM step checks the
|
||||
# outbox wiring at call time, so a detached outbox keeps chat events.
|
||||
for repository in (self._task_repo, self._task_run_repo):
|
||||
register = getattr(repository, "set_finalization_observer", None)
|
||||
if callable(register):
|
||||
register(observer)
|
||||
register(self._on_finalization)
|
||||
|
||||
async def _enqueue_finalization_notices(self, session, task, occurrence, *, events: tuple[str, ...]) -> None:
|
||||
"""Commit new lifecycle notices with their first terminal transition.
|
||||
async def _on_finalization(self, session, task, occurrence, *, events: tuple[str, ...]) -> None:
|
||||
"""Stage lifecycle obligations in the transaction that finalizes them.
|
||||
|
||||
Ordinary success/failure notification behavior remains in the existing
|
||||
completion hook. Only goal-unmet and automatic-pause obligations use
|
||||
this transaction-aware callback, including during crash recovery.
|
||||
Called by ``finalize_occurrence`` and ``finish_task_at_end_condition``
|
||||
under the parent lock, before their commit, so every row written here
|
||||
commits or rolls back with the state change it reports (also during
|
||||
crash and lease recovery). ``occurrence`` is None for an idle finish.
|
||||
|
||||
1. A manual trial drops only its ``run_*`` events (the user is watching
|
||||
it); lifecycle events are one-time transitions and are kept.
|
||||
2. Lifecycle events become rows for the originating chat.
|
||||
3. One IM notice per occurrence (only while the outbox is wired).
|
||||
|
||||
Lookups never raise for bad data (see ``_record_chat_events``); only
|
||||
database errors propagate, because swallowing them would break the
|
||||
same-transaction guarantee.
|
||||
"""
|
||||
selected = tuple(event for event in events if event in {"run_unmet", "task_paused"})
|
||||
if not selected or occurrence.trigger == "manual" or self._notification_repo is None or self._connection_repo is None:
|
||||
if occurrence is not None and occurrence.trigger == "manual":
|
||||
events = tuple(event for event in events if event not in RUN_EVENT_BY_STATUS.values())
|
||||
lifecycle = tuple(event for event in events if event in LIFECYCLE_EVENTS)
|
||||
if lifecycle:
|
||||
await self._record_chat_events(session, task, occurrence, lifecycle)
|
||||
if self._notification_repo is not None and self._connection_repo is not None:
|
||||
await self._enqueue_notice(session, task, occurrence, events)
|
||||
|
||||
async def _record_chat_events(self, session, task, occurrence, events: tuple[str, ...]) -> None:
|
||||
"""Write one display-only event row per lifecycle event for the originating chat.
|
||||
|
||||
Page-created tasks have no originating chat, and a deleted chat gets no
|
||||
new rows. Bad stored data falls back (blank title -> None, unreadable
|
||||
end time -> ``max_runs`` / ``seq:`` anchor, missing run -> None).
|
||||
"""
|
||||
origin_thread_id = getattr(task, "origin_thread_id", None)
|
||||
if not isinstance(origin_thread_id, str) or not origin_thread_id:
|
||||
return
|
||||
bindings = await session.scalars(select(ChannelConnectionRow).where(ChannelConnectionRow.owner_user_id == task.user_id, ChannelConnectionRow.status == "connected"))
|
||||
if await session.scalar(_origin_thread_share_lock(origin_thread_id)) is None:
|
||||
return
|
||||
now = _event_time(task, occurrence)
|
||||
anchor = lifecycle_anchor(task, occurrence, now=now)
|
||||
after_run_id = await session.scalar(select(RunRow.run_id).where(RunRow.thread_id == origin_thread_id, RunRow.operation_kind == "run").order_by(RunRow.created_at.desc(), RunRow.run_id.desc()).limit(1))
|
||||
run_number = None
|
||||
if occurrence is not None:
|
||||
value = await session.scalar(select(run_number_expression(ScheduledTaskRunRow)).where(ScheduledTaskRunRow.id == occurrence.id))
|
||||
run_number = _int_or_none(value)
|
||||
for event in events:
|
||||
reason = lifecycle_reason(task, event, now=now, occurrence=occurrence)
|
||||
await ScheduledTaskEventRepository.record_in_session(
|
||||
session,
|
||||
user_id=task.user_id,
|
||||
task_id=task.id,
|
||||
thread_id=origin_thread_id,
|
||||
occurrence_id=occurrence.id if occurrence is not None else None,
|
||||
anchor=anchor,
|
||||
event=event,
|
||||
reason_code=reason,
|
||||
after_run_id=after_run_id,
|
||||
payload=_chat_event_payload(task, occurrence, event, run_number=run_number),
|
||||
)
|
||||
|
||||
async def _enqueue_notice(self, session, task, occurrence, events: tuple[str, ...]) -> None:
|
||||
"""Stage the occurrence's one IM notice for each push-capable binding of the owner.
|
||||
|
||||
Rows are deduplicated on ``(task_run_id, event, provider, target)``,
|
||||
so recovery re-finalizing nothing (``finalize_occurrence`` returns
|
||||
False) and a replayed idle finish (same anchor) add nothing. Providers
|
||||
without proactive push get no row. Plain manual trials reach here
|
||||
without run events and send nothing; interrupted runs have no event.
|
||||
"""
|
||||
now = _event_time(task, occurrence)
|
||||
selected = _notice_event(task, occurrence, events, now=now)
|
||||
if selected is None:
|
||||
return
|
||||
event, reason = selected
|
||||
bindings = [
|
||||
binding
|
||||
for binding in await session.scalars(select(ChannelConnectionRow).where(ChannelConnectionRow.owner_user_id == task.user_id, ChannelConnectionRow.status == "connected"))
|
||||
if binding.provider and binding.external_account_id and supports_proactive_notifications(binding.provider)
|
||||
]
|
||||
if not bindings:
|
||||
return
|
||||
payload = _notice_payload(task, occurrence, event, reason, locale=await self._owner_locale(session, task.user_id))
|
||||
task_run_id = occurrence.id if occurrence is not None else _idle_notice_key(task.id, lifecycle_anchor(task, occurrence, now=now))
|
||||
for binding in bindings:
|
||||
if not binding.provider or not binding.external_account_id:
|
||||
continue
|
||||
for event in selected:
|
||||
await self._notification_repo.enqueue_in_session(
|
||||
session,
|
||||
task_id=task.id,
|
||||
task_run_id=occurrence.id,
|
||||
run_id=occurrence.run_id,
|
||||
event=event,
|
||||
provider=binding.provider,
|
||||
target=binding.external_account_id,
|
||||
owner_user_id=task.user_id,
|
||||
payload={"task_id": task.id, "task_title": task.title, "reason_code": "consecutive_unmet" if event == "task_paused" else occurrence.error},
|
||||
)
|
||||
await self._notification_repo.enqueue_in_session(
|
||||
session,
|
||||
task_id=task.id,
|
||||
task_run_id=task_run_id,
|
||||
run_id=occurrence.run_id if occurrence is not None else None,
|
||||
event=event,
|
||||
provider=binding.provider,
|
||||
target=binding.external_account_id,
|
||||
owner_user_id=task.user_id,
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
async def _owner_locale(session, user_id: str) -> str | None:
|
||||
"""The owner's UI language preference, or None (= ``channel_connections.notification_locale``).
|
||||
|
||||
A missing, invalid or unreadable stored value falls back instead of
|
||||
raising (liveness rule: the observer must not wedge recovery). Database
|
||||
errors still propagate.
|
||||
"""
|
||||
try:
|
||||
value = await session.scalar(select(UserPreferenceRow.value).where(UserPreferenceRow.user_id == user_id, UserPreferenceRow.key == "locale"))
|
||||
except (ValueError, TypeError, KeyError):
|
||||
# Undecodable stored JSON (json.JSONDecodeError is a ValueError).
|
||||
return None
|
||||
return value if isinstance(value, str) and value in NOTIFICATION_LOCALES else None
|
||||
|
||||
def detach_notification_outbox(self) -> None:
|
||||
"""Stop enqueueing run notifications (issue #4254).
|
||||
@ -377,6 +603,7 @@ class ScheduledTaskService:
|
||||
now=now,
|
||||
lease_seconds=self._lease_seconds,
|
||||
global_max_concurrent_runs=self._max_concurrent_runs,
|
||||
per_user_max_concurrent_runs=self._max_concurrent_runs_per_user,
|
||||
)
|
||||
if claimed is None:
|
||||
return self._queued_result(task_run_id, execution_thread_id)
|
||||
@ -666,7 +893,10 @@ class ScheduledTaskService:
|
||||
}
|
||||
|
||||
async def _drain_queue(self, *, now: datetime) -> None:
|
||||
queued_rows = await self._task_run_repo.list_queued_runs(limit=max(16, self._max_concurrent_runs * 4))
|
||||
queued_rows = await self._task_run_repo.list_queued_runs(
|
||||
limit=max(16, self._max_concurrent_runs * 4),
|
||||
per_user_max_concurrent_runs=self._max_concurrent_runs_per_user,
|
||||
)
|
||||
for queued in queued_rows:
|
||||
await self._task_repo.release_queued_admission_lease(queued["task_id"])
|
||||
task = await self._task_repo.get_internal(queued["task_id"])
|
||||
@ -732,7 +962,11 @@ class ScheduledTaskService:
|
||||
completion_kwargs = {}
|
||||
if metadata.get("scheduled_goal_objective") is not None:
|
||||
completion_kwargs["goal_verdict"] = getattr(record, "goal_verdict", None)
|
||||
completed = await self._task_repo.complete_run(
|
||||
# The outcome and its notices (chat event, IM outbox row) commit in one
|
||||
# transaction: the finalization observer stages them. A completion
|
||||
# that recorded nothing (another run's occurrence, already finalized
|
||||
# by recovery) announces nothing.
|
||||
await self._task_repo.complete_run(
|
||||
task_id,
|
||||
user_id=user_id,
|
||||
task_run_id=task_run_id,
|
||||
@ -742,110 +976,6 @@ class ScheduledTaskService:
|
||||
finished_at=datetime.now(UTC),
|
||||
**completion_kwargs,
|
||||
)
|
||||
if not completed:
|
||||
# The occurrence was not this run's to complete (missing row, another
|
||||
# run, another owner): nothing was recorded, so nothing is announced.
|
||||
return
|
||||
|
||||
if terminal_status == "unmet":
|
||||
# Its distinct notice was staged in the same transaction above.
|
||||
return
|
||||
|
||||
if metadata.get("scheduled_trigger") == "manual":
|
||||
# A manual "run now" happens with the user watching the UI; the
|
||||
# IM push would only echo what they already see. Missing
|
||||
# metadata fails safe towards notifying.
|
||||
return
|
||||
|
||||
if self._notification_repo is None or self._connection_repo is None:
|
||||
return
|
||||
# complete_run commits the outcome atomically without returning the task,
|
||||
# so the title is read afterwards. A task deleted meanwhile stays silent.
|
||||
# A failed read must not shadow the committed outcome or drop the push:
|
||||
# the title is optional and the message falls back to the task id.
|
||||
task_title = None
|
||||
try:
|
||||
task = await self._task_repo.get(task_id, user_id=user_id)
|
||||
except Exception:
|
||||
logger.exception("[Scheduler] failed to load task %s for notifications", task_id)
|
||||
else:
|
||||
if task is None:
|
||||
return
|
||||
task_title = task.get("title")
|
||||
|
||||
await self._enqueue_run_notifications(
|
||||
task_id=task_id,
|
||||
task_run_id=task_run_id,
|
||||
run_id=record.run_id,
|
||||
user_id=user_id,
|
||||
terminal_status=terminal_status,
|
||||
error=error,
|
||||
task_title=task_title,
|
||||
relied_on_assumption=(isinstance(getattr(record, "goal_verdict", None), dict) and record.goal_verdict.get("satisfied") is True and record.goal_verdict.get("relied_on_assumption") is True),
|
||||
)
|
||||
|
||||
async def _enqueue_run_notifications(
|
||||
self,
|
||||
*,
|
||||
task_id: str,
|
||||
task_run_id: str,
|
||||
run_id: str | None,
|
||||
user_id: str,
|
||||
terminal_status: str,
|
||||
error: str | None,
|
||||
task_title: str | None = None,
|
||||
relied_on_assumption: bool = False,
|
||||
) -> None:
|
||||
"""Write durable outbox rows for the run outcome (issue #4254).
|
||||
|
||||
Best-effort by design: a notification failure is logged and swallowed
|
||||
so delivery problems can never shadow the execution status written
|
||||
above. Interrupted runs are user-initiated cancels and intentionally
|
||||
produce no notification.
|
||||
"""
|
||||
if self._notification_repo is None or self._connection_repo is None:
|
||||
return
|
||||
event = {"success": "run_completed", "failed": "run_failed"}.get(terminal_status)
|
||||
if event is None:
|
||||
return
|
||||
try:
|
||||
connections = await self._connection_repo.list_connections(user_id)
|
||||
except Exception:
|
||||
logger.exception("[Scheduler] failed to list channel connections for notifications")
|
||||
return
|
||||
payload = {
|
||||
"run_status": terminal_status,
|
||||
"error": error,
|
||||
"task_id": task_id,
|
||||
"task_title": task_title,
|
||||
}
|
||||
if relied_on_assumption:
|
||||
payload["relied_on_assumption"] = True
|
||||
for connection in connections:
|
||||
if connection.get("status") != "connected":
|
||||
continue
|
||||
provider = connection.get("provider")
|
||||
target = connection.get("external_account_id")
|
||||
if not provider or not target:
|
||||
continue
|
||||
try:
|
||||
await self._notification_repo.enqueue(
|
||||
task_id=task_id,
|
||||
task_run_id=task_run_id,
|
||||
run_id=run_id,
|
||||
event=event,
|
||||
provider=provider,
|
||||
target=target,
|
||||
owner_user_id=user_id,
|
||||
payload=payload,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"[Scheduler] failed to enqueue notification task_run=%s provider=%s target=%s",
|
||||
task_run_id,
|
||||
provider,
|
||||
target,
|
||||
)
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._task is not None:
|
||||
|
||||
@ -36,7 +36,7 @@ return `503`.
|
||||
|
||||
### Account Preferences
|
||||
|
||||
`GET /api/v1/auth/preferences` returns the signed-in browser user's four
|
||||
`GET /api/v1/auth/preferences` returns the signed-in browser user's five
|
||||
preferences. `PATCH` updates only explicitly supplied fields and returns `204`.
|
||||
Both require `X-Expected-User-Id` matching the session user; PATCH also requires
|
||||
the normal `X-CSRF-Token` header. The expected ID is a stale-tab guard, not an
|
||||
@ -48,17 +48,22 @@ authorization credential. PAT, internal, and auth-disabled callers receive
|
||||
"notification_enabled": false,
|
||||
"model_name": "my-model",
|
||||
"mode": "pro",
|
||||
"reasoning_effort": "high"
|
||||
"reasoning_effort": "high",
|
||||
"locale": "zh-CN"
|
||||
}
|
||||
```
|
||||
|
||||
All four fields accept `null` to restore the default. `mode` accepts `flash`,
|
||||
All five fields accept `null` to restore the default. `mode` accepts `flash`,
|
||||
`thinking`, `pro`, or `ultra`; `reasoning_effort` accepts `minimal`, `low`,
|
||||
`medium`, or `high`; model names are at most 200 characters. Unknown fields and
|
||||
`medium`, or `high`; `locale` accepts `en-US` or `zh-CN`; model names are at
|
||||
most 200 characters. Unknown fields and
|
||||
invalid values return `422`. Missing preferences read as `null`. Separate-field
|
||||
patches preserve each other's changes, and same-field writes are last-commit-wins.
|
||||
Storage requires SQLite or PostgreSQL (`503` when unavailable). Browser
|
||||
notification permission remains device-local and is not changed by this API.
|
||||
`locale` is the web UI language; the web app writes it after sign-in and on
|
||||
every language switch, and scheduled-task IM notices are written in it (without
|
||||
it they use `channel_connections.notification_locale`).
|
||||
|
||||
### Personal Access Tokens
|
||||
|
||||
@ -1092,6 +1097,77 @@ DELETE /api/threads/{thread_id}
|
||||
- `422` for invalid thread IDs
|
||||
- `500` returns a generic `{"detail": "Failed to delete local thread data."}` response while full exception details stay in server logs
|
||||
|
||||
Deleting a thread also removes every user's read marker for it.
|
||||
|
||||
### Thread Origin, Activity and Unread State
|
||||
|
||||
Runs the server starts for a user (a schedule, an IM channel, a GitHub agent,
|
||||
an extension, an MCP notification) carry a server-owned origin in their
|
||||
metadata, and so does a thread the server creates for one:
|
||||
|
||||
```json
|
||||
{ "deerflow_origin": { "kind": "im_channel", "provider": "feishu" } }
|
||||
```
|
||||
|
||||
`kind` is one of `schedule`, `im_channel`, `github`, `extension`,
|
||||
`mcp_notification` (`contracts/thread_origin_contract.json`); `provider`
|
||||
(IM/GitHub) and `namespace` (extension plugin) are optional. Clients cannot
|
||||
set it: `POST /api/threads` and `PATCH /api/threads/{thread_id}` strip it, run
|
||||
admission drops it from `metadata` and `config.metadata`, and only server-side
|
||||
launchers and the internal channel caller may stamp it. The run's kind is also
|
||||
stored as `runs.origin_kind` (`null` for interactive and pre-upgrade runs). IM
|
||||
threads keep `metadata.channel_source` as their marker. This key is unrelated to
|
||||
the message-level `additional_kwargs.deerflow_scheduled_origin` of a scheduled
|
||||
prompt.
|
||||
|
||||
**Activity feed.** Requires SQL persistence (`503` on the memory backend);
|
||||
`GET /api/features` reports `{"thread_activity": {"available": true}}`.
|
||||
|
||||
```http
|
||||
GET /api/thread-activity?cursor=1834:run-42&limit=200
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"cursor": "1840:run-57",
|
||||
"threads": [{ "thread_id": "…", "origin_kind": "schedule", "status": "success" }],
|
||||
"truncated": false,
|
||||
"read_version": 17
|
||||
}
|
||||
```
|
||||
|
||||
- Without `cursor` the call only seeds: it returns the caller's current
|
||||
position (`"0:"` for a user with no runs) and no threads.
|
||||
- With a cursor it pages the caller's run changes in order (`limit` 1-500,
|
||||
default 200). `threads` lists each thread with a change from a
|
||||
server-originated run of the caller in the page, once, with the status of
|
||||
its latest such change. The caller's own interactive runs advance the cursor
|
||||
but are not listed.
|
||||
- `cursor` is the position of the last change used, so a `truncated` page
|
||||
continues on the next poll without gaps. Treat it as opaque.
|
||||
- `read_version` is the caller's read clock; it changes when a thread is
|
||||
marked read on any device.
|
||||
- A malformed cursor returns `422` with `detail.code` `invalid_cursor`.
|
||||
- Every query is scoped to the caller; another user's cursor reveals nothing.
|
||||
|
||||
**Unread state.** A thread is unread for a user while one of that user's
|
||||
server-originated runs in it changed after the user last read it. The user's
|
||||
own interactive runs, other users' runs in a shared thread and pre-upgrade runs
|
||||
never make a thread unread.
|
||||
|
||||
```http
|
||||
POST /api/threads/{thread_id}/read
|
||||
```
|
||||
|
||||
```json
|
||||
{ "unread": false, "read_version": 18 }
|
||||
```
|
||||
|
||||
The read position never moves backwards. A thread already read up to its
|
||||
latest run writes nothing and returns the unchanged `read_version`. Items of
|
||||
`POST /api/threads/search` carry `unread` (`true`/`false`; `null` without SQL
|
||||
persistence); other thread responses return `null`.
|
||||
|
||||
### Projects
|
||||
|
||||
#### Get Projects Config
|
||||
@ -1273,6 +1349,7 @@ POST /api/scheduled-tasks/{task_id}/resume
|
||||
POST /api/scheduled-tasks/{task_id}/trigger
|
||||
GET /api/scheduled-tasks/{task_id}/runs?status=&limit=&offset=
|
||||
GET /api/threads/{thread_id}/scheduled-tasks
|
||||
GET /api/threads/{thread_id}/scheduled-task-events?limit=50
|
||||
POST /api/scheduled-tasks/preview-cron
|
||||
```
|
||||
|
||||
@ -1304,6 +1381,33 @@ while its run executes, so check `active_run_status` for "is a run active".
|
||||
one of its runs) and `thread_run` (`{"run_number", "trigger", "scheduled_for",
|
||||
"status"}` for `run`, else null).
|
||||
|
||||
`GET /api/threads/{thread_id}/scheduled-task-events` (thread owner check;
|
||||
`limit` 1–200, default 50) returns the caller's lifecycle events for a chat
|
||||
that created tasks, oldest first:
|
||||
|
||||
```json
|
||||
{ "events": [ { "id": "evt-…", "task_id": "task-…", "event": "task_stopped", "reason_code": "agent_stop",
|
||||
"task_title": "Check the release checklist", "stop_condition": "all items are ticked",
|
||||
"run_thread_id": "…", "run_agent_name": "lead_agent", "run_number": 4, "run_status": "success", "max_runs": null, "end_at": null,
|
||||
"schedule_type": "cron", "after_run_id": "…", "created_at": "2026-10-06T09:00:03+00:00" } ] }
|
||||
```
|
||||
|
||||
`event` is `task_stopped` (the agent paused its own schedule, reason
|
||||
`agent_stop`), `task_paused` (automatic pause, `consecutive_unmet`) or
|
||||
`task_finished` (`max_runs`, `end_at`, or a one-time task's `once_done` /
|
||||
`once_failed`); the vocabulary is pinned in
|
||||
`contracts/scheduled_goal_notes_contract.json`. Each row is written in the
|
||||
same transaction as the state change it reports and is unique per task,
|
||||
transition and event, so recovery never adds a second one. `run_status` is the
|
||||
last run's outcome (a stop or finish after a failed run says so);
|
||||
`run_thread_id` is null when the occurrence never launched, and
|
||||
`run_agent_name` is the agent that ran it (its run chat's route); `after_run_id` is
|
||||
the newest run of the chat when the event was written (the chat shows the line
|
||||
after that turn, else at the end). Rows keep a title snapshot and stay after
|
||||
the task is deleted; deleting the chat removes them. Tasks created on the
|
||||
Scheduled tasks page have no originating chat and no rows. IDs are for links
|
||||
only.
|
||||
|
||||
**Create** returns `409 scheduler_not_running` (after the request validates)
|
||||
while this Gateway process's scheduler is not running; the tasks page's
|
||||
Duplicate is a create.
|
||||
@ -1391,6 +1495,19 @@ other runs in `timezone`. A new run conversation is titled
|
||||
when that zone is only the placeholder. The key is server-owned: it is
|
||||
stripped from client-supplied messages and state updates.
|
||||
|
||||
**IM notices ("scheduled task updates").** With `channel_connections.enabled`,
|
||||
each item of `GET /api/channels/providers` carries
|
||||
`proactive_notifications` (bool): whether scheduled-task updates are pushed to
|
||||
that app. Only providers with proactive push (WeCom today) get notices; the
|
||||
others get none, and Settings says so. An occurrence sends at most one message,
|
||||
in the owner's `locale` preference (see Account Preferences), else
|
||||
`channel_connections.notification_locale`. Notices carry no IDs and no links;
|
||||
see `backend/docs/CONFIGURATION.md` for the events and the merge rule.
|
||||
|
||||
```json
|
||||
{ "provider": "wecom", "display_name": "WeCom", "connection_status": "connected", "proactive_notifications": true }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Responses
|
||||
@ -1506,6 +1623,21 @@ Both endpoints return `Content-Location: /api/threads/{thread_id}/runs/{run_id}`
|
||||
The DeerFlow web UI and LangGraph SDK clients rely on this header to discover the
|
||||
assigned `thread_id` and `run_id` on the first message of a new chat.
|
||||
|
||||
Every SSE response (both create endpoints above, `GET /api/threads/{thread_id}/runs/{run_id}/join`
|
||||
and `GET`/`POST /api/threads/{thread_id}/runs/{run_id}/stream`) carries the same headers:
|
||||
|
||||
```http
|
||||
Content-Type: text/event-stream
|
||||
Cache-Control: no-cache, no-transform
|
||||
Connection: keep-alive
|
||||
X-Accel-Buffering: no
|
||||
```
|
||||
|
||||
`no-transform` keeps compressing proxies (for example the Next.js rewrite proxy
|
||||
when the frontend runs with `pnpm start` and no nginx) from gzipping and therefore
|
||||
buffering the stream; `X-Accel-Buffering: no` does the same for nginx. A reverse
|
||||
proxy you put in front of the Gateway should not compress `text/event-stream`.
|
||||
|
||||
### SSE replay retention and gaps
|
||||
|
||||
Clients may reconnect to a run stream with `Last-Event-ID`. Replay history is
|
||||
|
||||
@ -507,6 +507,7 @@ scheduler:
|
||||
poll_interval_seconds: 5
|
||||
lease_seconds: 120
|
||||
max_concurrent_runs: 3
|
||||
max_concurrent_runs_per_user: 2
|
||||
queue_timeout_seconds: 3600
|
||||
min_once_delay_seconds: 60
|
||||
recursion_limit: 1000
|
||||
@ -519,13 +520,16 @@ Notes:
|
||||
- The tasks page and REST API have parity with conversations for the per-run goal (`goal_objective`), safety cap (`max_runs`, `end_at`) and stop condition (`stop_condition`, stored in its own column and appended to the run message only at launch). Resume computes the next run from now without a catch-up run and refuses a one-time task whose time passed; reactivating a task whose cap is used up returns `409 limits_exhausted` unless the same request (for example the optional Resume body `{"max_runs", "end_at"}`) renews or clears the cap. Goal-check failures neither count toward nor reset the three-miss automatic pause, and changing the goal, instructions or stop condition, or adding a note, starts a new count (Resume keeps it). Creating a task while this Gateway process's poller is not running returns `409 scheduler_not_running`; `/api/features` reports `scheduled_tasks.running`. See `backend/docs/API.md#scheduled-tasks`.
|
||||
- `multi_instance: true` opts into lease-aware scheduler recovery across Gateway instances. It requires Postgres, `run_ownership.heartbeat_enabled: true`, and `run_events.backend: db`; otherwise startup fails fast. Leave it false for the default single-instance scheduler.
|
||||
- `max_concurrent_runs` is a shared global execution cap in multi-instance mode. Waiting `queued` rows do not consume capacity; an atomic `queued` → `launching` claim counts `launching`/`running` rows under a Postgres advisory lock so concurrent Pods cannot exceed the cap.
|
||||
- `queue_timeout_seconds` limits how long a persisted occurrence may wait for capacity or a reused thread to become available. Expired occurrences are marked `failed`; queued rows otherwise survive Gateway restarts.
|
||||
- `max_concurrent_runs_per_user` (default `2`, `0` = off, at most 32) caps how many scheduled runs one task owner may have `launching`/`running` at a time, whatever started them (manual "run now" included). The effective cap is `min(max_concurrent_runs_per_user, max_concurrent_runs)`, so with the defaults one owner holds at most 2 of the 3 global slots and the third stays free for other owners. The check runs in the same atomic claim, under the same database-wide lock, as the global cap, so it holds across workers and Pods. A row rejected by either cap stays `queued`.
|
||||
- The waiting queue is drained fairly whatever the per-owner cap: eligible rows are ranked within their owner and the drain batch takes every owner's oldest row before any owner's second one, so one owner's backlog cannot push another owner's run out of the batch. Owners already at their cap drop out of the batch before its size limit applies. Same-thread FIFO and the order within one owner (fewest launch attempts, then oldest) are unchanged; with a single owner the order is the same as before.
|
||||
- `queue_timeout_seconds` limits how long a persisted occurrence may wait for capacity or a reused thread to become available. Expired occurrences are marked `failed`; queued rows otherwise survive Gateway restarts. With the per-owner cap on, an owner whose runs take longer than this can see its waiting runs expire: the run history then reads "Skipped: it waited too long for a free slot", and no chat event or IM notice is sent for it. Keep `queue_timeout_seconds` well above an owner's typical run length.
|
||||
- A task definition is immutable while an occurrence is `queued`, `launching`, or `running`. This prevents a durable occurrence from mixing its admitted thread with a later prompt or schedule edit. Transitioning a task to paused or deleting it cancels a waiting row; PATCH and resume return a conflict until the active occurrence finishes or is cancelled.
|
||||
- A manual trigger remains explicit even while the recurring schedule is paused: it may wait in the durable queue and run later, while the task itself stays paused. Transitioning an enabled task to paused still cancels its waiting occurrence atomically.
|
||||
- Queue admission, PATCH/resume, pause, and delete serialize on the parent task row. Per-thread FIFO spans all active states, so an older `launching` or `running` occurrence blocks a newer queued occurrence on the same reused thread as well as an older `queued` occurrence.
|
||||
- Multi-instance reconciliation uses the run ownership lease: a live peer run is preserved, an expired lease is atomically taken over before its scheduled row is interrupted, and a stale Pod cannot overwrite a newer Pod's parent-task bookkeeping.
|
||||
- `recursion_limit` is the LangGraph super-step cap for scheduler-launched runs (default 1000, matching the web UI's interactive budget). Values above `max_recursion_limit` (default 1000) are clamped. This field is read at dispatch, so a YAML edit applies to the next scheduled run without a Gateway restart.
|
||||
- Poller fields (`enabled`, `multi_instance`, `poll_interval_seconds`, `lease_seconds`, `max_concurrent_runs`, `queue_timeout_seconds`, `min_once_delay_seconds`) are restart-required; edits need a Gateway restart.
|
||||
- Poller fields (`enabled`, `multi_instance`, `poll_interval_seconds`, `lease_seconds`, `max_concurrent_runs`, `max_concurrent_runs_per_user`, `queue_timeout_seconds`, `min_once_delay_seconds`) are restart-required; edits need a Gateway restart.
|
||||
- **Upgrade note (behavior change):** one owner now runs at most 2 scheduled runs at a time by default (before, up to `max_concurrent_runs`). Set `scheduler.max_concurrent_runs_per_user: 0` to restore the old behavior; the drain stays fair to owners either way.
|
||||
- **Upgrade note:** before upgrading a deployment with `GATEWAY_WORKERS > 1` and `scheduler.enabled: true`, either run the scheduler on exactly one Gateway worker or enable `scheduler.multi_instance: true` with shared Postgres, `run_ownership.heartbeat_enabled: true`, and `run_events.backend: db`. The startup gate now rejects the unsafe combination instead of allowing it to start silently.
|
||||
- **Upgrade note:** in multi-instance mode, `max_concurrent_runs` is cluster-wide rather than per Pod and counts `launching`/`running` occurrences. Waiting `queued` rows remain outside the execution cap; capacity does not multiply with the replica count.
|
||||
- **Upgrade note:** `scheduler.multi_instance` and its related scheduler, ownership, and run-event settings are startup-only. Restart all Gateway Pods together after changing them; a ConfigMap update without a coordinated restart leaves the running service on its previous mode.
|
||||
@ -535,7 +539,14 @@ Notes:
|
||||
- Create/update accept `once`, `cron`, and `interval`. Interval uses `schedule_spec.every_seconds` (UTC `now + N`, no missed-beat catch-up). N is at least `min_once_delay_seconds` (default 60) and at most 30 days.
|
||||
- Manual trigger uses the same scheduled-task resource and run lifecycle.
|
||||
- Scheduled task definitions and task-run history are persisted in the application database.
|
||||
- With `channel_connections.enabled: true`, the scheduler enqueues IM outcome notifications for the task owner's connected identities. A delivery worker (same poll cadence as the scheduler) pushes them. Scheduled runs that finish as success or failed notify. For goal tasks, an `unmet` scheduled occurrence sends a goal-unmet notice instead, and an automatic pause after three unmet occurrences sends an auto-pause notice; a stop requested by the agent adds no separate notice. Manual triggers and interrupts stay silent, and so do occurrences that end without a finished run (launch error, queue timeout, restart recovery). Channel/transport outages park rows without consuming the retry budget, for up to about a day; platform rejections exhaust ~15 minutes of counted retries then settle `failed`. The worker re-checks the binding right before sending: a target the owner has disconnected since enqueue is dropped as `failed`, never pushed. Only WeCom currently implements proactive `send_notification`.
|
||||
- With `channel_connections.enabled: true`, the scheduler sends IM notices ("scheduled task updates") to the task owner's connected identities. A delivery worker (same poll cadence as the scheduler) pushes them.
|
||||
- **Which apps:** only providers whose channel implements proactive push get notices; today that is WeCom (`proactive_notifications` in `app/channels/capabilities.py`, also returned by `GET /api/channels/providers`, where Settings shows "sent here" or "not available for this app yet"). Other providers get no outbox rows at all, and rows queued for them before the upgrade end once as `failed` without retries.
|
||||
- **Events:** `run_completed`, `run_failed`, `run_unmet` (goal missed), `task_paused` (automatic pause after three missed goals), `task_stopped` (paused by the agent because its stop condition was met) and `task_finished` (all `max_runs` done or `end_at` reached).
|
||||
- **One message per occurrence:** when one occurrence produces several events, only the first of `task_stopped` > `task_paused` > `task_finished` > `run_failed` > `run_unmet` > `run_completed` is sent, and that message still says how the last run went ("The last run failed." / "didn't meet the goal"). A one-time task sends only its run's outcome, never a separate `task_finished`.
|
||||
- **Transactional:** notices are queued by the finalization observer in the same database transaction that records the outcome and are deduplicated per occurrence (or per end time for a task that ends between runs), so an occurrence finalized by crash or lease recovery notifies exactly once. Plain manual "run now" trials and interrupted runs stay silent; a trial that pauses or finishes the task still sends that notice. Occurrences that end without a finished run (launch error, queue timeout, interrupted by a restart) stay silent.
|
||||
- **Text:** the task title, what happened, a one-line result (the same run summary the tasks page shows, redacted, only when the agent replied) and "Open DeerFlow → Scheduled tasks for details." Notices carry no task, run or thread IDs, no timestamps and no links: most deployments run on localhost or a LAN, where a link would be dead on the phone that receives it.
|
||||
- **Language:** the owner's web UI language (the `locale` user preference, which the web app keeps in sync), else `channel_connections.notification_locale` (`en-US` or `zh-CN`, default `en-US`; startup-scoped). Auth-disabled installs always use the config value.
|
||||
- **Delivery:** channel/transport outages park rows without consuming the retry budget, for up to about a day; platform rejections exhaust ~15 minutes of counted retries then settle `failed`. The worker re-checks the binding right before sending: a target the owner has disconnected since enqueue is dropped as `failed`, never pushed.
|
||||
|
||||
### Deployment topology (multi-instance)
|
||||
|
||||
|
||||
@ -2,8 +2,13 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
# Languages scheduled-task IM notices are rendered in (the web UI's languages).
|
||||
NotificationLocale = Literal["en-US", "zh-CN"]
|
||||
|
||||
|
||||
class SlackChannelConnectionConfig(BaseModel):
|
||||
enabled: bool = False
|
||||
@ -43,6 +48,10 @@ class ChannelConnectionsConfig(BaseModel):
|
||||
|
||||
enabled: bool = False
|
||||
require_bound_identity: bool = True
|
||||
# Language of scheduled-task IM notices for an owner who has no UI language
|
||||
# preference (auth disabled, or never signed in to the web app). The owner's
|
||||
# own preference, which the web app keeps in sync with its UI language, wins.
|
||||
notification_locale: NotificationLocale = "en-US"
|
||||
slack: SlackChannelConnectionConfig = Field(default_factory=SlackChannelConnectionConfig)
|
||||
telegram: TelegramChannelConnectionConfig = Field(default_factory=TelegramChannelConnectionConfig)
|
||||
discord: DiscordChannelConnectionConfig = Field(default_factory=DiscordChannelConnectionConfig)
|
||||
|
||||
@ -68,11 +68,12 @@ STARTUP_ONLY_FIELDS: dict[str, str] = {
|
||||
# config.yaml edits.
|
||||
"channels": ("start_channel_service() is invoked once during startup; the live IM channel clients (Feishu, Slack, Telegram, DingTalk) are not rebuilt when channels.* changes."),
|
||||
"channel_connections": (
|
||||
"start_channel_service() wires the connection repository and channel workers once at startup, and the channel-connections router caches the merged provider config on app.state; channel_connections.* edits need a restart."
|
||||
"start_channel_service() wires the connection repository and channel workers once at startup, and the channel-connections router caches the merged provider config on app.state; "
|
||||
"the scheduled-task notification delivery worker captures channel_connections.notification_locale when it starts. channel_connections.* edits need a restart."
|
||||
),
|
||||
"scheduler": (
|
||||
"ScheduledTaskService is constructed and started once during Gateway lifespan startup; enabled, poll_interval_seconds, lease_seconds, "
|
||||
"max_concurrent_runs, queue_timeout_seconds, and multi_instance are captured into the service instance and the background poller task is not rebuilt on config.yaml edits. "
|
||||
"max_concurrent_runs, max_concurrent_runs_per_user, queue_timeout_seconds, and multi_instance are captured into the service instance and the background poller task is not rebuilt on config.yaml edits. "
|
||||
"Changing multi-instance recovery prerequisites or lease behavior requires restarting every Gateway Pod together. "
|
||||
"scheduler.recursion_limit is not captured there: launch_scheduled_thread_run reads it from get_app_config() on each dispatch, so a YAML edit applies to the next scheduled run without a Gateway restart."
|
||||
),
|
||||
|
||||
@ -10,6 +10,12 @@ class SchedulerConfig(BaseModel):
|
||||
poll_interval_seconds: int = Field(default=5, ge=1, le=300)
|
||||
lease_seconds: int = Field(default=120, ge=5, le=3600)
|
||||
max_concurrent_runs: int = Field(default=3, ge=1, le=32)
|
||||
max_concurrent_runs_per_user: int = Field(
|
||||
default=2,
|
||||
ge=0,
|
||||
le=32,
|
||||
description="Cap on launching/running scheduled runs per task owner; 0 = no per-owner cap. The effective cap is min(this, max_concurrent_runs). Queue selection is fair to owners either way.",
|
||||
)
|
||||
queue_timeout_seconds: int = Field(default=3600, ge=60, le=604800)
|
||||
min_once_delay_seconds: int = Field(default=60, ge=1, le=86400)
|
||||
recursion_limit: int = Field(
|
||||
@ -28,6 +34,7 @@ class SchedulerConfig(BaseModel):
|
||||
"poll_interval_seconds",
|
||||
"lease_seconds",
|
||||
"max_concurrent_runs",
|
||||
"max_concurrent_runs_per_user",
|
||||
"queue_timeout_seconds",
|
||||
"min_once_delay_seconds",
|
||||
"recursion_limit",
|
||||
|
||||
@ -12,4 +12,8 @@ On SQLite, `BEGIN IMMEDIATE` takes the database-wide write lock, so every unrela
|
||||
|
||||
## Scheduled-task lifecycle
|
||||
|
||||
Scheduled-task lifecycle rules live in one place each. `ScheduledTaskRepository.update` is the single reactivation choke point: any `status -> enabled` from a non-enabled row re-checks `end_condition_reached` after applying the same request's renewal fields and raises `ScheduledTaskLimitsExhausted` (HTTP/tool `limits_exhausted`). The same method moves `unmet_streak_after_seq` when `goal_objective`, `prompt` or `stop_condition` actually changes (`append_standing_note` too; Resume never), and `_unmet_streak` skips both rows at/below that boundary and goal-check failures (`finalization.CHECK_FAILURE_CODES`). A host pause marker (`is_host_pause_marker`) survives a manual trial on a paused task. Pausing a terminal task returns `"finished"`. `stop_condition` has its own column; only `ScheduledTaskService` composes it into the launched message (`scheduler/stop_rule.launch_prompt`), so never write it into `prompt`. REST and capability validation share `app/gateway/scheduled_task_validation.py`, and every error raised there, in `routers/scheduled_tasks.py` and in `scheduled_task_access.py` uses `scheduler_error(status, code, message, **params)` with a code listed in `contracts/scheduled_task_errors_contract.json` (request models declare types only, so value rules never become FastAPI list details).
|
||||
Scheduled-task lifecycle rules live in one place each. `ScheduledTaskRepository.update` is the single reactivation choke point: any `status -> enabled` from a non-enabled row re-checks `end_condition_reached` after applying the same request's renewal fields and raises `ScheduledTaskLimitsExhausted` (HTTP/tool `limits_exhausted`). The same method moves `unmet_streak_after_seq` when `goal_objective`, `prompt` or `stop_condition` actually changes (`append_standing_note` too; Resume never), and `_unmet_streak` skips both rows at/below that boundary and goal-check failures (`finalization.CHECK_FAILURE_CODES`). A host pause marker (`is_host_pause_marker`) survives a manual trial on a paused task. A task is finished (`completed` by `max_runs`/`end_at`) only by `finalize_occurrence` or `finish_task_at_end_condition` (both in `scheduled_task_runs/finalization.py`), which report `task_stopped`/`task_paused`/`task_finished` to the finalization observer on a live -> paused/finished transition only, inside the caller's transaction; `test_no_inline_end_condition_completion` fails on any other `status = "completed"` in the two repositories. The scheduler's observer (`ScheduledTaskService._on_finalization`) is always installed and writes `scheduled_task_events` rows for the originating chat in that transaction; its lookups fall back on bad stored data and only database errors propagate, because it runs inside batch recovery. Pausing a terminal task returns `"finished"`. `stop_condition` has its own column; only `ScheduledTaskService` composes it into the launched message (`scheduler/stop_rule.launch_prompt`), so never write it into `prompt`. REST and capability validation share `app/gateway/scheduled_task_validation.py`, and every error raised there, in `routers/scheduled_tasks.py` and in `scheduled_task_access.py` uses `scheduler_error(status, code, message, **params)` with a code listed in `contracts/scheduled_task_errors_contract.json` (request models declare types only, so value rules never become FastAPI list details).
|
||||
|
||||
The per-owner run cap `scheduler.max_concurrent_runs_per_user` is effective `min(per_user, max_concurrent_runs)` (0 = off) and is checked in `claim_queued_run` right after the global count, under the same budget lock. `list_queued_runs` ranks same-thread FIFO heads per owner (`ROW_NUMBER() OVER (PARTITION BY owner)`) and orders the batch by that rank, so the drain is owner-fair even with the cap off; owners at their cap drop out before the batch limit, and orphan rows (task deleted while queued) are never held back by the cap. Scheduled-task repository boundaries coerce serialized timestamps before SQL `DateTime` binding.
|
||||
|
||||
Recovery locks task/run pairs in task-id/run-id order and restores `run_id`, `started_at` and live errors before releasing launch claims. Launch/failure/timeout updates use one parent-first transaction to prevent interleaved claims. Queue timeout fails the occurrence and advances scheduled work to prevent immediate requeue.
|
||||
|
||||
@ -27,8 +27,8 @@ The empty-DB path keeps using `create_all` because `Base.metadata` is the only a
|
||||
`0024_project_documents` → `0025_repair_run_change_seq` →
|
||||
`0026_mcp_task_lease_tokens` → `0027_notification_deliveries` →
|
||||
`0028_parked_attempts` → `0029_scheduler_agent_tasks` →
|
||||
`0030_notification_claim_tokens` → `0031_scheduled_streak_boundary`
|
||||
(current head). The preference
|
||||
`0030_notification_claim_tokens` → `0031_scheduled_streak_boundary` →
|
||||
`0032_activity_and_task_events` (current head). The preference
|
||||
revision adds a separate owner/key table with a cascading users foreign key and
|
||||
does not alter users; the project-documents revision adds a new owner-scoped
|
||||
shelf table, and the MCP lease-token revision adds two nullable token columns to
|
||||
@ -171,6 +171,7 @@ on installs that never enabled it. The convention is:
|
||||
- `migrations/versions/0027_notification_deliveries.py` — creates the scheduled-task IM notification outbox (`notification_deliveries`) with idempotency on `(task_run_id, event, provider, target)`; chains after `0026_mcp_task_lease_tokens`. Consumed by `ScheduledTaskService` enqueue + `NotificationDeliveryWorker` (issue #4254); no HTTP read surface yet
|
||||
- `migrations/versions/0028_parked_attempts.py` — adds `notification_deliveries.parked_attempts` so channel-down parking is capped; backfills with a temporary `server_default="0"` then drops it so the durable schema matches `create_all` (ORM Python-side `default=0` only). Chains after `0027_notification_deliveries`
|
||||
- `migrations/versions/0031_scheduled_streak_boundary.py` — adds nullable `scheduled_tasks.unmet_streak_after_seq` (occurrences at or below it never count toward the three-miss automatic pause; moved by goal/prompt/stop-condition/note edits, internal and never serialized) and nullable `scheduled_tasks.stop_condition` (the user's normalized "stop when …" rule, appended to the run message only at launch). Two nullable columns, so the bootstrap forward-compat floor is unchanged; chains after `0030_notification_claim_tokens`. Test: `tests/test_migration_0031_scheduled_streak_boundary.py`
|
||||
- `migrations/versions/0032_activity_and_task_events.py` — all schema of the scheduled-signals change in one revision: nullable `runs.origin_kind` (server-owned run origin; never backfilled, so legacy runs stay NULL, the "interactive or unknown" value) plus `ix_runs_thread_change_seq`, the per-user `thread_read_markers` (`(user_id, thread_id)` key, `seen_change_seq`) and `thread_read_versions` (`user_id` key, `version`) tables, and `scheduled_task_events` (lifecycle rows for the originating chat, unique on `(task_id, anchor, event)` as `uq_scheduled_task_event`, indexed on `(thread_id, created_at)` and `user_id`). `_helpers.py` guards only columns, so the revision guards every table and index on its own with a fresh inspector in both directions: a partially applied upgrade completes on the next start and an interrupted downgrade can run again. `RunRow` declares the index and column, and both new models are registered in `persistence/models`, because the empty-DB bootstrap uses `create_all` + `stamp head`. New tables and a nullable column, so the bootstrap forward-compat floor is unchanged; chains after `0031_scheduled_streak_boundary`. The `runs` index is built without `CONCURRENTLY` (a brief write block on large Postgres tables) and the SQLite downgrade rebuilds `runs`. Test: `tests/test_migration_0032_activity_and_task_events.py`
|
||||
- `persistence/bootstrap.py` — `bootstrap_schema(engine, backend=...)`, the three-branch provisioning decision, locked revision validation, and the narrow 0019 forward-compatibility exception
|
||||
- `extensions/loader.py::load_extensions` — registers each spec's `table_prefix` with `register_extension_table_prefix()`
|
||||
- Tests: `tests/test_persistence_bootstrap.py` (branches), `tests/test_persistence_bootstrap_concurrency.py` (concurrency), `tests/test_persistence_bootstrap_regression.py` (issue #3682), `tests/test_persistence_migrations_env.py` (filter, including extension-owned tables), `tests/test_extension_loader.py::TestTablePrefixRegistration` (spec-to-filter wiring), `tests/blocking_io/test_persistence_bootstrap.py` (asyncio.to_thread anchor), `tests/test_migration_0004_run_ownership_dedupe.py` + `tests/test_migration_0007_scheduled_run_active_dedupe.py` (dedupe-before-unique-index pre-steps), `tests/test_migration_0025_repair_run_change_seq.py` (issue #5516 skipped-revision heal)
|
||||
|
||||
@ -0,0 +1,122 @@
|
||||
"""Run origin, per-user thread read state and scheduled-task lifecycle events.
|
||||
|
||||
Revision ID: 0032_activity_and_task_events
|
||||
Revises: 0031_scheduled_streak_boundary
|
||||
|
||||
All schema of the scheduled-signals change in one revision:
|
||||
|
||||
- ``runs.origin_kind`` (nullable, never backfilled: legacy runs stay NULL, the
|
||||
"interactive or unknown" value) and ``ix_runs_thread_change_seq``;
|
||||
- ``thread_read_markers`` and ``thread_read_versions`` (per-user read state);
|
||||
- ``scheduled_task_events`` (lifecycle events for the originating chat,
|
||||
unique on ``(task_id, anchor, event)``).
|
||||
|
||||
``_helpers.py`` only guards columns, so every table and index is guarded on
|
||||
its own with a fresh inspector. A partially applied upgrade (for example one
|
||||
table created before a crash) therefore completes on the next start, and a
|
||||
downgrade can run again after an interrupted attempt.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision: str = "0032_activity_and_task_events"
|
||||
down_revision: str | Sequence[str] | None = "0031_scheduled_streak_boundary"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
_EVENT_INDEXES = (
|
||||
("ix_scheduled_task_events_thread", ["thread_id", "created_at"]),
|
||||
("ix_scheduled_task_events_user_id", ["user_id"]),
|
||||
)
|
||||
|
||||
|
||||
def _insp():
|
||||
return sa.inspect(op.get_bind())
|
||||
|
||||
|
||||
def _has_table(table: str) -> bool:
|
||||
return _insp().has_table(table)
|
||||
|
||||
|
||||
def _has_index(table: str, name: str) -> bool:
|
||||
if not _has_table(table):
|
||||
return False
|
||||
return name in {index["name"] for index in _insp().get_indexes(table)}
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
from deerflow.persistence.migrations._helpers import safe_add_column
|
||||
|
||||
# Denormalized run origin (NULL = interactive or legacy) and the per-thread
|
||||
# change clock index used by unread checks.
|
||||
safe_add_column("runs", sa.Column("origin_kind", sa.String(32), nullable=True))
|
||||
if not _has_index("runs", "ix_runs_thread_change_seq"):
|
||||
op.create_index("ix_runs_thread_change_seq", "runs", ["thread_id", "change_seq"])
|
||||
|
||||
# Per-user read markers.
|
||||
if not _has_table("thread_read_markers"):
|
||||
op.create_table(
|
||||
"thread_read_markers",
|
||||
sa.Column("user_id", sa.String(64), nullable=False),
|
||||
sa.Column("thread_id", sa.String(64), nullable=False),
|
||||
sa.Column("seen_change_seq", sa.BigInteger(), nullable=False, server_default=sa.text("0")),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.PrimaryKeyConstraint("user_id", "thread_id", name="pk_thread_read_markers"),
|
||||
)
|
||||
if not _has_index("thread_read_markers", "ix_thread_read_markers_thread_id"):
|
||||
op.create_index("ix_thread_read_markers_thread_id", "thread_read_markers", ["thread_id"])
|
||||
|
||||
# Per-user monotonic read clock, one row per user.
|
||||
if not _has_table("thread_read_versions"):
|
||||
op.create_table(
|
||||
"thread_read_versions",
|
||||
sa.Column("user_id", sa.String(64), nullable=False),
|
||||
sa.Column("version", sa.BigInteger(), nullable=False, server_default=sa.text("0")),
|
||||
sa.PrimaryKeyConstraint("user_id", name="pk_thread_read_versions"),
|
||||
)
|
||||
|
||||
# Durable lifecycle events for the originating chat.
|
||||
if not _has_table("scheduled_task_events"):
|
||||
op.create_table(
|
||||
"scheduled_task_events",
|
||||
sa.Column("id", sa.String(64), nullable=False),
|
||||
sa.Column("user_id", sa.String(64), nullable=False),
|
||||
sa.Column("task_id", sa.String(64), nullable=False),
|
||||
sa.Column("thread_id", sa.String(64), nullable=False),
|
||||
sa.Column("occurrence_id", sa.String(64), nullable=True),
|
||||
sa.Column("anchor", sa.String(96), nullable=False),
|
||||
sa.Column("event", sa.String(32), nullable=False),
|
||||
sa.Column("reason_code", sa.String(32), nullable=False),
|
||||
sa.Column("after_run_id", sa.String(64), nullable=True),
|
||||
sa.Column("payload_json", sa.JSON(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.PrimaryKeyConstraint("id", name="pk_scheduled_task_events"),
|
||||
sa.UniqueConstraint("task_id", "anchor", "event", name="uq_scheduled_task_event"),
|
||||
)
|
||||
for name, columns in _EVENT_INDEXES:
|
||||
if not _has_index("scheduled_task_events", name):
|
||||
op.create_index(name, "scheduled_task_events", columns)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
from deerflow.persistence.migrations._helpers import safe_drop_column
|
||||
|
||||
for name, _columns in reversed(_EVENT_INDEXES):
|
||||
if _has_index("scheduled_task_events", name):
|
||||
op.drop_index(name, table_name="scheduled_task_events")
|
||||
if _has_table("scheduled_task_events"):
|
||||
op.drop_table("scheduled_task_events")
|
||||
if _has_table("thread_read_versions"):
|
||||
op.drop_table("thread_read_versions")
|
||||
if _has_index("thread_read_markers", "ix_thread_read_markers_thread_id"):
|
||||
op.drop_index("ix_thread_read_markers_thread_id", table_name="thread_read_markers")
|
||||
if _has_table("thread_read_markers"):
|
||||
op.drop_table("thread_read_markers")
|
||||
if _has_index("runs", "ix_runs_thread_change_seq"):
|
||||
op.drop_index("ix_runs_thread_change_seq", table_name="runs")
|
||||
safe_drop_column("runs", "origin_kind")
|
||||
@ -29,10 +29,12 @@ from deerflow.persistence.notification_deliveries.model import NotificationDeliv
|
||||
from deerflow.persistence.personal_access_tokens.model import PersonalAccessTokenRow
|
||||
from deerflow.persistence.projects.model import ProjectDocumentRow, ProjectRow
|
||||
from deerflow.persistence.run.model import RunChangeClockRow, RunRow
|
||||
from deerflow.persistence.scheduled_task_events.model import ScheduledTaskEventRow
|
||||
from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow
|
||||
from deerflow.persistence.scheduled_tasks.model import ScheduledTaskRow
|
||||
from deerflow.persistence.subagent_batches.model import SubagentBatchItemRow, SubagentBatchRow
|
||||
from deerflow.persistence.thread_meta.model import ThreadMetaRow
|
||||
from deerflow.persistence.thread_reads.model import ThreadReadMarkerRow, ThreadReadVersionRow
|
||||
from deerflow.persistence.user.model import UserPreferenceRow, UserRow
|
||||
from deerflow.persistence.webhook_delivery.model import WebhookDeliveryRow
|
||||
|
||||
@ -52,11 +54,14 @@ __all__ = [
|
||||
"RunEventRow",
|
||||
"RunChangeClockRow",
|
||||
"RunRow",
|
||||
"ScheduledTaskEventRow",
|
||||
"ScheduledTaskRow",
|
||||
"ScheduledTaskRunRow",
|
||||
"SubagentBatchRow",
|
||||
"SubagentBatchItemRow",
|
||||
"ThreadMetaRow",
|
||||
"ThreadReadMarkerRow",
|
||||
"ThreadReadVersionRow",
|
||||
"UserPreferenceRow",
|
||||
"UserRow",
|
||||
"WebhookDeliveryRow",
|
||||
|
||||
@ -1,11 +1,12 @@
|
||||
"""ORM model for the scheduled-task notification delivery outbox (issue #4254).
|
||||
|
||||
A row records one pending IM notification for a scheduled-task outcome. The
|
||||
completion hook only enqueues; a separate delivery worker claims due rows and
|
||||
sends them. Execution status (``scheduled_task_runs``) and delivery status
|
||||
(this table) are deliberately independent: a failed delivery never changes the
|
||||
run outcome, and a re-fired completion hook cannot duplicate a notification
|
||||
because ``(task_run_id, event, provider, target)`` is unique.
|
||||
scheduler's finalization observer stages it in the transaction that records
|
||||
the outcome; a separate delivery worker claims due rows and sends them.
|
||||
Execution status (``scheduled_task_runs``) and delivery status (this table)
|
||||
are deliberately independent: a failed delivery never changes the run outcome,
|
||||
and a replayed finalization (recovery, a re-fired completion) cannot duplicate
|
||||
a notification because ``(task_run_id, event, provider, target)`` is unique.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@ -1,7 +1,8 @@
|
||||
"""SQL repository for the scheduled-task notification delivery outbox.
|
||||
|
||||
The completion hook calls :meth:`NotificationDeliveryRepository.enqueue`
|
||||
(idempotent), and the delivery worker uses ``claim_due_deliveries`` /
|
||||
The scheduler's finalization observer calls
|
||||
:meth:`NotificationDeliveryRepository.enqueue_in_session` (idempotent, inside
|
||||
the outcome's transaction), and the delivery worker uses ``claim_due_deliveries`` /
|
||||
``mark_sent`` / ``mark_failed``. Claim flips rows ``pending -> sending``
|
||||
inside one transaction guarded by ``status = 'pending'`` and stamps a fresh
|
||||
``claim_token``. Completion writes that carry the token are conditionally
|
||||
|
||||
@ -60,6 +60,10 @@ class RunRow(Base):
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC))
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
|
||||
change_seq: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0, server_default=text("0"))
|
||||
# Server-owned origin kind of the run (``deerflow_origin``: schedule,
|
||||
# im_channel, github, extension, mcp_notification). NULL means an
|
||||
# interactive or legacy run; legacy rows are never backfilled.
|
||||
origin_kind: Mapped[str | None] = mapped_column(String(32), nullable=True)
|
||||
|
||||
__table_args__ = (
|
||||
Index("ix_runs_thread_status", "thread_id", "status"),
|
||||
@ -67,6 +71,8 @@ class RunRow(Base):
|
||||
Index("uq_runs_idempotency_key", "idempotency_key", unique=True),
|
||||
Index("ix_runs_change_seq", "change_seq", "run_id"),
|
||||
Index("ix_runs_user_change_seq", "user_id", "change_seq", "run_id"),
|
||||
# Per-thread change clock: unread checks seek a thread's newest change.
|
||||
Index("ix_runs_thread_change_seq", "thread_id", "change_seq"),
|
||||
# Cross-process atomicity guarantee: at most one pending/running run per
|
||||
# thread. Must live in ORM ``__table_args__`` (not just the migration)
|
||||
# because the empty-DB bootstrap path runs ``create_all`` + ``stamp head``
|
||||
|
||||
@ -16,6 +16,7 @@ from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from deerflow.persistence.run.model import RunChangeClockRow, RunRow
|
||||
from deerflow.runtime.run_origin import origin_kind_of
|
||||
from deerflow.runtime.runs.store.base import (
|
||||
LeaseRenewal,
|
||||
RunIdempotencyConflict,
|
||||
@ -33,6 +34,15 @@ def _lease_expired_or_null(lease_col, cutoff: datetime):
|
||||
return or_(lease_col.is_(None), lease_col < cutoff)
|
||||
|
||||
|
||||
def _origin_kind(operation_kind: str, metadata: Any) -> str | None:
|
||||
"""Denormalized ``runs.origin_kind``: the server-owned origin of an agent run.
|
||||
|
||||
Thread operations (checkpoint writes, deletes) never carry one, and a run
|
||||
without a well-formed ``deerflow_origin`` stays NULL (interactive/legacy).
|
||||
"""
|
||||
return origin_kind_of(metadata) if operation_kind == "run" else None
|
||||
|
||||
|
||||
class RunRepository(RunStore):
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._sf = session_factory
|
||||
@ -155,6 +165,7 @@ class RunRepository(RunStore):
|
||||
"owner_worker_id": owner_worker_id,
|
||||
"lease_expires_at": lease_dt,
|
||||
"idempotency_key": idempotency_key,
|
||||
"origin_kind": _origin_kind(operation_kind, metadata),
|
||||
"updated_at": now,
|
||||
}
|
||||
async with self._sf() as session:
|
||||
@ -205,6 +216,17 @@ class RunRepository(RunStore):
|
||||
result = await session.execute(stmt)
|
||||
return [self._row_to_dict(row) for row in result.scalars()]
|
||||
|
||||
async def latest_change(self, *, user_id: str) -> tuple[int, str] | None:
|
||||
"""The caller's newest run-change position ``(change_seq, run_id)``, or ``None``.
|
||||
|
||||
Seeds the activity feed cursor: one backward seek on
|
||||
``ix_runs_user_change_seq``. ``user_id`` is always an explicit owner id.
|
||||
"""
|
||||
stmt = select(RunRow.change_seq, RunRow.run_id).where(RunRow.user_id == user_id).order_by(RunRow.change_seq.desc(), RunRow.run_id.desc()).limit(1)
|
||||
async with self._sf() as session:
|
||||
row = (await session.execute(stmt)).first()
|
||||
return (int(row.change_seq), str(row.run_id)) if row is not None else None
|
||||
|
||||
async def list_by_thread(
|
||||
self,
|
||||
thread_id,
|
||||
@ -864,6 +886,7 @@ class RunRepository(RunStore):
|
||||
"owner_worker_id": owner_worker_id,
|
||||
"lease_expires_at": lease_dt,
|
||||
"idempotency_key": idempotency_key,
|
||||
"origin_kind": _origin_kind(operation_kind, metadata),
|
||||
"created_at": created,
|
||||
"updated_at": now,
|
||||
}
|
||||
|
||||
@ -0,0 +1,9 @@
|
||||
"""Durable scheduled-task lifecycle events shown in the originating chat."""
|
||||
|
||||
from deerflow.persistence.scheduled_task_events.model import ScheduledTaskEventRow
|
||||
from deerflow.persistence.scheduled_task_events.sql import ScheduledTaskEventRepository
|
||||
|
||||
__all__ = [
|
||||
"ScheduledTaskEventRepository",
|
||||
"ScheduledTaskEventRow",
|
||||
]
|
||||
@ -0,0 +1,47 @@
|
||||
"""ORM model for scheduled-task lifecycle events (agent stop, auto-pause, finish).
|
||||
|
||||
The finalization observer writes one row per lifecycle transition in the same
|
||||
transaction as the task or occurrence state change. ``(task_id, anchor, event)``
|
||||
is unique, so crash and lease recovery that finalize again add nothing. The
|
||||
row is display-only chat history for the originating conversation: it is not
|
||||
part of the checkpoint and never becomes model context. Deleting the task
|
||||
keeps its rows (they carry a title snapshot); deleting the chat removes them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import JSON, DateTime, Index, PrimaryKeyConstraint, String, UniqueConstraint
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from deerflow.persistence.base import Base
|
||||
|
||||
|
||||
class ScheduledTaskEventRow(Base):
|
||||
__tablename__ = "scheduled_task_events"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
user_id: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
task_id: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
# The originating chat (the task's ``origin_thread_id``).
|
||||
thread_id: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
# NULL only for an idle finish (no occurrence ended the task).
|
||||
occurrence_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
# Dedupe anchor: the occurrence id, else ``end:<iso>`` / ``seq:<n>``.
|
||||
anchor: Mapped[str] = mapped_column(String(96), nullable=False)
|
||||
# task_stopped | task_paused | task_finished
|
||||
event: Mapped[str] = mapped_column(String(32), nullable=False)
|
||||
reason_code: Mapped[str] = mapped_column(String(32), nullable=False)
|
||||
# Newest run of the originating chat when the event was written; the chat
|
||||
# places the line after the turn that holds this run.
|
||||
after_run_id: Mapped[str | None] = mapped_column(String(64), nullable=True)
|
||||
payload_json: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, default=lambda: datetime.now(UTC))
|
||||
|
||||
__table_args__ = (
|
||||
PrimaryKeyConstraint("id", name="pk_scheduled_task_events"),
|
||||
UniqueConstraint("task_id", "anchor", "event", name="uq_scheduled_task_event"),
|
||||
Index("ix_scheduled_task_events_thread", "thread_id", "created_at"),
|
||||
Index("ix_scheduled_task_events_user_id", "user_id"),
|
||||
)
|
||||
@ -0,0 +1,119 @@
|
||||
"""SQL repository for scheduled-task lifecycle events.
|
||||
|
||||
``record_in_session`` runs inside the finalization transaction (the
|
||||
scheduler's finalization observer calls it with the caller's session), so the
|
||||
event commits or rolls back with the state change it reports. The insert is
|
||||
``ON CONFLICT (task_id, anchor, event) DO NOTHING``: recovery that finalizes
|
||||
the same transition again, or a replayed idle finish, adds no second row.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from deerflow.persistence.scheduled_task_events.model import ScheduledTaskEventRow
|
||||
from deerflow.utils.time import coerce_iso
|
||||
|
||||
# Payload keys the events route returns next to the row columns.
|
||||
_PAYLOAD_FIELDS = ("task_title", "stop_condition", "run_thread_id", "run_agent_name", "run_number", "run_status", "max_runs", "end_at", "schedule_type")
|
||||
|
||||
|
||||
class ScheduledTaskEventRepository:
|
||||
"""Persistence facade for ``scheduled_task_events``."""
|
||||
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._sf = session_factory
|
||||
|
||||
@staticmethod
|
||||
def to_api_dict(row: ScheduledTaskEventRow) -> dict[str, Any]:
|
||||
"""The events-route shape: row columns plus the display payload, no owner."""
|
||||
payload = row.payload_json if isinstance(row.payload_json, dict) else {}
|
||||
data: dict[str, Any] = {
|
||||
"id": row.id,
|
||||
"task_id": row.task_id,
|
||||
"event": row.event,
|
||||
"reason_code": row.reason_code,
|
||||
}
|
||||
for key in _PAYLOAD_FIELDS:
|
||||
data[key] = payload.get(key)
|
||||
data["after_run_id"] = row.after_run_id
|
||||
data["created_at"] = coerce_iso(row.created_at)
|
||||
return data
|
||||
|
||||
@staticmethod
|
||||
async def record_in_session(
|
||||
session: AsyncSession,
|
||||
*,
|
||||
user_id: str,
|
||||
task_id: str,
|
||||
thread_id: str,
|
||||
occurrence_id: str | None,
|
||||
anchor: str,
|
||||
event: str,
|
||||
reason_code: str,
|
||||
after_run_id: str | None,
|
||||
payload: dict[str, Any],
|
||||
created_at: datetime | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Stage one event row in the caller's transaction; a duplicate is ignored.
|
||||
|
||||
Never commits and never rolls back: conflict handling must not undo the
|
||||
caller's state change or open a second writer while it holds the parent
|
||||
lock. Returns the stored row (the existing one on a duplicate).
|
||||
"""
|
||||
dialect = session.get_bind().dialect.name
|
||||
if dialect not in {"sqlite", "postgresql"}:
|
||||
raise ValueError(f"Unsupported scheduled task event database dialect: {dialect}")
|
||||
insert = sqlite_insert if dialect == "sqlite" else pg_insert
|
||||
values = {
|
||||
"id": f"evt-{uuid.uuid4().hex}",
|
||||
"user_id": user_id,
|
||||
"task_id": task_id,
|
||||
"thread_id": thread_id,
|
||||
"occurrence_id": occurrence_id,
|
||||
"anchor": anchor,
|
||||
"event": event,
|
||||
"reason_code": reason_code,
|
||||
"after_run_id": after_run_id,
|
||||
"payload_json": dict(payload),
|
||||
"created_at": created_at or datetime.now(UTC),
|
||||
}
|
||||
statement = insert(ScheduledTaskEventRow).values(**values).on_conflict_do_nothing(index_elements=["task_id", "anchor", "event"])
|
||||
await session.execute(statement)
|
||||
row = await session.scalar(
|
||||
select(ScheduledTaskEventRow).where(
|
||||
ScheduledTaskEventRow.task_id == task_id,
|
||||
ScheduledTaskEventRow.anchor == anchor,
|
||||
ScheduledTaskEventRow.event == event,
|
||||
)
|
||||
)
|
||||
if row is None:
|
||||
raise RuntimeError("Scheduled task event insert produced no row")
|
||||
return ScheduledTaskEventRepository.to_api_dict(row)
|
||||
|
||||
async def list_for_thread(self, *, user_id: str, thread_id: str, limit: int = 50) -> list[dict[str, Any]]:
|
||||
"""The caller's newest ``limit`` events of one chat, returned oldest first."""
|
||||
stmt = select(ScheduledTaskEventRow).where(ScheduledTaskEventRow.thread_id == thread_id, ScheduledTaskEventRow.user_id == user_id).order_by(ScheduledTaskEventRow.created_at.desc(), ScheduledTaskEventRow.id.desc()).limit(limit)
|
||||
async with self._sf() as session:
|
||||
rows = list((await session.execute(stmt)).scalars())
|
||||
return [self.to_api_dict(row) for row in reversed(rows)]
|
||||
|
||||
async def delete_by_thread(self, thread_id: str, *, user_id: str | None) -> int:
|
||||
"""Delete a chat's event rows; an explicit ``user_id`` scopes it to that owner.
|
||||
|
||||
``None`` removes every owner's rows (migration and CLI callers).
|
||||
"""
|
||||
conditions = [ScheduledTaskEventRow.thread_id == thread_id]
|
||||
if user_id is not None:
|
||||
conditions.append(ScheduledTaskEventRow.user_id == user_id)
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(delete(ScheduledTaskEventRow).where(*conditions))
|
||||
await session.commit()
|
||||
return int(result.rowcount or 0)
|
||||
@ -13,12 +13,29 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow
|
||||
from deerflow.persistence.scheduled_task_runs.projection import account_launch, can_project
|
||||
from deerflow.scheduler.schedules import next_run_at
|
||||
from deerflow.utils.time import coerce_iso
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from deerflow.persistence.scheduled_tasks.model import ScheduledTaskRow
|
||||
|
||||
FinalizationObserver = Callable[..., Awaitable[None]]
|
||||
|
||||
# Lifecycle vocabulary, pinned as ``lifecycle_events``, ``lifecycle_reasons``
|
||||
# and ``notification_events`` in contracts/scheduled_goal_notes_contract.json.
|
||||
# ``task_paused`` is the automatic pause; the agent's own stop is
|
||||
# ``task_stopped``; ``task_finished`` is a once task's outcome or a reached
|
||||
# ``max_runs`` / ``end_at``.
|
||||
RUN_EVENT_BY_STATUS: dict[str, str] = {"success": "run_completed", "failed": "run_failed", "unmet": "run_unmet"}
|
||||
LIFECYCLE_EVENTS: tuple[str, ...] = ("task_stopped", "task_paused", "task_finished")
|
||||
LIFECYCLE_REASONS: dict[str, tuple[str, ...]] = {
|
||||
"task_stopped": ("agent_stop",),
|
||||
"task_paused": ("consecutive_unmet",),
|
||||
"task_finished": ("max_runs", "end_at", "once_done", "once_failed"),
|
||||
}
|
||||
NOTIFICATION_EVENTS: tuple[str, ...] = (*RUN_EVENT_BY_STATUS.values(), "task_paused", "task_stopped", "task_finished")
|
||||
# A once task's ``task_finished`` reports its own run outcome only for these.
|
||||
_ONCE_OUTCOME_STATUSES = frozenset(RUN_EVENT_BY_STATUS)
|
||||
|
||||
# Host-written task ``last_error`` values that the tasks page translates;
|
||||
# contracts/scheduled_goal_notes_contract.json pins them for the frontend.
|
||||
AGENT_STOP_LAST_ERROR_PREFIX = "stopped by the agent in run "
|
||||
@ -105,7 +122,7 @@ async def finalize_occurrence(
|
||||
observer: FinalizationObserver | None = None,
|
||||
) -> bool:
|
||||
"""Terminalize once; receipts may still be repaired by the launch helpers."""
|
||||
from deerflow.persistence.scheduled_tasks.model import ONCE_TASK_STATUS_BY_RUN_STATUS, TERMINAL_RUN_STATUSES
|
||||
from deerflow.persistence.scheduled_tasks.model import LIVE_TASK_STATUSES, ONCE_TASK_STATUS_BY_RUN_STATUS, TERMINAL_RUN_STATUSES
|
||||
|
||||
if occurrence.status in TERMINAL_RUN_STATUSES:
|
||||
return False
|
||||
@ -126,6 +143,9 @@ async def finalize_occurrence(
|
||||
occurrence.lease_expires_at = None
|
||||
if task is None:
|
||||
return True
|
||||
# The lifecycle events below fire only on a real transition, so read the
|
||||
# status before the projection turns ``running`` back into ``enabled``.
|
||||
previous_status = task.status
|
||||
if run_id is not None:
|
||||
account_launch(task, occurrence, run_id)
|
||||
if not can_project(task, occurrence):
|
||||
@ -144,17 +164,26 @@ async def finalize_occurrence(
|
||||
# agent" / "Auto-paused"); the trial's own outcome stays on its run row.
|
||||
if not (occurrence.trigger == "manual" and task.status == "paused" and is_host_pause_marker(task.last_error)):
|
||||
task.last_error = error
|
||||
events = {"success": ("run_completed",), "failed": ("run_failed",), "unmet": ("run_unmet",)}.get(status, ())
|
||||
events: tuple[str, ...] = (RUN_EVENT_BY_STATUS[status],) if status in RUN_EVENT_BY_STATUS else ()
|
||||
# A once task's result remains meaningful: an unmet once task failed.
|
||||
# Limits and agent stop apply to the recurring schedule lifecycle.
|
||||
if task.schedule_type == "once":
|
||||
task.status = ONCE_TASK_STATUS_BY_RUN_STATUS[status]
|
||||
# A cancel or a skip (``cancelled``) stays silent, like interrupted runs.
|
||||
if status in _ONCE_OUTCOME_STATUSES:
|
||||
events += ("task_finished",)
|
||||
elif await end_condition_reached(session, task, now=finished_at):
|
||||
task.status = "completed"
|
||||
task.next_run_at = None
|
||||
if previous_status in LIVE_TASK_STATUSES:
|
||||
events += ("task_finished",)
|
||||
elif occurrence.stop_requested_run_id is not None and occurrence.stop_requested_run_id == run_id:
|
||||
# The stop applies whatever the run's own outcome (PR1 semantics); the
|
||||
# notice carries ``run_status`` so it never claims a clean finish.
|
||||
task.status = "paused"
|
||||
task.last_error = f"{AGENT_STOP_LAST_ERROR_PREFIX}{run_id}"
|
||||
if previous_status != "paused":
|
||||
events += ("task_stopped",)
|
||||
elif occurrence.goal_objective is not None and await _unmet_streak(session, task, occurrence):
|
||||
was_paused = task.status == "paused"
|
||||
task.status = "paused"
|
||||
@ -165,3 +194,86 @@ async def finalize_occurrence(
|
||||
if observer is not None:
|
||||
await observer(session, task, occurrence, events=events)
|
||||
return True
|
||||
|
||||
|
||||
async def finish_task_at_end_condition(
|
||||
session: AsyncSession,
|
||||
task: ScheduledTaskRow,
|
||||
*,
|
||||
occurrence: ScheduledTaskRunRow | None,
|
||||
now: datetime,
|
||||
observer: FinalizationObserver | None,
|
||||
) -> None:
|
||||
"""Complete a task whose ``max_runs`` or ``end_at`` is reached, outside a run.
|
||||
|
||||
The single place, besides ``finalize_occurrence``, that marks a task
|
||||
finished: admission rejected as ended (``occurrence`` None), the claim-time
|
||||
skip of a queued row, and ``complete_if_ended``. The caller holds the parent
|
||||
lock and commits right after, so ``task_finished`` commits with the status
|
||||
change. It is emitted only on a live -> finished transition: a path that
|
||||
already went through ``finalize_occurrence`` (which emits for the same
|
||||
transition) adds nothing here. ``occurrence`` None means an idle finish.
|
||||
"""
|
||||
from deerflow.persistence.scheduled_tasks.model import LIVE_TASK_STATUSES
|
||||
|
||||
previous_status = task.status
|
||||
task.status = "completed"
|
||||
task.next_run_at = None
|
||||
task.lease_owner = None
|
||||
task.lease_expires_at = None
|
||||
task.updated_at = now
|
||||
if previous_status in LIVE_TASK_STATUSES and observer is not None:
|
||||
await observer(session, task, occurrence, events=("task_finished",))
|
||||
|
||||
|
||||
def _end_at_utc(task: ScheduledTaskRow) -> datetime | None:
|
||||
"""The task's end time as aware UTC, or None when unset or unreadable."""
|
||||
value = getattr(task, "end_at", None)
|
||||
if not isinstance(value, datetime):
|
||||
return None
|
||||
try:
|
||||
return utc(value)
|
||||
except (ValueError, TypeError, OverflowError):
|
||||
return None
|
||||
|
||||
|
||||
def lifecycle_reason(task: ScheduledTaskRow, event: str, *, now: datetime, occurrence: ScheduledTaskRunRow | None = None) -> str:
|
||||
"""Reason code of a lifecycle event (contract ``lifecycle_reasons``).
|
||||
|
||||
A once task's ``task_finished`` reports its run (``once_done`` /
|
||||
``once_failed``) when it comes from that run's outcome. A finish without
|
||||
such a run (an end time that passed before the once run) is ``end_at`` or
|
||||
``max_runs`` like a recurring task. An unreadable ``end_at`` reads as
|
||||
``max_runs``.
|
||||
"""
|
||||
if event == "task_stopped":
|
||||
return "agent_stop"
|
||||
if event == "task_paused":
|
||||
return "consecutive_unmet"
|
||||
if task.schedule_type == "once" and occurrence is not None and occurrence.status in _ONCE_OUTCOME_STATUSES:
|
||||
return "once_done" if task.status == "completed" else "once_failed"
|
||||
end_at = _end_at_utc(task)
|
||||
try:
|
||||
reached = end_at is not None and end_at <= utc(now)
|
||||
except (ValueError, TypeError, AttributeError):
|
||||
reached = False
|
||||
return "end_at" if reached else "max_runs"
|
||||
|
||||
|
||||
def lifecycle_anchor(task: ScheduledTaskRow, occurrence: ScheduledTaskRunRow | None, *, now: datetime) -> str:
|
||||
"""Dedupe anchor of a lifecycle event: one event per transition, replay-safe.
|
||||
|
||||
An occurrence anchors on its id. An idle finish anchors on its reason: the
|
||||
end time (``end:<iso>``) when that end time had passed at ``now`` (reason
|
||||
``end_at``), else the occurrence sequence high-water mark (``seq:<n>``,
|
||||
reason ``max_runs``). Re-running ``complete_if_ended`` cannot add a row,
|
||||
and an idle ``max_runs`` finish does not take the anchor of a later finish
|
||||
at the task's (then still future) end time. A resume with a new end time
|
||||
followed by a second finish gets a new anchor.
|
||||
"""
|
||||
if occurrence is not None:
|
||||
return occurrence.id
|
||||
end_at = _end_at_utc(task)
|
||||
if end_at is not None and lifecycle_reason(task, "task_finished", now=now) == "end_at":
|
||||
return f"end:{coerce_iso(end_at)}"
|
||||
return f"seq:{task.last_occurrence_seq}"
|
||||
|
||||
@ -12,7 +12,7 @@ from sqlalchemy.orm import aliased
|
||||
|
||||
from deerflow.persistence.run import RunRepository
|
||||
from deerflow.persistence.run.model import RunRow
|
||||
from deerflow.persistence.scheduled_task_runs.finalization import FinalizationObserver, end_condition_reached, finalize_occurrence, is_host_pause_marker
|
||||
from deerflow.persistence.scheduled_task_runs.finalization import FinalizationObserver, end_condition_reached, finalize_occurrence, finish_task_at_end_condition, is_host_pause_marker
|
||||
from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow
|
||||
from deerflow.persistence.scheduled_task_runs.projection import account_launch, can_project
|
||||
from deerflow.persistence.scheduled_tasks.model import (
|
||||
@ -283,10 +283,9 @@ class ScheduledTaskRunRepository:
|
||||
await session.rollback()
|
||||
raise ScheduledTaskAdmissionRejected(task_id, reason="stale")
|
||||
if trigger == "scheduled" and await end_condition_reached(session, task, now=scheduled_for):
|
||||
task.status = "completed"
|
||||
task.next_run_at = None
|
||||
task.lease_owner = None
|
||||
task.lease_expires_at = None
|
||||
# The row being admitted is never persisted, so this is an
|
||||
# idle finish (no occurrence anchor).
|
||||
await finish_task_at_end_condition(session, task, occurrence=None, now=scheduled_for, observer=self._finalization_observer)
|
||||
await session.commit()
|
||||
raise ScheduledTaskAdmissionRejected(task_id, reason="ended")
|
||||
active_status = await session.scalar(
|
||||
@ -442,40 +441,94 @@ class ScheduledTaskRunRepository:
|
||||
result = await session.execute(stmt)
|
||||
return int(result.scalar() or 0)
|
||||
|
||||
async def list_queued_runs(self, *, limit: int) -> list[dict[str, Any]]:
|
||||
async def list_queued_runs(self, *, limit: int, per_user_max_concurrent_runs: int = 0) -> list[dict[str, Any]]:
|
||||
"""The next bounded drain batch, fair to task owners (see ``_fair_queue_statement``)."""
|
||||
stmt = self._fair_queue_statement(limit=limit, per_user_max_concurrent_runs=per_user_max_concurrent_runs)
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(stmt)
|
||||
return [self._row_to_dict(row) for row in result.scalars()]
|
||||
|
||||
@staticmethod
|
||||
def _fair_queue_statement(*, limit: int, per_user_max_concurrent_runs: int = 0):
|
||||
"""SELECT for the next bounded drain batch, fair to task owners.
|
||||
|
||||
Only same-thread FIFO heads are eligible (an older active row on the
|
||||
same thread hides every newer one). Eligible rows are ranked within
|
||||
their owner by ``attempt_count, created_at, id`` and the batch is
|
||||
ordered by that rank first, so every owner's head comes before any
|
||||
owner's second row: one owner's backlog cannot fill the batch. With a
|
||||
per-owner cap, an owner only contributes as many rows as it has free
|
||||
slots (``cap - executing``), and owners already at the cap drop out
|
||||
before the limit is applied. With a single owner the order equals the
|
||||
plain ``attempt_count, created_at, id`` order.
|
||||
|
||||
Rows whose task is gone (deleted while queued) have no owner. They are
|
||||
never launched (the drain marks them interrupted), so the cap does not
|
||||
hold them back.
|
||||
"""
|
||||
candidate = aliased(ScheduledTaskRunRow)
|
||||
older = aliased(ScheduledTaskRunRow)
|
||||
older_same_thread = exists(
|
||||
select(older.id).where(
|
||||
older.thread_id == ScheduledTaskRunRow.thread_id,
|
||||
older.thread_id == candidate.thread_id,
|
||||
older.status.in_(ACTIVE_RUN_STATUSES),
|
||||
or_(
|
||||
older.created_at < ScheduledTaskRunRow.created_at,
|
||||
older.created_at < candidate.created_at,
|
||||
and_(
|
||||
older.created_at == ScheduledTaskRunRow.created_at,
|
||||
older.id < ScheduledTaskRunRow.id,
|
||||
older.created_at == candidate.created_at,
|
||||
older.id < candidate.id,
|
||||
),
|
||||
),
|
||||
)
|
||||
)
|
||||
stmt = (
|
||||
select(ScheduledTaskRunRow)
|
||||
.where(
|
||||
ScheduledTaskRunRow.status == "queued",
|
||||
~older_same_thread,
|
||||
)
|
||||
# Prefer rows that have had fewer launch attempts. A permanently
|
||||
# busy thread therefore cannot monopolize the bounded drain batch,
|
||||
# while created_at/id preserve FIFO order among equal attempts.
|
||||
.order_by(
|
||||
ScheduledTaskRunRow.attempt_count.asc(),
|
||||
ScheduledTaskRunRow.created_at.asc(),
|
||||
ScheduledTaskRunRow.id.asc(),
|
||||
)
|
||||
.limit(limit)
|
||||
executing_run = aliased(ScheduledTaskRunRow)
|
||||
executing_task = aliased(ScheduledTaskRow)
|
||||
executing = (
|
||||
select(executing_task.user_id.label("owner"), func.count().label("n"))
|
||||
.select_from(executing_run)
|
||||
.join(executing_task, executing_task.id == executing_run.task_id)
|
||||
.where(executing_run.status.in_(EXECUTING_RUN_STATUSES))
|
||||
.group_by(executing_task.user_id)
|
||||
.subquery("executing")
|
||||
)
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(stmt)
|
||||
return [self._row_to_dict(row) for row in result.scalars()]
|
||||
owner_task = aliased(ScheduledTaskRow)
|
||||
eligible = (
|
||||
select(
|
||||
candidate.id.label("id"),
|
||||
candidate.attempt_count.label("attempt_count"),
|
||||
candidate.created_at.label("created_at"),
|
||||
owner_task.user_id.label("owner"),
|
||||
func.coalesce(executing.c.n, 0).label("owner_executing"),
|
||||
func.row_number()
|
||||
.over(
|
||||
partition_by=owner_task.user_id,
|
||||
order_by=(candidate.attempt_count.asc(), candidate.created_at.asc(), candidate.id.asc()),
|
||||
)
|
||||
.label("owner_rank"),
|
||||
)
|
||||
.select_from(candidate)
|
||||
.outerjoin(owner_task, owner_task.id == candidate.task_id)
|
||||
.outerjoin(executing, executing.c.owner == owner_task.user_id)
|
||||
.where(candidate.status == "queued", ~older_same_thread)
|
||||
.subquery("eligible")
|
||||
)
|
||||
stmt = select(ScheduledTaskRunRow).join(eligible, eligible.c.id == ScheduledTaskRunRow.id)
|
||||
if per_user_max_concurrent_runs > 0:
|
||||
stmt = stmt.where(
|
||||
or_(
|
||||
eligible.c.owner.is_(None),
|
||||
eligible.c.owner_rank <= per_user_max_concurrent_runs - eligible.c.owner_executing,
|
||||
)
|
||||
)
|
||||
# Prefer rows that have had fewer launch attempts within an owner. A
|
||||
# permanently busy thread therefore cannot monopolize the bounded drain
|
||||
# batch, while created_at/id preserve FIFO order among equal attempts.
|
||||
return stmt.order_by(
|
||||
eligible.c.owner_rank.asc(),
|
||||
eligible.c.attempt_count.asc(),
|
||||
eligible.c.created_at.asc(),
|
||||
eligible.c.id.asc(),
|
||||
).limit(limit)
|
||||
|
||||
async def get_active_run(self, task_id: str) -> dict[str, Any] | None:
|
||||
stmt = (
|
||||
@ -499,8 +552,14 @@ class ScheduledTaskRunRepository:
|
||||
now: datetime,
|
||||
lease_seconds: int,
|
||||
global_max_concurrent_runs: int,
|
||||
per_user_max_concurrent_runs: int = 0,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Atomically move one waiting row into the lease-fenced launch phase."""
|
||||
"""Atomically move one waiting row into the lease-fenced launch phase.
|
||||
|
||||
Both budgets are checked under the same database-wide budget lock, so
|
||||
the per-owner cap is as atomic across workers and instances as the
|
||||
global one. A row rejected by either budget stays ``queued``.
|
||||
"""
|
||||
async with self._sf() as session:
|
||||
dialect = session.get_bind().dialect.name
|
||||
if dialect == "postgresql":
|
||||
@ -528,16 +587,29 @@ class ScheduledTaskRunRepository:
|
||||
return None
|
||||
if task is not None and row.trigger == "scheduled" and await end_condition_reached(session, task, now=now):
|
||||
await finalize_occurrence(session, task, row, status="skipped", error=RUN_ERROR_END_REACHED, finished_at=now, run_id=None, observer=self._finalization_observer)
|
||||
task.status = "completed"
|
||||
task.next_run_at = None
|
||||
task.lease_owner = None
|
||||
task.lease_expires_at = None
|
||||
# finalize_occurrence already finished a live recurring task
|
||||
# (and emitted); this only clears the lease and covers rows
|
||||
# that could not project, without emitting twice.
|
||||
await finish_task_at_end_condition(session, task, occurrence=row, now=now, observer=self._finalization_observer)
|
||||
await session.commit()
|
||||
return None
|
||||
executing = await session.scalar(select(func.count()).select_from(ScheduledTaskRunRow).where(ScheduledTaskRunRow.status.in_(EXECUTING_RUN_STATUSES)))
|
||||
if int(executing or 0) >= global_max_concurrent_runs:
|
||||
await session.rollback()
|
||||
return None
|
||||
if per_user_max_concurrent_runs > 0 and task is not None:
|
||||
owner_executing = await session.scalar(
|
||||
select(func.count())
|
||||
.select_from(ScheduledTaskRunRow)
|
||||
.join(ScheduledTaskRow, ScheduledTaskRow.id == ScheduledTaskRunRow.task_id)
|
||||
.where(
|
||||
ScheduledTaskRunRow.status.in_(EXECUTING_RUN_STATUSES),
|
||||
ScheduledTaskRow.user_id == task.user_id,
|
||||
)
|
||||
)
|
||||
if int(owner_executing or 0) >= per_user_max_concurrent_runs:
|
||||
await session.rollback()
|
||||
return None
|
||||
older = aliased(ScheduledTaskRunRow)
|
||||
older_same_thread = exists(
|
||||
select(older.id).where(
|
||||
|
||||
@ -84,6 +84,12 @@ ACTIVE_RUN_STATUSES: frozenset[str] = frozenset(
|
||||
}
|
||||
)
|
||||
|
||||
# Parent task lifecycle: a live task can still run (or be resumed); a terminal
|
||||
# one is finished. Defined here, next to the run statuses, so the finalization
|
||||
# module can use them without importing ``scheduled_tasks.sql`` (import cycle).
|
||||
TERMINAL_TASK_STATUSES: frozenset[str] = frozenset({"completed", "failed", "cancelled"})
|
||||
LIVE_TASK_STATUSES: frozenset[str] = frozenset({"enabled", "running", "paused"})
|
||||
|
||||
# Parent ``once`` task status projected from a terminal occurrence status.
|
||||
# Shared by the completion path and both recovery paths so the mapping
|
||||
# cannot drift between them.
|
||||
|
||||
@ -11,19 +11,16 @@ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from deerflow.persistence.run import RunRepository
|
||||
from deerflow.persistence.run.model import RunRow
|
||||
from deerflow.persistence.scheduled_task_runs.finalization import FinalizationObserver, automatic_runs_used, end_condition_reached, finalize_occurrence, is_host_pause_marker, utc
|
||||
from deerflow.persistence.scheduled_task_runs.finalization import FinalizationObserver, automatic_runs_used, end_condition_reached, finalize_occurrence, finish_task_at_end_condition, is_host_pause_marker, utc
|
||||
from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow
|
||||
from deerflow.persistence.scheduled_task_runs.projection import account_launch, can_project
|
||||
from deerflow.persistence.scheduled_tasks.model import ACTIVE_RUN_STATUSES, ONCE_TASK_STATUS_BY_RUN_STATUS, TERMINAL_RUN_STATUSES, ScheduledTaskRow
|
||||
from deerflow.persistence.scheduled_tasks.model import ACTIVE_RUN_STATUSES, LIVE_TASK_STATUSES, ONCE_TASK_STATUS_BY_RUN_STATUS, TERMINAL_RUN_STATUSES, TERMINAL_TASK_STATUSES, ScheduledTaskRow
|
||||
from deerflow.scheduler.host_notes import RUN_ERROR_END_REACHED
|
||||
from deerflow.utils.goal_objective import normalize_goal_objective
|
||||
from deerflow.utils.time import coerce_iso
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
TERMINAL_TASK_STATUSES: frozenset[str] = frozenset({"completed", "failed", "cancelled"})
|
||||
LIVE_TASK_STATUSES: frozenset[str] = frozenset({"enabled", "running", "paused"})
|
||||
|
||||
|
||||
class ScheduledTaskQuotaExceeded(ValueError):
|
||||
"""A tool-created task would exceed its owner's live schedule quota."""
|
||||
@ -775,12 +772,11 @@ class ScheduledTaskRepository:
|
||||
# Completion owns the lifecycle while work is already executing.
|
||||
return True
|
||||
if active is not None and active.trigger == "scheduled":
|
||||
# Skipping the waiting row finishes the task and emits there;
|
||||
# the helper below then only clears the lease.
|
||||
await finalize_occurrence(session, task, active, status="skipped", error=RUN_ERROR_END_REACHED, finished_at=now, run_id=None, observer=self._finalization_observer)
|
||||
task.status = "completed"
|
||||
task.next_run_at = None
|
||||
task.lease_owner = None
|
||||
task.lease_expires_at = None
|
||||
task.updated_at = now
|
||||
# No queued row (or a queued manual trial): an idle finish.
|
||||
await finish_task_at_end_condition(session, task, occurrence=None, now=now, observer=self._finalization_observer)
|
||||
await session.commit()
|
||||
return True
|
||||
|
||||
|
||||
@ -0,0 +1,10 @@
|
||||
"""Per-user read state of threads (server-created threads stay unread until opened)."""
|
||||
|
||||
from deerflow.persistence.thread_reads.model import ThreadReadMarkerRow, ThreadReadVersionRow
|
||||
from deerflow.persistence.thread_reads.sql import ThreadReadRepository
|
||||
|
||||
__all__ = [
|
||||
"ThreadReadMarkerRow",
|
||||
"ThreadReadRepository",
|
||||
"ThreadReadVersionRow",
|
||||
]
|
||||
@ -0,0 +1,40 @@
|
||||
"""ORM models for per-user thread read markers and the per-user read clock.
|
||||
|
||||
A marker stores the highest ``runs.change_seq`` of a thread the user has seen;
|
||||
a thread is unread while a server-originated run of that user changed after
|
||||
it. ``thread_read_versions`` is a per-user counter raised whenever a marker
|
||||
moves forward, so another device notices a read without a run change.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import BigInteger, DateTime, Index, PrimaryKeyConstraint, String, text
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from deerflow.persistence.base import Base
|
||||
|
||||
|
||||
class ThreadReadMarkerRow(Base):
|
||||
__tablename__ = "thread_read_markers"
|
||||
|
||||
user_id: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
thread_id: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
# Never moves backwards: concurrent reads from several devices keep the max.
|
||||
seen_change_seq: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0, server_default=text("0"))
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, default=lambda: datetime.now(UTC))
|
||||
|
||||
__table_args__ = (
|
||||
PrimaryKeyConstraint("user_id", "thread_id", name="pk_thread_read_markers"),
|
||||
Index("ix_thread_read_markers_thread_id", "thread_id"),
|
||||
)
|
||||
|
||||
|
||||
class ThreadReadVersionRow(Base):
|
||||
__tablename__ = "thread_read_versions"
|
||||
|
||||
user_id: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
version: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0, server_default=text("0"))
|
||||
|
||||
__table_args__ = (PrimaryKeyConstraint("user_id", name="pk_thread_read_versions"),)
|
||||
@ -0,0 +1,138 @@
|
||||
"""SQL repository for per-user thread read state.
|
||||
|
||||
A thread is **unread** for a user while one of that user's server-originated
|
||||
runs in it (``runs.origin_kind IS NOT NULL``) changed after the user last
|
||||
opened it. "Last opened" is ``thread_read_markers.seen_change_seq``: the
|
||||
highest ``runs.change_seq`` of the user's runs in that thread when they read
|
||||
it. The run-change clock is database-global and monotonic, so the marker
|
||||
works across workers and devices and never depends on wall-clock time.
|
||||
|
||||
``thread_read_versions`` is a per-user counter raised in the same transaction
|
||||
whenever a marker actually moves forward. The activity feed returns it, so
|
||||
another device notices a read even when no run changed.
|
||||
|
||||
Every method takes an explicit ``user_id`` string (never ``None`` or the
|
||||
``AUTO`` sentinel): the router resolves the caller once.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import and_, delete, func, select
|
||||
from sqlalchemy.dialects.postgresql import insert as pg_insert
|
||||
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from deerflow.persistence.run.model import RunRow
|
||||
from deerflow.persistence.thread_reads.model import ThreadReadMarkerRow, ThreadReadVersionRow
|
||||
|
||||
|
||||
def _insert_for(session: AsyncSession):
|
||||
dialect = session.get_bind().dialect.name
|
||||
if dialect == "postgresql":
|
||||
return pg_insert
|
||||
if dialect == "sqlite":
|
||||
return sqlite_insert
|
||||
raise ValueError(f"Unsupported thread read database dialect: {dialect}")
|
||||
|
||||
|
||||
class ThreadReadRepository:
|
||||
"""Persistence facade for ``thread_read_markers`` and ``thread_read_versions``."""
|
||||
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._sf = session_factory
|
||||
|
||||
async def mark_read(self, *, user_id: str, thread_id: str) -> int:
|
||||
"""Mark the thread read up to the user's newest run change; returns the stored position.
|
||||
|
||||
One transaction. The marker only ever moves forward (concurrent reads
|
||||
from several devices keep the maximum), and a read that changes
|
||||
nothing writes nothing: no marker update and no ``read_version``
|
||||
bump, so other tabs and devices do not refetch. A thread without any
|
||||
run of this user has nothing to read and gets no marker.
|
||||
"""
|
||||
async with self._sf() as session:
|
||||
insert = _insert_for(session)
|
||||
seen = int(
|
||||
await session.scalar(
|
||||
select(func.coalesce(func.max(RunRow.change_seq), 0)).where(
|
||||
RunRow.thread_id == thread_id,
|
||||
RunRow.user_id == user_id,
|
||||
)
|
||||
)
|
||||
or 0
|
||||
)
|
||||
raised = None
|
||||
if seen > 0:
|
||||
stmt = insert(ThreadReadMarkerRow).values(
|
||||
user_id=user_id,
|
||||
thread_id=thread_id,
|
||||
seen_change_seq=seen,
|
||||
updated_at=datetime.now(UTC),
|
||||
)
|
||||
stmt = stmt.on_conflict_do_update(
|
||||
index_elements=[ThreadReadMarkerRow.user_id, ThreadReadMarkerRow.thread_id],
|
||||
set_={"seen_change_seq": stmt.excluded.seen_change_seq, "updated_at": stmt.excluded.updated_at},
|
||||
where=ThreadReadMarkerRow.seen_change_seq < stmt.excluded.seen_change_seq,
|
||||
).returning(ThreadReadMarkerRow.seen_change_seq)
|
||||
raised = (await session.execute(stmt)).scalar_one_or_none()
|
||||
if raised is not None:
|
||||
version = insert(ThreadReadVersionRow).values(user_id=user_id, version=1)
|
||||
version = version.on_conflict_do_update(
|
||||
index_elements=[ThreadReadVersionRow.user_id],
|
||||
set_={"version": ThreadReadVersionRow.version + 1},
|
||||
)
|
||||
await session.execute(version)
|
||||
await session.commit()
|
||||
return int(raised)
|
||||
stored = await session.scalar(
|
||||
select(ThreadReadMarkerRow.seen_change_seq).where(
|
||||
ThreadReadMarkerRow.user_id == user_id,
|
||||
ThreadReadMarkerRow.thread_id == thread_id,
|
||||
)
|
||||
)
|
||||
await session.rollback()
|
||||
return int(stored) if stored is not None else seen
|
||||
|
||||
async def unread_thread_ids(self, *, user_id: str, thread_ids: Sequence[str]) -> set[str]:
|
||||
"""The subset of ``thread_ids`` that are unread for ``user_id``.
|
||||
|
||||
Only the viewer's own server-originated agent runs count: their own
|
||||
interactive runs, legacy rows (``origin_kind`` NULL) and other users'
|
||||
runs in a shared thread never make a thread unread.
|
||||
"""
|
||||
ids = list(dict.fromkeys(thread_ids))
|
||||
if not ids:
|
||||
return set()
|
||||
stmt = (
|
||||
select(RunRow.thread_id)
|
||||
.distinct()
|
||||
.select_from(RunRow)
|
||||
.outerjoin(
|
||||
ThreadReadMarkerRow,
|
||||
and_(ThreadReadMarkerRow.user_id == user_id, ThreadReadMarkerRow.thread_id == RunRow.thread_id),
|
||||
)
|
||||
.where(
|
||||
RunRow.thread_id.in_(ids),
|
||||
RunRow.user_id == user_id,
|
||||
RunRow.operation_kind == "run",
|
||||
RunRow.origin_kind.is_not(None),
|
||||
RunRow.change_seq > func.coalesce(ThreadReadMarkerRow.seen_change_seq, 0),
|
||||
)
|
||||
)
|
||||
async with self._sf() as session:
|
||||
return {str(thread_id) for thread_id in (await session.execute(stmt)).scalars()}
|
||||
|
||||
async def read_version(self, *, user_id: str) -> int:
|
||||
"""The user's read clock; 0 when they never marked anything read."""
|
||||
async with self._sf() as session:
|
||||
version = await session.scalar(select(ThreadReadVersionRow.version).where(ThreadReadVersionRow.user_id == user_id))
|
||||
return int(version or 0)
|
||||
|
||||
async def delete_by_thread(self, thread_id: str) -> None:
|
||||
"""Drop every user's marker for a deleted thread."""
|
||||
async with self._sf() as session:
|
||||
await session.execute(delete(ThreadReadMarkerRow).where(ThreadReadMarkerRow.thread_id == thread_id))
|
||||
await session.commit()
|
||||
80
backend/packages/harness/deerflow/runtime/run_origin.py
Normal file
80
backend/packages/harness/deerflow/runtime/run_origin.py
Normal file
@ -0,0 +1,80 @@
|
||||
"""Server-owned origin of a run or a thread (``deerflow_origin``).
|
||||
|
||||
A run started by the server on a user's behalf (a schedule, an IM channel, a
|
||||
GitHub agent, an extension, an MCP notification) carries
|
||||
``metadata["deerflow_origin"] = {"kind": ..., "provider"?: ..., "namespace"?: ...}``.
|
||||
Run admission derives ``runs.origin_kind`` from it, which drives the activity
|
||||
feed and the per-user unread state. A thread the server creates for such a run
|
||||
keeps the same marker in its metadata.
|
||||
|
||||
The value is never accepted from a browser, PAT or API client: the Gateway
|
||||
drops client copies and only stamps what a server-side launcher set (see
|
||||
``app.gateway.run_origin``). It is a different key from the message-level
|
||||
``deerflow_scheduled_origin`` that marks the scheduled prompt message inside a
|
||||
run; code must never read one for the other.
|
||||
|
||||
Pinned by ``contracts/thread_origin_contract.json``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
|
||||
DEERFLOW_ORIGIN_KEY = "deerflow_origin"
|
||||
ORIGIN_KINDS: frozenset[str] = frozenset({"schedule", "im_channel", "github", "extension", "mcp_notification"})
|
||||
|
||||
_PROVIDER_RE = re.compile(r"[a-z0-9_-]{1,32}")
|
||||
_MAX_NAMESPACE_CHARS = 96
|
||||
_ALLOWED_KEYS = frozenset({"kind", "provider", "namespace"})
|
||||
|
||||
|
||||
def admit_origin(value: object) -> dict[str, str] | None:
|
||||
"""Return a clean copy of a well-formed origin, else ``None``.
|
||||
|
||||
Shape: a mapping with ``kind`` in :data:`ORIGIN_KINDS`, an optional
|
||||
``provider`` (at most 32 chars of ``[a-z0-9_-]``) and an optional
|
||||
``namespace`` (1-96 chars, no ``:``). Anything else, including extra keys,
|
||||
is rejected as a whole rather than partially kept.
|
||||
"""
|
||||
if not isinstance(value, Mapping) or not set(value).issubset(_ALLOWED_KEYS):
|
||||
return None
|
||||
kind = value.get("kind")
|
||||
if not isinstance(kind, str) or kind not in ORIGIN_KINDS:
|
||||
return None
|
||||
origin = {"kind": kind}
|
||||
provider = value.get("provider")
|
||||
if provider is not None:
|
||||
if not isinstance(provider, str) or not _PROVIDER_RE.fullmatch(provider):
|
||||
return None
|
||||
origin["provider"] = provider
|
||||
namespace = value.get("namespace")
|
||||
if namespace is not None:
|
||||
if not isinstance(namespace, str) or not namespace or len(namespace) > _MAX_NAMESPACE_CHARS or ":" in namespace:
|
||||
return None
|
||||
origin["namespace"] = namespace
|
||||
return origin
|
||||
|
||||
|
||||
def make_origin(kind: str, *, provider: str | None = None, namespace: str | None = None) -> dict[str, str]:
|
||||
"""Build an origin for a server-side launcher; raises ``ValueError`` when malformed."""
|
||||
value: dict[str, str] = {"kind": kind}
|
||||
if provider is not None:
|
||||
value["provider"] = provider
|
||||
if namespace is not None:
|
||||
value["namespace"] = namespace
|
||||
origin = admit_origin(value)
|
||||
if origin is None:
|
||||
raise ValueError(f"invalid run origin: {value!r}")
|
||||
return origin
|
||||
|
||||
|
||||
def origin_kind_of(metadata: Mapping | None) -> str | None:
|
||||
"""``metadata["deerflow_origin"]["kind"]`` when well-formed, else ``None`` (interactive or legacy)."""
|
||||
if not isinstance(metadata, Mapping):
|
||||
return None
|
||||
origin = admit_origin(metadata.get(DEERFLOW_ORIGIN_KEY))
|
||||
return origin["kind"] if origin is not None else None
|
||||
|
||||
|
||||
__all__ = ["DEERFLOW_ORIGIN_KEY", "ORIGIN_KINDS", "admit_origin", "make_origin", "origin_kind_of"]
|
||||
40
backend/tests/_readable_helpers.py
Normal file
40
backend/tests/_readable_helpers.py
Normal file
@ -0,0 +1,40 @@
|
||||
"""Backend twin of the frontend ``expectNoRawIdentifiers`` helper.
|
||||
|
||||
User-visible text the backend writes itself (scheduled-task IM notices) must
|
||||
show no raw identifiers: UUIDs, task/run ids, ISO timestamps, five-field cron
|
||||
strings and internal enum or field names. The patterns are those of
|
||||
``frontend/tests/e2e/utils/raw-identifiers.ts``, plus URLs (IM notices carry no
|
||||
links) and any snake_case token (internal reason and event codes such as
|
||||
``consecutive_unmet`` or ``task_finished``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
RAW_IDENTIFIER_PATTERNS: tuple[re.Pattern[str], ...] = (
|
||||
re.compile(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", re.IGNORECASE),
|
||||
re.compile(r"\btask-(run-)?[0-9a-f]{16,}\b"),
|
||||
re.compile(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}"),
|
||||
re.compile(r"(^|\s)[\d*/,-]+(\s[\d*/,-]+){4}(\s|$)"),
|
||||
re.compile(r"\b(fresh_thread_per_run|reuse_thread|lead_agent|goal_objective|max_runs|end_at|unmet|stop_scheduled_task|schedule_task)\b"),
|
||||
# Backend additions: no links, no internal snake_case codes.
|
||||
re.compile(r"https?://", re.IGNORECASE),
|
||||
re.compile(r"\b[a-z][a-z0-9]*(?:_[a-z0-9]+)+\b"),
|
||||
)
|
||||
|
||||
|
||||
def find_raw_identifiers(text: str) -> list[str]:
|
||||
"""The patterns ``text`` matches, with the matched text, for a readable failure message."""
|
||||
found = []
|
||||
for pattern in RAW_IDENTIFIER_PATTERNS:
|
||||
match = pattern.search(text)
|
||||
if match:
|
||||
found.append(f'{pattern.pattern} -> "{match.group(0).strip()}"')
|
||||
return found
|
||||
|
||||
|
||||
def assert_no_raw_identifiers(text: str) -> None:
|
||||
"""Assert that user-visible ``text`` carries no raw identifiers, URLs or internal codes."""
|
||||
found = find_raw_identifiers(text)
|
||||
assert not found, f"raw identifiers in user-visible text: {found}\n---\n{text}"
|
||||
@ -46,7 +46,7 @@ def _channel(**overrides) -> BuzzChannel:
|
||||
def test_registered_in_framework_registries():
|
||||
assert _CHANNEL_REGISTRY["buzz"] == "app.channels.buzz:BuzzChannel"
|
||||
assert _CHANNEL_CREDENTIAL_KEYS["buzz"] == ["private_key"]
|
||||
assert CHANNEL_CAPABILITIES["buzz"] == {"supports_streaming": True}
|
||||
assert CHANNEL_CAPABILITIES["buzz"] == {"supports_streaming": True, "proactive_notifications": False}
|
||||
policy = CHANNEL_RUN_POLICY["buzz"]
|
||||
assert policy.serialize_thread_runs is True and policy.requires_bound_identity is False
|
||||
|
||||
|
||||
93
backend/tests/test_channel_capabilities.py
Normal file
93
backend/tests/test_channel_capabilities.py
Normal file
@ -0,0 +1,93 @@
|
||||
"""The static channel capability table stays in sync with the channel code.
|
||||
|
||||
The scheduler and the Settings page read ``proactive_notifications`` from
|
||||
``app.channels.capabilities`` without importing channel SDKs, so these tests
|
||||
pin that table to the channel registry and to the channel classes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.capabilities import CHANNEL_CAPABILITIES, supports_proactive_notifications
|
||||
from app.channels.service import _CHANNEL_REGISTRY
|
||||
from deerflow.reflection import resolve_class
|
||||
|
||||
_BACKEND_DIR = Path(__file__).resolve().parents[1]
|
||||
# Third-party modules the channel implementations import at module level.
|
||||
_CHANNEL_SDK_MODULES = ("lark_oapi", "slack_sdk", "telegram", "discord", "dingtalk_stream", "aibot", "websockets", "cryptography", "markdown_to_mrkdwn")
|
||||
|
||||
|
||||
def test_capabilities_cover_channel_registry():
|
||||
# A provider registered without a capability entry (as qq once was) fails here.
|
||||
assert set(CHANNEL_CAPABILITIES) == set(_CHANNEL_REGISTRY)
|
||||
assert CHANNEL_CAPABILITIES["qq"]["proactive_notifications"] is False
|
||||
|
||||
|
||||
def test_every_entry_declares_both_flags_as_bools():
|
||||
for provider, entry in CHANNEL_CAPABILITIES.items():
|
||||
assert set(entry) == {"supports_streaming", "proactive_notifications"}, provider
|
||||
assert all(isinstance(value, bool) for value in entry.values()), provider
|
||||
|
||||
|
||||
def test_only_wecom_pushes_proactively_today():
|
||||
assert {provider for provider, entry in CHANNEL_CAPABILITIES.items() if entry["proactive_notifications"]} == {"wecom"}
|
||||
assert supports_proactive_notifications("wecom") is True
|
||||
assert supports_proactive_notifications("feishu") is False
|
||||
|
||||
|
||||
@pytest.mark.parametrize("provider", ["", "unknown", "WeCom", None, 1])
|
||||
def test_unknown_provider_has_no_proactive_push(provider):
|
||||
assert supports_proactive_notifications(provider) is False
|
||||
|
||||
|
||||
def _missing_optional_module(exc: BaseException) -> str | None:
|
||||
"""The third-party module a resolution failure is missing, or None for any other failure."""
|
||||
current: BaseException | None = exc
|
||||
while current is not None:
|
||||
if isinstance(current, ModuleNotFoundError):
|
||||
name = current.name or ""
|
||||
return name if name and name != "app" and not name.startswith("app.") else None
|
||||
current = current.__cause__ or current.__context__
|
||||
return None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("provider", sorted(CHANNEL_CAPABILITIES))
|
||||
def test_proactive_flag_matches_send_notification_override(provider):
|
||||
# Resolved exactly as the channel service starts a channel.
|
||||
try:
|
||||
cls = resolve_class(_CHANNEL_REGISTRY[provider], base_class=None)
|
||||
except ImportError as exc:
|
||||
missing = _missing_optional_module(exc)
|
||||
if missing is None or provider == "wecom":
|
||||
# wecom imports no SDK at module level, so it must always resolve.
|
||||
raise
|
||||
pytest.skip(f"{provider}: optional SDK {missing} not installed")
|
||||
overrides = cls.send_notification is not Channel.send_notification
|
||||
assert CHANNEL_CAPABILITIES[provider]["proactive_notifications"] is overrides
|
||||
|
||||
|
||||
def test_importing_capabilities_loads_no_channel_sdk():
|
||||
code = (
|
||||
"import sys\n"
|
||||
"import app.channels.capabilities\n"
|
||||
f"sdk = {_CHANNEL_SDK_MODULES!r}\n"
|
||||
f"providers = {tuple(sorted(_CHANNEL_REGISTRY))!r}\n"
|
||||
"loaded = sorted(name for name in sys.modules if name.split('.')[0] in sdk or name in {'app.channels.' + p for p in providers})\n"
|
||||
"print(','.join(loaded))\n"
|
||||
)
|
||||
result = subprocess.run([sys.executable, "-c", code], cwd=_BACKEND_DIR, capture_output=True, text=True, check=True)
|
||||
assert result.stdout.strip() == ""
|
||||
|
||||
|
||||
def test_qq_streaming_is_unchanged_by_its_new_entry():
|
||||
from app.channels.manager import ChannelManager
|
||||
|
||||
# No running channel service: the manager falls back to the static table,
|
||||
# which says False for qq, as the missing entry did before.
|
||||
assert ChannelManager._channel_supports_streaming("qq") is False
|
||||
@ -1,5 +1,8 @@
|
||||
"""Tests for user-facing IM channel connection configuration."""
|
||||
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
from deerflow.config.channel_connections_config import ChannelConnectionsConfig
|
||||
|
||||
|
||||
@ -72,3 +75,14 @@ def test_qq_connection_configuration_is_opt_in():
|
||||
configured = ChannelConnectionsConfig.model_validate({"enabled": True, "qq": {"enabled": True}})
|
||||
assert configured.provider_status("qq") == {"enabled": True, "configured": True}
|
||||
assert configured.require_bound_identity is True
|
||||
|
||||
|
||||
def test_notification_locale_defaults_to_english():
|
||||
assert ChannelConnectionsConfig().notification_locale == "en-US"
|
||||
assert ChannelConnectionsConfig.model_validate({"notification_locale": "zh-CN"}).notification_locale == "zh-CN"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value", ["fr-FR", "zh", "en", "", None, 1])
|
||||
def test_notification_locale_accepts_only_the_ui_languages(value):
|
||||
with pytest.raises(ValidationError):
|
||||
ChannelConnectionsConfig.model_validate({"notification_locale": value})
|
||||
|
||||
@ -199,6 +199,17 @@ def test_get_providers_uses_existing_channels_config(tmp_path):
|
||||
"bot_id": "wecom-bot",
|
||||
"bot_secret": "********",
|
||||
}
|
||||
# Scheduled-task updates are pushed only where the provider implements
|
||||
# proactive push; Settings shows that per provider card.
|
||||
assert {name: item["proactive_notifications"] for name, item in by_provider.items()} == {
|
||||
"telegram": False,
|
||||
"slack": False,
|
||||
"discord": False,
|
||||
"feishu": False,
|
||||
"dingtalk": False,
|
||||
"wechat": False,
|
||||
"wecom": True,
|
||||
}
|
||||
|
||||
anyio.run(repo.close)
|
||||
|
||||
@ -1523,6 +1534,7 @@ def test_qq_provider_exposes_binding_flow_and_masks_secret(tmp_path):
|
||||
assert provider["auth_mode"] == "binding_code"
|
||||
assert provider["configured"] is True
|
||||
assert provider["connectable"] is True
|
||||
assert provider["proactive_notifications"] is False
|
||||
assert "fixture-private-secret" not in response.text
|
||||
fields = {field["name"]: field for field in provider["credential_fields"]}
|
||||
assert fields["app_id"]["type"] == "text"
|
||||
|
||||
@ -497,6 +497,7 @@ class TestChannelBase:
|
||||
from app.channels.feishu import FeishuChannel
|
||||
from app.channels.github import GitHubChannel
|
||||
from app.channels.manager import CHANNEL_CAPABILITIES
|
||||
from app.channels.qq import QQChannel
|
||||
from app.channels.slack import SlackChannel
|
||||
from app.channels.telegram import TelegramChannel
|
||||
from app.channels.wechat import WechatChannel
|
||||
@ -509,6 +510,7 @@ class TestChannelBase:
|
||||
"discord": DiscordChannel(bus=bus, config={}).supports_streaming,
|
||||
"feishu": FeishuChannel(bus=bus, config={}).supports_streaming,
|
||||
"github": GitHubChannel(bus=bus, config={}).supports_streaming,
|
||||
"qq": QQChannel(bus=bus, config={}).supports_streaming,
|
||||
"slack": SlackChannel(bus=bus, config={}).supports_streaming,
|
||||
"telegram": TelegramChannel(bus=bus, config={}).supports_streaming,
|
||||
"wechat": WechatChannel(bus=bus, config={}).supports_streaming,
|
||||
@ -2000,7 +2002,7 @@ class TestChannelManager:
|
||||
|
||||
history_by_checkpoint: dict[tuple[str, str], list[str]] = {}
|
||||
|
||||
async def _runs_wait(thread_id, assistant_id, *, input, config, context, multitask_strategy=None):
|
||||
async def _runs_wait(thread_id, assistant_id, *, input, config, context, multitask_strategy=None, metadata=None):
|
||||
del assistant_id, context # unused in this test, kept for signature parity
|
||||
|
||||
checkpoint_ns = config.get("configurable", {}).get("checkpoint_ns")
|
||||
|
||||
@ -87,6 +87,7 @@ def test_features_reports_agents_api_enabled() -> None:
|
||||
"scope_selection_enabled": False,
|
||||
},
|
||||
"scheduled_tasks": {"available": False, "running": False, "tool_enabled": False, "min_interval_seconds": 60},
|
||||
"thread_activity": {"available": False},
|
||||
}
|
||||
|
||||
|
||||
@ -109,6 +110,7 @@ def test_features_reports_agents_api_disabled() -> None:
|
||||
"scope_selection_enabled": False,
|
||||
},
|
||||
"scheduled_tasks": {"available": False, "running": False, "tool_enabled": False, "min_interval_seconds": 60},
|
||||
"thread_activity": {"available": False},
|
||||
}
|
||||
|
||||
|
||||
@ -255,3 +257,30 @@ def test_features_reports_scheduled_tasks_process_state(repo: bool, running: boo
|
||||
response = client.get("/api/features")
|
||||
assert response.status_code == 200
|
||||
assert response.json()["scheduled_tasks"] == {**expected, "min_interval_seconds": 60}
|
||||
|
||||
|
||||
class _SqlRunStore:
|
||||
async def latest_change(self, *, user_id):
|
||||
return None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("run_store", "read_repo", "available"),
|
||||
[
|
||||
(None, None, False),
|
||||
(SimpleNamespace(), object(), False), # memory run store: no run-change clock seek
|
||||
(_SqlRunStore(), None, False),
|
||||
(_SqlRunStore(), object(), True),
|
||||
],
|
||||
ids=["memory", "memory-run-store", "no-read-repo", "sql"],
|
||||
)
|
||||
def test_features_reports_thread_activity_only_with_sql_persistence(run_store, read_repo, available: bool) -> None:
|
||||
"""The frontend polls /api/thread-activity only when this says so; a
|
||||
missing block on an older backend means unavailable."""
|
||||
app = _app_with_config(agents_api_enabled=True)
|
||||
app.state.run_store = run_store
|
||||
app.state.thread_read_repo = read_repo
|
||||
with TestClient(app) as client:
|
||||
response = client.get("/api/features")
|
||||
assert response.status_code == 200
|
||||
assert response.json()["thread_activity"] == {"available": available}
|
||||
|
||||
@ -47,6 +47,9 @@ def test_enabled_scheduler_start_failure_aborts_gateway_lifespan():
|
||||
startup_config.scheduler.poll_interval_seconds = 5
|
||||
startup_config.scheduler.lease_seconds = 120
|
||||
startup_config.scheduler.max_concurrent_runs = 3
|
||||
# Set explicitly: a bare MagicMock attribute must never reach the
|
||||
# per-owner cap the drain query compares against.
|
||||
startup_config.scheduler.max_concurrent_runs_per_user = 2
|
||||
startup_config.scheduler.queue_timeout_seconds = 3600
|
||||
startup_config.run_ownership.grace_seconds = 10
|
||||
channel_service = MagicMock()
|
||||
@ -63,7 +66,7 @@ def test_enabled_scheduler_start_failure_aborts_gateway_lifespan():
|
||||
patch("app.gateway.app.auth.close_oidc_service", AsyncMock()),
|
||||
patch("app.channels.service.start_channel_service", start_channel_service),
|
||||
patch("app.channels.service.stop_channel_service", AsyncMock()),
|
||||
patch("app.scheduler.ScheduledTaskService", return_value=scheduler_service),
|
||||
patch("app.scheduler.ScheduledTaskService", return_value=scheduler_service) as service_class,
|
||||
patch("deerflow.skills.projection.ensure_public_skill_projection"),
|
||||
patch("deerflow.agents.memory.get_memory_manager", return_value=MagicMock()),
|
||||
):
|
||||
@ -73,6 +76,9 @@ def test_enabled_scheduler_start_failure_aborts_gateway_lifespan():
|
||||
|
||||
scheduler_service.start.assert_awaited_once()
|
||||
start_channel_service.assert_not_awaited()
|
||||
# The startup-captured per-owner cap reaches the service unchanged.
|
||||
assert service_class.call_args.kwargs["max_concurrent_runs_per_user"] == 2
|
||||
assert service_class.call_args.kwargs["max_concurrent_runs"] == 3
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
@ -774,6 +780,7 @@ def _notification_startup_config(*, channel_connections_enabled: bool = True):
|
||||
poll_interval_seconds=5,
|
||||
lease_seconds=30,
|
||||
max_concurrent_runs=1,
|
||||
max_concurrent_runs_per_user=2,
|
||||
multi_instance=False,
|
||||
queue_timeout_seconds=3600,
|
||||
),
|
||||
@ -868,6 +875,8 @@ def test_lifespan_starts_notification_worker_when_enqueue_and_channel_are_wired(
|
||||
resolve_connections = worker_on_app.factory_kwargs["resolve_connections"]
|
||||
assert resolve_connections is not None
|
||||
assert getattr(resolve_connections, "__name__", None) == "list_connections"
|
||||
# Notices of owners without a UI language preference use channel_connections.notification_locale.
|
||||
assert worker_on_app.factory_kwargs["default_locale"] == "en-US"
|
||||
scheduled_service.detach_notification_outbox.assert_not_called()
|
||||
worker_stop.assert_awaited_once()
|
||||
stop_channel_service.assert_awaited_once()
|
||||
|
||||
@ -44,6 +44,12 @@ def _seed_database_that_skipped_0023(db_path) -> None:
|
||||
try:
|
||||
Base.metadata.create_all(sync_engine)
|
||||
with sync_engine.begin() as conn:
|
||||
# 0032 had not run at this stamp; its runs index covers
|
||||
# change_seq, so it goes before the column.
|
||||
conn.execute(sa.text("DROP INDEX IF EXISTS ix_runs_thread_change_seq"))
|
||||
conn.execute(sa.text("ALTER TABLE runs DROP COLUMN origin_kind"))
|
||||
for table in ("thread_read_markers", "thread_read_versions", "scheduled_task_events"):
|
||||
conn.execute(sa.text(f"DROP TABLE IF EXISTS {table}"))
|
||||
# Remove everything 0023_run_change_seq owns.
|
||||
conn.execute(sa.text("DROP INDEX IF EXISTS ix_runs_change_seq"))
|
||||
conn.execute(sa.text("DROP INDEX IF EXISTS ix_runs_user_change_seq"))
|
||||
|
||||
@ -23,7 +23,7 @@ OUTBOX = "0027_notification_deliveries"
|
||||
PARKED = "0028_parked_attempts"
|
||||
SCHEDULER = "0029_scheduler_agent_tasks"
|
||||
CLAIM_TOKENS = "0030_notification_claim_tokens"
|
||||
HEAD = "0031_scheduled_streak_boundary"
|
||||
HEAD = "0032_activity_and_task_events"
|
||||
PREVIOUS = "0026_mcp_task_lease_tokens"
|
||||
TABLE = "notification_deliveries"
|
||||
|
||||
|
||||
@ -23,7 +23,7 @@ from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
|
||||
from deerflow.persistence.scheduled_tasks import ScheduledTaskRepository
|
||||
|
||||
REVISION = "0029_scheduler_agent_tasks"
|
||||
CURRENT_HEAD = "0031_scheduled_streak_boundary"
|
||||
CURRENT_HEAD = "0032_activity_and_task_events"
|
||||
NEXT = "0030_notification_claim_tokens"
|
||||
PREVIOUS = "0028_parked_attempts"
|
||||
TASK_FIELDS = {"origin_thread_id", "goal_objective", "max_runs", "end_at", "standing_notes"}
|
||||
|
||||
@ -18,14 +18,16 @@ from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
|
||||
|
||||
REVISION = "0031_scheduled_streak_boundary"
|
||||
PREVIOUS = "0030_notification_claim_tokens"
|
||||
NEXT = "0032_activity_and_task_events"
|
||||
COLUMNS = {"unmet_streak_after_seq", "stop_condition"}
|
||||
pytestmark = pytest.mark.asyncio
|
||||
|
||||
|
||||
async def test_0031_is_the_single_head_after_0030():
|
||||
async def test_0031_follows_0030_and_precedes_0032():
|
||||
script = ScriptDirectory(str(bootstrap._MIGRATIONS_DIR))
|
||||
assert script.get_heads() == [REVISION]
|
||||
assert len(script.get_heads()) == 1
|
||||
assert script.get_revision(REVISION).down_revision == PREVIOUS
|
||||
assert script.get_revision(NEXT).down_revision == REVISION
|
||||
# alembic_version.version_num is VARCHAR(32).
|
||||
assert len(REVISION) <= 32
|
||||
|
||||
|
||||
167
backend/tests/test_migration_0032_activity_and_task_events.py
Normal file
167
backend/tests/test_migration_0032_activity_and_task_events.py
Normal file
@ -0,0 +1,167 @@
|
||||
"""Migration 0032: run origin column and index, read-state tables, task events."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import uuid
|
||||
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
|
||||
|
||||
import pytest
|
||||
import sqlalchemy as sa
|
||||
from alembic import command
|
||||
from alembic.script import ScriptDirectory
|
||||
from sqlalchemy.ext.asyncio import create_async_engine
|
||||
|
||||
from deerflow.persistence import bootstrap
|
||||
from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
|
||||
|
||||
REVISION = "0032_activity_and_task_events"
|
||||
PREVIOUS = "0031_scheduled_streak_boundary"
|
||||
TABLES = {"thread_read_markers", "thread_read_versions", "scheduled_task_events"}
|
||||
INDEXES = {
|
||||
"runs": {"ix_runs_thread_change_seq"},
|
||||
"thread_read_markers": {"ix_thread_read_markers_thread_id"},
|
||||
"scheduled_task_events": {"ix_scheduled_task_events_thread", "ix_scheduled_task_events_user_id"},
|
||||
}
|
||||
pytestmark = pytest.mark.asyncio
|
||||
|
||||
|
||||
async def test_0032_is_the_single_head_after_0031():
|
||||
script = ScriptDirectory(str(bootstrap._MIGRATIONS_DIR))
|
||||
assert script.get_heads() == [REVISION]
|
||||
assert script.get_revision(REVISION).down_revision == PREVIOUS
|
||||
# alembic_version.version_num is VARCHAR(32).
|
||||
assert len(REVISION) <= 32
|
||||
|
||||
|
||||
def _engine(tmp_path, backend):
|
||||
if backend == "sqlite":
|
||||
return create_async_engine(f"sqlite+aiosqlite:///{tmp_path / 'migration.db'}"), None
|
||||
uri = os.environ.get("TEST_POSTGRES_URI")
|
||||
if not uri:
|
||||
pytest.skip("requires TEST_POSTGRES_URI (real Postgres migration 0032)")
|
||||
parts = urlsplit(uri)
|
||||
scheme = "postgresql+asyncpg" if parts.scheme in {"postgres", "postgresql"} else parts.scheme
|
||||
query = urlencode([(key, value) for key, value in parse_qsl(parts.query, keep_blank_values=True) if key not in {"sslmode", "channel_binding"}])
|
||||
schema = f"activity_events_{uuid.uuid4().hex}"
|
||||
return create_async_engine(urlunsplit(parts._replace(scheme=scheme, query=query)), connect_args=build_asyncpg_connect_args(schema)), schema
|
||||
|
||||
|
||||
async def _shape(engine) -> dict:
|
||||
async with engine.connect() as conn:
|
||||
|
||||
def read(sync):
|
||||
inspector = sa.inspect(sync)
|
||||
tables = set(inspector.get_table_names())
|
||||
return {
|
||||
"tables": tables,
|
||||
"run_columns": {col["name"]: col for col in inspector.get_columns("runs")},
|
||||
"indexes": {table: {index["name"] for index in inspector.get_indexes(table)} for table in INDEXES if table in tables},
|
||||
"uniques": {constraint["name"]: constraint["column_names"] for constraint in inspector.get_unique_constraints("scheduled_task_events")} if "scheduled_task_events" in tables else {},
|
||||
}
|
||||
|
||||
return await conn.run_sync(read)
|
||||
|
||||
|
||||
async def _insert_legacy_run(engine) -> None:
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(
|
||||
sa.text(
|
||||
"INSERT INTO runs (run_id,thread_id,user_id,status,operation_kind,multitask_strategy,metadata_json,kwargs_json,message_count,"
|
||||
"total_input_tokens,total_output_tokens,total_tokens,llm_call_count,lead_agent_tokens,subagent_tokens,middleware_tokens,"
|
||||
"token_usage_by_model,change_seq,created_at,updated_at) "
|
||||
"VALUES ('legacy-run','legacy-thread','owner','success','run','reject','{}','{}',0,0,0,0,0,0,0,0,'{}',7,CURRENT_TIMESTAMP,CURRENT_TIMESTAMP)"
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", ["sqlite", "postgres"])
|
||||
async def test_0032_upgrade_keeps_runs_and_downgrade_drops_everything(tmp_path, backend):
|
||||
engine, schema = _engine(tmp_path, backend)
|
||||
cfg = bootstrap._get_alembic_config(engine, postgres_schema=schema or "")
|
||||
try:
|
||||
if schema:
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(sa.text(f'CREATE SCHEMA "{schema}"'))
|
||||
await asyncio.to_thread(command.upgrade, cfg, PREVIOUS)
|
||||
await _insert_legacy_run(engine)
|
||||
|
||||
await asyncio.to_thread(command.upgrade, cfg, REVISION)
|
||||
shape = await _shape(engine)
|
||||
assert TABLES <= shape["tables"]
|
||||
assert shape["run_columns"]["origin_kind"]["nullable"] is True
|
||||
assert shape["run_columns"]["origin_kind"]["default"] is None
|
||||
for table, names in INDEXES.items():
|
||||
assert names <= shape["indexes"][table], table
|
||||
assert shape["uniques"]["uq_scheduled_task_event"] == ["task_id", "anchor", "event"]
|
||||
async with engine.connect() as conn:
|
||||
# Never backfilled: legacy runs stay "interactive or unknown".
|
||||
row = (await conn.execute(sa.text("SELECT origin_kind, change_seq, status FROM runs WHERE run_id='legacy-run'"))).one()
|
||||
assert tuple(row) == (None, 7, "success")
|
||||
|
||||
# The unique key deduplicates lifecycle events at the database level.
|
||||
insert = sa.text(
|
||||
"INSERT INTO scheduled_task_events (id,user_id,task_id,thread_id,occurrence_id,anchor,event,reason_code,after_run_id,payload_json,created_at) "
|
||||
"VALUES (:id,'owner','task','origin','occ','occ','task_finished','max_runs',NULL,'{}',CURRENT_TIMESTAMP)"
|
||||
)
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(insert, {"id": "evt-1"})
|
||||
with pytest.raises(sa.exc.IntegrityError):
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(insert, {"id": "evt-2"})
|
||||
|
||||
await asyncio.to_thread(command.downgrade, cfg, PREVIOUS)
|
||||
shape = await _shape(engine)
|
||||
assert TABLES.isdisjoint(shape["tables"])
|
||||
assert "origin_kind" not in shape["run_columns"]
|
||||
assert "ix_runs_thread_change_seq" not in shape["indexes"]["runs"]
|
||||
async with engine.connect() as conn:
|
||||
assert (await conn.execute(sa.text("SELECT change_seq FROM runs WHERE run_id='legacy-run'"))).scalar_one() == 7
|
||||
|
||||
# Downgrade then upgrade again is clean.
|
||||
await asyncio.to_thread(command.upgrade, cfg, REVISION)
|
||||
assert TABLES <= (await _shape(engine))["tables"]
|
||||
finally:
|
||||
if schema:
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(sa.text(f'DROP SCHEMA IF EXISTS "{schema}" CASCADE'))
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", ["sqlite", "postgres"])
|
||||
async def test_0032_completes_a_partially_applied_upgrade(tmp_path, backend):
|
||||
"""One table and the column already exist, the runs index is missing."""
|
||||
engine, schema = _engine(tmp_path, backend)
|
||||
cfg = bootstrap._get_alembic_config(engine, postgres_schema=schema or "")
|
||||
try:
|
||||
if schema:
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(sa.text(f'CREATE SCHEMA "{schema}"'))
|
||||
await asyncio.to_thread(command.upgrade, cfg, PREVIOUS)
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(sa.text("ALTER TABLE runs ADD COLUMN origin_kind VARCHAR(32)"))
|
||||
await conn.execute(sa.text("CREATE TABLE thread_read_versions (user_id VARCHAR(64) NOT NULL, version BIGINT DEFAULT 0 NOT NULL, CONSTRAINT pk_thread_read_versions PRIMARY KEY (user_id))"))
|
||||
await conn.execute(sa.text("INSERT INTO thread_read_versions (user_id, version) VALUES ('owner', 3)"))
|
||||
|
||||
await asyncio.to_thread(command.upgrade, cfg, REVISION)
|
||||
shape = await _shape(engine)
|
||||
assert TABLES <= shape["tables"]
|
||||
for table, names in INDEXES.items():
|
||||
assert names <= shape["indexes"][table], table
|
||||
async with engine.connect() as conn:
|
||||
assert (await conn.execute(sa.text("SELECT version FROM thread_read_versions WHERE user_id='owner'"))).scalar_one() == 3
|
||||
assert (await conn.execute(sa.text("SELECT version_num FROM alembic_version"))).scalar_one() == REVISION
|
||||
|
||||
# A downgrade that already removed part of the schema still completes.
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(sa.text("DROP TABLE scheduled_task_events"))
|
||||
await asyncio.to_thread(command.downgrade, cfg, PREVIOUS)
|
||||
shape = await _shape(engine)
|
||||
assert TABLES.isdisjoint(shape["tables"])
|
||||
assert "origin_kind" not in shape["run_columns"]
|
||||
finally:
|
||||
if schema:
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(sa.text(f'DROP SCHEMA IF EXISTS "{schema}" CASCADE'))
|
||||
await engine.dispose()
|
||||
@ -72,3 +72,57 @@ async def test_one_occurrence_can_commit_unmet_and_pause_notices_together(delive
|
||||
await session.commit()
|
||||
claimed = await delivery_repo.claim_due_deliveries(now=datetime.now(UTC), limit=5)
|
||||
assert {row["event"] for row in claimed} == {"run_unmet", "task_paused"}
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_observer_notice_commits_and_rolls_back_with_complete_run(delivery_repo):
|
||||
"""The finalization observer stages the v2 notice inside complete_run's transaction."""
|
||||
from app.scheduler.service import ScheduledTaskService
|
||||
from deerflow.persistence.channel_connections import ChannelConnectionRepository
|
||||
from deerflow.persistence.channel_connections.model import ChannelConnectionRow
|
||||
from deerflow.persistence.scheduled_task_runs import ScheduledTaskRunRepository
|
||||
from deerflow.persistence.scheduled_tasks import ScheduledTaskRepository
|
||||
|
||||
sf = delivery_repo.session_factory
|
||||
tasks, runs = ScheduledTaskRepository(sf), ScheduledTaskRunRepository(sf)
|
||||
service = ScheduledTaskService(task_repo=tasks, task_run_repo=runs, launch_run=None, poll_interval_seconds=60, lease_seconds=30, max_concurrent_runs=3, connection_repo=ChannelConnectionRepository(sf), notification_repo=delivery_repo)
|
||||
now = datetime(2026, 10, 6, 8, tzinfo=UTC)
|
||||
await tasks.create(
|
||||
task_id="task-a",
|
||||
user_id="alice",
|
||||
thread_id=None,
|
||||
context_mode="fresh_thread_per_run",
|
||||
assistant_id=None,
|
||||
title="Digest",
|
||||
prompt="Summarize.",
|
||||
schedule_type="interval",
|
||||
schedule_spec={"every_seconds": 3600},
|
||||
timezone="UTC",
|
||||
next_run_at=now,
|
||||
)
|
||||
await runs.create(run_record_id="occurrence-a", task_id="task-a", thread_id="thread-a", scheduled_for=now, trigger="scheduled", status="running")
|
||||
await runs.update_status("occurrence-a", status="running", run_id="run-a", started_at=now)
|
||||
async with sf() as session:
|
||||
session.add(ChannelConnectionRow(id="binding", owner_user_id="alice", provider="wecom", status="connected", external_account_id="alice-external"))
|
||||
await session.commit()
|
||||
|
||||
async def fails_after_enqueue(session, task, occurrence, *, events):
|
||||
await service._on_finalization(session, task, occurrence, events=events)
|
||||
assert await session.scalar(select(func.count()).select_from(NotificationDeliveryRow)) == 1
|
||||
raise RuntimeError("commit interrupted")
|
||||
|
||||
def complete():
|
||||
return tasks.complete_run("task-a", user_id="alice", task_run_id="occurrence-a", run_id="run-a", status="success", error=None, finished_at=now)
|
||||
|
||||
tasks.set_finalization_observer(fails_after_enqueue)
|
||||
with pytest.raises(RuntimeError, match="commit interrupted"):
|
||||
await complete()
|
||||
# The notice rolled back with the outcome; the occurrence is still active.
|
||||
assert await delivery_repo.claim_due_deliveries(now=datetime.now(UTC), limit=5) == []
|
||||
assert (await runs.list_by_task("task-a"))[0]["status"] == "running"
|
||||
|
||||
tasks.set_finalization_observer(service._on_finalization)
|
||||
assert await complete() is True
|
||||
(claimed,) = await delivery_repo.claim_due_deliveries(now=datetime.now(UTC), limit=5)
|
||||
assert (claimed["event"], claimed["task_run_id"], claimed["run_id"]) == ("run_completed", "occurrence-a", "run-a")
|
||||
assert claimed["payload"] == {"payload_version": 2, "task_id": "task-a", "task_title": "Digest", "locale": None, "reason_code": None, "latest_reason_code": None, "run_status": "success"}
|
||||
|
||||
@ -15,7 +15,7 @@ import pytest
|
||||
from app.channels.base import ChannelUnavailable
|
||||
from app.channels.message_bus import MessageBus
|
||||
from app.channels.wecom import WeComChannel
|
||||
from app.scheduler.notification_delivery import NotificationDeliveryWorker, redact_egress_text, render_notification_text
|
||||
from app.scheduler.notification_delivery import NotificationDeliveryWorker, render_notification_text
|
||||
|
||||
|
||||
class FakeDeliveryRepo:
|
||||
@ -108,20 +108,25 @@ def _connection(*, provider="wecom", external_account_id="GaoZhiChao", status="c
|
||||
}
|
||||
|
||||
|
||||
def test_render_run_completed_text_includes_task_and_run():
|
||||
text = render_notification_text(_delivery_row())
|
||||
def test_render_run_completed_text_has_no_raw_ids():
|
||||
text = render_notification_text(_delivery_row(payload={"run_status": "success", "error": None, "task_id": "task-1", "task_title": "Daily digest"}))
|
||||
|
||||
assert "completed" in text.lower()
|
||||
assert "task-1" in text
|
||||
assert "run-1" in text
|
||||
assert "Finished a run." in text
|
||||
assert "“Daily digest”" in text
|
||||
assert "task-1" not in text
|
||||
assert "run-1" not in text
|
||||
|
||||
|
||||
def test_render_prefers_task_title_over_id():
|
||||
text = render_notification_text(_delivery_row(payload={"run_status": "success", "error": None, "task_id": "task-1", "task_title": "Daily digest"}))
|
||||
|
||||
assert "Daily digest" in text
|
||||
task_line = next(line for line in text.splitlines() if line.startswith("Task:"))
|
||||
assert "task-1" not in task_line
|
||||
assert "“Daily digest”" in text
|
||||
assert "task-1" not in text
|
||||
assert not any(line.startswith("Task:") for line in text.splitlines())
|
||||
# Without a title the notice says so instead of printing the id.
|
||||
untitled = render_notification_text(_delivery_row())
|
||||
assert "Untitled task" in untitled
|
||||
assert "task-1" not in untitled
|
||||
|
||||
|
||||
def test_render_run_failed_text_does_not_forward_raw_error():
|
||||
@ -133,9 +138,9 @@ def test_render_run_failed_text_does_not_forward_raw_error():
|
||||
)
|
||||
)
|
||||
|
||||
assert "failed" in text.lower()
|
||||
assert "A run failed." in text
|
||||
assert secret not in text
|
||||
assert "workspace" in text.lower()
|
||||
assert "Open DeerFlow → Scheduled tasks for details." in text
|
||||
assert "Error:" not in text
|
||||
|
||||
|
||||
@ -151,43 +156,6 @@ def test_render_truncates_unbounded_error_text():
|
||||
assert "Error:" not in text
|
||||
|
||||
|
||||
def test_redact_egress_text_scrubs_seeded_secret():
|
||||
secret = "sk-" + ("S" * 40)
|
||||
redacted = redact_egress_text(f"answer used {secret} here")
|
||||
|
||||
assert secret not in redacted
|
||||
assert "[redacted]" in redacted
|
||||
|
||||
|
||||
def test_redact_egress_text_scrubs_entire_pem_block():
|
||||
pem_body = "ABCDEFSECRETKEYBODY"
|
||||
pem = "-----BEGIN PRIVATE KEY-----\n" + pem_body + "\n-----END PRIVATE KEY-----"
|
||||
redacted = redact_egress_text(f"key follows\n{pem}\nend")
|
||||
|
||||
assert pem_body not in redacted
|
||||
assert "BEGIN PRIVATE KEY" not in redacted
|
||||
assert "END PRIVATE KEY" not in redacted
|
||||
assert "[redacted]" in redacted
|
||||
|
||||
|
||||
def test_render_redacts_entire_pem_block_in_result_summary():
|
||||
pem_body = "ABCDEFSECRETKEYBODY"
|
||||
pem = "-----BEGIN PRIVATE KEY-----\n" + pem_body + "\n-----END PRIVATE KEY-----"
|
||||
text = render_notification_text(_delivery_row(payload={"run_status": "success", "error": None, "task_id": "task-1", "result_summary": pem}))
|
||||
|
||||
assert pem_body not in text
|
||||
assert "END PRIVATE KEY" not in text
|
||||
assert "[redacted]" in text
|
||||
|
||||
|
||||
def test_render_redacts_secret_in_result_summary():
|
||||
secret = "ghp_" + ("a" * 36)
|
||||
text = render_notification_text(_delivery_row(payload={"run_status": "success", "error": None, "task_id": "task-1", "result_summary": f"token={secret}"}))
|
||||
|
||||
assert secret not in text
|
||||
assert "[redacted]" in text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_once_delivers_claimed_row_and_marks_sent():
|
||||
repo = FakeDeliveryRepo([_delivery_row()])
|
||||
@ -201,7 +169,8 @@ async def test_run_once_delivers_claimed_row_and_marks_sent():
|
||||
assert len(channel.sent) == 1
|
||||
target, text = channel.sent[0]
|
||||
assert target == "GaoZhiChao"
|
||||
assert "task-1" in text
|
||||
assert text.splitlines()[0] == "Scheduled task “Untitled task”"
|
||||
assert "task-1" not in text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@ -500,17 +469,19 @@ async def test_start_stop_lifecycle():
|
||||
|
||||
|
||||
def test_render_includes_result_summary_for_completed_runs():
|
||||
text = render_notification_text(_delivery_row(payload={"run_status": "success", "error": None, "task_id": "task-1", "result_summary": "The answer is 42"}))
|
||||
text = render_notification_text(_delivery_row(payload={"run_status": "success", "error": None, "task_id": "task-1", "result_summary": "The answer is 42\nand a second line"}))
|
||||
|
||||
assert "The answer is 42" in text
|
||||
# The tasks page's run summary: the reply's first line only.
|
||||
assert "Result: The answer is 42" in text.splitlines()
|
||||
assert "second line" not in text
|
||||
|
||||
|
||||
def test_render_truncates_long_result_summary():
|
||||
text = render_notification_text(_delivery_row(payload={"run_status": "success", "error": None, "task_id": "task-1", "result_summary": "x" * 5000}))
|
||||
|
||||
# Bounded on the wire: 999 chars of summary + the ellipsis marker.
|
||||
# Bounded like the tasks page: at most 160 chars, ending in an ellipsis.
|
||||
result_line = next(line for line in text.splitlines() if line.startswith("Result: "))
|
||||
assert len(result_line) == len("Result: ") + 1000
|
||||
assert len(result_line) == len("Result: ") + 160
|
||||
assert result_line.endswith("\u2026")
|
||||
|
||||
|
||||
@ -596,7 +567,8 @@ async def test_delivery_falls_back_to_skeleton_when_resolver_raises():
|
||||
# A summary lookup failure must never block the notification itself.
|
||||
assert repo.sent == ["delivery-1"]
|
||||
assert repo.failed == []
|
||||
assert "task-1" in channel.sent[0][1]
|
||||
assert "Finished a run." in channel.sent[0][1]
|
||||
assert "Result:" not in channel.sent[0][1]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@ -609,10 +581,103 @@ async def test_delivery_sends_skeleton_when_summary_is_missing():
|
||||
|
||||
assert repo.sent == ["delivery-1"]
|
||||
text = channel.sent[0][1]
|
||||
assert "task-1" in text
|
||||
assert "Finished a run." in text
|
||||
assert "Result:" not in text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_legacy_row_for_unsupported_provider_fails_terminally_without_send():
|
||||
# Rows queued before only push-capable providers got rows: they can never
|
||||
# be sent, so they end at once instead of using up five retries.
|
||||
repo = FakeDeliveryRepo([_delivery_row(provider="feishu", target="ou_123")])
|
||||
channel = FakeChannel()
|
||||
resolved_channels, checked_connections = [], []
|
||||
|
||||
def resolve_channel(provider):
|
||||
resolved_channels.append(provider)
|
||||
return channel
|
||||
|
||||
def resolve_connections(owner):
|
||||
checked_connections.append(owner)
|
||||
return [_connection(provider="feishu", external_account_id="ou_123")]
|
||||
|
||||
worker = NotificationDeliveryWorker(delivery_repo=repo, resolve_channel=resolve_channel, resolve_connections=resolve_connections)
|
||||
|
||||
await worker.run_once(now=datetime.now(UTC))
|
||||
|
||||
assert channel.sent == []
|
||||
assert resolved_channels == [] and checked_connections == []
|
||||
assert repo.sent == []
|
||||
assert repo.terminal_failures == [("delivery-1", "provider does not support proactive notifications")]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_task_stopped_is_enriched_with_summary():
|
||||
row = _delivery_row(event="task_stopped", payload={"payload_version": 2, "task_title": "Release check", "reason_code": "agent_stop", "run_status": "success", "stop_condition": "all items are ticked"})
|
||||
repo = FakeDeliveryRepo([row])
|
||||
channel = FakeChannel()
|
||||
seen = []
|
||||
|
||||
def resolver(run_id, user_id):
|
||||
seen.append((run_id, user_id))
|
||||
return "## All 12 items are ticked\nmore"
|
||||
|
||||
await _make_worker(repo, channel, resolve_run_summary=resolver).run_once(now=datetime.now(UTC))
|
||||
|
||||
assert seen == [("run-1", "user-1")]
|
||||
assert channel.sent[0][1].splitlines() == [
|
||||
"Scheduled task “Release check”",
|
||||
"Paused by agent: its stop condition was met (all items are ticked).",
|
||||
"Result: All 12 items are ticked",
|
||||
"Open DeerFlow → Scheduled tasks for details.",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("event", "payload"),
|
||||
[
|
||||
("run_failed", {"payload_version": 2, "run_status": "failed"}),
|
||||
("task_finished", {"payload_version": 2, "reason_code": "max_runs", "run_status": "failed", "max_runs": 3}),
|
||||
("task_stopped", {"payload_version": 2, "reason_code": "agent_stop", "run_status": "interrupted"}),
|
||||
],
|
||||
)
|
||||
async def test_failed_run_is_not_enriched(event, payload):
|
||||
repo = FakeDeliveryRepo([_delivery_row(event=event, payload=payload)])
|
||||
channel = FakeChannel()
|
||||
called = []
|
||||
|
||||
def resolver(run_id, user_id):
|
||||
called.append(run_id)
|
||||
return "partial answer"
|
||||
|
||||
await _make_worker(repo, channel, resolve_run_summary=resolver).run_once(now=datetime.now(UTC))
|
||||
|
||||
assert called == []
|
||||
assert repo.sent == ["delivery-1"]
|
||||
assert "partial answer" not in channel.sent[0][1]
|
||||
assert "Result:" not in channel.sent[0][1]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_worker_uses_default_locale_for_rows_without_locale():
|
||||
legacy = _delivery_row(delivery_id="legacy", event="run_failed", payload={"run_status": "failed", "error": "boom", "task_id": "task-1"})
|
||||
no_preference = _delivery_row(delivery_id="no-preference", event="run_failed", payload={"payload_version": 2, "locale": None, "run_status": "failed"})
|
||||
english_owner = _delivery_row(delivery_id="english-owner", event="run_failed", payload={"payload_version": 2, "locale": "en-US", "run_status": "failed"})
|
||||
repo = FakeDeliveryRepo([legacy, no_preference, english_owner])
|
||||
channel = FakeChannel()
|
||||
worker = NotificationDeliveryWorker(delivery_repo=repo, resolve_channel=lambda provider: channel, default_locale="zh-CN")
|
||||
|
||||
await worker.run_once(now=datetime.now(UTC))
|
||||
|
||||
assert repo.sent == ["legacy", "no-preference", "english-owner"]
|
||||
texts = [text for _target, text in channel.sent]
|
||||
assert texts[0].splitlines()[1:] == ["一次运行出错了。", "在 DeerFlow 的定时任务页查看详情。"]
|
||||
assert texts[1] == texts[0].replace("Untitled task", "未命名任务")
|
||||
# The owner's own UI language wins over the configured default.
|
||||
assert texts[2].splitlines()[1] == "A run failed."
|
||||
|
||||
|
||||
class FakeWsClient:
|
||||
def __init__(self, *, errcode=0):
|
||||
self.calls = []
|
||||
|
||||
292
backend/tests/test_notification_text.py
Normal file
292
backend/tests/test_notification_text.py
Normal file
@ -0,0 +1,292 @@
|
||||
"""Scheduled-task IM notice text: localized, self-contained, no IDs, no links."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
import pytest
|
||||
from _readable_helpers import assert_no_raw_identifiers
|
||||
|
||||
from app.scheduler.notification_text import _GOAL_REASON_TEXT, _TEXT, NOTIFICATION_LOCALES, redact_egress_text, render_notification_text, wants_result_summary
|
||||
from deerflow.persistence.scheduled_task_runs.sql import run_summary
|
||||
|
||||
TASK_ID = "task-0f3c9a7d5b2e4c1a"
|
||||
OCCURRENCE_ID = "task-run-6a1d0c4e9b7f2a35"
|
||||
RUN_ID = "3f2b8c1e-6d4a-4b9e-9a7c-0e5d1f2a3b4c"
|
||||
CLOSING = {"en-US": "Open DeerFlow → Scheduled tasks for details.", "zh-CN": "在 DeerFlow 的定时任务页查看详情。"}
|
||||
_CJK = re.compile(r"[一-鿿]")
|
||||
# Stems of every English sentence; none may leak into the Chinese text.
|
||||
_EN_STEMS = ("Scheduled task", "Finished", "A run failed", "Ran, but", "Paused", "Result:", "Open DeerFlow", "The last run", "Goal met", "Untitled")
|
||||
|
||||
|
||||
def notice(event, *, locale="en-US", run_id=RUN_ID, **payload):
|
||||
base = {"payload_version": 2, "task_id": TASK_ID, "task_title": "Check the release checklist", "locale": locale}
|
||||
return {"id": "delivery-1", "task_id": TASK_ID, "task_run_id": OCCURRENCE_ID, "run_id": run_id, "event": event, "provider": "wecom", "target": "someone", "payload": {**base, **payload}}
|
||||
|
||||
|
||||
# Every event the scheduler enqueues, with the payloads it writes.
|
||||
CASES = {
|
||||
"run_completed": dict(reason_code=None, run_status="success", result_summary="## All 12 items are done\nmore detail"),
|
||||
"run_completed_goal": dict(reason_code=None, run_status="success", relied_on_assumption=False, result_summary="Report is ready"),
|
||||
"run_completed_assumption": dict(reason_code=None, run_status="success", relied_on_assumption=True, result_summary="Report is ready"),
|
||||
"run_failed": dict(reason_code=None, run_status="failed", result_summary="partial answer"),
|
||||
"run_unmet": dict(reason_code="blocked:needs_user_input", latest_reason_code="blocked:needs_user_input", run_status="unmet", result_summary="I need the budget file"),
|
||||
"task_paused": dict(reason_code="consecutive_unmet", latest_reason_code="goal_not_met_yet", run_status="unmet", result_summary="Still 3 open items"),
|
||||
"task_stopped": dict(reason_code="agent_stop", run_status="success", stop_condition="all items are ticked", result_summary="All items are ticked"),
|
||||
"task_stopped_no_condition": dict(reason_code="agent_stop", run_status="success"),
|
||||
"task_stopped_failed": dict(reason_code="agent_stop", run_status="failed", stop_condition="all items are ticked"),
|
||||
"task_stopped_interrupted": dict(reason_code="agent_stop", run_status="interrupted"),
|
||||
"task_finished_max_runs": dict(reason_code="max_runs", run_status="success", max_runs=5, result_summary="Done"),
|
||||
"task_finished_one_run": dict(reason_code="max_runs", run_status="success", max_runs=1),
|
||||
"task_finished_end_at": dict(reason_code="end_at", run_status=None),
|
||||
"task_finished_unmet": dict(reason_code="max_runs", run_status="unmet", latest_reason_code="missing_evidence", max_runs=3),
|
||||
"task_finished_failed": dict(reason_code="max_runs", run_status="failed", max_runs=3),
|
||||
}
|
||||
|
||||
|
||||
def _event_of(case):
|
||||
return next(event for event in ("run_completed", "run_failed", "run_unmet", "task_paused", "task_stopped", "task_finished") if case.startswith(event))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("locale", NOTIFICATION_LOCALES)
|
||||
@pytest.mark.parametrize("case", sorted(CASES))
|
||||
def test_every_notice_is_short_readable_and_self_contained(case, locale):
|
||||
text = render_notification_text(notice(_event_of(case), locale=locale, **CASES[case]))
|
||||
lines = text.splitlines()
|
||||
assert 3 <= len(lines) <= 4
|
||||
assert_no_raw_identifiers(text)
|
||||
assert TASK_ID not in text and OCCURRENCE_ID not in text and RUN_ID not in text
|
||||
assert not any(line.startswith(("Run:", "Task:", "Reason:")) for line in lines)
|
||||
assert "http" not in text and "`" not in text
|
||||
assert "Check the release checklist" in lines[0]
|
||||
assert lines[-1] == CLOSING[locale]
|
||||
if locale == "zh-CN":
|
||||
assert _CJK.search(text)
|
||||
assert not any(stem in text for stem in _EN_STEMS), text
|
||||
|
||||
|
||||
def test_english_templates():
|
||||
def line2(case):
|
||||
return render_notification_text(notice(_event_of(case), **CASES[case])).splitlines()[1]
|
||||
|
||||
assert line2("run_completed") == "Finished a run."
|
||||
assert line2("run_completed_goal") == "Finished a run. Goal met."
|
||||
assert line2("run_completed_assumption") == "Finished a run. Goal met, with an assumption."
|
||||
assert line2("run_failed") == "A run failed."
|
||||
assert line2("run_unmet") == "Ran, but the goal wasn't met: it needs your input."
|
||||
# goal_not_met_yet only restates the miss: the reason-less sentence.
|
||||
assert line2("task_paused") == "Paused after 3 runs in a row missed the goal."
|
||||
assert line2("task_stopped") == "Paused by agent: its stop condition was met (all items are ticked)."
|
||||
assert line2("task_stopped_no_condition") == "Paused by agent: its stop condition was met."
|
||||
assert line2("task_stopped_failed") == "Paused by agent: its stop condition was met (all items are ticked). The last run failed."
|
||||
assert line2("task_stopped_interrupted") == "Paused by agent: its stop condition was met. The last run was interrupted."
|
||||
assert line2("task_finished_max_runs") == "Finished: all 5 automatic runs are done."
|
||||
assert line2("task_finished_one_run") == "Finished: its one automatic run is done."
|
||||
assert line2("task_finished_end_at") == "Finished: its end time has been reached."
|
||||
assert line2("task_finished_unmet") == "Finished: all 3 automatic runs are done. The last run didn't meet the goal: the goal check found evidence missing."
|
||||
assert line2("task_finished_failed") == "Finished: all 3 automatic runs are done. The last run failed."
|
||||
|
||||
|
||||
def test_chinese_templates():
|
||||
def line2(case):
|
||||
return render_notification_text(notice(_event_of(case), locale="zh-CN", **CASES[case])).splitlines()[1]
|
||||
|
||||
assert line2("run_completed") == "完成了一次运行。"
|
||||
assert line2("run_completed_assumption") == "完成了一次运行。目标已达成(含假设)。"
|
||||
assert line2("run_failed") == "一次运行出错了。"
|
||||
assert line2("run_unmet") == "已运行,但目标未达成:需要你提供信息。"
|
||||
assert line2("task_paused") == "连续 3 次未达成目标,已自动暂停。"
|
||||
assert line2("task_stopped") == "已由智能体暂停:停止条件已满足(all items are ticked)。"
|
||||
assert line2("task_stopped_failed") == "已由智能体暂停:停止条件已满足(all items are ticked)。最后一次运行出错了。"
|
||||
assert line2("task_finished_max_runs") == "已结束:5 次自动运行已全部完成。"
|
||||
assert line2("task_finished_one_run") == "已结束:唯一一次自动运行已完成。"
|
||||
assert line2("task_finished_end_at") == "已结束:已到结束时间。"
|
||||
assert line2("task_finished_unmet") == "已结束:3 次自动运行已全部完成。最后一次运行未达成目标:目标检查发现缺少依据。"
|
||||
text = render_notification_text(notice("task_stopped", locale="zh-CN", **CASES["task_stopped"]))
|
||||
assert text.splitlines()[0] == "定时任务“Check the release checklist”"
|
||||
|
||||
|
||||
def test_documented_example():
|
||||
payload = {**CASES["task_stopped"], "result_summary": "All 12 checklist items are done; the release notes are published."}
|
||||
text = render_notification_text(notice("task_stopped", **payload))
|
||||
assert text == (
|
||||
"Scheduled task “Check the release checklist”\n"
|
||||
"Paused by agent: its stop condition was met (all items are ticked).\n"
|
||||
"Result: All 12 checklist items are done; the release notes are published.\n"
|
||||
"Open DeerFlow → Scheduled tasks for details."
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("locale", "fallback"), [("en-US", "Untitled task"), ("zh-CN", "未命名任务")])
|
||||
@pytest.mark.parametrize("title", [None, "", " ", 7])
|
||||
def test_missing_title_reads_untitled_never_the_id(title, locale, fallback):
|
||||
text = render_notification_text(notice("run_completed", locale=locale, task_title=title, run_status="success"))
|
||||
assert fallback in text.splitlines()[0]
|
||||
assert TASK_ID not in text
|
||||
|
||||
|
||||
def test_multiline_title_stays_on_the_title_line():
|
||||
text = render_notification_text(notice("run_failed", task_title="Weekly\nreport", run_status="failed"))
|
||||
assert text.splitlines()[0] == "Scheduled task “Weekly report”"
|
||||
|
||||
|
||||
def test_notification_locales_follow_the_config_type_and_have_templates():
|
||||
from typing import get_args
|
||||
|
||||
from app.gateway.routers.user_preferences import Preferences
|
||||
from deerflow.config.channel_connections_config import NotificationLocale
|
||||
|
||||
assert set(NOTIFICATION_LOCALES) == set(get_args(NotificationLocale)) == set(_TEXT) == set(_GOAL_REASON_TEXT)
|
||||
# /preferences accepts exactly the notice languages.
|
||||
for locale in NOTIFICATION_LOCALES:
|
||||
assert Preferences(locale=locale).locale == locale
|
||||
|
||||
|
||||
@pytest.mark.parametrize("locale", NOTIFICATION_LOCALES)
|
||||
@pytest.mark.parametrize("code", ["goal_not_met_yet", "unknown", None])
|
||||
@pytest.mark.parametrize(
|
||||
("event", "reason_code"),
|
||||
[("run_unmet", "same"), ("task_paused", "consecutive_unmet"), ("task_finished", "max_runs"), ("task_stopped", "agent_stop")],
|
||||
)
|
||||
def test_generic_reason_never_repeats_the_sentence(locale, code, event, reason_code):
|
||||
"""'Ran, but the goal wasn't met: the goal is not met yet.' says one thing twice."""
|
||||
payload = {"run_status": "unmet", "reason_code": code if reason_code == "same" else reason_code, "latest_reason_code": code, "max_runs": 3}
|
||||
line = render_notification_text(notice(event, locale=locale, **payload)).splitlines()[1]
|
||||
for text in (_GOAL_REASON_TEXT[locale]["goal_not_met_yet"], _GOAL_REASON_TEXT[locale]["unknown"]):
|
||||
assert text not in line, line
|
||||
assert ":。" not in line and ": ." not in line and "原因" not in line
|
||||
|
||||
|
||||
def test_every_reason_code_exists_in_both_locales():
|
||||
assert set(_GOAL_REASON_TEXT) == set(NOTIFICATION_LOCALES)
|
||||
assert set(_GOAL_REASON_TEXT["en-US"]) == set(_GOAL_REASON_TEXT["zh-CN"])
|
||||
assert _GOAL_REASON_TEXT["en-US"]["unknown"] == "no reason was recorded"
|
||||
for code, text in _GOAL_REASON_TEXT["zh-CN"].items():
|
||||
assert _CJK.search(text), code
|
||||
|
||||
|
||||
@pytest.mark.parametrize("locale", NOTIFICATION_LOCALES)
|
||||
def test_unknown_reason_is_never_forwarded(locale):
|
||||
text = render_notification_text(notice("run_unmet", locale=locale, reason_code="provider said: secret", run_status="unmet"))
|
||||
assert "secret" not in text
|
||||
# An unrecognized code says nothing more than "the goal wasn't met".
|
||||
assert text.splitlines()[1] == {"en-US": "Ran, but the goal wasn't met.", "zh-CN": "已运行,但目标未达成。"}[locale]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", ["success", "unmet"])
|
||||
def test_summary_line_is_the_tasks_page_run_summary(status):
|
||||
reply = "**Done:** all [12 items](http://example.invalid/list) are ticked\n\nDetails follow on many lines." + "x" * 400
|
||||
event = "run_completed" if status == "success" else "run_unmet"
|
||||
delivery = notice(event, run_status=status, reason_code="goal_not_met_yet", result_summary=reply)
|
||||
assert wants_result_summary(delivery)
|
||||
lines = render_notification_text(delivery).splitlines()
|
||||
assert lines[2] == f"Result: {run_summary(redact_egress_text(reply))}"
|
||||
assert lines[2] == "Result: Done: all 12 items are ticked"
|
||||
long_reply = "y" * 500
|
||||
long_line = render_notification_text(notice(event, run_status=status, result_summary=long_reply)).splitlines()[2]
|
||||
assert len(long_line.removeprefix("Result: ")) <= 160
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("event", "payload", "run_id"),
|
||||
[
|
||||
("run_failed", {"run_status": "failed"}, RUN_ID),
|
||||
("task_finished", {"reason_code": "max_runs", "run_status": "failed", "max_runs": 3}, RUN_ID),
|
||||
("task_stopped", {"reason_code": "agent_stop", "run_status": "interrupted"}, RUN_ID),
|
||||
("task_finished", {"reason_code": "end_at", "run_status": None}, None),
|
||||
("task_finished", {"reason_code": "max_runs", "run_status": "skipped", "max_runs": 3}, None),
|
||||
],
|
||||
)
|
||||
def test_no_summary_without_a_reply_of_a_finished_run(event, payload, run_id):
|
||||
delivery = notice(event, run_id=run_id, **payload, result_summary="partial answer")
|
||||
assert not wants_result_summary(delivery)
|
||||
text = render_notification_text(delivery)
|
||||
assert "partial answer" not in text
|
||||
assert "Result:" not in text
|
||||
|
||||
|
||||
def test_no_summary_line_when_no_reply_resolves():
|
||||
text = render_notification_text(notice("run_completed", run_status="success"))
|
||||
assert "Result:" not in text
|
||||
assert len(text.splitlines()) == 3
|
||||
assert "Result:" not in render_notification_text(notice("run_completed", run_status="success", result_summary=" \n "))
|
||||
|
||||
|
||||
def test_redact_egress_text_scrubs_seeded_secret():
|
||||
secret = "sk-" + ("S" * 40)
|
||||
redacted = redact_egress_text(f"answer used {secret} here")
|
||||
|
||||
assert secret not in redacted
|
||||
assert "[redacted]" in redacted
|
||||
|
||||
|
||||
def test_redact_egress_text_scrubs_entire_pem_block():
|
||||
pem_body = "ABCDEFSECRETKEYBODY"
|
||||
pem = "-----BEGIN PRIVATE KEY-----\n" + pem_body + "\n-----END PRIVATE KEY-----"
|
||||
redacted = redact_egress_text(f"key follows\n{pem}\nend")
|
||||
|
||||
assert pem_body not in redacted
|
||||
assert "BEGIN PRIVATE KEY" not in redacted
|
||||
assert "END PRIVATE KEY" not in redacted
|
||||
assert "[redacted]" in redacted
|
||||
|
||||
|
||||
def test_multiline_pem_block_never_reaches_the_text():
|
||||
pem_body = "ABCDEFSECRETKEYBODY"
|
||||
pem = "-----BEGIN PRIVATE KEY-----\n" + pem_body + "\n-----END PRIVATE KEY-----"
|
||||
text = render_notification_text(notice("run_completed", run_status="success", result_summary=f"{pem}\nThe key above was rotated"))
|
||||
assert pem_body not in text
|
||||
assert "PRIVATE KEY" not in text
|
||||
assert "Result: [redacted]" in text
|
||||
|
||||
|
||||
def test_secret_on_the_first_line_is_redacted():
|
||||
secret = "ghp_" + ("a" * 36)
|
||||
text = render_notification_text(notice("run_completed", run_status="success", result_summary=f"token={secret}\nsecond line"))
|
||||
assert secret not in text
|
||||
assert "Result: token=[redacted]" in text
|
||||
|
||||
|
||||
def test_stop_condition_is_redacted_and_truncated():
|
||||
secret = "sk-" + ("S" * 40)
|
||||
condition = f"stop when {secret} rotates " + "z" * 400
|
||||
line = render_notification_text(notice("task_stopped", reason_code="agent_stop", run_status="success", stop_condition=condition)).splitlines()[1]
|
||||
assert secret not in line
|
||||
assert "[redacted]" in line
|
||||
shown = line.removeprefix("Paused by agent: its stop condition was met (").removesuffix(").")
|
||||
assert len(shown) <= 200
|
||||
assert shown.endswith("…")
|
||||
|
||||
|
||||
def test_failed_run_never_forwards_raw_error_text():
|
||||
secret = "sk-" + ("x" * 40)
|
||||
text = render_notification_text(notice("run_failed", run_status="failed", error=secret + "x" * 5000))
|
||||
assert secret not in text
|
||||
assert len(text) < 300
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("delivery", "expected"),
|
||||
[
|
||||
# Rows written before payload version 2.
|
||||
({"event": "run_completed", "task_id": TASK_ID, "run_id": RUN_ID, "payload": {"run_status": "success", "error": None, "task_id": TASK_ID}}, "Finished a run."),
|
||||
({"event": "run_failed", "task_id": TASK_ID, "run_id": RUN_ID, "payload": {"run_status": "failed", "error": "boom", "task_id": TASK_ID, "task_title": "Digest"}}, "A run failed."),
|
||||
({"event": "run_unmet", "task_id": TASK_ID, "run_id": RUN_ID, "payload": {"task_id": TASK_ID, "reason_code": "goal_not_met_yet"}}, "Ran, but the goal wasn't met."),
|
||||
({"event": "task_paused", "task_id": TASK_ID, "run_id": RUN_ID, "payload": {"task_id": TASK_ID, "reason_code": "consecutive_unmet"}}, "Paused after 3 runs in a row missed the goal."),
|
||||
],
|
||||
)
|
||||
def test_legacy_rows_render_with_the_new_templates(delivery, expected):
|
||||
text = render_notification_text(delivery)
|
||||
assert text.splitlines()[1] == expected
|
||||
assert TASK_ID not in text and RUN_ID not in text
|
||||
assert_no_raw_identifiers(text)
|
||||
|
||||
|
||||
def test_locale_falls_back_to_the_configured_default():
|
||||
delivery = notice("run_failed", locale=None, run_status="failed")
|
||||
assert render_notification_text(delivery).splitlines()[-1] == CLOSING["en-US"]
|
||||
assert render_notification_text(delivery, default_locale="zh-CN").splitlines()[-1] == CLOSING["zh-CN"]
|
||||
# A row's own valid locale wins over the default; an invalid one does not.
|
||||
assert render_notification_text(notice("run_failed", locale="zh-CN", run_status="failed"), default_locale="en-US").splitlines()[-1] == CLOSING["zh-CN"]
|
||||
assert render_notification_text(notice("run_failed", locale="fr-FR", run_status="failed"), default_locale="zh-CN").splitlines()[-1] == CLOSING["zh-CN"]
|
||||
assert render_notification_text(delivery, default_locale="fr-FR").splitlines()[-1] == CLOSING["en-US"]
|
||||
439
backend/tests/test_run_origin.py
Normal file
439
backend/tests/test_run_origin.py
Normal file
@ -0,0 +1,439 @@
|
||||
"""Server-owned ``deerflow_origin`` on runs and threads.
|
||||
|
||||
Only server-side launchers (scheduler, MCP notifications, extension handles,
|
||||
via ``request.state.run_origin``) and the internal channel caller (via
|
||||
``body.metadata``) may stamp it; every client copy is dropped from both
|
||||
metadata forks of run admission (the run record and the live config, which is
|
||||
what reaches checkpoint metadata). ``RunRepository`` derives
|
||||
``runs.origin_kind`` from it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from fastapi import FastAPI, Request
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
|
||||
from app.gateway.auth.models import User
|
||||
from app.gateway.authz import AuthContext
|
||||
from app.gateway.extension_agent_runs import GatewayAgentRunsHost
|
||||
from app.gateway.internal_auth import INTERNAL_OWNER_USER_ID_HEADER_NAME, get_internal_user
|
||||
from app.gateway.run_models import RunCreateRequest
|
||||
from app.gateway.run_origin import resolve_request_origin
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.run import RunRepository
|
||||
from deerflow.persistence.run.model import RunChangeClockRow, RunRow
|
||||
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY, ORIGIN_KINDS, admit_origin, make_origin, origin_kind_of
|
||||
|
||||
CONTRACT = json.loads((Path(__file__).resolve().parents[2] / "contracts" / "thread_origin_contract.json").read_text())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The origin value itself
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_contract_matches_the_module():
|
||||
assert CONTRACT["version"] == 1
|
||||
assert CONTRACT["key"] == DEERFLOW_ORIGIN_KEY
|
||||
assert sorted(CONTRACT["kinds"]) == CONTRACT["kinds"]
|
||||
assert set(CONTRACT["kinds"]) == ORIGIN_KINDS
|
||||
|
||||
|
||||
def test_origin_key_is_distinct_from_the_message_level_scheduled_origin_key():
|
||||
from app.gateway.services import SCHEDULED_ORIGIN_KEY
|
||||
|
||||
notes = json.loads((Path(__file__).resolve().parents[2] / "contracts" / "scheduled_goal_notes_contract.json").read_text())
|
||||
assert DEERFLOW_ORIGIN_KEY != SCHEDULED_ORIGIN_KEY == notes["scheduled_origin_key"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"value",
|
||||
[
|
||||
{"kind": "schedule"},
|
||||
{"kind": "im_channel", "provider": "feishu"},
|
||||
{"kind": "github", "provider": "github"},
|
||||
{"kind": "extension", "namespace": "community.agent-teams"},
|
||||
{"kind": "mcp_notification"},
|
||||
],
|
||||
)
|
||||
def test_well_formed_origins_are_admitted(value):
|
||||
assert admit_origin(value) == value
|
||||
assert origin_kind_of({DEERFLOW_ORIGIN_KEY: value}) == value["kind"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"value",
|
||||
[
|
||||
None,
|
||||
"schedule",
|
||||
{},
|
||||
{"kind": "browser"},
|
||||
{"kind": "schedule", "extra": "x"},
|
||||
{"kind": "im_channel", "provider": "Feishu"},
|
||||
{"kind": "im_channel", "provider": "x" * 33},
|
||||
{"kind": "im_channel", "provider": 1},
|
||||
{"kind": "extension", "namespace": "a:b"},
|
||||
{"kind": "extension", "namespace": ""},
|
||||
{"kind": "extension", "namespace": "n" * 97},
|
||||
{"kind": True},
|
||||
],
|
||||
)
|
||||
def test_malformed_origins_are_dropped_whole(value):
|
||||
assert admit_origin(value) is None
|
||||
assert origin_kind_of({DEERFLOW_ORIGIN_KEY: value}) is None
|
||||
|
||||
|
||||
def test_make_origin_rejects_malformed_values():
|
||||
assert make_origin("im_channel", provider="slack") == {"kind": "im_channel", "provider": "slack"}
|
||||
with pytest.raises(ValueError):
|
||||
make_origin("browser")
|
||||
with pytest.raises(ValueError):
|
||||
make_origin("extension", namespace="bad:ns")
|
||||
|
||||
|
||||
def test_origin_kind_of_tolerates_missing_metadata():
|
||||
assert origin_kind_of(None) is None
|
||||
assert origin_kind_of({}) is None
|
||||
assert origin_kind_of([("deerflow_origin", {"kind": "schedule"})]) is None
|
||||
|
||||
|
||||
def test_request_origin_trusts_only_host_state_and_internal_callers():
|
||||
origin = {"kind": "im_channel", "provider": "slack"}
|
||||
body = {DEERFLOW_ORIGIN_KEY: origin}
|
||||
# Session/PAT/API callers: the body copy is ignored.
|
||||
assert resolve_request_origin(SimpleNamespace(state=SimpleNamespace(auth_source="session")), body) is None
|
||||
assert resolve_request_origin(SimpleNamespace(state=SimpleNamespace(auth_source="pat")), body) is None
|
||||
assert resolve_request_origin(SimpleNamespace(), body) is None
|
||||
# The internal channel caller may declare it; invalid shapes still drop.
|
||||
internal = SimpleNamespace(state=SimpleNamespace(auth_source="internal"))
|
||||
assert resolve_request_origin(internal, body) == origin
|
||||
assert resolve_request_origin(internal, {DEERFLOW_ORIGIN_KEY: {"kind": "schedule", "x": 1}}) is None
|
||||
assert resolve_request_origin(internal, None) is None
|
||||
# A host-set origin wins over anything in the body.
|
||||
host = SimpleNamespace(state=SimpleNamespace(auth_source="internal", run_origin={"kind": "schedule"}))
|
||||
assert resolve_request_origin(host, body) == {"kind": "schedule"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Run admission through a real Gateway composition (SQL run store, real graph)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_USER = User(email="origin@example.com")
|
||||
OWNER = str(_USER.id)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def gateway(monkeypatch, tmp_path):
|
||||
from langchain_core.messages import AIMessage
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
from langgraph.graph import MessagesState, StateGraph
|
||||
from langgraph.store.memory import InMemoryStore
|
||||
|
||||
from deerflow.config.app_config import AppConfig, reset_app_config, set_app_config
|
||||
from deerflow.persistence.thread_meta.memory import MemoryThreadMetaStore
|
||||
from deerflow.runtime import RunManager
|
||||
from deerflow.runtime.events.store.memory import MemoryRunEventStore
|
||||
from deerflow.runtime.stream_bridge.memory import MemoryStreamBridge
|
||||
|
||||
engine = create_async_engine(f"sqlite+aiosqlite:///{tmp_path / 'runs.db'}")
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(lambda sync: Base.metadata.create_all(sync, tables=[RunRow.__table__, RunChangeClockRow.__table__]))
|
||||
sf = async_sessionmaker(engine, expire_on_commit=False)
|
||||
|
||||
app = FastAPI()
|
||||
state = app.state
|
||||
state.checkpointer = InMemorySaver()
|
||||
state.store = InMemoryStore()
|
||||
state.thread_store = MemoryThreadMetaStore(state.store)
|
||||
state.run_store = RunRepository(sf)
|
||||
state.run_manager = RunManager(store=state.run_store)
|
||||
state.run_event_store = MemoryRunEventStore()
|
||||
state.stream_bridge = MemoryStreamBridge()
|
||||
monkeypatch.setenv("DEER_FLOW_HOME", str(tmp_path))
|
||||
set_app_config(AppConfig.model_validate({"sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"}}))
|
||||
|
||||
def factory(*, config):
|
||||
builder = StateGraph(MessagesState)
|
||||
builder.add_node("answer", lambda _state: {"messages": [AIMessage(content="done")]})
|
||||
builder.set_entry_point("answer")
|
||||
builder.set_finish_point("answer")
|
||||
return builder.compile()
|
||||
|
||||
monkeypatch.setattr("app.gateway.services.resolve_agent_factory", lambda _: factory)
|
||||
monkeypatch.setattr("app.gateway.services.get_local_provider", lambda: SimpleNamespace(get_user=AsyncMock(return_value=_USER)))
|
||||
host = GatewayAgentRunsHost(app, load_user=AsyncMock(return_value=_USER))
|
||||
state.agent_runs_host = host
|
||||
try:
|
||||
yield app, sf
|
||||
finally:
|
||||
host.close()
|
||||
await state.run_manager.shutdown()
|
||||
reset_app_config()
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
def _session_request(app):
|
||||
request = Request({"type": "http", "app": app, "headers": [], "state": {}})
|
||||
request.state.user = _USER
|
||||
request.state.auth_source = "session"
|
||||
request.state.auth = AuthContext(_USER, ["threads:write", "threads:read", "runs:create", "runs:read", "runs:cancel"])
|
||||
return request
|
||||
|
||||
|
||||
def _internal_request(app, owner=OWNER):
|
||||
return SimpleNamespace(app=app, headers={INTERNAL_OWNER_USER_ID_HEADER_NAME: owner}, state=SimpleNamespace(user=get_internal_user(), auth_source="internal"), cookies={})
|
||||
|
||||
|
||||
def _body(**kwargs):
|
||||
return RunCreateRequest(assistant_id="lead_agent", input={"messages": [{"role": "user", "content": "hi"}]}, **kwargs)
|
||||
|
||||
|
||||
async def _start(app, request, body, thread_id):
|
||||
from app.gateway.services import start_run
|
||||
|
||||
record = await start_run(body, thread_id, request)
|
||||
await record.task
|
||||
return record
|
||||
|
||||
|
||||
async def _origin_kind(sf, run_id):
|
||||
async with sf() as session:
|
||||
return (await session.get(RunRow, run_id)).origin_kind
|
||||
|
||||
|
||||
async def _checkpoint_metadata(app, thread_id):
|
||||
checkpoint = await app.state.checkpointer.aget_tuple({"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}})
|
||||
assert checkpoint is not None
|
||||
return checkpoint.metadata
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_session_caller_cannot_forge_an_origin_in_body_metadata(gateway):
|
||||
app, sf = gateway
|
||||
record = await _start(app, _session_request(app), _body(metadata={DEERFLOW_ORIGIN_KEY: {"kind": "schedule"}, "kept": 1}), "forged-body")
|
||||
assert DEERFLOW_ORIGIN_KEY not in record.metadata
|
||||
assert record.metadata["kept"] == 1
|
||||
assert await _origin_kind(sf, record.run_id) is None
|
||||
stored = await app.state.run_store.get(record.run_id, user_id=None)
|
||||
assert DEERFLOW_ORIGIN_KEY not in stored["metadata"]
|
||||
assert DEERFLOW_ORIGIN_KEY not in await _checkpoint_metadata(app, "forged-body")
|
||||
# The thread this run auto-created carries no marker either.
|
||||
assert DEERFLOW_ORIGIN_KEY not in (await app.state.thread_store.get("forged-body", user_id=None))["metadata"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_client_origin_in_config_metadata_is_dropped(gateway, monkeypatch):
|
||||
app, sf = gateway
|
||||
captured = {}
|
||||
from app.gateway import services
|
||||
|
||||
real_run_agent = services.run_agent
|
||||
|
||||
async def capture(*args, **kwargs):
|
||||
captured["config"] = kwargs["config"]
|
||||
return await real_run_agent(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr("app.gateway.services.run_agent", capture)
|
||||
body = _body(config={"metadata": {DEERFLOW_ORIGIN_KEY: {"kind": "schedule"}, "caller_key": "kept"}})
|
||||
record = await _start(app, _session_request(app), body, "forged-config")
|
||||
assert DEERFLOW_ORIGIN_KEY not in captured["config"]["metadata"]
|
||||
assert captured["config"]["metadata"]["caller_key"] == "kept"
|
||||
assert DEERFLOW_ORIGIN_KEY not in record.metadata
|
||||
assert await _origin_kind(sf, record.run_id) is None
|
||||
assert DEERFLOW_ORIGIN_KEY not in await _checkpoint_metadata(app, "forged-config")
|
||||
|
||||
# An internal caller's server-resolved origin overrides a conflicting copy.
|
||||
origin = {"kind": "im_channel", "provider": "slack"}
|
||||
body = _body(metadata={DEERFLOW_ORIGIN_KEY: origin}, config={"metadata": {DEERFLOW_ORIGIN_KEY: {"kind": "extension"}}})
|
||||
record = await _start(app, _internal_request(app), body, "internal-config")
|
||||
assert captured["config"]["metadata"][DEERFLOW_ORIGIN_KEY] == origin
|
||||
assert record.metadata[DEERFLOW_ORIGIN_KEY] == origin
|
||||
assert await _origin_kind(sf, record.run_id) == "im_channel"
|
||||
# LangGraph copies only scalar config metadata into checkpoints, so the
|
||||
# origin object lives on the run (record and runs.origin_kind), not there.
|
||||
assert DEERFLOW_ORIGIN_KEY not in await _checkpoint_metadata(app, "internal-config")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("value", "expected"),
|
||||
[
|
||||
({"kind": "github", "provider": "github"}, "github"),
|
||||
({"kind": "im_channel", "provider": "feishu", "extra": True}, None),
|
||||
({"kind": "browser"}, None),
|
||||
],
|
||||
ids=["valid", "extra-key", "unknown-kind"],
|
||||
)
|
||||
async def test_internal_caller_origin_is_kept_only_when_well_formed(gateway, value, expected):
|
||||
app, sf = gateway
|
||||
record = await _start(app, _internal_request(app), _body(metadata={DEERFLOW_ORIGIN_KEY: value}), f"internal-{expected}")
|
||||
assert await _origin_kind(sf, record.run_id) == expected
|
||||
assert record.user_id == OWNER
|
||||
if expected is None:
|
||||
assert DEERFLOW_ORIGIN_KEY not in record.metadata
|
||||
else:
|
||||
assert record.metadata[DEERFLOW_ORIGIN_KEY] == value
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_scheduled_launch_stamps_run_and_fresh_thread(gateway):
|
||||
from app.gateway.services import launch_scheduled_thread_run
|
||||
|
||||
app, sf = gateway
|
||||
result = await launch_scheduled_thread_run(
|
||||
app=app,
|
||||
thread_id="sched-run-thread",
|
||||
assistant_id="lead_agent",
|
||||
prompt="check",
|
||||
owner_user_id=OWNER,
|
||||
metadata={"scheduled_task_id": "task-1", "scheduled_task_run_id": "occurrence-1", "scheduled_trigger": "manual"},
|
||||
)
|
||||
await (await app.state.run_manager.get(result["run_id"])).task
|
||||
stored = await app.state.run_store.get(result["run_id"], user_id=None)
|
||||
assert stored["metadata"][DEERFLOW_ORIGIN_KEY] == {"kind": "schedule"}
|
||||
assert stored["origin_kind"] == "schedule"
|
||||
assert stored["user_id"] == OWNER
|
||||
thread = await app.state.thread_store.get("sched-run-thread", user_id=OWNER)
|
||||
assert thread["metadata"][DEERFLOW_ORIGIN_KEY] == {"kind": "schedule"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mcp_notification_launch_stamps_run_but_not_the_existing_thread(gateway):
|
||||
from app.gateway.services import launch_mcp_task_notification_run
|
||||
|
||||
app, sf = gateway
|
||||
await app.state.thread_store.create("mcp-thread", user_id=OWNER, metadata={"title": "chat"})
|
||||
result = await launch_mcp_task_notification_run(
|
||||
app=app,
|
||||
thread_id="mcp-thread",
|
||||
assistant_id="lead_agent",
|
||||
owner_user_id=OWNER,
|
||||
task_id="remote-1",
|
||||
dispatch_version=1,
|
||||
dispatch_attempt=1,
|
||||
event={"status": "completed"},
|
||||
)
|
||||
await (await app.state.run_manager.get(result["run_id"])).task
|
||||
assert await _origin_kind(sf, result["run_id"]) == "mcp_notification"
|
||||
thread = await app.state.thread_store.get("mcp-thread", user_id=OWNER)
|
||||
assert thread["metadata"] == {"title": "chat"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("namespace", [None, "community.agent-teams"])
|
||||
async def test_extension_handle_marks_its_threads_and_runs(gateway, monkeypatch, namespace):
|
||||
app, sf = gateway
|
||||
request = _session_request(app)
|
||||
monkeypatch.setattr("app.gateway.extension_agent_runs.resolve_route_permissions", AsyncMock(return_value=list(request.state.auth.permissions)))
|
||||
runs = app.state.agent_runs_host.bind(request)
|
||||
if namespace is not None:
|
||||
runs = runs.for_plugin(namespace)
|
||||
expected = make_origin("extension", namespace=namespace)
|
||||
|
||||
thread_id = await runs.create_thread(thread_id="extension-thread", metadata={DEERFLOW_ORIGIN_KEY: {"kind": "schedule"}, "keep": "me"})
|
||||
thread = await app.state.thread_store.get(thread_id, user_id=OWNER)
|
||||
assert thread["metadata"][DEERFLOW_ORIGIN_KEY] == expected
|
||||
assert thread["metadata"]["keep"] == "me"
|
||||
|
||||
run = await runs.start(thread_id=thread_id, input={"messages": [{"role": "user", "content": "go"}]})
|
||||
await runs.wait(thread_id=thread_id, run_id=run.run_id, timeout=5)
|
||||
assert await _origin_kind(sf, run.run_id) == "extension"
|
||||
assert (await app.state.run_store.get(run.run_id, user_id=None))["metadata"][DEERFLOW_ORIGIN_KEY] == expected
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# IM channels and GitHub: the manager declares the origin on every run call
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _manager():
|
||||
from app.channels.manager import ChannelManager
|
||||
from app.channels.message_bus import MessageBus
|
||||
from app.channels.store import ChannelStore
|
||||
|
||||
return ChannelManager(bus=MessageBus(), store=ChannelStore(path=Path(tempfile.mkdtemp()) / "store.json"))
|
||||
|
||||
|
||||
def _client(thread_id="im-thread"):
|
||||
client = MagicMock()
|
||||
client.threads.create = AsyncMock(return_value={"thread_id": thread_id})
|
||||
client.threads.update = AsyncMock(return_value={"thread_id": thread_id})
|
||||
client.threads.get = AsyncMock(return_value={"thread_id": thread_id})
|
||||
client.runs.wait = AsyncMock(return_value={"messages": [{"type": "ai", "content": "ok"}]})
|
||||
client.runs.create = AsyncMock(return_value={"run_id": "run-1", "status": "pending"})
|
||||
|
||||
async def _stream(*_args, **_kwargs):
|
||||
yield SimpleNamespace(event="values", data={"messages": [{"type": "ai", "content": "ok"}]})
|
||||
|
||||
client.runs.stream = MagicMock(side_effect=_stream)
|
||||
return client
|
||||
|
||||
|
||||
def _message(channel="slack", **kwargs):
|
||||
from app.channels.message_bus import InboundMessage
|
||||
|
||||
return InboundMessage(channel_name=channel, chat_id="chat-1", user_id="platform-user", owner_user_id="owner-1", text="hello", **kwargs)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_channel_wait_run_declares_im_origin():
|
||||
manager = _manager()
|
||||
client = _client()
|
||||
manager._client = client
|
||||
await manager._handle_chat(_message("slack"))
|
||||
client.runs.wait.assert_called_once()
|
||||
assert client.runs.wait.call_args.kwargs["metadata"] == {DEERFLOW_ORIGIN_KEY: {"kind": "im_channel", "provider": "slack"}}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_channel_stream_run_declares_im_origin():
|
||||
manager = _manager()
|
||||
client = _client()
|
||||
await manager._handle_streaming_chat(client, _message("feishu"), "im-thread", "lead_agent", {}, {}, {"role": "user", "content": "hello"})
|
||||
client.runs.stream.assert_called_once()
|
||||
assert client.runs.stream.call_args.kwargs["metadata"] == {DEERFLOW_ORIGIN_KEY: {"kind": "im_channel", "provider": "feishu"}}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_github_create_run_declares_github_origin():
|
||||
import app.gateway.github.run_policy # noqa: F401 - registers the fire-and-forget policy
|
||||
|
||||
manager = _manager()
|
||||
client = _client("gh-thread")
|
||||
manager._client = client
|
||||
await manager._handle_chat(_github_message())
|
||||
client.runs.create.assert_called_once()
|
||||
assert client.runs.create.call_args.kwargs["metadata"] == {DEERFLOW_ORIGIN_KEY: {"kind": "github", "provider": "github"}}
|
||||
|
||||
|
||||
def _github_message(text="please fix"):
|
||||
from app.channels.message_bus import InboundMessage
|
||||
|
||||
return InboundMessage(channel_name="github", chat_id="org/repo", user_id="octocat", owner_user_id="owner-1", text=text)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_followup_drain_declares_the_carrier_origin():
|
||||
manager = _manager()
|
||||
client = _client("gh-thread")
|
||||
for index in range(2):
|
||||
manager._buffer_followup("gh-thread", _github_message(f"comment {index}"))
|
||||
await manager._drain_followups_for_thread(client, "gh-thread", _github_message("carrier"))
|
||||
client.runs.create.assert_called_once()
|
||||
assert client.runs.create.call_args.kwargs["metadata"] == {DEERFLOW_ORIGIN_KEY: {"kind": "github", "provider": "github"}}
|
||||
|
||||
|
||||
def test_unusable_channel_name_sends_no_origin():
|
||||
from app.channels.manager import _apply_run_origin
|
||||
|
||||
kwargs = {}
|
||||
_apply_run_origin(kwargs, _message("Bad Channel"))
|
||||
assert "metadata" not in kwargs
|
||||
@ -51,15 +51,27 @@ def test_literal_stand_down_reasons_and_no_verdict_are_in_contract():
|
||||
|
||||
|
||||
@pytest.mark.parametrize("code", sorted(_CODES))
|
||||
def test_im_notice_translates_every_contract_code(code):
|
||||
text = render_notification_text({"event": "run_unmet", "task_id": "task-a", "payload": {"reason_code": code}})
|
||||
assert "Reason: unknown." not in text
|
||||
@pytest.mark.parametrize(("locale", "unknown"), [("en-US", "no reason was recorded"), ("zh-CN", "没有记录原因")])
|
||||
def test_im_notice_translates_every_contract_code(code, locale, unknown):
|
||||
text = render_notification_text({"event": "run_unmet", "task_id": "task-a", "payload": {"reason_code": code, "locale": locale}})
|
||||
assert unknown not in text
|
||||
assert code not in text
|
||||
|
||||
|
||||
def test_contract_version_two_keys():
|
||||
assert _CONTRACT["version"] == 2
|
||||
def test_contract_version_three_keeps_version_two_keys():
|
||||
assert _CONTRACT["version"] == 3
|
||||
assert _CONTRACT["scheduled_origin_key"] == "deerflow_scheduled_origin"
|
||||
assert {"agent_stop_last_error_prefix", "auto_pause_last_error", "unmet_reason_codes", "check_failure_codes", "host_run_errors"} <= set(_CONTRACT)
|
||||
|
||||
|
||||
def test_lifecycle_vocabulary_matches_finalization_constants():
|
||||
assert tuple(_CONTRACT["lifecycle_events"]) == finalization.LIFECYCLE_EVENTS
|
||||
assert {event: tuple(reasons) for event, reasons in _CONTRACT["lifecycle_reasons"].items()} == finalization.LIFECYCLE_REASONS
|
||||
assert set(_CONTRACT["lifecycle_reasons"]) == set(_CONTRACT["lifecycle_events"])
|
||||
assert tuple(_CONTRACT["notification_events"]) == finalization.NOTIFICATION_EVENTS
|
||||
# The existing outbox name of the automatic pause is kept, so queued rows stay valid.
|
||||
assert "task_paused" in _CONTRACT["notification_events"]
|
||||
assert set(finalization.RUN_EVENT_BY_STATUS.values()) <= set(_CONTRACT["notification_events"])
|
||||
|
||||
|
||||
def test_scheduled_origin_key_matches_the_gateway_constant():
|
||||
|
||||
572
backend/tests/test_scheduled_task_events.py
Normal file
572
backend/tests/test_scheduled_task_events.py
Normal file
@ -0,0 +1,572 @@
|
||||
"""Lifecycle events for the originating chat (agent stop, auto-pause, finish).
|
||||
|
||||
The scheduler's finalization observer writes ``scheduled_task_events`` rows in
|
||||
the transaction that changes the task or occurrence state, deduplicated on
|
||||
``(task_id, anchor, event)``, so recovery never writes a second row. These
|
||||
tests drive the real repositories and the real observer on SQLite (and on
|
||||
Postgres where marked, with ``TEST_POSTGRES_URI``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import uuid
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import patch
|
||||
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
|
||||
from uuid import UUID
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from _router_auth_helpers import make_authed_test_app
|
||||
from sqlalchemy import event as sa_event
|
||||
from sqlalchemy import func, select, text
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
from sqlalchemy.orm.attributes import set_committed_value
|
||||
|
||||
from app.gateway.auth.models import User
|
||||
from app.gateway.auth_disabled import AUTH_SOURCE_SESSION
|
||||
from app.gateway.routers import scheduled_tasks, threads
|
||||
from app.scheduler.service import ScheduledTaskService
|
||||
from deerflow.config.paths import Paths
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.channel_connections.model import ChannelConnectionRow
|
||||
from deerflow.persistence.notification_deliveries import NotificationDeliveryRepository, NotificationDeliveryRow
|
||||
from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
|
||||
from deerflow.persistence.run.model import RunRow
|
||||
from deerflow.persistence.scheduled_task_events import ScheduledTaskEventRepository, ScheduledTaskEventRow
|
||||
from deerflow.persistence.scheduled_task_runs import ScheduledTaskRunRepository
|
||||
from deerflow.persistence.scheduled_task_runs.finalization import LIFECYCLE_REASONS, lifecycle_anchor, lifecycle_reason
|
||||
from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow
|
||||
from deerflow.persistence.scheduled_tasks import ScheduledTaskRepository
|
||||
from deerflow.persistence.scheduled_tasks.model import ScheduledTaskRow
|
||||
from deerflow.persistence.thread_meta.model import ThreadMetaRow
|
||||
from deerflow.persistence.user.model import UserPreferenceRow, UserRow
|
||||
from deerflow.runtime import RunStatus
|
||||
from deerflow.runtime.runs.manager import RunRecord
|
||||
from deerflow.runtime.runs.schemas import DisconnectMode
|
||||
|
||||
NOW = datetime(2026, 10, 6, 8, tzinfo=UTC)
|
||||
_OWNER = User(id=UUID("6c1f3d0e-8a8f-4a35-9a51-0d2f5d1b7c11"), email="events@example.com", password_hash="unused", system_role="user")
|
||||
_OTHER = User(id=UUID("0b9d2c47-55e3-4d47-8f0c-a3b0f1f4d222"), email="other@example.com", password_hash="unused", system_role="user")
|
||||
OWNER = str(_OWNER.id)
|
||||
_TABLES = [ScheduledTaskRow, ScheduledTaskRunRow, RunRow, ThreadMetaRow, ScheduledTaskEventRow, NotificationDeliveryRow, ChannelConnectionRow, UserRow, UserPreferenceRow]
|
||||
_API_FIELDS = {"id", "task_id", "event", "reason_code", "task_title", "stop_condition", "run_thread_id", "run_agent_name", "run_number", "run_status", "max_runs", "end_at", "schedule_type", "after_run_id", "created_at"}
|
||||
|
||||
|
||||
@pytest.fixture(params=["sqlite", "postgres"])
|
||||
def database_backend(request):
|
||||
return request.param
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def database(tmp_path, *, backend="sqlite"):
|
||||
schema = None
|
||||
if backend == "postgres":
|
||||
uri = os.environ.get("TEST_POSTGRES_URI")
|
||||
if not uri:
|
||||
pytest.skip("requires TEST_POSTGRES_URI (real Postgres ON CONFLICT path)")
|
||||
parts = urlsplit(uri)
|
||||
scheme = "postgresql+asyncpg" if parts.scheme in {"postgres", "postgresql"} else parts.scheme
|
||||
query = urlencode([(key, value) for key, value in parse_qsl(parts.query, keep_blank_values=True) if key not in {"sslmode", "channel_binding"}])
|
||||
schema = f"task_events_{uuid.uuid4().hex}"
|
||||
engine = create_async_engine(urlunsplit(parts._replace(scheme=scheme, query=query)), connect_args=build_asyncpg_connect_args(schema))
|
||||
else:
|
||||
engine = create_async_engine(f"sqlite+aiosqlite:///{tmp_path / 'task-events.db'}")
|
||||
try:
|
||||
async with engine.begin() as conn:
|
||||
if schema:
|
||||
await conn.execute(text(f'CREATE SCHEMA "{schema}"'))
|
||||
await conn.run_sync(lambda sync: Base.metadata.create_all(sync, tables=[model.__table__ for model in _TABLES]))
|
||||
sf = async_sessionmaker(engine, expire_on_commit=False)
|
||||
tasks, runs = ScheduledTaskRepository(sf), ScheduledTaskRunRepository(sf)
|
||||
yield sf, tasks, runs
|
||||
finally:
|
||||
if schema:
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(text(f'DROP SCHEMA IF EXISTS "{schema}" CASCADE'))
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
def service_for(tasks, runs, **kwargs):
|
||||
"""The real scheduler service; its constructor installs the finalization observer."""
|
||||
return ScheduledTaskService(task_repo=tasks, task_run_repo=runs, launch_run=None, poll_interval_seconds=60, lease_seconds=30, max_concurrent_runs=3, **kwargs)
|
||||
|
||||
|
||||
async def make_task(tasks, task_id="task", *, origin_thread_id="origin", title="Check the release checklist", assistant_id=None, **extra):
|
||||
return await tasks.create(
|
||||
task_id=task_id,
|
||||
user_id=OWNER,
|
||||
thread_id=None,
|
||||
context_mode="fresh_thread_per_run",
|
||||
assistant_id=assistant_id,
|
||||
title=title,
|
||||
prompt="Check release-checklist.md.",
|
||||
schedule_type="interval",
|
||||
schedule_spec={"every_seconds": 3600},
|
||||
timezone="UTC",
|
||||
next_run_at=NOW,
|
||||
origin_thread_id=origin_thread_id,
|
||||
**extra,
|
||||
)
|
||||
|
||||
|
||||
async def chat(sf, thread_id="origin", *, runs=()):
|
||||
"""The originating chat (a threads_meta row) and its own runs ``(run_id, created_at, kind)``."""
|
||||
async with sf() as session:
|
||||
session.add(ThreadMetaRow(thread_id=thread_id, user_id=OWNER))
|
||||
for run_id, created_at, kind in runs:
|
||||
session.add(RunRow(run_id=run_id, thread_id=thread_id, user_id=OWNER, status="success", operation_kind=kind, created_at=created_at))
|
||||
await session.commit()
|
||||
|
||||
|
||||
async def occurrence(sf, runs, task_id="task", suffix="1", *, trigger="scheduled", durable_status="success", stop=False):
|
||||
occurrence_id, run_id = f"{task_id}-occ-{suffix}", f"{task_id}-run-{suffix}"
|
||||
await runs.create(run_record_id=occurrence_id, task_id=task_id, thread_id=f"{task_id}-thread-{suffix}", scheduled_for=NOW, trigger=trigger, status="running")
|
||||
await runs.update_status(occurrence_id, status="running", run_id=run_id, started_at=NOW)
|
||||
async with sf() as session:
|
||||
session.add(RunRow(run_id=run_id, thread_id=f"{task_id}-thread-{suffix}", user_id=OWNER, status=durable_status, metadata_json={"scheduled_task_id": task_id, "scheduled_task_run_id": occurrence_id}, created_at=NOW))
|
||||
if stop:
|
||||
(await session.get(ScheduledTaskRunRow, occurrence_id)).stop_requested_run_id = run_id
|
||||
await session.commit()
|
||||
return occurrence_id, run_id
|
||||
|
||||
|
||||
async def complete(tasks, occurrence_id, run_id, *, task_id="task", status="success", error=None):
|
||||
return await tasks.complete_run(task_id, user_id=OWNER, task_run_id=occurrence_id, run_id=run_id, status=status, error=error, finished_at=NOW)
|
||||
|
||||
|
||||
async def event_rows(sf):
|
||||
async with sf() as session:
|
||||
return list((await session.execute(select(ScheduledTaskEventRow).order_by(ScheduledTaskEventRow.created_at, ScheduledTaskEventRow.id))).scalars())
|
||||
|
||||
|
||||
async def bind_wecom(sf, *, owner=OWNER, target="wecom-user"):
|
||||
"""A connected WeCom identity of the owner (the only provider with proactive push)."""
|
||||
async with sf() as session:
|
||||
session.add(ChannelConnectionRow(id=f"binding-{target}", owner_user_id=owner, provider="wecom", status="connected", external_account_id=target))
|
||||
await session.commit()
|
||||
|
||||
|
||||
async def outbox_rows(sf):
|
||||
async with sf() as session:
|
||||
return list((await session.execute(select(NotificationDeliveryRow).order_by(NotificationDeliveryRow.created_at))).scalars())
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def http_client(sf, *, user=_OWNER, owner_check_passes=True, with_threads=False, tmp_path=None):
|
||||
app = make_authed_test_app(user_factory=lambda: user, owner_check_passes=owner_check_passes, bind_current_user=True)
|
||||
|
||||
@app.middleware("http")
|
||||
async def mark_session_source(request, call_next):
|
||||
request.state.auth_source = AUTH_SOURCE_SESSION
|
||||
return await call_next(request)
|
||||
|
||||
app.state.scheduled_task_event_repo = ScheduledTaskEventRepository(sf)
|
||||
app.include_router(scheduled_tasks.router)
|
||||
if with_threads:
|
||||
|
||||
class _RunManager:
|
||||
@asynccontextmanager
|
||||
async def reserve_thread_operation(self, _thread_id, **_kwargs):
|
||||
yield
|
||||
|
||||
app.state.run_manager = _RunManager()
|
||||
app.include_router(threads.router)
|
||||
with patch("app.gateway.routers.threads.get_paths", return_value=Paths(tmp_path or ".")):
|
||||
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app, raise_app_exceptions=False), base_url="http://events.test") as client:
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_event_row_commits_with_state_change(tmp_path):
|
||||
async with database(tmp_path) as (sf, tasks, runs):
|
||||
await make_task(tasks, max_runs=1)
|
||||
await chat(sf)
|
||||
service = service_for(tasks, runs)
|
||||
|
||||
async def fails_after_insert(session, parent, row, *, events):
|
||||
await service._on_finalization(session, parent, row, events=events)
|
||||
assert await session.scalar(select(func.count()).select_from(ScheduledTaskEventRow)) == 1
|
||||
raise RuntimeError("outbox unavailable")
|
||||
|
||||
tasks.set_finalization_observer(fails_after_insert)
|
||||
occurrence_id, run_id = await occurrence(sf, runs)
|
||||
with pytest.raises(RuntimeError, match="outbox unavailable"):
|
||||
await complete(tasks, occurrence_id, run_id)
|
||||
# Both the event and the state change rolled back.
|
||||
assert await event_rows(sf) == []
|
||||
assert (await runs.list_by_task("task"))[0]["status"] == "running"
|
||||
assert (await tasks.get("task", user_id=OWNER))["status"] == "enabled"
|
||||
# The next completion (or recovery) writes both, once.
|
||||
tasks.set_finalization_observer(service._on_finalization)
|
||||
assert await complete(tasks, occurrence_id, run_id) is True
|
||||
assert [row.event for row in await event_rows(sf)] == ["task_finished"]
|
||||
|
||||
|
||||
def _record(occurrence_id, run_id, *, task_id="task"):
|
||||
return RunRecord(
|
||||
run_id=run_id,
|
||||
thread_id=f"{task_id}-thread-1",
|
||||
assistant_id="lead_agent",
|
||||
status=RunStatus.success,
|
||||
on_disconnect=DisconnectMode.continue_,
|
||||
metadata={"scheduled_task_id": task_id, "scheduled_task_run_id": occurrence_id, "scheduled_trigger": "scheduled"},
|
||||
user_id=OWNER,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("recovery_first", [True, False])
|
||||
async def test_recovery_races_late_completion_writes_one_event(tmp_path, recovery_first, database_backend):
|
||||
"""The durable run is terminal while the occurrence is still ``running``.
|
||||
|
||||
Runs on Postgres too (``TEST_POSTGRES_URI``): the IM notice, its binding and
|
||||
locale lookups and both ON CONFLICT inserts share the recovery transaction.
|
||||
"""
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
await make_task(tasks, max_runs=1)
|
||||
await chat(sf)
|
||||
await bind_wecom(sf)
|
||||
service = service_for(tasks, runs, connection_repo=object(), notification_repo=NotificationDeliveryRepository(sf))
|
||||
occurrence_id, run_id = await occurrence(sf, runs)
|
||||
if recovery_first:
|
||||
assert await runs.reconcile_active_runs(error="lease lost", now=NOW) == 1
|
||||
await service.handle_run_completion(_record(occurrence_id, run_id))
|
||||
assert await tasks.complete_run("task", user_id=OWNER, task_run_id=occurrence_id, run_id=run_id, status="success", error=None, finished_at=NOW) is False
|
||||
else:
|
||||
await service.handle_run_completion(_record(occurrence_id, run_id))
|
||||
assert await runs.reconcile_active_runs(error="lease lost", now=NOW) == 0
|
||||
rows = await event_rows(sf)
|
||||
assert [(row.event, row.anchor) for row in rows] == [("task_finished", occurrence_id)]
|
||||
assert (await tasks.get("task", user_id=OWNER))["status"] == "completed"
|
||||
# Exactly one IM notice too, committed with the same finalization.
|
||||
assert [(row.event, row.task_run_id, row.provider) for row in await outbox_rows(sf)] == [("task_finished", occurrence_id, "wecom")]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_poisoned_row_does_not_block_recovery_of_other_rows(tmp_path):
|
||||
"""Three stale occurrences in one recovery pass; one task carries bad stored values."""
|
||||
async with database(tmp_path) as (sf, tasks, runs):
|
||||
await chat(sf)
|
||||
await bind_wecom(sf)
|
||||
service_for(tasks, runs, connection_repo=object(), notification_repo=NotificationDeliveryRepository(sf))
|
||||
await make_task(tasks, "task-a", max_runs=1)
|
||||
await make_task(tasks, "task-b", stop_condition="all items are ticked")
|
||||
await make_task(tasks, "task-c")
|
||||
await tasks.update("task-c", user_id=OWNER, updates={"schedule_type": "once", "schedule_spec": {"run_at": NOW.isoformat()}})
|
||||
await occurrence(sf, runs, "task-a")
|
||||
await occurrence(sf, runs, "task-b", stop=True)
|
||||
await occurrence(sf, runs, "task-c")
|
||||
async with sf() as session:
|
||||
# Bad stored values: a blank title and a non-numeric max_runs
|
||||
# (SQLite keeps text in an INTEGER column).
|
||||
await session.execute(text("UPDATE scheduled_tasks SET title = ' ', max_runs = 'many' WHERE id = 'task-c'"))
|
||||
# An unreadable locale preference (not even JSON) for the owner.
|
||||
await session.execute(text("INSERT INTO user_preferences (user_id, key, value) VALUES (:owner, 'locale', 'not json {')"), {"owner": OWNER})
|
||||
await session.commit()
|
||||
|
||||
def unreadable_end_at(target, *_args):
|
||||
# A stored end time the observer cannot parse (the DateTime column
|
||||
# itself cannot hold one, so it is injected at load time).
|
||||
if target.id == "task-c":
|
||||
set_committed_value(target, "end_at", "not a time")
|
||||
|
||||
sa_event.listen(ScheduledTaskRow, "load", unreadable_end_at)
|
||||
sa_event.listen(ScheduledTaskRow, "refresh", unreadable_end_at)
|
||||
try:
|
||||
assert await runs.mark_stale_active_runs(error="restarted") == 3
|
||||
finally:
|
||||
sa_event.remove(ScheduledTaskRow, "load", unreadable_end_at)
|
||||
sa_event.remove(ScheduledTaskRow, "refresh", unreadable_end_at)
|
||||
|
||||
for task_id in ("task-a", "task-b", "task-c"):
|
||||
assert (await runs.list_by_task(task_id))[0]["status"] == "success"
|
||||
rows = {row.task_id: row for row in await event_rows(sf)}
|
||||
assert {task_id: (row.event, row.reason_code) for task_id, row in rows.items()} == {
|
||||
"task-a": ("task_finished", "max_runs"),
|
||||
"task-b": ("task_stopped", "agent_stop"),
|
||||
"task-c": ("task_finished", "once_done"),
|
||||
}
|
||||
poisoned = rows["task-c"].payload_json
|
||||
assert (poisoned["task_title"], poisoned["end_at"], poisoned["max_runs"]) == (None, None, None)
|
||||
assert rows["task-c"].anchor == "task-c-occ-1"
|
||||
# Each occurrence also staged its one IM notice, the unreadable locale
|
||||
# falling back to the configured default (locale None).
|
||||
notices = {row.task_id: row for row in await outbox_rows(sf)}
|
||||
assert {task_id: row.event for task_id, row in notices.items()} == {"task-a": "task_finished", "task-b": "task_stopped", "task-c": "run_completed"}
|
||||
assert {row.payload_json["locale"] for row in notices.values()} == {None}
|
||||
assert notices["task-c"].payload_json["task_title"] is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_detached_outbox_keeps_chat_events(tmp_path):
|
||||
async with database(tmp_path) as (sf, tasks, runs):
|
||||
await chat(sf)
|
||||
await bind_wecom(sf)
|
||||
service = service_for(tasks, runs, connection_repo=object(), notification_repo=NotificationDeliveryRepository(sf))
|
||||
|
||||
async def auto_pause(task_id):
|
||||
await make_task(tasks, task_id, goal_objective="every item is checked")
|
||||
for suffix in ("1", "2", "3"):
|
||||
occurrence_id, run_id = await occurrence(sf, runs, task_id, suffix)
|
||||
await complete(tasks, occurrence_id, run_id, task_id=task_id, status="unmet", error="goal_not_met_yet")
|
||||
assert (await tasks.get(task_id, user_id=OWNER))["status"] == "paused"
|
||||
|
||||
await auto_pause("wired")
|
||||
wired_outbox = len(await outbox_rows(sf))
|
||||
assert wired_outbox > 0
|
||||
service.detach_notification_outbox()
|
||||
await auto_pause("detached")
|
||||
rows = await event_rows(sf)
|
||||
assert [(row.task_id, row.event, row.reason_code) for row in rows] == [("wired", "task_paused", "consecutive_unmet"), ("detached", "task_paused", "consecutive_unmet")]
|
||||
assert len(await outbox_rows(sf)) == wired_outbox
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_event_lines_survive_task_deletion(tmp_path):
|
||||
async with database(tmp_path) as (sf, tasks, runs):
|
||||
await make_task(tasks, max_runs=1)
|
||||
await chat(sf)
|
||||
service_for(tasks, runs)
|
||||
occurrence_id, run_id = await occurrence(sf, runs)
|
||||
await complete(tasks, occurrence_id, run_id)
|
||||
assert await tasks.delete("task", user_id=OWNER) is True
|
||||
async with http_client(sf) as client:
|
||||
response = await client.get("/api/threads/origin/scheduled-task-events")
|
||||
assert response.status_code == 200, response.text
|
||||
(line,) = response.json()["events"]
|
||||
assert (line["task_id"], line["event"], line["reason_code"], line["task_title"]) == ("task", "task_finished", "max_runs", "Check the release checklist")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_event_for_deleted_origin_thread(tmp_path):
|
||||
async with database(tmp_path) as (sf, tasks, runs):
|
||||
await make_task(tasks, max_runs=1)
|
||||
service_for(tasks, runs)
|
||||
occurrence_id, run_id = await occurrence(sf, runs)
|
||||
assert await complete(tasks, occurrence_id, run_id) is True
|
||||
assert (await tasks.get("task", user_id=OWNER))["status"] == "completed"
|
||||
assert await event_rows(sf) == []
|
||||
|
||||
|
||||
def test_origin_thread_lookup_takes_a_share_lock():
|
||||
"""A racing thread delete waits for the finalization (FOR SHARE), so its
|
||||
later delete_by_thread step removes the line instead of orphaning it."""
|
||||
from sqlalchemy.dialects import postgresql, sqlite
|
||||
|
||||
from app.scheduler.service import _origin_thread_share_lock
|
||||
|
||||
statement = _origin_thread_share_lock("origin")
|
||||
assert "FOR SHARE" in str(statement.compile(dialect=postgresql.dialect()))
|
||||
# SQLite has no row locks; its single writer already serializes the two.
|
||||
assert "FOR" not in str(statement.compile(dialect=sqlite.dialect()))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_skipped_occurrence_payload_has_no_run_thread(tmp_path):
|
||||
async with database(tmp_path) as (sf, tasks, runs):
|
||||
await make_task(tasks, end_at=NOW + timedelta(seconds=1))
|
||||
await chat(sf)
|
||||
service_for(tasks, runs)
|
||||
await runs.create(run_record_id="waiting", task_id="task", thread_id="never-created", scheduled_for=NOW, trigger="scheduled", status="queued")
|
||||
assert await runs.claim_queued_run("waiting", lease_owner="worker", now=NOW + timedelta(seconds=2), lease_seconds=30, global_max_concurrent_runs=1) is None
|
||||
(row,) = await event_rows(sf)
|
||||
assert (row.event, row.reason_code) == ("task_finished", "end_at")
|
||||
assert row.payload_json["run_thread_id"] is None
|
||||
assert row.payload_json["run_agent_name"] is None
|
||||
assert row.payload_json["run_number"] is None
|
||||
assert row.payload_json["run_status"] == "skipped"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_duplicate_insert_is_ignored(tmp_path, database_backend):
|
||||
"""Both ON CONFLICT DO NOTHING paths (event row and outbox row) on this dialect."""
|
||||
async with database(tmp_path, backend=database_backend) as (sf, _tasks, _runs):
|
||||
outbox = NotificationDeliveryRepository(sf)
|
||||
async with sf() as session:
|
||||
kwargs = {"user_id": OWNER, "task_id": "task", "thread_id": "origin", "occurrence_id": "occ", "anchor": "occ", "event": "task_stopped", "reason_code": "agent_stop", "after_run_id": None}
|
||||
first = await ScheduledTaskEventRepository.record_in_session(session, payload={"task_title": "first"}, **kwargs)
|
||||
second = await ScheduledTaskEventRepository.record_in_session(session, payload={"task_title": "second"}, **kwargs)
|
||||
delivery = {"task_id": "task", "task_run_id": "occ", "event": "task_stopped", "provider": "wecom", "target": "wecom-user", "owner_user_id": OWNER}
|
||||
first_delivery = await outbox.enqueue_in_session(session, payload={"n": 1}, **delivery)
|
||||
second_delivery = await outbox.enqueue_in_session(session, payload={"n": 2}, **delivery)
|
||||
await session.commit()
|
||||
assert first["id"] == second["id"]
|
||||
assert second["task_title"] == "first"
|
||||
assert first_delivery["id"] == second_delivery["id"]
|
||||
assert [row.payload_json for row in await event_rows(sf)] == [{"task_title": "first"}]
|
||||
assert [row.payload_json for row in await outbox_rows(sf)] == [{"n": 1}]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_page_created_task_writes_no_event(tmp_path):
|
||||
async with database(tmp_path) as (sf, tasks, runs):
|
||||
await make_task(tasks, origin_thread_id=None, max_runs=1)
|
||||
await chat(sf)
|
||||
service_for(tasks, runs)
|
||||
occurrence_id, run_id = await occurrence(sf, runs)
|
||||
await complete(tasks, occurrence_id, run_id)
|
||||
assert (await tasks.get("task", user_id=OWNER))["status"] == "completed"
|
||||
assert await event_rows(sf) == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_trial_writes_no_run_event(tmp_path):
|
||||
async with database(tmp_path) as (sf, tasks, runs):
|
||||
await make_task(tasks)
|
||||
await chat(sf)
|
||||
service_for(tasks, runs)
|
||||
occurrence_id, run_id = await occurrence(sf, runs, trigger="manual")
|
||||
assert await complete(tasks, occurrence_id, run_id) is True
|
||||
assert await event_rows(sf) == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_after_run_id_is_latest_origin_run(tmp_path):
|
||||
async with database(tmp_path) as (sf, tasks, runs):
|
||||
await make_task(tasks, max_runs=1)
|
||||
await chat(
|
||||
sf,
|
||||
runs=[
|
||||
("chat-run-1", NOW - timedelta(minutes=2), "run"),
|
||||
("chat-run-2", NOW - timedelta(minutes=1), "run"),
|
||||
# A thread operation reservation is not a turn of the chat.
|
||||
("chat-op", NOW, "checkpoint_write"),
|
||||
],
|
||||
)
|
||||
await chat(sf, "quiet-chat")
|
||||
service_for(tasks, runs)
|
||||
occurrence_id, run_id = await occurrence(sf, runs)
|
||||
await complete(tasks, occurrence_id, run_id)
|
||||
await make_task(tasks, "quiet", origin_thread_id="quiet-chat", max_runs=1)
|
||||
quiet_occurrence, quiet_run = await occurrence(sf, runs, "quiet")
|
||||
await complete(tasks, quiet_occurrence, quiet_run, task_id="quiet")
|
||||
rows = {row.task_id: row for row in await event_rows(sf)}
|
||||
assert rows["task"].after_run_id == "chat-run-2"
|
||||
# A chat without runs of its own: the line goes to the tail.
|
||||
assert rows["quiet"].after_run_id is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_events_route_is_owner_scoped(tmp_path):
|
||||
async with database(tmp_path) as (sf, tasks, runs):
|
||||
await make_task(tasks, max_runs=1)
|
||||
await chat(sf)
|
||||
service_for(tasks, runs)
|
||||
occurrence_id, run_id = await occurrence(sf, runs)
|
||||
await complete(tasks, occurrence_id, run_id)
|
||||
async with http_client(sf) as client:
|
||||
own = await client.get("/api/threads/origin/scheduled-task-events")
|
||||
assert own.status_code == 200
|
||||
assert len(own.json()["events"]) == 1
|
||||
# Another user who may read the thread still sees only their own rows.
|
||||
async with http_client(sf, user=_OTHER) as client:
|
||||
other = await client.get("/api/threads/origin/scheduled-task-events")
|
||||
assert other.status_code == 200
|
||||
assert other.json() == {"events": []}
|
||||
# Without access to the thread, the route refuses before listing.
|
||||
async with http_client(sf, user=_OTHER, owner_check_passes=False) as client:
|
||||
denied = await client.get("/api/threads/origin/scheduled-task-events")
|
||||
assert denied.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_thread_delete_removes_events(tmp_path):
|
||||
async with database(tmp_path) as (sf, tasks, runs):
|
||||
await chat(sf)
|
||||
await chat(sf, "kept-chat")
|
||||
service_for(tasks, runs)
|
||||
await make_task(tasks, max_runs=1)
|
||||
await make_task(tasks, "kept", origin_thread_id="kept-chat", max_runs=1)
|
||||
for task_id in ("task", "kept"):
|
||||
occurrence_id, run_id = await occurrence(sf, runs, task_id)
|
||||
await complete(tasks, occurrence_id, run_id, task_id=task_id)
|
||||
assert len(await event_rows(sf)) == 2
|
||||
async with http_client(sf, with_threads=True, tmp_path=tmp_path) as client:
|
||||
response = await client.delete("/api/threads/origin")
|
||||
assert response.status_code == 200, response.text
|
||||
assert [row.thread_id for row in await event_rows(sf)] == ["kept-chat"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_payload_has_title_condition_run_number(tmp_path):
|
||||
async with database(tmp_path) as (sf, tasks, runs):
|
||||
# The task runs on a custom agent: the run link must open on its route.
|
||||
await make_task(tasks, stop_condition="all items are ticked", end_at=NOW + timedelta(days=30), assistant_id="release-bot")
|
||||
await chat(sf, runs=[("chat-run", NOW - timedelta(minutes=5), "run")])
|
||||
service_for(tasks, runs)
|
||||
first, first_run = await occurrence(sf, runs, suffix="1")
|
||||
await complete(tasks, first, first_run)
|
||||
second, second_run = await occurrence(sf, runs, suffix="2", stop=True)
|
||||
await complete(tasks, second, second_run)
|
||||
(row,) = await event_rows(sf)
|
||||
assert (row.user_id, row.task_id, row.thread_id, row.occurrence_id, row.anchor) == (OWNER, "task", "origin", second, second)
|
||||
assert row.id.startswith("evt-")
|
||||
assert row.payload_json == {
|
||||
"task_title": "Check the release checklist",
|
||||
"stop_condition": "all items are ticked",
|
||||
"run_thread_id": "task-thread-2",
|
||||
"run_agent_name": "release-bot",
|
||||
"run_number": 2,
|
||||
"run_status": "success",
|
||||
"latest_reason_code": None,
|
||||
"max_runs": None,
|
||||
"end_at": (NOW + timedelta(days=30)).isoformat(),
|
||||
"schedule_type": "interval",
|
||||
}
|
||||
async with http_client(sf) as client:
|
||||
(line,) = (await client.get("/api/threads/origin/scheduled-task-events")).json()["events"]
|
||||
assert set(line) == _API_FIELDS
|
||||
assert (line["run_number"], line["stop_condition"], line["after_run_id"], line["run_thread_id"], line["run_agent_name"]) == (2, "all items are ticked", "chat-run", "task-thread-2", "release-bot")
|
||||
|
||||
|
||||
def _task(**values):
|
||||
base = {"schedule_type": "interval", "status": "completed", "end_at": None, "last_occurrence_seq": 4}
|
||||
return SimpleNamespace(**{**base, **values})
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("task_values", "event", "occurrence_status", "expected"),
|
||||
[
|
||||
({}, "task_stopped", "success", "agent_stop"),
|
||||
({}, "task_paused", "unmet", "consecutive_unmet"),
|
||||
({}, "task_finished", "success", "max_runs"),
|
||||
({"end_at": NOW - timedelta(seconds=1)}, "task_finished", None, "end_at"),
|
||||
({"end_at": (NOW + timedelta(hours=1)).replace(tzinfo=None)}, "task_finished", "success", "max_runs"),
|
||||
({"end_at": "not a time"}, "task_finished", None, "max_runs"),
|
||||
({"schedule_type": "once"}, "task_finished", "success", "once_done"),
|
||||
({"schedule_type": "once", "status": "failed"}, "task_finished", "unmet", "once_failed"),
|
||||
# A once task finished by its end time before it ran is not "has run".
|
||||
({"schedule_type": "once", "end_at": NOW - timedelta(seconds=1)}, "task_finished", None, "end_at"),
|
||||
],
|
||||
)
|
||||
def test_lifecycle_reason(task_values, event, occurrence_status, expected):
|
||||
occurrence = SimpleNamespace(status=occurrence_status) if occurrence_status is not None else None
|
||||
reason = lifecycle_reason(_task(**task_values), event, now=NOW, occurrence=occurrence)
|
||||
assert reason == expected
|
||||
assert reason in LIFECYCLE_REASONS[event]
|
||||
|
||||
|
||||
def test_lifecycle_anchor_is_deterministic():
|
||||
assert lifecycle_anchor(_task(), SimpleNamespace(id="occ-7"), now=NOW) == "occ-7"
|
||||
naive = datetime(2026, 12, 31, 18, 0)
|
||||
after_end = datetime(2027, 1, 1, tzinfo=UTC)
|
||||
assert lifecycle_anchor(_task(end_at=naive), None, now=after_end) == "end:2026-12-31T18:00:00+00:00"
|
||||
assert lifecycle_anchor(_task(end_at=naive.replace(tzinfo=UTC)), None, now=after_end) == lifecycle_anchor(_task(end_at=naive), None, now=after_end)
|
||||
assert lifecycle_anchor(_task(), None, now=after_end) == "seq:4"
|
||||
assert lifecycle_anchor(_task(end_at="not a time"), None, now=after_end) == "seq:4"
|
||||
assert len(lifecycle_anchor(_task(end_at=naive), None, now=after_end)) <= 96
|
||||
|
||||
|
||||
def test_idle_max_runs_finish_does_not_take_the_end_time_anchor():
|
||||
"""A max_runs idle finish before the end time keeps a seq anchor, so a later
|
||||
end-time finish (after a reactivation) is not dropped as a duplicate."""
|
||||
end = datetime(2026, 12, 31, 18, 0, tzinfo=UTC)
|
||||
before_end = lifecycle_anchor(_task(end_at=end), None, now=end - timedelta(days=1))
|
||||
after_end = lifecycle_anchor(_task(end_at=end), None, now=end + timedelta(minutes=1))
|
||||
assert before_end == "seq:4"
|
||||
assert after_end == f"end:{end.isoformat()}"
|
||||
assert before_end != after_end
|
||||
@ -3,20 +3,26 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import os
|
||||
import uuid
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from unittest.mock import patch
|
||||
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy import select, text, update
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
|
||||
from deerflow.config.database_config import DatabaseConfig
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.engine import close_engine, get_engine, get_session_factory, init_engine_from_config
|
||||
from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
|
||||
from deerflow.persistence.run import RunRepository
|
||||
from deerflow.persistence.run.model import RunRow
|
||||
from deerflow.persistence.scheduled_task_runs import ScheduledTaskRunRepository
|
||||
from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow
|
||||
from deerflow.persistence.scheduled_tasks import ScheduledTaskRepository
|
||||
from deerflow.persistence.scheduled_tasks.model import ScheduledTaskRow
|
||||
|
||||
POSTGRES_URL = os.environ.get("TEST_POSTGRES_URI")
|
||||
|
||||
@ -226,3 +232,78 @@ async def test_postgres_once_recovery_uses_occurrence_order_despite_clock_skew(p
|
||||
older_row = await session.get(ScheduledTaskRunRow, older["id"])
|
||||
newer_row = await session.get(ScheduledTaskRunRow, newer["id"])
|
||||
assert newer_row.occurrence_seq > older_row.occurrence_seq
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def _contender_engines(count: int):
|
||||
"""``count`` independent engines on one fresh schema (one pool per contender)."""
|
||||
assert POSTGRES_URL is not None
|
||||
parts = urlsplit(_postgres_url(POSTGRES_URL))
|
||||
scheme = "postgresql+asyncpg" if parts.scheme in {"postgres", "postgresql"} else parts.scheme
|
||||
url = urlunsplit(parts._replace(scheme=scheme))
|
||||
schema = f"scheduler_owner_cap_{uuid.uuid4().hex}"
|
||||
engines = [create_async_engine(url, connect_args=build_asyncpg_connect_args(schema)) for _ in range(count)]
|
||||
try:
|
||||
async with engines[0].begin() as conn:
|
||||
await conn.execute(text(f'CREATE SCHEMA "{schema}"'))
|
||||
await conn.run_sync(lambda sync: Base.metadata.create_all(sync, tables=[ScheduledTaskRow.__table__, ScheduledTaskRunRow.__table__, RunRow.__table__]))
|
||||
yield [async_sessionmaker(engine, expire_on_commit=False) for engine in engines]
|
||||
finally:
|
||||
async with engines[0].begin() as conn:
|
||||
await conn.execute(text(f'DROP SCHEMA IF EXISTS "{schema}" CASCADE'))
|
||||
for engine in engines:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("global_cap", [3, 1], ids=["owner-cap-binds", "global-cap-binds"])
|
||||
async def test_postgres_per_owner_cap_is_atomic_under_concurrent_claims(global_cap):
|
||||
per_user = 1
|
||||
async with _contender_engines(4) as factories:
|
||||
task_repo = ScheduledTaskRepository(factories[0])
|
||||
contenders = [ScheduledTaskRunRepository(sf) for sf in factories]
|
||||
seed = contenders[0]
|
||||
now = datetime.now(UTC)
|
||||
for round_index in range(20):
|
||||
rows = {f"a1-{round_index}": "owner-a", f"a2-{round_index}": "owner-a", f"b-{round_index}": "owner-b"}
|
||||
for row_id, owner in rows.items():
|
||||
await task_repo.create(
|
||||
task_id=f"task-{row_id}",
|
||||
user_id=owner,
|
||||
thread_id=None,
|
||||
context_mode="fresh_thread_per_run",
|
||||
assistant_id="lead_agent",
|
||||
title=row_id,
|
||||
prompt="p",
|
||||
schedule_type="cron",
|
||||
schedule_spec={"cron": "* * * * *"},
|
||||
timezone="UTC",
|
||||
next_run_at=None,
|
||||
)
|
||||
await seed.create(run_record_id=row_id, task_id=f"task-{row_id}", thread_id=f"thread-{row_id}", scheduled_for=now, trigger="scheduled", status="queued")
|
||||
# Four contenders on separate engines: both A rows, B's row, and a
|
||||
# second claimer racing for A's first row.
|
||||
targets = [f"a1-{round_index}", f"a2-{round_index}", f"b-{round_index}", f"a1-{round_index}"]
|
||||
await asyncio.gather(
|
||||
*(
|
||||
repo.claim_queued_run(target, lease_owner=f"pod-{index}", now=now, lease_seconds=60, global_max_concurrent_runs=global_cap, per_user_max_concurrent_runs=per_user)
|
||||
for index, (repo, target) in enumerate(zip(contenders, targets, strict=True))
|
||||
)
|
||||
)
|
||||
async with factories[0]() as session:
|
||||
launching = {
|
||||
row_id: owner
|
||||
for row_id, owner in (
|
||||
await session.execute(select(ScheduledTaskRunRow.id, ScheduledTaskRow.user_id).join(ScheduledTaskRow, ScheduledTaskRow.id == ScheduledTaskRunRow.task_id).where(ScheduledTaskRunRow.status == "launching"))
|
||||
).all()
|
||||
}
|
||||
owners = list(launching.values())
|
||||
assert all(owners.count(owner) <= per_user for owner in set(owners)), launching
|
||||
if global_cap >= 2:
|
||||
assert sorted(owners) == ["owner-a", "owner-b"], launching
|
||||
else:
|
||||
assert len(owners) == 1, launching
|
||||
# Free the slots before the next round.
|
||||
async with factories[0]() as session:
|
||||
await session.execute(update(ScheduledTaskRunRow).where(ScheduledTaskRunRow.status.in_(("queued", "launching"))).values(status="success", lease_owner=None, lease_expires_at=None))
|
||||
await session.commit()
|
||||
|
||||
@ -1,16 +1,28 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from uuid import UUID
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from _router_auth_helpers import make_authed_test_app
|
||||
from sqlalchemy import func, select
|
||||
|
||||
from app.gateway.auth.models import User
|
||||
from app.gateway.auth_disabled import AUTH_SOURCE_SESSION
|
||||
from app.gateway.routers import scheduled_tasks
|
||||
from app.scheduler.service import ScheduledTaskService
|
||||
from deerflow.config.database_config import DatabaseConfig
|
||||
from deerflow.persistence.engine import close_engine, get_session_factory, init_engine_from_config
|
||||
from deerflow.persistence.run import RunRepository
|
||||
from deerflow.persistence.scheduled_task_runs import ScheduledTaskRunRepository
|
||||
from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow
|
||||
from deerflow.persistence.scheduled_tasks import ScheduledTaskRepository
|
||||
from deerflow.persistence.scheduled_tasks.model import ScheduledTaskRow
|
||||
from deerflow.runtime import ConflictError
|
||||
|
||||
pytestmark = pytest.mark.asyncio
|
||||
@ -1052,3 +1064,297 @@ async def test_expired_launch_claim_attaches_existing_run_instead_of_relaunching
|
||||
assert (await task_repo.get("task-attached", user_id="user-1"))["run_count"] == 1
|
||||
finally:
|
||||
await close_engine()
|
||||
|
||||
|
||||
# --- PR2 M1: per-owner run cap and fair queue selection ------------------------------------
|
||||
|
||||
_QUEUE_CONTRACT = json.loads((Path(__file__).resolve().parents[2] / "contracts" / "scheduled_goal_notes_contract.json").read_text(encoding="utf-8"))
|
||||
_OWNER_A = User(id=UUID("aaaaaaaa-0000-4000-8000-000000000001"), email="owner-a@example.com", password_hash="unused", system_role="user")
|
||||
OWNER_A = str(_OWNER_A.id)
|
||||
OWNER_B = "bbbbbbbb-0000-4000-8000-000000000002"
|
||||
|
||||
|
||||
async def _owner_task(task_repo: ScheduledTaskRepository, task_id: str, *, user_id: str, next_run_at: datetime | None = None) -> dict:
|
||||
"""A page-created per-minute task with a fresh thread for every run."""
|
||||
return await task_repo.create(
|
||||
task_id=task_id,
|
||||
user_id=user_id,
|
||||
thread_id=None,
|
||||
context_mode="fresh_thread_per_run",
|
||||
assistant_id="lead_agent",
|
||||
title=task_id,
|
||||
prompt="Check the list",
|
||||
schedule_type="cron",
|
||||
schedule_spec={"cron": "* * * * *"},
|
||||
timezone="UTC",
|
||||
next_run_at=next_run_at,
|
||||
)
|
||||
|
||||
|
||||
async def _owned_occurrence(
|
||||
task_repo: ScheduledTaskRepository,
|
||||
run_repo: ScheduledTaskRunRepository,
|
||||
row_id: str,
|
||||
*,
|
||||
user_id: str,
|
||||
status: str = "queued",
|
||||
thread_id: str | None = None,
|
||||
task_id: str | None = None,
|
||||
) -> dict:
|
||||
"""One occurrence on its own task (one active occurrence per task)."""
|
||||
task_id = task_id or f"task-{row_id}"
|
||||
if await task_repo.get_internal(task_id) is None:
|
||||
await _owner_task(task_repo, task_id, user_id=user_id)
|
||||
return await run_repo.create(
|
||||
run_record_id=row_id,
|
||||
task_id=task_id,
|
||||
thread_id=thread_id or f"thread-{row_id}",
|
||||
scheduled_for=datetime.now(UTC),
|
||||
trigger="scheduled",
|
||||
status=status,
|
||||
)
|
||||
|
||||
|
||||
async def _executing_by_owner(sf, owner: str) -> int:
|
||||
async with sf() as session:
|
||||
return int(
|
||||
await session.scalar(
|
||||
select(func.count())
|
||||
.select_from(ScheduledTaskRunRow)
|
||||
.join(ScheduledTaskRow, ScheduledTaskRow.id == ScheduledTaskRunRow.task_id)
|
||||
.where(ScheduledTaskRunRow.status.in_(("launching", "running")), ScheduledTaskRow.user_id == owner)
|
||||
)
|
||||
or 0
|
||||
)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def sqlite_repos(tmp_path):
|
||||
await init_engine_from_config(DatabaseConfig(backend="sqlite", sqlite_dir=str(tmp_path)))
|
||||
sf = get_session_factory()
|
||||
assert sf is not None
|
||||
try:
|
||||
yield sf, ScheduledTaskRepository(sf), ScheduledTaskRunRepository(sf)
|
||||
finally:
|
||||
await close_engine()
|
||||
|
||||
|
||||
def _capped_service(task_repo, run_repo, launches: list[dict], **limits) -> ScheduledTaskService:
|
||||
"""The real service with the given caps; every launch is recorded in ``launches``."""
|
||||
|
||||
async def launch_run(**kwargs):
|
||||
launches.append(kwargs)
|
||||
return {"run_id": f"run-{len(launches)}", "thread_id": kwargs["thread_id"]}
|
||||
|
||||
return ScheduledTaskService(task_repo=task_repo, task_run_repo=run_repo, launch_run=launch_run, poll_interval_seconds=5, lease_seconds=120, **limits)
|
||||
|
||||
|
||||
async def test_fair_batch_includes_other_owner_when_one_owner_is_at_cap(sqlite_repos):
|
||||
_sf, task_repo, run_repo = sqlite_repos
|
||||
for index in range(2):
|
||||
await _owned_occurrence(task_repo, run_repo, f"a-running-{index}", user_id=OWNER_A, status="running")
|
||||
for index in range(50):
|
||||
await _owned_occurrence(task_repo, run_repo, f"a-queued-{index:02d}", user_id=OWNER_A)
|
||||
await _owned_occurrence(task_repo, run_repo, "b-queued", user_id=OWNER_B)
|
||||
|
||||
batch = await run_repo.list_queued_runs(limit=16, per_user_max_concurrent_runs=2)
|
||||
|
||||
# A is at its cap, so its 50 waiting rows drop out before the limit.
|
||||
assert [row["id"] for row in batch] == ["b-queued"]
|
||||
|
||||
|
||||
async def test_fair_batch_interleaves_owners_without_cap(sqlite_repos):
|
||||
_sf, task_repo, run_repo = sqlite_repos
|
||||
for index in range(50):
|
||||
await _owned_occurrence(task_repo, run_repo, f"a-queued-{index:02d}", user_id=OWNER_A)
|
||||
await _owned_occurrence(task_repo, run_repo, "b-queued", user_id=OWNER_B)
|
||||
|
||||
batch = [row["id"] for row in await run_repo.list_queued_runs(limit=16)]
|
||||
|
||||
# Without a cap the batch is still fair: B's only row, created last,
|
||||
# comes right after A's head instead of after 50 older A rows.
|
||||
assert len(batch) == 16
|
||||
assert batch.index("b-queued") <= 1
|
||||
assert batch[0] == "a-queued-00"
|
||||
# Within an owner the order is unchanged (attempt_count, created_at, id).
|
||||
assert [row_id for row_id in batch if row_id.startswith("a-")] == [f"a-queued-{index:02d}" for index in range(15)]
|
||||
|
||||
|
||||
async def test_owner_rank_respects_remaining_capacity(sqlite_repos):
|
||||
_sf, task_repo, run_repo = sqlite_repos
|
||||
await _owned_occurrence(task_repo, run_repo, "a-running", user_id=OWNER_A, status="running")
|
||||
for index in range(5):
|
||||
await _owned_occurrence(task_repo, run_repo, f"a-queued-{index}", user_id=OWNER_A)
|
||||
await _owned_occurrence(task_repo, run_repo, "b-queued", user_id=OWNER_B)
|
||||
|
||||
batch = [row["id"] for row in await run_repo.list_queued_runs(limit=16, per_user_max_concurrent_runs=3)]
|
||||
|
||||
assert [row_id for row_id in batch if row_id.startswith("a-")] == ["a-queued-0", "a-queued-1"]
|
||||
assert "b-queued" in batch
|
||||
|
||||
|
||||
@pytest.mark.parametrize("cap", [0, 2])
|
||||
async def test_fair_batch_keeps_same_thread_fifo(sqlite_repos, cap):
|
||||
_sf, task_repo, run_repo = sqlite_repos
|
||||
await _owned_occurrence(task_repo, run_repo, "a-older", user_id=OWNER_A, thread_id="reuse-thread")
|
||||
await _owned_occurrence(task_repo, run_repo, "a-newer", user_id=OWNER_A, thread_id="reuse-thread")
|
||||
|
||||
batch = [row["id"] for row in await run_repo.list_queued_runs(limit=16, per_user_max_concurrent_runs=cap)]
|
||||
|
||||
assert batch == ["a-older"]
|
||||
|
||||
|
||||
async def test_orphan_queued_row_is_still_drained(sqlite_repos):
|
||||
_sf, task_repo, run_repo = sqlite_repos
|
||||
await _owned_occurrence(task_repo, run_repo, "a-running", user_id=OWNER_A, status="running")
|
||||
# The parent task is gone (deleted while the row waited): no owner.
|
||||
await run_repo.create(run_record_id="orphan", task_id="task-deleted", thread_id="thread-orphan", scheduled_for=datetime.now(UTC), trigger="scheduled", status="queued")
|
||||
|
||||
assert [row["id"] for row in await run_repo.list_queued_runs(limit=16, per_user_max_concurrent_runs=1)] == ["orphan"]
|
||||
|
||||
launches: list[dict] = []
|
||||
service = _capped_service(task_repo, run_repo, launches, max_concurrent_runs=3, max_concurrent_runs_per_user=1)
|
||||
await service.run_once(now=datetime.now(UTC))
|
||||
|
||||
row = (await run_repo.list_by_task("task-deleted"))[0]
|
||||
assert (row["status"], row["error"]) == ("interrupted", _QUEUE_CONTRACT["host_run_errors"]["deleted_while_queued"])
|
||||
assert launches == []
|
||||
|
||||
|
||||
async def test_claim_rejects_owner_at_cap_but_admits_other_owner(sqlite_repos):
|
||||
_sf, task_repo, run_repo = sqlite_repos
|
||||
now = datetime.now(UTC)
|
||||
await _owned_occurrence(task_repo, run_repo, "a-running", user_id=OWNER_A, status="running")
|
||||
await _owned_occurrence(task_repo, run_repo, "a-queued", user_id=OWNER_A)
|
||||
await _owned_occurrence(task_repo, run_repo, "b-queued", user_id=OWNER_B)
|
||||
claim = {"now": now, "lease_seconds": 120, "global_max_concurrent_runs": 3, "per_user_max_concurrent_runs": 1}
|
||||
|
||||
assert await run_repo.claim_queued_run("a-queued", lease_owner="worker", **claim) is None
|
||||
assert (await run_repo.list_by_task("task-a-queued"))[0]["status"] == "queued"
|
||||
assert (await run_repo.list_by_task("task-a-queued"))[0]["attempt_count"] == 0
|
||||
|
||||
claimed = await run_repo.claim_queued_run("b-queued", lease_owner="worker", **claim)
|
||||
assert claimed is not None and claimed["status"] == "launching"
|
||||
|
||||
# Cap 0 keeps today's behavior: only the global budget applies.
|
||||
assert await run_repo.claim_queued_run("a-queued", lease_owner="worker", **{**claim, "per_user_max_concurrent_runs": 0}) is not None
|
||||
|
||||
|
||||
async def test_effective_owner_cap_is_bounded_by_global_cap(sqlite_repos):
|
||||
sf, task_repo, run_repo = sqlite_repos
|
||||
for index in range(3):
|
||||
await _owned_occurrence(task_repo, run_repo, f"a-queued-{index}", user_id=OWNER_A)
|
||||
launches: list[dict] = []
|
||||
service = _capped_service(task_repo, run_repo, launches, max_concurrent_runs=1, max_concurrent_runs_per_user=2)
|
||||
|
||||
assert service._max_concurrent_runs_per_user == 1
|
||||
await service.run_once(now=datetime.now(UTC))
|
||||
|
||||
assert len(launches) == 1
|
||||
assert await run_repo.count_active_runs() == 1
|
||||
assert await _executing_by_owner(sf, OWNER_A) == 1
|
||||
# The drain asks only for as many A rows as A may still start.
|
||||
assert await run_repo.list_queued_runs(limit=16, per_user_max_concurrent_runs=service._max_concurrent_runs_per_user) == []
|
||||
|
||||
|
||||
async def test_capped_owner_queue_expiry_writes_contract_queue_timeout_note(sqlite_repos):
|
||||
sf, task_repo, run_repo = sqlite_repos
|
||||
await _owned_occurrence(task_repo, run_repo, "a-running", user_id=OWNER_A, status="running")
|
||||
await _owned_occurrence(task_repo, run_repo, "a-waiting", user_id=OWNER_A)
|
||||
launches: list[dict] = []
|
||||
service = _capped_service(task_repo, run_repo, launches, max_concurrent_runs=3, max_concurrent_runs_per_user=1, queue_timeout_seconds=60)
|
||||
|
||||
# While A holds its only slot, the waiting row is not launched.
|
||||
await service.run_once(now=datetime.now(UTC))
|
||||
assert launches == []
|
||||
assert (await run_repo.list_by_task("task-a-waiting"))[0]["status"] == "queued"
|
||||
|
||||
await service.run_once(now=datetime.now(UTC) + timedelta(seconds=61))
|
||||
|
||||
expired = (await run_repo.list_by_task("task-a-waiting"))[0]
|
||||
assert (expired["status"], expired["run_id"]) == ("failed", None)
|
||||
assert expired["error"] == _QUEUE_CONTRACT["host_run_errors"]["queue_timeout"]
|
||||
assert launches == []
|
||||
|
||||
# The runs route hands that note to the page unchanged, so PR1's
|
||||
# runErrors.queueTimeout mapping ("waited too long") applies.
|
||||
app = make_authed_test_app(user_factory=lambda: _OWNER_A, bind_current_user=True)
|
||||
|
||||
@app.middleware("http")
|
||||
async def mark_session_source(request, call_next):
|
||||
request.state.auth_source = AUTH_SOURCE_SESSION
|
||||
return await call_next(request)
|
||||
|
||||
app.state.scheduled_task_repo = task_repo
|
||||
app.state.scheduled_task_run_repo = run_repo
|
||||
app.include_router(scheduled_tasks.router)
|
||||
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://queue.test") as client:
|
||||
response = await client.get("/api/scheduled-tasks/task-a-waiting/runs")
|
||||
assert response.status_code == 200, response.text
|
||||
assert [(row["status"], row["run_id"], row["error"]) for row in response.json()] == [("failed", None, _QUEUE_CONTRACT["host_run_errors"]["queue_timeout"])]
|
||||
|
||||
|
||||
async def test_service_drain_launches_other_owner_first(sqlite_repos):
|
||||
"""Two poll cycles: A's backlog never takes B's slot.
|
||||
|
||||
``claim_due_tasks`` admits ``max_concurrent_runs`` tasks per poll ordered
|
||||
by ``next_run_at, id``, so A's five tasks (due first) fill cycle 1. Each
|
||||
admitted occurrence is offered a launch at once; the per-owner cap lets
|
||||
one A row start and leaves the other two queued. In cycle 2 the drain
|
||||
must not start any A row, and B's occurrence (due one second later)
|
||||
launches although four older A rows are queued ahead of it.
|
||||
"""
|
||||
sf, task_repo, run_repo = sqlite_repos
|
||||
t0 = datetime.now(UTC).replace(microsecond=0) - timedelta(minutes=5)
|
||||
for index in range(5):
|
||||
await _owner_task(task_repo, f"task-a-{index}", user_id=OWNER_A, next_run_at=t0)
|
||||
await _owner_task(task_repo, "task-b", user_id=OWNER_B, next_run_at=t0 + timedelta(seconds=1))
|
||||
launches: list[dict] = []
|
||||
service = _capped_service(task_repo, run_repo, launches, max_concurrent_runs=3, max_concurrent_runs_per_user=1)
|
||||
|
||||
b_cycle = None
|
||||
for cycle in range(1, 5):
|
||||
await service.run_once(now=t0 + timedelta(seconds=cycle))
|
||||
assert await _executing_by_owner(sf, OWNER_A) <= 1
|
||||
b_rows = await run_repo.list_by_task("task-b")
|
||||
if b_rows:
|
||||
b_cycle = cycle
|
||||
break
|
||||
|
||||
assert b_cycle == 2
|
||||
assert b_rows[0]["status"] in {"launching", "running"}
|
||||
assert await _executing_by_owner(sf, OWNER_A) == 1
|
||||
queued_a = [row for index in range(5) for row in await run_repo.list_by_task(f"task-a-{index}") if row["status"] == "queued"]
|
||||
assert len(queued_a) == 4
|
||||
|
||||
|
||||
async def test_service_drain_launches_other_owner_from_a_seeded_queue(sqlite_repos):
|
||||
"""A seeded backlog larger than the drain batch cannot hide B's row."""
|
||||
sf, task_repo, run_repo = sqlite_repos
|
||||
for index in range(20):
|
||||
await _owned_occurrence(task_repo, run_repo, f"a-queued-{index:02d}", user_id=OWNER_A)
|
||||
await _owned_occurrence(task_repo, run_repo, "b-queued", user_id=OWNER_B)
|
||||
launches: list[dict] = []
|
||||
service = _capped_service(task_repo, run_repo, launches, max_concurrent_runs=3, max_concurrent_runs_per_user=1)
|
||||
|
||||
await service.run_once(now=datetime.now(UTC))
|
||||
|
||||
# The batch limit is max(16, 3 * 4) = 16 < 20 A rows: an unfair batch
|
||||
# would hold only A rows. The fair one starts A's head and B's row.
|
||||
assert (await run_repo.list_by_task("task-b-queued"))[0]["status"] == "running"
|
||||
assert (await run_repo.list_by_task("task-a-queued-00"))[0]["status"] == "running"
|
||||
assert await _executing_by_owner(sf, OWNER_A) == 1
|
||||
assert len(launches) == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize("dialect_name", ["postgresql", "sqlite"])
|
||||
async def test_fair_queue_statement_compiles_for_both_dialects(dialect_name):
|
||||
"""The fair drain is plain SQLAlchemy Core (no raw text) on both dialects."""
|
||||
from sqlalchemy.dialects import postgresql, sqlite
|
||||
|
||||
dialect = {"postgresql": postgresql.dialect(), "sqlite": sqlite.dialect()}[dialect_name]
|
||||
sql = str(ScheduledTaskRunRepository._fair_queue_statement(limit=16, per_user_max_concurrent_runs=2).compile(dialect=dialect)).lower()
|
||||
|
||||
assert "row_number() over (partition by" in sql
|
||||
assert "group by" in sql and "not (exists" in sql
|
||||
assert "limit" in sql
|
||||
|
||||
@ -2,6 +2,7 @@ import asyncio
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
|
||||
from app.scheduler.service import ScheduledTaskService
|
||||
from deerflow.runtime import ConflictError, RunStatus
|
||||
@ -79,7 +80,7 @@ class DummyRunRepo:
|
||||
async def count_active_runs(self):
|
||||
return self.active_count
|
||||
|
||||
async def list_queued_runs(self, *, limit):
|
||||
async def list_queued_runs(self, *, limit, **_kwargs):
|
||||
return []
|
||||
|
||||
async def expire_queued_runs(self, **_kwargs):
|
||||
@ -808,7 +809,7 @@ class _StatefulRunRepo:
|
||||
async def count_active_runs(self) -> int:
|
||||
return sum(1 for row in self.rows.values() if row["status"] in {"launching", "running"})
|
||||
|
||||
async def list_queued_runs(self, *, limit: int) -> list[dict]:
|
||||
async def list_queued_runs(self, *, limit: int, **_kwargs) -> list[dict]:
|
||||
return []
|
||||
|
||||
async def expire_queued_runs(self, **_kwargs) -> list[dict]:
|
||||
@ -1192,308 +1193,378 @@ async def test_manual_trigger_proceeds_when_global_budget_available():
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Notification enqueue on completion (issue #4254): the hook writes durable
|
||||
# outbox rows; a separate delivery worker does the actual IM send.
|
||||
# IM notices (issue #4254, RFC #6340 N1). The finalization observer stages one
|
||||
# notice per occurrence in the transaction that records the outcome; the
|
||||
# delivery worker sends it. These tests drive the real SQLite repositories.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class DummyConnectionRepo:
|
||||
def __init__(self, connections):
|
||||
self.connections = connections
|
||||
|
||||
async def list_connections(self, owner_user_id):
|
||||
return [dict(connection, owner_user_id=owner_user_id) for connection in self.connections]
|
||||
NOTICE_NOW = datetime(2026, 10, 6, 8, tzinfo=UTC)
|
||||
NOTICE_OWNER = "6c1f3d0e-8a8f-4a35-9a51-0d2f5d1b7c11"
|
||||
|
||||
|
||||
class DummyNotificationRepo:
|
||||
def __init__(self, *, fail=False):
|
||||
"""Records any enqueue attempt; the service must make none outside finalization."""
|
||||
|
||||
def __init__(self):
|
||||
self.enqueued = []
|
||||
self.fail = fail
|
||||
|
||||
async def enqueue(self, **kwargs):
|
||||
if self.fail:
|
||||
raise RuntimeError("delivery backend unavailable")
|
||||
self.enqueued.append(kwargs)
|
||||
return {"id": f"delivery-{len(self.enqueued)}", **kwargs}
|
||||
|
||||
async def enqueue_in_session(self, _session, **kwargs):
|
||||
self.enqueued.append(kwargs)
|
||||
|
||||
|
||||
def _make_notifying_service(task_repo, run_repo, *, connection_repo, notification_repo):
|
||||
return ScheduledTaskService(
|
||||
task_repo=task_repo,
|
||||
task_run_repo=run_repo,
|
||||
launch_run=lambda **_kwargs: None,
|
||||
poll_interval_seconds=5,
|
||||
lease_seconds=120,
|
||||
max_concurrent_runs=3,
|
||||
connection_repo=connection_repo,
|
||||
notification_repo=notification_repo,
|
||||
class DummyConnectionRepo:
|
||||
def __init__(self):
|
||||
self.reads = 0
|
||||
|
||||
async def list_connections(self, owner_user_id):
|
||||
self.reads += 1
|
||||
return [{"provider": "wecom", "external_account_id": "someone", "status": "connected", "owner_user_id": owner_user_id}]
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def outbox(tmp_path):
|
||||
from types import SimpleNamespace
|
||||
|
||||
from deerflow.persistence.channel_connections import ChannelConnectionRepository
|
||||
from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
|
||||
from deerflow.persistence.notification_deliveries import NotificationDeliveryRepository
|
||||
from deerflow.persistence.scheduled_task_runs import ScheduledTaskRunRepository
|
||||
from deerflow.persistence.scheduled_tasks import ScheduledTaskRepository
|
||||
from deerflow.persistence.user.model import UserRow
|
||||
|
||||
await init_engine("sqlite", url=f"sqlite+aiosqlite:///{tmp_path / 'notices.db'}", sqlite_dir=str(tmp_path))
|
||||
sf = get_session_factory()
|
||||
async with sf() as session:
|
||||
session.add(UserRow(id=NOTICE_OWNER, email="owner@example.com"))
|
||||
await session.commit()
|
||||
tasks, runs, deliveries = ScheduledTaskRepository(sf), ScheduledTaskRunRepository(sf), NotificationDeliveryRepository(sf)
|
||||
service = ScheduledTaskService(task_repo=tasks, task_run_repo=runs, launch_run=None, poll_interval_seconds=60, lease_seconds=30, max_concurrent_runs=3, connection_repo=ChannelConnectionRepository(sf), notification_repo=deliveries)
|
||||
try:
|
||||
yield SimpleNamespace(sf=sf, tasks=tasks, runs=runs, deliveries=deliveries, service=service)
|
||||
finally:
|
||||
await close_engine()
|
||||
|
||||
|
||||
async def _bind(env, provider="wecom", target="wecom-user", *, status="connected"):
|
||||
from deerflow.persistence.channel_connections.model import ChannelConnectionRow
|
||||
|
||||
async with env.sf() as session:
|
||||
session.add(ChannelConnectionRow(id=f"binding-{provider}-{target}", owner_user_id=NOTICE_OWNER, provider=provider, status=status, external_account_id=target))
|
||||
await session.commit()
|
||||
|
||||
|
||||
async def _set_locale(env, value):
|
||||
from sqlalchemy import delete
|
||||
|
||||
from deerflow.persistence.user.model import UserPreferenceRow
|
||||
|
||||
async with env.sf() as session:
|
||||
await session.execute(delete(UserPreferenceRow).where(UserPreferenceRow.user_id == NOTICE_OWNER, UserPreferenceRow.key == "locale"))
|
||||
if value is not None:
|
||||
session.add(UserPreferenceRow(user_id=NOTICE_OWNER, key="locale", value=value))
|
||||
await session.commit()
|
||||
|
||||
|
||||
async def _task(env, task_id="task-a", *, origin_thread_id=None, once=False, **extra):
|
||||
task = await env.tasks.create(
|
||||
task_id=task_id,
|
||||
user_id=NOTICE_OWNER,
|
||||
thread_id=None,
|
||||
context_mode="fresh_thread_per_run",
|
||||
assistant_id=None,
|
||||
title="Check the release checklist",
|
||||
prompt="Check release-checklist.md.",
|
||||
schedule_type="interval",
|
||||
schedule_spec={"every_seconds": 3600},
|
||||
timezone="UTC",
|
||||
next_run_at=NOTICE_NOW,
|
||||
origin_thread_id=origin_thread_id,
|
||||
**extra,
|
||||
)
|
||||
if once:
|
||||
await env.tasks.update(task_id, user_id=NOTICE_OWNER, updates={"schedule_type": "once", "schedule_spec": {"run_at": NOTICE_NOW.isoformat()}})
|
||||
return task
|
||||
|
||||
|
||||
async def _occurrence(env, task_id="task-a", suffix="1", *, trigger="scheduled", stop=False, reply="Report is ready", durable_status="running"):
|
||||
from deerflow.persistence.run.model import RunRow
|
||||
from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow
|
||||
|
||||
occurrence_id, run_id = f"task-run-{task_id}-{suffix}", f"run-{task_id}-{suffix}"
|
||||
await env.runs.create(run_record_id=occurrence_id, task_id=task_id, thread_id=f"thread-{task_id}-{suffix}", scheduled_for=NOTICE_NOW, trigger=trigger, status="running")
|
||||
await env.runs.update_status(occurrence_id, status="running", run_id=run_id, started_at=NOTICE_NOW)
|
||||
async with env.sf() as session:
|
||||
session.add(
|
||||
RunRow(
|
||||
run_id=run_id,
|
||||
thread_id=f"thread-{task_id}-{suffix}",
|
||||
user_id=NOTICE_OWNER,
|
||||
status=durable_status,
|
||||
last_ai_message=reply,
|
||||
metadata_json={"scheduled_task_id": task_id, "scheduled_task_run_id": occurrence_id},
|
||||
created_at=NOTICE_NOW,
|
||||
)
|
||||
)
|
||||
if stop:
|
||||
(await session.get(ScheduledTaskRunRow, occurrence_id)).stop_requested_run_id = run_id
|
||||
await session.commit()
|
||||
return occurrence_id, run_id
|
||||
|
||||
|
||||
async def _complete(env, occurrence_id, run_id, *, task_id="task-a", status="success", error=None, finished_at=NOTICE_NOW):
|
||||
return await env.tasks.complete_run(task_id, user_id=NOTICE_OWNER, task_run_id=occurrence_id, run_id=run_id, status=status, error=error, finished_at=finished_at)
|
||||
|
||||
|
||||
async def _notices(env):
|
||||
from sqlalchemy import select
|
||||
|
||||
from deerflow.persistence.notification_deliveries import NotificationDeliveryRow
|
||||
|
||||
async with env.sf() as session:
|
||||
rows = (await session.execute(select(NotificationDeliveryRow).order_by(NotificationDeliveryRow.created_at, NotificationDeliveryRow.id))).scalars()
|
||||
return [{"event": row.event, "provider": row.provider, "target": row.target, "task_run_id": row.task_run_id, "run_id": row.run_id, "payload": row.payload_json} for row in rows]
|
||||
|
||||
|
||||
async def _deliver_all(env, *, default_locale="en-US"):
|
||||
"""Send every queued notice through the real worker with a fake WeCom channel; return the texts."""
|
||||
from app.scheduler.notification_delivery import NotificationDeliveryWorker
|
||||
from deerflow.persistence.run import RunRepository
|
||||
|
||||
sent = []
|
||||
|
||||
class Channel:
|
||||
is_running = True
|
||||
|
||||
async def send_notification(self, *, target, text_markdown):
|
||||
sent.append(text_markdown)
|
||||
|
||||
run_repo = RunRepository(env.sf)
|
||||
|
||||
async def resolve_run_summary(run_id, user_id):
|
||||
row = await run_repo.get(run_id, user_id=user_id)
|
||||
return row.get("last_ai_message") if row else None
|
||||
|
||||
channel = Channel()
|
||||
worker = NotificationDeliveryWorker(delivery_repo=env.deliveries, resolve_channel=lambda _provider: channel, resolve_run_summary=resolve_run_summary, default_locale=default_locale)
|
||||
await worker.run_once(now=datetime.now(UTC) + timedelta(days=1))
|
||||
return sent
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_completion_enqueues_run_completed_for_connected_channels():
|
||||
task_repo = DummyTaskRepo([_once_task_row()])
|
||||
run_repo = DummyRunRepo()
|
||||
connection_repo = DummyConnectionRepo(
|
||||
[
|
||||
{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"},
|
||||
]
|
||||
)
|
||||
notification_repo = DummyNotificationRepo()
|
||||
service = _make_notifying_service(
|
||||
task_repo,
|
||||
run_repo,
|
||||
connection_repo=connection_repo,
|
||||
notification_repo=notification_repo,
|
||||
)
|
||||
|
||||
await service.handle_run_completion(_completion_record(RunStatus.success))
|
||||
|
||||
assert len(notification_repo.enqueued) == 1
|
||||
enqueued = notification_repo.enqueued[0]
|
||||
assert enqueued["event"] == "run_completed"
|
||||
assert enqueued["provider"] == "wecom"
|
||||
assert enqueued["target"] == "GaoZhiChao"
|
||||
assert enqueued["task_id"] == "task-once"
|
||||
assert enqueued["task_run_id"] == "task-run-x"
|
||||
assert enqueued["run_id"] == "run-x"
|
||||
assert enqueued["owner_user_id"] == "user-1"
|
||||
assert enqueued["payload"]["run_status"] == "success"
|
||||
assert enqueued["payload"]["error"] is None
|
||||
assert enqueued["payload"]["task_title"] == "Once summary"
|
||||
async def test_unsupported_provider_gets_no_outbox_row(outbox):
|
||||
await _task(outbox)
|
||||
await _bind(outbox, "feishu", "ou_feishu")
|
||||
await _bind(outbox, "wecom", "wecom-user")
|
||||
await _bind(outbox, "wecom", "revoked-user", status="revoked")
|
||||
occurrence_id, run_id = await _occurrence(outbox)
|
||||
assert await _complete(outbox, occurrence_id, run_id) is True
|
||||
assert [(row["event"], row["provider"], row["target"]) for row in await _notices(outbox)] == [("run_completed", "wecom", "wecom-user")]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_completion_enqueues_run_failed_event_on_error():
|
||||
task_repo = DummyTaskRepo([_once_task_row()])
|
||||
run_repo = DummyRunRepo()
|
||||
connection_repo = DummyConnectionRepo(
|
||||
[
|
||||
{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"},
|
||||
]
|
||||
)
|
||||
notification_repo = DummyNotificationRepo()
|
||||
service = _make_notifying_service(
|
||||
task_repo,
|
||||
run_repo,
|
||||
connection_repo=connection_repo,
|
||||
notification_repo=notification_repo,
|
||||
)
|
||||
|
||||
await service.handle_run_completion(_completion_record(RunStatus.error, error="boom"))
|
||||
|
||||
assert len(notification_repo.enqueued) == 1
|
||||
enqueued = notification_repo.enqueued[0]
|
||||
assert enqueued["event"] == "run_failed"
|
||||
assert enqueued["payload"]["run_status"] == "failed"
|
||||
assert enqueued["payload"]["error"] == "boom"
|
||||
async def test_auto_pause_sends_one_merged_notice(outbox):
|
||||
await _task(outbox, goal_objective="every item is checked")
|
||||
await _bind(outbox)
|
||||
for suffix, code in (("1", "goal_not_met_yet"), ("2", "goal_not_met_yet"), ("3", "blocked:needs_user_input")):
|
||||
occurrence_id, run_id = await _occurrence(outbox, suffix=suffix, reply="Two items are still open")
|
||||
await _complete(outbox, occurrence_id, run_id, status="unmet", error=code)
|
||||
assert (await outbox.tasks.get("task-a", user_id=NOTICE_OWNER))["status"] == "paused"
|
||||
notices = await _notices(outbox)
|
||||
# Runs 1 and 2 each send their unmet notice; run 3 sends only the merged pause.
|
||||
assert [(row["event"], row["task_run_id"]) for row in notices] == [
|
||||
("run_unmet", "task-run-task-a-1"),
|
||||
("run_unmet", "task-run-task-a-2"),
|
||||
("task_paused", "task-run-task-a-3"),
|
||||
]
|
||||
assert notices[-1]["payload"]["latest_reason_code"] == "blocked:needs_user_input"
|
||||
text = (await _deliver_all(outbox))[-1]
|
||||
assert text.splitlines() == [
|
||||
"Scheduled task “Check the release checklist”",
|
||||
"Paused after 3 runs in a row missed the goal: it needs your input.",
|
||||
"Result: Two items are still open",
|
||||
"Open DeerFlow → Scheduled tasks for details.",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_completion_does_not_notify_for_interrupted_runs():
|
||||
task_repo = DummyTaskRepo([_once_task_row()])
|
||||
run_repo = DummyRunRepo()
|
||||
connection_repo = DummyConnectionRepo(
|
||||
[
|
||||
{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"},
|
||||
]
|
||||
)
|
||||
notification_repo = DummyNotificationRepo()
|
||||
service = _make_notifying_service(
|
||||
task_repo,
|
||||
run_repo,
|
||||
connection_repo=connection_repo,
|
||||
notification_repo=notification_repo,
|
||||
)
|
||||
|
||||
await service.handle_run_completion(_completion_record(RunStatus.interrupted))
|
||||
|
||||
# An interrupt is a user-initiated cancel; the user is already aware.
|
||||
assert notification_repo.enqueued == []
|
||||
async def test_agent_stop_sends_task_stopped_only(outbox):
|
||||
await _task(outbox, stop_condition="all items are ticked")
|
||||
await _bind(outbox)
|
||||
occurrence_id, run_id = await _occurrence(outbox, stop=True, reply="All 12 items are ticked")
|
||||
await _complete(outbox, occurrence_id, run_id)
|
||||
notices = await _notices(outbox)
|
||||
assert [row["event"] for row in notices] == ["task_stopped"]
|
||||
assert notices[0]["payload"]["stop_condition"] == "all items are ticked"
|
||||
assert (await _deliver_all(outbox))[0].splitlines()[1:3] == ["Paused by agent: its stop condition was met (all items are ticked).", "Result: All 12 items are ticked"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_completion_does_not_notify_for_manual_triggers():
|
||||
"""A manual "run now" happens with the user watching the UI; pushing the
|
||||
same outcome to IM is noise, so only scheduled firings notify."""
|
||||
task_repo = DummyTaskRepo([_once_task_row()])
|
||||
run_repo = DummyRunRepo()
|
||||
connection_repo = DummyConnectionRepo(
|
||||
[
|
||||
{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"},
|
||||
]
|
||||
)
|
||||
notification_repo = DummyNotificationRepo()
|
||||
service = _make_notifying_service(
|
||||
task_repo,
|
||||
run_repo,
|
||||
connection_repo=connection_repo,
|
||||
notification_repo=notification_repo,
|
||||
)
|
||||
|
||||
await service.handle_run_completion(_completion_record(RunStatus.success, trigger="manual"))
|
||||
|
||||
# Execution status is still recorded; only the IM push is suppressed.
|
||||
assert task_repo.completions[-1][1]["status"] == "success"
|
||||
assert notification_repo.enqueued == []
|
||||
async def test_max_runs_finish_sends_task_finished_with_summary_enrichment(outbox):
|
||||
await _task(outbox, max_runs=1)
|
||||
await _bind(outbox)
|
||||
occurrence_id, run_id = await _occurrence(outbox, reply="**Done:** the checklist is complete\nDetails…")
|
||||
await _complete(outbox, occurrence_id, run_id)
|
||||
notices = await _notices(outbox)
|
||||
assert [(row["event"], row["run_id"]) for row in notices] == [("task_finished", run_id)]
|
||||
assert (notices[0]["payload"]["reason_code"], notices[0]["payload"]["max_runs"], notices[0]["payload"]["run_status"]) == ("max_runs", 1, "success")
|
||||
(text,) = await _deliver_all(outbox)
|
||||
assert text.splitlines()[1:3] == ["Finished: its one automatic run is done.", "Result: Done: the checklist is complete"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_completion_still_notifies_when_trigger_metadata_is_missing():
|
||||
"""Fail safe: an untagged completion (older rows, unexpected paths) keeps
|
||||
notifying -- dropping a push silently is worse than a rare extra one."""
|
||||
task_repo = DummyTaskRepo([_once_task_row()])
|
||||
run_repo = DummyRunRepo()
|
||||
connection_repo = DummyConnectionRepo(
|
||||
[
|
||||
{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"},
|
||||
]
|
||||
)
|
||||
notification_repo = DummyNotificationRepo()
|
||||
service = _make_notifying_service(
|
||||
task_repo,
|
||||
run_repo,
|
||||
connection_repo=connection_repo,
|
||||
notification_repo=notification_repo,
|
||||
)
|
||||
|
||||
record = _completion_record(RunStatus.success)
|
||||
record.metadata.pop("scheduled_trigger")
|
||||
await service.handle_run_completion(record)
|
||||
|
||||
assert len(notification_repo.enqueued) == 1
|
||||
async def test_max_runs_finish_after_failed_run_says_last_run_failed(outbox):
|
||||
await _task(outbox, max_runs=1)
|
||||
await _bind(outbox)
|
||||
occurrence_id, run_id = await _occurrence(outbox, reply="partial work")
|
||||
await _complete(outbox, occurrence_id, run_id, status="failed", error="model provider timeout")
|
||||
notices = await _notices(outbox)
|
||||
assert [row["event"] for row in notices] == ["task_finished"]
|
||||
(text,) = await _deliver_all(outbox)
|
||||
assert text.splitlines()[1] == "Finished: its one automatic run is done. The last run failed."
|
||||
assert "partial work" not in text and "timeout" not in text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_completion_skips_unusable_connections():
|
||||
task_repo = DummyTaskRepo([_once_task_row()])
|
||||
run_repo = DummyRunRepo()
|
||||
connection_repo = DummyConnectionRepo(
|
||||
[
|
||||
{"provider": "wecom", "external_account_id": "revoked-user", "status": "revoked"},
|
||||
{"provider": "wecom", "external_account_id": None, "status": "connected"},
|
||||
{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"},
|
||||
]
|
||||
)
|
||||
notification_repo = DummyNotificationRepo()
|
||||
service = _make_notifying_service(
|
||||
task_repo,
|
||||
run_repo,
|
||||
connection_repo=connection_repo,
|
||||
notification_repo=notification_repo,
|
||||
)
|
||||
|
||||
await service.handle_run_completion(_completion_record(RunStatus.success))
|
||||
|
||||
assert [item["target"] for item in notification_repo.enqueued] == ["GaoZhiChao"]
|
||||
async def test_manual_trigger_sends_no_run_notice(outbox):
|
||||
await _task(outbox)
|
||||
await _bind(outbox)
|
||||
occurrence_id, run_id = await _occurrence(outbox, trigger="manual")
|
||||
assert await _complete(outbox, occurrence_id, run_id) is True
|
||||
assert await _notices(outbox) == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_completion_notification_failure_does_not_break_status_writes():
|
||||
task_repo = DummyTaskRepo([_once_task_row()])
|
||||
run_repo = DummyRunRepo()
|
||||
connection_repo = DummyConnectionRepo(
|
||||
[
|
||||
{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"},
|
||||
]
|
||||
)
|
||||
notification_repo = DummyNotificationRepo(fail=True)
|
||||
service = _make_notifying_service(
|
||||
task_repo,
|
||||
run_repo,
|
||||
connection_repo=connection_repo,
|
||||
notification_repo=notification_repo,
|
||||
)
|
||||
|
||||
await service.handle_run_completion(_completion_record(RunStatus.success))
|
||||
|
||||
# Execution status is written regardless: delivery is best-effort and must
|
||||
# never shadow the run outcome (execution/delivery status separation).
|
||||
assert task_repo.completions[-1][1]["status"] == "success"
|
||||
|
||||
|
||||
def _notifying_service_with_wecom(task_repo, notification_repo):
|
||||
connection_repo = DummyConnectionRepo([{"provider": "wecom", "external_account_id": "GaoZhiChao", "status": "connected"}])
|
||||
return _make_notifying_service(task_repo, DummyRunRepo(), connection_repo=connection_repo, notification_repo=notification_repo)
|
||||
async def test_manual_trial_that_finishes_task_sends_task_finished(outbox):
|
||||
await _task(outbox, end_at=NOTICE_NOW + timedelta(minutes=1))
|
||||
await _bind(outbox)
|
||||
occurrence_id, run_id = await _occurrence(outbox, trigger="manual")
|
||||
await _complete(outbox, occurrence_id, run_id, finished_at=NOTICE_NOW + timedelta(minutes=2))
|
||||
assert (await outbox.tasks.get("task-a", user_id=NOTICE_OWNER))["status"] == "completed"
|
||||
notices = await _notices(outbox)
|
||||
assert [(row["event"], row["payload"]["reason_code"]) for row in notices] == [("task_finished", "end_at")]
|
||||
assert (await _deliver_all(outbox))[0].splitlines()[1] == "Finished: its end time has been reached."
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_detached_outbox_stops_the_enqueue():
|
||||
# The Gateway detaches the outbox when the delivery side cannot start.
|
||||
task_repo = DummyTaskRepo([_once_task_row()])
|
||||
notification_repo = DummyNotificationRepo()
|
||||
service = _notifying_service_with_wecom(task_repo, notification_repo)
|
||||
|
||||
await service.handle_run_completion(_completion_record(RunStatus.success))
|
||||
assert [row["event"] for row in notification_repo.enqueued] == ["run_completed"]
|
||||
|
||||
service.detach_notification_outbox()
|
||||
await service.handle_run_completion(_completion_record(RunStatus.success))
|
||||
|
||||
assert len(notification_repo.enqueued) == 1
|
||||
assert len(task_repo.completions) == 2
|
||||
@pytest.mark.parametrize(
|
||||
("trigger", "status", "expected"), [("scheduled", "success", ["run_completed"]), ("scheduled", "failed", ["run_failed"]), ("scheduled", "unmet", ["run_unmet"]), ("scheduled", "interrupted", []), ("manual", "success", [])]
|
||||
)
|
||||
async def test_once_task_sends_only_its_run_event(outbox, trigger, status, expected):
|
||||
await _task(outbox, once=True)
|
||||
await _bind(outbox)
|
||||
occurrence_id, run_id = await _occurrence(outbox, trigger=trigger)
|
||||
await _complete(outbox, occurrence_id, run_id, status=status, error="goal_not_met_yet" if status == "unmet" else None)
|
||||
# A once task's own outcome is its notice; no separate task_finished.
|
||||
assert [row["event"] for row in await _notices(outbox)] == expected
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_completion_that_was_not_recorded_does_not_notify():
|
||||
# complete_run returns False for a missing occurrence, another run's occurrence or another owner.
|
||||
class RejectingTaskRepo(DummyTaskRepo):
|
||||
async def test_idle_finish_dedupe_key_is_deterministic_for_page_tasks(outbox):
|
||||
await _task(outbox, end_at=NOTICE_NOW + timedelta(hours=1))
|
||||
await _bind(outbox)
|
||||
later = NOTICE_NOW + timedelta(hours=2)
|
||||
assert await outbox.tasks.complete_if_ended("task-a", user_id=NOTICE_OWNER, now=later) is True
|
||||
await outbox.tasks.complete_if_ended("task-a", user_id=NOTICE_OWNER, now=later)
|
||||
notices = await _notices(outbox)
|
||||
assert [(row["event"], row["run_id"], row["payload"]["reason_code"]) for row in notices] == [("task_finished", None, "end_at")]
|
||||
key = notices[0]["task_run_id"]
|
||||
assert key.startswith("idle:") and len(key) <= 64
|
||||
from app.scheduler.service import _idle_notice_key
|
||||
from deerflow.utils.time import coerce_iso
|
||||
|
||||
assert key == _idle_notice_key("task-a", f"end:{coerce_iso(NOTICE_NOW + timedelta(hours=1))}")
|
||||
(text,) = await _deliver_all(outbox)
|
||||
assert "Result:" not in text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_locale_from_preference_then_config_fallback(outbox):
|
||||
await _bind(outbox)
|
||||
for index, preference in enumerate(("zh-CN", None, "fr-FR", {"value": "zh-CN"})):
|
||||
await _set_locale(outbox, preference)
|
||||
task_id = f"task-{index}"
|
||||
await _task(outbox, task_id)
|
||||
occurrence_id, run_id = await _occurrence(outbox, task_id, reply="Done")
|
||||
await _complete(outbox, occurrence_id, run_id, task_id=task_id)
|
||||
assert [row["payload"]["locale"] for row in await _notices(outbox)] == ["zh-CN", None, None, None]
|
||||
texts = await _deliver_all(outbox, default_locale="en-US")
|
||||
assert [text.splitlines()[-1] for text in texts] == ["在 DeerFlow 的定时任务页查看详情。"] + ["Open DeerFlow → Scheduled tasks for details."] * 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_locale_without_preference_follows_notification_locale(outbox):
|
||||
await _bind(outbox)
|
||||
await _task(outbox)
|
||||
occurrence_id, run_id = await _occurrence(outbox, reply="Done")
|
||||
await _complete(outbox, occurrence_id, run_id)
|
||||
(text,) = await _deliver_all(outbox, default_locale="zh-CN")
|
||||
assert text.splitlines() == ["定时任务“Check the release checklist”", "完成了一次运行。", "结果:Done", "在 DeerFlow 的定时任务页查看详情。"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_recovered_success_notifies_exactly_once(outbox):
|
||||
await _task(outbox)
|
||||
await _bind(outbox)
|
||||
occurrence_id, run_id = await _occurrence(outbox, durable_status="success")
|
||||
# The completion hook never ran (crash); lease reconciliation finalizes it.
|
||||
assert await outbox.runs.reconcile_active_runs(error="lease lost", now=NOTICE_NOW) == 1
|
||||
assert await outbox.runs.reconcile_active_runs(error="lease lost", now=NOTICE_NOW) == 0
|
||||
await outbox.service.handle_run_completion(
|
||||
RunRecord(
|
||||
run_id=run_id,
|
||||
thread_id="thread-task-a-1",
|
||||
assistant_id="lead_agent",
|
||||
status=RunStatus.success,
|
||||
on_disconnect=DisconnectMode.continue_,
|
||||
metadata={"scheduled_task_id": "task-a", "scheduled_task_run_id": occurrence_id, "scheduled_trigger": "scheduled"},
|
||||
user_id=NOTICE_OWNER,
|
||||
)
|
||||
)
|
||||
assert [(row["event"], row["task_run_id"]) for row in await _notices(outbox)] == [("run_completed", occurrence_id)]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_detached_outbox_stops_the_enqueue(outbox):
|
||||
await _task(outbox)
|
||||
await _bind(outbox)
|
||||
occurrence_id, run_id = await _occurrence(outbox)
|
||||
await _complete(outbox, occurrence_id, run_id)
|
||||
outbox.service.detach_notification_outbox()
|
||||
occurrence_id, run_id = await _occurrence(outbox, suffix="2")
|
||||
await _complete(outbox, occurrence_id, run_id)
|
||||
assert [row["task_run_id"] for row in await _notices(outbox)] == ["task-run-task-a-1"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("completed", [True, False])
|
||||
async def test_handle_run_completion_no_longer_enqueues_outside_transaction(completed):
|
||||
"""Only the finalization observer enqueues; the completion hook just records the outcome."""
|
||||
|
||||
class RecordingTaskRepo(DummyTaskRepo):
|
||||
reads = 0
|
||||
|
||||
async def complete_run(self, task_id, **kwargs):
|
||||
await super().complete_run(task_id, **kwargs)
|
||||
return False
|
||||
|
||||
task_repo = RejectingTaskRepo([_once_task_row()])
|
||||
notification_repo = DummyNotificationRepo()
|
||||
|
||||
await _notifying_service_with_wecom(task_repo, notification_repo).handle_run_completion(_completion_record(RunStatus.success))
|
||||
|
||||
assert len(task_repo.completions) == 1
|
||||
assert notification_repo.enqueued == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_completion_for_a_task_deleted_meanwhile_does_not_notify():
|
||||
task_repo = DummyTaskRepo([])
|
||||
notification_repo = DummyNotificationRepo()
|
||||
|
||||
await _notifying_service_with_wecom(task_repo, notification_repo).handle_run_completion(_completion_record(RunStatus.success))
|
||||
|
||||
assert task_repo.completions[-1][1]["status"] == "success"
|
||||
assert notification_repo.enqueued == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_completion_still_notifies_when_the_task_read_fails():
|
||||
class UnreadableTaskRepo(DummyTaskRepo):
|
||||
async def get(self, task_id, *, user_id):
|
||||
raise RuntimeError("database unavailable")
|
||||
|
||||
task_repo = UnreadableTaskRepo([_once_task_row()])
|
||||
notification_repo = DummyNotificationRepo()
|
||||
|
||||
await _notifying_service_with_wecom(task_repo, notification_repo).handle_run_completion(_completion_record(RunStatus.success))
|
||||
|
||||
assert task_repo.completions[-1][1]["status"] == "success"
|
||||
# The title is optional: the push still goes out and falls back to the task id.
|
||||
assert [row["payload"]["task_title"] for row in notification_repo.enqueued] == [None]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_completion_without_the_outbox_does_not_read_the_task():
|
||||
class CountingTaskRepo(DummyTaskRepo):
|
||||
reads = 0
|
||||
return completed
|
||||
|
||||
async def get(self, task_id, *, user_id):
|
||||
self.reads += 1
|
||||
return await super().get(task_id, user_id=user_id)
|
||||
|
||||
task_repo = CountingTaskRepo([_once_task_row()])
|
||||
service = _make_notifying_service(task_repo, DummyRunRepo(), connection_repo=None, notification_repo=None)
|
||||
task_repo = RecordingTaskRepo([_once_task_row()])
|
||||
notification_repo, connection_repo = DummyNotificationRepo(), DummyConnectionRepo()
|
||||
service = ScheduledTaskService(task_repo=task_repo, task_run_repo=DummyRunRepo(), launch_run=None, poll_interval_seconds=5, lease_seconds=120, max_concurrent_runs=3, connection_repo=connection_repo, notification_repo=notification_repo)
|
||||
|
||||
await service.handle_run_completion(_completion_record(RunStatus.success))
|
||||
for status in (RunStatus.success, RunStatus.error, RunStatus.interrupted):
|
||||
await service.handle_run_completion(_completion_record(status, error="boom" if status == RunStatus.error else None))
|
||||
|
||||
assert task_repo.completions[-1][1]["status"] == "success"
|
||||
assert task_repo.reads == 0
|
||||
assert [completion["status"] for _task_id, completion in task_repo.completions] == ["success", "failed", "interrupted"]
|
||||
assert notification_repo.enqueued == []
|
||||
assert connection_repo.reads == 0 and task_repo.reads == 0
|
||||
assert not hasattr(service, "_enqueue_run_notifications")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@ -6,18 +6,20 @@ from datetime import UTC, datetime, timedelta
|
||||
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
|
||||
from deerflow.persistence.run import RunRepository
|
||||
from deerflow.persistence.run.model import RunRow
|
||||
from deerflow.persistence.scheduled_task_runs import ScheduledTaskRunRepository
|
||||
from deerflow.persistence.scheduled_task_events import ScheduledTaskEventRow
|
||||
from deerflow.persistence.scheduled_task_runs import ScheduledTaskAdmissionRejected, ScheduledTaskRunRepository
|
||||
from deerflow.persistence.scheduled_task_runs.model import ScheduledTaskRunRow
|
||||
from deerflow.persistence.scheduled_tasks import ScheduledTaskLimitsExhausted, ScheduledTaskRepository
|
||||
from deerflow.persistence.scheduled_tasks.model import ScheduledTaskRow
|
||||
from deerflow.persistence.scheduled_tasks.sql import ActiveScheduledTaskMutationConflict
|
||||
from deerflow.persistence.thread_meta.model import ThreadMetaRow
|
||||
|
||||
NOW = datetime(2026, 10, 3, 8, tzinfo=UTC)
|
||||
VERDICT = {"satisfied": False, "blocker": "goal_not_met_yet", "reason": "missing evidence", "evidence_summary": "none", "relied_on_assumption": False, "stand_down_reason": "max_continuations"}
|
||||
@ -47,7 +49,7 @@ async def database(tmp_path, *, backend="sqlite"):
|
||||
async with engine.begin() as conn:
|
||||
if schema:
|
||||
await conn.execute(text(f'CREATE SCHEMA "{schema}"'))
|
||||
await conn.run_sync(lambda sync: Base.metadata.create_all(sync, tables=[ScheduledTaskRow.__table__, ScheduledTaskRunRow.__table__, RunRow.__table__]))
|
||||
await conn.run_sync(lambda sync: Base.metadata.create_all(sync, tables=[ScheduledTaskRow.__table__, ScheduledTaskRunRow.__table__, RunRow.__table__, ThreadMetaRow.__table__, ScheduledTaskEventRow.__table__]))
|
||||
sf = async_sessionmaker(engine, expire_on_commit=False)
|
||||
yield sf, ScheduledTaskRepository(sf), ScheduledTaskRunRepository(sf)
|
||||
finally:
|
||||
@ -503,3 +505,276 @@ async def test_terminal_pause_cannot_race_a_create_out_of_its_quota_slot(tmp_pat
|
||||
assert results[0] == "finished"
|
||||
assert isinstance(results[1], dict)
|
||||
assert sum(row["status"] in {"enabled", "running", "paused"} for row in await tasks.list_by_origin_thread("owner", "origin")) == 20
|
||||
|
||||
|
||||
# --- Lifecycle events (task_stopped / task_paused / task_finished) ----------------------
|
||||
|
||||
|
||||
async def lifecycle_task(sf, tasks, runs, *, once=False, **extra):
|
||||
"""The task (``once`` makes it a once task) and its originating chat, with the
|
||||
scheduler's real finalization observer installed; returns each recorded call."""
|
||||
from app.scheduler.service import ScheduledTaskService
|
||||
|
||||
await task(tasks, **extra)
|
||||
if once:
|
||||
await tasks.update("task", user_id="owner", updates={"schedule_type": "once", "schedule_spec": {"run_at": NOW.isoformat()}})
|
||||
async with sf() as session:
|
||||
session.add(ThreadMetaRow(thread_id="origin", user_id="owner"))
|
||||
await session.commit()
|
||||
service = ScheduledTaskService(task_repo=tasks, task_run_repo=runs, launch_run=None, poll_interval_seconds=60, lease_seconds=30, max_concurrent_runs=3)
|
||||
observed = []
|
||||
|
||||
async def observer(session, parent, row, *, events):
|
||||
observed.append((row.id if row is not None else None, events))
|
||||
await service._on_finalization(session, parent, row, events=events)
|
||||
|
||||
for repository in (tasks, runs):
|
||||
repository.set_finalization_observer(observer)
|
||||
return observed
|
||||
|
||||
|
||||
def lifecycle_calls(observed):
|
||||
return [(occurrence_id, tuple(event for event in events if event.startswith("task_"))) for occurrence_id, events in observed if any(event.startswith("task_") for event in events)]
|
||||
|
||||
|
||||
async def event_rows(sf):
|
||||
async with sf() as session:
|
||||
return list((await session.execute(select(ScheduledTaskEventRow).order_by(ScheduledTaskEventRow.created_at, ScheduledTaskEventRow.id))).scalars())
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_agent_stop_emits_task_stopped_once(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
observed = await lifecycle_task(sf, tasks, runs)
|
||||
occurrence_id, run_id = await occurrence(sf, runs, durable_status="running")
|
||||
assert await runs.request_stop(occurrence_id, task_id="task", run_id=run_id, user_id="owner")
|
||||
assert await complete(tasks, occurrence_id, run_id) is True
|
||||
assert observed == [(occurrence_id, ("run_completed", "task_stopped"))]
|
||||
assert await complete(tasks, occurrence_id, run_id) is False
|
||||
assert len(observed) == 1
|
||||
rows = await event_rows(sf)
|
||||
assert [(row.event, row.reason_code, row.anchor, row.occurrence_id, row.thread_id) for row in rows] == [("task_stopped", "agent_stop", occurrence_id, occurrence_id, "origin")]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_max_runs_completion_emits_task_finished(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
observed = await lifecycle_task(sf, tasks, runs, max_runs=2)
|
||||
for suffix in ("1", "2"):
|
||||
occurrence_id, run_id = await occurrence(sf, runs, suffix=suffix)
|
||||
await complete(tasks, occurrence_id, run_id)
|
||||
assert observed == [("occ-1", ("run_completed",)), ("occ-2", ("run_completed", "task_finished"))]
|
||||
assert (await tasks.get("task", user_id="owner"))["status"] == "completed"
|
||||
rows = await event_rows(sf)
|
||||
assert [(row.event, row.reason_code, row.anchor) for row in rows] == [("task_finished", "max_runs", "occ-2")]
|
||||
assert rows[0].payload_json["max_runs"] == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_end_condition_wins_over_agent_stop_events(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
observed = await lifecycle_task(sf, tasks, runs, max_runs=1)
|
||||
occurrence_id, run_id = await occurrence(sf, runs, durable_status="running")
|
||||
assert await runs.request_stop(occurrence_id, task_id="task", run_id=run_id, user_id="owner")
|
||||
await complete(tasks, occurrence_id, run_id)
|
||||
assert observed == [(occurrence_id, ("run_completed", "task_finished"))]
|
||||
assert [row.event for row in await event_rows(sf)] == ["task_finished"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_idle_end_at_finish_emits_with_end_anchor_once(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
observed = await lifecycle_task(sf, tasks, runs, end_at=NOW)
|
||||
assert await tasks.complete_if_ended("task", user_id="owner", now=NOW + timedelta(seconds=1)) is True
|
||||
assert await tasks.complete_if_ended("task", user_id="owner", now=NOW + timedelta(seconds=2)) is True
|
||||
assert observed == [(None, ("task_finished",))]
|
||||
rows = await event_rows(sf)
|
||||
assert [(row.event, row.reason_code, row.anchor, row.occurrence_id) for row in rows] == [("task_finished", "end_at", f"end:{NOW.isoformat()}", None)]
|
||||
assert (await tasks.get("task", user_id="owner"))["status"] == "completed"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_idle_max_runs_finish_then_end_time_finish_writes_two_events(tmp_path, database_backend):
|
||||
"""An idle max_runs finish before the end time must not take the end-time
|
||||
anchor: after a reactivation, the real end-time finish still gets its line."""
|
||||
end = NOW + timedelta(hours=1)
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
observed = await lifecycle_task(sf, tasks, runs, end_at=end, max_runs=1)
|
||||
await runs.create(run_record_id="used", task_id="task", thread_id="thread", scheduled_for=NOW, trigger="scheduled", status="queued")
|
||||
async with sf() as session:
|
||||
row = await session.get(ScheduledTaskRunRow, "used")
|
||||
row.status, row.launch_accounted, row.finished_at = "success", True, NOW
|
||||
await session.commit()
|
||||
assert await tasks.complete_if_ended("task", user_id="owner", now=NOW + timedelta(seconds=1)) is True
|
||||
async with sf() as session:
|
||||
parent = await session.get(ScheduledTaskRow, "task")
|
||||
parent.status, parent.max_runs = "enabled", 5
|
||||
await session.commit()
|
||||
assert await tasks.complete_if_ended("task", user_id="owner", now=end + timedelta(seconds=1)) is True
|
||||
assert observed == [(None, ("task_finished",)), (None, ("task_finished",))]
|
||||
rows = await event_rows(sf)
|
||||
assert [(row.event, row.reason_code, row.anchor) for row in rows] == [
|
||||
("task_finished", "max_runs", "seq:1"),
|
||||
("task_finished", "end_at", f"end:{end.isoformat()}"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_claim_time_end_skip_emits_task_finished_once(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
observed = await lifecycle_task(sf, tasks, runs, end_at=NOW + timedelta(seconds=1))
|
||||
await runs.create(run_record_id="waiting", task_id="task", thread_id="thread", scheduled_for=NOW, trigger="scheduled", status="queued")
|
||||
assert await runs.claim_queued_run("waiting", lease_owner="worker", now=NOW + timedelta(seconds=2), lease_seconds=30, global_max_concurrent_runs=1) is None
|
||||
assert lifecycle_calls(observed) == [("waiting", ("task_finished",))]
|
||||
assert len(observed) == 1
|
||||
rows = await event_rows(sf)
|
||||
assert [(row.event, row.reason_code, row.anchor) for row in rows] == [("task_finished", "end_at", "waiting")]
|
||||
assert (await tasks.get("task", user_id="owner"))["status"] == "completed"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_complete_if_ended_with_queued_row_emits_once(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
observed = await lifecycle_task(sf, tasks, runs, end_at=NOW + timedelta(seconds=1))
|
||||
await runs.create(run_record_id="waiting", task_id="task", thread_id="thread", scheduled_for=NOW, trigger="scheduled", status="queued")
|
||||
assert await tasks.complete_if_ended("task", user_id="owner", now=NOW + timedelta(seconds=2)) is True
|
||||
assert observed == [("waiting", ("task_finished",))]
|
||||
assert [(row.event, row.anchor) for row in await event_rows(sf)] == [("task_finished", "waiting")]
|
||||
assert (await runs.list_by_task("task"))[0]["status"] == "skipped"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_complete_if_ended_with_queued_manual_row_emits_idle_finish(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
observed = await lifecycle_task(sf, tasks, runs, end_at=NOW + timedelta(seconds=1))
|
||||
await runs.create(run_record_id="trial", task_id="task", thread_id="thread", scheduled_for=NOW, trigger="manual", status="queued")
|
||||
assert await tasks.complete_if_ended("task", user_id="owner", now=NOW + timedelta(seconds=2)) is True
|
||||
assert observed == [(None, ("task_finished",))]
|
||||
rows = await event_rows(sf)
|
||||
assert [(row.event, row.anchor, row.occurrence_id) for row in rows] == [("task_finished", f"end:{(NOW + timedelta(seconds=1)).isoformat()}", None)]
|
||||
# The waiting trial is left alone; only scheduled rows are skipped.
|
||||
assert (await runs.list_by_task("task"))[0]["status"] == "queued"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_once_task_end_skip_stays_silent(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
observed = await lifecycle_task(sf, tasks, runs, once=True, end_at=NOW + timedelta(seconds=1))
|
||||
await runs.create(run_record_id="waiting", task_id="task", thread_id="thread", scheduled_for=NOW, trigger="scheduled", status="queued")
|
||||
assert await runs.claim_queued_run("waiting", lease_owner="worker", now=NOW + timedelta(seconds=2), lease_seconds=30, global_max_concurrent_runs=1) is None
|
||||
# As in PR1: skipped -> cancelled, then the end condition marks it completed.
|
||||
assert (await tasks.get("task", user_id="owner"))["status"] == "completed"
|
||||
assert lifecycle_calls(observed) == []
|
||||
assert await event_rows(sf) == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize(
|
||||
("status", "error", "reason"),
|
||||
[("success", None, "once_done"), ("failed", "boom", "once_failed"), ("unmet", "goal_not_met_yet", "once_failed"), ("interrupted", "cancelled", None), ("skipped", None, None)],
|
||||
)
|
||||
async def test_once_task_finish_reasons(tmp_path, database_backend, status, error, reason):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
observed = await lifecycle_task(sf, tasks, runs, once=True)
|
||||
occurrence_id, run_id = await occurrence(sf, runs)
|
||||
assert await complete(tasks, occurrence_id, run_id, status=status, error=error) is True
|
||||
rows = await event_rows(sf)
|
||||
if reason is None:
|
||||
assert lifecycle_calls(observed) == []
|
||||
assert rows == []
|
||||
else:
|
||||
assert lifecycle_calls(observed) == [(occurrence_id, ("task_finished",))]
|
||||
assert [(row.event, row.reason_code, row.anchor) for row in rows] == [("task_finished", reason, occurrence_id)]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_admission_time_end_emits_task_finished(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
observed = await lifecycle_task(sf, tasks, runs, end_at=NOW - timedelta(seconds=1))
|
||||
with pytest.raises(ScheduledTaskAdmissionRejected) as rejected:
|
||||
await runs.create(run_record_id="admitted", task_id="task", thread_id="thread", scheduled_for=NOW, trigger="scheduled", status="queued", coordinate_with_task=True, expected_task_user_id="owner")
|
||||
assert rejected.value.reason == "ended"
|
||||
assert (await tasks.get("task", user_id="owner"))["status"] == "completed"
|
||||
assert observed == [(None, ("task_finished",))]
|
||||
rows = await event_rows(sf)
|
||||
assert [(row.event, row.reason_code, row.occurrence_id) for row in rows] == [("task_finished", "end_at", None)]
|
||||
assert await runs.list_by_task("task") == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_trial_that_reaches_end_at_emits_task_finished(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
observed = await lifecycle_task(sf, tasks, runs, end_at=NOW)
|
||||
occurrence_id, run_id = await occurrence(sf, runs, suffix="trial", trigger="manual")
|
||||
await complete(tasks, occurrence_id, run_id)
|
||||
# Finalization reports the run event too; the observer drops only that for a trial.
|
||||
assert observed == [(occurrence_id, ("run_completed", "task_finished"))]
|
||||
assert [(row.event, row.reason_code, row.anchor) for row in await event_rows(sf)] == [("task_finished", "end_at", occurrence_id)]
|
||||
assert (await tasks.get("task", user_id="owner"))["status"] == "completed"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_manual_trial_agent_stop_emits_task_stopped(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
observed = await lifecycle_task(sf, tasks, runs)
|
||||
occurrence_id, run_id = await occurrence(sf, runs, suffix="trial", trigger="manual", durable_status="running")
|
||||
assert await runs.request_stop(occurrence_id, task_id="task", run_id=run_id, user_id="owner")
|
||||
await complete(tasks, occurrence_id, run_id)
|
||||
assert observed == [(occurrence_id, ("run_completed", "task_stopped"))]
|
||||
assert [row.event for row in await event_rows(sf)] == ["task_stopped"]
|
||||
assert (await tasks.get("task", user_id="owner"))["status"] == "paused"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_agent_stop_after_failed_run_pauses_and_says_run_failed(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, tasks, runs):
|
||||
observed = await lifecycle_task(sf, tasks, runs)
|
||||
occurrence_id, run_id = await occurrence(sf, runs, durable_status="running")
|
||||
assert await runs.request_stop(occurrence_id, task_id="task", run_id=run_id, user_id="owner")
|
||||
await complete(tasks, occurrence_id, run_id, status="failed", error="boom")
|
||||
# The transition is PR1's: a stop request pauses whatever the run outcome.
|
||||
stopped = await tasks.get("task", user_id="owner")
|
||||
assert stopped["status"] == "paused"
|
||||
assert stopped["last_error"] == f"stopped by the agent in run {run_id}"
|
||||
assert observed == [(occurrence_id, ("run_failed", "task_stopped"))]
|
||||
(row,) = await event_rows(sf)
|
||||
assert (row.event, row.payload_json["run_status"]) == ("task_stopped", "failed")
|
||||
|
||||
|
||||
def test_no_inline_end_condition_completion():
|
||||
"""Only finalize_occurrence and finish_task_at_end_condition finish a task.
|
||||
|
||||
The once-task repair in ``_associate_task_with_run`` mirrors a once run's
|
||||
own outcome (not an end condition) and is the single allowed assignment.
|
||||
"""
|
||||
import ast
|
||||
from pathlib import Path
|
||||
|
||||
from deerflow.persistence.scheduled_task_runs import sql as run_sql
|
||||
from deerflow.persistence.scheduled_tasks import sql as task_sql
|
||||
|
||||
class Finder(ast.NodeVisitor):
|
||||
def __init__(self):
|
||||
self.stack = []
|
||||
self.found = []
|
||||
|
||||
def _visit_function(self, node):
|
||||
self.stack.append(node.name)
|
||||
self.generic_visit(node)
|
||||
self.stack.pop()
|
||||
|
||||
visit_FunctionDef = _visit_function
|
||||
visit_AsyncFunctionDef = _visit_function
|
||||
|
||||
def visit_Assign(self, node):
|
||||
targets_status = any(isinstance(target, ast.Attribute) and target.attr == "status" for target in node.targets)
|
||||
if targets_status and isinstance(node.value, ast.Constant) and node.value.value == "completed":
|
||||
self.found.append(self.stack[-1] if self.stack else "<module>")
|
||||
self.generic_visit(node)
|
||||
|
||||
found = []
|
||||
for module in (run_sql, task_sql):
|
||||
finder = Finder()
|
||||
finder.visit(ast.parse(Path(module.__file__).read_text(encoding="utf-8")))
|
||||
found += [(module.__name__.rsplit(".", 2)[-2], name) for name in finder.found]
|
||||
assert found == [("scheduled_task_runs", "_associate_task_with_run")]
|
||||
|
||||
@ -139,20 +139,18 @@ async def test_auto_pause_and_notice_share_the_completion_transaction(scheduler,
|
||||
|
||||
def test_unmet_and_pause_notifications_have_distinct_safe_text():
|
||||
unmet = render_notification_text({"event": "run_unmet", "task_id": "task-a", "payload": {"reason_code": "blocked:needs_user_input", "error": "private raw failure"}})
|
||||
paused = render_notification_text({"event": "task_paused", "task_id": "task-a", "payload": {"reason_code": "consecutive_unmet"}})
|
||||
assert "goal was not met" in unmet
|
||||
assert "Reason: it needs your input." in unmet
|
||||
paused = render_notification_text({"event": "task_paused", "task_id": "task-a", "payload": {"reason_code": "consecutive_unmet", "latest_reason_code": "blocked:needs_user_input"}})
|
||||
assert "Ran, but the goal wasn't met: it needs your input." in unmet
|
||||
assert "needs_user_input" not in unmet
|
||||
assert "private raw failure" not in unmet
|
||||
assert "automatically paused" in paused
|
||||
assert "Reason: 3 scheduled runs in a row did not meet the goal." in paused
|
||||
assert "Paused after 3 runs in a row missed the goal: it needs your input." in paused
|
||||
assert "consecutive_unmet" not in paused
|
||||
assert "completed" not in unmet + paused
|
||||
assert "Finished a run" not in unmet + paused
|
||||
|
||||
|
||||
def test_assumption_marker_is_visible_without_forwarding_raw_error_text():
|
||||
text = render_notification_text({"event": "run_completed", "task_id": "task-a", "payload": {"relied_on_assumption": True, "result_summary": "Report is ready"}})
|
||||
assert "met, relying on stated assumptions" in text
|
||||
assert "Goal met, with an assumption." in text
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
@ -203,13 +201,14 @@ async def test_external_wait_does_not_advance_unmet_streak(scheduler, outcomes):
|
||||
)
|
||||
def test_known_host_stand_down_reasons_are_reported_readably(reason, readable):
|
||||
text = render_notification_text({"event": "run_unmet", "task_id": "task-a", "payload": {"reason_code": reason}})
|
||||
assert f"Reason: {readable}." in text
|
||||
assert f"Ran, but the goal wasn't met: {readable}." in text
|
||||
assert reason not in text
|
||||
|
||||
|
||||
def test_unknown_reason_code_is_not_forwarded():
|
||||
text = render_notification_text({"event": "run_unmet", "task_id": "task-a", "payload": {"reason_code": "provider said: secret"}})
|
||||
assert "Reason: unknown." in text
|
||||
# An unrecognized code adds nothing: the reason-less sentence.
|
||||
assert "Ran, but the goal wasn't met." in text.splitlines()
|
||||
assert "secret" not in text
|
||||
|
||||
|
||||
|
||||
@ -11,6 +11,8 @@ def test_scheduler_config_defaults():
|
||||
|
||||
assert config.enabled is False
|
||||
assert config.recursion_limit == 1000
|
||||
assert config.max_concurrent_runs == 3
|
||||
assert config.max_concurrent_runs_per_user == 2
|
||||
|
||||
|
||||
def test_scheduler_config_accepts_positive_recursion_limit():
|
||||
@ -24,3 +26,14 @@ def test_scheduler_config_rejects_non_positive_recursion_limit():
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
SchedulerConfig(recursion_limit=-5)
|
||||
|
||||
|
||||
def test_scheduler_config_per_user_cap_bounds():
|
||||
assert SchedulerConfig(max_concurrent_runs_per_user=0).max_concurrent_runs_per_user == 0
|
||||
assert SchedulerConfig(max_concurrent_runs_per_user=32).max_concurrent_runs_per_user == 32
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
SchedulerConfig(max_concurrent_runs_per_user=-1)
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
SchedulerConfig(max_concurrent_runs_per_user=33)
|
||||
|
||||
@ -18,6 +18,7 @@ SCHEDULER_INT_FIELDS = (
|
||||
"poll_interval_seconds",
|
||||
"lease_seconds",
|
||||
"max_concurrent_runs",
|
||||
"max_concurrent_runs_per_user",
|
||||
"queue_timeout_seconds",
|
||||
"min_once_delay_seconds",
|
||||
"recursion_limit",
|
||||
|
||||
21
backend/tests/test_sqlite_capabilities.py
Normal file
21
backend/tests/test_sqlite_capabilities.py
Normal file
@ -0,0 +1,21 @@
|
||||
"""The bundled SQLite must support what the scheduler's SQL relies on.
|
||||
|
||||
The fair drain (``ScheduledTaskRunRepository.list_queued_runs``) ranks queued
|
||||
rows per owner with ``ROW_NUMBER() OVER (PARTITION BY ...)``, which SQLite
|
||||
supports from 3.25. An older build should fail CI loudly here rather than at
|
||||
runtime on the first poll.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
|
||||
|
||||
def test_sqlite_supports_window_functions():
|
||||
assert sqlite3.sqlite_version_info >= (3, 25), f"SQLite {sqlite3.sqlite_version} lacks window functions (needs >= 3.25)"
|
||||
connection = sqlite3.connect(":memory:")
|
||||
try:
|
||||
connection.execute("CREATE TABLE t (owner TEXT, n INTEGER)")
|
||||
connection.executemany("INSERT INTO t VALUES (?, ?)", [("a", 1), ("a", 2), ("b", 1)])
|
||||
rows = connection.execute("SELECT owner, n, ROW_NUMBER() OVER (PARTITION BY owner ORDER BY n) FROM t ORDER BY owner, n").fetchall()
|
||||
finally:
|
||||
connection.close()
|
||||
assert rows == [("a", 1, 1), ("a", 2, 2), ("b", 1, 1)]
|
||||
119
backend/tests/test_sse_response_headers.py
Normal file
119
backend/tests/test_sse_response_headers.py
Normal file
@ -0,0 +1,119 @@
|
||||
"""Every SSE response sends ``Cache-Control: no-cache, no-transform``.
|
||||
|
||||
Compressing proxies (Next's rewrite proxy behind ``pnpm start``, and others)
|
||||
buffer a response they compress, so a plain ``no-cache`` stream reaches the
|
||||
browser in bursts. ``no-transform`` asks them to leave the body alone. These
|
||||
tests pin the header on all five SSE routes, keep ``Content-Location`` on the
|
||||
two routes that create a run, and fail if a route goes back to a literal
|
||||
header dict.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from _router_auth_helpers import make_authed_test_app
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.gateway.routers import runs, thread_runs
|
||||
from app.gateway.sse_headers import SSE_CACHE_CONTROL, sse_response_headers
|
||||
from deerflow.runtime import RunManager, RunStatus
|
||||
from deerflow.runtime.stream_bridge import MemoryStreamBridge
|
||||
|
||||
THREAD_ID = "thread-sse-headers"
|
||||
ROUTERS_DIR = Path(__file__).resolve().parents[1] / "app" / "gateway" / "routers"
|
||||
|
||||
|
||||
async def _one_frame(*_args, **_kwargs):
|
||||
yield "event: end\ndata: null\n\n"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(monkeypatch: pytest.MonkeyPatch) -> tuple[TestClient, str]:
|
||||
mgr = RunManager()
|
||||
|
||||
async def _seed():
|
||||
record = await mgr.create(THREAD_ID)
|
||||
await mgr.set_status(record.run_id, RunStatus.success)
|
||||
return record
|
||||
|
||||
record = asyncio.run(_seed())
|
||||
|
||||
async def _start_run(*_args, **_kwargs):
|
||||
return record
|
||||
|
||||
for module in (thread_runs, runs):
|
||||
monkeypatch.setattr(module, "sse_consumer", _one_frame)
|
||||
monkeypatch.setattr(module, "start_run", _start_run)
|
||||
|
||||
app = make_authed_test_app()
|
||||
app.include_router(thread_runs.router)
|
||||
app.include_router(runs.router)
|
||||
app.state.run_manager = mgr
|
||||
app.state.stream_bridge = MemoryStreamBridge()
|
||||
with TestClient(app, raise_server_exceptions=False) as test_client:
|
||||
yield test_client, record.run_id
|
||||
|
||||
|
||||
def _stream(test_client: TestClient, method: str, url: str, **kwargs):
|
||||
with test_client.stream(method, url, **kwargs) as response:
|
||||
body = "".join(response.iter_text())
|
||||
return response, body
|
||||
|
||||
|
||||
def _assert_sse_headers(response) -> None:
|
||||
assert response.status_code == 200
|
||||
assert response.headers["content-type"].startswith("text/event-stream")
|
||||
assert response.headers["cache-control"] == "no-cache, no-transform"
|
||||
assert response.headers["x-accel-buffering"] == "no"
|
||||
assert response.headers["connection"] == "keep-alive"
|
||||
|
||||
|
||||
def test_helper_shape():
|
||||
assert SSE_CACHE_CONTROL == "no-cache, no-transform"
|
||||
assert sse_response_headers() == {
|
||||
"Cache-Control": "no-cache, no-transform",
|
||||
"Connection": "keep-alive",
|
||||
"X-Accel-Buffering": "no",
|
||||
}
|
||||
assert sse_response_headers(content_location="/api/threads/t/runs/r")["Content-Location"] == "/api/threads/t/runs/r"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("method", "path", "json_body", "creates_run"),
|
||||
[
|
||||
("POST", f"/api/threads/{THREAD_ID}/runs/stream", {"input": {}}, True),
|
||||
("POST", "/api/runs/stream", {"input": {}, "config": {"configurable": {"thread_id": THREAD_ID}}}, True),
|
||||
("GET", f"/api/threads/{THREAD_ID}/runs/{{run_id}}/join", None, False),
|
||||
("GET", f"/api/threads/{THREAD_ID}/runs/{{run_id}}/stream", None, False),
|
||||
("POST", f"/api/threads/{THREAD_ID}/runs/{{run_id}}/stream", None, False),
|
||||
],
|
||||
ids=["thread-run-create", "stateless-create", "join", "existing-run-GET", "existing-run-POST"],
|
||||
)
|
||||
def test_sse_route_headers(client, method: str, path: str, json_body: dict | None, creates_run: bool):
|
||||
test_client, run_id = client
|
||||
kwargs = {} if json_body is None else {"json": json_body}
|
||||
response, body = _stream(test_client, method, path.format(run_id=run_id), **kwargs)
|
||||
_assert_sse_headers(response)
|
||||
if creates_run:
|
||||
# Only the two routes that create a run point at it.
|
||||
assert response.headers["content-location"] == f"/api/threads/{THREAD_ID}/runs/{run_id}"
|
||||
else:
|
||||
assert "content-location" not in response.headers
|
||||
assert "event: end" in body
|
||||
|
||||
|
||||
def test_no_literal_no_cache_header_left_in_routers():
|
||||
pattern = re.compile(r"""["']Cache-Control["']\s*:\s*["']no-cache["']""")
|
||||
offenders = [str(path.relative_to(ROUTERS_DIR)) for path in sorted(ROUTERS_DIR.rglob("*.py")) if pattern.search(path.read_text(encoding="utf-8"))]
|
||||
assert offenders == []
|
||||
|
||||
|
||||
def test_every_event_stream_route_uses_the_helper():
|
||||
for path in sorted(ROUTERS_DIR.rglob("*.py")):
|
||||
source = path.read_text(encoding="utf-8")
|
||||
streams = source.count('media_type="text/event-stream"')
|
||||
assert source.count("headers=sse_response_headers(") == streams, path.name
|
||||
221
backend/tests/test_thread_activity_router.py
Normal file
221
backend/tests/test_thread_activity_router.py
Normal file
@ -0,0 +1,221 @@
|
||||
"""``GET /api/thread-activity``: the per-user activity feed over the run-change clock.
|
||||
|
||||
Drives the real router against real SQLite repositories (``RunRepository``
|
||||
for the clock, ``ThreadReadRepository`` for the read clock), with runs written
|
||||
through ``RunRepository.put`` so ``origin_kind`` is derived as in production.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import asynccontextmanager
|
||||
from uuid import UUID
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from _router_auth_helpers import make_authed_test_app
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
|
||||
from app.gateway.auth.models import User
|
||||
from app.gateway.deps import get_config
|
||||
from app.gateway.routers import features, thread_activity, threads
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.run import RunRepository
|
||||
from deerflow.persistence.run.model import RunChangeClockRow, RunRow
|
||||
from deerflow.persistence.thread_reads import ThreadReadMarkerRow, ThreadReadRepository, ThreadReadVersionRow
|
||||
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY, make_origin
|
||||
from deerflow.runtime.runs.store.memory import MemoryRunStore
|
||||
|
||||
_ALICE = User(id=UUID("2a8c4f53-6a0b-4a51-9f5e-0c1d2e3f4a51"), email="alice@example.com", password_hash="unused", system_role="user")
|
||||
_BOB = User(id=UUID("7d1e2f30-4b5c-4d6e-8f70-81a2b3c4d5e6"), email="bob@example.com", password_hash="unused", system_role="user")
|
||||
ALICE = str(_ALICE.id)
|
||||
BOB = str(_BOB.id)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def repositories(tmp_path):
|
||||
engine = create_async_engine(f"sqlite+aiosqlite:///{tmp_path / 'activity.db'}")
|
||||
try:
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(lambda sync: Base.metadata.create_all(sync, tables=[m.__table__ for m in (RunRow, RunChangeClockRow, ThreadReadMarkerRow, ThreadReadVersionRow)]))
|
||||
sf = async_sessionmaker(engine, expire_on_commit=False)
|
||||
yield RunRepository(sf), ThreadReadRepository(sf)
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def client_for(run_store, read_repo, *, user=_ALICE):
|
||||
app = make_authed_test_app(user_factory=lambda: user, bind_current_user=True)
|
||||
app.state.run_store = run_store
|
||||
app.state.thread_read_repo = read_repo
|
||||
app.include_router(thread_activity.router)
|
||||
app.include_router(threads.router)
|
||||
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://activity.test") as client:
|
||||
yield client
|
||||
|
||||
|
||||
async def put_run(runs, run_id, thread_id, *, user_id=ALICE, origin=None, status="success"):
|
||||
metadata = {DEERFLOW_ORIGIN_KEY: make_origin(origin)} if origin else {}
|
||||
await runs.put(run_id, thread_id=thread_id, user_id=user_id, status=status, metadata=metadata)
|
||||
|
||||
|
||||
async def poll(client, cursor=None, **params):
|
||||
if cursor is not None:
|
||||
params["cursor"] = cursor
|
||||
response = await client.get("/api/thread-activity", params=params)
|
||||
assert response.status_code == 200, response.text
|
||||
return response.json()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_first_poll_only_seeds_at_the_callers_head(tmp_path):
|
||||
async with repositories(tmp_path) as (runs, reads), client_for(runs, reads) as client:
|
||||
await put_run(runs, "run-1", "sched", origin="schedule")
|
||||
seeded = await poll(client)
|
||||
assert seeded["threads"] == []
|
||||
assert seeded["truncated"] is False
|
||||
assert seeded["read_version"] == 0
|
||||
latest = await runs.latest_change(user_id=ALICE)
|
||||
assert seeded["cursor"] == f"{latest[0]}:{latest[1]}"
|
||||
# Nothing changed since the seed.
|
||||
again = await poll(client, seeded["cursor"])
|
||||
assert again["threads"] == [] and again["cursor"] == seeded["cursor"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_user_without_runs_seeds_at_zero_and_sees_the_first_scheduled_run(tmp_path):
|
||||
async with repositories(tmp_path) as (runs, reads), client_for(runs, reads) as client:
|
||||
seeded = await poll(client)
|
||||
assert seeded["cursor"] == "0:"
|
||||
await put_run(runs, "run-1", "sched", origin="schedule", status="running")
|
||||
page = await poll(client, seeded["cursor"])
|
||||
assert page["threads"] == [{"thread_id": "sched", "origin_kind": "schedule", "status": "running"}]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cursor_advances_to_the_last_returned_row_without_skips_or_repeats(tmp_path):
|
||||
async with repositories(tmp_path) as (runs, reads), client_for(runs, reads) as client:
|
||||
cursor = (await poll(client))["cursor"]
|
||||
for index in range(5):
|
||||
await put_run(runs, f"run-{index}", f"thread-{index}", origin="schedule")
|
||||
seen: list[str] = []
|
||||
truncated_flags = []
|
||||
for _ in range(3):
|
||||
page = await poll(client, cursor, limit=2)
|
||||
seen.extend(item["thread_id"] for item in page["threads"])
|
||||
truncated_flags.append(page["truncated"])
|
||||
if page["threads"]:
|
||||
last = await runs.get(f"run-{page['threads'][-1]['thread_id'].split('-')[1]}", user_id=None)
|
||||
assert page["cursor"] == f"{last['change_seq']}:{last['run_id']}"
|
||||
cursor = page["cursor"]
|
||||
assert seen == [f"thread-{index}" for index in range(5)]
|
||||
assert truncated_flags == [True, True, False]
|
||||
assert (await poll(client, cursor))["threads"] == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_interactive_changes_advance_the_cursor_but_are_not_returned(tmp_path):
|
||||
async with repositories(tmp_path) as (runs, reads), client_for(runs, reads) as client:
|
||||
cursor = (await poll(client))["cursor"]
|
||||
await put_run(runs, "chat-1", "chat", status="pending")
|
||||
await put_run(runs, "chat-1", "chat", status="running")
|
||||
await put_run(runs, "chat-1", "chat", status="success")
|
||||
page = await poll(client, cursor)
|
||||
assert page["threads"] == []
|
||||
assert page["cursor"] != cursor
|
||||
latest = await runs.latest_change(user_id=ALICE)
|
||||
assert page["cursor"] == f"{latest[0]}:{latest[1]}"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_new_server_runs_are_listed_once_per_thread_with_latest_status(tmp_path):
|
||||
async with repositories(tmp_path) as (runs, reads), client_for(runs, reads) as client:
|
||||
cursor = (await poll(client))["cursor"]
|
||||
await put_run(runs, "run-s", "sched", origin="schedule", status="running")
|
||||
await put_run(runs, "run-im", "im", origin="im_channel", status="running")
|
||||
await put_run(runs, "run-s", "sched", origin="schedule", status="success")
|
||||
page = await poll(client, cursor)
|
||||
assert page["threads"] == [
|
||||
{"thread_id": "im", "origin_kind": "im_channel", "status": "running"},
|
||||
{"thread_id": "sched", "origin_kind": "schedule", "status": "success"},
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_other_users_runs_never_appear_even_with_their_cursor(tmp_path):
|
||||
async with repositories(tmp_path) as (runs, reads):
|
||||
async with client_for(runs, reads, user=_BOB) as bob:
|
||||
bob_cursor = (await poll(bob))["cursor"]
|
||||
await put_run(runs, "run-bob", "bob-thread", user_id=BOB, origin="schedule")
|
||||
# Bob's run in a thread shared with Alice.
|
||||
await put_run(runs, "run-bob-shared", "shared", user_id=BOB, origin="im_channel")
|
||||
async with client_for(runs, reads) as alice:
|
||||
assert (await poll(alice, bob_cursor))["threads"] == []
|
||||
assert (await poll(alice, "0:"))["threads"] == []
|
||||
assert (await poll(alice))["cursor"] == "0:"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("cursor", ["", "abc", "12", "-1:run", "1:run:x", "01:run", "1: run", "9" * 20 + ":r", f"{2**63}:r", "1:" + "r" * 65, "1:" + "r" * 200])
|
||||
async def test_malformed_cursor_is_rejected(tmp_path, cursor):
|
||||
async with repositories(tmp_path) as (runs, reads), client_for(runs, reads) as client:
|
||||
response = await client.get("/api/thread-activity", params={"cursor": cursor})
|
||||
assert response.status_code == 422
|
||||
assert response.json()["detail"]["code"] == "invalid_cursor"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_largest_change_seq_is_a_valid_cursor(tmp_path):
|
||||
async with repositories(tmp_path) as (runs, reads), client_for(runs, reads) as client:
|
||||
cursor = f"{2**63 - 1}:r"
|
||||
page = await poll(client, cursor)
|
||||
assert page["threads"] == [] and page["cursor"] == cursor
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_read_version_moves_only_when_a_read_clears_something(tmp_path):
|
||||
async with repositories(tmp_path) as (runs, reads), client_for(runs, reads) as client:
|
||||
await put_run(runs, "run-1", "sched", origin="schedule")
|
||||
cursor = (await poll(client))["cursor"]
|
||||
|
||||
cleared = await client.post("/api/threads/sched/read")
|
||||
assert cleared.status_code == 200, cleared.text
|
||||
assert cleared.json() == {"unread": False, "read_version": 1}
|
||||
page = await poll(client, cursor)
|
||||
assert page["read_version"] == 1
|
||||
assert page["threads"] == []
|
||||
|
||||
again = await client.post("/api/threads/sched/read")
|
||||
assert again.json() == {"unread": False, "read_version": 1}
|
||||
assert (await poll(client, page["cursor"]))["read_version"] == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_memory_backend_reports_unavailable(tmp_path):
|
||||
app = make_authed_test_app(bind_current_user=True)
|
||||
app.state.run_store = MemoryRunStore()
|
||||
app.state.thread_read_repo = None
|
||||
app.include_router(thread_activity.router)
|
||||
app.include_router(threads.router)
|
||||
app.include_router(features.router)
|
||||
app.dependency_overrides[get_config] = lambda: _feature_config()
|
||||
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://activity.test") as client:
|
||||
assert (await client.get("/api/thread-activity")).status_code == 503
|
||||
assert (await client.post("/api/threads/some-thread/read")).status_code == 503
|
||||
response = await client.get("/api/features")
|
||||
assert response.status_code == 200
|
||||
assert response.json()["thread_activity"] == {"available": False}
|
||||
|
||||
|
||||
def _feature_config():
|
||||
from types import SimpleNamespace
|
||||
|
||||
config = SimpleNamespace(
|
||||
agents_api=SimpleNamespace(enabled=False),
|
||||
tools=[],
|
||||
subagent_runtime=SimpleNamespace(max_running=3),
|
||||
knowledge_base=SimpleNamespace(enabled=False, scope_selection_enabled=False),
|
||||
scheduler=SimpleNamespace(enabled=False, tool_enabled=False, min_once_delay_seconds=60),
|
||||
)
|
||||
config.get_tool_config = lambda _name: None
|
||||
return config
|
||||
@ -622,6 +622,26 @@ class TestThreadMetaRepository:
|
||||
assert row is not None and row.project_id is None
|
||||
assert "deerflow_project_id" not in row.metadata_json
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_run_admission_names_a_new_thread_from_its_input_title(self, repo):
|
||||
"""A scheduled run names its fresh thread in the input. The thread is
|
||||
created with that name, so a sidebar listing it at once shows it; the
|
||||
worker would copy the title only when the run ends."""
|
||||
from app.gateway.services import _ensure_thread_metadata
|
||||
from deerflow.persistence.thread_meta.model import ThreadMetaRow
|
||||
from deerflow.runtime.runs.manager import RunRecord
|
||||
from deerflow.runtime.runs.schemas import DisconnectMode, RunStatus
|
||||
from deerflow.runtime.runs.worker import RunContext
|
||||
|
||||
run_ctx = RunContext(checkpointer=None, thread_store=repo)
|
||||
cases = [("t1", {"title": "Checklist · 10-07 09:00", "messages": []}, "Checklist · 10-07 09:00"), ("t2", {"messages": []}, None), ("t3", {"title": " "}, None)]
|
||||
for thread_id, run_input, expected in cases:
|
||||
record = RunRecord(run_id=f"run-{thread_id}", thread_id=thread_id, assistant_id="lead-agent", status=RunStatus.pending, on_disconnect=DisconnectMode.cancel, kwargs={"input": run_input})
|
||||
await _ensure_thread_metadata(run_ctx, record, owner_user_id=None)
|
||||
async with repo._sf() as session:
|
||||
row = await session.get(ThreadMetaRow, thread_id)
|
||||
assert row is not None and row.display_name == expected
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_create_with_project_assignment_and_rejection(self, repo):
|
||||
from deerflow.persistence.projects import ProjectNotAssignableError, ProjectRepository
|
||||
|
||||
216
backend/tests/test_thread_reads_repository.py
Normal file
216
backend/tests/test_thread_reads_repository.py
Normal file
@ -0,0 +1,216 @@
|
||||
"""Per-user unread state over the run-change clock (``ThreadReadRepository``).
|
||||
|
||||
A thread is unread for a user while one of that user's server-originated runs
|
||||
in it changed after the user last read it. Runs are written through the real
|
||||
``RunRepository`` so ``runs.origin_kind`` is derived exactly as in production.
|
||||
Runs on SQLite, and on Postgres with ``TEST_POSTGRES_URI``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import uuid
|
||||
from contextlib import asynccontextmanager
|
||||
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.postgres_schema import build_asyncpg_connect_args
|
||||
from deerflow.persistence.run import RunRepository
|
||||
from deerflow.persistence.run.model import RunChangeClockRow, RunRow
|
||||
from deerflow.persistence.thread_reads import ThreadReadMarkerRow, ThreadReadRepository, ThreadReadVersionRow
|
||||
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY, make_origin
|
||||
|
||||
ALICE = "user-alice"
|
||||
BOB = "user-bob"
|
||||
_TABLES = [RunRow, RunChangeClockRow, ThreadReadMarkerRow, ThreadReadVersionRow]
|
||||
|
||||
|
||||
@pytest.fixture(params=["sqlite", "postgres"])
|
||||
def database_backend(request):
|
||||
return request.param
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def database(tmp_path, *, backend="sqlite"):
|
||||
schema = None
|
||||
if backend == "postgres":
|
||||
uri = os.environ.get("TEST_POSTGRES_URI")
|
||||
if not uri:
|
||||
pytest.skip("requires TEST_POSTGRES_URI (real Postgres upsert path)")
|
||||
parts = urlsplit(uri)
|
||||
scheme = "postgresql+asyncpg" if parts.scheme in {"postgres", "postgresql"} else parts.scheme
|
||||
query = urlencode([(key, value) for key, value in parse_qsl(parts.query, keep_blank_values=True) if key not in {"sslmode", "channel_binding"}])
|
||||
schema = f"thread_reads_{uuid.uuid4().hex}"
|
||||
engine = create_async_engine(urlunsplit(parts._replace(scheme=scheme, query=query)), connect_args=build_asyncpg_connect_args(schema))
|
||||
else:
|
||||
engine = create_async_engine(f"sqlite+aiosqlite:///{tmp_path / 'thread-reads.db'}")
|
||||
try:
|
||||
async with engine.begin() as conn:
|
||||
if schema:
|
||||
await conn.execute(text(f'CREATE SCHEMA "{schema}"'))
|
||||
await conn.run_sync(lambda sync: Base.metadata.create_all(sync, tables=[model.__table__ for model in _TABLES]))
|
||||
sf = async_sessionmaker(engine, expire_on_commit=False)
|
||||
yield sf, RunRepository(sf), ThreadReadRepository(sf)
|
||||
finally:
|
||||
if schema:
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(text(f'DROP SCHEMA IF EXISTS "{schema}" CASCADE'))
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
async def put_run(runs, run_id, thread_id, *, user_id=ALICE, origin=None, status="success"):
|
||||
"""Write a run as RunManager does; ``origin`` is the server-stamped origin kind."""
|
||||
metadata = {DEERFLOW_ORIGIN_KEY: make_origin(origin)} if origin else {}
|
||||
await runs.put(run_id, thread_id=thread_id, user_id=user_id, status=status, metadata=metadata)
|
||||
|
||||
|
||||
async def marker(sf, user_id, thread_id):
|
||||
async with sf() as session:
|
||||
return await session.get(ThreadReadMarkerRow, (user_id, thread_id))
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_server_origin_run_is_unread_until_read_and_again_after_a_new_run(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (_sf, runs, reads):
|
||||
await put_run(runs, "run-1", "sched-thread", origin="schedule")
|
||||
assert await reads.unread_thread_ids(user_id=ALICE, thread_ids=["sched-thread"]) == {"sched-thread"}
|
||||
|
||||
await reads.mark_read(user_id=ALICE, thread_id="sched-thread")
|
||||
assert await reads.unread_thread_ids(user_id=ALICE, thread_ids=["sched-thread"]) == set()
|
||||
|
||||
await put_run(runs, "run-2", "sched-thread", origin="schedule", status="running")
|
||||
assert await reads.unread_thread_ids(user_id=ALICE, thread_ids=["sched-thread"]) == {"sched-thread"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_interactive_and_legacy_runs_never_mark_unread(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, runs, reads):
|
||||
await put_run(runs, "run-chat", "chat-thread")
|
||||
# A legacy row written before 0032: no origin metadata, origin_kind NULL.
|
||||
async with sf() as session:
|
||||
session.add(RunRow(run_id="run-legacy", thread_id="legacy-thread", user_id=ALICE, status="success", change_seq=999, metadata_json={}))
|
||||
await session.commit()
|
||||
assert await reads.unread_thread_ids(user_id=ALICE, thread_ids=["chat-thread", "legacy-thread"]) == set()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_every_server_origin_kind_marks_unread(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (_sf, runs, reads):
|
||||
for kind in ("schedule", "im_channel", "github", "extension", "mcp_notification"):
|
||||
await put_run(runs, f"run-{kind}", f"thread-{kind}", origin=kind)
|
||||
ids = [f"thread-{kind}" for kind in ("schedule", "im_channel", "github", "extension", "mcp_notification")]
|
||||
assert await reads.unread_thread_ids(user_id=ALICE, thread_ids=ids) == set(ids)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mark_read_never_lowers_seen_change_seq(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, runs, reads):
|
||||
await put_run(runs, "run-1", "thread", origin="schedule")
|
||||
first = await reads.mark_read(user_id=ALICE, thread_id="thread")
|
||||
await put_run(runs, "run-2", "thread", origin="schedule")
|
||||
second = await reads.mark_read(user_id=ALICE, thread_id="thread")
|
||||
assert second > first
|
||||
|
||||
# A stale read (another device that computed an older position) loses.
|
||||
async with sf() as session:
|
||||
await session.execute(RunRow.__table__.delete().where(RunRow.run_id == "run-2"))
|
||||
await session.commit()
|
||||
assert await reads.mark_read(user_id=ALICE, thread_id="thread") == second
|
||||
assert (await marker(sf, ALICE, "thread")).seen_change_seq == second
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_concurrent_reads_keep_the_maximum(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, runs, reads):
|
||||
await put_run(runs, "run-1", "thread", origin="schedule")
|
||||
await put_run(runs, "run-2", "thread", origin="schedule")
|
||||
results = await asyncio.gather(*(reads.mark_read(user_id=ALICE, thread_id="thread") for _ in range(4)))
|
||||
stored = (await marker(sf, ALICE, "thread")).seen_change_seq
|
||||
assert set(results) == {stored}
|
||||
# Exactly one of the four raised the marker, so the clock moved once.
|
||||
assert await reads.read_version(user_id=ALICE) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mark_read_on_an_already_read_thread_writes_nothing(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, runs, reads):
|
||||
assert await reads.read_version(user_id=ALICE) == 0
|
||||
await put_run(runs, "run-1", "thread", origin="schedule")
|
||||
await reads.mark_read(user_id=ALICE, thread_id="thread")
|
||||
assert await reads.read_version(user_id=ALICE) == 1
|
||||
before = await marker(sf, ALICE, "thread")
|
||||
|
||||
await reads.mark_read(user_id=ALICE, thread_id="thread")
|
||||
after = await marker(sf, ALICE, "thread")
|
||||
assert after.updated_at == before.updated_at
|
||||
assert after.seen_change_seq == before.seen_change_seq
|
||||
assert await reads.read_version(user_id=ALICE) == 1
|
||||
|
||||
# A real raise increments the clock by exactly one.
|
||||
await put_run(runs, "run-2", "thread", origin="schedule")
|
||||
await reads.mark_read(user_id=ALICE, thread_id="thread")
|
||||
assert await reads.read_version(user_id=ALICE) == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mark_read_without_runs_writes_no_marker(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, _runs, reads):
|
||||
assert await reads.mark_read(user_id=ALICE, thread_id="empty") == 0
|
||||
assert await marker(sf, ALICE, "empty") is None
|
||||
assert await reads.read_version(user_id=ALICE) == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_shared_thread_other_users_runs_do_not_mark_unread(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (_sf, runs, reads):
|
||||
# Bob's schedule and IM runs in a thread Alice can also see.
|
||||
await put_run(runs, "run-bob-1", "shared", user_id=BOB, origin="schedule")
|
||||
await put_run(runs, "run-bob-2", "shared", user_id=BOB, origin="im_channel")
|
||||
assert await reads.unread_thread_ids(user_id=ALICE, thread_ids=["shared"]) == set()
|
||||
assert await reads.unread_thread_ids(user_id=BOB, thread_ids=["shared"]) == {"shared"}
|
||||
|
||||
# Alice's read does not clear Bob's state, and vice versa.
|
||||
await reads.mark_read(user_id=ALICE, thread_id="shared")
|
||||
assert await reads.unread_thread_ids(user_id=BOB, thread_ids=["shared"]) == {"shared"}
|
||||
await reads.mark_read(user_id=BOB, thread_id="shared")
|
||||
assert await reads.unread_thread_ids(user_id=BOB, thread_ids=["shared"]) == set()
|
||||
assert await reads.read_version(user_id=ALICE) == 0
|
||||
assert await reads.read_version(user_id=BOB) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_thread_operations_never_carry_an_origin(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, runs, reads):
|
||||
await runs.put("op-1", thread_id="thread", user_id=ALICE, operation_kind="checkpoint_write", metadata={DEERFLOW_ORIGIN_KEY: make_origin("schedule")})
|
||||
async with sf() as session:
|
||||
assert (await session.get(RunRow, "op-1")).origin_kind is None
|
||||
assert await reads.unread_thread_ids(user_id=ALICE, thread_ids=["thread"]) == set()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unread_query_handles_empty_and_unknown_ids(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (_sf, runs, reads):
|
||||
await put_run(runs, "run-1", "thread", origin="schedule")
|
||||
assert await reads.unread_thread_ids(user_id=ALICE, thread_ids=[]) == set()
|
||||
assert await reads.unread_thread_ids(user_id=ALICE, thread_ids=["missing", "thread", "thread"]) == {"thread"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_by_thread_drops_every_users_marker(tmp_path, database_backend):
|
||||
async with database(tmp_path, backend=database_backend) as (sf, runs, reads):
|
||||
await put_run(runs, "run-a", "thread", origin="schedule")
|
||||
await put_run(runs, "run-b", "thread", user_id=BOB, origin="schedule")
|
||||
await put_run(runs, "run-c", "other", origin="schedule")
|
||||
await reads.mark_read(user_id=ALICE, thread_id="thread")
|
||||
await reads.mark_read(user_id=BOB, thread_id="thread")
|
||||
await reads.mark_read(user_id=ALICE, thread_id="other")
|
||||
|
||||
await reads.delete_by_thread("thread")
|
||||
async with sf() as session:
|
||||
remaining = (await session.execute(select(ThreadReadMarkerRow.user_id, ThreadReadMarkerRow.thread_id))).all()
|
||||
assert [tuple(row) for row in remaining] == [(ALICE, "other")]
|
||||
@ -4830,3 +4830,152 @@ def test_task_notes_state_write_normalizes_and_replaces(monkeypatch, mode, fallb
|
||||
read = client.get("/api/threads/note-replacement/state")
|
||||
assert read.status_code == 200, read.text
|
||||
assert read.json()["values"]["task_notes"] == {"new": {"content": "keep backups", "source_ids": [], "authority": "model_report"}}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Server-owned origin marker and per-user unread state (deerflow_origin)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_strip_reserved_metadata_drops_client_origin():
|
||||
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY
|
||||
|
||||
assert DEERFLOW_ORIGIN_KEY in threads._SERVER_RESERVED_METADATA_KEYS
|
||||
assert threads._strip_reserved_metadata({DEERFLOW_ORIGIN_KEY: {"kind": "schedule"}, "keep": 1}) == {"keep": 1}
|
||||
|
||||
|
||||
def test_create_and_patch_strip_a_client_supplied_origin() -> None:
|
||||
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY
|
||||
|
||||
app, _store, checkpointer = _build_thread_app()
|
||||
forged = {DEERFLOW_ORIGIN_KEY: {"kind": "schedule"}, "title": "mine"}
|
||||
|
||||
with TestClient(app) as client:
|
||||
created = client.post("/api/threads", json={"thread_id": "origin-forge", "metadata": forged})
|
||||
assert created.status_code == 200, created.text
|
||||
assert created.json()["metadata"] == {"title": "mine"}
|
||||
patched = client.patch("/api/threads/origin-forge", json={"metadata": {DEERFLOW_ORIGIN_KEY: {"kind": "im_channel"}}})
|
||||
assert patched.status_code == 200, patched.text
|
||||
assert DEERFLOW_ORIGIN_KEY not in patched.json()["metadata"]
|
||||
fetched = client.post("/api/threads/search", json={"limit": 10})
|
||||
assert all(DEERFLOW_ORIGIN_KEY not in item["metadata"] for item in fetched.json())
|
||||
|
||||
checkpoint = asyncio.run(checkpointer.aget_tuple({"configurable": {"thread_id": "origin-forge", "checkpoint_ns": ""}}))
|
||||
assert DEERFLOW_ORIGIN_KEY not in checkpoint.metadata
|
||||
|
||||
|
||||
def test_create_thread_marks_a_host_origin_from_request_state() -> None:
|
||||
"""Only server code can set request.state.run_origin (an extension handle)."""
|
||||
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY
|
||||
|
||||
app, _store, _checkpointer = _build_thread_app()
|
||||
|
||||
@app.middleware("http")
|
||||
async def host_origin(request, call_next):
|
||||
request.state.run_origin = {"kind": "extension", "namespace": "community.teams"}
|
||||
return await call_next(request)
|
||||
|
||||
with TestClient(app) as client:
|
||||
response = client.post("/api/threads", json={"thread_id": "ext-thread", "metadata": {"title": "t"}})
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json()["metadata"] == {"title": "t", DEERFLOW_ORIGIN_KEY: {"kind": "extension", "namespace": "community.teams"}}
|
||||
|
||||
|
||||
def _unread_app(tmp_path, *, owner_check_passes: bool = True):
|
||||
from uuid import UUID
|
||||
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
|
||||
|
||||
from deerflow.persistence.base import Base
|
||||
from deerflow.persistence.run import RunRepository
|
||||
from deerflow.persistence.run.model import RunChangeClockRow, RunRow
|
||||
from deerflow.persistence.thread_reads import ThreadReadMarkerRow, ThreadReadRepository, ThreadReadVersionRow
|
||||
|
||||
user = User(id=UUID("31f5a8c2-1d4e-4b6f-8a9c-0d1e2f3a4b5c"), email="unread@example.com", password_hash="x", system_role="user")
|
||||
app = make_authed_test_app(user_factory=lambda: user, owner_check_passes=owner_check_passes, bind_current_user=True)
|
||||
store = InMemoryStore()
|
||||
app.state.store = store
|
||||
app.state.checkpointer = InMemorySaver()
|
||||
app.state.run_manager = _ThreadTestRunManager()
|
||||
app.state.thread_store = _PermissiveThreadMetaStore(store)
|
||||
engine = create_async_engine(f"sqlite+aiosqlite:///{tmp_path / 'unread.db'}")
|
||||
|
||||
async def _init():
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(lambda sync: Base.metadata.create_all(sync, tables=[m.__table__ for m in (RunRow, RunChangeClockRow, ThreadReadMarkerRow, ThreadReadVersionRow)]))
|
||||
|
||||
asyncio.run(_init())
|
||||
sf = async_sessionmaker(engine, expire_on_commit=False)
|
||||
app.state.thread_read_repo = ThreadReadRepository(sf)
|
||||
app.include_router(threads.router)
|
||||
return app, store, RunRepository(sf), str(user.id), engine
|
||||
|
||||
|
||||
def test_search_items_carry_unread_until_the_thread_is_read(tmp_path) -> None:
|
||||
from deerflow.runtime.run_origin import DEERFLOW_ORIGIN_KEY
|
||||
|
||||
app, store, runs, user_id, engine = _unread_app(tmp_path)
|
||||
|
||||
async def _seed():
|
||||
for thread_id, metadata in (("sched-thread", {DEERFLOW_ORIGIN_KEY: {"kind": "schedule"}}), ("chat-thread", {})):
|
||||
await store.aput(THREADS_NS, thread_id, {"thread_id": thread_id, "status": "idle", "created_at": "2026-10-06T00:00:00+00:00", "updated_at": "2026-10-06T00:00:00+00:00", "metadata": metadata})
|
||||
await runs.put("run-sched", thread_id="sched-thread", user_id=user_id, status="success", metadata={DEERFLOW_ORIGIN_KEY: {"kind": "schedule"}})
|
||||
await runs.put("run-chat", thread_id="chat-thread", user_id=user_id, status="success", metadata={})
|
||||
|
||||
asyncio.run(_seed())
|
||||
try:
|
||||
with TestClient(app) as client:
|
||||
items = {item["thread_id"]: item for item in client.post("/api/threads/search", json={"limit": 10}).json()}
|
||||
assert items["sched-thread"]["unread"] is True
|
||||
assert items["chat-thread"]["unread"] is False
|
||||
|
||||
read = client.post("/api/threads/sched-thread/read")
|
||||
assert read.status_code == 200, read.text
|
||||
assert read.json() == {"unread": False, "read_version": 1}
|
||||
items = {item["thread_id"]: item for item in client.post("/api/threads/search", json={"limit": 10}).json()}
|
||||
assert items["sched-thread"]["unread"] is False
|
||||
finally:
|
||||
asyncio.run(engine.dispose())
|
||||
|
||||
|
||||
def test_search_unread_is_null_without_sql_read_state() -> None:
|
||||
app, store, _checkpointer = _build_thread_app()
|
||||
asyncio.run(store.aput(THREADS_NS, "t", {"thread_id": "t", "status": "idle", "created_at": "2026-10-06T00:00:00+00:00", "updated_at": "2026-10-06T00:00:00+00:00", "metadata": {}}))
|
||||
with TestClient(app) as client:
|
||||
assert [item["unread"] for item in client.post("/api/threads/search", json={"limit": 10}).json()] == [None]
|
||||
|
||||
|
||||
def test_mark_read_on_another_users_thread_is_404(tmp_path) -> None:
|
||||
app, _store, _runs, user_id, engine = _unread_app(tmp_path, owner_check_passes=False)
|
||||
try:
|
||||
with TestClient(app) as client:
|
||||
response = client.post("/api/threads/someone-elses/read")
|
||||
assert response.status_code == 404
|
||||
assert asyncio.run(app.state.thread_read_repo.read_version(user_id=user_id)) == 0
|
||||
finally:
|
||||
asyncio.run(engine.dispose())
|
||||
|
||||
|
||||
def test_delete_thread_route_removes_read_markers(tmp_path) -> None:
|
||||
app, store, runs, user_id, engine = _unread_app(tmp_path)
|
||||
|
||||
async def _seed():
|
||||
await store.aput(THREADS_NS, "gone", {"thread_id": "gone", "status": "idle", "created_at": "2026-10-06T00:00:00+00:00", "updated_at": "2026-10-06T00:00:00+00:00", "metadata": {}})
|
||||
await runs.put("run-gone", thread_id="gone", user_id=user_id, status="success", metadata={"deerflow_origin": {"kind": "schedule"}})
|
||||
await app.state.thread_read_repo.mark_read(user_id=user_id, thread_id="gone")
|
||||
|
||||
asyncio.run(_seed())
|
||||
app.state.run_event_store = MagicMock(delete_by_thread=AsyncMock())
|
||||
app.state.run_store = runs
|
||||
try:
|
||||
with patch("app.gateway.routers.threads.get_paths", return_value=Paths(tmp_path)), TestClient(app) as client:
|
||||
assert client.delete("/api/threads/gone").status_code == 200
|
||||
from deerflow.persistence.thread_reads import ThreadReadMarkerRow
|
||||
|
||||
async def _count():
|
||||
async with app.state.thread_read_repo._sf() as session:
|
||||
return await session.get(ThreadReadMarkerRow, (user_id, "gone"))
|
||||
|
||||
assert asyncio.run(_count()) is None
|
||||
finally:
|
||||
asyncio.run(engine.dispose())
|
||||
|
||||
@ -59,7 +59,7 @@ class _StubTaskRepo:
|
||||
|
||||
|
||||
class _StubRunRepo:
|
||||
async def list_queued_runs(self, *, limit):
|
||||
async def list_queued_runs(self, *, limit, **_kwargs):
|
||||
return []
|
||||
|
||||
async def expire_queued_runs(self, **_kwargs):
|
||||
|
||||
@ -78,14 +78,14 @@ async def test_preferences_api_partial_patch_reset_and_owner_isolation(api):
|
||||
path = "/api/v1/auth/preferences"
|
||||
assert (await client.patch(path, json={"notification_enabled": False, "mode": "pro"})).status_code == 204
|
||||
assert (await client.patch(path, json={"model_name": "model-a"})).status_code == 204
|
||||
assert (await client.get(path)).json() == {"notification_enabled": False, "model_name": "model-a", "mode": "pro", "reasoning_effort": None}
|
||||
assert (await client.get(path)).json() == {"notification_enabled": False, "model_name": "model-a", "mode": "pro", "reasoning_effort": None, "locale": None}
|
||||
assert (await client.patch(path, json={"mode": None})).status_code == 204
|
||||
identity.id = "bob"
|
||||
# A stale tab with Alice's expected identity must not write using Bob's cookie.
|
||||
assert (await client.patch(path, json={"mode": "ultra"})).status_code == 409
|
||||
assert (await client.get(path)).status_code == 409
|
||||
client.headers["X-Expected-User-Id"] = "bob"
|
||||
assert (await client.get(path)).json() == {"notification_enabled": None, "model_name": None, "mode": None, "reasoning_effort": None}
|
||||
assert (await client.get(path)).json() == {"notification_enabled": None, "model_name": None, "mode": None, "reasoning_effort": None, "locale": None}
|
||||
identity.id = "alice"
|
||||
client.headers["X-Expected-User-Id"] = "alice"
|
||||
assert (await client.get(path)).json()["notification_enabled"] is False
|
||||
@ -93,7 +93,20 @@ async def test_preferences_api_partial_patch_reset_and_owner_isolation(api):
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
@pytest.mark.parametrize("body", [{"notification_enabled": "false"}, {"mode": "invalid"}, {"reasoning_effort": "Not Valid!"}, {"model_name": "a" * 201}, {"context": {"github_token": "not-a-real-token"}}, {"user_id": "bob"}])
|
||||
@pytest.mark.parametrize(
|
||||
"body",
|
||||
[
|
||||
{"notification_enabled": "false"},
|
||||
{"mode": "invalid"},
|
||||
{"locale": "fr-FR"},
|
||||
{"locale": "zh"},
|
||||
{"locale": 1},
|
||||
{"reasoning_effort": "Not Valid!"},
|
||||
{"model_name": "a" * 201},
|
||||
{"context": {"github_token": "not-a-real-token"}},
|
||||
{"user_id": "bob"},
|
||||
],
|
||||
)
|
||||
async def test_preferences_api_rejects_invalid_and_unrelated_fields(api, body):
|
||||
client, _ = api
|
||||
assert (await client.patch("/api/v1/auth/preferences", json=body)).status_code == 422
|
||||
@ -118,11 +131,24 @@ async def test_preferences_requires_authentication_and_expected_identity(api):
|
||||
assert (await client.get("/api/v1/auth/preferences")).status_code == 401
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
@pytest.mark.parametrize("locale", ["en-US", "zh-CN"])
|
||||
async def test_preferences_api_stores_the_ui_locale(api, preference_repo, locale):
|
||||
# The web app syncs its UI language here; scheduled-task IM notices use it.
|
||||
client, _ = api
|
||||
path = "/api/v1/auth/preferences"
|
||||
assert (await client.patch(path, json={"locale": locale})).status_code == 204
|
||||
assert (await client.get(path)).json()["locale"] == locale
|
||||
assert (await preference_repo.get("alice"))["locale"] == locale
|
||||
assert (await client.patch(path, json={"locale": None})).status_code == 204
|
||||
assert (await client.get(path)).json()["locale"] is None
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
async def test_preferences_read_discards_only_malformed_fields(api, preference_repo):
|
||||
client, _ = api
|
||||
await preference_repo.patch("alice", {"notification_enabled": False, "mode": "invalid", "unknown": "ignored"})
|
||||
assert (await client.get("/api/v1/auth/preferences")).json() == {"notification_enabled": False, "model_name": None, "mode": None, "reasoning_effort": None}
|
||||
await preference_repo.patch("alice", {"notification_enabled": False, "mode": "invalid", "locale": "fr-FR", "unknown": "ignored"})
|
||||
assert (await client.get("/api/v1/auth/preferences")).json() == {"notification_enabled": False, "model_name": None, "mode": None, "reasoning_effort": None, "locale": None}
|
||||
|
||||
|
||||
def test_preferences_migration_preserves_existing_users_and_downgrades(tmp_path):
|
||||
|
||||
@ -20,7 +20,7 @@
|
||||
# ============================================================================
|
||||
# Bump this number when the config schema changes.
|
||||
# Run `make config-upgrade` to merge new fields into your local config.yaml.
|
||||
config_version: 53
|
||||
config_version: 55
|
||||
|
||||
# Optional operator-owned literal prompt extensions (see CONFIGURATION.md):
|
||||
# lead_prompt_overlay:
|
||||
@ -2690,20 +2690,38 @@ agent_storage:
|
||||
# ============================================================================
|
||||
# Background scheduler for one-time, cron, and interval agent runs.
|
||||
# Poller fields (enabled, multi_instance, poll_interval_seconds, lease_seconds,
|
||||
# max_concurrent_runs, min_once_delay_seconds) are restart-required.
|
||||
# max_concurrent_runs, max_concurrent_runs_per_user, queue_timeout_seconds,
|
||||
# min_once_delay_seconds) are restart-required.
|
||||
# recursion_limit is read at dispatch and applies to the next scheduled run
|
||||
# without a Gateway restart. The default (1000) matches the web UI's
|
||||
# interactive budget so scheduled and interactive runs behave identically out
|
||||
# of the box. Values above max_recursion_limit are clamped.
|
||||
#
|
||||
# Execution slots are shared fairly between task owners. Each owner may hold at
|
||||
# most max_concurrent_runs_per_user launching/running scheduled runs (manual
|
||||
# "run now" included); the default 2 leaves one of the 3 default global slots
|
||||
# for everyone else. The waiting queue is drained owner by owner, so one
|
||||
# owner's backlog never pushes another owner's run out of the batch, even with
|
||||
# the per-owner cap off (0). A run that waits longer than queue_timeout_seconds
|
||||
# is skipped; its run history reads "Skipped: it waited too long for a free
|
||||
# slot" and no chat or IM notice is sent, so keep queue_timeout_seconds well
|
||||
# above an owner's typical run length.
|
||||
#
|
||||
# The scheduler is single-instance by default. Set multi_instance: true only
|
||||
# when every Gateway instance shares Postgres, run ownership heartbeats, and
|
||||
# database-backed run events; startup recovery then preserves live peer runs.
|
||||
#
|
||||
# When channel_connections.enabled is true and IM channels are running, the
|
||||
# same poll_interval_seconds also drives the notification delivery worker that
|
||||
# pushes scheduled-run outcomes to the owner's bound IM identities (WeCom
|
||||
# proactive push today). Manual "run now" and interrupts stay silent.
|
||||
# pushes scheduled-task updates to the owner's bound IM identities (only WeCom
|
||||
# supports proactive push today; other apps get no notices). Each occurrence
|
||||
# sends at most one message: a finished run, a failed run, a missed goal, the
|
||||
# automatic pause after three misses, a pause by the agent (stop condition met)
|
||||
# or the task finishing (all max_runs done, end_at reached). Notices are queued
|
||||
# in the same transaction that records the outcome, so runs finalized by crash
|
||||
# or lease recovery notify once. Plain manual "run now" trials and interrupted
|
||||
# runs stay silent. The text is in the owner's web UI language (else
|
||||
# channel_connections.notification_locale) and has no IDs and no links.
|
||||
#
|
||||
# scheduler:
|
||||
# enabled: false # Master switch for the background poller
|
||||
@ -2711,6 +2729,7 @@ agent_storage:
|
||||
# poll_interval_seconds: 5 # How often to scan for due tasks (+ IM outbox delivery)
|
||||
# lease_seconds: 120 # Claim lease; a crashed process's task becomes reclaimable after this
|
||||
# max_concurrent_runs: 3 # Global cap on launching/running scheduled runs across multi-instance Pods
|
||||
# max_concurrent_runs_per_user: 2 # Per-owner cap (0 = off); never more than max_concurrent_runs
|
||||
# queue_timeout_seconds: 3600 # Maximum durable queue wait before an occurrence fails
|
||||
# min_once_delay_seconds: 60 # Floor for one-time run_at offset and interval every_seconds
|
||||
# recursion_limit: 1000 # LangGraph super-step cap for scheduled runs (matches the web UI)
|
||||
@ -2721,6 +2740,7 @@ scheduler:
|
||||
poll_interval_seconds: 5
|
||||
lease_seconds: 120
|
||||
max_concurrent_runs: 3
|
||||
max_concurrent_runs_per_user: 2
|
||||
queue_timeout_seconds: 3600
|
||||
min_once_delay_seconds: 60
|
||||
recursion_limit: 1000
|
||||
@ -2861,15 +2881,20 @@ run_ownership:
|
||||
# - Telegram uses a deep link when `bot_username` is configured.
|
||||
# - Slack, Discord, Feishu, DingTalk, WeChat, WeCom, and QQ use `/connect <code>`
|
||||
# through the already-running bot/app.
|
||||
# - When enabled (and channels are running), scheduled-task completions/failures
|
||||
# also enqueue IM pushes to the owner's connected identities (see scheduler
|
||||
# section above). Manual triggers and interrupts do not notify.
|
||||
# - When enabled (and channels are running), scheduled-task updates (runs,
|
||||
# pauses, finishes) are pushed to the owner's connected identities on apps
|
||||
# with proactive push, WeCom today (see scheduler section above). Settings
|
||||
# shows per app whether updates are sent there. Plain manual trials and
|
||||
# interrupted runs do not notify. Notices carry no links.
|
||||
#
|
||||
# channel_connections:
|
||||
# enabled: false
|
||||
# # Security: keep this enabled unless you intentionally want legacy open-bot behavior.
|
||||
# # Disabling it lets unbound external IM users create DeerFlow threads/runs.
|
||||
# require_bound_identity: true
|
||||
# # Language of scheduled-task IM notices (en-US or zh-CN) for owners without
|
||||
# # a web UI language preference (e.g. auth disabled). Restart-required.
|
||||
# notification_locale: en-US
|
||||
#
|
||||
# telegram:
|
||||
# enabled: false
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
{
|
||||
"version": 2,
|
||||
"description": "Cross-language contract fixture for the host-written scheduled-task strings the tasks page and IM notices translate. The backend writes these values to scheduled_task_runs.error and scheduled_tasks.last_error during goal and stop finalization and host-side run bookkeeping; the frontend (core/scheduled-tasks/goal-outcome.ts) and the IM notice renderer map them to readable text, so neither shows the raw value.",
|
||||
"version": 3,
|
||||
"description": "Cross-language contract fixture for the host-written scheduled-task strings the tasks page and IM notices translate. The backend writes these values to scheduled_task_runs.error and scheduled_tasks.last_error during goal and stop finalization and host-side run bookkeeping; the frontend (core/scheduled-tasks/goal-outcome.ts) and the IM notice renderer map them to readable text, so neither shows the raw value. Version 3 adds the lifecycle vocabulary: lifecycle_events and their lifecycle_reasons (the rows the originating chat shows, GET /api/threads/{thread_id}/scheduled-task-events) and notification_events (the outbox event names).",
|
||||
"agent_stop_last_error_prefix": "stopped by the agent in run ",
|
||||
"auto_pause_last_error": "paused after 3 unmet scheduled goal runs",
|
||||
"unmet_reason_codes": [
|
||||
@ -33,5 +33,12 @@
|
||||
"deleted_while_queued": "scheduled task was deleted while queued",
|
||||
"end_reached": "schedule end condition reached",
|
||||
"interrupted": "run was interrupted before completion"
|
||||
}
|
||||
},
|
||||
"lifecycle_events": ["task_stopped", "task_paused", "task_finished"],
|
||||
"lifecycle_reasons": {
|
||||
"task_stopped": ["agent_stop"],
|
||||
"task_paused": ["consecutive_unmet"],
|
||||
"task_finished": ["max_runs", "end_at", "once_done", "once_failed"]
|
||||
},
|
||||
"notification_events": ["run_completed", "run_failed", "run_unmet", "task_paused", "task_stopped", "task_finished"]
|
||||
}
|
||||
|
||||
12
contracts/thread_origin_contract.json
Normal file
12
contracts/thread_origin_contract.json
Normal file
@ -0,0 +1,12 @@
|
||||
{
|
||||
"version": 1,
|
||||
"description": "Cross-language contract fixture for the server-owned run and thread origin. The Gateway stamps metadata[key] = {\"kind\": <one of kinds>, \"provider\"?: ..., \"namespace\"?: ...} on runs started by a server-side launcher, and on the thread when it creates one (backend/packages/harness/deerflow/runtime/run_origin.py); runs.origin_kind holds the kind. The frontend reads the same key to pick a thread's origin marker. Not to be confused with scheduled_origin_key in scheduled_goal_notes_contract.json, which marks the scheduled prompt message inside a run.",
|
||||
"key": "deerflow_origin",
|
||||
"kinds": [
|
||||
"extension",
|
||||
"github",
|
||||
"im_channel",
|
||||
"mcp_notification",
|
||||
"schedule"
|
||||
]
|
||||
}
|
||||
@ -135,7 +135,7 @@ they resolve from the `secrets` map):
|
||||
|
||||
```yaml
|
||||
config: |
|
||||
config_version: 53
|
||||
config_version: 55
|
||||
models:
|
||||
- name: gpt-4
|
||||
use: langchain_openai:ChatOpenAI
|
||||
|
||||
@ -324,7 +324,7 @@ ingress:
|
||||
# -- DeerFlow config.yaml content. Secrets MUST stay as $VAR references — never
|
||||
# inline literal secret values here. The default enables provisioner sandbox.
|
||||
config: |
|
||||
config_version: 53
|
||||
config_version: 55
|
||||
log_level: info
|
||||
recursion_limit: 100
|
||||
max_recursion_limit: 1000
|
||||
|
||||
@ -58,6 +58,9 @@ Run status `unmet` identifies a finished occurrence whose scheduled goal was not
|
||||
Scheduled-task views read state through the pure `core/scheduled-tasks` helpers (`status.ts`, `actions.ts`, `format.ts`, `describeTaskSchedule`, `errors.ts`) so list, detail and chat card agree. "Is a run active?" is `status === "running"` or `active_run_status` (a recurring task stays `enabled` while it runs). Default views show no IDs, ISO times, cron strings or enum names; errors localize by `detail.code` (`contracts/scheduled_task_errors_contract.json`) with raw text only behind Details. `tests/e2e/utils/readable.ts` and `tests/unit/helpers/readable.ts` assert this.
|
||||
The tasks page (`app/workspace/scheduled-tasks/page.tsx`) only composes `components/workspace/scheduled-tasks/*`: list with status tabs, detail (Runs / Stops when / goal / Does / notes / History), outcome notice, create/edit/duplicate dialog (PATCH sends only changed fields, `null` clears) and the renew dialog that a `limits_exhausted` Resume opens. `?task_id=` selects a task; `?thread_id=` scopes the list to one chat.
|
||||
In chat, a `schedule_task` result becomes an `assistant:scheduled-task` group (`core/scheduled-tasks/tool-result.ts`, last card per task per turn; not a turn boundary) rendered by `ScheduledTaskCard`, which polls the task only while on screen. A human message with `additional_kwargs.deerflow_scheduled_origin` stays a `human` group with `scheduledOrigin`: `ScheduledRunPrompt` renders the origin's user-language parts, never the launched text, and it is not editable.
|
||||
Server-created threads: `core/threads/origin.ts` reads `metadata.deerflow_origin` (`contracts/thread_origin_contract.json`), then `channel_source`, then a legacy `scheduled_task_id`; provider names come from `threads.origin.providers`. `core/threads/activity.ts` polls `/api/thread-activity` (mount `useThreadActivity` once; unmounting resets its cursor; first poll only seeds; invalidate lists only on server-origin threads, truncation or a higher `read_version`) and `useMarkThreadRead` posts reads. `core/scheduled-tasks/events.ts` loads a chat's lifecycle events, not gated on its task list; `placeTaskEvents` puts each at the end of its `after_run_id` turn. The UI language is saved as the `locale` preference once per session and on each switch, only when the Gateway's preferences include `locale`.
|
||||
Thread rows (sidebar, `/workspace/chats`) show `ThreadOriginIcon` and, if `unread === true` and not the open thread, `ThreadUnreadDot` with `threads.unreadLabel` as row label. `WorkspaceSidebar` mounts `useThreadActivity`. Chat pages call `useMarkOpenThreadRead` once metadata loads (marks read on load, on an unread flip, on tab visible, and via its callback from `onFinish`).
|
||||
Chat pages pass `useThreadScheduledTaskEvents` data to `MessageList` (`scheduledTaskEvents`; the custom-agent page also passes its agent for the run link), which renders `ScheduledTaskEventLine` through `VirtualMessageList.renderAfterGroup` at the `placeTaskEvents` position, inside the group's measured row. Lines are history: never gated on the task list. Channel provider cards (Settings and the sidebar list) render `ChannelScheduledUpdates` from `proactive_notifications`, and nothing when the field is absent.
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
@ -10,7 +10,11 @@ import { SidebarTrigger } from "@/components/ui/sidebar";
|
||||
import { AgentWelcome } from "@/components/workspace/agent-welcome";
|
||||
import { ArtifactTrigger } from "@/components/workspace/artifacts";
|
||||
import { BrowserTrigger } from "@/components/workspace/browser-view";
|
||||
import { ChatBox, useThreadChat } from "@/components/workspace/chats";
|
||||
import {
|
||||
ChatBox,
|
||||
useMarkOpenThreadRead,
|
||||
useThreadChat,
|
||||
} from "@/components/workspace/chats";
|
||||
import { ContextUsageBadge } from "@/components/workspace/context-usage-badge";
|
||||
import { ExportTrigger } from "@/components/workspace/export-trigger";
|
||||
import { GoalStatus } from "@/components/workspace/goal-status";
|
||||
@ -60,6 +64,7 @@ import {
|
||||
import { isHiddenFromUIMessage } from "@/core/messages/utils";
|
||||
import { useModels } from "@/core/models/hooks";
|
||||
import { useNotification } from "@/core/notification/hooks";
|
||||
import { useThreadScheduledTaskEvents } from "@/core/scheduled-tasks/events";
|
||||
import { useLocalSettings, useThreadSettings } from "@/core/settings";
|
||||
import { resolveThreadContext } from "@/core/settings/store";
|
||||
import {
|
||||
@ -107,6 +112,17 @@ export default function AgentChatPage() {
|
||||
enabled: !isNewThread && !isMock,
|
||||
isMock,
|
||||
});
|
||||
// A saved thread that exists on the server is being read while open:
|
||||
// clears its unread dot (sidebar and chats list) on every device.
|
||||
const markThreadRead = useMarkOpenThreadRead(threadId, {
|
||||
enabled: !isNewThread && !isMock && threadMetadata.data != null,
|
||||
});
|
||||
// Lifecycle lines of schedules created in this chat ("Paused by agent",
|
||||
// "Finished"); they stay after the task is deleted.
|
||||
const scheduledTaskEvents = useThreadScheduledTaskEvents(threadId, {
|
||||
isNewThread,
|
||||
enabled: !isMock,
|
||||
});
|
||||
const backendTokenUsage = threadTokenUsageToTokenUsage(threadTokenUsage.data);
|
||||
const contextUsage = selectContextUsage(threadTokenUsage.data);
|
||||
|
||||
@ -191,6 +207,9 @@ export default function AgentChatPage() {
|
||||
setIsNewThread(false);
|
||||
},
|
||||
onFinish: (state) => {
|
||||
// A run in this thread ended (a send, or a joined scheduled run) while
|
||||
// it is open: it has been read.
|
||||
markThreadRead();
|
||||
if (document.hidden || !document.hasFocus()) {
|
||||
let body = "Conversation finished";
|
||||
const lastMessage = state.messages[state.messages.length - 1];
|
||||
@ -433,6 +452,7 @@ export default function AgentChatPage() {
|
||||
testId="main-message-list"
|
||||
threadId={threadId}
|
||||
thread={thread}
|
||||
scheduledTaskEvents={scheduledTaskEvents.data}
|
||||
enableConversationOutline
|
||||
paddingBottom={MESSAGE_LIST_DEFAULT_PADDING_BOTTOM}
|
||||
hasMoreHistory={hasMoreHistory}
|
||||
|
||||
@ -10,7 +10,9 @@ import { ScrollArea } from "@/components/ui/scroll-area";
|
||||
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
|
||||
import {
|
||||
ThreadChannelBadge,
|
||||
ThreadChannelIcon,
|
||||
ThreadOriginIcon,
|
||||
ThreadUnreadDot,
|
||||
unreadLabelOfThread,
|
||||
} from "@/components/workspace/thread-channel-source";
|
||||
import { VirtualThreadList } from "@/components/workspace/thread-list-virtualizer";
|
||||
import { useThreadArchiveAction } from "@/components/workspace/use-thread-archive-action";
|
||||
@ -21,6 +23,7 @@ import {
|
||||
} from "@/components/workspace/workspace-container";
|
||||
import { useI18n } from "@/core/i18n/hooks";
|
||||
import { useInfiniteThreads } from "@/core/threads/hooks";
|
||||
import { threadOriginOf } from "@/core/threads/origin";
|
||||
import { buildThreadListModel } from "@/core/threads/thread-list-model";
|
||||
import {
|
||||
channelSourceOfThread,
|
||||
@ -144,29 +147,50 @@ export default function ChatsPage() {
|
||||
items={filteredThreads}
|
||||
scrollParentSelector='[data-slot="scroll-area-viewport"]'
|
||||
renderItem={(thread) => {
|
||||
const channelSource = channelSourceOfThread(thread);
|
||||
const channelSource = channelSourceOfThread(thread, t);
|
||||
const title = titleOfThread(thread, t.pages.untitled);
|
||||
const unread = thread.unread === true;
|
||||
const origin = threadOriginOf(thread);
|
||||
// The unread label replaces the row's name; the time
|
||||
// stays announced as its description.
|
||||
const timeId = `thread-time-${thread.thread_id}`;
|
||||
return (
|
||||
<div
|
||||
key={thread.thread_id}
|
||||
className="flex items-center gap-2 border-b"
|
||||
>
|
||||
<Link
|
||||
aria-label={
|
||||
unread
|
||||
? unreadLabelOfThread(title, t, origin)
|
||||
: undefined
|
||||
}
|
||||
aria-describedby={
|
||||
unread && thread.updated_at ? timeId : undefined
|
||||
}
|
||||
className="min-w-0 flex-1"
|
||||
data-unread={unread ? "true" : undefined}
|
||||
href={pathOfThread(thread)}
|
||||
>
|
||||
<div className="flex flex-col gap-2 p-4">
|
||||
<div className="flex min-w-0 items-center gap-2">
|
||||
<ThreadChannelIcon source={channelSource} />
|
||||
<div className="min-w-0 flex-1 truncate">
|
||||
{titleOfThread(thread, t.pages.untitled)}
|
||||
</div>
|
||||
<ThreadOriginIcon
|
||||
origin={origin}
|
||||
className="text-muted-foreground"
|
||||
/>
|
||||
<div className="min-w-0 truncate">{title}</div>
|
||||
{unread && <ThreadUnreadDot />}
|
||||
<div className="flex-1" />
|
||||
<ThreadChannelBadge
|
||||
source={channelSource}
|
||||
className="hidden sm:inline-flex"
|
||||
/>
|
||||
</div>
|
||||
{thread.updated_at && (
|
||||
<div className="text-muted-foreground text-sm">
|
||||
<div
|
||||
id={timeId}
|
||||
className="text-muted-foreground text-sm"
|
||||
>
|
||||
{formatTimeAgo(thread.updated_at)}
|
||||
</div>
|
||||
)}
|
||||
|
||||
@ -109,6 +109,22 @@ export function ChannelProviderIcon({
|
||||
);
|
||||
}
|
||||
|
||||
if (normalizedProvider === "github") {
|
||||
// GitHub agent threads (GitHub webhooks reach the channel manager as the
|
||||
// `github` provider). Octicons mark-github, in the text color.
|
||||
return (
|
||||
<svg
|
||||
viewBox="0 0 16 16"
|
||||
aria-hidden="true"
|
||||
fill="currentColor"
|
||||
className={cn("size-5", className)}
|
||||
{...props}
|
||||
>
|
||||
<path d="M8 0c4.42 0 8 3.58 8 8a8.013 8.013 0 0 1-5.45 7.59c-.4.08-.55-.17-.55-.38 0-.27.01-1.13.01-2.2 0-.75-.25-1.23-.54-1.48 1.78-.2 3.65-.88 3.65-3.95 0-.88-.31-1.59-.82-2.15.08-.2.36-1.02-.08-2.12 0 0-.67-.22-2.2.82-.64-.18-1.32-.27-2-.27-.68 0-1.36.09-2 .27-1.53-1.03-2.2-.82-2.2-.82-.44 1.1-.16 1.92-.08 2.12-.51.56-.82 1.28-.82 2.15 0 3.06 1.86 3.75 3.64 3.95-.23.2-.44.55-.51 1.07-.46.21-1.61.55-2.33-.66-.15-.24-.6-.83-1.23-.82-.67.01-.27.38.01.53.34.19.73.9.82 1.13.16.45.68 1.31 2.69.94 0 .67.01 1.3.01 1.49 0 .21-.15.45-.55.38A7.995 7.995 0 0 1 0 8c0-4.42 3.58-8 8-8Z" />
|
||||
</svg>
|
||||
);
|
||||
}
|
||||
|
||||
if (normalizedProvider === "feishu") {
|
||||
return (
|
||||
<svg
|
||||
|
||||
@ -26,6 +26,7 @@ import {
|
||||
} from "@/components/ui/dialog";
|
||||
import { Input } from "@/components/ui/input";
|
||||
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
|
||||
import { channelProviderName } from "@/core/channels/provider-state";
|
||||
import type {
|
||||
ChannelProvider,
|
||||
ChannelRuntimeConfigValues,
|
||||
@ -197,8 +198,8 @@ export function ChannelRuntimeConfigDialog({
|
||||
<DialogHeader className={cn(hasWechatQR && "gap-1 text-left")}>
|
||||
<DialogTitle>
|
||||
{isEditing && !completionProvider
|
||||
? t.channels.setupEditTitle(provider.display_name)
|
||||
: t.channels.setupTitle(provider.display_name)}
|
||||
? t.channels.setupEditTitle(channelProviderName(provider, t))
|
||||
: t.channels.setupTitle(channelProviderName(provider, t))}
|
||||
</DialogTitle>
|
||||
<DialogDescription
|
||||
className={cn(hasWechatQR && "text-xs leading-relaxed")}
|
||||
|
||||
@ -0,0 +1,52 @@
|
||||
"use client";
|
||||
|
||||
import { BellIcon, BellOffIcon } from "lucide-react";
|
||||
|
||||
import type { ChannelProvider } from "@/core/channels/types";
|
||||
import { useI18n } from "@/core/i18n/hooks";
|
||||
import { cn } from "@/lib/utils";
|
||||
|
||||
/**
|
||||
* Whether scheduled-task updates reach this app ("Scheduled task updates:
|
||||
* sent here" / "... available after you connect" / "... not available for
|
||||
* this app yet"). Only providers that
|
||||
* implement proactive push receive them (WeCom today). Renders nothing for
|
||||
* an older backend that does not report `proactive_notifications`, rather
|
||||
* than claiming an app it cannot judge is unsupported.
|
||||
*/
|
||||
export function ChannelScheduledUpdates({
|
||||
provider,
|
||||
connected = true,
|
||||
className,
|
||||
}: {
|
||||
provider: Pick<ChannelProvider, "proactive_notifications">;
|
||||
/** False for an app not connected yet: nothing is sent until it is. */
|
||||
connected?: boolean;
|
||||
className?: string;
|
||||
}) {
|
||||
const { t } = useI18n();
|
||||
if (typeof provider.proactive_notifications !== "boolean") {
|
||||
return null;
|
||||
}
|
||||
const supported = provider.proactive_notifications;
|
||||
const Icon = supported ? BellIcon : BellOffIcon;
|
||||
return (
|
||||
<p
|
||||
data-testid="channel-scheduled-updates"
|
||||
data-supported={supported ? "true" : "false"}
|
||||
className={cn(
|
||||
"text-muted-foreground flex items-start gap-1.5 text-xs",
|
||||
className,
|
||||
)}
|
||||
>
|
||||
<Icon className="mt-px size-3.5 shrink-0" aria-hidden />
|
||||
<span>
|
||||
{!supported
|
||||
? t.channels.scheduledUpdates.unsupported
|
||||
: connected
|
||||
? t.channels.scheduledUpdates.supported
|
||||
: t.channels.scheduledUpdates.supportedAfterConnect}
|
||||
</span>
|
||||
</p>
|
||||
);
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user