* feat: add scheduled tasks MVP
* fix: harden scheduled task execution semantics
* feat(scheduled-tasks): preset-driven schedule form with timezone and live preview
Replace the raw cron input with a preset Select (hourly/daily/weekly/monthly/custom)
plus structured inputs (time picker, weekday toggles, day-of-month), datetime-local
for one-time tasks, a timezone selector defaulting to the browser timezone, and a
live human-readable preview. Reuses one ScheduledTaskScheduleInput for create and
edit; backend contract unchanged; zero new deps (pure Intl + DST-safe offset helpers).
* feat(scheduled-tasks): full-page i18n + recipe templates + E2E locale pin
Localize the rest of the scheduled-tasks page (filters, detail pane, actions,
edit form, run list, enum values) via t.scheduledTasks.* in en/zh. Add four
built-in recipe templates (GitHub Trending, news digest, issue triage, weekly
report) exposed as a chip row that pre-fills title + prompt + schedule. Pin
Playwright locale to en-US so E2E selectors stay stable against i18n. No backend
change, no new deps.
* fix(scheduled-tasks): idempotent 0003 migration, update head constants, future-date once test
Merge with main surfaced three CI failures:
- 0003_scheduled_tasks create_table collided with legacy test seeds that
build from full metadata; guard with inspector.has_table so the revision
no-ops when the table already exists (0004/0005 are already idempotent via
_helpers.py).
- persistence bootstrap concurrency/regression tests pinned HEAD to main's
0002_runs_token_usage; bump to the new head 0005_scheduled_task_thread_nullable.
- once-task router test used a fixed past run_at and tripped the
must-be-in-the-future validation; use a future date.
* address review: ok-check, 502 for trigger failure, mock fields, migration filename, doc fences
- fetchThreadScheduledTasks now checks response.ok like the other fetchers.
- trigger endpoint returns 502 (not 409) when dispatch fails outright, so
clients can distinguish a real conflict from a server-side failure.
- E2E mock normalizes scheduled-task objects with context_mode/last_thread_id
and nullable thread_id, matching the backend contract the UI renders against.
- Rename 0002_scheduled_tasks.py -> 0003_scheduled_tasks.py to match its
revision id (file was renamed in spirit already; filename now follows).
- CONFIGURATION.md: close the Tool Groups yaml fence and drop the stray fence
after the Scheduler notes so the sections render correctly.
* fix(scheduled-tasks): harden lease, poller, config, and frontend UX after review
* fix(scheduled-tasks): harden run lifecycle, overlap skip, non_interactive gating, and DST conversion after review
- defer a once task's terminal status to the run-completion hook; the task
stays running until the real outcome, and a startup sweep cancels once
tasks orphaned by a crash (launch-time 'completed' could stick forever)
- record interrupted runs as a distinct 'interrupted' run status with a
readable message; an interrupted once task ends 'cancelled', not 'failed'
- enforce overlap_policy=skip for fresh_thread_per_run via an active-run
pre-check (same-thread ConflictError can never fire across fresh threads)
- protect terminal run statuses from the late launch-path 'running' write
- honor context.non_interactive only for internally-authenticated callers;
arbitrary clients can no longer strip ask_clarification
- fix DST-stale timezone offset in zonedLocalToUtcIso by re-deriving the
offset at the resolved instant (once tasks fired an hour late around
spring-forward and the create->edit round-trip diverged)
- drop dead ScheduledTaskRunRepository.update_by_run_id; share one Gateway
API error helper between channels and scheduled-tasks frontends
* fix(scheduled-tasks): close review round-3 gaps in guards, concurrency, and API ergonomics
- scrub internal-only context keys (non_interactive) from the assembled run
config for non-internal callers: gating body.context alone left the same
key smuggle-able through the free-form body.config copied verbatim by
build_run_config
- guard update_after_launch with protect_terminal so the launch bookkeeping
write cannot clobber a once task already finalized by a fast-failing run's
completion hook (parent-row sibling of the run-row guard)
- reject a manual trigger while the task has an active run (409) instead of
launching a duplicate concurrent run on fresh_thread_per_run
- re-arm a terminal once task to enabled when PATCH pushes run_at into the
future; previously the endpoint returned 200 with a next_run_at that could
never be claimed
- make max_concurrent_runs a real global cap: each poll claims only into the
remaining budget of active (queued/running) scheduled runs
- paginate GET /scheduled-tasks/{id}/runs (limit<=200, offset) and push the
thread filter of /threads/{id}/scheduled-tasks into SQL
- stamp context.user_id on scheduler-launched runs, matching IM channels, so
user-scoped guardrail providers see the owning user
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Update DeepSeek references from deprecated model names to the V4 lineup:
- deepseek-reasoner → deepseek-v4-pro
- deepseek-chat → deepseek-v4-flash
Keep docs and frontend mocks aligned with the wizard provider list.
* feat(skills): per-user skill isolation (#2905)
Implement user-scoped skill storage that isolates custom skills between
users while sharing public skills globally.
Key changes:
- Add UserScopedSkillStorage class for per-user custom skill directories
- Introduce get_or_new_user_skill_storage() factory with user_id context
- Auth middleware sets effective_user_id for request-scoped storage
- Agent/prompt/middleware now use user-scoped storage and prompt cache
- Sandbox mounts user-scoped skill directories for search/read tools
- Add validate_skill_file_path() to SkillStorage for path security
- Migration script supports --all-users bulk migration
- Frontend: add editable field to Skill type, error check in enableSkill
- All skill categories can be toggled (custom skills default to enabled)
- Update skill-creator SKILL.md with isolation-aware instructions
Tests:
- Add test_user_scoped_skill_storage.py (new)
- Update all existing skill tests for user-scoped storage
- Update sandbox, client, and router tests
* fix(skills): address second-round PR review feedback (#3889)
- P1-1: restrict legacy skill mount to users without custom skills
- P1-2: fail-closed for _is_disabled_skill_path (OSError → return True)
- P2-1: AND-merge global extensions_config skill disabled state
- P2-2: atomic write for _skill_states.json (mkstemp + replace)
- P2-3: normalize X-DeerFlow-Owner-User-Id in trusted boundary
- P2-4: LRU-bounded _enabled_skills_by_config_cache (OrderedDict, maxsize=256)
- P2-5: clear global prompt cache on PUBLIC skill toggle
- P2-6: invalidate skill caches on client.update_skill
* fix(tests): correct tool policy test after merge
* fix(skills): use DEFAULT_SKILLS_CONTAINER_PATH in UserScopedSkillStorage
The "/mnt/skills" literal in UserScopedSkillStorage.__init__ triggers
test_skill_container_path_defaults::test_mnt_skills_literal_is_owned_by_skill_constants_module
on CI. Migrate the default to the existing deerflow.constants constant,
matching the pattern already used by LocalSkillStorage, SkillStorage, and
the durable/tool_error middlewares.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: emit structured runtime metadata
* fix: avoid subagent import cycle in replay gateway
* fix: preserve legacy subtask result parsing
* refactor: tighten runtime metadata contracts
* fix(middleware): keep recovery hint on task exception wrapper content
The structured-metadata stamp overwrote the wrapper text with the bare
task-failure message, dropping the model-facing 'Continue with available
context, or choose an alternative tool.' guidance that every other tool
exception keeps. Append the shared hint after the formatted message.
* fix(subagents): require lowercase hex for result_sha256 reader
Length-only validation accepted any 64-char string; a faulty serializer
or relaying wrapper could store a non-digest value in the delegation
ledger. Enforce the producer's hexdigest shape with a fullmatch.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* Fix stale run reconnect and cancel handling
* fix the frontend
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Fix prettier formatting and document error/timeout reconnect intent
Wrap the over-long assertion in api-client.test.ts so the CI prettier --check
job passes, and add a comment above TERMINAL_RUN_STATUSES noting that
error/timeout short-circuit intentionally drops the transient onError toast on
reload (the persisted error state still loads from the checkpoint via
useThreadHistory).
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
* feat(frontend): add citation sources evidence panel
Inline [citation:Title](URL) links render as badges, but with many
citations the reader has no consolidated, deduplicated list of what a
message or report actually drew on. Add a collapsible "sources" panel
that extracts citation links from AI messages and markdown artifacts,
dedupes by URL, counts occurrences, and offers per-source copy of a
reusable markdown reference.
- extractCitationSources: parse [citation:...](url) links, skip images
and fenced code, dedupe by normalized URL, fall back to domain for
generic labels
- CitationSourcesPanel: collapsible list with per-source cite counts,
internal scroll for long lists, and copy-to-clipboard reference button
- Wire panel into AI message content and markdown artifact preview
- Add en-US/zh-CN citation strings and types
- Unit tests for extraction and panel rendering
* fix(frontend): preserve default link styling in message content override
MessageContent_ passes a custom `a` renderer to MarkdownContent, whose
default `a` (primary underline + external target/rel) is overridden
because MarkdownContent spreads props components last. Restore that
styling/external behavior in the fallback branch so normal links in
messages aren't regressed, while keeping citation: and /mnt/ handling.
* fix(frontend): harden citation source extraction and dedupe link renderer
Address review findings on the citation sources panel:
- Use a non-consuming lookbehind so back-to-back citations no longer drop
every other source.
- Match balanced parenthetical groups in URLs so disambiguation links like
.../Foo_(a)_(b) are no longer truncated.
- Mask inline code (and unclosed streaming fences) so example citations in
code aren't scraped as real sources; masking preserves indices.
- Extract a shared createMarkdownLinkComponent factory used by both message
content and markdown content, removing the duplicated `a` renderer.
* implement goal continuations
* fix(goal): address review findings for goal continuations
- goal: key the no-progress breaker on a signature of the latest visible
assistant evidence instead of the evaluator's volatile free-text, so it
actually fires on stalled turns; thread the signature through every
worker persist / no-progress call site
- goal: align _stand_down_reason default caps with should_continue_goal
(8 / 2) so the two gate functions agree on goals missing the fields
- runtime: offload the synchronous checkpointer fallback via
asyncio.to_thread (goal.py + worker.py) to keep blocking IO off the loop
- frontend: i18n the GoalStatus "Goal" label (goalLabel in en/zh/types)
- frontend: extract pure composer helpers into input-box-helpers.ts with
unit tests (parseGoalCommand, readGoalResponseError, skill suggestions)
- tests: cover the evidence-based no-progress and default-cap behavior
- docs: align backend/AGENTS.md goal paragraph with actual behavior
- e2e: prettier-format chat.spec.ts (fixes the lint-frontend CI failure)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(frontend): hide goal continuation counter until the agent continues
The goal status bar rendered a raw "0/8" before any auto-continuation, which
read as a mysterious score. Now the counter is hidden until
continuation_count > 0, then shows "Continuing N/M" with a tooltip explaining
the auto-continuation cap.
- Extract getGoalContinuationDisplay into a pure helper (hides at 0) + unit tests
- Add goalContinuing / goalContinuationTooltip i18n keys (en/zh/types)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(goal): address review findings for goal continuations
Frontend correctness
- Fix the optimistic /goal result permanently shadowing server goal state:
the streamed continuation counter never surfaced for a goal set in-session.
Extract a shared useActiveGoal hook (used by both chat pages) that reconciles
the optimistic copy with server state via a goalReconciliationKey, de-duping
the copy-pasted goal block across the two pages.
- Stop /goal status|clear failures from escaping handleSubmit as unhandled
rejections (handleGoalCommand now returns success; the run only starts when a
goal was actually saved).
- Use a function replacer for the goal-status toast so an objective containing
$&/$1 isn't treated as a replacement pattern.
Backend cleanliness / correctness
- De-duplicate four byte-identical helpers (_call_checkpointer_method,
_message_type, _additional_kwargs, _is_visible_message) by importing them
from runtime.goal instead of re-defining them in the run worker.
- Remove the dead `checkpoint_tuple.tasks` durability guard (CheckpointTuple has
no tasks field) and document that pending_writes is the durability signal.
- Decompose the 176-line _prepare_goal_continuation_input: extract
_reread_goal_and_checkpoint and a _persist closure so the thread-unchanged
guard and stand-down persistence aren't open-coded three times. Document the
last-writer-wins write-window limitation as a follow-up.
- Add a shared parse_goal_command helper and use it from the TUI and IM-channel
/goal handlers (one place for the status/clear/set semantics).
Tests
- Restore the 11 command-registry tests dropped by the previous goal change
(filter_commands ranking/description, build_registry builtins/skills, resolve
cases) alongside the new goal tests.
- Add coverage for the IM-channel _handle_goal_command, the TUI _handle_goal
handler, parse_goal_command, and goalReconciliationKey.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix goal review feedback
* fix goal continuation checkpoint races
* prioritize goal commands while streaming
Route composer submits through a shared helper so /goal commands can be handled before the streaming stop shortcut, while ordinary streaming submits still stop the active run.
Testing: cd frontend && pnpm exec rstest run tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; cd frontend && pnpm check
* preserve goal status during clarification
Keep omitted stream goal fields distinct from explicit null clears so clarification interrupts do not hide an active thread goal that is still present in the checkpoint.
Testing: pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check
* style: format active goal hook
Run Prettier on use-active-goal.ts to satisfy the frontend lint workflow formatting gate.
Testing: pnpm format; pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check
* fix goal review followups
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(gateway): add GET /api/features for frontend feature gating (#3757)
* feat(agents): add useAgentsApiEnabled feature-flag hook (#3757)
* feat(agents): gate /workspace/agents segment on agents_api flag (#3757)
* feat(sidebar): grey out Agents button with tooltip when agents_api disabled (#3757)
* fix(sidebar): show disabled Agents tooltip on hover, harden a11y + e2e (#3757)
- Wrap the disabled Agents button in a hoverable tooltip trigger so the
'feature not enabled' hint shows in the expanded sidebar, not only when
collapsed (the SidebarMenuButton tooltip prop is hidden unless collapsed).
- Add tabIndex={-1} and drop the redundant onClick preventDefault.
- Add e2e coverage for the disabled state + a default /api/features mock.
* style(sidebar): move cursor-not-allowed to the hoverable span (#3757)
* test(agents): anchor e2e agents-API request filter with a path regex (#3757)
* fix(agents): don't expose backend config in disabled message; tell user to contact admin (#3757)
The disabled panel previously said 'Set agents_api.enabled: true in
config.yaml', leaking backend configuration to end users. Replace with a
generic 'not enabled on this server, contact your administrator' message
(matching the existing nameStepApiDisabledError copy). e2e now asserts the
contact-admin message and that no config.yaml/agents_api text is rendered.
* make format
* fix(agents): keep agents_api flag sticky during /api/features outage (#3757)
Failing open re-mounted the agents UI and re-triggered the 403 storm when
agents_api was genuinely disabled and /api/features was down. Persist the
last definitive answer and fall back to it (sticky) before failing open,
only failing open when nothing has ever been observed. Read the cached
value after mount so the first client render matches the server (no
hydration mismatch on the non-loading-gated sidebar).
* fix(sidebar): make disabled Agents entry keyboard/SR accessible (#3757)
The disabled-state explanation was hover-only: tabIndex={-1} removed the
entry from the tab order and the reason lived only in a pointer-triggered
tooltip. Keep it in the tab order and wire aria-describedby to a
visually-hidden reason so keyboard and screen-reader users learn why it is
disabled.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(gateway): require admin for global skills management endpoints
The skills router exposed its management endpoints with no authorization
check (only Depends(get_config)), while the MCP router guards the
equivalent global extensions_config mutations with require_admin_user
(added in the #3425 security hardening). Skills storage is global/shared
across users (no per-user path), lives in the same extensions_config.json
as MCP servers, and custom skill SKILL.md content is injected into every
user's agent system prompt. Any authenticated non-admin user could mutate
or read global skills that affect all tenants:
- POST /api/skills/install: install from an arbitrary thread_id, writing
into the global custom skills tree
- PUT /api/skills/custom/{skill_name}: rewrite a global skill, injecting
instructions into all users' agent prompts (cross-tenant prompt injection)
- DELETE /api/skills/custom/{skill_name} and rollback: tamper with global skills
- GET /api/skills/custom, GET .../{skill_name}, GET .../history: read raw
global custom skill bodies/history
- PUT /api/skills/{skill_name} (enable toggle): writes the shared
extensions_config.json and refreshes the system prompt for every tenant,
so a non-admin could enable/disable any skill globally. There is no
per-user skill state, so this is a global mutation, not a preference.
Add require_admin_user to every endpoint above, mirroring the MCP router.
The shared read path used internally by update/rollback was extracted into
a non-auth helper (_read_custom_skill_response) so internal reuse does not
double-check auth.
Only the read-only GET /api/skills and GET /api/skills/{skill_name} stay
open to normal users: they return just name/description/enabled and back
the user-facing settings UI.
Tests:
- New tests/test_skills_router_authz.py: a non-admin user gets 403 on every
guarded endpoint (including the enable toggle); basic listing stays open;
admins can still toggle.
- Update tests/test_skills_custom_router.py to authenticate as admin.
- pytest tests/ -k skill -> 350 passed, 1 skipped; ruff clean.
Signed-off-by: DengY11 <151997860+DengY11@users.noreply.github.com>
* fix(frontend): gate skill enable/install UI behind admin + handle 403
Follow-up to the backend change that made the global skills mutations
admin-only. Without a matching UI change, a non-admin user would hit a
silent 403 when toggling a skill in Settings -> Skills or clicking
"Install skill" on a .skill artifact.
- core/skills/api.ts: add SkillRequestError (status + isAdminRequired),
throw it from loadSkills/enableSkill on non-ok responses and from
installSkill on 403 (other install errors keep the soft-failure
contract).
- core/skills/hooks.ts: useSkills no longer retries on SkillRequestError.
- skill-settings-page.tsx: show an "admin required" message on 403, and
disable the enable toggle for non-admins (mirrors the MCP tools page).
- artifact-file-detail.tsx / artifact-file-list.tsx: only render the
"Install skill" action for admins, and surface an admin-required toast
if a 403 still occurs.
- i18n: add settings.skills.adminRequired / installAdminRequired (en + zh).
Auth/no-auth and static-website modes synthesize an admin user, so these
gates do not affect single-user/local deployments.
Verified locally: pnpm check (eslint + tsc) passes with no new errors,
pnpm build succeeds, and the dev server renders / and /login (200) with
no compile/runtime errors.
Signed-off-by: DengY11 <151997860+DengY11@users.noreply.github.com>
* style(frontend): format skills hook for prettier
Keep the admin-guard skills hook aligned with Prettier output so the frontend format check passes in CI.
---------
Signed-off-by: DengY11 <151997860+DengY11@users.noreply.github.com>
* Add Brave image search community tool
* fix(community): length-cap Brave web_search queries
Apply _clean_query in web_search_tool so over-long queries are trimmed
to Brave's 400-char limit before the API call, matching image_search_tool
and avoiding HTTP 422 from the Brave Search API.
* fix(community): harden Brave image search SSRF guard and dimension mapping
Address PR review findings:
- Catch ValueError from urlparse so a malformed bracketed-IPv6 URL skips
one item instead of crashing the whole image_search call
- Reject IPv6 literals embedding a non-global IPv4 (IPv4-mapped, 6to4,
NAT64, IPv4-compatible), closing the loopback/private SSRF bypass
- Report width/height from the dict of the URL actually returned, so a
surviving thumbnail no longer reports the dropped original's dimensions
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(subagents): persist and display subagent step history (#3779)
Capture both assistant turns and tool outputs during subagent execution,
stream them in task_running events, and persist them as subagent.* run
events so the subtask card's step timeline survives a reload.
Backend:
- step_events.py: pure layer (capture_step_message, build_subagent_step,
subagent_run_event) shared by streaming and persistence
- executor.py: capture ToolMessage outputs, not just AIMessage turns
- worker.py: persist task_* custom events to RunEventStore (category
"subagent" keeps them out of the thread feed; list_events backfills)
Frontend:
- core/tasks/steps.ts + api.ts: SubtaskStep model, messageToStep,
eventsToSteps, mergeSteps, fetchSubtaskSteps
- subtask card accumulates live steps and backfills on expand
- carry run_id onto history content messages for the events endpoint
* fix(subagents): show AI turns in subtask card + paginate step backfill (#3779)
Two follow-ups to the subagent step-history feature:
Problem 1 — reload backfill could silently truncate the step timeline because
list_events capped at 500 events (seq-ASC) across the whole run. Add task_id
filtering + an after_seq forward cursor to list_events (all three stores +
abstract base + the /events route), and make fetchSubtaskSteps page through one
task's subagent.step events until a short page. No schema migration: the DB
filter rides the existing run-scoped index via event_metadata["task_id"].
Problem 2 — the card only rendered tool steps, so persisted AI turns were never
shown. Replace toolStepsForDisplay with stepsForDisplay: interleave AI reasoning
turns (with text) and tool steps by message_index, drop blank-text AI turns, and
drop the trailing final-answer AI turn when completed (already shown as result).
Card renders AI steps as muted clamped markdown with a sparkles icon.
Tests: store task_id/after_seq filtering + pagination across memory/db/jsonl,
the /events route forwarding, stepsForDisplay rules, and fetchSubtaskSteps
pagination. Docs updated in both AGENTS.md.
* make format
* fix(subagents): capture full multi-tool step tail, batch step persistence, cap tool-call args (#3779)
Address PR review findings on the subagent step-history feature:
1. executor.py streamed on stream_mode="values" and captured only
messages[-1] per chunk, so a multi-tool-call turn (ToolNode appends
one ToolMessage per call in a single super-step) lost all but the last
tool output in both the live task_running stream and the persisted
history. Replace with capture_new_step_messages, which walks the
newly-appended tail (and still re-checks the trailing message on
no-growth chunks so id-less in-place replacements survive).
2. worker.py persisted each step with the store's low-frequency put()
(a per-thread advisory lock per call); a deep subagent (max_turns=150)
emits hundreds of steps on the hot stream loop. Replace with
_SubagentEventBuffer, which batches via put_batch (flush on terminal
subagent.end, at FLUSH_THRESHOLD, and in the worker finally).
3. build_subagent_step capped only text; tool_calls[].args were copied
verbatim, so a large write_file/bash payload produced an unbounded
subagent.step row. Cap each call's serialized args at
SUBAGENT_STEP_MAX_CHARS, flagged args_truncated.
Tests updated/added for all three; AGENTS.md refreshed.
* fix(subagents): merge backfill into latest subtask state; reuse message_content_to_text (#3779)
Address the remaining two PR review findings:
4. subtask-card's fetchSubtaskSteps().then(updateSubtask) closed over a
stale tasks snapshot: a late-resolving backfill wrote setTasks({...stale}),
clobbering SSE steps/status and sibling subtasks that arrived during the
fetch. useUpdateSubtask now reads/writes through a tasksRef mirroring the
latest state (ref-to-latest), and the pure per-subtask transition is
extracted to core/tasks/subtask-update.ts::computeNextSubtask (unit-tested).
5. step_events._content_to_text duplicated deerflow.utils.messages.
message_content_to_text; call the shared helper instead (guarding None
content with 'or ""' so a tool-call-only turn still renders as "").
Tests added for computeNextSubtask and the None-content case; AGENTS.md docs updated.
* fix: generate title for interrupted first turn
* test(title): cover partial-exchange + dict-form messages
Harden the interrupted-run fallback path added in 19fc34fd:
- TitleMiddleware._should_generate_title now accepts a lone first-turn
user message when allow_partial_exchange=True, so the worker can still
derive a title if cancellation lands before any AI chunk is checkpointed.
- runtime/runs/worker._ensure_interrupted_title computes the next
checkpoint step defensively (treat missing/non-int step as 0) and
renames a shadowed ckpt_config local for readability.
- Add four unit tests in tests/test_title_middleware_core_logic.py:
partial-exchange allows user-only, partial-exchange still respects an
existing title, dict-form messages are recognized, and the sync
fallback path derives a title from dict-form messages — matching what
channel_values stores in the checkpoint.
Refs #3859.
* fix: persist interrupted-title via channel_versions bump
Address PR #3874 review feedback: ``_ensure_interrupted_title`` previously
called ``aput(..., new_versions={})``. LangGraph's DB-backed savers
(``PostgresSaver`` and the v4 ``SqliteSaver`` blob layout) strip inline
``channel_values`` from ``put`` and only persist blobs for channels named
in ``new_versions`` — so the fallback ``title`` channel was dropped on
read-back and ``threads_meta.display_name`` stayed ``"Untitled"`` after
refresh on those backends. The original in-memory e2e passed because
``InMemorySaver`` keeps the inline snapshot verbatim.
Fix mirrors ``_rollback_to_pre_run_checkpoint`` in the same file: bump
``channel_versions["title"]`` (via ``checkpointer.get_next_version`` when
available, else int/string fallbacks), persist the new version on the
checkpoint, and declare it in ``new_versions`` so the DB savers actually
write the blob.
Regression coverage in ``tests/test_run_worker_rollback.py``:
- ``test_ensure_interrupted_title_bumps_channel_version_and_declares_it_in_new_versions``
— exact ``aput`` invariants: ``new_versions == {"title": 1}``, the
written checkpoint's ``channel_versions["title"]`` is bumped, and the
pre-existing ``messages`` version is preserved.
- ``test_ensure_interrupted_title_bumps_existing_string_version`` —
string-shaped prior version (some savers use UUID-style versions);
bumped value must differ from the prior, no overwrite-in-place.
- ``test_ensure_interrupted_title_skips_when_title_already_set`` — title
short-circuit; no extra ``aput``.
- ``test_ensure_interrupted_title_returns_none_when_no_checkpoint`` —
no checkpoint yet; returns ``None`` without writing.
- ``test_ensure_interrupted_title_round_trip_with_real_sqlite_checkpointer``
— full round-trip against a real ``AsyncSqliteSaver`` on a disk-backed
DB, then closes and re-opens the saver to simulate a fresh
connection. The fallback title must still be present on the second
``aget_tuple``. This is the exact scenario the review flagged.
Validated locally with the full backend suite: 5195 passed, 18 skipped.
Refs #3859. Addresses review on #3874.
* test(worker, title): harden interrupted-title fallback for every saver
Defensive coverage on top of the channel_versions fix (commit 05253957),
addressing edge cases surfaced during a second-pass review of #3874.
Worker:
- Extract version bump into ``_bump_channel_version(checkpointer, current)``
with explicit fallbacks for int / float / numeric-string / UUID-shaped
string / None / bool, AND a wrap-around defense when the saver's
``get_next_version`` raises or returns an unchanged value. The
invariant is: returned version MUST differ from the prior. Without
this, a saver bug (or a custom backend) could leave
``new_versions={"title": v}`` no-op on DB savers — the very class of
bug the original review pointed out.
Title middleware:
- Coerce ``state.get("messages")`` from ``None`` to ``[]`` on both
``_should_generate_title`` and ``_build_title_prompt``. A
partially-initialized checkpoint can carry ``messages=None`` on the
channel_values channel (the worker reads raw channel_values, not
BaseMessages), and the default kwarg only protects against a missing
key. Repro: ``TypeError: 'NoneType' object is not iterable`` from
the next() generator — confirmed by reverting the fix and watching
``test_*_handles_none_messages_channel`` go red.
Tests (TDD-verified red→green for the new asserts):
- ``test_run_worker_rollback.py``:
* ``_bump_channel_version`` — 8 tests covering every version type
(int, float, numeric string, UUID-style string, None, bool) and
every saver-side fault mode (no ``get_next_version`` / raising /
stuck on identity).
* ``test_ensure_interrupted_title_*`` — 5 additional helper
boundary tests: title.enabled=false short-circuit; empty
messages list; messages=None; aput-error propagation (helper
contract: caller swallows, not the helper); idempotency on a
real InMemorySaver across two invocations.
* ``test_ensure_interrupted_title_preserves_non_title_channel_versions``
— pins that ``new_versions`` only contains ``"title"`` and that
other channels' versions are untouched (regression anchor for a
sloppier draft that bumped every channel).
* ``test_worker_finally_block_swallows_helper_exceptions`` — pins
the integration contract: even if the helper raises, the worker's
threads_meta status sync still runs and ``publish_end`` is still
awaited so the SSE stream closes cleanly.
- ``test_title_middleware_core_logic.py``:
* 4 additional tests: ``messages=None`` on both
``_should_generate_title`` and ``_build_title_prompt``; the
``role: user`` / ``role: assistant`` (OpenAI-style) dict
normalization; partial-exchange path with a dict-form message.
Verification:
- ``PYTHONPATH=. uv run pytest tests/ -x --ignore=tests/blocking_io -q``
→ 5215 passed, 18 skipped.
- ``ruff check`` + ``ruff format --check`` clean on every touched file.
- Red/green TDD verification: temporarily reverted the
``new_versions={}`` fix → 4 new tests went red as expected; restored
and the suite is green again. Same red/green dance for the
``messages=None`` coercion.
Refs #3859. Addresses second-pass review on #3874.
* fix(title): ignore dict context reminders in fallback
* fix(worker): link interrupted-title checkpoint to its parent
The title-bump checkpoint written by ``_ensure_interrupted_title`` was
landing without a ``parent_checkpoint_id`` — a real orphan in the
LangGraph history graph. Reproduction (disk-backed AsyncSqliteSaver):
[seed] checkpoint_id = 1f173dbc...
[helper] wrote title = "Why is the sky blue?"
[issue 1] new checkpoint = 1f173dbc..., parent = None
[issue 1] is new checkpoint orphaned? True
Root cause: ``_ensure_interrupted_title`` built ``write_config`` as
``{"thread_id": ..., "checkpoint_ns": ...}`` only. ``BaseCheckpointSaver``
implementations read ``configurable.checkpoint_id`` from that config as
the *parent* id when inserting (see ``langgraph/checkpoint/sqlite/aio.py``
``aput``: ``config["configurable"].get("checkpoint_id")`` becomes the
``parent_checkpoint_id`` column). With no value, the saver writes NULL —
the new checkpoint is a tree root.
Consequences:
- Any future LangGraph ``runs.resume_from`` / time-travel feature has no
backward edge to walk past the title-bump.
- History-visualization UIs built on ``alist()`` render the title-bump as
a sibling of the prior checkpoint, not its descendant.
Fix: read ``checkpoint_id`` off the tuple's own config and thread it into
``write_config["configurable"]["checkpoint_id"]`` before calling ``aput``,
the same pattern every middleware-driven write uses.
Three new regression tests against real ``AsyncSqliteSaver`` (disk-backed,
fresh connections so we exercise the on-disk read path):
- ``test_ensure_interrupted_title_links_new_checkpoint_to_its_parent`` —
asserts ``latest.parent_config["configurable"]["checkpoint_id"]`` equals
the seeded checkpoint id. TDD red-green verified: reverting the fix
flips this test red with ``AssertionError: title-bump checkpoint must
have a parent_config``.
- ``test_ensure_interrupted_title_appears_in_history_with_audit_marker``
— pins the audit contract: the title-bump entry in ``alist()`` carries
``metadata.source == "update"`` and ``metadata.writes`` contains
``runtime_interrupt_title``. This is a deliberate design choice — we
do NOT hide the entry from history (audit trail belongs in the saver),
but its source and writes marker MUST be unambiguous so UIs/tools can
identify it.
- ``test_ensure_interrupted_title_survives_immediate_next_turn`` —
cancel → immediate user follow-up scenario. Simulates the agent's next
turn appending a (user, ai) pair without touching the title channel,
then opens a fresh saver and verifies the title is still present after
the next-turn checkpoint write. Pins the channel-version-blob
invariant established by commit 05253957 — without the
``new_versions={"title": v}`` declaration there, the title blob would
vanish from the DB and this test would read back ``None``.
Verification:
- ``PYTHONPATH=. uv run pytest tests/ -x --ignore=tests/blocking_io -q``
→ 5222 passed, 15 skipped.
- ``ruff check`` + ``ruff format --check`` clean on every touched file.
- Reproduction script confirms ``parent_checkpoint_id`` is now non-null
and the next-turn read-back preserves the fallback title.
Refs #3859.
* Revert "fix(worker): link interrupted-title checkpoint to its parent"
This reverts commit c763ed9334781db1acdce0f5f33d663d8d5f80ff.
* test: trim over-engineered test coverage
Reduce review surface area on PR #3874 by dropping defensive tests that
don't pin a real invariant. After self-review:
- ``_bump_channel_version``: 8 tests → 2 (happy path + saver-error
fallback). Dropped float / bool / numeric-string / UUID-string /
missing-get-next-version / stuck-get-next-version branches — those
are speculative scaffolding for savers we don't ship.
- ``_ensure_interrupted_title``: dropped ``returns_none_when_title_disabled``,
``returns_none_with_no_user_message``, ``returns_none_when_no_checkpoint``
— boundary guards already exercised by the e2e test and the
``handles_none_messages_channel`` regression anchor.
Net: -107 lines of test code. Remaining coverage still pins every
red-green-verified invariant (channel_versions bump, string-version
bump, idempotency, sqlite round-trip, non-title channel preservation,
aput-error contract, worker finally swallowing, partial-exchange).
Verification: 5209 passed, 15 skipped.
* fix: harden interrupted title finalization
* fix: serialize interrupted title finalization
* fix: preserve interrupt semantics during title finalization
* fix: preserve delayed interrupted title recovery
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add Browserless web_capture screenshot tool
Add a web_capture tool that renders a page via Browserless /screenshot and
presents it through the artifact system, alongside the existing Browserless
web_fetch provider.
Hardening:
- SSRF guard: reject URLs resolving to private/loopback/link-local (incl. the
169.254.169.254 cloud-metadata endpoint)/reserved/multicast/unspecified
addresses; opt out via allow_private_addresses for internal targets.
- Surface a warning when Browserless renders a target page that itself
responded with a non-2xx/3xx status (X-Response-Code), so an error/anti-bot
page is not mistaken for valid visual evidence.
- Dedupe colliding output filenames instead of silently overwriting prior
captures.
Docs: comment out token: $BROWSERLESS_TOKEN in tool examples (an unset $VAR
fails AppConfig startup) and document allow_private_addresses.
* fix(community): format web_capture guard + document local Browserless startup
Address PR #3881 review: fix the lint-backend failure (ruff format on
browserless/tools.py) and add local Browserless startup instructions to
CONFIGURATION.md so reviewers can run the service to try web_fetch/web_capture.
A reasoning+content AI message with no tool calls was grouped into both an
`assistant:processing` group (the ChainOfThought panel above the bubble) and an
`assistant` group (the bubble's own Reasoning collapsible), so its reasoning
rendered twice. Such a message now becomes only the assistant bubble; the
ChainOfThought panel keeps handling intermediate reasoning and tool steps.
Closes#3868
Adds ``deerflow.community.e2b_sandbox.E2BSandboxProvider`` with parity
to AioSandboxProvider: metadata-keyed per-thread persistence, server-
side idle timeout, warm-pool reclaim with liveness checks, /mnt/user-data
bootstrap symlinks, dead-sandbox auto-rebuild, and release-time mirror
of agent outputs back to the host artifact directory.
Signed-off-by: joey <zchengjoey@gmail.com>
* fix(frontend): keep orphan tool messages visible
LangGraph `messages-tuple` stream mode can emit tool-result events
out of order or replay them from subagent state (e.g. the bash subagent
under LocalSandboxProvider with allow_host_bash: true). When that
happens, the tool message arrives after a terminal assistant/human
group, so getMessageGroups' lastOpenGroup() returns null.
The previous behaviour was console.error + drop, which silently hid
the tool result from the UI - the user could not see the tool output
or tool-call records even though the agent executed the action
correctly (backend + Langfuse trace were both fine).
Fallback now attaches the orphan tool message to the most recent group
so the UI shows it. Adds a unit test that covers the orphan-tool path
and a duplicate-stream regression test.
* test(frontend): make orphan-tool replay test actually reach the fallback
The previous fixture was `human → ai(tool_calls) → tool → tool(replay)`
with no terminal group in between, so both tool messages hit the
unchanged happy path (`open.messages.push(...)`). Neither message was
an orphan, the new `else if (groups.length > 0)` fallback was never
exercised, and the `>= 1` length assertion was satisfied trivially by
t-1a alone.
Interleave a terminal assistant message between the original tool
result and the replayed one, so t-1b arrives when lastOpenGroup()
returns null. Now the strict assertion that t-1b is reachable can only
pass via the new fallback branch.
Review feedback from @willem-bd on PR #3880.
* fix(frontend): satisfy noUncheckedIndexedAccess in orphan-tool fallback
The fallback branch pushed into `groups[groups.length - 1].messages`
directly, which trips `noUncheckedIndexedAccess` under the project's
strict TS config and breaks lint-frontend, e2e-tests, and the
full-stack render CI layer.
Take `groups[groups.length - 1]` into a local `lastGroup` and check
for `undefined` explicitly. The `else if (groups.length > 0)` guard
becomes redundant once `lastGroup` is checked, so the branches are
folded into the outer `else` to keep the structure flat. Behavior is
unchanged: orphan tools still attach to the most recent group, and
the empty-groups diagnostic still logs at ERROR level.
Review feedback from @willem-bd on PR #3880.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
After SummarizationMiddleware runs, the merged conversation view could drop
already-displayed messages (previous assistant output, current user input),
leaving a nearly-empty thread.
Root cause: the display merge combines `visibleHistory` (archived history, a
React `useState` in useThreadHistory) with `persistedMessages` (live thread,
the LangGraph SDK external store via useSyncExternalStore). On summarization
the backend removes every live message and onUpdateEvent re-archives them via
an async `appendMessages` setState. Those two state systems are scheduled
independently, so a render can observe the post-summary (shrunk) thread before
the archive setState commits — the rescued messages are then absent from BOTH
merge inputs and get dropped.
Fix: bridge the async gap with a synchronous `pendingArchivedMessagesRef`
buffer written the moment onUpdateEvent computes the moved messages and read by
the merge on every render, so correctness no longer depends on how the two
channels interleave. The buffer drains once history confirms absorption and
only injects messages missing from history (live copies stay authoritative,
order preserved). It is tagged with the thread it was captured from and the
merge overlays it only when that matches the viewed `threadId` (the same prop
visibleHistory is gated on), so it can never leak into another thread or the
new-chat screen — a read-only check, no render-phase ref mutation.
Extracts the moved-message derivation and the merge overlay into pure,
unit-tested helpers (computeSummarizationMovedMessages, resolvePreservedHistory,
pruneConfirmedArchivedMessages) with regression coverage for the full rescue
pipeline.
Closes#3192
Root cause
----------
The artifact preview header is driven by a Radix Select whose <SelectValue>
renders the label of the <SelectItem> matching the current value. The option
list was built solely from `artifacts` in ArtifactsContext, which is only
synced from `thread.values.artifacts` (chat-box.tsx). Artifacts surfaced via
the message-layer `present_files` tool call are never written back to
`thread.values.artifacts`, so when such a file is selected its filepath has
no matching <SelectItem>. Radix then renders an empty trigger and the header
filename appears blank, even though the preview body loads correctly.
Fix
---
Compute `artifactOptions` as a defensive union: if the currently selected
filepath is missing from `artifacts`, prepend it so a matching <SelectItem>
always exists. This keeps the header label in sync with the active file
without changing context semantics or coupling the UI to a future
auto-discovery mechanism (see existing TODO in chat-box.tsx).
Tests
-----
Add an e2e case that mocks a thread whose only artifact is delivered via
`present_files` (thread.values.artifacts = []) and asserts both the header
title and preview content render. All 21 e2e tests pass.
* docs: add root-level CLAUDE.md to orient the monorepo
Adds a thin top-level CLAUDE.md that maps the monorepo and delegates depth
to backend/CLAUDE.md and frontend/CLAUDE.md, per issue #3761.
Includes the project overview + service topology (Nginx 2026, Gateway 8001,
Frontend 3000, optional Provisioner 8002), a top-level repository map, root
`make` vs. per-module command sections, "where to go next" links to the module
guides and primary root docs, and the repo-wide cross-cutting conventions
(documentation-update policy, TDD expectation, format before pushing).
No code or behavior changes; root points down, modules own depth.
Closes#3761
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: make AGENTS.md the source of truth, CLAUDE.md a thin @AGENTS.md importer
Adopt the AGENTS.md convention so the same agent guidance serves Claude Code,
Codex, and other tools. At each level (root, backend, frontend) the content
lives in AGENTS.md and CLAUDE.md just imports it via `@AGENTS.md`.
- root: move the monorepo orientation layer to AGENTS.md; CLAUDE.md -> @AGENTS.md.
Fix an incorrect "TUI" reference (not present on main) and repoint the module
links to the AGENTS.md files.
- backend: move the guide to AGENTS.md (was an AGENTS.md -> @CLAUDE.md pointer;
direction is now flipped). Refresh stale content: rebuild the full middleware
chain (~26 ordered steps incl. InputSanitization, ToolOutputBudget,
DynamicContext, TokenBudget, SafetyFinishReason) from the actual build
functions; drop the brittle "11 middleware components" count; expand the
community-tools list to the real set.
- frontend: merge the practical Next.js guide with the existing AGENTS.md's
unique sections (LangGraph diagram, tech-stack versions, interaction
ownership, resources) into one AGENTS.md (CLAUDE.md -> @AGENTS.md). Fix the
stale src/ layout (remove the no-longer-present server/ better-auth entry;
add the now-active auth/agents/blog/... modules and routes) and drop a bogus
interaction-ownership bullet referencing files that don't exist.
Docs only; no code or behavior changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(frontend): improve chat math rendering
* fix(frontend): refine math rendering stability
* fix(frontend): address review feedback on math preprocessing
- Fix escaped-backslash blind spot: consume \\\\ as a unit so \\( and
\\[ are not mis-interpreted as math delimiters when the backslash
itself is escaped.
- Thread inInlineCode state across lines so multi-line backtick code
spans are protected from delimiter conversion.
- Remove double math normalization: preprocessStreamdownMarkdown now
only handles Mermaid; math normalization lives solely in
ClipboardSafeStreamdown, preventing non-idempotent double passes.
- Wire parseIncompleteMarkdown to isLoading in MarkdownContent so
Streamdown's streaming-incomplete-math handling is actually active
during streaming.
* fix(frontend): address streamdown math review issues
* refactor(frontend): move streamdown preprocessing out of ai elements
_lock was defined in __init__ and used only in reset(), leaving _apply(),
before_agent(), _clear_run_state(), and _drain_pending_warnings() unprotected.
Concurrent after_model callbacks for the same run_id could race on
usage_accum.input += diff_input, causing lost-update on token counters.
Also fix stray bare 'a' character in frontend/src/content/zh/introduction/index.mdx
that rendered as visible text on the documentation page.
Co-authored-by: kapilvus <kapilvus@gmail.com>
P0: DynamicContextMiddleware role isolation — framework data as SystemMessage,
memory as HumanMessage (OWASP LLM01), user input unchanged.
Includes E2E hash stabilization: frontend hide_from_ui filter fix in
suggest_agent, backend _canonical_messages() hide_from_ui skip, fixture
hash updates.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The login page used hardcoded English strings. Add a `login` section to
the i18n Translations interface and both locale files, then wire the
login page to `useI18n()` so all titles, labels, placeholders, buttons,
SSO hints, and error messages resolve from the active locale.
* feat(groundroute): add GroundRoute community web_search + web_fetch tools
GroundRoute is a meta search layer over six engines (Serper, Brave, Exa,
Tavily, Firecrawl, Perplexity) with price-based routing and failover. This
adds a self-contained community engine module (httpx only, no new required
deps) mirroring community/brave + community/tavily:
- web_search: POST /v1/search, normalize to {title,url,snippet,source_engine}.
- web_fetch: fetch a URL via mode=page.
- unit tests covering normalization, auth, clamping, and graceful errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(groundroute): register GroundRoute search + fetch in wizard and config
Add GroundRoute to the setup wizard provider lists (SEARCH_PROVIDERS +
WEB_FETCH_PROVIDERS) and as commented web_search + web_fetch examples in
config.example.yaml, mirroring tavily/serper/brave so SEARCH_API can select it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(groundroute): apply repo ruff format (line-length 240)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(groundroute): add GroundRoute to tools docs and config reference
Adds GroundRoute as a web_search and web_fetch option in the en + zh
tools.mdx pages (new tab alongside Tavily/Brave/Exa/etc.) and documents
GROUNDROUTE_API_KEY in CONFIGURATION.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(groundroute): define empty groundroute extra for clean install
The docs install line 'uv add deerflow-harness[groundroute]' (mirroring the
tavily/exa/firecrawl pattern) referenced an undefined extra, which uv accepts
but warns about. GroundRoute needs no extra packages (httpx is a core dep), so
declare an empty 'groundroute' extra in deerflow-harness optional-dependencies
so the documented command resolves without a warning.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(groundroute): per-tool api key + honor caller max_results (review)
Address maintainer review on PR #3675:
- _get_api_key(tool_name): web_fetch now reads the web_fetch config block's key
instead of always web_search, so a flow that pairs GroundRoute fetch with a
different search engine authenticates correctly. Mirrors serper/exa/firecrawl.
- web_search honors a caller-supplied max_results (sentinel default None),
falling back to the configured value only when omitted, so the documented
parameter is no longer silently discarded.
- warn-once is now keyed per tool. Tests cover both fixes (web_fetch key,
agent max_results honored).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): reset new chat on client-side navigation
Drive the chat reset effect with Next.js's reactive pathname instead of the render-time window.location-derived flag.
During App Router transitions, window.location may still point to the previous thread until commit, leaving chat state stale until another UI interaction triggers a render. Preserve the stale 'new' param guard so created thread UUIDs are not overwritten.
* test(frontend): add e2e to cover history-only new chat reset
* docs(frontend): clarify new chat pathname synchronization
* feat: persist AI turn duration in backend and UI
* chore: restore uv.lock to match main
* refactor(frontend): use Math.floor instead of Math.ceil for reasoning timer
