mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-07-31 02:15:59 +00:00
420 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Willem Jiang
|
8d3ff5d805
|
Merge branch 'main' into fix-2788 | ||
|
Xinmin Zeng
|
4d660b202a
|
feat(skills): bind request-scoped secrets for autonomously-invoked skills (A+) (#3938)
* feat(skills): bind request-scoped secrets for in-context (autonomously invoked) skills Extends the #3861 binding point A (slash-activation only) to A+: the injection set is recomputed on every model call from two unioned sources — the run's most recent slash activation (persisted on the run context so the tool loop keeps the binding) and skills the model actually loaded in this thread (ThreadState.skill_context), re-validated against the live registry each call. Authorization stays three-gated regardless of activation style: skill enabled by the operator, values supplied per-request by the caller in context.secrets (never persisted server-side, never from the host env), names declared in the skill's required-secrets frontmatter. Because the set is replaced per call, eviction from skill_context or a caller that stops supplying a value revokes injection on the next call. New frontmatter field secrets-autonomous (default true) lets a skill restrict binding to explicit slash activation; malformed values fail closed to false. Binding changes are recorded as a middleware:skill_secrets journal event carrying names only. Design informed by a survey of peer systems (Claude Code, Codex CLI, opencode, pi, deepagents, hermes-agent, QwenPaw) and specs (agentskills.io, MCP 2025-11-25): the industry trust boundary is enable-time consent plus caller-scoped credentials, not per-invocation ceremony; no surveyed system scopes secrets to an activation turn. Part of #3914 * refactor(skills): centralize secret context keys, document intentional per-call reload Review follow-ups (no behavior change): move the two private binding keys (__slash_skill_secret_source, __skill_secrets_binding_audit) into secret_context.py and add them to REDACTED_CONTEXT_KEYS so the redaction allowlist stays a complete guard even though both keys hold names only. Document why _in_context_secret_sources reloads skills every call rather than caching: load_skills re-reads enabled state so an operator disabling a skill revokes its binding on the next model call — an mtime cache would miss enable/disable toggles and keep injecting after a disable. * fix(skills): match in-context secret bindings by path only, never by name Review finding (confused deputy): _in_context_secret_sources fell back to name matching when a skill_context path did not resolve. DeerFlow lets a custom skill shadow a same-named public/legacy one (load_skills de-dupes by name, custom wins), so a thread that read public/foo could bind the custom foo's declared secrets although the custom skill was never loaded in the thread. The recent user-isolation path changes make by-path misses (and thus the dangerous fallback) more likely. Drop the by-name fallback: match strictly by the exact container file path the model read; an unresolved path simply does not bind (the safe direction). Regression tests cover the shadowing case and a stale path. Part of #3914 * fix(skills): resolve secret-binding sources via registry; strip caller __-keys Security review (willem-bd, #3938): 1. Forged `__slash_skill_secret_source` bypassed the enabled/allowlist/ secrets-autonomous gates. runtime.context is caller-mergeable, and the slash source was trusted as authoritative (its stored requirements were injected directly). Now the slash source records only the activated skill's canonical container path, and BOTH the slash and in-context sources resolve the live registry skill by normalized path each call (_resolve_registry_skill) — binding only that real, enabled, allowlisted skill's own declared secrets. A forged path resolves to nothing. As defense in depth, build_run_config strips caller-supplied __-prefixed context keys at the gateway boundary. 2. Malformed caller requirements crashed the run (unguarded tuple unpack / DoS). The middleware no longer unpacks caller-provided requirement data at all — declarations come from the registry — so a malformed source fails closed instead of raising. 3. Path-normalization asymmetry silently disabled in-context binding on a trailing-slash container_path config. Both the registry keys and the lookup path are now posixpath.normpath'd. Regression tests: forged source rejected, forged-but-real path ignores caller requirements + allowlist, malformed source fails closed, trailing- slash config binds, gateway strips __-keys. Part of #3914 * docs(skills): correct _SLASH_SECRET_SOURCE_KEY comment and note fail-closed trade-off Post-review cleanup: the key now stores only the canonical container path (the comment still described the pre-fix skill-name+requirements shape), and document that a transient registry-load failure fails closed (drops the binding for that call) rather than trusting stale data. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
codeingforcoffee
|
4669d3c089
|
feat(gateway): cache-aware cost accounting (#3920)
* feat(gateway): cache-aware cost accounting + /api/console observability endpoints - Capture prompt-cache hits (usage_metadata.input_token_details.cache_read) in RunJournal and SubagentTokenCollector as a sparse cache_read_tokens key in token_usage_by_model (JSON field — no schema migration; legacy bucket shapes unchanged) - New read-only /api/console router: GET /stats (headline counters), GET /runs (cross-thread paginated history joined with thread titles), GET /usage (zero-filled daily token series + per-model breakdown); user-scoped, 503 on the memory database backend - Optional models[*].pricing (currency, input_per_million, output_per_million, input_cache_hit_per_million) powers real spend estimation; cache-hit input tokens are billed at the hit price (omitted hit price falls back to the miss price as a conservative upper bound); unpriced models yield cost: null - create_chat_model strips the presentation-only pricing block so it never reaches the provider client (unknown kwargs are forwarded into the completion payload and break live calls) - Tests: console router SQLite round-trips, journal/collector cache capture incl. a DeepSeek raw-usage pin test, factory strip regression Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: address review feedback on cost sum and sparse cache_read_tokens - console.py: replace the walrus-in-generator total-cost sum with an explicit loop (review noted the multi-line form reads ambiguously) - token_collector.py: omit cache_read_tokens from usage records when the provider reported no cache hits, matching the journal's sparse per-model bucket shape; absent is treated as 0 downstream - add a regression test pinning the sparse record shape Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: coffeeFish <codeingforcoffee@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Tianye Song
|
15454b6fec
|
feat(skills): deferred skill discovery via describe_skill tool (#3775)
Replace the full-metadata <available_skills> system-prompt block with a compact <skill_index> (names only) and an on-demand describe_skill tool when skills.deferred_discovery: true (default: false / backward compat). New modules: - skills/catalog.py — SkillCatalog (immutable, searchable; select: has no cap, keyword/prefix search caps at MAX_RESULTS=5) - skills/describe.py — build_describe_skill_tool(catalog) closure; build_skill_search_setup() wires SkillSearchSetup into both the LangGraph agent factory (agent.py) and DeerFlowClient (client.py) Changes: - Skill @dataclass(frozen=True); allowed_tools/required_secrets list→tuple - Skill First prompt line gated on skill_names (deferred vs legacy wording) - get_skills_prompt_section: short-circuit storage on deferred path; merge user_id (upstream) + skill_names (this PR) params - describe_skill tool parameter named "name" (matches prompt wording) - select: branch removes [:MAX_RESULTS] cap (exact request, not ranking) - AGENTS.md: document deferred_discovery config field + new modules Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zheng Feng
|
dcb2e687d5
|
feat(channels): add GitHub as a webhook-driven channel (#3754)
* feat(channels): add GitHub event-driven agents (#3754) Add a webhook-driven GitHub channel with fail-closed webhook routing, deterministic per-agent PR/issue threads, mention-gated trigger fan-out, GitHub App token injection for sandboxed gh/git commands, and backend/AGENTS.md documentation. * fix(llm-middleware): classify bare IndexError as transient Upstream chat providers occasionally return 200 OK with an empty generations list (observed against Volces "coding" on ark.cn-beijing.volces.com). When that happens, langchain_core.language_models.chat_models.ainvoke raises ``IndexError: list index out of range`` at ``llm_result.generations[0][0].message`` and kills the run. Treat a bare IndexError reaching the middleware as a transient upstream-payload glitch and route it through the existing retry/backoff path instead of failing the whole agent run. The retry budget and backoff schedule are unchanged. Adds three regression tests covering the classifier and both the recover-on-retry and exhausted-retries paths. * fix(runtime): ignore stale LLM fallback markers from prior runs When a run on a thread ends with the LLM-error-handling middleware emitting a `deerflow_error_fallback`-marked AIMessage (e.g. after the IndexError empty-generations classification fix lands), that message is persisted to the thread's checkpoint as part of the messages channel. LangGraph replays the full message history in `stream_mode="values"` chunks, so every subsequent run on the same thread re-streams the stale fallback marker — and the worker's chunk scanner faithfully picks it up, flipping `RunStatus.success` to `RunStatus.error` for runs that themselves had no LLM failure at all. Snapshot the set of pre-existing message ids from the pre-run checkpoint and thread it through `_extract_llm_error_fallback_message` / `_try_extract_from_message` as a filter. Markers on history messages are ignored; markers on fresh messages produced during this run still trip the error path. Falls back to an empty set when the checkpointer is absent or the snapshot can't be captured, preserving the prior behavior on first-run / no-state paths. Adds unit tests for the new filter (helper-level and `_collect_pre_existing_message_ids`) plus an integration test exercising the full `run_agent` path with a stale history checkpointer. * fix(channels): make github channel fire-and-forget to avoid httpx.ReadTimeout on long runs GitHub agent runs (clone -> edit -> test -> push -> PR) routinely exceed the langgraph_sdk default 300s read deadline. The manager's runs.wait call kept an HTTP stream open for the entire run lifetime, so the long run blew up with httpx.ReadTimeout and the outer except branch then released the dedupe key and emitted a false 'internal error' outbound. The GitHub channel's outbound send is log-only by design: agents post to the issue/PR via the gh CLI in the sandbox when they choose to comment or create a PR. There is nothing for the manager to ferry back, so the long-poll was pure overhead. This change adds ChannelRunPolicy.fire_and_forget (default False) and sets it True for the github channel. When fire_and_forget is True, _handle_chat dispatches via client.runs.create (short POST, returns once the run is pending) instead of client.runs.wait, and skips the response-extraction + outbound-publish block. ConflictError on a busy thread still trips the standard THREAD_BUSY_MESSAGE path so behavior on the busy case is preserved for any future non-github fire-and-forget channel. Other (non-github) channels are unchanged: their policy defaults fire_and_forget=False and they continue to dispatch via runs.wait. Adds 6 regression tests in tests/test_channels.py::TestGithubFireAndForget: - Default ChannelRunPolicy.fire_and_forget is False. - The github policy registers fire_and_forget=True. - github inbound calls runs.create, not runs.wait, with the right kwargs. - github inbound publishes no outbound on success. - ConflictError from runs.create still emits THREAD_BUSY_MESSAGE. - Non-github channels (slack) still dispatch via runs.wait. * test(lead-agent): accept user_id kwarg in skill-policy test stubs The two GitHub-channel tests added in #3754 stubbed _load_enabled_skills_for_tool_policy with a lambda that only accepted `available_skills` and `app_config`, but the real function (and its call site in agent.py) also passes `user_id`. This raised TypeError on every run, failing backend-unit-tests. Add `user_id=None` to match the three sibling stubs in the same file. * refactor(gateway): disambiguate context-key set names The two frozensets _INTERNAL_ONLY_CONTEXT_KEYS and _CONTEXT_ONLY_KEYS shared a confusable "CONTEXT_ONLY" token in different orders, and the first broke the _CONTEXT_<X>_KEYS pattern of its sibling _CONTEXT_CONFIGURABLE_KEYS. Rename to make the distinct axes explicit: _CONTEXT_INTERNAL_CALLER_KEYS - WHO: internal callers (scheduler) only _CONTEXT_RUNTIME_ONLY_KEYS - WHERE: runtime context only, never configurable Pure rename, no behavior change. |
||
|
Xinmin Zeng
|
4fc08b4f15
|
feat: add scheduled tasks MVP (#3898)
* feat: add scheduled tasks MVP
* fix: harden scheduled task execution semantics
* feat(scheduled-tasks): preset-driven schedule form with timezone and live preview
Replace the raw cron input with a preset Select (hourly/daily/weekly/monthly/custom)
plus structured inputs (time picker, weekday toggles, day-of-month), datetime-local
for one-time tasks, a timezone selector defaulting to the browser timezone, and a
live human-readable preview. Reuses one ScheduledTaskScheduleInput for create and
edit; backend contract unchanged; zero new deps (pure Intl + DST-safe offset helpers).
* feat(scheduled-tasks): full-page i18n + recipe templates + E2E locale pin
Localize the rest of the scheduled-tasks page (filters, detail pane, actions,
edit form, run list, enum values) via t.scheduledTasks.* in en/zh. Add four
built-in recipe templates (GitHub Trending, news digest, issue triage, weekly
report) exposed as a chip row that pre-fills title + prompt + schedule. Pin
Playwright locale to en-US so E2E selectors stay stable against i18n. No backend
change, no new deps.
* fix(scheduled-tasks): idempotent 0003 migration, update head constants, future-date once test
Merge with main surfaced three CI failures:
- 0003_scheduled_tasks create_table collided with legacy test seeds that
build from full metadata; guard with inspector.has_table so the revision
no-ops when the table already exists (0004/0005 are already idempotent via
_helpers.py).
- persistence bootstrap concurrency/regression tests pinned HEAD to main's
0002_runs_token_usage; bump to the new head 0005_scheduled_task_thread_nullable.
- once-task router test used a fixed past run_at and tripped the
must-be-in-the-future validation; use a future date.
* address review: ok-check, 502 for trigger failure, mock fields, migration filename, doc fences
- fetchThreadScheduledTasks now checks response.ok like the other fetchers.
- trigger endpoint returns 502 (not 409) when dispatch fails outright, so
clients can distinguish a real conflict from a server-side failure.
- E2E mock normalizes scheduled-task objects with context_mode/last_thread_id
and nullable thread_id, matching the backend contract the UI renders against.
- Rename 0002_scheduled_tasks.py -> 0003_scheduled_tasks.py to match its
revision id (file was renamed in spirit already; filename now follows).
- CONFIGURATION.md: close the Tool Groups yaml fence and drop the stray fence
after the Scheduler notes so the sections render correctly.
* fix(scheduled-tasks): harden lease, poller, config, and frontend UX after review
* fix(scheduled-tasks): harden run lifecycle, overlap skip, non_interactive gating, and DST conversion after review
- defer a once task's terminal status to the run-completion hook; the task
stays running until the real outcome, and a startup sweep cancels once
tasks orphaned by a crash (launch-time 'completed' could stick forever)
- record interrupted runs as a distinct 'interrupted' run status with a
readable message; an interrupted once task ends 'cancelled', not 'failed'
- enforce overlap_policy=skip for fresh_thread_per_run via an active-run
pre-check (same-thread ConflictError can never fire across fresh threads)
- protect terminal run statuses from the late launch-path 'running' write
- honor context.non_interactive only for internally-authenticated callers;
arbitrary clients can no longer strip ask_clarification
- fix DST-stale timezone offset in zonedLocalToUtcIso by re-deriving the
offset at the resolved instant (once tasks fired an hour late around
spring-forward and the create->edit round-trip diverged)
- drop dead ScheduledTaskRunRepository.update_by_run_id; share one Gateway
API error helper between channels and scheduled-tasks frontends
* fix(scheduled-tasks): close review round-3 gaps in guards, concurrency, and API ergonomics
- scrub internal-only context keys (non_interactive) from the assembled run
config for non-internal callers: gating body.context alone left the same
key smuggle-able through the free-form body.config copied verbatim by
build_run_config
- guard update_after_launch with protect_terminal so the launch bookkeeping
write cannot clobber a once task already finalized by a fast-failing run's
completion hook (parent-row sibling of the run-row guard)
- reject a manual trigger while the task has an active run (409) instead of
launching a duplicate concurrent run on fresh_thread_per_run
- re-arm a terminal once task to enabled when PATCH pushes run_at into the
future; previously the endpoint returned 200 with a next_run_at that could
never be claimed
- make max_concurrent_runs a real global cap: each poll claims only into the
remaining budget of active (queued/running) scheduled runs
- paginate GET /scheduled-tasks/{id}/runs (limit<=200, offset) and push the
thread filter of /threads/{id}/scheduled-tasks into SQL
- stamp context.user_id on scheduler-launched runs, matching IM channels, so
user-scoped guardrail providers see the owning user
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
AnoobFeng
|
5acd0b3ba8
|
fix(gateway): offload gateway upload file IO (#3935)
* fix(gateway): offload gateway upload file IO Move Gateway upload router filesystem work off the asyncio event loop by using a dedicated ContextVar-preserving file IO executor. Use async sandbox acquisition for non-mounted sandbox uploads and offload remote sandbox sync together with host file reads. Add blocking-IO regression coverage for upload, list, delete, and remote sandbox sync paths. * fix(gateway): align file IO worker env var prefix Rename the file IO executor worker-count environment variable from DEERFLOW_FILE_IO_WORKERS to DEER_FLOW_FILE_IO_WORKERS to match the repo's existing runtime configuration prefix convention. |
||
|
Xinmin Zeng
|
576577bd32
|
feat(channels): expose IM channel_user_id to sandbox commands as DEERFLOW_CHANNEL_USER_ID (#3926)
* feat(channels): expose channel_user_id to sandbox commands as DEERFLOW_CHANNEL_USER_ID IM-channel skills need the sender's platform identity (Feishu open_id, Slack Uxxx, ...). The channel manager already writes channel_user_id into body.context, but the Gateway whitelist dropped it. Forward it into the runtime context only (never configurable, which is checkpointed), and have bash_tool export it as a fixed env var through a shell-quoted command prefix. The identity deliberately does not ride execute_command(env=...): that channel is reserved for request-scoped secrets, and a non-empty env switches AioSandbox onto the bash.exec path (fresh session per call, image >= 1.9.3 required), which would have broken every IM bash command on older sandbox images and abandoned persistent-shell semantics on new ones. A command-string export keeps the legacy path, stays visible in audit logs (it is an identifier, not a secret), and gives per-call correctness in group chats where one thread and sandbox are shared by senders with different platform ids. Skipped on the Windows local sandbox, whose PowerShell/cmd.exe fallback has no POSIX export. Part of #3914 * feat(channels): propagate channel_user_id to subagents; cap value length Review findings from the pre-PR verification pass: - Subagent delegation dropped the sender identity: task_tool now captures channel_user_id from the parent runtime context and the executor forwards it into the subagent's context, mirroring the guardrail attribution fields (user_role/oauth_*/run_id). Without this, bash commands delegated via task lost the group-chat sender's id. - body.context is client-writable on web requests, so values over 256 chars are ignored instead of bloating every command string sent to the sandbox. * fix(channels): set-or-unset channel_user_id so identity is per-call regardless of AIO session persistence Review (willem-bd): the identity export could leak across senders in a shared group-chat AIO sandbox. The AIO no-env path reuses a persistent shell session (the class-lock reason, #1433), and the 256-char/type guard made some commands carry no prefix — so a dropped-id command could resolve the id a previous sender exported. Make per-call correctness independent of session semantics: an IM-channel command (channel_user_id present in context) now always carries an explicit prefix — export VAR=<quoted> for a valid id, or unset VAR for an unusable one (empty / non-str / over the cap). Non-IM runs (no key) are untouched. A prefix unset has none of the '& ; unset' suffix hazard raised earlier. Verified on a real AIO 1.11.0 container: the no-id shell path auto-creates a session per call (does not persist today), but an explicit shared session DOES persist (export stale-A -> readback [stale-A]); the unset prefix clears it (-> []). So the fix holds even on an image whose no-id path persists. Regression tests cover the dropped-id group-chat window and the non-IM passthrough. Part of #3914 * test(channels): align channel_user_id task test with new Command return shape The merge from main changed task_tool to return a Command(update=...) instead of a plain string; update the assertion to extract the tool message via the existing _task_tool_message helper, matching the sibling tests. Fixes the CI backend-unit-tests failure introduced by the merge. |
||
|
hataa
|
80e031dcc6
|
fix(subagents): inherit LoopDetectionMiddleware to break tool loops (#3875) (#3931)
Subagents build their runtime chain via build_subagent_runtime_middlewares, which attached none of the lead's runaway guards — no LoopDetectionMiddleware. A degenerate subagent tool loop therefore ran unchecked until max_turns, re-sending a growing context each turn (the #3875 4.4M-token burn). Append LoopDetectionMiddleware.from_config(app_config.loop_detection) to the subagent chain, gated by the existing loop_detection.enabled config (default on) — no new config field. Registered before SafetyFinishReasonMiddleware to match the lead's after-model ordering (the safety middleware requires placement after LoopDetection). Subagents disallow task, so only the tool-loop heuristic can fire here — no recursive-delegation false positives to worry about. Phase 1 of #3875 (smallest blast radius). Phase 2 adds a deterministic turn/token budget with a lead-visible stop reason; Phase 3 defers summarization. Tests: enabled/disabled wiring + before-SafetyFinish ordering, mirroring the lead chain's coverage. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Janlay
|
38342b15a3
|
fix(redis): stream retention recovery (#3933)
* fix(gateway): retain redis streams safely * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Zhipeng Zheng
|
53a80d3ad1
|
feat(skills): per-user custom skill isolation with sandbox mounting (#3889)
* feat(skills): per-user skill isolation (#2905) Implement user-scoped skill storage that isolates custom skills between users while sharing public skills globally. Key changes: - Add UserScopedSkillStorage class for per-user custom skill directories - Introduce get_or_new_user_skill_storage() factory with user_id context - Auth middleware sets effective_user_id for request-scoped storage - Agent/prompt/middleware now use user-scoped storage and prompt cache - Sandbox mounts user-scoped skill directories for search/read tools - Add validate_skill_file_path() to SkillStorage for path security - Migration script supports --all-users bulk migration - Frontend: add editable field to Skill type, error check in enableSkill - All skill categories can be toggled (custom skills default to enabled) - Update skill-creator SKILL.md with isolation-aware instructions Tests: - Add test_user_scoped_skill_storage.py (new) - Update all existing skill tests for user-scoped storage - Update sandbox, client, and router tests * fix(skills): address second-round PR review feedback (#3889) - P1-1: restrict legacy skill mount to users without custom skills - P1-2: fail-closed for _is_disabled_skill_path (OSError → return True) - P2-1: AND-merge global extensions_config skill disabled state - P2-2: atomic write for _skill_states.json (mkstemp + replace) - P2-3: normalize X-DeerFlow-Owner-User-Id in trusted boundary - P2-4: LRU-bounded _enabled_skills_by_config_cache (OrderedDict, maxsize=256) - P2-5: clear global prompt cache on PUBLIC skill toggle - P2-6: invalidate skill caches on client.update_skill * fix(tests): correct tool policy test after merge * fix(skills): use DEFAULT_SKILLS_CONTAINER_PATH in UserScopedSkillStorage The "/mnt/skills" literal in UserScopedSkillStorage.__init__ triggers test_skill_container_path_defaults::test_mnt_skills_literal_is_owned_by_skill_constants_module on CI. Migrate the default to the existing deerflow.constants constant, matching the pattern already used by LocalSkillStorage, SkillStorage, and the durable/tool_error middlewares. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
AochenShen99
|
66b9e7f212
|
feat: emit structured runtime metadata (follow-up#3887) (#3906)
* feat: emit structured runtime metadata * fix: avoid subagent import cycle in replay gateway * fix: preserve legacy subtask result parsing * refactor: tighten runtime metadata contracts * fix(middleware): keep recovery hint on task exception wrapper content The structured-metadata stamp overwrote the wrapper text with the bare task-failure message, dropping the model-facing 'Continue with available context, or choose an alternative tool.' guidance that every other tool exception keeps. Append the shared hint after the formatted message. * fix(subagents): require lowercase hex for result_sha256 reader Length-only validation accepted any 64-char string; a faulty serializer or relaying wrapper could store a non-digest value in the delegation ledger. Enforce the producer's hexdigest shape with a fullmatch. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
AochenShen99
|
9a088805d7
|
fix(skills): close skill install security scan coverage gap (#3924)
* fix(skills): close install-path scan coverage gap Skill installs only sent scripts/* and document files under references/ and templates/ to the LLM security scanner. Code at the skill root or under lib/, bin/, src/, etc., and binary files could be installed without any scan. - Scan code files anywhere in the skill tree (by extension, plus shebang detection for extensionless files) with the executable policy: only an explicit allow admits them. - Reject ELF/PE/Mach-O executable binaries by magic bytes during safe archive extraction; non-executable binary assets remain allowed. Interim hardening ahead of the SkillScan framework (RFC #2634); the deterministic full-tree scanner from PR #3033 supersedes the per-file LLM coverage when it lands. * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * test(skills): pin magic coverage and shebang offload boundary Follow-up to the applied review suggestions: cover every Mach-O magic variant with tests (plus the fat64 pair and a partial-prefix asset that must stay installable), name the pure classification helper so the call-site logic reads as policy, drop the now-unused sync _is_code_file, and pin that only extensionless files get the shebang sniff. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Aari
|
94c852ac4a
|
fix(mcp): route tools by source server, not name prefix (#3812)
* fix(mcp): route tools by source server to fix prefix-collision mis-routing get_mcp_tools() flattened the per-server tool lists and then re-derived each tool's server by scanning servers_config for a name that is a prefix of the (prefixed) tool name, taking the first match. When one server name is a prefix of another (e.g. "web" and "web_scraper"), a "web_scraper_search" tool matches "web" first, so it is pooled under the wrong server and invoked with the wrong stripped name — the call fails, or for mixed transports the stdio tool loses session pooling. Route each tool by the server that actually produced it (tools_by_server[i] corresponds to the i-th server in servers_config), keeping the prefix guard so unprefixed tools still fall through unwrapped. Add a regression test covering overlapping server-name prefixes. * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Janlay
|
72f033fbbe
|
feat(gateway): add redis stream bridge (#3191)
* feat: add redis stream bridge * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(gateway): address redis stream bridge review Redis was imported eagerly through deerflow.runtime and declared as a hard dependency, which made memory-only installs load redis.asyncio at startup and left the lazy factory import ineffective. Move redis behind an optional extra, remove the public eager re-export, and keep make_stream_bridge as the only runtime import path with an actionable install hint when the extra is missing. Because Docker deployments now default the stream bridge to Redis via DEER_FLOW_STREAM_BRIDGE_REDIS_URL, install the redis extra explicitly in Docker/dev container flows and teach the local uv-extra detector to infer redis from both stream_bridge.type and the Redis URL env var. This keeps Docker working while preserving slim non-Docker installs. Harden the Redis bridge by batching XREAD replay, replacing brittle ResponseError string matching with a single fallback to 0-0 for malformed Last-Event-ID values, documenting connection/retention/fail-hard behavior, and adding fake plus opt-in real Redis coverage for XADD/XREAD, replay, invalid IDs, and MAXLEN trimming. * fix(config): bump config version for stream bridge * fix redis stream bridge terminal handling * fix: repair uv.lock, format redis.py, and align Dockerfile extras test The uv.lock file was missing a closing bracket for the redis extras section, redis.py had a formatting issue caught by ruff, and the Dockerfile extras test did not account for the hardcoded --extra redis flag. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
thefoolgy
|
76aa599107
|
fix(backend): limit uploaded file context manifest (#3917)
* limit uploaded file context manifest * fix: address uploaded file context review |
||
|
Xinmin Zeng
|
48477d868b
|
fix(sandbox): fail fast when the AIO image lacks bash.exec for env injection (#3922)
Images older than all-in-one-sandbox 1.9.x have no /v1/bash/* routes, so every env-bearing command (skills declaring required-secrets) surfaced a raw nginx 404 that the model kept retrying. Detect the 404, remember the capability gap per sandbox instance, and return an actionable error that names the minimum image version and the sandbox.image remediation. No fallback through the legacy shell path on purpose: /v1/shell/exec has no env parameter, and every workaround puts the secret values back into the command string or on disk — the exact leak surfaces the request-scoped secrets design closed. Closes #3921 |
||
|
kongxiangxin
|
9d7e131340
|
fix(skills): preserve read_file for lead skill loading (#3862) (#3863) | ||
|
Nan Gao
|
b81334ccfe
|
feat(middlewares): deterministic read-before-write version gate for file tools (#3911) (#3912)
* docs(specs): read-before-write gate design for issue #3857 output layer * docs(plans): read-before-write gate implementation plan (#3857) * refactor(sandbox): extract read_current_file_content helper (#3857) * feat(middlewares): read-before-write version gate for file tools (#3857) * test(middlewares): pin async read-before-write gate paths (#3857) * feat(config): wire ReadBeforeWriteMiddleware into runtime chain, default on (#3857) * docs(sandbox): document read-before-write gate in tool docstrings and AGENTS.md (#3857) * docs(plans): align plan doc with landed config_version (17) and drop machine-specific paths Addresses Copilot review comments on #3912. * fix(middlewares): read-before-write gate — error-string sandboxes fail open; serialize gate+execution per path (#3912 review) - AIO/E2B read_file reports failures (incl. missing files) as 'Error: ...' strings instead of raising; the gate treated that string as existing file content and blocked first-write creation. Error-string reads now count as uninspectable: gate fails open, no mark is stamped. - LangGraph runs one AIMessage's tool calls concurrently, so two same-turn writes could both pass on one stale mark before either mutation landed (and a read mark could hash a version the model never saw). Gate check + tool execution (and read + mark stamping) now share a per-(thread, path) critical section, separate from the tool-internal file_operation_lock. |
||
|
Miracle778
|
e5d361876a
|
feat(guardrails): persist security interventions as run events (#3837)
* feat: record guardrail decisions in run events Persist security-relevant GuardrailMiddleware outcomes (deny and provider-error fail-open/fail-closed) as middleware:guardrail run events via RunJournal.record_middleware(), mirroring SafetyFinishReasonMiddleware. Recording is best-effort: it reads __run_journal from the runtime context and swallows persistence failures, so tool execution behavior is unchanged. The audit payload records tool name/id, agent id, subagent flag, user role, allow decision, policy id, reason codes/messages, fail_closed mode, and provider_error flag. Tool input/args and identity fields (user_id, oauth_*) are deliberately excluded to avoid persisting the sensitive content being blocked. The fail-closed provider-error branch returns the denied message directly from the except block so it records exactly once and does not fall through to the generic deny branch. * docs: clarify guardrail journal runtime boundary * fix: preserve subagent attribution in guardrail events * test: align guardrail event attribution fixtures * refactor(guardrails): resolve runtime context once per tool call Extract `_resolve_context` and thread the already-resolved context dict through `_build_request` and `_record_guardrail_event` so the getattr/runtime.context chain runs once per wrap_tool_call instead of twice. Also document the `is_subagent` field boundary: native subagents do not inherit __run_journal, so the field is structurally False in persisted records today; custom runtimes may still supply it with attribution. Addresses review feedback on #3837 (context-read-twice cleanup and the is_subagent trade-off note). No behavior change. |
||
|
bjtolo
|
c9a5f23e7b
|
feat(mcp): add per-server tool_call_timeout for MCP tool calls (#3843)
* feat(mcp): add per-server tool_call_timeout for MCP tool calls Add a configurable timeout for individual MCP tool calls to prevent agent runs from blocking indefinitely when an MCP server becomes unresponsive (e.g., rate-limited HTTP API, hung subprocess). Uses the MCP SDK's built-in read_timeout_seconds parameter on ClientSession.call_tool, which handles the timeout within the session's own task — avoiding cross-task cancellation issues with the session pool (ref #3379, #3203). Config field is named tool_call_timeout (not timeout) to avoid collision with langchain-mcp-adapters' existing timeout field on HTTP/SSE connections. Closes #3840 * fix(mcp): read tool_call_timeout from McpServerConfig, not connection dict The previous implementation put tool_call_timeout into the connection dict returned by build_server_params, which langchain's create_session then passed to _create_stdio_session(), causing TypeError. Now reads the timeout directly from ExtensionsConfig.mcp_servers where the wrapper is built, keeping it out of the connection dict entirely. Fixes P1 bug from review on #3843. * test(mcp): regression test for tool_call_timeout not leaking into connection dict Adds two tests: - test_build_server_params_excludes_tool_call_timeout: verifies the connection dict returned by build_server_params() does NOT contain tool_call_timeout - test_stdio_tool_call_timeout_does_not_raise_typeerror: end-to-end test that get_mcp_tools() with a stdio server having tool_call_timeout configured loads tools without TypeError from _create_stdio_session() Regression for PR #3843 P1 bug. * fix(mcp): only pass read_timeout_seconds when tool_call_timeout is set When tool_call_timeout is None, don't pass read_timeout_seconds=None to session.call_tool(). This avoids breaking existing tests that assert on exact call_tool arguments without the extra kwarg. * docs(mcp): clarify stdio tool timeout |
||
|
bjtolo
|
cd982d6751
|
fix(sandbox): normalize Windows backslash paths to forward slashes in bash commands (#3869)
* fix(sandbox): normalize Windows backslash paths to forward slashes in bash commands
On Windows, `replace_virtual_paths_in_command()` and
`LocalSandbox._resolve_paths_in_command()` resolve virtual paths
(/mnt/skills, /mnt/user-data, /mnt/acp-workspace, custom mounts) to
host paths with backslashes (C:\Users\admin\...). Bash interprets
\U, \a, \d, \s, \n, \t as escape sequences, mangling the path.
Fix: add `.replace("\\", "/")` to all path resolution callbacks so
resolved paths use forward slashes, which bash handles correctly on
all platforms and Windows APIs/Python's open() accept natively.
Fixes #3865
* test(sandbox): regression tests for Windows backslash path normalization
Covers replace_virtual_paths_in_command and LocalSandbox._resolve_paths_in_command
with Windows-style backslash paths, asserting no backslashes survive in output.
Closes #3865
* test(sandbox): fix ACP test failure, keep 6 passing regression tests
Removed test_acp_workspace_no_backslash which triggered path traversal
validation. Remaining 6 tests cover user-data (3), skills (1), and
LocalSandbox custom-mount (2) paths — all with Windows backslash paths.
* test(sandbox): restore ACP Windows path normalization coverage
|
||
|
DanielWalnut
|
25ea6970a6
|
feat(runtime): implement goal continuations (#3858)
* implement goal continuations * fix(goal): address review findings for goal continuations - goal: key the no-progress breaker on a signature of the latest visible assistant evidence instead of the evaluator's volatile free-text, so it actually fires on stalled turns; thread the signature through every worker persist / no-progress call site - goal: align _stand_down_reason default caps with should_continue_goal (8 / 2) so the two gate functions agree on goals missing the fields - runtime: offload the synchronous checkpointer fallback via asyncio.to_thread (goal.py + worker.py) to keep blocking IO off the loop - frontend: i18n the GoalStatus "Goal" label (goalLabel in en/zh/types) - frontend: extract pure composer helpers into input-box-helpers.ts with unit tests (parseGoalCommand, readGoalResponseError, skill suggestions) - tests: cover the evidence-based no-progress and default-cap behavior - docs: align backend/AGENTS.md goal paragraph with actual behavior - e2e: prettier-format chat.spec.ts (fixes the lint-frontend CI failure) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): hide goal continuation counter until the agent continues The goal status bar rendered a raw "0/8" before any auto-continuation, which read as a mysterious score. Now the counter is hidden until continuation_count > 0, then shows "Continuing N/M" with a tooltip explaining the auto-continuation cap. - Extract getGoalContinuationDisplay into a pure helper (hides at 0) + unit tests - Add goalContinuing / goalContinuationTooltip i18n keys (en/zh/types) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(goal): address review findings for goal continuations Frontend correctness - Fix the optimistic /goal result permanently shadowing server goal state: the streamed continuation counter never surfaced for a goal set in-session. Extract a shared useActiveGoal hook (used by both chat pages) that reconciles the optimistic copy with server state via a goalReconciliationKey, de-duping the copy-pasted goal block across the two pages. - Stop /goal status|clear failures from escaping handleSubmit as unhandled rejections (handleGoalCommand now returns success; the run only starts when a goal was actually saved). - Use a function replacer for the goal-status toast so an objective containing $&/$1 isn't treated as a replacement pattern. Backend cleanliness / correctness - De-duplicate four byte-identical helpers (_call_checkpointer_method, _message_type, _additional_kwargs, _is_visible_message) by importing them from runtime.goal instead of re-defining them in the run worker. - Remove the dead `checkpoint_tuple.tasks` durability guard (CheckpointTuple has no tasks field) and document that pending_writes is the durability signal. - Decompose the 176-line _prepare_goal_continuation_input: extract _reread_goal_and_checkpoint and a _persist closure so the thread-unchanged guard and stand-down persistence aren't open-coded three times. Document the last-writer-wins write-window limitation as a follow-up. - Add a shared parse_goal_command helper and use it from the TUI and IM-channel /goal handlers (one place for the status/clear/set semantics). Tests - Restore the 11 command-registry tests dropped by the previous goal change (filter_commands ranking/description, build_registry builtins/skills, resolve cases) alongside the new goal tests. - Add coverage for the IM-channel _handle_goal_command, the TUI _handle_goal handler, parse_goal_command, and goalReconciliationKey. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix goal review feedback * fix goal continuation checkpoint races * prioritize goal commands while streaming Route composer submits through a shared helper so /goal commands can be handled before the streaming stop shortcut, while ordinary streaming submits still stop the active run. Testing: cd frontend && pnpm exec rstest run tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; cd frontend && pnpm check * preserve goal status during clarification Keep omitted stream goal fields distinct from explicit null clears so clarification interrupts do not hide an active thread goal that is still present in the checkpoint. Testing: pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check * style: format active goal hook Run Prettier on use-active-goal.ts to satisfy the frontend lint workflow formatting gate. Testing: pnpm format; pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check * fix goal review followups --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
AnoobFeng
|
e3e5c73b03
|
feat(observability): add trace-id correlation and enhanced logging (#3902)
* feat(observability): add trace-id correlation and enhanced logging - add opt-in gateway request trace correlation via X-Trace-Id - enhance logging with configurable trace_id-aware formatting - propagate deerflow_trace_id into runtime context and Langfuse metadata - keep enhanced logging disabled by default to preserve existing behavior * fix: harden trace correlation wiring - Make logging enhancement a restart-required startup snapshot and remove per-request config reads from TraceMiddleware - Restrict trace ids to printable ASCII before writing them to response headers, logs, and Langfuse metadata - Gate implicit DeerFlowClient trace-id creation behind logging.enhance.enabled while preserving explicit caller opt-in - Bind embedded client trace context per stream step to avoid generator ContextVar leaks and cross-context reset errors - Rebind memory update trace ids in Timer/executor worker paths so enhanced logs keep the captured correlation id - Remove unrelated __run_journal context overwrite from the trace-correlation change set * fix(gateway): avoid eager app construction on package import * fix(gateway): avoid config load during app import Keep Gateway app construction import-safe when config.yaml is absent by disabling TraceMiddleware only for that construction-time fallback path. Startup lifespan still performs strict config loading before serving. |
||
|
Xinmin Zeng
|
09988caf95
|
feat(skills): request-scoped secrets for skills (closes #3861) (#3871)
* feat(sandbox): per-call env injection + platform-secret scrubbing for skills Add an env parameter to Sandbox.execute_command (abstract + local + AIO) so request-scoped secrets can be injected into skill subprocesses, and scrub platform credentials (*KEY*/*SECRET*/*TOKEN*/*PASSWORD*/*CREDENTIAL*) from the inherited environment by default so scoped injection is not security theatre. LocalSandbox always passes an explicit scrubbed env; AioSandbox routes env-bearing commands through bash.exec(env=) on a fresh session and leaves the legacy persistent-shell path unchanged. Part of #3861. BEHAVIOR CHANGE: execute_command no longer inherits the full os.environ; Windows encoding tests updated to assert the scrubbed dict. * feat(skills): parse required-secrets frontmatter declaration Add SecretRequirement and Skill.required_secrets, and parse the required-secrets SKILL.md frontmatter field (a string list or {name, optional} mappings), dropping malformed entries with a warning so one bad declaration does not invalidate the skill. The declared name is both the context.secrets key and the env var injected at activation. Part of #3861. * feat(runtime): request-scoped secret carrier (context.secrets) Add SECRETS_CONTEXT_KEY + extract_request_secrets, centralising the context.secrets carrier contract. The existing context passthrough (build_run_config -> _build_runtime_context) already carries the sub-key to runtime.context without mirroring it into configurable; characterization tests lock that behaviour. Part of #3861. * feat(skills): inject declared secrets at slash-activation into bash env Binding point A: when a skill is slash-activated, SkillActivationMiddleware resolves its declared required-secrets against the request's context.secrets and writes the per-run injection set to runtime.context. The bash tool forwards that set to execute_command(env=). A skill cannot harvest a host platform credential (is_host_platform_secret guard, cf. GHSA-rhgp-j443-p4rf), and injected values are redacted from bash output (mask_secret_values) so an echoed secret never re-enters the prompt/trace. Part of #3861. * test(skills): lock the five secret leak surfaces + add trace redaction helper Regression tests assert the secret value is absent from all five surfaces: prompt (activation message), checkpoint (graph state vs context separation), audit (journal records names only), trace (metadata builder never copies context; never mirrored to configurable), and stdout (mask_secret_values). Add redact_secret_context_keys as a defensive helper for any context serialization. Part of #3861. * docs(backend): document request-scoped secrets for skills Add Request-Scoped Secrets subsection (Skills) + env policy note (Sandbox) and the execute_command(env=) signature change, per the doc-sync policy. Part of #3861. * fix(skills): close gaps found by end-to-end verification of request-scoped secrets Real-gateway e2e + independent review of #3861 surfaced three defects, now fixed: 1. Slash activation never fired in the live chain. InputSanitizationMiddleware wraps user input in BEGIN/END markers before SkillActivationMiddleware sees it, and the original text was only preserved when an upload or IM channel set it. For a plain text message the slash command became undetectable, so no secret was ever resolved. Fix: the sanitizer now setdefaults the pre-wrap text into ORIGINAL_USER_CONTENT_KEY (additive; sanitization behaviour unchanged), so slash activation works for all messages. Pre-existing latent bug surfaced here. 2. The raw request config (with context.secrets) was persisted to runs.kwargs_json and echoed by the run API (RunResponse.kwargs). Fix: redact_config_secrets() strips secret-bearing context keys from the persisted/echoed copy in start_run; the live config that drives the run keeps them. build_run_config now also sets configurable.thread_id on the context path (the checkpointer requires it). 3. Connection-string credentials (DATABASE_URL, REDIS_URL, SENTRY_DSN, GH_PAT, ...) were not scrubbed from the inherited sandbox env. Fix: env_policy adds a *DSN* pattern plus an explicit connection-string denylist (no blanket *URL* — benign service URLs stay readable). Verified end-to-end via a real gateway run (real LLM + skill activation + bash): the secret reaches the sandbox subprocess and appears in NONE of prompt, trace, checkpoint, audit, stdout, runs.kwargs_json, or the run API. Part of #3861. * docs(backend): document the env scrub, persistence redaction, and sanitizer interaction Sync the Request-Scoped Secrets section with the verification-driven fixes: inherited-env scrub (incl. connection-string denylist), run-record/run-API redaction as the 6th sealed leak surface, and the sanitizer preserving original content so slash activation fires. Part of #3861. * fix(skills): inject caller secret over scrubbed host value; drop redundant host-name guard A real-world demo (a skill calling a third-party cloud API with a request-scoped key) exposed that the is_host_platform_secret guard was both wrong and harmful: it refused to inject a caller-supplied secret whenever a same-named variable existed in the Gateway env — which is exactly the #3861 use case (a per-user key overriding a shared platform key). The guard was also redundant: build_sandbox_env already scrubs secret-looking names from the inherited env before injection, so a skill can never read a host credential — it only ever receives the caller's value. Remove the guard; the injected (caller) value simply wins over the scrubbed host value. Verified end-to-end: the agent called the real cloud API successfully with the caller's key, the host's same-named key was scrubbed and never used, and the caller's key leaked to none of the surfaces. Part of #3861. * fix(skills): address review on request-scoped secrets (#3861) Review fixes from PR #3871: - E2BSandbox.execute_command now accepts env/timeout and routes them to commands.run(envs=, timeout=). The bash tool passes env= unconditionally, so the prior signature (command only) raised TypeError on every e2b bash call and broke e2b deployments entirely. env=None stays backward-compatible. - SkillActivationMiddleware clears the active-secret set before resolving each activation, so a later skill in the same run never inherits an earlier skill's injection set (the #3861 contract: a skill only receives what the caller supplied AND that skill declared). - AioSandbox env path uses a dedicated _DEFAULT_HARD_TIMEOUT — bash.exec exposes no idle/no-change timeout, so the prior reuse of the legacy idle constant conflated wall-clock vs idle semantics. The env path also retries on the ErrorObservation signature now, sharing the legacy persistent-shell recovery contract. - mask_secret_values skips values below a minimum length floor so a short declared secret (e.g. "42") cannot shred unrelated bytes (exit codes, timestamps, sizes) of tool output. The secret is still injected into the subprocess; only the output mask skips it. session_id reuse on the env path is intentionally NOT added: a shared session could let request-scoped secrets ride the session env into later commands, which the SDK does not contractually forbid. The fresh-session choice matches the LocalSandbox model (each call is a fresh subprocess); the trade-off (consecutive env-bearing calls do not share cwd/venv/exports) is documented on _execute_with_env. |
||
|
Huixin615
|
b476c7a18d
|
fix(store): honor unified database configuration (#3904)
* fix(store): align store backend with database config * fix(store): preserve no-config memory fallback --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Yi Deng
|
4e6248f013
|
fix(gateway): clamp client-supplied recursion_limit to prevent runaway runs (#3903)
build_run_config() copied every top-level request key (except
configurable/context) verbatim into the LangGraph RunnableConfig, including
recursion_limit. The server default of 100 was fully overridable by the caller
with no upper bound, so a request like {"config": {"recursion_limit": 100000000}}
could make a single run execute effectively unbounded LangGraph super-steps
(each >= 1 LLM call), enabling runaway API cost / DoS.
Validate the client value server-side and clamp it into a safe range:
- valid positive ints are capped at a configurable ceiling
(AppConfig.max_recursion_limit, default 1000 to match the existing
frontend/public-skill default so legitimate deep runs are unaffected)
- invalid/non-positive/bool/None values fall back to the 100 server default
- applied on both the configurable and the LangGraph >= 0.6.0 context paths
- WARNING logged on clamp for observability
Add unit tests (including a configurable-ceiling case), expose
max_recursion_limit in config.example.yaml (config_version 16), and document
the ceiling/fallback in backend/docs/API.md.
Co-authored-by: DengY11 <DengY11@users.noreply.github.com>
|
||
|
ajayr
|
1f74082987
|
feat(community): add Crawl4AI web_fetch provider (#3821)
* feat(community): add Crawl4AI web_fetch provider
Crawl4AI is a self-hosted, no-API-key web fetcher: it runs headless
Chromium and returns server-cleaned "fit" markdown directly via its
POST /md endpoint, so no client-side readability extraction is needed.
It sits alongside the existing self-hosted Browserless provider.
- deerflow.community.crawl4ai: async Crawl4AiClient + web_fetch_tool
(reads base_url/timeout_s/token/filter from config; "Error:" string
convention; 4096-char cap), mirroring the browserless provider
- tests: 17 unit cases (success, HTTP error, success:false, empty,
timeout, request error, token header, truncation, config reads)
- config.example.yaml: commented web_fetch example
- doctor: register as a no-key (free) web_fetch provider
- setup wizard: add to WEB_FETCH_PROVIDERS (no API key)
- docs: README, CONTRIBUTING, CONFIGURATION, AGENTS provider lists
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(community): address Crawl4AI provider review feedback
- timeout: robust _coerce_timeout (bool / non-numeric -> default) mirroring
jina, so 'timeout: off' no longer becomes 0.0 and times out every request
- read web_fetch config once per invocation and pass values into the client,
so a concurrent hot-reload can't split base_url from filter
- rename config key timeout_s -> timeout to match jina/infoquest (the
default providers); update config.example.yaml + setup wizard
- validate + normalize the markdown filter against {fit,raw,bm25,llm};
unknown values fall back to fit with a warning instead of an opaque HTTP 400
- client: a non-JSON 200 body (reverse proxy / auth wall) now reports the
content-type + snippet instead of a generic JSONDecodeError
- tests: 22 cases (added non-JSON-200, _coerce_timeout, _coerce_filter,
invalid-filter fallback, read-config-once)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Ryker_Feng
|
ddb097a72f
|
feat(community): add Brave image search community tool (#3866)
* Add Brave image search community tool * fix(community): length-cap Brave web_search queries Apply _clean_query in web_search_tool so over-long queries are trimmed to Brave's 400-char limit before the API call, matching image_search_tool and avoiding HTTP 422 from the Brave Search API. * fix(community): harden Brave image search SSRF guard and dimension mapping Address PR review findings: - Catch ValueError from urlparse so a malformed bracketed-IPv6 URL skips one item instead of crashing the whole image_search call - Reject IPv6 literals embedding a non-global IPv4 (IPv4-mapped, 6to4, NAT64, IPv4-compatible), closing the loopback/private SSRF bypass - Report width/height from the dict of the URL actually returned, so a surviving thumbnail no longer reports the dropped original's dimensions --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Syed Osama Ali Shah
|
d9305989f3
|
fix: don't flag outline as truncated at exactly MAX_OUTLINE_ENTRIES headings (#3856)
extract_outline appended the {truncated: True} sentinel as soon as len(outline) >= MAX_OUTLINE_ENTRIES, i.e. right after the 50th heading, before knowing whether a 51st exists. A document with exactly 50 headings was therefore reported as truncated, and uploads_middleware injected a misleading 'showing first 50 headings' hint into the agent's context. Use a lookahead: only mark truncated once a heading beyond the limit is actually seen, then drop that extra entry. Adds an exact-boundary regression test.
|
||
|
Nan Gao
|
4fcb4bc366
|
feat(subagents): persist and display subagent step history (#3779) (#3845)
* feat(subagents): persist and display subagent step history (#3779) Capture both assistant turns and tool outputs during subagent execution, stream them in task_running events, and persist them as subagent.* run events so the subtask card's step timeline survives a reload. Backend: - step_events.py: pure layer (capture_step_message, build_subagent_step, subagent_run_event) shared by streaming and persistence - executor.py: capture ToolMessage outputs, not just AIMessage turns - worker.py: persist task_* custom events to RunEventStore (category "subagent" keeps them out of the thread feed; list_events backfills) Frontend: - core/tasks/steps.ts + api.ts: SubtaskStep model, messageToStep, eventsToSteps, mergeSteps, fetchSubtaskSteps - subtask card accumulates live steps and backfills on expand - carry run_id onto history content messages for the events endpoint * fix(subagents): show AI turns in subtask card + paginate step backfill (#3779) Two follow-ups to the subagent step-history feature: Problem 1 — reload backfill could silently truncate the step timeline because list_events capped at 500 events (seq-ASC) across the whole run. Add task_id filtering + an after_seq forward cursor to list_events (all three stores + abstract base + the /events route), and make fetchSubtaskSteps page through one task's subagent.step events until a short page. No schema migration: the DB filter rides the existing run-scoped index via event_metadata["task_id"]. Problem 2 — the card only rendered tool steps, so persisted AI turns were never shown. Replace toolStepsForDisplay with stepsForDisplay: interleave AI reasoning turns (with text) and tool steps by message_index, drop blank-text AI turns, and drop the trailing final-answer AI turn when completed (already shown as result). Card renders AI steps as muted clamped markdown with a sparkles icon. Tests: store task_id/after_seq filtering + pagination across memory/db/jsonl, the /events route forwarding, stepsForDisplay rules, and fetchSubtaskSteps pagination. Docs updated in both AGENTS.md. * make format * fix(subagents): capture full multi-tool step tail, batch step persistence, cap tool-call args (#3779) Address PR review findings on the subagent step-history feature: 1. executor.py streamed on stream_mode="values" and captured only messages[-1] per chunk, so a multi-tool-call turn (ToolNode appends one ToolMessage per call in a single super-step) lost all but the last tool output in both the live task_running stream and the persisted history. Replace with capture_new_step_messages, which walks the newly-appended tail (and still re-checks the trailing message on no-growth chunks so id-less in-place replacements survive). 2. worker.py persisted each step with the store's low-frequency put() (a per-thread advisory lock per call); a deep subagent (max_turns=150) emits hundreds of steps on the hot stream loop. Replace with _SubagentEventBuffer, which batches via put_batch (flush on terminal subagent.end, at FLUSH_THRESHOLD, and in the worker finally). 3. build_subagent_step capped only text; tool_calls[].args were copied verbatim, so a large write_file/bash payload produced an unbounded subagent.step row. Cap each call's serialized args at SUBAGENT_STEP_MAX_CHARS, flagged args_truncated. Tests updated/added for all three; AGENTS.md refreshed. * fix(subagents): merge backfill into latest subtask state; reuse message_content_to_text (#3779) Address the remaining two PR review findings: 4. subtask-card's fetchSubtaskSteps().then(updateSubtask) closed over a stale tasks snapshot: a late-resolving backfill wrote setTasks({...stale}), clobbering SSE steps/status and sibling subtasks that arrived during the fetch. useUpdateSubtask now reads/writes through a tasksRef mirroring the latest state (ref-to-latest), and the pure per-subtask transition is extracted to core/tasks/subtask-update.ts::computeNextSubtask (unit-tested). 5. step_events._content_to_text duplicated deerflow.utils.messages. message_content_to_text; call the shared helper instead (guarding None content with 'or ""' so a tool-call-only turn still renders as ""). Tests added for computeNextSubtask and the None-content case; AGENTS.md docs updated. |
||
|
xiawiie
|
2e15e3fe0d
|
fix: generate fallback title for interrupted first-turn runs (#3874)
* fix: generate title for interrupted first turn * test(title): cover partial-exchange + dict-form messages Harden the interrupted-run fallback path added in 19fc34fd: - TitleMiddleware._should_generate_title now accepts a lone first-turn user message when allow_partial_exchange=True, so the worker can still derive a title if cancellation lands before any AI chunk is checkpointed. - runtime/runs/worker._ensure_interrupted_title computes the next checkpoint step defensively (treat missing/non-int step as 0) and renames a shadowed ckpt_config local for readability. - Add four unit tests in tests/test_title_middleware_core_logic.py: partial-exchange allows user-only, partial-exchange still respects an existing title, dict-form messages are recognized, and the sync fallback path derives a title from dict-form messages — matching what channel_values stores in the checkpoint. Refs #3859. * fix: persist interrupted-title via channel_versions bump Address PR #3874 review feedback: ``_ensure_interrupted_title`` previously called ``aput(..., new_versions={})``. LangGraph's DB-backed savers (``PostgresSaver`` and the v4 ``SqliteSaver`` blob layout) strip inline ``channel_values`` from ``put`` and only persist blobs for channels named in ``new_versions`` — so the fallback ``title`` channel was dropped on read-back and ``threads_meta.display_name`` stayed ``"Untitled"`` after refresh on those backends. The original in-memory e2e passed because ``InMemorySaver`` keeps the inline snapshot verbatim. Fix mirrors ``_rollback_to_pre_run_checkpoint`` in the same file: bump ``channel_versions["title"]`` (via ``checkpointer.get_next_version`` when available, else int/string fallbacks), persist the new version on the checkpoint, and declare it in ``new_versions`` so the DB savers actually write the blob. Regression coverage in ``tests/test_run_worker_rollback.py``: - ``test_ensure_interrupted_title_bumps_channel_version_and_declares_it_in_new_versions`` — exact ``aput`` invariants: ``new_versions == {"title": 1}``, the written checkpoint's ``channel_versions["title"]`` is bumped, and the pre-existing ``messages`` version is preserved. - ``test_ensure_interrupted_title_bumps_existing_string_version`` — string-shaped prior version (some savers use UUID-style versions); bumped value must differ from the prior, no overwrite-in-place. - ``test_ensure_interrupted_title_skips_when_title_already_set`` — title short-circuit; no extra ``aput``. - ``test_ensure_interrupted_title_returns_none_when_no_checkpoint`` — no checkpoint yet; returns ``None`` without writing. - ``test_ensure_interrupted_title_round_trip_with_real_sqlite_checkpointer`` — full round-trip against a real ``AsyncSqliteSaver`` on a disk-backed DB, then closes and re-opens the saver to simulate a fresh connection. The fallback title must still be present on the second ``aget_tuple``. This is the exact scenario the review flagged. Validated locally with the full backend suite: 5195 passed, 18 skipped. Refs #3859. Addresses review on #3874. * test(worker, title): harden interrupted-title fallback for every saver Defensive coverage on top of the channel_versions fix (commit 05253957), addressing edge cases surfaced during a second-pass review of #3874. Worker: - Extract version bump into ``_bump_channel_version(checkpointer, current)`` with explicit fallbacks for int / float / numeric-string / UUID-shaped string / None / bool, AND a wrap-around defense when the saver's ``get_next_version`` raises or returns an unchanged value. The invariant is: returned version MUST differ from the prior. Without this, a saver bug (or a custom backend) could leave ``new_versions={"title": v}`` no-op on DB savers — the very class of bug the original review pointed out. Title middleware: - Coerce ``state.get("messages")`` from ``None`` to ``[]`` on both ``_should_generate_title`` and ``_build_title_prompt``. A partially-initialized checkpoint can carry ``messages=None`` on the channel_values channel (the worker reads raw channel_values, not BaseMessages), and the default kwarg only protects against a missing key. Repro: ``TypeError: 'NoneType' object is not iterable`` from the next() generator — confirmed by reverting the fix and watching ``test_*_handles_none_messages_channel`` go red. Tests (TDD-verified red→green for the new asserts): - ``test_run_worker_rollback.py``: * ``_bump_channel_version`` — 8 tests covering every version type (int, float, numeric string, UUID-style string, None, bool) and every saver-side fault mode (no ``get_next_version`` / raising / stuck on identity). * ``test_ensure_interrupted_title_*`` — 5 additional helper boundary tests: title.enabled=false short-circuit; empty messages list; messages=None; aput-error propagation (helper contract: caller swallows, not the helper); idempotency on a real InMemorySaver across two invocations. * ``test_ensure_interrupted_title_preserves_non_title_channel_versions`` — pins that ``new_versions`` only contains ``"title"`` and that other channels' versions are untouched (regression anchor for a sloppier draft that bumped every channel). * ``test_worker_finally_block_swallows_helper_exceptions`` — pins the integration contract: even if the helper raises, the worker's threads_meta status sync still runs and ``publish_end`` is still awaited so the SSE stream closes cleanly. - ``test_title_middleware_core_logic.py``: * 4 additional tests: ``messages=None`` on both ``_should_generate_title`` and ``_build_title_prompt``; the ``role: user`` / ``role: assistant`` (OpenAI-style) dict normalization; partial-exchange path with a dict-form message. Verification: - ``PYTHONPATH=. uv run pytest tests/ -x --ignore=tests/blocking_io -q`` → 5215 passed, 18 skipped. - ``ruff check`` + ``ruff format --check`` clean on every touched file. - Red/green TDD verification: temporarily reverted the ``new_versions={}`` fix → 4 new tests went red as expected; restored and the suite is green again. Same red/green dance for the ``messages=None`` coercion. Refs #3859. Addresses second-pass review on #3874. * fix(title): ignore dict context reminders in fallback * fix(worker): link interrupted-title checkpoint to its parent The title-bump checkpoint written by ``_ensure_interrupted_title`` was landing without a ``parent_checkpoint_id`` — a real orphan in the LangGraph history graph. Reproduction (disk-backed AsyncSqliteSaver): [seed] checkpoint_id = 1f173dbc... [helper] wrote title = "Why is the sky blue?" [issue 1] new checkpoint = 1f173dbc..., parent = None [issue 1] is new checkpoint orphaned? True Root cause: ``_ensure_interrupted_title`` built ``write_config`` as ``{"thread_id": ..., "checkpoint_ns": ...}`` only. ``BaseCheckpointSaver`` implementations read ``configurable.checkpoint_id`` from that config as the *parent* id when inserting (see ``langgraph/checkpoint/sqlite/aio.py`` ``aput``: ``config["configurable"].get("checkpoint_id")`` becomes the ``parent_checkpoint_id`` column). With no value, the saver writes NULL — the new checkpoint is a tree root. Consequences: - Any future LangGraph ``runs.resume_from`` / time-travel feature has no backward edge to walk past the title-bump. - History-visualization UIs built on ``alist()`` render the title-bump as a sibling of the prior checkpoint, not its descendant. Fix: read ``checkpoint_id`` off the tuple's own config and thread it into ``write_config["configurable"]["checkpoint_id"]`` before calling ``aput``, the same pattern every middleware-driven write uses. Three new regression tests against real ``AsyncSqliteSaver`` (disk-backed, fresh connections so we exercise the on-disk read path): - ``test_ensure_interrupted_title_links_new_checkpoint_to_its_parent`` — asserts ``latest.parent_config["configurable"]["checkpoint_id"]`` equals the seeded checkpoint id. TDD red-green verified: reverting the fix flips this test red with ``AssertionError: title-bump checkpoint must have a parent_config``. - ``test_ensure_interrupted_title_appears_in_history_with_audit_marker`` — pins the audit contract: the title-bump entry in ``alist()`` carries ``metadata.source == "update"`` and ``metadata.writes`` contains ``runtime_interrupt_title``. This is a deliberate design choice — we do NOT hide the entry from history (audit trail belongs in the saver), but its source and writes marker MUST be unambiguous so UIs/tools can identify it. - ``test_ensure_interrupted_title_survives_immediate_next_turn`` — cancel → immediate user follow-up scenario. Simulates the agent's next turn appending a (user, ai) pair without touching the title channel, then opens a fresh saver and verifies the title is still present after the next-turn checkpoint write. Pins the channel-version-blob invariant established by commit 05253957 — without the ``new_versions={"title": v}`` declaration there, the title blob would vanish from the DB and this test would read back ``None``. Verification: - ``PYTHONPATH=. uv run pytest tests/ -x --ignore=tests/blocking_io -q`` → 5222 passed, 15 skipped. - ``ruff check`` + ``ruff format --check`` clean on every touched file. - Reproduction script confirms ``parent_checkpoint_id`` is now non-null and the next-turn read-back preserves the fallback title. Refs #3859. * Revert "fix(worker): link interrupted-title checkpoint to its parent" This reverts commit c763ed9334781db1acdce0f5f33d663d8d5f80ff. * test: trim over-engineered test coverage Reduce review surface area on PR #3874 by dropping defensive tests that don't pin a real invariant. After self-review: - ``_bump_channel_version``: 8 tests → 2 (happy path + saver-error fallback). Dropped float / bool / numeric-string / UUID-string / missing-get-next-version / stuck-get-next-version branches — those are speculative scaffolding for savers we don't ship. - ``_ensure_interrupted_title``: dropped ``returns_none_when_title_disabled``, ``returns_none_with_no_user_message``, ``returns_none_when_no_checkpoint`` — boundary guards already exercised by the e2e test and the ``handles_none_messages_channel`` regression anchor. Net: -107 lines of test code. Remaining coverage still pins every red-green-verified invariant (channel_versions bump, string-version bump, idempotency, sqlite round-trip, non-title channel preservation, aput-error contract, worker finally swallowing, partial-exchange). Verification: 5209 passed, 15 skipped. * fix: harden interrupted title finalization * fix: serialize interrupted title finalization * fix: preserve interrupt semantics during title finalization * fix: preserve delayed interrupted title recovery --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Ryker_Feng
|
a8f950feb6
|
feat(community): add Browserless web_capture screenshot tool (#3881)
* feat(community): add Browserless web_capture screenshot tool Add a web_capture tool that renders a page via Browserless /screenshot and presents it through the artifact system, alongside the existing Browserless web_fetch provider. Hardening: - SSRF guard: reject URLs resolving to private/loopback/link-local (incl. the 169.254.169.254 cloud-metadata endpoint)/reserved/multicast/unspecified addresses; opt out via allow_private_addresses for internal targets. - Surface a warning when Browserless renders a target page that itself responded with a non-2xx/3xx status (X-Response-Code), so an error/anti-bot page is not mistaken for valid visual evidence. - Dedupe colliding output filenames instead of silently overwriting prior captures. Docs: comment out token: $BROWSERLESS_TOKEN in tool examples (an unset $VAR fails AppConfig startup) and document allow_private_addresses. * fix(community): format web_capture guard + document local Browserless startup Address PR #3881 review: fix the lint-backend failure (ruff format on browserless/tools.py) and add local Browserless startup instructions to CONFIGURATION.md so reviewers can run the service to try web_fetch/web_capture. |
||
|
AochenShen99
|
442248dd06
|
feat: preserve durable context across summarization (#3887)
* feat: preserve durable context across summarization * fix: harden durable context review gaps * style: format delegation ledger live test * chore: remove stale delegation ledger prefix * fix: address durable context review feedback |
||
|
Minh Vu
|
8fa6ed2b54
|
fix(gateway): fix oversized upload replacements deleting existing files (#3822)
* fix upload rollback on size limit * fix upload staging cleanup and listings |
||
|
chengjoey
|
e5424cbab9
|
feat(sandbox): add E2B sandbox provider (#3883)
Adds ``deerflow.community.e2b_sandbox.E2BSandboxProvider`` with parity to AioSandboxProvider: metadata-keyed per-thread persistence, server- side idle timeout, warm-pool reclaim with liveness checks, /mnt/user-data bootstrap symlinks, dead-sandbox auto-rebuild, and release-time mirror of agent outputs back to the host artifact directory. Signed-off-by: joey <zchengjoey@gmail.com> |
||
|
Nan Gao
|
af0d14d296
|
feat(subagents): system-maintained delegation ledger to stop redundant re-delegation (#3877)
* feat(subagents): add delegations ledger field + reducer to ThreadState * feat(subagents): pure helpers to derive + format the delegation ledger * feat(subagents): DelegationLedgerMiddleware records + injects the ledger * feat(subagents): register DelegationLedgerMiddleware for lead when subagents enabled + docs * add runtime log * chore(subagents): make delegation-ledger injection log production-ready * test(subagents): make delegation-ledger registration tests config-free; refresh ultra replay golden for delegations channel * refactor(subagents): derive TERMINAL_STATUSES from SUBAGENT_STATUS_VALUES + pin it Make thread_state's TERMINAL_STATUSES a frozenset over the status contract's SUBAGENT_STATUS_VALUES instead of a hardcoded literal, so the terminal-status set can never drift from the contract. Add a pinning test asserting the derivation and that the non-terminal "in_progress" stays excluded. Addresses PR #3877 review. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
xiawiie
|
2453718acd
|
fix(title): avoid default LLM call before stream end (#3885)
* fix(title): avoid default LLM call before stream end * fix(title): keep default fallback local * fix(title): harden fallback and replay e2e * docs(title): align fallback title behavior |
||
|
DanielWalnut
|
fe82552023
|
fix(sandbox): stop blocking bash commands (e.g. servers) from hanging the turn (#3864)
* fix(sandbox): stop blocking bash commands from hanging the turn Starting a server through the host bash tool (e.g. `python -m http.server`) could hang the whole turn for the full 600s timeout. `LocalSandbox.execute_command` used `subprocess.run(capture_output=True)`, whose captured pipes are inherited by any process the command spawns — so a backgrounded long-lived process (`server &`) keeps the read end open and blocks `communicate()` until the timeout fires, even though the foreground command already returned. Commands that read stdin blocked the same way, and on timeout only the direct child was killed, leaving orphaned process groups. Rework the POSIX path to capture stdout/stderr via temp files instead of pipes, take stdin from /dev/null, and run the command in its own session/process group: - Backgrounded long-lived processes (servers) now return immediately while the process keeps running. - A command reading stdin gets immediate EOF instead of blocking. - A genuinely blocking foreground command is bounded by a configurable wall-clock timeout; on timeout the whole process group is killed and the agent gets an explanatory notice telling it to background long-lived processes. The timeout is configurable via `sandbox.bash_command_timeout` (default 600). The Windows path is unchanged. Adds focused regression tests and updates docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sandbox): instruct the agent to background long-lived processes The code fix bounds a foreground server with a timeout, but the turn still waits the full timeout before the run continues. Add the prompt-side half: the bash tool description now tells the model to ALWAYS start long-lived processes (e.g. web servers) in the background with output redirected, so the tool returns immediately. The timeout notice points at the same readable workspace log path. Pins the guidance with a test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sandbox): make fallback-kill exception explicit and observable Address automated review: the inner `except OSError: pass` in _terminate_process_group silently swallowed the case where the direct-child fallback kill found the process already gone. Make the intent explicit with a comment and a debug log instead of a bare pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sandbox): address bash timeout review feedback * fix(sandbox): document fd cleanup races --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
heart-scalpel
|
6d7b94c5f8
|
refactor(middlewares): declarative layered builder; pin chain order with tests (#3809)
Split `_build_runtime_middlewares`'s flat list into three named declarative sublists (outer_wrappers / thread_hooks / tail) and drop the `middlewares.insert(2, UploadsMiddleware())` magic-index pattern. The declarative structure makes the layering self-documenting and immune to position drift when the head of the list changes. Move UploadsMiddleware to run after ThreadDataMiddleware in the chain. Under the previous order (a magic-index artifact introduced when #3662 prepended InputSanitizationMiddleware), UploadsMiddleware scanned the uploads directory before ThreadDataMiddleware created it under lazy_init=False, so historical files could be missed on the first run of a thread. Narrow path — the upload endpoint normally pre-creates the directory — but the order is the correct semantic and is now locked. Documentation: - backend/AGENTS.md middleware chain renumbered: ThreadDataMiddleware is now #3, UploadsMiddleware #4 (was reversed). Tests (backend/tests/test_tool_error_handling_middleware.py): - test_build_lead_runtime_middlewares_orders_thread_data_before_uploads — focused td_idx < um_idx assertion. - test_build_lead_runtime_middlewares_chain_order_matches_agents_md — full-chain order pin using real classes, so a swap between any pair is caught (the existing FakeMiddleware-stubbed tests cannot detect this). - test_lead_runtime_chain_finds_historical_uploads_under_lazy_init_false — integration anchor: under lazy_init=False, ThreadDataMiddleware creates the dirs, then UploadsMiddleware surfaces a pre-existing historical file in the injected <uploaded_files> context. |
||
|
Huixin615
|
e5a610e845
|
fix: ignore middleware prompts in run journal input (#3780) | ||
|
Yufeng He
|
2e789eae18
|
fix(skills): synchronize skill storage singleton lifecycle (#3778)
get_or_new_skill_storage() and reset_skill_storage() touch the process-global skill storage singleton without a lock - the same unsynchronized check-then-create that #3730 just fixed in sandbox_provider.py, the module this file documents itself as mirroring. Two callers racing a cold start can both see _default_skill_storage is None and each build a SkillStorage, so the second overwrites the first; a reset_skill_storage() racing a get can also null the global between the None-check and the return. Guard the build/return and the reset with a module-level threading.Lock and a double check, mirroring get_memory_storage(). Construction stays inside the lock (rather than sandbox_provider's build-outside-then- discard-the-loser) because SkillStorage has no teardown hook, so a losing racer's orphan could not be cleaned up. Add backend/tests/test_skill_storage_lifecycle.py with concurrency regression tests (8-thread cold-start race asserting a single instance; reset racing gets asserting no None is returned). Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com> |
||
|
Huixin615
|
73874a9b35
|
fix(mcp): isolate tool discovery failures (#3772) | ||
|
Huixin615
|
b0ce4b513c
|
fix: allow valid heredoc commands in sandbox audit (#3786)
* fix: allow valid heredoc sandbox audit commands * test: cover unparseable heredoc sandbox audit path |
||
|
DanielWalnut
|
ef5f54c5bf
|
feat(tui): Hermes-like terminal workbench (deerflow) backed by DeerFlowClient (#3760)
* feat(tui): add Hermes-like terminal workbench backed by DeerFlowClient Implements the `deerflow` TUI from RFC #3540: a terminal-native, embedded workbench over the existing harness (no Gateway/frontend/nginx/Docker), built Python-native with Textual and learning UX patterns from tao-pi. Architecture — every layer except the Textual app is pure and unit-tested: - view_state.py: ViewState + reduce(state, action), the testable heart - runtime.py: StreamEvent -> reducer actions (pure translate + threaded driver) - message_format / command_registry / input_history / render / theme: pure - app.py: Textual App; runs the sync DeerFlowClient.stream() on a worker thread and marshals actions back to the UI thread. Slash command palette, model and thread modal pickers, ↑/↓ history, Ctrl+C interrupt, TTY-aware fallback. - cli.py: pure launch-mode planning + headless --print/--json + `deerflow` console script (textual is an optional [tui] extra; degrades to headless help) Web UI visibility (the RFC's key decision): persistence.py writes a threads_meta row under the local default user into the same DB the Gateway reads, so terminal sessions appear in the Web UI sidebar without running the Gateway. Best-effort, no-op on the memory backend; all DB work on one long-lived background loop. Tests: 95 TUI tests — pure layers via pytest, app/palette/overlays via Textual's pilot harness with a fake session, and a threads_meta read/write round-trip. ruff clean; respects the harness->app import boundary. Docs: backend/docs/TUI.md plus CLAUDE.md/README updates and preview screenshots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tui): de-duplicate streamed assistant text and tool cards; keep Tab in composer Self-test surfaced three issues, all root-caused to consuming non-strict streaming from DeerFlowClient (proven by the client's own test_dedup_requires_messages_before_values_invariant, which shows the client can re-emit a message id's full content twice): - Assistant text was doubled (e.g. "answer answer") because the reducer blindly concatenated same-id deltas. Now merges by content: a re-send or cumulative snapshot replaces; only genuine increments append. - Tool activity showed duplicate and empty "gear" cards from partial/re-emitted tool-call chunks. ToolStarted now de-dupes by tool_call_id, drops id-less noise chunks, and fills the name on a later chunk; a tool result with no prior card still surfaces as a completed card. - Tab moved focus off the composer to the scroll region (felt like broken cursor logic). Tab is now consumed by the composer (completes a command when the palette is open, no-op otherwise). Adds reducer tests for each case plus a Tab-focus test; 102 TUI tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tui): make Esc interrupt an active run (matches the status hint) The status line advertised "esc interrupt" but Esc was only wired to close the slash palette, so it did nothing during a run. Esc now: closes the palette when open, interrupts the active run when streaming, and is a no-op when idle. The interrupt logic is shared with Ctrl+C via _interrupt_run(). Adds a regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tui): stop prior answers duplicating on threads with history On a thread with history, DeerFlowClient re-emits every prior message on each new turn (its streamed_ids dedup is per-stream-call), and a re-emitted older message can arrive after a newer message has already started. The reducer only matched the *most recent* assistant row by id and otherwise appended, so each re-emitted older answer was duplicated verbatim at the end of the transcript. Match an assistant row by id anywhere in the transcript and merge in place. Tool cards already de-dupe by call id globally, so they were unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tui): correct CJK cursor drift in the composer Confirmed a Textual Input bug (latest 8.2.7): Input._cursor_offset adds an unconditional +1 at the end of the value, overshooting by one cell after double-width (CJK) characters. That misplaces the hardware/IME cursor — the drift seen when typing Chinese in iTerm2 (the on-screen block cursor, drawn separately in render_line, is fine; English doesn't use an IME so it looks correct). Reproduced with a bare Input, so it's upstream, not our layout. Add ComposerInput(Input) overriding _cursor_offset to the true cell position and use it for the composer. Numeric tests pin the CJK end/mid and ASCII cases. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tui): render finalized assistant messages as Markdown The transcript showed raw Markdown (literal **bold**, ## headings, - lists, links). Finalized assistant messages now render as Rich Markdown — headings, bold/italic, lists, inline code + code blocks, blockquotes, horizontal rules and links — with the ● speaker marker aligned to the top of the body. The actively-streaming message stays plain text so partial Markdown doesn't reflow/jump, then snaps to its rendered form when the run ends. Transcript re-renders are coalesced on a ~60ms timer (dirty flag) so per-token Markdown re-parsing stays smooth on long threads. Tests cover both the rendered and the streaming-plain paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style(tui): apply ruff format CI lint runs `ruff format --check` via uvx (latest ruff); apply the formatter so the lint-backend job passes. No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(tui): address code-quality review comments From github-code-quality[bot] on #3760: - runtime.py: give the `_ClientLike` Protocol method a docstring body instead of a bare `...` (flagged as a no-effect statement), matching the harness convention for Protocol stubs (e.g. SafetyTerminationDetector). - test_tui_cli_main.py: drop the unnecessary `lambda: _FakeSession()` wrappers in monkeypatch.setattr; pass `_FakeSession` directly (same behavior). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tui): keep history Markdown-rendered when a follow-up run starts Previously the transcript rendered "the last assistant row" as plain text while streaming. But when a follow-up turn starts, the last assistant row is the *previous, finalized* answer until the new message begins — and the client re-emits prior messages early in the turn — so sending a follow-up reverted the previous answer from rendered Markdown back to raw text. Track the actively-streaming message id in ViewState instead: it's reset on RunStarted, set only when an AssistantDelta actually adds new content (history re-emits are no-ops and don't mark it), and cleared on RunEnded. The renderer keeps only that one message plain; all history stays Markdown. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): add Terminal Workbench (TUI) section to root README Mention the new `deerflow` TUI alongside the Embedded Python Client in the root README.md and README_zh.md (install, launch/headless commands, feature summary, Web UI visibility), with a ToC entry and a preview screenshot. Links to backend/docs/TUI.md for the full guide. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tui): address review feedback (willem-bd) Ten findings from the TUI code review: 1. /resume was dead-ended — registered + in /help + tested as a builtin, but no dispatch branch. Wired it to thread resolution / the switcher. 2. --resume <title> was forwarded raw into the checkpointer (blank thread). Added Session.resolve_ref() to resolve id-or-title via list_threads; used by --resume and /resume. 3. str(get("id","")) returned "None" for an explicit id:None (truthy), defeating the empty-id guard so unrelated null-id tool calls collapsed into one card. Coerce via a None-safe helper. 4. Headless --print/--json no longer spin up the persistence loop/engine/pool (open_session(persistence=False)). 5. _LoopThread + engine are now closed: Session.close() (dispose engine + stop loop) called from a try/finally around app.run(). 6. --cli --continue (and piped --cli) now run headless instead of erroring. 7. Cancelled runs no longer persist a truncated title (guard on _cancelled). 8. Palette highlight resets to the top when the filter set changes. 9. Dropped the never-populated tools count from the header. 10. Documented the `not row.error` merge guard. Adds regression tests for each; 126 TUI tests pass, ruff check + format clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Czile
|
30841c3b9f
|
fix(sandbox): synchronize sandbox provider singleton lifecycle (+ concurrency regression tests) (#3730)
* fix(sandbox): synchronize sandbox provider singleton lifecycle get_sandbox_provider() used an unsynchronized check-then-create, so two OS threads (e.g. the main event loop and the Feishu channel thread, which runs its own loop) could double-initialize the provider. With AioSandboxProvider the overwritten instance leaks its idle-checker thread, since the only code that joins it (shutdown()) is reachable only through the reference that was overwritten. reset_sandbox_provider(), shutdown_sandbox_provider() and set_sandbox_provider() also touched the global without a lock, so a reset/shutdown racing an in-flight create could clear it mid-construction or tear down an instance another thread was about to return. Guard all four lifecycle sites with a single module-level threading.Lock and use double-checked locking in the getter, mirroring get_memory_storage(). * test(sandbox): add concurrent regression tests for provider singleton - 8 threads racing on cold start, synchronized with a threading.Barrier so the check-then-create race fires deterministically; asserts exactly one provider instance is created. - reset racing concurrent gets: asserts every returned value is a fully constructed provider (never None / half-built). * fix(sandbox): lock get read path, run provider callbacks outside the lock Addresses the three review findings on #3730: 1. get_sandbox_provider()'s read+return ran outside _provider_lock, so a concurrent reset/shutdown/set could null or tear the global between the check and the return, handing callers None / a torn instance. The hot read and the install reconciliation now both happen under the lock. 2. The non-reentrant _provider_lock was held across plugin-supplied callbacks (resolve_class import + provider __init__ in get; provider.reset()/shutdown() in reset/shutdown). A custom provider that re-entered these lifecycle functions would self-deadlock, and a slow teardown blocked every concurrent get(). resolve_class + construction now run outside the lock; reset/shutdown detach the reference under the lock and invoke the callback outside it. Tradeoff: racing cold-start callers may each construct a candidate. Exactly one is installed and returned to everyone; the losers (e.g. an AioSandbox instance that already started an idle-checker thread) are shut down so they do not leak the orphan thread #3721 is about. set_sandbox_provider() documents that it replaces but does not shut down the prior instance. 3. test_reset_racing_get reset the singleton to None *before* the barrier, so the racing reset was a no-op and never exercised reset-of-a-live-provider. It now populates the singleton up front so the reset tears down a live instance while getters read it. Tests: rename the cold-start test to assert "one installed singleton, observed by all" (construction is no longer single under the new design); add shutdown-vs-get, set-vs-get, and a losing-racer-shuts-down-its-orphan case. All five pass; the existing sandbox/middleware/mounts/uploads suites are green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Zhipeng Zheng
|
f656440245
|
fix(middleware): prevent ID-swap recursive injection and orphan peer compression (#3746)
- Add endswith('__user') guard in _is_user_injection_target to prevent
unbounded suffix growth (id__user__user__user...) when a prior ID-swap
peer is mistakenly treated as a new injection target
- Add peer rescue in _preserve_dynamic_context_reminders to keep
ID-swap __user and __memory messages out of summary compression,
preventing orphaned SystemMessage reminders and lost user context
- Add 6 regression tests covering both failure modes
|
||
|
ly-wang19
|
820560e576
|
perf(runtime): index MemoryRunEventStore events by run_id to avoid O(n) scans (#3686)
* perf(runtime): index MemoryRunEventStore events by run_id to avoid O(n) scans list_messages was already served from a thread-wide messages projection (#3531), but list_events and list_messages_by_run still scanned the whole thread's event log (every run, every category) to return one run's events -- O(N_thread) on every run-scoped /messages page-load and /events request. Add _events_by_run / _messages_by_run projections (same dict objects, kept in lockstep in _put_one / delete_by_run / delete_by_thread), so both reads are O(M_run), with bisect cursor pagination for messages. Semantics are unchanged, pinned by a brute-force parity test over interleaved traces and both cursors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: ruff format test_run_event_store_by_run_index.py Clears the lint-backend (ruff format --check) failure on the PR; the original commit ran `ruff check` but not `ruff format`. No behavior change (test fixture formatting only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
AnoobFeng
|
debb0fd161
|
feat(persistence): wire alembic migrations, bootstrap schema on startup (#3706)
* feat(persistence): wire alembic migrations + bootstrap schema on startup Closes #3682. Pre-#3658 DBs lack the `runs.token_usage_by_model` column because alembic was never wired up — startup only ran `create_all`, which never ALTERs existing tables. Adds a hybrid bootstrap in FastAPI lifespan (replaces bare `create_all`): - empty DB → create_all + stamp head - legacy DB → stamp 0001_baseline + upgrade head - versioned DB → upgrade head Concurrency: Postgres `pg_advisory_lock` (cross-process); SQLite per-engine `asyncio.Lock` + 30s `PRAGMA busy_timeout` on both prod and alembic engines. Column revisions use `safe_add_column` / `safe_drop_column` idempotent helpers as fallback. Other bits: - 0001 baseline (chain root) + 0002 add `runs.token_usage_by_model` - `include_object` filter so alembic ignores LangGraph checkpointer tables - `make migrate-rev MSG="..."` for authoring new revisions (no migrate/stamp targets — startup is the only execution path) - Tests: three-branch decision, concurrency, #3682 regression, env filter, blocking-IO gate anchor - CLAUDE.md: new "Schema migrations" section * fix(style): fix lint error * perf(persistence): address review feedback on alembic bootstrap Behavioural fixes - _SQLITE_LOCKS now keyed via WeakKeyDictionary so id-reuse after GC cannot return a stale, loop-bound lock and the cache cannot leak one entry per disposed engine. - safe_add_column compares nullable / server_default against the desired column when the name already exists and emits a warning on drift, surfacing manual-ALTER workarounds instead of silently no-op'ing. - _postgres_lock issues SET LOCAL idle_in_transaction_session_timeout=0 before pg_advisory_lock, so managed Postgres cannot kill the idle lock-holding session mid-upgrade and silently release the advisory lock. - legacy branch now backfills missing baseline tables via a restricted create_all (Base.metadata.create_all scoped to _BASELINE_TABLE_NAMES). Restores pre-#1930 upgraders whose channel_* tables were never provisioned, without pre-empting future create_table revisions for newly-added models. Schema parity - runs.token_usage_by_model gains server_default=text("'{}'") in both the ORM model and the 0001_baseline create_table, matching what 0002 adds via ALTER. create_all and alembic-upgrade paths now produce identical column definitions. - New parity test compares Base.metadata.create_all output against a pure alembic upgrade base->head, asserting column-set, nullable, and server_default agree across all tables (normalized through the same helper safe_add_column's drift check uses). Guards - test_baseline_table_names_constant_matches_0001 pins _BASELINE_TABLE_NAMES to 0001_baseline.upgrade()'s actual output -- the constant cannot drift silently when someone edits 0001. - test_legacy_backfill_skips_non_baseline_tables verifies the restricted backfill does not create a phantom table on Base.metadata, modelling a future revision that would otherwise collide on op.create_table. Doc residuals - Three-branch decision table is now consistent across bootstrap.py top docstring, engine.py comment, test module docstring, and CLAUDE.md. - Stale test anchor in blocking_io/test_persistence_engine_sqlite.py docstring now points at the real file. * fix(style): fix lint error * fix(persistence): close drift detection holes - _check_column_drift compares column type via a family equivalence allowlist ({JSON, JSONB}). Catches the wrong-type workaround `TEXT NOT NULL DEFAULT '{}'` that previously slipped through silently, while keeping Postgres JSON/JSONB dialect reflection quiet. Reflected and desired type are also echoed in every drift warning's payload for operator triage. - Extract _escape_url_for_alembic so bootstrap._alembic_safe_url and scripts/_autogen_revision share the ConfigParser % escape rule instead of duplicating it. - backend/README.md: add `make migrate-rev MSG=...` to Commands and a Schema Migrations section per the repo's README/CLAUDE.md sync policy. - test_base_to_dict.py: scope the test ORM class to an isolated MetaData so the create_all-vs-alembic parity test (added in the previous commit) is not polluted by the phantom table on the full pytest session. |
||
|
ly-wang19
|
346164d740
|
perf(runtime): index MemoryRunStore by thread_id to avoid O(n) scans (#3562)
* perf(runtime): index MemoryRunStore by thread_id to avoid O(n) scans MemoryRunStore is the default run backend (database.backend=memory) and backs RunManager.list_by_thread, which calls it on every thread-runs query to hydrate persisted runs. list_by_thread scanned every run in the store (O(total runs)) to filter by thread_id, so listing one thread's runs got linearly slower as unrelated runs accumulated across all threads. Add a thread_id -> insertion-ordered run_id set secondary index, maintained in lockstep with _runs in put()/delete(), and use it in list_by_thread for an O(runs-in-thread) lookup. This mirrors the index RunManager already keeps over its own in-memory records (#3499); the store extraction — whose docstring notes it is "Equivalent to the original RunManager._runs dict behavior" — did not carry the index across. Behavior is unchanged: same user_id filtering, newest-first ordering, and limit. The store runs each method without awaits on the event loop, so the index and _runs stay consistent without a lock. Extends tests/test_persistence_scaffold.py::TestMemoryRunStore with coverage for unknown-thread, newest-first ordering, limit, and index cleanup on delete (including empty-bucket removal). * perf(runtime): route aggregate_tokens_by_thread through the thread index too list_by_thread already uses the _runs_by_thread index this PR adds, but aggregate_tokens_by_thread (the /token-usage endpoint) still scanned every run in the process to pick out one thread's runs. Route it through the same index for an O(runs-in-thread) lookup, completing the thread-scoped read coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |