* fix(skills): offload update_skill's blocking IO and share the config write lock
Re-applied on top of main rather than merged: update_skill has since gained
admin gating, user-scoped storage and a PUBLIC vs CUSTOM/LEGACY split, so the
offload is applied per path.
- PUBLIC: the extensions_config.json read-modify-write (path resolve, snapshot,
merge, write, reload) moves to a worker thread via asyncio.to_thread. The
payload is built from a snapshot so the cached singleton is never mutated in
place while the write is still in flight.
- CUSTOM/LEGACY: set_skill_enabled_state is offloaded; the non-user-scoped
fallback takes the same shared-file RMW path as PUBLIC.
- Both load_skills calls (and storage construction) are offloaded.
The RMW lock now lives next to reload_extensions_config as
get_extensions_config_write_lock() and is acquired by both the skills router and
the MCP router, which performs the same read-modify-write on the same file.
Previously each router held its own module-local lock, so once both sides
offloaded, a PUT /api/mcp/config could run inside a skill toggle's read->write
window and the later write would silently drop the other's change.
The lock is keyed by the running event loop rather than being a module-level
singleton: asyncio primitives bind to the first loop that awaits them, which
makes a plain module-level lock unusable in a process that runs more than one
loop.
Adds a cross-router regression anchor asserting a skill toggle and an MCP update
never overlap inside the RMW (max in-flight 1); it observes 2 when the routers
use separate locks.
* fix(config): own the extensions_config RMW lock from the worker thread
An asyncio.Lock held around `await asyncio.to_thread(...)` protects only the
awaiting task. If that task is cancelled the context manager releases the lock
immediately while Python keeps running the worker thread, so a second skills or
MCP writer could acquire it and operate on extensions_config.json concurrently
with the first worker — reopening the lost-update window this was meant to close.
The per-event-loop keying had a second hole: writers on different loops got
different locks and so did not exclude each other at all.
Replace it with a process-wide threading.Lock acquired *inside* the worker that
performs the RMW (`_write_extensions_skill_state` and `_apply_mcp_config_update`),
so ownership belongs to the thread doing the writing and is held until the write
and reload actually finish, regardless of what happens to the caller. A
threading.Lock also has no event-loop affinity.
Adds a cancellation regression: the skills worker is paused inside the lock, its
route task is cancelled, and the MCP writer is started — the MCP RMW must not
enter until the skills worker is released. Against the previous asyncio-lock
design this test fails with ['skills-enter', 'mcp-enter'].
The two existing serialization tests instrumented by replacing the worker
functions, which now bypasses the lock under test; they instead patch inside the
real workers (each module's reload_extensions_config, the last step under the
lock) so the production lock is exercised.
---------
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: add lark cli integration
* fix: polish lark integration actions
* feat: support lark incremental permissions
* fix: detect lark authorization completion
* fix: harden lark integration install
* feat: expand lark auth scopes and reuse host auth in sandbox
Default lark auth to least-privilege (recommend=false, base sign-in only)
and expose the full set of lark-cli --domain business domains as native
--domain grants instead of a 4-domain read-only mapping. Resolve the
skill pack from the latest larksuite/cli GitHub release at install time
with content-hash integrity, and surface version/runtime drift in status.
Share the per-user lark-cli config/data profile between the Gateway
Settings auth flow and agent conversations by mounting the integration
dirs into the AIO sandbox and injecting the matching env for lark-cli
commands, with an allowlisted extra_mounts path in the provisioner/K8s
backend and traversal guards on integration paths.
* style: fix lint issues from ruff and prettier
Sort imports in the provisioner PVC test and re-wrap two long i18n
description strings to satisfy backend ruff and frontend prettier CI.
* fix(lark): address managed integration review feedback
* fix(frontend): stabilize integrations settings e2e
* test(sandbox): isolate remote backend legacy visibility check
* test: fix backend unit failures after merge
* Harden Lark integration review fixes
* Format Lark integration E2E test
* fix(lark): harden sandbox credential exposure and status disclosure
Address willem_bd's security review on PR #3971:
- Mount the per-user lark-cli config dir (long-lived appSecret) read-only
into the AIO sandbox; only the refreshable-token data dir stays writable.
- Redact host filesystem paths (install_path, cli.path) from
GET /lark/status and the config/auth complete responses for non-admin
callers, fail-closed on any auth error.
- Document the npm postinstall trade-off (--ignore-scripts is not viable
because @larksuite/cli fetches its platform binary in postinstall).
- Document the sandbox credential trust boundary in AGENTS.md and README,
pointing at the sidecar-broker follow-up (#4338).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(config): close out #4124 review follow-ups
Extracts the (mtime, size, sha256) content-signature helper that was
duplicated between config/app_config.py and mcp/cache.py into a new
config/file_signature.py, and fixes ExtensionsConfig.resolve_config_path()
to return None instead of raising FileNotFoundError when an explicit
config_path argument or DEER_FLOW_EXTENSIONS_CONFIG_PATH points at a file
that has since been deleted -- the exact resolution mode Docker dev/prod
uses per AGENTS.md, so the MCP tools-cache staleness check could raise
instead of degrading to "not stale". Both were flagged by willem-bd in
review on #4124 and explicitly deferred there ("flagging for visibility",
"leaving the extraction as the follow-up you suggested").
* fix(config): keep explicit extensions-config paths fail-loud
resolve_config_path() previously turned every missing-file case into a
clean None, including an explicit config_path argument or
DEER_FLOW_EXTENSIONS_CONFIG_PATH (the exact mode Docker dev/prod uses).
That silently downgrades a bad Docker mount, typo, or deleted
production config to "no extensions" instead of surfacing the
misconfiguration, per fancyboi999's review [P1] and willem-bd's
follow-up notes on this PR.
Restores FileNotFoundError for the two explicit modes (config_path
argument, DEER_FLOW_EXTENSIONS_CONFIG_PATH); only the fallback search
mode (no explicit path/env var, nothing found in the usual locations)
still returns None, since that is the legitimate "extensions were
never configured" case.
The one caller that needs the old fail-soft behavior -- the MCP
tools-cache staleness check, which re-resolves the path on every
get_cached_mcp_tools() call -- gets a narrow, local catch instead
(deerflow.mcp.cache._resolve_config_path) so a config file going
missing mid-run still degrades the cache to "not stale" rather than
crashing a hot per-request path. Also hoists the double os.getenv()
read in the env-var branch into a local, per willem-bd's nit.
Adds resolver-level tests for both explicit-path and env-var raises,
a dedicated search-mode None regression test, and updates the
existing MCP-cache docstrings to describe the corrected split.
* feat(skills): per-user skill isolation (#2905)
Implement user-scoped skill storage that isolates custom skills between
users while sharing public skills globally.
Key changes:
- Add UserScopedSkillStorage class for per-user custom skill directories
- Introduce get_or_new_user_skill_storage() factory with user_id context
- Auth middleware sets effective_user_id for request-scoped storage
- Agent/prompt/middleware now use user-scoped storage and prompt cache
- Sandbox mounts user-scoped skill directories for search/read tools
- Add validate_skill_file_path() to SkillStorage for path security
- Migration script supports --all-users bulk migration
- Frontend: add editable field to Skill type, error check in enableSkill
- All skill categories can be toggled (custom skills default to enabled)
- Update skill-creator SKILL.md with isolation-aware instructions
Tests:
- Add test_user_scoped_skill_storage.py (new)
- Update all existing skill tests for user-scoped storage
- Update sandbox, client, and router tests
* fix(skills): address second-round PR review feedback (#3889)
- P1-1: restrict legacy skill mount to users without custom skills
- P1-2: fail-closed for _is_disabled_skill_path (OSError → return True)
- P2-1: AND-merge global extensions_config skill disabled state
- P2-2: atomic write for _skill_states.json (mkstemp + replace)
- P2-3: normalize X-DeerFlow-Owner-User-Id in trusted boundary
- P2-4: LRU-bounded _enabled_skills_by_config_cache (OrderedDict, maxsize=256)
- P2-5: clear global prompt cache on PUBLIC skill toggle
- P2-6: invalidate skill caches on client.update_skill
* fix(tests): correct tool policy test after merge
* fix(skills): use DEFAULT_SKILLS_CONTAINER_PATH in UserScopedSkillStorage
The "/mnt/skills" literal in UserScopedSkillStorage.__init__ triggers
test_skill_container_path_defaults::test_mnt_skills_literal_is_owned_by_skill_constants_module
on CI. Migrate the default to the existing deerflow.constants constant,
matching the pattern already used by LocalSkillStorage, SkillStorage, and
the durable/tool_error middlewares.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(mcp): add per-server tool_call_timeout for MCP tool calls
Add a configurable timeout for individual MCP tool calls to prevent
agent runs from blocking indefinitely when an MCP server becomes
unresponsive (e.g., rate-limited HTTP API, hung subprocess).
Uses the MCP SDK's built-in read_timeout_seconds parameter on
ClientSession.call_tool, which handles the timeout within the
session's own task — avoiding cross-task cancellation issues with
the session pool (ref #3379, #3203).
Config field is named tool_call_timeout (not timeout) to avoid
collision with langchain-mcp-adapters' existing timeout field on
HTTP/SSE connections.
Closes#3840
* fix(mcp): read tool_call_timeout from McpServerConfig, not connection dict
The previous implementation put tool_call_timeout into the connection dict
returned by build_server_params, which langchain's create_session then
passed to _create_stdio_session(), causing TypeError. Now reads the timeout
directly from ExtensionsConfig.mcp_servers where the wrapper is built,
keeping it out of the connection dict entirely.
Fixes P1 bug from review on #3843.
* test(mcp): regression test for tool_call_timeout not leaking into connection dict
Adds two tests:
- test_build_server_params_excludes_tool_call_timeout: verifies the connection
dict returned by build_server_params() does NOT contain tool_call_timeout
- test_stdio_tool_call_timeout_does_not_raise_typeerror: end-to-end test that
get_mcp_tools() with a stdio server having tool_call_timeout configured loads
tools without TypeError from _create_stdio_session()
Regression for PR #3843 P1 bug.
* fix(mcp): only pass read_timeout_seconds when tool_call_timeout is set
When tool_call_timeout is None, don't pass read_timeout_seconds=None to
session.call_tool(). This avoids breaking existing tests that assert on
exact call_tool arguments without the extra kwarg.
* docs(mcp): clarify stdio tool timeout
The official MCP configuration schema uses `transport` to specify the
transport mechanism (stdio/sse/http), but `McpServerConfig` only honored
`type` and defaulted to `stdio`. Remote MCP servers configured with just
`transport: sse` were therefore misidentified as stdio and failed with
"with stdio transport requires 'command' field".
Add a model validator that promotes `transport` to `type` when only
`transport` is provided, while keeping `type` authoritative when both
are set. This matches the MCP-spec field name without breaking existing
configurations.
Fixes#3238
* Fix env resolution in MCP config lists
* fix:unset env variable and consistent function
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(harness): resolve runtime paths from project root
* docs(config): update
* fix(config): address runtime path review feedback
* test(config): fix skills path e2e root
* test(config): cover legacy config fallback when project root lacks config files
Verifies that when DEER_FLOW_PROJECT_ROOT is unset and cwd has no
config.yaml/extensions_config.json, AppConfig and ExtensionsConfig fall back
to the legacy backend/repo-root candidates — the backward-compat path
requested in PR #2642 review.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor: extract shared utils to break harness→app cross-layer imports
Move _validate_skill_frontmatter to src/skills/validation.py and
CONVERTIBLE_EXTENSIONS + convert_file_to_markdown to src/utils/file_conversion.py.
This eliminates the two reverse dependencies from client.py (harness layer)
into gateway/routers/ (app layer), preparing for the harness/app package split.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: split backend/src into harness (deerflow.*) and app (app.*)
Physically split the monolithic backend/src/ package into two layers:
- **Harness** (`packages/harness/deerflow/`): publishable agent framework
package with import prefix `deerflow.*`. Contains agents, sandbox, tools,
models, MCP, skills, config, and all core infrastructure.
- **App** (`app/`): unpublished application code with import prefix `app.*`.
Contains gateway (FastAPI REST API) and channels (IM integrations).
Key changes:
- Move 13 harness modules to packages/harness/deerflow/ via git mv
- Move gateway + channels to app/ via git mv
- Rename all imports: src.* → deerflow.* (harness) / app.* (app layer)
- Set up uv workspace with deerflow-harness as workspace member
- Update langgraph.json, config.example.yaml, all scripts, Docker files
- Add build-system (hatchling) to harness pyproject.toml
- Add PYTHONPATH=. to gateway startup commands for app.* resolution
- Update ruff.toml with known-first-party for import sorting
- Update all documentation to reflect new directory structure
Boundary rule enforced: harness code never imports from app.
All 429 tests pass. Lint clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: add harness→app boundary check test and update docs
Add test_harness_boundary.py that scans all Python files in
packages/harness/deerflow/ and fails if any `from app.*` or
`import app.*` statement is found. This enforces the architectural
rule that the harness layer never depends on the app layer.
Update CLAUDE.md to document the harness/app split architecture,
import conventions, and the boundary enforcement test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add config versioning with auto-upgrade on startup
When config.example.yaml schema changes, developers' local config.yaml
files can silently become outdated. This adds a config_version field and
auto-upgrade mechanism so breaking changes (like src.* → deerflow.*
renames) are applied automatically before services start.
- Add config_version: 1 to config.example.yaml
- Add startup version check warning in AppConfig.from_file()
- Add scripts/config-upgrade.sh with migration registry for value replacements
- Add `make config-upgrade` target
- Auto-run config-upgrade in serve.sh and start-daemon.sh before starting services
- Add config error hints in service failure messages
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix comments
* fix: update src.* import in test_sandbox_tools_security to deerflow.*
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: handle empty config and search parent dirs for config.example.yaml
Address Copilot review comments on PR #1131:
- Guard against yaml.safe_load() returning None for empty config files
- Search parent directories for config.example.yaml instead of only
looking next to config.yaml, fixing detection in common setups
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: correct skills root path depth and config_version type coercion
- loader.py: fix get_skills_root_path() to use 5 parent levels (was 3)
after harness split, file lives at packages/harness/deerflow/skills/
so parent×3 resolved to backend/packages/harness/ instead of backend/
- app_config.py: coerce config_version to int() before comparison in
_check_config_version() to prevent TypeError when YAML stores value
as string (e.g. config_version: "1")
- tests: add regression tests for both fixes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: update test imports from src.* to deerflow.*/app.* after harness refactor
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
