mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-09-21 20:16:18 +00:00
36 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Aari
|
236a068e77
|
fix(docker): let the Gateway write extensions_config.json in production (#4852)
* fix(docker): let the Gateway write extensions_config.json in production AGENTS.md states extensions_config.json may be edited at runtime through the Gateway API, and the Gateway implements that for the MCP enable switch, PUT/PATCH /api/mcp/config and the skill update route. Two properties of the production compose stack made every one of those writes fail: - the file was mounted read-only, and - Docker mounts it as its own mount point, so the temp-file-plus-rename in atomic_write_extensions_config hit EBUSY. Linux refuses rename() over a mount point whether or not the mount is writable, so making the mount read-write alone is not enough. Mount it read-write and fall back to an in-place overwrite on EBUSY only. The fallback is deliberately non-atomic and says so in a warning; it is reached only where the atomic route cannot work, and any other errno still propagates. config.yaml stays read-only: no API writes it. docker-compose-dev.yaml mounts the whole project directory, so the destination is an ordinary file there and this never surfaced in development. * test(docker): parse mount options instead of matching a :ro suffix Docker's short-syntax options segment is comma-separated, so a read-only mount can legally be spelled ":ro,z" or ":z,ro" — common with SELinux relabelling. Matching the raw string for a ":ro" suffix reads those as writable, which silently defeats the guard: the writability assertion would pass on a read-only mount, and the config.yaml assertion would fail on a correctly read-only one. Parse the options segment and test membership instead, and cover the parser with the spellings that broke the suffix check. * fix(config): harden mutable extensions config |
||
|
Aari
|
5d520e44a8
|
fix(docker): wait for gateway readiness (#4806)
* fix(docker): verify gateway startup readiness * fix(docker): clarify compose wait requirement |
||
|
Nan Gao
|
c542185a7f
|
feat(extensions): add gateway contribution points and packaged extension management (#4780)
* feat(extensions): add gateway services and routers * feat(extensions): add standalone reference extension * fix(extensions): harden contributed gateway routes * docs(extensions): document gateway contribution points * feat(extensions): add operator CLI for packaged extension management Add `deerflow extensions install/list/enable/disable/remove` plus the root `make extension-*` wrappers, backed by an `ExtensionManager` that owns one transaction over backend/pyproject.toml, backend/uv.lock, the managed source snapshot, the uv environment, and the `plugins:` block in config.yaml. Install accepts a package requirement, a public HTTPS Git URL, or a local directory. Local directories are copied to backend/extensions/sources/ as deployable snapshots rather than editable installs, and the root .dockerignore re-includes that tree so snapshots reach the backend builder. Remote sources are HTTPS-only; SSH Git, file:// and local wheels are rejected because the stock Docker builder cannot reproduce them. Because environment configuration can still resolve a plain package name to a local wheel (a UV_FIND_LINKS wheelhouse, say), every uv add/remove is followed by an audit of the new lock: any local reference the stock image build cannot reproduce rolls back the whole transaction. A config carrying duplicate top-level `plugins:` keys is rejected outright rather than managed against one block while the Gateway reads another. Dependency synchronization now has one lock authority. The `extensions` dependency group joins [tool.uv].default-groups, every startup path syncs the same lock with --locked and launches with --no-sync, and the Docker images move to uv 0.11.1 for the --no-workspace boundary the manager needs. Loader gains `enabled`, `name` and `package` fields so a disabled extension is skipped before resolution and import. Co-authored-by: Codex <codex@openai.com> * fix(extensions): stop the managed plugins rewrite from destroying config Two data-safety defects in the managed `plugins:` block writer. The "next top-level key" boundary was a regex matching only `[A-Za-z_][A-Za-z0-9_-]*` or a quoted key. `AppConfig` is `extra="allow"`, so a config may legally carry any top-level key, and a key the pattern cannot recognize did not fail loudly — it read as "no next section", and the rewrite replaced that neighbour and its entire subtree with the managed block. `my.key`, `2fa`, `$schema`, `my key` and non-ASCII keys were all silently deleted by a plain `extension-enable`/`disable`. Both boundaries now come from the YAML parser's node marks, so key shape is irrelevant. The file-final branch never consulted the trailing-comment scan the has-next-key branch used, so any comment below the block was dropped. Since the manager appends `plugins:` at end of file, that is the steady-state shape for most installs: an operator note below the block was destroyed on the next toggle. Separately, every managed install wrote `required: true` while the loader defaults to false. That turned any later load failure — broken wheel, missing native library, deleted snapshot — into a Gateway startup abort recoverable only with shell access. New records are now written `required: false`, with an explicit `install --required` opt-in; adopting an existing hand-written record still preserves the operator's own choice. * fix(extensions): harden the manager transaction and correct its docs Follow-up hardening on the extension package manager. Security posture, which the docs already claimed: - Scrub `UV_PYTHON`, `UV_INSECURE_HOST`, `UV_CONSTRAINT` and `UV_NO_BUILD_ISOLATION` from the controlled uv environment. `UV_PYTHON` swaps the interpreter that the entry-point probe then imports and calls, and every later `uv run --no-sync` startup uses; `UV_INSECURE_HOST` removes the TLS validation the HTTPS-only source rule depends on. Neither is an index, proxy, cache or credential-provider setting, so neither was covered by the carve-out. - Recognize run-together and all-caps secret query parameters (`accesstoken`, `ACCESSTOKEN`, `key`, `pw`, `sas`, `code`). The camel-case splitter only fires on case transitions, so only the separated spellings were caught. Short generic words stay boundary-anchored, so `?keyword=` remains installable. - Validate the config before running any uv command. `uv add`/`uv sync` execute the package's build backend, so a config the manager could never write to must fail before that code runs rather than afterwards through rollback. Transaction integrity: - Run the second dependency-file restore from a `finally`. The recovery sync runs without `--locked` when the checkout had no lock, so uv writes one while resolving; if that sync then failed, the restore was skipped and the operator kept a lock file they never had. A failing recovery sync now also reports the original failure instead of replacing it. - Skip the recovery sync on cancellation. Answering Ctrl-C with a full dependency resolve invites a second interrupt that escapes the handler and strands the checkout mid-transaction; the declarations are already restored and the next locked startup sync reconciles the environment. - Retry a non-blocking lock on Windows instead of using `msvcrt.LK_LOCK`, which gives up after ~10s — far shorter than a real `uv add` plus `uv sync`, so contention surfaced as `Permission denied` rather than serializing. - Locate the entry-point probe's JSON payload instead of parsing stdout's first line, so a `sitecustomize`/`.pth` banner cannot roll back a good install. - Warn when the lock records a loopback source. `127.0.0.1` inside the image builder is a different machine, but unlike an environment-driven wheelhouse resolution this is a source the operator typed deliberately, so it is reported rather than rolled back. Private-network indexes are untouched: a builder on that network can reach them. Docs: the blanket claim that failed operations restore the config file was wrong — the conflict branches deliberately preserve a concurrent external edit and leave `remove` deactivated. Document that, the `required: false` default, the config preflight, the interrupt behaviour, and where the plugins-block boundaries come from. * test(gateway): pin the request-path projection agreement `get_request_route_path()` imports the private `starlette._utils.get_route_path` so the auth and CSRF predicates classify the exact string Starlette's router matches on. Its requirement is not "strip root_path correctly" but "return what the dispatcher is matching", so delegating to the router's own implementation keeps the two in lockstep by construction. Keep the private import rather than vendoring a copy: an import that disappears fails loudly at startup, while a stale copy diverges silently at a security boundary. Cover the property directly instead of the mechanism, so the tests survive a future reimplementation: - projection edge cases, including the segment-boundary guard that keeps root_path="/api" from slicing "/apifoo/models" into a string the router would never match - agreement with the router under nested mounts - the two bypasses these predicates exist to prevent: a protected route mounted under the "/health" public prefix must still 401, and a POST mounted under "/api/webhooks" must still require a CSRF token Both are verified to fail when the projection is reverted to `request.url.path` (9/13 red) and when a plausible vendored copy omits the boundary guard (the 2 boundary cases red). Declare starlette as a bounded direct dependency so a bump — which is security-relevant here — shows up in review rather than arriving silently through FastAPI. * ci: pin uv to the version production ships ExtensionManager is not a consumer of uv the build tool -- it is a program whose whole job is driving `uv` as a subprocess, depending on its CLI behavior (`--no-workspace`, `--no-sync`, what `uv add` writes into `[dependency-groups] extensions`) and on the `uv.lock` serialization format. uv is closer to a runtime dependency with a contract than to incidental tooling. backend/Dockerfile pins that binary to 0.11.1, but all eight astral-sh/setup-uv steps installed whatever was latest at run time, so CI exercised the manager against a uv that is not the uv production runs. The sharpest failure that allows: a newer uv bumps uv.lock's `revision`, CI stays green because the same uv reads back what it wrote, and the pinned uv in the production image cannot read the committed lock. `uv lock --check` is version-sensitive for the same reason -- it verifies the lock is what *this* uv would produce, and two versions can emit equivalent but non-identical output. Pin every step to 0.11.1 and lift the one lingering setup-uv@v3 to v7 so the steps share input and caching behavior. Pinning alone drifts apart again on the next bump, so add a constraint test in the style of test_compose_default_bind_host.py: the Dockerfile's UV_IMAGE tag is the single source of truth, and both compose defaults plus every setup-uv step must match it. Verified to fail when a pin drifts, when a step omits `version`, and -- the real scenario -- when the Dockerfile is bumped alone, which lights up the workflows and both compose files at once. * fix(gateway): state the extension route auth limit and abort a failed dev sync Two scoped review follow-ups. README: contributed routers cannot enter the host's reserved public prefixes, which makes every extension endpoint session-authenticated -- there is no way to expose an unauthenticated route. The rejection rule was documented but its consequence was not, so inbound provider webhooks and public status endpoints read as merely undocumented rather than out of scope for this release. docker/dev-entrypoint.sh: the self-heal retry reuses `--locked`, so it repairs a corrupt .venv but never a lock that disagrees with pyproject.toml. `set -e` already stopped the script there -- uvicorn was not being started against a stale environment -- but it exited on a bare uv exit code with no indication of what to do. Abort explicitly with the cause and the fix. Tests slice the sync block out of the real script and run it against a stub uv, so they exercise the shipped code rather than a copy of it (/app/backend only exists inside the container). They cover the success path, the retry that recovers, the abort, and the guidance. Verified against the pre-fix script: only the guidance case goes red, confirming the abort itself was already correct. * fix(extensions): point Git SSH shorthand at the HTTPS correction Git's SCP-like shorthand carries no URL scheme, so `git+git@host:org/repo.git` reached the scheme rules looking like a bare path and was rejected with "local path references are not deployable; pass a local directory so DeerFlow can snapshot it". The operator asked for a remote source, so that guidance points at the wrong fix. Detect the shorthand ahead of the scheme rules and report the public-HTTPS correction instead. The bare `git@host:org/repo.git` spelling took a different wrong turn: packaging parses it as a direct reference named `git`, leaving `host:org/repo.git`, whose `host` reads as a URL scheme and produced the generic HTTPS message. Both spellings now share one message, as does the PEP 508 named form. * docs: keep the root extension summary within its new budget #4799 split the depth out of the module guides and added a size gate; the root file's job is now orientation, and this branch had pushed it 192 bytes past the soft limit. The manager transaction, source rules, and lock discipline are already stated in full in the extensions guide, so the root keeps the one-line orientation and points there instead of restating them. --------- Co-authored-by: Codex <codex@openai.com> |
||
|
Nan Gao
|
2a143dced6
|
fix(docker): bind the published entry port to loopback by default (#4618)
README documents DeerFlow as deployed by default "in a local trusted
environment (accessible only via the 127.0.0.1 loopback interface)", but both
compose files published nginx as `"${PORT:-2026}:2026"`, which Docker binds to
0.0.0.0 and [::]. The shipped artifact did not match its own documented
default, so running it on a LAN or cloud host produced a wider surface than
the docs implied without the operator changing anything -- and the agent can
execute commands.
Publish as `"${BIND_HOST:-127.0.0.1}:${PORT:-2026}:2026"` in both compose
files, so the default matches the documented model while operators who front
the stack with their own TLS/auth can still widen it via BIND_HOST. The
Gateway keeps binding 0.0.0.0:8001 inside the container (nginx reaches it over
the compose network) and its port stays unpublished, so the published nginx
port is the entire external surface.
BREAKING CHANGE: a deployment that relied on the previous 0.0.0.0 default
becomes unreachable from other hosts after this upgrade. Set BIND_HOST=0.0.0.0
in .env to restore it, after putting authentication in front and completing
first-run setup.
Also:
- .env.example documents BIND_HOST and PORT with the reasoning.
- deploy.sh reports the address the stack actually bound and, when it is not
loopback, tells the operator to complete first-run setup immediately. It
reads BIND_HOST/PORT from .env via a new read_dotenv_value helper following
compose precedence; the shell does not source .env, so reading the
environment alone would have reported "loopback only" for a stack .env had
exposed. The pre-existing ${PORT} summary line had the same defect and is
fixed with it.
- test_compose_default_bind_host.py pins the loopback default, that BIND_HOST
stays overridable, and that no service in either compose file publishes a
port without an explicit bind address, so a later addition cannot drift back
to 0.0.0.0 unnoticed.
|
||
|
Xinmin Zeng
|
f2e832330e
|
fix(sandbox): enforce disabled skills in filesystem views (#4178)
* fix(sandbox): project enabled skills into sandbox views * fix(skills): keep projection mutations consistent * fix(skills): fail closed on projection errors * fix(skills): isolate per-scope failures during boot projection rebuild rebuild_all_skill_projections() propagated any exception from the public rebuild or from a single user's rebuild straight out of the gateway lifespan startup, uncaught. A single broken user directory (bad permissions, corrupted _skill_states.json, unreadable content) would therefore abort gateway boot for every user, not just that one - _rebuild_*_locked already fails closed internally (clears the view and re-raises), so the boot loop only needed to stop treating that re-raise as fatal. Each scope's rebuild now fails closed independently and boot continues; a scope left empty by a boot failure self-heals on the next sandbox acquire via ensure_skill_projections(). Also patches deerflow.skills.projection.rebuild_all_skill_projections in the memory-flush lifespan test fixture, matching the two sibling fixtures in the same file — this call is now on the lifespan startup path and the fixture's minimal SimpleNamespace config predates it. * test(skills): update authz test for the projection-aware public toggle _persist_shared_skill_state (introduced earlier in this branch) reads the shared extensions_config.json fresh from disk under the projection lock instead of through the cached get_extensions_config() singleton - that's the whole point of the fix (stale worker caches must not clobber another worker's concurrent update). The name no longer exists on the skills router module, so the test's monkeypatch of it started raising AttributeError instead of exercising the endpoint. The mock storage in this test isn't a real LocalSkillStorage instance, so _persist_shared_skill_state's projection-mutation branch is already skipped (nullcontext) and it falls back to a fresh ExtensionsConfig() for the nonexistent tmp config_path - no replacement monkeypatch needed. * fix(sandbox): make skill projection ensure best-effort in acquire acquire() called _ensure_skills_projection() directly, outside any try/except, in both LocalSandboxProvider and AioSandboxProvider. Every other skill-mount setup path in these providers has always caught exceptions and logged a warning rather than failing sandbox acquire outright (e.g. when config.yaml can't be resolved) - these two new call sites broke that contract, so any projection failure (including simply not having a config.yaml, as in CI's test environment) now failed acquire() itself instead of just leaving skill mounts off. _ensure_skills_projection now catches its own exceptions and returns None; both providers' callers already tolerate that (a None projection skips the skill-specific mounts, matching the existing degrade path) after making _append_public_skill_mapping and the custom/legacy mount block in LocalSandboxProvider explicitly None-safe. Caught by running the full suite with config.yaml removed, matching CI's environment - not caught locally because a real config.yaml was present, masking the failure. * fix(sandbox): make E2B skill projection mounts best-effort _skill_projection_mounts called ensure_skill_projections with no guard, unlike Local/AIO's _ensure_skills_projection. A raise propagated out of _apply_mounts before the configured-mounts loop ran, so a skills projection failure dropped the operator's own configured mounts too - only caught by create()'s outer warning, with nothing applied at all. Swallow here and return an empty mount list on failure, matching the Local/AIO pattern: still fail-closed for skills, but no longer widens the blast radius to unrelated configured mounts. Review feedback from PR #4178. * docs(skills): document projection trade-offs flagged in review - _update_tree_digest: note the metadata-only (not content) hashing trade-off and why runtime writes through this codebase are still covered regardless (rebuild-under-lock + rename always changes inode). - LocalSandboxProvider.acquire: note the acquire-time self-heal cost (cheap on a fresh manifest, ~400ms rebuild under lock on stale/drift). - skill_projection_mutation: drop the no-op except-Exception-then-raise; a raise from the mutation already propagates past the yield with the view left cleared, no explicit re-raise needed. - provisioner README: spell out that hostPath skills volumes require the gateway and K8s node to share DEER_FLOW_HOST_BASE_DIR (single-node or shared storage), and that the custom/legacy volumes' hostPath type Directory (not DirectoryOrCreate) makes a violation of that assumption a visible Pod-creation failure instead of a silent empty mount. Review feedback from PR #4178. * fix(skills): lazily repair user projections * fix(skills): close projection review gaps * fix(skills): refresh user projection enable state * fix(skills): close projection review follow-ups * fix(skills): preserve state across projection writes --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
RongfuShuiping
|
2aaf74b0f8
|
feat(memory): add OpenViking HTTP backend (#4509)
* feat(memory): add OpenViking HTTP backend * fix(memory): harden OpenViking lifecycle --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Ryker_Feng
|
aacb99cfd2
|
feat(lark): sidecar credential broker for sandbox lark-cli (Pattern B) (#4501)
* feat(lark): sidecar credential broker for sandbox lark-cli (Pattern B) Removes the plaintext Lark credential mounts (appSecret + OAuth tokens) from the sandbox container. A long-running broker sidecar owns lark-cli and the per-user config/data dirs and serves the command surface over Pod loopback; the sandbox gets only a forwarding shim on PATH, so the raw credential files never exist in the sandbox filesystem. - lark_broker.py: stdlib-only loopback broker (argv passthrough with shell=False, server-injected credential env, bounded I/O) + shim script constant + install-shim mode. - docker/lark-cli-broker: init(install-shim) + serve image. - provisioner: LARK_CLI_BROKER_IMAGE + provision_lark_cli_broker → shim init container + lark-cli-broker sidecar (config/data mounted sidecar-only); credentials dropped from the sandbox container; /api/capabilities reports lark_cli_broker_image. Broker supersedes the Pattern A init-container binary when both are configured. - gateway: lark_cli_env_overlay(broker=True) omits config/data env; sandbox_lark_broker_active() TTL-cached mode resolver; broker added to sandbox_runtime_mode / readiness and the settings UI. Opt-in and off by default (empty LARK_CLI_BROKER_IMAGE ⇒ no change). Closes #4338 * fix(lark): address Pattern B broker review findings (#4501) Follow-up to the sidecar credential broker addressing the PR #4501 review: - shim: split the on-PATH lark-cli into a /bin/sh launcher + Python shim body so broker mode fails loudly (exit 127, actionable message) instead of ENOEXEC when the sandbox image ships no python3; interpreter pinnable via DEERFLOW_LARK_BROKER_PYTHON. Launcher bakes in the shim's absolute path since $0 is the bare command name when run off PATH. - broker: drop the dead cwd payload field (broker can't see the sandbox FS) and document the command-surface-only / no-file-IO limitation. - broker: return a structured 500 JSON on unexpected exec errors so the shim gets a meaningful message, not an opaque transport failure; set a handler socket timeout to bound slow/stuck connections. - broker: add an opt-in DEERFLOW_LARK_BROKER_DENY_SUBCOMMANDS denylist that refuses secret-dumping subcommands before spawning the binary, forwarded from the provisioner sidecar. - gateway: tighten the per-bash-call broker probe timeout (1.5s) and cache negatives longer (300s) so non-broker remote-provisioner users don't pay a latency hit; guard the mode cache with a lock; drop the dead _probe_provisioner_lark_cli_init_image wrapper. - docs: remove the broken design-doc link from the broker README. Adds tests for launcher python resolution, cwd omission, denylist enforcement, 500-on-error, hot-path probe timeout + negative caching, and provisioner denylist-env wiring. |
||
|
Ryker_Feng
|
7aa314b4c1
|
feat: add Lark CLI integration (#3971)
* feat: add lark cli integration * fix: polish lark integration actions * feat: support lark incremental permissions * fix: detect lark authorization completion * fix: harden lark integration install * feat: expand lark auth scopes and reuse host auth in sandbox Default lark auth to least-privilege (recommend=false, base sign-in only) and expose the full set of lark-cli --domain business domains as native --domain grants instead of a 4-domain read-only mapping. Resolve the skill pack from the latest larksuite/cli GitHub release at install time with content-hash integrity, and surface version/runtime drift in status. Share the per-user lark-cli config/data profile between the Gateway Settings auth flow and agent conversations by mounting the integration dirs into the AIO sandbox and injecting the matching env for lark-cli commands, with an allowlisted extra_mounts path in the provisioner/K8s backend and traversal guards on integration paths. * style: fix lint issues from ruff and prettier Sort imports in the provisioner PVC test and re-wrap two long i18n description strings to satisfy backend ruff and frontend prettier CI. * fix(lark): address managed integration review feedback * fix(frontend): stabilize integrations settings e2e * test(sandbox): isolate remote backend legacy visibility check * test: fix backend unit failures after merge * Harden Lark integration review fixes * Format Lark integration E2E test * fix(lark): harden sandbox credential exposure and status disclosure Address willem_bd's security review on PR #3971: - Mount the per-user lark-cli config dir (long-lived appSecret) read-only into the AIO sandbox; only the refreshable-token data dir stays writable. - Redact host filesystem paths (install_path, cli.path) from GET /lark/status and the config/auth complete responses for non-admin callers, fail-closed on any auth error. - Document the npm postinstall trade-off (--ignore-scripts is not viable because @larksuite/cli fetches its platform binary in postinstall). - Document the sandbox credential trust boundary in AGENTS.md and README, pointing at the sidecar-broker follow-up (#4338). --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Vanzeren
|
e3137b13cf
|
fix(provisioner): gate legacy skills mount by user visibility (#3985)
* fix(provisioner): gate legacy skills mount by user visibility * fix(test): aio sandbox provider * fix: use shared legacy skill visibility helper for sandbox mounts |
||
|
Janlay
|
72f033fbbe
|
feat(gateway): add redis stream bridge (#3191)
* feat: add redis stream bridge * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(gateway): address redis stream bridge review Redis was imported eagerly through deerflow.runtime and declared as a hard dependency, which made memory-only installs load redis.asyncio at startup and left the lazy factory import ineffective. Move redis behind an optional extra, remove the public eager re-export, and keep make_stream_bridge as the only runtime import path with an actionable install hint when the extra is missing. Because Docker deployments now default the stream bridge to Redis via DEER_FLOW_STREAM_BRIDGE_REDIS_URL, install the redis extra explicitly in Docker/dev container flows and teach the local uv-extra detector to infer redis from both stream_bridge.type and the Redis URL env var. This keeps Docker working while preserving slim non-Docker installs. Harden the Redis bridge by batching XREAD replay, replacing brittle ResponseError string matching with a single fallback to 0-0 for malformed Last-Event-ID values, documenting connection/retention/fail-hard behavior, and adding fake plus opt-in real Redis coverage for XADD/XREAD, replay, invalid IDs, and MAXLEN trimming. * fix(config): bump config version for stream bridge * fix redis stream bridge terminal handling * fix: repair uv.lock, format redis.py, and align Dockerfile extras test The uv.lock file was missing a closing bracket for the redis extras section, redis.py had a formatting issue caught by ruff, and the Dockerfile extras test did not account for the hardcoded --extra redis flag. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Willem Jiang
|
0fb2a75bfb | fix(doc):update the document for the docker configuration | ||
|
Xinmin Zeng
|
5d61718c80
|
fix(security): mount host Docker socket only in aio (DooD) sandbox mode (#3517)
* fix(security): mount host Docker socket only in aio (DooD) sandbox mode The default Compose stack mounted /var/run/docker.sock read-write into the root gateway container in every sandbox mode, including the default `local` mode that never uses it -- an unnecessary host-escape surface (DooD = root-equivalent host control). deploy.sh already gated the socket *check* on sandbox_mode != local, but the Compose files mounted it unconditionally. Move the socket mount to an opt-in docker/docker-compose.dood.yaml overlay that deploy.sh / docker.sh append only when detect_sandbox_mode() returns `aio`. Default (local) and provisioner/Kubernetes modes no longer expose the host daemon. Tighten the socket existence check from != local to == aio. Document the DooD threat model in SECURITY.md. Reported by @greatmengqi. * refactor(docker): address review on socket-hardening PR - docker.sh: use absolute path for the dood overlay (match deploy.sh, drop cwd dependency) - deploy.sh: drop now-dead DEER_FLOW_DOCKER_SOCKET exports in down/build paths - docker-compose.yaml: fix stale header comment to point at the overlay Addresses codex + reviewer feedback on #3517. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Xinmin Zeng
|
474c89bac2
|
fix(security): do not bind-mount host CLI auth dirs by default (#3521)
* fix(security): do not bind-mount host CLI auth dirs by default The Compose stack bind-mounted the entire ~/.claude and ~/.codex dirs (read-only) into the root gateway container in every configuration -- exposing not just credentials but full conversation history, per-project session data, and global CLI config. The default OpenAI-compatible model providers and the local sandbox never use them. Move the mounts to an opt-in docker/docker-compose.cli-auth.yaml overlay. Document env-token paths (CLAUDE_CODE_OAUTH_TOKEN, CODEX_AUTH_PATH) in .env.example -- the Gateway credential loader reads env first, so most setups need no mount at all. Document the exposure and per-mode options in SECURITY.md. Reported by @greatmengqi. * docs: clarify ACP adapter auth and add Claude single-file credential option - ACP adapters authenticate independently (many take an env API key like ANTHROPIC_API_KEY and need no mount); the cli-auth overlay is only for adapters that read the full CLI config dir. Avoids steering users toward mounting the whole dir for ACP when env auth usually suffices. - Add CLAUDE_CODE_CREDENTIALS_PATH (single .credentials.json) as a Claude one-file option, matching codex CODEX_AUTH_PATH and the README. * docs: cite claude-code-acp env auth and CLAUDE_CONFIG_DIR in ACP guidance Replace the generic 'some adapters' wording with the verified behavior of the common claude-code-acp adapter (env ANTHROPIC_API_KEY startup + CLAUDE_CONFIG_DIR), so the 'no ~/.claude mount needed for ACP' guidance is backed by a concrete adapter. |
||
|
Xinmin Zeng
|
05ae4467ae
|
fix(docker): default Gateway to a single worker to prevent multi-worker breakage (#3475)
The default `make up` started the Gateway with `--workers 4`, but run state (RunManager and the stream bridge) is held in-process and nginx uses no sticky sessions. With the default config, same-run requests scatter across workers that each keep their own run state, breaking run cancellation (409), SSE reconnect (hangs on heartbeats), multitask de-duplication, and IM channels (duplicate replies). The shared cross-worker stream bridge does not exist yet. Default GATEWAY_WORKERS to 1 so the out-of-the-box deployment is correct, document the single-worker boundary in the README, and add a regression test pinning the default while keeping it overridable. This is a stop-gap, not a multi-worker implementation; the full fix (shared run state + stream bridge) is tracked in #3191. Refs #3239, #3260 |
||
|
Ryker_Feng
|
f92a26d56f
|
fix(web_fetch): support proxy for Jina reader in restricted networks (#3418) (#3430)
* fix(web_fetch): support proxy for Jina reader in restricted networks The web_fetch tool built a bare httpx.AsyncClient() with no proxy awareness, so users behind a corporate proxy / in Docker / WSL could not reach https://r.jina.ai and web_fetch timed out. - Add optional `proxy` / `trust_env` params to JinaClient.crawl and wire them from the `web_fetch` tool config (with type coercion for YAML string values). - Pass internal service hostnames through NO_PROXY in both compose files so proxy env inherited via env_file does not break in-cluster calls (gateway/provisioner/etc). - Load proxy vars from .env into the shell in scripts/docker.sh so the NO_PROXY interpolation can merge user-provided values on `make` path. - Document proxy/trust_env options in config.example.yaml. Closes #3418 * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Stellar鱼
|
b00749a8a6
|
fix(auth): share internal gateway token across workers (#3184)
* fix(auth): share internal gateway token across workers * fix: restore deploy script executable bit * Update deploy.sh to skip the auth_token setup for the down command --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
c09c334544
|
fix(harness): resolve runtime paths from project root (#2642)
* fix(harness): resolve runtime paths from project root * docs(config): update * fix(config): address runtime path review feedback * test(config): fix skills path e2e root * test(config): cover legacy config fallback when project root lacks config files Verifies that when DEER_FLOW_PROJECT_ROOT is unset and cwd has no config.yaml/extensions_config.json, AppConfig and ExtensionsConfig fall back to the legacy backend/repo-root candidates — the backward-compat path requested in PR #2642 review. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
JeffJiang
|
7bf618de67 |
Refactor DeerFlow to use Gateway's LangGraph-compatible API
- Updated documentation and comments to reflect the transition from LangGraph Server to Gateway. - Changed default URLs in ChannelManager and tests to point to Gateway. - Removed references to LangGraph Server in deployment scripts and configurations. - Updated Nginx configuration to route API traffic to Gateway. - Adjusted frontend configurations to utilize Gateway's API. - Removed LangGraph service from Docker Compose files, consolidating services under Gateway. - Added regression tests to ensure Gateway integration works as expected. Co-authored-by: Copilot <copilot@github.com> |
||
|
rayhpeng
|
d8ecaf46c9 |
feat(persistence): add unified persistence layer with event store, token tracking, and feedback (#1930)
* feat(persistence): add SQLAlchemy 2.0 async ORM scaffold
Introduce a unified database configuration (DatabaseConfig) that
controls both the LangGraph checkpointer and the DeerFlow application
persistence layer from a single `database:` config section.
New modules:
- deerflow.config.database_config — Pydantic config with memory/sqlite/postgres backends
- deerflow.persistence — async engine lifecycle, DeclarativeBase with to_dict mixin, Alembic skeleton
- deerflow.runtime.runs.store — RunStore ABC + MemoryRunStore implementation
Gateway integration initializes/tears down the persistence engine in
the existing langgraph_runtime() context manager. Legacy checkpointer
config is preserved for backward compatibility.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(persistence): add RunEventStore ABC + MemoryRunEventStore
Phase 2-A prerequisite for event storage: adds the unified run event
stream interface (RunEventStore) with an in-memory implementation,
RunEventsConfig, gateway integration, and comprehensive tests (27 cases).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(persistence): add ORM models, repositories, DB/JSONL event stores, RunJournal, and API endpoints
Phase 2-B: run persistence + event storage + token tracking.
- ORM models: RunRow (with token fields), ThreadMetaRow, RunEventRow
- RunRepository implements RunStore ABC via SQLAlchemy ORM
- ThreadMetaRepository with owner access control
- DbRunEventStore with trace content truncation and cursor pagination
- JsonlRunEventStore with per-run files and seq recovery from disk
- RunJournal (BaseCallbackHandler) captures LLM/tool/lifecycle events,
accumulates token usage by caller type, buffers and flushes to store
- RunManager now accepts optional RunStore for persistent backing
- Worker creates RunJournal, writes human_message, injects callbacks
- Gateway deps use factory functions (RunRepository when DB available)
- New endpoints: messages, run messages, run events, token-usage
- ThreadCreateRequest gains assistant_id field
- 92 tests pass (33 new), zero regressions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(persistence): add user feedback + follow-up run association
Phase 2-C: feedback and follow-up tracking.
- FeedbackRow ORM model (rating +1/-1, optional message_id, comment)
- FeedbackRepository with CRUD, list_by_run/thread, aggregate stats
- Feedback API endpoints: create, list, stats, delete
- follow_up_to_run_id in RunCreateRequest (explicit or auto-detected
from latest successful run on the thread)
- Worker writes follow_up_to_run_id into human_message event metadata
- Gateway deps: feedback_repo factory + getter
- 17 new tests (14 FeedbackRepository + 3 follow-up association)
- 109 total tests pass, zero regressions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test+config: comprehensive Phase 2 test coverage + deprecate checkpointer config
- config.example.yaml: deprecate standalone checkpointer section, activate
unified database:sqlite as default (drives both checkpointer + app data)
- New: test_thread_meta_repo.py (14 tests) — full ThreadMetaRepository coverage
including check_access owner logic, list_by_owner pagination
- Extended test_run_repository.py (+4 tests) — completion preserves fields,
list ordering desc, limit, owner_none returns all
- Extended test_run_journal.py (+8 tests) — on_chain_error, track_tokens=false,
middleware no ai_message, unknown caller tokens, convenience fields,
tool_error, non-summarization custom event
- Extended test_run_event_store.py (+7 tests) — DB batch seq continuity,
make_run_event_store factory (memory/db/jsonl/fallback/unknown)
- Extended test_phase2b_integration.py (+4 tests) — create_or_reject persists,
follow-up metadata, summarization in history, full DB-backed lifecycle
- Fixed DB integration test to use proper fake objects (not MagicMock)
for JSON-serializable metadata
- 157 total Phase 2 tests pass, zero regressions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* config: move default sqlite_dir to .deer-flow/data
Keep SQLite databases alongside other DeerFlow-managed data
(threads, memory) under the .deer-flow/ directory instead of a
top-level ./data folder.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(persistence): remove UTFJSON, use engine-level json_serializer + datetime.now()
- Replace custom UTFJSON type with standard sqlalchemy.JSON in all ORM
models. Add json_serializer=json.dumps(ensure_ascii=False) to all
create_async_engine calls so non-ASCII text (Chinese etc.) is stored
as-is in both SQLite and Postgres.
- Change ORM datetime defaults from datetime.now(UTC) to datetime.now(),
remove UTC imports.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(gateway): simplify deps.py with getter factory + inline repos
- Replace 6 identical getter functions with _require() factory.
- Inline 3 _make_*_repo() factories into langgraph_runtime(), call
get_session_factory() once instead of 3 times.
- Add thread_meta upsert in start_run (services.py).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(docker): add UV_EXTRAS build arg for optional dependencies
Support installing optional dependency groups (e.g. postgres) at
Docker build time via UV_EXTRAS build arg:
UV_EXTRAS=postgres docker compose build
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(journal): fix flush, token tracking, and consolidate tests
RunJournal fixes:
- _flush_sync: retain events in buffer when no event loop instead of
dropping them; worker's finally block flushes via async flush().
- on_llm_end: add tool_calls filter and caller=="lead_agent" guard for
ai_message events; mark message IDs for dedup with record_llm_usage.
- worker.py: persist completion data (tokens, message count) to RunStore
in finally block.
Model factory:
- Auto-inject stream_usage=True for BaseChatOpenAI subclasses with
custom api_base, so usage_metadata is populated in streaming responses.
Test consolidation:
- Delete test_phase2b_integration.py (redundant with existing tests).
- Move DB-backed lifecycle test into test_run_journal.py.
- Add tests for stream_usage injection in test_model_factory.py.
- Clean up executor/task_tool dead journal references.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(events): widen content type to str|dict in all store backends
Allow event content to be a dict (for structured OpenAI-format messages)
in addition to plain strings. Dict values are JSON-serialized for the DB
backend and deserialized on read; memory and JSONL backends handle dicts
natively. Trace truncation now serializes dicts to JSON before measuring.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(events): use metadata flag instead of heuristic for dict content detection
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(converters): add LangChain-to-OpenAI message format converters
Pure functions langchain_to_openai_message, langchain_to_openai_completion,
langchain_messages_to_openai, and _infer_finish_reason for converting
LangChain BaseMessage objects to OpenAI Chat Completions format, used by
RunJournal for event storage. 15 unit tests added.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(converters): handle empty list content as null, clean up test
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(events): human_message content uses OpenAI user message format
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(events): ai_message uses OpenAI format, add ai_tool_call message event
- ai_message content now uses {"role": "assistant", "content": "..."} format
- New ai_tool_call message event emitted when lead_agent LLM responds with tool_calls
- ai_tool_call uses langchain_to_openai_message converter for consistent format
- Both events include finish_reason in metadata ("stop" or "tool_calls")
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(events): add tool_result message event with OpenAI tool message format
Cache tool_call_id from on_tool_start keyed by run_id as fallback for on_tool_end,
then emit a tool_result message event (role=tool, tool_call_id, content) after each
successful tool completion.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(events): summary content uses OpenAI system message format
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(events): replace llm_start/llm_end with llm_request/llm_response in OpenAI format
Add on_chat_model_start to capture structured prompt messages as llm_request events.
Replace llm_end trace events with llm_response using OpenAI Chat Completions format.
Track llm_call_index to pair request/response events.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(events): add record_middleware method for middleware trace events
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* test(events): add full run sequence integration test for OpenAI content format
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(events): align message events with checkpoint format and add middleware tag injection
- Message events (ai_message, ai_tool_call, tool_result, human_message) now use
BaseMessage.model_dump() format, matching LangGraph checkpoint values.messages
- on_tool_end extracts tool_call_id/name/status from ToolMessage objects
- on_tool_error now emits tool_result message events with error status
- record_middleware uses middleware:{tag} event_type and middleware category
- Summarization custom events use middleware:summarize category
- TitleMiddleware injects middleware:title tag via get_config() inheritance
- SummarizationMiddleware model bound with middleware:summarize tag
- Worker writes human_message using HumanMessage.model_dump()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(threads): switch search endpoint to threads_meta table and sync title
- POST /api/threads/search now queries threads_meta table directly,
removing the two-phase Store + Checkpointer scan approach
- Add ThreadMetaRepository.search() with metadata/status filters
- Add ThreadMetaRepository.update_display_name() for title sync
- Worker syncs checkpoint title to threads_meta.display_name on run completion
- Map display_name to values.title in search response for API compatibility
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(threads): history endpoint reads messages from event store
- POST /api/threads/{thread_id}/history now combines two data sources:
checkpointer for checkpoint_id, metadata, title, thread_data;
event store for messages (complete history, not truncated by summarization)
- Strip internal LangGraph metadata keys from response
- Remove full channel_values serialization in favor of selective fields
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove duplicate optional-dependencies header in pyproject.toml
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(middleware): pass tagged config to TitleMiddleware ainvoke call
Without the config, the middleware:title tag was not injected,
causing the LLM response to be recorded as a lead_agent ai_message
in run_events.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: resolve merge conflict in .env.example
Keep both DATABASE_URL (from persistence-scaffold) and WECOM
credentials (from main) after the merge.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(persistence): address review feedback on PR #1851
- Fix naive datetime.now() → datetime.now(UTC) in all ORM models
- Fix seq race condition in DbRunEventStore.put() with FOR UPDATE
and UNIQUE(thread_id, seq) constraint
- Encapsulate _store access in RunManager.update_run_completion()
- Deduplicate _store.put() logic in RunManager via _persist_to_store()
- Add update_run_completion to RunStore ABC + MemoryRunStore
- Wire follow_up_to_run_id through the full create path
- Add error recovery to RunJournal._flush_sync() lost-event scenario
- Add migration note for search_threads breaking change
- Fix test_checkpointer_none_fix mock to set database=None
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: update uv.lock
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(persistence): address 22 review comments from CodeQL, Copilot, and Code Quality
Bug fixes:
- Sanitize log params to prevent log injection (CodeQL)
- Reset threads_meta.status to idle/error when run completes
- Attach messages only to latest checkpoint in /history response
- Write threads_meta on POST /threads so new threads appear in search
Lint fixes:
- Remove unused imports (journal.py, migrations/env.py, test_converters.py)
- Convert lambda to named function (engine.py, Ruff E731)
- Remove unused logger definitions in repos (Ruff F841)
- Add logging to JSONL decode errors and empty except blocks
- Separate assert side-effects in tests (CodeQL)
- Remove unused local variables in tests (Ruff F841)
- Fix max_trace_content truncation to use byte length, not char length
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* style: apply ruff format to persistence and runtime files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Potential fix for pull request finding 'Statement has no effect'
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
* refactor(runtime): introduce RunContext to reduce run_agent parameter bloat
Extract checkpointer, store, event_store, run_events_config, thread_meta_repo,
and follow_up_to_run_id into a frozen RunContext dataclass. Add get_run_context()
in deps.py to build the base context from app.state singletons. start_run() uses
dataclasses.replace() to enrich per-run fields before passing ctx to run_agent.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(gateway): move sanitize_log_param to app/gateway/utils.py
Extract the log-injection sanitizer from routers/threads.py into a shared
utils module and rename to sanitize_log_param (public API). Eliminates the
reverse service → router import in services.py.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* perf: use SQL aggregation for feedback stats and thread token usage
Replace Python-side counting in FeedbackRepository.aggregate_by_run with
a single SELECT COUNT/SUM query. Add RunStore.aggregate_tokens_by_thread
abstract method with SQL GROUP BY implementation in RunRepository and
Python fallback in MemoryRunStore. Simplify the thread_token_usage
endpoint to delegate to the new method, eliminating the limit=10000
truncation risk.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: annotate DbRunEventStore.put() as low-frequency path
Add docstring clarifying that put() opens a per-call transaction with
FOR UPDATE and should only be used for infrequent writes (currently
just the initial human_message event). High-throughput callers should
use put_batch() instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(threads): fall back to Store search when ThreadMetaRepository is unavailable
When database.backend=memory (default) or no SQL session factory is
configured, search_threads now queries the LangGraph Store instead of
returning 503. Returns empty list if neither Store nor repo is available.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(persistence): introduce ThreadMetaStore ABC for backend-agnostic thread metadata
Add ThreadMetaStore abstract base class with create/get/search/update/delete
interface. ThreadMetaRepository (SQL) now inherits from it. New
MemoryThreadMetaStore wraps LangGraph BaseStore for memory-mode deployments.
deps.py now always provides a non-None thread_meta_repo, eliminating all
`if thread_meta_repo is not None` guards in services.py, worker.py, and
routers/threads.py. search_threads no longer needs a Store fallback branch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(history): read messages from checkpointer instead of RunEventStore
The /history endpoint now reads messages directly from the
checkpointer's channel_values (the authoritative source) instead of
querying RunEventStore.list_messages(). The RunEventStore API is
preserved for other consumers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(persistence): address new Copilot review comments
- feedback.py: validate thread_id/run_id before deleting feedback
- jsonl.py: add path traversal protection with ID validation
- run_repo.py: parse `before` to datetime for PostgreSQL compat
- thread_meta_repo.py: fix pagination when metadata filter is active
- database_config.py: use resolve_path for sqlite_dir consistency
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Implement skill self-evolution and skill_manage flow (#1874)
* chore: ignore .worktrees directory
* Add skill_manage self-evolution flow
* Fix CI regressions for skill_manage
* Address PR review feedback for skill evolution
* fix(skill-evolution): preserve history on delete
* fix(skill-evolution): tighten scanner fallbacks
* docs: add skill_manage e2e evidence screenshot
* fix(skill-manage): avoid blocking fs ops in session runtime
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(config): resolve sqlite_dir relative to CWD, not Paths.base_dir
resolve_path() resolves relative to Paths.base_dir (.deer-flow),
which double-nested the path to .deer-flow/.deer-flow/data/app.db.
Use Path.resolve() (CWD-relative) instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Feature/feishu receive file (#1608)
* feat(feishu): add channel file materialization hook for inbound messages
- Introduce Channel.receive_file(msg, thread_id) as a base method for file materialization; default is no-op.
- Implement FeishuChannel.receive_file to download files/images from Feishu messages, save to sandbox, and inject virtual paths into msg.text.
- Update ChannelManager to call receive_file for any channel if msg.files is present, enabling downstream model access to user-uploaded files.
- No impact on Slack/Telegram or other channels (they inherit the default no-op).
* style(backend): format code with ruff for lint compliance
- Auto-formatted packages/harness/deerflow/agents/factory.py and tests/test_create_deerflow_agent.py using `ruff format`
- Ensured both files conform to project linting standards
- Fixes CI lint check failures caused by code style issues
* fix(feishu): handle file write operation asynchronously to prevent blocking
* fix(feishu): rename GetMessageResourceRequest to _GetMessageResourceRequest and remove redundant code
* test(feishu): add tests for receive_file method and placeholder replacement
* fix(manager): remove unnecessary type casting for channel retrieval
* fix(feishu): update logging messages to reflect resource handling instead of image
* fix(feishu): sanitize filename by replacing invalid characters in file uploads
* fix(feishu): improve filename sanitization and reorder image key handling in message processing
* fix(feishu): add thread lock to prevent filename conflicts during file downloads
* fix(test): correct bad merge in test_feishu_parser.py
* chore: run ruff and apply formatting cleanup
fix(feishu): preserve rich-text attachment order and improve fallback filename handling
* fix(docker): restore gateway env vars and fix langgraph empty arg issue (#1915)
Two production docker-compose.yaml bugs prevent `make up` from working:
1. Gateway missing DEER_FLOW_CONFIG_PATH and DEER_FLOW_EXTENSIONS_CONFIG_PATH
environment overrides. Added in fb2d99f (#1836) but accidentally reverted
by ca2fb95 (#1847). Without them, gateway reads host paths from .env via
env_file, causing FileNotFoundError inside the container.
2. Langgraph command fails when LANGGRAPH_ALLOW_BLOCKING is unset (default).
Empty $${allow_blocking} inserts a bare space between flags, causing
' --no-reload' to be parsed as unexpected extra argument. Fix by building
args string first and conditionally appending --allow-blocking.
Co-authored-by: cooper <cooperfu@tencent.com>
* fix(frontend): resolve invalid HTML nesting and tabnabbing vulnerabilities (#1904)
* fix(frontend): resolve invalid HTML nesting and tabnabbing vulnerabilities
Fix `<button>` inside `<a>` invalid HTML in artifact components and add
missing `noopener,noreferrer` to `window.open` calls to prevent reverse
tabnabbing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(frontend): address Copilot review on tabnabbing and double-tab-open
Remove redundant parent onClick on web_fetch ChainOfThoughtStep to
prevent opening two tabs on link click, and explicitly null out
window.opener after window.open() for defensive tabnabbing hardening.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(persistence): organize entities into per-entity directories
Restructure the persistence layer from horizontal "models/ + repositories/"
split into vertical entity-aligned directories. Each entity (thread_meta,
run, feedback) now owns its ORM model, abstract interface (where applicable),
and concrete implementations under a single directory with an aggregating
__init__.py for one-line imports.
Layout:
persistence/thread_meta/{base,model,sql,memory}.py
persistence/run/{model,sql}.py
persistence/feedback/{model,sql}.py
models/__init__.py is kept as a facade so Alembic autogenerate continues to
discover all ORM tables via Base.metadata. RunEventRow remains under
models/run_event.py because its storage implementation lives in
runtime/events/store/db.py and has no matching repository directory.
The repositories/ directory is removed entirely. All call sites in
gateway/deps.py and tests are updated to import from the new entity
packages, e.g.:
from deerflow.persistence.thread_meta import ThreadMetaRepository
from deerflow.persistence.run import RunRepository
from deerflow.persistence.feedback import FeedbackRepository
Full test suite passes (1690 passed, 14 skipped).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(gateway): sync thread rename and delete through ThreadMetaStore
The POST /threads/{id}/state endpoint previously synced title changes
only to the LangGraph Store via _store_upsert. In sqlite mode the search
endpoint reads from the ThreadMetaRepository SQL table, so renames never
appeared in /threads/search until the next agent run completed (worker.py
syncs title from checkpoint to thread_meta in its finally block).
Likewise the DELETE /threads/{id} endpoint cleaned up the filesystem,
Store, and checkpointer but left the threads_meta row orphaned in sqlite,
so deleted threads kept appearing in /threads/search.
Fix both endpoints by routing through the ThreadMetaStore abstraction
which already has the correct sqlite/memory implementations wired up by
deps.py. The rename path now calls update_display_name() and the delete
path calls delete() — both work uniformly across backends.
Verified end-to-end with curl in gateway mode against sqlite backend.
Existing test suite (1690 passed) and focused router/repo tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(gateway): route all thread metadata access through ThreadMetaStore
Following the rename/delete bug fix in PR1, migrate the remaining direct
LangGraph Store reads/writes in the threads router and services to the
ThreadMetaStore abstraction so that the sqlite and memory backends behave
identically and the legacy dual-write paths can be removed.
Migrated endpoints (threads.py):
- create_thread: idempotency check + write now use thread_meta_repo.get/create
instead of dual-writing the LangGraph Store and the SQL row.
- get_thread: reads from thread_meta_repo.get; the checkpoint-only fallback
for legacy threads is preserved.
- patch_thread: replaced _store_get/_store_put with thread_meta_repo.update_metadata.
- delete_thread_data: dropped the legacy store.adelete; thread_meta_repo.delete
already covers it.
Removed dead code (services.py):
- _upsert_thread_in_store — redundant with the immediately following
thread_meta_repo.create() call.
- _sync_thread_title_after_run — worker.py's finally block already syncs
the title via thread_meta_repo.update_display_name() after each run.
Removed dead code (threads.py):
- _store_get / _store_put / _store_upsert helpers (no remaining callers).
- THREADS_NS constant.
- get_store import (router no longer touches the LangGraph Store directly).
New abstract method:
- ThreadMetaStore.update_metadata(thread_id, metadata) merges metadata into
the thread's metadata field. Implemented in both ThreadMetaRepository (SQL,
read-modify-write inside one session) and MemoryThreadMetaStore. Three new
unit tests cover merge / empty / nonexistent behaviour.
Net change: -134 lines. Full test suite: 1693 passed, 14 skipped.
Verified end-to-end with curl in gateway mode against sqlite backend
(create / patch / get / rename / search / delete).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: JilongSun <965640067@qq.com>
Co-authored-by: jie <49781832+stan-fu@users.noreply.github.com>
Co-authored-by: cooper <cooperfu@tencent.com>
Co-authored-by: yangzheli <43645580+yangzheli@users.noreply.github.com>
|
||
|
jie
|
2d068cc075
|
fix(docker): restore gateway env vars and fix langgraph empty arg issue (#1915)
Two production docker-compose.yaml bugs prevent `make up` from working: 1. Gateway missing DEER_FLOW_CONFIG_PATH and DEER_FLOW_EXTENSIONS_CONFIG_PATH environment overrides. Added in fb2d99f (#1836) but accidentally reverted by ca2fb95 (#1847). Without them, gateway reads host paths from .env via env_file, causing FileNotFoundError inside the container. 2. Langgraph command fails when LANGGRAPH_ALLOW_BLOCKING is unset (default). Empty $${allow_blocking} inserts a bare space between flags, causing ' --no-reload' to be parsed as unexpected extra argument. Fix by building args string first and conditionally appending --allow-blocking. Co-authored-by: cooper <cooperfu@tencent.com> |
||
|
Chincherry93
|
d7a3eff23e
|
fix(docker): command syntax for LANGGRAPH_ALLOW_BLOCKING (#1891) | ||
|
Willem Jiang
|
993fb0ff9d
|
fix: escape shell variables in production langgraph command (#1877) (#1880)
Escape shell variables to prevent Docker Compose from attempting substitution at parse time. Rename allow_blocking_flag to allow_blocking for consistency with dev version. Fixes the 'allow_blocking_flag not set' warning and enables --allow-blocking flag to work correctly. |
||
|
greatmengqi
|
ca2fb95ee6
|
feat: unified serve.sh with gateway mode support (#1847) | ||
|
DanielWalnut
|
2a150f5d4a
|
fix: unblock concurrent threads and workspace hydration (#1839)
* fix: unblock concurrent threads and workspace hydration * fix: restore async title generation * fix: address PR review feedback * style: format lead agent prompt |
||
|
Octopus
|
fb2d99fd86
|
fix: add missing DEER_FLOW_CONFIG_PATH and DEER_FLOW_EXTENSIONS_CONFIG_PATH env vars to gateway service (fixes #1829) (#1836)
The gateway service was missing these two environment variables that tell it where to find the config files inside the container. Without them, the gateway reads DEER_FLOW_CONFIG_PATH from the host's .env file (set to a host filesystem path), which is not accessible inside the container, causing FileNotFoundError on startup. The langgraph service already had these variables set correctly. |
||
|
1316151417
|
ac9a6ee6a2
|
fix(langgraph): correct config.yaml mount path in docker-compose (#1679)
Co-authored-by: zhoujie172 <zhoujie172@ke.com> |
||
|
ZHANG Ning
|
64e0f5329a
|
fix: remove LANGSMITH_TRACING override that ignores .env value (#1640)
The `environment` section in docker-compose.yaml set
`LANGSMITH_TRACING=${LANGSMITH_TRACING:-false}`, which always resolves
to `false` because Docker Compose evaluates `${}` substitutions from
the host shell environment, not from `env_file`.
Since `environment` entries take precedence over `env_file`, setting
`LANGSMITH_TRACING=true` in `.env` had no effect — tracing stayed
disabled despite following the documented instructions.
Remove the explicit `LANGSMITH_TRACING` from `environment` so the
value from `.env` (loaded via `env_file`) is used as intended.
|
||
|
Admire
|
9e3d484858
|
fix(frontend): route agent checks to gateway (#1572)
* fix(frontend): route agent checks to gateway * fix(frontend): proxy langgraph requests locally * fix(frontend): keep zh-CN text readable * fix(frontend): add exact local api rewrites * fix(frontend): support docker-safe internal rewrites * Update frontend/src/core/agents/api.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Admire
|
4bb3c101a8
|
chore(uv): speed up Docker builds with mirrors (#1600)
* docker mirror defaults * fix: make docker mirror defaults overridable * fix docker compose default pypi index * fix: restore upstream pypi defaults * docs: remove misleading env example mirrors --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
ZHANG Ning
|
0f1b023a2a
|
fix: add --n-jobs-per-worker 10 to langgraph dev command in Docker (#1623)
`langgraph dev` defaults `n_jobs_per_worker` to 1 when the flag is not explicitly passed (see langgraph_api/cli.py), even though the `N_JOBS_PER_WORKER` env-var default is 10. This causes the LangGraph server to run with a single background worker, meaning all conversation runs are processed serially. When one run is busy (e.g. summarization or long tool-calling chains), all other threads are blocked until it finishes. Add `--n-jobs-per-worker 10` to both production and dev Docker Compose files to match the intended default concurrency. |
||
|
DanielWalnut
|
c2dd8937ed
|
Fix IM channel backend URLs in Docker (#1497)
* Fix IM channel backend URLs in Docker * Address Copilot review comments |
||
|
yangzheli
|
a4e4bb21e3
|
docs: add LangSmith tracing configuration and documentation (#1414)
Add LangSmith tracing setup instructions across the project: - .env.example: add LANGSMITH_* env vars (commented out) - README.md + translations (zh/ja/fr/ru): add LangSmith Tracing section under Advanced with setup steps and env var reference - backend/README.md: add detailed LangSmith Tracing section with setup, env var table, how-it-works explanation, and Docker notes - docker-compose.yaml: update LANGCHAIN_TRACING_V2 to LANGSMITH_TRACING for naming consistency with the rest of the project Made-with: Cursor Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Kaushik Rajan
|
8ae023574e
|
fix: add build-arg support for proxies and mirrors in Docker builds (#1346)
* fix: add build-arg support for proxies and mirrors in Docker builds (#1260) Pin Debian images to bookworm, make UV source image configurable, and pass APT_MIRROR/NPM_REGISTRY/UV_IMAGE through docker-compose. * fix: ensure build args use consistent defaults across compose and Dockerfiles UV_IMAGE: ${UV_IMAGE:-} resolved to empty when unset, overriding the Dockerfile ARG default and breaking `FROM ${UV_IMAGE}`. Also configure COREPACK_NPM_REGISTRY before pnpm download and propagate NPM_REGISTRY into the prod stage. * fix: dearmor NodeSource GPG key to resolve signing error Pipe the downloaded key through gpg --dearmor so apt can verify the repository signature (fixes NO_PUBKEY 2F59B5F99B1BE0B4). --------- Co-authored-by: JeffJiang <for-eleven@hotmail.com> |
||
|
Purricane
|
835ba041f8
|
feat: add Claude Code OAuth and Codex CLI as LLM providers (#1166)
* feat: add Claude Code OAuth and Codex CLI providers Port of bytedance/deer-flow#1136 from @solanian's feat/cli-oauth-providers branch.\n\nCarries the feature forward on top of current main without the original CLA-blocked commit metadata, while preserving attribution in the commit message for review. * fix: harden CLI credential loading Align Codex auth loading with the current ~/.codex/auth.json shape, make Docker credential mounts directory-based to avoid broken file binds on hosts without exported credential files, and add focused loader tests. * refactor: tighten codex auth typing Replace the temporary Any return type in CodexChatModel._load_codex_auth with the concrete CodexCliCredential type after the credential loader was stabilized. * fix: load Claude Code OAuth from Keychain Match Claude Code's macOS storage strategy more closely by checking the Keychain-backed credentials store before falling back to ~/.claude/.credentials.json. Keep explicit file overrides and add focused tests for the Keychain path. * fix: require explicit Claude OAuth handoff * style: format thread hooks reasoning request * docs: document CLI-backed auth providers * fix: address provider review feedback * fix: harden provider edge cases * Fix deferred tools, Codex message normalization, and local sandbox paths * chore: narrow PR scope to OAuth providers * chore: remove unrelated frontend changes * chore: reapply OAuth branch frontend scope cleanup * fix: preserve upload guards with reasoning effort wiring --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
DanielWalnut
|
76803b826f
|
refactor: split backend into harness (deerflow.*) and app (app.*) (#1131)
* refactor: extract shared utils to break harness→app cross-layer imports Move _validate_skill_frontmatter to src/skills/validation.py and CONVERTIBLE_EXTENSIONS + convert_file_to_markdown to src/utils/file_conversion.py. This eliminates the two reverse dependencies from client.py (harness layer) into gateway/routers/ (app layer), preparing for the harness/app package split. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: split backend/src into harness (deerflow.*) and app (app.*) Physically split the monolithic backend/src/ package into two layers: - **Harness** (`packages/harness/deerflow/`): publishable agent framework package with import prefix `deerflow.*`. Contains agents, sandbox, tools, models, MCP, skills, config, and all core infrastructure. - **App** (`app/`): unpublished application code with import prefix `app.*`. Contains gateway (FastAPI REST API) and channels (IM integrations). Key changes: - Move 13 harness modules to packages/harness/deerflow/ via git mv - Move gateway + channels to app/ via git mv - Rename all imports: src.* → deerflow.* (harness) / app.* (app layer) - Set up uv workspace with deerflow-harness as workspace member - Update langgraph.json, config.example.yaml, all scripts, Docker files - Add build-system (hatchling) to harness pyproject.toml - Add PYTHONPATH=. to gateway startup commands for app.* resolution - Update ruff.toml with known-first-party for import sorting - Update all documentation to reflect new directory structure Boundary rule enforced: harness code never imports from app. All 429 tests pass. Lint clean. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: add harness→app boundary check test and update docs Add test_harness_boundary.py that scans all Python files in packages/harness/deerflow/ and fails if any `from app.*` or `import app.*` statement is found. This enforces the architectural rule that the harness layer never depends on the app layer. Update CLAUDE.md to document the harness/app split architecture, import conventions, and the boundary enforcement test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add config versioning with auto-upgrade on startup When config.example.yaml schema changes, developers' local config.yaml files can silently become outdated. This adds a config_version field and auto-upgrade mechanism so breaking changes (like src.* → deerflow.* renames) are applied automatically before services start. - Add config_version: 1 to config.example.yaml - Add startup version check warning in AppConfig.from_file() - Add scripts/config-upgrade.sh with migration registry for value replacements - Add `make config-upgrade` target - Auto-run config-upgrade in serve.sh and start-daemon.sh before starting services - Add config error hints in service failure messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix comments * fix: update src.* import in test_sandbox_tools_security to deerflow.* Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: handle empty config and search parent dirs for config.example.yaml Address Copilot review comments on PR #1131: - Guard against yaml.safe_load() returning None for empty config files - Search parent directories for config.example.yaml instead of only looking next to config.yaml, fixing detection in common setups Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: correct skills root path depth and config_version type coercion - loader.py: fix get_skills_root_path() to use 5 parent levels (was 3) after harness split, file lives at packages/harness/deerflow/skills/ so parent×3 resolved to backend/packages/harness/ instead of backend/ - app_config.py: coerce config_version to int() before comparison in _check_config_version() to prevent TypeError when YAML stores value as string (e.g. config_version: "1") - tests: add regression tests for both fixes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: update test imports from src.* to deerflow.*/app.* after harness refactor Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
JeffJiang
|
08ea9d3038
|
feat: enhance Docker support with production setup and deployment script (#1086)
* feat: add `make start` command for local previewing * Update Makefile Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: update help text for `make dev` and `make start` commands * feat: enhance Docker support with production setup and deployment script * feat: add production commands to Makefile * feat: remove PostgreSQL and Redis services from Docker Compose and update deploy script * fix: address Copilot review suggestions from Docker production PR #1086 (#10) * Initial plan * fix: address all review suggestions from PR #1086 Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com> * Update docker/docker-compose.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * feat: remove deprecated Dockerfile.langgraph to clean up repository --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: foreleven <4785594+foreleven@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |