* fix(multi-instance): publish skill_manage cache resets and tolerate a concurrent namespace create
Two leftovers from the multi-instance work (#7097).
The agent's skill_manage tool refreshed only its own process's skills
prompt cache after a create, edit, patch, delete, write_file or
remove_file, so a Gateway replica sharing the home volume kept rendering
that user's previous skill list. Every mutation now runs one drained tail,
as the skill routes do: the write and its history record, the local
refresh, then publish_skills_cache_reset(user_id=...) off the event loop.
A run cancelled while the write is in flight still refreshes and
publishes before the cancellation propagates. A marker that cannot be
written is logged inside the tail and raised as SkillCacheResetPublishError
with a message saying the change was applied, so the model does not retry
a write that succeeded.
Two provisioner replicas starting together against a missing
K8S_NAMESPACE both read 404 and both create; the loser's 409 escaped the
lifespan and crashed it. A 409 on create now counts as the namespace
existing, as the Pod and Service creates already do.
Also drops a stale duplicate of the multi-instance "declaration also
drives" note in CONFIGURATION.md.
* docs(changelog): reference #7099 for the multi-instance leftovers
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(helm): let the sandbox provisioner run with several replicas
The chart hard-coded replicas: 1 into the provisioner Deployment, gave it
no rollout strategy and no PodDisruptionBudget, and values.yaml had no
provisioner.replicas key, so a multi-replica gateway deployment still sent
every sandbox create/discover/destroy call through one Pod: while that Pod
restarted or its node drained no sandbox could be created anywhere.
- values.yaml provisioner gains replicas (default 1, unchanged), the
gateway's surge-then-drain RollingUpdate strategy and
podDisruptionBudget {enabled: true, minAvailable: 1}.
- provisioner-deployment.yaml renders replicas and strategy from values; at
1 replica the result equals the Kubernetes default strategy.
- New provisioner-pdb.yaml, rendered only while the provisioner is enabled,
the budget is enabled and replicas > 1, selecting on the Deployment's
labels and named <fullname>-provisioner so it never collides with the
gateway budget. No multiInstance override: nothing else in the chart
renders differently per provisioner replica count.
- docker/provisioner/app.py is unchanged. It keeps no state between
requests: the labelled sandbox Pods and Services are the only registry,
create tolerates the 409 a concurrent creator produces, destroy tolerates
404, NodePorts come from the API server and there is no background loop.
The chart README and docker/provisioner/README.md record this, and
tests/test_helm_provisioner_replicas.py pins it at source level next to
the values, template and helm-render assertions.
* docs(changelog): link the provisioner replicas entry to #6543
* fix(helm): preserve percentage PodDisruptionBudget values instead of casting them to 0
Review of #6543: both PDB templates rendered minAvailable through
`| default 1 | int`, and sprig's int turns a percentage string such as
"50%" -- a value Kubernetes accepts -- into 0, a budget that permits every
Pod to be evicted. The gateway budget had the same pre-existing cast.
- New shared helper deer-flow.pdbMinAvailable in _helpers.tpl: a YAML or
--set number renders as an integer, a digit string from --set-string
becomes that integer, a percentage is preserved quoted, and 0, "0%",
more than 100%, a non-integer or anything else fails the render with a
message naming the values key and the enabled switch to use instead.
Unset still falls back to 1.
- gateway-pdb.yaml and provisioner-pdb.yaml render minAvailable through it.
- values.yaml comments, the chart README multi-replica note and the
CHANGELOG entry say integer-or-percentage and that zero is rejected.
- tests/test_helm_provisioner_replicas.py pins the helper and the two
templates at text level and adds render cases for both budgets: "50%"
with three replicas renders "50%", 2 via --set and --set-string renders
2, and abc / 0 / 0% / 150% fail naming the key.
* fix(helm): normalize json.Number before the minAvailable string branches
Review of #6543: Helm 3.18.0 loads YAML numbers as json.Number, whose
kind is "string" but which regexMatch and the other string-typed sprig
functions reject at argument validation ("expected string; got
json.Number"). deer-flow.pdbMinAvailable gated its string branch on
kindIs "string", so the unoverridden chart default minAvailable: 1 made
every gateway.replicas=2 / provisioner.replicas=2 render fail on that
release, inside the README's documented 3.8+ support window.
- The helper keeps the explicit float64/int64/int branch and now converts
everything else with toString into $text before the digit, percentage
and fail branches; regexMatch, atoi, trimSuffix and quote only ever see
that plain string. A json.Number 1 renders 1, a json.Number 1.5 still
fails (the digit regex rejects it) and the hint still prints the raw
value; bools and other kinds still fail. The helper comment records the
Helm 3.18.0 reason.
- tests/test_helm_provisioner_replicas.py pins that the string branch
runs on the toString-normalized variable, that no string-typed function
is applied to $raw and that kindIs "string" is gone, and adds an
explicit render case for both budgets with only the replica count set
(the unoverridden default must render minAvailable: 1), marked as the
Helm 3.18.0 json.Number regression guard.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(migration): read thread owners from the configured database
migrate_user_isolation.py looked for thread owners in
{base_dir}/deer-flow.db, a file DeerFlow never creates. The database is
{sqlite_dir}/deerflow.db or PostgreSQL, so the owner map was always empty
and every legacy thread was moved to users/default/.
Read threads_meta through the configured database's sync SQLAlchemy URL
(SQLite or PostgreSQL, honoring postgres_schema). When legacy threads
exist but owners cannot be read (no database file, a memory backend, a
missing driver, or a failed query), stop before migrating anything;
--allow-missing-thread-owners keeps the old assign-to-default behavior
for installs that never recorded owners.
* docs(changelog): reference #6450
* fix(migration): document recovery and report unusable database URLs
Installs that already ran the old script have every legacy thread under
users/default/threads/, and re-running finds no threads/ to migrate. The
provisioner README and AUTH_DESIGN now give the recovery: move the owned
threads back to threads/ and re-run; an existing owner directory is never
overwritten and the old copy goes to migration-conflicts/. A test pins
that re-run behavior.
create_engine errors other than ImportError (e.g. ArgumentError from an
empty postgres_url) now stop with the same "Nothing was migrated." message
instead of a traceback.
The memory AGENTS.md note said threads go to --user-id; they go to their
threads_meta owner.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): project enabled skills into sandbox views
* fix(skills): keep projection mutations consistent
* fix(skills): fail closed on projection errors
* fix(skills): isolate per-scope failures during boot projection rebuild
rebuild_all_skill_projections() propagated any exception from the public
rebuild or from a single user's rebuild straight out of the gateway
lifespan startup, uncaught. A single broken user directory (bad
permissions, corrupted _skill_states.json, unreadable content) would
therefore abort gateway boot for every user, not just that one -
_rebuild_*_locked already fails closed internally (clears the view and
re-raises), so the boot loop only needed to stop treating that re-raise
as fatal.
Each scope's rebuild now fails closed independently and boot continues;
a scope left empty by a boot failure self-heals on the next sandbox
acquire via ensure_skill_projections().
Also patches deerflow.skills.projection.rebuild_all_skill_projections in
the memory-flush lifespan test fixture, matching the two sibling
fixtures in the same file — this call is now on the lifespan startup
path and the fixture's minimal SimpleNamespace config predates it.
* test(skills): update authz test for the projection-aware public toggle
_persist_shared_skill_state (introduced earlier in this branch) reads
the shared extensions_config.json fresh from disk under the projection
lock instead of through the cached get_extensions_config() singleton -
that's the whole point of the fix (stale worker caches must not clobber
another worker's concurrent update). The name no longer exists on the
skills router module, so the test's monkeypatch of it started raising
AttributeError instead of exercising the endpoint.
The mock storage in this test isn't a real LocalSkillStorage instance,
so _persist_shared_skill_state's projection-mutation branch is already
skipped (nullcontext) and it falls back to a fresh ExtensionsConfig()
for the nonexistent tmp config_path - no replacement monkeypatch needed.
* fix(sandbox): make skill projection ensure best-effort in acquire
acquire() called _ensure_skills_projection() directly, outside any
try/except, in both LocalSandboxProvider and AioSandboxProvider. Every
other skill-mount setup path in these providers has always caught
exceptions and logged a warning rather than failing sandbox acquire
outright (e.g. when config.yaml can't be resolved) - these two new call
sites broke that contract, so any projection failure (including simply
not having a config.yaml, as in CI's test environment) now failed
acquire() itself instead of just leaving skill mounts off.
_ensure_skills_projection now catches its own exceptions and returns
None; both providers' callers already tolerate that (a None projection
skips the skill-specific mounts, matching the existing degrade path)
after making _append_public_skill_mapping and the custom/legacy mount
block in LocalSandboxProvider explicitly None-safe.
Caught by running the full suite with config.yaml removed, matching
CI's environment - not caught locally because a real config.yaml was
present, masking the failure.
* fix(sandbox): make E2B skill projection mounts best-effort
_skill_projection_mounts called ensure_skill_projections with no guard,
unlike Local/AIO's _ensure_skills_projection. A raise propagated out of
_apply_mounts before the configured-mounts loop ran, so a skills
projection failure dropped the operator's own configured mounts too -
only caught by create()'s outer warning, with nothing applied at all.
Swallow here and return an empty mount list on failure, matching the
Local/AIO pattern: still fail-closed for skills, but no longer widens
the blast radius to unrelated configured mounts.
Review feedback from PR #4178.
* docs(skills): document projection trade-offs flagged in review
- _update_tree_digest: note the metadata-only (not content) hashing
trade-off and why runtime writes through this codebase are still
covered regardless (rebuild-under-lock + rename always changes inode).
- LocalSandboxProvider.acquire: note the acquire-time self-heal cost
(cheap on a fresh manifest, ~400ms rebuild under lock on stale/drift).
- skill_projection_mutation: drop the no-op except-Exception-then-raise;
a raise from the mutation already propagates past the yield with the
view left cleared, no explicit re-raise needed.
- provisioner README: spell out that hostPath skills volumes require
the gateway and K8s node to share DEER_FLOW_HOST_BASE_DIR (single-node
or shared storage), and that the custom/legacy volumes' hostPath type
Directory (not DirectoryOrCreate) makes a violation of that assumption
a visible Pod-creation failure instead of a silent empty mount.
Review feedback from PR #4178.
* fix(skills): lazily repair user projections
* fix(skills): close projection review gaps
* fix(skills): refresh user projection enable state
* fix(skills): close projection review follow-ups
* fix(skills): preserve state across projection writes
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(lark): sidecar credential broker for sandbox lark-cli (Pattern B)
Removes the plaintext Lark credential mounts (appSecret + OAuth tokens)
from the sandbox container. A long-running broker sidecar owns lark-cli
and the per-user config/data dirs and serves the command surface over
Pod loopback; the sandbox gets only a forwarding shim on PATH, so the
raw credential files never exist in the sandbox filesystem.
- lark_broker.py: stdlib-only loopback broker (argv passthrough with
shell=False, server-injected credential env, bounded I/O) + shim
script constant + install-shim mode.
- docker/lark-cli-broker: init(install-shim) + serve image.
- provisioner: LARK_CLI_BROKER_IMAGE + provision_lark_cli_broker →
shim init container + lark-cli-broker sidecar (config/data mounted
sidecar-only); credentials dropped from the sandbox container;
/api/capabilities reports lark_cli_broker_image. Broker supersedes
the Pattern A init-container binary when both are configured.
- gateway: lark_cli_env_overlay(broker=True) omits config/data env;
sandbox_lark_broker_active() TTL-cached mode resolver; broker added
to sandbox_runtime_mode / readiness and the settings UI.
Opt-in and off by default (empty LARK_CLI_BROKER_IMAGE ⇒ no change).
Closes#4338
* fix(lark): address Pattern B broker review findings (#4501)
Follow-up to the sidecar credential broker addressing the PR #4501 review:
- shim: split the on-PATH lark-cli into a /bin/sh launcher + Python shim body
so broker mode fails loudly (exit 127, actionable message) instead of ENOEXEC
when the sandbox image ships no python3; interpreter pinnable via
DEERFLOW_LARK_BROKER_PYTHON. Launcher bakes in the shim's absolute path since
$0 is the bare command name when run off PATH.
- broker: drop the dead cwd payload field (broker can't see the sandbox FS) and
document the command-surface-only / no-file-IO limitation.
- broker: return a structured 500 JSON on unexpected exec errors so the shim
gets a meaningful message, not an opaque transport failure; set a handler
socket timeout to bound slow/stuck connections.
- broker: add an opt-in DEERFLOW_LARK_BROKER_DENY_SUBCOMMANDS denylist that
refuses secret-dumping subcommands before spawning the binary, forwarded from
the provisioner sidecar.
- gateway: tighten the per-bash-call broker probe timeout (1.5s) and cache
negatives longer (300s) so non-broker remote-provisioner users don't pay a
latency hit; guard the mode cache with a lock; drop the dead
_probe_provisioner_lark_cli_init_image wrapper.
- docs: remove the broken design-doc link from the broker README.
Adds tests for launcher python resolution, cwd omission, denylist enforcement,
500-on-error, hot-path probe timeout + negative caching, and provisioner
denylist-env wiring.
* feat: add lark cli integration
* fix: polish lark integration actions
* feat: support lark incremental permissions
* fix: detect lark authorization completion
* fix: harden lark integration install
* feat: expand lark auth scopes and reuse host auth in sandbox
Default lark auth to least-privilege (recommend=false, base sign-in only)
and expose the full set of lark-cli --domain business domains as native
--domain grants instead of a 4-domain read-only mapping. Resolve the
skill pack from the latest larksuite/cli GitHub release at install time
with content-hash integrity, and surface version/runtime drift in status.
Share the per-user lark-cli config/data profile between the Gateway
Settings auth flow and agent conversations by mounting the integration
dirs into the AIO sandbox and injecting the matching env for lark-cli
commands, with an allowlisted extra_mounts path in the provisioner/K8s
backend and traversal guards on integration paths.
* style: fix lint issues from ruff and prettier
Sort imports in the provisioner PVC test and re-wrap two long i18n
description strings to satisfy backend ruff and frontend prettier CI.
* fix(lark): address managed integration review feedback
* fix(frontend): stabilize integrations settings e2e
* test(sandbox): isolate remote backend legacy visibility check
* test: fix backend unit failures after merge
* Harden Lark integration review fixes
* Format Lark integration E2E test
* fix(lark): harden sandbox credential exposure and status disclosure
Address willem_bd's security review on PR #3971:
- Mount the per-user lark-cli config dir (long-lived appSecret) read-only
into the AIO sandbox; only the refreshable-token data dir stays writable.
- Redact host filesystem paths (install_path, cli.path) from
GET /lark/status and the config/auth complete responses for non-admin
callers, fail-closed on any auth error.
- Document the npm postinstall trade-off (--ignore-scripts is not viable
because @larksuite/cli fetches its platform binary in postinstall).
- Document the sandbox credential trust boundary in AGENTS.md and README,
pointing at the sidecar-broker follow-up (#4338).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(provisioner): keep k8s calls off event loop
* test(provisioner): scan blocking IO by module name
* test(provisioner): exercise create path so BlockBuster has real work
* feat(provisioner): add optional PVC support for sandbox volumes (#1978)
Add SKILLS_PVC_NAME and USERDATA_PVC_NAME env vars to allow sandbox
Pods to use PersistentVolumeClaims instead of hostPath volumes. This
prevents data loss in production when pods are rescheduled across nodes.
When USERDATA_PVC_NAME is set, a subPath of threads/{thread_id}/user-data
is used so a single PVC can serve multiple threads. Falls back to hostPath
when the new env vars are not set, preserving backward compatibility.
* add unit test for provisioner pvc volumes
* refactor: extract shared provisioner_module fixture to conftest.py
Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/e7ccf708-c6ba-40e4-844a-b526bdb249dd
Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JeffJiang <for-eleven@hotmail.com>
* Adds Kubernetes sandbox provisioner support
* Improves Docker dev setup by standardizing host paths
Replaces hardcoded host paths with a configurable root directory,
making the development environment more portable and easier to use
across different machines. Automatically sets the root path if not
already defined, reducing manual setup steps.
