* feat(ci): split backend unit tests into parallel CI shards
Split the single offline backend `make test` job into four GitHub Actions
matrix shards (SPLITS=4, GROUP=1..4) via pytest-split, so the ~12k-test suite
runs in parallel instead of in one 15-minute job. Each shard runs on its own
runner with its own Postgres/Redis services; fail-fast: false lets a failing
shard report its owned tests without cancelling its peers.
`make test` stays the canonical full-suite entry point; CI now calls the new
`make test-shard SPLITS=4 GROUP=N`. tests/blocking_io remains owned solely by
the dedicated blocking-I/O workflow (excluded via --ignore), extending #5105.
Fixes#5088
* test(ci): make backend test shards duration-aware and pin the contract
Make `make test-shard` an explicit least_duration split that READS
backend/.test_durations (read-only for shards, so concurrent CI jobs never
race writes on it), and add `make test-shard-durations` to regenerate that
file from the full offline suite. Update the CI unit-test workflow contract to
call `make test-shard SPLITS=4 GROUP=<n>` and assert the shard command carries
--splits 4, --group 2, -m "not live", --ignore=tests/blocking_io and
--splitting-algorithm least_duration. Verified on the real 13,140-test normal
suite that the four shards are pairwise disjoint and their union equals the
unsplit suite.
Refs #5088
* test(ci): fail fast when the duration baseline is missing
`make test-shard` now requires backend/.test_durations and exits with a clear
error instead of letting pytest-split silently degrade to an even (count-based)
split. Harden the CI contract test to pin `--durations-path=.test_durations` and
to assert the repo ships the committed duration baseline.
Refs #5088
* docs: trim backend/AGENTS.md within guidance budget
* test(ci): add backend test duration baseline
Add the duration baseline generated by a full offline backend run on a
GitHub-hosted ubuntu-latest runner (the same runner type the shards use), so
`make test-shard` balances the four matrix shards by real wall-clock cost.
Refs #5088
* test(ci): make the duration writer honor DURATIONS_FILE
`test-shard-durations` now writes `--durations-path=$(DURATIONS_FILE)` instead of a
hard-coded .test_durations, so the reader and writer stay consistent when the
path is overridden.
Refs #5088
* test(ci): address sharding review feedback
* test: isolate subagent execution capacity state
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): harden local Docker sandbox containers and port binding
Root causes (security audit SBX-1/SBX-2) in the local container backend:
- _resolve_docker_bind_host published sandbox ports on 0.0.0.0 whenever
DEER_FLOW_SANDBOX_HOST was non-loopback (docker-compose defaults to
host.docker.internal), exposing the unauthenticated /v1/shell/* exec
API on every host interface.
- _start_container ran every sandbox with seccomp=unconfined and no
capability, privilege-escalation, or resource limits, so untrusted
model-authored code could exhaust the host, escalate privileges, and
reach internal networks / cloud metadata endpoints directly.
Hardening changes and defaults:
- Port binding: non-loopback sandbox hosts now bind the Docker default
bridge gateway instead of 0.0.0.0, discovered dynamically via
`docker network inspect bridge` with a static 172.17.0.1 fallback.
host.docker.internal resolves to that gateway through host-gateway,
so DooD gateways and the Docker host still reach the sandbox while
external interfaces no longer see the port.
DEER_FLOW_SANDBOX_BIND_HOST=0.0.0.0 restores the legacy broad bind.
- seccomp=unconfined is no longer unconditional: sandboxes run with
Docker's default seccomp profile; opt back in with
DEER_FLOW_SANDBOX_SECCOMP_UNCONFINED=1, only when the sandbox image
is verified to require syscalls the default profile blocks.
- Add --cap-drop=ALL and --security-opt no-new-privileges (Docker only;
the Apple Container CLI does not support these flags).
- Bounded resources with env overrides: --memory 2g
(DEER_FLOW_SANDBOX_MEMORY), --cpus 2 (DEER_FLOW_SANDBOX_CPUS),
--pids-limit 512 (DEER_FLOW_SANDBOX_PIDS_LIMIT); each also accepts
"0"/"none" to disable the limit.
- No --user is forced by default (the default AIO sandbox image's user
is upstream-controlled and unverified), but
DEER_FLOW_SANDBOX_CONTAINER_USER passes one through for deployments
that know their image.
- DEER_FLOW_SANDBOX_NETWORK passes --network so sandboxes can be
attached to a dedicated egress-controlled network; default networking
is unchanged.
backend/docs/CONFIGURATION.md documents the new bind behavior and every
override; tests cover each default and escape hatch.
* fix(sandbox): follow host-gateway mapping for binds; keep image-required seccomp default
Review follow-ups on the hardening change:
- Bind: resolve the sandbox host itself and bind that address, instead of
assuming the default bridge IPv4. host.docker.internal follows the
daemon host-gateway-ip mapping (customizable, possibly IPv6), so the
resolved address is exactly where the gateway connects — the published
port and advertised URL always match. IPv6 is bracketed for docker -p,
zone ids stripped, wildcard resolutions ignored; unresolved hosts fall
back to the bridge gateway with a warning pointing at
DEER_FLOW_SANDBOX_BIND_HOST.
- seccomp: the shipped AIO image needs seccomp=unconfined for its
Chromium browser (upstream quick-start always passes it; the upstream
FAQ documents the browser failing under Docker default profile), so
that option returns as the default. Tightening stays possible via
DEER_FLOW_SANDBOX_SECCOMP_PROFILE=<path to a restricted,
Chromium-compatible profile> or DEER_FLOW_SANDBOX_SECCOMP_UNCONFINED=0
for images verified to work with Docker's default profile.
- cap-drop/no-new-privileges and the resource limits are unchanged.
- Tests updated for both behaviors; 37 pass.
* fix(sandbox): bracket bare IPv6 bind overrides; state seccomp default accurately
DEER_FLOW_SANDBOX_BIND_HOST was returned verbatim, so a bare IPv6 literal
like fd00::1 produced an invalid publish spec (fd00::1:port:8080); Docker
requires the bracketed form. Normalize raw and already-bracketed IPv6
literals (IPv4/hostnames untouched), with resolver-level and argv-level
tests covering the explicit IPv6 override.
The CONFIGURATION.md overview claimed Docker's default seccomp profile
stays active, contradicting the seccomp=unconfined default the table (and
the code) actually ship for the Chromium-based image; spell out the relaxed
default and where to change it.
* style(sandbox): apply ruff format to local_backend
* fix(sandbox): reject host networking, force builtin seccomp opt-out, resolve hostname binds
Review follow-up on #4986 (willem-bd):
- P1: DEER_FLOW_SANDBOX_NETWORK=host (and container:<name>) now raise a
RuntimeError at start instead of silently voiding the hardened port
bind — Docker discards -p/--publish in host mode and shares the
network namespace for container:<name>, which would re-expose the
unauthenticated exec API on the host's interfaces. Two regression
tests cover both rejections.
- P2: the seccomp opt-out now passes seccomp=builtin explicitly instead
of omitting the option, so a daemon configured with an unconfined or
custom default cannot weaken the documented opt-out; the test asserts
the flag.
- P2: hostname values in DEER_FLOW_SANDBOX_BIND_HOST resolve to an
address before use (Docker publish specs require an IP literal as the
host part, so host.docker.internal previously produced an invalid
spec that prevented every sandbox from starting); unresolvable names
raise a clear configuration error. Tests cover resolution and
rejection; CONFIGURATION.md updated for all three behaviors.
43/43 pass in tests/test_aio_sandbox_local_backend.py; ruff check +
format clean.
* fix(sandbox): reject DEER_FLOW_SANDBOX_NETWORK=none (loopback-only, breaks published API port)
* fix(sandbox): validate the effective Docker network target; normalize IPv6 sandbox hosts once
name=host / name=none dodge raw-string checks but attach like the bare
words; strip name= prefixes and validate the effective target (network IDs
keep passing). Bracketed IPv6 sandbox hosts now resolve for the bind and
bare IPv6 hosts produce bracketed URL authorities — both input forms give
identical bind and URL addresses.
* fix(sandbox): parse the full Docker network long syntax before validating
Docker accepts comma-separated key=value fields in any order (name=, gw-priority=,
alias=, ...); a name=host field hides the host network behind surrounding fields.
Parse the CSV and validate the parsed name= target (last occurrence wins, fields
lowercased, mirroring opts/network.go); no-name values fall through like Docker's
own rejection.
* fix(sandbox): keep CHOWN/SETUID/SETGID through cap-drop=ALL for the default image
The shipped image's entrypoint starts as root, creates the gem user,
chowns /opt/jupyter and drops to that user via su; without those three
capabilities the set -e script dies before the readiness endpoint exists.
no-new-privileges stays (it blocks gaining privileges via exec, not using
the added caps). Adds a docker-gated real-image startup smoke test.
* fix(sandbox): let pre-initialized non-root images drop the startup capabilities
The CHOWN/SETUID/SETGID re-add only exists for the shipped image's root
entrypoint handoff. A custom image that never runs as root gets an explicit
opt-out (DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS=0) so those capabilities are
not left available to sandboxed code (chown on bind mounts, UID/GID
impersonation).
* test(sandbox): gate the real-image smoke test behind the live marker
The default offline suite (make test = -m 'not live') must not depend on a
third-party registry: mark the smoke test live, probe the daemon inside the
test body (never at collection time), and allow pinning the image reference
via DEER_FLOW_SANDBOX_SMOKE_IMAGE for a dedicated integration job.
* test/docs: isolate DEER_FLOW_SANDBOX_IMAGE_STARTUP_CAPS in tests; add table row; split custom-image guidance
_clear_hardening_env now clears the new knob so a developer shell or .env
preset cannot flip the default-path tests. CONFIGURATION.md gains the table
row, and the custom-image guidance becomes its own paragraph with the
no-new-privileges scope stated correctly (it does not mitigate the retained
CAP_SETUID/SETGID risk).
* test(sandbox): make the live smoke test diagnosable
300s readiness budget (cold pull + cold start must not be conflated with
broken capabilities) and dump the container's last 40 log lines on failure
so the next live run tells us whether the capability set is incomplete
(chown/useradd/su errors) or the services are merely slow.
* test(ci): align the smoke test with the 60s provider deadline; add a dedicated live smoke workflow
Single-source the readiness deadline as SANDBOX_LOCAL_PROVIDER_READY_TIMEOUT
(used by both provider paths and the smoke test) so the validation cannot
drift from the production contract again. New sandbox-image-smoke.yml runs
the live test on a dedicated job, with the image reference pinnable via the
SANDBOX_SMOKE_IMAGE repository variable (digest resolved and recorded in the
job summary when falling back to :latest).
* test(sandbox): pull the failing program's own logs on smoke failure
supervisord only surfaces exit codes in docker logs; nginx's stderr lands in
files inside the container. Dump supervisor program logs, nginx -t, and the
nginx error log on failure so the next run names the exact broken line.
* ci(sandbox): export an immutable repo@digest reference for the smoke run
docker pull once on the runner platform, resolve RepoDigests[0], and pass
that immutable reference to the test via GITHUB_ENV — the recorded and
executed images can no longer diverge when the tag moves, and platform
selection is left to the daemon instead of jq over the manifest index.
* fix(sandbox): add DAC_OVERRIDE — the root nginx master writes gem-owned logs
The image's root nginx master opens /var/log/nginx/{access,error}.log,
which belong to the gem user, for the container's lifetime; without
CAP_DAC_OVERRIDE it dies with 'open() failed (13: Permission denied)' on
every start (FATAL under supervisord) and readiness never arrives. Four
capabilities now: CHOWN/SETUID/SETGID for the entrypoint handoff plus this
runtime log-write need.
* feat(extensions): add gateway services and routers
* feat(extensions): add standalone reference extension
* fix(extensions): harden contributed gateway routes
* docs(extensions): document gateway contribution points
* feat(extensions): add operator CLI for packaged extension management
Add `deerflow extensions install/list/enable/disable/remove` plus the root
`make extension-*` wrappers, backed by an `ExtensionManager` that owns one
transaction over backend/pyproject.toml, backend/uv.lock, the managed source
snapshot, the uv environment, and the `plugins:` block in config.yaml.
Install accepts a package requirement, a public HTTPS Git URL, or a local
directory. Local directories are copied to backend/extensions/sources/ as
deployable snapshots rather than editable installs, and the root .dockerignore
re-includes that tree so snapshots reach the backend builder. Remote sources are
HTTPS-only; SSH Git, file:// and local wheels are rejected because the stock
Docker builder cannot reproduce them.
Because environment configuration can still resolve a plain package name to a
local wheel (a UV_FIND_LINKS wheelhouse, say), every uv add/remove is followed
by an audit of the new lock: any local reference the stock image build cannot
reproduce rolls back the whole transaction. A config carrying duplicate
top-level `plugins:` keys is rejected outright rather than managed against one
block while the Gateway reads another.
Dependency synchronization now has one lock authority. The `extensions`
dependency group joins [tool.uv].default-groups, every startup path syncs the
same lock with --locked and launches with --no-sync, and the Docker images move
to uv 0.11.1 for the --no-workspace boundary the manager needs.
Loader gains `enabled`, `name` and `package` fields so a disabled extension is
skipped before resolution and import.
Co-authored-by: Codex <codex@openai.com>
* fix(extensions): stop the managed plugins rewrite from destroying config
Two data-safety defects in the managed `plugins:` block writer.
The "next top-level key" boundary was a regex matching only
`[A-Za-z_][A-Za-z0-9_-]*` or a quoted key. `AppConfig` is `extra="allow"`, so a
config may legally carry any top-level key, and a key the pattern cannot
recognize did not fail loudly — it read as "no next section", and the rewrite
replaced that neighbour and its entire subtree with the managed block. `my.key`,
`2fa`, `$schema`, `my key` and non-ASCII keys were all silently deleted by a
plain `extension-enable`/`disable`. Both boundaries now come from the YAML
parser's node marks, so key shape is irrelevant.
The file-final branch never consulted the trailing-comment scan the has-next-key
branch used, so any comment below the block was dropped. Since the manager
appends `plugins:` at end of file, that is the steady-state shape for most
installs: an operator note below the block was destroyed on the next toggle.
Separately, every managed install wrote `required: true` while the loader
defaults to false. That turned any later load failure — broken wheel, missing
native library, deleted snapshot — into a Gateway startup abort recoverable only
with shell access. New records are now written `required: false`, with an
explicit `install --required` opt-in; adopting an existing hand-written record
still preserves the operator's own choice.
* fix(extensions): harden the manager transaction and correct its docs
Follow-up hardening on the extension package manager.
Security posture, which the docs already claimed:
- Scrub `UV_PYTHON`, `UV_INSECURE_HOST`, `UV_CONSTRAINT` and
`UV_NO_BUILD_ISOLATION` from the controlled uv environment. `UV_PYTHON` swaps
the interpreter that the entry-point probe then imports and calls, and every
later `uv run --no-sync` startup uses; `UV_INSECURE_HOST` removes the TLS
validation the HTTPS-only source rule depends on. Neither is an index, proxy,
cache or credential-provider setting, so neither was covered by the carve-out.
- Recognize run-together and all-caps secret query parameters (`accesstoken`,
`ACCESSTOKEN`, `key`, `pw`, `sas`, `code`). The camel-case splitter only fires
on case transitions, so only the separated spellings were caught. Short
generic words stay boundary-anchored, so `?keyword=` remains installable.
- Validate the config before running any uv command. `uv add`/`uv sync` execute
the package's build backend, so a config the manager could never write to must
fail before that code runs rather than afterwards through rollback.
Transaction integrity:
- Run the second dependency-file restore from a `finally`. The recovery sync
runs without `--locked` when the checkout had no lock, so uv writes one while
resolving; if that sync then failed, the restore was skipped and the operator
kept a lock file they never had. A failing recovery sync now also reports the
original failure instead of replacing it.
- Skip the recovery sync on cancellation. Answering Ctrl-C with a full
dependency resolve invites a second interrupt that escapes the handler and
strands the checkout mid-transaction; the declarations are already restored
and the next locked startup sync reconciles the environment.
- Retry a non-blocking lock on Windows instead of using `msvcrt.LK_LOCK`, which
gives up after ~10s — far shorter than a real `uv add` plus `uv sync`, so
contention surfaced as `Permission denied` rather than serializing.
- Locate the entry-point probe's JSON payload instead of parsing stdout's first
line, so a `sitecustomize`/`.pth` banner cannot roll back a good install.
- Warn when the lock records a loopback source. `127.0.0.1` inside the image
builder is a different machine, but unlike an environment-driven wheelhouse
resolution this is a source the operator typed deliberately, so it is reported
rather than rolled back. Private-network indexes are untouched: a builder on
that network can reach them.
Docs: the blanket claim that failed operations restore the config file was
wrong — the conflict branches deliberately preserve a concurrent external edit
and leave `remove` deactivated. Document that, the `required: false` default,
the config preflight, the interrupt behaviour, and where the plugins-block
boundaries come from.
* test(gateway): pin the request-path projection agreement
`get_request_route_path()` imports the private
`starlette._utils.get_route_path` so the auth and CSRF predicates classify
the exact string Starlette's router matches on. Its requirement is not
"strip root_path correctly" but "return what the dispatcher is matching",
so delegating to the router's own implementation keeps the two in lockstep
by construction. Keep the private import rather than vendoring a copy: an
import that disappears fails loudly at startup, while a stale copy diverges
silently at a security boundary.
Cover the property directly instead of the mechanism, so the tests survive
a future reimplementation:
- projection edge cases, including the segment-boundary guard that keeps
root_path="/api" from slicing "/apifoo/models" into a string the router
would never match
- agreement with the router under nested mounts
- the two bypasses these predicates exist to prevent: a protected route
mounted under the "/health" public prefix must still 401, and a POST
mounted under "/api/webhooks" must still require a CSRF token
Both are verified to fail when the projection is reverted to
`request.url.path` (9/13 red) and when a plausible vendored copy omits the
boundary guard (the 2 boundary cases red).
Declare starlette as a bounded direct dependency so a bump — which is
security-relevant here — shows up in review rather than arriving silently
through FastAPI.
* ci: pin uv to the version production ships
ExtensionManager is not a consumer of uv the build tool -- it is a program
whose whole job is driving `uv` as a subprocess, depending on its CLI
behavior (`--no-workspace`, `--no-sync`, what `uv add` writes into
`[dependency-groups] extensions`) and on the `uv.lock` serialization format.
uv is closer to a runtime dependency with a contract than to incidental
tooling.
backend/Dockerfile pins that binary to 0.11.1, but all eight
astral-sh/setup-uv steps installed whatever was latest at run time, so CI
exercised the manager against a uv that is not the uv production runs. The
sharpest failure that allows: a newer uv bumps uv.lock's `revision`, CI
stays green because the same uv reads back what it wrote, and the pinned uv
in the production image cannot read the committed lock. `uv lock --check`
is version-sensitive for the same reason -- it verifies the lock is what
*this* uv would produce, and two versions can emit equivalent but
non-identical output.
Pin every step to 0.11.1 and lift the one lingering setup-uv@v3 to v7 so
the steps share input and caching behavior.
Pinning alone drifts apart again on the next bump, so add a constraint test
in the style of test_compose_default_bind_host.py: the Dockerfile's
UV_IMAGE tag is the single source of truth, and both compose defaults plus
every setup-uv step must match it. Verified to fail when a pin drifts, when
a step omits `version`, and -- the real scenario -- when the Dockerfile is
bumped alone, which lights up the workflows and both compose files at once.
* fix(gateway): state the extension route auth limit and abort a failed dev sync
Two scoped review follow-ups.
README: contributed routers cannot enter the host's reserved public prefixes,
which makes every extension endpoint session-authenticated -- there is no way
to expose an unauthenticated route. The rejection rule was documented but its
consequence was not, so inbound provider webhooks and public status endpoints
read as merely undocumented rather than out of scope for this release.
docker/dev-entrypoint.sh: the self-heal retry reuses `--locked`, so it repairs
a corrupt .venv but never a lock that disagrees with pyproject.toml. `set -e`
already stopped the script there -- uvicorn was not being started against a
stale environment -- but it exited on a bare uv exit code with no indication of
what to do. Abort explicitly with the cause and the fix.
Tests slice the sync block out of the real script and run it against a stub uv,
so they exercise the shipped code rather than a copy of it (/app/backend only
exists inside the container). They cover the success path, the retry that
recovers, the abort, and the guidance. Verified against the pre-fix script:
only the guidance case goes red, confirming the abort itself was already
correct.
* fix(extensions): point Git SSH shorthand at the HTTPS correction
Git's SCP-like shorthand carries no URL scheme, so `git+git@host:org/repo.git`
reached the scheme rules looking like a bare path and was rejected with
"local path references are not deployable; pass a local directory so DeerFlow
can snapshot it". The operator asked for a remote source, so that guidance
points at the wrong fix. Detect the shorthand ahead of the scheme rules and
report the public-HTTPS correction instead.
The bare `git@host:org/repo.git` spelling took a different wrong turn: packaging
parses it as a direct reference named `git`, leaving `host:org/repo.git`, whose
`host` reads as a URL scheme and produced the generic HTTPS message. Both
spellings now share one message, as does the PEP 508 named form.
* docs: keep the root extension summary within its new budget
#4799 split the depth out of the module guides and added a size gate; the root
file's job is now orientation, and this branch had pushed it 192 bytes past the
soft limit. The manager transaction, source rules, and lock discipline are
already stated in full in the extensions guide, so the root keeps the one-line
orientation and points there instead of restating them.
---------
Co-authored-by: Codex <codex@openai.com>
* docs: govern agent guidance size
* refactor: split agent guidance by code scope
* Clarify virtual path handling in AGENTS.md
Updated the translation section to clarify the role of `LocalSandboxProvider` and the handling of virtual paths in the tool layer.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* ci(lark): publish lark-cli-init/broker images (#4532)
Add .github/workflows/lark-cli-images.yaml to build and push the two
optional Lark sandbox runtime images (Pattern A init container and
Pattern B broker sidecar) to GHCR.
These track the upstream larksuite/cli version, not the DeerFlow v*
release, so the workflow is decoupled from container.yaml / the
verify-versions gate:
- Trigger via workflow_dispatch (lark_cli_version input) or a lark-cli-v*
tag (version read from after the prefix).
- Multi-arch linux/amd64,linux/arm64 (the images stage a real
arch-dispatched lark-cli binary), via QEMU + Buildx.
- Per-component build context: lark-cli-init builds from its own dir
(relative COPY), lark-cli-broker from the repo root (it copies the
shared build-runtime.sh + the harness lark_broker.py).
- Tagged by lark-cli version; no latest; gated on the upstream repo.
Docs: document the independent publishing in RELEASING.md and both image
READMEs, replacing the "publishing is a fast-follow" notes.
Closes#4532
* fix(ci): scope broker build context past root .dockerignore
The repo-root .dockerignore excludes the whole `docker/` tree, but the
lark-cli-broker image builds from a repo-root context and must COPY
`docker/lark-cli-init/build-runtime.sh` and
`docker/lark-cli-broker/entrypoint.sh` (plus the harness lark_broker.py).
Under root .dockerignore those COPYs fail (excluded from context).
Add docker/lark-cli-broker/Dockerfile.dockerignore: BuildKit uses this
per-Dockerfile ignore-file instead of the root one for `-f
docker/lark-cli-broker/Dockerfile` builds, keeping `docker/` and
`backend/` in context while still dropping .git/venv/frontend/docs noise.
lark-cli-init is unaffected (it builds from its own dir context).
* fix(ci): harden lark-cli version input against shell injection
Address PR #4558 review: pass the workflow_dispatch input through an env
var instead of interpolating ${{ inputs.lark_cli_version }} directly into
the run: script, so a dispatched value can't be expression-injected into
the runner shell if the repo gate ever widens. Also fix the multi-arch
comment verb (stage -> ships).
* feat(channels): share inbound webhook dedupe across pods via Postgres (#4120)
* ci: run cross-pod inbound dedupe integration tests in CI
Expose the job Postgres service via DEDUPE_TEST_POSTGRES_URL so the integration tests (issue #4120) actually execute instead of silently skipping. Normalize the URL for asyncpg (postgresql:// -> +asyncpg, drop libpq-only sslmode) and await the now-async _is_duplicate_inbound in test_github_dispatcher.
* fix(auth): recover from setup status timeouts
* test(auth): cover setup status recovery flows
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(checkpoint): dual-mode checkpoint storage with LangGraph DeltaChannel
Add a restart-required database.checkpoint_channel_mode ("full" default,
"delta") that stores the messages channel via LangGraph 1.2 DeltaChannel,
cutting checkpoint storage from O(n^2) to O(n) for append-only history.
Existing full checkpoints seed delta state transparently; no data migration.
- config: mode schema + freeze-on-first-use with
CheckpointModeReconfigurationError; mode marker persisted in checkpoint
metadata; unsafe delta->full downgrade rejected fail-closed with
CheckpointModeMismatchError (run-level error, failed state read)
- state: delta message state schema; CheckpointStateAccessor centralizes
materialized reads for all consumers (threads API, branches,
regeneration, compaction, state updates, memory, goal workers)
- runtime: raw writers (run durations, interrupted title, thread goal)
parent their checkpoints to the checkpoint they derive from, preserving
delta ancestry; rollback forks the pre-run lineage through a state
mutation graph with Overwrite restores; InMemorySaver delta-history
override delegates to the base walk (fixes dropped first write after
migration, also present upstream)
- tests: conformance suite over {memory, sqlite, postgres} covering
migration replay, stable message IDs, storage shape and writer
preservation; conftest fixture isolates the frozen mode between tests;
stale config fakes refreshed
- ci: backend unit tests gain a postgres service
* fix(checkpoint): close materialization gaps in goal flow, guard public factory
- Route goal-continuation message reads through CheckpointStateAccessor:
raw channel_values reads see the delta sentinel in delta mode, which
disabled goal continuation (stand_down=no_durable_end_of_turn) after
durable assistant turns. Raw tuples remain for tuple-only metadata
(checkpoint id, pending_writes).
- Reject checkpoint_channel_mode='delta' + checkpointer in
create_deerflow_agent at construction: factory-built persisted graphs
bypass mode-marker injection and the fail-closed gate, reproducing
silent mixed-mode state loss. Delta without persistence stays allowed.
- Import the postgres saver lazily (pytest.importorskip in the fixture)
so the documented default install collects the suite; add a CI job
running pytest --collect-only on uv sync --group dev without extras.
- Fix test_checkpointer fallback test to patch get_app_config at its
use site (provider module), making it deterministic when a local
config.yaml selects a persistent backend.
* fix(gateway): preserve extension-owned channels in state mutations, bump config version
- build_state_mutation_graph / build_checkpoint_state_mutation_accessor
accept an explicit state_schema; branch and POST /state now compile the
mutation graph from the thread's effective schema
(graph_state_schema on the assistant graph). The base-ThreadState
fallback silently discarded channels contributed by custom
AgentMiddleware.state_schema on branch (data loss) and returned a
false-success 200 on POST /state.
- POST /state validates values keys against the mutation graph's
channels and rejects unknown fields with 422 instead of ignoring
them; reducer detection covers extension channels
(BinaryOperatorAggregate or DeltaChannel) so Overwrite replace
semantics work for middleware reducers in both modes.
- Endpoint regression: custom AgentMiddleware.state_schema value
survives branch, updates through POST /state, and an unknown field
receives 422.
- config_version 26 -> 27 for the new database.checkpoint_channel_mode
(example, Helm chart values + README, support-bundle fixture), so
existing installs get the outdated-config warning and
make config-upgrade merges the field; covered by a test driving the
real example file and the real config-upgrade script.
* fix(gateway): resolve assistant schema via one boundary, copy branch reducer values with Overwrite
GET /threads/{id}/state now resolves the thread's assistant_id through a
single reusable boundary (thread metadata -> assistant_id -> effective
graph), so channels contributed by a custom AgentMiddleware.state_schema
are materialized instead of dropped by the default lead schema. POST
/state uses the same boundary instead of resolving the schema ad hoc.
Branch writes wrap every copied reducer channel in Overwrite (derived
from the effective mutation graph: BinaryOperatorAggregate + DeltaChannel),
not just messages, so already-aggregated values are never re-merged.
Regression tests use a real AgentMiddleware.state_schema with a
non-identity reducer in both full and delta modes: GET /state returns the
extension value, POST /state replaces it, branch preserves it
byte-for-byte; the unknown-field 422 is a separate assertion.
* refactor(checkpoint): collapse read-path round-trips and ship dual-mode parity tests
Address review round 4 on PR #4292:
- Push ahistory/history limit through Pregel into checkpointer.alist
(SQL LIMIT) instead of materializing all rows and breaking in Python
- Fold the read-side mode-compat gate onto the returned snapshot's
metadata; only writes keep the pre-write tuple fetch (fail-closed)
- Cache factory-built accessor graphs per (assistant_id, mode) with
factory-identity revalidation; state reads no longer build a lead
agent per request
- get_thread: one snapshot fetch + one raw pending_writes fetch on the
resolved checkpoint (post-checkpoint __error__ writes never surface
in snapshot.tasks; verified empirically)
- DeerFlowClient.get_thread: single checkpointer.list walk collects
pending_writes per checkpoint instead of N get_tuple calls
- InMemorySaver delta-history patch: stand-down when the upstream
override disappears, try/except guard, validated-version warning,
guard tests
- make_lead_agent mode precedence: first freeze is owned by app_config
(client-supplied configurable key ignored); once frozen, injected
key/app_config must match or fail closed
- Rollback: lock in non-message channel restoration via fork
inheritance with a dedicated reducer-channel test
- Add tests/test_threads_checkpoint_mode.py and
tests/test_gateway_checkpoint_mode.py referenced by AGENTS.md and
the PR validation section: lifecycle parity (memory + sqlite),
per-step blob-count storage guard, gateway endpoint parity
Counted-saver tests pin checkpoint round-trips for aget/ahistory so
these regressions cannot silently return.
* fix(checkpoint): precise mode-mismatch HTTP mapping, gate E2E, and accessor resilience
- threads router: map CheckpointModeMismatchError to 409 (with cause and
thread id) and CheckpointModeReconfigurationError to 503 across all state
endpoints instead of swallowing both into a generic 500
- gate coverage: seed a real delta checkpoint into AsyncSqliteSaver and
assert aget/aupdate/ahistory fail closed; assert 409 at the HTTP boundary
through the real route stack
- rollback: compile the restore mutation graph with the thread's effective
state schema per the build_state_mutation_graph contract
- inheritance contract locks: rollback and manual compaction preserve
middleware-contributed channels via checkpoint fork cloning
- services: revalidate the accessor-graph cache against app_config identity
so config.yaml hot-reloads never serve a stale compiled graph
- services: degrade full-mode state reads to raw checkpointer reads when the
agent factory is unavailable (delta gate still applies; delta mode has no
fallback)
- deps: override websockets==16.0 (langgraph-sdk 0.4.2's <16 pin silently
downgraded 16.0 -> 15.0.1; pin is not grounded in any API incompatibility)
and bump the langchain lower bound to what the lockfile actually resolves
* fix(checkpoint): include anchor checkpoint in degraded history walk + cover get_thread
- _RawCheckpointReadAccessor.ahistory: alist(before=...) is exclusive while
pregel's get_state_history treats config.checkpoint_id as the inclusive
start; fetch the anchor explicitly so both read paths paginate identically
- extend the degraded-path gateway test: GET /thread returns raw values, and
POST /history with before starts at the anchor checkpoint
* fix(gateway): preserve degraded checkpoint timestamps
* fix(gateway): harden degraded checkpoint access
* fix(gateway): resolve assistants for checkpoint reads
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(helm): default sandbox Services to ClusterIP (#3929)
The K8s sandbox provisioner supports both NodePort and ClusterIP via
SANDBOX_SERVICE_TYPE (added in #4016), but the Helm chart never set it,
so real-cluster installs inherited the NodePort default. That bound the
code-execution sandbox on every node's interfaces - including externally
reachable ones on GKE/EKS/AKS - and pinned every sandbox URL to one node
IP (SPOF on node reboot/drain/ephemeral-IP).
Default the chart to ClusterIP: the provisioner returns a cluster-DNS URL
(http://sandbox-<id>-svc.<ns>.svc.cluster.local:8080) so the gateway->
sandbox hop stays inside the cluster network - no node IP, no 30xxx port,
no external exposure. The chart always runs the gateway in-cluster, so
ClusterIP is always correct there.
NodePort remains an opt-in (provisioner.sandboxServiceType: NodePort +
nodeHost) for the Docker-Compose/hybrid path where the gateway is not in
K8s and cannot resolve .svc.cluster.local; the provisioner code default
stays NodePort for that path.
- values.yaml: add provisioner.sandboxServiceType ("ClusterIP")
- provisioner-deployment.yaml: emit SANDBOX_SERVICE_TYPE; gate the
NODE_HOST block on NodePort mode (default "ClusterIP" for upgrade safety)
- NOTES.txt + README.md: document ClusterIP default + NodePort opt-in
No change to docker/provisioner/app.py (already mode-aware since #4016)
or RBAC (services verbs already cover ClusterIP).
* test(helm): assert sandbox Service-type gating + CHANGELOG the default flip (#3929)
Address review on #4190:
- Add scripts/check_chart_sandbox_service.sh: renders the chart for the
default (ClusterIP, no NODE_HOST), the NodePort opt-in (both emitted),
and NodePort+nodeHost (literal value, not downward API). Locks in the
#3929 gating so a regression (e.g. re-adding an unconditional NODE_HOST,
or dropping the `default "ClusterIP"` upgrade-safety fallback) fails CI.
Wired into .github/workflows/chart.yaml validate-chart job. (#2)
- CHANGELOG [Unreleased] -> Changed: note the NodePort->ClusterIP default
flip on upgrade + the `sandboxServiceType: NodePort` opt-back-in. (#4)
No chart template changes (the gating itself landed in the first commit).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* ci(helm): publish chart to charts/ namespace prefix on GHCR
Push the Helm chart to ghcr.io/<owner>/charts/deer-flow (via a `charts/`
prefix on the `helm push` target) instead of the bare `deer-flow`
package. This namespaces the chart apart from the image packages
(deer-flow-{backend,frontend,provisioner}) without renaming the chart:
Chart.yaml `name` stays `deer-flow`, so dir = chart name = in-cluster
resource/selector names, and no `nameOverride` hack is needed.
The chart is new in 2.1.0 (chart infra landed in #3987, after v2.0.0),
so 2.1.0 is the first chart release. Early nightly builds remain at the
legacy non-prefixed ghcr.io/<owner>/deer-flow.
Also refresh chart release docs:
- Replace the removed scripts/build-and-push.sh in the chart README with
raw `docker build`/`push` commands (contexts/args match container.yaml).
- Point NOTES.txt's empty-registry warning at the README section instead
of the removed script.
- Retarget RELEASING.md version examples from 2.2.0 to 2.1.0.
- Bump the chart README's helm prerequisite to 3+.
* docs(helm): require helm 3.8+ and note legacy chart package cleanup
Address PR #4175 review:
- bump documented minimum to helm 3.8 (OCI registry support stabilized
there; earlier 3.x needs HELM_EXPERIMENTAL_OCI=1)
- add a post-release note to delete/revoke the legacy bare
ghcr.io/<owner>/deer-flow chart package after 2.1.0
* feat(frontend): show real project version on About page
The About page heading was hardcoded "About DeerFlow 2.0" while the real
version is 2.1.0, kept in lockstep across backend/pyproject.toml,
frontend/package.json, and Chart.yaml by scripts/verify_versions.sh.
With the nightly workflow shipping unreleased main daily, a hardcoded
version drifts further from reality every day.
Wire the heading to the real version, nightly-aware:
- src/version.ts reads NEXT_PUBLIC_APP_VERSION (nightly override) with a
package.json fallback for release/local builds.
- about-content.ts interpolates APP_VERSION into the heading; the
"DeerFlow 1.0 and 2.0" milestone copy in acknowledgments is left as-is.
- Dockerfile adds ARG APP_VERSION / ENV NEXT_PUBLIC_APP_VERSION in the
builder stage so nightly CI can stamp the version.
- nightly.yaml computes the nightly string once in prepare
(<base>-nightly.<date>-<sha>, mirroring the chart scheme) and feeds it
to both the frontend image build-arg and the chart publish (refactored
to consume the shared output, so the displayed version and chart
version can't drift). Release builds (container.yaml) are untouched
and fall back to package.json.
Tests cover the version branches and the heading interpolation;
RELEASING.md notes the About page as a derived version consumer.
* test(frontend): assert positive About-page version value
Address review feedback on #4126: the unset-env assertion was negative-only
(checked the stale "2.0" literal was gone), so it would still pass if
APP_VERSION interpolated to "" or undefined. Mirror version.test.ts by
asserting the heading carries the real resolved APP_VERSION.
* ci(nightly): fail loudly on empty base version
Address review feedback on #4126: the prepare step's BASE (parsed from
Chart.yaml) had no empty-guard. A malformed/missing version would yield
`-nightly.<date>-<sha>` (invalid semver) that now flows into both the
chart publish and the frontend About-page version. Add `set -euo pipefail`
plus a `test -n "$BASE"` guard mirroring publish-chart. `pipefail` matters
- without it the pipeline's exit code is awk's, so `set -e` alone wouldn't
catch a grep miss.
* ci: add nightly build workflow for images + helm chart
Nightly build of backend/frontend/provisioner images and the helm chart,
pushed to GHCR with nightly + nightly-YYYYMMDD tags (latest stays on v*
releases). amd64 only. Gated to the upstream repo (bytedance/deer-flow).
Documented in RELEASING.md.
* ci(nightly): harden chart patch + dedupe config_version check
Address PR #4050 review feedback:
- Gate the in-workflow chart patches with grep assertions so a drifted
Chart.yaml/values.yaml fails loudly instead of silently shipping a chart
that pulls the release `latest` images (sed exits 0 on zero matches).
- Suffix the nightly chart version with the short SHA
(<base>-nightly.<date>-<sha>) so a same-day re-dispatch re-publishes
cleanly; OCI chart versions are immutable and otherwise collide.
- Note in the image-tags comment that :nightly-<date> is mutable within a
day and :sha-<short> is the only truly immutable pin.
- Extract the config_version drift check into scripts/check_config_version.sh,
shared by chart.yaml and nightly.yaml, so the parsing logic lives in one
place.
The tag-publish workflow (container.yaml) built backend/Dockerfile with no
build-args, so UV_EXTRAS was empty and the published *-backend image shipped
without the Postgres driver (only --extra redis). Multi-replica deployments
(K8s/Helm) that need shared Postgres persistence instead of file-based SQLite
could not use the release image without rebuilding it.
Pass UV_EXTRAS=postgres so the release image includes
deerflow-harness[postgres]. Additive only: single-replica sqlite/redis setups
keep working; the Postgres driver is added, mirroring how redis is already
always baked in.
* feat(helm): add production-ready Helm chart for Kubernetes deployment
Adds deploy/helm/deer-flow, a native-Kubernetes translation of the
production docker-compose stack, plus CI to publish its images and chart.
* ci(release): gate releases on version-source consistency
Add a reusable verify-versions workflow invoked by both chart.yaml and
container.yaml on v* tags. It runs scripts/verify_versions.sh against the
tag and fails the release — skipping all image and chart publishing — when
Chart.yaml (version + appVersion), backend/pyproject.toml, or
frontend/package.json don't all match the tag.
Add scripts/verify_versions.sh (the check, also runnable locally) and
scripts/bump_version.sh (bumps all four sources in lockstep, then
self-verifies). Document the release flow in RELEASING.md and link it from
AGENTS.md.
* fix(deploy): address Helm chart review feedback (#3987)
Three review items from willem-bd:
1. nginx IPv6 listen strip never matched. The sed pattern required a `;`
immediately after `2026`, but the rendered config emits
`listen [::]:2026 default_server;` (space + `default_server` before the
`;`), so the line was never deleted and nginx crash-looped on pods
without IPv6 (`socket() :::2026 failed (97: Address family not
supported)`). Drop the trailing `;` from the pattern so it matches.
Same latent bug fixed in docker-compose-dev.yaml.
2. Passwords were spliced into DSNs verbatim, so a password containing
URL-special chars (@ : / # ? % [ ] space) produced a malformed DSN and
a confusing parse error. Add a `deer-flow.urlEscape` helper
(replace-based: Sprig lacks urlqueryescape, and regexReplaceAllLiteral
treats the replacement as a regex template so `[`/`]`/`?` break it) and
apply it to the password in the postgres and redis DSNs. The raw
`postgres-password` / `redis-password` keys stay unencoded - they back
POSTGRES_PASSWORD / REDIS_PASSWORD, not a URL segment.
3. NODE_HOST defaulted to "gateway", which can never route: the gateway
Service is ClusterIP:8001 and knows nothing of a sandbox NodePort, so a
user who skips the caveat gets unreachable sandboxes with no error at
install time. Default NODE_HOST to the provisioner pod's node IP via
the downward API (status.hostIP) - a NodePort is exposed on every node,
so <node-IP>:<NodePort> routes from the gateway on most clusters.
`provisioner.nodeHost` remains an override for CNIs/policies that block
pod->node-IP traffic. Updated NOTES.txt, values.yaml, and the chart
README. (#3929 remains the long-term fix - ClusterIP + cluster-DNS URL
removes NODE_HOST and the NodePort exposure entirely.)
Validated with helm lint, helm template (incl. a special-char password
rendering the encoded DSNs), and a sed pattern-match check.
* fix(deploy): address round-2 Helm chart review feedback (#3987)
Three "Medium" items from willem-bd:
1. No helm lint / helm template gate before publish. A template regression
ships as an immutable OCI artifact (GHCR won't overwrite --version), so
gate packaging on `helm lint` + `helm template --include-crds` in
chart.yaml before `helm package`. (ct lint / helm-unittest deferred.)
2. Action pinning inconsistent + PR body overstates it. SHA-pin
actions/checkout (v6.0.3, df4cb1c0) and actions/attest-build-provenance
(v2.4.0, e8998f94) across the publishing workflows (chart.yaml,
container.yaml, verify-versions.yml), matching the existing docker/*
SHA-pin pattern. Resolves the checkout @v4/@v6 mismatch and makes the
"SHA-pinned actions" claim accurate. Other pre-existing workflows left
untouched (out of scope for this PR).
3. Provisioner RBAC broader than needed. Dropped the unused update/patch
verbs and the pods/exec + events rules from the provisioner Role -
audited against docker/provisioner/app.py, which only calls
get/create/delete on pods and get/list/create/delete on services. Fixed
NOTES.txt to accurately describe the grant instead of understating it as
"create Pods and Services". The remaining scope concern - verbs apply to
all Pods in the namespace, not just sandbox Pods - is still deferred
(RBAC can't scope by label; needs a dedicated namespace or admission
control), now noted in NOTES.txt and README.
Validated with helm lint + helm template (narrowed Role renders with
exactly get/list/watch/create/delete).
* feat(helm): enable sandbox+web tools out of the box
The chart's default config loaded zero agent tools (config.tools empty ->
"Total tools loaded: 0"), so a fresh install gave an agent that could do
nothing useful. Add tool_groups + tools to the default config block:
- web: web_search (ddg), web_fetch (jina), image_search - no API key
- file:read: ls, read_file, glob, grep
- file:write: write_file, str_replace
- bash
The file/bash tools run inside the AIO sandbox the chart already
configures; the web tools need outbound internet from the gateway pod
(swap backends or drop entries for air-gapped clusters - see
config.example.yaml).
Also bump config_version 15 -> 19 to match config.example.yaml (the chart
had drifted behind). NOTES.txt and the README example updated to match.
* ci(helm): add chart validation + config_version drift check on PR
Extend the chart workflow with a PR-triggered validate-chart job that runs
helm lint, helm template --include-crds, and a config_version drift check:
it parses config_version from both config.example.yaml and the chart's
values.yaml and fails the build (with a ::error:: naming the files to bump)
if the chart is behind the example. This catches the kind of drift this
PR is fixing - the chart sat at v15 while the example moved to v19 - before
it can merge again.
verify-versions and publish-chart stay tag-only; publish-chart now
needs: [verify-versions, validate-chart]. validate-chart runs on both
PRs and tag pushes: the tag arm is required because a job that `needs`
a skipped job is itself skipped under the default success() check, so
validate-chart must actually run on tag pushes or publish-chart would
never fire.
* Bump config version to 20
Add a uv lock --check guard in two places so a stale uv.lock cannot be
committed unnoticed (as happened with the groundroute extra):
- pre-commit: a local uv-lock-check hook, scoped to the backend
pyproject.toml files and uv.lock.
- CI: a "Check uv.lock is in sync" step in the lint-backend job, run
before uv sync so the install step cannot mask a stale lock by
regenerating it.
* fix(ci): consolidate PR/issue labeling into one triage.yml; fix reviewing crash & label thrash
- Replace pr-labeler + pr-triage + issue-triage with a single triage.yml; drop actions/labeler.
Its sync-labels removed labels outside its config (clobbered size/risk/needs-validation and
could clobber maintainer labels). Area is now computed in-script and reconciled only within
owned namespaces (area:/size//risk:/needs-validation); first-time/reviewing are add-only.
- reviewing: gate on author_association in {OWNER,MEMBER,COLLABORATOR} + user.type==='User'
instead of getCollaboratorPermissionLevel, which 404'd on bot reviewers ('Copilot is not a
user') and crashed the job. Excludes all review bots with no denylist and no API call.
- Read live state (listFiles + listLabelsOnIssue) not the stale event payload, so rapid
synchronize events converge instead of thrashing. Size churn excludes lockfiles/snapshots.
* fix(ci): read labels live via paginate in reviewing & issue-triage jobs
Address review feedback on #3455:
- reviewing: listLabelsOnIssue now paginates (per_page:100) instead of the
default 30, matching pr-labels, so a 'reviewing' label is never missed on
PRs with many labels.
- issue-triage: read live labels via the API instead of the event payload,
consistent with the live-state reads documented in the header.
* test(e2e): record/replay front-back contract verification
Guards the front-back contract with a deterministic, key-free record/replay
harness (mirrors open-design's golden-trace approach):
- ReplayChatModel (tests/replay_provider.py): replays recorded LLM turns by a
normalized hash of the model input. Strips <system-reminder>/date/uuid/tmp-path
so one fixture replays across days and from both the browser and direct-POST
paths; a miss raises loudly (no silent divergence).
- Recording is record-through-browser (scripts/record_gateway.py +
build_fixture_from_jsonl.py + frontend/tests/e2e-record): a real run is driven
through the real frontend so captured inputs match exactly what the browser
sends; fixtures contain no API key.
- Layer 1 — backend golden (tests/test_replay_golden.py): replay through the real
gateway, assert the SSE event sequence == committed golden.
- Layer 2 — full-stack render (frontend/tests/e2e-real-backend): real Next.js +
real gateway (replay model) + Chromium; assert the replayed auto-title and
follow-up suggestions render. DOM assertions are the gate; visual regression is
a local dev gate (CI uploads the render as an artifact).
- CI (.github/workflows/replay-e2e.yml): both layers, triggered on EITHER side of
the contract (frontend/** or backend gateway/harness/fixtures).
* test(e2e): multi-run render-order cross-stack scenario (#3352)
Guards the dangerous front-back class where a backend ordering change
silently breaks a frontend assumption while both sides' unit tests stay
green. Reproduces issue #3352: backend list_by_thread returns runs
newest-first (#2932) and the frontend prepended per-run pages, inverting
chronological order once the checkpoint no longer held the older messages.
- tests/seed_runs_router.py: test-only seeder, mounted on the replay
gateway only when DEERFLOW_ENABLE_TEST_SEED=1 (never in the production
app). Seeds a thread with >=2 runs + per-run message events and no
checkpoint -- the #3352 precondition -- so the frontend per-run reload
path is the sole source of truth and the prepend inversion is observable.
- frontend/tests/e2e-real-backend/multi-run-order.spec.ts: drives the real
frontend against the real gateway, asserts the first run renders above
the second. Reverting the #3354 fix turns it red.
- replay-e2e.yml: trigger on the new replay test-infra paths.
- docs: REPLAY_E2E.md cross-stack scenario section.
* test(e2e): address Copilot review on the replay harness
- Fix stale recorder references (scripts/record_traces.py ->
scripts/record_gateway.py + scripts/build_fixture_from_jsonl.py) in
replay_provider.py, test_replay_golden.py, _replay_fixture.py.
- MODE_CONTEXT['ultra']: thinking_enabled False -> True, mirroring the
frontend's `context.mode !== 'flash'` (hooks.ts). It did not affect the
hashed input (Layer 1 golden still green), but the table now matches the
real frontend context it claims to mirror.
- replay_provider.py docstring: stop claiming memory is recorded-enabled;
the replay config disables memory/summarization for determinism (title
stays, as an in-graph deterministic call).
- record_gateway.py / run_replay_gateway.py: override DEER_FLOW_HOME instead
of setdefault, so an outer value can't leak into the hermetic harness.
- record_gateway.py: clear error when DEERFLOW_RECORD_OUT is unset (was a
bare KeyError).
- playwright.record.config.ts: forward OPENAI_*/DEERFLOW_RECORD_OUT only when
set, so the gateway raises a clear 'missing env' error instead of getting ''.
* test(e2e): address Copilot review round 2
- seed_runs_router.py: constrain SeedMessage.role to Literal['human','ai']
so a bad value is a clean 422 at the boundary instead of a 500
(KeyError on _EVENT_TYPE).
- record-write-read-file.spec.ts: waitForCaptureStable now throws on
timeout instead of returning the last count, so a truncated/partial
recording can't pass silently.
- real-backend-render.spec.ts: guard the suggestions JSON.parse; a
bracket-prefixed non-JSON turn falls back to '' so the existing
not.toBe('') assertion fails clearly instead of a generic parse throw.
- .github/labels.yml: declarative source of truth (29 namespaced labels)
- scripts/sync_labels.py + label-sync.yml: idempotent label sync (self-bootstraps on merge)
- labeler.yml + pr-labeler.yml: area:* labels by changed path (actions/labeler)
- pr-triage.yml: size/*, risk:*, needs-validation, first-time-contributor, reviewing
- issue-triage.yml: needs-triage on new issues (self-healing)
All PR workflows use pull_request_target but never check out or run PR code
(read changed-file metadata via the API only).
* feat(tests): add Blockbuster runtime gate for event-loop blocking IO
Adds a strict runtime gate that fails CI when sync blocking IO calls run
on the asyncio event loop thread through DeerFlow business code.
Components:
- backend/tests/support/detectors/blocking_io_runtime.py — Blockbuster
context scoped to `app.*` and `deerflow.*` so test infrastructure,
pytest internals, and third-party libraries stay silent.
- backend/tests/blocking_io/conftest.py — pytest_runtest_protocol
hookwrapper that wraps every item (setup + call + teardown) with the
strict context. Respects `@pytest.mark.allow_blocking_io` opt-out.
- backend/tests/blocking_io/test_skills_load.py — regression anchor for
the #1917 fix (asyncio.to_thread offload around
LocalSkillStorage.load_skills).
- backend/tests/blocking_io/test_sqlite_lifespan.py — regression anchor
for the #1912 fix (asyncio.to_thread offload around
ensure_sqlite_parent_dir).
- backend/tests/blocking_io/test_gate_smoke.py — meta-test asserting the
gate actually catches unoffloaded blocking IO and that the
`@pytest.mark.allow_blocking_io` opt-out works.
- backend/Makefile — `make test-blocking-io` target.
- .github/workflows/backend-blocking-io-tests.yml — hard-fail PR gate on
ubuntu-latest. Windows matrix deferred to follow-up.
Dependencies:
- blockbuster>=1.5.26,<1.6 added to dev group.
Coverage boundary (called out in PR body): the gate only catches blocking
IO on code paths the test suite actually exercises. Static AST inventory
(separate, informational) is the complementary coverage tool. Three blind
spot categories — untested paths, mocked-away paths, env-mismatched paths
— are documented in the PR description.
Findings surfaced while authoring this PR:
- resolve_sqlite_conn_str in runtime/store/_sqlite_utils.py:19 does sync
Path.resolve() -> os.path.abspath on the lifespan loop thread, ahead of
the #1912 fix. Not addressed here; tracked as follow-up.
Tests: 4 passed locally (`make test-blocking-io`).
Lint/format: clean (`ruff check` and `ruff format --check`).
* fix(tests): scope Blockbuster gate to blocking-io suite
* fix(tests): harden Blockbuster runtime gate
* test(blocking-io): add project rule extension point
* test(blocking-io): address review cleanup
* feat: set up Vitest frontend testing infrastructure with CI workflow
Migrate existing 4 frontend test files from Node.js native test runner
(node:test + node:assert/strict) to Vitest, reorganize test directory
structure under tests/unit/ mirroring src/ layout, and add a dedicated
CI workflow for frontend unit tests.
- Add vitest as devDependency, remove tsx
- Create vitest.config.ts with @/ path alias
- Migrate tests to Vitest API (test/expect/vi)
- Rename .mjs test files to .ts
- Move tests from src/ to tests/unit/ (mirrors src/ layout)
- Add frontend/Makefile `test` target
- Add .github/workflows/frontend-unit-tests.yml (parallel to backend)
- Update CONTRIBUTING.md, README.md, AGENTS.md, CLAUDE.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: fix the lint error
* style: fix the lint error
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(subagents): make subagent timeout configurable via config.yaml
- Add SubagentsAppConfig supporting global and per-agent timeout_seconds
- Load subagents config section in AppConfig.from_file()
- Registry now applies config.yaml overrides without mutating builtin defaults
- Polling safety-net in task_tool is now dynamic (execution timeout + 60s buffer)
- Document subagents section in config.example.yaml
- Add make test command and enforce TDD policy in CLAUDE.md
- Add 38 unit tests covering config validation, timeout resolution, registry
override behavior, and polling timeout formula
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(subagents): add logging for subagent timeout config and execution
- Log loaded timeout config (global default + per-agent overrides) on startup
- Log debug message in registry when config.yaml overrides a builtin timeout
- Include timeout in executor's async execution start log
- Log effective timeout and polling limit when a task is dispatched
- Fix UnboundLocalError: move max_poll_count assignment before logger.info
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* ci(backend): add lint step and run all unit tests via Makefile
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix lint
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: resolve issue #588 - react key warnings from duplicate message IDs + establish jest testing framework
* Update the makefile and workflow with the js test
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: using mongomock for the checkpoint test
* Add postgres mock setting to the unit test
* Added utils file of postgres_mock_utils
* fixed the runtime loading error of deerflow server
* fix: using commit hash as the action version
* fix: using commit hash as the action version
---------
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
* feat: Enhance chat streaming and tool call processing
- Added support for MongoDB checkpointer in the chat streaming workflow.
- Introduced functions to process tool call chunks and sanitize arguments.
- Improved event message creation with additional metadata.
- Enhanced error handling for JSON serialization in event messages.
- Updated the frontend to convert escaped characters in tool call arguments.
- Refactored the workflow input preparation and initial message processing.
- Added new dependencies for MongoDB integration and tool argument sanitization.
* fix: Update MongoDB checkpointer configuration to use LANGGRAPH_CHECKPOINT_DB_URL
* feat: Add support for Postgres checkpointing and update README with database recommendations
* feat: Implement checkpoint saver functionality and update MongoDB connection handling
* refactor: Improve code formatting and readability in app.py and json_utils.py
* refactor: Clean up commented code and improve formatting in server.py
* refactor: Remove unused imports and improve code organization in app.py
* refactor: Improve code organization and remove unnecessary comments in app.py
* chore: use langgraph-checkpoint-postgres==2.0.21 to avoid the JSON convert issue in the latest version, implement chat stream persistant with Postgres
* feat: add MongoDB and PostgreSQL support for LangGraph checkpointing, enhance environment variable handling
* fix: update comments for clarity on Windows event loop policy
* chore: remove empty code changes in MongoDB and PostgreSQL checkpoint tests
* chore: clean up unused imports and code in checkpoint-related files
* chore: remove empty code changes in test_checkpoint.py
* chore: remove empty code changes in test_checkpoint.py
* chore: remove empty code changes in test_checkpoint.py
* test: update status code assertions in MCP endpoint tests to allow for 403 responses
* test: update MCP endpoint tests to assert specific status codes and enable MCP server configuration
* chore: remove unnecessary environment variables from unittest workflow
* fix: invert condition for MCP server configuration check to raise 403 when disabled
* chore: remove pymongo from test dependencies in uv.lock
* chore: optimize the _get_agent_name method
* test: enhance ChatStreamManager tests for PostgreSQL and MongoDB initialization
* test: add persistence tests for ChatStreamManager with PostgreSQL and MongoDB
* test: add unit tests for ChatStreamManager initialization with PostgreSQL and MongoDB
* test: enhance persistence tests for ChatStreamManager with PostgreSQL and MongoDB to verify message aggregation
* test: add unit tests for ChatStreamManager with PostgreSQL and MongoDB
* test: add unit tests for ChatStreamManager initialization with PostgreSQL and MongoDB
* test: add unit tests for ChatStreamManager initialization with PostgreSQL and MongoDB
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
