The tag-publish workflow (container.yaml) built backend/Dockerfile with no
build-args, so UV_EXTRAS was empty and the published *-backend image shipped
without the Postgres driver (only --extra redis). Multi-replica deployments
(K8s/Helm) that need shared Postgres persistence instead of file-based SQLite
could not use the release image without rebuilding it.
Pass UV_EXTRAS=postgres so the release image includes
deerflow-harness[postgres]. Additive only: single-replica sqlite/redis setups
keep working; the Postgres driver is added, mirroring how redis is already
always baked in.
* feat(helm): add production-ready Helm chart for Kubernetes deployment
Adds deploy/helm/deer-flow, a native-Kubernetes translation of the
production docker-compose stack, plus CI to publish its images and chart.
* ci(release): gate releases on version-source consistency
Add a reusable verify-versions workflow invoked by both chart.yaml and
container.yaml on v* tags. It runs scripts/verify_versions.sh against the
tag and fails the release — skipping all image and chart publishing — when
Chart.yaml (version + appVersion), backend/pyproject.toml, or
frontend/package.json don't all match the tag.
Add scripts/verify_versions.sh (the check, also runnable locally) and
scripts/bump_version.sh (bumps all four sources in lockstep, then
self-verifies). Document the release flow in RELEASING.md and link it from
AGENTS.md.
* fix(deploy): address Helm chart review feedback (#3987)
Three review items from willem-bd:
1. nginx IPv6 listen strip never matched. The sed pattern required a `;`
immediately after `2026`, but the rendered config emits
`listen [::]:2026 default_server;` (space + `default_server` before the
`;`), so the line was never deleted and nginx crash-looped on pods
without IPv6 (`socket() :::2026 failed (97: Address family not
supported)`). Drop the trailing `;` from the pattern so it matches.
Same latent bug fixed in docker-compose-dev.yaml.
2. Passwords were spliced into DSNs verbatim, so a password containing
URL-special chars (@ : / # ? % [ ] space) produced a malformed DSN and
a confusing parse error. Add a `deer-flow.urlEscape` helper
(replace-based: Sprig lacks urlqueryescape, and regexReplaceAllLiteral
treats the replacement as a regex template so `[`/`]`/`?` break it) and
apply it to the password in the postgres and redis DSNs. The raw
`postgres-password` / `redis-password` keys stay unencoded - they back
POSTGRES_PASSWORD / REDIS_PASSWORD, not a URL segment.
3. NODE_HOST defaulted to "gateway", which can never route: the gateway
Service is ClusterIP:8001 and knows nothing of a sandbox NodePort, so a
user who skips the caveat gets unreachable sandboxes with no error at
install time. Default NODE_HOST to the provisioner pod's node IP via
the downward API (status.hostIP) - a NodePort is exposed on every node,
so <node-IP>:<NodePort> routes from the gateway on most clusters.
`provisioner.nodeHost` remains an override for CNIs/policies that block
pod->node-IP traffic. Updated NOTES.txt, values.yaml, and the chart
README. (#3929 remains the long-term fix - ClusterIP + cluster-DNS URL
removes NODE_HOST and the NodePort exposure entirely.)
Validated with helm lint, helm template (incl. a special-char password
rendering the encoded DSNs), and a sed pattern-match check.
* fix(deploy): address round-2 Helm chart review feedback (#3987)
Three "Medium" items from willem-bd:
1. No helm lint / helm template gate before publish. A template regression
ships as an immutable OCI artifact (GHCR won't overwrite --version), so
gate packaging on `helm lint` + `helm template --include-crds` in
chart.yaml before `helm package`. (ct lint / helm-unittest deferred.)
2. Action pinning inconsistent + PR body overstates it. SHA-pin
actions/checkout (v6.0.3, df4cb1c0) and actions/attest-build-provenance
(v2.4.0, e8998f94) across the publishing workflows (chart.yaml,
container.yaml, verify-versions.yml), matching the existing docker/*
SHA-pin pattern. Resolves the checkout @v4/@v6 mismatch and makes the
"SHA-pinned actions" claim accurate. Other pre-existing workflows left
untouched (out of scope for this PR).
3. Provisioner RBAC broader than needed. Dropped the unused update/patch
verbs and the pods/exec + events rules from the provisioner Role -
audited against docker/provisioner/app.py, which only calls
get/create/delete on pods and get/list/create/delete on services. Fixed
NOTES.txt to accurately describe the grant instead of understating it as
"create Pods and Services". The remaining scope concern - verbs apply to
all Pods in the namespace, not just sandbox Pods - is still deferred
(RBAC can't scope by label; needs a dedicated namespace or admission
control), now noted in NOTES.txt and README.
Validated with helm lint + helm template (narrowed Role renders with
exactly get/list/watch/create/delete).
* feat(helm): enable sandbox+web tools out of the box
The chart's default config loaded zero agent tools (config.tools empty ->
"Total tools loaded: 0"), so a fresh install gave an agent that could do
nothing useful. Add tool_groups + tools to the default config block:
- web: web_search (ddg), web_fetch (jina), image_search - no API key
- file:read: ls, read_file, glob, grep
- file:write: write_file, str_replace
- bash
The file/bash tools run inside the AIO sandbox the chart already
configures; the web tools need outbound internet from the gateway pod
(swap backends or drop entries for air-gapped clusters - see
config.example.yaml).
Also bump config_version 15 -> 19 to match config.example.yaml (the chart
had drifted behind). NOTES.txt and the README example updated to match.
* ci(helm): add chart validation + config_version drift check on PR
Extend the chart workflow with a PR-triggered validate-chart job that runs
helm lint, helm template --include-crds, and a config_version drift check:
it parses config_version from both config.example.yaml and the chart's
values.yaml and fails the build (with a ::error:: naming the files to bump)
if the chart is behind the example. This catches the kind of drift this
PR is fixing - the chart sat at v15 while the example moved to v19 - before
it can merge again.
verify-versions and publish-chart stay tag-only; publish-chart now
needs: [verify-versions, validate-chart]. validate-chart runs on both
PRs and tag pushes: the tag arm is required because a job that `needs`
a skipped job is itself skipped under the default success() check, so
validate-chart must actually run on tag pushes or publish-chart would
never fire.
* Bump config version to 20
Add a uv lock --check guard in two places so a stale uv.lock cannot be
committed unnoticed (as happened with the groundroute extra):
- pre-commit: a local uv-lock-check hook, scoped to the backend
pyproject.toml files and uv.lock.
- CI: a "Check uv.lock is in sync" step in the lint-backend job, run
before uv sync so the install step cannot mask a stale lock by
regenerating it.
* fix(ci): consolidate PR/issue labeling into one triage.yml; fix reviewing crash & label thrash
- Replace pr-labeler + pr-triage + issue-triage with a single triage.yml; drop actions/labeler.
Its sync-labels removed labels outside its config (clobbered size/risk/needs-validation and
could clobber maintainer labels). Area is now computed in-script and reconciled only within
owned namespaces (area:/size//risk:/needs-validation); first-time/reviewing are add-only.
- reviewing: gate on author_association in {OWNER,MEMBER,COLLABORATOR} + user.type==='User'
instead of getCollaboratorPermissionLevel, which 404'd on bot reviewers ('Copilot is not a
user') and crashed the job. Excludes all review bots with no denylist and no API call.
- Read live state (listFiles + listLabelsOnIssue) not the stale event payload, so rapid
synchronize events converge instead of thrashing. Size churn excludes lockfiles/snapshots.
* fix(ci): read labels live via paginate in reviewing & issue-triage jobs
Address review feedback on #3455:
- reviewing: listLabelsOnIssue now paginates (per_page:100) instead of the
default 30, matching pr-labels, so a 'reviewing' label is never missed on
PRs with many labels.
- issue-triage: read live labels via the API instead of the event payload,
consistent with the live-state reads documented in the header.
* test(e2e): record/replay front-back contract verification
Guards the front-back contract with a deterministic, key-free record/replay
harness (mirrors open-design's golden-trace approach):
- ReplayChatModel (tests/replay_provider.py): replays recorded LLM turns by a
normalized hash of the model input. Strips <system-reminder>/date/uuid/tmp-path
so one fixture replays across days and from both the browser and direct-POST
paths; a miss raises loudly (no silent divergence).
- Recording is record-through-browser (scripts/record_gateway.py +
build_fixture_from_jsonl.py + frontend/tests/e2e-record): a real run is driven
through the real frontend so captured inputs match exactly what the browser
sends; fixtures contain no API key.
- Layer 1 — backend golden (tests/test_replay_golden.py): replay through the real
gateway, assert the SSE event sequence == committed golden.
- Layer 2 — full-stack render (frontend/tests/e2e-real-backend): real Next.js +
real gateway (replay model) + Chromium; assert the replayed auto-title and
follow-up suggestions render. DOM assertions are the gate; visual regression is
a local dev gate (CI uploads the render as an artifact).
- CI (.github/workflows/replay-e2e.yml): both layers, triggered on EITHER side of
the contract (frontend/** or backend gateway/harness/fixtures).
* test(e2e): multi-run render-order cross-stack scenario (#3352)
Guards the dangerous front-back class where a backend ordering change
silently breaks a frontend assumption while both sides' unit tests stay
green. Reproduces issue #3352: backend list_by_thread returns runs
newest-first (#2932) and the frontend prepended per-run pages, inverting
chronological order once the checkpoint no longer held the older messages.
- tests/seed_runs_router.py: test-only seeder, mounted on the replay
gateway only when DEERFLOW_ENABLE_TEST_SEED=1 (never in the production
app). Seeds a thread with >=2 runs + per-run message events and no
checkpoint -- the #3352 precondition -- so the frontend per-run reload
path is the sole source of truth and the prepend inversion is observable.
- frontend/tests/e2e-real-backend/multi-run-order.spec.ts: drives the real
frontend against the real gateway, asserts the first run renders above
the second. Reverting the #3354 fix turns it red.
- replay-e2e.yml: trigger on the new replay test-infra paths.
- docs: REPLAY_E2E.md cross-stack scenario section.
* test(e2e): address Copilot review on the replay harness
- Fix stale recorder references (scripts/record_traces.py ->
scripts/record_gateway.py + scripts/build_fixture_from_jsonl.py) in
replay_provider.py, test_replay_golden.py, _replay_fixture.py.
- MODE_CONTEXT['ultra']: thinking_enabled False -> True, mirroring the
frontend's `context.mode !== 'flash'` (hooks.ts). It did not affect the
hashed input (Layer 1 golden still green), but the table now matches the
real frontend context it claims to mirror.
- replay_provider.py docstring: stop claiming memory is recorded-enabled;
the replay config disables memory/summarization for determinism (title
stays, as an in-graph deterministic call).
- record_gateway.py / run_replay_gateway.py: override DEER_FLOW_HOME instead
of setdefault, so an outer value can't leak into the hermetic harness.
- record_gateway.py: clear error when DEERFLOW_RECORD_OUT is unset (was a
bare KeyError).
- playwright.record.config.ts: forward OPENAI_*/DEERFLOW_RECORD_OUT only when
set, so the gateway raises a clear 'missing env' error instead of getting ''.
* test(e2e): address Copilot review round 2
- seed_runs_router.py: constrain SeedMessage.role to Literal['human','ai']
so a bad value is a clean 422 at the boundary instead of a 500
(KeyError on _EVENT_TYPE).
- record-write-read-file.spec.ts: waitForCaptureStable now throws on
timeout instead of returning the last count, so a truncated/partial
recording can't pass silently.
- real-backend-render.spec.ts: guard the suggestions JSON.parse; a
bracket-prefixed non-JSON turn falls back to '' so the existing
not.toBe('') assertion fails clearly instead of a generic parse throw.
- .github/labels.yml: declarative source of truth (29 namespaced labels)
- scripts/sync_labels.py + label-sync.yml: idempotent label sync (self-bootstraps on merge)
- labeler.yml + pr-labeler.yml: area:* labels by changed path (actions/labeler)
- pr-triage.yml: size/*, risk:*, needs-validation, first-time-contributor, reviewing
- issue-triage.yml: needs-triage on new issues (self-healing)
All PR workflows use pull_request_target but never check out or run PR code
(read changed-file metadata via the API only).
* feat(tests): add Blockbuster runtime gate for event-loop blocking IO
Adds a strict runtime gate that fails CI when sync blocking IO calls run
on the asyncio event loop thread through DeerFlow business code.
Components:
- backend/tests/support/detectors/blocking_io_runtime.py — Blockbuster
context scoped to `app.*` and `deerflow.*` so test infrastructure,
pytest internals, and third-party libraries stay silent.
- backend/tests/blocking_io/conftest.py — pytest_runtest_protocol
hookwrapper that wraps every item (setup + call + teardown) with the
strict context. Respects `@pytest.mark.allow_blocking_io` opt-out.
- backend/tests/blocking_io/test_skills_load.py — regression anchor for
the #1917 fix (asyncio.to_thread offload around
LocalSkillStorage.load_skills).
- backend/tests/blocking_io/test_sqlite_lifespan.py — regression anchor
for the #1912 fix (asyncio.to_thread offload around
ensure_sqlite_parent_dir).
- backend/tests/blocking_io/test_gate_smoke.py — meta-test asserting the
gate actually catches unoffloaded blocking IO and that the
`@pytest.mark.allow_blocking_io` opt-out works.
- backend/Makefile — `make test-blocking-io` target.
- .github/workflows/backend-blocking-io-tests.yml — hard-fail PR gate on
ubuntu-latest. Windows matrix deferred to follow-up.
Dependencies:
- blockbuster>=1.5.26,<1.6 added to dev group.
Coverage boundary (called out in PR body): the gate only catches blocking
IO on code paths the test suite actually exercises. Static AST inventory
(separate, informational) is the complementary coverage tool. Three blind
spot categories — untested paths, mocked-away paths, env-mismatched paths
— are documented in the PR description.
Findings surfaced while authoring this PR:
- resolve_sqlite_conn_str in runtime/store/_sqlite_utils.py:19 does sync
Path.resolve() -> os.path.abspath on the lifespan loop thread, ahead of
the #1912 fix. Not addressed here; tracked as follow-up.
Tests: 4 passed locally (`make test-blocking-io`).
Lint/format: clean (`ruff check` and `ruff format --check`).
* fix(tests): scope Blockbuster gate to blocking-io suite
* fix(tests): harden Blockbuster runtime gate
* test(blocking-io): add project rule extension point
* test(blocking-io): address review cleanup
* feat: set up Vitest frontend testing infrastructure with CI workflow
Migrate existing 4 frontend test files from Node.js native test runner
(node:test + node:assert/strict) to Vitest, reorganize test directory
structure under tests/unit/ mirroring src/ layout, and add a dedicated
CI workflow for frontend unit tests.
- Add vitest as devDependency, remove tsx
- Create vitest.config.ts with @/ path alias
- Migrate tests to Vitest API (test/expect/vi)
- Rename .mjs test files to .ts
- Move tests from src/ to tests/unit/ (mirrors src/ layout)
- Add frontend/Makefile `test` target
- Add .github/workflows/frontend-unit-tests.yml (parallel to backend)
- Update CONTRIBUTING.md, README.md, AGENTS.md, CLAUDE.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: fix the lint error
* style: fix the lint error
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(subagents): make subagent timeout configurable via config.yaml
- Add SubagentsAppConfig supporting global and per-agent timeout_seconds
- Load subagents config section in AppConfig.from_file()
- Registry now applies config.yaml overrides without mutating builtin defaults
- Polling safety-net in task_tool is now dynamic (execution timeout + 60s buffer)
- Document subagents section in config.example.yaml
- Add make test command and enforce TDD policy in CLAUDE.md
- Add 38 unit tests covering config validation, timeout resolution, registry
override behavior, and polling timeout formula
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(subagents): add logging for subagent timeout config and execution
- Log loaded timeout config (global default + per-agent overrides) on startup
- Log debug message in registry when config.yaml overrides a builtin timeout
- Include timeout in executor's async execution start log
- Log effective timeout and polling limit when a task is dispatched
- Fix UnboundLocalError: move max_poll_count assignment before logger.info
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* ci(backend): add lint step and run all unit tests via Makefile
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix lint
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: resolve issue #588 - react key warnings from duplicate message IDs + establish jest testing framework
* Update the makefile and workflow with the js test
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: using mongomock for the checkpoint test
* Add postgres mock setting to the unit test
* Added utils file of postgres_mock_utils
* fixed the runtime loading error of deerflow server
* fix: using commit hash as the action version
* fix: using commit hash as the action version
---------
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
* feat: Enhance chat streaming and tool call processing
- Added support for MongoDB checkpointer in the chat streaming workflow.
- Introduced functions to process tool call chunks and sanitize arguments.
- Improved event message creation with additional metadata.
- Enhanced error handling for JSON serialization in event messages.
- Updated the frontend to convert escaped characters in tool call arguments.
- Refactored the workflow input preparation and initial message processing.
- Added new dependencies for MongoDB integration and tool argument sanitization.
* fix: Update MongoDB checkpointer configuration to use LANGGRAPH_CHECKPOINT_DB_URL
* feat: Add support for Postgres checkpointing and update README with database recommendations
* feat: Implement checkpoint saver functionality and update MongoDB connection handling
* refactor: Improve code formatting and readability in app.py and json_utils.py
* refactor: Clean up commented code and improve formatting in server.py
* refactor: Remove unused imports and improve code organization in app.py
* refactor: Improve code organization and remove unnecessary comments in app.py
* chore: use langgraph-checkpoint-postgres==2.0.21 to avoid the JSON convert issue in the latest version, implement chat stream persistant with Postgres
* feat: add MongoDB and PostgreSQL support for LangGraph checkpointing, enhance environment variable handling
* fix: update comments for clarity on Windows event loop policy
* chore: remove empty code changes in MongoDB and PostgreSQL checkpoint tests
* chore: clean up unused imports and code in checkpoint-related files
* chore: remove empty code changes in test_checkpoint.py
* chore: remove empty code changes in test_checkpoint.py
* chore: remove empty code changes in test_checkpoint.py
* test: update status code assertions in MCP endpoint tests to allow for 403 responses
* test: update MCP endpoint tests to assert specific status codes and enable MCP server configuration
* chore: remove unnecessary environment variables from unittest workflow
* fix: invert condition for MCP server configuration check to raise 403 when disabled
* chore: remove pymongo from test dependencies in uv.lock
* chore: optimize the _get_agent_name method
* test: enhance ChatStreamManager tests for PostgreSQL and MongoDB initialization
* test: add persistence tests for ChatStreamManager with PostgreSQL and MongoDB
* test: add unit tests for ChatStreamManager initialization with PostgreSQL and MongoDB
* test: enhance persistence tests for ChatStreamManager with PostgreSQL and MongoDB to verify message aggregation
* test: add unit tests for ChatStreamManager with PostgreSQL and MongoDB
* test: add unit tests for ChatStreamManager initialization with PostgreSQL and MongoDB
* test: add unit tests for ChatStreamManager initialization with PostgreSQL and MongoDB
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
