* fix(ragflow): batch validation for large document selections
* docs(ragflow): align documentation language with repository conventions
* docs(ragflow): preserve spacing before validation heading
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash
Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md
(issue #5160, tracker #5129) in the slice order of the spec's §16.
Slices:
- A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY
admission pinning (both server-owned sets + worker hoist); latest-only
request-scoped <project> block via DynamicContextMiddleware
wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID
prefix + marker + provenance, never persisted); journal audit
fingerprints; Instructions tab.
- B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with
locked check-and-set; hash-qualified immutable shelf storage with
Paths helpers; upload/list/content/delete-to-trash routes; project
delete trashes the shelf in-transaction; request-scoped bounded
<documents> index with honest count/shown + actionable overflow note;
list_project_documents/read_project_document tools registered only on
pinned runs; PAT allowlist + drift guards; blocking-IO anchors.
- C: shared thread-upload ingestion service (uploads router refactored to
parity); POST from-thread with provenance; attach-to-thread with
lock-staged copy (archived source allowed); read-only thread-files
view with per-group truncation reporting.
- D: restore (restored/merged/not_found/no_target/content_missing; no
file moves), purge (continuous row lock across unlink/delete/commit,
retryable on FS errors), retention sweep (lazy + startup, 24h orphan
guard, row-side reconciliation never deletes).
- E: Documents tab (shelf + conversation-files browser, provenance,
archived banner, content-missing rows), /workspace/trash route,
sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks
+ specs.
Review hardening folded in (10 rounds, all with tests):
- force active shelf content (HTML/XML family) to download; nosniff on
artifact + content responses; unified unsandboxed-iframe PDF preview
(fixes the pre-existing Chromium sandbox blank in the artifact viewer)
- scope document trash to the URL project under the document lock
- atomic no-overwrite filename reservation for ALL ingestion (seeded
claims + os.link commit with suffix retry; same-name re-upload now
unique-names instead of replacing); hidden staging only, no visible
placeholders; lease cleanup on setup failure
- serialize conversion under the document lock with post-lock active
revalidation; drain locked filesystem work on cancellation; preserve
bytes when an insert's commit state is uncertain (including trashed
rows)
- original-integrity checks before serving text or cached conversions;
content_missing surfaced in list responses (UI reads the flag, no
409-probe); downloads always serve original bytes
- bounded streaming document reads with cached char counts; shelf limits
declared in middleware release identity
- thread-root confinement for from-thread sources; config fallback
rejects fractional/infinite values; composer counts staged
attachments; pending attachments persist until submission or removal;
in-flight instruction/rename edits survive save refetches; shelf and
trash pagination; conversation-file and thread-files pages stay
subscribed to refetches
Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md
contracts, config.example.yaml projects block.
Review follow-ups (head b4807477 → this revision):
- The trash retention sweep is split so repeated lazy triggers stay
bounded: the indexed expiry purge still runs on every trigger
(GET /api/trash/documents, POST /api/trash/purge) while the
O(all rows + all files) reconciliation is throttled to one run per
user per 15 minutes (process-local, per-user window). The startup
sweep now runs as a background task instead of blocking gateway
readiness, and shutdown awaits it (bounded).
- The export scrub (stripInternalMarkers) is fence- and indentation-aware
like the render path, so a pasted, fenced <project>/<documents> snippet
survives markdown export while real injected blocks (never fenced) are
still removed. Fence regexes moved to a dependency-free leaf module to
avoid the messages↔streamdown import cycle.
- The artifact viewer's PDF iframe no longer carries an added title
attribute (the upstream e2e contract locates it via :not([title])), and
the upstream artifact-preview spec now pins the new contract: PDFs
render unsandboxed, images keep sandbox="".
* fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title
- Shutdown cancelled only the shield around the background startup sweep,
so an all-users reconciliation that outlived the 5s budget kept walking
rows and files while the document repo and DB engine were disposed
underneath it. The wait now lives in `_shutdown_startup_trash_sweep`,
which cancels the task and drains it before worker exit: the shield
keeps the wait bounded, the cancel makes it final (CancelledError lands
at the sweep's next await, and `_run_startup_trash_sweep` only catches
`Exception`, so nothing swallows it).
- The browser-preview iframe lost `title={getFileName(filepath)}` in the
previous fix round, leaving the PDF frame without an accessible name
while its siblings keep theirs. Restore it (WCAG frame titles), assert
it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]`
instead of `iframe:not([title])`.
* fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel
`Task.cancel()` returns False when the sweep already finished inside the
window between the deadline firing and the cancel, so the shutdown log
claimed a cancellation that never happened. Branch on that outcome: the
warning stays for a real cancel, a late finish is logged at info, and both
paths still reap the task before worker exit.
* fix(projects): round-4 review — make Empty trash delete what it confirms
`POST /api/trash/purge` only ran the retention sweep, and the sweep's
candidate selection is age-gated, so a freshly trashed document survived
"Empty trash" even though the confirmation promises that every listed
document is permanently deleted. With one trashed row the route answered
`{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps
expired rows before listing, so the visible rows were normally ineligible
for the action by construction.
Empty trash now drives `purge_all_trashed`: the caller's trashed rows
(`list_all_trashed`, no age filter) each go through the same guarded,
row-locked `purge` as the single-document delete — bytes first, then the
row, in one transaction — so a row restored mid-flight is skipped instead of
force-deleted, and an unlink failure rolls that row back and answers 500 with
a retryable message. Retention expiry stays where it was: the sweep's
`purge_candidates` is now the only age-gated selection, and the lazy
retention sweep still runs on the listing and at startup.
Tests: the router suite replaces the retention-gated expectation with the
reviewer's repro (fresh row purged, bytes unlinked, shelf and other users'
trash untouched, a failing unlink stays retryable and 500); a blocking-I/O
anchor drives the new entry point through the offload; the mocked e2e covers
the action end to end; a new real-backend spec performs it against the real
gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE
and the phase-2 design docs (en+zh) state the age-independent contract.
* feat(scheduled-tasks): filter run history by occurrence status
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
* fix(scheduled-tasks): share occurrence status contract
---------
Signed-off-by: tiammomo <26957354+tiammomo@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* docs(zh): add the missing Project Membership section to README_zh
#5395 added the Project Membership section to the English README; this
mirrors it in the Chinese README, in the same position between the
embedded Python Client and Scheduled Tasks sections.
* docs(zh): use half-width parentheses in the Project Membership heading
Review nit: match the heading style of 定时任务 (Scheduled Tasks) and
终端工作台 (TUI).
* feat(scheduler): add interval schedule type
Allow scheduled tasks to fire every N seconds from last dispatch, not
only wall-clock cron or a single run_at. Cadence is UTC now+N with no
missed-beat catch-up, bounded by min_once_delay_seconds and 30 days.
* fix(scheduler): let interval tasks create, edit, and keep next run
Create/edit now keep every_seconds. Unchanged interval spec no longer
resets next_run_at, including timezone-only PATCH.
* fix(scheduler): keep non-minute intervals on edit
Stop rounding every_seconds to whole minutes in the form. Values that
are not whole minutes or hours now use a seconds unit so edit/duplicate
round-trips the stored cadence instead of rewriting it and resetting
next_run_at. Document that min_once_delay_seconds is also the interval
floor.
* fix(scheduler): clamp interval seconds to the default 60s floor
The new seconds unit allowed 1–59, which the API rejects under the
default min_once_delay_seconds. Clamp the form to >= 60 and show the
floor next to the preview. Also mention interval in the scheduler
field_doc, matching config.example.yaml.
* fix(scheduler): do not clamp interval amount while typing
Keystroke clamp made 90 become 9 -> 60, then 600, and backspace could
not leave 60. Keep the raw field text and apply the 60s floor on blur
and emit only.
* test(scheduler): cover interval input editing
* fix(frontend): preserve saved interval cadence until edited
* style(tests): format scheduled task router tests
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(scheduler): let scheduled tasks pin a custom agent
Create and update accept optional assistant_id, defaulting to lead_agent.
Custom names are normalized and must already exist for the task owner.
The workspace form exposes the same choice, and duplicate copies it.
Fixes#5286
* fix(scheduler): keep assistant-id PR free of interval tests
Drop the six interval tests that belonged to the interval schedule PR
and fail here because this tree still only accepts once/cron.
Treat lead_agent case-insensitively so LEAD_AGENT / lead-agent store
as the default. Omit unchanged assistant_id on edit so a deleted custom
agent does not 422 unrelated PATCH (rename, reschedule).
* fix(scheduler): format task page and browser tests
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Continues the zh/en README drift follow-up (#5222 covered the LangGraph
Studio section). The English README documents Request Trace Correlation
as a standalone section between the slash-command table and LangSmith
Tracing; the zh README only mentioned the correlation id in passing
inside its Langfuse section.
Add the translated section in the same position (after the
slash-command table, before LangSmith 链路追踪), covering: X-Trace-Id
inheritance/generation and the response header, propagation to detached
runs / subagents / background memory threads, the logging.enhance
config, the deerflow_trace_id semantics (not a run id, not a provider
trace id, not a lookup key, ignored-and-overwritten when supplied by the
caller), terminal run.delivery receipts with orphan-recovery behavior,
and the loop-detection / deferred-tool-promotion run event records.
The Chinese README had drifted from the English one: the entire
"LangGraph Studio (Optional)" section (including the standalone
langgraph dev workflow, server-owned assistant provenance in that mode,
the persisted-store repair note, the authenticated-identity consumption
for custom agents/skills/uploads/memory, the /mnt/user-data/outputs
native-delivery enforcement, and the dual LangGraph streaming interfaces
for custom events) was never translated and had no zh counterpart.
Add the translated section in the same position it occupies in the
English README (after the local development walkthrough, before
进阶配置), keeping the en section structure aligned. Verified the
section heading now exists on both sides and that no relative links are
affected.
* refactor(gateway): issue request trace ids unconditionally
The request trace id was gated behind logging.enhance.enabled at every
entry point, so downstream code had to keep asking whether one existed:
a header-provenance flag in its own ContextVar, a precedence resolver,
and three-level carrier fallbacks at each consumer.
Bind one unconditionally instead. TraceMiddleware covers Gateway HTTP;
ensure_trace_context covers the entry points that never touch ASGI --
scheduled occurrences, MCP task notification runs, IM channel messages,
and the embedded client -- each scoped to one unit of work so a
long-lived worker task cannot leak one occurrence's id into the next.
The ContextVar becomes the only source; the response header, runtime
context, run metadata and log records are derived outputs.
Consumers now use ensure_trace_id() or resolve_trace_id(*carriers) and
drop their presence guards. Removed: resolve_deerflow_trace_id, the
header-provenance flag and its three helpers, set/reset_current_trace_id,
is_trace_correlation_enabled and its gateway alias.
BREAKING CHANGE: every Gateway HTTP response now carries X-Trace-Id and
it cannot be turned off; logging.enhance.enabled controls log output
only. Installations on the default enabled: false will start seeing the
header. No config keys were added or removed.
* fix(gateway): stop persisting a caller-supplied trace id on the run record
body.metadata forks two ways: through build_run_config into the live run
config, which the run worker restamps, and through create_or_reject into
the run record that the runs API echoes verbatim. Only the first was
covered, so a client sending metadata.deerflow_trace_id made the most
durable and most visible surface of a run disagree with the X-Trace-Id
and the log lines the same request produced -- a correlation id that
does not match the logs is worse than none.
Stamp the server-issued id once at the trust boundary so both forks
receive it, preserving the caller's own metadata keys. Close the same
gap on config.context, which reaches the runtime context by a separate
path: _build_runtime_context no longer merges server-owned keys from the
caller, and _install_runtime_context assigns rather than setdefaults.
A thread's metadata is no longer seeded with the run-scoped id of
whichever run created it -- one thread spans many runs and as many
trace ids.
Found by driving a real run through the Gateway and reading the run back
from the runs API; every unit test built its metadata by hand and so
could not see it.
* fix(gateway): expose X-Trace-Id to split-origin browser clients
X-Trace-Id is not on the CORS safelist, so a browser client served from
a separate origin could not read it -- and those are exactly the clients
that cannot read the Gateway's logs either, leaving them with nothing to
quote in a bug report. Same-origin nginx deployments were unaffected,
which is why this stayed hidden.
Add it to CORS_EXPOSED_HEADERS beside Content-Location, referencing
TRACE_ID_HEADER rather than repeating the literal.
* fix(gateway): keep X-Trace-Id on unhandled-exception 500s
Starlette's ServerErrorMiddleware sits outside every user middleware and
emits unhandled-exception 500s through the raw send, so those responses
never pass TraceMiddleware's header-writing wrapper. The 500 for a server
bug is exactly the response a user most needs to correlate with a log line,
and it was the one response that shipped without the id.
TraceMiddleware now tracks whether http.response.start has been sent. On an
exception with no response started it emits its own plain 500 carrying the
header, then re-raises: the outer ServerErrorMiddleware sees the response
already started and only re-raises too, so the server's exception logging is
untouched. An exception mid-stream keeps propagating unchanged — a second
response start cannot be sent, and the already-written header stands.
The trace id is printable ASCII by construction (normalize_trace_id /
generate_trace_id), which is what makes the raw latin-1 header encoding
safe.
* fix(gateway): strip the forged trace id from the persisted request echo
The run-record fix stopped a forged metadata.deerflow_trace_id on the
authoritative metadata surface, but the raw request echo still carried one:
create_or_reject persists body.config verbatim as runs.kwargs_json, which
the runs API serves back. A client posting config.context.deerflow_trace_id
therefore still got its forged value stored and echoed on one API surface
while the header, logs, run metadata, and checkpoint all carried the real
id — the id is ignored as input there, so echoing it back only manufactures
disagreement.
Two changes close it. redact_config_secrets — already the shared scrub for
that echo, applied at admission and again at serve time, so historical
records are covered too — now also drops deerflow_trace_id from
config.metadata and config.context. And build_run_config now merges run
metadata onto a copy of the caller's config["metadata"] instead of updating
it in place: the nested values of the request config are reference copies,
so the in-place merge was writing the server-stamped key through into
body.config, contaminating the "what the client sent" record before it was
persisted (and incidentally masking the forged-value echo on the metadata
container).
The regression test posts a forged id through body.metadata,
config.metadata, and config.context at once and reads the kwargs echo back
off the run record, failing if either leak returns.
* docs(harness): record the trace-echo scrub, 500 fallback, and accepted retry divergence
The trace section of the harness AGENTS.md now covers the two fixes that
close the derived-output rule (the kwargs-echo scrub in
redact_config_secrets plus build_run_config's copy merge, and
TraceMiddleware's own 500 for unhandled exceptions), and CHANGELOG gains
their Fixed entries.
It also writes down the one accepted divergence: a crash-recovered
scheduled launch reuses the durable run through its idempotency key, and
start_run returns early on idempotency_reused without restamping — so the
run record keeps the first attempt's deerflow_trace_id while the retry's
own log lines carry the freshly minted id of its ensure_trace_context
binding. The divergence is confined to the crash-recovery window and is
accepted rather than fixed: restamping on reuse would rewrite a persisted
record for a run that already exists, which is worse than two ids that each
correlate their own attempt's logs. Written down so the next reader of the
scheduler recovery path does not diagnose it as a bug.
* docs(config): align the logging.enhance schema note with the unconditional trace id
The config-module AGENTS.md still described logging.enhance as the gate for
the Gateway X-Trace-Id header and Langfuse deerflow_trace_id. That model is
gone: ids are issued unconditionally and this block decides log output only.
Left as-is, the stale wording invites an agent to "restore" a header gate it
believes was lost. Reworded to match the sibling AGENTS.md files and
config.example.yaml, with a pointer to the Request Trace Context section
that owns the full model.
* docs(changelog): link the trace entries to #5119
The five new entries pointed at the ([#XXXX]) placeholder with no reference
definition, rendering as literal text instead of a link — and RELEASING.md
step 2 relies on those references when the section becomes release notes.
All five now point at #5119, with the definition appended to the reference
block.
* refactor(harness): rename _stream_without_trace_context to _stream_turn
The name asserted the opposite of what the method now does. It was accurate
while logging.enhance.enabled could route stream() around the trace scope;
with the gate gone it is the only stream implementation left, and it binds
the id itself via ensure_trace_id(). Private, so the rename touches only the
definition and the one stream() call site.
* docs(harness): fit the trace-context guidance inside the AGENTS.md chain budget
The expanded Request Trace Context section pushed the effective AGENTS.md
chain for agents/middlewares to 99,815 bytes, past the 98,304 hard limit
scripts/check_agent_guidance.py enforces in CI (AG002). Compressed the
section from 7,359 to 4592 bytes with no facts removed: the entry-point
table, the derived-output rule and its enforcement points, the accepted
scheduled-retry divergence, the two resolution helpers, the stream()
binding rationale, the log-output-only gate, the CORS listing, the 500
fallback, and the test map all remain.
Sized against the merge, not just the branch: current main grew the same
chain by ~724 bytes, so the check was verified on the merged tree as well
(97,772 bytes; branch tree 97,048).
* fix(gateway): declare content-length on the fallback 500
The pre-response 500 declared content-type but no content-length, leaving
the framing to the ASGI server: chunked on HTTP/1.1, close-delimited on
HTTP/1.0 — the one wire difference from the ServerErrorMiddleware response
it replaces, which sends content-length: 21. The explicit header keeps the
fallback byte-identical to what clients saw before.
* docs(readme): drop the trace-correlation condition from the translations
The zh/ja/fr/ru Langfuse sections still said metadata.deerflow_trace_id
matches X-Trace-Id "when request trace correlation is enabled". The id now
always matches and that condition no longer exists, so each bullet states
the unconditional match and that logging.enhance.enabled only controls
whether the id is printed into logs — the one piece of the feature a user
can still configure.
* test(gateway): pin TraceMiddleware wiring through create_app()
Every X-Trace-Id test exercised a hand-built four-route app, so the real
stack's add_middleware(TraceMiddleware) line was pinned by nothing: deleting
it — or short-circuiting above it — passed CI while silently dropping both
the response header and the ambient id the run-record stamp and enhanced log
records derive from. One case now drives /health through create_app() and
asserts the inbound id round-trips; mutation-checked by removing the wiring
line, which fails exactly this test.
* docs(gateway): note the fallback 500 is CORS-opaque
The pre-response 500 is emitted outside CORSMiddleware — the exception has
already unwound past it — so it carries no Access-Control-Allow-Origin and
a split-origin browser client cannot read the id on this one response,
unchanged from the ServerErrorMiddleware 500 it replaces. Documented on the
class and in the CHANGELOG entry rather than fixed: replicating the origin
allowlist outside CORSMiddleware would let the two policies drift.
* fix(harness): keep abandoned-stream cleanup inside the trace binding
stream() binds the turn's id around each next(inner) and resets it before
yielding, but the finally's inner.close() ran after that binding was gone.
Abandoning the stream therefore drove the inner LangGraph generator's
GeneratorExit/finally path with no trace id — or an unrelated ambient one
from whichever context ran the close — so cancellation and finalization
logs and callbacks did not correlate with the turn they belong to.
inner.close() is now wrapped in a local bind/reset of the same turn id. The
token is set and reset in the same frame, never across a yield, so the
per-step cross-context safety is preserved even when GC closes the
generator from another Context — pinned by the existing copy_context close
test, which now exercises this path. The regression test records the id
from the inner generator's finally and fails without the binding.
* test(harness): teach the worker-trace fake about RunManager.cleanup
Upstream #5112 (bound gateway memory after terminal runs) added a
run_manager.cleanup(run_id) call to run_agent's finalization, so the
merge-commit CI run failed all five worker-trace-binding tests with
AttributeError on this PR's _FakeRunManager. The fake gains the same no-op
shape as its other methods.
* docs(gateway): bring the gateway AGENTS.md back under its soft budget
Upstream #5092 grew backend/app/gateway/AGENTS.md to 40,966 bytes, 6 over
the 40,960 soft budget that
test_agent_guidance_check.py::test_repository_guidance_stays_below_soft_budgets_and_avoids_doc_indexes
enforces — its Unit Tests run on main was cancelled by push concurrency, so
main is currently red on that test and every PR merge-run inherits the
failure. Two whitespace/wording trims in the row #5092 touched (a doubled
space, and "its configured `context_window`" → "its `context_window`")
bring the file to 40,953 with no content change.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The English Security Notice has four subsections; the Chinese one had two.
Translate the two that were missing:
- Gateway Admin Is Equivalent to Code Execution
- Deployment Defaults
Both describe deployment-time security behavior, so a stale translation
leaves Chinese-speaking operators without the loopback-default and
first-run-setup guidance that English readers get.
Verified against the code rather than translated blind:
- stdio MCP allowlist defaults to {npx, uvx} and is extended via
DEER_FLOW_MCP_STDIO_COMMAND_ALLOWLIST
(backend/app/gateway/routers/mcp.py)
- entry port publishes as ${BIND_HOST:-127.0.0.1}:${PORT:-2026}
(docker/docker-compose.yaml)
- /setup exists as the first-run admin creation route
(frontend/src/app/(auth)/setup/page.tsx)
Co-authored-by: zaoshangduziteng <309590849+zaoshangduziteng@users.noreply.github.com>
* fix(docker): create compose env files and keep Windows compose paths relative
Windows Docker reports a generic file-not-found when env_file targets are missing, or when compose paths are doubled. Make docker-start copy .env examples and invoke compose with filenames relative to docker/.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(docker): make dev compose env files optional and repair test fixture
Address review feedback on #4956.
[P1] prepare_compose_env aborted before the mocked COMPOSE_CMD in
test_compose_commands_set_deer_flow_root_before_compose, because the
temp root had no compose file or .env examples. Seed them in the
fixture so the preflight reaches the mock.
[P2] .env is gitignored, so a fresh clone has none and a direct
`docker compose -f docker/docker-compose-dev.yaml up --build` aborts on
Windows before scripts/docker.sh can help. Mark the dev env_file entries
`required: false` so a missing .env is not fatal, and document that
direct Compose must be run from the repository root.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(docker): declare Compose 2.24 floor and keep non-start commands read-only
Address the second review round on #4956.
- Document Compose >= 2.24 (CONTRIBUTING, README, compose header) and fail
early from make docker-start with an actionable message; probe both
`docker compose` and the hyphenated `docker-compose` binary.
- Document DEER_FLOW_ROOT for direct Compose callers (bash + PowerShell);
leave the variable without a $PWD fallback because PowerShell/cmd do not
export it.
- Split prepare_compose_env: compose_preflight is shared and read-only;
ensure_env_files runs only from start.
- Expand tests for version boundaries, hyphenated fallback, env-file
creation, and read-only stop/logs/restart behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(docker): reuse the probed Compose binary for wrapper operations
The version probe could accept a standalone docker-compose install while
COMPOSE_CMD stayed hardcoded to `docker compose`, so preflight passed and
start/logs/stop/restart then failed. Keep the selected executable in
COMPOSE_BIN (array), refresh COMPOSE_CMD from it in the current shell, and
extend the fallback test through an actual stop invocation.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
The Star History charts in the README files no longer render because the underlying chart service relies on GitHub stargazer data that is currently restricted. This switches the charts to a working alternative that needs no API token, updating the English, Simplified Chinese, Japanese, French, and Russian READMEs at the same time.
Co-authored-by: OctoBored <212877535+OctoBored@users.noreply.github.com>
* feat: integrate MiniMax Code as an ACP agent
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
The English README has a "Sister Projects" section (linking to the LLM
Space desktop tool) that was missing from the Chinese, Japanese, French,
and Russian translations. Add the translated section to all four so the
i18n READMEs stay in sync with the English source.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: add assistant turn branching
* fix(threads): skip workspace clone when branching from historical turn
Workspace files are not checkpointed, so cloning them onto a branch
rooted at an older assistant turn leaked files created in a later
timeline. Restrict the best-effort workspace copy to branches taken from
the latest turn; historical-turn branches now report
workspace_clone_mode="skipped_historical_turn" and keep only the
restored message history.
* style(frontend): fix prettier formatting in e2e mock-api
Collapse the branch-title normalization chain onto a single line to
satisfy the frontend lint (prettier --check) CI gate.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add Browserless web_capture screenshot tool
Add a web_capture tool that renders a page via Browserless /screenshot and
presents it through the artifact system, alongside the existing Browserless
web_fetch provider.
Hardening:
- SSRF guard: reject URLs resolving to private/loopback/link-local (incl. the
169.254.169.254 cloud-metadata endpoint)/reserved/multicast/unspecified
addresses; opt out via allow_private_addresses for internal targets.
- Surface a warning when Browserless renders a target page that itself
responded with a non-2xx/3xx status (X-Response-Code), so an error/anti-bot
page is not mistaken for valid visual evidence.
- Dedupe colliding output filenames instead of silently overwriting prior
captures.
Docs: comment out token: $BROWSERLESS_TOKEN in tool examples (an unset $VAR
fails AppConfig startup) and document allow_private_addresses.
* fix(community): format web_capture guard + document local Browserless startup
Address PR #3881 review: fix the lint-backend failure (ruff format on
browserless/tools.py) and add local Browserless startup instructions to
CONFIGURATION.md so reviewers can run the service to try web_fetch/web_capture.
* feat(scripts): add redacted community support bundle generator
Add `make support-bundle` (scripts/support_bundle.py) to help users file
high-signal, privacy-safe GitHub issues for local setup/config/runtime
problems.
The command produces:
- `*-issue-summary.md` to paste into the issue body
- `*-issue-draft.md` scaffold for AI-assisted filing (REQUIRED placeholders,
never invents repro/expected/summary facts)
- an optional evidence zip under `.deer-flow/support-bundles/` containing a
stable `triage.json` plus redacted environment/config/extensions/git/doctor
evidence
Privacy: secrets are redacted across config values, URL userinfo, query
strings, CLI flags, custom headers, bearer/sk- tokens, and home paths. The
bundle never includes `.env`, raw conversation messages, or user file
contents; optional `--thread-id` adds file manifests only. `thread_id` input
is validated against path traversal.
Wire it into the Makefile, AGENTS.md, README/README_zh, CONTRIBUTING, and the
bug-report issue template. Covered by backend/tests/test_support_bundle.py.
* fix(scripts): redact MCP env values by default in support bundle
Address PR #3886 review (willem-bd, P2): the key-name allowlist let literal
secrets under non-standard env keys (e.g. SUPABASE_SERVICE_ROLE_KEY,
R2_ACCESS_KEY, hardcoded AIza… keys) leak verbatim into the bundle that users
are told is safe to share publicly.
Mask all MCP `env` values by default, keeping only `$VAR`/`${VAR}` references
visible, and broaden SECRET_KEY_RE (access_key, pwd, private_key). Add tests
for non-keyword env secrets, broadened key names, and end-to-end zip redaction.
* feat(tui): add Hermes-like terminal workbench backed by DeerFlowClient
Implements the `deerflow` TUI from RFC #3540: a terminal-native, embedded
workbench over the existing harness (no Gateway/frontend/nginx/Docker), built
Python-native with Textual and learning UX patterns from tao-pi.
Architecture — every layer except the Textual app is pure and unit-tested:
- view_state.py: ViewState + reduce(state, action), the testable heart
- runtime.py: StreamEvent -> reducer actions (pure translate + threaded driver)
- message_format / command_registry / input_history / render / theme: pure
- app.py: Textual App; runs the sync DeerFlowClient.stream() on a worker thread
and marshals actions back to the UI thread. Slash command palette, model and
thread modal pickers, ↑/↓ history, Ctrl+C interrupt, TTY-aware fallback.
- cli.py: pure launch-mode planning + headless --print/--json + `deerflow`
console script (textual is an optional [tui] extra; degrades to headless help)
Web UI visibility (the RFC's key decision): persistence.py writes a threads_meta
row under the local default user into the same DB the Gateway reads, so terminal
sessions appear in the Web UI sidebar without running the Gateway. Best-effort,
no-op on the memory backend; all DB work on one long-lived background loop.
Tests: 95 TUI tests — pure layers via pytest, app/palette/overlays via Textual's
pilot harness with a fake session, and a threads_meta read/write round-trip.
ruff clean; respects the harness->app import boundary. Docs: backend/docs/TUI.md
plus CLAUDE.md/README updates and preview screenshots.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): de-duplicate streamed assistant text and tool cards; keep Tab in composer
Self-test surfaced three issues, all root-caused to consuming non-strict
streaming from DeerFlowClient (proven by the client's own
test_dedup_requires_messages_before_values_invariant, which shows the client can
re-emit a message id's full content twice):
- Assistant text was doubled (e.g. "answer answer") because the reducer blindly
concatenated same-id deltas. Now merges by content: a re-send or cumulative
snapshot replaces; only genuine increments append.
- Tool activity showed duplicate and empty "gear" cards from partial/re-emitted
tool-call chunks. ToolStarted now de-dupes by tool_call_id, drops id-less
noise chunks, and fills the name on a later chunk; a tool result with no prior
card still surfaces as a completed card.
- Tab moved focus off the composer to the scroll region (felt like broken cursor
logic). Tab is now consumed by the composer (completes a command when the
palette is open, no-op otherwise).
Adds reducer tests for each case plus a Tab-focus test; 102 TUI tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): make Esc interrupt an active run (matches the status hint)
The status line advertised "esc interrupt" but Esc was only wired to close the
slash palette, so it did nothing during a run. Esc now: closes the palette when
open, interrupts the active run when streaming, and is a no-op when idle. The
interrupt logic is shared with Ctrl+C via _interrupt_run(). Adds a regression
test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): stop prior answers duplicating on threads with history
On a thread with history, DeerFlowClient re-emits every prior message on each
new turn (its streamed_ids dedup is per-stream-call), and a re-emitted older
message can arrive after a newer message has already started. The reducer only
matched the *most recent* assistant row by id and otherwise appended, so each
re-emitted older answer was duplicated verbatim at the end of the transcript.
Match an assistant row by id anywhere in the transcript and merge in place.
Tool cards already de-dupe by call id globally, so they were unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): correct CJK cursor drift in the composer
Confirmed a Textual Input bug (latest 8.2.7): Input._cursor_offset adds an
unconditional +1 at the end of the value, overshooting by one cell after
double-width (CJK) characters. That misplaces the hardware/IME cursor — the
drift seen when typing Chinese in iTerm2 (the on-screen block cursor, drawn
separately in render_line, is fine; English doesn't use an IME so it looks
correct). Reproduced with a bare Input, so it's upstream, not our layout.
Add ComposerInput(Input) overriding _cursor_offset to the true cell position and
use it for the composer. Numeric tests pin the CJK end/mid and ASCII cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(tui): render finalized assistant messages as Markdown
The transcript showed raw Markdown (literal **bold**, ## headings, - lists,
links). Finalized assistant messages now render as Rich Markdown — headings,
bold/italic, lists, inline code + code blocks, blockquotes, horizontal rules and
links — with the ● speaker marker aligned to the top of the body.
The actively-streaming message stays plain text so partial Markdown doesn't
reflow/jump, then snaps to its rendered form when the run ends. Transcript
re-renders are coalesced on a ~60ms timer (dirty flag) so per-token Markdown
re-parsing stays smooth on long threads. Tests cover both the rendered and the
streaming-plain paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style(tui): apply ruff format
CI lint runs `ruff format --check` via uvx (latest ruff); apply the formatter so
the lint-backend job passes. No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(tui): address code-quality review comments
From github-code-quality[bot] on #3760:
- runtime.py: give the `_ClientLike` Protocol method a docstring body instead of
a bare `...` (flagged as a no-effect statement), matching the harness
convention for Protocol stubs (e.g. SafetyTerminationDetector).
- test_tui_cli_main.py: drop the unnecessary `lambda: _FakeSession()` wrappers in
monkeypatch.setattr; pass `_FakeSession` directly (same behavior).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): keep history Markdown-rendered when a follow-up run starts
Previously the transcript rendered "the last assistant row" as plain text while
streaming. But when a follow-up turn starts, the last assistant row is the
*previous, finalized* answer until the new message begins — and the client
re-emits prior messages early in the turn — so sending a follow-up reverted the
previous answer from rendered Markdown back to raw text.
Track the actively-streaming message id in ViewState instead: it's reset on
RunStarted, set only when an AssistantDelta actually adds new content (history
re-emits are no-ops and don't mark it), and cleared on RunEnded. The renderer
keeps only that one message plain; all history stays Markdown.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(readme): add Terminal Workbench (TUI) section to root README
Mention the new `deerflow` TUI alongside the Embedded Python Client in the root
README.md and README_zh.md (install, launch/headless commands, feature summary,
Web UI visibility), with a ToC entry and a preview screenshot. Links to
backend/docs/TUI.md for the full guide.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): address review feedback (willem-bd)
Ten findings from the TUI code review:
1. /resume was dead-ended — registered + in /help + tested as a builtin, but no
dispatch branch. Wired it to thread resolution / the switcher.
2. --resume <title> was forwarded raw into the checkpointer (blank thread).
Added Session.resolve_ref() to resolve id-or-title via list_threads; used by
--resume and /resume.
3. str(get("id","")) returned "None" for an explicit id:None (truthy), defeating
the empty-id guard so unrelated null-id tool calls collapsed into one card.
Coerce via a None-safe helper.
4. Headless --print/--json no longer spin up the persistence loop/engine/pool
(open_session(persistence=False)).
5. _LoopThread + engine are now closed: Session.close() (dispose engine + stop
loop) called from a try/finally around app.run().
6. --cli --continue (and piped --cli) now run headless instead of erroring.
7. Cancelled runs no longer persist a truncated title (guard on _cancelled).
8. Palette highlight resets to the top when the filter set changes.
9. Dropped the never-populated tools count from the header.
10. Documented the `not row.error` merge guard.
Adds regression tests for each; 126 TUI tests pass, ruff check + format clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The Official Website hero image and the Coding Plan banner referenced
deleted GitHub user-attachment assets that now 404. No working
replacement exists, so remove them; section headings and prose/bullet
links already cover the same destinations.
Closes#3715
* fix(harness): resolve runtime paths from project root
* docs(config): update
* fix(config): address runtime path review feedback
* test(config): fix skills path e2e root
* test(config): cover legacy config fallback when project root lacks config files
Verifies that when DEER_FLOW_PROJECT_ROOT is unset and cwd has no
config.yaml/extensions_config.json, AppConfig and ExtensionsConfig fall back
to the legacy backend/repo-root candidates — the backward-compat path
requested in PR #2642 review.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(channels): add DingTalk channel integration
Add a new DingTalk messaging channel using the dingtalk-stream SDK
with Stream Push (WebSocket), requiring no public IP. Supports both
plain sampleMarkdown replies and optional AI Card streaming for a
typewriter effect when card_template_id is configured.
- Add DingTalkChannel implementation with token management, message
routing, allowed_users filtering, and markdown adaptation
- Register dingtalk in channel service registry and capability map
- Propagate inbound metadata to outbound messages in ChannelManager
for DingTalk sender context (sender_staff_id, conversation_type)
- Add dingtalk-stream dependency to pyproject.toml
- Add configuration examples in config.example.yaml and .env.example
- Update all README translations with setup instructions
- Add comprehensive test suite (test_dingtalk_channel.py) and
metadata propagation test in test_channels.py
- Update backend CLAUDE.md to document DingTalk channel
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(channels): address PR review feedback for DingTalk integration
- Replace runtime mutation of CHANNEL_CAPABILITIES with a
`supports_streaming` property on the Channel base class, overridden
by DingTalkChannel, FeishuChannel, and WeComChannel
- Store stream client reference and attempt graceful disconnect in
stop(); guard _on_chatbot_message with _running check to prevent
post-stop message processing
- Use msg.chat_id as the primary routing key in send/send_file via
a shared _resolve_routing helper, with metadata as fallback
- Fix process() return type annotation from tuple[str, str] to
tuple[int, str] to match AckMessage.STATUS_OK
- Protect _incoming_messages with threading.Lock for cross-thread
safety between the Stream Push thread and the asyncio loop
- Re-add Docker Compose URL guidance removed during DingTalk setup
docs addition in README.md
- Fix incomplete sentence in README_zh.md (missing verb "启用")
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(docs): restore plain paragraph format for Docker Compose note
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(channels): fix isinstance TypeError and add file size guard in DingTalk channel
Use tuple syntax for isinstance() type check to avoid runtime TypeError
with PEP 604 union types. Add upload size limit (20MB) before reading
files into memory. Narrow exception handlers to specific types.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(channels): propagate markdown fallback errors and validate access token response
- Re-raise exceptions in _send_markdown_fallback to prevent partial
deliveries (files sent without accompanying text)
- Validate _get_access_token response: reject non-dict bodies, empty
tokens, and coerce invalid expireIn to a safe default
- Add tests for both fixes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(channels): validate upload response and broaden send_file exception handling
- Validate _upload_media JSON response: handle JSONDecodeError and
non-dict payloads gracefully by returning None
- Broaden send_file exception tuple to include TypeError and
AttributeError for unexpected JSON shapes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(channels): fix streaming race on channel registration and slim outbound metadata
- Register channel in service before calling start() to avoid race
where background receiver publishes inbound before registration,
causing manager to fall back to static CHANNEL_CAPABILITIES
- Strip known-large metadata keys (raw_message, ref_msg) from outbound
messages to prevent memory bloat from propagated inbound payloads
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Update service.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update CLAUDE.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* windows check and dev fixes
* fix windows startup scripts
* fix windows startup scripts
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The lark-oapi SDK defaults to open.feishu.cn (China), but apps on the
international Lark platform (open.larksuite.com) fail to connect with
error 1000040351 'Incorrect domain name'.
Changes:
- Add 'domain' config option to feishu channel (default: open.feishu.cn)
- Pass domain to both API client and WebSocket client
- Update config.example.yaml and all README files
