Part of #3742. RunJournal._message_text and the gateway thread-messages
helper (thread_runs._message_text) reimplemented the same 'extract display
text from a message' logic — str / list of string|{text}|nested{content}
blocks joined without a separator / mapping with text|content key. They
differed only in two ways: journal reads a BaseMessage attribute while
thread_runs reads dict-shaped run_events rows, and journal falls back to
message.text.
Add deerflow.utils.messages.message_to_text(message, *,
text_attribute_fallback=False) that handles both message shapes (attribute
or mapping content access) and gates the .text fallback behind a flag, and
have both call sites delegate. journal passes text_attribute_fallback=True;
thread_runs uses the default. Behavior is unchanged at both sites.
Verified behavior-preserving with an equivalence harness running both
original implementations vs the shared helper over 98 inputs (BaseMessage
and dict messages; str/list/mapping/None/numeric content; mixed blocks;
.text attribute present/absent/non-str) -> 0 mismatches. Added
tests/test_utils_messages.py; the journal last_ai_message extraction tests
still pass.
These three module-private helpers have no callers anywhere in the repo
(verified by a repo-wide identifier scan + direct greps; each appears only
at its own definition):
- agents/memory/updater.py: _create_empty_memory (a no-caller wrapper around
storage.create_empty_memory, which is used directly elsewhere)
- runtime/runs/worker.py: _extract_human_message (also drops the now-unused
HumanMessage TYPE_CHECKING import and TYPE_CHECKING from typing)
- sandbox/tools.py: _looks_like_unsafe_cwd_target (cwd safety is enforced via
_is_allowed_local_bash_absolute_path)
Underscore-prefixed = module-internal, so zero references is conclusive.
Route handlers and @compiles-registered functions were excluded as false
positives. Public-surface candidates are noted in the issue, not touched here.
Fixes#3748.
DynamicContextMiddleware (PR #3630, p0) uses the ID-swap technique to
inject a SystemMessage(reminder) into the middle of the conversation.
create_agent then prepends the static system_prompt as another
SystemMessage at request time, so strict OpenAI-compatible backends
(vLLM, SGLang, Qwen) and Anthropic reject the request with
'System message must be at the beginning'.
New SystemMessageCoalescingMiddleware runs in wrap_model_call — after
create_agent prepends system_prompt — and merges every SystemMessage
into a single leading one before the request reaches the provider.
Non-system messages keep their original order; the merged SystemMessage
preserves the id of the first system message. Only the request payload
is touched; checkpoint state is unchanged, so every consumer that scans
history (memory builder, journal, summarization, dynamic-context
detection) keeps working.
Mirrors the per-request coalescing already done for Claude in
claude_provider._coalesce_system_messages (PR #3702) but at a
provider-agnostic layer so every backend benefits from a single fix.
Closes#3707
* fix(middleware): fix positional fallback consuming unrelated todo when same-content list is exhausted
When next_todos contains the same content string twice (e.g. two "A"
entries), the first iteration pops the matching previous todo from
previous_by_content["A"], leaving an empty list. The second iteration
finds that empty list, treats it as falsy, and sets previous_match=None.
The positional fallback then fires unconditionally — consuming
previous_todos[index] even if it holds a completely different entry
(e.g. "B"). That marks "B" as matched, so the final loop never emits
a todo_remove action for it.
Fix: guard the positional fallback with `content not in previous_by_content`
so it only fires for genuinely new content (the key was never in previous),
not for same-content entries whose list has simply been exhausted.
Add a regression test to _build_todo_actions covering this exact case.
* style: ruff-format the token usage middleware test
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
get_model_config / get_tool_config / get_tool_group_config did a next(...)
linear scan of self.models / self.tools / self.tool_groups on every call.
These sit on hot paths: get_tool_config runs 2-3x per community-tool
invocation (web_search etc.) and get_model_config several times per agent
build, across 40+ call sites.
Build name -> config dicts once in a mode="after" validator (stored as
PrivateAttr), so each getter is an O(1) dict lookup. A config reload
constructs a fresh AppConfig, which rebuilds the indexes; setdefault keeps
first-match-wins on duplicate names, matching the prior next(...) semantics.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MemoryStreamBridge._resolve_start_offset scanned the retained event buffer
(up to queue_maxsize=256 entries) on every subscribe/reconnect carrying a
Last-Event-ID. Event ids are "{ts}-{seq}" where seq is a per-run monotonic
counter that equals the event's absolute offset, so the offset is computable
arithmetically. Parse seq, index into the buffer, and verify the id matches
exactly -- a stale/evicted/foreign/malformed id falls back to
replay-from-earliest, identical to the previous scan.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Middleware-injected hidden messages (TodoMiddleware.todo_reminder,
ViewImageMiddleware, p0 DynamicContextMiddleware.__memory) were being fed
to the memory-updating LLM as if they were real user input, polluting
long-term memory with framework-internal text. The p0 __memory payload
could also trigger a self-amplification loop.
Fix: skip HumanMessages with additional_kwargs['hide_from_ui'] in
filter_messages_for_memory, consistent with the frontend logic.
Closes#3695
mask_local_paths_in_output runs once per glob/grep match (tools.py:1540,
:1625) and once per bash/ls output, and each call rebuilt every skills /
ACP / user-data masking regex from scratch — Path.resolve() syscalls +
re.escape + re.compile. A grep returning up to 100 matches recompiled the
same patterns 100x.
Compile the host->virtual patterns once per ordered source set via
functools.lru_cache (keyed on the config-stable + per-thread (host,
virtual) pairs, so it self-invalidates when they change) and collapse the
three byte-identical replacer blocks into one applier. Behavior is
unchanged: same patterns, same application order, same per-thread mappings.
Verified behavior-preserving with an equivalence harness (original vs
refactored logic over 54 inputs incl. slash-style variants, base-only
matches, subpaths, overlapping mappings, missing skills/ACP, thread_data
None) -> 0 mismatches. Added cache/consistency tests alongside the
existing masking tests.
Fixes#3712.
* feat(memory): add guaranteed injection for correction facts with graceful fallback
When the token budget is tight, high-value facts (e.g. user corrections)
can be silently evicted by lower-priority regular facts. This change:
- Introduces configurable 'guaranteed_categories' (default: [correction])
whose facts draw from a separate 'guaranteed_token_budget', ensuring
they are never dropped due to budget pressure.
- Adds a graceful fallback to confidence-only ranking when the
guaranteed-category path raises an unexpected exception.
- Refactors fact selection into a header-agnostic helper
(_select_fact_lines) with explicit token accounting in the caller,
eliminating double-counting of separators.
- Emits a single 'Facts:' header regardless of whether both guaranteed
and regular facts are present.
- Extends the final safety truncation limit to account for the
additional guaranteed budget so guaranteed facts survive end-to-end.
* refactor(memory): address review feedback on guaranteed injection
- Restore strict break-on-overflow in `_select_fact_lines` to preserve
the caller's confidence-ordered ranking; add a regression test locking
in the invariant that a shorter lower-confidence fact never slips
ahead of a skipped higher-confidence one.
- Account for the inter-group `\n` separator between guaranteed and
regular fact blocks in the regular budget (1-token precision fix).
- Clarify docstrings on `format_memory_for_injection` and
`MemoryConfig.guaranteed_token_budget` to distinguish the common
*displacement* case (total stays within `max_tokens`) from the rarer
*additive* case (safety-truncation ceiling raised when guaranteed
lines alone would overflow).
* fix(memory): address P1 safety truncation + P2s from review
- Structure-aware safety truncation: Facts block is now a protected
suffix so guaranteed-category facts can never be silently discarded
by a prefix-cut on overflow. Only the preceding (user/history)
sections are eligible for truncation.
- Extend the same protected-suffix treatment to the except/fallback
path by returning fact lines alongside the formatted section from
_fallback_format_facts, avoiding string parsing.
- Single inter-section separator: facts section no longer embeds its
own leading \n\n; the final "\n\n".join(sections) is the single
source of truth for section-to-section spacing.
- Bare string for guaranteed_categories now raises TypeError instead
of silently iterating single characters.
- Category-less / malformed facts no longer default-promote into the
guaranteed "context" pool — only facts with an explicit category
field qualify.
- Lift valid_facts pre-filter outside the try so the fallback path
reuses it instead of re-doing validation work.
- MemoryConfigResponse + DeerFlowClient.get_memory_config now expose
guaranteed_categories / guaranteed_token_budget.
- config.example.yaml: document the two new fields and bump
config_version from 12 to 13.
- Add regression tests for every finding.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Atlas Cloud (https://atlascloud.ai) exposes a single OpenAI-compatible
endpoint in front of many open models (DeepSeek, Qwen, Kimi, GLM,
MiniMax, Llama, ...). It needs no new provider code — it uses the same
ChatOpenAI + base_url pattern already documented for OpenRouter, Novita
and other gateways.
Add a commented example to config.example.yaml: a plain ChatOpenAI entry
plus a PatchedChatOpenAI variant for *-thinking model ids so
reasoning_content is replayed across multi-turn tool calls. The key is
read from the ATLASCLOUD_API_KEY environment variable via the $VAR form,
consistent with the other examples.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
_lock was defined in __init__ and used only in reset(), leaving _apply(),
before_agent(), _clear_run_state(), and _drain_pending_warnings() unprotected.
Concurrent after_model callbacks for the same run_id could race on
usage_accum.input += diff_input, causing lost-update on token counters.
Also fix stray bare 'a' character in frontend/src/content/zh/introduction/index.mdx
that rendered as visible text on the documentation page.
Co-authored-by: kapilvus <kapilvus@gmail.com>
The Official Website hero image and the Coding Plan banner referenced
deleted GitHub user-attachment assets that now 404. No working
replacement exists, so remove them; section headings and prose/bullet
links already cover the same destinations.
Closes#3715
* test(agents): multi-turn message-stream invariants (graph integration)
Add a graph-integration net for the class of bug behind #3684: build a real
create_agent graph with DynamicContextMiddleware plus a checkpointer, drive two
user turns on one thread with a deterministic fake model and memory injection
stubbed on, then assert the message stream stays well-formed -- the newest user
message is the latest human turn, no duplicate ids, no __user__user suffix
explosion.
Runs at unit speed in `make test` (backend-unit-tests), with no gateway, SSE,
fixtures, or API key. Verified red on the pre-fix middleware and green after.
Catches this class earlier than e2e replay, which disables memory, uses a
single-turn golden, and asserts SSE shape only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(agents): address review — semantic-first asserts, _STREAM_MIDDLEWARES, DRY
- Check the semantic invariant (newest user message is the latest human turn)
before the structural id checks, so a regression surfaces as a meaning-level
failure; verified it now fails first on the pre-#3685 middleware.
- Add module-level _STREAM_MIDDLEWARES (the docstring referenced it; it did not
exist) so the net is trivially widened with more state-touching middlewares.
- _last_human_text reuses _msg_text instead of re-implementing content flattening.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(agents): skip dateless reminders in dynamic-context date scan
_last_injected_date returned at the first message flagged
dynamic_context_reminder regardless of whether it carried a
<current_date>. Since #3630 split injected context into a date
SystemMessage plus a separate dateless <memory> HumanMessage, the
reverse scan hit the memory message first and returned None, making a
later turn look like the first turn. The middleware then re-injected,
re-targeted the previous turn's __user message via the ID swap, and left
the stale message as the latest human turn -- so the model re-answered
the previous message and the persisted message stream got scrambled.
Skip flagged reminders without a <current_date> so the scan reaches the
real date SystemMessage. Add a regression test for the 2nd-turn +
memory-enabled scenario.
Fixes#3684
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(agents): make injected reminder date authoritative via metadata
Promote the dynamic-context date out of message content into
additional_kwargs["reminder_date"] on the date SystemMessage.
_last_injected_date reads that key instead of regex-parsing content, with a
SystemMessage-scoped content fallback for checkpoints written before the key
existed.
This removes two structural fragilities raised in review of #3685:
- the boolean dynamic_context_reminder flag no longer has to also mean
"carries a date" -- dateless reminders simply lack reminder_date and are
skipped, so a future reminder type cannot reproduce the shadowing bug.
- the date regex no longer runs on the user-influenceable memory HumanMessage,
so a memory fact containing a literal <current_date> cannot spoof the
injected date (false same-day skip / false midnight crossing).
Add regression tests for memory date-spoofing, structured-date stamping, and
legacy (no-key) detection; update existing date-reminder fixtures to the
production shape.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
_aexecute collects AI messages from agent.astream(stream_mode="values"),
which re-yields the full state every super-step. The duplicate check rescanned
the append-only ai_messages list on every chunk -- any(m["id"] == message_id) --
so a run with M messages did O(M^2) work, and M reaches max_turns=150 for the
general-purpose / deep-research subagent.
Track an id-keyed set alongside ai_messages: id-bearing messages become O(1)
set lookups, and the id-less full-dict-compare fallback is preserved. Behavior
is unchanged.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(config): coerce null object config sections to their defaults
#3434 made commented-out list sections (models/tools/tool_groups) parse as []
instead of crashing, but the same scenario still crashes for object sections:
commenting out every key under e.g. memory: / summarization: / guardrails:
makes PyYAML parse the value as None, and AppConfig then raises "Input should
be a valid dictionary" for that section — breaking the documented
`cp config.example.yaml config.yaml` first-run flow.
Generalize the handling: a model_validator(mode="before") drops None-valued
sections so each field falls back to its default (list sections -> [], object
sections -> their default config). This subsumes the previous list-only
field_validator and the database special-case. Required sections without a
default (sandbox) still error when null.
Adds test_app_config_coerces_commented_out_object_sections; the existing
list-section regression test still passes.
* docs+test: address review on null-section coercion
- Correct the _drop_null_config_sections docstring: it does not subsume the
database special-case; _apply_database_defaults still owns `database` and
applies concrete defaults beyond null-coercion.
- Strengthen test_app_config_coerces_commented_out_object_sections to assert
each null section falls back to its expected default config type, not just
that it is non-None.
- Add test_app_config_null_required_section_still_errors covering the
required-section (sandbox) "still errors when null" claim.
---------
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
list_thread_messages matched event_type == "ai_message" to find each
run's last AI message, but RunJournal stores AI messages as
"llm.ai.response" and the event store returns that verbatim. No code
writes "ai_message", so the match never hit: feedback was never attached
(every message returned feedback=null) and the grouped-feedback query ran
on every request for nothing.
Match "llm.ai.response", and only run the grouped-feedback query when the
thread actually has an AI message to attach it to. Adds a regression test
for the per-run attachment and the no-AI-message lazy-query path.
Fixes#3650.
P0: DynamicContextMiddleware role isolation — framework data as SystemMessage,
memory as HumanMessage (OWASP LLM01), user input unchanged.
Includes E2E hash stabilization: frontend hide_from_ui filter fix in
suggest_agent, backend _canonical_messages() hide_from_ui skip, fixture
hash updates.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Add a uv lock --check guard in two places so a stale uv.lock cannot be
committed unnoticed (as happened with the groundroute extra):
- pre-commit: a local uv-lock-check hook, scoped to the backend
pyproject.toml files and uv.lock.
- CI: a "Check uv.lock is in sync" step in the lint-backend job, run
before uv sync so the install step cannot mask a stale lock by
regenerating it.
The deerflow-harness pyproject.toml declares a groundroute optional
extra (added in #3675), but uv.lock provides-extras was left out of
sync. Regenerate the lock so the declared extra is recorded.
The login page used hardcoded English strings. Add a `login` section to
the i18n Translations interface and both locale files, then wire the
login page to `useI18n()` so all titles, labels, placeholders, buttons,
SSO hints, and error messages resolve from the active locale.
init_engine runs on the FastAPI lifespan event loop, but created the SQLite
data directory with a synchronous os.makedirs (a stat + mkdir syscall),
blocking startup. Dispatch it via asyncio.to_thread, mirroring the #1912 fix
for the checkpointer's ensure_sqlite_parent_dir.
Adds a Blockbuster-gated regression test in tests/blocking_io/ that drives the
real init_engine path with a not-yet-existing sqlite_dir; it trips
BlockingError if the makedirs regresses onto the event loop.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
* feat(groundroute): add GroundRoute community web_search + web_fetch tools
GroundRoute is a meta search layer over six engines (Serper, Brave, Exa,
Tavily, Firecrawl, Perplexity) with price-based routing and failover. This
adds a self-contained community engine module (httpx only, no new required
deps) mirroring community/brave + community/tavily:
- web_search: POST /v1/search, normalize to {title,url,snippet,source_engine}.
- web_fetch: fetch a URL via mode=page.
- unit tests covering normalization, auth, clamping, and graceful errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(groundroute): register GroundRoute search + fetch in wizard and config
Add GroundRoute to the setup wizard provider lists (SEARCH_PROVIDERS +
WEB_FETCH_PROVIDERS) and as commented web_search + web_fetch examples in
config.example.yaml, mirroring tavily/serper/brave so SEARCH_API can select it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(groundroute): apply repo ruff format (line-length 240)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(groundroute): add GroundRoute to tools docs and config reference
Adds GroundRoute as a web_search and web_fetch option in the en + zh
tools.mdx pages (new tab alongside Tavily/Brave/Exa/etc.) and documents
GROUNDROUTE_API_KEY in CONFIGURATION.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(groundroute): define empty groundroute extra for clean install
The docs install line 'uv add deerflow-harness[groundroute]' (mirroring the
tavily/exa/firecrawl pattern) referenced an undefined extra, which uv accepts
but warns about. GroundRoute needs no extra packages (httpx is a core dep), so
declare an empty 'groundroute' extra in deerflow-harness optional-dependencies
so the documented command resolves without a warning.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(groundroute): per-tool api key + honor caller max_results (review)
Address maintainer review on PR #3675:
- _get_api_key(tool_name): web_fetch now reads the web_fetch config block's key
instead of always web_search, so a flow that pairs GroundRoute fetch with a
different search engine authenticates correctly. Mirrors serper/exa/firecrawl.
- web_search honors a caller-supplied max_results (sentinel default None),
falling back to the configured value only when omitted, so the documented
parameter is no longer silently discarded.
- warn-once is now keyed per tool. Tests cover both fixes (web_fetch key,
agent max_results honored).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): reset new chat on client-side navigation
Drive the chat reset effect with Next.js's reactive pathname instead of the render-time window.location-derived flag.
During App Router transitions, window.location may still point to the previous thread until commit, leaving chat state stale until another UI interaction triggers a render. Preserve the stale 'new' param guard so created thread UUIDs are not overwritten.
* test(frontend): add e2e to cover history-only new chat reset
* docs(frontend): clarify new chat pathname synchronization
* feat: persist AI turn duration in backend and UI
* chore: restore uv.lock to match main
* refactor(frontend): use Math.floor instead of Math.ceil for reasoning timer
Follow-up to #3301 / #3344. The Gateway-embedded runtime is the standard
topology — there is no standalone LangGraph service — but two stale
references slipped past the cleanup guard:
- .github/copilot-instructions.md still told contributors that `make dev`
"Starts LangGraph (2024)" and wrote logs/langgraph.log.
- SandboxAuditMiddleware's docstring pointed audit logs at langgraph.log.
Update both to the Gateway-embedded reality (gateway.log) and extend
test_gateway_runtime_cleanup.py to pin the agent-instruction docs so the
standalone references can't reappear.
Stays within the safe scope of #3304: does not touch langgraph_auth.py,
langgraph.json, or the langgraph-api / langgraph-cli / langgraph-runtime-inmem
deps, which the issue gates on maintainer confirmation of Studio / direct
LangGraph Server support.
Dependabot security update #3620 bumped cryptography to 48.0.1 in uv.lock
but left the harness manifest at >=43.0.0, since the dependency lives in
the workspace member packages/harness, not backend/pyproject.toml. The
drift caused 'uv lock' to keep rewriting the recorded specifier. Align the
source constraint with the locked version.
