* feat(persistence): wire alembic migrations + bootstrap schema on startup
Closes#3682. Pre-#3658 DBs lack the `runs.token_usage_by_model` column
because alembic was never wired up — startup only ran `create_all`,
which never ALTERs existing tables.
Adds a hybrid bootstrap in FastAPI lifespan (replaces bare `create_all`):
- empty DB → create_all + stamp head
- legacy DB → stamp 0001_baseline + upgrade head
- versioned DB → upgrade head
Concurrency: Postgres `pg_advisory_lock` (cross-process); SQLite
per-engine `asyncio.Lock` + 30s `PRAGMA busy_timeout` on both prod and
alembic engines. Column revisions use `safe_add_column` /
`safe_drop_column` idempotent helpers as fallback.
Other bits:
- 0001 baseline (chain root) + 0002 add `runs.token_usage_by_model`
- `include_object` filter so alembic ignores LangGraph checkpointer tables
- `make migrate-rev MSG="..."` for authoring new revisions
(no migrate/stamp targets — startup is the only execution path)
- Tests: three-branch decision, concurrency, #3682 regression, env
filter, blocking-IO gate anchor
- CLAUDE.md: new "Schema migrations" section
* fix(style): fix lint error
* perf(persistence): address review feedback on alembic bootstrap
Behavioural fixes
- _SQLITE_LOCKS now keyed via WeakKeyDictionary so id-reuse after GC
cannot return a stale, loop-bound lock and the cache cannot leak one
entry per disposed engine.
- safe_add_column compares nullable / server_default against the desired
column when the name already exists and emits a warning on drift,
surfacing manual-ALTER workarounds instead of silently no-op'ing.
- _postgres_lock issues SET LOCAL idle_in_transaction_session_timeout=0
before pg_advisory_lock, so managed Postgres cannot kill the idle
lock-holding session mid-upgrade and silently release the advisory
lock.
- legacy branch now backfills missing baseline tables via a restricted
create_all (Base.metadata.create_all scoped to _BASELINE_TABLE_NAMES).
Restores pre-#1930 upgraders whose channel_* tables were never
provisioned, without pre-empting future create_table revisions for
newly-added models.
Schema parity
- runs.token_usage_by_model gains server_default=text("'{}'") in both
the ORM model and the 0001_baseline create_table, matching what 0002
adds via ALTER. create_all and alembic-upgrade paths now produce
identical column definitions.
- New parity test compares Base.metadata.create_all output against a
pure alembic upgrade base->head, asserting column-set, nullable, and
server_default agree across all tables (normalized through the same
helper safe_add_column's drift check uses).
Guards
- test_baseline_table_names_constant_matches_0001 pins
_BASELINE_TABLE_NAMES to 0001_baseline.upgrade()'s actual output --
the constant cannot drift silently when someone edits 0001.
- test_legacy_backfill_skips_non_baseline_tables verifies the restricted
backfill does not create a phantom table on Base.metadata, modelling
a future revision that would otherwise collide on op.create_table.
Doc residuals
- Three-branch decision table is now consistent across bootstrap.py
top docstring, engine.py comment, test module docstring, and
CLAUDE.md.
- Stale test anchor in blocking_io/test_persistence_engine_sqlite.py
docstring now points at the real file.
* fix(style): fix lint error
* fix(persistence): close drift detection holes
- _check_column_drift compares column type via a family equivalence
allowlist ({JSON, JSONB}). Catches the wrong-type workaround
`TEXT NOT NULL DEFAULT '{}'` that previously slipped through silently,
while keeping Postgres JSON/JSONB dialect reflection quiet. Reflected
and desired type are also echoed in every drift warning's payload for
operator triage.
- Extract _escape_url_for_alembic so bootstrap._alembic_safe_url and
scripts/_autogen_revision share the ConfigParser % escape rule
instead of duplicating it.
- backend/README.md: add `make migrate-rev MSG=...` to Commands and a
Schema Migrations section per the repo's README/CLAUDE.md sync policy.
- test_base_to_dict.py: scope the test ORM class to an isolated MetaData
so the create_all-vs-alembic parity test (added in the previous
commit) is not polluted by the phantom table on the full pytest
session.
* perf(runtime): index MemoryRunStore by thread_id to avoid O(n) scans
MemoryRunStore is the default run backend (database.backend=memory) and backs
RunManager.list_by_thread, which calls it on every thread-runs query to hydrate
persisted runs. list_by_thread scanned every run in the store (O(total runs))
to filter by thread_id, so listing one thread's runs got linearly slower as
unrelated runs accumulated across all threads.
Add a thread_id -> insertion-ordered run_id set secondary index, maintained in
lockstep with _runs in put()/delete(), and use it in list_by_thread for an
O(runs-in-thread) lookup. This mirrors the index RunManager already keeps over
its own in-memory records (#3499); the store extraction — whose docstring notes
it is "Equivalent to the original RunManager._runs dict behavior" — did not
carry the index across.
Behavior is unchanged: same user_id filtering, newest-first ordering, and limit.
The store runs each method without awaits on the event loop, so the index and
_runs stay consistent without a lock.
Extends tests/test_persistence_scaffold.py::TestMemoryRunStore with coverage for
unknown-thread, newest-first ordering, limit, and index cleanup on delete
(including empty-bucket removal).
* perf(runtime): route aggregate_tokens_by_thread through the thread index too
list_by_thread already uses the _runs_by_thread index this PR adds, but
aggregate_tokens_by_thread (the /token-usage endpoint) still scanned every run
in the process to pick out one thread's runs. Route it through the same index
for an O(runs-in-thread) lookup, completing the thread-scoped read coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
get_artifact ran its filesystem work directly on the event loop: virtual-path
resolution (os.path.abspath via .resolve()), exists/is_file probes, MIME sniffing
(mimetypes lazily stats the system MIME DB on first use), full-file
read_text/read_bytes, is_text_file_by_content (open+read), and .skill ZIP
open+extract. So serving any artifact blocked the loop for the whole read;
`make detect-blocking-io` flagged it. Same class as #3457 / #3529.
Offload each branch's IO via asyncio.to_thread: one sync helper per branch
(_load_skill_archive_member, _read_artifact_payload) folds stat + MIME + read /
extract into a single worker hop and returns a small (kind, mime, payload) plan
the handler turns into the response on the loop. FileResponse (download / active
content) keeps streaming the file itself. Behavior, branching, error codes, and
security boundaries are unchanged.
Add tests/blocking_io/test_artifacts_router.py anchor (text / binary / .skill
member), verified red->green under the strict Blockbuster gate. The gate also
caught a blocking call the static scan missed: resolve_thread_virtual_path's
.resolve() (os.path.abspath), now offloaded too.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(channels): let UI runtime channel config win over config.yaml
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* test(channels): update assertion to expect runtime/UI value to win over yaml
The test was written when yaml took precedence over runtime config.
This PR inverts that precedence so UI-entered credentials win; the
assertion now correctly reflects that runtime value (xapp-ui) beats
the yaml value (xapp) on a shared key.
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Part of #3742. RunJournal._message_text and the gateway thread-messages
helper (thread_runs._message_text) reimplemented the same 'extract display
text from a message' logic — str / list of string|{text}|nested{content}
blocks joined without a separator / mapping with text|content key. They
differed only in two ways: journal reads a BaseMessage attribute while
thread_runs reads dict-shaped run_events rows, and journal falls back to
message.text.
Add deerflow.utils.messages.message_to_text(message, *,
text_attribute_fallback=False) that handles both message shapes (attribute
or mapping content access) and gates the .text fallback behind a flag, and
have both call sites delegate. journal passes text_attribute_fallback=True;
thread_runs uses the default. Behavior is unchanged at both sites.
Verified behavior-preserving with an equivalence harness running both
original implementations vs the shared helper over 98 inputs (BaseMessage
and dict messages; str/list/mapping/None/numeric content; mixed blocks;
.text attribute present/absent/non-str) -> 0 mismatches. Added
tests/test_utils_messages.py; the journal last_ai_message extraction tests
still pass.
DynamicContextMiddleware (PR #3630, p0) uses the ID-swap technique to
inject a SystemMessage(reminder) into the middle of the conversation.
create_agent then prepends the static system_prompt as another
SystemMessage at request time, so strict OpenAI-compatible backends
(vLLM, SGLang, Qwen) and Anthropic reject the request with
'System message must be at the beginning'.
New SystemMessageCoalescingMiddleware runs in wrap_model_call — after
create_agent prepends system_prompt — and merges every SystemMessage
into a single leading one before the request reaches the provider.
Non-system messages keep their original order; the merged SystemMessage
preserves the id of the first system message. Only the request payload
is touched; checkpoint state is unchanged, so every consumer that scans
history (memory builder, journal, summarization, dynamic-context
detection) keeps working.
Mirrors the per-request coalescing already done for Claude in
claude_provider._coalesce_system_messages (PR #3702) but at a
provider-agnostic layer so every backend benefits from a single fix.
Closes#3707
* fix(middleware): fix positional fallback consuming unrelated todo when same-content list is exhausted
When next_todos contains the same content string twice (e.g. two "A"
entries), the first iteration pops the matching previous todo from
previous_by_content["A"], leaving an empty list. The second iteration
finds that empty list, treats it as falsy, and sets previous_match=None.
The positional fallback then fires unconditionally — consuming
previous_todos[index] even if it holds a completely different entry
(e.g. "B"). That marks "B" as matched, so the final loop never emits
a todo_remove action for it.
Fix: guard the positional fallback with `content not in previous_by_content`
so it only fires for genuinely new content (the key was never in previous),
not for same-content entries whose list has simply been exhausted.
Add a regression test to _build_todo_actions covering this exact case.
* style: ruff-format the token usage middleware test
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
get_model_config / get_tool_config / get_tool_group_config did a next(...)
linear scan of self.models / self.tools / self.tool_groups on every call.
These sit on hot paths: get_tool_config runs 2-3x per community-tool
invocation (web_search etc.) and get_model_config several times per agent
build, across 40+ call sites.
Build name -> config dicts once in a mode="after" validator (stored as
PrivateAttr), so each getter is an O(1) dict lookup. A config reload
constructs a fresh AppConfig, which rebuilds the indexes; setdefault keeps
first-match-wins on duplicate names, matching the prior next(...) semantics.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MemoryStreamBridge._resolve_start_offset scanned the retained event buffer
(up to queue_maxsize=256 entries) on every subscribe/reconnect carrying a
Last-Event-ID. Event ids are "{ts}-{seq}" where seq is a per-run monotonic
counter that equals the event's absolute offset, so the offset is computable
arithmetically. Parse seq, index into the buffer, and verify the id matches
exactly -- a stale/evicted/foreign/malformed id falls back to
replay-from-earliest, identical to the previous scan.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Middleware-injected hidden messages (TodoMiddleware.todo_reminder,
ViewImageMiddleware, p0 DynamicContextMiddleware.__memory) were being fed
to the memory-updating LLM as if they were real user input, polluting
long-term memory with framework-internal text. The p0 __memory payload
could also trigger a self-amplification loop.
Fix: skip HumanMessages with additional_kwargs['hide_from_ui'] in
filter_messages_for_memory, consistent with the frontend logic.
Closes#3695
mask_local_paths_in_output runs once per glob/grep match (tools.py:1540,
:1625) and once per bash/ls output, and each call rebuilt every skills /
ACP / user-data masking regex from scratch — Path.resolve() syscalls +
re.escape + re.compile. A grep returning up to 100 matches recompiled the
same patterns 100x.
Compile the host->virtual patterns once per ordered source set via
functools.lru_cache (keyed on the config-stable + per-thread (host,
virtual) pairs, so it self-invalidates when they change) and collapse the
three byte-identical replacer blocks into one applier. Behavior is
unchanged: same patterns, same application order, same per-thread mappings.
Verified behavior-preserving with an equivalence harness (original vs
refactored logic over 54 inputs incl. slash-style variants, base-only
matches, subpaths, overlapping mappings, missing skills/ACP, thread_data
None) -> 0 mismatches. Added cache/consistency tests alongside the
existing masking tests.
Fixes#3712.
* feat(memory): add guaranteed injection for correction facts with graceful fallback
When the token budget is tight, high-value facts (e.g. user corrections)
can be silently evicted by lower-priority regular facts. This change:
- Introduces configurable 'guaranteed_categories' (default: [correction])
whose facts draw from a separate 'guaranteed_token_budget', ensuring
they are never dropped due to budget pressure.
- Adds a graceful fallback to confidence-only ranking when the
guaranteed-category path raises an unexpected exception.
- Refactors fact selection into a header-agnostic helper
(_select_fact_lines) with explicit token accounting in the caller,
eliminating double-counting of separators.
- Emits a single 'Facts:' header regardless of whether both guaranteed
and regular facts are present.
- Extends the final safety truncation limit to account for the
additional guaranteed budget so guaranteed facts survive end-to-end.
* refactor(memory): address review feedback on guaranteed injection
- Restore strict break-on-overflow in `_select_fact_lines` to preserve
the caller's confidence-ordered ranking; add a regression test locking
in the invariant that a shorter lower-confidence fact never slips
ahead of a skipped higher-confidence one.
- Account for the inter-group `\n` separator between guaranteed and
regular fact blocks in the regular budget (1-token precision fix).
- Clarify docstrings on `format_memory_for_injection` and
`MemoryConfig.guaranteed_token_budget` to distinguish the common
*displacement* case (total stays within `max_tokens`) from the rarer
*additive* case (safety-truncation ceiling raised when guaranteed
lines alone would overflow).
* fix(memory): address P1 safety truncation + P2s from review
- Structure-aware safety truncation: Facts block is now a protected
suffix so guaranteed-category facts can never be silently discarded
by a prefix-cut on overflow. Only the preceding (user/history)
sections are eligible for truncation.
- Extend the same protected-suffix treatment to the except/fallback
path by returning fact lines alongside the formatted section from
_fallback_format_facts, avoiding string parsing.
- Single inter-section separator: facts section no longer embeds its
own leading \n\n; the final "\n\n".join(sections) is the single
source of truth for section-to-section spacing.
- Bare string for guaranteed_categories now raises TypeError instead
of silently iterating single characters.
- Category-less / malformed facts no longer default-promote into the
guaranteed "context" pool — only facts with an explicit category
field qualify.
- Lift valid_facts pre-filter outside the try so the fallback path
reuses it instead of re-doing validation work.
- MemoryConfigResponse + DeerFlowClient.get_memory_config now expose
guaranteed_categories / guaranteed_token_budget.
- config.example.yaml: document the two new fields and bump
config_version from 12 to 13.
- Add regression tests for every finding.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* test(agents): multi-turn message-stream invariants (graph integration)
Add a graph-integration net for the class of bug behind #3684: build a real
create_agent graph with DynamicContextMiddleware plus a checkpointer, drive two
user turns on one thread with a deterministic fake model and memory injection
stubbed on, then assert the message stream stays well-formed -- the newest user
message is the latest human turn, no duplicate ids, no __user__user suffix
explosion.
Runs at unit speed in `make test` (backend-unit-tests), with no gateway, SSE,
fixtures, or API key. Verified red on the pre-fix middleware and green after.
Catches this class earlier than e2e replay, which disables memory, uses a
single-turn golden, and asserts SSE shape only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(agents): address review — semantic-first asserts, _STREAM_MIDDLEWARES, DRY
- Check the semantic invariant (newest user message is the latest human turn)
before the structural id checks, so a regression surfaces as a meaning-level
failure; verified it now fails first on the pre-#3685 middleware.
- Add module-level _STREAM_MIDDLEWARES (the docstring referenced it; it did not
exist) so the net is trivially widened with more state-touching middlewares.
- _last_human_text reuses _msg_text instead of re-implementing content flattening.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(agents): skip dateless reminders in dynamic-context date scan
_last_injected_date returned at the first message flagged
dynamic_context_reminder regardless of whether it carried a
<current_date>. Since #3630 split injected context into a date
SystemMessage plus a separate dateless <memory> HumanMessage, the
reverse scan hit the memory message first and returned None, making a
later turn look like the first turn. The middleware then re-injected,
re-targeted the previous turn's __user message via the ID swap, and left
the stale message as the latest human turn -- so the model re-answered
the previous message and the persisted message stream got scrambled.
Skip flagged reminders without a <current_date> so the scan reaches the
real date SystemMessage. Add a regression test for the 2nd-turn +
memory-enabled scenario.
Fixes#3684
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(agents): make injected reminder date authoritative via metadata
Promote the dynamic-context date out of message content into
additional_kwargs["reminder_date"] on the date SystemMessage.
_last_injected_date reads that key instead of regex-parsing content, with a
SystemMessage-scoped content fallback for checkpoints written before the key
existed.
This removes two structural fragilities raised in review of #3685:
- the boolean dynamic_context_reminder flag no longer has to also mean
"carries a date" -- dateless reminders simply lack reminder_date and are
skipped, so a future reminder type cannot reproduce the shadowing bug.
- the date regex no longer runs on the user-influenceable memory HumanMessage,
so a memory fact containing a literal <current_date> cannot spoof the
injected date (false same-day skip / false midnight crossing).
Add regression tests for memory date-spoofing, structured-date stamping, and
legacy (no-key) detection; update existing date-reminder fixtures to the
production shape.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
_aexecute collects AI messages from agent.astream(stream_mode="values"),
which re-yields the full state every super-step. The duplicate check rescanned
the append-only ai_messages list on every chunk -- any(m["id"] == message_id) --
so a run with M messages did O(M^2) work, and M reaches max_turns=150 for the
general-purpose / deep-research subagent.
Track an id-keyed set alongside ai_messages: id-bearing messages become O(1)
set lookups, and the id-less full-dict-compare fallback is preserved. Behavior
is unchanged.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(config): coerce null object config sections to their defaults
#3434 made commented-out list sections (models/tools/tool_groups) parse as []
instead of crashing, but the same scenario still crashes for object sections:
commenting out every key under e.g. memory: / summarization: / guardrails:
makes PyYAML parse the value as None, and AppConfig then raises "Input should
be a valid dictionary" for that section — breaking the documented
`cp config.example.yaml config.yaml` first-run flow.
Generalize the handling: a model_validator(mode="before") drops None-valued
sections so each field falls back to its default (list sections -> [], object
sections -> their default config). This subsumes the previous list-only
field_validator and the database special-case. Required sections without a
default (sandbox) still error when null.
Adds test_app_config_coerces_commented_out_object_sections; the existing
list-section regression test still passes.
* docs+test: address review on null-section coercion
- Correct the _drop_null_config_sections docstring: it does not subsume the
database special-case; _apply_database_defaults still owns `database` and
applies concrete defaults beyond null-coercion.
- Strengthen test_app_config_coerces_commented_out_object_sections to assert
each null section falls back to its expected default config type, not just
that it is non-None.
- Add test_app_config_null_required_section_still_errors covering the
required-section (sandbox) "still errors when null" claim.
---------
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
list_thread_messages matched event_type == "ai_message" to find each
run's last AI message, but RunJournal stores AI messages as
"llm.ai.response" and the event store returns that verbatim. No code
writes "ai_message", so the match never hit: feedback was never attached
(every message returned feedback=null) and the grouped-feedback query ran
on every request for nothing.
Match "llm.ai.response", and only run the grouped-feedback query when the
thread actually has an AI message to attach it to. Adds a regression test
for the per-run attachment and the no-AI-message lazy-query path.
Fixes#3650.
P0: DynamicContextMiddleware role isolation — framework data as SystemMessage,
memory as HumanMessage (OWASP LLM01), user input unchanged.
Includes E2E hash stabilization: frontend hide_from_ui filter fix in
suggest_agent, backend _canonical_messages() hide_from_ui skip, fixture
hash updates.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
init_engine runs on the FastAPI lifespan event loop, but created the SQLite
data directory with a synchronous os.makedirs (a stat + mkdir syscall),
blocking startup. Dispatch it via asyncio.to_thread, mirroring the #1912 fix
for the checkpointer's ensure_sqlite_parent_dir.
Adds a Blockbuster-gated regression test in tests/blocking_io/ that drives the
real init_engine path with a not-yet-existing sqlite_dir; it trips
BlockingError if the makedirs regresses onto the event loop.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
* feat(groundroute): add GroundRoute community web_search + web_fetch tools
GroundRoute is a meta search layer over six engines (Serper, Brave, Exa,
Tavily, Firecrawl, Perplexity) with price-based routing and failover. This
adds a self-contained community engine module (httpx only, no new required
deps) mirroring community/brave + community/tavily:
- web_search: POST /v1/search, normalize to {title,url,snippet,source_engine}.
- web_fetch: fetch a URL via mode=page.
- unit tests covering normalization, auth, clamping, and graceful errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(groundroute): register GroundRoute search + fetch in wizard and config
Add GroundRoute to the setup wizard provider lists (SEARCH_PROVIDERS +
WEB_FETCH_PROVIDERS) and as commented web_search + web_fetch examples in
config.example.yaml, mirroring tavily/serper/brave so SEARCH_API can select it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(groundroute): apply repo ruff format (line-length 240)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(groundroute): add GroundRoute to tools docs and config reference
Adds GroundRoute as a web_search and web_fetch option in the en + zh
tools.mdx pages (new tab alongside Tavily/Brave/Exa/etc.) and documents
GROUNDROUTE_API_KEY in CONFIGURATION.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(groundroute): define empty groundroute extra for clean install
The docs install line 'uv add deerflow-harness[groundroute]' (mirroring the
tavily/exa/firecrawl pattern) referenced an undefined extra, which uv accepts
but warns about. GroundRoute needs no extra packages (httpx is a core dep), so
declare an empty 'groundroute' extra in deerflow-harness optional-dependencies
so the documented command resolves without a warning.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(groundroute): per-tool api key + honor caller max_results (review)
Address maintainer review on PR #3675:
- _get_api_key(tool_name): web_fetch now reads the web_fetch config block's key
instead of always web_search, so a flow that pairs GroundRoute fetch with a
different search engine authenticates correctly. Mirrors serper/exa/firecrawl.
- web_search honors a caller-supplied max_results (sentinel default None),
falling back to the configured value only when omitted, so the documented
parameter is no longer silently discarded.
- warn-once is now keyed per tool. Tests cover both fixes (web_fetch key,
agent max_results honored).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: persist AI turn duration in backend and UI
* chore: restore uv.lock to match main
* refactor(frontend): use Math.floor instead of Math.ceil for reasoning timer
Follow-up to #3301 / #3344. The Gateway-embedded runtime is the standard
topology — there is no standalone LangGraph service — but two stale
references slipped past the cleanup guard:
- .github/copilot-instructions.md still told contributors that `make dev`
"Starts LangGraph (2024)" and wrote logs/langgraph.log.
- SandboxAuditMiddleware's docstring pointed audit logs at langgraph.log.
Update both to the Gateway-embedded reality (gateway.log) and extend
test_gateway_runtime_cleanup.py to pin the agent-instruction docs so the
standalone references can't reappear.
Stays within the safe scope of #3304: does not touch langgraph_auth.py,
langgraph.json, or the langgraph-api / langgraph-cli / langgraph-runtime-inmem
deps, which the issue gates on maintainer confirmation of Studio / direct
LangGraph Server support.
should_ignore_name runs once per directory entry during glob/grep tree
walks and looped over ~57 IGNORE_PATTERNS calling fnmatch per pattern.
Precompute the literal names into a frozenset (O(1)) and the few glob
patterns into one combined compiled regex via fnmatch.translate; per name
it's now one normcase + a set lookup + at most one regex match.
os.path.normcase preserves fnmatch's platform case behavior, so results
are identical (covered by an equivalence test against the old loop).
Fixes#3655.
* fix(mcp): migrate local MCP-produced files into sandbox outputs (#3597)
Stdio MCP servers (e.g. Playwright) write files to host paths that the
sandbox/artifact API cannot resolve, since it only serves paths under
/mnt/user-data. Copy local files referenced by ResourceLink results into
the thread's sandbox outputs dir and rewrite their URIs to
/mnt/user-data/outputs/... so they become readable.
Also scope pooled MCP sessions by user_id:thread_id instead of thread_id
alone, matching the per-(user_id, thread_id) filesystem isolation.
* fix(mcp): restrict file migration to trusted source roots (#3597)
Add a source-root allowlist to the MCP file-migration path so a
malicious or buggy MCP server cannot have us copy arbitrary host files
(e.g. /etc/passwd) into a thread's outputs directory, from where the
artifact API would serve them. Files are migrated only when located
under the OS temp dir (Playwright's default), the thread's own
user-data tree, or an operator-configured root via
DEERFLOW_MCP_MIGRATION_SOURCE_ROOTS.
Expand test coverage with allowlist/security cases (path escape refusal,
trusted-root acceptance), URL-encoded file:// paths, converter content
branches (image/embedded/error/structured), and copy/resolve failure
fallbacks.
* fix(mcp): harden local file migration into sandbox outputs
Address robustness and security gaps in the MCP ResourceLink file
migration:
- Set migrated files to 0o644 so a differently-UID sandbox container can
read them, instead of inheriting the source's (possibly 0o600) mode.
- Enforce the 100MB size cap during the copy (chunked, byte-counted)
rather than from a prior stat(), closing the grow-after-stat TOCTOU.
- Create the destination atomically with O_CREAT|O_EXCL to remove the
check-then-create name-collision race.
- Document the shared-$TMPDIR multi-tenant read surface and mitigation.
Add regression tests: symlink escape refusal, explicit $TMPDIR source
migration, 0o644 mode, and the outputs/user-data resolve() OSError
fallback branches.
* fix(mcp): migrate playwright text file outputs
* fix(mcp): translate MCP file outputs to virtual paths instead of copying (#3597)
Pin stdio MCP subprocess cwd and TMPDIR/TMP/TEMP under the thread workspace
so produced files always land in the mounted user-data tree, then rewrite
returned references via deterministic host->virtual path translation. Free
text is best-effort only: a reference is rewritten only when it resolves to
an existing file inside the thread's tree, and bare filenames are matched
against files created/modified by the same tool call. Replaces the previous
copy-into-outputs + regex approach (which missed cases like temp/page-*.yml).
* style(mcp): apply ruff format to mcp path translation tests
* perf(mcp): offload stdio FS work off event loop and gate on transport
Address review on #3600:
- Wrap the workspace dir prep, snapshot diff, and per-token path
resolution in asyncio.to_thread so they no longer block the event
loop (matches the repo's blocking-IO gate convention).
- Gate the cwd/temp pinning and snapshots on stdio transport only;
SSE/HTTP servers skip the filesystem work entirely.
- Skip the post-call snapshot diff when the result has no text content.
* test(mcp): cover stdio transport gating and text-content after-walk skip
Add unit/integration coverage for the new review-driven behavior:
- _prepare_stdio_workspace dir/temp/snapshot bundle
- _result_has_text_content detection (text, embedded text, image, empty)
- non-stdio transport skips cwd/temp pinning and touches no workspace dirs
- post-call snapshot diff is skipped without text content and runs with it
* fix(mcp): address stdio path rewrite review feedback
- Restrict the stdio MCP temp directory to 0700 instead of 0777.
- Preserve operator-provided stdio cwd values while keeping injected cwd values as strings.
- Add debug logging for deterministic path rewrites and bare-filename rewrite decisions.
- Document the stdio cwd/temp pinning, virtual-path translation, and user/thread session scope.
- Cover explicit cwd preservation and temp-dir permissions in session-pool tests.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* perf(sandbox): cache LocalSandbox path-rewrite regexes per instance
LocalSandbox re-sorted path_mappings and re-compiled its path-rewrite
regex on every tool call: _resolve_paths_in_command (every bash),
_resolve_paths_in_content (every write_file), and
_reverse_resolve_paths_in_output (every bash output / read_file). The
inputs are derived solely from self.path_mappings, which is assigned once
in __init__ and never mutated, so the work is identical every call.
Compile the patterns once per sandbox via functools.cached_property and
reuse them; hoist 'import re' to module scope. Behavior is unchanged —
only the per-call sort+escape+compile on the agent's hot path is removed.
Fixes#3647. Adds tests covering caching identity, unchanged rewriting,
path-segment boundary matching, and the empty-mappings pass-through.
* perf(sandbox): also cache resolved local paths and sorted mapping views
Beyond the regex compilation, _find_path_mapping, _is_read_only_path,
_resolve_path_with_mapping and _reverse_resolve_path re-sorted
path_mappings and re-ran Path(local_path).resolve() (a filesystem
syscall) on every call. Since path_mappings is immutable, cache the
resolved local root per mapping and the two sorted views via
cached_property and reuse them. Behavior is unchanged; the reverse-output
pattern builder now reuses the same resolved-path cache.
* fix(gateway): attribute token usage to actual models
Capture per-call model names from LLM response metadata for lead, middleware, and subagent calls.
Persist a per-run token_usage_by_model breakdown and aggregate by that map in both SQL and memory stores, with legacy fallback to the run-level model_name for older rows.
Add regression coverage for by_model totals, caller consistency, active progress snapshots, store parity, and SubagentTokenCollector model propagation.
* fix(gateway): harden by-model token aggregation
Use usage.get("total_tokens", 0) when reducing per-model token usage maps so aggregation tolerates partially written or manually edited JSON blobs without changing behavior for journal-written rows.
* docs(gateway): clarify by-model run count semantics
Document that by_model[*].runs counts the number of runs in which a model appeared, so multi-model runs can increment multiple model buckets.
Base.to_dict() and __repr__() ran sqlalchemy.inspect(type(self)).mapper
reflection on every call, but to_dict() is invoked once per row when
serializing ORM results (e.g. every event in a messages/events page).
The mapped columns are fixed at class-definition time, so cache the
column keys per class with functools.cache and iterate the cached tuple.
Behavior is unchanged.
Fixes#3653.
* feat(suggest_agent): stop frontend from fetching when suggestions disabled
* wait for suggestions config to load before fetching
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The SSE stream's `values` snapshots serialize the full state, including the
hide_from_ui human messages that ViewImageMiddleware fills with base64 image
payloads. #3535 stripped those from the REST wait/history/state endpoints via
serialize_channel_values_for_api, but the streaming path (worker publishes
serialize(chunk, mode="values")) still went through serialize_channel_values,
so the same base64 leaked to the frontend over SSE. Route values-mode
serialization through serialize_channel_values_for_api as well. Non-hidden
messages and https image URLs are left untouched.
* fix(channels): scope IM files and helper commands to owner
* fix(memory): honor bound IM owner for /memory gateway endpoints
The channel manager already attaches X-DeerFlow-Owner-User-Id for /memory
and /models, but the memory router resolved user_id solely from
get_effective_user_id(), which returns the synthetic internal user
(DEFAULT_USER_ID) for channel workers. A bound IM /memory therefore read
the default/internal memory instead of the connection owner's.
Resolve the owner via _resolve_memory_user_id(request) across all
/api/memory* endpoints: trusted internal callers act for the owner header,
browser/API callers fall back to get_effective_user_id(). Mirrors the
threads router's get_trusted_internal_owner_user_id pattern, completing
acceptance criterion #3 of #3539.
Add end-to-end tests asserting the resolved user_id (not just that the
header is sent) and that a spoofed owner header from a browser user is
ignored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): align memory bucket and reuse cached storage owner
Address PR #3579 review feedback:
- Memory router now sanitizes the trusted owner header via make_safe_user_id
before routing, matching the channel file pipeline
(_safe_user_id_for_run/prepare_user_dir_for_raw_id). A bound owner id needing
sanitization now resolves to the same bucket as its files/uploads instead of
500ing in _validate_user_id.
- _handle_chat reuses the storage_user_id cached at the top of the method for
artifact delivery instead of re-deriving _channel_storage_user_id(msg), so
uploads and outputs cannot drift to different buckets if a channel rewrites
the InboundMessage in receive_file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): stage unbound IM files under the run's user bucket
Address PR #3579 review feedback (#5): _channel_storage_user_id now mirrors
_resolve_run_params' identity policy, falling back to safe(msg.user_id) instead
of returning None for unbound auth-enabled channels.
Previously an unbound msg ran under safe(platform_user_id) but staged uploads
under get_effective_user_id() in the dispatcher task (unset contextvar ->
"default"), so files landed in users/default/... while the agent read from
users/{safe_platform_user_id}/.... Bound and unbound channels now write where
the agent reads. Returns None only when no identity is available.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): reuse cached storage owner in streaming artifact delivery
Address PR #3579 review feedback (#6): thread the storage_user_id resolved in
_handle_chat into _handle_streaming_chat instead of re-deriving
_channel_storage_user_id(msg) in the finally block. Avoids re-running
_safe_user_id_for_run (and its possible filesystem touch) on the streaming-error
path and guarantees artifact delivery targets the same bucket as the uploads.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(channels): document owner-scoped IM file storage
Address PR #3579 review feedback (#4): the IM Channels and File Upload sections
still described pre-PR default-bucket behaviour. Document that receive_file,
_ingest_inbound_files/ensure_uploads_dir/get_uploads_dir, and
_resolve_attachments/_prepare_artifact_delivery are owner-scoped via the user_id
kwarg, and that the bucket matches the memory bucket from _resolve_memory_user_id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(channels): unify run identity and storage bucket resolution
Address PR #3579 review feedback (#3): _resolve_run_params no longer duplicates
the owner-resolution rule inline. After the #5 fix the inline block and
_channel_storage_user_id computed the identical sanitized-with-platform-fallback
value, so the run identity now calls the same helper, making it the single
source of truth for run_context["user_id"] and the file/artifact storage bucket.
_owner_headers stays deliberately separate: it sends the raw owner id over HTTP
for the gateway to re-resolve (no sanitize, no platform fallback), documented on
both helpers. test_run_identity_matches_storage_bucket pins the two together so
they cannot drift again.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): harden runtime credential management APIs
* fix(channels): address review feedback on credential hardening
Follow-up to the runtime credential-hardening pass, resolving five review
findings:
- WeChat auth persistence now writes through a 0o600 NamedTemporaryFile +
Path.replace instead of write_text-then-chmod, so the iLink bot_token is
never briefly readable at umask defaults (mirrors ChannelRuntimeConfigStore).
- The post-write chmod is split into its own try/except: a chmod failure on a
filesystem without POSIX perms now logs at debug instead of masquerading as
a "failed to persist" warning.
- Extracted the three near-identical _require_admin_user helpers (mcp,
channel_connections, channels) into a single require_admin_user(request, *,
detail) in app/gateway/deps.py; each router supplies its own detail string.
- Strengthened the runtime-config-store chmod coverage: a new test injects a
temp-file chmod failure and asserts it is logged at debug while the
destination is still owner-only (mutation-verified to fail if the chmod is
dropped), plus a loose-pre-existing-file case.
- Removed the unused _FakeRepo from the blocking-io test: its isinstance gate
routes through the repo-less 503 path, so neither stub was ever invoked.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): make channel connect flow deterministic
* make format
* fix(channels): apply connect-code before allowed_users on telegram and wechat
The bind-bootstrap reorder shipped for slack/dingtalk only. Telegram and
WeChat still gate _check_user/allowed_users before connect-code dispatch, so
a newly allowlisted-but-unbound user is silently rejected when binding via the
browser deep-link / connect-code flow — the same deadlock the PR fixes.
- telegram: consume the /start deep-link token before the allowed_users gate.
- wechat: handle the /connect code before the allowed_users gate, and defer
inbound file extraction + context-token tracking past the gate so blocked
senders no longer trigger CDN downloads or token bookkeeping.
Adds regression tests for both adapters mirroring the slack/dingtalk coverage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): enforce single-active-owner invariant at the DB layer
_revoke_other_active_owners did a SELECT-then-UPDATE in app code with no row
lock or constraint covering active rows. Under READ COMMITTED, two concurrent
connect-code consumes for the same (provider, external_account_id, workspace_id)
from different owners could each observe "no other active owner" and both commit
a connected row, leaving find_connection_by_external_identity nondeterministic.
- Add a partial unique index on (provider, external_account_id, workspace_id)
WHERE status != 'revoked' (portable to SQLite >= 3.8.0 and PostgreSQL) so the
database guarantees at most one non-revoked row per external identity.
- Reorder upsert_connection to revoke other owners' active rows before the new
connected row is flushed (so the index is satisfied at commit), wrapped in a
bounded rollback-and-retry loop. A losing concurrent writer now retries
against the now-visible state instead of committing a duplicate.
Adds DB-constraint, revoked-slot-reuse, and concurrent-upsert regression tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): harden connect-status polling primitive
pollChannelConnectionUntilResolved was a free-floating recursive setTimeout
started from onSuccess with no cancellation, no per-provider dedup, a redundant
second endpoint per tick, and an unbounded loop on a non-finite expires_in.
- Extract a framework-agnostic, cancellable poller (connect-poll.ts) that polls
only listChannelConnections() and invalidates the providers query once when the
bind resolves, instead of fetching both endpoints every tick.
- Guard expires_in with a finite check + default window so undefined/NaN can no
longer produce a poll loop that runs until the page closes.
- Track one active poll handle per provider in useConnectChannelProvider via a
ref Map: a new connect cancels the prior poll for that provider, and a useEffect
cleanup cancels all polls on unmount.
Adds unit tests for resolve-and-stop, cancellation, and non-finite-expiry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): stop leaking blocked-sender content in DingTalk INFO log; document bind semantics
Moving the allowed_users gate past _extract_text meant the parsed-message INFO
log (text=%r, first 100 chars) fired for senders that allowed_users would have
rejected, defeating the filter's noise/privacy role. Move that log to after the
allowed_users gate so blocked senders' message text never reaches INFO logs.
Also document the two operator-relevant semantic changes in backend/CLAUDE.md:
connect-code dispatch runs before allowed_users (so allowed_users is no longer a
bind-time defense; the model relies on code confidentiality + 600s TTL + one-time
consumption), and the single-active-owner-per-external-identity transfer semantics
now backed by the partial unique index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(channels): note connect-code-vs-allowlist and ownership transfer in operator guide
Mirror the backend/CLAUDE.md notes in the operator-facing IM_CHANNEL_CONNECTIONS.md:
connect codes are consumed before allowed_users (so a not-yet-allowlisted user can
still complete a first bind, and allowed_users is not a bind-time defense), and an
external identity has at most one active owner with last-bind-wins transfer enforced
at the DB layer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(channels): lift connect-code dispatch into Channel base class
Each adapter duplicated the ordering-sensitive boilerplate of extracting a
/connect code and guarding on the connection repo before its allowed_users gate.
The duplication is what let telegram/wechat drift and keep the gate ahead of the
bind. Centralize it:
- Move `_connection_repo` onto Channel.__init__ (removing 7 duplicate assignments).
- Add Channel._pending_connect_code(text), which guards on the repo and extracts
the code, documenting that adapters MUST consult it before authorization so a
browser-initiated bind can bootstrap a not-yet-authorized identity.
- Route slack, discord, feishu, dingtalk, wechat, and wecom through the helper.
This also fixes a latent inconsistency where slack dispatched a bind even when
no connection repo was configured.
Pure refactor — the full channel suite stays green; adds a direct unit test for
the base helper's contract.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* make format
* fix(channels): redact DingTalk parsed-message INFO log content
Log text_len instead of the first 100 chars of message text, so message
content never reaches INFO logs (the after-gate move already keeps blocked
senders out entirely). This takes over the redaction from #3584 so only this
PR touches dingtalk.py, letting the two PRs merge in any order conflict-free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): add operational guardrails
* make format
* fix(channels): converge with #3582 to avoid merge-order conflicts
Drop this PR's DingTalk INFO-log redaction and hand it to #3582, which
already restructures that handler and will redact the same log there. This
PR no longer touches dingtalk.py, so the two PRs can merge to main in any
order without a conflict.
For WeChat, drop the contested thread_ts priority reorder (review #3) and
keep only what inbound dedupe needs: a server-stable message_id in the
inbound metadata (message_id/msg_id, no client_id per review #6). This is a
single added line inside the metadata dict, a region #3582 never touches, so
it auto-merges regardless of order.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): address three correctness review findings
1. Connect-code cap was racy (willem #1): _create_state ran delete-expired,
count, and insert as three separate transactions, so concurrent connect
POSTs from one owner could each see count < cap and all insert past it. Add
ChannelConnectionRepository.create_oauth_state_within_cap which does
delete+count+insert in a single transaction serialized per (owner,
provider) — Postgres via pg_advisory_xact_lock, SQLite via the write lock
the leading DELETE takes — and have the router use it.
2. Inbound dedupe key fell back to "" workspace (willem #3): two workspaces
delivering without team/guild/aibotid would collapse to the same key and
dedupe each other's messages. _inbound_dedupe_key now fails closed
(returns None) when no workspace identifier is present.
3. Dedupe key was recorded on receipt and never released on failure
(ShenAC #1): a transient error (DB blip, Gateway 503) left the key in place
for the full TTL, so a provider redelivery of the same message_id — exactly
the retry dedupe should absorb — was silently dropped. _handle_message now
releases the key in the unexpected-exception branch so redelivery can
recover, while keeping record-on-receipt so retries during handling are
still deduped.
Tests: repo cap enforcement incl. concurrent-issuance non-leak; dedupe
fail-closed; dedupe key release-on-failure redelivery recovery.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): address cleanup/efficiency and test review findings
Efficiency / cleanup:
- Dedupe key set drops client-generated ids (client_msg_id, client_id);
keep only server-stable event_id/message_id/msg_id, which a provider's own
redelivery preserves (ShenAC #6). Every provider already emits message_id.
- TTL/overflow pruning of _recent_inbound_events is now O(k): switch to an
OrderedDict and popitem(last=False) from the front instead of scanning all
4096 entries on every inbound (willem #4).
- Log "received inbound" only after the dedupe check so a provider retrying N
times no longer logs N accepts; document that manager dedupe covers the
agent run/final answer, not provider ack side-effects (willem #5, ShenAC #2).
- Slack drops the redundant `team_id or event.get("team")` fallback the caller
already resolved (willem #6).
- create_oauth_state_within_cap prunes only this owner/provider's expired codes
instead of a global DELETE on every connect POST; global cleanup still runs
on consume_oauth_state (willem #7).
Tests:
- Dedupe test uses tmp_path instead of a leaked mkdtemp, uses distinct objects
per publish, and adds a negative control: a different message_id is still
processed, catching over-dedupe regressions (willem #8, ShenAC #4).
- Slack HTTP-mode rejection test supplies app_token so the missing-token early
return can't mask the guard, giving the state assertions teeth (ShenAC #3).
- count_oauth_states test pins that the active row survives, not just the count
(ShenAC #5).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* make format
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sandbox): stop flagging string-literal path fragments as unsafe paths
The host-bash absolute-path guard scans the raw command string, so /segment
sequences inside string literals, f-strings, and templates were treated as
absolute path arguments and rejected — e.g. python -c "print(f'/端口{port}')"
or a REST template /devices/{id}/port. Whether a fragment tripped the guard
depended on the character right before the slash (a word char suppressed the
match), so the same literal could pass or fail unpredictably, pushing the model
into retry loops that bloat context and wall-clock time.
Exempt matches carrying non-ASCII characters or format braces: real host paths
a command would open contain neither, so these are text, not paths. The guard
is best-effort (not a security boundary), and plain ASCII host paths like
/etc/passwd — including ones written inside a code string such as
open('/etc/passwd') — stay rejected.
* fix(sandbox): only exempt identifier-template braces, not bash brace expansion
The literal-fragment exemption exempted any path fragment containing { or },
which let bash brace expansion (cat /etc/{passwd,shadow}) and ${VAR} expansion
reconstitute real host paths past validate_local_bash_command_paths. Tighten
the brace branch to only exempt fragments where every {...} block is a single
identifier-like placeholder (/devices/{id}/port, f-string /{port}); reject
${VAR} shell-variable expansion. Add parametrized regression tests for the
brace-expansion and shell-var bypasses.
* feat(community): add Serper Google Images provider for image_search
Add a Serper-backed `image_search` tool alongside the existing Serper
`web_search` provider, so users with a SERPER_API_KEY can pull Google
Images results as reference images for downstream image generation.
- Share request/response handling between web_search and image_search
via `_serper_post` / `_response_items`, with bounded `max_results`
(capped at 10) and query normalization.
- Add a best-effort SSRF guard (`_safe_public_url`) that rejects
non-http(s), localhost and private/non-global IP image URLs; filtered
entries are dropped and never consume the result limit.
- doctor: flag literal `api_key` values in config as a warning and steer
users toward `.env` + `$SERPER_API_KEY`.
- Docs/config: document the Serper image_search provider and SERPER_API_KEY,
and discourage committing literal keys to config.yaml.
- Tests: cover the provider end-to-end (100% line coverage on tools.py)
and the doctor literal-key warning path.
* fix(community): block obfuscated IPv4 literals in Serper image SSRF guard
The image_search SSRF guard only rejected dotted-decimal IP literals; encoded
forms such as decimal (http://2130706433/), hex (0x7f000001) and octal
(0177.0.0.1) raised ValueError in ip_address() and were allowed through, even
though many HTTP clients resolve them to private addresses like 127.0.0.1.
Add _decode_ipv4() to permissively decode these inet_aton-style encodings and
apply the same is_global check; hostnames that do not decode to an IP (e.g.
cafe.com) are still treated as hosts and left to fetch-time re-validation.
Addresses PR review feedback. Tests cover decimal/hex/octal loopback and
private encodings plus non-IP edge cases; tools.py stays at 100% line coverage.
* test(community): cover IPv4-mapped IPv6 URL filtering
* fix(community): address Serper image search review feedback
- Block trailing-dot hostname SSRF bypass (localhost./127.0.0.1.) in
_safe_public_url by stripping the FQDN root label before checks.
- Keep a filtered image/thumbnail URL empty instead of collapsing onto
its counterpart, preserving the high-res/preview contract.
- Evaluate the SSRF guard once per field rather than twice.
- Treat a null-typed organic/images field as "no results" rather than a
malformed payload.
- doctor.py: when a config $VAR is unset, fall through to the default env
var before reporting it as not set.
