* fix(mindie): degrade unserializable tool args to str() in XML replay
The XML parser's literal_eval fallback puts values json.dumps rejects
(e.g. Python sets from set-literal arguments) into AIMessage tool_calls
args. Replaying that message from history hit the unguarded json.dumps
in _fix_messages' args join, raising TypeError and failing the whole
request — and every later turn, since the poisoned message rides the
history. Mirror the tool-result json block handling already in the same
function: degrade the value to str() so one unserializable argument
cannot fail the request.
* fix(mindie): reject non-finite floats in literal sets and tuples
* fix(mindie): validate literal dictionary keys recursively
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The no-tool native stream applied _decode_escaped_newlines_outside_fences
to each streamed chunk independently, but a fenced code block almost never
opens and closes within one token-sized chunk, so the per-chunk fence regex
never saw a complete ``` pair and decoded every literal \n inside the fence.
Code like print("a\nb") reached the streamed answer and the persisted
message content with a real newline, while the non-streaming path and the
tool-mode simulated stream preserved it.
Decode the native stream with a fence-stateful decoder instead: it carries
whether the stream is inside a fence across chunks and holds back a
trailing partial ```/`\n` sequence, so the joined stream equals the one-shot
decode of the same reply. A fence left unclosed by a truncated reply keeps
its escapes, matching how an unterminated fence still renders as code.
Document native no-tool streaming in the MindIE decode contract.
* fix(models): preserve structured MindIE tool results
Signed-off-by: lovett55 <xbzz101800@163.com>
* fix(models): redact and bound structured MindIE tool results
* fix(models): keep structured MindIE ToolMessage content visible
_fix_messages flattened list content keeping only string and type=text
blocks, so a JSON tool result became an empty <tool_response> and the
model could not see the tool output on the next turn. Serialize type=json
blocks onto the same text channel; the existing XML escaping applies to
the serialized text unchanged.
Fixes#6010
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(models): harden the MindIE json channel from review
Review follow-ups on the structured-content fix:
- json.dumps can also raise ValueError (circular reference); catch it
alongside TypeError so the payload degrades to str() instead of
failing the whole request normalization.
- A {"type": "json"} block with no json key keeps the old drop
behavior instead of emitting a literal "null" into the frame.
- PiiRedactionMiddleware now redacts json block payloads leaf-wise
(nested dict/list strings scanned, keys and non-string leaves kept),
so the newly serialized channel participates in the configured
redaction policy for user messages and remote tool results alike.
Without it, redaction-enabled deployments would ship structured PII
raw once the provider serializes the block.
Tests: fallback degrade for both exception paths, missing-key drop,
leaf-wise redaction on the model-call and tool boundaries, and a
middleware-to-MindIE regression pinning redaction through provider
serialization. Red/green verified: the 6 new redaction/json tests
fail without the source changes, 155 pass with them.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(models): close MindIE structured tool redaction gaps
* fix(agents): budget structured json tool output before provider normalization
Willem's P2 on #6208: _message_text() returned None for {"type": "json"}
blocks, so a large structured ToolMessage slipped past both the tool-call
and history budget gates and reached MindIE's serializer at full size,
ignoring the externalize threshold and the fallback cap.
Render those blocks with the same json.dumps(..., ensure_ascii=False)
serialization the MindIE adapter applies (str() fallback included), so
they count toward the budget everywhere _message_text is consulted.
Images and other non-textual blocks still skip enforcement.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(mindie): render json blocks for tool messages only
InputSanitizationMiddleware neutralizes strings and text blocks, so a
{"type": "json"} block smuggled into a genuine user message reaches
_fix_messages with its payload untouched. Rendering it into the text
channel handed the model an unescaped, unframed injection (e.g. a
literal <system-reminder>), bypassing the guardrail entirely.
Gate the json-block serialization on ToolMessage: structured tool
results keep riding the text channel (and still get html.escape plus
the <tool_response> framing), while user/assistant json blocks go back
to the old drop behavior. Adds unit coverage for the drop paths plus a
chain regression that runs a genuine user message through the
sanitization middleware and then _fix_messages, asserting the smuggled
payload never reaches the model-bound text.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(models): preserve configured budgets in combined MindIE fix
---------
Signed-off-by: lovett55 <xbzz101800@163.com>
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: lovett55 <xbzz101800@163.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
In tool-enabled requests MindIEChatModel._astream falls back to awaiting the full _agenerate response and re-emitting it as simulated AIMessageChunks. The full response carries usage_metadata, but none of the simulated chunks copied it, so chunk aggregation (add_ai_message_chunks) produced a final message with usage_metadata=None. Token usage therefore vanished from token accounting, run stats, persistence and the UI for every tool-enabled streamed turn.
Mirror OpenAI's terminal-usage-frame convention: attach msg.usage_metadata to exactly the last simulated chunk (the trailing tool-call chunk when present, else the last text chunk / the single tool-only chunk) so the aggregated message carries it exactly once. add_usage() is per-chunk additive, so attaching usage to every chunk would multiply the totals.
Scope: MindIEChatModel only; other providers keep native streaming and ainvoke/non-tool astream were already correct.
Tests: regression guard asserting exactly one carrier chunk equals the last one and that merged usage equals the original across all three simulated-stream branches, plus chain-level tests driving the public astream() wrapper and asserting the persisted model_dump() shape.
Closes#5192
* feat(channels): enhance Discord with mention-only mode, thread routing, and typing indicators
Add mention_only config to only respond when bot is mentioned, with
allowed_channels override. Add thread_mode for Hermes-style auto-thread
creation. Add periodic typing indicators while bot is processing.
* fix(discord): include allowed_channels in mention_only skip condition (line 274)
* docs: fix Discord config example to match boolean thread_mode implementation
* style: format with ruff
* fix(discord): apply Copilot review fixes and resolve lint errors
- Remove unused Optional import
- Fix thread_ts type hints to str | None
- Fix has_mention logic for None values
- Implement thread_mode fallback to channel replies on thread creation failure
- Fix thread_mode docstring alignment
- Fix allowed_channels comment formatting in config.example.yaml
* fix(discord): reset context for orphaned threads in mention_only mode
When a message arrives in a thread not tracked by _active_threads,
clear thread_id and typing_target so the message falls through to
the standard channel handling pipeline, which creates a fresh thread
instead of incorrectly routing to the stale thread.
* fix(discord): create new thread on @ when channel has existing tracked thread
When mention_only is enabled and a user @-s the bot in a channel
that already has a tracked thread, create a new thread instead of
incorrectly routing to the old one.
* fix(discord): allow no-@ thread replies while skipping no-@ channel messages
The skip block for no-@ messages was too aggressive — it blocked
continuation replies within tracked threads AND incorrectly routed
no-@ channel messages to the existing thread.
Now:
- Thread message, no @ → routed to existing tracked thread
- Channel message, no @ → skipped
- Channel message, with @ → creates new thread
* feat(discord): add checkmark reaction to acknowledge received messages
* Move discord.py to optional dependency and auto-detect from config.yaml
- Add discord extra to [project.optional-dependencies] in pyproject.toml
- Update detect_uv_extras.py to map channels.discord.enabled: true -> --extra discord
- Set UV_EXTRAS=discord in docker-compose-dev.yaml gateway env
* fix(discord): persist thread-channel mappings to store for recovery after restart
Discord's _active_threads dict was purely in-memory, so all channel-to-thread
mappings were lost on server restart. This fix bridges ChannelStore into
DiscordChannel:
- Save thread mappings to store.json after every thread creation
- Restore active threads from store on DiscordChannel startup
- Pass channel_store to all channels via service.py config injection
Store keys follow the pattern: discord:<channel_id>:<thread_id>
* fix(discord): address Copilot review — fix types, typing targets, cross-thread safety, and config comments
* fix(tests): add multitask_strategy param to mock for clarification follow-up test
* fix(tests): explicitly set model_name=None for title middleware test isolation
* fix(discord): use trigger_typing() instead of typing() for typing indicators
discord.py 2.x TextChannel.typing() and Thread.typing() are async context
managers, not one-shot coroutines. Use trigger_typing() for periodic
typing indicator pings.
* fix(discord): cancel typing tasks on channel shutdown
Prevents 'Task was destroyed but it is pending' warnings when the
Discord client stops while typing indicator loops are still running.
* fix(scripts): detect nested YAML config for discord extra
section_value() only matched top-level YAML sections. Added
nested_section_value() that handles two-level nesting (e.g.,
channels.discord.enabled), so auto-detection of the discord
extra works when config uses the standard nested format.
* fix(docker): remove hard-coded UV_EXTRAS=discord from dev compose
Relies on auto-detection via detect_uv_extras.py instead of forcing
discord.py install even when channels.discord.enabled is false.
Matches production docker-compose.yaml behavior (UV_EXTRAS:-).
* refactor(nginx): move proxy_buffering/proxy_cache to server level
DRY cleanup — these directives were repeated in 14 location blocks.
Set at server level once, reducing duplication and risk of drift.
* fix(discord): use dedicated JSON file for thread persistence
Replace ChannelStore usage for Discord thread-ID persistence with a
dedicated discord_threads.json file. ChannelStore is designed to map
IM conversations to DeerFlow thread IDs — using it to persist Discord
thread IDs was semantically wrong and confusing.
Changes:
- _save_thread() now reads/writes a simple {channel_id: thread_id} JSON dict
- _load_active_threads() reads directly from the JSON file
- File path derived from ChannelStore directory (when available) or
defaults to ~/.deer-flow/channels/discord_threads.json
- Removed unused ChannelStore import
* fix(discord): address WillemJiang's code review comments on PR #2842
1. Remove semantically incorrect message_in_thread variable. At this code
point (after the Thread case is handled above), we're guaranteed to be in
a channel, not a thread. Always apply mention_only check here.
2. Add _active_thread_ids reverse-lookup set for O(1) thread ID membership
checks instead of O(n) scan of _active_threads.values(). Keep the set
in sync with _active_threads in _load_active_threads() and _save_thread().
3. Add _thread_store_lock (threading.Lock) to protect _active_threads and
the JSON file from concurrent access between the Discord loop thread
(_run_client) and the main thread (_load_active_threads, _save_thread).
* feat(models): 适配 MindIE引擎的模型
* test: add unit tests for MindIEChatModel adapter and fix PR review comments
* chore: update uv.lock with pytest-asyncio
* build: add pytest-asyncio to test dependencies
* fix: address PR review comments (lazy import, cache clients, safe newline escape, strict xml regex)
* fix(mindie): preserve string args without JSON quotes in XML tool call serialization
* fix(mindie): preserve string args without JSON quotes in XML tool call serialization
* test_mindie_provider:format
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(mindie): prevent nested tool_call params from leaking into outer args
* fixed by escaping XML entities in _fix_messages and unescaping during parse, with regression tests added.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
