* fix(skillscan): report secrets bound by unpacking and container literals
The AST walk behind secret-env-assignment claims "precision, never less
coverage", but it only read a single assignment target and a bare string
value, so a credential moved into `host, api_key = endpoint, "..."` or
into `api_key = ["..."]` stopped being reported even though the pre-#5648
line sweep still reported the same bytes.
Pair unpacking targets with values positionally (fixed head and tail only,
so a runtime-sized slice names nothing) and assert on each literal a
container or conditional holds, in source order.
* fix(skillscan): distinguish mapping labels from credentials
---------
Co-authored-by: sxh313 <sxh313@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The channel runtime config store's atomic 0o600-temp + replace + chmod
cannot land a POSIX mode on Windows, where chmod is a no-op
approximation. Skip the two POSIX-bound tests and guard the trailing
mode assertion of the chmod-failure test, keeping its platform-
independent log-behavior coverage everywhere.
Fixes#6256
Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>
Four of the eight packages/harness/deerflow citations in this file name modules
that are not in the tree: sandbox/local.py, mcp/manager.py, skills/loader.py,
and tools/__init__ (missing its extension). The LocalSandboxProvider class lives
under sandbox/local/, the MCP and skills subsystems are packages rather than
single manager/loader modules, and get_available_tools is re-exported by
tools/__init__.py.
* refactor(config): converge the boolean guards into one shared helper
Willem-bd's #6171 review follow-up: a tiny shared helper lets the next numeric field get the guard in one line and unifies the messages. Consolidates 14 validator copies across 9 config files into deerflow/config/_boolean_guards.reject_boolean(value, info, kind=...); the bare per-file messages gain their field name prefix, command_timeout keeps its distinct wording.
Signed-off-by: yetuge <2219677952@qq.com>
* refactor(config): converge stream_bridge_config and pin the guard wording
Review follow-ups on #6226: convert the last surviving per-file copy (stream_bridge_config) so deerflow/config is fully converged, make kind required keyword-only so a float field cannot silently inherit integer wording, and pin the unified field-prefixed messages with direct tests.
Signed-off-by: yetuge <2219677952@qq.com>
* test(config): cast the boolean-guard test probe to ValidationInfo
The _info probe was annotated -> object, so a type checker flags every call
site that passes its result to reject_boolean's info: ValidationInfo
parameter (runtime was fine since the helper only reads .field_name).
Return cast(ValidationInfo, _Probe(field_name)) to keep the probe honest
for both.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(memory): sign DeerMem reload before reading the document
reload() read the document before computing its cache signature, so a
write committed in between cached the old document under the new
signature and later load() calls served stale memory until the next
write. Compute the signature first, matching load().
* test(memory): cover markdown storage and agent scope in reload race test
* docs(changelog): reference #6238 for the DeerMem reload fix
* fix(telegram): deny all users when allowed_users has no valid ID
TelegramChannel parsed channels.telegram.allowed_users with int() and
dropped every entry that failed without logging, and _check_user treated
an empty result as "no allowlist". A list of @usernames therefore opened
the bot to everyone. A single string ID was iterated character by
character ("123456" allowed users 1-6 and blocked 123456), which is what
an allowed_users: $ENV reference resolves to, and null or a bare integer
crashed the channel at startup.
The allowlist is now parsed into frozenset[int] | None: None (unset,
null, empty list, blank string) still allows everyone, a single ID is a
one-entry list, entries that are not positive numeric IDs (@usernames,
floats, booleans, comma-joined strings) are dropped with a warning, and
a configured list with no valid ID denies every user and logs an error.
* docs(changelog): reference #6230 for the telegram allowed_users fix
* fix(telegram): say "positive" in the allowed_users drop warning
0 and negative integers are numeric but still dropped, so "expected one
numeric Telegram user ID" misdescribed them. The warning now asks for a
positive numeric ID, matching _parse_telegram_user_id, and the drop test
covers 0 and -5 alongside @alice.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(extensions): add host-bound Agent run control
* fix(extensions): preserve internal launches and scope run control
* docs(extensions): use a plugin-local run operation key
* feat(frontend): support extension providers in composer mentions
* fix(mentions): retain settled plugin suggestions during search
* fix(extensions): validate identifiers and skip empty mention searches
* test: stabilize history pagination after latest page refresh
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): run Discord connection-repository work on the Gateway loop
discord.py delivers messages on a private loop in the client thread, and
the Discord adapter awaited the channel-connection repository there even
though its SQLAlchemy engine and pool belong to the Gateway loop. On
PostgreSQL the first Discord message after the Gateway used the pool
failed with "got Future attached to a different loop" and was dropped; on
SQLite a pool-exhausting burst failed with "Queue is bound to a different
event loop" and left the pool's wait queue bound to the Discord loop, so
the Gateway's own queries then failed the same way.
Resolve the identity together with the intake commit, and run /connect
binding, on the Gateway loop through _submit_threadsafe_coroutine, as
Telegram, Feishu and DingTalk already do. Bind replies return through
_run_on_discord_loop, and start()/stop() now open and drain the
cross-thread submissions.
* docs(changelog): reference #6214 for the Discord connection-loop fix
* fix(channels): correct the unscheduled Discord bind comment and pin it
When _submit_threadsafe_coroutine refuses the bind, base.py closes the
coroutine, so consume_oauth_state never runs and the one-time code stays
valid; only the Discord message is swallowed. Say that instead of
claiming the code is consumed, and pin both halves with a test: the
attempt is handled (never a chat turn) and the code still binds later.
* fix(channels): undo Discord typing when the Gateway hand-off fails
Since identity resolution moved to the Gateway loop, both inbound paths
started typing and the ack reaction as soon as the hand-off was
scheduled. A lookup that then raised (database outage, pool-checkout
timeout) dropped the message but left an endless typing loop and a
success reaction behind.
The hand-off now returns its completion future (new
Channel._submit_threadsafe_coroutine_future, which the bool helper wraps)
and the Discord side waits on it through asyncio.shield. Typing still
registers before the hand-off settles, so a fast reply's _stop_typing
cannot run first; the ack reaction waits for the commit. On failure, or
when stop() drains the hand-off, the indicator this message started is
stopped (never one another in-flight message owns) and no ack is added.
* fix(channels): keep shared Discord typing alive for dependent messages
Two messages to the same target share one typing task: the first starts
it, a later one reuses it. If the creator's hand-off failed after a
follower had committed, or while the follower was still pending, the
creator stopped the indicator the follower relied on, and nothing
restored it while the follower's request ran.
Count the hand-offs that depend on each typing task. A pending or
committed hand-off holds the task (a commit keeps its hold until a reply
stops the indicator), a failed one releases its hold, and the indicator
is stopped only when the failed message started it and no hold remains.
The count is dropped when the task ends or the channel cancels typing.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): ignore literal base tags in HTML artifact previews
* fix(frontend): address HTML preview base review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* docs(readme): sync TOC with headings and fix deploy.sh paths
* docs(readme): move skills content stranded in the RAGFlow chapter back into Skills & Tools
* fix(lark): match broker subcommand denylist through separate option values
The opt-in DEERFLOW_LARK_BROKER_DENY_SUBCOMMANDS check dropped only
`-`-prefixed tokens and compared the rest from the start, so the value in
`--profile work config show` became the leading positional and a
`config show` rule never matched. Real lark-cli 1.0.65 runs `config show`
for that shape and for `config --profile work show`.
The broker cannot know which options take a value, so a rule now matches
when its tokens appear in order among the non-flag tokens. A contiguous
match would still miss a value placed between the path tokens. Argument
values that spell a denied path in order are refused too (fail-closed).
* docs(changelog): reference #6212 for the lark broker denylist fix
* docs(lark): call denylist entries command paths, not prefixes
Entries are matched as in-order subsequences now, so the remaining
'prefix' wording in the BrokerConfig comment, the parse_deny_subcommands
and run_lark_cli docstrings, and the broker README undersold the rule.
Also rewrap the comment and docstring lines this PR had pushed past ~80
columns.
* fix(sandbox): scope ignore patterns to the search root in glob/grep
Rebuilt on top of current main so the changelog entry lands in the
Unreleased (2.2.0) Fixed section rather than the now-released 2.1.0 block.
* test(sandbox): cover the include_dirs glob branch at an ignored search root
The file.list_path branch only ran against a non-ignored tree, so a
regression there would go unnoticed. Two cases now exercise it with an
ignored root: the requested root keeps its contents, and ignored
descendants stay hidden.
* fix(config): reject boolean integer settings in app_config
Follows the loop-detection and circuit-breaker guards: YAML booleans next
to integer knobs load cleanly, and Pydantic coerces booleans to 1/0.
- recursion_limit coerced from `true` to 1, so every Gateway run that does
not supply its own limit hit the LangGraph recursion ceiling at the
first super-step (max_recursion_limit likewise clamped the ceiling to 1).
- The five llm_call integers coerced similarly: retry_max_attempts as
`true` disabled retries, max_concurrent_calls as `true` serialized the
pipeline.
All seven integer fields now fail configuration loading with a
field-specific error, while valid integers and numeric strings keep their
existing behavior. Mirrors the loop-detection helper (#6017) and the
in-file circuit-breaker validator.
Signed-off-by: yetuge <2219677952@qq.com>
* docs(changelog): record the app-config boolean rejection
Changelog entries for #6171, mirroring the #6017 entry format in both
languages.
Signed-off-by: yetuge <2219677952@qq.com>
* test(config): pin the False-side coercion regression in the boolean guards
Per review: the tests only exercised `True`. On the `ge=1` fields `false`
already failed the constraint pre-fix, but on the `ge=0` fields
(`max_concurrent_calls`, the two delay fields) `false` coerced silently to
the harmless-looking `0`, disabling the concurrency cap with no error.
Parametrize both boolean tests over `[True, False]` to lock that regression
in as well.
Signed-off-by: yetuge <2219677952@qq.com>
---------
Signed-off-by: yetuge <2219677952@qq.com>
* fix(mcp): drain the config read-modify-write across cancellation
Route the five MCP config endpoints through the canonical
run_drained_write helper and fold the tools-cache reset into the drained
unit, so a cancelled caller cannot detach from the worker holding the
extensions-config locks or skip the cache invalidation behind a settled
write. Caller-facing HTTPException contracts re-raise unlogged.
Fixes#6209
* fix(mcp): type the drain helper against the worker signature and unlist drain adopters
- _run_drained_mcp_apply now mirrors the canonical helper's [**P, T]
Callable[P, T] typing so each call site's argument list is checked;
- AGENTS.md stops enumerating the adopted routers, so the line stops
needing an edit on every new adopter.
---------
Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>
* fix(community): refuse non-global addresses in the shared SSRF guard
is_blocked_address ORed is_private, is_loopback, is_link_local,
is_reserved, is_multicast and is_unspecified. None of them covers the
100.64.0.0/10 shared address space, which holds CGNAT and Tailscale
hosts and Alibaba Cloud's 100.100.100.200 instance metadata endpoint, so
web_fetch, web_capture, the agentic browser and personal MCP connections
could reach it, including through the IPv4-mapped ::ffff:100.100.100.200
form a DNS answer can carry.
Also refuse any address that is not is_global, the classification the
repo's other SSRF filters already use. Keep the existing flags alongside
it: some non-public forms still report is_global, such as the NAT64
spelling of a metadata address (64:ff9b::a9fe:a9fe) and 224.0.0.1, so an
is_global-only check would newly allow them.
* docs(changelog): reference #6202 in the shared address space entry
* test(community): pin the IPv4-mapped metadata spelling at the web_fetch and personal MCP entry points
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(browser): pin browser connections to the SSRF-vetted addresses
The navigate screen and the per-request guard resolve a hostname to vet
it, but Chromium resolved the name again to open the connection. A
rebinding DNS server could answer both checks with a public address and
the connection with a private or cloud-metadata one, and the browser
followed it.
Launch each private browser through a per-session loopback SOCKS5 proxy.
Over SOCKS5 Chromium sends the hostname itself, so the proxy resolves it
exactly once through the session's egress resolver and connects only to
the addresses that resolver vetted. resolve_browser_egress applies the
same allow_private_addresses policy as validate_browser_url, on top of a
new url_safety.resolve_public_addresses helper that
validate_public_http_url now shares. Loopback is proxied explicitly with
bypass "<-loopback>". The tools, navigate_and_capture and the Live
WebSocket all pass the resolver, because a thread's first caller fixes
its launch options.
CDP-attached Chrome is unchanged. Delegated fetch services (crawl4ai,
Browserless, fastcrw) still resolve on their own side, which the Gateway
cannot pin.
* docs(changelog): reference #6201 in the browser egress pinning entry
* fix(browser): bound egress proxy faults and close, and record the WebRTC gap
Review follow-ups for the pinned browser egress proxy:
- A resolver exception other than ValueError escaped the connection
handler as an unretrieved task error. Log it with its traceback and
reply SOCKS5 general failure (0x01), so a resolver fault stays
distinguishable from a policy refusal (0x02).
- Bound the resolve with its own timeout. The handshake timeout did not
cover it, so a wedged getaddrinfo held the half-open tunnel until
Chromium gave up. A timeout replies host unreachable.
- Bound close(): Server.wait_closed() also waits for a connection whose
handler registered after the writer sweep, which could stall session
close and eviction for the 15 s handshake. A handler that starts after
close now drops its connection without opening a tunnel.
- Docs said a launched browser connects "solely" through the proxy.
WebRTC UDP does not traverse a SOCKS proxy, so AGENTS.md, README (en/zh)
and both changelogs now scope the guarantee to TCP and name the gap.
The English and Chinese READMEs document the Projects feature
(shared instructions, document shelf, archive read semantics, trash),
but the Japanese, French, and Russian READMEs skip it: their heading
structure jumps straight from the embedded Python client to Scheduled
Tasks, and their tables of contents have no Projects entry.
Add the translated section at the same position between the embedded
Python client and Scheduled Tasks, and add the matching table of
contents entry, so all five language variants cover the same 21 H1/H2
sections.
Follow-up in spirit to #4803 and #6129, which synced the Sister
Projects and Startup Modes sections across the localized READMEs.
* fix(tui): preserve multiline pasted input
The TUI composer used a single-line `Input`, so a bracketed paste
containing code, a stack trace, or multiple paragraphs kept only the
first line when submitted. The composer is now a multiline `TextArea`:
Enter still submits, Up/Down move inside the document before falling
back to input history at the first/last line, and the slash palette is
unchanged.
Originally #4630 by @felix-windsor, cherry-picked onto current main;
conflicts in app.py key gating (transcript paging keys), test imports,
and the relocated TUI docs were resolved by hand.
Fixes#4613
* fix(tui): submit multiline input unchanged and document the widgets layer
Address the review on #4630:
- Send non-command input to the agent and history exactly as typed or
pasted. Only the empty-input check uses strip(), so leading
indentation and trailing newlines in pasted code and stack traces
survive. Slash commands are still trimmed so they resolve as before.
- widgets/composer.py imports Textual, so the module-layout notes in
tui/AGENTS.md and docs/TUI.md now exempt widgets/ alongside app.py.
- Cover pressing Down on the last document line (history fallback),
whitespace-only input, and slash commands with surrounding whitespace.
Co-authored-by: Felix <felixwindsor3344@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(tui): treat soft-wrapped rows as composer navigation boundaries
TextArea soft-wraps long lines, and its Up/Down move between the wrapped
rows of one logical line. The history fallback only looked at logical
rows, so Up from the second visual row of a long prompt recalled history
instead of moving the cursor, and Down from its first visual row could
replace the draft with history's empty value.
Use the TextArea navigator's first/last wrapped-row checks for the
boundaries, and cover wrapped prompts in both directions. Also restore a
tui/AGENTS.md sentence about PageUp/PageDown to the app.py entry it
belongs to.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Felix <felixwindsor3344@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(sandbox): pin utf-8 decoding on the local sandbox backend's subprocess calls
The local container backend shells out to docker / the container runtime
with text-mode subprocess calls that do not pass an encoding, so output is
decoded with the platform's preferred encoding. On hosts whose ANSI code
page is not UTF-8 (cp936 Chinese Windows, cp1252 Western European), any
non-ASCII byte in the CLI output raises inside subprocess's reader thread:
stdout/stderr come back None, the JSON-parsing call sites then crash on
None or silently mis-parse, and stderr diagnostics are dropped.
Pin encoding="utf-8" with errors="replace" on every text-mode call in
the module. The container CLIs write UTF-8 on every platform; replace
keeps the parseable portion of any non-conforming output alive instead of
discarding it wholesale.
* test(sandbox): validate UTF-8 subprocess settings per call
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(community): normalize max_results before handing it to Firecrawl and fastCRW
* test(community): pin the explicit-null max_results shape
The parametrized `omitted` case reused None as its "key absent" marker, so
`extra["max_results"] = configured` never ran for None and the key-present,
value-null YAML shape never reached _coerce_max_results. Use a distinct
sentinel for absence and add an explicit-null case per provider.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(typesafe): expose instructions/criteria overrides in the risk-gate eval
The pre-enablement evaluation could not answer the question an operator
actually asks when the gates fail: is the miss/false-block a model limit or
a rubric limit? --instructions/--criteria-true/--criteria-false pass the
provider's rubric overrides through, so threshold, max_state_chars and the
rubric can be calibrated in one loop.
First online run (28 labeled cases, jev-latest): defaults failed
risky_misses_zero (cat ~/.aws/credentials at p=0.47) and
system_false_block_within_target (two long safe calls locally refused over
max_state_chars). With threshold 0.4, max_state_chars 8000 and a
criteria.false clarifying that long-but-fully-shown workspace writes are
local and reversible, every gate passed in two independent runs
(0 missed risky, 0.0% false block; risky floor 0.48, safe ceiling 0.28).
Network p95 passed at 505-620ms in three runs and failed once at 1023ms —
re-measure in the target environment before enablement.
* feat(typesafe): record the effective risk policy in the eval report
A custom-rubric run was indistinguishable from a default-policy run in the
recorded configuration, so its verdict could not be reproduced. Publish the
provider's built-in defaults and write the resolved instructions and both
criteria as report["policy"] alongside model and threshold.
* feat(typesafe): mark the eval policy on stdout and pin flag passthrough
Address the PR #6179 review notes: read the rubric-override flags by
direct attribute access so a plumbing mistake fails loudly instead of
silently evaluating the default policy, print policy=default|custom in
the report header so a custom-rubric run is distinguishable on the
console too, and pin the flag-to-provider passthrough with an offline
provider-construction test.
* fix(channels): send Discord typing indicators through the discord.py 2.x API
_start_typing() called channel.trigger_typing(), which discord.py removed
in 2.0; the project requires discord.py>=2.7.0. The typing loop swallowed
every exception, so each tick raised AttributeError and no indicator was
ever sent, and the tests never noticed because they mocked the removed
method on a SimpleNamespace.
Await channel.typing() instead (awaiting the returned Typing sends one
indicator), and log a failed tick at DEBUG rather than discarding it. The
existing typing-lifecycle tests now mock typing(), and new tests drive
_start_typing() against a fake shaped like the 2.x Messageable, which has
no trigger_typing.
* docs(changelog): reference #6138
* fix(channels): warn on a Discord typing loop's first failed indicator
Logging every failed typing tick at DEBUG left a permanently dead
indicator invisible at the default INFO level, the same silence that hid
the trigger_typing AttributeError. A bot that lost Send Messages in a
channel (403 on every tick) or is being rate limited could not be told
apart from a working one without enabling DEBUG.
Log the first failure of each typing loop at WARNING with the traceback,
and later ticks of that loop at DEBUG so a retrying loop cannot flood the
log. The level is per loop, so the next reply's loop warns again.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
terminate() is TerminateProcess on Windows: langgraph dev never runs its
shutdown hooks, so .langgraph_ops.pckl is never flushed and state does not
survive the restart the two cross-restart tests exercise. Create the child
with CREATE_NEW_PROCESS_GROUP and stop it via CTRL_BREAK_EVENT (handled by
uvicorn as SIGBREAK) on Windows, keeping terminate() on POSIX and a hard
kill as the timeout fallback.
Fixes#6196
Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>
* test(blocking-io): catch socket.getaddrinfo on the event loop
Blockbuster wraps socket.socket methods but not the module-level
resolver, so a synchronous hostname lookup on the loop passes the strict
gate. The SSRF guard in deerflow.community.url_safety resolves with
socket.getaddrinfo from async web tools, and the anchors added next stay
green against that unfixed code without this rule.
Add socket.getaddrinfo to _PROJECT_BLOCKING_RULES, scoped to app.* and
deerflow.* callers. loop.getaddrinfo and asyncio.to_thread resolve in a
worker thread and are not flagged. The full make test-blocking-io suite
stays green with the rule active (187 passed).
* fix(community): resolve SSRF-guarded hostnames off the event loop
validate_public_http_url resolves hostnames with blocking
socket.getaddrinfo. The crawl4ai and Browserless web_fetch, web_capture,
browser_navigate, the Gateway browser navigate route, and the Live
stream's navigate input and seed all called it directly from async code,
so a slow DNS answer for a model- or user-chosen URL stalled the Gateway
event loop for the whole lookup.
Run the guard through asyncio.to_thread at each of those call sites, the
same way personal_network already offloads it. The screening result is
unchanged. fastcrw's web_fetch is a sync tool and personal MCP validation
already runs in a worker, so neither needed a change.
* docs(changelog): reference #6140 in the SSRF guard event-loop entry
* fix(browser): run the Playwright request guard off the shared browser loop
BrowserSession._install_request_guard screens every redirect hop, iframe
and subresource by calling the SSRF guard inside the async route
callback. The guard resolves hostnames with blocking socket.getaddrinfo,
and that callback runs on the process-wide Playwright loop, which also
pumps Live frames and input for every browser session, so one slow
lookup stalled all of them.
Run the guard through asyncio.to_thread there as well, and pin it with a
strict-gate anchor that drives the real route handler. Also note in the
url_safety docstring that the screen is check-time only: a caller that
re-resolves at connect time is not pinned to the vetted IPs.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
