* refactor(skills): keep allowed-tools as compatibility metadata
* docs(skills): align current guides with agent tool authority
* test(skills): simplify authority regressions and sync Korean docs
* fix(channels): keep staging cleanup from replacing the save error
The channel stores publish state through a staged temporary file and, if the
rename fails, close and unlink that file inside the handler before re-raising.
Neither step was guarded, so a failing close skipped the unlink and a failing
unlink replaced the error that actually caused the save to fail -- leaving the
staged copy, which for the runtime-config and WeChat stores holds an 0600 IM
token, behind with nothing reporting it.
Make both steps independently best-effort, name the leaked file in a WARNING,
and suppress the diagnostic itself so a broken log handler cannot become the
reported failure. This is the same contract PR #6871 establishes for browser
captures, applied to the three JSON publishers in app/channels.
* fix(channels): address staging cleanup review follow-ups
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): preserve Markdown table cells in data-analysis exports
* docs(skills): document Markdown cell escaping contract
Clarify handling of Markdown exports and escaping rules.
* docs: describe data-analysis Markdown export escaping
Clarify data-analysis skill features and behavior in README.
* test(skills): cover Markdown export headers and multiline cells
Add a test for markdown export functionality in data analysis.
* feat(decisions): add an OpenAI Decisions backend for the decision slots
Issue #7005 asked for a second decision provider behind the interfaces the
TypeSafe (Jev) integration already uses, not a new decision system.
`deerflow/openai_decisions` speaks `POST /v1/decisions`: a `noul` question
becomes a `predicate` whose true/false rubric folds into `instructions` (the
endpoint has no criteria field, and dropping it would move the decision
boundary while the adapter kept reporting the same criteria in its policy
identity), answers arrive as an array rather than a mapping, and a refusal is an
answer `type` rather than an HTTP failure -- so it lands in the existing
per-question error channel and each consumer keeps its own failure direction.
Transport is reused rather than copied. `provider_label` and the connection's
`endpoint_path` are the only two seams a dialect needs, so retry and backoff,
the deadline budget, the bounded never-decoded body read and the error taxonomy
keep one owner; `resolve_decisions_connection` pairs the two so a Decisions
connection cannot be built pointing at System One. The three adapters override
their connection resolution, their client, and the value the endpoint judges
(`input` text instead of a `state` object), and inherit thresholds, rubrics,
caches and audit policy unchanged. Wiring stays a `use:` class path, so no
config schema is added.
* fix(decisions): isolate endpoint identities and correct diagnostics
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): quarantine unreadable memory.json in the read-error branch too
MarkdownMemoryStorage._load_memory_file quarantined summaries that failed
to decode as JSON/Markdown, but the earlier except (OSError, UnicodeError)
branch (unreadable bytes, e.g. a UTF-16 BOM) still returned None without
quarantining. The leftover file was then silently overwritten by the next
save, destroying the user's memories with no backup.
Call the existing best-effort self._quarantine_unreadable(path) there as
well, matching the documented quarantine contract, and add a regression
test asserting the original bytes survive a subsequent load+save.
Fixes#7072
* fix(memory): preserve summaries on transient read failures
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(tui): accept whitespace separators in slash commands
* fix(tui): align slash palette with Unicode whitespace
Close the slash palette on every whitespace separator accepted by the command resolver, including pasted tabs and Unicode spaces. Document the accepted delimiters and cover parser arguments, palette closure, Enter submission and Tab completion.
Validation: 352 focused tests, 34,089 offline backend tests and 262 blocking-I/O tests passed. Backend formatting, frontend check and agent guidance passed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(docker): make dev provisioner opt-in via compose profile
Gate the Kubernetes provisioner behind a Compose profile and have the
wrapper enable it only when config selects provisioner sandbox mode.
This lets the default Docker Compose stack start without kubeconfig.
Fixes#3232
* test(docker): render every compose profile in the env-file regression test
main added backend/tests/test_compose_default_bind_host.py, whose _render helper asserted that the dev provisioner still loads ../.env. `docker compose config` without a profile omits profile-gated services, so that assertion failed once the provisioner became opt-in.
Pass `--profile *` from the helper: the test is about an existing .env reaching every service, so it has to render the services a profile gates rather than only the ones `up` starts by default.
* docs(docker): restore script mode and localize direct Compose guidance
The dev provisioner is now opt-in, but the earlier edit also flipped scripts/docker.sh from 100755 to 100644 even though README.md still documents direct `./scripts/docker.sh` invocation. Restore the executable bit.
Also add the direct-Compose guidance to the translated READMEs so readers of README_zh.md, README_ja.md, and README_fr.md learn that `--profile provisioner` selects the Kubernetes mode and the default stack does not need Kubernetes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skill-creator): tolerate non-object timing measurements
Treat optional timing JSON values that are not objects as missing measurements so they cannot abort benchmark aggregation. Preserve grading durations, character-count fallback and recorded zeros; add regression coverage for both layouts and subsequent runs.
* fix(skill-creator): warn and validate timing measurements
* fix(skill-creator): tolerate unreadable timing files
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): parse a BOM-prefixed DeerMem fact instead of calling it corruption
Facts are hand-editable Markdown files, so a Windows editor can save one as
"UTF-8 with BOM" -- the same exposure #7083 fixed for memory.json. The fact
parser checks `text.startswith("---\n")` against a plain-utf-8 read, so the
BOM makes a valid fact read as "missing YAML front matter" and raises
MemoryStorageCorruption, which the module docs describe as taking down the
whole agent.
Read the fact with utf-8-sig, matching the summary loader and the SOUL.md /
SKILL.md convention already recorded in AGENTS.md. `_parse_listed_fact`
delegates here, so the directory-scan path is covered by the same change.
Measured: the new BOM test fails on the base revision with exactly that
"missing YAML front matter" corruption and passes at head; the no-BOM control
passes both. 49 tests in test_memory_storage.py, 106 across the memory
suites, ruff check/format clean. Mutation-checked: reverting to plain utf-8
fails only the BOM case.
* test(memory): cover BOM facts through the public storage API
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(github-research): resolve the tree against the repository's default branch
The tree command asked for "main" and, when that was missing, for "master".
A repository whose default branch is neither gets a name-guessing fallback, and
when the stale "master" still resolves GitHub returns it happily: the agent is
handed the tree of an abandoned branch and reports it as the current structure.
Ask for HEAD instead, which GitHub resolves to the repository's own default
branch, so no candidate names are guessed and no extra request is added.
* test(github-research): clarify stale branch tree comment
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: support client context capabilities
(cherry picked from commit b7c223f9415dcd545cbe05a5f5a6e47073a42d68)
[Totoro-qaq: resolved onto main. The persisted-reminder middleware hunks and their tests are dropped (delivery is redesigned in a follow-up); the RunCreateRequest description moved to app/gateway/run_models.py.]
* fix: address client context review comments
(cherry picked from commit bce6da6b819228eda58e887872720b6f9b5fab69)
[Totoro-qaq: middleware hunks dropped with the redesigned delivery.]
* feat(agents): deliver client context as a request-scoped data block
The cherry-picked #3106 put context.client into the persisted first-turn
reminder, with change detection and a "not provided" clear marker. Main
has since moved the date reminder to a SystemMessage, made request-scoped
context (the project block, #5443) request-only, and agents/AGENTS.md now
requires user-influenced values to ride the HumanMessage data channel,
never system text.
Render runtime.context["client"] on every model call in
DynamicContextMiddleware.wrap_model_call/awrap_model_call as one hidden
HumanMessage right after the leading SystemMessages, where DurableContext
puts its data. It is never a state update and never flagged
dynamic_context_reminder, so nothing is checkpointed, summarized or
replayed, and a run without a client gets no block. An unchanged client
keeps the previous request as the next request's prefix.
The renderer sanitizes again (direct graph callers bypass the Gateway),
html-escapes values and frames them with frame_untrusted_text: the same
--- BEGIN/END USER INPUT --- boundary the user's own turn and the MCP task
notification get. The lead prompt treats everything outside those markers
as framework data, so unframed values would rank above the user's own
text; framed, context.client can say nothing the same caller could not
already say in input.messages. client_context joins _BLOCKED_TAG_NAMES so
a forged block in user input or a tool result is escaped, and one static,
value-free sentence in the lead prompt names the block as untrusted hints,
never instructions, permissions or identity. The now unused empty-context
marker is removed.
* fix(gateway): log dropped context.client fields without values
A run request is never rejected over context.client: unknown fields,
nested values and malformed entries are silently dropped, the same
leniency as the other context keys. That left no signal when a custom
client sent something the Gateway ignored (review on #3106).
Route both the config.context.client and the body context.client paths
through one helper that emits a debug line whenever sanitizing changed
the value. Like the malformed client_timezone branch, the line carries no
values; unlike client_timezone, client.name and preference strings are
free caller-controlled text, so they are never logged, not even
sanitized. A clean client, an explicit null or an absent key logs
nothing.
Tests also pin the config-only path (sanitized before the early return)
and the end-to-end start_run admission: the sanitized client reaches
config["context"] and never configurable.
* fix(runtime): bound numeric client context preferences
String preferences are capped at 200 characters, but integers were kept
at any size. A JSON body may carry integers of up to 4300 digits, so 32 of
them rendered a block of about 138,000 characters, re-sent on every model
call of the run. In-process callers could also pass an int past the str()
digit limit, which failed the model call, or a number subclass with its
own __str__.
Keep integers within JavaScript's Number.MAX_SAFE_INTEGER, the range a
browser client can send exactly, and coerce numbers to plain int and
float.
* docs(api): describe client context as advisory request-scoped hints
The cherry-picked section described a hidden reminder that no longer
exists. Document what a custom client can rely on: the allow-list and its
limits (strings, the integer range, and only the first 32 entries of each
map), debug-level logging of dropped fields, runtime-only forwarding, and
delivery as a per-request user-role block, framed like user input, rather
than system text or thread history.
Only top-level context.client is documented. An explicit
config.context.client is sanitized for the run as well, but the run record
keeps the caller's config as sent, so it is not offered as an alternative.
It also answers the #3106 review: capabilities and preferences are
advisory hints, not enforcement, which belongs outside the model; name is
informational and never for authorization; the same object should be
sent on every run to keep the cached prefix stable; subagents and
embedded DeerFlowClient runs do not receive it, and server-started runs
(schedules, IM channels, GitHub agents, MCP notifications, extensions
unless the plugin passes one) carry none.
* fix(context): declare client delivery and keep client out of configurable
The request-only <client_context> block is model-visible, so
DynamicContextMiddleware now declares its delivery and the sanitizer's
limits in release_policy_parameters(); runs before and after this change
no longer share a policy fingerprint.
build_run_config copies body.config['configurable'] verbatim and
configurable is persisted with checkpoints, so a client placed there is
dropped instead of being stored raw. Client context stays runtime-only.
Document context.client in the README and in the middleware guide.
* fix(context): apply the PII redaction policy to the client block
DynamicContextMiddleware adds the <client_context> message inside the
outer PiiRedactionMiddleware, and the hidden message is not re-sanitized,
so with pii_redaction enabled a preference such as an email address
reached the model verbatim while the same address in the user turn was
redacted. Pass the rendered block through the shared redact_text seam
with this middleware's app_config.pii_redaction, and declare the setting
in release_policy_parameters().
* fix(context): keep client hints out of stored run config
---------
Co-authored-by: Blair Leng <25675774+BlairLeng@users.noreply.github.com>
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
* fix(sandbox): search long lines for literal grep patterns
* test(sandbox): cover literal long lines across providers
* fix(sandbox): bound case-insensitive literal grep work
Replace repeated-prefix regex searches with KMP over single-character Unicode case matches. Keep local long-line literal results and existing regex and preview limits. Add a subprocess-deadline regression for the review counterexample and case-semantics coverage.
* docs(sandbox): keep grep guidance within inherited size limit
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skill-creator): preserve recorded token counts in benchmarks
Read timing.json token measurements independently of grading duration and retain recorded zero values. Keep the existing character-count fallback for missing measurements and add benchmark/report regression coverage.
* fix(skillscan): distinguish token metrics from credentials
Recognize typed token counts, numeric benchmark summaries, and signed deltas so skill-creator documentation passes deterministic review. Preserve credential names, string values, duplicate-key detection, and cloud-token findings; add regression coverage and document the boundary.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(searxng): contain malformed search payloads instead of crashing on them
`_search_page` indexed the decoded body directly:
data = resp.json()
return data.get("results") or []
so a self-hosted instance that answers HTTP 200 with something other than a
result page -- an HTML or plain-text error page from a reverse proxy or captive
portal -- reached the model as a raw Python error, or as an empty search:
{"results": "maintenance window"} -> {"error": "'str' object has no attribute 'get'"}
{"results": {"a": 1}} -> {"error": "'str' object has no attribute 'get'"}
{"results": [1, 2, 3]} -> {"error": "'int' object has no attribute 'get'"}
{"results": [None]} -> {"error": "'NoneType' object has no attribute 'get'"}
Worse, a single malformed entry in an otherwise valid page crashed the whole
call and discarded every good row:
{"results": [{"title": "ok", ...}, 7]} -> {"error": "'int' object has no attribute 'get'"}
This is the containment rule the sibling providers already follow (brave #6550,
groundroute #6603, tavily #6645, sofya #6719, serper #6779, serply #6796,
tencent-wsa #6799): a payload that is not a result page is a format error, not an
empty result, because "No results found" tells the agent the query legitimately
matched nothing and ends a research loop as "no hits".
The client now raises `SearxngResponseError` for a payload that is not an object,
a body that is not JSON, a `results` container that is not a list, and a
non-empty list holding no object entries, while still skipping non-object entries
in an otherwise valid page so the valid rows survive. `web_search_tool` reports
the provider's format error instead of leaking the exception text. Missing, null,
and empty `results` keep the existing "No results found" behavior.
Diagnostics log the container field name and its type, or the absence of usable
objects -- never the query, credentials, or payload values.
* test(searxng): pin the mid-walk format-error trade-off and fix the docstring
The paginating client aborts the walk and discards already-collected rows when
a later page comes back malformed, which is a different outcome from the
single-request siblings. Pin that trade-off with a test so the behaviour is a
deliberate decision rather than an accident, and reword the
_unexpected_format_error docstring: it promises the query is not echoed, but
the sibling providers all echo it back on purpose and only withhold the
payload.
* test(searxng): log skipped entries and pin the dedicated format-error handler
Two follow-ups from review, both about making the family contract observable
and pinned:
- Skipped non-object entries now log their index ("Skipping malformed SearXNG
result entry at results[1]"), matching the merged InfoQuest sibling. SearXNG
is the provider where drift is most expected -- a self-hosted instance serves
whatever sits behind its reverse proxy -- and a search spans up to
_MAX_PAGES requests, so rows could otherwise be shed with no trace at all.
- The dedicated `except SearxngResponseError` branch had no test that could
fail: the exception is a ValueError whose message is already the format-error
string, so the generic handler returns byte-identical JSON. The new test
asserts the generic branch's log line is absent, which is what actually
distinguishes the two paths.
* docs(searxng): name the empty result it actually returns, and pin the dedicated except
Two low findings from the follow-up review:
- The README said missing/null/empty `results` keep the existing "No results
found" response. That response does not exist in this provider: all three
produce a JSON empty list, and the string appears only in comments
explaining what must NOT be reported. The family convention distinguishes
(Tavily: "`[]` for search, Error: No results found for fetch"; InfoQuest:
"keep the existing empty result"), so this now reads "keep the existing
empty result (`[]`)".
- The client-level `except SearxngResponseError: raise` was unpinned: deleting
it left the whole file green, because the tool-level test mocks the client
and the payload tests only assert the raise type. The observable regression
is the generic branch logging "An unexpected error occurred" for every
contained page, so the existing payload test now asserts that line is absent
from caplog.
Mutation check: removing the clause now fails 8 parameterised cases (was 33
green); with the clause in place the suite is 33 passed and 91 across the two
provider files, ruff check/format clean.
* feat(channels): share encrypted runtime channel config across Gateway instances
IM bot credentials entered in the web UI lived in each host's plaintext
channels/runtime-config.json, and the router cached the merged config on
app.state for the life of the process. With several Gateway instances a
save or disconnect served by an instance that does not hold the
provider's leader lease never reached the transport holder: it kept the
old credentials until it next started the channel, and a disconnect
could leave the old bot connected.
- New shared table channel_runtime_configs (migration 0044): one row per
provider with the config block encrypted under DEER_FLOW_CREDENTIALS_KEY,
a plaintext disconnect tombstone, and a version bumped by every save.
- SqlChannelRuntimeConfigStore replaces the JSON file for sqlite/postgres
(memory keeps the file); it is synchronous on app_sync_sqlalchemy_url
because its callers already run in worker threads.
- The router re-reads the store on every request instead of caching the
merged config, and refuses (503) to save credentials without a key.
- The leader-lease supervisor compares the versions on each renewal tick
and restarts or stops the holder's transport when a provider's config
changed; a takeover reads a disconnect instead of reviving cached
credentials.
- The first start imports an existing runtime-config.json (rows already
in the table win) and deletes the plaintext file.
* docs(changelog): reference #6988 for the shared runtime channel config
* ci: rerun checks after #6999 fixed the frontend guidance budget on main
* fix(channels): start a held transport that is down when its runtime config changes
When the lease holder's transport was down (a start that failed on the
old credentials), applying a peer-saved config only updated the cached
config, and the transport started on the next lease round: two renewal
ticks instead of the one the docs promise. The apply step now starts it
on the held lease right away, with the lease round's backoff bookkeeping.
* fix(browser): never replace an existing output file with browser_screenshot
browser_screenshot published straight onto outputs/<name> through the
replacing atomic publisher, so a screenshot saved as report.png silently
replaced the user's own report.png, an earlier capture, or another
default-named capture taken in the same second (the default stem has
second resolution). The sibling web_capture tool already keeps existing
files and takes a -1, -2, ... suffix.
Stage the complete capture under a hidden sibling and hard-link it onto
the first free candidate name. The link is the whole no-overwrite guard:
it fails with FileExistsError when the name exists as anything, so no
name is checked before it is claimed and a concurrent writer cannot be
replaced; the final name never shows a partial file. Staging shares the
outputs directory, so the link is same-filesystem, and like Gateway
uploads other link failures propagate instead of falling back to
replacement. New files keep the publisher's normal creation mode.
Hidden per-action progress frames keep the replacing _write_screenshot:
their names carry microsecond timestamps in a server-owned directory.
* docs(changelog): reference #6913 for the browser_screenshot no-overwrite fix
* fix(browser): refuse a staged screenshot swapped before it is linked
The sandbox can write outputs/, so it can see the hidden staged capture
and replace it with a symlink before the hard link. os.link follows
symlinks by default, which would publish a hard link to the host file as
the screenshot artifact (the replacing publisher this PR moved away from
never followed one).
Stage the capture with an exclusively created descriptor and keep its
identity from fstat, link without following symlinks where the platform
supports it (decided once at import), and require the published name to
be that inode; anything else is removed and the capture fails, like
Gateway uploads' identity check. A failed staging write still leaves no
files behind.
* docs(readme): note that browser_screenshot never replaces an output file
* fix(browser): keep the staged screenshot open until its link is verified
Linux filesystems reuse a freed inode number at once. Closing the staged
capture before linking let a file swapped in at the staged name take the
same number and pass the samestat check, so the symlink and planted-file
swap tests failed on Linux CI while passing on macOS. Hold the staged
descriptor through the link and identity check, as Gateway uploads do,
and require the published name to be a regular file.
* test(browser): race the peer claim against the real publication step
The claimed-name test patched _publish_bytes_atomically, which the
screenshot path no longer calls, so its hook never fired and the race
assertions passed vacuously. Plant the peer file right before os.link or
os.replace lands bytes on a visible name, and assert that the claim
happened, the capture moved to report-1.png, and the peer file survived.
* refactor(capture): share one output-name candidate sequence
browser_screenshot hand-copied web_capture's probe bound and suffix
order, and the copies had already drifted: Browserless retried a held
timestamped name five times, the screenshot tool once. Move the bound,
the retry count and the candidate order into utils/output_names.py and
have both tools iterate it, so the naming cannot drift again. Browserless
behavior is unchanged; its tests now patch the shared module.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(infoquest): screen result URLs through the shared public-URL owner
Three providers already delegate result-URL screening to
community/url_safety.safe_public_result_url; infoquest registers the
image_search tool too but imported the owner zero times, so organic,
news and image rows all handed raw provider URLs to the model. A row the
owner rejects is now dropped rather than raised, keeping the containment
rule #6932 established for this client.
tests: test_infoquest_result_url_safety.py (new, red on main),
test_image_result_url_safety.py (infoquest added to the pinned modules).
* fix(infoquest): reject unscreened search response fallbacks
---------
Co-authored-by: ZHOU YANG <236804849+Zhouyang66639@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(streaming): retire memory buffers and wake existing subscribers
* fix(streaming): preserve end for fully consumed retired streams
* docs: keep frontend guidance within its hard budget
* fix(streaming): report gaps when retirement wins initial attachment
* docs(docker): keep UV_INDEX_URL on the index pinned by uv.lock
The README told users on restricted networks to export
UV_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple, but the gateway
image installs with `uv sync --locked` and uv.lock is locked to
https://pypi.org/simple, so any other index fails the build with "The
lockfile at `uv.lock` needs to be updated, but `--locked` was provided"
(#7003).
Document the constraint instead of the mirror: name the pinned index,
quote the exact failure, and note that installing from a mirror requires
regenerating the lock or dropping --locked. CONTRIBUTING.md already
showed the working value and gains the same note, and README_pt.md and
README_ko.md carried the same sentence and are updated to match.
* docs(docker): correct the NPM_REGISTRY scope and qualify the repro
NPM_REGISTRY is not frontend-only: the gateway build also consumes it.
docker/docker-compose.yaml (lines 95 and 117) and
docker/docker-compose-dev.yaml (lines 154 and 189) pass it as a build arg
to both the frontend and the gateway image, and backend/Dockerfile declares
ARG NPM_REGISTRY (line 16) and applies it via npm config set registry
(line 43) for the @larksuite/cli install.
While re-verifying the uv repro on this head, also found that the quoted
uv sync --locked --check only discriminates between indexes when the local
environment is already in sync; if the environment is stale it reports the
stale environment for either index. Point at uv lock --check for that case.
Reported-by: willem-bd
---------
Co-authored-by: D41910 <d41910@users.noreply.github.com>
