* feat(deploy): select production Compose env files
Use an operator-selected dotenv file consistently for interpolation, container environment, secret probes and build extras. Preserve default selection and shutdown after a profile is removed.
Co-Authored-By: Codex <noreply@openai.com>
* fix(deploy): address production env selection review
Clarify Docker launcher entry points and render real Compose through the launcher to verify both service environments.
Co-Authored-By: Codex <noreply@openai.com>
* test(deploy): cover unreadable env selectors
Reject unreadable explicit selectors before Docker is invoked, using POSIX permission checks and restoring the fixture mode in finally.
Co-Authored-By: Codex <noreply@openai.com>
* test(deploy): scope dotenv assertions to selected secrets
---------
Co-authored-by: Codex <noreply@openai.com>
* chore(doc):sync the chinese changelog info
* chore(doc):sync the changelog with 2.2.0 milestone
* chroe(doc):sync the Chinese changelog with the english one
* fix(memory): persist the DeerMem extraction watermark across restarts and replicas
The updater's per-thread extraction watermark lived only in an in-process
LRU, so a restart or a second Gateway replica on the same storage root fed
already-extracted turns to the memory LLM again.
FileMemoryStorage now keeps it in users/{user}/agents/{agent}/.metadata/
watermarks.json, keyed by thread id, written atomically under the per-user
lock without touching memory.json and capped by watermark_max_keys. Each
update feeds after the furthest of the LRU entry and the persisted one, and
writes through after an extraction or a pre-screen skip; bypass_watermark is
unchanged and sidecar IO failures fall back to the LRU. Memory clears remove
the file with the other .metadata sidecars. Content-based message identities
are stored as SHA-256 digests so the sidecar holds no conversation text.
* docs(changelog): link the memory watermark entry to #7204 and #7205
* fix(memory): never replace an unreadable watermark sidecar
set_extraction_watermark rewrites the whole watermarks.json, and the shared
sidecar reader turns an OSError into an empty mapping, so a failed read could
replace every other thread's watermark with the current one (or delete the
file). The watermark sidecar now has its own reader: a missing file reads as
empty, malformed content reads as empty (nothing to keep; the next write
repairs it), and any other OSError propagates before anything is written. The
updater already logs that and keeps the extraction successful.
* fix(memory): keep the persisted watermark when the last message has no identity
_message_identity returns None for a last message with neither an id nor
text. The updater wrote that None through, which deleted the thread's entry
from watermarks.json, losing a watermark a peer may have advanced and forcing
a full feed on every replica. The write-through now skips None: the persisted
entry still marks an extracted prefix and the furthest-of-both read keeps
using it.
* fix(skills): keep load error text out of user-facing skill read 500s
GET /api/skills, /api/skills/custom and /api/skills/{name} are open to
every signed-in user, but their 500 detail echoed str(e). A malformed or
unreadable extensions_config.json (re-read on every load by the
user-scoped storage) made all three return the file's absolute host path,
plus the [Errno 13] text for a permission error.
Return a fixed detail like the diagnostics and reload routes already do,
chain the cause, and keep the full error in the Gateway log.
* docs(changelog): reference #7155 for the skill read 500 detail fix
* test(skills): pin capability skill discovery 500 through the real Gateway app
GET /api/capabilities/installations/skills reads the same user-scoped
storage as the skill routes but has no handler of its own. Its 500 body
is TraceMiddleware's fixed fallback, so drive create_app() with a
malformed extensions_config.json and assert no host path reaches the
body on any of the four reads.
* fix(sandbox): measure warm-pool idle age on a monotonic clock
Warm-pool expiry measures a duration, but every provider stamped and reaped
entries using the wall clock. A forward correction could therefore destroy
fresh sandboxes, while a backward correction could retain expired ones until
the clock caught up.
Move every warm-pool writer and reader to time.monotonic(), including the
Tenki and OpenSandbox reset paths missed by the earlier closed attempt. Keep
absolute SandboxInfo.created_at and reconciliation grace timestamps on the
wall clock. Convert AIO's sibling active-idle timestamps as one clock domain,
and update finite clock mocks for the additional duration read.
Fixes#6970
* test(sandbox): keep active-idle race fixtures in the monotonic domain
The production AIO activity clock moved to monotonic time, but the shared
orphan-reconciliation helper still stamped active fixtures with wall time.
CI then computed a negative idle age, so two teardown race tests never reached
the ownership gate. Use the same monotonic domain as production.
* test(sandbox): read warm-pool idle fixtures on the monotonic clock
The idle-age conversion left several fixtures seeding the fields the reaper
now reads with values from the clock it no longer reads:
- `test_idle_reaping_forces_a_new_remote` froze `warm_pool_lifecycle.time.time`,
so the benchmark's reaper compared against real elapsed monotonic seconds —
zero over the test's lifetime. Nothing expired, the entry survived, and the
second turn came back as a warm hit.
- Three orphan-reconciliation fixtures seeded `_last_activity` with `0.0` and
two aio fixtures with `time.time()`, both of which are far below a running
monotonic clock: the computed idle age went negative, so the reaper never
reached the ownership claim these tests are built to exercise.
Seed them from `time.monotonic()` instead, and keep `created_at` (a wall-clock
field) on `time.time()`.
* refactor(skills): keep allowed-tools as compatibility metadata
* docs(skills): align current guides with agent tool authority
* test(skills): simplify authority regressions and sync Korean docs
* fix(channels): keep staging cleanup from replacing the save error
The channel stores publish state through a staged temporary file and, if the
rename fails, close and unlink that file inside the handler before re-raising.
Neither step was guarded, so a failing close skipped the unlink and a failing
unlink replaced the error that actually caused the save to fail -- leaving the
staged copy, which for the runtime-config and WeChat stores holds an 0600 IM
token, behind with nothing reporting it.
Make both steps independently best-effort, name the leaked file in a WARNING,
and suppress the diagnostic itself so a broken log handler cannot become the
reported failure. This is the same contract PR #6871 establishes for browser
captures, applied to the three JSON publishers in app/channels.
* fix(channels): address staging cleanup review follow-ups
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): preserve Markdown table cells in data-analysis exports
* docs(skills): document Markdown cell escaping contract
Clarify handling of Markdown exports and escaping rules.
* docs: describe data-analysis Markdown export escaping
Clarify data-analysis skill features and behavior in README.
* test(skills): cover Markdown export headers and multiline cells
Add a test for markdown export functionality in data analysis.
* feat(decisions): add an OpenAI Decisions backend for the decision slots
Issue #7005 asked for a second decision provider behind the interfaces the
TypeSafe (Jev) integration already uses, not a new decision system.
`deerflow/openai_decisions` speaks `POST /v1/decisions`: a `noul` question
becomes a `predicate` whose true/false rubric folds into `instructions` (the
endpoint has no criteria field, and dropping it would move the decision
boundary while the adapter kept reporting the same criteria in its policy
identity), answers arrive as an array rather than a mapping, and a refusal is an
answer `type` rather than an HTTP failure -- so it lands in the existing
per-question error channel and each consumer keeps its own failure direction.
Transport is reused rather than copied. `provider_label` and the connection's
`endpoint_path` are the only two seams a dialect needs, so retry and backoff,
the deadline budget, the bounded never-decoded body read and the error taxonomy
keep one owner; `resolve_decisions_connection` pairs the two so a Decisions
connection cannot be built pointing at System One. The three adapters override
their connection resolution, their client, and the value the endpoint judges
(`input` text instead of a `state` object), and inherit thresholds, rubrics,
caches and audit policy unchanged. Wiring stays a `use:` class path, so no
config schema is added.
* fix(decisions): isolate endpoint identities and correct diagnostics
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): quarantine unreadable memory.json in the read-error branch too
MarkdownMemoryStorage._load_memory_file quarantined summaries that failed
to decode as JSON/Markdown, but the earlier except (OSError, UnicodeError)
branch (unreadable bytes, e.g. a UTF-16 BOM) still returned None without
quarantining. The leftover file was then silently overwritten by the next
save, destroying the user's memories with no backup.
Call the existing best-effort self._quarantine_unreadable(path) there as
well, matching the documented quarantine contract, and add a regression
test asserting the original bytes survive a subsequent load+save.
Fixes#7072
* fix(memory): preserve summaries on transient read failures
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(tui): accept whitespace separators in slash commands
* fix(tui): align slash palette with Unicode whitespace
Close the slash palette on every whitespace separator accepted by the command resolver, including pasted tabs and Unicode spaces. Document the accepted delimiters and cover parser arguments, palette closure, Enter submission and Tab completion.
Validation: 352 focused tests, 34,089 offline backend tests and 262 blocking-I/O tests passed. Backend formatting, frontend check and agent guidance passed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(models): fall through an unusable credential source instead of ending the lookup
load_claude_code_credential's env branch returned None even when
_credential_from_direct_token rejected the value, so a valid
~/.claude/.credentials.json was never consulted for a blank token.
load_codex_cli_credential's 'or' chain stops at the first truthy layout,
so a blank or non-string top-level access_token hid a usable 'token' or
'tokens.access_token' in the same file -- and made the loader disagree
with scripts/doctor.py, which validates all three layouts.
Fixes#7139
* fix(models): try both env token sources and log the rejected one
A whitespace-only CLAUDE_CODE_OAUTH_TOKEN is rejected by _credential_from_direct_token, but os.getenv(A) or os.getenv(B) never reads ANTHROPIC_AUTH_TOKEN while A is truthy, so the blank first variable hid a usable second one. Test each source with the same non-blank rule, and log the rejected variable at debug (no token material) so the resolution outcome stays observable.
Also refresh the doctor.py mirror comment: the loader rejects whitespace-only tokens in every layout now, so the documented strictness divergence is gone.
* style(tests): format credential fallback log assertion
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(docker): make dev provisioner opt-in via compose profile
Gate the Kubernetes provisioner behind a Compose profile and have the
wrapper enable it only when config selects provisioner sandbox mode.
This lets the default Docker Compose stack start without kubeconfig.
Fixes#3232
* test(docker): render every compose profile in the env-file regression test
main added backend/tests/test_compose_default_bind_host.py, whose _render helper asserted that the dev provisioner still loads ../.env. `docker compose config` without a profile omits profile-gated services, so that assertion failed once the provisioner became opt-in.
Pass `--profile *` from the helper: the test is about an existing .env reaching every service, so it has to render the services a profile gates rather than only the ones `up` starts by default.
* docs(docker): restore script mode and localize direct Compose guidance
The dev provisioner is now opt-in, but the earlier edit also flipped scripts/docker.sh from 100755 to 100644 even though README.md still documents direct `./scripts/docker.sh` invocation. Restore the executable bit.
Also add the direct-Compose guidance to the translated READMEs so readers of README_zh.md, README_ja.md, and README_fr.md learn that `--profile provisioner` selects the Kubernetes mode and the default stack does not need Kubernetes.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): answer invalid custom-skill history names with 400
GET /api/skills/custom/{skill_name}/history let the ValueError from
validate_skill_name fall through to its catch-all, so a malformed name
returned 500 and logged a traceback on every request, while delete and
rollback answer 400.
Validate the name up front instead of adding a handler-wide
except ValueError: a corrupt history line raises JSONDecodeError, also a
ValueError, and must stay a server error. Both outcomes are pinned.
* fix(skills): keep corrupt history a 500 on rollback, validate delete names early
Sibling sweep of the history fix. Rollback's handler-wide except ValueError
also caught json.JSONDecodeError, so a corrupt history record answered 400
with the parser message and no log; it is now a logged 500 like history.
Delete only failed an invalid name inside the drained mutation tail, after
taking the owner's mutation lock, and logged it as a tail error although
_drain_skill_mutation keeps caller 4xx unlogged. It now rejects the name
before any storage work. Each guard is pinned by a mutation-checked test.
* docs(changelog): reference #7123 for the skill history status fix
* fix(skills): treat non-UTF-8 skill history as a server fault on rollback
UnicodeDecodeError is a ValueError too, so a non-UTF-8 history file still
reached rollback's ValueError -> 400 mapping while GET .../history answered
the same file with 500. Catch it alongside JSONDecodeError and pin both
corruption shapes on both routes.
Also drop the whitespace-only separator lines the changelog merge added
between other entries.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skill-creator): tolerate non-object timing measurements
Treat optional timing JSON values that are not objects as missing measurements so they cannot abort benchmark aggregation. Preserve grading durations, character-count fallback and recorded zeros; add regression coverage for both layouts and subsequent runs.
* fix(skill-creator): warn and validate timing measurements
* fix(skill-creator): tolerate unreadable timing files
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(memory): parse a BOM-prefixed DeerMem fact instead of calling it corruption
Facts are hand-editable Markdown files, so a Windows editor can save one as
"UTF-8 with BOM" -- the same exposure #7083 fixed for memory.json. The fact
parser checks `text.startswith("---\n")` against a plain-utf-8 read, so the
BOM makes a valid fact read as "missing YAML front matter" and raises
MemoryStorageCorruption, which the module docs describe as taking down the
whole agent.
Read the fact with utf-8-sig, matching the summary loader and the SOUL.md /
SKILL.md convention already recorded in AGENTS.md. `_parse_listed_fact`
delegates here, so the directory-scan path is covered by the same change.
Measured: the new BOM test fails on the base revision with exactly that
"missing YAML front matter" corruption and passes at head; the no-BOM control
passes both. 49 tests in test_memory_storage.py, 106 across the memory
suites, ruff check/format clean. Mutation-checked: reverting to plain utf-8
fails only the BOM case.
* test(memory): cover BOM facts through the public storage API
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): run each IM message once across instances, restarts and late redeliveries
Inbound IM messages were deduplicated only by the 10-minute inbound dedupe
store (#7103). Discord direct messages had no key at all, a Discord thread
reply got an instance-dependent key, a Discord thread was created before
dedupe ran, a redelivery past the window started a second run, and two
instances racing on a conversation's first messages bound it to two
threads.
- Discord messages are keyed (discord, guild_id or "dm", posting channel,
message_id); the posting channel is the same on every instance.
- MessageBus.claim_inbound lets an adapter record the manager's dedupe key
before a side effect; Discord claims it on the Gateway loop before
_create_thread, marks the published message dedupe_claimed, and gives the
claim back when the message is never committed.
- runs.wait / runs.stream / runs.create send an Idempotency-Key derived from
the dedupe key, and keyed human message ids are uuid5 of it (a uuid4 per
attempt made every keyed retry a 409). A reused run's status-only /wait or
stream gap is resolved to this message's own turn via threads.get_state;
409s about the message's own keyed run keep the key instead of reading as
a busy thread; an ended reused GitHub run gets no follow-up watcher.
- A conversation's first thread binding is claimed insert-if-absent
(claim_thread_id on channel_conversations and channel_thread_bindings);
the loser adopts the winner and deletes the empty thread it minted.
/new and /agent use still rebind.
Docs: IM_CHANNEL_CONNECTIONS.md, CONFIGURATION.md, channels AGENTS.md,
CHANGELOG (en/zh).
* fix(channels): give back Discord dedupe claims for messages never processed
Review round 1 on #7104. A Discord claim records the manager's dedupe key
before the message is queued, so every path that drops the message unrun
must release it, or a peer refuses the redelivery for the rest of the TTL:
- The hand-off task releases the claim on cancellation too (stop() drains it
while it resolves the connection identity), via await_drained so the
cancellation still propagates.
- ChannelManager._discard_pending_inbound releases the key of a claimed
message that shutdown drops from the queue, after the grace period and on
a cancelled shutdown.
- The adapter tracks claims until they are committed. A release refused
because stop() already closed cross-loop intake, or a hand-off cancelled
before it ran, leaves the claim tracked, and stop() releases it last.
* fix(gateway): keep IM run idempotency stable when /new rebinds the conversation
Review round 2 on #7104. The Gateway scoped every Idempotency-Key by owner
and thread. /new rebinds an IM conversation to a fresh thread, so a late
redelivery of a message admitted on the old thread was requested on the new
one, got a different scoped key and started a second run.
- Keys from trusted internal callers (the IM channel manager) are scoped by
owner only (`internal-run:` prefix); browser and API keys keep the
owner-and-thread scope (`http-run:`).
- A key that resolves to a run of another thread or user raises
RunIdempotencyKeyMismatch, a RuntimeError subclass so in-process launchers
see the same type, and start_run answers it with 409 "Idempotency-Key
already used with a different request" instead of a 500. The channel
manager already treats that 409 as the message's admitted run.
Docs: API.md, IM_CHANNEL_CONNECTIONS.md, gateway AGENTS.md (reworded to stay
under its budget), CHANGELOG (en/zh).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(multi-instance): publish skill_manage cache resets and tolerate a concurrent namespace create
Two leftovers from the multi-instance work (#7097).
The agent's skill_manage tool refreshed only its own process's skills
prompt cache after a create, edit, patch, delete, write_file or
remove_file, so a Gateway replica sharing the home volume kept rendering
that user's previous skill list. Every mutation now runs one drained tail,
as the skill routes do: the write and its history record, the local
refresh, then publish_skills_cache_reset(user_id=...) off the event loop.
A run cancelled while the write is in flight still refreshes and
publishes before the cancellation propagates. A marker that cannot be
written is logged inside the tail and raised as SkillCacheResetPublishError
with a message saying the change was applied, so the model does not retry
a write that succeeded.
Two provisioner replicas starting together against a missing
K8S_NAMESPACE both read 404 and both create; the loser's 409 escaped the
lifespan and crashed it. A 409 on create now counts as the namespace
existing, as the Pod and Service creates already do.
Also drops a stale duplicate of the multi-instance "declaration also
drives" note in CONFIGURATION.md.
* docs(changelog): reference #7099 for the multi-instance leftovers
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(projects): check the project before staging shelf uploads
The shelf upload and save-from-thread routes staged the body under
users/{user_id}/projects/{project_id}/ before checking the project. A
missing, foreign, or archived id left an empty {project_id}/documents/
directory that the retention sweep keeps forever, and an id longer than
one path component failed with ENAMETOOLONG (500).
Both routes now require an owned, active project first and answer the
same 404 without writing. The shelf insert still re-checks under the
project row lock.
* test(projects): pin the shelf insert's re-check behind the early project check
With the early check in place, no route test reached the insert's
row-lock re-check any more. Archive the project between the check and
the insert and assert both routes still answer 404 and leave no staging
file.
* docs(changelog): reference #7040 for the shelf upload project check
* docs(changelog): drop the whitespace-only line after the zh #7040 entry
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The official DeepSeek Chat Completions endpoint rejects the generic
"minimal" effort token (issue #4514). The declared effort contract for
DeepSeek models now advertises the provider's real vocabulary
(low/medium/high/max/xhigh), defaults to "low", and aliases the generic
"minimal" preset onto "low" so the "flash" dialogue mode serializes a
value the endpoint accepts.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): exempt awk field arithmetic from the shell-variable path guard
Since #6930 every `$NAME/` in a host-bash command was treated as a
variable-prefixed host path unless both sides were digits, so legitimate
awk programs such as `awk '{print $NF/2}' data.txt` or
`awk '{print $sum/$n}' data.txt` on files under /mnt/user-data were
rejected with "Unsafe absolute paths".
Exempt a `$` only inside the single-quoted inline program of awk, gawk,
mawk or nawk (after `-F`, `-v` and `--`), where bash expands nothing and
awk reads `$NAME` as a field reference. awk string literals, option
values, input operands, double-quoted programs, text re-read by `sh -c`
or `eval`, heredocs, awk functions/aliases, unknown options such as
gawk's `--assign`, `-f` programs, and programs whose `"` may sit in a
regex or comment keep the strict scan.
Fixesbytedance/deer-flow#7021
* fix(sandbox): read awk string literals and comments before judging quotes
Review follow-up: the quote-reliability check gave up the awk exemption
for any `/` not preceded by an identifier, number or `]`, including a `/`
inside a plain string literal, so `awk '{n=split($0,p,"/"); print
$1/p[n]}'` was still rejected. Walk the program once with the same
string-literal tracker that computes the code spans: `/` and `#` inside
a literal are text, `#` outside one starts a comment to end of line, and
only a `/` outside a literal that cannot be division (regex literal)
forfeits the exemption.
* fix(sandbox): scan the awk exemption the way the shell parses the command
Review follow-up for the awk field-arithmetic exemption.
Bypasses closed: a backslash-newline let `aw\<nl>k() { eval "$1"; }`
define an awk function past the literal blocker regex, so any
continuation now forfeits the exemption; the word scanner emitted `)`,
`$(` and `>|` as command separators, so an `awk` that was only an
argument (`echo $(true) awk '...'`) looked like a command; `$'...'`,
comments and quotes nested in `"$(...)"` could desynchronise quote
pairing; the alias/function blocker only matched literal spellings and
ignored eval, source, hash, PATH, autoload and zsh's
`functions[awk]=` / `aliases[awk]=` arrays.
The scanner now follows bash: single and double quotes with escapes,
`#` comments at a word start, fd redirections, the usual separators,
and `$(...)` as a nested command context with fresh quoting. Anything
it does not model (backslash-newline, backticks, `$'...'`, heredocs, an
unmatched `)`, unbalanced quotes) returns no exemption. The command
word is found like `_validate_local_bash_shell_tokens` finds it:
after assignments, prefix keywords and wrappers, reduced to its
basename, inside every `$(...)`. A command that could redefine awk or
re-read quoted text gets no exemption at all, and the docstring says
the enumeration is best-effort.
False positives removed: `($2+$3)/2` and `length($0)/2` (a `/` after
`)` is division unless the `)` closes an if/while/for condition, and
the absolute-path scan is now exempt inside awk code too),
`/usr/bin/awk`, `LC_ALL=C awk`, `echo "$(awk ...)"`, `"<<"` inside an
awk string. The awk spans are computed lazily, only when a `$NAME/` or
absolute-path match needs them, and the division check scans backward
instead of re-slicing the program.
* fix(sandbox): exempt only single-segment absolute-path matches in awk code
Review follow-up. Inside awk code spans the absolute-path scan now skips
only a match with no second `/` (`/2`, `/n`, `/NR`), so a `/a/b` shape
is still rejected even if a span were ever misjudged; previously
`awk '{print ($1)/etc/passwd}'` and `awk 'BEGIN{x = 1 /etc/shadow}'`
were accepted.
`enable` joins the commands that withhold the exemption, since
`enable -f` can load a builtin under any name. The division check reads
the preceding `)` through the backward scan instead of re-slicing the
program for every `/`.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
