mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-09 12:30:37 +00:00
148 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Eilen Shin
|
2a8927c5e1
|
feat(models): add StepFun provider preset (#7988)
* feat(models): add StepFun provider preset * fix(models): standardize the StepFun preset label --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Wenchao An
|
c1f9064e9b
|
fix(models): route managed OpenAI GPT-6 presets through Responses (#7870)
* fix(models): route managed OpenAI GPT-6 presets through Responses * test(memory): pin capacity scenarios to a stable clock * fix(models): honor Flash mode for managed OpenAI GPT-6 * fix(models): reconcile preset effort with explicit reasoning contracts * test(memory): share the fixed clock with updater capacity tests * fix(models): normalize GPT-6 reasoning overrides and sampling --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Wenchao An
|
2145c9ab1f
|
feat(models): discover provider catalogs and search chat models (#7871)
* feat(models): discover provider catalogs and search chat models * docs(models): scope catalog guidance to owning directories * test(guidance): register model catalog scoped instructions * test(memory): pin capacity scenarios to a stable clock * fix(models): guard catalog imports and align picker interactions * test(memory): share the fixed clock with updater capacity tests * fix(models): clear provider filters removed by catalog refresh * refactor(models): share connection fields across model forms * fix(models): keep active provider filters clearable --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
8062d4ba12
|
fix(scripts): include nginx logs in dev_multi_instance.sh logs all (#7723)
* fix(scripts): include nginx logs in dev_multi_instance.sh logs all `logs all` selected only the two Gateway logs, the same as running `logs` with no argument, so nginx's error and access logs never showed up although the help lists all next to nginx. `all` now adds the nginx logs, and the nginx selection also covers nginx.log, where the start command's own output goes and which no option printed before. * docs(changelog): reference #7723 for the dev_multi_instance logs fix * test(scripts): pin dev_multi_instance logs when nginx never started Cover the layout start_nginx leaves behind when it bails on a missing binary or a busy port: only gateway-*.log exists, so `logs all` prints the two Gateway files and `logs nginx` prints nothing, both exiting 0. |
||
|
huidongchen148
|
612140f183
|
feat(models): add managed Kimi K3.1 Pro capabilities (#7566)
Declare required thinking, native reasoning effort and image support for both K3.1 Pro IDs. Normalize fixed parameters at the final Moonshot request boundary and cover real SDK serialization with offline regression tests. Fixes #7425 |
||
|
Totoro
|
c48447a4ae
|
fix(runtime): present outputs before a loop wrap-up and explain step-limit stops (#7588)
* fix(runtime): present outputs before a loop wrap-up and explain step-limit stops Loop-detection warnings now ask the agent to present the files it already created in /mnt/user-data/outputs before its final answer, when present_files is available. A run that exhausts recursion_limit now ends with stop_reason recursion_capped and a plain message that lists outputs it created but did not present. An active goal stands down without an evaluator call. The /wait error envelope keeps the GraphRecursionError name, and IM channels reply with the step-limit message instead of a generic failure. * docs(changelog): note run-end delivery and step-limit stops (#7588) * test(runtime): stop pinning LangGraph's step count in the step-limit test The produced paths and the message already follow the observed model calls, so the exact count only pinned LangGraph's per-step accounting. Also note on the step-limit goal branch that its caller passes no evaluator factory, so it has to stay ahead of the evaluator section. --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
creed
|
3c482e0f58
|
feat(jina): add bounded crawl completion diagnostics (#6504)
* feat(jina): add bounded crawl completion diagnostics Signed-off-by: 97three <2212371308@qq.com> * test(jina): cover diagnostics with request admission Signed-off-by: 97three <2212371308@qq.com> --------- Signed-off-by: 97three <2212371308@qq.com> |
||
|
spud
|
aabfdeebc9
|
fix(streaming): make Redis stream END publication idempotent (#7572)
* fix(streaming): make Redis stream END publication idempotent * fix(streaming): align Redis test fake with Lua END checks Remove the unreachable owned missing-stream branch, mirror tail-only data checks in both fake script paths, and pin legacy END and owned missing-buffer behavior against real Redis. * fix(streaming): gate legacy END scans after writer rollout * fix(helm): synchronize embedded config version with schema * test(runtime): use real stream bridge config in close coverage Keep new configuration defaults in lifecycle fixtures and surface owner-task failures instead of masking them as event timeouts. Drain every helper task and cover context entry/close failures. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Totoro
|
c36bb699de
|
fix(channels): honor the configured recursion limit in IM runs (#7521)
* fix(channels): honor the configured recursion limit in IM runs Every IM channel run asked the Gateway for recursion_limit 100, because the channel manager seeded each run config with it. The Gateway honors a request value, so the top-level recursion_limit never reached IM runs, and a budget of 100 could stop a run after a handful of tool-calling turns (six in one measurement with the default lead-agent middleware; the count changes with the middleware chain). IM runs now send no recursion_limit of their own and get the Gateway's configured default, clamped by max_recursion_limit, like Gateway API runs. A session config value still takes precedence, and GitHub channel runs keep their floor of 250. The README and config.example.yaml IM examples no longer pin session.config.recursion_limit to 100. * docs(changelog): reference #7521 for the IM recursion-limit fix --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
haochen
|
f7f9a7971b
|
fix(sandbox): normalize mount container_path trailing slashes at config load (#7393)
* fix(sandbox): normalize mount container_path trailing slashes at config load A container_path declared with a trailing slash (e.g. /mnt/reports/) was accepted by VolumeMountConfig and normalized by LocalSandboxProvider when building path mappings, but the sandbox tool path gates compare the raw configured value, so every ls/read/write/glob/grep/bash access to the mount failed with a confusing PermissionError. Normalize trailing slashes in the config model so both layers agree, and reject the container root '/' which the gates cannot express. * fix(sandbox): normalize skills.container_path and pin empty-value rejection Review follow-up on #7393: - Share the container-path normalizer with SkillsConfig, whose prefix feeds the same as-is comparison in _is_skills_path and had the same trailing-slash dead-path failure mode. - Pin the empty-string rejection: on main an empty container_path made the gates match every absolute path, applying one mount's read_only flag to all paths. - Cover both config fields in CONFIGURATION.md. --------- Co-authored-by: Zhangchenhao-111 <285911435+Zhangchenhao-111@users.noreply.github.com> |
||
|
Nan Gao
|
a9917e986b
|
docs(channels): align leader-lease failover and runtime-config notes with the merged code (#7420)
The two-Gateway harness checklist in CONFIGURATION.md said a crashed leader hands its providers over within lease_seconds + grace_seconds; a peer only tries to acquire on its fixed lease_seconds // 3 tick, so the worst case is one renewal interval more (50 s with the defaults), as the leader-lease section of the same file already says. The grace_seconds field description had the same shorter bound. The channels.leader_lease comment in config.example.yaml still said web UI credential changes reach the holder only when it next starts the channel; since #6988 the holder applies them within one renewal interval. The harness `check` footer still listed the IM channel gates as waiting for leader election, which landed in #6843; it now points at the manual channel item in CONFIGURATION.md, which also gains the runtime-config step. |
||
|
CodeAlex
|
4f9a76d5ef
|
fix(mindie): decode escaped newlines correctly inside tilde-fenced code blocks (#7285)
* fix(stream): tolerate ~~~-fenced code and \-prefixed closer when decoding escaped newlines _decode_escaped_newlines_outside_fences only recognised ``` fences, so replies that used ~~~ fences (e.g. code blocks rendered by the model with ~~~python ...) kept literal backslash-n sequences escaped instead of decoding the ones outside fences, and a fence whose closer was split as backslash+tilde across chunks was misread mid-stream. Fixes #7262. - detect the opening fence with (?:```|~~~) and remember which delimiter the fence was opened with, so closing requires the same delimiter - while inside a fence, hold back a closer that begins with a backslash so a fence ending in backslash+tilde mid-stream is not misread; the held tail is revisited on the next push and normalised on flush - keep backtick hold semantics unchanged * fix(mindie): preserve unclosed fences during newline decoding Match one-shot decoding to streaming by choosing the earliest backtick or tilde opener and preserving the remainder when its matching closer is absent. Correct the string-return docstring and cover mixed delimiters across every two-chunk split and character-by-character streaming. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Lachlan Allen
|
7a32a5e177
|
feat(exa): support a configurable Exa-compatible base URL (#7306)
Exa web_search and web_fetch accept a per-tool base_url, falling back to EXA_BASE_URL and then the SDK default. Overrides are validated like the Serper base URL. web_fetch screens a configured backend with the shared delegated-backend guard. Co-authored-by: lachy3433 <lachlanallen.co@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Kiko0301
|
1a5f8edc2f
|
docs(config): explain skills and sandbox paths across platforms (#7304)
* docs(config): explain skills and sandbox paths across platforms * docs: clarify YAML paths and trim startup guidance --------- Co-authored-by: Kiko0301 <191240073+Kiko0301@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
f39cdd6077
|
fix(mcp): keep rotating OAuth refresh tokens across restarts and Gateway instances (#7207)
* fix(mcp): keep rotating OAuth refresh tokens across restarts and Gateway instances
MCP servers on the OAuth refresh_token grant lost their rotated refresh
token whenever a new process or manager started (restart, tool-cache
reset for non-task servers, every personal-connection run, a second
Gateway instance) and re-seeded with the consumed configured token,
getting invalid_grant for good.
- New table mcp_oauth_tokens (migration 0047_mcp_oauth_tokens), one row
per refresh_token connection keyed (scope, owner_user_id, server_key):
('deployment', '', name) or ('personal', owner, personal runtime
name). Access and refresh tokens are DEER_FLOW_CREDENTIALS_KEY
ciphertext; a config fingerprint (token_url, client_id, grant, scope,
audience, seed) makes a changed seed win.
- OAuthTokenManager re-reads the row under its process lock, adopts a
peer's valid token, and otherwise claims the refresh with a lease plus
a version compare-and-set (database clock), fetches outside any
transaction, writes under the lease and releases it on failure.
- The Gateway installs the store (sync engine via to_thread) around the
LangGraph runtime for sqlite/postgres; personal managers pass their
owner. client_credentials stays in process; an unloadable key falls
back to process memory; database errors fail the refresh.
- credentials_key_consumers lists refresh_token MCP OAuth servers, so a
declared multi-instance deployment needs the shared key.
Docs: MCP_SERVER.md, CONFIGURATION.md, Helm README, README, mcp and
migrations AGENTS.md, CHANGELOG (en/zh).
* docs(changelog): link the MCP OAuth token store entry to #7207
* fix(mcp): prune removed connections' OAuth token rows and ask owners to re-authorize on invalid_grant
Review follow-up (willem-bd):
- Both MCP config writers drop mcp_oauth_tokens rows of connections a
save removes (deployment: _commit_mcp_config_write; personal:
_mutate, keeping each connection's enabled/disabled runtime-name
variants so a toggle keeps its chain). Best effort; no startup sweep,
since a missing config file would look like every server removed.
- invalid_grant on a refresh_token request raises
McpOAuthReauthorizationRequired: the tool error names the server and
asks for a new refresh token, plus the shared-key hint when this
instance could not decrypt the stored chain.
- Annotate the gateway helpers' config as AppConfig.
* fix(mcp): renew the OAuth refresh lease, keep unstored rotations, and key access tokens by request parameters
Review follow-up (willem-bd, second round):
- HTTPX timeouts bound each I/O wait, not a whole response: the refresh
holder now renews its lease every third of REFRESH_LEASE_SECONDS while
the token request runs and abandons it after REFRESH_MAX_SECONDS (120 s);
waiters outlast a live holder. Regression uses a real loopback endpoint
that streams its body for three lease lengths.
- A refresh whose tokens could not be stored keeps them pending
(unsaved_base_version): they are written back before an older stored
token is adopted or refreshed with, retried every few seconds while the
access token is still valid, and dropped once a peer stored a newer
chain. A failed save releases the lease.
- New token_request_fingerprint (extra_token_params and response fields):
a stored access token is reused only under the same request parameters;
otherwise the stored refresh-token chain is refreshed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
ningkexin96
|
4f948c7b8d
|
feat(deploy): select production Compose env files (#6522)
* feat(deploy): select production Compose env files Use an operator-selected dotenv file consistently for interpolation, container environment, secret probes and build extras. Preserve default selection and shutdown after a profile is removed. Co-Authored-By: Codex <noreply@openai.com> * fix(deploy): address production env selection review Clarify Docker launcher entry points and render real Compose through the launcher to verify both service environments. Co-Authored-By: Codex <noreply@openai.com> * test(deploy): cover unreadable env selectors Reject unreadable explicit selectors before Docker is invoked, using POSIX permission checks and restoring the fixture mode in finally. Co-Authored-By: Codex <noreply@openai.com> * test(deploy): scope dotenv assertions to selected secrets --------- Co-authored-by: Codex <noreply@openai.com> |
||
|
Nan Gao
|
dc6f0d7421
|
fix(memory): persist the DeerMem extraction watermark across restarts and replicas (#7205)
* fix(memory): persist the DeerMem extraction watermark across restarts and replicas
The updater's per-thread extraction watermark lived only in an in-process
LRU, so a restart or a second Gateway replica on the same storage root fed
already-extracted turns to the memory LLM again.
FileMemoryStorage now keeps it in users/{user}/agents/{agent}/.metadata/
watermarks.json, keyed by thread id, written atomically under the per-user
lock without touching memory.json and capped by watermark_max_keys. Each
update feeds after the furthest of the LRU entry and the persisted one, and
writes through after an extraction or a pre-screen skip; bypass_watermark is
unchanged and sidecar IO failures fall back to the LRU. Memory clears remove
the file with the other .metadata sidecars. Content-based message identities
are stored as SHA-256 digests so the sidecar holds no conversation text.
* docs(changelog): link the memory watermark entry to #7204 and #7205
* fix(memory): never replace an unreadable watermark sidecar
set_extraction_watermark rewrites the whole watermarks.json, and the shared
sidecar reader turns an OSError into an empty mapping, so a failed read could
replace every other thread's watermark with the current one (or delete the
file). The watermark sidecar now has its own reader: a missing file reads as
empty, malformed content reads as empty (nothing to keep; the next write
repairs it), and any other OSError propagates before anything is written. The
updater already logs that and keeps the extraction successful.
* fix(memory): keep the persisted watermark when the last message has no identity
_message_identity returns None for a last message with neither an id nor
text. The updater wrote that None through, which deleted the thread's entry
from watermarks.json, losing a watermark a peer may have advanced and forcing
a full feed on every replica. The write-through now skips None: the persisted
entry still marks an extracted prefix and the furthest-of-both read keeps
using it.
|
||
|
trent87
|
0d73798beb
|
feature(skills): keep allowed-tools as compatibility metadata (#6935)
* refactor(skills): keep allowed-tools as compatibility metadata * docs(skills): align current guides with agent tool authority * test(skills): simplify authority regressions and sync Korean docs |
||
|
Nan Gao
|
085924a3d0
|
fix(channels): run each IM message once across instances, restarts and late redeliveries (#7104)
* fix(channels): run each IM message once across instances, restarts and late redeliveries Inbound IM messages were deduplicated only by the 10-minute inbound dedupe store (#7103). Discord direct messages had no key at all, a Discord thread reply got an instance-dependent key, a Discord thread was created before dedupe ran, a redelivery past the window started a second run, and two instances racing on a conversation's first messages bound it to two threads. - Discord messages are keyed (discord, guild_id or "dm", posting channel, message_id); the posting channel is the same on every instance. - MessageBus.claim_inbound lets an adapter record the manager's dedupe key before a side effect; Discord claims it on the Gateway loop before _create_thread, marks the published message dedupe_claimed, and gives the claim back when the message is never committed. - runs.wait / runs.stream / runs.create send an Idempotency-Key derived from the dedupe key, and keyed human message ids are uuid5 of it (a uuid4 per attempt made every keyed retry a 409). A reused run's status-only /wait or stream gap is resolved to this message's own turn via threads.get_state; 409s about the message's own keyed run keep the key instead of reading as a busy thread; an ended reused GitHub run gets no follow-up watcher. - A conversation's first thread binding is claimed insert-if-absent (claim_thread_id on channel_conversations and channel_thread_bindings); the loser adopts the winner and deletes the empty thread it minted. /new and /agent use still rebind. Docs: IM_CHANNEL_CONNECTIONS.md, CONFIGURATION.md, channels AGENTS.md, CHANGELOG (en/zh). * fix(channels): give back Discord dedupe claims for messages never processed Review round 1 on #7104. A Discord claim records the manager's dedupe key before the message is queued, so every path that drops the message unrun must release it, or a peer refuses the redelivery for the rest of the TTL: - The hand-off task releases the claim on cancellation too (stop() drains it while it resolves the connection identity), via await_drained so the cancellation still propagates. - ChannelManager._discard_pending_inbound releases the key of a claimed message that shutdown drops from the queue, after the grace period and on a cancelled shutdown. - The adapter tracks claims until they are committed. A release refused because stop() already closed cross-loop intake, or a hand-off cancelled before it ran, leaves the claim tracked, and stop() releases it last. * fix(gateway): keep IM run idempotency stable when /new rebinds the conversation Review round 2 on #7104. The Gateway scoped every Idempotency-Key by owner and thread. /new rebinds an IM conversation to a fresh thread, so a late redelivery of a message admitted on the old thread was requested on the new one, got a different scoped key and started a second run. - Keys from trusted internal callers (the IM channel manager) are scoped by owner only (`internal-run:` prefix); browser and API keys keep the owner-and-thread scope (`http-run:`). - A key that resolves to a run of another thread or user raises RunIdempotencyKeyMismatch, a RuntimeError subclass so in-process launchers see the same type, and start_run answers it with 409 "Idempotency-Key already used with a different request" instead of a 500. The channel manager already treats that 409 as the message's admitted run. Docs: API.md, IM_CHANNEL_CONNECTIONS.md, gateway AGENTS.md (reworded to stay under its budget), CHANGELOG (en/zh). --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
0550816c50
|
fix(multi-instance): publish skill_manage cache resets and tolerate a concurrent namespace create (#7099)
* fix(multi-instance): publish skill_manage cache resets and tolerate a concurrent namespace create Two leftovers from the multi-instance work (#7097). The agent's skill_manage tool refreshed only its own process's skills prompt cache after a create, edit, patch, delete, write_file or remove_file, so a Gateway replica sharing the home volume kept rendering that user's previous skill list. Every mutation now runs one drained tail, as the skill routes do: the write and its history record, the local refresh, then publish_skills_cache_reset(user_id=...) off the event loop. A run cancelled while the write is in flight still refreshes and publishes before the cancellation propagates. A marker that cannot be written is logged inside the tail and raised as SkillCacheResetPublishError with a message saying the change was applied, so the model does not retry a write that succeeded. Two provisioner replicas starting together against a missing K8S_NAMESPACE both read 404 and both create; the loser's 409 escaped the lifespan and crashed it. A 409 on create now counts as the namespace existing, as the Pod and Service creates already do. Also drops a stale duplicate of the multi-instance "declaration also drives" note in CONFIGURATION.md. * docs(changelog): reference #7099 for the multi-instance leftovers --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
RongJie G
|
bf2cb0e1df
|
fix(runtime): recover terminal Redis streams after producer loss (#7093)
Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
8d5f5fd0a8
|
feat(channels): share encrypted runtime channel config across Gateway instances (#6988)
* feat(channels): share encrypted runtime channel config across Gateway instances IM bot credentials entered in the web UI lived in each host's plaintext channels/runtime-config.json, and the router cached the merged config on app.state for the life of the process. With several Gateway instances a save or disconnect served by an instance that does not hold the provider's leader lease never reached the transport holder: it kept the old credentials until it next started the channel, and a disconnect could leave the old bot connected. - New shared table channel_runtime_configs (migration 0044): one row per provider with the config block encrypted under DEER_FLOW_CREDENTIALS_KEY, a plaintext disconnect tombstone, and a version bumped by every save. - SqlChannelRuntimeConfigStore replaces the JSON file for sqlite/postgres (memory keeps the file); it is synchronous on app_sync_sqlalchemy_url because its callers already run in worker threads. - The router re-reads the store on every request instead of caching the merged config, and refuses (503) to save credentials without a key. - The leader-lease supervisor compares the versions on each renewal tick and restarts or stops the holder's transport when a provider's config changed; a takeover reads a disconnect instead of reviving cached credentials. - The first start imports an existing runtime-config.json (rows already in the table win) and deletes the plaintext file. * docs(changelog): reference #6988 for the shared runtime channel config * ci: rerun checks after #6999 fixed the frontend guidance budget on main * fix(channels): start a held transport that is down when its runtime config changes When the lease holder's transport was down (a start that failed on the old credentials), applying a peer-saved config only updated the cached config, and the transport started on the next lease round: two renewal ticks instead of the one the docs promise. The apply step now starts it on the held lease right away, with the lease round's backoff bookkeeping. |
||
|
Totoro
|
16d36ccd21
|
feat(sandbox): add bounded AIO prewarming and lifecycle benchmarks (#6998)
* fix(benchmark): validate sandbox lifecycle and concurrent measurements * feat(sandbox): prewarm likely AIO tool turns (cherry picked from commit 3a1017ed032ed9331987bc3dc61f23dcdac4e905) * fix(sandbox): bind bounded prewarming to execution leases * docs: trim frontend guidance below its existing size limit * fix(benchmark): separate failed lifecycle stage timings * test(sandbox): measure prewarm overlap and unused resources through agent graphs * fix(sandbox): finish lease and sampler cleanup after cancellation or errors * docs(benchmark): distinguish initialization wait from model overlap * docs(helm): align the configuration example with version 61 * fix(sandbox): preserve synchronous agent graphs when prewarming is enabled * test(cargo): isolate native fixtures from inherited make jobserver flags * fix(sandbox): require managed cleanup before prewarming * fix(sandbox): log denied prewarming as an expected skip * fix(benchmark): filter provider summary input records --------- Co-authored-by: nothingyuancando <1579035512@qq.com> |
||
|
Totoro
|
c994826d06
|
feat(channels): generalize group notifications and add opt-in task/goal IM notices (#6903)
* feat(channels): generalize IM group targets and add task/goal run notifications
- Generalize proactive group notification capability (proactive_group_notifications), binding validation, and scheduled task group selection across Feishu, QQ, and WeCom while keeping WeChat personal-only.
- Add QQ group active notification delivery via POST /v2/groups/{group_openid}/messages (msg_type=2) and WeCom group active notification delivery via WS Client.sendMessage(chatid, body) with bot_id:group:chatid workspace isolation.
- Generalize notification outbox sources via migration 0043_runtime_im_notices (source_kind, source_key, run_id, nullable task_id/task_run_id) with backward-compatible SQLite/PostgreSQL upgrades.
- Add opt-in ordinary task & goal run IM notifications (run_completed, run_failed, human_input_requested) on the shared outbox, delivery worker, renderer, and settings UI.
* fix(notifications): address review feedback and group/runtime notification edge cases
* test(migrations): update CURRENT_HEAD to 0043_runtime_im_notices across migration chain tests
* fix(channels): preserve group binding names and legacy authorization
Carry owner-chosen group names through existing connect state, keep private bindings out of legacy group fallback, and restore completion metadata retries. Cover real binding and worker finalization paths and synchronize notification docs.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
XY
|
2d6b406801
|
fix(runtime): preserve run event timestamp offsets in SQLite (#6894)
* fix(runtime): preserve run event timestamp offsets in SQLite * docs: clarify run event timestamp upgrade behavior --------- Co-authored-by: XY-code1 <231763295+XY-code1@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
33b8e84f11
|
fix(channels): keep IM runs alive when the creating request disconnects (#6876)
* fix(channels): keep IM runs alive when the creating request disconnects The channel manager holds a `runs.wait` / `runs.stream` request open for the whole run, and that request dies with the Gateway instance that received the IM message: a leader hand-off, a rolling restart, a crash. The Gateway cancels a run whose creating request disconnects unless the request asked for `on_disconnect: continue`, so every IM run in flight was cancelled on failover. Channel runs now ask for `continue`, like scheduled-task and MCP-task runs already do. The run finishes on the Gateway executing it and its thread state is persisted, so the answer stays readable in the web UI even though the IM reply is lost with the old transport. The key is set after the fire-and-forget branch because the SDK's `runs.create` has no `on_disconnect` parameter. Docs: CONFIGURATION.md and the Helm/deployment guides no longer list "failover cancels IM runs" as a known limit; IM_CHANNEL_CONNECTIONS.md and the channels AGENTS.md describe the policy. * docs(changelog): link #6876 * fix(channels): keep the inbound dedupe key once a run may be executing With `on_disconnect: continue` a run outlives the manager worker that requested it, but the worker loop released the inbound dedupe key on cancellation, so a provider redelivery on the replacement instance admitted the same event again and executed the turn twice (review on #6876). Each inbound message now carries an admission record: `requested` flips before the run request leaves the manager, `confirmed` once the Gateway proved it admitted the run (a run id, a stream frame, a completed response). The generic release paths (worker cancellation, worker/handler exceptions, a broken stream) keep the key when the run may continue: after confirmation, on cancellation after the request was sent, and on post-request transport failures (read timeout, read error, protocol error). Pre-request failures, Gateway rejections (busy thread, stale pin, status errors, connection failures) and the stream's terminal `error` frame still release it so the redelivery can retry. |
||
|
Nan Gao
|
e8c7df9042
|
feat(channels): arbitrate IM transports with a per-provider leader lease (#6843)
* feat(channels): arbitrate IM transports with a per-provider leader lease Every Gateway instance connected every enabled IM platform with the same bot credentials, so channels could not run on more than one instance (Telegram 409 storms, Discord answering twice, scattered per-process state). channels.leader_lease was reserved by #6746; this implements it. - channel_leases table (migration 0041): one row per provider. Acquire is one dialect-native upsert decided against the database clock and bumps a fencing epoch; renew extends only a live lease at the holder's epoch; release sets the expiry to 0 so a peer takes over on its next tick. - app/channels/leadership.py: the holder's local monotonic deadline (send time of the last confirmed renewal + lease_seconds); holds() turns false one renewal interval early, so the transport stops at least grace_seconds before a peer may connect. DB calls are bounded. - ChannelService: start fence in _start_channel (no DB call) behind every start path; fixed-rate supervisor renews/acquires and runs transport starts/stops as per-provider tasks; takeover reloads the provider's saved config; failed starts back off; shutdown releases only leases whose transport stopped. get_status reports leader/leader_here and the holder's transport state on every instance. - Notification worker leaves rows of providers another instance holds. - Gate: the enabled=false refusal now recommends auto/true; an explicit true with database.backend memory is refused when a channel is enabled. - Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated. * docs(changelog): link the channel leader lease entry to #6843 * fix(channels): see peer-held leases for providers absent from local config Review on #6843: a provider connected from the web UI on a peer never enters this instance's channel config, so the notification worker here still claimed its rows (and parked them for 900s) and GET /api/channels/ reported no leader for it. Every supervision round, and startup, now also reads the whole lease table once (bounded), and notification exclusions and status use those observations: a provider whose live lease a peer holds is left to it even without local config. Running that read concurrently with renewals exposed that, through SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on another connection can hand back one of its own bind values instead of the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do not show it). A lease must never take such a row for a confirmed renewal, so acquire/renew/release now decide by affected-row count and acquire reads its epoch back in the same transaction. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Eilen Shin
|
e98179a9d3
|
feat(models): register managed models with provider presets (#6757)
* feat(models): register managed models with provider presets * docs: add PR 6757 UI comparison screenshots * test: fix model management CI checks * docs: remove PR 6757 screenshots --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Totoro
|
5ae95e6a34
|
feat: add Feishu group notifications and fix mention caret placement (#6762)
* feat(feishu): deliver scheduled-task notifications to bound users * feat: add Feishu group notifications and fix mention caret placement * fix(frontend): preserve group targets while capabilities load --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> |
||
|
Nan Gao
|
d18d2937e7
|
feat(channels): add channels.leader_lease and gate an explicitly disabled lease under multi-process signals (#6746)
* feat(channels): add channels.leader_lease and gate it under multi-process signals IM channels other than GitHub connect outbound to their platform. With more than one Gateway instance each one opens that connection with the same bot credentials: Telegram rejects the second poller with 409, Discord answers every message twice and per-process ChannelManager state diverges. The multi-process startup gate said nothing about channels; the only guard was a comment in the Helm values. Add the `channels.leader_lease` section (`enabled: auto|true|false`, `lease_seconds`, `grace_seconds`; startup-only, registered in the reload boundary) that the per-provider leader lease will read, validated by a Pydantic model from the non-schema `channels` mapping. The gate now refuses `enabled: false` under a multi-process signal while a non-GitHub provider is enabled under `channels.*` or connectable through `channel_connections`, naming the providers and the rollback; a malformed block is refused on every topology. `auto` follows the multi-process signal, so a single instance is unchanged. ChannelService strips the block before its per-provider loops and exposes it as `leader_lease`; `ensure_ready_channels` only visits registered providers. The two-Gateway harness overlay leaves the block alone. config_version 60 in the example, the Helm chart and its README; docs replace the comment-only "keep 1 replica" guidance with the gate. * docs(changelog): link #6746 * fix(channels): stop recommending the unimplemented leader lease as a remedy Review on #6746: the refusal told operators to switch channels.leader_lease from false to auto/true "so one instance owns each provider's connection", but the transport arbitration ships separately, so that change only bypassed the clause and every worker still started the same bot. The refusal now offers only the remedies that work today (a single instance, or disabling the channels) and states that auto/true are reserved for the lease that ships separately. A regression helper in the gate tests asserts the remedies start with the rollback and that auto/true are never offered. The config example, Helm values/README, CONFIGURATION.md, README, the deployment guide (en/zh), the model docstrings and both changelogs describe auto/true as reserved settings rather than an ownership guarantee. * fix(channels): warn on an enabled unknown channel key; reword the changelog entry Review on #6746: the registry guard in ensure_ready_channels skipped a misspelled provider key (channels.telegam) silently, where it previously reached _start_channel and logged 'Unknown channel type'. An enabled dict block that is neither a registered provider nor leader_lease now logs a warning that lists the registry. The changelog entry claimed the chart and docs drop the "keep 1 replica" guidance; they keep it (auto/true do not make several replicas safe yet) and only stop relying on the comment alone. * fix(dev): disable UI channel providers in multi-instance overlay --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Totoro
|
4b36e680c3
|
feat(feishu): deliver scheduled-task notifications to bound users (#6753)
Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com> |
||
|
AochenShen99
|
5b2a1352cd
|
fix(agents): allow SQL updates with retained legacy files (#6747) | ||
|
Value Add
|
4beb6c4ea4
|
feat(community): add You.com web search provider (#6515)
* feat(community): add You.com web search provider Adds You.com as an opt-in community `web_search` engine, mirroring the existing community engines (tavily, brave, exa, firecrawl, groundroute). You.com runs its own web index and returns extracted page snippets per hit rather than a single meta description, so a research loop gets usable context without a follow-up fetch. The provider picks its endpoint by whether a key is configured, so it works out of the box and scales up when the operator adds one: - no key -> /v1/agents/search, keyless free tier - YDC_API_KEY -> /v1/search with X-API-Key, full Search API Endpoint and headers are always chosen together; the keyless endpoint rejects an auth header, so a stale key can never be sent there. Both return the same body shape, so one normalizer covers both. httpx only, no new dependency and no pyproject extra, so uv.lock is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(community/youcom): review follow-ups — drop stray .pr-body.md, guard OverflowError, add 429 hint * fix(doctor): make the You.com free-provider detail true with or without YDC_API_KEY The doctor detail returned before any key check, so operators who had set YDC_API_KEY still saw 'set YDC_API_KEY for the full Search API'. Reworded to match the sibling free-provider entries and stay accurate in both states. * fix(search): sweep youcom max_results onto the shared owner The single-owner gate that landed with #6563 flags this branch's private _coerce_max_results as a newly hand-rolled copy, so backend-unit-tests now fails after merging main. Follow the coordination note from review: delegate to deerflow.community.search_max_results.coerce_max_results, which owns the rejection profile (booleans, non-integral floats, OverflowError), and keep only the 1-100 mirror of You.com's server-side count clamp on top — clamping policy stays per-provider. The .inf regression test now goes through the public tool path and also pins the bool and non-integral-float shapes of the shared bar. * docs(tools): use the bare you.com platform link in config and docs Match every other provider entry: no other provider in config.example.yaml or the en/zh tools.mdx uses campaign-tagged URLs, so drop the utm params from the three you.com/platform links. Integration attribution stays in the identifying User-Agent sent to You.com hosts. * test(youcom): cover shared max_results ownership and bounds Register You.com among the shared-owner providers and assert default handling and the 100-result cap through outbound request parameters. --------- Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: valueadd-lgtm <339269770+valueadd-lgtm@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
creed
|
3461c12896
|
feat(jina): add process-local request admission (#6489)
* feat(jina): add process-local request admission Signed-off-by: 97three <2212371308@qq.com> * test(jina): cover closed admission owner loop Signed-off-by: 97three <2212371308@qq.com> * test(jina): retain admission across API redirects Signed-off-by: 97three <2212371308@qq.com> * fix(jina): drain redirect response cleanup before releasing admission Signed-off-by: 97three <2212371308@qq.com> --------- Signed-off-by: 97three <2212371308@qq.com> |
||
|
Nan Gao
|
35f6a98cd9
|
feat(dev): add a two-Gateway multi-instance harness script (#6613)
* feat(dev): add a two-Gateway multi-instance harness script There was no way to run the deployment.multi_instance topology locally: scripts/serve.sh pins one Gateway to port 8001, the local nginx config has a single upstream, and getting two processes past the multi-instance startup gate needs Postgres, Redis, a shared DEER_FLOW_HOME, shared auth secrets and a per-process DeerMem retrieval index by hand. scripts/dev_multi_instance.sh (make dev-multi / dev-multi-check / dev-multi-down) starts throwaway Postgres and Redis containers on loopback, renders a config from the developer's config.yaml (or config.example.yaml), and runs Gateway A on 8001 and Gateway B on 8011 with one shared home, one copy of extensions_config.json and generated AUTH_JWT_SECRET, DEER_FLOW_INTERNAL_AUTH_TOKEN and DEER_FLOW_CREDENTIALS_KEY values, plus an optional round-robin nginx on 2027. B starts after A answers /health: Alembic migrations are advisory-locked, but the LangGraph checkpointer/store setup() that follows is not. scripts/dev_multi_instance.py holds the testable parts: the config overlay (neutralizing and reporting settings the gate refuses), the secrets file, the nginx config, and `check`, which automates the scriptable cross-instance acceptance checks: readiness with stream_bridge ok, A-issued sessions and the internal token accepted by B, thread/upload/artifact visibility, a skill toggle seen by B, nginx balancing, and SSE join on B with Last-Event-ID resume on A. Tests pin that the rendered config passes the real _enforce_postgres_for_multi_worker gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): link the two-Gateway harness entries to #6613 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dev): keep the multi-instance harness inside its own state Address review findings on the two-Gateway harness (#6613): - Containers carry a deerflow.harness.state-dir label; up/down remove only containers of their own state dir and refuse those of another checkout or DEERFLOW_MI_STATE_DIR instead of force-removing them. - Containers are removed with `docker rm -f -v`, so the anonymous volumes the postgres and redis images declare no longer leak on every up/down. - `up` installs an EXIT trap once it starts containers, so any later failure (including B's port being taken after preflight) tears down A and both containers; start_gateway returns failures instead of exiting the script, and the Gateway is disowned so stopping it prints no job notice. - The overlay removes any checkpointer section (a postgres one would write checkpoints and the Store into the developer's database), points an explicit sandbox.ownership and a Redis checkpoint_cache at the harness Redis (keeping ownership timing), drops DeerMem storage_path (both spellings) and the blob_storage root so data lands in the harness home, gives a local AIO sandbox a state-dir-specific container_prefix, and reports a non-DeerMem memory backend it keeps sharing. - nginx paths are rendered as quoted string literals ('$' and control characters are refused), so a state dir with spaces works. - `check` skips the run-based SSE check when model discovery did not succeed, unless --with-llm, and enforces the SSE time budget per raw line so heartbeat-only traffic cannot hold it open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(scripts): keep the harness guidance under the AGENTS.md soft limit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nan Gao
|
db7340d2fa
|
fix(security): add DEER_FLOW_CREDENTIALS_KEY and wire the channel credentials cipher (#6611)
* fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption
There was no at-rest encryption key. ChannelCredentialCipher had no
production caller: the channel service, the scheduled-task notification
outbox and the channel-connections router all built
ChannelConnectionRepository without a cipher, so store_credentials refused
(RuntimeError) and get_credentials always returned None.
- New harness module deerflow.config.credentials_key: env-only
DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts,
all decrypt via MultiFernet), CredentialsCipher with encrypt_text /
decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1:
values still readable, and get_credentials_cipher() as the process-wide
entry point. Malformed keys are refused without echoing key material.
Without the variable a single instance generates
{base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET.
- New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL
create (0600) plus read-back with a bounded settle window, shared by
.credentials_key, .jwt_secret (fixes two replicas cold-starting on a
shared volume keeping different JWT secrets) and the managed-model key.
- ChannelCredentialCipher now builds on CredentialsCipher (same
constructors); the Gateway loads the cipher off the event loop at
startup when channel_connections is enabled and passes it to every
ChannelConnectionRepository. Undecryptable values stay "missing + warn".
- Startup gate: a malformed key stops startup in any topology; a declared
multi-instance deployment with channel_connections enabled and no key
is refused. Plain multi-worker shares the generated file and is allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh
- Helm: the app Secret generates a Fernet key (randBytes 32 translated to
the urlsafe alphabet) and preserves it across upgrades via lookup like
AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef
so a user-managed existingAppSecret without it still boots (the Gateway
gate refuses only multi-instance deployments that store credentials).
README, NOTES and values document backup, rotation and existingAppSecret.
- scripts/deploy.sh resolves the key shell -> .env -> persisted file ->
generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600,
noclobber/O_EXCL with read-back): the same file the Gateway generates in
its runtime home, so either side's key is the one both keep.
- Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway.
- .env.example entry with a generation hint; README and CHANGELOG notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): make abandoned secret-file replacement single-winner
A rename is last-writer-wins, so replicas that restart together and all
find the same abandoned empty secret file could each replace it and keep
a different value. The process that exclusively creates
<name>.replacing now owns the replacement, re-checks the file is still
empty, and publishes by renaming the claim over it; peers wait for that
value, bounded so a crashed replacer cannot block startup forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): name the setting that turns off a credentials-key consumer
The multi-instance refusal ended with "or disable
channel_connections.enabled=true"; it now says "or set
channel_connections.enabled=false".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): never expose a secret file before its value is written
O_EXCL published the name before its content, so a creator that stalled
mid-write left an empty file a peer replaced after the settle window; the
resumed creator then returned a value nobody persisted, or deleted the
peer's valid file when its own write failed. Creators now write and sync
a private temporary file and hard-link it to the name, which fails if
the name exists; a failed creator only removes its own temporary file.
Filesystems without hard links publish through the same single-winner
<name>.replacing claim that replaces abandoned empty files, so creation
and replacement are serialized there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): recover an empty managed-model key when no catalog exists
An interrupted first save could leave an empty key file; every later save
then passed empty bytes to Fernet and the endpoint returned 503 until the
file was removed by hand. An empty key with no catalog encrypted nothing,
so it now goes through the secret-file recovery path; next to an existing
catalog it is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start
scripts/docker.sh runs Compose from docker/ without --env-file, so the dev
gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry
interpolated to empty and overrode the key the checkout .env supplied via
env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
yangfucheng123
|
3b8141aa43
|
fix(community): Fail closed on delegated fetch backends to close SSRF gap (#6531)
* Fail closed on delegated fetch backends to close SSRF gap * fix(community): resolve delegated-fetch credentials from one config snapshot Address review feedback on the fail-closed SSRF guard: - firecrawl/fastcrw client construction no longer re-reads get_app_config(), so the API key and endpoint come from the same revision the endpoint screen validated (prevents cross-revision credential misrouting). - fix the browserless test double to accept the new two-argument _get_browserless_client signature. - document the check-only (DNS rebinding) caveat on validate_delegated_backend_url. - add a CHANGELOG upgrade note for the default self-hosted backend break. * fix(doctor): consolidate delegated fetch backend checks and deployment guidance --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
6c4c74a504
|
fix(channels): share chat-to-thread bindings through the database (#6558)
* fix(channels): share chat-to-thread bindings through the database ChannelManager kept its IM chat-to-thread bindings (channel_name:chat_id[:topic_id] -> thread_id) in a per-process JSON file, channels/store.json, loaded once and rewritten whole on every change. With several Gateway processes a binding created on one replica was invisible to the others (the next message for the same chat opened a second thread) and concurrent writers clobbered each other's file. - New channel_thread_bindings table (migration 0036_channel_thread_bindings) and harness package deerflow.persistence.channel_thread_bindings: key PK (the legacy composite string), channel_name, chat_id, nullable topic_id, thread_id, user_id, epoch created_at/updated_at; index (channel_name, chat_id). Dialect upsert keeps created_at; prefix remove and batched ON CONFLICT DO NOTHING import. - app.channels.store.ChannelStore is an async protocol: SqlChannelStore for sqlite/postgres, JsonChannelStore (file I/O via asyncio.to_thread) for database.backend: memory. resolve_channel_store picks at ChannelService construction; ChannelService.start() imports an existing store.json once into an empty table and renames it store.json.migrated (a concurrently importing peer neither duplicates nor clobbers; a populated table leaves the file alone). - Callers updated: manager, connection_identity.lookup_thread_id, Feishu (send() awaits the mapping writer; the synchronous lark callback bridges its two store calls to the Gateway loop with a bounded wait), Discord (channels_dir). - Head pins in the 0027_0028/0029/0032/0034/0035 migration tests point at the new head; docs (channels and migrations AGENTS.md, CONFIGURATION.md, README) and CHANGELOG updated. * docs(changelog): link the channel thread bindings entry to #6558 * fix(channels): reject only the Gateway loop in the Feishu store bridge lark-oapi 1.5.5 invokes the message callback synchronously from its async frame handler, on the private event loop that FeishuChannel._run_ws starts for the SDK thread. The bridge's guard treated any running loop as unsafe, so every real inbound message skipped the stored-topic lookup and the pending-mapping restore: a reply whose parent card was mapped but whose root was not fell back to the unknown root, and could consume an unrelated pending clarification. Only a wait issued from the Gateway loop itself (the loop that executes the submitted coroutine) is refused now; the SDK's own loop may block for the bounded wait, as that thread blocked on the JSON rewrite before. Tests drive _on_message from a worker thread running its own loop, the production shape, for both the lookup and the write path, and pin the Gateway-loop refusal. * fix(channels): defer unresolved Feishu lookups and harden the store.json import Review follow-ups on the chat-to-thread binding store (#6558): - Import: a legacy entry whose key, channel_name, chat_id, topic_id, thread_id or user_id exceeds its column width is skipped and counted with the malformed ones instead of failing the whole batch on PostgreSQL (SQLite silently accepted the same row, so the backends diverged on one file). The corrupt-file warning now says what each caller does: the JSON store starts fresh, the import is skipped and the file kept. - Feishu bridge: store calls run on the Gateway loop under a 3s asyncio.wait_for (the SDK loop also services ws keepalives; the old 10s bound could stall it per message), and the lark-thread wait is that budget plus 1s. A deadline or store error is StoreLookupUnresolved, never a miss: _on_message no longer routes such a reply onto its unknown root (which created a second thread and later overwrote the parent alias). The message joins a per-chat FIFO drained on the Gateway loop with backoff for up to 30s, then is dropped with a warning. _on_message is split into parse / resolve / dispatch so the deferred path dispatches the same way; later messages for the chat queue behind the deferred one. * fix(tests): expect follow_redirects in the Jina client kwargs after #5808 #5808 made JinaClient pass follow_redirects=True to httpx.AsyncClient and updated test_jina_client.py, but test_jina_response_limit.py and test_jina_retry_after.py still pin the exact client kwargs to proxy and trust_env, so every backend-unit-tests shard that lands on them fails on main (the backend unit-test workflow only runs on pull requests, which is why the push to main did not surface it). Update the two expectations to include the new keyword. (cherry picked from commit d5aa23156ce19ce36d95df03953442f83ae6d050) * Revert "fix(tests): expect follow_redirects in the Jina client kwargs after #5808" This reverts commit 5bee00bfb09c65a30f77fb4df47a1f82c47ecef7. * fix(channels): bound deferred Feishu routing by the inbound queue capacity Review follow-up on #6558: a store outage let every subsequent message for a chat join the deferred queue with no capacity check (the bus reservation was only taken after routing succeeded), and each queued message started a fresh 30s retry budget when it reached the head, so a burst could be retained for N x 30s and grow without bound. - Intake is reserved BEFORE routing for every store-backed message (bus.reserve_inbound is thread-safe and already used from SDK threads), so deferred and live messages share the one configured bound, channels.inbound_queue_maxsize; a message that cannot be admitted is dropped at once with a warning, never queued or routed as a miss. - The reservation travels with the message into dispatch (committed by _prepare_inbound), is released on the batch path (the flush reserves at publish time), on drop after the budget, on a failed deferral, on dispatch failure and for every leftover when the drain task is cancelled by stop(). - The retry budget runs from the message's arrival: a queued message whose budget is spent when it reaches the head is dropped without a lookup. - The bounded resolver returns the unresolved outcome instead of raising it through the tracked task, which logged an ERROR for an expected, handled store timeout. - Upstream's new blocking-io test constructs the store by its old class name; it now uses JsonChannelStore. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
creed
|
be62fff070
|
feat(serper): add opt-in bounded transient retries (#6456)
* feat(serper): add opt-in bounded transient retries Signed-off-by: 97three <2212371308@qq.com> * fix(serper): validate HTTP retry dates precisely Signed-off-by: 97three <2212371308@qq.com> * fix(serper): address retry review feedback --------- Signed-off-by: 97three <2212371308@qq.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Daoyuan Li
|
215758818e
|
fix(models): keep Claude retries behind request admission (#6545)
* fix(models): keep Claude retries behind request admission * fix(models): keep Codex retries behind admission and log overrides * fix(models): retry paced Codex HTTP 529 errors through middleware * test(jina): allow independent HTTPX options in compatibility checks --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
b99ac916b6
|
fix(auth): count login lockouts in the database across Gateway replicas (#6501)
* fix(auth): count login lockouts in the database across Gateway replicas POST /api/v1/auth/login/local throttled failed logins per client IP in a module-level dict, so with N Gateway replicas behind one load balancer an attacker got N x auth.local.max_login_attempts guesses and a lockout on one replica was invisible to the others. Move the counter behind a LoginThrottleStore (deerflow.persistence. login_throttle): MemoryLoginThrottleStore keeps the historical per-process semantics and 10k tracked-IP bound; SqlLoginThrottleStore shares one login_throttle row per IP through the application database, incrementing and starting the lock in a single dialect-native upsert, applying check decisions with compare-and-set predicates in a separate write transaction, and sweeping served locks plus 24h-idle counters in bounded batches. Both stores pin the existing contract: the lock starts when the count reaches max_login_attempts, the committed duration is honored when the policy shrinks mid-lock, a served sentence is never resurrected, tightening keeps the count, success resets the IP, and the 429 status and message are unchanged. New startup-only selector auth.local.throttle_storage (auto | memory | db, default auto = the database whenever database.backend is sqlite or postgres). The Gateway installs the store once at startup after the persistence engine; deps._validate_login_throttle_storage warns when a multi-process deployment keeps memory. Migration 0034_login_throttle creates the table; config_version 55 -> 56 in config.example.yaml and the Helm chart. Docs: README, CONFIGURATION.md, AUTH_DESIGN.md, config and migrations AGENTS.md, CHANGELOG. * docs(changelog): link the login throttle entry to #6501 * fix(auth): re-check expiry on the row the login-throttle sweep deletes The bounded cleanup in SqlLoginThrottleStore.record_failure deleted `WHERE ip IN (SELECT ip ... WHERE <served or stale> LIMIT 200)`. On PostgreSQL READ COMMITTED the expiry predicate applied only inside the snapshot-bound subquery: when the DELETE had to wait on a row lock held by a concurrent check() that was extending the same lock under a raised lockout_seconds, PostgreSQL re-evaluated only the outer WHERE on the updated tuple, IP membership still held for the old candidate, and the now-active lock was deleted; the next attempt from that IP got a clean record and reached password verification (review of #6501, P2). Repeat the served/stale predicate on the DELETE target as well (SqlLoginThrottleStore.sweep_statement), keeping the bounded candidate subquery, so the re-evaluation sees the updated row's expiry. The other writes already carry their full compare-and-set predicate on the target row: _discard and the check() UPDATE both use _matches (ip, fail_count, locked_at, lock_duration_seconds), so a tuple changed while they waited no longer matches and the stale decision is skipped. _discard keeps the equality compare-and-set rather than adding an expiry condition: its early-release branch deletes a lock whose committed sentence has not elapsed under the lowered live policy, which an expiry predicate would wrongly block, and the equality predicate already pins every input of the decision. reset stays unconditional by design. Tests: a dialect-compile test asserts the DELETE's own WHERE carries the expiry predicate for postgresql and sqlite (red on the previous HEAD), and a PostgreSQL-only test (skipped without TEST_POSTGRES_URI) reproduces the update-versus-sweep interleaving with two engines, waiting on pg_stat_activity for the sweep to block before the extender commits. Docs: sql.py concurrency contract and the migrations guide's 0034 bullet. * fix(auth): keep an active login lock's clock and fail closed on explicit db Address the "changes requested" review on #6501. Lock clock (confirmed defect): record_failure re-stamped locked_at = now for every failure on an already-locked row, so a sustained attacker's sentence slid forward with each attempt. Both stores now apply one rule in one atomic decision: new_count = fail_count + 1; below max_attempts the IP counts with no lock (a failure after the operator raised the threshold clears an existing lock); at or over it, a failure during an *active* lock keeps locked_at and the committed lock_duration unchanged, while a never-locked or served row starts anew with now / lockout_seconds (so the failure that reaches a raised threshold starts a fresh lock rather than keeping a stale locked_at). The SQL upsert expresses it as nested case() on the row's own values; the memory store mirrors it. The sweeps in both stores now leave the row being recorded to that decision instead of deleting a served lock or stale counter first, which split the contract (SQL would have produced a fresh (1, NULL, NULL) row). base.py states the rule. Explicit db downgrade: resolve_login_throttle_store raises SystemExit — the failure type of the other startup validations in app.gateway.deps — when throttle_storage='db' resolves to the shared table but no engine is initialised, so langgraph_runtime refuses to start instead of silently running every replica on per-process counters; the lazy bare-app path applies the same rule and installs nothing. 'auto' keeps warning and falling back. 'db' on a memory database still degrades with a warning (there is no shared table to insist on). Non-blocking items: _key() logs a warning when a client key is truncated to 255 characters (never the key itself); the sql.py docstring covers wall-clock locked_at across replicas (run NTP; skew shifts evaluation by that much) and why STALE_COUNTER_SECONDS=24h / SWEEP_BATCH_SIZE=200 were chosen; a store-level blocking-I/O anchor covers the SQL reset path; the _validate_login_throttle_storage warning was verified to render 'throttle_storage=memory' for the StrEnum (the gate test now passes the enum member and asserts that text). reset stays unconditional: a peer failure that commits after a reset leaves fail_count=1, one genuine failure after the cleared history, which a new test documents. Docs: CHANGELOG entry, config.example.yaml, CONFIGURATION.md, AUTH_DESIGN.md and the throttle_storage field description. * fix(auth): re-evaluate a login-throttle check whose compare-and-set missed SqlLoginThrottleStore.check applied its decision (commit the live duration, or clear a served / released lock) with a compare-and-set predicate on the snapshot it had read, but ignored the affected-row count. When a peer changed the row between the read and the write — an admitted wrong password incrementing fail_count on another replica, or a successful login clearing the row — the UPDATE or DELETE matched zero rows, yet check reported the decision as applied: it could return an extended 60s sentence while the row kept its 1s duration, so the next check cleared the lock and let password verification through; a live duration decrease could be dropped the same way; and a row a peer had just cleared was still reported as locked (#6501 review, P2). Every compare-and-set write in check now checks result.rowcount. On a miss the row is re-read and re-evaluated from the top, up to CHECK_CAS_ATTEMPTS (3) times, applying a fresh compare-and-set decision on the new snapshot and never clobbering a reset or replacement: a row that is gone is reported as allowed, a fresh peer lock is reported as locked, and an incremented count gets the extension or decrease committed on it. Exhausting the bound fails closed: nothing is written and the answer is the last snapshot's lock under the live policy, logged at debug. The read stays in its own short transaction and each write in a separate one. The evaluation is now one shared pure function, evaluate_check in base.py (CheckDecision), used by both stores so the contract cannot diverge. The memory store keeps a single pass: its compare-and-set cannot miss because there is no await between the read and the write on the loop thread, which its docstring now states. Tests (red first on df26c2acf) use a session proxy that runs the peer mutation on a second engine right after the snapshot read ends its transaction: the reviewer's raised-duration interleaving (59.5s reported and committed, the increment survives, a later check still locked), the mirrored decrease, early release on the fresh row, a reset landing between read and write (0.0, no row recreated), and the bound-exhausted fail-closed case. * docs(changelog): resolve merge markers left in the Unreleased section * test(migrations): pin 0035_login_throttle as the head in the 0034 watermark test Upstream #6477 added test_migration_0034_run_event_seq_watermark.py asserting 0034 is the single head; this branch chains 0035_login_throttle after it, so the test now pins the current head and walks the chain like its siblings. * perf(auth): one session per login-throttle check, one transaction per failure Two non-blocking performance suggestions from the #6501 review. Single session per check. The router probed the row with store.get() and then called store.check(), which re-read it: a recorded IP cost two connection-pool checkouts and two SELECTs per request on the unauthenticated login path, letting a lockout flood compete with legitimate traffic for Postgres connections. LoginThrottleStore.check now takes the policy as an async callable (LoginThrottlePolicy) and resolves it lazily inside the store: the probe, the policy resolution and the compare-and-set loop share one session, a clean IP returns 0.0 without ever calling the policy (the config.yaml read is still skipped for the common case), and a recorded IP resolves it exactly once, also across compare-and-set retries. The decision is made on a snapshot read after the resolution: the policy read is a yield point, and a decision that needs no write has no compare-and-set to catch a row a peer changed meanwhile — so one session, but still a read after the yield. The memory store takes the same signature and re-reads its record after the await for the same reason. The router passes its asyncio.to_thread policy read as the callable and drops the separate probe; get stays on the protocol for tests and anchors. No short-TTL verdict cache: it would mask a successful login's reset for the TTL window, and the single session already halves the hot path. Sweep in the upsert's transaction, plus an index. record_failure ran the housekeeping DELETE in its own transaction and the upsert in a second one, two commits (two WAL flushes on PostgreSQL) per failed login; the sweep now rides in the upsert's transaction with one commit, the keep exclusion already protecting the row being recorded. The new ix_login_throttle_updated_at (ORM __table_args__ and the unreleased 0035 migration, created after the table and dropped before it, each guarded separately so a table left by an interrupted attempt still gets its index) serves the sweep's stale-counter predicate; the served-lock predicate scans a table the sweep itself keeps small. Tests (red first on a137fbc47): a clean IP never resolves the policy and a recorded IP resolves it exactly once (both stores), the SQL store opens exactly one session per check and commits once per record_failure (counting wrappers), the migration test asserts the index after upgrade, after completing a partial upgrade that lacked it, and gone after downgrade while compare_metadata stays empty; the router-level "mutated during policy resolution" tests now inject their mutation through the policy callable. * docs(changelog): note config_version 57 in the login throttle entry |
||
|
Nan Gao
|
5ae953fc2f
|
fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers (#6494)
* fix(memory): keep the retrieval index Pod-local and refresh scopes written by peers
DeerMem's derived SQLite FTS5 index for every user was one WAL database pinned
to {storage_path}/.retrieval. With several Gateway Pods sharing one home volume
each Pod opened that file over a network filesystem (unsupported by SQLite),
every Pod start emptied and refilled it under its peers, one Pod's corruption
recovery deleted files the others held open, and a Pod never learned that a
peer had written a user's facts, so memory_search served stale results.
- New backend-private key memory.backend_config.retrieval_index_path: empty
keeps {storage_path}/.retrieval, a relative path resolves against
storage_path, an absolute path is used as-is. The full rebuild and the
corruption recovery act only on that configured index.
- rebuild_index records each agent scope's manifest signature
(mtime_ns, size, revision) before reading its facts; search_facts rebuilds a
scope whose live signature differs because another process wrote the user's
memory. An own commit advances the recorded signatures of that user's scopes
only when they were in sync at the previous revision, so own writes never
rebuild while an interleaved peer write still does.
- Gateway startup warns, in the style of _validate_agent_storage, when a
declared multi-instance deployment keeps the DeerMem index inside
storage_path.
- config.example.yaml documents the key; config_version 55 -> 56, mirrored in
the Helm chart and its README.
- Helm mounts an emptyDir at /var/lib/deerflow/memory-index, points the key at
it, drops the legacy memory.storage_path: memory.json line, and documents the
memory layout next to the other multi-replica prerequisites.
* docs(changelog): link the memory retrieval index entry to #6494
* fix(memory): never advance the index signature on a no-op commit and resync whole scopes
Two review findings on the cross-replica retrieval re-sync:
- save() and apply_changes() inferred the replaced manifest revision as
"returned revision - 1". The commit helper returns the existing manifest
unchanged for a no-op (unchanged-value or empty change set), so after a peer
bumped the manifest an own no-op write advanced this process's stale index
signature to the peer's revision without carrying the peer's delta: a fact the
peer deleted stayed searchable until the manifest changed again. Both paths
now read the real pre-commit revision under the user lock and pass
committed=(pre_revision, signature) only when the commit produced a new
revision; a no-op passes None so the next search still rebuilds the scope.
- The scoped rebuild (signature mismatch and the pre-existing dirty-scope path)
listed facts through list_facts() with its default limit of 100 and then
replaced every row of the scope, so a scope with more than 100 facts lost its
tail on the first re-sync. It now reads the scope's complete fact list.
* fix(memory): fence index signature promotion against a concurrent scope rebuild
Review finding on the cross-replica re-sync: a successful adapter notification
does not prove its delta is still in the index when the signature is promoted.
The SQLite lock covers single adapter calls only, and neither the scope lock
nor DeerMem's warm-up lock spans "notify, then promote", so rebuild_index can
replace a scope's rows in between. After a peer summary-only commit (r+1), A's
search starts a scoped refresh that captures the r+1 signature and reads f1/f2;
A's own deletion of f2 commits r+2 and removes the row; the refresh then
installs its stale f1/f2 and marks the scope synced at r+1; the deletion
resumes and promoted that signature to r+2, so later searches saw the live
signature as equal and kept returning the resurrected f2 until another manifest
change.
Mechanism: a generation fence rather than a lock. rebuild_index bumps a
per-scope generation (scoped rebuild) or a storage-wide generation (full
rebuild) under _cache_lock whenever it publishes or forgets a scope's rows, in
both mark_synced and forget_synced. _dispatch_retrieval_notifications snapshots
the generations under _cache_lock before its first adapter call and, at
promotion time, skips every scope whose generation moved (and everything when
the storage-wide generation moved), leaving the rebuild's own captured
signature in place; that signature predates the commit, so the next search's
comparison rebuilds the scope. A lock would have had to serialize notification
plus promotion with bulk install plus publication while search_facts ->
rebuild_index -> load() already takes the scope, file and cache locks; the
fence adds no lock-order obligations and only costs a redundant rebuild in the
rare race. Every interleaving is covered: a rebuild that completes before the
snapshot is reflected in the captured generation and promotion proceeds on an
index that carries both the rebuild's rows and this write's delta; one that
publishes between snapshot and promotion is skipped; one that publishes after
the promotion records its stale captured signature, which the next search
catches.
Tests reproduce both orders deterministically with threading.Event ordering
(adapter wrapper that holds the bulk install and the remove notification):
test_promotion_is_fenced_against_a_refresh_that_installs_in_between (red before
the fix) and
test_stale_refresh_installing_after_a_promotion_is_caught_by_the_next_search
(regression guard for the mirrored order).
* fix(memory): publish a rebuild's signature under the lock that replaced its rows
Review finding on the cross-replica re-sync: two peer-triggered refreshes of
one scope could replace rows in one order and publish their captured
signatures in the other. On a warmed Pod A, refresh R1 captures r+1 with f1/f2
and waits before installing; a peer deletes f2 at r+2; refresh R2 captures
r+2, installs f1 only and pauses after bulk_rebuild returns, before
mark_synced; R1 installs its stale f1/f2 and publishes r+1; R2 then publishes
r+2 over R1's rows. Later searches saw the live signature as equal and kept
returning the deleted fact. The generation fence only guards own-write
promotion, and the adapter's SQLite lock ends when the install returns.
Mechanism: a per-storage _retrieval_publish_lock held across the row
replacement and the signature publication, for the bulk path (bulk_rebuild +
mark_synced / forget_synced) and the clear+upsert fallback, scoped and full
alike. Overlapping refreshes of a scope therefore serialize their
install+publish pairs: whichever installs last also publishes last, and a stale
captured signature published last differs from the live manifest, so the next
search's compare rebuilds the scope. Fact reading (load() / the glob) stays
outside the lock so a slow read never blocks refreshes of other scopes. The
startup full rebuild holds the lock for its single bulk install; scoped
refreshes wait for it, which is acceptable because before warm-up completes a
search would be rebuilding its scope anyway (DeerMem.warm_retrieval takes its
own _retrieval_lock only after rebuild() returns, so the two never nest).
Lock order: the publish lock is acquired with no storage lock held --
search_facts releases _cache_lock before calling rebuild_index, reload() leaves
its scope/file-lock block before rebuilding, and DeerMem._ensure_retrieval_scopes
holds only its outer RLock, which nothing inside rebuild_index acquires. Inside
the lock only the adapter's own lock and _cache_lock are taken, and both are
leaf locks that never call back into storage, so no cycle is possible.
Test: test_rebuild_publication_stays_ordered_with_row_replacement drives the
reviewer's interleaving with threading.Event ordering (red before the fix). The
pause harness now holds a refresh after its fact read, outside the publish
lock, instead of inside the adapter's rebuild, so the earlier fence tests keep
their semantics without holding the new lock while paused.
* fix(memory): read each manifest once per full rebuild and serve the local index on a compare read error
Two suggestion-level review items on the cross-replica retrieval re-sync.
- The full rebuild cached manifest signatures keyed by (manifest path, agent
name), but _scope_signature reads only the manifest file -- every agent
bucket of a user shares one memory.json -- so the startup scan stat+read+
parsed the same file once per bucket that has facts. The cache is now keyed
by the manifest path alone while synced signatures are still recorded per
(user, agent) scope: users manifest reads instead of users x agents.
- The per-search freshness compare put a shared-volume manifest read on every
search_facts call, and _load_memory_file raises MemoryStorageCorruption on
OSError and malformed JSON, so a transient network-filesystem read error or a
corrupt memory.json failed a search the Pod-local index could still answer.
The compare now catches OSError, ValueError and MemoryStorageCorruption: it
logs a warning naming the scope (traceback at debug), serves the current
local index for that scope and leaves the recorded synced signature
untouched so the next search compares again. It deliberately does not mark
the scope dirty: a scoped rebuild would hit the same failing read, forget the
scope's signature and fail the search path too.
* docs(changelog): resolve merge markers left in the Unreleased section
* fix(memory): forget a scope's index signature when an own delta cannot be proved compatible
Review finding on the cross-replica re-sync: skipping promotion left a newer
rebuild's signature trusted even after this dispatcher had overwritten its rows
with an older notification. On a warmed Pod A, an own PATCH of f2 commits r+1
and pauses after releasing the user lock, before its adapter upsert; Pod B
deletes f2 at r+2; a concurrent search on A detects the peer change, rebuilds
the scope without f2 and publishes r+2; the pending r+1 upsert then reinserts
the deleted fact, and the revision check merely skipped because the synced
revision was r+2 rather than the PATCH's previous r. The signature stayed equal
to the live manifest, so searches returned the deleted fact indefinitely. The
generation fence could not help: it only suppresses promotion, the stale
adapter mutation still happens.
Rule: after an own commit's notifications are applied, the mutated scope's
recorded signature may stay only when the delta is provably compatible with
the published snapshot -- the scope is synced at exactly previous_revision and
no rebuild published rows for it (or storage-wide) since the fence snapshot --
in which case it is promoted as before. In every other case where this
dispatcher mutated the adapter for the scope (synced at another revision,
newer or older; generation moved; not synced at all; or notifications without
a committed revision, i.e. migration and clear paths) the scope's signature is
forgotten so the next search's compare rebuilds it. The user's other scopes
were not mutated by this delta: they are promoted when provable and otherwise
left untouched. The decision runs under _cache_lock together with the
generation reads, so a rebuild publishing concurrently is either observed or
observes the forget; the failure path (dirty + forget) is unchanged.
Orders covered: the reviewer's (publish r+2, then the stale upsert) forgets and
the next search rebuilds without f2; the earlier fence order (publish between
adapter op and promotion) now forgets too, one redundant rebuild; no concurrent
rebuild promotes exactly as before.
Test: test_stale_own_delta_replayed_over_a_newer_snapshot_forgets_the_scope
holds the PATCH's adapter upsert after the user lock is released (new
pause_next_upsert hook on the test adapter), Event-ordered, red before the fix.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Hyeonsang Cho
|
fb0ed9c960
|
fix(browserless): send the web_fetch token as a query parameter (#6484)
* fix(browserless): send the web_fetch token as a query parameter fetch_html_with_status put the configured token in the /content JSON body. Browserless reads the token only from the token query parameter or the Authorization header, and checks it before reading the body, so with BROWSERLESS_TOKEN set every web_fetch failed with HTTP 401; against an instance started without TOKEN the unknown body key failed schema validation with HTTP 400. capture_screenshot already sent ?token=, so web_capture worked with the same token. Both endpoints now take the token from BrowserlessClient._auth_params(). The test that pinned the body token is rewritten, and a stand-in for the Browserless request gate (query/header auth, no extra body keys) drives both endpoints through the real httpx request path. * docs(changelog): reference #6484 --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
litescraper
|
e1458eefee
|
feat: support configurable Serper base URL (#6483)
* feat: support configurable Serper base URL * fix(serper): address configurable endpoint review feedback * fix(serper): address endpoint configuration review * fix(serper): pair provider settings and validate endpoints * test(serper): account for URL redaction in endpoint diagnostics --------- Co-authored-by: litescraper <322634161+litescraper@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Daoyuan Li
|
46eecf2950
|
fix(models): align MindIE synchronous streaming compatibility (#6497) | ||
|
哈基米
|
3772f7c7c3
|
fix(models): keep escaped newlines inside fences in MindIE native stream (#6478)
The no-tool native stream applied _decode_escaped_newlines_outside_fences
to each streamed chunk independently, but a fenced code block almost never
opens and closes within one token-sized chunk, so the per-chunk fence regex
never saw a complete ``` pair and decoded every literal \n inside the fence.
Code like print("a\nb") reached the streamed answer and the persisted
message content with a real newline, while the non-streaming path and the
tool-mode simulated stream preserved it.
Decode the native stream with a fence-stateful decoder instead: it carries
whether the stream is inside a fence across chunks and holds back a
trailing partial ```/`\n` sequence, so the joined stream equals the one-shot
decode of the same reply. A fence left unclosed by a truncated reply keeps
its escapes, matching how an unterminated fence still renders as code.
Document native no-tool streaming in the MindIE decode contract.
|
||
|
Nan Gao
|
5e3c24fe94
|
fix(gateway): probe the Redis stream bridge and provisioner in /health/ready (#6447)
* fix(gateway): probe the Redis stream bridge and provisioner in /health/ready GET /health/ready only probed the ORM engine and the LangGraph checkpointer. The Redis stream bridge client connects lazily and nothing ever pinged it, so a Gateway whose Redis was down started, answered 200 ready to the Kubernetes readinessProbe and Compose healthcheck, and kept receiving traffic while every run's first publish failed. Since #6328 the Redis bridge is mandatory for multi-instance deployments, so such a replica is not serviceable at all. - StreamBridge gains `async ping() -> bool`; the base returns True (no external backend), RedisStreamBridge round-trips PING and answers False on RedisError. - /health/ready adds a `stream_bridge` probe (not_configured for the memory bridge; unreachable flips the endpoint to 503; a missing bridge fails closed) and a report-only `provisioner` probe of `{sandbox.provisioner_url}/health` that never changes the status code, since every replica shares one provisioner. The provisioner URL is snapshotted on app.state at startup. - A probe that overruns the endpoint deadline is now the only one marked unreachable; finished verdicts are kept. - The public probe gate is one lock per probe kind, so one request's probes still run concurrently while a thundering herd opens at most one probe connection per kind per process. - Tests: bridge ping cases, stream-bridge/provisioner probe cases, verdict and deadline cases, and the first route-level /health/ready tests via TestClient. - Docs: gateway AGENTS.md, CONFIGURATION.md multi-instance notes, Helm README and probe comment, Compose healthcheck comment. * docs(changelog): note the readiness probe fix (#6447) * fix(gateway): apply the sandbox proxy policy to the provisioner probe Review follow-up for #6447. - The provisioner readiness probe built its httpx client with the default trust_env=True. httpx 0.28 treats CIDR entries in NO_PROXY as host patterns rather than subnet bypasses, so with HTTP_PROXY set and NO_PROXY=10.0.0.0/8 a private provisioner that the requests-based sandbox calls reach directly would have been probed through the proxy and reported unreachable on every check. The probe now reuses `sandbox_http_trust_env`: loopback, private, link-local and cluster-local addresses bypass the proxy, external hosts keep the environment's proxy settings. - Pin the liveness/readiness separation with a route test: `GET /health` stays 200 while the stream bridge and database are unreachable and `/health/ready` is 503, so a Redis blip can never become a restart loop. |
||
|
Felix
|
f154cfd221
|
feat(wizard): add Opper as an LLM provider (#6399)
* feat(wizard): add Opper as an LLM provider Opper (https://opper.ai) is an EU-hosted AI gateway: 700+ models from 50+ providers behind one OpenAI-compatible API and one key. This adds it to the setup wizard's LLM_PROVIDERS next to OpenRouter, with the same shape: langchain_openai:ChatOpenAI pointed at https://api.opper.ai/v3/compat, env var OPPER_API_KEY. Models use Opper's bare model ids: claude-sonnet-4-6 (default), gpt-5.5, gemini-3.8-flash and gpt-5.4-mini. Because the GPT ids are bare, langchain-openai drops the 0.7 temperature for them the same way it does for direct OpenAI gpt-5 models. Checked: python -m py_compile scripts/wizard/providers.py, and each model id against Opper's public model list (GET https://api.opper.ai/v3/models). Not run locally: make lint and make test. Disclosure: Opper is my startup. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs: add Opper to the config example and READMEs Add an Opper model example next to the other OpenAI-compatible gateways, as requested in review: - config.example.yaml: commented opper-claude-sonnet-4-6 block after the other gateway examples (ChatOpenAI + base_url, OPPER_API_KEY) - README.md and the translated READMEs: the same entry in the manual model configuration examples, after OpenRouter - .env.example and backend/docs/CONFIGURATION.md: OPPER_API_KEY, in the shape of the Novita lines claude-sonnet-4-6 is a pooled id in Opper's public catalogue (GET https://api.opper.ai/v3/models). config.example.yaml still parses with yaml.safe_load; config_version is unchanged because the schema is. Disclosure: Opper is my startup. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
haochen
|
cde9602631
|
docs: run configuration and migration commands in the backend environment (#6425)
* docs: run configuration and migration commands in the backend environment * docs: preserve runtime paths and extras in backend commands --------- Co-authored-by: Zhangchenhao-111 <285911435+Zhangchenhao-111@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
cangen185-sudo
|
4918bea98b
|
fix(models): preserve escaped newlines in MindIE XML JSON arguments (#6334)
* fix(models): preserve escaped newlines in MindIE XML JSON arguments * fix(models): restore multiline raw MindIE tool arguments |