mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-30 23:20:35 +00:00
4571 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
RongJie G
|
b708145001
|
fix(runtime): require explicit resume identity capability (#6571)
* fix(runtime): require explicit resume identity capability Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com> * fix(runtime): guard key-only idempotency admission --------- Signed-off-by: CorgiBoyG <111257566+CorgiBoyG@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
TYY-2005
|
34b7a2d13b
|
fix(frontend): preserve newer reconnect pointers during gap recovery (#6662)
* fix(frontend): preserve newer reconnect pointers during gap recovery * test(frontend): cover same-run replay-gap pointer restoration --------- Co-authored-by: TYY-2005 <334029581+TYY-2005@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
ranjianghong526-hash
|
04a45db836
|
fix(skills): encode GitHub API fallback query parameters (#6661)
Co-authored-by: ranjianghong526-hash <228174768+ranjianghong526-hash@users.noreply.github.com> |
||
|
ZJPex
|
128dafd0ce
|
feat(tools): add opt-in literal keyword discovery (#6659) | ||
|
spud
|
75ce6f4920
|
fix(mcp): preserve server binding history across gateway workers (#6617)
* fix(mcp): preserve server binding history across gateway workers A peer Gateway worker could keep a stale pooled session after another worker deleted and identically re-added a stdio server, because the final effective MCP slice matched the revision it had already applied. Record a per-server lifecycle generation in the shared extensions config and consume it in the reader, discovery claim, and committed handoff so an observed generation change force-rebinds only the affected server. Legacy configs derive a stable in-memory token; deletions leave permanent tombstones. * fix(mcp): fail closed on an invalid mcpLifecycle instead of retrying An unverifiable mcpLifecycle made initialize_mcp_tools() unpublishable, so get_cached_mcp_tools() retried discovery forever and could keep respawning stdio servers. Abort discovery on an invalid ledger after conservatively retiring local MCP state; a fixed config recovers on the next call. Also documents the rolling-upgrade prerequisite: cross-worker ABA detection requires configuration writes to go through a lifecycle-aware writer. * refactor(mcp): share stdio fingerprinting across lifecycle paths Reuse lifecycle._stdio_fingerprints from the applied-revision builder so the connection-identity rule has a single implementation, and move the unusable-server diagnostic into the shared helper. Add a regression proving the Gateway Skills writer preserves mcpLifecycle byte-for-value, and document that DeerFlowClient.update_mcp_config can raise RuntimeError after the config was committed. * docs(mcp): condense lifecycle guidance under the AGENTS chain limit The effective mcp/AGENTS.md chain was 15 bytes under the 98,304-byte CI hard limit, so the standalone cross-worker lifecycle bullet pushed it over. Fold the mcpLifecycle and rolling-upgrade facts into the existing selective-reconciliation bullet and condense it, keeping the chain under the limit while preserving the contract. * docs: trim inherited agent guidance below CI size limit --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Dragon ze
|
e5d9a5f42d
|
refactor(community): give retrieval credential handling a single owner (#6638) | ||
|
Dragon ze
|
6709c58256
|
refactor(models): give reasoning capture plumbing a single owner (#6639) | ||
|
Hyeonsang Cho
|
b9b74d2338
|
fix(scripts): reject local BIND_HOST values nginx would misread (#6635)
* fix(scripts): reject local BIND_HOST values nginx would misread nginx-local-conf.sh checked only for address and hostname characters. BIND_HOST=0 rendered `listen 0:2026;`, which nginx resolves to 0.0.0.0 and binds on every IPv4 interface, and 1.2.3 bound 1.2.0.3. Values nginx rejects outright ([[fd00::1]], -foo) passed the serve.sh preflight and failed only after stop_all had torn down the running stack. Accept exactly a dotted-quad IPv4 address, an IPv6 address (brackets allowed around IPv6 only), or an RFC 1123 hostname. Returning to the loopback default removes a temp/nginx.local.conf rendered for an earlier value, and stop_all now says why it can quit a rendered-config nginx through the tracked config; a test pins the shared pid directive. * docs(changelog): reference #6635 * docs(changelog): match the zh reference spacing * fix(scripts): check every BIND_HOST hostname character before splitting labels read -r -a stops at the first newline, so is_hostname validated only the first line of a value like $'localhost\nbad' and the rest reached the rendered listen directive. Check the allowed characters across the whole value first, and cover multi-line, CR and tab values for each validator. |
||
|
creed
|
3461c12896
|
feat(jina): add process-local request admission (#6489)
* feat(jina): add process-local request admission Signed-off-by: 97three <2212371308@qq.com> * test(jina): cover closed admission owner loop Signed-off-by: 97three <2212371308@qq.com> * test(jina): retain admission across API redirects Signed-off-by: 97three <2212371308@qq.com> * fix(jina): drain redirect response cleanup before releasing admission Signed-off-by: 97three <2212371308@qq.com> --------- Signed-off-by: 97three <2212371308@qq.com> |
||
|
jiujiu
|
3234563db8
|
fix(utils): skip malformed response block types (#6614)
Co-authored-by: songjianglin <270247441+2017955725@users.noreply.github.com> |
||
|
IronMurphy
|
05297813ae
|
fix(lark): publish the CLI runtime files atomically (#6618)
* fix(lark): publish the CLI runtime files atomically install_shim wrote the shim, the launcher and the runtime marker with a truncating open. The sandbox chmods the launcher 0o755 and execs it, so a crash mid-write left an executable fragment, and the marker is parsed as JSON. Publish through a temporary file plus os.replace, as the storage backends already do. * style: drop the unused import ruff flags * fix(lark): publish runtime permissions atomically Set file permissions on temporary files before os.replace() to ensure the launcher, shim, and runtime marker are atomically published with correct permissions (0o755 for executables, 0o644 for the marker). This addresses reviewer feedback from PR #6591: - Permissions are now set before the atomic replace operation - Prevents brief window where files have incorrect permissions - The runtime marker now has mode 0o644 instead of 0600 (mkstemp default) Also updates test_lark_shim_atomic.py: - Fixes test_atomic_helper to match new function signature - Adds test_install_shim_publishes_correct_permissions to verify launcher/shim (0o755) and marker (0o644) permissions Co-authored-by: IronMurphy <hepushuo@gmail.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
NanPan
|
38c55c67f6
|
fix(streaming): report replay gap for future Redis Last-Event-ID (#6605)
* fix(stream): report replay gap for future Redis stream cursors * test(stream): future reconnect cursors report gap on live and ended runs |
||
|
Jason Cat
|
65a186187d
|
fix(workspace-changes): keep the package importable as a first import (#6602) (#6608)
`workspace_changes/api.py` imports `deerflow.runtime.user_context` for the AUTO sentinel (#6282). Importing that submodule runs the runtime package init, which imports `.runs` -> `worker` -> `deerflow.workspace_changes` back for `capture_workspace_snapshot` and friends, so importing `.api` as the package's first statement failed with ImportError while those names were still unbound. Resolve the re-export lazily via a module-level `__getattr__`, the pattern `deerflow/subagents/__init__.py` already uses. |
||
|
Nan Gao
|
35f6a98cd9
|
feat(dev): add a two-Gateway multi-instance harness script (#6613)
* feat(dev): add a two-Gateway multi-instance harness script There was no way to run the deployment.multi_instance topology locally: scripts/serve.sh pins one Gateway to port 8001, the local nginx config has a single upstream, and getting two processes past the multi-instance startup gate needs Postgres, Redis, a shared DEER_FLOW_HOME, shared auth secrets and a per-process DeerMem retrieval index by hand. scripts/dev_multi_instance.sh (make dev-multi / dev-multi-check / dev-multi-down) starts throwaway Postgres and Redis containers on loopback, renders a config from the developer's config.yaml (or config.example.yaml), and runs Gateway A on 8001 and Gateway B on 8011 with one shared home, one copy of extensions_config.json and generated AUTH_JWT_SECRET, DEER_FLOW_INTERNAL_AUTH_TOKEN and DEER_FLOW_CREDENTIALS_KEY values, plus an optional round-robin nginx on 2027. B starts after A answers /health: Alembic migrations are advisory-locked, but the LangGraph checkpointer/store setup() that follows is not. scripts/dev_multi_instance.py holds the testable parts: the config overlay (neutralizing and reporting settings the gate refuses), the secrets file, the nginx config, and `check`, which automates the scriptable cross-instance acceptance checks: readiness with stream_bridge ok, A-issued sessions and the internal token accepted by B, thread/upload/artifact visibility, a skill toggle seen by B, nginx balancing, and SSE join on B with Last-Event-ID resume on A. Tests pin that the rendered config passes the real _enforce_postgres_for_multi_worker gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): link the two-Gateway harness entries to #6613 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dev): keep the multi-instance harness inside its own state Address review findings on the two-Gateway harness (#6613): - Containers carry a deerflow.harness.state-dir label; up/down remove only containers of their own state dir and refuse those of another checkout or DEERFLOW_MI_STATE_DIR instead of force-removing them. - Containers are removed with `docker rm -f -v`, so the anonymous volumes the postgres and redis images declare no longer leak on every up/down. - `up` installs an EXIT trap once it starts containers, so any later failure (including B's port being taken after preflight) tears down A and both containers; start_gateway returns failures instead of exiting the script, and the Gateway is disowned so stopping it prints no job notice. - The overlay removes any checkpointer section (a postgres one would write checkpoints and the Store into the developer's database), points an explicit sandbox.ownership and a Redis checkpoint_cache at the harness Redis (keeping ownership timing), drops DeerMem storage_path (both spellings) and the blob_storage root so data lands in the harness home, gives a local AIO sandbox a state-dir-specific container_prefix, and reports a non-DeerMem memory backend it keeps sharing. - nginx paths are rendered as quoted string literals ('$' and control characters are refused), so a state dir with spaces works. - `check` skips the run-based SSE check when model discovery did not succeed, unless --with-llm, and enforces the SSE time budget per raw line so heartbeat-only traffic cannot hold it open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(scripts): keep the harness guidance under the AGENTS.md soft limit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
ZJPex
|
7e0e3013d0
|
feat: return hit-centered history search excerpts with source offsets (#6008)
* feat: return history search excerpts near keyword matches * docs: use English for history search guidance and comments * docs: sync Chinese history search excerpt contract * docs: clarify history excerpt fallback for oversized matches * fix(history): skip oversized occurrences when selecting excerpts Continue scanning casefolded substring matches until an occurrence fits the 600-character excerpt, including overlaps. Cover later fitting hits through synchronous and asynchronous active/archive search and read tools, and update the documented contract. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Storm
|
588ad010c0
|
fix(mcp): stop persisting URL-sourced ResourceLink blocks and heal bricked threads (#6606)
* fix(agents): sanitize persisted URL content blocks before model calls
Threads whose checkpoints carry URL-sourced file blocks (written before
the MCP conversion-layer fix) fail every model call at serialization: the
Chat Completions translator rejects any file block carrying a url, and the
error handler answers each later turn with a non-retriable fallback. Add a
read-time sanitizer that rewrites those blocks in the model-bound request
view only, so poisoned threads heal on their next turn with no migration
and no user action.
- ModelContentCompatibilityMiddleware wraps the model call (sync and
async) and rewrites ToolMessage list content: URL-sourced file blocks
become "[Resource ({mime or unknown type}) available at {url}]" text
blocks, and image blocks whose scheme is not http/https/data get the
same placeholder. Fetchable image URLs, base64 payloads, text, and
non-ToolMessage messages pass through. The rewrite is order-preserving,
idempotent, and never writes state or mutates checkpoints, so artifact
capture and other state readers keep seeing the original blocks.
- Register on all three chains that replay checkpointed ToolMessages:
build_middlewares (lead; outer of the MODEL_PHYSICAL extension anchor so
its final-request guarantee holds), the SDK factory
_assemble_from_features (new step 13a; docstring updated and its stale
omission of TokenBudgetMiddleware fixed), and
build_subagent_runtime_middlewares (after DurableContext/summarization,
before the date/coalescer tail).
- Subagent coverage verified by tracing every subagent agent-creation
site: executor.py _create_agent is the only caller of
build_subagent_runtime_middlewares and serves both the task tool and the
durable batch worker (SubagentRuntime itself only carries dependencies),
and a wiring test now asserts the middleware is present in that
builder's output; the lead and factory chains gained the same wiring
assertions.
- Update the chain-order contract tests for the new slot: full-chain
order, loop-detection adjacency, custom loop-detection tail positions,
and the subagent chain count (22 -> 23).
* fix(mcp): stop emitting URL-sourced file blocks from ResourceLink results
MCP ResourceLink conversion produced a URL-sourced file content block for
every non-image link, which LangChain's Chat Completions translator rejects
at serialization time. Once such a block was checkpointed in the
ToolMessage, every subsequent turn of the thread failed at model-call time,
permanently bricking the conversation.
- Downgrade every ResourceLink that is not an http(s) image to a short text
placeholder carrying the resource name, MIME type, and the
already-virtualized URL, so host paths never leak and the model still
learns a resource was produced and where it lives.
- Preserve each downgraded link as structured data in the tool-message
artifact under resource_links (merged with structured_content when
present) for artifact capture and future MCP Apps rendering.
- Keep http(s) image links as image blocks; embedded base64 payloads, text
path rewriting, and the error-to-ToolException path are unchanged.
- Re-contract the file-migration tests to the new behavior and document
the conversion contract in the MCP AGENTS.md.
* docs(middlewares): add ModelContentCompatibilityMiddleware to the chain inventory
Insert the read-time sanitizer at its actual lead-chain position (after the
token-budget guard, before the custom-middleware injection point) and renumber
the tail entries 34-39 to 35-40.
* fix(agents): rewrite file blocks on url key presence, not truthiness
langchain-core's OpenAI translator raises for ANY file block carrying a
url key — an empty string or None included — so the read-time sanitizer
must not let such blocks through just because the value is falsy. A
persisted block with url="" or url=None would still brick every later
model call in the thread.
- Gate the file-block rewrite on "url" in block instead of a truthiness
check; a non-empty string URL keeps the
"[Resource (mime) available at url]" placeholder, while an empty or
None URL gets a location-less "[Resource (mime)]" placeholder, so no
unusable location is advertised to the model.
- Pin both shapes with new middleware tests (url="" and url=None file
blocks are both rewritten to text). The image branch is unchanged: an
empty-image case fails provider-side with a 400, not a bricked thread.
* fix(mcp): keep data URIs out of model text and resource_links
A data: scheme ResourceLink fell into the generic placeholder branch,
inlining megabytes of base64 into the model-visible text and into
resource_links, which is checkpointed state — against the repo policy
that data:/blob: URIs never enter state.
- Special-case data: links in _convert_call_tool_result: the
model-visible placeholder becomes "[Resource: name (mime) embedded
inline]" and the raw URI is not added to resource_links at all, so the
payload can neither bloat the context window nor persist in the
checkpoint.
- Pin the behavior with a conversion test asserting both the placeholder
wording and the absence of the payload/URI from every emitted block and
the artifact.
- Add the missing passthrough pin for an HTTPS:// uppercase-scheme image
link: scheme matching is case-insensitive, so the link stays an image
block (the block's URL validation normalizes the scheme to lowercase).
* fix(tools): consume resource_links in extract_artifacts_from_result
The MCP conversion layer preserves downgraded ResourceLinks as
{"name", "uri", "mime_type"} entries under the tool message
artifact's resource_links key, but the artifact extractor only read
structured_content — a result whose only artifact signal was
resource_links produced no entries and the file silently vanished from
the artifacts panel.
- Add a fourth extraction path: each resource_links entry becomes a file
artifact entry, reusing _is_referenceable_url like the
structured-content and content-block paths, so data:/blob: and other
unresolvable URIs stay rejected per existing policy. The entry is named
after the link when a name is present, falling back to the URI's final
segment; a string mime_type is carried through.
- Integration-test a result whose only signal is resource_links, plus the
referenceability gate (data:, ui://, and absolute-path URIs in one
artifact).
* docs(mcp): update the ResourceLink conversion contract bullet
Mention the new artifact-registry consumer and the data: special case so
the bullet stays accurate: uppercase http(s) schemes match
case-insensitively, data: URIs get the embedded-inline placeholder and
are dropped from resource_links, and extract_artifacts_from_result turns
resource_links into registry entries through the shared
referenceability gate.
* refactor(tools): add the canonical resource placeholder formatter
Both the MCP conversion layer and the read-time
ModelContentCompatibilityMiddleware downgrade model-incompatible resource
references to a text placeholder, but each built its own literal. Extract
the single formatter they will share so a thread healed at read time shows
the model the same placeholder shape a fresh conversion produces; the name
segment is optional because persisted content blocks carry no resource name.
* fix(mcp): keep host paths out of ResourceLink placeholder text
Review follow-up on the ResourceLink conversion contract. The placeholder
embedded the resolved URL unconditionally, so a link outside the user-data
tree (or converted without thread context) leaked a raw host path — a
file:// URI or bare filesystem path — into model-visible text, violating
US-16. Compute the resolved URL's scheme and withhold the location segment
when it spells out a host filesystem location (empty scheme, file, or a
single-letter Windows drive scheme); virtual paths and remote schemes keep
their location. The unresolved URI still lands in the artifact channel,
which the model never sees.
- add _model_visible_link_location gate and route the generic placeholder
through it, re-contracting the outside-user-data and no-context tests to
the location-less shape and pinning the Windows-drive case
- treat blob: like data: as non-referenceable inline input: drop its raw
URI from checkpointed resource_links and emit the location-less
placeholder instead of the mismatched 'embedded inline' wording, and fix
the comment that claimed blob: was already excluded
- replace every inline placeholder f-string in the ResourceLink branch
with the shared resource_placeholder_text formatter so the conversion
layer and the read-time middleware emit byte-identical shapes; data:
links now render in the canonical location-less form
- sync the ResourceLink conversion contract bullet in mcp/AGENTS.md and
the _convert_call_tool_result docstring with the host-path suppression
rule and blob: handling
* fix(agents): sanitize URL content blocks on any message role
Address the second-pass review of the request sanitizer with two
changes that share the same code path.
- Adopt the canonical resource_placeholder_text formatter from
deerflow.tools.resource_placeholder instead of the module-local
placeholder helpers, so the read-time rewrite and the MCP conversion
layer emit byte-identical placeholder text from one definition.
- Widen the rewrite from ToolMessage-only to every request message with
list-form content. User uploads are verified to produce base64 blocks,
but imported or cross-client history can carry a URL-sourced file
block on a HumanMessage or AIMessage, and the Chat Completions
serializer rejects such a block regardless of the role carrying it.
The rewrite stays request-view-only: state and checkpoints are never
mutated, and base64/file-id blocks and string content pass through
untouched on any role.
* docs(agents): state the ModelContentCompatibility placement contract
The comment above the registration claimed the sanitizer sits just
before Clarification. In the lead chain it is actually followed by the
custom/extension injection point and the whole terminal tail
(terminal-response, finish-reason, and the always-last
ClarificationMiddleware). Describe the real contract: after every
built-in content transform, before extension injection and the
terminal tail.
* docs(middlewares): clarify provenance stamps for request-view-only rewrites
The message-provenance rule read as if every rewrite needs a
provenance_kwargs stamp. Add one line noting the exception: rewrites
that only modify the model-bound request view and never enter state
need no stamps, citing InputSanitizationMiddleware,
DanglingToolCallMiddleware, and ModelContentCompatibilityMiddleware as
precedents.
* fix(agents): keep host paths and data URIs out of healed placeholders
The read-time ModelContentCompatibilityMiddleware rewrote URL-sourced
file blocks and non-fetchable image blocks into placeholders that
inlined the block's URL verbatim. Pre-fix checkpoints — exactly the
threads this middleware exists to heal — can carry a file:// URI, a bare
or Windows host path (links that never resolved to virtual paths), or a
data: URI embedding megabytes of base64, so healing leaked deployment
filesystem layout into prompts (US-16) and could inject the whole inline
payload into model-visible text.
The conversion layer already suppresses those forms via
_model_visible_link_location. Move that gate into the shared
deerflow.tools.resource_placeholder module as model_visible_location
(now also collapsing data:/blob: URIs) and route both middleware
branches through it, so both layers apply identical suppression rules
with one definition. Conversion behavior is unchanged: its data:/blob:
branch short-circuits before the gate, and the gate covers those schemes
for defense in depth.
- resource_placeholder.py gains model_visible_location alongside the
shared formatter; mcp/tools.py drops its private copy and imports it.
- The middleware's file and image branches pass the URL through the gate
before building the placeholder; module docstring documents the rules.
- Tests: gate unit tests (virtual/remote visible; file://, bare/Windows
host paths, data:/blob: withheld) and middleware regression tests
pinning that a pre-fix data: payload, file:// URI, and Windows drive
path heal to location-less placeholders.
- AGENTS.md bullets in mcp/ and middlewares/ state the shared gate.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(agents): downgrade malformed image URLs instead of raising in the sanitizer
The read-time ModelContentCompatibilityMiddleware gated persisted image
blocks on urlparse(url).scheme, but urlparse raises ValueError on
structurally invalid URLs — a bad IPv6 literal, invalid netloc
characters — instead of returning an unusable scheme. Pre-fix
checkpoints and imported history can carry exactly those URLs, and the
block stays in the checkpoint, so every later model call in the thread
raised again: the sanitizer re-bricked the very threads it exists to
heal, only with a framework-side exception instead of the langchain
serialization error.
Catch ValueError and treat the URL as non-fetchable, so the block
degrades into the same text placeholder as any other unresolvable
scheme (the shared model_visible_location gate already swallows
ValueError, so the location segment is withheld as well).
- wrap the image branch's urlparse call in try/except ValueError and
fall back to an empty scheme, keeping http/https/data pass-through
behavior unchanged for well-formed URLs
- add a regression test pinning that an image block with a bad IPv6
literal heals to the location-less placeholder without raising
Co-Authored-By: Codex <noreply@openai.com>
* fix(tools): dedupe artifact refs across extraction sources
One downgraded ResourceLink produced two registry entries: the typed
resource_links channel added an entry for the URI, and the free-text
scan over the conversion layer's placeholder text — which embeds the
same virtual path or remote URL — matched it again. The two handles
pointed at one resource, spent double of the render budget, and showed
the model two handles for a single artifact.
extract_artifacts_from_result now tracks seen real_refs across all
sources and keeps only the first entry per ref. Extraction order makes
the typed sources (structured content, resource_links, content blocks)
win over the free-text scan, so the surviving entry keeps the link's
display name and mime type.
- route all six entry-producing sites through a local _add_unique
helper keyed on real_ref; per-source behavior is unchanged
- document the cross-source dedup in the function docstring
- add a regression test covering a virtual-path and a remote-URL
placeholder together with their resource_links entries
Co-Authored-By: Codex <noreply@openai.com>
* docs(mcp): state the intentional data: image downgrade asymmetry
The conversion layer downgrades image/* data: ResourceLinks to
location-less text placeholders, while the read-time middleware lets
persisted data: image blocks pass through to the model. The two
contracts looked contradictory but are deliberately layered: the
conversion layer keeps new inline payloads out of checkpointed state,
and the middleware only preserves blocks already occupying one.
- mcp/AGENTS.md's ResourceLink conversion contract and the conversion
comment in mcp/tools.py now state the image/* data: downgrade and
why it differs from the read-time pass-through
- middlewares/AGENTS.md's ModelContentCompatibilityMiddleware entry
states the symmetric read-time rule
- pin the behavior with an image-mime data: ResourceLink conversion
test (location-less placeholder, nothing in the artifact channel)
Co-Authored-By: Codex <noreply@openai.com>
* docs(backend): compact AGENTS.md guidance to fit the chain byte budget
The branch's new contract bullets pushed the middlewares and mcp
effective AGENTS.md chains past the 98304-byte AG002 hard limit (the
sandbox/subagents chains on main were already over it). Compact the
guidance without dropping contracts:
- middlewares item 34 points at the MCP ResourceLink conversion
contract and the module docstring instead of restating both
- the mcp ResourceLink conversion contract bullet is tightened
- backend/AGENTS.md merges the File Upload and Context Summarization
bullet sprawl, folds the stray ATX test note into the upload outline
rule, drops the Nginx routing block the root guide already covers,
merges the duplicate MCP-task pointer bullet, compacts Vision
Support, and scopes the Project Structure tree to the backend
subtree (the root guide maps the full repo)
Co-Authored-By: Codex <noreply@openai.com>
* fix(agents): gate the file-URL downgrade on provider support and neutralize placeholder metadata
Two review findings on the read-time sanitizer:
- The file-block downgrade ran for every provider, but only Chat
Completions rejects URL-sourced file blocks: the OpenAI Responses API
(input_file file_url) and Anthropic (document url source) accept
http(s) document URLs, so a persisted block created for those
providers lost its document from the model request. Keep the block
when the selected model serializes http(s) file URLs natively; every
other URL shape (non-http schemes, empty/None, malformed) still
degrades to the placeholder.
- The placeholder interpolated persisted MIME/location metadata
verbatim, but the middleware runs after the input/tool-result
sanitization passes and historical URL blocks never went through them
(non-text content is left untouched), so values carrying
<system-reminder>-style tags or user-input boundary markers reached
the model raw. Route the interpolated metadata through the shared
neutralize_untrusted_tags guard in both the file and image branches.
Tests: provider gating (Responses/Anthropic preserve, Chat Completions
still downgrades, non-http and malformed URLs downgrade even for
capable providers) and metadata neutralization (injection tags
HTML-escaped, boundary tokens neutralized, file and image branches).
Co-Authored-By: Codex <noreply@openai.com>
* fix(tools): require a directory boundary before exposing virtual locations
model_visible_location admitted any URL starting with the virtual
prefix, so a sibling like /mnt/user-data-backups/ops/private/report.pdf
was treated as a virtual path and its host location reached
model-visible placeholder text — defeating the US-16 host-path
suppression the gate exists for. Require the exact virtual root or a
slash-delimited descendant; siblings now collapse to the location-less
placeholder like any other host path. Both the file and image
downgrade branches share the gate, and the conversion layer applies
the same rules to fresh results.
Regressions: gate unit tests (exact root and descendants stay visible;
/mnt/user-data-backups and /mnt/user-database siblings withheld) and
middleware tests pinning the location-less placeholder for both
branches.
Co-Authored-By: Codex <noreply@openai.com>
* docs(agents): scope the placeholder artifact-visibility claim to the conversion layer
The module docstring claimed the healed placeholder keeps the location
visible to the artifact free-text scan and that the link survives in
the structured artifact channel — true for the conversion layer, whose
placeholders are checkpointed, but not on the healing path: the request
view never reaches artifact capture, and no resource_links entry is
created for a downgraded block. Spell out what actually happens here:
a referenceable original block (http(s) URL or absolute host path) is
still captured, from the capture-time scan of the state block rather
than the placeholder, while a non-referenceable one (file://, data:,
blob:) yields no entry and survives only in the untouched checkpoint.
Co-Authored-By: Codex <noreply@openai.com>
* docs(mcp): clarify healing and artifact preservation contracts
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Jason Cat
|
d37616e659
|
test: route four unguarded symlink fixtures through symlink_or_skip (#6619)
* test: route four unguarded symlink fixtures through symlink_or_skip `tests/support/symlinks.py` was added in #5414 so suites that need a real symlink skip cleanly when the host cannot create one. On a stock Windows host (no Developer Mode, not elevated) `Path.symlink_to` / `os.symlink` raise `OSError: [WinError 1314]`, and four fixtures still called them directly, so they reported hard errors instead of skipping: - tests/test_shared_reset_marker.py::test_marker_path_follows_a_symlinked_config - tests/test_view_image_middleware.py::test_host_image_copy_is_scoped_to_current_user_and_thread - tests/test_review_skill_package_tool.py::test_review_skill_package_rejects_symlink_planted_in_callers_outputs - tests/test_channel_file_attachments.py::TestInboundFileSandboxPerms::test_make_inbound_file_sandbox_readable_swap_after_lstat_does_not_follow_symlink These were the only symlink-related failures in a Windows run of the full backend suite; the other bare symlink call sites there are already guarded by a `skipif` or a `try/except OSError`. The view-image test plants its symlink after its scoping assertions, so those still run before the skip. The inbound-attachment test stages a symlink from inside a monkeypatched `os.lstat` to exercise a TOCTOU race; a host that cannot create a symlink cannot stage that race at all, so skipping is the only honest outcome. * test: make channel symlink race fixture portable --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
db7340d2fa
|
fix(security): add DEER_FLOW_CREDENTIALS_KEY and wire the channel credentials cipher (#6611)
* fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption
There was no at-rest encryption key. ChannelCredentialCipher had no
production caller: the channel service, the scheduled-task notification
outbox and the channel-connections router all built
ChannelConnectionRepository without a cipher, so store_credentials refused
(RuntimeError) and get_credentials always returned None.
- New harness module deerflow.config.credentials_key: env-only
DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts,
all decrypt via MultiFernet), CredentialsCipher with encrypt_text /
decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1:
values still readable, and get_credentials_cipher() as the process-wide
entry point. Malformed keys are refused without echoing key material.
Without the variable a single instance generates
{base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET.
- New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL
create (0600) plus read-back with a bounded settle window, shared by
.credentials_key, .jwt_secret (fixes two replicas cold-starting on a
shared volume keeping different JWT secrets) and the managed-model key.
- ChannelCredentialCipher now builds on CredentialsCipher (same
constructors); the Gateway loads the cipher off the event loop at
startup when channel_connections is enabled and passes it to every
ChannelConnectionRepository. Undecryptable values stay "missing + warn".
- Startup gate: a malformed key stops startup in any topology; a declared
multi-instance deployment with channel_connections enabled and no key
is refused. Plain multi-worker shares the generated file and is allowed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh
- Helm: the app Secret generates a Fernet key (randBytes 32 translated to
the urlsafe alphabet) and preserves it across upgrades via lookup like
AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef
so a user-managed existingAppSecret without it still boots (the Gateway
gate refuses only multi-instance deployments that store credentials).
README, NOTES and values document backup, rotation and existingAppSecret.
- scripts/deploy.sh resolves the key shell -> .env -> persisted file ->
generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600,
noclobber/O_EXCL with read-back): the same file the Gateway generates in
its runtime home, so either side's key is the one both keep.
- Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway.
- .env.example entry with a generation hint; README and CHANGELOG notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): make abandoned secret-file replacement single-winner
A rename is last-writer-wins, so replicas that restart together and all
find the same abandoned empty secret file could each replace it and keep
a different value. The process that exclusively creates
<name>.replacing now owns the replacement, re-checks the file is still
empty, and publishes by renaming the claim over it; peers wait for that
value, bounded so a crashed replacer cannot block startup forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): name the setting that turns off a credentials-key consumer
The multi-instance refusal ended with "or disable
channel_connections.enabled=true"; it now says "or set
channel_connections.enabled=false".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(security): never expose a secret file before its value is written
O_EXCL published the name before its content, so a creator that stalled
mid-write left an empty file a peer replaced after the settle window; the
resumed creator then returned a value nobody persisted, or deleted the
peer's valid file when its own write failed. Creators now write and sync
a private temporary file and hard-link it to the name, which fails if
the name exists; a failed creator only removes its own temporary file.
Filesystems without hard links publish through the same single-winner
<name>.replacing claim that replaces abandoned empty files, so creation
and replacement are serialized there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(models): recover an empty managed-model key when no catalog exists
An interrupted first save could leave an empty key file; every later save
then passed empty bytes to Fernet and the endpoint returned 503 until the
file was removed by hand. An empty key with no catalog encrypted nothing,
so it now goes through the secret-file recovery path; next to an existing
catalog it is still refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start
scripts/docker.sh runs Compose from docker/ without --env-file, so the dev
gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry
interpolated to empty and overrode the key the checkout .env supplied via
env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
xiaodu55
|
e3f21ce105
|
test: fix two Windows-host failures (subprocess encoding, budget-boundary epsilon) (#6627)
* test: fix two Windows-host failures (subprocess encoding, budget-boundary epsilon) - test_env_file: pin PYTHONIOENCODING=utf-8 in the stripped child env; without it the child writes stderr in the host's ANSI codepage while the UTF-8-mode parent decodes it, the reader thread dies on the first non-ASCII byte, and stderr comes back None; - test_jina_retries: give the budget-cap assertion a 1e-9 tolerance — deadline = now + budget rounds, so the cap lands a few ulps above the exact 0.05 (observed 0.05000000000291038). Fixes #6626 * test: pin both sides of the capture and use an uptime-proof budget tolerance - test_env_file: the parent's capture decodes with the host locale when pytest runs without the Makefile's UTF-8 pins, so the child-side pin alone mirrored the crash rather than eliminating it. Pin the capture explicitly (encoding=utf-8, errors=replace, per the test_local_sandbox_encoding.py precedent) and carry both suite pins in the child env (PYTHONUTF8=1 + PYTHONIOENCODING=utf-8). - test_jina_retries: the float error scales with loop-time magnitude (host uptime), not with 0.05; a 1e-9 absolute slack leaves a ~389-day- uptime flake window. Use a relative tolerance instead. --------- Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com> |
||
|
haochen
|
8377aa3d39
|
fix(support-bundle): retain diagnostics when dotenv cannot be read (#6536)
Co-authored-by: Zhangchenhao-111 <285911435+Zhangchenhao-111@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
ordinary being
|
737492d4e1
|
fix(runtime): drain staged-terminal runs on shutdown before checkpointer teardown (#6620)
* fix(runtime): drain staged-terminal runs on shutdown before checkpointer teardown RunManager.shutdown() selected only pending/running records, but with an event store a run stages its terminal status in memory and keeps finalizing: journal flush, delivery receipt, workspace scan, the duration checkpoint write and the deferred terminal commit, all while terminal_commit_pending is set. A Gateway shutdown landing in that window did not await the run, so langgraph_runtime could close the checkpointer before the duration write landed - the #3373 class the drain exists to prevent. Drain those records too, using the existing _awaits_terminal_commit state that _renew_leases already relies on; await them without cancelling (a cancel would skip their terminal tail) and never overwrite their staged terminal status with interrupted. * fix(runtime): surface shutdown terminal finalizer failures Cover timed-out staged terminal runs for both commit barriers and terminal outcomes. Log failed finalizers with their run ID, staged status, and traceback while preserving status and completing the drain. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Jason Cat
|
f623e64e07
|
fix(acp): resolve a bare agent command through PATH before spawning (#6598)
* fix(acp): resolve a bare agent command through PATH before spawning `invoke_acp_agent` handed `acp_agents.<name>.command` straight to the ACP SDK, which spawns it with `asyncio.create_subprocess_exec`. That API applies no `PATHEXT` lookup on Windows, so a bare npm launcher (`npx`, `mcode`) raised `FileNotFoundError` although its `.cmd` shim was on `PATH`. The documented `command: npx` and `command: mcode` shapes could not start, and the tool reported "was not found on PATH" for an installed launcher. Resolve the configured command with `shutil.which` before spawning — the way the MCP Python SDK normalizes stdio server commands — and keep the configured value when nothing matches, so the existing install guidance still fires. Fixes #6597 * fix(acp): resolve the ACP agent command off the event loop `_resolve_agent_command` calls `shutil.which`, which stats the filesystem. Calling it inline in the async `_invoke` tripped the strict BlockBuster gate (`BlockingError: Blocking call to os.access`) for every command that resolves, and would block the Gateway event loop in production. Offload the lookup with `asyncio.to_thread`, matching the neighbouring `_get_work_dir`, `_build_acp_mcp_servers`, and `_format_invocation_error` calls. The blocking-io anchor configured `command="test-agent"`, a name that resolves nowhere, so `shutil.which` short-circuited before `os.access` and the gate never covered the new call. Point it at `sys.executable` so the lookup is really exercised, and cover the unresolvable-guidance branch end-to-end with a new test. Refs #6597 * fix(acp): keep the configured command in ACP remediation text Review follow-ups on the PATH resolution: - Only bare names are resolved. A configured path is interpreted by the spawn against the per-thread ACP workspace, so resolving it against the Gateway's working directory could pick a different file or hand the spawn a path that then fails. - `_format_invocation_error` receives `agent_config.command` again. A command can resolve on PATH and still fail to launch (a broken shebang, for example), and reporting the resolved path made "was not found on PATH" false and dropped the `codex-acp` hint. - The blocking-io anchor uses a bare name backed by a shim on a controlled PATH instead of `sys.executable`, which the bare-name scoping no longer resolves. Tests: the pass-through assertion in `test_invoke_acp_agent_uses_fixed_acp_workspace` pins PATH, and new coverage covers the bare-name scoping and the spawn-failure message. Refs #6597 * docs(acp): correct the path-resolution rationale A configured path is resolved by the spawn in its own process, so the pass-through reason must not claim it is resolved against the ACP workspace: that only holds on POSIX, while Windows resolves a relative executable against the calling process's directory. State the platform-neutral reason instead. Refs #6597 * fix(acp): resolve the launcher against the agent's own PATH The ACP SDK merges acp_agents.<name>.env over its inherited environment at spawn time, so a configured PATH, not the Gateway's, decides which launcher the agent would find. Resolve a bare command against that effective PATH, and use the Gateway's only when the agent does not override it. Return a successful lookup as an absolute path: a relative PATH entry makes shutil.which yield a result relative to the Gateway's cwd, while the spawn runs with cwd set to the ACP workspace, where that path resolves elsewhere or not at all. Regressions: a real-subprocess test with same-named launchers in the Gateway and agent PATHs (asserting which one ran), and a real-subprocess test whose PATH holds a relative directory while the ACP workspace is elsewhere. Both are red on the pre-fix code. |
||
|
yang rui
|
a333ac3189
|
feat(search): add Webz news search provider and setup integration (#5880)
* feat(search): add Webz news search provider and setup integration * fix(webz): honor caller result limits and clarify page counts * fix(frontend): render Webz search source links * fix(frontend): skip malformed search source entries |
||
|
yangfucheng123
|
3b8141aa43
|
fix(community): Fail closed on delegated fetch backends to close SSRF gap (#6531)
* Fail closed on delegated fetch backends to close SSRF gap * fix(community): resolve delegated-fetch credentials from one config snapshot Address review feedback on the fail-closed SSRF guard: - firecrawl/fastcrw client construction no longer re-reads get_app_config(), so the API key and endpoint come from the same revision the endpoint screen validated (prevents cross-revision credential misrouting). - fix the browserless test double to accept the new two-argument _get_browserless_client signature. - document the check-only (DNS rebinding) caveat on validate_delegated_backend_url. - add a CHANGELOG upgrade note for the default self-hosted backend break. * fix(doctor): consolidate delegated fetch backend checks and deployment guidance --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
四野
|
b398921538
|
fix(skills): preserve Unicode separators in JSONL history (#6615) | ||
|
Jason Cat
|
8834a759b8
|
fix(groundroute): contain malformed web search and fetch payloads (#6603)
* fix(groundroute): contain malformed web search and fetch payloads web_search and web_fetch assumed the GroundRoute 200 body was an object with a `results` list of objects, and read it outside the request try/except. A body that arrived as a bare list, as an object with a non-list `results`, or with non-object entries raised AttributeError / KeyError out of the tool boundary, so the tool call failed with a stack trace instead of the provider's format error. Validate the payload before normalizing it, following the containment rule already applied to brave (#6550), serper, serply, sofya, tencent_wsa and unbrowse: a non-object payload, a non-list `results` container, or a non-empty list holding no objects returns the structured format error, non-object entries in an otherwise valid list are skipped in order, and missing, null or empty results keep returning "No results found". * fix(groundroute): coerce non-string fetch fields web_fetch sliced the fetched page's `content` (falling back to `snippet`) to the snippet limit without checking its type, so a 200 body whose result object held a number, boolean, or object there raised TypeError / KeyError out of the tool boundary -- the same escape the previous commit closed for the container shape. A null title also rendered as "# None". Coerce non-string content to text before truncating, and render a missing or non-string title as empty, matching the coercion sofya and unbrowse already apply to their fetch fields. * docs(groundroute): scope the falsy-title note to actual behavior The guidance said a non-string title renders as empty, but the code only empties falsy titles; a truthy non-string title still renders as its text form via the f-string. Say "falsy" instead. |
||
|
hataa
|
47d8ff13b2
|
feat(authz): account disable/enable across every authentication surface (#4063 gap 3) (#6566)
* feat(authz): account disable/enable across every authentication surface (#4063 gap 3) Vertical slice 1 of the user-lifecycle work: an operator can suspend an account and every authentication surface rejects it, while the row — credentials, role, OAuth linkage — is retained so re-enabling restores exactly what was suspended. - users.disabled (NOT NULL, server default 0) via migration 0036_user_disabled; User/UserResponse carry and expose it. - Enforcement at all four surfaces: password login (no credential comparison happens), JWT session resolution (401 ACCOUNT_DISABLED — sessions die at the next request), PAT resolution (the generic 401, no oracle), OAuth provisioning (403 with a contact-admin message; the link is kept so re-enable restores access). - SQLiteUserRepository.set_disabled: single-column, serialized write (same idiom as update_system_role) with the last-ACTIVE-admin guard — disabled admins cannot authenticate, so they cannot restore management access; re-enabling is always allowed. Field-scoping completed both ways: the credential writer mirrors and never writes disabled, so a stale password-change snapshot cannot re-enable a suspended account. - PATCH /api/v1/admin/users/{id} becomes a partial account update (system_role and/or disabled), each field routed to its field-scoped writer; empty bodies 422. Tests 11->18: route (disable/re-enable, last-active-admin 409, empty 422, combined update), real-repo (field isolation both directions, disabled-admin-not-active, guard inside the serialized write, re-enable always), login rejection with the CORRECT password (only the disabled gate can reject) plus resolver-visible flag. Mutations verified red: guard removed (M19), login gate removed (M20 — after hardening the test with a real hash so the gate is load-bearing). * fix(authz): portable boolean default, alternate-authenticator suspension, active-admin role guard Review round 1 on #6566 — three P1s: - [PG default] sa.text("0") is a raw integer expression PostgreSQL rejects as a BOOLEAN default, breaking startup migration and fresh create_all bootstraps alike. Both the migration and the ORM model now use sa.false()/false() — dialect-rendered, portable. - [alternate authenticators] WebSockets bypass AuthMiddleware, so a disabled user's still-valid cookie could open a NEW browser/stream connection and drive the retained session; the standalone LangGraph authenticate callback accepted the same suspended JWT. Both now route through a shared post-lookup validator (validate_resolved_session_user in deps.py — token_version + disabled in one place) that the HTTP dependency also uses, so a lifecycle change lands everywhere at once. - [active-admin role guard] update_system_role counted every admin row, so 'disable B, then demote self' — or a combined {system_role, disabled} self-update — could strand the deployment with zero administrators able to authenticate or re-enable anyone. The demotion guard now counts ACTIVE admins only (disabled admins cannot authenticate), under the same serialization boundary. Tests 18->22: shared-validator verdicts, WS admission after suspension (real token, real repo), LangGraph-callback admission after suspension, disable-then-demote-self + combined self-update both rejected (fake-repo route level and real-repo update_system_role level). Mutations verified red: helper's disabled check removed (M21 — three admission tests fail), active-admin predicate reverted to all-admins (M22 — repo-level strand scenario fails). 366 tests green on the affected surface. * fix(migrations): chain 0036_user_disabled after upstream 0037 The maintainer main-merge brought 0037_project_document_summaries, which branched from 0035 independently of this PR's 0036 — alembic now sees two heads and every engine bootstrap fails. Chain 0036 after 0037 (single head; revision numbers are labels, the graph is what matters). * fix(migrations): renumber user_disabled to 0038 and follow the chain-test convention Upstream landed its own 0036 (run_idempotency_request) with 0037 chained after it while this PR was in review — my 0036_user_disabled collided on the number and my interim chain-fix (0036 revises 0037) left the graph in a state upstream's chain tests reject (heads == [0037] hardcoded). - 0036_user_disabled -> 0038_user_disabled, revises 0037_project_document_summaries: single head 0038. - Chain tests follow the repo convention (each migration PR appends the new head): 0032/0035 gain the DOC_SUMMARIES link and CURRENT_HEAD = 0038; 0037's single-head assertion moves to CURRENT_HEAD with its own-revision link check unchanged. * fix(migrations): pin the remaining chain tests to the 0038_user_disabled head The 0027/0028, 0029 and 0034 chain tests still pinned the single head to 0037_project_document_summaries (and 0034 asserted the head's parent was 0036_run_idempotency_request), so the renumbered lifecycle migration broke two unit-test shards. Follow the chain-test convention: every pinned head moves to 0038_user_disabled and 0034 walks the chain through 0037. * fix(authz): enforce account suspension on the internal-auth dispatch surface Round-3 review finding: the internal-auth principal is synthesized from trusted headers without a users-row lookup, so a suspended user with an active channel binding kept driving the agent, and scheduler-owned runs (their fabricated in-process requests skip AuthMiddleware entirely) kept launching. Two gates now enforce the lifecycle there: - AuthMiddleware rejects owner-bound internal calls whose users row is disabled (401 account_disabled) at the single gate every internal HTTP call passes through. Auth-disabled mode is exempt (no suspendable accounts, and its owner header carries the synthetic identity), and owner ids without a users row stay allowed — a nonexistent account cannot be suspended. - start_run re-asserts it at run admission after the owner is resolved for attribution: the scheduler records the rejection as the occurrence's launch error and no run record is created. Also addresses the remaining round-3 nits: drop the inline token_version check duplicated by validate_resolved_session_user, correct the local_provider comment (login never surfaces ACCOUNT_DISABLED), refresh the stale 0038 docstring header, and restore the deps module caches in the two suspension tests instead of leaking a closed engine. * fix(auth): carry the disabled field on the frontend synthetic users UserResponse now always includes disabled, so the auth-disabled contract e2e (gateway /auth/me deep-equals the frontend synthetic user) failed without the field on AUTH_DISABLED_USER. Add disabled to both synthetic users and declare it optional on the zod schema — absent still parses as not disabled for pre-gap-3 backends. * fix(authz): log when the internal-owner suspension check is skipped The middleware gate's users-lookup failure path silently degraded to allow; keep the fail-open behavior but leave a WARNING with the exception, so a store that cannot be read is observable in production logs instead of invisible. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
NanPan
|
29d654e86d
|
fix(checkpoints): treat corrupt Redis history cache entries as misses (#6592)
* fix(checkpoint): treat malformed Redis history cache entries as misses * test(checkpoint): cover malformed Redis history cache entries * style(checkpoint): format corrupt cache regression test * test(checkpoint): cover non-dict cache entries and pinned Ruff formatting * fix(checkpoint): include cache key in malformed-entry diagnostics * docs: trim inherited agent guidance below size limit --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
stanGSA
|
f71999d99c
|
fix(frontend): export complete persisted conversation history (#6530)
* fix(frontend): export complete persisted conversation history * fix(frontend): preserve canonical export order across runs --------- Co-authored-by: stanGSA <225717802+stanGSA@users.noreply.github.com> |
||
|
xuan
|
f5156b6762
|
docs(zh): document the /agent channel commands in README_zh (#6593)
The Chinese command table skipped straight from /memory to /help, so the /agent list and /agent use <name> rows and the conversation-scoped agent-selection paragraph documented in README.md were missing entirely. Mirror them from the English README. |
||
|
Hyeonsang Cho
|
f8b419ab38
|
fix(scripts): bind local make dev and make start to loopback (#6587)
* fix(scripts): bind local make dev and make start to loopback serve.sh and backend/Makefile started the Gateway with --host 0.0.0.0, nginx.local.conf listened on every interface, and Next.js kept its all-interfaces default outside Windows, so a local stack on a LAN or VPN exposed ports 2026, 8001 and 3000, including /setup before the first admin existed. The Docker stack and the README already document a loopback-only default. Bind the Gateway and frontend to 127.0.0.1 and keep nginx on loopback unless BIND_HOST is set, the variable the Docker stack already honors. nginx-local-conf.sh renders a copy listening on BIND_HOST, and serve.sh resolves it before stopping anything so an invalid value cannot tear down a running stack. * docs(changelog): reference #6587 * docs(changelog): separate the loopback entry from the next bullet --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
yetuge
|
ab9920f25f
|
fix(sandbox): frame remote search records on LF only (#6595)
* fix(sandbox): frame remote search records on LF only
BoxLite, OpenSandbox and Tenki sliced find/grep output with str.splitlines(),
which also ends a line at a bare carriage return, form feed, vertical tab,
file/group/record separator, next-line, and U+2028/U+2029. Every one of those
is legal inside a Linux filename and inside matched text, and the shared
parser already documents the opposite contract ("Split on \"\n\" only:
splitlines() would also split on characters that are legal in Linux
filenames").
Effects on these three providers, both silent:
- A path such as notes\x0bdraft.txt is reported as two unrelated paths, so
glob() returns a path that does not exist (dropping the real one) and
grep() reports the match against that wrong path.
- A matched line containing U+2028 -- which occurs inside real JavaScript and
JSON string literals -- is truncated at that character.
Split on "\n" only, matching LocalSandbox, the AIO backend and E2B (#6573).
The colon half of #6573 is deliberately not included: it needs grep --null,
and these providers restrict their command to busybox-portable flags by
design. BusyBox grep rejects --null/-Z (it prints its usage line instead of
searching), so adding the flag here would break every grep on a BusyBox
image. Splitting the record from the right is not a substitute because
"path:12:needle: hello:world" is genuinely ambiguous without a NUL separator.
Tests: a cross-provider file pins the framing through each provider's own
transport seam (canned output, so it runs on any host) plus real-shell cases
for a separator-named file and separator-bearing matched text. 81 of them fail
on main and pass here; the provider suites are unchanged at 268 passed.
Verified with PYTHONUTF8=0.
Fixes #6594
* test(sandbox): make the real-shell fixtures POSIX-correct
Both real-shell tests were written against a Windows host, where _RS_POSIX
skips them, so neither had ever executed. On a POSIX host (macOS, and CI's
ubuntu/macOS runners, which do have sh/find/grep/head) both failed:
- The separator-named-file fixture emitted one octal escape per code point, so
U+2028 became `\20050` -- printf reads that as a bare 0x80 byte plus the
literal digits `50`, not U+2028. Dead `len(separator) == 1` branch removed;
the code point is now emitted as one octal escape per UTF-8 byte
(`\342\200\250`).
- The matched-text fixture used `printf '%s'`, which does not interpret
backslash escapes, and interpolated a payload whose own double quotes closed
the shell word; the file ended up as `const s = au2028b;\n` with no real
U+2028. The escape now sits in printf's format string, with the octals
assembled from the code point's UTF-8 bytes.
Verified by running the three cases directly on a POSIX-style root, which is
what the CI runners do: all three pass. The canned-output tests, which pin the
production change, are unchanged.
Also replaced the raw control characters in both changelog entries with
escape-style text (`notes\x0bdraft.txt`, `"a\u2028b"`, `"\n"`): a raw vertical
tab or U+2028 renders as nothing or a replacement glyph, and one code span had
a literal newline in it, which CommonMark collapses to a space and would have
read as `split on " " only`. Matches the #6520 entry's practice of naming the
characters.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
yetuge
|
42df8f3e5f
|
fix(integrations): decode lark-cli and npm output as UTF-8 (#6590)
* fix(integrations): decode lark-cli and npm output as UTF-8 Every text-mode subprocess.run in lark_cli.py that captures lark-cli or npm output omits encoding, so the bytes are decoded with the host locale. Both write UTF-8 to a pipe: the @larksuite/cli npm package's bin is a Node wrapper around a native binary that emits UTF-8 (lark-cli --help writes U+2014 as E2 80 94), and npm output is UTF-8 too. On a host whose ANSI code page is not UTF-8 (cp936, cp1252) the locale decode silently mangles every non-ASCII field -- auth status --json returns a garbled userName, so the status card shows mojibake -- and an undecodable byte raises inside subprocess's reader thread, which is not re-raised: stdout comes back as None or "", so a healthy CLI is reported as unavailable and a successful run raises "lark-cli did not return JSON output." Decode these captures as UTF-8, matching #5440 (PowerShell), #5904 (AIO sandbox docker CLI) and #6548 (Git Bash output in LocalSandbox). The whoami /user capture stays on the locale: whoami is a native tool that writes the host code page, the same UTF-8/native split #6548 drew. Tests drive the real functions with a stand-in CLI emitting UTF-8 CJK JSON and pin the pre-fix decode path, so they fail on main and pass here on any host locale. Fixes #6589 * test(lark): isolate the paths singleton in the encoding suite The new tests reached ensure_lark_cli_credential_tree through the global get_paths() singleton, so every run created and hardened a real <cwd>/.deer-flow/users/alice/integrations/lark-cli/{config,data} tree in whatever directory pytest ran from -- the same runtime home a live deployment keeps real per-user Lark credentials in. An autouse fixture now points the singleton at tmp_path, matching what test_lark_cli_integration.py does. Red/green is unchanged: reverting only lark_cli.py still fails the two assertions that pin the decode, and the suite leaves no .deer-flow behind. * test(lark): make the version probe pin the decode, drop the inert locale patch Two test-rigor notes from review: - test_probe_lark_cli_at_path only asserted available/error, and the probe returns available=True with any non-empty output and a zero exit code, so the test stayed green even with the encoding pin removed from _probe_lark_cli_at_path. The stand-in now answers --version with a non-ASCII payload and the test names the expected text, which fails on the unpinned call site (verified) and passes with the pin. - The locale.getencoding patch was inert: the subprocess.run wrapper injects the codec for unpinned text-mode calls and lark_cli never consults the locale itself, so nothing reached the real locale-decode path. Dropped it, along with the now-unused import, rather than implying a redirection that does not happen. The Windows shim forwards its arguments (%*) so the stand-in sees --version. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
db9b0b4bf0
|
fix(release): reject a stale backend/uv.lock in the version gate (#6588)
* fix(release): reject a stale backend/uv.lock in the version gate verify_versions.sh compared only Chart.yaml, pyproject.toml and package.json, so bumping those three by hand passed the v* gate while backend/uv.lock still recorded the previous root package version. The backend image installs with uv sync --locked, so the chart and the other images published while the backend image failed to build. Run uv lock --check in backend/ as part of the gate, letting uv own the PEP 440 normalization, and fail when uv is missing. The reusable verify-versions workflow installs the uv version the backend image pins. * docs(changelog): reference #6588 * fix(release): explain an unparsed lock version and stop blaming the lock for every uv failure Print a fallback when the lock's root entry is not found, so the job log does not show an empty field, and word the uv lock --check annotation so a network error or uv crash is not reported as a stale lock. |
||
|
Nan Gao
|
6c4c74a504
|
fix(channels): share chat-to-thread bindings through the database (#6558)
* fix(channels): share chat-to-thread bindings through the database ChannelManager kept its IM chat-to-thread bindings (channel_name:chat_id[:topic_id] -> thread_id) in a per-process JSON file, channels/store.json, loaded once and rewritten whole on every change. With several Gateway processes a binding created on one replica was invisible to the others (the next message for the same chat opened a second thread) and concurrent writers clobbered each other's file. - New channel_thread_bindings table (migration 0036_channel_thread_bindings) and harness package deerflow.persistence.channel_thread_bindings: key PK (the legacy composite string), channel_name, chat_id, nullable topic_id, thread_id, user_id, epoch created_at/updated_at; index (channel_name, chat_id). Dialect upsert keeps created_at; prefix remove and batched ON CONFLICT DO NOTHING import. - app.channels.store.ChannelStore is an async protocol: SqlChannelStore for sqlite/postgres, JsonChannelStore (file I/O via asyncio.to_thread) for database.backend: memory. resolve_channel_store picks at ChannelService construction; ChannelService.start() imports an existing store.json once into an empty table and renames it store.json.migrated (a concurrently importing peer neither duplicates nor clobbers; a populated table leaves the file alone). - Callers updated: manager, connection_identity.lookup_thread_id, Feishu (send() awaits the mapping writer; the synchronous lark callback bridges its two store calls to the Gateway loop with a bounded wait), Discord (channels_dir). - Head pins in the 0027_0028/0029/0032/0034/0035 migration tests point at the new head; docs (channels and migrations AGENTS.md, CONFIGURATION.md, README) and CHANGELOG updated. * docs(changelog): link the channel thread bindings entry to #6558 * fix(channels): reject only the Gateway loop in the Feishu store bridge lark-oapi 1.5.5 invokes the message callback synchronously from its async frame handler, on the private event loop that FeishuChannel._run_ws starts for the SDK thread. The bridge's guard treated any running loop as unsafe, so every real inbound message skipped the stored-topic lookup and the pending-mapping restore: a reply whose parent card was mapped but whose root was not fell back to the unknown root, and could consume an unrelated pending clarification. Only a wait issued from the Gateway loop itself (the loop that executes the submitted coroutine) is refused now; the SDK's own loop may block for the bounded wait, as that thread blocked on the JSON rewrite before. Tests drive _on_message from a worker thread running its own loop, the production shape, for both the lookup and the write path, and pin the Gateway-loop refusal. * fix(channels): defer unresolved Feishu lookups and harden the store.json import Review follow-ups on the chat-to-thread binding store (#6558): - Import: a legacy entry whose key, channel_name, chat_id, topic_id, thread_id or user_id exceeds its column width is skipped and counted with the malformed ones instead of failing the whole batch on PostgreSQL (SQLite silently accepted the same row, so the backends diverged on one file). The corrupt-file warning now says what each caller does: the JSON store starts fresh, the import is skipped and the file kept. - Feishu bridge: store calls run on the Gateway loop under a 3s asyncio.wait_for (the SDK loop also services ws keepalives; the old 10s bound could stall it per message), and the lark-thread wait is that budget plus 1s. A deadline or store error is StoreLookupUnresolved, never a miss: _on_message no longer routes such a reply onto its unknown root (which created a second thread and later overwrote the parent alias). The message joins a per-chat FIFO drained on the Gateway loop with backoff for up to 30s, then is dropped with a warning. _on_message is split into parse / resolve / dispatch so the deferred path dispatches the same way; later messages for the chat queue behind the deferred one. * fix(tests): expect follow_redirects in the Jina client kwargs after #5808 #5808 made JinaClient pass follow_redirects=True to httpx.AsyncClient and updated test_jina_client.py, but test_jina_response_limit.py and test_jina_retry_after.py still pin the exact client kwargs to proxy and trust_env, so every backend-unit-tests shard that lands on them fails on main (the backend unit-test workflow only runs on pull requests, which is why the push to main did not surface it). Update the two expectations to include the new keyword. (cherry picked from commit d5aa23156ce19ce36d95df03953442f83ae6d050) * Revert "fix(tests): expect follow_redirects in the Jina client kwargs after #5808" This reverts commit 5bee00bfb09c65a30f77fb4df47a1f82c47ecef7. * fix(channels): bound deferred Feishu routing by the inbound queue capacity Review follow-up on #6558: a store outage let every subsequent message for a chat join the deferred queue with no capacity check (the bus reservation was only taken after routing succeeded), and each queued message started a fresh 30s retry budget when it reached the head, so a burst could be retained for N x 30s and grow without bound. - Intake is reserved BEFORE routing for every store-backed message (bus.reserve_inbound is thread-safe and already used from SDK threads), so deferred and live messages share the one configured bound, channels.inbound_queue_maxsize; a message that cannot be admitted is dropped at once with a warning, never queued or routed as a miss. - The reservation travels with the message into dispatch (committed by _prepare_inbound), is released on the batch path (the flush reserves at publish time), on drop after the budget, on a failed deferral, on dispatch failure and for every leftover when the drain task is cancelled by stop(). - The retry budget runs from the message's arrival: a queued message whose budget is spent when it reaches the head is dropped without a lookup. - The bounded resolver returns the unresolved outcome instead of raising it through the tracked task, which logged an ERROR for an expected, handled store timeout. - Upstream's new blocking-io test constructs the store by its old class name; it now uses JsonChannelStore. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
spud
|
cd15687284
|
fix(mcp): reconcile committed Gateway config mutations (#6567)
* fix(mcp): reconcile committed Gateway config mutations Install selective MCP binding fences inside the Gateway config commit boundary and finish teardown after releasing config locks. Reject frozen durable-task server changes before persistence. Preserve drained cancellation semantics and cover delete/re-add, durable-only bindings, multi-server diffs, shared writers, and post-write reconciliation failure. * test(mcp): pin committed handoff boundaries Prove the fallback fence is installed before releasing the config lock, a committed writer cannot swallow an earlier shared reset, delete/re-add advances the applied baseline and cache generation, and retries after a post-write 500 have explicit conflict or missing semantics. * fix(mcp): fence committed config after reload failure Treat a post-write reload failure as a committed handoff failure so local MCP state is detached before releasing the config lock and teardown remains outside it. * fix(mcp): always finish committed pool teardown Run finish_mcp_reconciliation from a post-lock try/finally so response or logging failures cannot strand a retired full-reset pool. * test(mcp): fence cold-start binding race Pin the in-flight discovery path that keeps the committed fast path from admitting a late old-fingerprint binding. * docs(mcp): document committed teardown guarantee * fix(mcp): sanitize committed response errors Keep post-write response conversion failures behind a fixed HTTP detail while preserving the retired-pool teardown guarantee. * fix(mcp): avoid duplicate committed reconciliation error logs --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
sjr666666
|
f4d4f776f6
|
fix(skills): record the content actually overwritten in custom-skill history (#6153)
* fix(skills): record the content actually overwritten in custom-skill history Scope: - backend/app/gateway/routers/skills.py: move the prev_content read of update_custom_skill and the current-content read of rollback_custom_skill into their drained persist tails (read -> write -> append_history in one worker thread, no awaits in between). The blocked-rollback branch now builds its informational entry separately. No behavior contract changes: status codes, cancellation draining, and prompt-cache refresh order are untouched. - backend/tests/test_skills_custom_router.py: two regression tests that inject a concurrent write during the scan phase and assert the history entry's prev_content equals the revision actually overwritten on disk. Why: prev_content was read before the security scans, so a concurrent revision landing between that read and the persisted write was overwritten on disk but absent from the history chain, making it unreachable by a later rollback and desynchronizing the audit record. Tests: uv run pytest tests/test_skills_custom_router.py --basetemp=... -> 36/36 passed (34 existing + 2 new); ruff check and ruff format --check clean on both files. Python 3.12.13, Windows 11. * fix(skills): serialize custom-skill mutations and pin the concurrency regression to the real window - Add a module-level mutation lock shared by the edit, delete, and rollback paths, covering the full read -> write -> history-append sequence so a concurrent PUT cannot land between a request's predecessor read and its write (previously the projection lock started inside write_custom_skill, leaving that window open). - Replace the two regression tests: they now pause the first mutation inside its storage write after the predecessor read (deterministic gate), complete a second real HTTP PUT inside that window, and assert the history chain stays replayable. Verified both tests fail against the base implementation (2 failed) and pass with the lock (2 passed); full file 36 passed, ruff check + format clean. * fix(skills): scope the custom-skill mutation lock to the owning user The lock serializing the read -> write -> append sequence was one process-wide lock, and that sequence wraps write_custom_skill's entire user projection rebuild. A user waiting on their own projection lock, or copying a large or slow linked package, therefore stalled every other user's edit, rollback, and delete of unrelated custom skills, even though their skill trees, histories, and projection roots are disjoint. The base storage only serialized those mutations per user projection. Key the lock on the canonical owning-user storage root instead, so one user's mutations remain a single critical section — including across separately constructed storage objects, which a hot-reloaded config or a later request creates — while two users never share it. The read -> write -> append sequence still cannot interleave within a user, and the lock is still acquired and released inside the worker thread's own with block, so caller cancellation cannot strand it. * fix(skills): serialize custom history across gateway processes --------- Co-authored-by: sjr666666 <sjr666666@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Heran Liu
|
be69d993c1
|
fix(e2b): preserve special characters in search results (#6573)
* fix(e2b): preserve colon-containing paths in grep results * fix(e2b): preserve non-LF separators in search results --------- Co-authored-by: Heran Liu <210811826+heranliu@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
creed
|
be62fff070
|
feat(serper): add opt-in bounded transient retries (#6456)
* feat(serper): add opt-in bounded transient retries Signed-off-by: 97three <2212371308@qq.com> * fix(serper): validate HTTP retry dates precisely Signed-off-by: 97three <2212371308@qq.com> * fix(serper): address retry review feedback --------- Signed-off-by: 97three <2212371308@qq.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
kuku
|
2d55c75870
|
fix(agents): preserve complete remote artifact URLs (#6574)
* fix(agents): preserve complete remote artifact URLs * fix(agents): handle CJK punctuation and mixed-case artifact suffixes * fix(agents): preserve balanced CJK artifact references --------- Co-authored-by: kuku <186267668+xbzz1018@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
四野
|
90aa5d4302
|
fix(brave): contain malformed web search results (#6550)
* fix(brave): contain malformed web search results * fix(brave): log malformed responses and isolate test transport --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Storm
|
061a7dfc14
|
feat(channels): select the model per conversation via /model (#6534)
* feat(channels): select the model per conversation via /model IM channels could list models with /models but had no way to choose one: the model for a conversation came only from channel/session config or the server default, so a user who needed a stronger or cheaper model for one task had to reconfigure the whole channel or move to the web UI. - /model <name> pins a model to the current conversation, validated against the caller-visible /api/models list (list membership doubles as the authorization check), persisted in thread metadata so it survives a Gateway restart and applies from the next message in place — unlike /agent use, which must start a new conversation. - The pin is written to all three runtime carriers (run context, configurable, context) after the session layers merge, so it wins over channel/session configured models; like any explicitly requested model it also takes precedence over a Custom Agent's configured model. - /model (no args) reports the effective model and its source through one resolver that mirrors the run merge order (default -> channel -> user, context carrier over configurable, both session-config carriers scanned); /model default clears the pin; /models names the pinned model in a footer. - Pin lookups degrade instead of failing messages: the status query reports "Current model: unknown" on a lookup failure (matching the /models footer), and the reuse-thread preload leaves the in-process cache untouched so the next turn retries a transient threads.get. - Reserve "model" in the slash-skill reserved-name contract (backend, contract fixture, frontend), register the Telegram handler, and document the command in /help, README, and README_zh. * fix(channels): fence model pin cache and mirror runtime model resolution Review follow-up for /model: - Serialize cold pin loads with persist+cache publication under a per-manager lock, so a slow threads.get that captured pre-pin metadata can never overwrite a concurrently committed selection (commands are not run-serialized). - _resolve_configured_model_name now replicates the run path's shallow session merges and applies the real runtime precedence — config.context (preserved by build_run_config, unbeatable by the setdefault-filled body context) > top-level context > configurable — instead of scanning per-layer carriers; a later layer replacing a nested carrier wholesale no longer leaks a stale value into the status reply. - /api/models resolves the trusted bound owner for internal channel calls and filters the list with the owner's role, so /model selection gates on the same principal that run admission admits for that account. Tests: pin-write-vs-in-flight-load race, status compared against the real merged runtime config (parametrized across carriers), nested-carrier replacement, and owner-role filtering with the real RBAC provider; each verified red by reverting exactly its guard. * fix(channels): share model resolvers with the run path and harden the pin lifecycle The /model status reply re-implemented the session-layer merge and the Gateway carrier precedence, and it ignored the custom-agent model fallback — on a thread pinned to a custom agent whose config sets ``model:`` it reported "server default" while the run actually used the agent's model. Review also found the pin lock spanned a network call and a stale pin bricked its conversation behind the generic error fallback. - Extract ``_select_carried_model_name`` as the single carrier-precedence rule; the status reply reuses the run path's exact ``_merge_dicts`` layer shapes, so a future layering change has one place to touch. - Extract ``_resolve_message_assistant_id`` / ``_resolve_assistant_id`` and share the assistant precedence chain between the run path and the status reply. - Fall back to the resolved custom agent's configured ``model`` when no session layer carries one, mirroring the agent factory's ``_resolve_model_name(requested or agent_model_name)``; the reply now distinguishes "(from channel configuration)" and "(from agent configuration)". - Run the cold ``threads.get`` in ``_load_thread_model`` outside ``_model_pin_lock``; only cache publication takes the lock, with a post-GET re-check so an in-flight /model write still wins over the older snapshot. - Detect the run-admission allowlist rejection in ``_handle_message`` and recover: clear the stale pin (durable write plus cache under the lock) and reply with the reset and the /model recovery instead of the generic internal-error fallback. - Document /model and the ``channel_model_name`` pin semantics in backend/app/channels/AGENTS.md. * fix(channels): fence stale-pin recovery and cover streaming admission failures Review round 4 found the stale-pin recovery itself unsafe and incomplete: - Recovery cleared whatever pin was current when the delayed admission rejection arrived, so a rejection for an old pin could erase a replacement selection committed mid-flight (and tell the user the new model was unavailable). ``_stale_pin_recovery`` now takes the failed run's own selection (``expected_pin``, read from its run context) and clears only while that selection is still current under the lock; a replaced or already-reset selection gets a non-destructive reply instead. - Feishu/Telegram/DingTalk streaming turns never reached the recovery hook because ``_handle_streaming_chat`` swallows start_run's 400 into its terminal-error reply. The recovery now runs in the streaming terminal-error path too, delivered as the single final outbound with the same dedupe cleanup; blocking, streaming, and fire-and-forget run-creation paths all handle the rejection at the site that knows the submitted selection. * docs(channels): fit the /model entry within the agent-guidance size budget The base file sits one byte under the AGENTS.md hard limit, so documenting /model requires net-zero growth: keep the command-inventory addition and a compact pin-semantics sentence, and losslessly tighten the surrounding prose (the file was already at the limit). * fix(channels): apply the review suggestions on /model replies Three non-blocking findings from the latest review: - The /models pin footer decorated the failure string too, advertising a pin next to an error that lists nothing. The command now fetches the listing through ``_fetch_gateway_json`` and appends the footer only on a successful fetch. - The /model status reply labeled every session-layer model "from channel configuration", misreporting user- and default-layer models. The label is now "session configuration" for every session layer. - Stale-pin recovery replies read as if the turn went through, but the run was rejected before starting and the triggering message never reached the agent. All recovery variants now end with an explicit "Your message was not processed — please send it again." * fix(channels): recover stale model pins on the follow-up drain path The drain is the fourth run-creation path and the only one without stale-pin recovery: a /model pin that left the allowlist while follow-ups were buffered made every drained runs.create fail admission, requeueing the batch into the same 400 on every cycle with no user-facing recovery — contradicting the documented "stale pins self-clear" invariant. - Compute expected_pin from the drain's run context (mirroring _handle_chat_on_thread, initialized None since _resolve_run_params is itself inside the guarded try) and run _stale_pin_recovery on drain failure before requeueing; the pin self-clears and the requeued batch succeeds on the next cycle. - Pass the caller-authoritative thread_id into _stale_pin_recovery: every call site already holds it, so recovery never re-derives it from the message — a store-mapping rewrite between dispatch and recovery can never clear the wrong conversation's pin, and the lookup's failure mode (and its guard) disappears entirely. - Return StalePinRecovery(text, resolved) so requeueing callers can tell a cleared pin from a repeat failure: the drain logs INFO when the next cycle is free of this rejection and WARNING when recovery could not remove it (persist failed, or the selection came from session configuration, which still carries it) — no silent infinite requeue loop. Drain logs carry structured facts only: the reply text ends with a "please resend" note that would double-process buffered messages retried automatically (GitHub, the only buffering channel, sends log-only anyway). - Reword the config-carried recovery reply: with no pin held, the rejected selection came from session configuration — name that source and the fix instead of claiming a default takeover that would not happen. - Correct the _load_thread_model lock docstring (writes deliberately hold _model_pin_lock across threads.update) and read the agent config for /model status off the event loop via asyncio.to_thread, with a Blockbuster-gated blocking_io regression test. - Pin the under-lock re-check in _stale_pin_recovery with a concurrency test (a /model committed during the lock-wait window wins; recovery persists nothing), mutation-verified — the drain is the only caller running on a background task without the serialized-run lock, so it is the path most exposed to that window. * Revert "fix(channels): recover stale model pins on the follow-up drain path" This reverts commit 0b4652c689afb56791e5e670901c97a4bf44a156. The drain recovery work moves to a dedicated PR stacked on this branch, keeping #6534 scoped to the /model feature itself. * fix(channels): recover stale model pins on the follow-up drain path The drain is the fourth run-creation path and the only one without stale-pin recovery: a /model pin that left the allowlist while follow-ups were buffered made every drained runs.create fail admission, requeueing the batch into the same 400 on every cycle with no user-facing recovery — contradicting the documented "stale pins self-clear" invariant. - Compute expected_pin from the drain's run context (mirroring _handle_chat_on_thread, initialized None since _resolve_run_params is itself inside the guarded try) and run _stale_pin_recovery on drain failure before requeueing; the pin self-clears and the requeued batch succeeds on the next cycle. - Pass the caller-authoritative thread_id into _stale_pin_recovery: every call site already holds it, so recovery never re-derives it from the message — a store-mapping rewrite between dispatch and recovery can never clear the wrong conversation's pin, and the lookup's failure mode (and its guard) disappears entirely. - Return StalePinRecovery(text, resolved) so requeueing callers can tell a cleared pin from a repeat failure: the drain logs INFO when the next cycle is free of this rejection and WARNING when recovery could not remove it (persist failed, or the selection came from session configuration, which still carries it) — no silent infinite requeue loop. Drain logs carry structured facts only: the reply text ends with a "please resend" note that would double-process buffered messages retried automatically (GitHub, the only buffering channel, sends log-only anyway). - Reword the config-carried recovery reply: with no pin held, the rejected selection came from session configuration — name that source and the fix instead of claiming a default takeover that would not happen. - Correct the _load_thread_model lock docstring (writes deliberately hold _model_pin_lock across threads.update) and read the agent config for /model status off the event loop via asyncio.to_thread, with a Blockbuster-gated blocking_io regression test. - Pin the under-lock re-check in _stale_pin_recovery with a concurrency test (a /model committed during the lock-wait window wins; recovery persists nothing), mutation-verified — the drain is the only caller running on a background task without the serialized-run lock, so it is the path most exposed to that window. (cherry picked from commit 0b4652c689afb56791e5e670901c97a4bf44a156) * fix(channels): apply the third review round on /model command replies - Delete the dead kind == "models" branch in _fetch_gateway: /models has rendered the listing inline since the pin-footer fix, so the helper's copy was unreachable and had already diverged (unguarded m["name"], no pin footer). The internal-auth-header test now drives the surviving /memory kind. - Guard the inline /models listing comprehension like the _handle_model_command sibling (isinstance(m, Mapping) and isinstance(m.get("name"), str)), so a malformed payload entry can no longer turn the command into the generic internal-error reply; regression: test_models_command_skips_malformed_entries. - Drop the unused user parameter from _resolve_bound_owner_for_model_list: the resolved owner comes solely from the trusted header, and the parameter implied the caller's own principal shaped the filter. * fix(channels): apply the fourth review round on the /model lifecycle - Preserve fresh model pins across cache eviction: a cold threads.get racing a successful /model write could still publish its stale snapshot after clear-on-pressure evicted the fresh entry — the in-cache re-check cannot see an eviction. The model-pin cache now carries a write generation; a load whose generation moved during the GET refetches once instead of publishing over the committed selection. Regression: test_fresh_model_pin_survives_cache_eviction_during_cold_load (blocked GET, write, eviction, then the refetch wins), mutation-verified. - Resolve context-carried agents before reporting their model: /model status now mirrors the run path's effective-agent rule — explicit message/thread choice overwrites every carrier, then the merged carriers' agent_name (new _select_carried_agent_name, sharing the model-name precedence via _select_carried_value), then the session-layer assistant_id that only setdefaults into context. A session with context.agent_name=coder now reports coder-model instead of server default, and assistant_id=writer alongside context.agent_name=coder reports coder-model instead of writer-model. Regressions: context-carried, context-beats-assistant_id, and thread-pin-beats-context status tests. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
16877b92fc
|
fix(skills): confine review_skill_package local targets to the caller (#6580)
* fix(skills): confine review_skill_package local targets to the caller Local path targets were allowed anywhere under the Gateway cwd or /tmp. Every documented deployment keeps DEER_FLOW_HOME under the cwd, so a user who knew another user's id could pass .deer-flow/users/<id>/skills/custom/<skill> and get that skill's SKILL.md and references/ back in the model response. The tool is always available, so skill tool policy could not remove it. Local targets are now confined to the configured skills root and the caller's own user directory (resolved from the runtime user id through make_safe_user_id, like the user-scoped skill storage). skill:// and inline:// targets are unchanged. * test(skills): pin dot-dot and sandbox-planted symlink review targets * docs(changelog): reference #6580 * fix(skills): ignore a review skills root that contains per-user directories A skills root configured at or above DEER_FLOW_HOME would re-admit every user's directory as a local review target. Drop such a root and log a warning instead of trusting it silently. * test(skills): use the platform temp dir for the shared-temp review case A hardcoded /tmp does not exist on Windows dev hosts. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
ZJPex
|
4cb8e3096b
|
feat(mcp): filter background tasks by status (#6527)
* feat(mcp): filter background tasks by status * feat(mcp): expose task list filters in Gateway * docs(mcp): describe background task query filters --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
d3887531e9
|
docs: list the summarize event tag and scope the extension example's claims (#6582)
* docs: list the summarize event tag and scope the extension example's claims RUN_EVENT_STREAM.md listed seven middleware tags, but the catalog and the run event stream contract carry eight; `summarize` is emitted by SummarizationMiddleware for every successful summary result. Add the tag, describe the event, and pin the documented list to MIDDLEWARE_EVENT_TAGS so the next tag cannot drift silently. summarization.md now lists all eight `changes` fields of that event. The extension example said it showed all five (README, pyproject, module docstrings, test name) or every contribution kind, while ExtensionRegistry has eight. Say it demonstrates five, name the three it does not, point to the observers guide and the bookmarks plugin, and assert in the example's test that the two observer kinds stay unregistered. * docs(changelog): reference #6582 * docs: clarify extension example package layout scope --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Daoyuan Li
|
be44c36b2b
|
fix(events): preserve JSONL sequences after run deletion and restart (#6546)
* fix(events): preserve JSONL sequences after run deletion and restart * fix(events): match JSONL watermark permissions to run files --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Storm
|
b9a75b869d
|
fix(sandbox): attest lark broker mode per Pod and quarantine uncertain shell outcomes (#6436)
* fix(sandbox): attest lark broker mode per Pod and quarantine uncertain shell outcomes Why this change: Two fail-open paths in the AIO sandbox lifecycle. First, the lark broker capability probe collapsed "provisioner unreachable" into "broker not configured": one 1.5s timeout cached a negative for 300s, so new sandboxes were created without the broker sidecar while the per-user credential directories were still bind-mounted into the container, and existing Pods kept being reused without their actual mode ever being read. Second, an implicit-shell command that timed out or returned an unknown status only marked the current client object; warm-pool reclaim and rediscovery built a fresh client that re-entered a container whose shell state could not be proven clean, and nothing about that ambiguity survived a Gateway restart. What changed: - The capability cache stores confirmed observations only, keyed per provisioner endpoint/auth identity; an unknown answer (timeout, malformed payload) raises instead of selecting credential placement, while a confirmed broker requirement is retained through outages. One in-flight probe per identity shares success and failure, and a Pod-attested mode contradicting the cache forces one shared refresh. - Execution uses the mode attested for the actual sandbox: SandboxInfo.lark_cli_broker flows from the provisioner's atomic Pod observation into the client, so a probe outage delays drift detection but never denies reuse of an attested runtime, and never trusts an unattested one. - The provisioner reports each Pod's actual broker mode, generation (Pod UID) and shell capacity from one read, rejects lark requests conflicting with its configuration (409/503 carrying a capability-refresh marker the Gateway uses to drop the stale cache entry), rejects conflicting or Terminating Pods on coordination creates while reporting Terminating as data to status/get/list, and answers Pod presence independently of Service readiness. - A persistent quarantine (DEER_FLOW_HOME/sandbox-quarantine, keyed by scope/sandbox/generation) fences containers after uncertain implicit shell outcomes across client rebuilds and restarts; in-process reuse, warm reclaim, discovery, create responses and orphan reconciliation all pass one policy gate, and replacement runs under the existing ownership and teardown fences. A fenced sandbox still tracked after a failed release-recycle is destroyed inline on the next acquire. - Quarantine records retire only after confirmed absence, retrying partial cleanup for the quarantined generation and releasing remembered ports under the same fences; local stop failures now propagate instead of being logged away, and remote destroy errors surface to callers. Problem solved: A provisioner blip could silently strip broker isolation from new lark sandboxes for minutes at a time, and ambiguous shell outcomes let corrupted default shells persist across warm reuse, rediscovery and restarts. Docs updated: README, aio_sandbox and skills AGENTS.md. * docs(changelog): record sandbox attestation and quarantine fix (#6436) * fix(sandbox): preserve active leases and fail closed on unattested broker modes Concurrent uploads could recycle a healthy AIO runtime while another holder was creating or cleaning up a command session. Older provisioner responses could also trigger destructive replacement before failing admission. - Distinguish pending session creation from ambiguous outcomes and reserve idle teardown until execution and upload holders finish scope cleanup. - Reject stale fork lookups and reconcile cancelled acquires without releasing a sandbox still borrowed by another holder. - Default Lark broker mode to unverified and preserve unattested Pods until the provisioner is upgraded. - Recognize Docker 29 absence diagnostics while matching resource names exactly. - Preserve Gateway upload errors and cover admission and lifecycle regressions. * fix(sandbox): scope the unverified-broker refuse path to attested providers The fail-closed raise in _lark_cli_env_from_runtime treated every Sandbox without a lark_cli_broker attribute as unverified, but only AioSandbox carries the attestation — E2B, OpenSandbox, Tenki, and Boxlite have no broker sidecar, no credential mounts, and no provisioner to upgrade, so every lark-cli command on those providers failed with an unactionable error. Declare the tri-state on the Sandbox base class (defaulting to False, the only mode brokerless providers have) so the None fail-closed path stays scoped to AioSandbox's deliberate unverified default. - Add lark_cli_broker: bool | None = False to the Sandbox ABC, mirroring the persistent_shell_sessions tri-state pattern - Add a regression test pinning the non-broker overlay for a provider without the attribute * fix(sandbox): close review gaps in broker attestation and teardown lifecycle Address the latest #6436 review round: - The provisioner's post-build broker-mode 409 now carries the capability-refresh marker, so an incompatible-mode conflict surfaced by idempotent re-create, attestation drift or post-create verification drops the Gateway's cached capability observation instead of repeating the same 409 for the full negative-cache TTL. - AioSandboxProvider.destroy() returns the _destroy_tracked outcome: False when teardown is deferred because an execution/upload holder or a peer teardown is still active, so callers can no longer mistake a deferred destroy for a completed one. - A live replacement under the same logical ID proves the fenced old generations are absent; retire_replaced_generations prunes only those stale records so they cannot strand future creates, while the live generation's fence and unversioned crash fences are never touched. - SandboxInfo.from_dict keeps a missing lark_cli_broker attestation unverified (None) instead of silently rehydrating as attested non-broker. - The restricted-network inspect path shares the strict _is_no_such_network_error parser, closing the substring loophole for context-name collisions and wrong-target errors. * docs(sandbox): condense lease notes to keep the AGENTS.md chain under the size gate The three lease notes added in 26c307f3 pushed the effective guidance chain to 98,312 bytes — 8 over the 98,304 hard limit enforced by agent-guidance (AG002). Fold them into two lines carrying the same rules. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zhengxu (Joshua) Jin
|
8fb44f1e48
|
fix(sandbox): number remote read_file line ranges like LocalSandbox (#6585)
* fix(sandbox): number remote read_file line ranges like LocalSandbox E2B, BoxLite, Tenki and OpenSandbox sliced ranged reads with str.splitlines(), which also ends lines at a bare "\r", \f, \v, \x1c-\x1e, \x85, \u2028 and \u2029. LocalSandbox.read_file, count_file_lines and the read_file truncation marker end lines only at "\n", so on those providers a ranged read of such a file returned different lines, and following a truncation marker's start_line went back into text already shown. Add split_file_lines next to count_file_lines and use it in the four providers. * test(sandbox): separate remote and host read paths --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
NanPan
|
479067ea8c
|
fix(channels): fence manager startup across completed shutdown (#6521)
* fix(channels): fence service startup after concurrent shutdown * test(channels): pin delayed manager startup after stop * fix(channels): avoid duplicate manager drain during shutdown * test(channels): name regression for shutdown generation * test(channels): remove ambiguous epoch-named regression * fix(channels): drain late manager start during in-flight shutdown * test(channels): cover manager startup during in-flight channel drain * fix(channels): serialize startup and stale cleanup --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Zeren Wang
|
553f85e788
|
feat(projects): best-effort LLM summaries for shelf documents (#6581)
Opt-in per-document summary column rendered into the injected <documents> index, list_project_documents JSON, and the documents API. Pipeline (projects/summaries.py, behind the narrow enqueue_summary interface): - The two shelf write routes enqueue a job on created=True; dedup hits never regenerate. Off by default (projects.summaries_enabled). - Bounded worker pool (summary_concurrency + bounded summary_queue_size; a full queue drops the job) fed by FIFO; shutdown drains queued jobs, delivers one sentinel per worker via blocking put (queue smaller than the worker pool cannot strand idle workers), and bounds only genuinely in-flight jobs with the grace period. - Jobs carry (document_id, project_id, user_id) explicitly; the job's owner is bound into the user ContextVar around the model call so tracing/usage attribution never falls back to 'default'. - Text source reuses read_text_serving_path (convertible-first rule, content validation); conversion has no deadline (await_drained), the model call has a caller-side wait limit — the worker slot is retained until the provider call actually finishes and late results are discarded. - Output contract: strip leading <think> reasoning -> unwrap fence/ quotes -> collapse to one line -> empty is NULL -> UTF-8 byte cap on a character boundary; length-capped responses (provider-normalized detectors) are failures. - set_summary pins updated_at (literal self-assignment) so a landing summary never reorders the shelf; owner-predicated UPDATE is a no-op on rows purged mid-generation. - Metrics: eligible/success/failed/skipped/queue_dropped/ uncapped_accepted. Schema/config: - project_documents.summary (migration 0036, nullable, no backfill; forward-compat floor unchanged). - Six projects.summary_* keys, all registered in STARTUP_ONLY_FIELDS (captured once at lifespan startup); config_version 57 -> 58. - run_oneshot_llm gains an optional model_overrides passthrough (passed only when set, existing callers unchanged) and a detailed variant exposing the raw response message for termination-signal detection. Docs: API.md response field, ARCHITECTURE.md pipeline paragraph, migrations AGENTS.md chain head, root AGENTS.md records the scoped all-core exception with the lift-out path. |