mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-11-16 00:10:42 +00:00
98 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
1cd42cf42e
|
fix(doctor): check the config file the Gateway actually loads (#5987)
* fix(doctor): check the config file the Gateway actually loads make doctor always inspected <checkout>/config.yaml and ignored DEER_FLOW_CONFIG_PATH and DEER_FLOW_PROJECT_ROOT. A missing override that stops the Gateway from starting still reported "config.yaml found" and "config.yaml loadable", and a valid override pointing elsewhere got the wrong file checked. Doctor now resolves the path through the harness's own resolver, AppConfig.resolve_config_path, after defaulting DEER_FLOW_PROJECT_ROOT to the checkout as scripts/serve.sh does for make dev. An override the Gateway would reject fails "config.yaml found" with the Gateway's error and the config checks skip; an unimportable harness is reported rather than raised. * docs(changelog): reference #5987 * fix(doctor): hand the config-location variables over like serve.sh Review follow-up. serve.sh sources the checkout .env over the shell and then replaces an unset or empty DEER_FLOW_PROJECT_ROOT with the checkout. Doctor loaded .env shell-first and only defaulted an unset root, so a shell DEER_FLOW_CONFIG_PATH / DEER_FLOW_PROJECT_ROOT that differed from .env, or an empty root, made it check a different file than the Gateway loads. For those two variables the .env value now overrides the shell (other keys keep doctor's shell-first loading), and an empty root becomes the checkout, before the harness resolver runs. * fix(doctor): report any harness import failure and expand ~ like source Review follow-up. - The harness-import guard caught only ImportError, so a harness that is present but broken differently (TypeError/SyntaxError while the module body runs) still crashed doctor. Catch Exception, as check_config_loadable does, and name the exception type in the report. - serve.sh's `source .env` expands an unquoted leading `~` in DEER_FLOW_CONFIG_PATH / DEER_FLOW_PROJECT_ROOT and keeps a quoted one literal; dotenv_values expands neither. Expand the unquoted form so a `~/...` path in .env no longer fails a setup that starts, and keep the quoted form literal so doctor does not pass one the Gateway rejects. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
b87ebd9402
|
fix(database): validate pool settings and command_timeout against booleans and non-finite values (#6009) | ||
|
|
dfad2dcc74
|
fix(frontend): keep additional_kwargs on the optimistic bubble through an upload (#5982)
* fix(frontend): keep additional_kwargs on the optimistic bubble through an upload When a send carried files, the upload-complete update replaced the optimistic human message's additional_kwargs with only the uploaded files. Quote and conversation reference chips vanished until the server echoed the message, and project attachments staged with the send were missing from the bubble both while uploading and after. The submitted message was always intact. The optimistic copy (before and after the upload) and the submit now build additional_kwargs through one helper, buildHumanMessageAdditionalKwargs, and share a single uploaded-files value, so the bubble always matches what was sent. * docs(changelog): reference #5982 --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
f889ce7517
|
feat(community): add Unbrowse web fetch provider (#5981)
* feat(community): add Unbrowse web fetch provider Add a community web_fetch provider backed by Unbrowse (https://unbrowse.ai), a hosted service that returns a page as markdown, over plain HTTP when that is enough and through a cloud browser when the page needs JavaScript. The tool sends one JSON-RPC tools/call POST (unbrowse.scrape) to Unbrowse's MCP endpoint, so it needs no extra Python package and no MCP client (uses httpx, already a dependency). Unbrowse is not a general web search engine, so only web_fetch is added. Changes: - backend/packages/harness/deerflow/community/unbrowse/{__init__,tools}.py web_fetch_tool with the same output shape as the Sofya, Exa and Firecrawl fetch tools (# <title> plus markdown, 4KB cap). API key is read from the config.yaml api_key field or the UNBROWSE_API_KEY env var. Optional render: auto|never|always; unknown values log a warning and fall back to auto. JSON-RPC errors, isError tool results, HTTP errors and malformed payloads come back as "Error: ..." strings. - backend/tests/test_unbrowse_tools.py: key resolution, render handling, request shape, structuredContent vs text payloads, error paths. - scripts/doctor.py, scripts/wizard/providers.py: register the key and the web_fetch provider; backend/tests/test_doctor.py covers both doctor paths. - config.example.yaml, .env.example, backend/docs/CONFIGURATION.md, tools/AGENTS.md, EN/ZH harness/tools.mdx: document the provider. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs(changelog): add Unbrowse web_fetch provider entry Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
1e26f00664
|
fix(scheduler): stop interval-task metadata edits from failing with 500 on the reused next_run_at (#5964)
* fix(scheduler): coerce serialized timestamps in ScheduledTaskRepository.update()
Editing an interval task's title or prompt returned 500. The edit dialog
always sends schedule_spec beside the changed field, and when the cadence
is unchanged the PATCH route keeps existing["next_run_at"] — a value the
repository had handed back serialized as an ISO string. update() assigned
it to the DateTime column untouched, so SQLite raised StatementError
("only accepts Python datetime") and Postgres a DataError; only a cadence
change worked, because that path computes a fresh datetime.
Coerce every serialized timestamp update() accepts through the existing
_coerce_datetime, sharing one _TIMESTAMP_KEYS definition with the row
serializer, as update_after_launch() already does (#4785). The router test
now drives the real SQLite repository for the same-cadence rename and the
cadence change; the in-memory stand-in the other router tests use never
saw the type mismatch.
* docs(changelog): reference #5964 in the scheduled-task PATCH entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
|
c89ba45f7f
|
chore(doc):fix the changelog section error (#5868) | ||
|
|
b72753c226
|
fix(gateway): load skills off the event loop in the listing and detail routes (#5945)
* fix(gateway): load skills off the event loop in the listing and detail routes
GET /api/skills, GET /api/skills/custom and GET /api/skills/{name} called
load_skills() inline. Resolving the caller's storage builds absolute
paths, and load_skills walks every public and custom skill directory and
parses each SKILL.md, so the work grows with the number of installed
skills. #5747 moved the same call off the loop for get_custom_skill and
documented why; these three routes were missed, and the strict
Blockbuster gate raises BlockingError on each of them.
Offload the load through one shared _load_user_skills helper (also used
by update_skill's reload), and add a gate module covering the three
routes including get_skill's 404 branch.
* docs(changelog): reference #5945 in the skills listing blocking I/O entries
|
||
|
|
931d2eeb81
|
fix(models): deep-merge when_thinking_enabled on the legacy path (#5894)
* fix(models): deep-merge when_thinking_enabled on the legacy path The legacy thinking path (profiles without a `reasoning:` block, i.e. every existing config.yaml) applied `when_thinking_enabled` with a shallow `dict.update`, so a template such as `extra_body.thinking.type=enabled` replaced the profile's whole `extra_body` and silently dropped sibling keys like GLM's `extra_body.tool_stream` — but only while thinking was on. The disable branch of the same function already deep-merged into `extra_body`, and the contract path added in #5780 deep-merges through `_merge_settings` for exactly this reason (its docstring names `tool_stream`). The on/off asymmetry cannot be intended: the same profile sent `tool_stream` with thinking off and lost it with thinking on. Route the enable template through `_merge_settings` as well. Precedence is unchanged (template values win on conflicts); the only observable difference is that keys present only in the profile now survive a thinking-on request. Everything else on the legacy path — the synthesized `reasoning_effort=minimal`, the disable payloads, the supports_thinking guard — is untouched. Tests pin both directions for a base extra_body beside a thinking template, recursive merging of nested chat_template_kwargs, and template precedence. * docs(changelog): reference #5894 in the when_thinking_enabled entries * fix(models): keep the template's vLLM switch spelling authoritative when merging Review follow-up. Deep-merging the thinking template into a profile keeps both spellings of vLLM's toggle when they differ — base `chat_template_kwargs.enable_thinking: false` plus a template using the legacy `thinking: true` alias — and VllmChatModel's normalizer maps the alias onto `enable_thinking` only when the latter is absent, so the profile's `false` won and thinking stayed off despite thinking_enabled=True. The previous shallow replacement never hit this because it discarded the base switch. The contract path had the same trap since it already deep-merged, and both disable branches had the mirror image (a base `enable_thinking: true` outlived a template `thinking: false`). Route every thinking on/off merge through _merge_thinking_payload: it drops the profile's *other* spelling of the switch before the deep merge, so the payload's spelling decides in both directions on both paths, and unrelated chat_template_kwargs still survive. Tests cover both spelling combinations, both directions, and both paths, asserting the switch as the provider will send it after normalization. * fix(models): deep-merge the legacy disable template and mirror the vLLM switch instead of pruning Review of the previous head found two regressions. The legacy when_thinking_disabled branch was still dict.update-ed, so a profile that declares both templates (the shape of most extra_body examples) kept tool_stream while thinking was on and lost it while off. And the merge pruned the profile's other vLLM switch spelling for every model class, although only VllmChatModel maps `thinking` onto `enable_thinking`: on a plain ChatOpenAI profile the prune removed the only key vLLM reads, so thinking-off could no longer switch reasoning off. Route when_thinking_disabled through _merge_thinking_payload too, and replace the prune with mirroring: after the deep merge, the value the payload declares for its switch spelling is copied onto the profile's other spelling when the profile carries it. VllmChatModel collapses the pair, plain OpenAI-compatible classes send both, and either way the server reads the template's intent. No spelling is invented, a payload that spells both is left alone, and non-mapping template values are forwarded unchanged instead of raising. _VLLM_THINKING_SWITCHES is now the single definition used by the disable, toggle and dialect helpers. Docs state the merge semantics (keys are never removed; a template can only add or override) and drop the "byte-for-byte" wording; tests cover the disable template on/off, nested inheritance, non-mapping values, the raw payload for a non-normalizing class, and the no-invention rule. * fix(models): copy thinking templates on merge and document the never-remove rule Review follow-up. Templates merged into the constructor settings were stored by reference wherever the profile lacked the key, so the kwargs handed to the model class aliased the cached ModelConfig; the mirroring step now writes into that mapping, so make _merge_settings and _deep_merge_dicts deep-copy every template mapping they store. Pinned at every depth by a test that also checks the profile is unchanged. Deep-merging the legacy when_thinking_disabled template means it can no longer clear a key the profile's base extra_body sets: a base extra_body.thinking {type: enabled, budget_tokens} reaches the provider as {type: disabled, budget_tokens}, which Anthropic-style APIs reject. The synthesized disable payloads and the contract path already behaved this way. State the rule and the migration (enable-only keys belong in when_thinking_enabled) in the changelog, models/AGENTS.md and configuration.mdx, and pin it with a test for both shapes. Route the legacy synthesized OpenAI disable through _merge_thinking_payload too (identical output), narrow the docstring and guidance wording about the native-Anthropic disable, and make the two vLLM switch suites resolve the real VllmChatModel / ChatOpenAI classes so a class-gated merge would fail the case it is named after. |
||
|
|
5b8b69bd45
|
fix(gateway): keep /api/user-profile filesystem work off the event loop (#5935)
* fix(gateway): keep /api/user-profile filesystem work off the event loop GET and PUT /api/user-profile resolved the per-user USER.md path (which builds absolute paths on every call), stat'ed, read, created the user bucket and wrote the file inline on the event loop, while every other handler in the custom-agent router offloads that work with asyncio.to_thread. The strict Blockbuster gate raises BlockingError on both; in production a slow disk stalls every other request on the worker. Offload the whole resolve-stat-read and resolve-mkdir-write sequences, capturing the effective user id on the loop first like the sibling handlers. The blocking-IO gate test for this router now covers both profile routes (missing profile, write, read back). * docs(changelog): reference #5935 in the user-profile blocking I/O entries |
||
|
|
a253faefdf
|
fix(docker): give project-document uploads their own nginx location so they are not capped at 1 MB (#5934)
* fix(docker): give project-document uploads their own nginx location
POST /api/projects/{id}/documents is a multipart upload that Gateway
accepts up to uploads.max_file_size (50 MiB by default), but no nginx
location matched it. It fell through to the /api/ catch-all, which has no
client_max_body_size, so nginx's 1m default answered a 2 MB PDF with a
bare 413 before Gateway saw the request. Reproduced on nginx:alpine with
the shipped config; the same body to /api/threads/{id}/uploads passed.
Add a `~ ^/api/projects/[^/]+/documents` location to the Docker, make dev
and Helm configs with the thread-uploads settings (100M, streamed body)
and the 600s read timeout the catch-all already granted these routes. The
catch-all keeps nginx's defaults. The shared location-block helpers move
to tests/support/nginx_conf.py so the new test and the existing body-size
test use one implementation.
* docs(changelog): reference #5934 in the project-documents nginx entries
|
||
|
|
231335d309
|
fix(deploy): honor BETTER_AUTH_SECRET and DEER_FLOW_INTERNAL_AUTH_TOKEN from .env in make up (#5928)
* fix(deploy): honor BETTER_AUTH_SECRET and DEER_FLOW_INTERNAL_AUTH_TOKEN from .env deploy.sh only checked the shell before reloading a persisted secret or generating a fresh one and exporting it. Compose interpolation lets shell variables outrank --env-file, so a secret written to the repo-root .env -- the surface every deployment doc points at -- was silently replaced by the generated one: sessions were signed with a secret the operator never chose and Gateway workers outside the stack got 401 with the configured token. Leave a .env-provided secret unexported so Compose parses it itself (shell -> .env -> persisted file -> generated), and keep generating when the shell exports an empty value, since Compose would otherwise pass the empty string through. * docs(changelog): reference #5928 in the deploy.sh dotenv secrets entries * fix(deploy): decide whether .env provides a secret with Compose's own parser read_dotenv_value only matches KEY=VALUE and returns the literal text, but Compose also accepts `KEY: VALUE` lines and interpolates `${VAR}` inside values. So `TOKEN: abc` looked absent (and got shadowed by a generated token) while `TOKEN=${UNSET}` looked present (and suppressed generation, leaving the stack with an empty token). Ask Compose for its resolved interpolation environment instead (`docker compose config --environment`, same --env-file, stub project on stdin, project directory docker/) and treat a resolved-empty value as missing. Clients older than Compose 2.28 lack the flag and fall back to the plain reader. Tests cover the canned Compose answers, the fallback, and every dotenv spelling against the real Compose client when installed. * fix(deploy): resolve .env secrets through Compose on every supported client `config --environment` needs Compose 2.28, so 2.24–2.27 clients took the plain KEY=VALUE fallback and kept both defects: `TOKEN: abc` was missed and shadowed, `TOKEN=${UNSET}` counted as present and left the stack with an empty token. Drop the fallback. Render a stub project whose only environment entry is `${KEY}` through `docker compose config` — same --env-file and project directory as the real command — and read the interpolated value back; `""` means empty or unset. This is plain interpolation, available on every Compose v2, and verified identical on 2.24.0, 2.27.0 and 2.32. A failing probe stops the script instead of guessing. The real-Compose tests now run against the docker CLI and any standalone binaries listed in DEER_FLOW_TEST_COMPOSE_BINARIES. |
||
|
|
e8cb5e9c4f
|
fix(skills): keep unmatched markdown brackets out of quadratic backtracking (#5884)
* fix(skills): bail out of the markdown-link scan when no link construct exists (#5714) * fix(skills): scan markdown links linearly and skip failed target runs (#5714) * fix(skills): make link scan exactly match finditer, blank spans by position Address review feedback: drive the scan by openers (the regex engine's own start-position scan) so it reproduces finditer exactly, blank matched spans by position instead of str.replace, pin the equivalence with tests, and add CHANGELOG entries. * fix: resolve skill review test lint errors --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
1dfd307c92
|
fix(docker): strip the IPv6 listen in the production nginx launcher too (#5900)
* fix(docker): strip the IPv6 listen in the production nginx launcher too docker/nginx/nginx.conf listens on 2026 and [::]:2026. On a host booted with ipv6.disable=1 (or a kernel without IPv6) nginx refuses to start — socket() [::]:2026 failed (97: Address family not supported by protocol) — so the container restart-loops under `restart: unless-stopped` and `make up` waits on a readiness probe that never turns healthy. #2027 added a launcher guard that deletes the IPv6 listen when /proc/net/if_inet6 is absent, but only to docker-compose-dev.yaml; the Helm chart mirrors it; docker-compose.yaml — the file behind `make up` and scripts/deploy.sh — still ran the bare copy-and-start command. Give the production nginx service the same launcher as the dev file: `set -e`, copy the template, guard, and `exec nginx` so nginx is PID 1 and receives docker stop's signal directly. The new test pins the guard in all three launchers (prod, dev, Helm), requires the three sed patterns to be identical so they cannot drift again, checks the guard edits the file nginx actually loads, and applies the pattern to the shipped compose and Helm configs to confirm it removes only the IPv6 listen. Verified on nginx:alpine: BusyBox sed strips exactly that line and `nginx -t` accepts the result; the rendered production launcher runs unchanged when IPv6 is present. * docs(changelog): reference #5900 in the nginx IPv6 guard entries |
||
|
|
bf9a019f28
|
fix(skills): let remove_file/write_file handle binary support files and reject directories (#5893)
* fix(skills): let remove_file/write_file handle binary support files and reject directories
A .skill archive may carry binary support files such as assets/logo.png:
the installer only rejects executable binaries by magic bytes. But both
skill_manage file actions read the previous content as UTF-8 text before
touching the file — purely to fill the history record's prev_content — so
a binary member raised UnicodeDecodeError out of
SkillStorage.remove_custom_skill_file (and out of the tool's write_file
prelude) and was never removed or overwritten.
A bare support directory ("assets") also slipped through
ensure_safe_support_path, because it resolves to its own allowed root and
passes the containment check, and then raised IsADirectoryError from the
same read.
Add read_text_or_none for the history reads: UTF-8 text is returned as
before, undecodable content yields None (the record simply has no previous
text), and IO errors still propagate. Require a filename below the support
subdirectory in ensure_safe_support_path, and reject an existing directory
with ValueError in both remove_custom_skill_file and the tool's write_file,
so the model gets an actionable validation error instead of a crash.
prev_content is informational for support files — rollback only restores
SKILL.md — so recording None loses nothing that could be used.
* docs(changelog): reference #5893 in the support-file entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
|
59aa91d622
|
fix(projects): label untitled threads in the conversation-files view (#5879)
* fix(projects): label untitled threads in the conversation-files view
GET /api/projects/{id}/threads/files returns each member thread's
display_name as `str | None`: the thread meta row is created with NULL and
only filled once title generation has run after the first reply, or never
if that step fails. The frontend's ProjectThreadFileGroup declared the field
as a required string and ConversationFileGroup rendered it verbatim, so a
thread whose files were uploaded before its first reply showed them under
an empty heading. The sibling ProjectThread type already models the field
as nullable and every consumer of it falls back to t.projects.untitled.
Make the group type nullable to match the wire, and render the heading
through a shared threadTitle() helper that both the thread-name map and the
file group now use, so the "Untitled" rule lives in one place.
The DOM test covers null, missing and whitespace-only titles.
* docs(changelog): reference #5879 in the conversation-files entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
|
9c53ebb06f
|
fix(skills): unlink a symlinked custom skill package on delete (#5881)
* fix(skills): unlink a symlinked custom skill package on delete
The storage contract (_is_external_skill_directory_symlink) accepts a
one-level symlink under custom/ that points at an operator-managed skill
package, and every read path honours it: the skill loads, lists, validates
and can be enabled. delete_custom_skill, however, removed the package with
shutil.rmtree, which refuses symlinks. The delete raised OSError after the
history record had already been appended, skill_projection_mutation cleared
the user's projected custom view on the way out (rebuilt on the next
acquire), and DELETE /api/skills/custom/{name} answered 500 — for both
LocalSkillStorage and UserScopedSkillStorage, which inherits the method.
Unlink the entry when it is a symlink and keep rmtree for real directories.
The external tree is not ours to delete: removing the skill means removing
the link. Checking is_symlink() first also covers a dangling link, which
exists() would report as absent.
Tests cover both storages: the link is gone, the skill no longer exists,
the external package (SKILL.md and a support file) is untouched, the
history record was written, and the sibling skill's projected view
survives the delete.
* docs(changelog): reference #5881 in the symlinked-skill delete entries
|
||
|
|
74f6dce489
|
fix(dev): keep whole worktree paths when serve.sh builds DEERFLOW_ROOTS (#5856)
* fix(dev): keep whole worktree paths when serve.sh builds DEERFLOW_ROOTS
serve.sh collects every deer-flow worktree root so `make stop` / `make dev`
can reclaim the shared dev ports (8001/3000/2026) from a sibling checkout.
It parsed `git worktree list --porcelain` with `awk '{print $2}'`, but that
output does not quote paths, so a worktree at `.../deer flow two` was
recorded as `.../deer`. `_is_deerflow_pid` matches open files against
"<root>/", which the truncated root never satisfies, so a Gateway or
frontend started from that worktree was treated as a foreign process and
the start aborted with "port already in use".
Take the remainder of the line after the `worktree ` key instead. The
dedupe and longest-first ordering that follow are unchanged.
The new test builds a scratch repo with a spaced linked worktree and runs
the DEERFLOW_ROOTS block from serve.sh against it, from both the main
checkout and the spaced one.
* docs(changelog): reference #5856 in the worktree-roots entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
|
bbab1e84d0
|
fix(uploads): never let a malformed files[*].size abort the run (#5855)
* fix(uploads): never let a malformed files[*].size abort the run
UploadsMiddleware._files_from_kwargs validates every field of a
client-supplied file entry fail-soft — non-dict entries, non-basename or
staging filenames, and files missing from the uploads dir are all skipped —
except `size`, which went straight to int(). A value int() cannot convert
("abc", a list, inf) raised ValueError/TypeError/OverflowError out of
before_agent (and the executor-wrapped abefore_agent), failing the whole run
before the model was called. The Gateway strips only server-owned metadata
keys and edit/regenerate deep-copies `files` verbatim, so the same entry
failed again on every retry of that message.
The size only feeds the human-readable "(N KB)" line in <current_uploads>,
so route it through _coerce_file_size: a non-negative finite number (or a
numeric string such as "2048") is kept as int; None, bools, negatives,
non-finite floats and anything unconvertible fall back to 0, the value a
missing size already gets.
Tests pin the fallback for each unusable shape, the kept values, and that
before_agent still injects an existing upload whose size is malformed.
* docs(changelog): reference #5855 in the uploads size entries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
|
65ed502e54
|
fix(release): refresh backend/uv.lock when bumping the version (#5859)
* fix(release): refresh backend/uv.lock when bumping the version `scripts/bump_version.sh` rewrote backend/pyproject.toml, frontend/package.json and the Helm chart's `version`/`appVersion`, but `backend/uv.lock` also records the root package version (uv keeps its PEP 440 form, so `2.1.0-rc0` is stored as `2.1.0rc0`). The documented release step left the lock pinning the previous version, and that is not cosmetic drift: `uv lock --check` (lint CI) fails on a stale lock and `uv sync --locked` (e2e / plugin workflows) refuses the tree, so the release commit could not pass CI. RELEASING.md only mentioned the manual `cd backend && uv lock` step for release candidates. The script now refreshes the lock via `uv lock` and refuses to start without `uv` — it exits before editing anything rather than leaving a half-bumped tree whose lock CI rejects. `uv lock` stays the authority for the normalized prerelease form; the lock diff for a bump is a single line. RELEASING.md and AGENTS.md now describe the lock as part of the bump. Tests: backend/tests/test_bump_version_lockfile_sync.py (red on main, green here) pins both halves — the lock moves with the sources, and a missing uv is an error instead of a silent no-op. * docs(changelog): record the release bump lockfile fix (#5859) --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
c8d9ed1b08
|
chore(doc): updated the documents of the develope of Version 2.2 (#5862)
* chore(version):update the version to 2.2.0-dev * chore(doc):Updated the CHANGLOG with the 2.1.0 release version * chore(doc): Updated the CHANGLOG for the latest changes * chore(doc): Added the release SOP document |
||
|
|
38b82057e1
|
fix(config): sign the bytes the config cache parsed, not a second read (#5848)
* fix(config): sign the bytes the config cache parsed, not a second read _load_and_cache_app_config parsed config.yaml through AppConfig.from_file and then called get_config_signature, which opens and hashes the file again. A write landing between those two reads left the cache holding the older content under the newer content's signature. get_app_config() compares the on-disk signature against the recorded one, so from then on it saw "unchanged" and the edit only surfaced when yet another edit changed the signature — silently stale until then, with no self-healing. Read the file once through a new shared helper, file_signature.read_config_with_signature, which returns the bytes together with the (mtime, size, sha256) of exactly those bytes, and parse them through AppConfig._from_yaml_text (from_file's body, split out; from_file itself is unchanged). The recorded signature now always describes the parsed content: a write that races the load can only produce a signature mismatch on the next call, i.e. one extra reload. Signing before parsing but re-reading for the parse would not be enough — an edit that lands between the two reads and is then reverted would be served forever — so the tests pin that the parsed bytes are the signed bytes. The stale-metadata reload test now routes the loader's read through the same stale-stat simulation it already applies to the probe, since the loader no longer records the probe's result. * docs(changelog): reference #5848 in the config cache entries --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
3448c3426e
|
fix(config): honor $VAR substitution for request_admission integer fields (#5838)
* fix(config): honor $VAR substitution for request_admission integer fields AppConfig.resolve_env_variables replaces every `$VAR` reference with the raw environment string. RequestAdmissionConfig.requests_per_minute and max_queue_size are declared `strict=True` so a bool or float cannot slip in, but strict mode also refuses the string "60", so `requests_per_minute: $RPM` failed the whole config load with "Input should be a valid integer" even when RPM=60 — contradicting config.example.yaml's promise that environment variables work for every field. They were the only strict fields in the config schema; every other integer coerces in lax mode. Wrap both fields in a BeforeValidator that converts a decimal integer literal arriving as a string and leaves everything else untouched, so the strict check behind it still rejects bools, floats, "60.0" and non-numeric strings. Tests pin the accepted literals, the rejections that must survive, and the end-to-end path from a config.yaml `$VAR` reference through AppConfig.from_file. * docs(changelog): reference #5838 in the request_admission entries |
||
|
|
174c6f7fd3
|
fix(mcp): run lazy MCP discovery once when initialization itself raises (#5836)
* fix(mcp): run lazy MCP discovery once when initialization itself raises get_cached_mcp_tools() wrapped the whole lazy-initialization call in the `except RuntimeError` that was only meant to catch asyncio.get_event_loop() raising when no loop is set. run_until_complete() and Future.result() re-raise the coroutine's own exception, so a RuntimeError from discovery — e.g. the deliberately re-raised McpTaskConfigurationError — took the same fallback and ran asyncio.run(initialize_mcp_tools()) a second time, respawning every stdio server and re-fetching OAuth tokens before giving up. Inside a running loop that fallback fails with "asyncio.run() cannot be called from a running event loop", so the logged traceback hid the real cause. Resolve the loop first, in its own try/except, and drive initialization exactly once: asyncio.run when no loop is set or the current loop is closed, a worker thread when the loop is running, run_until_complete otherwise. Any failure is logged once and returns [] as before. Regression tests pin one discovery call and the real logged cause with and without a running loop, and keep both fallbacks (no loop set, closed loop) covered. * test(mcp): surface worker-thread failures in the no-loop fallback test Thread.join() does not propagate exceptions, so a regression of the asyncio.run fallback failed this test with a bare KeyError: 'result' instead of the real traceback. Capture sys.exc_info() in the worker and re-raise it on the main thread before the assertions. |
||
|
|
c943d953cd
|
docs(checkpoints): add a ten-chapter checkpoint storage manual in both languages (#5845)
* docs(checkpoints): add a bilingual checkpoint storage manual Add the missing docs-site coverage for dual-mode checkpoint storage (`database.checkpoint_channel_mode: full | delta`) as a ten-chapter manual under `harness/checkpoints/` in both languages: concepts, quick start, channel modes, snapshot cadence, history cache, resume and rollback, operating checkpoints, observability, troubleshooting by symptom, and a reference of every configuration key, error message, HTTP status mapping, public symbol, load-bearing test, and pinned upstream defect. The index page is `asIndexPage`, so `/docs/harness/checkpoints` is the section itself and its `_meta.ts` lists only the nine child chapters. Discovery: `harness/_meta.ts` registers the section, the harness configuration module table gains a `database` row, and the thread persistence section of `agents-and-threads` links to the manual. Content is grounded in the code. Only the three measured values recorded in `backend/docs/runtime-guidance-details.md` are quoted; the upstream-derived 41x storage figure in the untracked delta-channel RFC is deliberately not cited. Pinned upstream defects are documented with their issue and PR numbers and the versions they are pinned at (`langgraph 1.2.9`, `langgraph-checkpoint 4.2.0`, `langgraph-checkpoint-postgres 3.1.2`). * docs(checkpoints): correct the history-cache restart requirement The history cache is built with the checkpointer at Gateway startup (`langgraph_runtime()` -> `make_checkpointer()` -> `make_checkpoint_cache()`), so editing `database.checkpoint_cache.*` does not change a running process. The manual previously marked the whole section "not restart-required", which conflated "not frozen for correctness" with "hot-reloadable" and could mislead an operator changing capacity, credentials, or retention. Distinguish the two properties everywhere the section is described: history-cache.mdx marks every field as needing a restart and adds a callout explaining the difference, reference.mdx redefines the Restart column and fixes the five rows and the effect list (adding the `asyncprov` evidence alias), troubleshooting.mdx replaces the "two keys need no restart" claim and gains a dedicated symptom entry, and observability.mdx notes that changed counters need a restart. The Chinese text is corrected in parallel. Verified: the docs link test and the docs unit suite pass, frontmatter parses for all 20 pages, and every in-page anchor resolves on the rendered English and Chinese history-cache, troubleshooting, reference, and observability pages. * docs(checkpoints): account for periodic snapshots in the storage-growth claim Every periodic snapshot stores the entire message list as of that step (`_DeltaSnapshot(ch.get())`, persisted as a checkpoint blob), so with a fixed snapshot frequency K the retained total after N turns is O(N²/K + N), not O(N). The manual claimed unconditional linear growth, which understated capacity planning — the more so because nothing in DeerFlow prunes those snapshots yet. Distinguish per-step delta writes from total retained storage in index.mdx (prose and table), modes.mdx (step-cost row), and operations.mdx; note in cadence.mdx that K is also the capacity knob that scales the snapshot bytes. The English and Chinese chapters are corrected in parallel. The reference chapter keeps the released #4292 wording as a verbatim quotation and now names the retained-storage bound beside it. Verified: `_DeltaSnapshot(ch.get())` and its blob serialization in the pinned langgraph (channels/delta.py:196-198, checkpoint/serde/types.py:19-25) confirm the premise; the docs link test and docs unit suite pass; all ten changed pages render with resolving anchors and no stale linear-growth wording. * docs(checkpoints): correct the run-error surfaces and repair two evidence pointers Review follow-ups on the checkpoint manual, both languages. Error-surface corrections: - Run creation is not always asynchronous: when a thread's run-event feed is empty but a checkpoint head exists, ensure_checkpoint_history_seeded reads that head inside start_run, where only ConflictError (409) and UnsupportedStrategyError (501) are mapped, so a mode mismatch returns a plain 500 before admission. Troubleshooting, observability and the reference route tables now scope the "does not pre-gate" claim. - The 503 symptom is retargeted. Only Gateway startup, make_lead_agent and DeerFlowClient.__init__ call the freeze helpers, so a config.yaml edit cannot make a state/history/thread route answer 503. A mode edit is injected over by the run worker and ignored until a restart; a cadence edit fails the next run with the restart-required message. The router's 503 mapping is marked defensive in modes and the reference. - The regenerate section title no longer groups the state endpoints, which map a mismatch to 409. - operations: the Textual TUI catches the mismatch through stream_actions(); only the headless --print/--json runners print a traceback. Evidence-pointer repairs: - reference: this branch's CHANGELOG insertion shifted the #5734 entry from 3032-3039 to 3040-3048; the citation is re-pointed. - reference: the #8548 row cited an untracked worktree plan file; replaced by the in-repo test anchor, and "mirror issue #8551" reworded to "related". - reference: new `agent` alias for the agent-factory freeze sites. - history-cache: the bare `deps:`/`asyncprov:` aliases are expanded to full paths; the alias legend lives only on the reference page. * docs(checkpoints): say a delta fork is accepted, not refused "Delta mode cannot fork" overstated the constraint. Nothing in the stack rejects the fork: LangGraph stores the second child, no saver checks for an existing sibling, and ensure_checkpoint_mode_compatible returns immediately when the process mode is delta. What breaks is the read — the walk replays the abandoned sibling's writes stored under the shared parent. Reproduced on the pinned stack (langgraph 1.2.9 / langgraph-checkpoint 4.2.0, AsyncSqliteSaver) by running this repo's own #4458 fixture without the linearization call: the forked resume returns ["h1", "a1", "h2", "a2", "a2-new"] instead of ["h1", "a1", "h2", "a2-new"], i.e. the superseded answer comes back. - resume-and-rollback: title/description/heading now say "cannot fork correctly"; new "Forks are accepted, not refused" subsection covering the unguarded write side. - troubleshooting: the "old answer reappears" cause states the same split. - reference: the POST /threads/{id}/state row notes that a client-supplied checkpoint_id older than the head is applied as a delta fork with no linearization — the write shape upstream tracks as #8551 (open), which the run-path linearization does not cover. * docs(checkpoints): scope the unguarded state-update fork to what the probe showed The previous commit said a mutation against an older checkpoint stores its writes under the shared parent and implied the sibling branch suffers for it. Probing the pinned stack (AsyncSqliteSaver, delta messages channel) shows: - the write does land under the shared parent and joins the write set the parent's other children walk (raw walk for a sibling includes the abandoned head's own write); - but reducer channels are written with Overwrite, which replaces the accumulated value, so the forked branch reads clean and the abandoned branch is unchanged in that shape — #8551 did not reproduce; state the masking mechanism and the upstream tracking issue, not a corruption claim. * docs(checkpoints): fix a frontmatter description that broke the docs build `resume-and-rollback.mdx` shipped `description: Delta mode cannot fork a thread correctly: a forked delta read ...` in the previous commit. The unquoted ": " turns the value into a nested mapping, so Nextra's YAML frontmatter parse failed and the e2e lane could not start its web server (YAMLParseError at line 2, column 14) — the exact failure class this manual warns docs authors about, introduced by me two commits ago. - Replace the colon with an em dash in the English description (the Chinese one uses a full-width colon and was unaffected). - Add tests/unit/content/docs-frontmatter.test.ts: every `src/content/**/*.mdx` frontmatter block must be YAML-parseable, implemented as the one rule that actually breaks it (an unquoted scalar containing ": "), because the unit suite never compiles MDX and only a build or a dev render catches it. Verified the guard fails on the broken line and passes on the fix. Verified with a clean `BETTER_AUTH_SECRET=local-dev-secret pnpm build` on this tree: all pages prerender, resume-and-rollback EN/ZH included. |
||
|
|
bcaf6db36a
|
feat(models): add declarative reasoning capability contract (#5780)
* feat(models): add declarative reasoning capability contract (#5073) Introduce an optional per-model `reasoning:` block beside the legacy `supports_thinking` / `supports_reasoning_effort` booleans. It declares thinking availability (unsupported | optional | required), the accepted effort values with aliases, default, and serialization path, the payload dialect, and the reasoning-history requirement. When present the booleans are derived from it and contradictory profiles fail at config load. `deerflow.models.reasoning` normalizes legacy and declared profiles into one immutable contract and resolves every caller's generic request against it. `create_chat_model` is the single enforcement point: required-thinking models never receive a synthesized disable payload, unsupported effort values never reach the provider, and the dialect synthesizes the on/off payload when no template exists. Legacy profiles keep the historical path byte-for-byte. The lead agent and subagent descriptor resolve the same policy first so run metadata reports effective values. `/api/models` and `DeerFlowClient` project the contract as `reasoning`; the frontend derives mode clamping and effort choices from it through `core/models/reasoning.ts`, and user preferences accept provider-specific effort tokens. The setup wizard's Z.AI GLM-5.3-Flash profile migrates to the contract, restoring its low/high/max effort control. * docs(models): move the reasoning contract guidance into models/AGENTS.md backend/AGENTS.md sits within 46 bytes of the 96 KiB effective-chain hard limit for the middlewares, runtime, sandbox, and subagents guides, so the new section pushed scripts/check_agent_guidance.py over it. The models guide is the nearest file for the factory and the contract module, and its chain has ample room. * fix(models): address review on the reasoning contract - vLLM dialect: the enable path now mirrors whichever chat_template_kwargs switch the template declares, like the disable path already did, so a template using the older `thinking` key no longer also grows `enable_thinking`. - Validate operator-supplied effort values at `effort.path` inside `when_thinking_enabled`, `when_thinking_disabled`, and the `thinking` shortcut against the contract, not only the profile-level value; they are forwarded when the caller chooses nothing, so the "unknown values never reach the provider" guarantee now holds for them too. - Constrain `effort.path` to dotted identifiers and reject single-segment paths that would replace a whole mapping (`extra_body`, `thinking`, `model_kwargs`, `default_headers`, `default_query`). - Z.AI GLM-5.3-Flash profile: `default: high` instead of `max`, because the default also governs summarization, title generation, and subagent calls that never choose an effort; `max` stays selectable. Documented in the config example, the configuration manual, and the design note. * fix(models): close reasoning contract review gaps * fix(models): preserve native Ollama reasoning config * fix(models): keep native reasoning strings and project reasoning into the descriptor - `ModelConfig.reasoning` also accepts the level string ChatOllama takes for gpt-oss style models (`low` / `medium` / `high`); the factory forwards both the boolean and the string on the legacy path, so profiles that passed `reasoning: high` through before the contract field existed keep loading and keep sending it. - The assembly descriptor no longer excludes `reasoning` from `model_parameters`: the native ChatOllama value and a declared contract's dialect/history/effort change the request payload, so they now move the assembly fingerprint. - e2e: the sidecar spec pinned the pre-contract body where a model without effort control still received `reasoning_effort: minimal`; the preset is now dropped for such models (the Gateway discarded it anyway), so the assertion expects the key to be absent. |
||
|
|
509151d006
|
fix(scheduler): read the lease owner under SQLite's writer lock (#5777)
* fix(scheduler): read the lease owner under SQLite's writer lock release_dispatch_lease guards on the lease owner and then writes the status its caller asked for — "enabled" for a scheduled dispatch. It read the row with session.get(..., with_for_update=True), and FOR UPDATE is ignored on SQLite, so that guard ran against whatever snapshot the plain SELECT saw. A user pausing a task while its dispatch was in flight therefore lost the pause: pause clears lease_owner, which is exactly what the guard checks, but the release had already read the pre-pause row, passed the check, and overwrote status with "enabled". The API had answered "paused", the task row went back to enabled with next_run_at untouched, and the scheduler kept firing it. The read now goes through _lock_task, the helper every other mutating path here uses, which takes the writer up front on SQLite (FOR UPDATE keeps doing the work on Postgres, where a pause that commits first is seen and the release declines). The reconciliation paths in this file keep their documented "no SQLite writer emulation" trade-off. * docs(changelog): note the scheduler lease-release lock fix (#5777) --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
e489bf2c92
|
fix(auth): claim the first admin atomically instead of counting first (#5776)
* fix(auth): claim the first admin atomically instead of counting first POST /api/v1/auth/initialize read the admin count in one session and created the admin in another, with no lock between them. Two requests arriving together with different emails both saw an empty system and both became admins: reproduced over HTTP, two 201s and two admin rows, after which setup-status reports an initialized system. The endpoint is public and unthrottled, and during the pre-setup window anyone who can reach the gateway can claim admin with a single request anyway — that part is the first-boot design. What the race removes is the only signal: without it exactly one request wins and the loser gets 409, so an operator who loses sees "already initialized" and looks into it. With it, both succeed and the operator's setup looks ordinary. The check and the insert now share one transaction in the repository, with writers serialized first — SQLite takes the write lock up front (BEGIN IMMEDIATE) and Postgres a transaction-scoped advisory lock, the idiom the project and channel repositories already use. The same repository serves both backends, and multi-replica deployments make the race genuinely parallel rather than an event-loop interleaving. create_first_admin returns None when the claim is lost, which the route reports as 409; uniqueness conflicts still raise ValueError and stay a 400. The count in the handler is now only a fast path that avoids hashing a password for an already-initialized system. AUTH_DESIGN.md said concurrent initialization was backstopped by a database unique constraint. No such constraint exists — email uniqueness only catches the same address submitted twice — so it now describes the claim that does the work. * docs(changelog): note the atomic first-admin claim (#5776) * fix(auth): refuse a dialect the first-admin claim cannot serialize Review follow-up. The claim's serialization handled sqlite and postgresql and fell through silently for anything else, which would leave create_first_admin as the plain check-then-act it exists to prevent — the one place the abstract method's atomicity contract is enforced. An unknown dialect now raises instead. The engine builds only these two today, so this guards a future backend: it fails loudly at first boot rather than quietly reopening the race. |
||
|
|
5d8a9492ee
|
docs(extensions v0.2.3): cover full-stack plugins and request-scoped run evidence (#5775)
* docs(extensions): add an eleven-page extension developer manual
Add a harness/extensions/ section in English and Chinese documenting the
deerflow-extension-api 0.2.3 contract:
index when to write an extension vs a tool, MCP server or
skill; contribution kinds; loading; failure and
trust model; versioning
quick-start build, test, install and remove a working extension
runtime load sequence, API version rules, scopes and
ExtensionData, fail-open and cancellation, diagnostics
middleware placements and where each lands, scope, ordering,
observe-only wrap hooks, failure isolation
observers task lifecycle, system model calls, agent assembly
and fingerprint, context compaction
services-and-routes service lifecycle, router rejection rules, auth,
principals
run-evidence service and request-scoped readers (#5727), cursor
semantics, deletions, redaction, store differences
plugins experimental full-stack plugins, inline modules and
manifest-based browser assets (#5685)
operations extension manager CLI, sources, upgrade, rollback,
Docker, Helm, recovery
troubleshooting indexed by the exact log and error strings
reference every public name and the contract version history
Also fix stale descriptions: the contribution kinds listed in AGENTS.md
and deerflow/extensions/AGENTS.md (now eight, including plugins), the
run-evidence redaction claim (only auth_token is removed), and the plugin
mount return value in docs/full-stack-plugins.md ({ dispose }, not a
function). Changelog entries added in both languages.
* docs(agents): trim root AGENTS.md to keep inherited chains under the size limit
* docs(extensions): scope the manual to the 0.2.1 contract
Plugins and request-scoped run evidence are not in 2.1.x-dev yet; they
move to a stacked follow-up so this change can be cherry-picked.
* docs(extensions): cover full-stack plugins and request-scoped run evidence
Brings the extension manual up to deerflow-extension-api 0.2.3. Stacked on
the 0.2.1 manual so that part can be cherry-picked to 2.1.x-dev alone.
* docs(extensions): clarify PAT rejection on contributed routes
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
|
7cbcbef52c
|
docs(extensions v0.2.1): add a ten-page extension developer manual (#5769)
* docs(extensions): add an eleven-page extension developer manual
Add a harness/extensions/ section in English and Chinese documenting the
deerflow-extension-api 0.2.3 contract:
index when to write an extension vs a tool, MCP server or
skill; contribution kinds; loading; failure and
trust model; versioning
quick-start build, test, install and remove a working extension
runtime load sequence, API version rules, scopes and
ExtensionData, fail-open and cancellation, diagnostics
middleware placements and where each lands, scope, ordering,
observe-only wrap hooks, failure isolation
observers task lifecycle, system model calls, agent assembly
and fingerprint, context compaction
services-and-routes service lifecycle, router rejection rules, auth,
principals
run-evidence service and request-scoped readers (#5727), cursor
semantics, deletions, redaction, store differences
plugins experimental full-stack plugins, inline modules and
manifest-based browser assets (#5685)
operations extension manager CLI, sources, upgrade, rollback,
Docker, Helm, recovery
troubleshooting indexed by the exact log and error strings
reference every public name and the contract version history
Also fix stale descriptions: the contribution kinds listed in AGENTS.md
and deerflow/extensions/AGENTS.md (now eight, including plugins), the
run-evidence redaction claim (only auth_token is removed), and the plugin
mount return value in docs/full-stack-plugins.md ({ dispose }, not a
function). Changelog entries added in both languages.
* docs(agents): trim root AGENTS.md to keep inherited chains under the size limit
* docs(extensions): scope the manual to the 0.2.1 contract
Plugins and request-scoped run evidence are not in 2.1.x-dev yet; they
move to a stacked follow-up so this change can be cherry-picked.
* docs(extensions): clarify PAT rejection on contributed routes
|
||
|
|
7d1aa00136
|
docs(subagents): restructure the subagent docs into an eleven-chapter user manual (#5761)
* docs(subagents): restructure the subagent docs into an eleven-chapter user manual
Replace the single harness/subagents.mdx page (en + zh) with a
harness/subagents/ section of eleven chapters per language:
1. index concepts, delegation flow, inheritance, terminology
2. quick-start Ultra mode, task card, results, stop behaviour
3. catalog built-ins, config.yaml / managed sources, precedence,
Custom Agent delegation scope, ACP agents
4. delegation task parameters, snapshot context, acceptance criteria,
batch_task, skills / MCP / uploads inside a subagent
5. results terminal statuses, stop_reason, report contract,
receipt verification, acceptance checklist, ledger
6. limits every limit with key / default / range / behaviour,
runaway guards, subagent_runtime capacity
7. sandbox leases, per-subagent shell sessions, MAX_SHELL_SESSIONS,
middleware chain, execution isolation
8. observability task card, SSE + persisted events, metadata keys,
token attribution, Langfuse, trace ids, batch API
9. troubleshooting symptom-indexed FAQ with the fixing PRs
10. developers create_deerflow_agent, SubagentRuntime, contracts,
extensions, security boundaries, background registry
11. reference config keys, context keys, tool signatures, enums,
events, routes, June-September 2026 change log
The old page becomes the section index (asIndexPage), so existing links to
/docs/harness/subagents keep working; the two anchor links in
middlewares.mdx now point at limits#runaway-guards. Every chapter compiles
with @mdx-js/mdx and the docs link tests pass. Changelog entries added in
both languages.
* docs(subagents): fix the docs build and align the manual with the code
Remove the `index` key from both subagents `_meta.ts` files. The index page
is marked `asIndexPage`, so Nextra treats it as the folder itself; listing it
as a child failed `_meta` validation and returned 500 for every docs page.
Correct claims that disagreed with the backend, in both languages:
- GET /api/subagents is open to all users; only writes need an admin
- subagents use their own subagents.token_budget, not the Lead Agent's
- warn_threshold injects a model-visible warning, not just a log line
- [SUBAGENT LIMIT REACHED] and subagent_limit_capped fire only when the
per-run total was already exhausted before the response
- per-response concurrency defaults to subagent_runtime.max_running
- batch tools are not registered when a supplied runtime lacks a batch
service
- ask_clarification / present_files are default denies that a config.yaml
agent can lift
- smaller fixes to result text formats, event payloads, batch item states,
MAX_SHELL_SESSIONS handling, and UI labels
The changelog entry now says only page-level links survive the split.
|
||
|
|
31bfd40f38
|
chore(deps): bump langgraph-checkpoint to 4.2.0 / postgres 3.1.2 and delete the saver patch (#5734)
* chore(checkpoint): bump to checkpoint 4.2.0 and drop the stale saver patch langgraph-checkpoint 4.2.0 fixes the write dropped after a full -> delta migration (upstream langchain-ai/langgraph#8526) and keeps its own InMemorySaver override, so the compatibility patch in checkpoint_patches.py could neither detect the fix nor stand down: it read langgraph's version while InMemorySaver ships in langgraph-checkpoint, and it would have kept shadowing the upstream implementation on 4.2.0. The postgres release 3.1.2 locates plain-value delta seeds (upstream #8535). - floor langgraph-checkpoint>=4.2.0,<5.0 in the harness and raise the postgres extra to >=3.1.2,<3.2; langgraph's own >=4.1.0 would otherwise let a resolver reintroduce the bug now that the patch is gone - delete the patch, its version guard, and the three implementation-assertion tests; test_full_to_delta_migration_replays_on_same_thread stays as the gate and fails on 4.1.1 without the patch (verified by a one-off unpatch run) - langgraph stays 1.2.9 and langgraph-checkpoint-sqlite stays 3.1.1 Verification: make lint; delta checkpointer/cache/worker-resume suites green with the migration gate passing on memory and sqlite (postgres params skip without TEST_POSTGRES_URI). The full make test run before the packaging-pin fix below was 13 failed / 18523 passed; 12 of those failures reproduce on main with the same environment (git stash baseline, identical failure sets), 7 of them because the local config.yaml runs checkpoint_channel_mode: delta. The remaining one, the postgres-extra string pin in tests/test_checkpointer.py, is updated here. Upstream status for the surrounding gaps: cache parity across off/cold/warm states, postgres pagination (langchain-ai/langgraph#8448) and parallel-superstep replay order (#8382) are still open upstream, as is the abandoned-branch fork fix (#8548) that the delta resume linearization in runtime/runs/worker.py works around, so that path stays. * docs(changelog): keep checkpoint upgrade in Unreleased --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
90f0866fa8
|
fix(uploads): stop deleting a converted companion we cannot prove we wrote (#5673)
* fix(uploads): stop deleting a converted companion we cannot prove we wrote Conversion names a document's Markdown companion after the document's stem and falls back to a _N suffix when that name is already taken, so the .md beside a document may belong to another document sharing the stem, or to the user. Delete removed it anyway: uploading a.docx and a.pdf produces a.md and a_1.md, and deleting a.pdf destroyed a.docx's companion while orphaning a.pdf's own. Delete now removes only the file it was asked to remove. The companion stays listed and can be deleted on its own. Orphans are the cost of not guessing; issue #5672 covers giving companions a provable owner, which is what a safe cleanup needs, along with the two related readers that still guess (the outline injected for a document and the agent's file listing). convertible_extensions loses its last use and is dropped from the signature and both call sites. The gateway router keeps importing CONVERTIBLE_EXTENSIONS for the ingestion bridge and now declares it in __all__, where that module documents its re-exports. * docs(changelog): note that delete keeps the converted markdown (#5673) |
||
|
|
29d285731b
|
fix(uploads): convert the bytes we wrote, not the name they landed under (#5611)
* fix(uploads): convert the bytes we wrote, not the name they landed under Document conversion re-opened the upload by name after it was already visible in the thread's uploads directory: the Gateway converted the committed file_path, and DeerFlowClient converted the copy it had just placed there. That directory is writable from local and AIO sandboxes, so a process watching it can replace the name with a symlink in the window between the upload landing and the converter opening it. The converter then reads whatever host file the link points at and writes that content back into the thread as the .md companion, which the sandbox can read. Reproduced end to end on both paths with a real xlsx: the companion came back holding the host file's rows. The Gateway now duplicates the descriptor of the staged file before the link-commit, copies those bytes into a private directory outside the uploads tree, and converts there. A descriptor cannot be redirected by replacing a name, so the conversion input is the content this request wrote. The client converts the caller's own source file instead of the copy in uploads; the source is the file the caller handed in, which the sandbox cannot reach. Both already wrote the companion without following a symlink, so only the read side changes. The uploads directory still receives exactly the same files. * docs(changelog): note upload conversion source fix (#5611) * fix(uploads): close the conversion descriptor when staging its copy fails Review follow-up. The private directory for the conversion copy was created before the try that owns the duplicated descriptor, so a failure there — a full or unwritable temporary filesystem — propagated without closing it. The upload's own cleanup only unlinks the committed name and releases the sandbox lease, so the descriptor stayed open for the life of the process and kept the unlinked staged bytes allocated with it; repeated failures accumulated both. Directory creation now happens inside that try, and the finally removes the directory only once it exists. * fix(uploads): keep the conversion descriptor owned across cancellation Review follow-up. run_file_io cannot interrupt its worker, so cancelling the await around os.dup only abandoned the result: the duplicate was created moments later with nothing left to close it, and it pinned the staged bytes of an upload whose name the cleanup had already unlinked. Cancellation after the duplication was just as leaky, because the commit-path handler caught Exception and CancelledError is not one. The duplication now runs as its own task, shielded from the caller's cancellation, and closes its own result when the caller is gone by the time the worker finishes. The commit path catches BaseException, closing the descriptor it already owns before re-raising. Both windows are pinned: one test stalls the duplication worker after it allocates and cancels ingestion, the other stalls the commit so the cancellation lands while the descriptor is owned. * fix(uploads): drain the conversion copy so its descriptor always closes Review follow-up. The copy worker owns the duplicated descriptor and closes it in its own finally, but a bare await let a cancellation cancel the executor job while it was still queued: the worker never ran, so that finally never ran either, and the enclosing scope had already handed ownership away and saw None. Draining also keeps a late worker from writing into a private directory this scope has since removed. The copy now goes through await_drained, the shield-and-drain helper the Gateway already uses for offloads that must not be abandoned mid-flight. Pinned by a test that holds the copy job queued, cancels ingestion, then releases it and requires the descriptor to come back closed. |
||
|
|
c52ad191f4
|
fix(subagents): recognize empty regular files in remote acceptance pr… (#5559)
* fix(subagents): recognize empty regular files in remote acceptance probes * test(subagents): address empty-file acceptance review feedback * docs(subagents): condense empty artifact guidance --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
2b6254f76d
|
fix(client): stop embedded uploads from writing through symlinks (#5578)
* fix(client): stop embedded uploads from writing through symlinks DeerFlowClient.upload_files copied each file with shutil.copy2 and let convert_file_to_markdown write the companion straight into the uploads directory. Local and AIO sandboxes can write to that directory, so a symlink planted at an upload name or at the companion's name was followed: the upload's bytes and the converted Markdown landed in whatever host file the link pointed to, and the call reported success. The Gateway refuses symlinked destinations and the IM channels write through write_upload_file_no_symlink; the embedded client never adopted either. Uploads now go through copy_upload_file_no_symlink, a new helper next to write_upload_file_no_symlink. It keeps copy2's content, permission bits and timestamps, so files stay readable to Docker sandboxes, but applies them to the descriptor opened with O_NOFOLLOW and opens the source first so a missing source cannot truncate an existing upload. As in the Gateway, a file with an unsafe destination is skipped and listed in skipped_files, success turns false, and the message says how many were skipped. The companion is converted inside a private temporary directory and then written with write_upload_file_no_symlink; one whose name is unsafe is left out like a failed conversion, and the original upload is kept. * docs(changelog): note embedded upload symlink fix (#5578) * fix(client): keep copy2's same-file guard and companion permissions Review follow-up. Two regressions in the previous commit. copy_upload_file_no_symlink opened the destination before comparing it with the source, and that open truncates. Passing a file that already sits in the thread's uploads directory therefore copied an emptied file over itself: the upload reported success with size 0 and the original bytes were gone, where copy2 raised SameFileError and left the file alone. The destination is now compared with the source through os.path.samestat before anything is opened, so identity — including a hardlink or another spelling of the same path — raises SameFileError as before. The Markdown companion was published with write_upload_file_no_symlink, which creates a new file as 0600 and ignores the converted file's mode. Under umask 022 the companion became 0600 while its own document stayed 0644, so a bind-mounted sandbox running as another uid could read the upload but not the Markdown the response advertises. It now goes through the same copy helper as the upload, which preserves the converter's permission bits. |
||
|
|
492e2ac2cc
|
fix(sandbox): report an exactly-full search result as complete in the remote providers (#5534)
* fix(sandbox): report an exactly-full search result as complete in the remote providers `glob` and `grep` decide `truncated` twice: once for the raw output cap (`parse_remote_search_output`, unchanged) and once for `max_results` after the Python-side filters have run. The second decision returned as soon as `max_results` matches had been collected, which cannot tell a search that held exactly that many from one that held more — a tree holding exactly `max_results` eligible matches came back flagged as cut off, and the tool then told the model the result was incomplete. These providers hold the whole listing (the raw stream is capped at `max(max_results * 4, max_results + 50)` lines and reports its own cut-off), so like AIO's `glob` branches they can look one match past the cap before deciding: `AioSandbox.grep`, plus `glob`/`grep` in E2B, OpenSandbox, Tenki and BoxLite now use the same `len(matches) > max_results` rule. This completes what #5449 started for AIO's `glob`; the local provider's half is #5491. Co-Authored-By: Claude Code <noreply@anthropic.com> * fix(sandbox): let remote grep see one match past the per-file cap E2B and OpenSandbox stopped each file's grep at max(max_results, 50) matches, so a single file holding more than max_results hits — with a raw stream far below its limit — ended the Python loop exactly at the cap and reported the result as complete (#5534 review). Retain one extra match per file so the one-match lookahead can observe the overflow and report truncation. A single-file regression at max_results=50 covers 50 matches (complete) vs 51 (truncated) for both providers. Co-Authored-By: Claude Code <noreply@anthropic.com> --------- Co-authored-by: Claude Code <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
f9f3127dc1
|
fix(uploads): delete the requested upload, not a symlink's target (#5547)
* fix(uploads): delete the requested upload, not a symlink's target delete_file_safe resolved the requested path before unlinking it. The uploads directory is writable from local and AIO sandboxes, so a symlink planted under an upload name was followed: deleting alias.pdf removed the victim.pdf it pointed to, and the companion cleanup then removed victim.md, while the link itself survived and the call reported "Deleted alias.pdf". A link resolving outside the directory was already refused by the traversal check, so the damage stayed inside the thread's uploads. The function now checks and unlinks the requested entry itself and treats a symlink as not found, the same way list_files_in_dir already hides it. unlink() never follows the final component, so a file swapped for a link between the check and the unlink removes only the link. Tests cover the helper, the Gateway DELETE route, and DeerFlowClient.delete_upload. * docs(changelog): note upload delete symlink fix (#5547) |
||
|
|
114b78d7db
|
test(persistence): cover historical run-change repair and rollback (#5518)
* fix(persistence): repair run-change clock schema skipped by the 0023 insertion 0023_run_change_seq was chained ahead of the already-shipped 0023_user_preferences revision, so databases stamped at that revision or later treat it as an applied ancestor and never execute it: the run_change_clock table and runs.change_seq column are permanently missing and the first thread deletion fails with 'no such table: run_change_clock' (#5516). 0025_repair_run_change_seq re-applies the same guarded DDL on upgrade and no-ops on healthy shapes. RunChangeClockRow and UserPreferenceRow are also registered in the ORM model registry. Fixes #5516 * fix(persistence): preserve run-change schema when rolling back repair --------- Co-authored-by: 1553126902 <1553126902@qq.com> |
||
|
|
d540be7e21
|
fix(frontend): gate tool-step links through the href scheme allowlist (#5526)
* fix(frontend): gate tool-step links through the href scheme allowlist The chain-of-thought renderer turned web_fetch args and web_search / image_search result URLs straight into <a href>. Markdown links already pass isSafeHref, but these tool-step links bypassed it, so a prompt-injected tool call could put file:, ms-msdt:, vscode: or other OS protocol-handler links into the chat. React 19 only rewrites javascript: hrefs. All three sites now reuse the markdown allowlist and render an unsafe URL as plain text (the image thumbnail stays, unlinked). Tests render MessageGroup for each tool with unsafe schemes plus a web-URL control. * docs(changelog): note tool-step link scheme gating (#5526) * fix(frontend): mark omitted tool-step links and guard web_fetch url type Review follow-up. Tool steps dropped an unsafe URL to bare text, while markdown and artifact links show a dotted "Unsafe link omitted" span, so the two surfaces applying the same rule degraded differently. That span was already duplicated between markdown-link.tsx and artifact-link.tsx; it is now one UnsafeLink component used by all three renderers. It passes extra props through so the image tile still works as a Radix tooltip trigger. web_fetch also read args.url with a cast only. A non-string url (models occasionally emit one mid-stream) reached JSX as an object and threw, taking down the message list. It is now typeof-guarded. * fix(frontend): default missing tool-call args before rendering steps Review follow-up. The web_fetch typeof guard dropped the optional chaining of the cast it replaced, so a tool call without an args object threw again. Other branches were already exposed the same way: seven tool kinds (web_fetch, web_search, image_search, read_file, write_file, str_replace, browser_*) threw on a missing or null args while building their labels. convertToSteps now defaults args to {} once, so every ToolCall branch receives an object. |
||
|
|
db6130861d
|
fix(persistence): repair run-change clock schema skipped by the 0023 insertion (#5517)
0023_run_change_seq was chained ahead of the already-shipped 0023_user_preferences revision, so databases stamped at that revision or later treat it as an applied ancestor and never execute it: the run_change_clock table and runs.change_seq column are permanently missing and the first thread deletion fails with 'no such table: run_change_clock' (#5516). 0025_repair_run_change_seq re-applies the same guarded DDL on upgrade and no-ops on healthy shapes. RunChangeClockRow and UserPreferenceRow are also registered in the ORM model registry. Fixes #5516 Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
9f79ddf9b6
|
fix(nginx): allow model-bound /api/ and /api/skills requests past 60 seconds (#5524)
* fix(nginx): allow model-bound /api/ and /api/skills requests past 60 seconds Two locations were left on nginx's 60s default while the routes behind them wait on Gateway. The /api/ catch-all carries the stateless POST /api/runs/wait, which blocks on wait_for_run_completion and cancels its run when the client disconnects, so a caller waiting on a longer run got a 504 and lost the run; it also carries POST /api/input-polish, which waits for a one-shot model call. /api/skills carries POST /api/skills/install, which runs one LLM security scan per file in the archive, and the custom-skill edit and rollback routes, which run one more each. None of them sets an application-level timeout, and only the sibling /api/skills/install/upload endpoint had been given the longer timeout, so the same archive failed at 60s depending on which endpoint installed it. Allow 600s on both locations, matching /api/langgraph/ and /api/threads, in all three copies of the nginx config. Each directive is pinned by its own test that parses the active directive per config. * docs(changelog): link the nginx /api/ and /api/skills timeout entry to #5524 |
||
|
|
3cfc9c58fd
|
fix(nginx): allow model-bound /api/threads requests past 60 seconds (#5505)
* fix(nginx): allow model-bound /api/threads requests past 60 seconds The browser calls /api/threads/* directly rather than through /api/langgraph/, and the generic `location ~ ^/api/threads` block set no proxy_read_timeout, so nginx's 60s default applied. /compact and /suggestions hold the response open for a whole model call, and /runs/wait for a whole run. Past 60s nginx returned 504 mid-work: the compaction still committed behind the failed request, and /runs/wait cancelled its run on the disconnect (on_disconnect defaults to cancel). Allow 600s on that location, matching /api/langgraph/, in all three copies of the nginx config: Docker, local dev, and the Helm ConfigMap. The regression test parses the active directive per config, so a missing, commented-out, lowered, or misplaced timeout fails. * docs(changelog): link the nginx /api/threads timeout entry to #5505 --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
849d4910ad
|
docs(changelog): complete the 2.1.0 milestone entries in both languages (#5520)
Every merged pull request in milestone 2 (765, confirmed by paginating the GitHub GraphQL milestone query) now has an entry in CHANGELOG.md, and the Chinese mirror carries the same 119 new blocks with its reference block rebuilt to match. The cited sets are identical between the two files (962 each, the extra 197 being pre-2.0.0 history), with no orphan and no unused reference in either direction. Section skeletons match too: 42 headings in the same order with the same nesting. Two merge defects were fixed while splicing the Chinese entries, both of which would have corrupted the file silently. Anchors were searched across the whole English block list, so a walk could cross a `### ` boundary and land on a bullet from the previous section -- four `新增/调度器` entries were about to be filed under Breaking Changes; anchors are now confined to each entry's own section and subsection, with an assertion at merge time. And the one entry whose Chinese counterpart already narrated the same pull request without citing it had its replacement inserted at the first line the merge then deleted, so the deletion pass discarded the new text and kept the old block; the splice is now a single forward pass over original line indices. Verification lives in /tmp/zhwork (temporary): translation validation, the splice, and a final pass asserting milestone coverage, reference hygiene, cited-set parity, section agreement for all 119 blocks, and line widths. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
582a632868
|
fix(agents): key read_file loop detection on its exact line window (#5486)
Layer 1 quantized read_file's line range into 200-line buckets, which erased the offset inside a bucket: every read shorter than a bucket collapsed onto its neighbours. Five sequential 40-line reads hashed identically and tripped the hard stop, ending the run with a forced final answer and stop_reason=loop_capped — on exactly the ranged reads that read_file's own truncation notice tells the model to make. Bucketing cannot separate progress from repetition in general: an equality key can only approximate range overlap, and the approximation was erasing the offset that distinguishes the two. Key on the exact window instead, with an omitted end_line kept open-ended so a bare read and an explicit start_line=1 still share one key. Repeating a single range is still caught at the same threshold, and a read loop that varies its bounds remains covered by the per-tool frequency layer. Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
53798b44cd
|
fix(subagents): scale max_turns into the graph's super-step budget (#5485)
* fix(subagents): scale max_turns into the graph's super-step budget
max_turns was handed to LangGraph as recursion_limit, but the two count
different things. recursion_limit counts super-steps, one per graph node,
and create_agent compiles a node for every middleware lifecycle hook, so
one turn costs before_model + model + after_model + tools nodes — seven to
eight through the subagent chain. The built-in general-purpose agent's
max_turns=150 therefore bought about 18 tool-using turns before failing as
turn_capped, and every middleware added to the chain shrank the effective
budget again.
Resolve the limit from the chain each subagent was actually assembled with
(subagents/turn_budget.py) instead of passing the turn count through, so
raising max_turns buys the turns it names.
No config keys or defaults changed; existing max_turns values now grant
their full budget, bounded as before by subagents.timeout_seconds and
subagents.token_budget.
* fix(subagents): warn when a counted hook can jump the agent loop
Review follow-up. The per-turn cost is a flat multiplier over the straight
before_model -> model -> tools loop. A hook that declares can_jump_to and
returns {"jump_to": ...} re-enters the loop without traversing tools,
spending another before_model + model + after_model pass that buys no tool
result, so the resolved limit becomes a lower bound rather than an exact
budget — silently re-creating the short budget this translation fixes.
Measured against a compiled graph: with one jumping after_model hook,
three tool turns need the resolved limit plus one jump pass, and the run
raises GraphRecursionError at the resolved limit.
How often a jump fires is data-dependent and unbounded, so it cannot be
folded into the arithmetic. find_jumping_hooks reports the condition off
the same __can_jump_to__ attribute the factory reads, and the executor
warns when a counted hook declares one. Nothing in today's subagent chain
does, so this changes no budget.
* fix(subagents): detect jumps declared on agent-level hooks too
Review follow-up. find_jumping_hooks exempted before_agent/after_agent on
the grounds that a jump out of them lands in the loop the budget already
pays for. That does not hold on langchain 1.3.14:
- after_agent jumps re-enter the loop after it finished, and the hook runs
again on the next exit, so the extra passes are unbounded. Even
jump_to "end" is routed to exit_node, the head of the after_agent chain,
so it reruns the chain; destinations are no safe filter.
- a before_agent hook that stages a tool call and jumps to tools runs a
tools step no model turn paid for. It is O(1), but the resolved limit
has zero headroom, so one step caps the last turn.
The detector now scans every hook pair the factory wires jump edges for.
The compiled-graph pin is parametrized over after_model->model,
after_agent->model, after_agent->end and before_agent->tools, each raising
GraphRecursionError at the resolved limit and completing once the jump's
cost is added. No middleware in the subagent chain declares a jump on any
hook, so this changes no budget.
|
||
|
|
b0cb3a3a8c
|
fix(scheduler): serialize the SQLite launch-budget claim (#5469)
* fix(scheduler): serialize the SQLite launch-budget claim claim_queued_run counts the executing occurrences and then promotes one row to launching. Postgres serializes that pair with a transaction advisory lock; SQLite had no counterpart. pysqlite does not begin a transaction for a SELECT, so the budget count ran in autocommit and the deferred transaction reserved the writer only at the promoting UPDATE. Claimers racing on distinct rows therefore read the same stale count, each passed its own status == 'queued' CAS, and max_concurrent_runs was exceeded. A manual trigger overlapping the poller reaches this concurrently within one process, and scheduler.multi_instance over a shared database file reaches it across processes. Two claims of the same row were already safe, which is why the existing coverage did not catch it. Take the writer before the count with BEGIN IMMEDIATE, the idiom ThreadMetaRepository already uses for its read-modify-write paths and the same reservation _lock_task makes for a parent row. The claim targets one row but the budget is global, so this has to be the database-wide writer rather than a row lock. * docs(changelog): reference #5469 in the SQLite launch-budget entry * test(scheduler): pin the launch-budget test's connection reuse The warm-up gather is what makes the claimers actually overlap, but it silently depended on the SQLite engine keeping pooled connections. If that engine ever moved to a non-pooling class, every claimer would open its own connection, the per-connection PRAGMA setup would stagger them, and this test would pass against an unserialized claim instead of failing -- the cold-pool case it exists to avoid. Assert that the warm-up left connections checked in. A non-pooling class does not implement checkedin() at all, so a missing counter reads as zero reuse and reports the same explanation rather than an AttributeError. Verified against NullPool: the guard fails with "NullPool left 0 connections pooled after the warm-up". Only pool_size connections survive the gather (the overflow is discarded), which is why the pre-fix failure is exactly five claimants over a cap of one rather than eight. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
6bab87aca4
|
fix(sandbox): report an exactly-full AIO glob result as complete (#5449)
* fix(sandbox): report an exactly-full AIO glob result as complete AioSandbox.glob's include_dirs branch returned as soon as it had collected max_results matches, without looking at the rest of the listing. A listing that held exactly that many matches and nothing more was therefore reported as truncated, and the glob tool told the model the result was incomplete — prompting a re-search or distrust of a complete answer. The same line returned one match for max_results=0, one past the caller's cap. Look one match past the cap before deciding, which is what the include_dirs=False branch in the same function already does and what #5427 moved parse_remote_search_output to for BoxLite, Tenki, E2B and OpenSandbox. * review: filtered-tail cases, the glob contract docstring, and the cap wording Addresses the three items from the review on #5449. - Two regression cases over a tail of ignored / out-of-root / pattern-miss entries: an exactly-full result stays complete when only filtered entries follow, and a third eligible match after that tail still reports truncation. Both fail against the previous return-on-the-max-th-match behaviour. - 'Sandbox.glob' promised the conservative flag ('``max_results`` was reached') that this change deliberately stops producing on the AIO branch. The contract now reads as 'may be incomplete' and records that providers differ in how precisely they can decide it. - The changelog no longer lumps 'parse_remote_search_output' in with the filtered-match cap: its raw-output cap is a separate limit with its own one-line-past accounting, and the other providers' filtered-match cap is unchanged. Also corrects the docstring on the existing test, which still described the removed early return in the present tense. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
f7f4a022e6
|
fix(agents): remove provider tool-call blocks when guards strip calls (#5447)
* fix(agents): remove provider tool-call blocks when guards strip calls Token-budget and loop-detection hard stops, subagent-limit truncation, and safety-finish-reason suppression removed calls from tool_calls and the raw additional_kwargs payload, but left the provider's own tool-call blocks in AIMessage.content. Provider adapters re-serialize those blocks: langchain_anthropic sends a tool_use block whose id is not in tool_calls, and the OpenAI Responses input builder sends every function_call block. ChatAnthropic stores any tool-calling response as a block list, so a guard firing on a Claude tool call always left a tool_use without a tool_result. A truncated subagent call failed the next model request of the same run; a hard stop was checkpointed under the same message id and failed every later turn of the thread. clone_ai_message_with_tool_calls now trims content tool-call blocks to the calls that remain on the message: tool_use and LangChain v1 tool_call/tool_call_chunk by id, Responses function_call and custom_tool_call by call_id (their id is the fc_ item id), Google GenAI function_call by id, and id-less blocks by name in order. Blocks for calls still on invalid_tool_calls stay, because DanglingToolCallMiddleware answers those calls with placeholder results. The token-budget and loop-detection hard stops now build their messages through the helper instead of their own copies, and ClarificationMiddleware drops its private filter, which matched Responses blocks by item id. * docs(changelog): reference #5447 in the orphaned tool-call block entry * fix(agents): skip id-matched calls in the id-less block budget The name budget for id-less content tool-call blocks counted every retained call, including calls whose own id-bearing block had already matched. In mixed-shape content, a retained call "a" with a function_call block carrying id "a" also let a same-named id-less block survive, leaving the unpaired block this helper exists to remove. Collect the retained ids that id-bearing blocks matched first, and build the name budget only from retained calls outside that set. Content with no id-bearing blocks keeps the full budget, so the Gemini path is unchanged. |
||
|
|
6469833886
|
fix(skills): close SkillScan bypasses in the skill review gate (#5431)
* fix(skills): close SkillScan bypasses in the skill review gate The public skill review gate re-materialized a package snapshot into a temp directory for SkillScan, but copied only entries the reader had decoded as text and skipped every file under any evals/fixtures/ directory. Executable binaries and nested archives never reached the package rules, and a fixture-shaped path hid any script from the scan. Readers now keep binary bytes as content_base64, the analyzer writes every non-symlink file byte for byte, and only eval fixture SKILL.md samples stay exempt. Files are created exclusively, so a duplicate archive member or a case-folded name fails the scan closed instead of overwriting an earlier file. SkillScan itself skipped any file that was not NUL-free UTF-8. One Latin-1 byte in a comment hid a reverse shell from the review gate, and a NUL byte skipped static analysis at install. Code files that fail strict decoding now raise package-undecodable-script (HIGH) and are analyzed over a lossy decode, so CRITICAL matches keep blocking. "Code file" and "executable magic" were defined separately in the installer and SkillScan and had drifted: SkillScan missed 32-bit little-endian and fat Mach-O variants the installer blocks. Both rules now live in skills/package_files.py, shared by the installer, the export guard, and SkillScan. * docs(changelog): link the skill review gate fix to #5431 * fix(skills): fail closed on bytes-less snapshot entries and skip text rules for executables The review analyzer skipped any snapshot entry it could not turn into bytes. Readers only emit such entries for oversized files, and they also mark the snapshot truncated, but content_base64 is optional in the contract, so a reader regression or a hand-built snapshot would silently drop a file from SkillScan. An entry without bytes now fails the scan closed (not_assessed: skillscan) unless the snapshot is truncated, and a text entry without content no longer materializes as an empty file. A real executable under scripts/ is a code file, so SkillScan decoded it lossily and ran the text rules over its string tables. An OpenSSH binary produced a CRITICAL secret-private-key finding from the key-format banner it embeds. An undecodable file with executable magic still reports package-undecodable-script, and its CRITICAL package-executable-binary finding already blocks it, so it now skips the text rules. Decodable files keep full text analysis. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
ed986a10ef
|
fix(sandbox): report truncated remote glob and grep results (#5427)
* fix(sandbox): report truncated remote glob and grep results
BoxLite, Tenki, E2B, and OpenSandbox run find/grep in the sandbox, cap
the raw output with `| head`, and then filter those lines in Python:
ignored directories such as node_modules are dropped and grep's glob
scope is applied. They reported truncated only when max_results matches
survived the filter. When the capped lines were mostly filtered out, a
search with real matches past the cap came back short or empty with
truncated=False, and glob_tool/grep_tool rendered it as "No files
matched" / "No matches found". With the default max_results=200 and
1,200 files under node_modules, glob("**/*.py") reported no matches for
a workspace that has src/app.py.
remote_search_command now lets one line past its limit through, and
parse_remote_search_output(..., limit=) returns RemoteSearchOutput(text,
truncated): the first `limit` lines and whether the extra line arrived.
Exactly `limit` lines stays a complete result. Each provider passes the
cap it already computed to both calls and returns that truncated from
glob and grep when fewer than max_results results survive filtering.
The glob and grep tools now describe an empty truncated result as
incomplete instead of reporting no matches, which also covers AIO grep's
forwarded truncated flag. Sandbox.glob/grep document truncated as "the
matches may be incomplete".
* docs(changelog): reference #5427 in the remote search truncation entry
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|