* fix(projects): delete read-only documents when purging trash
A permanently deleted project document could never be purged on Windows:
the read-only file attribute makes `Path.unlink` fail with `PermissionError`
(`WinError 5`), where POSIX needs only a writable parent. A restored archive
or a mode-preserving sync sets that attribute, and because the purge re-raises,
the row rolls back and the trashed document keeps coming back with the same
error on every retry.
Clear the attribute and retry the unlink once, mirroring the retry that
`config/paths.py::_clear_read_only_tree` already performs for thread
directories. Any other unlink error still propagates, so the row stays
retryable.
* fix(projects): harden trash purge permission retries
---------
Co-authored-by: Aling33 <along308@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(runtime): backfill run.terminal on confirmed recovery takeover
* fix(runtime): isolate recovery terminal publication failures
* fix(runtime): fence recovery terminal publication lifecycle
* fix(runtime): unify recovery receipt publication
* docs(runtime): document lifecycle guard lock constraints
* fix(runtime): reuse event transaction for recovery guards
Read Run lifecycle through the active event session when SQL stores share an engine, preserving deletion fencing without a second pool checkout. Keep custom lookups and different engines on their ordinary read path. Scope memory recovery guards and deletions to per-thread locks. Add SQLite/PostgreSQL pool pressure, stale identity, fallback, and independent-thread regressions, and document the contracts.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* Track DingTalk stream message tasks to avoid dangling tasks
The stream loop handed each frame to client.background_task() via a
bare asyncio.create_task(...) whose result was dropped. asyncio only
keeps a weak reference to a task once it awaits, so the task could be
garbage-collected mid-execution (silently dropping a message) and any
exception it raised was never observed.
Keep a strong reference to each handler task and discard it via a done
callback, matching the background-task tracking used elsewhere (e.g.
the subagent executor and the MCP session pool).
* fix(channels): observe DingTalk stream task failures
---------
Co-authored-by: lisry-1 <lisry-1@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): make doctor.py --help print usage without running diagnostics
scripts/doctor.py never parsed its arguments: `python scripts/doctor.py
--help` printed no usage and ran the whole diagnostic suite (subprocess
probes, config resolution, provider checks) and exited 0; any unrecognized
flag was silently ignored the same way. Closes#7386.
- Parse arguments with argparse.ArgumentParser in the __main__ entry point:
--help prints usage and exits 0, unknown flags exit 2.
- The no-argument invocation used by `make doctor` is unchanged, and main()
stays directly callable from the existing unit tests (they run in-process
under pytest, whose argv must not reach argparse).
- Add backend/tests/test_doctor_script.py: offline subprocess cases for
--help (usage, no diagnostics run), an unknown flag (exit 2), and the
no-argument run (report footer present, exit 0 or 1).
* test(scripts): isolate doctor CLI subprocess checks
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(readability): preserve <a> links when Readability.js silently degrades on Windows
On Windows, simple_json_from_html_string(use_readability=True) invokes the
Readability.js subprocess, which succeeds without error but returns the
unextracted document instead of the extracted article. The pure-Python
fallback (use_readability=False) also drops every <a> destination. The net
effect is that web_fetch returns link-free Markdown on Windows.
Fix: after extraction, check whether the output retained the <a> elements
that the resolved HTML carried. When they are all gone, fall back to
converting the resolved HTML directly with markdownify — which preserves
<a> as [text](href) — so links survive regardless of the Readability.js
outcome.
Measured: 17 of 22 test_web_fetch_relative_links.py failures resolved;
43/48 pass on Windows (5 remaining are pre-existing malformed-markup
extraction tests, unrelated to this change).
Closes#6459
* test(readability): add degraded-extraction and scope-pinning tests
* test(readability): detect unwanted article fallback
* test(readability): classify HTML body in authority-tag census
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(uploads): never truncate an existing upload on same-name collision
Same-name uploads silently destroyed the earlier file on disk: the shared
write path (open_upload_file_no_symlink) opened the destination with
O_WRONLY|O_CREAT (no O_EXCL) and ftruncate(0), so re-uploading report.pdf
opened and emptied the report.pdf already stored for the thread, and two
concurrent same-name uploads could race on one inode. The Gateway
ingestion pipeline already refuses overwrites (staged no-overwrite link
commit), but the embedded client upload path (copy_upload_file_no_symlink)
and the channel ingress paths (write_upload_file_no_symlink) still went
through the truncating open.
Open every upload destination with O_EXCL. On FileExistsError, claim the
next free _N suffix against the directory's current contents via the
existing claim_unique_filename helper and retry — the same deterministic
behavior the Gateway pipeline documents (name_1.ext). The existing file is
never opened, so its bytes survive intact under any future naming policy.
Embedded-client and channel inbound-file responses now report the name the
bytes actually landed under instead of the requested name.
* test(uploads): cover concurrent same-name writers
* fix(uploads): skip known copy collisions before staging
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Document that a failed write, fsync, or replace in
_publish_bytes_atomically leaves the previous file intact, and that
temporary removal must not replace that error, including interruptions.
Add site-specific regression tests for unlink and warning-log failures.
The cleanup_failed_atomic_write call itself already landed on main in #7200.
Co-authored-by: wannT-1 <200890097+wannT-1@users.noreply.github.com>
* fix(extensions): keep atomic-write cleanup from replacing the save error
Extension snapshot rollback, the managed plugins rewrite, and skill-archive
upload copies now use cleanup_failed_atomic_write so a failed unlink or
warning log cannot mask the original publication error.
Co-authored-by: wannT-1 <wannT-1@users.noreply.github.com>
* fix(skills): keep upload and storage cleanup from replacing the save error
Route the skill-archive upload discard, custom-skill file writes, and
per-user skill-state writes through cleanup_failed_atomic_write. Close the
extension rollback descriptor on fdopen failure, and record the plugins
temporary path before the write so a short write is still removed.
Co-authored-by: wannT-1 <wannT-1@users.noreply.github.com>
* fix(skills): keep form close and snapshot delete from masking the save error
Discard the uploaded skill archive before closing the multipart form, and
suppress close errors so a flush failure cannot replace the install outcome.
Snapshot rollback of a missing original now uses cleanup_failed_atomic_write,
and skill-state writes clean up on BaseException including KeyboardInterrupt.
Co-authored-by: wannT-1 <wannT-1@users.noreply.github.com>
* test(extensions): assert logging cleanup cannot replace the delete-branch error
The log-handler check only ran for the unlink parameter, where the handler
is never patched. Move it under the logging failure so a handler exception
must stay out of the surfaced install error.
Co-authored-by: wannT-1 <wannT-1@users.noreply.github.com>
---------
Co-authored-by: wannT-1 <200890097+wannT-1@users.noreply.github.com>
Co-authored-by: wannT-1 <wannT-1@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor(helm): derive the postgres Secret name from one helper
`databaseUrlSecret` re-encoded the `postgresql.external.existingSecret` then
`postgresql.existingSecret` chain in its own `if`/`else if`. Three consumers
read it — the gateway's `DATABASE_URL`, the postgres StatefulSet's
`POSTGRES_PASSWORD`, and `postgres-secret.yaml`'s own `metadata.name` — so a
source added to one helper and not the other would make the gateway and the
StatefulSet resolve different Secrets. That is the class of bug #7149 fixed
for the redis pair.
Extract `databaseUserSecret` as the single encoding of the chain and let
`databaseUrlSecret` fall back to the chart-managed name when it is empty,
mirroring `redisUserSecret` / `redisUrlSecret`.
Pure refactor: the rendered manifest is byte-identical before and after across
the four configurations (chart-managed, `existingSecret`,
`external.existingSecret`, and both set), compared with `helm template`.
Adds `test_helm_database_secret_precedence.py`, which pins that both consumers
name the same Secret and that only the chart-managed branch renders a Secret.
Dropping either precedence source fails the two cases that depend on it.
Suggested in review of #7161.
* fix(helm): keep the postgres Secret helper's truthiness
Review follow-up on the previous commit. Two things were still wrong:
`postgres-secret.yaml` kept its own copy of the precedence chain for the
render gate, so "the only place the chain is encoded" was not yet true — a
source added to `databaseUserSecret` alone would have left that gate empty
and rendered an unreferenced `<release>-postgres` Secret. It now reads the
helper, like `redis-secret.yaml` already did.
`or` returns its last operand even when it is falsy, so
`postgresql.existingSecret: false` (how people spell "not set") named the
Secret `false` instead of falling back to the chart-managed one. Chaining
`default` keeps the old if/else-if truthiness and yields an empty string
for any falsy value; the gateway's `$pgConfigured` now reads the same
helper rather than re-listing the two keys.
Two falsy cases join the parametrize list: `false` in the lower-precedence
slot and in both slots. Both fail against the previous `or` version with
`assert [] == [False]` — the Secret literally named `false` — so they are
regressions this test can actually catch. The other four configurations
render byte-identically before and after.
* fix(helm): keep the redis user-Secret helper's truthiness
Same defect the postgres helper had, one release later: `redisUserSecret`
used `or`, which returns its last operand even when falsy, so
`redis.existingSecret: false` named both the gateway's `redis-url` ref and
the StatefulSet's `redis-password` ref after a Secret called `false`. The
render gate in `redis-secret.yaml` saw the string "false" as truthy, so it
skipped creating the chart-managed Secret — leaving two dangling refs and
a StatefulSet that cannot start.
`default` chaining restores the pre-`redisUserSecret` if/else-if behaviour:
any falsy value yields an empty string, so the chart-managed branch is
taken. `redisConfigured` now reads the helper instead of re-listing the two
keys, matching the gateway's `$pgConfigured`.
Verified against the four string-valued configurations (chart-managed,
`existingSecret`, `external.existingSecret`, both): renders are
byte-identical before and after. The two new falsy cases fail on the
previous version, and `--basetemp` makes all 14 cases in this file pass.
* docs(helm): drop a stray article in the helper comment
* fix(helm): honor redis-password from a user-managed Secret
The bundled Redis StatefulSet derived its AUTH mode only from
`redis.auth.password`, so a `redis.existingSecret` that carried both
`redis-url` and `redis-password` left the gateway authenticating against
a server started without `--requirepass`. Both probes used the no-auth
path as well, so the Pod reported healthy while the gateway could not
connect to it.
Whether a user-managed Secret carries `redis-password` is only knowable
at runtime, so the key is now read with `optional: true` and the
entrypoint enables AUTH only when the value is nonempty. A Secret
carrying only `redis-url` keeps the no-auth behavior instead of failing
to start, and a stale `redis.auth.password` no longer wins over the
selected Secret, matching what values.yaml documents.
Refs #7149
* test(helm): cover Redis Secret authentication and share resolution
Add the issue #7149 render-and-shell regression suite for chart-managed,
user-managed and external Redis credentials. Resolve Secret precedence once
for the Gateway, bundled server and chart-owned Secret, and clarify how
optional user-managed passwords select AUTH at runtime.
Regression coverage adapted from dakjdakd's commit
7e20745d678276ed33e8f8eafd2c576fa8ae9aec.
Co-authored-by: dakjdakd <1428823446@qq.com>
* test(helm): assert the stale value instead of an unused constant
`test_bundled_redis_uses_optional_existing_secret_password` checked the
module-level `PASSWORD` constant against the rendered manifest, but that
constant is never an input to any of its six renders — the parametrized
value is `ignored_password`. The guard could therefore never fail, so a
template edit that started rendering `redis.auth.password` alongside a
user-managed Secret would have gone unnoticed.
Assert the parametrized value instead: it is the stale input the case is
meant to keep out of the manifest. Verified the value is detectable in a
render that does carry it (the chart-managed path renders it twice), so
the assertion is no longer vacuous.
Also fix the garbled sentence in `tests/AGENTS.md` that described this
coverage as "stale values passwords".
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: dakjdakd <1428823446@qq.com>
* fix(harness): complete Lark device auth via a single-flight login run
The web authorization polling loop reused the request wait budget as the
lark-cli auth login subprocess lifetime: each /lark/auth/complete poll ran
the CLI with subprocess timeout=wait_timeout_seconds (8s from the frontend)
and killed it on timeout. The login transaction is not atomic — poll, redeem
the single-use device code, fetch user info, then persist credentials — so a
kill after redemption but before persistence stranded the flow, and the next
poll with the same code failed with 'device_code is invalid'.
Own the login process by the flow generation instead: the first poll spawns
one background run per (user_id, generation) and later polls attach to it;
wait_timeout_seconds only bounds the caller's wait (unchanged 504 contract)
and never terminates the CLI. Advancing the generation or re-issuing a device
code abandons and kills the orphaned process, and a finished run's outcome
stays attachable for 120s so late polls return the same result instead of
re-consuming the redeemed code.
* fix(lark): tolerate vanished credential-tree entries and bound owner-alive grace
Publication could kill the login worker with an unhandled FileNotFoundError
when a concurrently-written temp file vanished between iterdir() and lstat(),
leaving the marker live but no result file ever published; every foreign poll
then blocked 504 until the 600 s attempt bound, and could respawn an
already-redeemed code. Also, a bare-pid owner_alive check trusted for the
full 600 s let a recycled-pid or crashed-thread marker outlive any process
that could still publish.
- _credential_tree_path_kind returns "gone" on FileNotFoundError so the
walk skips vanished entries instead of aborting; the same tolerance is
applied to iterdir().
- _write_lark_login_result_once treats the re-harden as best-effort: a
filesystem hiccup is logged and publication continues so the marker sweep
still runs.
- The marker now records owner_pid_started_at and
_read_live_lark_login_marker_locked only keeps an owner-alive marker
within a 10 s publication grace, so a stale marker is swept and the flow
respawns instead of wedging every foreign poll.
Regression: test_walk_and_harden_tolerates_a_vanished_entry,
test_write_result_once_continues_when_hardening_fails,
test_abandoned_marker_with_dead_owner_is_swept_within_grace.
Focused suites: 139 passed, 12 skipped.
* fix(lark): harden surviving credentials through filesystem churn
Atomic login publication can remove a descendant after lstat but before
chmod. Catch FileNotFoundError around the entire per-entry walk so that
lstat, chmod and directory enumeration tolerate those disappearances.
A vanished root, permission error or symlink remains a hard failure.
Exercise real removals at each boundary and concurrent result publication;
assert that injection ran and surviving directories/files reach 0700/0600.
The regressions detect a skipped walk, swallowed permission failures and
loss of directory-churn tolerance in isolated mutation runs.
* fix(lark): retain login ownership through durable completion
Browser approval can take longer than the publication grace. Start that
grace at the first observed CLI exit, then reject abandoned attempts
without retrying a device code that may already have been redeemed.
Serialize atomic result publication under the credential lock. Preserve
published results through cleanup failures and retain terminal failures
on the marker when result storage fails. Never bypass credential hardening.
Reserve storage before spawning and roll back failed marker, process or
thread startup by reaping the CLI, clearing the registry and waking waiters.
Add process/filesystem failure regressions, a long-login two-worker case,
and strict unhandled-thread gates. Fix invocation logging in both workers
and reap forked test CLIs after the parent observes their launch. Document
the authorization lifecycle and required regression assertions.
Validated the reproduced failures and ten isolated mutations; final
focused, blocking-I/O, full-suite and independent review details are in PR #7182.
* fix(channels): pin JSON state encoding across platforms
* test(channels): check encoding pins at each state I/O call
---------
Co-authored-by: huyengiang101086-spec <270684046+huyengiang101086-spec@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): keep staging cleanup from replacing the save error
The channel stores publish state through a staged temporary file and, if the
rename fails, close and unlink that file inside the handler before re-raising.
Neither step was guarded, so a failing close skipped the unlink and a failing
unlink replaced the error that actually caused the save to fail -- leaving the
staged copy, which for the runtime-config and WeChat stores holds an 0600 IM
token, behind with nothing reporting it.
Make both steps independently best-effort, name the leaked file in a WARNING,
and suppress the diagnostic itself so a broken log handler cannot become the
reported failure. This is the same contract PR #6871 establishes for browser
captures, applied to the three JSON publishers in app/channels.
* fix(channels): address staging cleanup review follow-ups
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(infoquest): reject invalid extracted fetch content
* fix(infoquest): address fetch response review feedback
Preserve reader_result precedence so malformed primary content is reported instead of silently masked by a secondary field. Include request URLs in invalid/empty content diagnostics and reject non-object JSON before field selection.
* test(infoquest): respect URL redaction in fetch diagnostics
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(tests): pass Windows base system env to isolated pytest child
The live-test policy regression tests spawn a pytest child with a minimal,
from-scratch env. On Windows that env lacks SystemRoot/TEMP/USERPROFILE, so
the child dies during import asyncio (WinError 10106, winsock provider init)
and tempfile cannot resolve a directory, making the tests fail for
environment reasons instead of testing the opt-in policy.
Extract the env assembly into _child_pytest_env() and pass through a small
allowlist of Windows base system variables (SystemRoot, windir, COMSPEC,
PATHEXT, TEMP, TMP, USERPROFILE) when present. These carry no credentials,
so the isolation contract (no API keys, no real .env) is unchanged, and
POSIX behavior is byte-identical. A new unit test pins the contract: only
the allowlisted base vars may pass through, and SystemRoot must be present
on Windows.
* test(live-policy): seed Windows base env vars in the bootability contract
The contract test asserted TEMP/TMP pass-through without seeding them, so
on Windows hosts that do not export those optional variables (CPython then
falls back to %USERPROFILE%\AppData\Local\Temp) it failed on host
configuration rather than on the helper's contract.
Seed the representative base variables with monkeypatch and assert the
forwarded set exactly, so the isolation contract is deterministic on every
host. Red/green evidence: with TEMP/TMP stripped from the environment the
old assertion fails (extra = {COMSPEC, PATHEXT, SystemRoot, USERPROFILE,
windir}) while the seeded assertion passes.
Addresses the P3 finding in the review of PR #6439.
* ci(tests): cover Windows live-test policy
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(community): reserve Browserless capture names before publishing
Name selection and publication were two separate operations:
`_dedupe_output_name` probed with `Path.exists()` and then
`publish_bytes_atomically` ran `os.replace`, so two `web_capture` calls in the
same thread outputs directory could both settle on `report.png` before either
wrote, and the second publication replaced the first capture while both reported
success for the same artifact path (#6883). `web_capture_tool` runs the writer
through `asyncio.to_thread`, and same-second default names collide the same way
because the generated stem's timestamp has second resolution.
Claim the name instead of checking for it: an `O_CREAT | O_EXCL` open either
creates an empty placeholder or reports the name held, so the probe walk
(-1, -2, ... then the microsecond timestamp) is race-free. `_write_capture_output`
releases the placeholder when publication fails, catching BaseException so an
interruption also frees the name, and a failing unlink is logged rather than
allowed to mask the original error - the same error-masking family #6871 is
addressing in the shared publisher, which this change deliberately does not
touch.
When even the timestamped names are held, the function now raises instead of
returning a name it cannot own; returning one would reintroduce the silent
overwrite. `tests/test_browser_outputs_atomic.py` pins the publisher call as
source text, so its expectation moved to the new call shape in the same commit.
Refs #6883
* fix(community): open the reserved capture name with new-file permissions
An `os.open` without an explicit mode creates the file with `0o777 & ~umask`, and
`publish_bytes_atomically` preserves an existing destination's mode, so the
reservation added in the previous commit made every capture executable (`0o755`
under the usual umask) where main produced `0o666 & ~umask` (`0o644`) through the
publisher's new-file branch. Pass `0o666` explicitly and pin the mode against a
`Path.write_bytes` reference in the same directory, which is the permission the
publisher documents for new files.
Refs #6883
* fix(browserless): warn when capture reservation cleanup fails
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scripts): feed config-upgrade inline python on stdin, not argv
On Windows setups where python/uv resolve through shim layers that
reparse arguments with cmd.exe (e.g. pyenv-win), the multi-line
`python -c "..."` payloads in config-upgrade.sh get mangled mid-flight
(observed as `IndentationError: unexpected indent` on a stray
`|| goto :error` line) and `make dev` fails before the Gateway starts.
Feed both inline scripts on stdin instead, which is safe across every
spawn chain, and load the checkout .env explicitly inside the resolver
probe: the default load_dotenv() discovery walks up from cwd/__file__,
which a stdin-fed script (`python -`) cannot satisfy. Pin both with a
static regression test.
* test(config): isolate upgrade fixtures from inherited dotenv selectors
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): bound viewed image context per model request
Keep recent image payloads within a count and byte budget, and move re-viewed paths to the newest position. Add focused regressions for budget enforcement and state ordering.
* test(agents): verify cumulative image budgets and re-view recovery
* fix(agents): clarify image omission and read-attempt limits
* docs(middleware): shorten image context guidance for CI budget
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(client): convert the captured upload source
* test(uploads): cover ACL sources and scope capture failures
* fix(uploads): preserve concurrent uploads from one captured source
Combine the embedded client's source capture with no-replace publication for
first-time filenames and Markdown companions. Hold one private capture across
collision retries, retain original descriptor metadata, and keep replacement
behavior for names present when the call starts.
Port concurrent-upload and staging-publication regressions from PR #6687.
Extend source-edit/removal and real XLSX cases to both publication paths, and
cover edits or removal during two filename collisions in ordinary and event-loop
calls. Document capture, retry, publication, and cleanup contracts.
Validation: related suites 469 passed, 24 skipped; full offline suite 28,788
passed, 488 skipped; blocking-I/O suite 245 passed, 1 skipped; Ruff, guidance,
and diff checks passed. Nine race/collision cases failed on the #6632 baseline.
Co-authored-by: haotianqi666 <250384991+haotianqi666@users.noreply.github.com>
* docs: keep agent guidance within CI budgets
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: haotianqi666 <250384991+haotianqi666@users.noreply.github.com>
* fix(uploads): convert the same source version the upload saved (#6600)
DeerFlowClient.upload_files copied the caller's file into the uploads
directory, then reopened the original path for Markdown conversion. A
save landing between the two reads left the saved upload at version A
while the companion registered version B, and resolve_companion still
accepted the mismatched Markdown.
Snapshot convertible sources into a private temp dir up front and run
both the upload copy and the conversion from that one read. The snapshot
keeps the #5611 invariant (conversion never reopens a name inside the
sandbox-writable uploads dir), preserves metadata via copy2, and keeps
the re-upload SameFileError rejection through an explicit same-file
probe before the snapshot.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* fix(uploads): pin the snapshot copy to one held source descriptor
Review on #6621 found two gaps in the snapshot step: stat-by-name then
copy2-by-name split identity from content across two path resolutions,
and copy2 drops ACLs so a source readable only via ACL made the private
snapshot unreadable. Open the source once, fstat it for the same-file
rejection, and copyfileobj out of the held fd; restore the mtime from
that same stat so the upload keeps the source's timestamps.
Tests: the version-parity regression's injection now also fires when a
future regression reads the caller's path during conversion, and a real
macOS ACL case (mode 000 + read grant) uploads and converts cleanly.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* test: catch upload conversion source regressions
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(lark): publish the skill file and manifest atomically
_write_skill_file and _write_manifest wrote their final paths with a truncating
write, while the runtime files in the same module already go through
_publish_bytes_atomically. A partial skill file or manifest is unparseable, and
the manifest carries the content hash the install is verified against, so a
half-written one cannot be repaired. Route both through the existing helper.
* fix(lark): address skill publication review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* test(local-dev): normalize the upgrade script's MSYS path rendering on Windows
nginx-local-conf.sh computes its repo root with pwd -P, which inside
Git Bash yields the MSYS mount form (/tmp/... for the Windows temp dir)
— correct for its bash-native consumers (serve.sh, nginx.sh) but never
equal to the test's Windows-form Path.resolve(). Map the rendering back
through cygpath -w before comparing, so the tests read and compare the
rendered config on Git Bash hosts too.
Fixes#6657
* test(local-dev): harden Git Bash path normalization
---------
Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(community): add You.com web search provider
Adds You.com as an opt-in community `web_search` engine, mirroring the
existing community engines (tavily, brave, exa, firecrawl, groundroute).
You.com runs its own web index and returns extracted page snippets per
hit rather than a single meta description, so a research loop gets usable
context without a follow-up fetch.
The provider picks its endpoint by whether a key is configured, so it
works out of the box and scales up when the operator adds one:
- no key -> /v1/agents/search, keyless free tier
- YDC_API_KEY -> /v1/search with X-API-Key, full Search API
Endpoint and headers are always chosen together; the keyless endpoint
rejects an auth header, so a stale key can never be sent there. Both
return the same body shape, so one normalizer covers both.
httpx only, no new dependency and no pyproject extra, so uv.lock is
untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(community/youcom): review follow-ups — drop stray .pr-body.md, guard OverflowError, add 429 hint
* fix(doctor): make the You.com free-provider detail true with or without YDC_API_KEY
The doctor detail returned before any key check, so operators who had
set YDC_API_KEY still saw 'set YDC_API_KEY for the full Search API'.
Reworded to match the sibling free-provider entries and stay accurate in
both states.
* fix(search): sweep youcom max_results onto the shared owner
The single-owner gate that landed with #6563 flags this branch's private
_coerce_max_results as a newly hand-rolled copy, so backend-unit-tests
now fails after merging main. Follow the coordination note from review:
delegate to deerflow.community.search_max_results.coerce_max_results,
which owns the rejection profile (booleans, non-integral floats,
OverflowError), and keep only the 1-100 mirror of You.com's server-side
count clamp on top — clamping policy stays per-provider.
The .inf regression test now goes through the public tool path and also
pins the bool and non-integral-float shapes of the shared bar.
* docs(tools): use the bare you.com platform link in config and docs
Match every other provider entry: no other provider in config.example.yaml
or the en/zh tools.mdx uses campaign-tagged URLs, so drop the utm params
from the three you.com/platform links. Integration attribution stays in
the identifying User-Agent sent to You.com hosts.
* test(youcom): cover shared max_results ownership and bounds
Register You.com among the shared-owner providers and assert default handling and the 100-result cap through outbound request parameters.
---------
Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: valueadd-lgtm <339269770+valueadd-lgtm@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(skills): record the content actually overwritten in custom-skill history
Scope:
- backend/app/gateway/routers/skills.py: move the prev_content read of
update_custom_skill and the current-content read of rollback_custom_skill
into their drained persist tails (read -> write -> append_history in one
worker thread, no awaits in between). The blocked-rollback branch now
builds its informational entry separately. No behavior contract changes:
status codes, cancellation draining, and prompt-cache refresh order are
untouched.
- backend/tests/test_skills_custom_router.py: two regression tests that
inject a concurrent write during the scan phase and assert the history
entry's prev_content equals the revision actually overwritten on disk.
Why: prev_content was read before the security scans, so a concurrent
revision landing between that read and the persisted write was overwritten
on disk but absent from the history chain, making it unreachable by a later
rollback and desynchronizing the audit record.
Tests: uv run pytest tests/test_skills_custom_router.py --basetemp=... ->
36/36 passed (34 existing + 2 new); ruff check and ruff format --check clean
on both files. Python 3.12.13, Windows 11.
* fix(skills): serialize custom-skill mutations and pin the concurrency regression to the real window
- Add a module-level mutation lock shared by the edit, delete, and rollback
paths, covering the full read -> write -> history-append sequence so a
concurrent PUT cannot land between a request's predecessor read and its
write (previously the projection lock started inside write_custom_skill,
leaving that window open).
- Replace the two regression tests: they now pause the first mutation inside
its storage write after the predecessor read (deterministic gate), complete
a second real HTTP PUT inside that window, and assert the history chain
stays replayable. Verified both tests fail against the base implementation
(2 failed) and pass with the lock (2 passed); full file 36 passed,
ruff check + format clean.
* fix(skills): scope the custom-skill mutation lock to the owning user
The lock serializing the read -> write -> append sequence was one process-wide
lock, and that sequence wraps write_custom_skill's entire user projection
rebuild. A user waiting on their own projection lock, or copying a large or
slow linked package, therefore stalled every other user's edit, rollback, and
delete of unrelated custom skills, even though their skill trees, histories,
and projection roots are disjoint. The base storage only serialized those
mutations per user projection.
Key the lock on the canonical owning-user storage root instead, so one user's
mutations remain a single critical section — including across separately
constructed storage objects, which a hot-reloaded config or a later request
creates — while two users never share it.
The read -> write -> append sequence still cannot interleave within a user, and
the lock is still acquired and released inside the worker thread's own with
block, so caller cancellation cannot strand it.
* fix(skills): serialize custom history across gateway processes
---------
Co-authored-by: sjr666666 <sjr666666@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): number remote read_file line ranges like LocalSandbox
E2B, BoxLite, Tenki and OpenSandbox sliced ranged reads with str.splitlines(), which also ends lines at a bare "\r", \f, \v, \x1c-\x1e, \x85, \u2028 and \u2029. LocalSandbox.read_file, count_file_lines and the read_file truncation marker end lines only at "\n", so on those providers a ranged read of such a file returned different lines, and following a truncation marker's start_line went back into text already shown. Add split_file_lines next to count_file_lines and use it in the four providers.
* test(sandbox): separate remote and host read paths
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor(community): give max_results coercion a single owner
#5865 audited every bundled provider's max_results coercion, and #5866 /
#5867 are fixing seven copies of it. Each one had to be read on its own
because four of the copies were byte-identical apart from the provider
label in their warning line.
Move the shared bar into deerflow.community.search_max_results and have
ddg_search, image_search, fastcrw and firecrawl delegate to it. Warnings
keep their existing text and their existing logger: logger is a parameter
so records keep their module attribution, which
tests/test_ddg_search_tools.py filters on.
image_search also stops spelling its own default twice -- the two tool
signatures hardcoded 5 while the same file defined DEFAULT_MAX_RESULTS.
No bound is added to any provider, and brave / groundroute / serper /
serply / sofya / tencent_wsa are left untouched so #5866 and #5867 land
without conflict. The new test pins the 13-value behaviour, the delegation
and the historical labels, and asserts that the set of providers still
hand-rolling a coercer is exactly the declared six -- so a twelfth copy
fails CI rather than being audited by hand later.
* test(search): cover generic and inline max-results coercion
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(scanner): cache decoded text atomically
_cache_text_file keyed the cache on the virtual path and returned the path, and
the caller serves that path as the file's content, so a write that failed
part-way left a truncated entry that every later read handed back as the preview.
Publish through a temporary file plus os.replace.
* test(scanner): pin the cache entry against a failed publish
The helper-level case cannot distinguish the call site, so this drives the real
entry point with a failing publish and asserts the cached entry is untouched.
* docs(sandbox): keep agent guidance within its size limit
* fix(scanner): preserve publish errors during cleanup
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(deploy): fail fast when DEER_FLOW_HOME is not writable
`make docker-start` bind-mounts the host `backend/` directory into the gateway
container, whose process runs as root, so any state it creates under
`backend/.deer-flow` ends up owned by root on the host. A later `make up` runs
`scripts/deploy.sh` as the invoking user and only fails deep into the run with a
bare "Permission denied" while persisting generated secrets (deploy.sh:234).
Check writability right after `mkdir -p "$DEER_FLOW_HOME"` and, when it fails,
print the exact `chown` recovery command instead of letting the run get halfway
through. `make down` is excluded so stopping the stack is never blocked.
Fixes#6462
* test(deploy): cover runtime home writability preflight
* fix(deploy): diagnose persisted secret permissions and preserve teardown
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* test(auth): make user-list ordering independent of clock resolution
* docs(test): clarify deterministic user-list ordering
Explain the lexical tie-break over stored canonical UUID strings and give
user-repository ordering its own guidance section, separate from router fixtures.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(deploy): let make up start on a checkout without .env files
.env and frontend/.env are gitignored and make up does not create them,
but docker/docker-compose.yaml listed both as short-form env_file entries,
which Compose treats as required. On a fresh clone make up, make down and
make prod-logs all stopped with "env file .../.env not found" (exit 14)
before building or touching anything.
Mark the three entries optional with the long-form path/required syntax
the development compose file already uses. Compose loads them when they
exist and skips them otherwise; an unreadable or non-file .env still
fails. This needs Docker Compose 2.24+, the floor the README already
documents, and deploy.sh's startup-failure hint now names it for older
clients.
deploy.sh deliberately does not seed the files from the examples the way
docker.sh start does: .env.example carries placeholder API keys the
production Gateway would receive, and make config skips files that exist.
* docs(changelog): reference #6370
* test(deploy): gate Compose clients and cover make up
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(agents): bind read marks to the current tool call
* test(agents): make file and proxy baseline checks portable
* fix(agents): address read-mark review diagnostics and cleanup
* docs(agents): keep read-mark guidance within the chain budget
* fix(models): exclude thinking blocks from Claude cache breakpoints
* fix(models): declare Claude test SDK and clarify caching docs
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
