* fix(gateway,integrations): keep temp cleanup from replacing the caller's outcome
* fix(gateway): adopt the family helper so a published edit stays silent
Document that a failed write, fsync, or replace in
_publish_bytes_atomically leaves the previous file intact, and that
temporary removal must not replace that error, including interruptions.
Add site-specific regression tests for unlink and warning-log failures.
The cleanup_failed_atomic_write call itself already landed on main in #7200.
Co-authored-by: wannT-1 <200890097+wannT-1@users.noreply.github.com>
* fix(uploads,integrations): keep atomic-write cleanup from replacing the caller's outcome
* fix(lark): keep the download error when archive cleanup fails
* test(lark): keep the download fixtures inside the test directory
* fix(skills): sniff executable binaries past leading filler bytes
The executable-binary guard (package_files.is_executable_binary_prefix)
matched ELF/PE/Mach-O magics strictly at file offset 0 across all three
layers: archive preflight, the extraction guard, and the directory static
scan. Prepending a single filler byte (newline, space, NUL, ...) evades
every layer simultaneously: a whitespace-prefixed ELF still loads through
ld.so, and the existing test asserting the scanner and the extraction
guard agree passes for both the plain and the padded form after this fix,
because both layers share this helper.
The sniff now tolerates a bounded (64-byte) run of leading ASCII
whitespace/NUL filler before matching, and scan windows widen from 8 to
EXEC_SNIFF_BYTES so the magic plus filler always fits the probe. The cap
keeps indented prose out of the false-positive zone (the existing
"full magics only" tests stay green).
Repro before the fix: archive with SKILL.md and assets/payload whose
bytes are b'\n' + ELF header: preflight passes, extraction writes the
file, the directory scan reports nothing.
Tests: 10 new regression cases across test_skillscan_native.py and
test_skills_installer.py; 701 related tests pass locally on py3.12.
* fix(skills): clarify executable filler sniffing
Address review feedback by clarifying the offset-0 probe rationale, handling UTF-8 BOM prefixes without truncating Mach-O magics, and pinning the 63/64-byte boundary.
* test(skillscan): describe what the filler-prefix regression asserts
The docstring said filler "must not evade the offset-0 probe", which reads
oddly for a test whose payloads all sit past offset 0. State the asserted
behavior instead: a bounded filler run still yields a blocking CRITICAL
finding.
* test(skillscan): pin executable sniff window and BOM handling
* test(skillscan): cover every executable sniff entry point
* test(lark): isolate windows runtime branch
* fix(harness): complete Lark device auth via a single-flight login run
The web authorization polling loop reused the request wait budget as the
lark-cli auth login subprocess lifetime: each /lark/auth/complete poll ran
the CLI with subprocess timeout=wait_timeout_seconds (8s from the frontend)
and killed it on timeout. The login transaction is not atomic — poll, redeem
the single-use device code, fetch user info, then persist credentials — so a
kill after redemption but before persistence stranded the flow, and the next
poll with the same code failed with 'device_code is invalid'.
Own the login process by the flow generation instead: the first poll spawns
one background run per (user_id, generation) and later polls attach to it;
wait_timeout_seconds only bounds the caller's wait (unchanged 504 contract)
and never terminates the CLI. Advancing the generation or re-issuing a device
code abandons and kills the orphaned process, and a finished run's outcome
stays attachable for 120s so late polls return the same result instead of
re-consuming the redeemed code.
* fix(lark): tolerate vanished credential-tree entries and bound owner-alive grace
Publication could kill the login worker with an unhandled FileNotFoundError
when a concurrently-written temp file vanished between iterdir() and lstat(),
leaving the marker live but no result file ever published; every foreign poll
then blocked 504 until the 600 s attempt bound, and could respawn an
already-redeemed code. Also, a bare-pid owner_alive check trusted for the
full 600 s let a recycled-pid or crashed-thread marker outlive any process
that could still publish.
- _credential_tree_path_kind returns "gone" on FileNotFoundError so the
walk skips vanished entries instead of aborting; the same tolerance is
applied to iterdir().
- _write_lark_login_result_once treats the re-harden as best-effort: a
filesystem hiccup is logged and publication continues so the marker sweep
still runs.
- The marker now records owner_pid_started_at and
_read_live_lark_login_marker_locked only keeps an owner-alive marker
within a 10 s publication grace, so a stale marker is swept and the flow
respawns instead of wedging every foreign poll.
Regression: test_walk_and_harden_tolerates_a_vanished_entry,
test_write_result_once_continues_when_hardening_fails,
test_abandoned_marker_with_dead_owner_is_swept_within_grace.
Focused suites: 139 passed, 12 skipped.
* fix(lark): harden surviving credentials through filesystem churn
Atomic login publication can remove a descendant after lstat but before
chmod. Catch FileNotFoundError around the entire per-entry walk so that
lstat, chmod and directory enumeration tolerate those disappearances.
A vanished root, permission error or symlink remains a hard failure.
Exercise real removals at each boundary and concurrent result publication;
assert that injection ran and surviving directories/files reach 0700/0600.
The regressions detect a skipped walk, swallowed permission failures and
loss of directory-churn tolerance in isolated mutation runs.
* fix(lark): retain login ownership through durable completion
Browser approval can take longer than the publication grace. Start that
grace at the first observed CLI exit, then reject abandoned attempts
without retrying a device code that may already have been redeemed.
Serialize atomic result publication under the credential lock. Preserve
published results through cleanup failures and retain terminal failures
on the marker when result storage fails. Never bypass credential hardening.
Reserve storage before spawning and roll back failed marker, process or
thread startup by reaping the CLI, clearing the registry and waking waiters.
Add process/filesystem failure regressions, a long-login two-worker case,
and strict unhandled-thread gates. Fix invocation logging in both workers
and reap forked test CLIs after the parent observes their launch. Document
the authorization lifecycle and required regression assertions.
Validated the reproduced failures and ten isolated mutations; final
focused, blocking-I/O, full-suite and independent review details are in PR #7182.
* fix(lark): retry the Windows install lock instead of failing after ~10s
msvcrt.LK_LOCK gives up after roughly ten seconds and surfaces contention
as Permission denied, while a real install (download + setup of the
lark/sandbox CLI) can hold the lock far longer. Retry the non-blocking
acquisition on the Windows fallback branch so concurrent Gateway workers
serialize instead of failing, mirroring the extensions manager's lock
introduced in #6020. The POSIX flock branch is unchanged.
Closes#6879
* test(lark): inject msvcrt stub with raising=False for Linux runners
The regression test patches the module-level ``msvcrt`` attribute, which only
exists on Windows (the POSIX branch imports ``fcntl`` instead), so
``monkeypatch.setattr`` with the default ``raising=True`` raised AttributeError
on Linux CI (backend-unit-tests shard 4).
Pass ``raising=False`` so the stub is injected on POSIX runners: ``fcntl`` is
patched to ``None`` first, and ``_exclusive_install_lock`` then takes the
Windows fallback branch against the fake. Windows behavior is unchanged.
Scope: backend/tests/test_lark_cli_install_lock.py, 1 line.
Verification: pytest tests/test_lark_cli_install_lock.py on Windows
(Python 3.12, pytest 9.0.3) -- 3 passed incl. the real-contention case;
Linux CI rerun is the authoritative check.
* fix(lark): harden and share advisory lock retries
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(lark): publish the skill file and manifest atomically
_write_skill_file and _write_manifest wrote their final paths with a truncating
write, while the runtime files in the same module already go through
_publish_bytes_atomically. A partial skill file or manifest is unparseable, and
the manifest carries the content hash the install is verified against, so a
half-written one cannot be repaired. Route both through the existing helper.
* fix(lark): address skill publication review feedback
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
_write_lark_cli_runtime_from_archive and _write_lark_cli_sandbox_launcher wrote
their file with a truncating open and chmod 0o755 afterwards, so a crash mid-write
left an executable fragment behind - for the extracted binary that file can be
large. Publish through a temporary file plus os.replace, as the flow-state writer
in the same module already does.
* fix(lark): publish the CLI runtime files atomically
install_shim wrote the shim, the launcher and the runtime marker with a
truncating open. The sandbox chmods the launcher 0o755 and execs it, so a crash
mid-write left an executable fragment, and the marker is parsed as JSON. Publish
through a temporary file plus os.replace, as the storage backends already do.
* style: drop the unused import ruff flags
* fix(lark): publish runtime permissions atomically
Set file permissions on temporary files before os.replace() to ensure
the launcher, shim, and runtime marker are atomically published with
correct permissions (0o755 for executables, 0o644 for the marker).
This addresses reviewer feedback from PR #6591:
- Permissions are now set before the atomic replace operation
- Prevents brief window where files have incorrect permissions
- The runtime marker now has mode 0o644 instead of 0600 (mkstemp default)
Also updates test_lark_shim_atomic.py:
- Fixes test_atomic_helper to match new function signature
- Adds test_install_shim_publishes_correct_permissions to verify
launcher/shim (0o755) and marker (0o644) permissions
Co-authored-by: IronMurphy <hepushuo@gmail.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(integrations): decode lark-cli and npm output as UTF-8
Every text-mode subprocess.run in lark_cli.py that captures lark-cli or npm
output omits encoding, so the bytes are decoded with the host locale. Both
write UTF-8 to a pipe: the @larksuite/cli npm package's bin is a Node wrapper
around a native binary that emits UTF-8 (lark-cli --help writes U+2014 as
E2 80 94), and npm output is UTF-8 too.
On a host whose ANSI code page is not UTF-8 (cp936, cp1252) the locale decode
silently mangles every non-ASCII field -- auth status --json returns a garbled
userName, so the status card shows mojibake -- and an undecodable byte raises
inside subprocess's reader thread, which is not re-raised: stdout comes back
as None or "", so a healthy CLI is reported as unavailable and a successful
run raises "lark-cli did not return JSON output."
Decode these captures as UTF-8, matching #5440 (PowerShell), #5904 (AIO
sandbox docker CLI) and #6548 (Git Bash output in LocalSandbox). The
whoami /user capture stays on the locale: whoami is a native tool that writes
the host code page, the same UTF-8/native split #6548 drew.
Tests drive the real functions with a stand-in CLI emitting UTF-8 CJK JSON
and pin the pre-fix decode path, so they fail on main and pass here on any
host locale.
Fixes#6589
* test(lark): isolate the paths singleton in the encoding suite
The new tests reached ensure_lark_cli_credential_tree through the global
get_paths() singleton, so every run created and hardened a real
<cwd>/.deer-flow/users/alice/integrations/lark-cli/{config,data} tree in
whatever directory pytest ran from -- the same runtime home a live deployment
keeps real per-user Lark credentials in. An autouse fixture now points the
singleton at tmp_path, matching what test_lark_cli_integration.py does.
Red/green is unchanged: reverting only lark_cli.py still fails the two
assertions that pin the decode, and the suite leaves no .deer-flow behind.
* test(lark): make the version probe pin the decode, drop the inert locale patch
Two test-rigor notes from review:
- test_probe_lark_cli_at_path only asserted available/error, and the probe
returns available=True with any non-empty output and a zero exit code, so
the test stayed green even with the encoding pin removed from
_probe_lark_cli_at_path. The stand-in now answers --version with a non-ASCII
payload and the test names the expected text, which fails on the unpinned
call site (verified) and passes with the pin.
- The locale.getencoding patch was inert: the subprocess.run wrapper injects
the codec for unpinned text-mode calls and lark_cli never consults the
locale itself, so nothing reached the real locale-decode path. Dropped it,
along with the now-unused import, rather than implying a redirection that
does not happen.
The Windows shim forwards its arguments (%*) so the stand-in sees --version.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): attest lark broker mode per Pod and quarantine uncertain shell outcomes
Why this change:
Two fail-open paths in the AIO sandbox lifecycle. First, the lark broker
capability probe collapsed "provisioner unreachable" into "broker not
configured": one 1.5s timeout cached a negative for 300s, so new sandboxes
were created without the broker sidecar while the per-user credential
directories were still bind-mounted into the container, and existing Pods
kept being reused without their actual mode ever being read. Second, an
implicit-shell command that timed out or returned an unknown status only
marked the current client object; warm-pool reclaim and rediscovery built a
fresh client that re-entered a container whose shell state could not be
proven clean, and nothing about that ambiguity survived a Gateway restart.
What changed:
- The capability cache stores confirmed observations only, keyed per
provisioner endpoint/auth identity; an unknown answer (timeout, malformed
payload) raises instead of selecting credential placement, while a
confirmed broker requirement is retained through outages. One in-flight
probe per identity shares success and failure, and a Pod-attested mode
contradicting the cache forces one shared refresh.
- Execution uses the mode attested for the actual sandbox:
SandboxInfo.lark_cli_broker flows from the provisioner's atomic Pod
observation into the client, so a probe outage delays drift detection but
never denies reuse of an attested runtime, and never trusts an unattested
one.
- The provisioner reports each Pod's actual broker mode, generation (Pod
UID) and shell capacity from one read, rejects lark requests conflicting
with its configuration (409/503 carrying a capability-refresh marker the
Gateway uses to drop the stale cache entry), rejects conflicting or
Terminating Pods on coordination creates while reporting Terminating as
data to status/get/list, and answers Pod presence independently of
Service readiness.
- A persistent quarantine (DEER_FLOW_HOME/sandbox-quarantine, keyed by
scope/sandbox/generation) fences containers after uncertain implicit
shell outcomes across client rebuilds and restarts; in-process reuse,
warm reclaim, discovery, create responses and orphan reconciliation all
pass one policy gate, and replacement runs under the existing ownership
and teardown fences. A fenced sandbox still tracked after a failed
release-recycle is destroyed inline on the next acquire.
- Quarantine records retire only after confirmed absence, retrying partial
cleanup for the quarantined generation and releasing remembered ports
under the same fences; local stop failures now propagate instead of being
logged away, and remote destroy errors surface to callers.
Problem solved:
A provisioner blip could silently strip broker isolation from new lark
sandboxes for minutes at a time, and ambiguous shell outcomes let corrupted
default shells persist across warm reuse, rediscovery and restarts. Docs
updated: README, aio_sandbox and skills AGENTS.md.
* docs(changelog): record sandbox attestation and quarantine fix (#6436)
* fix(sandbox): preserve active leases and fail closed on unattested broker modes
Concurrent uploads could recycle a healthy AIO runtime while another holder
was creating or cleaning up a command session. Older provisioner responses
could also trigger destructive replacement before failing admission.
- Distinguish pending session creation from ambiguous outcomes and reserve
idle teardown until execution and upload holders finish scope cleanup.
- Reject stale fork lookups and reconcile cancelled acquires without
releasing a sandbox still borrowed by another holder.
- Default Lark broker mode to unverified and preserve unattested Pods until
the provisioner is upgraded.
- Recognize Docker 29 absence diagnostics while matching resource names exactly.
- Preserve Gateway upload errors and cover admission and lifecycle regressions.
* fix(sandbox): scope the unverified-broker refuse path to attested providers
The fail-closed raise in _lark_cli_env_from_runtime treated every Sandbox
without a lark_cli_broker attribute as unverified, but only AioSandbox
carries the attestation — E2B, OpenSandbox, Tenki, and Boxlite have no
broker sidecar, no credential mounts, and no provisioner to upgrade, so
every lark-cli command on those providers failed with an unactionable
error.
Declare the tri-state on the Sandbox base class (defaulting to False,
the only mode brokerless providers have) so the None fail-closed path
stays scoped to AioSandbox's deliberate unverified default.
- Add lark_cli_broker: bool | None = False to the Sandbox ABC, mirroring
the persistent_shell_sessions tri-state pattern
- Add a regression test pinning the non-broker overlay for a provider
without the attribute
* fix(sandbox): close review gaps in broker attestation and teardown lifecycle
Address the latest #6436 review round:
- The provisioner's post-build broker-mode 409 now carries the
capability-refresh marker, so an incompatible-mode conflict surfaced by
idempotent re-create, attestation drift or post-create verification drops
the Gateway's cached capability observation instead of repeating the same
409 for the full negative-cache TTL.
- AioSandboxProvider.destroy() returns the _destroy_tracked outcome: False
when teardown is deferred because an execution/upload holder or a peer
teardown is still active, so callers can no longer mistake a deferred
destroy for a completed one.
- A live replacement under the same logical ID proves the fenced old
generations are absent; retire_replaced_generations prunes only those
stale records so they cannot strand future creates, while the live
generation's fence and unversioned crash fences are never touched.
- SandboxInfo.from_dict keeps a missing lark_cli_broker attestation
unverified (None) instead of silently rehydrating as attested non-broker.
- The restricted-network inspect path shares the strict
_is_no_such_network_error parser, closing the substring loophole for
context-name collisions and wrong-target errors.
* docs(sandbox): condense lease notes to keep the AGENTS.md chain under the size gate
The three lease notes added in 26c307f3 pushed the effective guidance chain
to 98,312 bytes — 8 over the 98,304 hard limit enforced by agent-guidance
(AG002). Fold them into two lines carrying the same rules.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(lark): match broker subcommand denylist through separate option values
The opt-in DEERFLOW_LARK_BROKER_DENY_SUBCOMMANDS check dropped only
`-`-prefixed tokens and compared the rest from the start, so the value in
`--profile work config show` became the leading positional and a
`config show` rule never matched. Real lark-cli 1.0.65 runs `config show`
for that shape and for `config --profile work show`.
The broker cannot know which options take a value, so a rule now matches
when its tokens appear in order among the non-flag tokens. A contiguous
match would still miss a value placed between the path tokens. Argument
values that spell a denied path in order are refused too (fail-closed).
* docs(changelog): reference #6212 for the lark broker denylist fix
* docs(lark): call denylist entries command paths, not prefixes
Entries are matched as in-order subsequences now, so the remaining
'prefix' wording in the BrokerConfig comment, the parse_deny_subcommands
and run_lark_cli docstrings, and the broker README undersold the rule.
Also rewrap the comment and docstring lines this PR had pushed past ~80
columns.
Why this change:
In Lark broker mode the shim forwards piped stdin to the broker only
after EOF. AIO's /v1/bash transport keeps a subprocess stdin pipe open
for writes, so a bare `lark-cli ...` never saw EOF and the shim blocked
forever on input that would never arrive. An earlier brace-group/eval
approach fixed the hang but changed top-level parsing (aliases/extglob)
or let a trailing backslash consume the appended delimiter.
What changed:
- _run_bash_exec prefixes the original command with `exec < /dev/null`
on its own line, giving the fresh, per-invocation session immediate
EOF on default stdin. Explicit pipes/heredocs/files still override
fd0. The plain prefix keeps top-level parsing and appends no
delimiter a trailing backslash can consume.
- The persistent /v1/shell PTY transport keeps commands unchanged; the
shim already treats TTY stdin as no input.
- The shim drains non-TTY stdin under grace/tail idle budgets
(DEERFLOW_LARK_BROKER_STDIN_{GRACE,TAIL}_SECONDS, default 2s, valid
(0, 600]); an idle pipe exits 124 without contacting the broker, and
read errors fail loudly instead of becoming empty input.
- Broker logs each exec as argc/exit/elapsed only (never argument
values) and warns on concurrency-cap rejections.
Problem solved:
Lark commands without explicit stdin hung indefinitely in broker mode,
stalled producers could execute partial writes, and broker logs risked
capturing argument values. Docs updated: README, harness AGENTS.md,
docker/lark-cli-broker README.
* fix(integrations): keep Lark runtime status consistent
Probe sandbox runtime readiness for every Lark mutation response outside the credential lock. Expose explicit sandbox_runtime_probed state so the frontend can distinguish conservative fallbacks without inferring semantics from runtime detail text, while cancelling stale reads before updating the cache.
* style(frontend): format Lark cache test
* fix(integrations): snapshot Lark credential status under the lock
Why this change is needed:
PR #5994 moved the post-mutation status read outside the per-user
credential lock so the new sandbox runtime probe would not extend the
critical section. As flagged in review, that lets a concurrent app
switch or auth flow land between the committed mutation and the status
read; in remote-provisioner mode the runtime probe can spend seconds
before probe_lark_auth runs, so the raced switch can clear the new
tokens and make a completed login report success=false, or mix an old
app_id with a newer auth state.
What changed:
- add _LarkCredentialSnapshot (app config + auth probe) and capture it
in the same _lark_credential_lock section that commits the mutation
- thread the snapshot through _get_lark_mutation_status into
get_lark_integration_status(credential_snapshot=...); only the
credential-independent runtime probe still runs after the lock
- apply to complete_lark_auth, complete_lark_config, and
set_lark_app_credentials; install keeps the one-shot status build
Problem solved:
Mutation responses stay tied to the just-committed credential
generation: a raced app switch can no longer flip a completed
device-code login to success=false or blend a stale app_id with newer
auth state, while the runtime probe stays off the lock.
* fix(integrations): address Lark status review follow-ups
- install_lark_integration: snapshot credential-derived status fields
under _lark_credential_lock so the install response cannot mix newer
credential state read across the unlocked runtime probe
- useInstallLarkIntegration: route the cache write through
cacheLarkMutationStatus so install cancels in-flight reads and keeps
probed runtime fields under frontend/backend version skew
- README: soften the UI claim - the status API exposes
sandbox_runtime_probed and the mutation cache honors it
* test(integrations): tighten Lark status fixtures and guidance docs
- cache.test: use a distinct mutation runtime mode in the unprobed-merge
case so the assertions prove the cached runtime fields win
- e2e mock: include broker in the sandbox_runtime_mode union cast so the
fixture cannot drift from LarkSandboxRuntimeMode
- AGENTS.md (gateway + skills): mention sandbox_runtime_probed alongside
the existing runtime readiness fields, keeping the gateway guidance
file under its byte budget
* fix(integrations): harden Lark mutation cache against skew and races
- cacheLarkMutationStatus: await cancelQueries before writing so an
in-flight GET whose fetch already resolved cannot clobber the fresher
mutation status, matching the existing threads/archive pattern
- self-heal version skew: an unprobed (pre-field backend) mutation
response now triggers a status refetch instead of risking the
conservative runtime fallback lingering until the next focus/remount
- api: normalize a missing sandbox_runtime_probed to false at the
boundary so the declared-non-optional type matches skewed runtimes
- tests: pin the skew contract (missing-field cases for cache and api)
and add a useInstallLarkIntegration regression test proving the
install cache write goes through cacheLarkMutationStatus
- README: describe the probed flag as cross-version semantics instead of
claiming the same-version API surfaces both states
- lark_cli: note that install's credential snapshot runs the auth probe
subprocess while holding the credential lock
* refactor(integrations): derive sandbox_runtime_probed inside the resolver
_resolve_sandbox_runtime_readiness now returns a fourth probed element
next to the branch that produces the unevaluated remote fallback, so the
'was readiness actually evaluated' invariant can no longer drift from the
fallback logic it mirrors. Callers and resolver-level tests updated to
pin probed per branch.
Follows #5801 for the remaining numeric env reads that break on an empty
value (`-e VAR=`, or `${VAR}` with VAR unset):
- The WeChat QR worker guard read a blank GATEWAY_WORKERS as an invalid
count and refused QR login, although the compose command starts one
worker for it (`--workers ${GATEWAY_WORKERS:-1}`) and the other worker
checks treat it as 1. A blank value now falls through to WEB_CONCURRENCY
and then 1, so the guard still fails closed for real multi-worker setups.
- The Lark broker aborted startup on int("") for a blank
DEERFLOW_LARK_BROKER_PORT or DEERFLOW_LARK_BROKER_TIMEOUT; it now uses the
defaults.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(sandbox): platform-aware Lark CLI runtime validation for Windows hosts
The managed Lark CLI sandbox runtime validation and its tests assumed POSIX
semantics that Windows hosts cannot satisfy, breaking the focused AIO/Lark CLI
suites (5 failures on current main).
- _validate_lark_cli_sandbox_runtime keeps the strict executable-bit contract
on POSIX; on Windows it validates the Linux-only artifacts by content
instead (ELF/PE/Mach-O image magic for linux-*/lark-cli, shebang for the
bin/lark-cli launcher), since NTFS cannot represent the exec bit.
- The AIO runtime-mounts test now asserts the explicit Windows credential
contract: an owner-only inheritable DACL (via the existing PowerShell ACL
resolvers) instead of exact 0o700 modes, which remain asserted on POSIX.
- Accept-path runtime tests stage ELF-prefixed payloads so both platforms
exercise realistic artifact content; the extractor's mode assertion is
POSIX-only with a writability check on Windows.
Focused suites: 5 failed / 162 passed -> 167 passed, 3 skipped on Windows 11;
POSIX behavior unchanged (POSIX branches keep the original assertions).
* refactor(tests): share Windows ACL resolvers via a helper module and cover the Windows shebang gate
Review follow-ups on #5442:
- Move the PowerShell ACL resolvers (_windows_acl_env/_windows_acl_sids/
_windows_acl_protected/_windows_acl_owner_sid) from
tests/test_lark_cli_integration.py into tests/_windows_acl_helpers.py,
following the existing shared-helper convention, so the aio suite no longer
imports the full lark-cli integration module (which drags in app.gateway
routers and the FastAPI TestClient at collection time).
- Add test_managed_sandbox_runtime_rejects_launcher_without_shebang_on_windows:
a launcher without a shebang plus ELF-magic binaries, with lark_cli.os
monkeypatched via the existing Windows stub so the shebang-missing reject
branch of _runtime_artifact_is_executable is covered on every platform.
- Comment the rejects-non-executable prestaged-binary test to record that on
Windows the rejection comes from the payload's non-magic content, since
chmod() cannot clear the exec bit there.
Focused suites: 168 passed / 3 skipped on Windows 11.
* fix(skills): close SkillScan bypasses in the skill review gate
The public skill review gate re-materialized a package snapshot into a
temp directory for SkillScan, but copied only entries the reader had
decoded as text and skipped every file under any evals/fixtures/
directory. Executable binaries and nested archives never reached the
package rules, and a fixture-shaped path hid any script from the scan.
Readers now keep binary bytes as content_base64, the analyzer writes
every non-symlink file byte for byte, and only eval fixture SKILL.md
samples stay exempt. Files are created exclusively, so a duplicate
archive member or a case-folded name fails the scan closed instead of
overwriting an earlier file.
SkillScan itself skipped any file that was not NUL-free UTF-8. One
Latin-1 byte in a comment hid a reverse shell from the review gate, and
a NUL byte skipped static analysis at install. Code files that fail
strict decoding now raise package-undecodable-script (HIGH) and are
analyzed over a lossy decode, so CRITICAL matches keep blocking.
"Code file" and "executable magic" were defined separately in the
installer and SkillScan and had drifted: SkillScan missed 32-bit
little-endian and fat Mach-O variants the installer blocks. Both rules
now live in skills/package_files.py, shared by the installer, the export
guard, and SkillScan.
* docs(changelog): link the skill review gate fix to #5431
* fix(skills): fail closed on bytes-less snapshot entries and skip text rules for executables
The review analyzer skipped any snapshot entry it could not turn into
bytes. Readers only emit such entries for oversized files, and they also
mark the snapshot truncated, but content_base64 is optional in the
contract, so a reader regression or a hand-built snapshot would silently
drop a file from SkillScan. An entry without bytes now fails the scan
closed (not_assessed: skillscan) unless the snapshot is truncated, and a
text entry without content no longer materializes as an empty file.
A real executable under scripts/ is a code file, so SkillScan decoded it
lossily and ran the text rules over its string tables. An OpenSSH binary
produced a CRITICAL secret-private-key finding from the key-format
banner it embeds. An undecodable file with executable magic still
reports package-undecodable-script, and its CRITICAL
package-executable-binary finding already blocks it, so it now skips the
text rules. Decodable files keep full text analysis.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(lark): enforce private ACLs on Windows credential tree
On Windows, posix chmod(0o700/0o600) does not map to NTFS ACLs, so the
secret-bearing Lark CLI credential tree was not actually owner-restricted
and existing trees were not repaired.
Branch the permission application by platform:
- POSIX: directories 0o700, files 0o600 (behavior unchanged).
- Windows: disable inherited ACLs, grant the Gateway process user Full
Control (resolved via its SID from whoami /user /fo csv /nh so it is
locale-independent), and remove broad non-administrative grants
(Everyone, Authenticated Users, Users). Fail closed on identity or
icacls failures so a tree is never left accessible silently.
Existing-tree handling is covered by asserting every entry in the tree is
repaired, and the Windows command contract is covered by mocked tests run
in CI.
* fix(lark): harden Windows credential tree against TOCTOU and hard-link races
This replaces the path-based Windows hardening (lstat -> SetFileSecurityW(path) -> iterdir) with a handle-relative walker, so validation, the ACL update, and traversal are bound to the opened object rather than a re-resolved pathname.
Every credential object is opened no-follow; children are enumerated with GetFileInformationByHandleEx(FileFullDirectoryInfo) and opened/created relative to an already-open parent handle (NtOpenFile/NtCreateFile with OBJECT_ATTRIBUTES.RootDirectory), so a pathname swap cannot redirect the walk. Credential directories are opened exclusively (share=0): SetSecurityInfo therefore does not propagate the final owner-only OI|CI DACL into as-yet-unvalidated children, and the namespace is locked for the duration of the walk (concurrent child rename/replacement and hard-link insertion fail with sharing violations). Any file with nNumberOfLinks != 1 is rejected before its security descriptor is touched, so an NTFS hard link to an external file cannot change that file owner/DACL. POSIX keeps the lstat-before-descent walk.
Tests: native regressions for exclusive no-propagation, late hard-link insertion being blocked, mid-walk junction swap being blocked, static hard-link rejection, and both real NTFS junction rejections. Mock seams updated for the handle-relative API, and Windows portability fixes make the suite green on Windows except the known #5116 sandbox-runtime executable-bit failures.
* test(lark): keep the credential-tree symlink assertion portable
The credential-tree symlink rejection is a ValueError; POSIX reports a symlink while the Windows handle-relative walker reports a reparse point. Use a platform-dependent regex so the test passes on Linux/macOS and Windows.
* fix(lark): close remaining credential-tree hardening gaps
Review follow-up for the handle-relative credential-tree walker:
- Stage the transaction snapshot under the owner-only root, copying only config/ and data/.
- Serialize ensure() per-user across threads and processes with a dedicated lock.
- Make the walker iterative so deep trees cannot hit the recursion limit.
- Re-reject a symlinked POSIX root before mkdir; drop the over-strict ancestor-chain check.
- Soften the SetSecurityInfo failure claim; add regressions for each and carry os.SEEK_END in the os stub.
* fix(lark): anchor hardening lock under trusted base and keep POSIX untouched
Follow-up refinements to the credential-tree hardening:
- The per-user hardening lock file now lives directly under the trusted base_dir
instead of the unverified per-user chain, so it is never written through an
ancestor that has not yet passed reparse validation.
- ensure() takes the hardening lock only on the Windows branch; POSIX keeps the
original contract, so no new lock-file side effect.
- Strengthen the ancestor-junction regression (lock not written to the external
target) and fix two test docstrings to match the parent-first order and the
no-prior-broadening failure claim.
* fix(lark): anchor credential-operation lock under trusted base on Windows
The per-user credential lock (_lark_credential_lock) created its advisory lock file
under the unverified per-user chain (users/<id>/integrations/.lark-cli.credentials.lock)
before ensure() validated the ancestor chain. On Windows it is now anchored directly
under the trusted paths.base_dir (mirroring the hardening lock), so a junction at
integrations can no longer cause the credential lock to be written into an external
target before reparse validation. POSIX keeps the original location unchanged.
Tests:
- Public-flow regression (start_lark_config -> credential lock -> ensure) uses an empty
sentinel lock file to prove the old credential-lock path is never opened/written.
- CLI-write re-harden tests restore the POSIX outcome assertion (file tightened to 0600).
* feat(lark): sidecar credential broker for sandbox lark-cli (Pattern B)
Removes the plaintext Lark credential mounts (appSecret + OAuth tokens)
from the sandbox container. A long-running broker sidecar owns lark-cli
and the per-user config/data dirs and serves the command surface over
Pod loopback; the sandbox gets only a forwarding shim on PATH, so the
raw credential files never exist in the sandbox filesystem.
- lark_broker.py: stdlib-only loopback broker (argv passthrough with
shell=False, server-injected credential env, bounded I/O) + shim
script constant + install-shim mode.
- docker/lark-cli-broker: init(install-shim) + serve image.
- provisioner: LARK_CLI_BROKER_IMAGE + provision_lark_cli_broker →
shim init container + lark-cli-broker sidecar (config/data mounted
sidecar-only); credentials dropped from the sandbox container;
/api/capabilities reports lark_cli_broker_image. Broker supersedes
the Pattern A init-container binary when both are configured.
- gateway: lark_cli_env_overlay(broker=True) omits config/data env;
sandbox_lark_broker_active() TTL-cached mode resolver; broker added
to sandbox_runtime_mode / readiness and the settings UI.
Opt-in and off by default (empty LARK_CLI_BROKER_IMAGE ⇒ no change).
Closes#4338
* fix(lark): address Pattern B broker review findings (#4501)
Follow-up to the sidecar credential broker addressing the PR #4501 review:
- shim: split the on-PATH lark-cli into a /bin/sh launcher + Python shim body
so broker mode fails loudly (exit 127, actionable message) instead of ENOEXEC
when the sandbox image ships no python3; interpreter pinnable via
DEERFLOW_LARK_BROKER_PYTHON. Launcher bakes in the shim's absolute path since
$0 is the bare command name when run off PATH.
- broker: drop the dead cwd payload field (broker can't see the sandbox FS) and
document the command-surface-only / no-file-IO limitation.
- broker: return a structured 500 JSON on unexpected exec errors so the shim
gets a meaningful message, not an opaque transport failure; set a handler
socket timeout to bound slow/stuck connections.
- broker: add an opt-in DEERFLOW_LARK_BROKER_DENY_SUBCOMMANDS denylist that
refuses secret-dumping subcommands before spawning the binary, forwarded from
the provisioner sidecar.
- gateway: tighten the per-bash-call broker probe timeout (1.5s) and cache
negatives longer (300s) so non-broker remote-provisioner users don't pay a
latency hit; guard the mode cache with a lock; drop the dead
_probe_provisioner_lark_cli_init_image wrapper.
- docs: remove the broken design-doc link from the broker README.
Adds tests for launcher python resolution, cwd omission, denylist enforcement,
500-on-error, hot-path probe timeout + negative caching, and provisioner
denylist-env wiring.
* feat: add lark cli integration
* fix: polish lark integration actions
* feat: support lark incremental permissions
* fix: detect lark authorization completion
* fix: harden lark integration install
* feat: expand lark auth scopes and reuse host auth in sandbox
Default lark auth to least-privilege (recommend=false, base sign-in only)
and expose the full set of lark-cli --domain business domains as native
--domain grants instead of a 4-domain read-only mapping. Resolve the
skill pack from the latest larksuite/cli GitHub release at install time
with content-hash integrity, and surface version/runtime drift in status.
Share the per-user lark-cli config/data profile between the Gateway
Settings auth flow and agent conversations by mounting the integration
dirs into the AIO sandbox and injecting the matching env for lark-cli
commands, with an allowlisted extra_mounts path in the provisioner/K8s
backend and traversal guards on integration paths.
* style: fix lint issues from ruff and prettier
Sort imports in the provisioner PVC test and re-wrap two long i18n
description strings to satisfy backend ruff and frontend prettier CI.
* fix(lark): address managed integration review feedback
* fix(frontend): stabilize integrations settings e2e
* test(sandbox): isolate remote backend legacy visibility check
* test: fix backend unit failures after merge
* Harden Lark integration review fixes
* Format Lark integration E2E test
* fix(lark): harden sandbox credential exposure and status disclosure
Address willem_bd's security review on PR #3971:
- Mount the per-user lark-cli config dir (long-lived appSecret) read-only
into the AIO sandbox; only the refreshable-token data dir stays writable.
- Redact host filesystem paths (install_path, cli.path) from
GET /lark/status and the config/auth complete responses for non-admin
callers, fail-closed on any auth error.
- Document the npm postinstall trade-off (--ignore-scripts is not viable
because @larksuite/cli fetches its platform binary in postinstall).
- Document the sandbox credential trust boundary in AGENTS.md and README,
pointing at the sidecar-broker follow-up (#4338).
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
