mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-30 23:20:35 +00:00
10 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Storm
|
b9a75b869d
|
fix(sandbox): attest lark broker mode per Pod and quarantine uncertain shell outcomes (#6436)
* fix(sandbox): attest lark broker mode per Pod and quarantine uncertain shell outcomes Why this change: Two fail-open paths in the AIO sandbox lifecycle. First, the lark broker capability probe collapsed "provisioner unreachable" into "broker not configured": one 1.5s timeout cached a negative for 300s, so new sandboxes were created without the broker sidecar while the per-user credential directories were still bind-mounted into the container, and existing Pods kept being reused without their actual mode ever being read. Second, an implicit-shell command that timed out or returned an unknown status only marked the current client object; warm-pool reclaim and rediscovery built a fresh client that re-entered a container whose shell state could not be proven clean, and nothing about that ambiguity survived a Gateway restart. What changed: - The capability cache stores confirmed observations only, keyed per provisioner endpoint/auth identity; an unknown answer (timeout, malformed payload) raises instead of selecting credential placement, while a confirmed broker requirement is retained through outages. One in-flight probe per identity shares success and failure, and a Pod-attested mode contradicting the cache forces one shared refresh. - Execution uses the mode attested for the actual sandbox: SandboxInfo.lark_cli_broker flows from the provisioner's atomic Pod observation into the client, so a probe outage delays drift detection but never denies reuse of an attested runtime, and never trusts an unattested one. - The provisioner reports each Pod's actual broker mode, generation (Pod UID) and shell capacity from one read, rejects lark requests conflicting with its configuration (409/503 carrying a capability-refresh marker the Gateway uses to drop the stale cache entry), rejects conflicting or Terminating Pods on coordination creates while reporting Terminating as data to status/get/list, and answers Pod presence independently of Service readiness. - A persistent quarantine (DEER_FLOW_HOME/sandbox-quarantine, keyed by scope/sandbox/generation) fences containers after uncertain implicit shell outcomes across client rebuilds and restarts; in-process reuse, warm reclaim, discovery, create responses and orphan reconciliation all pass one policy gate, and replacement runs under the existing ownership and teardown fences. A fenced sandbox still tracked after a failed release-recycle is destroyed inline on the next acquire. - Quarantine records retire only after confirmed absence, retrying partial cleanup for the quarantined generation and releasing remembered ports under the same fences; local stop failures now propagate instead of being logged away, and remote destroy errors surface to callers. Problem solved: A provisioner blip could silently strip broker isolation from new lark sandboxes for minutes at a time, and ambiguous shell outcomes let corrupted default shells persist across warm reuse, rediscovery and restarts. Docs updated: README, aio_sandbox and skills AGENTS.md. * docs(changelog): record sandbox attestation and quarantine fix (#6436) * fix(sandbox): preserve active leases and fail closed on unattested broker modes Concurrent uploads could recycle a healthy AIO runtime while another holder was creating or cleaning up a command session. Older provisioner responses could also trigger destructive replacement before failing admission. - Distinguish pending session creation from ambiguous outcomes and reserve idle teardown until execution and upload holders finish scope cleanup. - Reject stale fork lookups and reconcile cancelled acquires without releasing a sandbox still borrowed by another holder. - Default Lark broker mode to unverified and preserve unattested Pods until the provisioner is upgraded. - Recognize Docker 29 absence diagnostics while matching resource names exactly. - Preserve Gateway upload errors and cover admission and lifecycle regressions. * fix(sandbox): scope the unverified-broker refuse path to attested providers The fail-closed raise in _lark_cli_env_from_runtime treated every Sandbox without a lark_cli_broker attribute as unverified, but only AioSandbox carries the attestation — E2B, OpenSandbox, Tenki, and Boxlite have no broker sidecar, no credential mounts, and no provisioner to upgrade, so every lark-cli command on those providers failed with an unactionable error. Declare the tri-state on the Sandbox base class (defaulting to False, the only mode brokerless providers have) so the None fail-closed path stays scoped to AioSandbox's deliberate unverified default. - Add lark_cli_broker: bool | None = False to the Sandbox ABC, mirroring the persistent_shell_sessions tri-state pattern - Add a regression test pinning the non-broker overlay for a provider without the attribute * fix(sandbox): close review gaps in broker attestation and teardown lifecycle Address the latest #6436 review round: - The provisioner's post-build broker-mode 409 now carries the capability-refresh marker, so an incompatible-mode conflict surfaced by idempotent re-create, attestation drift or post-create verification drops the Gateway's cached capability observation instead of repeating the same 409 for the full negative-cache TTL. - AioSandboxProvider.destroy() returns the _destroy_tracked outcome: False when teardown is deferred because an execution/upload holder or a peer teardown is still active, so callers can no longer mistake a deferred destroy for a completed one. - A live replacement under the same logical ID proves the fenced old generations are absent; retire_replaced_generations prunes only those stale records so they cannot strand future creates, while the live generation's fence and unversioned crash fences are never touched. - SandboxInfo.from_dict keeps a missing lark_cli_broker attestation unverified (None) instead of silently rehydrating as attested non-broker. - The restricted-network inspect path shares the strict _is_no_such_network_error parser, closing the substring loophole for context-name collisions and wrong-target errors. * docs(sandbox): condense lease notes to keep the AGENTS.md chain under the size gate The three lease notes added in 26c307f3 pushed the effective guidance chain to 98,312 bytes — 8 over the 98,304 hard limit enforced by agent-guidance (AG002). Fold them into two lines carrying the same rules. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Hyeonsang Cho
|
14cd840bbf
|
fix(gateway): reject non-ASCII tokens instead of raising in constant-time compares (#6076)
* fix(gateway): reject non-ASCII tokens instead of raising in constant-time compares
hmac.compare_digest raises TypeError when a str operand holds non-ASCII
characters. Starlette decodes header bytes as latin-1 and query values as
UTF-8, so a single 0xE9 byte in a CSRF token, GitHub webhook signature,
internal auth token, or OIDC state turned the expected 403/401 into a 500.
Route the five client-facing comparisons through a new
app.gateway.utils.constant_time_equals helper that compares the UTF-8
encodings ("surrogatepass" so even a lone surrogate cannot raise). ASCII
results are unchanged, and no bypass was possible before the fix.
* fix(provisioner): reject non-ASCII API keys instead of raising; drive OIDC test over HTTP
The standalone provisioner compared X-API-Key with secrets.compare_digest on
str operands, so a non-ASCII header raised TypeError (500) the same way the
Gateway sites did. It cannot import app.gateway, so encode inline.
The OIDC state regression test now mounts the auth router and sends the
percent-encoded state through a real signed state cookie instead of stubbing
get_state_cookie and calling the handler directly.
* docs(changelog): reference #6076
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
Onefly
|
d8d110c637
|
fix(sandbox): prevent AIO subagent session eviction (#5178)
* fix(sandbox): prevent AIO subagent session eviction * fix(sandbox): address PR 5178 review issues * fix(sandbox): handle transient session and metadata failures * fix(sandbox): fence capacity upgrades and validate reused limits * docs(sandbox): restore list indentation and trim guidance * fix(ci): stabilize Buzz persistence test and trim sandbox guidance --------- Co-authored-by: ranxi2001 <ranxi2001@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Aari
|
317577e285
|
fix: enforce custom agent skill allowlists in sandboxes (#5077)
* fix: enforce agent skill allowlists in sandboxes * fix: guard E2B skill projection resets * fix: preserve agent skill isolation across delegation * fix: close sandbox skill isolation bypasses * fix(sandbox): close skill isolation review gaps * fix(sandbox): harden skill isolation lifecycle |
||
|
Vanzeren
|
095092418c
|
fix(gateway):unify thread id validation (#4589)
* fix(gateway): unify thread ID validation at the API boundary
Thread ID entry points accepted arbitrary strings while downstream
consumers (filesystem paths, Kubernetes Provisioner, JSONL event store)
each enforced different character restrictions, so invalid IDs were
persisted first and only failed later during sandbox/workspace init.
Centralize validation in deerflow.utils.thread_id (pattern
^[A-Za-z0-9_-]{1,64}$): validate at routers, RunCreateRequest,
scheduler dispatch, paths.py, JSONL store, embedded client, and align
the Provisioner pattern (pinned by a parity test). UUIDs are still
generated only when no ID is supplied; caller-supplied opaque IDs stay
supported.
Deliberate exceptions: DELETE /threads/{id} keeps str as the legacy
cleanup escape hatch (filesystem cleanup guarded), read-only
client.get_thread stays unvalidated, and scheduler rows with legacy
invalid IDs record a failed dispatch instead of raising out of the
poll loop.
* docs: document canonical thread ID contract
README: caller-supplied thread IDs need not be UUIDs; the canonical
pattern and per-endpoint behavior. AGENTS.md: the shared
deerflow.utils.thread_id contract, its enforcement boundaries, and the
legacy-ID escape hatches.
* fix(gateway): close thread ID validation gaps at remaining entry points
Follow-up to the canonical thread ID contract: a full audit found the
uniform-422 coverage only reached about half of the thread_id surfaces.
- routers: 18 routes still took a bare thread_id: str — 13 in
thread_runs.py (including the five messages/events/workspace-changes
reads that returned 500 on the JSONL event store vs 404/empty on the
DB store), 4 read routes in threads.py, and the suggestions route
flagged in review. DELETE /api/threads/{id} keeps str as the declared
legacy-cleanup escape hatch.
- client: upload_files/delete_upload/list_uploads/get_artifact now
validate up front, fulfilling the RFC's 'all mutating entry points'
clause (get_thread stays unvalidated as the declared legacy read path).
- tui: the /resume literal-ref fallback validates against the canonical
contract and reports a descriptive error instead of failing deep in
the client.
- scripts/support_bundle.py: replace the drifted dot-allowing pattern
with a byte-identical copy of THREAD_ID_PATTERN (kept local so the
script still runs with a broken venv).
* test(gateway): guard the canonical thread ID contract against regressions
- test_thread_id_route_contract.py: static AST sweep asserting every
route handler with a thread_id parameter annotates ThreadId
(whitelist: the DELETE escape hatch), plus a runtime sweep hitting
all 44 thread_id routes with a non-canonical ID and asserting a 422
that names thread_id, plus a websocket upgrade-rejection case.
- test_thread_id_validation.py: client entry-point validation,
support_bundle pattern parity, and TUI literal-ref fallback tests.
- Align two tests that encoded the old contract (dotted IDs).
|
||
|
Xinmin Zeng
|
f2e832330e
|
fix(sandbox): enforce disabled skills in filesystem views (#4178)
* fix(sandbox): project enabled skills into sandbox views * fix(skills): keep projection mutations consistent * fix(skills): fail closed on projection errors * fix(skills): isolate per-scope failures during boot projection rebuild rebuild_all_skill_projections() propagated any exception from the public rebuild or from a single user's rebuild straight out of the gateway lifespan startup, uncaught. A single broken user directory (bad permissions, corrupted _skill_states.json, unreadable content) would therefore abort gateway boot for every user, not just that one - _rebuild_*_locked already fails closed internally (clears the view and re-raises), so the boot loop only needed to stop treating that re-raise as fatal. Each scope's rebuild now fails closed independently and boot continues; a scope left empty by a boot failure self-heals on the next sandbox acquire via ensure_skill_projections(). Also patches deerflow.skills.projection.rebuild_all_skill_projections in the memory-flush lifespan test fixture, matching the two sibling fixtures in the same file — this call is now on the lifespan startup path and the fixture's minimal SimpleNamespace config predates it. * test(skills): update authz test for the projection-aware public toggle _persist_shared_skill_state (introduced earlier in this branch) reads the shared extensions_config.json fresh from disk under the projection lock instead of through the cached get_extensions_config() singleton - that's the whole point of the fix (stale worker caches must not clobber another worker's concurrent update). The name no longer exists on the skills router module, so the test's monkeypatch of it started raising AttributeError instead of exercising the endpoint. The mock storage in this test isn't a real LocalSkillStorage instance, so _persist_shared_skill_state's projection-mutation branch is already skipped (nullcontext) and it falls back to a fresh ExtensionsConfig() for the nonexistent tmp config_path - no replacement monkeypatch needed. * fix(sandbox): make skill projection ensure best-effort in acquire acquire() called _ensure_skills_projection() directly, outside any try/except, in both LocalSandboxProvider and AioSandboxProvider. Every other skill-mount setup path in these providers has always caught exceptions and logged a warning rather than failing sandbox acquire outright (e.g. when config.yaml can't be resolved) - these two new call sites broke that contract, so any projection failure (including simply not having a config.yaml, as in CI's test environment) now failed acquire() itself instead of just leaving skill mounts off. _ensure_skills_projection now catches its own exceptions and returns None; both providers' callers already tolerate that (a None projection skips the skill-specific mounts, matching the existing degrade path) after making _append_public_skill_mapping and the custom/legacy mount block in LocalSandboxProvider explicitly None-safe. Caught by running the full suite with config.yaml removed, matching CI's environment - not caught locally because a real config.yaml was present, masking the failure. * fix(sandbox): make E2B skill projection mounts best-effort _skill_projection_mounts called ensure_skill_projections with no guard, unlike Local/AIO's _ensure_skills_projection. A raise propagated out of _apply_mounts before the configured-mounts loop ran, so a skills projection failure dropped the operator's own configured mounts too - only caught by create()'s outer warning, with nothing applied at all. Swallow here and return an empty mount list on failure, matching the Local/AIO pattern: still fail-closed for skills, but no longer widens the blast radius to unrelated configured mounts. Review feedback from PR #4178. * docs(skills): document projection trade-offs flagged in review - _update_tree_digest: note the metadata-only (not content) hashing trade-off and why runtime writes through this codebase are still covered regardless (rebuild-under-lock + rename always changes inode). - LocalSandboxProvider.acquire: note the acquire-time self-heal cost (cheap on a fresh manifest, ~400ms rebuild under lock on stale/drift). - skill_projection_mutation: drop the no-op except-Exception-then-raise; a raise from the mutation already propagates past the yield with the view left cleared, no explicit re-raise needed. - provisioner README: spell out that hostPath skills volumes require the gateway and K8s node to share DEER_FLOW_HOST_BASE_DIR (single-node or shared storage), and that the custom/legacy volumes' hostPath type Directory (not DirectoryOrCreate) makes a violation of that assumption a visible Pod-creation failure instead of a silent empty mount. Review feedback from PR #4178. * fix(skills): lazily repair user projections * fix(skills): close projection review gaps * fix(skills): refresh user projection enable state * fix(skills): close projection review follow-ups * fix(skills): preserve state across projection writes --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
OrbisAI Security
|
8cc4b3abeb
|
fix(sandbox): the sandbox provisioner api exposes endpoints need to check API KEY (#4116)
* fix: V-001 security vulnerability Automated security fix generated by OrbisAI Security * fix(sandbox): wire provisioner API key through backend client and config RemoteSandboxBackend now accepts an api_key parameter and sends it as X-API-Key on all five provisioner HTTP calls (list, create, destroy, is_alive, discover). AioSandboxProvider reads provisioner_api_key from SandboxConfig and forwards it at construction time. SandboxConfig formally declares the field; config.example.yaml documents it under Option 4; docker-compose-dev.yaml threads PROVISIONER_API_KEY into both the provisioner and gateway containers so a single .env entry covers both sides. Tests: monkeypatch PROVISIONER_API_KEY and send X-API-Key headers in the five parametrized threading tests (previously 401-failing); new test_auth_middleware asserts /health is open, /api/* rejects no-header and wrong-key with 401, and accepts the correct key. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(provisioner): correct fail-closed auth docs and fix test mock signatures The first round of PR #4116 fixes left four blocking issues: - Mock signatures in test_remote_sandbox_backend.py (17) and test_aio_sandbox_provider.py (2) didn't accept the new headers= kwarg, causing TypeError on every provisioner HTTP call test. - sandbox_config.py and config.example.yaml described the auth as optional ("leave unset to disable") but the middleware is fail-closed: an unset PROVISIONER_API_KEY causes 401 on every /api/* request. - .env.example had no PROVISIONER_API_KEY entry, leaving users with an empty value and silent 401s. - No test covered the PROVISIONER_API_KEY="" fail-closed path. Fix all four: add headers=None to all mock signatures, correct the field description and example comment to state that both sides must have the same key set, add PROVISIONER_API_KEY to .env.example with generation guidance, add test_auth_middleware_unset_key, and add a logger.warning on auth rejection for observability. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
明年我18
|
488ec17899
|
feat(provisioner): support ClusterIP services and scoped skills PVC mounts (#4016)
* feat(provisioner): add ClusterIP mode for sandbox services * feat(provisioner): support optional skills pvc subpath |
||
|
Vanzeren
|
e3137b13cf
|
fix(provisioner): gate legacy skills mount by user visibility (#3985)
* fix(provisioner): gate legacy skills mount by user visibility * fix(test): aio sandbox provider * fix: use shared legacy skill visibility helper for sandbox mounts |
||
|
AnoobFeng
|
e9de187408
|
fix(provisioner): keep k8s calls off event loop (#3941)
* fix(provisioner): keep k8s calls off event loop * test(provisioner): scan blocking IO by module name * test(provisioner): exercise create path so BlockBuster has real work |