github/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-10-26 13:16:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
deer-flow/docker/lark-cli-broker
History
Storm 69004bd52c
fix(sandbox): give fresh bash.exec runs immediate stdin EOF (#6117) 
Why this change:
In Lark broker mode the shim forwards piped stdin to the broker only
after EOF. AIO's /v1/bash transport keeps a subprocess stdin pipe open
for writes, so a bare `lark-cli ...` never saw EOF and the shim blocked
forever on input that would never arrive. An earlier brace-group/eval
approach fixed the hang but changed top-level parsing (aliases/extglob)
or let a trailing backslash consume the appended delimiter.

What changed:
- _run_bash_exec prefixes the original command with `exec < /dev/null`
  on its own line, giving the fresh, per-invocation session immediate
  EOF on default stdin. Explicit pipes/heredocs/files still override
  fd0. The plain prefix keeps top-level parsing and appends no
  delimiter a trailing backslash can consume.
- The persistent /v1/shell PTY transport keeps commands unchanged; the
  shim already treats TTY stdin as no input.
- The shim drains non-TTY stdin under grace/tail idle budgets
  (DEERFLOW_LARK_BROKER_STDIN_{GRACE,TAIL}_SECONDS, default 2s, valid
  (0, 600]); an idle pipe exits 124 without contacting the broker, and
  read errors fail loudly instead of becoming empty input.
- Broker logs each exec as argc/exit/elapsed only (never argument
  values) and warns on concurrency-cap rejections.

Problem solved:
Lark commands without explicit stdin hung indefinitely in broker mode,
stalled producers could execute partial writes, and broker logs risked
capturing argument values. Docs updated: README, harness AGENTS.md,
docker/lark-cli-broker README.
2026-10-01 07:34:32 +08:00
..
Dockerfile
feat(lark): sidecar credential broker for sandbox lark-cli (Pattern B) (#4501)
2026-07-28 22:54:44 +08:00
Dockerfile.dockerignore
ci(lark): publish lark-cli-init/broker images (#4558)
2026-07-29 15:24:17 +08:00
entrypoint.sh
feat(lark): sidecar credential broker for sandbox lark-cli (Pattern B) (#4501)
2026-07-28 22:54:44 +08:00
README.md
fix(sandbox): give fresh bash.exec runs immediate stdin EOF (#6117)
2026-10-01 07:34:32 +08:00

README.md

lark-cli broker image (Pattern B)

This image implements Pattern B (issue #4338): instead of mounting the per-user Lark credential directories into the sandbox (Pattern A still does), a long-running sidecar holds lark-cli + the credentials and serves only the command surface over loopback. The sandbox gets a tiny lark-cli shim on PATH that forwards argv/stdin to the sidecar.

Result: the raw appSecret / OAuth token files never exist in the sandbox filesystem, so a compromised or prompt-injected agent can no longer cat/exfiltrate them — while any authorized lark-cli subcommand still runs.

Two modes, one image

Dispatched by the first CLI argument:

  • install-shim <dest> — init container: writes the launcher + Python shim + .deerflow-lark-cli-runtime.json (kind: "shim") into the shared emptyDir at <dest> (default /mnt/integrations/lark-cli/runtime), then exits 0. The sandbox then finds bin/lark-cli exactly where lark_cli_env_overlay(sandbox_paths=True) points PATH — same layout the Pattern A init image produces.
  • serve (default CMD) — sidecar: runs the broker HTTP server on 127.0.0.1:8788 with the real lark-cli and the credential env pointing at the sidecar-only /var/lark/{config,data} mounts.

The executable on PATH (bin/lark-cli) is a /bin/sh launcher that resolves a Python 3 interpreter and execs the shim body (bin/lark-cli-shim.py) beside it (by its baked-in absolute path, since $0 is the bare command name when run off PATH); both are written from the in-process LARK_CLI_BROKER_LAUNCHER_TEMPLATE / LARK_CLI_BROKER_SHIM_SCRIPT (deerflow.integrations.lark_broker), so the image's copies can never drift from the Gateway's. Splitting the sh launcher from the Python body means broker mode does not hard-depend on python3 resolving via a #!/usr/bin/env python3 shebang: if no python3/python is on the sandbox PATH, the launcher exits 127 with an actionable message (set DEERFLOW_LARK_BROKER_PYTHON to a known interpreter path) instead of an opaque ENOEXEC. The stock all-in-one-sandbox image ships Python 3, so the default path needs no configuration.

Build

Build context is the repo root (the broker module lives under backend/):

docker build -t deer-flow/lark-cli-broker:v1.0.65 \
  --build-arg LARK_CLI_VERSION=v1.0.65 \
  -f docker/lark-cli-broker/Dockerfile .

The tag should encode the lark-cli version so it can be bumped independently of the upstream all-in-one-sandbox image.

CI publishes multi-arch (linux/amd64,linux/arm64) images to ghcr.io/<owner>/deer-flow-lark-cli-broker:<lark-cli-version> via .github/workflows/lark-cli-images.yaml (run it with a lark_cli_version input, or push a lark-cli-v* tag). This is decoupled from the DeerFlow v* release because the image tracks the upstream larksuite/cli version.

Wiring it into the provisioner

Broker mode is opt-in and off by default. Enable it by publishing this image and pointing the provisioner at it:

  • Set LARK_CLI_BROKER_IMAGE on the provisioner to the published tag. Empty ⇒ broker off (Pattern A / legacy path, no behavior change).
  • When set, and the Gateway sends provision_lark_cli_broker on sandbox create, the provisioner adds:
    • a lark-cli-runtime emptyDir shared by an init container and the sandbox;
    • a lark-cli-shim-init init container (install-shim) that stages the shim;
    • a lark-cli-broker sidecar (serve) with the per-user config (RO) / data (RW) credential mounts — into the sidecar only;
    • the sandbox container gets the runtime RO mount + DEERFLOW_LARK_BROKER_URL and no config/data mounts.
  • Broker mode supersedes Pattern A when both are configured.
  • The provisioner reports it via GET /api/capabilities ({"lark_cli_broker_image": true|false}), which the Gateway surfaces as the Lark integration sandbox-runtime readiness signal in /api/integrations/lark/status (sandbox_runtime_mode: "broker").

Opt-in note: broker mode stays off until LARK_CLI_BROKER_IMAGE is set on the provisioner, so an unpublished or unconfigured image is a no-op (Pattern A / legacy path, no behavior change).

Broker HTTP contract (loopback)

  • POST /v1/exec — body {"args": [...], "stdin_b64": "..."}; response {"exit_code", "stdout_b64", "stderr_b64", "truncated"}. args is run with shell=False, so a sandbox-supplied argument can never be shell-injected. The broker injects the credential env itself; the client cannot override it. Unexpected broker-side errors return a 500 {"error": ...} so the shim always gets a structured response rather than an opaque transport failure.
  • GET /v1/health — {"ok": true}.

Bound to loopback only. In K8s the sandbox and sidecar share the Pod network namespace, so 127.0.0.1 reaches the sidecar and nothing outside the Pod can.

Stdin and execution logs

AIO's fresh non-interactive Bash runs close inherited pipe stdin, so commands without input receive immediate EOF. Persistent Shell terminal commands remain unchanged; the shim ignores their TTY stdin. Explicit pipelines, heredocs, and file redirections supply input normally. The shim forwards pipe/file stdin only after EOF; a pipe that exceeds its idle budget exits 124 before contacting the broker, so incomplete input cannot execute a partial write.

The sandbox process can override the first-byte budget with DEERFLOW_LARK_BROKER_STDIN_GRACE_SECONDS and the between-chunk idle budget with DEERFLOW_LARK_BROKER_STDIN_TAIL_SECONDS. Both default to 2 seconds and accept finite values in (0, 600]; invalid values fall back to the default. Increase the appropriate budget when an explicit input producer needs longer startup or pauses. These settings belong on the sandbox invocation, not the broker sidecar.

Execution logs record argument count, exit code, elapsed time, and output truncation, but never argument values or stdin. Updating the Gateway alone does not update an installed shim: rebuild and publish this broker image with a new immutable tag or digest, update LARK_CLI_BROKER_IMAGE, and recreate sandbox Pods.

No file I/O relative to the sandbox cwd

The broker runs lark-cli in the sidecar's working directory and cannot see the sandbox filesystem, so the sandbox's cwd is intentionally not forwarded. lark-cli subcommands that read or write files by a path relative to the sandbox cwd (e.g. uploading a local file) are therefore unsupported in broker mode — this is a command-surface-only bridge, not a filesystem bridge. Absolute paths still refer to the sidecar's filesystem, not the sandbox's.

Optional subcommand denylist (hardening)

The broker removes the credential files from the sandbox, but the full lark-cli command surface stays reachable, so any subcommand that prints/exports tokens could still exfiltrate them. Set DEERFLOW_LARK_BROKER_DENY_SUBCOMMANDS on the sidecar to a comma-separated list of command prefixes the broker should refuse (matched against the leading non-flag tokens), e.g. DEERFLOW_LARK_BROKER_DENY_SUBCOMMANDS="config show, auth token". Denied calls return exit 126 with a subcommand ... is disabled message and never spawn the binary. Empty by default (no behavior change); confirm the deployed lark-cli version's subcommand surface has no trivial secret-dump command before enabling broker mode in production.