diff --git a/AGENTS.md b/AGENTS.md index 3d396ebaf..9f1567c63 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -50,6 +50,7 @@ deer-flow/ ├── frontend/ # Next.js frontend (pnpm) — see frontend/AGENTS.md ├── docker/ # docker-compose files, nginx config, provisioner ├── skills/ # Agent skills: public/ (committed), custom/ (gitignored) +│ # Managed integration skill packs are installed per user under .deer-flow/users/{user_id}/skills/integrations/ ├── contracts/ # Cross-component JSON contracts (e.g. subagent status, skill review) ├── scripts/ # Root orchestration scripts invoked by the Makefile (check, configure, doctor, support_bundle, serve, nginx, docker, deploy, setup_wizard) ├── tests/ # Root-level tests (currently tests/skills/ — public skill tests) diff --git a/README.md b/README.md index 9fc1a6507..5c802a919 100644 --- a/README.md +++ b/README.md @@ -664,6 +664,70 @@ An enabled skill's `allowed-tools` policy applies only after that skill is expli When you install `.skill` archives through the Gateway, DeerFlow accepts standard optional frontmatter metadata such as `version`, `author`, and `compatibility` instead of rejecting otherwise valid external skills. +Managed integrations install shared read-only skill packs without mixing them +into custom skills. The Lark/Feishu CLI integration is available under +`Settings → Integrations → Lark / Feishu CLI`; an administrator installs or +upgrades the official `lark-*` pack once under +`{DEER_FLOW_HOME}/integrations/skills/lark-cli`, and every user discovers that +same pack with an independent enabled state. Each user's app configuration and +OAuth data remain isolated under +`{DEER_FLOW_HOME}/users/{user_id}/integrations/lark-cli/{config,data}`. These +secret directories are restricted to `0700`, regular credential files to +`0600`, and symlinks are rejected. + +After installation, users can click **Connect Lark** to open a browser +authorization link; no terminal authorization is required. The same UI can +request additional permission domains such as Calendar, Docs, or Drive, or a +specific OAuth scope reported by `lark-cli`. A cheap status refresh only +inspects the local credential tree, so the UI reports **Credentials configured +(not live-verified)** until an explicit browser completion performs live token +verification. The action then remains **Reconnect Lark** so users can replace +or extend authorization. If an agent hits missing Lark authorization during a +conversation, the managed `lark-shared` guidance points the user back to the +same settings entry with `?settings=integrations`. + +Installing the Lark skill pack resolves the latest official `larksuite/cli` +release from GitHub and downloads that version's skills at install time, so the +Gateway needs outbound internet access for that step (it falls back to a +bottom-line pinned version if the release lookup fails). The settings page shows +the installed version and, when available, the newest published version so an +admin can reinstall to upgrade. Air-gapped deployments can pre-stage the archive +and point `DEER_FLOW_LARK_CLI_SKILLS_ARCHIVE` at the local file. Integrity does +not depend on a pinned archive byte hash (GitHub does not guarantee stable +source-archive bytes); instead the download is restricted to the official GitHub +host, every archive member passes structural safety guards, and a content hash +of the effective installed skill tree (including DeerFlow's injected shared +guidance) is recorded so content changes are auditable across reinstalls. + +When `sandbox.use` selects the AIO provider, the same install also downloads the +official Linux amd64 and arm64 CLI release archives, verifies their published +SHA-256 checksums, safely extracts one executable per architecture, and mounts +the resulting runtime read-only at `/mnt/integrations/lark-cli/runtime`. An +architecture-selecting launcher in that mount makes `lark-cli` available in the +sandbox `PATH`. Air-gapped AIO deployments can pre-stage a symlink-free runtime +tree containing `bin/lark-cli` plus both `linux-{amd64,arm64}/lark-cli` files and +set `DEER_FLOW_LARK_CLI_SANDBOX_RUNTIME_DIR` to that directory. + +> **Sandbox trust boundary:** the browser never receives the Lark app secret, but +> agent conversations run `lark-cli` inside the sandbox, so the per-user +> credential directories are mounted into it: `config` (holding the long-lived +> `appSecret`) is mounted **read-only** and `data` (refreshable OAuth tokens) +> writable. Both remain *readable* by any process the agent runs there, so code +> reached via prompt injection in a tool result could read them. Treat the +> sandbox as inside the Lark credential trust boundary until the sidecar +> credential-broker follow-up removes these mounts from sandbox execution. + +For remote/Kubernetes deployments (the provisioner backend), the sandbox +`lark-cli` runtime can instead be supplied by an optional init container that +copies the binaries into a shared `emptyDir` — no install-time GitHub download and +no hostPath/PVC runtime mount. Publish the image under +[`docker/lark-cli-init`](docker/lark-cli-init/README.md) and set +`LARK_CLI_INIT_IMAGE` on the provisioner; it stays off (legacy behavior) when +unset. The Lark integration status (`GET /api/integrations/lark/status`) reports +`sandbox_runtime_mode` and `sandbox_runtime_ready` so the Settings UI shows +whether `lark-cli` will actually be present in the sandbox at chat time, rather +than a green status hiding a later `command not found`. + If a trusted operator manages the configured skills directory through an external mount such as MinIO, NFS, or CSI, an administrator can call `POST /api/skills/reload` after changing files. This invalidates skill prompt caches for the current Gateway process and waits up to the bounded refresh timeout so subsequent runs rescan the latest files; running tasks are unchanged. A loader-level filesystem failure returns a generic server error and preserves the last successfully loaded process cache rather than publishing an empty catalog. Uvicorn workers and Kubernetes Pods must each be targeted separately. Direct mount writes bypass the validation, SkillScan, and history applied by DeerFlow's install/edit APIs, so only operator-controlled systems should have write access. Skill installs and agent-managed skill edits run through **SkillScan**, a native deterministic safety scanner before the LLM-based skill scanner. Phase 1 runs offline with no Semgrep/OpenGrep dependency, blocks high-confidence `CRITICAL` findings such as private keys or shell execution, and passes warning findings to the LLM scanner for contextual review. Python instance-client exfiltration checks follow a minimal same-scope evidence chain: a simple name bound to a known client constructor, optional name-to-name aliases, and an actual outbound method or context-manager use supported by that constructor. Constructor roots must be proven imports; bare canonical-looking names are not inferred as modules. Nested scopes do not inherit client handles and inherit only constructor import aliases that are never rebound in the enclosing scope. Comprehensions, walrus-bearing statements, annotations, complex binding targets, unsupported operations, and ambiguous branch flows produce no finding from this signal; skipped constructs conservatively invalidate every name they may bind so stale client state cannot create a finding. A deterministic work budget or recursion limit reached by this best-effort analysis does not discard findings already collected for the file. Set `skill_scan.enabled: false` in `config.yaml` to disable only the deterministic analyzers; safe archive extraction and the LLM scanner still run. @@ -713,6 +777,9 @@ Web UI chat links percent-encode custom thread identifiers before placing them i /mnt/skills/custom └── your-custom-skill/SKILL.md ← yours + +/mnt/skills/integrations +└── lark-cli/lark-doc/SKILL.md ← managed, read-only ``` #### Claude Code Integration diff --git a/backend/AGENTS.md b/backend/AGENTS.md index 73caca7c8..88677c7fa 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -48,6 +48,7 @@ deer-flow/ │ │ │ └── registry.py # Agent registry │ │ ├── tools/builtins/ # Built-in tools (present_files, ask_clarification, view_image, review_skill_package) │ │ ├── mcp/ # MCP integration (tools, cache, client) +│ │ ├── integrations/ # Managed first-party integration installers (e.g. Lark CLI skill pack) │ │ ├── models/ # Model factory with thinking/vision support │ │ ├── skills/ # Skills discovery, loading, parsing │ │ ├── config/ # Configuration system (app, model, sandbox, tool, etc.) @@ -219,6 +220,9 @@ Blocking-IO runtime gate (`tests/blocking_io/`): SQLite path resolution plus `ensure_sqlite_parent_dir`, fix for #1912); `test_jsonl_run_event_store.py` (locks `JsonlRunEventStore`'s async API offloading its file IO via `asyncio.to_thread`); + `test_integrations_router.py` (locks Lark integration install and auth + completion route handlers offloading archive filesystem work and `lark-cli` + subprocesses); `test_uploads_middleware.py` (locks `UploadsMiddleware.abefore_agent` offloading the uploads-directory scan off the event loop); `test_uploads_router.py` (locks Gateway upload/list/delete endpoints @@ -425,6 +429,7 @@ Localhost persistence deliberately reads the direct request `Host` and ignores ` | **Console** (`/api/console`) | Read-only cross-thread observability for the current user (the data layer for an operations dashboard or external monitoring): `GET /stats` - headline counters (runs/threads/agents/tokens/cost); `GET /runs` - paginated run history joined with thread titles (per-run cost); `GET /usage` - zero-filled daily token series + per-model breakdown with spend. Queries `runs`/`threads_meta` directly as a reporting layer (no new `RunStore` methods); requires a SQL database backend — returns 503 on `database.backend: memory`. Real-cost estimation reads optional `models[*].pricing` (`currency`, `input_per_million`, `output_per_million`, `input_cache_hit_per_million`; `ModelConfig` is `extra="allow"`, so no schema change) and prices each run from its `token_usage_by_model` input/output split. Pricing is **cache-aware**: `RunJournal` accumulates prompt-cache hits from `usage_metadata.input_token_details.cache_read` into a sparse `cache_read_tokens` bucket key (also threaded through `SubagentTokenCollector` → `record_external_llm_usage_records`), and cache-hit input tokens are billed at `input_cache_hit_per_million` (omitted → billed at the miss price, a conservative upper bound). Legacy rows fall back to run-level totals at `model_name`; unpriced models yield `cost: null` and cost fields are null when no pricing is configured | | **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - update config (saves to extensions_config.json) | | **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`); `POST /reload` - admin-only process-local prompt-cache invalidation after trusted external filesystem changes | +| **Integrations** (`/api/integrations`) | `GET /lark/status` - inspect managed Lark/Feishu CLI integration state, including `sandbox_runtime_mode` / `sandbox_runtime_ready` (whether `lark-cli` will actually be present in the sandbox at chat time); `POST /lark/install` - admin-only install of the official `lark-*` managed skill pack; `POST /lark/config/start` and `/lark/config/complete` - internal first-time Lark connection setup; `POST /lark/auth/start` and `/lark/auth/complete` - browser device-flow user authorization without terminal access, with optional `domains` / exact `scope` for incremental permission grants | | **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data | | **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete | | **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; `POST /branches` - create a new main-thread branch from a completed assistant turn checkpoint and, when an addressable pre-user replay checkpoint exists, materialize it into the branch namespace so the inherited response remains regeneratable. Workspace files are not checkpointed, so the branch only best-effort copies the current workspace when branching from the **latest** turn (`workspace_clone_mode="current_thread_best_effort"`); branching from an older/historical turn skips the copy (`workspace_clone_mode="skipped_historical_turn"`) so the branch never inherits files that only exist in a later timeline. Branch creation also seeds the new thread's run-event feed from the branch checkpoint's visible messages (`history_seed_mode` in the response): the thread feed reads run_events, not checkpoints, so without the seed the inherited history disappears from the UI after the branch's first run (#4380). Seeded rows are grouped into one synthetic run per inherited turn (`branch-seed-{thread_id}-{n}`, a new turn opening at every persisted human message, including an allowlisted hidden `ask_clarification` reply) because `run_id` is a turn identity to the feed's consumers, not a provenance tag: regenerating an inherited answer supersedes that row's whole `run_id` in `GET /messages/page`, so one shared id for the entire seed deleted the complete inherited history on a branch's first regenerate (#4458); `GET /goal`, `PUT /goal`, `DELETE /goal` - read, set, and clear the active thread goal; `POST /compact` - manually summarize older active context into `summary_text` and retain the recent message window, blocked while a run is in flight; unexpected failures are logged server-side and return a generic 500 detail | @@ -614,9 +619,9 @@ E2B output sync records remote file versions and actual host file metadata in a ### Skills System (`packages/harness/deerflow/skills/`) -- **Location**: `deer-flow/skills/{public,custom}/` +- **Location**: global public skills live under `deer-flow/skills/public/`; user-authored custom skills live under `{DEER_FLOW_HOME}/users/{user_id}/skills/custom/`; globally managed integration skills live under `{DEER_FLOW_HOME}/integrations/skills/{provider}/`; per-user integration credentials remain under `{DEER_FLOW_HOME}/users/{user_id}/integrations/{provider}/{config,data}` - **Format**: Directory with `SKILL.md` (YAML frontmatter: name, description, license, allowed-tools, required-secrets) -- **Loading**: `load_skills()` recursively scans namespace directories under `skills/{public,custom}`, but stops descending once it finds a `SKILL.md`; that directory is a package boundary, so no nested `SKILL.md` is registered as a runtime skill. SkillScan has a deliberately narrower packaging rule: known eval fixtures are permitted as support data, while other nested `SKILL.md` files are reported as package defects. It parses runtime metadata and reads enabled state from extensions_config.json. +- **Loading**: `load_skills()` recursively scans public, per-user custom, global integration, and legacy custom locations for `SKILL.md`, parses metadata, and reads enabled state from extensions_config.json plus per-user skill state for non-public categories; that directory is a package boundary, so no nested `SKILL.md` is registered as a runtime skill. SkillScan has a deliberately narrower packaging rule: known eval fixtures are permitted as support data, while other nested `SKILL.md` files are reported as package defects. It parses runtime metadata and reads enabled state from extensions_config.json. - **External reload**: `POST /api/skills/reload` is an admin-only, process-local invalidation hook for trusted MinIO/NFS/CSI writes. `SkillStorage` instances do not cache a catalog — `load_skills()` scans on every call — so the route clears all `(app_config, user_id)` entries and the rendered prompt-section LRU, then waits up to the shared refresh timeout for the existing off-loop single-flight refresh. Each invalidation receives a generation-bound result handle; a successful scan atomically replaces the global enabled-skills cache, while a loader-level failure propagates to the HTTP waiter and preserves the last-known-good global cache. Per-user/config scans capture the refresh version and cannot repopulate shared caches if invalidation occurs while they are loading. A timed-out HTTP wait fails generically while the daemon refresh worker continues. Subsequent runs rescan after a successful reload; active runs keep their existing snapshot. Each Uvicorn worker/Kubernetes Pod must be targeted separately. Direct mount writes bypass install/edit validation, SkillScan, and history, so mounted roots are an operator-controlled trust boundary. - **Tool policy**: Lead-agent `allowed-tools` declarations apply dynamically only to slash-activated skills and skills captured in `ThreadState.skill_context` through configured `read_file` loads; passive enabled skills and custom-agent skill allowlists remain discoverable without clamping the global toolset. Slash policy is dominant for its run, preventing subsequently read skills from widening explicit authority; autonomous captured skills use the existing union only when no slash source exists. `tool_search` and `describe_skill` stay available as framework discovery infrastructure, while every discovered or promoted business tool still requires active-policy permission for schema visibility and execution; `task` likewise requires an explicit declaration. Each active model call intentionally reloads the full live registry so enable/disable changes, frontmatter edits, and custom/public name-shadow winners take effect without a stale TTL or unsafe direct-path cache; all tool calls produced by that model step reuse the resulting source-and-path-signed decision. Registry failures and all-invalid active sets fail closed, while stale individual paths are skipped when another valid skill remains. This is best-effort behavioral scoping, not a hard security boundary: alternate loading paths are not captured and bounded autonomous context may evict entries. Subagents still filter statically because their configured skills are all loaded into the session at startup. - **Injection (legacy / default)**: Enabled skills are listed in the agent system prompt with full metadata and container paths (`` block). Controlled by `skills.deferred_discovery: false` (default). @@ -625,6 +630,7 @@ E2B output sync records remote file versions and actual host file metadata in a - `skills/describe.py` — `build_describe_skill_tool(catalog)` builds the `describe_skill` tool as a closure; `build_skill_search_setup(skills, enabled, ...)` produces a `SkillSearchSetup(describe_skill_tool, skill_names)` that is wired into both the LangGraph agent factory (`agent.py`) and the embedded client (`client.py`). - **Slash activation**: `/skill-name task` loads that enabled skill's `SKILL.md` for the current model call only. The resolver rejects leading whitespace, missing separators, reserved channel commands (`/new`, `/help`, `/bootstrap`, `/status`, `/models`, `/memory`, `/goal`), disabled skills, and skills outside a custom agent's whitelist. - **Installation**: `POST /api/skills/install` extracts .skill ZIP archive to custom/ directory +- **Managed integrations**: Lark/Feishu CLI support installs one global official `lark-*` pack as read-only `SkillCategory.INTEGRATION` entries under `/mnt/skills/integrations/lark-cli/...`; enabled flags, app configuration, and OAuth data remain per-user. Install resolves the newest `larksuite/cli` release from GitHub (`releases/latest`) at install time (falling back to a bottom-line pinned version if the lookup fails) rather than hard-coding the pack version; integrity relies on the official host + structural archive guards + a recorded hash of the effective installed tree after shared guidance injection (not a pinned archive-byte SHA, which GitHub does not keep stable). The Gateway image still installs a pinned `@larksuite/cli` binary, so `get_lark_integration_status` surfaces `latest_available_version` and `runtime_version_mismatch` for the UI. AIO installs additionally verify and publish official Linux amd64/arm64 binaries under `{DEER_FLOW_HOME}/integrations/lark-cli/sandbox-cli`, mounted read-only at `/mnt/integrations/lark-cli/runtime`; `/mnt/integrations/lark-cli/config` (app credentials, incl. the long-lived `appSecret`) is mounted **read-only** into the sandbox and `/mnt/integrations/lark-cli/data` (refreshable OAuth tokens) stays writable, both mapping to owner-only per-user credential directories. **Sandbox trust boundary:** those two dirs are still *readable* by arbitrary sandbox processes (the agent's `bash` tool, or code reached via prompt-injection in a tool result), so the app secret and tokens are exposed to sandbox-side code even though they never reach the browser — the read-only config mount only prevents in-sandbox tampering, not read/exfiltration. The sidecar credential-broker follow-up (issue #4338) is the planned fix that removes these plaintext mounts from sandbox execution. `DEER_FLOW_LARK_CLI_SANDBOX_RUNTIME_DIR` supplies a validated, symlink-free pre-staged runtime for air-gapped deployments. For the remote provisioner (K8s), the runtime is instead provisioned by an optional init container + shared `emptyDir` (Pattern A): set `LARK_CLI_INIT_IMAGE` on the provisioner (see `docker/lark-cli-init/`) and the Gateway sends `provision_lark_cli_runtime` on sandbox create once the pack is installed, so remote installs skip the Gateway-side GitHub download entirely. `get_lark_integration_status(check_runtime=True)` surfaces `sandbox_runtime_mode` (`none` / `gateway-download` / `init-container`) and `sandbox_runtime_ready` (init-container mode reads the provisioner `GET /api/capabilities`) so a green UI can't hide a chat-time `lark-cli: command not found`. Cheap status probes are explicitly not live-verified; users authorize or reconnect through the browser device-flow endpoints instead of running terminal commands. - **SkillScan**: `packages/harness/deerflow/skills/skillscan/` is the native deterministic scanner for `.skill` archives and agent-managed skill writes. It runs offline before the LLM scanner, emits structured findings (`rule_id`, `severity`, `file`, `line`, `message`, `remediation`, redacted `evidence` — category/analyzer are encoded in the `rule_id` prefix), blocks `CRITICAL`, and passes warning findings into `scan_skill_content()`. `scan_archive_preflight()` / `scan_skill_dir()` are pure sync functions (dispatch off the event loop); `enforce_static_scan()` applies the blocking policy and the `skill_scan.enabled` kill switch. The Python instance-client signal deliberately follows only a one-level, same-scope evidence chain (PR #4265 review): a proven imported constructor bound to a simple name, optional name-to-name alias propagation, rebinding invalidation, and a constructor-supported outbound method or context-manager use; bare canonical-looking names never fall back to module identity. Nested scopes never inherit client handles and inherit only constructor aliases proven stable by a binding-only enclosing-scope prepass. Comprehensions, walrus-bearing statements, annotations, executable expressions inside complex binding targets, unsupported operations, and ambiguous flows produce no finding from this signal; skipped constructs invalidate all names they may bind, while representative false negatives are pinned by `test_python_declared_false_negatives_stay_unreported`. Compound bodies are walked from isolated copies so wrapping code in `if True:` is not a bypass, while copied scope entries, binding-only prepasses, and AST visits consume a deterministic work budget and the walk stops after its first sink. Budget or recursion exhaustion skips only this best-effort signal and retains deterministic findings already collected for the file. Do not add Semgrep/OpenGrep or YAML rule-engine dependencies to the core path; Phase 1 rule specs live in Python constants next to their analyzers in `skillscan/orchestrator.py`. - **Skill Review Core**: `packages/harness/deerflow/skills/review/` provides read-only package snapshots, deterministic facts, resource/eval analysis, report rendering, and the CLI (`python -m deerflow.skills.review.cli`). It reuses the shared frontmatter helper and SkillScan; it must not import `app.*`, execute target scripts, install dependencies, or call networks. JSON contracts live in `contracts/skill_review/`. The `review_skill_package` built-in tool labels results with `review_subject_entry` and never `skill_context_entry`, so reviewing a target does not activate it, bind its `required-secrets`, or apply its `allowed-tools`. Its model-visible `ToolMessage.content` is a compact JSON payload with untrusted control tags neutralized; the full raw review payload, including Markdown renders, stays in `ToolMessage.artifact`. CI should run the CLI with `--fail-on error --fail-on-incomplete` so blocker/error findings and truncated/not-assessed packages fail the gate. The public `skills/public/skill-reviewer` skill owns semantic readiness review and suggestions only; mutation and runtime experiments remain owned by `skill-creator`. diff --git a/backend/Dockerfile b/backend/Dockerfile index 110a182a3..a284bb19b 100644 --- a/backend/Dockerfile +++ b/backend/Dockerfile @@ -13,6 +13,8 @@ FROM python:3.12-slim-bookworm AS builder ARG NODE_MAJOR=22 ARG APT_MIRROR ARG UV_INDEX_URL +ARG NPM_REGISTRY +ARG LARK_CLI_NPM_VERSION=1.0.65 # Optional extras to install (e.g. "postgres" for PostgreSQL support) # Usage: docker build --build-arg UV_EXTRAS=postgres,discord ... ARG UV_EXTRAS @@ -36,6 +38,11 @@ RUN apt-get update && apt-get install -y \ && apt-get install -y nodejs \ && rm -rf /var/lib/apt/lists/* +# Install the official Lark/Feishu CLI used by the managed Lark integration. +RUN if [ -n "${NPM_REGISTRY}" ]; then npm config set registry "${NPM_REGISTRY}"; fi \ + && npm install -g @larksuite/cli@${LARK_CLI_NPM_VERSION} \ + && lark-cli --version + # Install uv (source image overridable via UV_IMAGE build arg) COPY --from=uv-source /uv /uvx /usr/local/bin/ @@ -95,7 +102,10 @@ ENV PYTHONIOENCODING=utf-8 COPY --from=builder /usr/bin/node /usr/bin/node COPY --from=builder /usr/lib/node_modules /usr/lib/node_modules RUN ln -s ../lib/node_modules/npm/bin/npm-cli.js /usr/bin/npm \ - && ln -s ../lib/node_modules/npm/bin/npx-cli.js /usr/bin/npx + && ln -s ../lib/node_modules/npm/bin/npx-cli.js /usr/bin/npx \ + && LARK_CLI_BIN="$(node -p 'const bin=require("/usr/lib/node_modules/@larksuite/cli/package.json").bin; typeof bin === "string" ? bin : (bin["lark-cli"] || Object.values(bin)[0])')" \ + && ln -s "../lib/node_modules/@larksuite/cli/${LARK_CLI_BIN#./}" /usr/bin/lark-cli \ + && lark-cli --version # Install Docker CLI (for DooD: allows starting sandbox containers via host Docker socket) COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker diff --git a/backend/app/gateway/app.py b/backend/app/gateway/app.py index 8fde90f27..08b16bc9d 100644 --- a/backend/app/gateway/app.py +++ b/backend/app/gateway/app.py @@ -25,6 +25,7 @@ from app.gateway.routers import ( feedback, github_webhooks, input_polish, + integrations, mcp, memory, models, @@ -516,6 +517,9 @@ This gateway provides runtime endpoints for agent runs plus custom endpoints for # Skills API is mounted at /api/skills app.include_router(skills.router) + # First-party integrations API is mounted at /api/integrations + app.include_router(integrations.router) + # Artifacts API is mounted at /api/threads/{thread_id}/artifacts app.include_router(artifacts.router) diff --git a/backend/app/gateway/routers/integrations.py b/backend/app/gateway/routers/integrations.py new file mode 100644 index 000000000..1620a57aa --- /dev/null +++ b/backend/app/gateway/routers/integrations.py @@ -0,0 +1,354 @@ +import asyncio +import logging + +from fastapi import APIRouter, Depends, HTTPException, Request +from pydantic import BaseModel, Field + +from app.gateway.deps import get_config, require_admin_user +from deerflow.agents.lead_agent.prompt import refresh_skills_system_prompt_cache_async +from deerflow.config.app_config import AppConfig +from deerflow.integrations.lark_cli import ( + LARK_AUTH_COMPLETE_DEFAULT_WAIT_SECONDS, + LARK_AUTH_COMPLETE_MAX_WAIT_SECONDS, + LARK_AUTH_COMPLETE_MIN_WAIT_SECONDS, + LarkAuthCompleteResult, + LarkAuthProbe, + LarkAuthStartResult, + LarkCliProbe, + LarkConfigCompleteResult, + LarkConfigStartResult, + LarkInstallResult, + LarkIntegrationStatus, + complete_lark_auth, + complete_lark_config, + get_lark_integration_status, + install_lark_integration, + start_lark_auth, + start_lark_config, +) +from deerflow.runtime.user_context import get_effective_user_id + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/api/integrations", tags=["integrations"]) + +_ADMIN_REQUIRED_DETAIL = "Admin privileges required to install integrations." + + +async def _is_admin_user(request: Request) -> bool: + """Non-raising admin check used to gate host-path disclosure in responses. + + Fails closed: any error (missing middleware state, auth failure) is treated + as non-admin so host paths are redacted rather than accidentally exposed. + """ + try: + await require_admin_user(request, detail=_ADMIN_REQUIRED_DETAIL) + except Exception: + return False + return True + + +class LarkCliProbeResponse(BaseModel): + available: bool = Field(..., description="Whether lark-cli is available to the Gateway, either managed by DeerFlow or on PATH") + path: str | None = Field(None, description="Resolved lark-cli executable path") + version: str | None = Field(None, description="lark-cli --version output") + error: str | None = Field(None, description="Probe failure message") + + +class LarkAuthProbeResponse(BaseModel): + status: str = Field(..., description="Auth status: authenticated, not_configured, unavailable, or error") + message: str | None = Field(None, description="Human-readable status detail") + user: str | None = Field(None, description="Authenticated Lark/Feishu user display value when available") + verified: bool = Field(False, description="Whether this status came from a live token verification") + + +class LarkIntegrationStatusResponse(BaseModel): + installed: bool = Field(..., description="Whether the managed Lark skill pack is installed") + version: str = Field(..., description="Installed Lark CLI skill-pack version (from manifest, resolved at install time)") + manifest_version: str | None = Field(None, description="Installed manifest version") + latest_available_version: str | None = Field(None, description="Newest larksuite/cli release version available on GitHub, when known") + runtime_version_mismatch: bool = Field(False, description="Whether the installed skill-pack version differs from the Gateway runtime lark-cli binary") + app_configured: bool = Field(..., description="Whether lark-cli has app_id/app_secret configured for this user") + app_id: str | None = Field(None, description="Configured Lark app ID") + app_brand: str | None = Field(None, description="Configured Lark brand: feishu or lark") + skills_expected: int = Field(..., description="Number of skills expected in the official pack") + skills_installed: int = Field(..., description="Number of installed managed Lark skills") + installed_skills: list[str] = Field(default_factory=list, description="Installed managed Lark skill names") + enabled_skills: list[str] = Field(default_factory=list, description="Installed Lark skills currently enabled for this user") + install_path: str = Field(..., description="Host path of the managed Lark skill pack") + cli: LarkCliProbeResponse + auth: LarkAuthProbeResponse + sandbox_runtime_mode: str = Field("none", description="How lark-cli is provisioned into the sandbox: none, gateway-download, or init-container") + sandbox_runtime_ready: bool = Field(False, description="Whether the sandbox lark-cli runtime is provisioned and usable at chat time") + sandbox_runtime_detail: str | None = Field(None, description="Human-readable reason when the sandbox runtime is not ready") + + +class LarkInstallResponse(BaseModel): + success: bool + installed_skills: list[str] + message: str + status: LarkIntegrationStatusResponse + + +class LarkAuthStartRequest(BaseModel): + recommend: bool = Field(default=False, description="Request the official recommended auto-approve scopes") + domains: list[str] = Field(default_factory=list, description="Optional Lark auth domains, e.g. calendar or docs") + scope: str | None = Field(default=None, description="Optional explicit OAuth scope string") + + +class LarkConfigStartRequest(BaseModel): + brand: str = Field(default="feishu", description="Lark brand to start app registration for: feishu or lark") + + +class LarkConfigStartResponse(BaseModel): + verification_url: str = Field(..., description="URL the user should open in a browser to configure the Lark app") + device_code: str = Field(..., description="Device code used by config/complete after browser approval") + expires_in: int | None = Field(None, description="Seconds before the configuration URL expires") + interval: int | None = Field(None, description="Suggested polling interval from Lark") + user_code: str | None = Field(None, description="Optional user code shown by Lark") + brand: str = Field(..., description="Brand used for this app registration flow") + + +class LarkConfigCompleteRequest(BaseModel): + device_code: str = Field(..., description="Device code returned by config/start") + brand: str = Field(default="feishu", description="Brand returned by config/start") + interval: int | None = Field(default=None, description="Polling interval returned by config/start") + expires_in: int | None = Field(default=None, description="Expiration returned by config/start") + + +class LarkConfigCompleteResponse(BaseModel): + success: bool + message: str + status: LarkIntegrationStatusResponse + + +class LarkAuthStartResponse(BaseModel): + verification_url: str = Field(..., description="URL the user should open in a browser to authorize") + device_code: str = Field(..., description="Device code used by the complete endpoint after browser approval") + expires_in: int | None = Field(None, description="Seconds before the authorization URL expires") + user_code: str | None = Field(None, description="Optional user code shown by Lark") + hint: str | None = Field(None, description="Optional guidance returned by lark-cli") + + +class LarkAuthCompleteRequest(BaseModel): + device_code: str = Field(..., description="Device code returned by auth/start") + wait_timeout_seconds: int = Field( + default=LARK_AUTH_COMPLETE_DEFAULT_WAIT_SECONDS, + ge=LARK_AUTH_COMPLETE_MIN_WAIT_SECONDS, + le=LARK_AUTH_COMPLETE_MAX_WAIT_SECONDS, + description="Maximum seconds for this device-code poll; automatic UI polling uses a shorter wait", + ) + + +class LarkAuthCompleteResponse(BaseModel): + success: bool + message: str + status: LarkIntegrationStatusResponse + + +def _cli_probe_to_response(probe: LarkCliProbe) -> LarkCliProbeResponse: + return LarkCliProbeResponse( + available=probe.available, + path=probe.path, + version=probe.version, + error=probe.error, + ) + + +def _auth_probe_to_response(probe: LarkAuthProbe) -> LarkAuthProbeResponse: + return LarkAuthProbeResponse( + status=probe.status, + message=probe.message, + user=probe.user, + verified=probe.verified, + ) + + +def _status_to_response(status: LarkIntegrationStatus, *, include_host_paths: bool = True) -> LarkIntegrationStatusResponse: + cli = _cli_probe_to_response(status.cli) + if not include_host_paths: + # Host filesystem paths (Gateway layout) are admin-only info; redact them + # for non-admin callers of the otherwise non-gated status/complete routes. + cli = cli.model_copy(update={"path": None}) + return LarkIntegrationStatusResponse( + installed=status.installed, + version=status.version, + manifest_version=status.manifest_version, + latest_available_version=status.latest_available_version, + runtime_version_mismatch=status.runtime_version_mismatch, + app_configured=status.app_configured, + app_id=status.app_id, + app_brand=status.app_brand, + skills_expected=status.skills_expected, + skills_installed=status.skills_installed, + installed_skills=list(status.installed_skills), + enabled_skills=list(status.enabled_skills), + install_path=status.install_path if include_host_paths else "", + cli=cli, + auth=_auth_probe_to_response(status.auth), + sandbox_runtime_mode=status.sandbox_runtime_mode, + sandbox_runtime_ready=status.sandbox_runtime_ready, + sandbox_runtime_detail=status.sandbox_runtime_detail, + ) + + +def _install_to_response(result: LarkInstallResult) -> LarkInstallResponse: + return LarkInstallResponse( + success=result.success, + installed_skills=list(result.installed_skills), + message=result.message, + status=_status_to_response(result.status), + ) + + +def _config_start_to_response(result: LarkConfigStartResult) -> LarkConfigStartResponse: + return LarkConfigStartResponse( + verification_url=result.verification_url, + device_code=result.device_code, + expires_in=result.expires_in, + interval=result.interval, + user_code=result.user_code, + brand=result.brand, + ) + + +def _config_complete_to_response(result: LarkConfigCompleteResult, *, include_host_paths: bool = True) -> LarkConfigCompleteResponse: + return LarkConfigCompleteResponse( + success=result.success, + message=result.message, + status=_status_to_response(result.status, include_host_paths=include_host_paths), + ) + + +def _auth_start_to_response(result: LarkAuthStartResult) -> LarkAuthStartResponse: + return LarkAuthStartResponse( + verification_url=result.verification_url, + device_code=result.device_code, + expires_in=result.expires_in, + user_code=result.user_code, + hint=result.hint, + ) + + +def _auth_complete_to_response(result: LarkAuthCompleteResult, *, include_host_paths: bool = True) -> LarkAuthCompleteResponse: + return LarkAuthCompleteResponse( + success=result.success, + message=result.message, + status=_status_to_response(result.status, include_host_paths=include_host_paths), + ) + + +@router.get("/lark/status", response_model=LarkIntegrationStatusResponse, summary="Get Lark/Feishu Integration Status") +async def get_lark_status(request: Request, config: AppConfig = Depends(get_config)) -> LarkIntegrationStatusResponse: + try: + status = await asyncio.to_thread(get_lark_integration_status, get_effective_user_id(), config, check_latest=True, check_runtime=True) + return _status_to_response(status, include_host_paths=await _is_admin_user(request)) + except Exception as e: + logger.error("Failed to get Lark integration status: %s", e, exc_info=True) + raise HTTPException(status_code=500, detail="Failed to get Lark integration status.") + + +@router.post("/lark/install", response_model=LarkInstallResponse, summary="Install Lark/Feishu Skill Pack") +async def install_lark(request: Request, config: AppConfig = Depends(get_config)) -> LarkInstallResponse: + await require_admin_user(request, detail=_ADMIN_REQUIRED_DETAIL) + try: + result = await asyncio.to_thread(install_lark_integration, get_effective_user_id(), config) + await refresh_skills_system_prompt_cache_async() + return _install_to_response(result) + except FileNotFoundError as e: + raise HTTPException(status_code=404, detail=str(e)) + except ValueError as e: + raise HTTPException(status_code=400, detail=str(e)) + except HTTPException: + raise + except Exception as e: + logger.error("Failed to install Lark integration: %s", e, exc_info=True) + raise HTTPException(status_code=500, detail="Failed to install Lark integration.") + + +@router.post("/lark/config/start", response_model=LarkConfigStartResponse, summary="Start Lark/Feishu App Configuration") +async def start_lark_app_config(body: LarkConfigStartRequest) -> LarkConfigStartResponse: + try: + result = await asyncio.to_thread( + start_lark_config, + get_effective_user_id(), + brand=body.brand, + ) + return _config_start_to_response(result) + except FileNotFoundError as e: + raise HTTPException(status_code=404, detail=str(e)) + except ValueError as e: + raise HTTPException(status_code=400, detail=str(e)) + except TimeoutError as e: + raise HTTPException(status_code=504, detail=str(e)) + except Exception as e: + logger.error("Failed to start Lark connection setup: %s", e, exc_info=True) + raise HTTPException(status_code=500, detail="Failed to start Lark connection setup.") + + +@router.post("/lark/config/complete", response_model=LarkConfigCompleteResponse, summary="Complete Lark/Feishu App Configuration") +async def complete_lark_app_config(request: Request, body: LarkConfigCompleteRequest, config: AppConfig = Depends(get_config)) -> LarkConfigCompleteResponse: + try: + result = await asyncio.to_thread( + complete_lark_config, + get_effective_user_id(), + config, + device_code=body.device_code, + brand=body.brand, + interval=body.interval, + expires_in=body.expires_in, + ) + return _config_complete_to_response(result, include_host_paths=await _is_admin_user(request)) + except FileNotFoundError as e: + raise HTTPException(status_code=404, detail=str(e)) + except ValueError as e: + raise HTTPException(status_code=400, detail=str(e)) + except TimeoutError as e: + raise HTTPException(status_code=504, detail=str(e)) + except Exception as e: + logger.error("Failed to complete Lark connection setup: %s", e, exc_info=True) + raise HTTPException(status_code=500, detail="Failed to complete Lark connection setup.") + + +@router.post("/lark/auth/start", response_model=LarkAuthStartResponse, summary="Start Lark/Feishu Browser Authorization") +async def start_lark_browser_auth(body: LarkAuthStartRequest) -> LarkAuthStartResponse: + try: + result = await asyncio.to_thread( + start_lark_auth, + get_effective_user_id(), + domains=tuple(body.domains), + scope=body.scope, + recommend=body.recommend, + ) + return _auth_start_to_response(result) + except FileNotFoundError as e: + raise HTTPException(status_code=404, detail=str(e)) + except ValueError as e: + raise HTTPException(status_code=400, detail=str(e)) + except TimeoutError as e: + raise HTTPException(status_code=504, detail=str(e)) + except Exception as e: + logger.error("Failed to start Lark authorization: %s", e, exc_info=True) + raise HTTPException(status_code=500, detail="Failed to start Lark authorization.") + + +@router.post("/lark/auth/complete", response_model=LarkAuthCompleteResponse, summary="Complete Lark/Feishu Browser Authorization") +async def complete_lark_browser_auth(request: Request, body: LarkAuthCompleteRequest, config: AppConfig = Depends(get_config)) -> LarkAuthCompleteResponse: + try: + result = await asyncio.to_thread( + complete_lark_auth, + get_effective_user_id(), + config, + device_code=body.device_code, + wait_timeout_seconds=body.wait_timeout_seconds, + ) + return _auth_complete_to_response(result, include_host_paths=await _is_admin_user(request)) + except FileNotFoundError as e: + raise HTTPException(status_code=404, detail=str(e)) + except ValueError as e: + raise HTTPException(status_code=400, detail=str(e)) + except TimeoutError as e: + raise HTTPException(status_code=504, detail=str(e)) + except Exception as e: + logger.error("Failed to complete Lark authorization: %s", e, exc_info=True) + raise HTTPException(status_code=500, detail="Failed to complete Lark authorization.") diff --git a/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py b/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py index 09bccad91..f3f7df088 100644 --- a/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py +++ b/backend/packages/harness/deerflow/community/aio_sandbox/aio_sandbox_provider.py @@ -39,6 +39,8 @@ from deerflow.community.warm_pool_lifecycle import ( ) from deerflow.config import get_app_config from deerflow.config.paths import VIRTUAL_PATH_PREFIX, get_paths, join_host_path +from deerflow.integrations.lark_cli import INTEGRATION_ID as LARK_CLI_INTEGRATION_ID +from deerflow.integrations.lark_cli import LARK_CLI_SANDBOX_CONFIG_DIR, LARK_CLI_SANDBOX_DATA_DIR, LARK_CLI_SANDBOX_RUNTIME_DIR, ensure_lark_cli_credential_tree, lark_skills_installed from deerflow.runtime.user_context import get_effective_user_id from deerflow.sandbox.sandbox import Sandbox from deerflow.sandbox.sandbox_provider import SandboxProvider @@ -744,7 +746,40 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): mounts.extend(skills_mounts) logger.info(f"Adding skills mounts: {skills_mounts}") - return mounts + user_skill_mounts = self._get_user_skill_mounts(user_id=user_id) + if user_skill_mounts: + mounts.extend(user_skill_mounts) + logger.info(f"Adding user skill mounts: {user_skill_mounts}") + + lark_cli_mounts = self._get_lark_cli_runtime_mounts(user_id=user_id) + if lark_cli_mounts: + mounts.extend(lark_cli_mounts) + logger.info(f"Adding Lark CLI runtime mounts: {lark_cli_mounts}") + + return self._dedupe_mounts_by_container_path(mounts) + + @staticmethod + def _dedupe_mounts_by_container_path(mounts: list[tuple[str, str, bool]]) -> list[tuple[str, str, bool]]: + """Keep the first mount for each container path. + + Duplicate container paths are rejected by the provisioner and can also + fail local Docker creation. The earlier mount wins because mount helpers + are appended in priority order: thread data, skill roots, integration + skill roots, then integration runtimes/credentials. + """ + seen: set[str] = set() + deduped: list[tuple[str, str, bool]] = [] + for host_path, container_path, read_only in mounts: + if container_path in seen: + logger.warning( + "Skipping duplicate sandbox mount for container path %s from host %s", + container_path, + host_path, + ) + continue + seen.add(container_path) + deduped.append((host_path, container_path, read_only)) + return deduped @staticmethod def _get_thread_mounts(thread_id: str, *, user_id: str | None = None) -> list[tuple[str, str, bool]]: @@ -839,6 +874,84 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): return mounts + @staticmethod + def _get_user_skill_mounts(*, user_id: str | None = None) -> list[tuple[str, str, bool]]: + """Mount managed integration skills into AIO sandboxes. + + Per-user custom skills are already mounted by ``_get_skills_mounts``. + This helper adds the shared integration skill root so sandbox paths match + the skill registry without duplicating ``/mnt/skills/custom``. + """ + try: + config = get_app_config() + paths = get_paths() + skills_container_path = config.skills.container_path + paths.integration_skills_dir().mkdir(parents=True, exist_ok=True) + return [ + (paths.host_integration_skills_dir(), f"{skills_container_path}/integrations", True), + ] + except Exception as e: + logger.warning(f"Could not setup user skill mounts: {e}") + return [] + + @staticmethod + def _lark_integration_active(user_id: str | None = None) -> bool: + """Whether the managed Lark skill pack is installed for this user. + + Drives whether a sandbox requests the lark-cli runtime (init container / + Gateway-download mount). Independent of whether a local ``sandbox-cli`` + dir exists, so remote/K8s can opt in without a Gateway-side download. + """ + try: + effective_user_id = AioSandboxProvider._effective_acquire_user_id(user_id) + return lark_skills_installed(effective_user_id) + except Exception as e: # pragma: no cover - defensive + logger.warning(f"Could not determine Lark integration state: {e}") + return False + + @staticmethod + def _get_lark_cli_runtime_mounts(*, user_id: str | None = None) -> list[tuple[str, str, bool]]: + """Mount the per-user lark-cli config/data dirs used by Settings auth. + + Settings endpoints run ``lark-cli`` on the Gateway with + ``LARKSUITE_CLI_CONFIG_DIR`` / ``DATA_DIR`` pointing at + ``users/{user}/integrations/lark-cli``. Agent conversations run + ``lark-cli`` inside the sandbox, so those same directories must be + mounted into the container or the CLI sees a separate unauthenticated + profile. + + The ``config`` dir holds the long-lived Lark ``appSecret`` (written by + ``lark-cli config init`` on the Gateway, never in-sandbox), so it is + mounted **read-only**: sandbox processes only need to read it, and a + read-only bind stops a compromised agent from tampering with or + replacing the app credentials. The ``data`` dir holds refreshable OAuth + tokens that ``lark-cli auth`` updates in-sandbox, so it stays writable. + This is defense-in-depth only — both dirs remain readable to arbitrary + sandbox processes until the auth-proxy follow-up (issue #4338) lands. + See the sandbox trust-boundary note in ``backend/AGENTS.md``. + """ + try: + paths = get_paths() + effective_user_id = AioSandboxProvider._effective_acquire_user_id(user_id) + ensure_lark_cli_credential_tree(effective_user_id, paths=paths) + mounts = [ + (paths.host_user_integration_config_dir(effective_user_id, LARK_CLI_INTEGRATION_ID), LARK_CLI_SANDBOX_CONFIG_DIR, True), + (paths.host_user_integration_data_dir(effective_user_id, LARK_CLI_INTEGRATION_ID), LARK_CLI_SANDBOX_DATA_DIR, False), + ] + runtime_dir = paths.base_dir / "integrations" / LARK_CLI_INTEGRATION_ID / "sandbox-cli" + if runtime_dir.is_dir(): + mounts.append( + ( + join_host_path(str(paths.host_base_dir), "integrations", LARK_CLI_INTEGRATION_ID, "sandbox-cli"), + LARK_CLI_SANDBOX_RUNTIME_DIR, + True, + ) + ) + return mounts + except Exception as e: + logger.warning(f"Could not setup Lark CLI runtime mounts: {e}") + return [] + # ── Idle timeout management ────────────────────────────────────────── def _cleanup_idle_resources(self, idle_timeout: float) -> None: @@ -1694,6 +1807,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): """ effective_user_id = self._effective_acquire_user_id(user_id) extra_mounts = self._get_extra_mounts(thread_id, user_id=effective_user_id) + provision_lark_cli_runtime = self._lark_integration_active(effective_user_id) # Enforce replicas: only warm-pool containers count toward eviction budget. # Active sandboxes are in use by live threads and must not be forcibly stopped. @@ -1702,7 +1816,13 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): evicted = self._evict_oldest_warm() self._log_replicas_soft_cap(replicas, sandbox_id, evicted) - info = self._backend.create(thread_id, sandbox_id, extra_mounts=extra_mounts or None, user_id=effective_user_id) + info = self._backend.create( + thread_id, + sandbox_id, + extra_mounts=extra_mounts or None, + user_id=effective_user_id, + provision_lark_cli_runtime=provision_lark_cli_runtime, + ) # Wait for sandbox to be ready if not wait_for_sandbox_ready(info.sandbox_url, timeout=60): @@ -1715,6 +1835,7 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): """Async counterpart to ``_create_sandbox``.""" effective_user_id = self._effective_acquire_user_id(user_id) extra_mounts = await asyncio.to_thread(self._get_extra_mounts, thread_id, user_id=effective_user_id) + provision_lark_cli_runtime = await asyncio.to_thread(self._lark_integration_active, effective_user_id) # Enforce replicas: only warm-pool containers count toward eviction budget. # Active sandboxes are in use by live threads and must not be forcibly stopped. @@ -1723,7 +1844,14 @@ class AioSandboxProvider(WarmPoolLifecycleMixin[SandboxInfo], SandboxProvider): evicted = await asyncio.to_thread(self._evict_oldest_warm) self._log_replicas_soft_cap(replicas, sandbox_id, evicted) - info = await asyncio.to_thread(self._backend.create, thread_id, sandbox_id, extra_mounts=extra_mounts or None, user_id=effective_user_id) + info = await asyncio.to_thread( + self._backend.create, + thread_id, + sandbox_id, + extra_mounts=extra_mounts or None, + user_id=effective_user_id, + provision_lark_cli_runtime=provision_lark_cli_runtime, + ) # Wait for sandbox to be ready without blocking the event loop. if not await wait_for_sandbox_ready_async(info.sandbox_url, timeout=60): diff --git a/backend/packages/harness/deerflow/community/aio_sandbox/backend.py b/backend/packages/harness/deerflow/community/aio_sandbox/backend.py index 0ff9cb3ea..28e614890 100644 --- a/backend/packages/harness/deerflow/community/aio_sandbox/backend.py +++ b/backend/packages/harness/deerflow/community/aio_sandbox/backend.py @@ -81,6 +81,7 @@ class SandboxBackend(ABC): extra_mounts: list[tuple[str, str, bool]] | None = None, *, user_id: str | None = None, + provision_lark_cli_runtime: bool = False, ) -> SandboxInfo: """Create/provision a new sandbox. @@ -90,6 +91,9 @@ class SandboxBackend(ABC): extra_mounts: Additional volume mounts as (host_path, container_path, read_only) tuples. Ignored by backends that don't manage containers (e.g., remote). user_id: User bucket that the sandbox should mount or provision for. + provision_lark_cli_runtime: Ask the backend to provision the sandbox + lark-cli runtime via its native mechanism (e.g. the provisioner's + init container + emptyDir). Backends that can't do this ignore it. Returns: SandboxInfo with connection details. diff --git a/backend/packages/harness/deerflow/community/aio_sandbox/local_backend.py b/backend/packages/harness/deerflow/community/aio_sandbox/local_backend.py index 6876a3ebb..29f211405 100644 --- a/backend/packages/harness/deerflow/community/aio_sandbox/local_backend.py +++ b/backend/packages/harness/deerflow/community/aio_sandbox/local_backend.py @@ -271,6 +271,7 @@ class LocalContainerBackend(SandboxBackend): extra_mounts: list[tuple[str, str, bool]] | None = None, *, user_id: str | None = None, + provision_lark_cli_runtime: bool = False, ) -> SandboxInfo: """Start a new container and return its connection info. @@ -280,6 +281,8 @@ class LocalContainerBackend(SandboxBackend): extra_mounts: Additional volume mounts as (host_path, container_path, read_only) tuples. user_id: User bucket already reflected in extra_mounts. Accepted for interface compatibility with remote backends. + provision_lark_cli_runtime: Ignored — the local backend provisions the + lark-cli runtime via the Gateway-download bind mount in extra_mounts. Returns: SandboxInfo with container details. @@ -287,7 +290,7 @@ class LocalContainerBackend(SandboxBackend): Raises: RuntimeError: If the container fails to start. """ - del user_id + del user_id, provision_lark_cli_runtime container_name = f"{self._container_prefix}-{sandbox_id}" # Retry loop: if Docker rejects the port (e.g. a stale container still diff --git a/backend/packages/harness/deerflow/community/aio_sandbox/remote_backend.py b/backend/packages/harness/deerflow/community/aio_sandbox/remote_backend.py index fb2692199..75e8372dd 100644 --- a/backend/packages/harness/deerflow/community/aio_sandbox/remote_backend.py +++ b/backend/packages/harness/deerflow/community/aio_sandbox/remote_backend.py @@ -29,6 +29,48 @@ from .sandbox_info import SandboxInfo logger = logging.getLogger(__name__) +_PROVISIONER_EXTRA_MOUNT_PATHS = { + "/mnt/acp-workspace", + "/mnt/skills/custom", + "/mnt/skills/integrations", + "/mnt/integrations/lark-cli/config", + "/mnt/integrations/lark-cli/data", + "/mnt/integrations/lark-cli/runtime", +} + +_LARK_CLI_RUNTIME_CONTAINER_PATH = "/mnt/integrations/lark-cli/runtime" + + +def _provisioner_extra_mounts_payload( + extra_mounts: list[tuple[str, str, bool]] | None, + *, + provision_lark_cli_runtime: bool = False, +) -> list[dict[str, object]]: + """Return only extra mounts the provisioner knows how to recreate safely. + + When ``provision_lark_cli_runtime`` is set, the provisioner supplies the + lark-cli runtime via an init container + emptyDir, so the runtime extra mount + is dropped here to avoid a colliding hostPath/PVC mount at the same path. The + per-user config/data credential mounts are always forwarded. + """ + if not extra_mounts: + return [] + + payload: list[dict[str, object]] = [] + for host_path, container_path, read_only in extra_mounts: + if container_path not in _PROVISIONER_EXTRA_MOUNT_PATHS: + continue + if provision_lark_cli_runtime and container_path == _LARK_CLI_RUNTIME_CONTAINER_PATH: + continue + payload.append( + { + "host_path": host_path, + "container_path": container_path, + "read_only": read_only, + } + ) + return payload + class RemoteSandboxBackend(SandboxBackend): """Backend that delegates sandbox lifecycle to the provisioner service. @@ -72,13 +114,20 @@ class RemoteSandboxBackend(SandboxBackend): extra_mounts: list[tuple[str, str, bool]] | None = None, *, user_id: str | None = None, + provision_lark_cli_runtime: bool = False, ) -> SandboxInfo: """Create a sandbox Pod + Service via the provisioner. Calls ``POST /api/sandboxes`` which creates a dedicated Pod + NodePort Service in k3s. """ - return self._provisioner_create(thread_id, sandbox_id, extra_mounts, user_id=user_id) + return self._provisioner_create( + thread_id, + sandbox_id, + extra_mounts, + user_id=user_id, + provision_lark_cli_runtime=provision_lark_cli_runtime, + ) def destroy(self, info: SandboxInfo) -> None: """Destroy a sandbox Pod + Service via the provisioner.""" @@ -149,20 +198,28 @@ class RemoteSandboxBackend(SandboxBackend): extra_mounts: list[tuple[str, str, bool]] | None = None, *, user_id: str | None = None, + provision_lark_cli_runtime: bool = False, ) -> SandboxInfo: """POST /api/sandboxes → create Pod + Service.""" - del extra_mounts effective_user_id = user_id or get_effective_user_id() include_legacy_skills = user_should_see_legacy_skills(effective_user_id) + payload = { + "sandbox_id": sandbox_id, + "thread_id": thread_id, + "user_id": effective_user_id, + "include_legacy_skills": include_legacy_skills, + "provision_lark_cli_runtime": provision_lark_cli_runtime, + } + provisioner_extra_mounts = _provisioner_extra_mounts_payload( + extra_mounts, + provision_lark_cli_runtime=provision_lark_cli_runtime, + ) + if provisioner_extra_mounts: + payload["extra_mounts"] = provisioner_extra_mounts try: resp = requests.post( f"{self._provisioner_url}/api/sandboxes", - json={ - "sandbox_id": sandbox_id, - "thread_id": thread_id, - "user_id": effective_user_id, - "include_legacy_skills": include_legacy_skills, - }, + json=payload, headers=self._auth_headers(), timeout=30, ) diff --git a/backend/packages/harness/deerflow/config/extensions_config.py b/backend/packages/harness/deerflow/config/extensions_config.py index 1e886387f..0f2040879 100644 --- a/backend/packages/harness/deerflow/config/extensions_config.py +++ b/backend/packages/harness/deerflow/config/extensions_config.py @@ -315,7 +315,7 @@ class ExtensionsConfig(BaseModel): skill_config = self.skills.get(skill_name) if skill_config is None: # Default to enabled for all skill categories - return skill_category in ("public", "custom", "legacy") + return skill_category in ("public", "custom", "legacy", "integrations") return skill_config.enabled diff --git a/backend/packages/harness/deerflow/config/paths.py b/backend/packages/harness/deerflow/config/paths.py index be52e1950..1d846f7e4 100644 --- a/backend/packages/harness/deerflow/config/paths.py +++ b/backend/packages/harness/deerflow/config/paths.py @@ -12,6 +12,7 @@ VIRTUAL_PATH_PREFIX = "/mnt/user-data" _SAFE_THREAD_ID_RE = re.compile(r"^[A-Za-z0-9_\-]+$") _SAFE_USER_ID_RE = re.compile(r"^[A-Za-z0-9_\-]+$") +_SAFE_INTEGRATION_ID_RE = re.compile(r"^[A-Za-z0-9_.\-]+$") _UNSAFE_USER_ID_CHAR_RE = re.compile(r"[^A-Za-z0-9_\-]") _SAFE_USER_ID_DIGEST_HEX_LEN = 16 @@ -37,6 +38,18 @@ def _validate_user_id(user_id: str) -> str: return user_id +def _validate_integration_id(integration_id: str) -> str: + """Validate an integration ID before using it in filesystem paths.""" + if not _SAFE_INTEGRATION_ID_RE.match(integration_id): + raise ValueError(f"Invalid integration_id {integration_id!r}: only alphanumeric characters, dots, hyphens, and underscores are allowed.") + # The charset allows dots for names like ``some.integration``; reject the + # bare ``.``/``..`` path components so a future caller cannot escape the + # per-integration namespace via ``_join_host_path(..., integration_id, ...)``. + if integration_id in {".", ".."}: + raise ValueError(f"Invalid integration_id {integration_id!r}: '.' and '..' are not allowed.") + return integration_id + + def make_safe_user_id(raw: str) -> str: """Normalize an external identity into the user-id charset (``[A-Za-z0-9_-]``). @@ -236,6 +249,15 @@ class Paths: """ return self.user_skills_dir(user_id) / "custom" + def integration_skills_dir(self) -> Path: + """Globally installed managed integration skills. + + Layout: ``{base_dir}/integrations/skills/{provider}/{skill}/``. The + package contents are shared and read-only; credentials and enabled + state remain user-scoped elsewhere under ``users/{user_id}``. + """ + return self.base_dir / "integrations" / "skills" + def thread_dir(self, thread_id: str, *, user_id: str | None = None) -> Path: """ Host path for a thread's data. @@ -325,6 +347,22 @@ class Paths: """Host path for the ACP workspace mount source.""" return _join_host_path(self.host_thread_dir(thread_id, user_id=user_id), "acp-workspace") + def host_user_custom_skills_dir(self, user_id: str) -> str: + """Host path for a user's custom skills directory, preserving Windows path syntax.""" + return _join_host_path(self._host_base_dir_str(), "users", _validate_user_id(user_id), "skills", "custom") + + def host_integration_skills_dir(self) -> str: + """Host path for globally installed managed integration skills.""" + return _join_host_path(self._host_base_dir_str(), "integrations", "skills") + + def host_user_integration_config_dir(self, user_id: str, integration_id: str) -> str: + """Host path for a user's managed integration runtime config directory.""" + return _join_host_path(self._host_base_dir_str(), "users", _validate_user_id(user_id), "integrations", _validate_integration_id(integration_id), "config") + + def host_user_integration_data_dir(self, user_id: str, integration_id: str) -> str: + """Host path for a user's managed integration runtime data directory.""" + return _join_host_path(self._host_base_dir_str(), "users", _validate_user_id(user_id), "integrations", _validate_integration_id(integration_id), "data") + def ensure_thread_dirs(self, thread_id: str, *, user_id: str | None = None) -> None: """Create all standard sandbox directories for a thread. diff --git a/backend/packages/harness/deerflow/integrations/__init__.py b/backend/packages/harness/deerflow/integrations/__init__.py new file mode 100644 index 000000000..83d0c5cb6 --- /dev/null +++ b/backend/packages/harness/deerflow/integrations/__init__.py @@ -0,0 +1 @@ +"""First-party integration installers and status helpers.""" diff --git a/backend/packages/harness/deerflow/integrations/lark_cli.py b/backend/packages/harness/deerflow/integrations/lark_cli.py new file mode 100644 index 000000000..1ed11413c --- /dev/null +++ b/backend/packages/harness/deerflow/integrations/lark_cli.py @@ -0,0 +1,1637 @@ +"""Managed Lark/Feishu CLI integration support. + +The integration installs the official ``lark-*`` AI-agent skills into a +global read-only managed integration skill directory. It deliberately +does not use the ordinary custom-skill archive path: this is a trusted, +versioned first-party integration package, not user-authored mutable content. + +Version resolution & integrity +------------------------------- +The installed skill-pack version follows the Gateway runtime ``lark-cli`` +binary version (``lark-cli --version``). This keeps the managed skills aligned +with the server-side CLI that will execute them. ``FALLBACK_LARK_CLI_VERSION`` +matches the Dockerfile/npm pin and is used only when the runtime binary is +unavailable or does not report a parseable version. + +Integrity is enforced without pinning a per-version archive byte hash (GitHub +does not guarantee source-archive bytes are stable across their internal git +upgrades, and pinning conflicts with tracking latest). Instead: + +* the download source is fixed to the official GitHub host over HTTPS and the + version only comes from the Gateway runtime CLI version or the pinned + fallback (no external URL injection); +* every archive member passes structural guards (zip-slip / symlink / + executable-binary / size / required-skill completeness / ``SKILL.md`` parse); +* a **content** SHA-256 over the extracted skill tree, after DeerFlow's shared + guidance is injected, is recorded in the manifest, so a reinstall whose + effective skill content changed is detectable/auditable even when GitHub + re-packs identical content with different archive bytes. + +Runtime coupling: the npm-installed ``lark-cli`` binary version is pinned in +``backend/Dockerfile`` (``ARG LARK_CLI_NPM_VERSION``) and +``docker/docker-compose*.yaml`` as a bootstrap fallback. The admin install path +also manages a writable DeerFlow-owned Gateway CLI under +``.deer-flow/integrations/lark-cli/gateway-cli`` and prefers it over the system +PATH, so users do not need to run terminal installation commands. Reinstalling +the integration refreshes both the managed Gateway CLI and the skill pack to the +same version when network access is available. ``get_lark_integration_status`` +surfaces ``latest_available_version`` and ``runtime_version_mismatch`` for +operators, and ``test_python_and_docker_lark_cli_versions_match`` pins the +fallback constant to the Dockerfile ARG so packaged deployments do not silently +diverge. +""" + +from __future__ import annotations + +import hashlib +import io +import json +import logging +import os +import posixpath +import re +import shutil +import subprocess +import tarfile +import tempfile +import threading +import time +import urllib.parse +import urllib.request +import zipfile +from contextlib import contextmanager +from dataclasses import dataclass +from datetime import UTC, datetime +from pathlib import Path, PurePosixPath +from typing import Any + +try: + import fcntl +except ImportError: # pragma: no cover - Windows fallback + fcntl = None # type: ignore[assignment] + import msvcrt + +from deerflow.config.app_config import AppConfig +from deerflow.config.paths import Paths, get_paths +from deerflow.skills.installer import is_executable_binary_prefix, is_symlink_member, is_unsafe_zip_member +from deerflow.skills.parser import parse_skill_file +from deerflow.skills.permissions import make_skill_tree_sandbox_readable +from deerflow.skills.types import SKILL_MD_FILE, SkillCategory + +logger = logging.getLogger(__name__) + +INTEGRATION_ID = "lark-cli" +# Matches the Gateway image/npm pin. Used when the runtime binary is unavailable +# or reports an unparsable version. +FALLBACK_LARK_CLI_VERSION = "v1.0.65" +LARK_CLI_NPM_VERSION = FALLBACK_LARK_CLI_VERSION.removeprefix("v") +LARK_CLI_NPM_PACKAGE = "@larksuite/cli" +LARK_CLI_GITHUB_REPO = "larksuite/cli" +LARK_CLI_LATEST_RELEASE_API = f"https://api.github.com/repos/{LARK_CLI_GITHUB_REPO}/releases/latest" +LARK_CLI_SOURCE_ARCHIVE_ENV = "DEER_FLOW_LARK_CLI_SKILLS_ARCHIVE" +LARK_CLI_SANDBOX_RUNTIME_SOURCE_ENV = "DEER_FLOW_LARK_CLI_SANDBOX_RUNTIME_DIR" +LARK_CLI_DOWNLOAD_TIMEOUT_SECONDS = 60 +LARK_CLI_NPM_INSTALL_TIMEOUT_SECONDS = 180 +LARK_HTTP_TIMEOUT_SECONDS = 20 +LARK_CONFIG_POLL_TIMEOUT_SECONDS = 45 +LARK_AUTH_COMPLETE_DEFAULT_WAIT_SECONDS = 45 +LARK_AUTH_COMPLETE_MIN_WAIT_SECONDS = 5 +LARK_AUTH_COMPLETE_MAX_WAIT_SECONDS = 45 +LARK_CLI_LATEST_VERSION_TTL_SECONDS = 3600 +LARK_CLI_MAX_ARCHIVE_BYTES = 128 * 1024 * 1024 +LARK_CLI_MAX_EXTRACTED_BYTES = 256 * 1024 * 1024 +LARK_CLI_MAX_RUNTIME_ASSET_BYTES = 128 * 1024 * 1024 +LARK_CLI_MANIFEST_FILE = ".deerflow-lark-cli-manifest.json" +LARK_CLI_SANDBOX_CONFIG_DIR = "/mnt/integrations/lark-cli/config" +LARK_CLI_SANDBOX_DATA_DIR = "/mnt/integrations/lark-cli/data" +LARK_CLI_SANDBOX_RUNTIME_DIR = "/mnt/integrations/lark-cli/runtime" +LARK_CLI_LINUX_ARCHES = ("amd64", "arm64") +LARK_CLI_RUNTIME_MANIFEST_FILE = ".deerflow-lark-cli-runtime.json" + +# Arch-dispatch launcher for the sandbox runtime layout. Shared by the Gateway +# writer (`_write_lark_cli_sandbox_launcher`) and the `docker/lark-cli-init` +# init image so the two producers of `bin/lark-cli` never drift. +LARK_CLI_SANDBOX_LAUNCHER_SCRIPT = """#!/bin/sh +set -eu +case "$(uname -m)" in + x86_64|amd64) arch=amd64 ;; + aarch64|arm64) arch=arm64 ;; + *) echo "Unsupported sandbox architecture: $(uname -m)" >&2; exit 126 ;; +esac +script_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +exec "$script_dir/../linux-$arch/lark-cli" "$@" +""" +_VERSION_TAG_RE = re.compile(r"v?\d+\.\d+\.\d+") +_DEERFLOW_LARK_SHARED_GUIDANCE_MARKER = "" +_DEERFLOW_LARK_SHARED_GUIDANCE_LEGACY_MARKERS = ("",) +_LARK_APP_REGISTRATION_PATH = "/oauth/v1/app/registration" + +LARK_SKILL_NAMES: tuple[str, ...] = ( + "lark-approval", + "lark-apps", + "lark-attendance", + "lark-base", + "lark-calendar", + "lark-contact", + "lark-doc", + "lark-drive", + "lark-event", + "lark-im", + "lark-mail", + "lark-markdown", + "lark-minutes", + "lark-note", + "lark-okr", + "lark-openapi-explorer", + "lark-shared", + "lark-sheets", + "lark-skill-maker", + "lark-slides", + "lark-task", + "lark-vc", + "lark-vc-agent", + "lark-whiteboard", + "lark-wiki", + "lark-workflow-meeting-summary", + "lark-workflow-standup-report", +) +LARK_SKILL_NAME_SET = frozenset(LARK_SKILL_NAMES) +_LARK_INSTALL_THREAD_LOCK = threading.Lock() +_LARK_RUNTIME_INSTALL_THREAD_LOCK = threading.Lock() + + +@dataclass(frozen=True) +class LarkCliProbe: + available: bool + path: str | None = None + version: str | None = None + error: str | None = None + + +@dataclass(frozen=True) +class LarkAuthProbe: + status: str + message: str | None = None + user: str | None = None + verified: bool = False + + +@dataclass(frozen=True) +class LarkIntegrationStatus: + installed: bool + version: str + manifest_version: str | None + latest_available_version: str | None + runtime_version_mismatch: bool + app_configured: bool + app_id: str | None + app_brand: str | None + skills_expected: int + skills_installed: int + installed_skills: tuple[str, ...] + enabled_skills: tuple[str, ...] + install_path: str + cli: LarkCliProbe + auth: LarkAuthProbe + sandbox_runtime_mode: str = "none" + sandbox_runtime_ready: bool = False + sandbox_runtime_detail: str | None = None + + +@dataclass(frozen=True) +class LarkInstallResult: + success: bool + installed_skills: tuple[str, ...] + status: LarkIntegrationStatus + message: str + + +@dataclass(frozen=True) +class LarkConfigStartResult: + verification_url: str + device_code: str + expires_in: int | None = None + interval: int | None = None + user_code: str | None = None + brand: str = "feishu" + + +@dataclass(frozen=True) +class LarkConfigCompleteResult: + success: bool + status: LarkIntegrationStatus + message: str + + +@dataclass(frozen=True) +class LarkAuthStartResult: + verification_url: str + device_code: str + expires_in: int | None = None + user_code: str | None = None + hint: str | None = None + + +@dataclass(frozen=True) +class LarkAuthCompleteResult: + success: bool + status: LarkIntegrationStatus + message: str + + +def lark_integration_root(_user_id: str | None = None) -> Path: + """Return the shared root for globally installed managed Lark skills. + + ``_user_id`` is accepted temporarily for source compatibility with the + pre-global-install API; it does not influence the shared package path. + """ + return get_paths().integration_skills_dir() / INTEGRATION_ID + + +def lark_manifest_path(user_id: str) -> Path: + return lark_integration_root(user_id) / LARK_CLI_MANIFEST_FILE + + +def lark_skills_installed(user_id: str | None = None) -> bool: + """Whether the managed Lark skill pack is installed. + + Mirrors the ``installed`` field of :func:`get_lark_integration_status` + (manifest present and ``lark-shared`` extracted) without probing the CLI or + auth. Used to decide whether a sandbox should request the lark-cli runtime. + """ + root = lark_integration_root(user_id) + manifest = _read_manifest(root) + if not manifest: + return False + return "lark-shared" in _installed_lark_skill_names(root) + + +def lark_cli_config_dir(user_id: str) -> Path: + return get_paths().user_dir(user_id) / "integrations" / INTEGRATION_ID / "config" + + +def lark_cli_data_dir(user_id: str) -> Path: + return get_paths().user_dir(user_id) / "integrations" / INTEGRATION_ID / "data" + + +def ensure_lark_cli_credential_tree(user_id: str, *, paths: Paths | None = None) -> None: + """Make the user's secret-bearing Lark CLI tree owner-only. + + The CLI writes plaintext app secrets and OAuth tokens beneath this tree. + Reject links before changing modes so a compromised tree cannot redirect a + chmod or subsequent CLI write outside the user's integration directory. + """ + paths = paths or get_paths() + root = paths.user_dir(user_id) / "integrations" / INTEGRATION_ID + if root.is_symlink(): + raise ValueError(f"Lark CLI credential path must not be a symlink: {root}") + root.mkdir(parents=True, exist_ok=True, mode=0o700) + root.chmod(0o700) + for required in (root / "config", root / "data"): + if required.is_symlink(): + raise ValueError(f"Lark CLI credential path must not be a symlink: {required}") + required.mkdir(parents=True, exist_ok=True, mode=0o700) + for path in root.rglob("*"): + if path.is_symlink(): + raise ValueError(f"Lark CLI credential path must not be a symlink: {path}") + if path.is_dir(): + path.chmod(0o700) + elif path.is_file(): + path.chmod(0o600) + else: + raise ValueError(f"Unsupported file type in Lark CLI credential tree: {path}") + + +def lark_cli_managed_gateway_dir() -> Path: + """Gateway-scoped DeerFlow-managed lark-cli install root.""" + return get_paths().base_dir / "integrations" / INTEGRATION_ID / "gateway-cli" + + +def lark_cli_managed_sandbox_dir() -> Path: + """Gateway-visible source directory mounted into Linux AIO sandboxes.""" + return get_paths().base_dir / "integrations" / INTEGRATION_ID / "sandbox-cli" + + +def _lark_cli_release_asset_name(version: str, arch: str) -> str: + tag = _normalize_lark_cli_version_tag(version) + if tag is None: + raise ValueError(f"Invalid Lark CLI version tag: {version!r}") + if arch not in LARK_CLI_LINUX_ARCHES: + raise ValueError(f"Unsupported Lark CLI Linux architecture: {arch!r}") + return f"lark-cli-{tag.removeprefix('v')}-linux-{arch}.tar.gz" + + +def _lark_cli_release_asset_url(version: str, asset_name: str) -> str: + tag = _normalize_lark_cli_version_tag(version) + if tag is None: + raise ValueError(f"Invalid Lark CLI version tag: {version!r}") + quoted_asset = urllib.parse.quote(asset_name, safe="") + return f"https://github.com/{LARK_CLI_GITHUB_REPO}/releases/download/{tag}/{quoted_asset}" + + +def _download_lark_release_asset(version: str, asset_name: str, *, max_bytes: int = LARK_CLI_MAX_RUNTIME_ASSET_BYTES) -> bytes: + """Download one official release asset with a strict size bound.""" + request = urllib.request.Request( + _lark_cli_release_asset_url(version, asset_name), + headers={"Accept": "application/octet-stream", "User-Agent": "deer-flow"}, + ) + try: + with urllib.request.urlopen(request, timeout=LARK_CLI_DOWNLOAD_TIMEOUT_SECONDS) as response: + chunks: list[bytes] = [] + total = 0 + while chunk := response.read(1024 * 1024): + total += len(chunk) + if total > max_bytes: + raise ValueError(f"Lark CLI release asset {asset_name!r} is too large.") + chunks.append(chunk) + except ValueError: + raise + except Exception as exc: # noqa: BLE001 - network boundary + raise ValueError(f"Could not download official Lark CLI release asset {asset_name!r} for {version}.") from exc + return b"".join(chunks) + + +def _release_checksums(raw: bytes) -> dict[str, str]: + checksums: dict[str, str] = {} + try: + text = raw.decode("utf-8") + except UnicodeDecodeError as exc: + raise ValueError("Lark CLI release checksums are not valid UTF-8.") from exc + for line in text.splitlines(): + parts = line.strip().split() + if len(parts) < 2 or not re.fullmatch(r"[0-9a-fA-F]{64}", parts[0]): + continue + checksums[parts[-1].lstrip("*")] = parts[0].lower() + return checksums + + +def _extract_lark_cli_runtime_binary(archive: bytes, destination: Path) -> None: + """Safely extract the single CLI executable from an official tar archive.""" + candidate: bytes | None = None + total = 0 + try: + with tarfile.open(fileobj=io.BytesIO(archive), mode="r:*") as tf: + for member in tf.getmembers(): + normalized = posixpath.normpath(member.name.replace("\\", "/")) + parts = PurePosixPath(normalized).parts + if normalized.startswith("/") or ".." in parts or member.issym() or member.islnk() or not (member.isdir() or member.isfile()): + raise ValueError(f"Unsafe Lark CLI runtime archive member: {member.name}") + if member.isfile(): + total += member.size + if total > LARK_CLI_MAX_RUNTIME_ASSET_BYTES: + raise ValueError("Lark CLI runtime archive expands beyond the allowed size.") + if PurePosixPath(normalized).name == "lark-cli": + extracted = tf.extractfile(member) + if extracted is None or candidate is not None: + raise ValueError("Lark CLI runtime archive must contain exactly one lark-cli executable.") + candidate = extracted.read() + except tarfile.TarError as exc: + raise ValueError("Lark CLI runtime archive is not a valid tar archive.") from exc + if not candidate: + raise ValueError("Lark CLI runtime archive does not contain a lark-cli executable.") + destination.parent.mkdir(parents=True, exist_ok=True) + destination.write_bytes(candidate) + destination.chmod(0o755) + + +def _write_lark_cli_sandbox_launcher(staging: Path) -> None: + launcher = staging / "bin" / "lark-cli" + launcher.parent.mkdir(parents=True, exist_ok=True) + launcher.write_text(LARK_CLI_SANDBOX_LAUNCHER_SCRIPT, encoding="utf-8") + launcher.chmod(0o755) + + +def _validate_lark_cli_sandbox_runtime(root: Path) -> None: + if root.is_symlink() or not root.is_dir(): + raise ValueError("Managed Lark CLI sandbox runtime root must be a regular directory, not a symlink.") + for path in root.rglob("*"): + if path.is_symlink(): + raise ValueError(f"Managed Lark CLI sandbox runtime must not contain a symlink: {path}") + if not (path.is_dir() or path.is_file()): + raise ValueError(f"Managed Lark CLI sandbox runtime contains an unsupported file type: {path}") + for relative in (Path("bin/lark-cli"), *(Path(f"linux-{arch}/lark-cli") for arch in LARK_CLI_LINUX_ARCHES)): + candidate = root / relative + if not candidate.is_file(): + raise ValueError(f"Managed Lark CLI sandbox runtime is missing a regular file: {relative}") + if candidate.stat().st_mode & 0o111 == 0: + raise ValueError(f"Managed Lark CLI sandbox runtime file is not executable: {relative}") + + +def _read_json_object_file(path: Path) -> dict[str, Any] | None: + if not path.is_file(): + return None + try: + parsed = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return None + return parsed if isinstance(parsed, dict) else None + + +@contextmanager +def _exclusive_install_lock(lock_path: Path, thread_lock): + """Hold one advisory file lock plus its in-process counterpart.""" + with thread_lock, lock_path.open("a+b") as lock_file: + lock_file.seek(0, os.SEEK_END) + if lock_file.tell() == 0: + lock_file.write(b"\0") + lock_file.flush() + lock_file.seek(0) + if fcntl is not None: + fcntl.flock(lock_file, fcntl.LOCK_EX) + else: # pragma: no cover - Windows fallback + msvcrt.locking(lock_file.fileno(), msvcrt.LK_LOCK, 1) + try: + yield + finally: + lock_file.seek(0) + if fcntl is not None: + fcntl.flock(lock_file, fcntl.LOCK_UN) + else: # pragma: no cover - Windows fallback + msvcrt.locking(lock_file.fileno(), msvcrt.LK_UNLCK, 1) + + +def _ensure_managed_sandbox_lark_cli(version: str) -> Path: + """Install verified official Linux binaries for AIO sandbox execution.""" + tag = _normalize_lark_cli_version_tag(version) + if tag is None: + raise ValueError(f"Invalid Lark CLI version tag: {version!r}") + target = lark_cli_managed_sandbox_dir() + parent = target.parent + parent.mkdir(parents=True, exist_ok=True) + with _exclusive_install_lock(parent / ".sandbox-cli.install.lock", _LARK_RUNTIME_INSTALL_THREAD_LOCK): + return _ensure_managed_sandbox_lark_cli_locked(tag, target, parent) + + +def _ensure_managed_sandbox_lark_cli_locked(tag: str, target: Path, parent: Path) -> Path: + manifest = _read_json_object_file(target / LARK_CLI_RUNTIME_MANIFEST_FILE) + if manifest and manifest.get("version") == tag: + _validate_lark_cli_sandbox_runtime(target) + return target + + staging = Path(tempfile.mkdtemp(prefix=".installing-sandbox-cli-", dir=str(parent))) + backup: Path | None = None + try: + source_override = os.getenv(LARK_CLI_SANDBOX_RUNTIME_SOURCE_ENV) + if source_override: + source = Path(source_override) + _validate_lark_cli_sandbox_runtime(source) + shutil.copytree(source, staging, dirs_exist_ok=True, symlinks=False) + else: + checksums = _release_checksums(_download_lark_release_asset(tag, "checksums.txt", max_bytes=1024 * 1024)) + for arch in LARK_CLI_LINUX_ARCHES: + asset_name = _lark_cli_release_asset_name(tag, arch) + archive = _download_lark_release_asset(tag, asset_name) + expected = checksums.get(asset_name) + actual = hashlib.sha256(archive).hexdigest() + if expected is None or actual != expected: + raise ValueError(f"Lark CLI release asset checksum mismatch: {asset_name}") + _extract_lark_cli_runtime_binary(archive, staging / f"linux-{arch}" / "lark-cli") + _write_lark_cli_sandbox_launcher(staging) + + _validate_lark_cli_sandbox_runtime(staging) + (staging / LARK_CLI_RUNTIME_MANIFEST_FILE).write_text( + json.dumps({"version": tag}, indent=2) + "\n", + encoding="utf-8", + ) + if target.exists(): + backup = parent / f".replacing-sandbox-cli-{os.getpid()}" + if backup.exists(): + shutil.rmtree(backup, ignore_errors=True) + target.rename(backup) + staging.rename(target) + if backup is not None: + shutil.rmtree(backup, ignore_errors=True) + return target + except Exception: + if backup is not None and backup.exists() and not target.exists(): + backup.rename(target) + raise + finally: + shutil.rmtree(staging, ignore_errors=True) + + +def _lark_cli_managed_bin_dir() -> Path: + return lark_cli_managed_gateway_dir() / "node_modules" / ".bin" + + +def _lark_cli_managed_path() -> str | None: + for name in ("lark-cli", "lark-cli.cmd"): + candidate = _lark_cli_managed_bin_dir() / name + if candidate.exists(): + return str(candidate) + return None + + +def lark_cli_env_overlay(user_id: str, *, sandbox_paths: bool = False) -> dict[str, str]: + """Environment overlay for lark-cli using DeerFlow-managed credentials. + + The directories are per-user so a local trusted-mode login cannot bleed + across accounts. Auth Proxy support can later replace these directories for + sandbox execution without changing the status API contract. + """ + if sandbox_paths: + config_dir: Path | str = LARK_CLI_SANDBOX_CONFIG_DIR + data_dir: Path | str = LARK_CLI_SANDBOX_DATA_DIR + else: + config_dir = lark_cli_config_dir(user_id) + data_dir = lark_cli_data_dir(user_id) + ensure_lark_cli_credential_tree(user_id) + overlay = { + "LARKSUITE_CLI_CONFIG_DIR": str(config_dir), + "LARKSUITE_CLI_DATA_DIR": str(data_dir), + "LARKSUITE_CLI_NO_UPDATE_NOTIFIER": "1", + "LARKSUITE_CLI_NO_SKILLS_NOTIFIER": "1", + } + if not sandbox_paths and _lark_cli_managed_path() is not None: + overlay["PATH"] = f"{_lark_cli_managed_bin_dir()}{os.pathsep}{os.environ.get('PATH', '')}" + elif sandbox_paths: + overlay["PATH"] = f"{LARK_CLI_SANDBOX_RUNTIME_DIR}/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" + return overlay + + +def lark_cli_env(user_id: str) -> dict[str, str]: + """Full environment for Gateway-side lark-cli probes.""" + return {**os.environ, **lark_cli_env_overlay(user_id)} + + +def probe_lark_cli() -> LarkCliProbe: + path = _resolve_lark_cli_path() + if path is None: + return LarkCliProbe(available=False, error="lark-cli is not installed on the Gateway") + return _probe_lark_cli_at_path(path) + + +def _probe_lark_cli_at_path(path: str) -> LarkCliProbe: + try: + result = subprocess.run( + [path, "--version"], + check=False, + capture_output=True, + text=True, + timeout=5, + ) + except Exception as exc: # noqa: BLE001 - probe boundary + return LarkCliProbe(available=False, path=path, error=str(exc)) + + output = (result.stdout or result.stderr or "").strip() + if result.returncode != 0: + return LarkCliProbe(available=False, path=path, error=output or f"exit code {result.returncode}") + return LarkCliProbe(available=True, path=path, version=output or None) + + +def probe_lark_auth(user_id: str, *, verify: bool = False) -> LarkAuthProbe: + """Probe the user's Lark authorization state. + + By default this only checks local token presence (``auth status --json``), + which is cheap and offline — suitable for the frequently-polled status + endpoint. Pass ``verify=True`` to add ``--verify`` for a live token check + against Lark; reserve that for the explicit "complete authorization" step + since it costs a network round-trip on every call. + """ + path = _resolve_lark_cli_path() + if path is None: + return LarkAuthProbe(status="unavailable", message="lark-cli is not installed on the Gateway") + app_config = read_lark_app_config(user_id) + if not app_config["configured"]: + return LarkAuthProbe(status="not_configured", message="Lark app is not configured") + args = [path, "auth", "status", "--json"] + if verify: + args.append("--verify") + try: + result = subprocess.run( + args, + check=False, + capture_output=True, + text=True, + timeout=8, + env=lark_cli_env(user_id), + ) + except subprocess.TimeoutExpired: + return LarkAuthProbe(status="error", message="lark-cli auth status timed out") + except Exception as exc: # noqa: BLE001 - probe boundary + return LarkAuthProbe(status="error", message=str(exc)) + + raw = (result.stdout or result.stderr or "").strip() + data: dict[str, Any] | None = None + if raw: + try: + parsed = json.loads(raw) + data = parsed if isinstance(parsed, dict) else None + except json.JSONDecodeError: + data = None + + if result.returncode != 0: + message = _auth_error_message(data) if data else raw + return LarkAuthProbe(status="not_authorized", message=message or "Lark user authorization is not configured") + + user = None + if data: + identities = data.get("identities") + if isinstance(identities, dict): + user_info = identities.get("user") + if isinstance(user_info, dict): + user = str(user_info.get("userName") or user_info.get("openId") or "") or None + if user is None and data.get("userName"): + user = str(data["userName"]) + if verify: + return LarkAuthProbe( + status="authenticated", + user=user, + message="Lark/Feishu authorization is live-verified.", + verified=True, + ) + return LarkAuthProbe( + status="authenticated", + user=user, + message="Lark/Feishu credentials are configured locally but not live-verified.", + verified=False, + ) + + +def _resolve_sandbox_runtime_readiness( + config: AppConfig, + *, + probe: bool, +) -> tuple[str, bool, str | None]: + """Resolve the sandbox lark-cli runtime mode and readiness. + + Modes: + - ``none``: sandboxes don't run lark-cli (non-AIO provider). + - ``gateway-download``: local AIO — the Gateway stages a ``sandbox-cli`` dir + and bind-mounts it; ready when that dir validates. + - ``init-container``: remote provisioner — a lark-cli init image provisions + the runtime; ready when the provisioner reports the init image is configured. + + ``probe`` gates the (best-effort, short-timeout) provisioner capability call. + """ + if not _uses_aio_sandbox(config): + return "none", False, "Sandbox does not run lark-cli in this configuration." + + if _uses_remote_provisioner(config): + if not probe: + return "init-container", False, None + init_image = _probe_provisioner_lark_cli_init_image(config) + if init_image is None: + return "init-container", False, "Could not reach the provisioner to confirm the lark-cli init image." + if not init_image: + return "init-container", False, "The provisioner has no lark-cli init image configured (LARK_CLI_INIT_IMAGE)." + return "init-container", True, None + + # Local AIO: Gateway-download runtime dir. + runtime_dir = lark_cli_managed_sandbox_dir() + try: + _validate_lark_cli_sandbox_runtime(runtime_dir) + except (ValueError, OSError): + return "gateway-download", False, "The managed sandbox lark-cli runtime is not installed." + return "gateway-download", True, None + + +def get_lark_integration_status( + user_id: str, + config: AppConfig, + *, + verify_auth: bool = False, + check_latest: bool = False, + check_runtime: bool = False, +) -> LarkIntegrationStatus: + root = lark_integration_root(user_id) + manifest = _read_manifest(root) + app_config = read_lark_app_config(user_id) + installed_skills = tuple(sorted(_installed_lark_skill_names(root))) + enabled_skills = tuple(sorted(_enabled_lark_skill_names(user_id, config))) + manifest_version = str(manifest.get("version")) if manifest else None + cli = probe_lark_cli() + latest_available = _cached_latest_lark_cli_version() if check_latest else None + runtime_mode, runtime_ready, runtime_detail = _resolve_sandbox_runtime_readiness(config, probe=check_runtime) + return LarkIntegrationStatus( + installed=bool(manifest) and "lark-shared" in installed_skills, + version=manifest_version or FALLBACK_LARK_CLI_VERSION, + manifest_version=manifest_version, + latest_available_version=latest_available, + runtime_version_mismatch=_versions_drifted(manifest_version, cli.version), + app_configured=bool(app_config["configured"]), + app_id=app_config["app_id"], + app_brand=app_config["brand"], + skills_expected=len(LARK_SKILL_NAMES), + skills_installed=len(installed_skills), + installed_skills=installed_skills, + enabled_skills=enabled_skills, + install_path=str(root), + cli=cli, + auth=probe_lark_auth(user_id, verify=verify_auth), + sandbox_runtime_mode=runtime_mode, + sandbox_runtime_ready=runtime_ready, + sandbox_runtime_detail=runtime_detail, + ) + + +def _normalize_version(value: str | None) -> str | None: + """Extract a comparable ``major.minor.patch`` from a version-ish string.""" + if not value: + return None + match = re.search(r"\d+\.\d+\.\d+", value) + return match.group(0) if match else None + + +def _versions_drifted(manifest_version: str | None, cli_version: str | None) -> bool: + """True when both versions are known and their numeric cores differ. + + The manifest records the installed skill-pack version; ``cli_version`` is + the Gateway runtime ``lark-cli`` binary. Unknown on either side means we + cannot claim a mismatch, so we stay quiet. + """ + left = _normalize_version(manifest_version) + right = _normalize_version(cli_version) + if left is None or right is None: + return False + return left != right + + +def _resolve_runtime_lark_cli_version() -> str: + """Resolve the skill-pack version that matches the Gateway runtime CLI. + + Managed Lark skills are executed by the server-side ``lark-cli`` binary, so + integration installs should align to that binary rather than blindly taking + GitHub's newest release. Packaged deployments install the pinned fallback in + the Gateway image; local/dev deployments can override this by putting a + newer ``lark-cli`` on the Gateway PATH and restarting the backend. + """ + cli = probe_lark_cli() + version = _normalize_version(cli.version) + return f"v{version}" if version is not None else FALLBACK_LARK_CLI_VERSION + + +def read_lark_app_config(user_id: str) -> dict[str, str | bool | None]: + ensure_lark_cli_credential_tree(user_id) + config_path = lark_cli_config_dir(user_id) / "config.json" + if not config_path.is_file(): + return {"configured": False, "app_id": None, "brand": None} + try: + data = json.loads(config_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return {"configured": False, "app_id": None, "brand": None} + if not isinstance(data, dict): + return {"configured": False, "app_id": None, "brand": None} + apps = data.get("apps") + if not isinstance(apps, list) or not apps: + return {"configured": False, "app_id": None, "brand": None} + current = data.get("currentApp") + app = None + if isinstance(current, str) and current: + app = next((candidate for candidate in apps if isinstance(candidate, dict) and (candidate.get("name") == current or candidate.get("appId") == current)), None) + if app is None: + app = apps[0] if isinstance(apps[0], dict) else None + if not isinstance(app, dict): + return {"configured": False, "app_id": None, "brand": None} + app_id = str(app.get("appId") or "").strip() + app_secret = app.get("appSecret") + brand = str(app.get("brand") or "feishu").strip() or "feishu" + return {"configured": bool(app_id and app_secret), "app_id": app_id or None, "brand": brand} + + +def install_lark_integration( + user_id: str, + config: AppConfig, + *, + source_archive: str | Path | None = None, +) -> LarkInstallResult: + env_archive = os.getenv(LARK_CLI_SOURCE_ARCHIVE_ENV) + if source_archive is not None: + archive_path = Path(source_archive) + resolved_version = None + created_temp_archive = False + elif env_archive: + archive_path = Path(env_archive) + resolved_version = None + created_temp_archive = False + else: + cli = _ensure_managed_gateway_lark_cli() + runtime_version = _normalize_version(cli.version) + resolved_version = f"v{runtime_version}" if runtime_version is not None else FALLBACK_LARK_CLI_VERSION + archive_path = _download_lark_archive(resolved_version) + created_temp_archive = True + + previous = _read_manifest(lark_integration_root(user_id)) + previous_content_sha = str(previous.get("content_sha256")) if previous else None + try: + installed_skills, content_sha = _install_lark_skills_from_archive(user_id, archive_path, version=resolved_version) + finally: + if created_temp_archive: + try: + archive_path.unlink(missing_ok=True) + except OSError: + pass + + if _uses_aio_sandbox(config) and not _uses_remote_provisioner(config): + installed_manifest = _read_manifest(lark_integration_root()) or {} + sandbox_version = str(installed_manifest.get("version") or resolved_version or FALLBACK_LARK_CLI_VERSION) + _ensure_managed_sandbox_lark_cli(sandbox_version) + + status = get_lark_integration_status(user_id, config) + content_changed = previous_content_sha is not None and previous_content_sha != content_sha + message = f"Installed {len(installed_skills)} Lark/Feishu skills." + if content_changed: + message += " Skill content changed since the previous install." + return LarkInstallResult( + success=True, + installed_skills=installed_skills, + status=status, + message=message, + ) + + +def _uses_aio_sandbox(config: AppConfig) -> bool: + sandbox = getattr(config, "sandbox", None) + use = getattr(sandbox, "use", None) + if use is None and isinstance(sandbox, dict): + use = sandbox.get("use") + return isinstance(use, str) and "aio_sandbox" in use.lower() + + +def _sandbox_config_value(config: AppConfig, key: str) -> str: + """Read a sandbox config value as a string, tolerating dict/attr configs.""" + sandbox = getattr(config, "sandbox", None) + value = getattr(sandbox, key, None) + if value is None and isinstance(sandbox, dict): + value = sandbox.get(key) + return str(value).strip() if value else "" + + +def _uses_remote_provisioner(config: AppConfig) -> bool: + """True when sandboxes are provisioned by a remote provisioner (K8s mode).""" + return bool(_sandbox_config_value(config, "provisioner_url")) + + +def _probe_provisioner_lark_cli_init_image(config: AppConfig) -> bool | None: + """Best-effort read of the provisioner's lark-cli init-image capability. + + Returns True/False when the provisioner answers, or None when it can't be + reached. Used only to surface a sandbox-runtime readiness signal; failures + degrade to "not ready" rather than raising. + """ + base = _sandbox_config_value(config, "provisioner_url") + if not base: + return None + api_key = _sandbox_config_value(config, "provisioner_api_key") + headers = {"X-API-Key": api_key} if api_key else {} + url = f"{base.rstrip('/')}/api/capabilities" + try: + request = urllib.request.Request(url, headers={"User-Agent": "deer-flow", **headers}) + with urllib.request.urlopen(request, timeout=5) as response: + payload = json.loads(response.read().decode("utf-8")) + return bool(payload.get("lark_cli_init_image")) + except Exception: + return None + + +def start_lark_config(user_id: str, *, brand: str = "feishu") -> LarkConfigStartResult: + """Start the browser flow that creates/binds a Lark OAuth app for this user.""" + parsed_brand = _normalize_lark_brand(brand) + begin_data = _request_lark_app_registration_begin(parsed_brand) + user_code = str(begin_data.get("user_code") or "").strip() + device_code = str(begin_data.get("device_code") or "").strip() + if not user_code or not device_code: + raise ValueError("Lark app registration did not return a user_code and device_code.") + verification_url = _build_lark_config_verification_url(parsed_brand, user_code) + return LarkConfigStartResult( + verification_url=verification_url, + device_code=device_code, + expires_in=_int_or_none(begin_data.get("expires_in")), + interval=_int_or_none(begin_data.get("interval")), + user_code=user_code, + brand=parsed_brand, + ) + + +def complete_lark_config( + user_id: str, + config: AppConfig, + *, + device_code: str, + brand: str = "feishu", + interval: int | None = None, + expires_in: int | None = None, +) -> LarkConfigCompleteResult: + """Complete app registration and persist app credentials through lark-cli.""" + device_code = device_code.strip() + if not device_code: + raise ValueError("device_code is required.") + parsed_brand = _normalize_lark_brand(brand) + result = _poll_lark_app_registration( + device_code=device_code, + brand=parsed_brand, + interval=interval or 5, + expires_in=expires_in or 300, + ) + if not result.get("client_secret") and _tenant_brand(result) == "lark": + # Lark CLI starts polling on the Feishu accounts host for both brands. + # For Lark tenants that response can include user_info.tenant_brand and + # client_id but omit client_secret; polling the Lark accounts host with + # the same device_code returns the complete app credentials. + result = _poll_lark_app_registration( + device_code=device_code, + brand="lark", + interval=interval or 5, + expires_in=expires_in or 300, + ) + + app_id = str(result.get("client_id") or "").strip() + app_secret = str(result.get("client_secret") or "").strip() + final_brand = _tenant_brand(result) or parsed_brand + if not app_id or not app_secret: + raise ValueError("Lark app registration succeeded but did not return app credentials.") + + _save_lark_app_config_with_cli(user_id, app_id=app_id, app_secret=app_secret, brand=final_brand) + status = get_lark_integration_status(user_id, config) + return LarkConfigCompleteResult( + success=True, + status=status, + message="Lark/Feishu connection setup completed.", + ) + + +def start_lark_auth( + user_id: str, + *, + domains: tuple[str, ...] = (), + scope: str | None = None, + recommend: bool = False, +) -> LarkAuthStartResult: + """Start a non-blocking Lark device authorization flow. + + The returned URL is safe to show in the browser UI or in a chat message. + ``device_code`` must be sent back to :func:`complete_lark_auth` after the + user finishes authorization in Lark/Feishu. + """ + path = _require_lark_cli_path() + args = [path, "auth", "login", "--no-wait", "--json"] + if recommend: + args.append("--recommend") + if scope: + args.extend(["--scope", scope]) + for domain in domains: + if domain: + args.extend(["--domain", domain]) + + data = _run_lark_cli_json(args, user_id=user_id, timeout=20) + verification_url = str(data.get("verification_url") or data.get("verification_uri_complete") or "").strip() + device_code = str(data.get("device_code") or "").strip() + if not verification_url or not device_code: + raise ValueError("lark-cli did not return a verification_url and device_code.") + + return LarkAuthStartResult( + verification_url=verification_url, + device_code=device_code, + expires_in=_int_or_none(data.get("expires_in")), + user_code=str(data.get("user_code") or "") or None, + hint=str(data.get("hint") or "") or None, + ) + + +def complete_lark_auth( + user_id: str, + config: AppConfig, + *, + device_code: str, + wait_timeout_seconds: int = LARK_AUTH_COMPLETE_DEFAULT_WAIT_SECONDS, +) -> LarkAuthCompleteResult: + """Complete a Lark device authorization flow after the user approves it.""" + device_code = device_code.strip() + if not device_code: + raise ValueError("device_code is required.") + if not LARK_AUTH_COMPLETE_MIN_WAIT_SECONDS <= wait_timeout_seconds <= LARK_AUTH_COMPLETE_MAX_WAIT_SECONDS: + raise ValueError(f"wait_timeout_seconds must be between {LARK_AUTH_COMPLETE_MIN_WAIT_SECONDS} and {LARK_AUTH_COMPLETE_MAX_WAIT_SECONDS}.") + + path = _require_lark_cli_path() + _run_lark_cli_json( + [path, "auth", "login", "--device-code", device_code, "--json"], + user_id=user_id, + timeout=wait_timeout_seconds, + allow_empty_success=True, + ) + status = get_lark_integration_status(user_id, config, verify_auth=True) + return LarkAuthCompleteResult( + success=status.auth.status == "authenticated", + status=status, + message="Lark/Feishu authorization completed." if status.auth.status == "authenticated" else (status.auth.message or "Lark/Feishu authorization status is still pending."), + ) + + +def _resolve_lark_cli_path() -> str | None: + return _lark_cli_managed_path() or shutil.which("lark-cli") + + +def _ensure_managed_gateway_lark_cli() -> LarkCliProbe: + """Install/update the DeerFlow-managed Gateway lark-cli. + + This is called by the admin install endpoint so non-technical users do not + need to install ``@larksuite/cli`` in a terminal. If npm/GitHub are not + reachable but an existing CLI is already available (managed or on PATH), we + keep using it and let the skill-pack install align to that runtime version. + """ + target_version = _resolve_latest_lark_cli_version() + current = probe_lark_cli() + current_version = _normalize_version(current.version) + if current.available and current_version == _normalize_version(target_version): + return current + + try: + return _install_managed_gateway_lark_cli(target_version) + except Exception: + fallback = probe_lark_cli() + if fallback.available: + logger.warning("Could not update managed lark-cli; using existing Gateway lark-cli", exc_info=True) + return fallback + raise + + +def _install_managed_gateway_lark_cli(version: str) -> LarkCliProbe: + normalized = _normalize_lark_cli_version_tag(version) + if normalized is None: + raise ValueError(f"Invalid Lark CLI npm version: {version!r}") + npm_version = normalized.removeprefix("v") + npm = shutil.which("npm") + if npm is None: + raise FileNotFoundError("npm is not available on the Gateway; cannot install managed @larksuite/cli.") + + install_root = lark_cli_managed_gateway_dir() + install_root.mkdir(parents=True, exist_ok=True) + # NOTE: this runs @larksuite/cli's install scripts (postinstall fetches the + # platform lark-cli binary), so `--ignore-scripts` is not viable here — the + # CLI would be unusable without it. The tradeoff: an admin-triggered install + # executes the official package's install scripts (and those of its deps) + # with Gateway privileges, so a supply-chain compromise of that package is + # the blast radius. This mirrors the pinned `npm install -g` in the Gateway + # Dockerfile; both are gated behind the admin install action. + result = subprocess.run( + [ + npm, + "install", + "--prefix", + str(install_root), + "--no-audit", + "--no-fund", + f"{LARK_CLI_NPM_PACKAGE}@{npm_version}", + ], + check=False, + capture_output=True, + text=True, + timeout=LARK_CLI_NPM_INSTALL_TIMEOUT_SECONDS, + env={**os.environ, "npm_config_update_notifier": "false"}, + ) + if result.returncode != 0: + raw = (result.stderr or result.stdout or "").strip() + raise ValueError(raw or f"npm install {LARK_CLI_NPM_PACKAGE}@{npm_version} exited with code {result.returncode}") + + path = _lark_cli_managed_path() + if path is None: + raise FileNotFoundError("Managed lark-cli install completed, but no lark-cli binary was found.") + probe = _probe_lark_cli_at_path(path) + if not probe.available: + raise ValueError(probe.error or "Managed lark-cli install did not produce a runnable CLI.") + return probe + + +def _require_lark_cli_path() -> str: + path = _resolve_lark_cli_path() + if path is None: + raise FileNotFoundError("lark-cli is not installed on the Gateway. Install the managed Lark integration as an admin, or rebuild the Gateway image with @larksuite/cli installed.") + return path + + +def _normalize_lark_brand(brand: str) -> str: + return "lark" if brand.strip().lower() == "lark" else "feishu" + + +def _lark_endpoints(brand: str) -> dict[str, str]: + if _normalize_lark_brand(brand) == "lark": + return { + "open": "https://open.larksuite.com", + "accounts": "https://accounts.larksuite.com", + } + return { + "open": "https://open.feishu.cn", + "accounts": "https://accounts.feishu.cn", + } + + +def _request_lark_app_registration_begin(brand: str) -> dict[str, Any]: + # lark-cli uses the Feishu accounts endpoint for the begin step, then + # switches to the tenant brand only if the poll response indicates Lark. + accounts_url = _lark_endpoints("feishu")["accounts"] + _LARK_APP_REGISTRATION_PATH + body = urllib.parse.urlencode( + { + "action": "begin", + "archetype": "PersonalAgent", + "auth_method": "client_secret", + "request_user_info": "open_id tenant_brand", + } + ).encode("utf-8") + data = _post_lark_form(accounts_url, body) + if "error" in data: + raise ValueError(str(data.get("error_description") or data.get("error") or "Lark app registration failed.")) + return data + + +def _build_lark_config_verification_url(brand: str, user_code: str) -> str: + base = f"{_lark_endpoints(brand)['open']}/page/cli" + # lpv/ocv mirror the *runtime* lark-cli client version doing the auth, which + # is the server-side Gateway binary — not the latest available skill-pack + # version. + runtime_version = _resolve_runtime_lark_cli_version() + query = urllib.parse.urlencode( + { + "user_code": user_code, + "lpv": runtime_version, + "ocv": runtime_version, + "from": "cli", + } + ) + return f"{base}?{query}" + + +def _poll_lark_app_registration( + *, + device_code: str, + brand: str, + interval: int, + expires_in: int, +) -> dict[str, Any]: + accounts_url = _lark_endpoints(brand)["accounts"] + _LARK_APP_REGISTRATION_PATH + deadline = time.monotonic() + min(max(expires_in, 1), LARK_CONFIG_POLL_TIMEOUT_SECONDS) + poll_interval = max(min(interval, 10), 1) + last_error = "authorization_pending" + while time.monotonic() < deadline: + body = urllib.parse.urlencode({"action": "poll", "device_code": device_code}).encode("utf-8") + data = _post_lark_form(accounts_url, body) + if not data.get("error") and data.get("client_id"): + return data + error = str(data.get("error") or "") + last_error = str(data.get("error_description") or error or "Lark app registration is still pending.") + if error == "authorization_pending": + time.sleep(poll_interval) + continue + if error == "slow_down": + poll_interval = min(poll_interval + 5, 30) + time.sleep(poll_interval) + continue + raise ValueError(last_error) + raise TimeoutError(f"Lark app registration is still pending: {last_error}") + + +def _post_lark_form(url: str, body: bytes) -> dict[str, Any]: + request = urllib.request.Request( + url, + data=body, + headers={"Content-Type": "application/x-www-form-urlencoded"}, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=LARK_HTTP_TIMEOUT_SECONDS) as response: + raw = response.read().decode("utf-8") + except Exception as exc: # noqa: BLE001 - network boundary + raise ValueError(f"Lark app registration request failed: {exc}") from exc + parsed = _parse_json_object(raw) + if parsed is None: + raise ValueError("Lark app registration returned non-JSON response.") + return parsed + + +def _tenant_brand(result: dict[str, Any]) -> str | None: + user_info = result.get("user_info") + if not isinstance(user_info, dict): + return None + brand = str(user_info.get("tenant_brand") or "").strip().lower() + return brand if brand in {"feishu", "lark"} else None + + +def _save_lark_app_config_with_cli(user_id: str, *, app_id: str, app_secret: str, brand: str) -> None: + path = _require_lark_cli_path() + try: + try: + result = subprocess.run( + [path, "config", "init", "--app-id", app_id, "--app-secret-stdin", "--brand", _normalize_lark_brand(brand)], + input=app_secret + "\n", + check=False, + capture_output=True, + text=True, + timeout=15, + env=lark_cli_env(user_id), + ) + except subprocess.TimeoutExpired as exc: + raise TimeoutError("Timed out while saving Lark connection setup.") from exc + finally: + ensure_lark_cli_credential_tree(user_id) + if result.returncode != 0: + raw = (result.stderr or result.stdout or "").strip() + parsed = _parse_json_object(raw) + message = _auth_error_message(parsed) if parsed else raw + raise ValueError(message or f"lark-cli config init exited with code {result.returncode}") + + +def _run_lark_cli_json( + args: list[str], + *, + user_id: str, + timeout: int, + allow_empty_success: bool = False, +) -> dict[str, Any]: + try: + try: + result = subprocess.run( + args, + check=False, + capture_output=True, + text=True, + timeout=timeout, + env=lark_cli_env(user_id), + ) + except subprocess.TimeoutExpired as exc: + raise TimeoutError("Timed out waiting for Lark/Feishu authorization. Complete authorization in the browser, then try again.") from exc + finally: + # OAuth commands may create new plaintext token files after the + # pre-command environment guard has run. Re-harden every file even on + # timeout or CLI failure before returning control to the Gateway. + ensure_lark_cli_credential_tree(user_id) + + stdout = (result.stdout or "").strip() + stderr = (result.stderr or "").strip() + raw = stdout or stderr + parsed = _parse_json_object(raw) + + if result.returncode != 0: + message = _auth_error_message(parsed) if parsed else raw + raise ValueError(message or f"lark-cli exited with code {result.returncode}") + + if not raw and allow_empty_success: + return {} + if parsed is None: + if allow_empty_success: + return {} + raise ValueError(raw or "lark-cli did not return JSON output.") + return parsed + + +def _parse_json_object(raw: str) -> dict[str, Any] | None: + if not raw: + return None + try: + data = json.loads(raw) + except json.JSONDecodeError: + return None + return data if isinstance(data, dict) else None + + +def _int_or_none(value: Any) -> int | None: + try: + return int(value) + except (TypeError, ValueError): + return None + + +def _auth_error_message(data: dict[str, Any] | None) -> str | None: + if not data: + return None + error = data.get("error") + if isinstance(error, dict): + for key in ("message", "hint", "type"): + value = error.get(key) + if value: + return str(value) + for key in ("message", "msg", "hint"): + value = data.get(key) + if value: + return str(value) + return None + + +def _read_manifest(root: Path) -> dict[str, Any] | None: + path = root / LARK_CLI_MANIFEST_FILE + if not path.is_file(): + return None + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return None + return data if isinstance(data, dict) else None + + +def _installed_lark_skill_names(root: Path) -> set[str]: + names: set[str] = set() + if not root.is_dir(): + return names + for skill_name in LARK_SKILL_NAMES: + if (root / skill_name / SKILL_MD_FILE).is_file(): + names.add(skill_name) + return names + + +def _enabled_lark_skill_names(user_id: str, config: AppConfig) -> set[str]: + from deerflow.skills.storage import get_or_new_user_skill_storage + + try: + storage = get_or_new_user_skill_storage(user_id, app_config=config) + return {skill.name for skill in storage.load_skills(enabled_only=True) if skill.name in LARK_SKILL_NAME_SET} + except Exception: + return set() + + +def _resolve_latest_lark_cli_version() -> str: + """Resolve the newest published ``larksuite/cli`` release tag. + + Queries the official ``releases/latest`` API. Any failure (rate limit, + offline, air-gapped, malformed payload) falls back to + ``FALLBACK_LARK_CLI_VERSION`` so an install can still proceed with a known + good version rather than aborting. + """ + try: + request = urllib.request.Request( + LARK_CLI_LATEST_RELEASE_API, + headers={"Accept": "application/vnd.github+json", "User-Agent": "deer-flow"}, + ) + with urllib.request.urlopen(request, timeout=LARK_HTTP_TIMEOUT_SECONDS) as response: + raw = response.read().decode("utf-8") + data = json.loads(raw) + tag = str(data.get("tag_name") or "").strip() if isinstance(data, dict) else "" + version = _normalize_lark_cli_version_tag(tag) + if version is not None: + return version + except Exception: # noqa: BLE001 - version discovery is best-effort + pass + return FALLBACK_LARK_CLI_VERSION + + +def _cached_latest_lark_cli_version() -> str | None: + """Best-effort latest version for status display, cached with a short TTL. + + Returns ``None`` on failure so the status endpoint never blocks the UI on a + GitHub outage; the install path uses :func:`_resolve_latest_lark_cli_version` + which has its own fallback. + """ + now = time.monotonic() + cached = getattr(_cached_latest_lark_cli_version, "_cache", None) + if cached is not None and now - cached[0] < LARK_CLI_LATEST_VERSION_TTL_SECONDS: + return cached[1] + try: + request = urllib.request.Request( + LARK_CLI_LATEST_RELEASE_API, + headers={"Accept": "application/vnd.github+json", "User-Agent": "deer-flow"}, + ) + with urllib.request.urlopen(request, timeout=LARK_HTTP_TIMEOUT_SECONDS) as response: + data = json.loads(response.read().decode("utf-8")) + tag = str(data.get("tag_name") or "").strip() if isinstance(data, dict) else "" + version = _normalize_lark_cli_version_tag(tag) + except Exception: # noqa: BLE001 - status probe is best-effort + version = None + _cached_latest_lark_cli_version._cache = (now, version) # type: ignore[attr-defined] + return version + + +def _lark_archive_url(version: str) -> str: + tag = _normalize_lark_cli_version_tag(version) + if tag is None: + raise ValueError(f"Invalid Lark CLI version tag: {version!r}") + return f"https://codeload.github.com/{LARK_CLI_GITHUB_REPO}/zip/refs/tags/{tag}" + + +def _normalize_lark_cli_version_tag(value: str | None) -> str | None: + tag = (value or "").strip() + if not _VERSION_TAG_RE.fullmatch(tag): + return None + return tag if tag.startswith("v") else f"v{tag}" + + +def _download_lark_archive(version: str) -> Path: + fd, archive_name = tempfile.mkstemp(prefix="lark-cli-skills-", suffix=".zip") + os.close(fd) + archive_path = Path(archive_name) + url = _lark_archive_url(version) + try: + with urllib.request.urlopen(url, timeout=LARK_CLI_DOWNLOAD_TIMEOUT_SECONDS) as response: + total = 0 + with archive_path.open("wb") as out: + while chunk := response.read(1024 * 1024): + total += len(chunk) + if total > LARK_CLI_MAX_ARCHIVE_BYTES: + raise ValueError("Lark CLI source archive is too large.") + out.write(chunk) + except ValueError: + archive_path.unlink(missing_ok=True) + raise + except Exception as exc: # noqa: BLE001 - network boundary + archive_path.unlink(missing_ok=True) + raise ValueError(f"Could not download the Lark skill pack ({version}) from GitHub. Check the Gateway's internet access, or pre-stage the archive via {LARK_CLI_SOURCE_ARCHIVE_ENV}.") from exc + return archive_path + + +def _content_sha256(root: Path, skill_names: set[str]) -> str: + """SHA-256 over effective installed skill contents (not archive bytes). + + The caller computes this after injecting DeerFlow's shared guidance, so the + digest covers both official extracted files and the guidance users/agents + actually read. It remains stable across GitHub re-packs of identical + content. Paths and bytes are hashed in sorted order for determinism. + """ + digest = hashlib.sha256() + for skill_name in sorted(skill_names): + skill_dir = root / skill_name + if not skill_dir.is_dir(): + continue + for file_path in sorted(p for p in skill_dir.rglob("*") if p.is_file()): + rel = file_path.relative_to(root).as_posix() + digest.update(rel.encode("utf-8")) + digest.update(b"\0") + digest.update(file_path.read_bytes()) + digest.update(b"\0") + return digest.hexdigest() + + +def _infer_lark_archive_version(zf: zipfile.ZipFile) -> str | None: + """Infer version from GitHub source archive roots such as ``cli-1.0.65/``. + + This keeps air-gapped / pre-staged archives from being mislabeled as the + fallback version when the archive itself clearly identifies its release. + """ + for info in zf.infolist(): + normalized = posixpath.normpath(info.filename.replace("\\", "/")) + parts = PurePosixPath(normalized).parts + if not parts: + continue + match = re.fullmatch(r"cli-(\d+\.\d+\.\d+)", parts[0]) + if match: + return f"v{match.group(1)}" + return None + + +def _install_lark_skills_from_archive(user_id: str, archive_path: Path, *, version: str | None = None) -> tuple[tuple[str, ...], str]: + if not archive_path.is_file(): + raise FileNotFoundError(f"Lark CLI skills archive not found: {archive_path}") + + parent = get_paths().integration_skills_dir() + parent.mkdir(parents=True, exist_ok=True) + with _lark_install_lock(parent): + return _install_lark_skills_from_archive_locked(archive_path, parent, version=version) + + +@contextmanager +def _lark_install_lock(parent: Path): + """Serialize the cross-process atomic replacement of the global pack.""" + with _exclusive_install_lock(parent / ".lark-cli.install.lock", _LARK_INSTALL_THREAD_LOCK): + yield + + +def _install_lark_skills_from_archive_locked( + archive_path: Path, + parent: Path, + *, + version: str | None = None, +) -> tuple[tuple[str, ...], str]: + target = parent / INTEGRATION_ID + staging_parent = Path(tempfile.mkdtemp(prefix=".installing-lark-cli-", dir=str(parent))) + staging_target = staging_parent / INTEGRATION_ID + staging_target.mkdir(parents=True, exist_ok=True) + + backup: Path | None = None + try: + with zipfile.ZipFile(archive_path, "r") as zf: + archive_version = version or _infer_lark_archive_version(zf) + extracted = _extract_lark_skills(zf, staging_target) + _validate_extracted_lark_skills(staging_target, extracted) + _append_deerflow_lark_shared_guidance(staging_target) + content_sha = _content_sha256(staging_target, extracted) + _write_manifest(staging_target, extracted, version=archive_version, content_sha256=content_sha) + make_skill_tree_sandbox_readable(staging_target) + + if target.exists(): + backup = parent / f".replacing-{INTEGRATION_ID}-{os.getpid()}" + if backup.exists(): + shutil.rmtree(backup) + target.rename(backup) + staging_target.rename(target) + if backup is not None: + # Best-effort: the new skills are already live after the rename, so a + # transient error deleting the old backup must not flip a successful + # install into a failure (the except-branch restore guard would also + # not fire because ``target`` now exists with the new content). + shutil.rmtree(backup, ignore_errors=True) + return tuple(sorted(extracted)), content_sha + except Exception: + if backup is not None and backup.exists() and not target.exists(): + backup.rename(target) + raise + finally: + shutil.rmtree(staging_parent, ignore_errors=True) + + +def _extract_lark_skills(zf: zipfile.ZipFile, destination: Path) -> set[str]: + extracted: set[str] = set() + total_written = 0 + dest_root = destination.resolve() + + for info in zf.infolist(): + if info.is_dir(): + continue + if is_unsafe_zip_member(info) or is_symlink_member(info): + raise ValueError(f"Unsafe Lark CLI archive member: {info.filename!r}") + + skill_name, relative = _resolve_lark_skill_member(info.filename) + if skill_name is None or relative is None: + continue + + target = dest_root / skill_name / relative + if not target.resolve().is_relative_to(dest_root): + raise ValueError(f"Archive member escapes destination: {info.filename!r}") + target.parent.mkdir(parents=True, exist_ok=True) + + with zf.open(info) as src, target.open("wb") as out: + first_chunk = True + while chunk := src.read(65536): + if first_chunk and is_executable_binary_prefix(chunk): + raise ValueError(f"Archive contains executable binary member: {info.filename!r}") + first_chunk = False + total_written += len(chunk) + if total_written > LARK_CLI_MAX_EXTRACTED_BYTES: + raise ValueError("Lark CLI skills archive expands to too much data.") + out.write(chunk) + extracted.add(skill_name) + + return extracted + + +def _resolve_lark_skill_member(raw_name: str) -> tuple[str | None, Path | None]: + normalized = posixpath.normpath(raw_name.replace("\\", "/")) + if normalized in {"", "."} or normalized.startswith("../"): + return None, None + parts = PurePosixPath(normalized).parts + + if "skills" in parts: + idx = parts.index("skills") + if len(parts) <= idx + 2: + return None, None + skill_name = parts[idx + 1] + rel_parts = parts[idx + 2 :] + elif parts and parts[0] in LARK_SKILL_NAME_SET: + skill_name = parts[0] + rel_parts = parts[1:] + else: + return None, None + + if skill_name not in LARK_SKILL_NAME_SET or not rel_parts: + return None, None + if any(part in {"", ".", ".."} for part in rel_parts): + raise ValueError(f"Unsafe Lark skill archive member: {raw_name!r}") + return skill_name, Path(*rel_parts) + + +def _validate_extracted_lark_skills(root: Path, extracted: set[str]) -> None: + missing = sorted(set(LARK_SKILL_NAMES) - extracted) + if missing: + raise ValueError(f"Lark CLI archive is missing required skills: {', '.join(missing)}") + + for skill_name in LARK_SKILL_NAMES: + skill_file = root / skill_name / SKILL_MD_FILE + parsed = parse_skill_file(skill_file, SkillCategory.INTEGRATION, relative_path=Path(INTEGRATION_ID) / skill_name) + if parsed is None: + raise ValueError(f"Invalid Lark skill metadata: {skill_name}/{SKILL_MD_FILE}") + if parsed.name != skill_name: + raise ValueError(f"Lark skill directory {skill_name!r} declares name {parsed.name!r}") + + +def _append_deerflow_lark_shared_guidance(root: Path) -> None: + skill_file = root / "lark-shared" / SKILL_MD_FILE + content = skill_file.read_text(encoding="utf-8") + if _DEERFLOW_LARK_SHARED_GUIDANCE_MARKER in content: + return + for legacy_marker in _DEERFLOW_LARK_SHARED_GUIDANCE_LEGACY_MARKERS: + if legacy_marker in content: + content = content.split(legacy_marker, maxsplit=1)[0].rstrip() + break + guidance = f""" + +{_DEERFLOW_LARK_SHARED_GUIDANCE_MARKER} + +## DeerFlow 授权入口 + +在 DeerFlow 中，如果 `lark-cli auth status` 或业务命令提示未配置、未登录、token 过期或缺少用户授权： + +1. 不要要求用户在终端执行 `lark-cli config init`、`lark-cli auth login` 或 `lark-cli auth login --device-code`。 +2. 回复用户这个可点击链接：[打开飞书授权设置](?settings=integrations)。 +3. 告诉用户在 **Settings → Integrations → Lark / Feishu CLI** 点击“连接飞书”，在浏览器里完成授权后再回来继续当前任务。 +4. 如果错误中包含缺失的 `scope`、`permission_violations` 或建议的 `--domain`，告诉用户在该设置页选择对应权限域（例如日历选择 Calendar），或把具体 scope 填入“Exact OAuth scope / 具体 OAuth scope”后重新授权。 + +只有在用户明确说明已经完成授权后，才继续调用具体的 `lark-cli` 业务命令。 +""" + skill_file.write_text(content.rstrip() + guidance + "\n", encoding="utf-8") + + +def _write_manifest(root: Path, installed_skills: set[str], *, version: str | None, content_sha256: str) -> None: + resolved_version = version or FALLBACK_LARK_CLI_VERSION + manifest = { + "provider": INTEGRATION_ID, + "version": resolved_version, + "source": _lark_archive_url(resolved_version), + "content_sha256": content_sha256, + "installed_at": datetime.now(UTC).isoformat(), + "skills": sorted(installed_skills), + } + (root / LARK_CLI_MANIFEST_FILE).write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") diff --git a/backend/packages/harness/deerflow/sandbox/local/local_sandbox_provider.py b/backend/packages/harness/deerflow/sandbox/local/local_sandbox_provider.py index c02f2cb32..470f690bd 100644 --- a/backend/packages/harness/deerflow/sandbox/local/local_sandbox_provider.py +++ b/backend/packages/harness/deerflow/sandbox/local/local_sandbox_provider.py @@ -136,6 +136,7 @@ class LocalSandboxProvider(SandboxProvider): _RESERVED_CONTAINER_PREFIXES = [ f"{container_path}/public", f"{container_path}/custom", + f"{container_path}/integrations", f"{container_path}/legacy", _ACP_WORKSPACE_VIRTUAL_PREFIX, _USER_DATA_VIRTUAL_PREFIX, @@ -287,7 +288,9 @@ class LocalSandboxProvider(SandboxProvider): config = get_app_config() skills_container_path = config.skills.container_path user_custom_path = paths.user_custom_skills_dir(effective_user_id) + integrations_path = paths.integration_skills_dir() user_custom_path.mkdir(parents=True, exist_ok=True) + integrations_path.mkdir(parents=True, exist_ok=True) mappings.append( PathMapping( @@ -296,6 +299,13 @@ class LocalSandboxProvider(SandboxProvider): read_only=True, ) ) + mappings.append( + PathMapping( + container_path=f"{skills_container_path}/integrations", + local_path=str(integrations_path), + read_only=True, + ) + ) except Exception as exc: logger.warning("Could not setup per-thread custom skills mount: %s", exc, exc_info=True) diff --git a/backend/packages/harness/deerflow/sandbox/tools.py b/backend/packages/harness/deerflow/sandbox/tools.py index 4955fb7e4..48db29f7b 100644 --- a/backend/packages/harness/deerflow/sandbox/tools.py +++ b/backend/packages/harness/deerflow/sandbox/tools.py @@ -163,6 +163,7 @@ def _extract_skill_name_from_skills_path(path: str) -> str | None: /mnt/skills/public/bootstrap/SKILL.md → "bootstrap" /mnt/skills/custom/my-skill/SKILL.md → "my-skill" /mnt/skills/legacy/my-skill/references/... → "my-skill" + /mnt/skills/integrations/lark-cli/lark-doc/SKILL.md → "lark-doc" /mnt/skills/public/bootstrap/ → "bootstrap" Returns None if the path doesn't contain a recognizable skill name pattern. """ @@ -173,16 +174,22 @@ def _extract_skill_name_from_skills_path(path: str) -> str | None: relative = path[len(skills_prefix) :].lstrip("/") if not relative: return None - # Expected patterns: "public//...", "custom//...", "legacy//..." + # Expected patterns: "public//...", "custom//...", + # "legacy//...", "integrations///..." # or "/..." (direct skill access). Empty segments are dropped so a # directory entry ("public/", as `ls` emits for dirs) is still recognized as # a category root rather than yielding an empty skill name. parts = [part for part in relative.split("/") if part] if len(parts) >= 2 and parts[0] in ("public", "custom", "legacy"): return parts[1] - if len(parts) == 1 and parts[0] in ("public", "custom", "legacy"): + if len(parts) >= 3 and parts[0] == "integrations": + return parts[2] + if len(parts) == 1 and parts[0] in ("public", "custom", "legacy", "integrations"): # Category root like /mnt/skills/custom — not a skill path. return None + if len(parts) == 2 and parts[0] == "integrations": + # Provider root like /mnt/skills/integrations/lark-cli. + return None if len(parts) >= 1: # Direct path like /mnt/skills/my-skill/SKILL.md return parts[0] @@ -215,6 +222,8 @@ def _is_disabled_skill_path(path: str, *, user_id: str | None = None) -> bool: category = "custom" elif relative.startswith("legacy/"): category = "legacy" + elif relative.startswith("integrations/"): + category = "integrations" else: # Try to infer from storage effective_uid = user_id or get_effective_user_id() @@ -307,7 +316,7 @@ def _resolve_skills_path(path: str) -> str: relative = path[len(skills_container) :].lstrip("/") - # Per-user custom skills: resolve to user-specific directory. + # Per-user custom and globally managed integration skills. # ``skill_manage_tool`` writes custom skills to the per-user directory, # and ``LocalSandboxProvider._build_thread_path_mappings`` mounts # ``/mnt/skills/custom`` to that same per-user dir. Without this @@ -327,6 +336,22 @@ def _resolve_skills_path(path: str) -> str: return str(user_custom_dir / custom_relative) return str(user_custom_dir) + if relative == "integrations" or relative.startswith("integrations/"): + from deerflow.config.paths import get_paths + + paths = get_paths() + integrations_dir = paths.integration_skills_dir() + integrations_relative = relative[len("integrations") :].lstrip("/") + if not integrations_relative: + return str(integrations_dir) + # Defense-in-depth: even though _reject_path_traversal runs upstream for + # sandbox callers, confirm the resolved path stays within the global + # integration dir so a lexical ``../`` cannot escape it here. + resolved = (integrations_dir / integrations_relative).resolve() + if not resolved.is_relative_to(integrations_dir.resolve()): + raise PermissionError("Access denied: path traversal detected") + return str(integrations_dir / integrations_relative) + return _join_path_preserving_style(skills_host, relative) @@ -758,11 +783,11 @@ def mask_local_paths_in_output(output: str, thread_data: ThreadDataState | None) """Mask host absolute paths from local sandbox output using virtual paths. Handles user-data paths (per-thread), skills paths (global + per-user - custom), and ACP workspace paths (per-thread). + custom + managed integrations), and ACP workspace paths (per-thread). """ # Build the ordered (host_base, virtual_base) source list. Order is # preserved from the original implementation: skills, then per-user - # custom skills, then ACP workspace, then user-data mappings (longest + # custom/integration skills, then ACP workspace, then user-data mappings (longest # host path first). Custom mount host paths are masked by # LocalSandbox._reverse_resolve_paths_in_output(). sources: list[tuple[str, str]] = [] @@ -782,9 +807,13 @@ def mask_local_paths_in_output(output: str, thread_data: ThreadDataState | None) user_id = get_effective_user_id() user_custom_dir = get_paths().user_custom_skills_dir(user_id) + integrations_dir = get_paths().integration_skills_dir() if user_custom_dir.exists(): skills_container = _get_skills_container_path() sources.append((str(user_custom_dir), f"{skills_container}/custom")) + if integrations_dir.exists(): + skills_container = _get_skills_container_path() + sources.append((str(integrations_dir), f"{skills_container}/integrations")) except Exception: pass @@ -1697,6 +1726,30 @@ def _github_env_from_runtime(runtime: Runtime) -> dict[str, str] | None: return {"GH_TOKEN": token, "GITHUB_TOKEN": token} +_LARK_CLI_COMMAND_RE = re.compile(r"(? dict[str, str] | None: + """Expose Settings-page Lark auth to sandbox ``lark-cli`` commands. + + Settings authorizes ``lark-cli`` under DeerFlow's per-user integration + config/data directories. Agent conversations invoke ``lark-cli`` through the + sandbox, so lark commands must receive those same directories or they see an + unrelated unauthenticated profile. Keep this scoped to commands that + actually call ``lark-cli`` so ordinary bash calls do not switch AIO into the + env-bearing execution path. + """ + if not _LARK_CLI_COMMAND_RE.search(command): + return None + try: + from deerflow.integrations.lark_cli import lark_cli_env_overlay + + return lark_cli_env_overlay(resolve_runtime_user_id(runtime), sandbox_paths=sandbox_paths) + except Exception: + logger.warning("Could not build Lark CLI env overlay; running command without managed auth", exc_info=True) + return None + + @tool("bash", parse_docstring=True) def bash_tool(runtime: Runtime, description: str, command: str) -> str: """Execute a bash command in a Linux environment. @@ -1723,8 +1776,11 @@ def bash_tool(runtime: Runtime, description: str, command: str) -> str: injected_env = read_active_secrets(getattr(runtime, "context", None)) or None identity_prefix = _channel_identity_prefix(runtime) github_env = _github_env_from_runtime(runtime) + lark_cli_env = _lark_cli_env_from_runtime(runtime, command, sandbox_paths=not is_local_sandbox(runtime)) if github_env: injected_env = {**(injected_env or {}), **github_env} + if lark_cli_env: + injected_env = {**(injected_env or {}), **lark_cli_env} if is_local_sandbox(runtime): if not is_host_bash_allowed(): return f"Error: {LOCAL_HOST_BASH_DISABLED_MESSAGE}" diff --git a/backend/packages/harness/deerflow/skills/storage/user_scoped_skill_storage.py b/backend/packages/harness/deerflow/skills/storage/user_scoped_skill_storage.py index 785be182a..123ee43a4 100644 --- a/backend/packages/harness/deerflow/skills/storage/user_scoped_skill_storage.py +++ b/backend/packages/harness/deerflow/skills/storage/user_scoped_skill_storage.py @@ -8,6 +8,7 @@ Layout:: /public//SKILL.md ← global, read-only //SKILL.md ← per-user, read-write + ///SKILL.md ← per-user, read-only /.history/.jsonl ← per-user history /_skill_states.json ← per-user enabled state //SKILL.md ← legacy fallback, read-only @@ -82,6 +83,7 @@ class UserScopedSkillStorage(LocalSkillStorage): self._user_id = _validate_user_id(user_id) paths = get_paths() self._user_custom_root: Path = paths.user_custom_skills_dir(self._user_id) + self._integrations_root: Path = paths.integration_skills_dir() self._user_skills_root: Path = paths.user_skills_dir(self._user_id) self._global_custom_root: Path = self._host_root / SkillCategory.CUSTOM.value self._skill_states_file: Path = self._user_skills_root / "_skill_states.json" @@ -256,8 +258,11 @@ class UserScopedSkillStorage(LocalSkillStorage): is_global_custom_fallback = (self._global_custom_root / normalized_name / SKILL_MD_FILE).exists() if is_global_public: raise ValueError(f"'{name}' is a built-in skill. Use the skill_manage tool to create your own version — it will shadow the built-in one.") + is_integration = any((candidate / SKILL_MD_FILE).exists() for candidate in self._integrations_root.glob(f"*/{normalized_name}") if candidate.is_dir()) if is_global_custom_fallback: raise ValueError(f"'{name}' is a legacy shared skill (not editable). To customise it, create your own version with the same name — it will shadow the shared one.") + if is_integration: + raise ValueError(f"'{name}' is a managed integration skill and cannot be edited. Create a custom skill with another name if you need a modified workflow.") raise FileNotFoundError(f"Custom skill '{name}' not found.") def _iter_skill_files(self) -> Iterable[tuple[SkillCategory, Path, Path]]: @@ -271,7 +276,17 @@ class UserScopedSkillStorage(LocalSkillStorage): dir_names.clear() yield SkillCategory.PUBLIC, public_path, Path(current_root) / SKILL_MD_FILE - # 2. Custom skills: prefer user-level directory + # 2. Managed integration skills: globally installed, read-only. Their + # enabled state is still merged from this user's _skill_states.json. + integration_path = self._integrations_root + if integration_path.exists() and integration_path.is_dir(): + for current_root, dir_names, file_names in os.walk(integration_path, followlinks=True): + dir_names[:] = sorted(name for name in dir_names if not name.startswith(".")) + if SKILL_MD_FILE not in file_names: + continue + yield SkillCategory.INTEGRATION, integration_path, Path(current_root) / SKILL_MD_FILE + + # 3. Custom skills: prefer user-level directory user_custom_exists = False user_custom_path = self._user_custom_root if user_custom_path.exists() and user_custom_path.is_dir(): @@ -283,7 +298,7 @@ class UserScopedSkillStorage(LocalSkillStorage): user_custom_exists = True yield SkillCategory.CUSTOM, user_custom_path, Path(current_root) / SKILL_MD_FILE - # 3. Fallback: if user has no custom skills, load from global custom + # 4. Fallback: if user has no custom skills, load from global custom # as LEGACY (read-only) so legacy skills are visible but not # editable/deletable by the user. LEGACY skills are mounted at # /mnt/skills/legacy// in the sandbox so their supporting @@ -370,6 +385,10 @@ class UserScopedSkillStorage(LocalSkillStorage): """Host path to this user's custom skills root directory.""" return self._user_custom_root + def get_user_integrations_root(self) -> Path: + """Host path to this user's managed integration skills root directory.""" + return self._user_integrations_root + # ------------------------------------------------------------------ # Path validation — accept per-user custom root as well as global root # ------------------------------------------------------------------ @@ -382,7 +401,7 @@ class UserScopedSkillStorage(LocalSkillStorage): would reject them. This override allows both roots. """ resolved_file = skill_file.resolve() - for allowed_root in (self._host_root.resolve(), self._user_custom_root.resolve()): + for allowed_root in (self._host_root.resolve(), self._user_custom_root.resolve(), self._user_integrations_root.resolve()): try: resolved_file.relative_to(allowed_root) return resolved_file diff --git a/backend/packages/harness/deerflow/skills/types.py b/backend/packages/harness/deerflow/skills/types.py index b9ea5fe5c..53b67a981 100644 --- a/backend/packages/harness/deerflow/skills/types.py +++ b/backend/packages/harness/deerflow/skills/types.py @@ -12,6 +12,7 @@ class SkillCategory(StrEnum): - ``PUBLIC``: built-in skill bundled with the platform, read-only. - ``CUSTOM``: user-authored skill that can be edited or deleted. + - ``INTEGRATION``: managed third-party integration skill, read-only. - ``LEGACY``: global custom skill from before user-isolation migration, presented as read-only (visible but not editable/deletable). These skills are mounted at ``/mnt/skills/legacy//`` in the sandbox. @@ -19,6 +20,7 @@ class SkillCategory(StrEnum): PUBLIC = "public" CUSTOM = "custom" + INTEGRATION = "integrations" LEGACY = "legacy" diff --git a/backend/tests/blocking_io/test_integrations_router.py b/backend/tests/blocking_io/test_integrations_router.py new file mode 100644 index 000000000..7f8fa3482 --- /dev/null +++ b/backend/tests/blocking_io/test_integrations_router.py @@ -0,0 +1,130 @@ +"""Regression anchors: integrations router must not block the event loop. + +The Lark integration handlers are async FastAPI route handlers, but the work +they dispatch includes zip reads, filesystem staging, manifest writes, and +``lark-cli`` subprocess calls. Those phases must stay behind +``asyncio.to_thread``; if a future refactor runs them inline, the strict +Blockbuster gate raises ``BlockingError`` and these anchors fail. +""" + +from __future__ import annotations + +import asyncio +import os +import zipfile +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from app.gateway.routers import integrations +from deerflow.config import paths as paths_module +from deerflow.integrations import lark_cli + +pytestmark = pytest.mark.asyncio + + +def _skill_content(name: str) -> str: + return f"---\nname: {name}\ndescription: {name} integration skill\n---\n\n# {name}\n" + + +def _build_lark_archive(archive: Path) -> None: + archive.parent.mkdir(parents=True, exist_ok=True) + with zipfile.ZipFile(archive, "w") as zf: + for skill_name in lark_cli.LARK_SKILL_NAMES: + zf.writestr(f"cli-1.0.65/skills/{skill_name}/SKILL.md", _skill_content(skill_name)) + zf.writestr(f"cli-1.0.65/skills/{skill_name}/references/readme.md", f"# {skill_name}\n") + + +def _write_stub_lark_cli(path: Path) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + """#!/bin/sh +if [ "$1" = "--version" ]; then + echo "v1.0.65" + exit 0 +fi + +if [ "$1" = "auth" ] && [ "$2" = "login" ]; then + echo "{}" + exit 0 +fi + +if [ "$1" = "auth" ] && [ "$2" = "status" ]; then + echo '{"identities":{"user":{"userName":"Alice"}}}' + exit 0 +fi + +echo "{}" +exit 0 +""", + encoding="utf-8", + ) + path.chmod(0o755) + + +def _reset_paths(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("DEER_FLOW_HOME", str(tmp_path / "home")) + monkeypatch.setattr(paths_module, "_paths", None) + + +async def _config(tmp_path: Path) -> SimpleNamespace: + skills_root = tmp_path / "skills" + await asyncio.to_thread((skills_root / "public").mkdir, parents=True, exist_ok=True) + await asyncio.to_thread((skills_root / "custom").mkdir, parents=True, exist_ok=True) + return SimpleNamespace( + skills=SimpleNamespace( + get_skills_path=lambda: skills_root, + container_path="/mnt/skills", + use="deerflow.skills.storage.local_skill_storage:LocalSkillStorage", + ) + ) + + +async def test_lark_install_route_does_not_block_event_loop(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + _reset_paths(tmp_path, monkeypatch) + config = await _config(tmp_path) + archive = tmp_path / "fixtures" / "lark-cli.zip" + await asyncio.to_thread(_build_lark_archive, archive) + + async def _allow_admin(*_args, **_kwargs) -> None: + return None + + refresh_calls = 0 + + async def _refresh_cache() -> None: + nonlocal refresh_calls + refresh_calls += 1 + + monkeypatch.setenv(lark_cli.LARK_CLI_SOURCE_ARCHIVE_ENV, str(archive)) + monkeypatch.setattr(lark_cli, "probe_lark_cli", lambda: lark_cli.LarkCliProbe(available=True, path="/usr/bin/lark-cli", version="v1.0.65")) + monkeypatch.setattr(lark_cli, "probe_lark_auth", lambda _user_id, **_kwargs: lark_cli.LarkAuthProbe(status="not_configured", message="not configured")) + monkeypatch.setattr(integrations, "get_effective_user_id", lambda: "loop-user") + monkeypatch.setattr(integrations, "require_admin_user", _allow_admin) + monkeypatch.setattr(integrations, "refresh_skills_system_prompt_cache_async", _refresh_cache, raising=False) + + response = await integrations.install_lark(request=None, config=config) + + assert response.success is True + assert refresh_calls == 1 + install_root = await asyncio.to_thread(lark_cli.lark_integration_root, "loop-user") + assert await asyncio.to_thread((install_root / "lark-doc" / "SKILL.md").exists) + + +async def test_lark_auth_complete_route_does_not_block_event_loop(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + _reset_paths(tmp_path, monkeypatch) + config = await _config(tmp_path) + cli_path = tmp_path / "bin" / "lark-cli" + await asyncio.to_thread(_write_stub_lark_cli, cli_path) + + monkeypatch.setenv("PATH", f"{cli_path.parent}{os.pathsep}{os.environ.get('PATH', '')}") + monkeypatch.setattr(integrations, "get_effective_user_id", lambda: "loop-user") + + response = await integrations.complete_lark_browser_auth( + request=None, + body=integrations.LarkAuthCompleteRequest(device_code="device-code"), + config=config, + ) + + assert response.status.cli.available is True + assert response.status.cli.version == "v1.0.65" diff --git a/backend/tests/test_aio_sandbox_provider.py b/backend/tests/test_aio_sandbox_provider.py index f34e0d999..4da31b581 100644 --- a/backend/tests/test_aio_sandbox_provider.py +++ b/backend/tests/test_aio_sandbox_provider.py @@ -2,6 +2,7 @@ import asyncio import importlib +import stat from types import SimpleNamespace from unittest.mock import MagicMock, patch @@ -119,6 +120,108 @@ def test_get_thread_mounts_uses_explicit_user_id(tmp_path, monkeypatch): assert container_paths["/mnt/user-data/outputs"] == str(tmp_path / "users" / "ou-user" / "threads" / "thread-4" / "user-data" / "outputs") +def test_get_lark_cli_runtime_mounts_uses_user_auth_dirs(tmp_path, monkeypatch): + """Sandbox lark-cli commands must read the same auth dirs as Settings.""" + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + lark_cli = importlib.import_module("deerflow.integrations.lark_cli") + monkeypatch.setattr(aio_mod, "get_paths", lambda: Paths(base_dir=tmp_path)) + monkeypatch.setattr(aio_mod, "get_effective_user_id", lambda: "default") + runtime_dir = tmp_path / "integrations" / "lark-cli" / "sandbox-cli" + runtime_dir.mkdir(parents=True) + + mounts = aio_mod.AioSandboxProvider._get_lark_cli_runtime_mounts(user_id="alice") + container_paths = {container_path: (host_path, read_only) for host_path, container_path, read_only in mounts} + + assert container_paths[lark_cli.LARK_CLI_SANDBOX_CONFIG_DIR] == ( + str(tmp_path / "users" / "alice" / "integrations" / "lark-cli" / "config"), + True, + ) + assert container_paths[lark_cli.LARK_CLI_SANDBOX_DATA_DIR] == ( + str(tmp_path / "users" / "alice" / "integrations" / "lark-cli" / "data"), + False, + ) + assert stat.S_IMODE((tmp_path / "users" / "alice" / "integrations" / "lark-cli" / "config").stat().st_mode) == 0o700 + assert stat.S_IMODE((tmp_path / "users" / "alice" / "integrations" / "lark-cli" / "data").stat().st_mode) == 0o700 + assert container_paths["/mnt/integrations/lark-cli/runtime"] == ( + str(runtime_dir), + True, + ) + + +def test_get_user_skill_mounts_mounts_only_global_integrations(tmp_path, monkeypatch): + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + skills_root = tmp_path / "skills" + (skills_root / "public").mkdir(parents=True) + config = SimpleNamespace( + skills=SimpleNamespace( + get_skills_path=lambda: skills_root, + container_path="/mnt/skills", + ) + ) + monkeypatch.setattr(aio_mod, "get_app_config", lambda: config) + monkeypatch.setattr(aio_mod, "get_paths", lambda: Paths(base_dir=tmp_path / "home")) + + alice = {container: host for host, container, _read_only in aio_mod.AioSandboxProvider._get_user_skill_mounts(user_id="alice")} + bob = {container: host for host, container, _read_only in aio_mod.AioSandboxProvider._get_user_skill_mounts(user_id="bob")} + + assert set(alice) == {"/mnt/skills/integrations"} + assert set(bob) == {"/mnt/skills/integrations"} + assert alice["/mnt/skills/integrations"] == bob["/mnt/skills/integrations"] + assert alice["/mnt/skills/integrations"] == str(tmp_path / "home" / "integrations" / "skills") + + +def test_get_extra_mounts_provisioner_payload_has_unique_container_paths(tmp_path, monkeypatch, provisioner_module): + """Full AIO mount composition must not send duplicate paths to provisioner.""" + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + lark_cli = importlib.import_module("deerflow.integrations.lark_cli") + remote_backend = importlib.import_module("deerflow.community.aio_sandbox.remote_backend") + skills_root = tmp_path / "skills" + (skills_root / "public").mkdir(parents=True) + home = tmp_path / "home" + config = SimpleNamespace( + skills=SimpleNamespace( + get_skills_path=lambda: skills_root, + container_path="/mnt/skills", + ) + ) + runtime_dir = home / "integrations" / "lark-cli" / "sandbox-cli" + runtime_dir.mkdir(parents=True) + + monkeypatch.setattr(aio_mod, "get_app_config", lambda: config) + monkeypatch.setattr(aio_mod, "get_paths", lambda: Paths(base_dir=home)) + monkeypatch.setattr(aio_mod, "get_effective_user_id", lambda: "default") + monkeypatch.setattr(aio_mod, "user_should_see_legacy_skills", lambda *_args, **_kwargs: False) + + provider = _make_provider(tmp_path) + mounts = provider._get_extra_mounts("thread-1", user_id="alice") + container_paths = [container for _host, container, _read_only in mounts] + + assert len(container_paths) == len(set(container_paths)) + assert "/mnt/skills/custom" in container_paths + assert "/mnt/skills/integrations" in container_paths + assert lark_cli.LARK_CLI_SANDBOX_CONFIG_DIR in container_paths + assert lark_cli.LARK_CLI_SANDBOX_DATA_DIR in container_paths + assert lark_cli.LARK_CLI_SANDBOX_RUNTIME_DIR in container_paths + + payload = remote_backend._provisioner_extra_mounts_payload(mounts) + payload_paths = [str(item["container_path"]) for item in payload] + assert len(payload_paths) == len(set(payload_paths)) + + provisioner_module.DEER_FLOW_HOST_BASE_DIR = str(home) + validated = provisioner_module._validated_extra_mounts([provisioner_module.ExtraMount(**item) for item in payload]) + validated_paths = [mount.container_path for mount in validated] + + assert len(validated_paths) == len(set(validated_paths)) + assert set(validated_paths) == { + "/mnt/acp-workspace", + "/mnt/skills/custom", + "/mnt/skills/integrations", + lark_cli.LARK_CLI_SANDBOX_CONFIG_DIR, + lark_cli.LARK_CLI_SANDBOX_DATA_DIR, + lark_cli.LARK_CLI_SANDBOX_RUNTIME_DIR, + } + + def test_join_host_path_preserves_windows_drive_letter_style(): base = r"C:\Users\demo\deer-flow\backend\.deer-flow" @@ -405,6 +508,7 @@ def test_remote_backend_create_forwards_effective_user_id(monkeypatch): "thread_id": "thread-42", "user_id": "user-7", "include_legacy_skills": True, + "provision_lark_cli_runtime": False, } @@ -435,6 +539,62 @@ def test_remote_backend_create_prefers_explicit_user_id(monkeypatch): assert posted["json"]["include_legacy_skills"] is False +def test_create_sandbox_requests_runtime_when_lark_installed(tmp_path, monkeypatch): + """The provider must request lark-cli runtime provisioning when Lark is installed.""" + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + provider = _make_provider(tmp_path) + provider._config = {"replicas": 3} + provider._thread_locks = {} + provider._warm_pool = {} + provider._sandbox_infos = {} + provider._thread_sandboxes = {} + provider._last_activity = {} + provider._lock = aio_mod.threading.Lock() + + captured: dict = {} + + def _create(thread_id, sandbox_id, *, extra_mounts=None, user_id=None, provision_lark_cli_runtime=False): + captured["provision_lark_cli_runtime"] = provision_lark_cli_runtime + return aio_mod.SandboxInfo(sandbox_id=sandbox_id, sandbox_url="http://sandbox") + + provider._backend = SimpleNamespace(create=_create, destroy=MagicMock(), discover=MagicMock(return_value=None)) + monkeypatch.setattr(aio_mod, "wait_for_sandbox_ready", lambda *_a, **_k: True) + monkeypatch.setattr(provider, "_get_extra_mounts", lambda *_a, **_k: []) + monkeypatch.setattr(aio_mod.AioSandboxProvider, "_lark_integration_active", staticmethod(lambda user_id=None: True)) + monkeypatch.setattr(provider, "_register_created_sandbox", lambda *a, **k: "sandbox-lark") + + provider._create_sandbox("thread-lark", "sandbox-lark", user_id="alice") + assert captured["provision_lark_cli_runtime"] is True + + +def test_create_sandbox_skips_runtime_when_lark_absent(tmp_path, monkeypatch): + """No runtime provisioning request when the Lark skill pack is not installed.""" + aio_mod = importlib.import_module("deerflow.community.aio_sandbox.aio_sandbox_provider") + provider = _make_provider(tmp_path) + provider._config = {"replicas": 3} + provider._thread_locks = {} + provider._warm_pool = {} + provider._sandbox_infos = {} + provider._thread_sandboxes = {} + provider._last_activity = {} + provider._lock = aio_mod.threading.Lock() + + captured: dict = {} + + def _create(thread_id, sandbox_id, *, extra_mounts=None, user_id=None, provision_lark_cli_runtime=False): + captured["provision_lark_cli_runtime"] = provision_lark_cli_runtime + return aio_mod.SandboxInfo(sandbox_id=sandbox_id, sandbox_url="http://sandbox") + + provider._backend = SimpleNamespace(create=_create, destroy=MagicMock(), discover=MagicMock(return_value=None)) + monkeypatch.setattr(aio_mod, "wait_for_sandbox_ready", lambda *_a, **_k: True) + monkeypatch.setattr(provider, "_get_extra_mounts", lambda *_a, **_k: []) + monkeypatch.setattr(aio_mod.AioSandboxProvider, "_lark_integration_active", staticmethod(lambda user_id=None: False)) + monkeypatch.setattr(provider, "_register_created_sandbox", lambda *a, **k: "sandbox-nolark") + + provider._create_sandbox("thread-nolark", "sandbox-nolark", user_id="alice") + assert captured["provision_lark_cli_runtime"] is False + + # ── Sandbox client teardown (#2872) ────────────────────────────────────────── diff --git a/backend/tests/test_gateway_imports.py b/backend/tests/test_gateway_imports.py index 09772ce67..7272ea2f1 100644 --- a/backend/tests/test_gateway_imports.py +++ b/backend/tests/test_gateway_imports.py @@ -8,23 +8,28 @@ import sys from pathlib import Path +def _gateway_import_env() -> dict[str, str]: + backend_root = Path(__file__).resolve().parents[1] + harness_root = backend_root / "packages" / "harness" + python_path_entries = [str(backend_root), str(harness_root)] + if existing_python_path := os.environ.get("PYTHONPATH"): + python_path_entries.append(existing_python_path) + return {**os.environ, "PYTHONPATH": os.pathsep.join(python_path_entries)} + + def test_gateway_app_imports_first_without_subagent_import_cycle() -> None: """The replay gateway imports app.gateway.app in a clean process.""" - backend_root = Path(__file__).resolve().parents[1] - env = {**os.environ, "PYTHONPATH": str(backend_root)} result = subprocess.run( [sys.executable, "-c", "from app.gateway.app import app"], capture_output=True, text=True, - env=env, + env=_gateway_import_env(), ) assert result.returncode == 0, result.stderr def test_subagent_package_public_executor_exports_are_lazy_importable() -> None: """The package-level executor exports must not re-enter their own import.""" - backend_root = Path(__file__).resolve().parents[1] - env = {**os.environ, "PYTHONPATH": str(backend_root)} result = subprocess.run( [ sys.executable, @@ -33,7 +38,7 @@ def test_subagent_package_public_executor_exports_are_lazy_importable() -> None: ], capture_output=True, text=True, - env=env, + env=_gateway_import_env(), ) assert result.returncode == 0, result.stderr assert "SubagentExecutor SubagentResult" in result.stdout diff --git a/backend/tests/test_lark_cli_integration.py b/backend/tests/test_lark_cli_integration.py new file mode 100644 index 000000000..a91ebcd50 --- /dev/null +++ b/backend/tests/test_lark_cli_integration.py @@ -0,0 +1,1599 @@ +from __future__ import annotations + +import hashlib +import inspect +import io +import json +import multiprocessing +import re +import shutil +import stat +import subprocess +import tarfile +import threading +import time +import zipfile +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +from types import SimpleNamespace +from uuid import uuid4 + +import pytest +from _router_auth_helpers import make_authed_test_app +from fastapi.testclient import TestClient + +from app.gateway.auth.models import User +from app.gateway.deps import get_config +from app.gateway.routers import integrations as integrations_router +from deerflow.config import paths as paths_module +from deerflow.config.paths import Paths +from deerflow.integrations import lark_cli +from deerflow.sandbox.tools import _lark_cli_env_from_runtime +from deerflow.skills.storage import reset_skill_storage +from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage +from deerflow.skills.types import SkillCategory + + +def _skill_content(name: str) -> str: + return f"---\nname: {name}\ndescription: {name} integration skill\n---\n\n# {name}\n" + + +def _make_lark_cli_source_zip(tmp_path: Path, *, omit_skill: str | None = None, renamed_skill: str | None = None) -> Path: + archive = tmp_path / "lark-cli.zip" + with zipfile.ZipFile(archive, "w") as zf: + for skill_name in lark_cli.LARK_SKILL_NAMES: + if skill_name == omit_skill: + continue + declared_name = f"{skill_name}-renamed" if skill_name == renamed_skill else skill_name + zf.writestr(f"cli-1.0.65/skills/{skill_name}/SKILL.md", _skill_content(declared_name)) + zf.writestr(f"cli-1.0.65/skills/{skill_name}/references/readme.md", f"# {skill_name}\n") + return archive + + +def _make_lark_cli_binary_tar(payload: bytes, *, member_name: str = "lark-cli") -> bytes: + buffer = io.BytesIO() + with tarfile.open(fileobj=buffer, mode="w:gz") as tf: + info = tarfile.TarInfo(member_name) + info.mode = 0o755 + info.size = len(payload) + tf.addfile(info, io.BytesIO(payload)) + return buffer.getvalue() + + +def _assert_lark_root_missing(user_id: str) -> None: + root = lark_cli.lark_integration_root(user_id) + assert not root.exists() + + +def _config(skills_root: Path): + return SimpleNamespace( + skills=SimpleNamespace( + get_skills_path=lambda: skills_root, + container_path="/mnt/skills", + use="deerflow.skills.storage.local_skill_storage:LocalSkillStorage", + ) + ) + + +def _patch_paths(monkeypatch, base_dir: Path) -> None: + monkeypatch.setattr(paths_module, "_paths", Paths(base_dir=base_dir)) + + +def test_sandbox_lark_cli_env_prepends_managed_linux_runtime() -> None: + overlay = lark_cli.lark_cli_env_overlay("alice", sandbox_paths=True) + + assert overlay["PATH"].split(":", 1)[0] == "/mnt/integrations/lark-cli/runtime/bin" + assert overlay["LARKSUITE_CLI_CONFIG_DIR"] == "/mnt/integrations/lark-cli/config" + assert overlay["LARKSUITE_CLI_DATA_DIR"] == "/mnt/integrations/lark-cli/data" + + +def test_init_image_launcher_matches_python_constant() -> None: + """The init image's build script must embed the same launcher as the Gateway. + + Both produce ``bin/lark-cli``; if they drift, the sandbox PATH contract + breaks for one provisioning mode. + """ + repo_root = Path(lark_cli.__file__).resolve().parents[5] + build_script = repo_root / "docker" / "lark-cli-init" / "build-runtime.sh" + assert build_script.is_file(), f"missing init image build script at {build_script}" + body = build_script.read_text(encoding="utf-8") + # The launcher heredoc in the build script must contain the exact script body. + assert lark_cli.LARK_CLI_SANDBOX_LAUNCHER_SCRIPT.strip() in body + + +def test_managed_sandbox_runtime_verifies_and_installs_linux_archives(monkeypatch, tmp_path) -> None: + assert hasattr(lark_cli, "_ensure_managed_sandbox_lark_cli"), "managed sandbox runtime installer is missing" + _patch_paths(monkeypatch, tmp_path / "home") + archives = { + "lark-cli-1.0.65-linux-amd64.tar.gz": _make_lark_cli_binary_tar(b"amd64-binary"), + "lark-cli-1.0.65-linux-arm64.tar.gz": _make_lark_cli_binary_tar(b"arm64-binary"), + } + checksums = "".join(f"{hashlib.sha256(payload).hexdigest()} {name}\n" for name, payload in archives.items()).encode() + assets = {"checksums.txt": checksums, **archives} + + monkeypatch.setattr(lark_cli, "_download_lark_release_asset", lambda _version, name, **_kwargs: assets[name]) + + runtime = lark_cli._ensure_managed_sandbox_lark_cli("v1.0.65") + + assert (runtime / "linux-amd64" / "lark-cli").read_bytes() == b"amd64-binary" + assert (runtime / "linux-arm64" / "lark-cli").read_bytes() == b"arm64-binary" + assert stat.S_IMODE((runtime / "linux-amd64" / "lark-cli").stat().st_mode) == 0o755 + launcher = (runtime / "bin" / "lark-cli").read_text(encoding="utf-8") + assert "uname -m" in launcher + assert "x86_64" in launcher and "aarch64" in launcher + + +def test_managed_sandbox_runtime_rejects_checksum_mismatch(monkeypatch, tmp_path) -> None: + assert hasattr(lark_cli, "_ensure_managed_sandbox_lark_cli"), "managed sandbox runtime installer is missing" + _patch_paths(monkeypatch, tmp_path / "home") + archives = { + "lark-cli-1.0.65-linux-amd64.tar.gz": _make_lark_cli_binary_tar(b"amd64-binary"), + "lark-cli-1.0.65-linux-arm64.tar.gz": _make_lark_cli_binary_tar(b"arm64-binary"), + } + bad_checksums = "".join(f"{'0' * 64} {name}\n" for name in archives).encode() + assets = {"checksums.txt": bad_checksums, **archives} + monkeypatch.setattr(lark_cli, "_download_lark_release_asset", lambda _version, name, **_kwargs: assets[name]) + + with pytest.raises(ValueError, match="checksum"): + lark_cli._ensure_managed_sandbox_lark_cli("v1.0.65") + + assert not lark_cli.lark_cli_managed_sandbox_dir().exists() + + +def test_managed_sandbox_runtime_rejects_unsafe_tar_member(monkeypatch, tmp_path) -> None: + assert hasattr(lark_cli, "_ensure_managed_sandbox_lark_cli"), "managed sandbox runtime installer is missing" + _patch_paths(monkeypatch, tmp_path / "home") + unsafe = _make_lark_cli_binary_tar(b"binary", member_name="../lark-cli") + safe = _make_lark_cli_binary_tar(b"binary") + archives = { + "lark-cli-1.0.65-linux-amd64.tar.gz": unsafe, + "lark-cli-1.0.65-linux-arm64.tar.gz": safe, + } + checksums = "".join(f"{hashlib.sha256(payload).hexdigest()} {name}\n" for name, payload in archives.items()).encode() + assets = {"checksums.txt": checksums, **archives} + monkeypatch.setattr(lark_cli, "_download_lark_release_asset", lambda _version, name, **_kwargs: assets[name]) + + with pytest.raises(ValueError, match="Unsafe Lark CLI runtime archive member"): + lark_cli._ensure_managed_sandbox_lark_cli("v1.0.65") + + +def test_managed_sandbox_runtime_accepts_prestaged_airgapped_tree(monkeypatch, tmp_path) -> None: + _patch_paths(monkeypatch, tmp_path / "home") + source = tmp_path / "pre-staged" + for arch in ("amd64", "arm64"): + binary = source / f"linux-{arch}" / "lark-cli" + binary.parent.mkdir(parents=True) + binary.write_bytes(f"{arch}-binary".encode()) + binary.chmod(0o755) + launcher = source / "bin" / "lark-cli" + launcher.parent.mkdir(parents=True) + launcher.write_text("#!/bin/sh\n", encoding="utf-8") + launcher.chmod(0o755) + monkeypatch.setenv(lark_cli.LARK_CLI_SANDBOX_RUNTIME_SOURCE_ENV, str(source)) + monkeypatch.setattr( + lark_cli, + "_download_lark_release_asset", + lambda *_args, **_kwargs: (_ for _ in ()).throw(AssertionError("air-gapped install must not download")), + ) + + runtime = lark_cli._ensure_managed_sandbox_lark_cli("v1.0.65") + + assert (runtime / "linux-amd64" / "lark-cli").read_bytes() == b"amd64-binary" + assert (runtime / "linux-arm64" / "lark-cli").read_bytes() == b"arm64-binary" + + +def test_managed_sandbox_runtime_rejects_any_symlink_in_prestaged_tree(monkeypatch, tmp_path) -> None: + _patch_paths(monkeypatch, tmp_path / "home") + source = tmp_path / "pre-staged" + for arch in ("amd64", "arm64"): + binary = source / f"linux-{arch}" / "lark-cli" + binary.parent.mkdir(parents=True) + binary.write_bytes(f"{arch}-binary".encode()) + binary.chmod(0o755) + launcher = source / "bin" / "lark-cli" + launcher.parent.mkdir(parents=True) + launcher.write_text("#!/bin/sh\n", encoding="utf-8") + launcher.chmod(0o755) + outside = tmp_path / "outside-secret" + outside.write_text("must-not-be-copied", encoding="utf-8") + try: + (source / "extra-link").symlink_to(outside) + except OSError as exc: + pytest.skip(f"symlinks are not available: {exc}") + monkeypatch.setenv(lark_cli.LARK_CLI_SANDBOX_RUNTIME_SOURCE_ENV, str(source)) + + with pytest.raises(ValueError, match="symlink"): + lark_cli._ensure_managed_sandbox_lark_cli("v1.0.65") + + assert not lark_cli.lark_cli_managed_sandbox_dir().exists() + + +def test_managed_sandbox_runtime_rejects_non_executable_prestaged_binary(monkeypatch, tmp_path) -> None: + _patch_paths(monkeypatch, tmp_path / "home") + source = tmp_path / "pre-staged" + for arch in ("amd64", "arm64"): + binary = source / f"linux-{arch}" / "lark-cli" + binary.parent.mkdir(parents=True) + binary.write_bytes(f"{arch}-binary".encode()) + binary.chmod(0o755) + (source / "linux-arm64" / "lark-cli").chmod(0o644) + launcher = source / "bin" / "lark-cli" + launcher.parent.mkdir(parents=True) + launcher.write_text("#!/bin/sh\n", encoding="utf-8") + launcher.chmod(0o755) + monkeypatch.setenv(lark_cli.LARK_CLI_SANDBOX_RUNTIME_SOURCE_ENV, str(source)) + + with pytest.raises(ValueError, match="executable"): + lark_cli._ensure_managed_sandbox_lark_cli("v1.0.65") + + assert not lark_cli.lark_cli_managed_sandbox_dir().exists() + + +def test_concurrent_managed_sandbox_runtime_installs_serialize_replacement(monkeypatch, tmp_path) -> None: + _patch_paths(monkeypatch, tmp_path / "home") + source = tmp_path / "pre-staged" + for arch in ("amd64", "arm64"): + binary = source / f"linux-{arch}" / "lark-cli" + binary.parent.mkdir(parents=True) + binary.write_bytes(f"{arch}-binary".encode()) + binary.chmod(0o755) + launcher = source / "bin" / "lark-cli" + launcher.parent.mkdir(parents=True) + launcher.write_text("#!/bin/sh\n", encoding="utf-8") + launcher.chmod(0o755) + monkeypatch.setenv(lark_cli.LARK_CLI_SANDBOX_RUNTIME_SOURCE_ENV, str(source)) + + real_validate = lark_cli._validate_lark_cli_sandbox_runtime + start = threading.Barrier(2) + state_lock = threading.Lock() + active = 0 + max_active = 0 + + def _slow_validate(root): + nonlocal active, max_active + with state_lock: + active += 1 + max_active = max(max_active, active) + try: + time.sleep(0.15) + return real_validate(root) + finally: + with state_lock: + active -= 1 + + def _install(): + start.wait() + return lark_cli._ensure_managed_sandbox_lark_cli("v1.0.65") + + monkeypatch.setattr(lark_cli, "_validate_lark_cli_sandbox_runtime", _slow_validate) + with ThreadPoolExecutor(max_workers=2) as pool: + results = [future.result(timeout=5) for future in [pool.submit(_install) for _ in range(2)]] + + assert results[0] == results[1] == lark_cli.lark_cli_managed_sandbox_dir() + assert max_active == 1 + assert not list(results[0].parent.glob(".replacing-sandbox-cli-*")) + + +def test_install_lark_integration_installs_one_readonly_pack_for_all_users(monkeypatch, tmp_path): + reset_skill_storage() + _patch_paths(monkeypatch, tmp_path / "home") + skills_root = tmp_path / "skills" + (skills_root / "public").mkdir(parents=True) + (skills_root / "custom").mkdir() + config = _config(skills_root) + archive = _make_lark_cli_source_zip(tmp_path) + + monkeypatch.setattr(lark_cli, "probe_lark_cli", lambda: lark_cli.LarkCliProbe(available=True, path="/usr/bin/lark-cli", version="v1.0.65")) + monkeypatch.setattr(lark_cli, "probe_lark_auth", lambda _user_id, **_kwargs: lark_cli.LarkAuthProbe(status="not_configured", message="not configured")) + + result = lark_cli.install_lark_integration("alice", config, source_archive=archive) + + assert result.success is True + assert "lark-doc" in result.installed_skills + assert result.status.installed is True + root = lark_cli.lark_integration_root("alice") + assert root == lark_cli.lark_integration_root("bob") + assert root == tmp_path / "home" / "integrations" / "skills" / "lark-cli" + assert (root / "lark-doc" / "SKILL.md").is_file() + assert (root / lark_cli.LARK_CLI_MANIFEST_FILE).is_file() + shared_content = (root / "lark-shared" / "SKILL.md").read_text(encoding="utf-8") + assert "?settings=integrations" in shared_content + assert "不要要求用户在终端执行" in shared_content + assert "Exact OAuth scope" in shared_content + + storage = UserScopedSkillStorage("alice", host_path=str(skills_root), app_config=config) + skills = storage.load_skills(enabled_only=False) + lark_doc = next(skill for skill in skills if skill.name == "lark-doc") + assert lark_doc.category == SkillCategory.INTEGRATION + assert lark_doc.get_container_file_path("/mnt/skills") == "/mnt/skills/integrations/lark-cli/lark-doc/SKILL.md" + assert lark_doc.enabled is True + + bob_storage = UserScopedSkillStorage("bob", host_path=str(skills_root), app_config=config) + bob_lark_doc = next(skill for skill in bob_storage.load_skills(enabled_only=False) if skill.name == "lark-doc") + assert bob_lark_doc.category == SkillCategory.INTEGRATION + assert bob_lark_doc.skill_file == root / "lark-doc" / "SKILL.md" + reset_skill_storage() + + +def test_aio_install_provisions_matching_linux_sandbox_runtime(monkeypatch, tmp_path) -> None: + reset_skill_storage() + _patch_paths(monkeypatch, tmp_path / "home") + skills_root = tmp_path / "skills" + (skills_root / "public").mkdir(parents=True) + (skills_root / "custom").mkdir() + config = _config(skills_root) + config.sandbox = SimpleNamespace(use="deerflow.community.aio_sandbox:AioSandboxProvider") + archive = _make_lark_cli_source_zip(tmp_path) + provisioned_versions: list[str] = [] + + monkeypatch.setattr(lark_cli, "probe_lark_cli", lambda: lark_cli.LarkCliProbe(available=True, path="/usr/bin/lark-cli", version="v1.0.65")) + monkeypatch.setattr(lark_cli, "probe_lark_auth", lambda _user_id, **_kwargs: lark_cli.LarkAuthProbe(status="not_configured", message="not configured")) + monkeypatch.setattr( + lark_cli, + "_ensure_managed_sandbox_lark_cli", + lambda version: provisioned_versions.append(version) or lark_cli.lark_cli_managed_sandbox_dir(), + ) + + result = lark_cli.install_lark_integration("alice", config, source_archive=archive) + + assert result.success is True + assert provisioned_versions == ["v1.0.65"] + reset_skill_storage() + + +def test_remote_provisioner_install_skips_gateway_sandbox_runtime(monkeypatch, tmp_path) -> None: + reset_skill_storage() + _patch_paths(monkeypatch, tmp_path / "home") + skills_root = tmp_path / "skills" + (skills_root / "public").mkdir(parents=True) + (skills_root / "custom").mkdir() + config = _config(skills_root) + config.sandbox = SimpleNamespace( + use="deerflow.community.aio_sandbox:AioSandboxProvider", + provisioner_url="http://provisioner:8002", + ) + archive = _make_lark_cli_source_zip(tmp_path) + provisioned_versions: list[str] = [] + + monkeypatch.setattr(lark_cli, "probe_lark_cli", lambda: lark_cli.LarkCliProbe(available=True, path="/usr/bin/lark-cli", version="v1.0.65")) + monkeypatch.setattr(lark_cli, "probe_lark_auth", lambda _user_id, **_kwargs: lark_cli.LarkAuthProbe(status="not_configured", message="not configured")) + monkeypatch.setattr( + lark_cli, + "_ensure_managed_sandbox_lark_cli", + lambda version: provisioned_versions.append(version) or lark_cli.lark_cli_managed_sandbox_dir(), + ) + + result = lark_cli.install_lark_integration("alice", config, source_archive=archive) + + assert result.success is True + # Remote provisioner mode gets the runtime from an init container, so the + # Gateway must not download Linux binaries at install time. + assert provisioned_versions == [] + reset_skill_storage() + + +def test_status_runtime_mode_none_for_non_aio(monkeypatch, tmp_path) -> None: + config = _config(tmp_path / "skills") + config.sandbox = SimpleNamespace(use="deerflow.sandbox.local:LocalSandboxProvider") + mode, ready, detail = lark_cli._resolve_sandbox_runtime_readiness(config, probe=True) + assert mode == "none" + assert ready is False + assert detail + + +def test_status_runtime_mode_gateway_download_ready(monkeypatch, tmp_path) -> None: + _patch_paths(monkeypatch, tmp_path / "home") + config = _config(tmp_path / "skills") + config.sandbox = SimpleNamespace(use="deerflow.community.aio_sandbox:AioSandboxProvider") + + # Stage a valid runtime dir so validation passes. + runtime = lark_cli.lark_cli_managed_sandbox_dir() + (runtime / "bin").mkdir(parents=True) + (runtime / "bin" / "lark-cli").write_text("#!/bin/sh\n", encoding="utf-8") + (runtime / "bin" / "lark-cli").chmod(0o755) + for arch in lark_cli.LARK_CLI_LINUX_ARCHES: + (runtime / f"linux-{arch}").mkdir(parents=True) + target = runtime / f"linux-{arch}" / "lark-cli" + target.write_bytes(b"\x7fELF") + target.chmod(0o755) + + mode, ready, detail = lark_cli._resolve_sandbox_runtime_readiness(config, probe=True) + assert mode == "gateway-download" + assert ready is True + assert detail is None + + +def test_status_runtime_mode_gateway_download_not_ready(monkeypatch, tmp_path) -> None: + _patch_paths(monkeypatch, tmp_path / "home") + config = _config(tmp_path / "skills") + config.sandbox = SimpleNamespace(use="deerflow.community.aio_sandbox:AioSandboxProvider") + mode, ready, detail = lark_cli._resolve_sandbox_runtime_readiness(config, probe=True) + assert mode == "gateway-download" + assert ready is False + assert detail + + +def test_status_runtime_mode_init_container_ready(monkeypatch, tmp_path) -> None: + config = _config(tmp_path / "skills") + config.sandbox = SimpleNamespace( + use="deerflow.community.aio_sandbox:AioSandboxProvider", + provisioner_url="http://provisioner:8002", + ) + monkeypatch.setattr(lark_cli, "_probe_provisioner_lark_cli_init_image", lambda _config: True) + mode, ready, detail = lark_cli._resolve_sandbox_runtime_readiness(config, probe=True) + assert mode == "init-container" + assert ready is True + assert detail is None + + +def test_status_runtime_mode_init_container_not_configured(monkeypatch, tmp_path) -> None: + config = _config(tmp_path / "skills") + config.sandbox = SimpleNamespace( + use="deerflow.community.aio_sandbox:AioSandboxProvider", + provisioner_url="http://provisioner:8002", + ) + monkeypatch.setattr(lark_cli, "_probe_provisioner_lark_cli_init_image", lambda _config: False) + mode, ready, detail = lark_cli._resolve_sandbox_runtime_readiness(config, probe=True) + assert mode == "init-container" + assert ready is False + assert detail + + +def test_status_runtime_mode_init_container_unreachable(monkeypatch, tmp_path) -> None: + config = _config(tmp_path / "skills") + config.sandbox = SimpleNamespace( + use="deerflow.community.aio_sandbox:AioSandboxProvider", + provisioner_url="http://provisioner:8002", + ) + monkeypatch.setattr(lark_cli, "_probe_provisioner_lark_cli_init_image", lambda _config: None) + mode, ready, detail = lark_cli._resolve_sandbox_runtime_readiness(config, probe=True) + assert mode == "init-container" + assert ready is False + assert detail + + +def test_status_runtime_probe_skipped_when_not_requested(monkeypatch, tmp_path) -> None: + config = _config(tmp_path / "skills") + config.sandbox = SimpleNamespace( + use="deerflow.community.aio_sandbox:AioSandboxProvider", + provisioner_url="http://provisioner:8002", + ) + + def _fail(_config): # pragma: no cover - must not be called + raise AssertionError("provisioner should not be probed when probe=False") + + monkeypatch.setattr(lark_cli, "_probe_provisioner_lark_cli_init_image", _fail) + mode, ready, detail = lark_cli._resolve_sandbox_runtime_readiness(config, probe=False) + assert mode == "init-container" + assert ready is False + assert detail is None + + +def test_install_lark_integration_is_idempotent_across_reinstalls(monkeypatch, tmp_path): + reset_skill_storage() + _patch_paths(monkeypatch, tmp_path / "home") + skills_root = tmp_path / "skills" + (skills_root / "public").mkdir(parents=True) + (skills_root / "custom").mkdir() + config = _config(skills_root) + archive = _make_lark_cli_source_zip(tmp_path) + + monkeypatch.setattr(lark_cli, "probe_lark_cli", lambda: lark_cli.LarkCliProbe(available=True, path="/usr/bin/lark-cli", version="v1.0.65")) + monkeypatch.setattr(lark_cli, "probe_lark_auth", lambda _user_id, **_kwargs: lark_cli.LarkAuthProbe(status="not_configured", message="not configured")) + + first = lark_cli.install_lark_integration("alice", config, source_archive=archive) + root = lark_cli.lark_integration_root("alice") + # Drop a stray file so the reinstall must replace the whole tree, not merge. + stray = root / "lark-doc" / "stray.txt" + stray.write_text("stale", encoding="utf-8") + + second = lark_cli.install_lark_integration("alice", config, source_archive=archive) + + assert first.installed_skills == second.installed_skills + assert second.status.installed is True + assert (root / "lark-doc" / "SKILL.md").is_file() + assert not stray.exists() + # No leftover backup/staging dirs beside the target after a reinstall. + parent = root.parent + leftovers = [p.name for p in parent.iterdir() if p.name not in {lark_cli.INTEGRATION_ID, ".lark-cli.install.lock"}] + assert leftovers == [] + reset_skill_storage() + + +@pytest.mark.parametrize("_attempt", range(5)) +def test_concurrent_lark_skill_reinstalls_serialize_atomic_replacement(monkeypatch, tmp_path, _attempt) -> None: + _patch_paths(monkeypatch, tmp_path / "home") + archive = _make_lark_cli_source_zip(tmp_path) + real_extract = lark_cli._extract_lark_skills + start = threading.Barrier(2) + state_lock = threading.Lock() + active = 0 + max_active = 0 + + def _slow_extract(zf, destination): + nonlocal active, max_active + with state_lock: + active += 1 + max_active = max(max_active, active) + try: + time.sleep(0.15) + return real_extract(zf, destination) + finally: + with state_lock: + active -= 1 + + def _install(): + start.wait() + return lark_cli._install_lark_skills_from_archive("alice", archive, version="v1.0.65") + + monkeypatch.setattr(lark_cli, "_extract_lark_skills", _slow_extract) + with ThreadPoolExecutor(max_workers=2) as pool: + futures = [pool.submit(_install) for _ in range(2)] + results = [future.result(timeout=5) for future in futures] + + assert results[0] == results[1] + assert max_active == 1 + root = lark_cli.lark_integration_root() + assert (root / "lark-doc" / "SKILL.md").is_file() + assert not list(root.parent.glob(".replacing-lark-cli-*")) + + +@pytest.mark.skipif( + "fork" not in multiprocessing.get_all_start_methods() or lark_cli.fcntl is None, + reason="requires POSIX fork and fcntl", +) +def test_concurrent_lark_skill_reinstalls_serialize_across_processes(monkeypatch, tmp_path) -> None: + _patch_paths(monkeypatch, tmp_path / "home") + archive = _make_lark_cli_source_zip(tmp_path) + real_extract = lark_cli._extract_lark_skills + context = multiprocessing.get_context("fork") + start = context.Barrier(2) + active = context.Value("i", 0) + max_active = context.Value("i", 0) + results = context.Queue() + + def _slow_extract(zf, destination): + with active.get_lock(), max_active.get_lock(): + active.value += 1 + max_active.value = max(max_active.value, active.value) + try: + time.sleep(0.2) + return real_extract(zf, destination) + finally: + with active.get_lock(): + active.value -= 1 + + def _install(): + try: + start.wait(timeout=5) + installed, digest = lark_cli._install_lark_skills_from_archive("alice", archive, version="v1.0.65") + results.put((installed, digest, None)) + except BaseException as exc: # noqa: BLE001 - propagate child failure + results.put((None, None, repr(exc))) + + monkeypatch.setattr(lark_cli, "_extract_lark_skills", _slow_extract) + processes = [context.Process(target=_install) for _ in range(2)] + for process in processes: + process.start() + for process in processes: + process.join(timeout=10) + + assert [process.exitcode for process in processes] == [0, 0] + child_results = [results.get(timeout=2) for _ in processes] + assert all(error is None for _installed, _digest, error in child_results), child_results + assert child_results[0][:2] == child_results[1][:2] + assert max_active.value == 1 + assert not list(lark_cli.lark_integration_root().parent.glob(".replacing-lark-cli-*")) + + +def test_install_lark_integration_succeeds_when_backup_cleanup_fails(monkeypatch, tmp_path): + reset_skill_storage() + _patch_paths(monkeypatch, tmp_path / "home") + skills_root = tmp_path / "skills" + (skills_root / "public").mkdir(parents=True) + (skills_root / "custom").mkdir() + config = _config(skills_root) + archive = _make_lark_cli_source_zip(tmp_path) + + monkeypatch.setattr(lark_cli, "probe_lark_cli", lambda: lark_cli.LarkCliProbe(available=True, path="/usr/bin/lark-cli", version="v1.0.65")) + monkeypatch.setattr(lark_cli, "probe_lark_auth", lambda _user_id, **_kwargs: lark_cli.LarkAuthProbe(status="not_configured", message="not configured")) + + # First install lays down the target so the reinstall has a backup to clean. + lark_cli.install_lark_integration("alice", config, source_archive=archive) + + real_rmtree = shutil.rmtree + forced_raises = {"count": 0} + + def _rmtree(path, *args, **kwargs): + # The post-rename backup deletion is best-effort and now passes + # ignore_errors=True, so a transient FS error there must not flip a + # successful install into a failure. Force any rmtree that does *not* + # ignore errors to raise, proving the success path no longer depends on + # a fragile backup cleanup. + if kwargs.get("ignore_errors"): + return real_rmtree(path, *args, **kwargs) + forced_raises["count"] += 1 + raise OSError("transient FS error during backup cleanup") + + monkeypatch.setattr(lark_cli.shutil, "rmtree", _rmtree) + + result = lark_cli.install_lark_integration("alice", config, source_archive=archive) + + assert result.success is True + root = lark_cli.lark_integration_root("alice") + assert (root / "lark-doc" / "SKILL.md").is_file() + # No non-ignoring rmtree is relied upon on the success path, and no leftover + # backup dir remains beside the target after the reinstall. + assert forced_raises["count"] == 0 + leftovers = [p.name for p in root.parent.iterdir() if p.name not in {lark_cli.INTEGRATION_ID, ".lark-cli.install.lock"}] + assert leftovers == [] + reset_skill_storage() + + +def test_install_lark_integration_records_content_sha_in_manifest(monkeypatch, tmp_path): + reset_skill_storage() + _patch_paths(monkeypatch, tmp_path / "home") + skills_root = tmp_path / "skills" + (skills_root / "public").mkdir(parents=True) + (skills_root / "custom").mkdir() + config = _config(skills_root) + archive = _make_lark_cli_source_zip(tmp_path) + + monkeypatch.setattr(lark_cli, "probe_lark_cli", lambda: lark_cli.LarkCliProbe(available=True, path="/usr/bin/lark-cli", version="v1.0.65")) + monkeypatch.setattr(lark_cli, "probe_lark_auth", lambda _user_id, **_kwargs: lark_cli.LarkAuthProbe(status="not_configured", message="not configured")) + + lark_cli.install_lark_integration("alice", config, source_archive=archive) + + manifest = json.loads((lark_cli.lark_integration_root("alice") / lark_cli.LARK_CLI_MANIFEST_FILE).read_text(encoding="utf-8")) + assert manifest["version"] == "v1.0.65" + assert isinstance(manifest["content_sha256"], str) + assert len(manifest["content_sha256"]) == 64 + reset_skill_storage() + + +def test_install_lark_integration_reports_content_change_on_reinstall(monkeypatch, tmp_path): + reset_skill_storage() + _patch_paths(monkeypatch, tmp_path / "home") + skills_root = tmp_path / "skills" + (skills_root / "public").mkdir(parents=True) + (skills_root / "custom").mkdir() + config = _config(skills_root) + archive = _make_lark_cli_source_zip(tmp_path) + + monkeypatch.setattr(lark_cli, "probe_lark_cli", lambda: lark_cli.LarkCliProbe(available=True, path="/usr/bin/lark-cli", version="v1.0.65")) + monkeypatch.setattr(lark_cli, "probe_lark_auth", lambda _user_id, **_kwargs: lark_cli.LarkAuthProbe(status="not_configured", message="not configured")) + + first = lark_cli.install_lark_integration("alice", config, source_archive=archive) + assert "content changed" not in first.message + + changed_dir = tmp_path / "changed" + changed_dir.mkdir() + changed_archive = _make_lark_cli_source_zip(changed_dir) + with zipfile.ZipFile(changed_archive, "a") as zf: + zf.writestr("cli-1.0.65/skills/lark-doc/references/extra.md", "# extra content\n") + + second = lark_cli.install_lark_integration("alice", config, source_archive=changed_archive) + assert "content changed" in second.message + reset_skill_storage() + + +def test_install_lark_integration_rejects_zip_slip_member(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + config = _config(tmp_path / "skills") + archive = _make_lark_cli_source_zip(tmp_path) + with zipfile.ZipFile(archive, "a") as zf: + zf.writestr("../evil.txt", "escape") + + with pytest.raises(ValueError, match="Unsafe Lark CLI archive member"): + lark_cli.install_lark_integration("alice", config, source_archive=archive) + + _assert_lark_root_missing("alice") + + +def test_install_lark_integration_rejects_symlink_member(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + config = _config(tmp_path / "skills") + archive = _make_lark_cli_source_zip(tmp_path) + link_info = zipfile.ZipInfo("cli-1.0.65/skills/lark-doc/references/link") + link_info.external_attr = (stat.S_IFLNK | 0o777) << 16 + with zipfile.ZipFile(archive, "a") as zf: + zf.writestr(link_info, "target") + + with pytest.raises(ValueError, match="Unsafe Lark CLI archive member"): + lark_cli.install_lark_integration("alice", config, source_archive=archive) + + _assert_lark_root_missing("alice") + + +def test_install_lark_integration_rejects_executable_binary_member(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + config = _config(tmp_path / "skills") + archive = _make_lark_cli_source_zip(tmp_path) + with zipfile.ZipFile(archive, "a") as zf: + zf.writestr("cli-1.0.65/skills/lark-doc/bin/tool", b"\x7fELFbinary") + + with pytest.raises(ValueError, match="executable binary member"): + lark_cli.install_lark_integration("alice", config, source_archive=archive) + + _assert_lark_root_missing("alice") + + +def test_install_lark_integration_rejects_oversized_extraction(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + config = _config(tmp_path / "skills") + archive = _make_lark_cli_source_zip(tmp_path) + monkeypatch.setattr(lark_cli, "LARK_CLI_MAX_EXTRACTED_BYTES", 128) + + with pytest.raises(ValueError, match="expands to too much data"): + lark_cli.install_lark_integration("alice", config, source_archive=archive) + + _assert_lark_root_missing("alice") + + +def test_install_lark_integration_rejects_missing_required_skill(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + config = _config(tmp_path / "skills") + archive = _make_lark_cli_source_zip(tmp_path, omit_skill="lark-doc") + + with pytest.raises(ValueError, match="missing required skills: lark-doc"): + lark_cli.install_lark_integration("alice", config, source_archive=archive) + + _assert_lark_root_missing("alice") + + +def test_install_lark_integration_rejects_renamed_skill_metadata(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + config = _config(tmp_path / "skills") + archive = _make_lark_cli_source_zip(tmp_path, renamed_skill="lark-doc") + + with pytest.raises(ValueError, match="declares name 'lark-doc-renamed'"): + lark_cli.install_lark_integration("alice", config, source_archive=archive) + + _assert_lark_root_missing("alice") + + +def test_fallback_and_docker_lark_cli_versions_match(): + dockerfile = Path(__file__).resolve().parents[1] / "Dockerfile" + match = re.search(r"^ARG LARK_CLI_NPM_VERSION=(?P\S+)$", dockerfile.read_text(encoding="utf-8"), re.MULTILINE) + + assert match is not None + assert lark_cli.LARK_CLI_NPM_VERSION == match.group("version") + assert lark_cli.FALLBACK_LARK_CLI_VERSION == f"v{lark_cli.LARK_CLI_NPM_VERSION}" + + +def test_resolve_lark_cli_path_prefers_managed_gateway_cli(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + managed_bin = lark_cli.lark_cli_managed_gateway_dir() / "node_modules" / ".bin" / "lark-cli" + managed_bin.parent.mkdir(parents=True) + managed_bin.write_text("#!/bin/sh\n", encoding="utf-8") + + monkeypatch.setattr(lark_cli.shutil, "which", lambda _name: "/usr/bin/lark-cli") + + assert lark_cli._resolve_lark_cli_path() == str(managed_bin) + + +def test_install_managed_gateway_lark_cli_uses_deerflow_prefix(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + captured: dict[str, object] = {} + + monkeypatch.setattr(lark_cli.shutil, "which", lambda name: "/usr/bin/npm" if name == "npm" else None) + + def _run(args, **kwargs): + captured["args"] = args + captured["kwargs"] = kwargs + managed_bin = lark_cli.lark_cli_managed_gateway_dir() / "node_modules" / ".bin" / "lark-cli" + managed_bin.parent.mkdir(parents=True) + managed_bin.write_text("#!/bin/sh\n", encoding="utf-8") + return subprocess.CompletedProcess(args=args, returncode=0, stdout="", stderr="") + + monkeypatch.setattr(lark_cli.subprocess, "run", _run) + monkeypatch.setattr(lark_cli, "_probe_lark_cli_at_path", lambda path: lark_cli.LarkCliProbe(available=True, path=path, version="lark-cli VERSION 1.2.3")) + + result = lark_cli._install_managed_gateway_lark_cli("v1.2.3") + + assert result.available is True + assert result.version == "lark-cli VERSION 1.2.3" + assert captured["args"] == [ + "/usr/bin/npm", + "install", + "--prefix", + str(lark_cli.lark_cli_managed_gateway_dir()), + "--no-audit", + "--no-fund", + "@larksuite/cli@1.2.3", + ] + + +def test_install_lark_integration_installs_managed_gateway_cli_before_skill_pack(monkeypatch, tmp_path): + reset_skill_storage() + _patch_paths(monkeypatch, tmp_path / "home") + skills_root = tmp_path / "skills" + (skills_root / "public").mkdir(parents=True) + (skills_root / "custom").mkdir() + config = _config(skills_root) + archive = _make_lark_cli_source_zip(tmp_path) + downloaded_versions: list[str] = [] + + monkeypatch.setattr(lark_cli, "probe_lark_auth", lambda _user_id, **_kwargs: lark_cli.LarkAuthProbe(status="not_configured", message="not configured")) + monkeypatch.setattr(lark_cli, "_ensure_managed_gateway_lark_cli", lambda: lark_cli.LarkCliProbe(available=True, path="/managed/bin/lark-cli", version="lark-cli VERSION 9.9.9")) + + def _download(version: str) -> Path: + downloaded_versions.append(version) + return archive + + monkeypatch.setattr(lark_cli, "_download_lark_archive", _download) + + result = lark_cli.install_lark_integration("alice", config) + + assert downloaded_versions == ["v9.9.9"] + assert result.status.manifest_version == "v9.9.9" + reset_skill_storage() + + +def test_resolve_latest_lark_cli_version_uses_release_tag(monkeypatch): + class _Resp: + def __enter__(self): + return self + + def __exit__(self, *args): + return False + + def read(self): + return json.dumps({"tag_name": "v1.2.3"}).encode("utf-8") + + monkeypatch.setattr(lark_cli.urllib.request, "urlopen", lambda *a, **k: _Resp()) + assert lark_cli._resolve_latest_lark_cli_version() == "v1.2.3" + + +def test_resolve_latest_lark_cli_version_falls_back_on_error(monkeypatch): + def _boom(*a, **k): + raise OSError("network down") + + monkeypatch.setattr(lark_cli.urllib.request, "urlopen", _boom) + assert lark_cli._resolve_latest_lark_cli_version() == lark_cli.FALLBACK_LARK_CLI_VERSION + + +def test_lark_archive_url_rejects_invalid_version_tag(): + with pytest.raises(ValueError, match="Invalid Lark CLI version tag"): + lark_cli._lark_archive_url("v1.2.3/../../evil") + + +def test_start_lark_auth_returns_browser_url(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + captured: dict[str, object] = {} + + def _run(args, **kwargs): + captured["args"] = args + captured["env"] = kwargs["env"] + return subprocess.CompletedProcess( + args=args, + returncode=0, + stdout=json.dumps( + { + "verification_url": "https://open.feishu.cn/auth/mock", + "device_code": "device-code", + "expires_in": 600, + } + ), + stderr="", + ) + + monkeypatch.setattr(lark_cli.shutil, "which", lambda _name: "/usr/bin/lark-cli") + monkeypatch.setattr(lark_cli.subprocess, "run", _run) + + result = lark_cli.start_lark_auth("alice", domains=("calendar",), recommend=True) + + assert result.verification_url == "https://open.feishu.cn/auth/mock" + assert result.device_code == "device-code" + assert captured["args"] == [ + "/usr/bin/lark-cli", + "auth", + "login", + "--no-wait", + "--json", + "--recommend", + "--domain", + "calendar", + ] + env = captured["env"] + assert isinstance(env, dict) + assert env["LARKSUITE_CLI_CONFIG_DIR"].endswith("users/alice/integrations/lark-cli/config") + + +def test_start_lark_auth_uses_minimal_login_by_default(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + captured: dict[str, object] = {} + + def _run(args, **kwargs): + captured["args"] = args + captured["env"] = kwargs["env"] + return subprocess.CompletedProcess( + args=args, + returncode=0, + stdout=json.dumps( + { + "verification_url": "https://open.feishu.cn/auth/mock", + "device_code": "device-code", + "expires_in": 600, + } + ), + stderr="", + ) + + monkeypatch.setattr(lark_cli.shutil, "which", lambda _name: "/usr/bin/lark-cli") + monkeypatch.setattr(lark_cli.subprocess, "run", _run) + + result = lark_cli.start_lark_auth("alice") + + assert result.verification_url == "https://open.feishu.cn/auth/mock" + assert captured["args"] == [ + "/usr/bin/lark-cli", + "auth", + "login", + "--no-wait", + "--json", + ] + + +def test_lark_cli_env_from_runtime_exposes_settings_auth_to_lark_commands(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + runtime = SimpleNamespace(context={"user_id": "alice"}) + + env = _lark_cli_env_from_runtime(runtime, "lark-cli auth status --json", sandbox_paths=False) + + assert env is not None + assert env["LARKSUITE_CLI_CONFIG_DIR"].endswith("users/alice/integrations/lark-cli/config") + assert env["LARKSUITE_CLI_DATA_DIR"].endswith("users/alice/integrations/lark-cli/data") + + +def test_lark_cli_env_hardens_existing_credential_tree(monkeypatch, tmp_path) -> None: + _patch_paths(monkeypatch, tmp_path / "home") + config_dir = lark_cli.lark_cli_config_dir("alice") + data_dir = lark_cli.lark_cli_data_dir("alice") + config_dir.mkdir(parents=True) + data_dir.mkdir(parents=True) + secret_file = config_dir / "config.json" + token_file = data_dir / "auth.json" + secret_file.write_text('{"appSecret":"secret"}', encoding="utf-8") + token_file.write_text('{"token":"secret"}', encoding="utf-8") + config_dir.chmod(0o755) + data_dir.chmod(0o777) + secret_file.chmod(0o644) + token_file.chmod(0o666) + + lark_cli.lark_cli_env_overlay("alice") + + assert stat.S_IMODE(config_dir.stat().st_mode) == 0o700 + assert stat.S_IMODE(data_dir.stat().st_mode) == 0o700 + assert stat.S_IMODE(secret_file.stat().st_mode) == 0o600 + assert stat.S_IMODE(token_file.stat().st_mode) == 0o600 + + +def test_lark_cli_env_rejects_symlinks_in_credential_tree(monkeypatch, tmp_path) -> None: + _patch_paths(monkeypatch, tmp_path / "home") + config_dir = lark_cli.lark_cli_config_dir("alice") + config_dir.mkdir(parents=True) + outside = tmp_path / "outside-secret" + outside.write_text("secret", encoding="utf-8") + try: + (config_dir / "config.json").symlink_to(outside) + except (NotImplementedError, OSError) as exc: + pytest.skip(f"symlinks are not available: {exc}") + + with pytest.raises(ValueError, match="symlink"): + lark_cli.lark_cli_env_overlay("alice") + + +def test_save_lark_app_config_rehardens_files_written_by_cli(monkeypatch, tmp_path) -> None: + _patch_paths(monkeypatch, tmp_path / "home") + monkeypatch.setattr(lark_cli, "_require_lark_cli_path", lambda: "/usr/bin/lark-cli") + + def _run(args, **kwargs): + config_file = Path(kwargs["env"]["LARKSUITE_CLI_CONFIG_DIR"]) / "config.json" + config_file.write_text('{"appSecret":"secret"}', encoding="utf-8") + config_file.chmod(0o644) + return subprocess.CompletedProcess(args=args, returncode=0, stdout="", stderr="") + + monkeypatch.setattr(lark_cli.subprocess, "run", _run) + + lark_cli._save_lark_app_config_with_cli("alice", app_id="cli_app", app_secret="secret", brand="feishu") + + config_file = lark_cli.lark_cli_config_dir("alice") / "config.json" + assert stat.S_IMODE(config_file.stat().st_mode) == 0o600 + + +def test_lark_cli_json_rehardens_auth_files_written_by_cli(monkeypatch, tmp_path) -> None: + _patch_paths(monkeypatch, tmp_path / "home") + + def _run(args, **kwargs): + token_file = Path(kwargs["env"]["LARKSUITE_CLI_DATA_DIR"]) / "auth.json" + token_file.write_text('{"token":"secret"}', encoding="utf-8") + token_file.chmod(0o644) + return subprocess.CompletedProcess(args=args, returncode=0, stdout="{}", stderr="") + + monkeypatch.setattr(lark_cli.subprocess, "run", _run) + + lark_cli._run_lark_cli_json(["/usr/bin/lark-cli", "auth", "login"], user_id="alice", timeout=5) + + token_file = lark_cli.lark_cli_data_dir("alice") / "auth.json" + assert stat.S_IMODE(token_file.stat().st_mode) == 0o600 + + +def test_lark_cli_env_from_runtime_uses_container_paths_for_sandbox_lark_commands(): + runtime = SimpleNamespace(context={"user_id": "alice"}) + + env = _lark_cli_env_from_runtime(runtime, "/usr/bin/lark-cli auth status", sandbox_paths=True) + + assert env is not None + assert env["LARKSUITE_CLI_CONFIG_DIR"] == lark_cli.LARK_CLI_SANDBOX_CONFIG_DIR + assert env["LARKSUITE_CLI_DATA_DIR"] == lark_cli.LARK_CLI_SANDBOX_DATA_DIR + + +def test_lark_cli_env_from_runtime_ignores_non_lark_commands(tmp_path, monkeypatch): + _patch_paths(monkeypatch, tmp_path / "home") + runtime = SimpleNamespace(context={"user_id": "alice"}) + + assert _lark_cli_env_from_runtime(runtime, "echo hello", sandbox_paths=False) is None + + +def test_lark_auth_probe_distinguishes_local_configuration_from_live_verification(monkeypatch, tmp_path) -> None: + assert "verified" in lark_cli.LarkAuthProbe.__dataclass_fields__ + _patch_paths(monkeypatch, tmp_path / "home") + config_file = lark_cli.lark_cli_config_dir("alice") / "config.json" + config_file.parent.mkdir(parents=True) + config_file.write_text( + json.dumps( + { + "currentApp": "cli_app", + "apps": [ + { + "name": "cli_app", + "appId": "cli_app", + "appSecret": "secret", + "brand": "feishu", + } + ], + } + ), + encoding="utf-8", + ) + calls: list[list[str]] = [] + + monkeypatch.setattr(lark_cli, "_resolve_lark_cli_path", lambda: "/usr/bin/lark-cli") + + def _run(args, **_kwargs): + calls.append(args) + return subprocess.CompletedProcess( + args=args, + returncode=0, + stdout='{"identities":{"user":{"userName":"Alice"}}}', + stderr="", + ) + + monkeypatch.setattr(lark_cli.subprocess, "run", _run) + + configured = lark_cli.probe_lark_auth("alice", verify=False) + live_verified = lark_cli.probe_lark_auth("alice", verify=True) + + assert configured.status == "authenticated" + assert configured.verified is False + assert "not live-verified" in (configured.message or "") + assert live_verified.verified is True + assert "live-verified" in (live_verified.message or "") + assert calls[0] == ["/usr/bin/lark-cli", "auth", "status", "--json"] + assert calls[1] == ["/usr/bin/lark-cli", "auth", "status", "--json", "--verify"] + + +def test_complete_lark_auth_polls_device_code_and_returns_status(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + skills_root = tmp_path / "skills" + (skills_root / "public").mkdir(parents=True) + (skills_root / "custom").mkdir() + config = _config(skills_root) + captured: dict[str, object] = {} + + def _run_lark_cli_json(args, **kwargs): + captured["args"] = args + captured["kwargs"] = kwargs + return {} + + monkeypatch.setattr(lark_cli, "_resolve_lark_cli_path", lambda: "/usr/bin/lark-cli") + monkeypatch.setattr(lark_cli, "_run_lark_cli_json", _run_lark_cli_json) + monkeypatch.setattr( + lark_cli, + "get_lark_integration_status", + lambda _user_id, _config, **_kwargs: lark_cli.LarkIntegrationStatus( + installed=True, + version="v1.0.65", + manifest_version="v1.0.65", + latest_available_version=None, + runtime_version_mismatch=False, + app_configured=True, + app_id="cli_mock", + app_brand="feishu", + skills_expected=27, + skills_installed=27, + installed_skills=("lark-doc",), + enabled_skills=("lark-doc",), + install_path="/tmp/lark", + cli=lark_cli.LarkCliProbe(available=True), + auth=lark_cli.LarkAuthProbe(status="authenticated", user="Alice"), + ), + ) + + result = lark_cli.complete_lark_auth("alice", config, device_code="device-code") + + assert result.success is True + assert captured["args"] == [ + "/usr/bin/lark-cli", + "auth", + "login", + "--device-code", + "device-code", + "--json", + ] + assert captured["kwargs"] == { + "user_id": "alice", + "timeout": 45, + "allow_empty_success": True, + } + + +def test_complete_lark_auth_accepts_short_automatic_poll_timeout(monkeypatch, tmp_path) -> None: + assert "wait_timeout_seconds" in inspect.signature(lark_cli.complete_lark_auth).parameters + _patch_paths(monkeypatch, tmp_path / "home") + config = _config(tmp_path / "skills") + captured: dict[str, object] = {} + + monkeypatch.setattr(lark_cli, "_require_lark_cli_path", lambda: "/usr/bin/lark-cli") + monkeypatch.setattr( + lark_cli, + "_run_lark_cli_json", + lambda _args, **kwargs: captured.update(kwargs) or {}, + ) + monkeypatch.setattr( + lark_cli, + "get_lark_integration_status", + lambda _user_id, _config, **_kwargs: lark_cli.LarkIntegrationStatus( + installed=True, + version="v1.0.65", + manifest_version="v1.0.65", + latest_available_version=None, + runtime_version_mismatch=False, + app_configured=True, + app_id="cli_mock", + app_brand="feishu", + skills_expected=27, + skills_installed=27, + installed_skills=("lark-doc",), + enabled_skills=("lark-doc",), + install_path="/tmp/lark", + cli=lark_cli.LarkCliProbe(available=True), + auth=lark_cli.LarkAuthProbe(status="authenticated", user="Alice"), + ), + ) + + result = lark_cli.complete_lark_auth( + "alice", + config, + device_code="device-code", + wait_timeout_seconds=8, + ) + + assert result.success is True + assert captured["timeout"] == 8 + + +def test_auth_complete_request_bounds_poll_timeout() -> None: + model = integrations_router.LarkAuthCompleteRequest(device_code="device-code", wait_timeout_seconds=8) + assert "wait_timeout_seconds" in type(model).model_fields + assert model.wait_timeout_seconds == 8 + with pytest.raises(ValueError): + integrations_router.LarkAuthCompleteRequest(device_code="device-code", wait_timeout_seconds=4) + with pytest.raises(ValueError): + integrations_router.LarkAuthCompleteRequest(device_code="device-code", wait_timeout_seconds=46) + + +def test_start_lark_config_returns_app_registration_url(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + monkeypatch.setattr( + lark_cli, + "_request_lark_app_registration_begin", + lambda _brand: { + "user_code": "abc", + "device_code": "config-device-code", + "expires_in": 600, + "interval": 5, + }, + ) + + result = lark_cli.start_lark_config("alice", brand="feishu") + + assert result.device_code == "config-device-code" + assert result.user_code == "abc" + assert result.verification_url.startswith("https://open.feishu.cn/page/cli?") + assert "user_code=abc" in result.verification_url + + +def test_complete_lark_config_saves_app_credentials_and_returns_status(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + skills_root = tmp_path / "skills" + (skills_root / "public").mkdir(parents=True) + (skills_root / "custom").mkdir() + config = _config(skills_root) + captured: dict[str, object] = {} + + monkeypatch.setattr( + lark_cli, + "_poll_lark_app_registration", + lambda **_kwargs: { + "client_id": "cli_mock", + "client_secret": "secret", + "user_info": {"tenant_brand": "feishu"}, + }, + ) + monkeypatch.setattr( + lark_cli, + "_save_lark_app_config_with_cli", + lambda user_id, **kwargs: captured.update({"user_id": user_id, **kwargs}), + ) + monkeypatch.setattr( + lark_cli, + "get_lark_integration_status", + lambda _user_id, _config, **_kwargs: lark_cli.LarkIntegrationStatus( + installed=True, + version="v1.0.65", + manifest_version="v1.0.65", + latest_available_version=None, + runtime_version_mismatch=False, + app_configured=True, + app_id="cli_mock", + app_brand="feishu", + skills_expected=27, + skills_installed=27, + installed_skills=("lark-doc",), + enabled_skills=("lark-doc",), + install_path="/tmp/lark", + cli=lark_cli.LarkCliProbe(available=True), + auth=lark_cli.LarkAuthProbe(status="not_authorized", user=None), + ), + ) + + result = lark_cli.complete_lark_config("alice", config, device_code="config-device-code", brand="feishu") + + assert result.success is True + assert captured == { + "user_id": "alice", + "app_id": "cli_mock", + "app_secret": "secret", + "brand": "feishu", + } + + +def test_complete_lark_config_repolls_lark_tenant_for_client_secret(monkeypatch, tmp_path): + _patch_paths(monkeypatch, tmp_path / "home") + skills_root = tmp_path / "skills" + (skills_root / "public").mkdir(parents=True) + (skills_root / "custom").mkdir() + config = _config(skills_root) + poll_calls: list[dict[str, object]] = [] + captured: dict[str, object] = {} + + def _poll_lark_app_registration(**kwargs): + poll_calls.append(kwargs) + if kwargs["brand"] == "feishu": + return { + "client_id": "cli_mock", + "user_info": {"tenant_brand": "lark"}, + } + return { + "client_id": "cli_mock", + "client_secret": "secret", + "user_info": {"tenant_brand": "lark"}, + } + + monkeypatch.setattr(lark_cli, "_poll_lark_app_registration", _poll_lark_app_registration) + monkeypatch.setattr( + lark_cli, + "_save_lark_app_config_with_cli", + lambda user_id, **kwargs: captured.update({"user_id": user_id, **kwargs}), + ) + monkeypatch.setattr( + lark_cli, + "get_lark_integration_status", + lambda _user_id, _config, **_kwargs: lark_cli.LarkIntegrationStatus( + installed=True, + version="v1.0.65", + manifest_version="v1.0.65", + latest_available_version=None, + runtime_version_mismatch=False, + app_configured=True, + app_id="cli_mock", + app_brand="lark", + skills_expected=27, + skills_installed=27, + installed_skills=("lark-doc",), + enabled_skills=("lark-doc",), + install_path="/tmp/lark", + cli=lark_cli.LarkCliProbe(available=True), + auth=lark_cli.LarkAuthProbe(status="not_authorized", user=None), + ), + ) + + result = lark_cli.complete_lark_config("alice", config, device_code="config-device-code", brand="feishu") + + assert result.success is True + assert [call["brand"] for call in poll_calls] == ["feishu", "lark"] + assert captured == { + "user_id": "alice", + "app_id": "cli_mock", + "app_secret": "secret", + "brand": "lark", + } + + +def _make_user(system_role: str) -> User: + return User(email=f"{system_role}-integration@example.com", password_hash="x", system_role=system_role, id=uuid4()) + + +def _make_app(*, system_role: str, config): + app = make_authed_test_app(user_factory=lambda: _make_user(system_role)) + app.dependency_overrides[get_config] = lambda: config + app.include_router(integrations_router.router) + return app + + +def test_lark_install_requires_admin(monkeypatch, tmp_path): + config = _config(tmp_path / "skills") + app = _make_app(system_role="user", config=config) + + def _should_not_install(*args, **kwargs): + raise AssertionError("install should be admin-gated") + + monkeypatch.setattr(integrations_router, "install_lark_integration", _should_not_install) + + with TestClient(app) as client: + response = client.post("/api/integrations/lark/install") + + assert response.status_code == 403 + + +def test_lark_status_is_available_to_authenticated_users(monkeypatch, tmp_path): + config = _config(tmp_path / "skills") + app = _make_app(system_role="user", config=config) + + monkeypatch.setattr( + integrations_router, + "get_lark_integration_status", + lambda _user_id, _config, **_kwargs: lark_cli.LarkIntegrationStatus( + installed=False, + version="v1.0.65", + manifest_version=None, + latest_available_version=None, + runtime_version_mismatch=False, + app_configured=False, + app_id=None, + app_brand=None, + skills_expected=27, + skills_installed=0, + installed_skills=(), + enabled_skills=(), + install_path="/tmp/lark-cli", + cli=lark_cli.LarkCliProbe(available=False, error="missing"), + auth=lark_cli.LarkAuthProbe(status="unavailable", message="missing"), + ), + ) + + with TestClient(app) as client: + response = client.get("/api/integrations/lark/status") + + assert response.status_code == 200 + assert response.json()["installed"] is False + + +def _status_with_host_paths() -> lark_cli.LarkIntegrationStatus: + return lark_cli.LarkIntegrationStatus( + installed=True, + version="v1.0.65", + manifest_version="v1.0.65", + latest_available_version=None, + runtime_version_mismatch=False, + app_configured=True, + app_id="cli_mock", + app_brand="feishu", + skills_expected=27, + skills_installed=27, + installed_skills=("lark-doc",), + enabled_skills=("lark-doc",), + install_path="/home/deer-flow/.deer-flow/integrations/skills/lark-cli", + cli=lark_cli.LarkCliProbe(available=True, path="/usr/bin/lark-cli", version="1.0.65"), + auth=lark_cli.LarkAuthProbe(status="authenticated", user="alice"), + ) + + +def test_lark_status_redacts_host_paths_for_non_admin(monkeypatch, tmp_path): + config = _config(tmp_path / "skills") + app = _make_app(system_role="user", config=config) + monkeypatch.setattr(integrations_router, "get_lark_integration_status", lambda *_a, **_k: _status_with_host_paths()) + + with TestClient(app) as client: + body = client.get("/api/integrations/lark/status").json() + + assert body["install_path"] == "" + assert body["cli"]["path"] is None + # Non-sensitive fields are still reported. + assert body["installed"] is True + assert body["cli"]["version"] == "1.0.65" + + +def test_lark_status_exposes_host_paths_for_admin(monkeypatch, tmp_path): + config = _config(tmp_path / "skills") + app = _make_app(system_role="admin", config=config) + monkeypatch.setattr(integrations_router, "get_lark_integration_status", lambda *_a, **_k: _status_with_host_paths()) + + with TestClient(app) as client: + body = client.get("/api/integrations/lark/status").json() + + assert body["install_path"] == "/home/deer-flow/.deer-flow/integrations/skills/lark-cli" + assert body["cli"]["path"] == "/usr/bin/lark-cli" + + +def test_lark_config_start_route_returns_browser_url(monkeypatch, tmp_path): + config = _config(tmp_path / "skills") + app = _make_app(system_role="user", config=config) + + monkeypatch.setattr( + integrations_router, + "start_lark_config", + lambda _user_id, **_kwargs: lark_cli.LarkConfigStartResult( + verification_url="https://open.feishu.cn/page/cli?user_code=config", + device_code="config-device-code", + expires_in=600, + interval=5, + user_code="config", + brand="feishu", + ), + ) + + with TestClient(app) as client: + response = client.post("/api/integrations/lark/config/start", json={"brand": "feishu"}) + + assert response.status_code == 200 + assert response.json()["verification_url"] == "https://open.feishu.cn/page/cli?user_code=config" + assert response.json()["device_code"] == "config-device-code" + + +def test_lark_config_complete_route_saves_app_credentials(monkeypatch, tmp_path): + config = _config(tmp_path / "skills") + app = _make_app(system_role="user", config=config) + + monkeypatch.setattr( + integrations_router, + "complete_lark_config", + lambda _user_id, _config, *, device_code, **_kwargs: lark_cli.LarkConfigCompleteResult( + success=True, + message=f"configured {device_code}", + status=lark_cli.LarkIntegrationStatus( + installed=True, + version="v1.0.65", + manifest_version="v1.0.65", + latest_available_version=None, + runtime_version_mismatch=False, + app_configured=True, + app_id="cli_mock", + app_brand="feishu", + skills_expected=27, + skills_installed=27, + installed_skills=("lark-doc",), + enabled_skills=("lark-doc",), + install_path="/tmp/lark", + cli=lark_cli.LarkCliProbe(available=True), + auth=lark_cli.LarkAuthProbe(status="not_authorized", user=None), + ), + ), + ) + + with TestClient(app) as client: + response = client.post( + "/api/integrations/lark/config/complete", + json={"device_code": "config-device-code", "brand": "feishu", "interval": 5, "expires_in": 600}, + ) + + assert response.status_code == 200 + assert response.json()["success"] is True + assert response.json()["status"]["app_configured"] is True + + +def test_lark_auth_start_route_returns_browser_url(monkeypatch, tmp_path): + config = _config(tmp_path / "skills") + app = _make_app(system_role="user", config=config) + captured_kwargs: dict[str, object] = {} + + monkeypatch.setattr( + integrations_router, + "start_lark_auth", + lambda _user_id, **kwargs: ( + captured_kwargs.update(kwargs) + or lark_cli.LarkAuthStartResult( + verification_url="https://open.feishu.cn/auth/mock", + device_code="device-code", + expires_in=600, + ) + ), + ) + + with TestClient(app) as client: + response = client.post("/api/integrations/lark/auth/start", json={}) + + assert response.status_code == 200 + assert response.json()["verification_url"] == "https://open.feishu.cn/auth/mock" + assert response.json()["device_code"] == "device-code" + assert captured_kwargs == {"domains": (), "scope": None, "recommend": False} + + +def test_lark_auth_start_route_passes_explicit_recommend(monkeypatch, tmp_path): + config = _config(tmp_path / "skills") + app = _make_app(system_role="user", config=config) + captured_kwargs: dict[str, object] = {} + + monkeypatch.setattr( + integrations_router, + "start_lark_auth", + lambda _user_id, **kwargs: ( + captured_kwargs.update(kwargs) + or lark_cli.LarkAuthStartResult( + verification_url="https://open.feishu.cn/auth/mock", + device_code="device-code", + expires_in=600, + ) + ), + ) + + with TestClient(app) as client: + response = client.post("/api/integrations/lark/auth/start", json={"recommend": True}) + + assert response.status_code == 200 + assert response.json()["verification_url"] == "https://open.feishu.cn/auth/mock" + assert response.json()["device_code"] == "device-code" + assert captured_kwargs == {"domains": (), "scope": None, "recommend": True} + + +def test_lark_auth_complete_route_polls_device_code(monkeypatch, tmp_path): + config = _config(tmp_path / "skills") + app = _make_app(system_role="user", config=config) + captured_kwargs = {} + + def _complete_auth(_user_id, _config, **kwargs): + captured_kwargs.update(kwargs) + return lark_cli.LarkAuthCompleteResult( + success=True, + message=f"completed {kwargs['device_code']}", + status=lark_cli.LarkIntegrationStatus( + installed=True, + version="v1.0.65", + manifest_version="v1.0.65", + latest_available_version=None, + runtime_version_mismatch=False, + app_configured=True, + app_id="cli_mock", + app_brand="feishu", + skills_expected=27, + skills_installed=27, + installed_skills=("lark-doc",), + enabled_skills=("lark-doc",), + install_path="/tmp/lark", + cli=lark_cli.LarkCliProbe(available=True), + auth=lark_cli.LarkAuthProbe(status="authenticated", user="Alice", verified=True), + ), + ) + + monkeypatch.setattr(integrations_router, "complete_lark_auth", _complete_auth) + + with TestClient(app) as client: + response = client.post("/api/integrations/lark/auth/complete", json={"device_code": "device-code"}) + + assert response.status_code == 200 + assert response.json()["success"] is True + assert response.json()["status"]["auth"]["status"] == "authenticated" + assert response.json()["status"]["auth"]["verified"] is True + assert captured_kwargs == {"device_code": "device-code", "wait_timeout_seconds": 45} diff --git a/backend/tests/test_local_sandbox_provider_mounts.py b/backend/tests/test_local_sandbox_provider_mounts.py index 1628ad9f5..f072d6378 100644 --- a/backend/tests/test_local_sandbox_provider_mounts.py +++ b/backend/tests/test_local_sandbox_provider_mounts.py @@ -544,6 +544,35 @@ class TestMultipleMounts: class TestLocalSandboxProviderMounts: + def test_thread_mappings_mount_global_integrations_for_every_user(self, tmp_path): + from deerflow.config.paths import Paths + + paths = Paths(base_dir=tmp_path / "home") + skills_dir = tmp_path / "skills" + (skills_dir / "public").mkdir(parents=True) + (skills_dir / "custom").mkdir() + config = SimpleNamespace( + skills=SimpleNamespace( + container_path="/mnt/skills", + get_skills_path=lambda: skills_dir, + use="deerflow.skills.storage.local_skill_storage:LocalSkillStorage", + ) + ) + + with ( + patch("deerflow.config.get_app_config", return_value=config), + patch("deerflow.config.paths.get_paths", return_value=paths), + ): + alice = LocalSandboxProvider._build_thread_path_mappings("thread-a", user_id="alice") + bob = LocalSandboxProvider._build_thread_path_mappings("thread-b", user_id="bob") + + alice_integrations = next(mapping for mapping in alice if mapping.container_path == "/mnt/skills/integrations") + bob_integrations = next(mapping for mapping in bob if mapping.container_path == "/mnt/skills/integrations") + expected = str(tmp_path / "home" / "integrations" / "skills") + assert alice_integrations.local_path == expected + assert bob_integrations.local_path == expected + assert alice_integrations.read_only is True + def test_setup_path_mappings_uses_configured_skills_container_path_as_reserved_prefix(self, tmp_path): skills_dir = tmp_path / "skills" skills_dir.mkdir() diff --git a/backend/tests/test_paths_user_isolation.py b/backend/tests/test_paths_user_isolation.py index 5f91e32e1..a7197bcec 100644 --- a/backend/tests/test_paths_user_isolation.py +++ b/backend/tests/test_paths_user_isolation.py @@ -66,6 +66,31 @@ class TestMakeSafeUserId: make_safe_user_id("") +class TestValidateIntegrationId: + def test_accepts_dotted_integration_id(self): + from deerflow.config.paths import _validate_integration_id + + assert _validate_integration_id("lark-cli") == "lark-cli" + assert _validate_integration_id("some.integration") == "some.integration" + + @pytest.mark.parametrize("integration_id", [".", ".."]) + def test_rejects_dot_and_dotdot(self, integration_id): + from deerflow.config.paths import _validate_integration_id + + with pytest.raises(ValueError, match="Invalid integration_id"): + _validate_integration_id(integration_id) + + @pytest.mark.parametrize("integration_id", [".", ".."]) + def test_host_integration_config_dir_rejects_dot_traversal(self, paths: Paths, integration_id): + with pytest.raises(ValueError, match="Invalid integration_id"): + paths.host_user_integration_config_dir("alice", integration_id) + + @pytest.mark.parametrize("integration_id", [".", ".."]) + def test_host_integration_data_dir_rejects_dot_traversal(self, paths: Paths, integration_id): + with pytest.raises(ValueError, match="Invalid integration_id"): + paths.host_user_integration_data_dir("alice", integration_id) + + class TestUserDir: def test_user_dir(self, paths: Paths): assert paths.user_dir("alice") == paths.base_dir / "users" / "alice" diff --git a/backend/tests/test_provisioner_mount_contract.py b/backend/tests/test_provisioner_mount_contract.py new file mode 100644 index 000000000..403b890e7 --- /dev/null +++ b/backend/tests/test_provisioner_mount_contract.py @@ -0,0 +1,28 @@ +"""Keep Gateway/provisioner extra-mount allowlists in lockstep.""" + +from __future__ import annotations + +import ast +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[2] + + +def _literal_assignment(path: Path, name: str): + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + for node in tree.body: + if isinstance(node, ast.Assign) and any(isinstance(target, ast.Name) and target.id == name for target in node.targets): + return ast.literal_eval(node.value) + raise AssertionError(f"{name} not found in {path}") + + +def test_gateway_and_provisioner_extra_mount_contracts_match() -> None: + gateway_path = REPO_ROOT / "backend/packages/harness/deerflow/community/aio_sandbox/remote_backend.py" + provisioner_path = REPO_ROOT / "docker/provisioner/app.py" + + gateway_paths = _literal_assignment(gateway_path, "_PROVISIONER_EXTRA_MOUNT_PATHS") + provisioner_paths = _literal_assignment(provisioner_path, "ALLOWED_EXTRA_MOUNT_PATHS") + + assert gateway_paths == provisioner_paths + assert "/mnt/integrations/lark-cli/runtime" in gateway_paths + assert _literal_assignment(provisioner_path, "MAX_EXTRA_MOUNTS") == 9 diff --git a/backend/tests/test_provisioner_pvc_volumes.py b/backend/tests/test_provisioner_pvc_volumes.py index c5d03f83a..6c61eedc5 100644 --- a/backend/tests/test_provisioner_pvc_volumes.py +++ b/backend/tests/test_provisioner_pvc_volumes.py @@ -1,5 +1,6 @@ """Regression tests for provisioner three-way skills + PVC volume support.""" +import pytest # ── _build_volumes ───────────────────────────────────────────────────── @@ -113,6 +114,67 @@ class TestBuildVolumes: assert volumes[0].name == "skills" assert volumes[-1].name == "user-data" + def test_extra_mount_uses_hostpath_when_userdata_pvc_is_disabled(self, provisioner_module): + """Controlled extra mounts should become extra hostPath volumes by default.""" + provisioner_module.SKILLS_PVC_NAME = "" + provisioner_module.USERDATA_PVC_NAME = "" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + extra_mounts = [ + provisioner_module.ExtraMount( + host_path="/state/users/alice/integrations/lark-cli/config", + container_path="/mnt/integrations/lark-cli/config", + read_only=False, + ) + ] + + volumes = provisioner_module._build_volumes("thread-1", extra_mounts=extra_mounts) + + # skills-public + skills-custom + user-data (3 base) + 1 extra volume. + assert len(volumes) == 4 + extra_vol = volumes[-1] + assert extra_vol.name == "extra-0" + assert extra_vol.host_path.path == "/state/users/alice/integrations/lark-cli/config" + assert extra_vol.host_path.type == "DirectoryOrCreate" + + def test_extra_mount_uses_userdata_pvc_when_configured(self, provisioner_module): + """PVC mode should use the same DeerFlow data PVC for runtime config mounts.""" + provisioner_module.SKILLS_PVC_NAME = "" + provisioner_module.USERDATA_PVC_NAME = "userdata-pvc" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + extra_mounts = [ + provisioner_module.ExtraMount( + host_path="/state/users/alice/integrations/lark-cli/config", + container_path="/mnt/integrations/lark-cli/config", + read_only=False, + ) + ] + + volumes = provisioner_module._build_volumes("thread-1", extra_mounts=extra_mounts) + + # skills-public + skills-custom + user-data (3 base) + 1 extra volume. + assert len(volumes) == 4 + extra_vol = volumes[-1] + assert extra_vol.name == "extra-0" + assert extra_vol.persistent_volume_claim is not None + assert extra_vol.persistent_volume_claim.claim_name == "userdata-pvc" + assert extra_vol.host_path is None + + def test_extra_mount_rejects_paths_outside_deerflow_state(self, provisioner_module): + """Provisioner must not accept arbitrary hostPath mounts from clients.""" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + extra_mounts = [ + provisioner_module.ExtraMount( + host_path="/etc", + container_path="/mnt/integrations/lark-cli/config", + read_only=False, + ) + ] + + with pytest.raises(provisioner_module.HTTPException) as exc_info: + provisioner_module._build_volumes("thread-1", extra_mounts=extra_mounts) + + assert exc_info.value.status_code == 400 + # ── _build_volume_mounts ─────────────────────────────────────────────── @@ -226,6 +288,64 @@ class TestBuildVolumeMounts: userdata_mount = mounts[-1] assert userdata_mount.sub_path == "deer-flow/users/default/threads/thread-42/user-data" + # ── Managed integration extra mounts ─────────────────────────────── + + def test_extra_mount_adds_volume_mount(self, provisioner_module): + """Controlled extra mounts should be mounted at their requested container path.""" + provisioner_module.SKILLS_PVC_NAME = "" + provisioner_module.USERDATA_PVC_NAME = "" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + extra_mounts = [ + provisioner_module.ExtraMount( + host_path="/state/users/alice/integrations/lark-cli/config", + container_path="/mnt/integrations/lark-cli/config", + read_only=False, + ) + ] + + mounts = provisioner_module._build_volume_mounts("thread-1", extra_mounts=extra_mounts) + + extra_mount = mounts[-1] + assert extra_mount.name == "extra-0" + assert extra_mount.mount_path == "/mnt/integrations/lark-cli/config" + assert extra_mount.read_only is False + assert extra_mount.sub_path is None + + def test_extra_mount_uses_pvc_subpath(self, provisioner_module): + """PVC extra mounts should point at the same user-scoped DeerFlow path.""" + provisioner_module.SKILLS_PVC_NAME = "" + provisioner_module.USERDATA_PVC_NAME = "userdata-pvc" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + extra_mounts = [ + provisioner_module.ExtraMount( + host_path="/state/users/alice/integrations/lark-cli/config", + container_path="/mnt/integrations/lark-cli/config", + read_only=False, + ) + ] + + mounts = provisioner_module._build_volume_mounts("thread-1", extra_mounts=extra_mounts) + + extra_mount = mounts[-1] + assert extra_mount.name == "extra-0" + assert extra_mount.sub_path == "deer-flow/users/alice/integrations/lark-cli/config" + + def test_extra_mount_rejects_unknown_container_path(self, provisioner_module): + """Only first-party managed mount paths are accepted.""" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + extra_mounts = [ + provisioner_module.ExtraMount( + host_path="/state/users/alice/integrations/lark-cli/config", + container_path="/mnt/secrets", + read_only=False, + ) + ] + + with pytest.raises(provisioner_module.HTTPException) as exc_info: + provisioner_module._build_volume_mounts("thread-1", extra_mounts=extra_mounts) + + assert exc_info.value.status_code == 400 + # ── _build_pod integration ───────────────────────────────────────────── @@ -293,6 +413,37 @@ class TestBuildPodVolumes: userdata_mount = pod.spec.containers[0].volume_mounts[-1] assert userdata_mount.sub_path == "deer-flow/users/user-7/threads/thread-1/user-data" + def test_pod_includes_extra_mounts(self, provisioner_module): + """Provisioner-created pods should include managed integration runtime mounts.""" + provisioner_module.SKILLS_PVC_NAME = "" + provisioner_module.USERDATA_PVC_NAME = "" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + extra_mounts = [ + provisioner_module.ExtraMount( + host_path="/state/users/alice/integrations/lark-cli/config", + container_path="/mnt/integrations/lark-cli/config", + read_only=False, + ), + provisioner_module.ExtraMount( + host_path="/state/users/alice/integrations/lark-cli/data", + container_path="/mnt/integrations/lark-cli/data", + read_only=False, + ), + ] + + pod = provisioner_module._build_pod( + "sandbox-1", + "thread-1", + user_id="alice", + extra_mounts=extra_mounts, + ) + + # skills-public + skills-custom + user-data (3 base) + 2 extra mounts. + assert len(pod.spec.volumes) == 5 + mount_paths = {mount.mount_path for mount in pod.spec.containers[0].volume_mounts} + assert "/mnt/integrations/lark-cli/config" in mount_paths + assert "/mnt/integrations/lark-cli/data" in mount_paths + def test_pod_three_way_skills_mount_paths(self, provisioner_module): """Ensure public/custom/legacy mount paths are correct.""" provisioner_module.SKILLS_PVC_NAME = "" @@ -315,3 +466,98 @@ class TestBuildPodVolumes: pod = provisioner_module._build_pod("sandbox-1", "thread-1", user_id="user-7") skills_mount = pod.spec.containers[0].volume_mounts[0] assert skills_mount.sub_path == "deer-flow/users/user-7/threads/thread-1/skills" + + +class TestLarkCliInitContainer: + """Init-container + emptyDir provisioning of the sandbox lark-cli runtime.""" + + def test_no_init_container_when_image_unset(self, provisioner_module): + provisioner_module.SKILLS_PVC_NAME = "" + provisioner_module.USERDATA_PVC_NAME = "" + provisioner_module.LARK_CLI_INIT_IMAGE = "" + pod = provisioner_module._build_pod( + "sandbox-1", + "thread-1", + provision_lark_cli_runtime=True, + ) + assert not pod.spec.init_containers + volume_names = {v.name for v in pod.spec.volumes} + assert provisioner_module.LARK_CLI_RUNTIME_VOLUME_NAME not in volume_names + + def test_no_init_container_when_flag_disabled(self, provisioner_module): + provisioner_module.SKILLS_PVC_NAME = "" + provisioner_module.USERDATA_PVC_NAME = "" + provisioner_module.LARK_CLI_INIT_IMAGE = "deer-flow/lark-cli-init:v1.0.65" + pod = provisioner_module._build_pod( + "sandbox-1", + "thread-1", + provision_lark_cli_runtime=False, + ) + assert not pod.spec.init_containers + + def test_init_container_and_emptydir_when_enabled(self, provisioner_module): + provisioner_module.SKILLS_PVC_NAME = "" + provisioner_module.USERDATA_PVC_NAME = "" + provisioner_module.LARK_CLI_INIT_IMAGE = "deer-flow/lark-cli-init:v1.0.65" + pod = provisioner_module._build_pod( + "sandbox-1", + "thread-1", + provision_lark_cli_runtime=True, + ) + + # emptyDir volume shared by init + sandbox containers. + runtime_volumes = [v for v in pod.spec.volumes if v.name == provisioner_module.LARK_CLI_RUNTIME_VOLUME_NAME] + assert len(runtime_volumes) == 1 + assert runtime_volumes[0].empty_dir is not None + + # Exactly one init container, pointing at the configured image and + # writing the runtime path. + assert pod.spec.init_containers is not None + assert len(pod.spec.init_containers) == 1 + init = pod.spec.init_containers[0] + assert init.image == "deer-flow/lark-cli-init:v1.0.65" + init_mount = init.volume_mounts[0] + assert init_mount.name == provisioner_module.LARK_CLI_RUNTIME_VOLUME_NAME + assert init_mount.mount_path == provisioner_module.LARK_CLI_RUNTIME_CONTAINER_PATH + assert init_mount.read_only is False + + # Sandbox container gets a read-only runtime mount at the same path. + sandbox_runtime_mounts = [m for m in pod.spec.containers[0].volume_mounts if m.name == provisioner_module.LARK_CLI_RUNTIME_VOLUME_NAME] + assert len(sandbox_runtime_mounts) == 1 + assert sandbox_runtime_mounts[0].mount_path == provisioner_module.LARK_CLI_RUNTIME_CONTAINER_PATH + assert sandbox_runtime_mounts[0].read_only is True + + def test_runtime_extra_mount_dropped_when_init_container_enabled(self, provisioner_module): + provisioner_module.SKILLS_PVC_NAME = "" + provisioner_module.USERDATA_PVC_NAME = "" + provisioner_module.DEER_FLOW_HOST_BASE_DIR = "/state" + provisioner_module.LARK_CLI_INIT_IMAGE = "deer-flow/lark-cli-init:v1.0.65" + extra_mounts = [ + provisioner_module.ExtraMount( + host_path="/state/users/alice/integrations/lark-cli/config", + container_path="/mnt/integrations/lark-cli/config", + read_only=False, + ), + provisioner_module.ExtraMount( + host_path="/state/integrations/lark-cli/sandbox-cli", + container_path="/mnt/integrations/lark-cli/runtime", + read_only=True, + ), + ] + + pod = provisioner_module._build_pod( + "sandbox-1", + "thread-1", + user_id="alice", + extra_mounts=extra_mounts, + provision_lark_cli_runtime=True, + ) + + # The credential config mount stays; the hostPath runtime extra mount is + # replaced by the emptyDir supplied by the init container (so the runtime + # path is not backed by an extra-* hostPath volume). + runtime_mounts = [m for m in pod.spec.containers[0].volume_mounts if m.mount_path == "/mnt/integrations/lark-cli/runtime"] + assert len(runtime_mounts) == 1 + assert runtime_mounts[0].name == provisioner_module.LARK_CLI_RUNTIME_VOLUME_NAME + mount_paths = {m.mount_path for m in pod.spec.containers[0].volume_mounts} + assert "/mnt/integrations/lark-cli/config" in mount_paths diff --git a/backend/tests/test_remote_sandbox_backend.py b/backend/tests/test_remote_sandbox_backend.py index c784b25e9..6affb8ddf 100644 --- a/backend/tests/test_remote_sandbox_backend.py +++ b/backend/tests/test_remote_sandbox_backend.py @@ -165,11 +165,12 @@ def test_create_delegates_to_provisioner_create(monkeypatch, expected_user_id): backend = RemoteSandboxBackend("http://provisioner:8002") expected = SandboxInfo(sandbox_id="abc123", sandbox_url="http://k3s:31001") - def mock_create(thread_id: str, sandbox_id: str, extra_mounts=None, *, user_id=None): + def mock_create(thread_id: str, sandbox_id: str, extra_mounts=None, *, user_id=None, provision_lark_cli_runtime=False): assert thread_id == "thread-1" assert sandbox_id == "abc123" assert extra_mounts == [("/host", "/container", False)] assert user_id == expected_user_id + assert provision_lark_cli_runtime is True return expected monkeypatch.setattr(backend, "_provisioner_create", mock_create) @@ -179,6 +180,7 @@ def test_create_delegates_to_provisioner_create(monkeypatch, expected_user_id): "abc123", extra_mounts=[("/host", "/container", False)], user_id=expected_user_id, + provision_lark_cli_runtime=True, ) assert result == expected @@ -194,6 +196,7 @@ def test_provisioner_create_returns_sandbox_info(monkeypatch): "thread_id": "thread-1", "user_id": "test-user-autouse", "include_legacy_skills": True, + "provision_lark_cli_runtime": False, } assert timeout == 30 return _StubResponse(payload={"sandbox_id": "abc123", "sandbox_url": "http://k3s:31001"}) @@ -205,6 +208,103 @@ def test_provisioner_create_returns_sandbox_info(monkeypatch): assert info.sandbox_url == "http://k3s:31001" +def test_provisioner_create_forwards_supported_extra_mounts(monkeypatch): + backend = RemoteSandboxBackend("http://provisioner:8002") + monkeypatch.setattr(remote_backend_mod, "user_should_see_legacy_skills", lambda user_id: False) + + def mock_post(url: str, json: dict, timeout: int, headers=None): + assert url == "http://provisioner:8002/api/sandboxes" + assert json["include_legacy_skills"] is False + assert json["extra_mounts"] == [ + { + "host_path": "/state/users/alice/skills/integrations", + "container_path": "/mnt/skills/integrations", + "read_only": True, + }, + { + "host_path": "/state/users/alice/integrations/lark-cli/config", + "container_path": "/mnt/integrations/lark-cli/config", + "read_only": False, + }, + ] + assert timeout == 30 + return _StubResponse(payload={"sandbox_id": "abc123", "sandbox_url": "http://k3s:31001"}) + + monkeypatch.setattr(requests, "post", mock_post) + + backend._provisioner_create( + "thread-1", + "abc123", + extra_mounts=[ + ("/state/users/alice/threads/thread-1/user-data/workspace", "/mnt/user-data/workspace", False), + ("/skills", "/mnt/skills", True), + ("/state/users/alice/skills/integrations", "/mnt/skills/integrations", True), + ("/state/users/alice/integrations/lark-cli/config", "/mnt/integrations/lark-cli/config", False), + ], + user_id="alice", + ) + + +def test_provisioner_create_strips_runtime_mount_when_init_container_enabled(monkeypatch): + backend = RemoteSandboxBackend("http://provisioner:8002") + monkeypatch.setattr(remote_backend_mod, "user_should_see_legacy_skills", lambda user_id: False) + + captured: dict = {} + + def mock_post(url: str, json: dict, timeout: int, headers=None): + captured.update(json) + return _StubResponse(payload={"sandbox_id": "abc123", "sandbox_url": "http://k3s:31001"}) + + monkeypatch.setattr(requests, "post", mock_post) + + backend._provisioner_create( + "thread-1", + "abc123", + extra_mounts=[ + ("/state/users/alice/integrations/lark-cli/config", "/mnt/integrations/lark-cli/config", False), + ("/state/users/alice/integrations/lark-cli/data", "/mnt/integrations/lark-cli/data", False), + ("/state/integrations/lark-cli/sandbox-cli", "/mnt/integrations/lark-cli/runtime", True), + ], + user_id="alice", + provision_lark_cli_runtime=True, + ) + + assert captured["provision_lark_cli_runtime"] is True + container_paths = {mount["container_path"] for mount in captured["extra_mounts"]} + # The init container supplies the runtime, so its mount is dropped, but the + # per-user credential mounts are still forwarded. + assert "/mnt/integrations/lark-cli/runtime" not in container_paths + assert "/mnt/integrations/lark-cli/config" in container_paths + assert "/mnt/integrations/lark-cli/data" in container_paths + + +def test_provisioner_create_keeps_runtime_mount_when_init_container_disabled(monkeypatch): + backend = RemoteSandboxBackend("http://provisioner:8002") + monkeypatch.setattr(remote_backend_mod, "user_should_see_legacy_skills", lambda user_id: False) + + captured: dict = {} + + def mock_post(url: str, json: dict, timeout: int, headers=None): + captured.update(json) + return _StubResponse(payload={"sandbox_id": "abc123", "sandbox_url": "http://k3s:31001"}) + + monkeypatch.setattr(requests, "post", mock_post) + + backend._provisioner_create( + "thread-1", + "abc123", + extra_mounts=[ + ("/state/integrations/lark-cli/sandbox-cli", "/mnt/integrations/lark-cli/runtime", True), + ], + user_id="alice", + provision_lark_cli_runtime=False, + ) + + assert captured["provision_lark_cli_runtime"] is False + container_paths = {mount["container_path"] for mount in captured["extra_mounts"]} + assert "/mnt/integrations/lark-cli/runtime" in container_paths + + def test_provisioner_create_accepts_anonymous_thread_id(monkeypatch): backend = RemoteSandboxBackend("http://provisioner:8002") monkeypatch.setattr(remote_backend_mod, "user_should_see_legacy_skills", lambda user_id: False) @@ -216,6 +316,7 @@ def test_provisioner_create_accepts_anonymous_thread_id(monkeypatch): "thread_id": None, "user_id": "test-user-autouse", "include_legacy_skills": False, + "provision_lark_cli_runtime": False, } assert timeout == 30 return _StubResponse(payload={"sandbox_id": "anon123", "sandbox_url": "http://k3s:31002"}) diff --git a/backend/tests/test_sandbox_tools_security.py b/backend/tests/test_sandbox_tools_security.py index 4356d13b9..59d0bc068 100644 --- a/backend/tests/test_sandbox_tools_security.py +++ b/backend/tests/test_sandbox_tools_security.py @@ -10,6 +10,7 @@ from deerflow.sandbox.tools import ( VIRTUAL_PATH_PREFIX, _apply_cwd_prefix, _compiled_mask_patterns, + _extract_skill_name_from_skills_path, _get_custom_mount_for_path, _get_custom_mounts, _is_acp_workspace_path, @@ -258,6 +259,26 @@ def test_mask_local_paths_no_thread_data_still_masks_skills() -> None: assert "/mnt/skills/a/b.md" in masked +def test_mask_local_paths_hides_global_integration_skill_paths(tmp_path: Path) -> None: + from deerflow.config.paths import Paths + + paths = Paths(base_dir=tmp_path) + integration_dir = tmp_path / "integrations" / "skills" / "lark-cli" / "lark-doc" + integration_dir.mkdir(parents=True) + output = f"Reading: {integration_dir / 'SKILL.md'}" + + with ( + patch("deerflow.sandbox.tools._get_skills_container_path", return_value="/mnt/skills"), + patch("deerflow.sandbox.tools._get_skills_host_path", return_value="/home/user/deer-flow/skills"), + patch("deerflow.config.paths.get_paths", return_value=paths), + patch("deerflow.runtime.user_context.get_effective_user_id", return_value="alice"), + ): + masked = mask_local_paths_in_output(output, _THREAD_DATA) + + assert str(integration_dir) not in masked + assert "/mnt/skills/integrations/lark-cli/lark-doc/SKILL.md" in masked + + # ---------- _reject_path_traversal ---------- @@ -372,6 +393,42 @@ def test_resolve_skills_path_resolves_root() -> None: assert resolved == "/home/user/deer-flow/skills" +def test_extract_skill_name_from_integration_skill_path() -> None: + with patch("deerflow.sandbox.tools._get_skills_container_path", return_value="/mnt/skills"): + assert _extract_skill_name_from_skills_path("/mnt/skills/integrations/lark-cli/lark-doc/SKILL.md") == "lark-doc" + assert _extract_skill_name_from_skills_path("/mnt/skills/integrations/lark-cli") is None + + +def test_resolve_skills_path_resolves_global_integration_skills(tmp_path: Path) -> None: + from deerflow.config.paths import Paths + + paths = Paths(base_dir=tmp_path) + expected = tmp_path / "integrations" / "skills" / "lark-cli" / "lark-doc" / "SKILL.md" + with ( + patch("deerflow.sandbox.tools._get_skills_container_path", return_value="/mnt/skills"), + patch("deerflow.sandbox.tools._get_skills_host_path", return_value="/home/user/deer-flow/skills"), + patch("deerflow.config.paths.get_paths", return_value=paths), + patch("deerflow.runtime.user_context.get_effective_user_id", return_value="alice"), + ): + resolved = _resolve_skills_path("/mnt/skills/integrations/lark-cli/lark-doc/SKILL.md") + + assert resolved == str(expected) + + +def test_resolve_skills_path_blocks_integration_traversal(tmp_path: Path) -> None: + from deerflow.config.paths import Paths + + paths = Paths(base_dir=tmp_path) + with ( + patch("deerflow.sandbox.tools._get_skills_container_path", return_value="/mnt/skills"), + patch("deerflow.sandbox.tools._get_skills_host_path", return_value="/home/user/deer-flow/skills"), + patch("deerflow.config.paths.get_paths", return_value=paths), + patch("deerflow.runtime.user_context.get_effective_user_id", return_value="alice"), + ): + with pytest.raises(PermissionError, match="path traversal detected"): + _resolve_skills_path("/mnt/skills/integrations/../../etc/passwd") + + def test_resolve_skills_path_raises_when_not_configured() -> None: """Should raise FileNotFoundError when skills directory is not available.""" with ( diff --git a/backend/tests/test_user_scoped_skill_storage.py b/backend/tests/test_user_scoped_skill_storage.py index 6ca2830e6..532fb070f 100644 --- a/backend/tests/test_user_scoped_skill_storage.py +++ b/backend/tests/test_user_scoped_skill_storage.py @@ -136,6 +136,24 @@ class TestSkillLoading: assert len(public_skills) == 1 assert public_skills[0].name == "deep-research" + def test_managed_integration_skills_are_global_but_enabled_per_user(self, base_dir: Path, skills_root: Path, config): + integration_dir = base_dir / "integrations" / "skills" / "lark-cli" / "lark-doc" + integration_dir.mkdir(parents=True) + (integration_dir / "SKILL.md").write_text(_skill_content("lark-doc"), encoding="utf-8") + + with patch("deerflow.config.paths.get_paths", return_value=Paths(base_dir=base_dir)): + alice = UserScopedSkillStorage("alice", host_path=str(skills_root), app_config=config) + bob = UserScopedSkillStorage("bob", host_path=str(skills_root), app_config=config) + + alice.set_skill_enabled_state("lark-doc", False) + + alice_skill = next(skill for skill in alice.load_skills(enabled_only=False) if skill.name == "lark-doc") + bob_skill = next(skill for skill in bob.load_skills(enabled_only=False) if skill.name == "lark-doc") + assert alice_skill.category == SkillCategory.INTEGRATION + assert alice_skill.skill_file == integration_dir / "SKILL.md" + assert alice_skill.enabled is False + assert bob_skill.enabled is True + def test_public_skill_package_children_are_not_registered(self, user_storage: UserScopedSkillStorage, skills_root: Path): public_dir = skills_root / "public" / "reviewer" fixture_dir = public_dir / "evals" / "fixtures" / "injection" diff --git a/docker/docker-compose-dev.yaml b/docker/docker-compose-dev.yaml index a623a32f3..8fb8b79fe 100644 --- a/docker/docker-compose-dev.yaml +++ b/docker/docker-compose-dev.yaml @@ -46,6 +46,10 @@ services: environment: - K8S_NAMESPACE=deer-flow - SANDBOX_IMAGE=enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest + # Optional lark-cli init image (Pattern A). Empty ⇒ legacy runtime mount. + # Set to a published tag (e.g. deer-flow/lark-cli-init:v1.0.65) to provision + # the sandbox lark-cli runtime via an init container + emptyDir. + - LARK_CLI_INIT_IMAGE=${LARK_CLI_INIT_IMAGE:-} # Host paths for K8s HostPath volumes (must be absolute paths accessible by K8s node) # On Docker Desktop/OrbStack, use your actual host paths like /Users/username/... # Set these in your shell before running docker-compose: @@ -146,6 +150,8 @@ services: APT_MIRROR: ${APT_MIRROR:-} UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:0.7.20} UV_INDEX_URL: ${UV_INDEX_URL:-https://pypi.org/simple} + NPM_REGISTRY: ${NPM_REGISTRY:-} + LARK_CLI_NPM_VERSION: ${LARK_CLI_NPM_VERSION:-1.0.65} container_name: deer-flow-gateway # Startup logic lives in docker/dev-entrypoint.sh — UV_EXTRAS validation, # `uv sync --all-packages`, .venv self-heal, and uvicorn handoff. Keeps diff --git a/docker/docker-compose.yaml b/docker/docker-compose.yaml index 32b07f7a7..2e33c49e2 100644 --- a/docker/docker-compose.yaml +++ b/docker/docker-compose.yaml @@ -88,6 +88,8 @@ services: UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:0.7.20} UV_INDEX_URL: ${UV_INDEX_URL:-https://pypi.org/simple} UV_EXTRAS: ${UV_EXTRAS:-} + NPM_REGISTRY: ${NPM_REGISTRY:-} + LARK_CLI_NPM_VERSION: ${LARK_CLI_NPM_VERSION:-1.0.65} container_name: deer-flow-gateway # Gateway hosts the agent runtime with in-process RunManager + StreamBridge # singletons -- run state lives in this worker's memory. Default to a single @@ -156,6 +158,10 @@ services: environment: - K8S_NAMESPACE=deer-flow - SANDBOX_IMAGE=enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest + # Optional lark-cli init image (Pattern A). Empty ⇒ legacy runtime mount. + # Set to a published tag (e.g. deer-flow/lark-cli-init:v1.0.65) to provision + # the sandbox lark-cli runtime via an init container + emptyDir. + - LARK_CLI_INIT_IMAGE=${LARK_CLI_INIT_IMAGE:-} - SKILLS_HOST_PATH=${DEER_FLOW_REPO_ROOT}/skills - THREADS_HOST_PATH=${DEER_FLOW_HOME}/threads - DEER_FLOW_HOST_BASE_DIR=${DEER_FLOW_HOME} diff --git a/docker/lark-cli-init/Dockerfile b/docker/lark-cli-init/Dockerfile new file mode 100644 index 000000000..25941df6e --- /dev/null +++ b/docker/lark-cli-init/Dockerfile @@ -0,0 +1,63 @@ +# syntax=docker/dockerfile:1 +# +# DeerFlow "lark-cli init" image (Pattern A). +# +# Purpose: provide the sandbox `lark-cli` runtime binary to a Kubernetes sandbox +# Pod via an init container + shared `emptyDir`, instead of downloading Linux +# binaries from GitHub at install time and mounting them via hostPath/PVC. +# +# At BUILD time (network available) this image downloads and SHA-256-verifies the +# official `larksuite/cli` Linux release binaries and stages the runtime layout: +# +# /opt/lark-cli/bin/lark-cli # arch-dispatch launcher (uname -m) +# /opt/lark-cli/linux-amd64/lark-cli +# /opt/lark-cli/linux-arm64/lark-cli +# /opt/lark-cli/.deerflow-lark-cli-runtime.json # {"version": "vX.Y.Z"} +# +# This layout is byte-identical to what the Gateway writer +# (`_write_lark_cli_sandbox_launcher`) produces and what +# `_validate_lark_cli_sandbox_runtime` enforces, so the sandbox PATH contract +# (`/mnt/integrations/lark-cli/runtime/bin/lark-cli`) is unchanged. +# +# At RUN time the default command copies `/opt/lark-cli/.` into the shared +# emptyDir mounted at `/mnt/integrations/lark-cli/runtime` and exits. +# +# The image tag should encode the lark-cli version so it can be bumped +# independently of the upstream `all-in-one-sandbox` image. +# +# Build (defaults to the pinned version below): +# docker build -t deer-flow/lark-cli-init:v1.0.65 \ +# --build-arg LARK_CLI_VERSION=v1.0.65 docker/lark-cli-init + +FROM debian:bookworm-slim AS builder + +ARG APT_MIRROR +# Version tag on the larksuite/cli GitHub releases (with or without leading "v"). +ARG LARK_CLI_VERSION=v1.0.65 + +RUN if [ -n "${APT_MIRROR}" ]; then \ + sed -i "s|deb.debian.org|${APT_MIRROR}|g" /etc/apt/sources.list.d/debian.sources 2>/dev/null || true; \ + sed -i "s|deb.debian.org|${APT_MIRROR}|g" /etc/apt/sources.list 2>/dev/null || true; \ + fi + +RUN apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates \ + curl \ + && rm -rf /var/lib/apt/lists/* + +COPY build-runtime.sh /usr/local/bin/build-runtime.sh +RUN chmod +x /usr/local/bin/build-runtime.sh \ + && LARK_CLI_VERSION="${LARK_CLI_VERSION}" /usr/local/bin/build-runtime.sh /opt/lark-cli + +# ── Final image: minimal, just the staged runtime + a copy entrypoint ──────── +FROM debian:bookworm-slim + +COPY --from=builder /opt/lark-cli /opt/lark-cli +COPY entrypoint.sh /usr/local/bin/lark-cli-init +RUN chmod +x /usr/local/bin/lark-cli-init + +# Destination is the emptyDir the provisioner mounts into both this init +# container and the main sandbox container. +ENV LARK_CLI_RUNTIME_DEST=/mnt/integrations/lark-cli/runtime + +ENTRYPOINT ["/usr/local/bin/lark-cli-init"] diff --git a/docker/lark-cli-init/README.md b/docker/lark-cli-init/README.md new file mode 100644 index 000000000..9d958022e --- /dev/null +++ b/docker/lark-cli-init/README.md @@ -0,0 +1,63 @@ +# lark-cli init image (Pattern A) + +This image provisions the sandbox `lark-cli` runtime binary into a Kubernetes +sandbox Pod via an **init container + shared `emptyDir`**, instead of the Gateway +downloading Linux binaries from GitHub at install time and mounting them via +hostPath/PVC. + +See the design at +[`docs/superpowers/specs/2026-07-21-lark-sandbox-init-container-design.md`](../../docs/superpowers/specs/2026-07-21-lark-sandbox-init-container-design.md). + +## What it does + +- **Build time** (network available): downloads and SHA-256-verifies the official + `larksuite/cli` Linux release binaries and stages the runtime layout under + `/opt/lark-cli`: + + ``` + /opt/lark-cli/bin/lark-cli # arch-dispatch launcher (uname -m) + /opt/lark-cli/linux-amd64/lark-cli + /opt/lark-cli/linux-arm64/lark-cli + /opt/lark-cli/.deerflow-lark-cli-runtime.json # {"version": "vX.Y.Z"} + ``` + + This is byte-identical to what the Gateway writer + (`_write_lark_cli_sandbox_launcher`) produces and what + `_validate_lark_cli_sandbox_runtime` enforces, so the sandbox PATH contract + (`/mnt/integrations/lark-cli/runtime/bin/lark-cli`) is unchanged. + +- **Run time**: copies `/opt/lark-cli/.` into the emptyDir mounted at + `${LARK_CLI_RUNTIME_DEST}` (default `/mnt/integrations/lark-cli/runtime`) and + exits `0`. + +## Build + +```bash +docker build -t deer-flow/lark-cli-init:v1.0.65 \ + --build-arg LARK_CLI_VERSION=v1.0.65 \ + docker/lark-cli-init +``` + +The tag should encode the lark-cli version so it can be bumped independently of +the upstream `all-in-one-sandbox` sandbox image. + +## Wiring it into the provisioner + +The init-container runtime path is **opt-in** and off by default. Enable it by +publishing this image and pointing the provisioner at it: + +- Set `LARK_CLI_INIT_IMAGE` on the provisioner service to the published tag + (e.g. `deer-flow/lark-cli-init:v1.0.65`). Empty ⇒ legacy hostPath / Gateway + download path (no behavior change). +- When set, the provisioner adds a `lark-cli-runtime` `emptyDir` volume, an + `lark-cli-init` init container, and a read-only runtime mount on the sandbox + container — and ignores any `/mnt/integrations/lark-cli/runtime` hostPath/PVC + extra mount (the init container supersedes it). The per-user `config` / `data` + credential mounts are unchanged. +- The provisioner reports whether it is configured via `GET /api/capabilities` + (`{"lark_cli_init_image": true|false}`), which the Gateway surfaces as the + Lark integration sandbox-runtime readiness signal in `/api/integrations/lark/status`. + +> Publishing note: this repository currently ships only backend/frontend images. +> Publishing a `lark-cli-init` tag is a fast-follow; until then the feature stays +> behind the empty-default `LARK_CLI_INIT_IMAGE`. diff --git a/docker/lark-cli-init/build-runtime.sh b/docker/lark-cli-init/build-runtime.sh new file mode 100644 index 000000000..62090a36d --- /dev/null +++ b/docker/lark-cli-init/build-runtime.sh @@ -0,0 +1,87 @@ +#!/bin/sh +# Stage the DeerFlow sandbox lark-cli runtime layout from official release +# binaries. Runs at image BUILD time (network available). +# +# Usage: LARK_CLI_VERSION=v1.0.65 build-runtime.sh /opt/lark-cli +# +# Produces: +# /bin/lark-cli arch-dispatch launcher (uname -m) +# /linux-amd64/lark-cli +# /linux-arm64/lark-cli +# /.deerflow-lark-cli-runtime.json {"version": "vX.Y.Z"} +# +# The layout mirrors the Gateway writer (_write_lark_cli_sandbox_launcher) and +# satisfies _validate_lark_cli_sandbox_runtime, so the sandbox PATH contract is +# unchanged. +set -eu + +DEST="${1:?destination directory required}" +VERSION="${LARK_CLI_VERSION:?LARK_CLI_VERSION required}" +REPO="larksuite/cli" +ARCHES="amd64 arm64" + +# Normalize to a leading-v tag and a bare version. +case "$VERSION" in + v*) TAG="$VERSION" ;; + *) TAG="v$VERSION" ;; +esac +BARE="${TAG#v}" + +BASE_URL="https://github.com/${REPO}/releases/download/${TAG}" +WORK="$(mktemp -d)" +trap 'rm -rf "$WORK"' EXIT + +echo "Downloading lark-cli ${TAG} release checksums..." +curl -fsSL "${BASE_URL}/checksums.txt" -o "${WORK}/checksums.txt" + +mkdir -p "${DEST}/bin" + +for arch in $ARCHES; do + asset="lark-cli-${BARE}-linux-${arch}.tar.gz" + echo "Downloading ${asset}..." + curl -fsSL "${BASE_URL}/${asset}" -o "${WORK}/${asset}" + + expected="$(awk -v f="${asset}" '$2 == f || $2 == "*"f {print $1}' "${WORK}/checksums.txt" | head -n1)" + if [ -z "$expected" ]; then + echo "ERROR: no checksum for ${asset} in checksums.txt" >&2 + exit 1 + fi + actual="$(sha256sum "${WORK}/${asset}" | awk '{print $1}')" + if [ "$expected" != "$actual" ]; then + echo "ERROR: checksum mismatch for ${asset} (expected ${expected}, got ${actual})" >&2 + exit 1 + fi + + mkdir -p "${DEST}/linux-${arch}" + # Extract only the single lark-cli executable from the archive. + tar -xzf "${WORK}/${asset}" -C "${WORK}" + found="$(find "${WORK}" -type f -name lark-cli ! -path "${DEST}/*" | head -n1)" + if [ -z "$found" ]; then + echo "ERROR: lark-cli executable not found in ${asset}" >&2 + exit 1 + fi + install -m 0755 "$found" "${DEST}/linux-${arch}/lark-cli" + rm -f "$found" +done + +# Arch-dispatch launcher. Kept byte-identical to +# LARK_CLI_SANDBOX_LAUNCHER_SCRIPT in +# backend/packages/harness/deerflow/integrations/lark_cli.py +# (a unit test asserts the two never drift). +cat > "${DEST}/bin/lark-cli" <<'LAUNCHER' +#!/bin/sh +set -eu +case "$(uname -m)" in + x86_64|amd64) arch=amd64 ;; + aarch64|arm64) arch=arm64 ;; + *) echo "Unsupported sandbox architecture: $(uname -m)" >&2; exit 126 ;; +esac +script_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +exec "$script_dir/../linux-$arch/lark-cli" "$@" +LAUNCHER +chmod 0755 "${DEST}/bin/lark-cli" + +printf '{\n "version": "%s"\n}\n' "$TAG" > "${DEST}/.deerflow-lark-cli-runtime.json" + +echo "Staged lark-cli ${TAG} runtime at ${DEST}:" +ls -R "${DEST}" diff --git a/docker/lark-cli-init/entrypoint.sh b/docker/lark-cli-init/entrypoint.sh new file mode 100644 index 000000000..1a1bc9166 --- /dev/null +++ b/docker/lark-cli-init/entrypoint.sh @@ -0,0 +1,24 @@ +#!/bin/sh +# DeerFlow lark-cli init container entrypoint (Pattern A). +# +# Copies the staged sandbox runtime layout into the shared emptyDir the +# provisioner mounts into both this init container and the main sandbox +# container, then exits. The sandbox container then finds the launcher at +# ${LARK_CLI_RUNTIME_DEST}/bin/lark-cli — exactly where +# lark_cli_env_overlay(sandbox_paths=True) points PATH. +set -eu + +SRC="/opt/lark-cli" +DEST="${LARK_CLI_RUNTIME_DEST:-/mnt/integrations/lark-cli/runtime}" + +if [ ! -x "${SRC}/bin/lark-cli" ]; then + echo "ERROR: staged runtime missing at ${SRC}/bin/lark-cli" >&2 + exit 1 +fi + +mkdir -p "${DEST}" +# Copy contents (including the dotfile manifest) into the destination. +cp -a "${SRC}/." "${DEST}/" + +echo "Provisioned lark-cli runtime into ${DEST}:" +ls -R "${DEST}" diff --git a/docker/provisioner/README.md b/docker/provisioner/README.md index 3f2875b50..455717ac9 100644 --- a/docker/provisioner/README.md +++ b/docker/provisioner/README.md @@ -145,6 +145,7 @@ The provisioner is configured via environment variables (set in [docker-compose- |----------|---------|-------------| | `K8S_NAMESPACE` | `deer-flow` | Kubernetes namespace for sandbox resources | | `SANDBOX_IMAGE` | `enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest` | AIO-compatible container image for sandbox Pods | +| `LARK_CLI_INIT_IMAGE` | empty (feature off) | Optional lark-cli init image (Pattern A). When set, sandbox Pods requesting the lark-cli runtime get an init container + shared `emptyDir` that provisions `lark-cli`, instead of a hostPath/PVC runtime mount. See [`docker/lark-cli-init`](../lark-cli-init/README.md) | | `SKILLS_HOST_PATH` | - | **Host machine** path to skills directory (must be absolute) | | `THREADS_HOST_PATH` | - | **Host machine** path to threads data directory (must be absolute) | | `SKILLS_PVC_NAME` | empty (use hostPath) | PVC name for skills volume; when set, sandbox Pods use PVC instead of hostPath | @@ -163,6 +164,29 @@ For persistent dependencies, build an image that extends the default `all-in-one See [Building a Custom AIO Sandbox Image](../../backend/docs/CONFIGURATION.md#building-a-custom-aio-sandbox-image) for the runtime contract and a minimal Dockerfile example. +### Lark CLI sandbox runtime (Pattern A) + +Agents run `lark-cli` **inside** the sandbox, so the binary must exist in the +sandbox container. Instead of the Gateway downloading Linux binaries from GitHub +at install time and mounting them via hostPath/PVC, the provisioner can inject +`lark-cli` with an **init container + shared `emptyDir`**: + +1. Publish a lark-cli init image (see [`docker/lark-cli-init`](../lark-cli-init/README.md)) + and set `LARK_CLI_INIT_IMAGE` on the provisioner. +2. When a sandbox is created with `provision_lark_cli_runtime: true` (the Gateway + sends this automatically once the managed Lark skill pack is installed), the + Pod gets a `lark-cli-runtime` `emptyDir`, an `lark-cli-init` init container + that copies the runtime into it, and a read-only runtime mount on the sandbox + container at `/mnt/integrations/lark-cli/runtime`. Any hostPath/PVC extra + mount at that path is dropped (the init container supersedes it); the per-user + `config`/`data` credential mounts are unchanged. + +`GET /api/capabilities` returns `{"lark_cli_init_image": true|false}` so the +Gateway can surface a sandbox-runtime readiness signal in +`/api/integrations/lark/status` — a green UI can't then hide a chat-time +`lark-cli: command not found`. + + ### PVC User-Data Upgrade Note Older provisioner versions mounted PVC user-data from `threads/{thread_id}/user-data`. The user-scoped layout mounts from `deer-flow/users/{user_id}/threads/{thread_id}/user-data`. diff --git a/docker/provisioner/app.py b/docker/provisioner/app.py index 8f3dfc83e..6796736b8 100644 --- a/docker/provisioner/app.py +++ b/docker/provisioner/app.py @@ -31,6 +31,7 @@ from __future__ import annotations import logging import os +import posixpath import re import secrets import time @@ -59,6 +60,13 @@ SANDBOX_IMAGE = os.environ.get( "SANDBOX_IMAGE", "enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest", ) +# Optional "lark-cli init" image (Pattern A). When set, sandbox Pods get an init +# container + shared emptyDir that provisions the lark-cli runtime binary, instead +# of a hostPath/PVC runtime mount fed by a Gateway-side GitHub download. Empty ⇒ +# feature off (legacy behavior). +LARK_CLI_INIT_IMAGE = os.environ.get("LARK_CLI_INIT_IMAGE", "") +LARK_CLI_RUNTIME_CONTAINER_PATH = "/mnt/integrations/lark-cli/runtime" +LARK_CLI_RUNTIME_VOLUME_NAME = "lark-cli-runtime" SKILLS_HOST_PATH = os.environ.get("SKILLS_HOST_PATH", "/skills") THREADS_HOST_PATH = os.environ.get("THREADS_HOST_PATH", "/.deer-flow/threads") DEER_FLOW_HOST_BASE_DIR = os.environ.get("DEER_FLOW_HOST_BASE_DIR", "/.deer-flow") @@ -78,6 +86,15 @@ if SANDBOX_SERVICE_TYPE not in {"NodePort", "ClusterIP"}: SAFE_THREAD_ID_PATTERN = r"^[A-Za-z0-9_\-]+$" SAFE_USER_ID_PATTERN = r"^[A-Za-z0-9_\-]+$" DEFAULT_USER_ID = "default" +MAX_EXTRA_MOUNTS = 9 +ALLOWED_EXTRA_MOUNT_PATHS = { + "/mnt/acp-workspace", + "/mnt/skills/custom", + "/mnt/skills/integrations", + "/mnt/integrations/lark-cli/config", + "/mnt/integrations/lark-cli/data", + "/mnt/integrations/lark-cli/runtime", +} # Path to the kubeconfig *inside* the provisioner container. # Typically the host's ~/.kube/config is mounted here. @@ -111,6 +128,109 @@ def join_host_path(base: str, *parts: str) -> str: return str(result) +def _host_base_dir_for_extra_mounts() -> str: + """Return the host-visible DeerFlow state root used for controlled mounts.""" + if DEER_FLOW_HOST_BASE_DIR: + return os.path.normpath(DEER_FLOW_HOST_BASE_DIR) + + normalized_threads = os.path.normpath(THREADS_HOST_PATH) + if os.path.basename(normalized_threads) == "threads": + return os.path.dirname(normalized_threads) + return "" + + +def _is_path_under_base(path: str, base: str) -> bool: + """Return whether *path* is inside *base* after normalization.""" + if not base: + return False + try: + return os.path.commonpath([os.path.normpath(path), os.path.normpath(base)]) == os.path.normpath(base) + except ValueError: + return False + + +def _normalize_extra_mount_container_path(container_path: str) -> str: + normalized = posixpath.normpath(container_path) + if not normalized.startswith("/"): + raise HTTPException(status_code=400, detail=f"Extra mount path must be absolute: {container_path}") + if normalized not in ALLOWED_EXTRA_MOUNT_PATHS: + raise HTTPException(status_code=400, detail=f"Unsupported extra mount path: {container_path}") + return normalized + + +def _validated_extra_mounts(extra_mounts: list["ExtraMount"] | None) -> list["ExtraMount"]: + """Validate extra mounts before converting them into K8s hostPath/PVC mounts.""" + if not extra_mounts: + return [] + if len(extra_mounts) > MAX_EXTRA_MOUNTS: + raise HTTPException(status_code=400, detail=f"Too many extra mounts; max is {MAX_EXTRA_MOUNTS}") + + host_base_dir = _host_base_dir_for_extra_mounts() + seen_container_paths: set[str] = set() + validated: list[ExtraMount] = [] + for mount in extra_mounts: + host_path = os.path.normpath(mount.host_path) + if not os.path.isabs(host_path): + raise HTTPException(status_code=400, detail=f"Extra mount host path must be absolute: {mount.host_path}") + if not _is_path_under_base(host_path, host_base_dir): + raise HTTPException(status_code=400, detail=f"Extra mount host path is outside DeerFlow state: {mount.host_path}") + + container_path = _normalize_extra_mount_container_path(mount.container_path) + if container_path in seen_container_paths: + raise HTTPException(status_code=400, detail=f"Duplicate extra mount path: {container_path}") + seen_container_paths.add(container_path) + + validated.append( + ExtraMount( + host_path=host_path, + container_path=container_path, + read_only=mount.read_only, + ) + ) + return validated + + +def _extra_mount_volume_name(index: int) -> str: + return f"extra-{index}" + + +def _lark_cli_runtime_enabled(provision_lark_cli_runtime: bool) -> bool: + """Whether to provision the lark-cli runtime via init container + emptyDir.""" + return bool(LARK_CLI_INIT_IMAGE) and provision_lark_cli_runtime + + +def _runtime_provided_extra_mounts( + extra_mounts: list["ExtraMount"] | None, + *, + provision_lark_cli_runtime: bool, +) -> list["ExtraMount"]: + """Drop the lark-cli runtime extra mount when the init container supersedes it. + + The init container + emptyDir provides ``/mnt/integrations/lark-cli/runtime``, + so a hostPath/PVC mount at the same path would collide. The per-user + ``config`` / ``data`` credential mounts are left untouched. + """ + if not extra_mounts or not _lark_cli_runtime_enabled(provision_lark_cli_runtime): + return list(extra_mounts or []) + return [ + mount + for mount in extra_mounts + if posixpath.normpath(mount.container_path) != LARK_CLI_RUNTIME_CONTAINER_PATH + ] + + +def _extra_mount_pvc_sub_path(host_path: str) -> str: + host_base_dir = _host_base_dir_for_extra_mounts() + if not _is_path_under_base(host_path, host_base_dir): + raise HTTPException(status_code=400, detail=f"Extra mount host path is outside DeerFlow state: {host_path}") + + rel_path = os.path.relpath(os.path.normpath(host_path), host_base_dir) + rel_parts = [part for part in rel_path.replace(os.sep, "/").split("/") if part and part != "."] + if not rel_parts or any(part == ".." for part in rel_parts): + raise HTTPException(status_code=400, detail=f"Invalid extra mount host path: {host_path}") + return posixpath.join("deer-flow", *rel_parts) + + # ── K8s client setup ──────────────────────────────────────────────────── core_v1: k8s_client.CoreV1Api | None = None @@ -219,11 +339,22 @@ async def verify_api_key(request: Request, call_next): # ── Request / Response models ─────────────────────────────────────────── +class ExtraMount(BaseModel): + host_path: str + container_path: str + read_only: bool = False + + class CreateSandboxRequest(BaseModel): sandbox_id: str - thread_id: str = Field(pattern=SAFE_THREAD_ID_PATTERN) + thread_id: str | None = Field(default=None, pattern=SAFE_THREAD_ID_PATTERN) user_id: str = Field(default=DEFAULT_USER_ID, pattern=SAFE_USER_ID_PATTERN) + extra_mounts: list[ExtraMount] = Field(default_factory=list) include_legacy_skills: bool = False + # When true (and LARK_CLI_INIT_IMAGE is configured), provision the sandbox + # lark-cli runtime via an init container + emptyDir instead of a runtime + # hostPath/PVC extra mount. + provision_lark_cli_runtime: bool = False class SandboxResponse(BaseModel): @@ -252,11 +383,53 @@ def _sandbox_url(sandbox_id: str, node_port: int | None = None) -> str: return f"http://{NODE_HOST}:{node_port}" +def _build_extra_volumes(extra_mounts: list[ExtraMount] | None = None) -> list[k8s_client.V1Volume]: + volumes: list[k8s_client.V1Volume] = [] + for index, mount in enumerate(_validated_extra_mounts(extra_mounts)): + if USERDATA_PVC_NAME: + volumes.append( + k8s_client.V1Volume( + name=_extra_mount_volume_name(index), + persistent_volume_claim=k8s_client.V1PersistentVolumeClaimVolumeSource( + claim_name=USERDATA_PVC_NAME, + ), + ) + ) + continue + + volumes.append( + k8s_client.V1Volume( + name=_extra_mount_volume_name(index), + host_path=k8s_client.V1HostPathVolumeSource( + path=mount.host_path, + type="Directory" if mount.read_only else "DirectoryOrCreate", + ), + ) + ) + return volumes + + +def _build_extra_volume_mounts(extra_mounts: list[ExtraMount] | None = None) -> list[k8s_client.V1VolumeMount]: + mounts: list[k8s_client.V1VolumeMount] = [] + for index, mount in enumerate(_validated_extra_mounts(extra_mounts)): + volume_mount = k8s_client.V1VolumeMount( + name=_extra_mount_volume_name(index), + mount_path=mount.container_path, + read_only=mount.read_only, + ) + if USERDATA_PVC_NAME: + volume_mount.sub_path = _extra_mount_pvc_sub_path(mount.host_path) + mounts.append(volume_mount) + return mounts + + def _build_volumes( thread_id: str, user_id: str = DEFAULT_USER_ID, *, include_legacy_skills: bool = False, + extra_mounts: list[ExtraMount] | None = None, + provision_lark_cli_runtime: bool = False, ) -> list[k8s_client.V1Volume]: """Build volume list: PVC when configured, otherwise hostPath. @@ -343,6 +516,21 @@ def _build_volumes( ) volumes.append(userdata_vol) + volumes.extend( + _build_extra_volumes( + _runtime_provided_extra_mounts( + extra_mounts, + provision_lark_cli_runtime=provision_lark_cli_runtime, + ) + ) + ) + if _lark_cli_runtime_enabled(provision_lark_cli_runtime): + volumes.append( + k8s_client.V1Volume( + name=LARK_CLI_RUNTIME_VOLUME_NAME, + empty_dir=k8s_client.V1EmptyDirVolumeSource(), + ) + ) return volumes @@ -351,6 +539,8 @@ def _build_volume_mounts( user_id: str = DEFAULT_USER_ID, *, include_legacy_skills: bool = False, + extra_mounts: list[ExtraMount] | None = None, + provision_lark_cli_runtime: bool = False, ) -> list[k8s_client.V1VolumeMount]: """Build volume mount list, mirroring three-way skills layout. @@ -405,18 +595,71 @@ def _build_volume_mounts( if USERDATA_PVC_NAME: userdata_mount.sub_path = f"deer-flow/users/{user_id}/threads/{thread_id}/user-data" mounts.append(userdata_mount) + mounts.extend( + _build_extra_volume_mounts( + _runtime_provided_extra_mounts( + extra_mounts, + provision_lark_cli_runtime=provision_lark_cli_runtime, + ) + ) + ) + if _lark_cli_runtime_enabled(provision_lark_cli_runtime): + mounts.append( + k8s_client.V1VolumeMount( + name=LARK_CLI_RUNTIME_VOLUME_NAME, + mount_path=LARK_CLI_RUNTIME_CONTAINER_PATH, + read_only=True, + ) + ) return mounts +def _build_lark_cli_init_containers( + provision_lark_cli_runtime: bool, +) -> list[k8s_client.V1Container]: + """Init container that copies the lark-cli runtime into the shared emptyDir.""" + if not _lark_cli_runtime_enabled(provision_lark_cli_runtime): + return [] + return [ + k8s_client.V1Container( + name="lark-cli-init", + image=LARK_CLI_INIT_IMAGE, + image_pull_policy="IfNotPresent", + env=[ + k8s_client.V1EnvVar( + name="LARK_CLI_RUNTIME_DEST", + value=LARK_CLI_RUNTIME_CONTAINER_PATH, + ) + ], + volume_mounts=[ + k8s_client.V1VolumeMount( + name=LARK_CLI_RUNTIME_VOLUME_NAME, + mount_path=LARK_CLI_RUNTIME_CONTAINER_PATH, + read_only=False, + ) + ], + security_context=k8s_client.V1SecurityContext( + privileged=False, + allow_privilege_escalation=False, + ), + ) + ] + + def _build_pod( sandbox_id: str, thread_id: str, user_id: str = DEFAULT_USER_ID, *, include_legacy_skills: bool = False, + extra_mounts: list[ExtraMount] | None = None, + provision_lark_cli_runtime: bool = False, ) -> k8s_client.V1Pod: """Construct a Pod manifest for a single sandbox.""" + init_containers = ( + _build_lark_cli_init_containers(provision_lark_cli_runtime) or None + ) return k8s_client.V1Pod( metadata=k8s_client.V1ObjectMeta( name=_pod_name(sandbox_id), @@ -477,6 +720,8 @@ def _build_pod( thread_id, user_id=user_id, include_legacy_skills=include_legacy_skills, + extra_mounts=extra_mounts, + provision_lark_cli_runtime=provision_lark_cli_runtime, ), security_context=k8s_client.V1SecurityContext( privileged=False, @@ -484,10 +729,13 @@ def _build_pod( ), ) ], + init_containers=init_containers, volumes=_build_volumes( thread_id, user_id=user_id, include_legacy_skills=include_legacy_skills, + extra_mounts=extra_mounts, + provision_lark_cli_runtime=provision_lark_cli_runtime, ), restart_policy="Always", ), @@ -573,6 +821,17 @@ async def health(): return {"status": "ok"} +@app.get("/api/capabilities") +async def capabilities(): + """Report provisioner-side capabilities the Gateway cannot infer statically. + + ``lark_cli_init_image`` reflects whether a lark-cli init image is configured, + which the Gateway surfaces as the Lark integration sandbox-runtime readiness + signal so a green UI can't hide a chat-time ``command not found``. + """ + return {"lark_cli_init_image": bool(LARK_CLI_INIT_IMAGE)} + + @app.post("/api/sandboxes", response_model=SandboxResponse) def create_sandbox(req: CreateSandboxRequest): """Create a sandbox Pod + Service for *sandbox_id*. @@ -581,16 +840,18 @@ def create_sandbox(req: CreateSandboxRequest): (idempotent). """ sandbox_id = req.sandbox_id - thread_id = req.thread_id + thread_id = req.thread_id or sandbox_id user_id = req.user_id include_legacy_skills = req.include_legacy_skills + provision_lark_cli_runtime = req.provision_lark_cli_runtime logger.info( - "Received request to create sandbox '%s' for thread '%s' user '%s' include_legacy_skills=%s", + "Received request to create sandbox '%s' for thread '%s' user '%s' include_legacy_skills=%s provision_lark_cli_runtime=%s", sandbox_id, thread_id, user_id, include_legacy_skills, + _lark_cli_runtime_enabled(provision_lark_cli_runtime), ) # ── Fast path: sandbox already exists ──────────────────────────── @@ -611,6 +872,8 @@ def create_sandbox(req: CreateSandboxRequest): thread_id, user_id=user_id, include_legacy_skills=include_legacy_skills, + extra_mounts=req.extra_mounts, + provision_lark_cli_runtime=provision_lark_cli_runtime, ), ) logger.info(f"Created Pod {_pod_name(sandbox_id)}") diff --git a/frontend/AGENTS.md b/frontend/AGENTS.md index ca6c2ca3c..feaf46c79 100644 --- a/frontend/AGENTS.md +++ b/frontend/AGENTS.md @@ -55,7 +55,7 @@ The frontend is a stateful chat application. Users create **threads** (conversat - `workspace/` — Chat page components (messages, artifacts, settings) - `landing/` — Landing page sections - `docs/` — Docs / MDX rendering components -- **`core/`** — Business logic, the heart of the app. Domains include `threads/` (creation, streaming, state), `api/` (LangGraph client singleton), `agents/` (custom agents), `auth/` (authentication), `artifacts/`, `channels/` (IM connections), `i18n/` (en-US, zh-CN), `settings/`, `memory/`, `skills/`, `messages/`, `mcp/`, `models/`, `input-polish/` (pre-send draft rewrite API), `voice-input/` (browser speech-recognition helpers), `suggestions/`, `tasks/`, `todos/`, `tools/`, `workspace-changes/` (run-scoped changed-file summaries and diff fetching), `config/`, `notification/`, `blog/`, plus rendering helpers (`rehype/`, `streamdown/`) and `utils/`. +- **`core/`** — Business logic, the heart of the app. Domains include `threads/` (creation, streaming, state), `api/` (LangGraph client singleton), `agents/` (custom agents), `auth/` (authentication), `artifacts/`, `channels/` (IM connections), `integrations/` (managed third-party integration status/install clients such as Lark CLI), `i18n/` (en-US, zh-CN), `settings/`, `memory/`, `skills/`, `messages/`, `mcp/`, `models/`, `input-polish/` (pre-send draft rewrite API), `voice-input/` (browser speech-recognition helpers), `suggestions/`, `tasks/`, `todos/`, `tools/`, `workspace-changes/` (run-scoped changed-file summaries and diff fetching), `config/`, `notification/`, `blog/`, plus rendering helpers (`rehype/`, `streamdown/`) and `utils/`. - **`hooks/`** — Shared React hooks - **`lib/`** — Utilities (`cn()` from clsx + tailwind-merge) - **`content/`** — MDX content (blog posts, docs) rendered by the app diff --git a/frontend/src/app/mock/api/integrations/lark/auth/complete/route.ts b/frontend/src/app/mock/api/integrations/lark/auth/complete/route.ts new file mode 100644 index 000000000..b046c4ddd --- /dev/null +++ b/frontend/src/app/mock/api/integrations/lark/auth/complete/route.ts @@ -0,0 +1,36 @@ +export function POST() { + return Response.json({ + success: true, + message: "Lark/Feishu authorization completed.", + status: { + installed: true, + version: "v1.0.65", + manifest_version: "v1.0.65", + latest_available_version: "v1.0.65", + runtime_version_mismatch: false, + app_configured: true, + app_id: "cli_mock", + app_brand: "feishu", + skills_expected: 27, + skills_installed: 4, + installed_skills: ["lark-doc", "lark-im", "lark-shared", "lark-sheets"], + enabled_skills: ["lark-doc", "lark-im", "lark-shared", "lark-sheets"], + install_path: "/mock/integrations/skills/lark-cli", + cli: { + available: true, + path: "/usr/bin/lark-cli", + version: "lark-cli version v1.0.65", + error: null, + }, + auth: { + status: "authenticated", + message: "Lark authorization is live-verified.", + user: "Alice", + verified: true, + }, + sandbox_runtime_mode: "init-container", + sandbox_runtime_ready: true, + sandbox_runtime_detail: null, + }, + }); +} diff --git a/frontend/src/app/mock/api/integrations/lark/auth/start/route.ts b/frontend/src/app/mock/api/integrations/lark/auth/start/route.ts new file mode 100644 index 000000000..d798f82fe --- /dev/null +++ b/frontend/src/app/mock/api/integrations/lark/auth/start/route.ts @@ -0,0 +1,9 @@ +export function POST() { + return Response.json({ + verification_url: "https://open.feishu.cn/auth/mock-device", + device_code: "mock-device-code", + expires_in: 600, + user_code: null, + hint: null, + }); +} diff --git a/frontend/src/app/mock/api/integrations/lark/config/complete/route.ts b/frontend/src/app/mock/api/integrations/lark/config/complete/route.ts new file mode 100644 index 000000000..45dd0b98b --- /dev/null +++ b/frontend/src/app/mock/api/integrations/lark/config/complete/route.ts @@ -0,0 +1,36 @@ +export function POST() { + return Response.json({ + success: true, + message: "Lark/Feishu connection setup completed.", + status: { + installed: true, + version: "v1.0.65", + manifest_version: "v1.0.65", + latest_available_version: "v1.0.65", + runtime_version_mismatch: false, + app_configured: true, + app_id: "cli_mock", + app_brand: "feishu", + skills_expected: 27, + skills_installed: 4, + installed_skills: ["lark-doc", "lark-im", "lark-shared", "lark-sheets"], + enabled_skills: ["lark-doc", "lark-im", "lark-shared", "lark-sheets"], + install_path: "/mock/integrations/skills/lark-cli", + cli: { + available: true, + path: "/usr/bin/lark-cli", + version: "lark-cli version v1.0.65", + error: null, + }, + auth: { + status: "not_authorized", + message: "Lark user authorization is not configured", + user: null, + verified: false, + }, + sandbox_runtime_mode: "none", + sandbox_runtime_ready: false, + sandbox_runtime_detail: null, + }, + }); +} diff --git a/frontend/src/app/mock/api/integrations/lark/config/start/route.ts b/frontend/src/app/mock/api/integrations/lark/config/start/route.ts new file mode 100644 index 000000000..567e5ed85 --- /dev/null +++ b/frontend/src/app/mock/api/integrations/lark/config/start/route.ts @@ -0,0 +1,10 @@ +export function POST() { + return Response.json({ + verification_url: "https://open.feishu.cn/page/cli?user_code=config", + device_code: "mock-config-device-code", + expires_in: 600, + interval: 5, + user_code: "config", + brand: "feishu", + }); +} diff --git a/frontend/src/app/mock/api/integrations/lark/install/route.ts b/frontend/src/app/mock/api/integrations/lark/install/route.ts new file mode 100644 index 000000000..91e7e8dbc --- /dev/null +++ b/frontend/src/app/mock/api/integrations/lark/install/route.ts @@ -0,0 +1,39 @@ +const installedSkills = ["lark-doc", "lark-im", "lark-shared", "lark-sheets"]; + +export function POST() { + return Response.json({ + success: true, + installed_skills: installedSkills, + message: `Installed ${installedSkills.length} Lark/Feishu skills.`, + status: { + installed: true, + version: "v1.0.65", + manifest_version: "v1.0.65", + latest_available_version: "v1.0.65", + runtime_version_mismatch: false, + app_configured: false, + app_id: null, + app_brand: null, + skills_expected: 27, + skills_installed: installedSkills.length, + installed_skills: installedSkills, + enabled_skills: installedSkills, + install_path: "/mock/integrations/skills/lark-cli", + cli: { + available: true, + path: "/usr/bin/lark-cli", + version: "lark-cli version v1.0.65", + error: null, + }, + auth: { + status: "not_configured", + message: "lark-cli auth is not configured", + user: null, + verified: false, + }, + sandbox_runtime_mode: "none", + sandbox_runtime_ready: false, + sandbox_runtime_detail: null, + }, + }); +} diff --git a/frontend/src/app/mock/api/integrations/lark/status/route.ts b/frontend/src/app/mock/api/integrations/lark/status/route.ts new file mode 100644 index 000000000..c694379ec --- /dev/null +++ b/frontend/src/app/mock/api/integrations/lark/status/route.ts @@ -0,0 +1,34 @@ +const status = { + installed: false, + version: "v1.0.65", + manifest_version: null, + latest_available_version: "v1.0.65", + runtime_version_mismatch: false, + app_configured: false, + app_id: null, + app_brand: null, + skills_expected: 27, + skills_installed: 0, + installed_skills: [], + enabled_skills: [], + install_path: "/mock/integrations/skills/lark-cli", + cli: { + available: false, + path: null, + version: null, + error: "lark-cli is not on PATH", + }, + auth: { + status: "unavailable", + message: "lark-cli is not installed on the Gateway", + user: null, + verified: false, + }, + sandbox_runtime_mode: "none", + sandbox_runtime_ready: false, + sandbox_runtime_detail: null, +}; + +export function GET() { + return Response.json(status); +} diff --git a/frontend/src/app/workspace/workspace-content.tsx b/frontend/src/app/workspace/workspace-content.tsx index ded8c81ee..3e712eb29 100644 --- a/frontend/src/app/workspace/workspace-content.tsx +++ b/frontend/src/app/workspace/workspace-content.tsx @@ -5,6 +5,8 @@ import { QueryClientProvider } from "@/components/query-client-provider"; import { SidebarInset, SidebarProvider } from "@/components/ui/sidebar"; import { CommandPalette } from "@/components/workspace/command-palette"; import { GatewayOfflineBanner } from "@/components/workspace/gateway-offline-banner"; +import { SettingsDialogHost } from "@/components/workspace/settings"; +import { WorkspaceSettingsDeepLink } from "@/components/workspace/workspace-settings-deep-link"; import { WorkspaceSidebar } from "@/components/workspace/workspace-sidebar"; function parseSidebarOpenCookie( @@ -37,6 +39,8 @@ export async function WorkspaceContent({ + + ); diff --git a/frontend/src/components/workspace/command-palette.tsx b/frontend/src/components/workspace/command-palette.tsx index 1d754f61d..38bbdc305 100644 --- a/frontend/src/components/workspace/command-palette.tsx +++ b/frontend/src/components/workspace/command-palette.tsx @@ -27,14 +27,14 @@ import { import { useI18n } from "@/core/i18n/hooks"; import { useGlobalShortcuts } from "@/hooks/use-global-shortcuts"; -import { SettingsDialog } from "./settings"; +import { useSettingsDialog } from "./settings"; export function CommandPalette() { const { t } = useI18n(); const router = useRouter(); + const { openSettings } = useSettingsDialog(); const [open, setOpen] = useState(false); const [shortcutsOpen, setShortcutsOpen] = useState(false); - const [settingsOpen, setSettingsOpen] = useState(false); const [isMac, setIsMac] = useState(false); const handleNewChat = useCallback(() => { @@ -44,8 +44,8 @@ export function CommandPalette() { const handleOpenSettings = useCallback(() => { setOpen(false); - setSettingsOpen(true); - }, []); + openSettings("appearance"); + }, [openSettings]); const handleShowShortcuts = useCallback(() => { setOpen(false); @@ -72,7 +72,6 @@ export function CommandPalette() { return ( <> - diff --git a/frontend/src/components/workspace/settings/index.ts b/frontend/src/components/workspace/settings/index.ts index 658450689..861c89520 100644 --- a/frontend/src/components/workspace/settings/index.ts +++ b/frontend/src/components/workspace/settings/index.ts @@ -1 +1,7 @@ -export { SettingsDialog } from "./settings-dialog"; +export { SettingsDialog, type SettingsSection } from "./settings-dialog"; +export { SettingsDialogHost } from "./settings-dialog-host"; +export { + openSettingsDialog, + setSettingsDialogOpen, + useSettingsDialog, +} from "./settings-dialog-store"; diff --git a/frontend/src/components/workspace/settings/integrations-settings-page.tsx b/frontend/src/components/workspace/settings/integrations-settings-page.tsx new file mode 100644 index 000000000..6424108af --- /dev/null +++ b/frontend/src/components/workspace/settings/integrations-settings-page.tsx @@ -0,0 +1,882 @@ +"use client"; + +import { + CheckCircle2Icon, + CopyIcon, + ExternalLinkIcon, + PlugZapIcon, + RefreshCwIcon, + XCircleIcon, +} from "lucide-react"; +import { useEffect, useRef, useState } from "react"; +import { toast } from "sonner"; + +import { Alert, AlertDescription, AlertTitle } from "@/components/ui/alert"; +import { Badge } from "@/components/ui/badge"; +import { Button } from "@/components/ui/button"; +import { + Card, + CardAction, + CardContent, + CardDescription, + CardHeader, + CardTitle, +} from "@/components/ui/card"; +import { Input } from "@/components/ui/input"; +import { useAuth } from "@/core/auth/AuthProvider"; +import { useI18n } from "@/core/i18n/hooks"; +import { + LarkIntegrationRequestError, + type LarkAuthStartRequest, + type LarkAuthStartResponse, + type LarkConfigStartResponse, + type LarkIntegrationStatus, + useCompleteLarkAuthorization, + useCompleteLarkConfiguration, + useInstallLarkIntegration, + useLarkIntegrationStatus, + useStartLarkAuthorization, + useStartLarkConfiguration, +} from "@/core/integrations/lark"; +import { env } from "@/env"; +import { cn } from "@/lib/utils"; + +import { SettingsSection } from "./settings-section"; + +type PendingLarkFlow = + | ({ kind: "config" } & LarkConfigStartResponse) + | ({ kind: "auth" } & LarkAuthStartResponse); + +type LarkAuthDomain = + | "approval" + | "apps" + | "attendance" + | "base" + | "calendar" + | "contact" + | "docs" + | "drive" + | "event" + | "im" + | "mail" + | "markdown" + | "mindnotes" + | "minutes" + | "note" + | "okr" + | "sheets" + | "slides" + | "task" + | "vc" + | "wiki" + | "all"; + +// Mirrors `lark-cli auth login --domain` (available business domains + all). +const LARK_AUTH_DOMAINS: LarkAuthDomain[] = [ + "calendar", + "im", + "docs", + "drive", + "sheets", + "base", + "wiki", + "task", + "mail", + "vc", + "minutes", + "note", + "slides", + "markdown", + "mindnotes", + "contact", + "approval", + "attendance", + "okr", + "event", + "apps", + "all", +]; + +const AUTOMATIC_LARK_AUTH_WAIT_SECONDS = 8; + +function splitScopes(value: string) { + return value + .split(/[\s,]+/) + .map((scope) => scope.trim()) + .filter(Boolean); +} + +function uniqueScopes(scopes: string[]) { + return Array.from(new Set(scopes)); +} + +export function IntegrationsSettingsPage() { + const { t } = useI18n(); + return ( + + + + ); +} + +function LarkIntegrationCard() { + const { t } = useI18n(); + const { user } = useAuth(); + const isAdmin = user?.system_role === "admin"; + const { data, isLoading, error, refetch, isFetching } = + useLarkIntegrationStatus(); + const install = useInstallLarkIntegration(); + const startConfig = useStartLarkConfiguration(); + const completeConfig = useCompleteLarkConfiguration(); + const startAuth = useStartLarkAuthorization(); + const completeAuth = useCompleteLarkAuthorization(); + const [pendingFlow, setPendingFlow] = useState(null); + const [isCheckingConnection, setIsCheckingConnection] = useState(false); + const [selectedAuthDomains, setSelectedAuthDomains] = useState< + LarkAuthDomain[] + >([]); + const [customAuthScope, setCustomAuthScope] = useState(""); + const browserWindowRef = useRef(null); + const authRequestRef = useRef({ recommend: false }); + const authToastIdRef = useRef(null); + const authRetryTimeoutRef = useRef | null>( + null, + ); + const authAttemptIdRef = useRef(0); + const authDeadlineRef = useRef(0); + const isMountedRef = useRef(true); + const connectBusy = + startConfig.isPending || completeConfig.isPending || startAuth.isPending; + const connectActionBusy = connectBusy || isCheckingConnection; + const credentialsConfigured = data?.auth.status === "authenticated"; + const isConnected = credentialsConfigured && data?.auth.verified === true; + // The sandbox-runtime readiness row only applies when the sandbox actually + // runs lark-cli (AIO / provisioner modes report a non-"none" mode). + const showSandboxRuntime = !!data && data.sandbox_runtime_mode !== "none"; + const trimmedCustomAuthScope = customAuthScope.trim(); + const hasAdditionalPermissionRequest = + selectedAuthDomains.length > 0 || trimmedCustomAuthScope.length > 0; + + const handleInstall = () => { + install.mutate(undefined, { + onSuccess: (result) => toast.success(result.message), + onError: (err) => { + if (err instanceof LarkIntegrationRequestError && err.isAdminRequired) { + toast.error(t.settings.integrations.adminRequired); + return; + } + toast.error(err instanceof Error ? err.message : String(err)); + }, + }); + }; + + const clearAuthRetryTimer = () => { + if (authRetryTimeoutRef.current != null) { + clearTimeout(authRetryTimeoutRef.current); + authRetryTimeoutRef.current = null; + } + }; + + useEffect( + () => () => { + isMountedRef.current = false; + if (authRetryTimeoutRef.current != null) { + clearTimeout(authRetryTimeoutRef.current); + } + }, + [], + ); + + const openPendingBrowserWindow = () => { + const browserWindow = window.open("about:blank", "_blank"); + if (browserWindow) { + browserWindow.opener = null; + browserWindowRef.current = browserWindow; + } + return browserWindow; + }; + + const closePendingBrowserWindow = (browserWindow: Window | null) => { + if (!browserWindow) return; + browserWindow.close(); + if (browserWindowRef.current === browserWindow) { + browserWindowRef.current = null; + } + }; + + const openAuthorizationUrl = ( + url: string, + browserWindow = browserWindowRef.current, + ) => { + if (browserWindow && !browserWindow.closed) { + browserWindow.location.href = url; + browserWindowRef.current = browserWindow; + return; + } + browserWindowRef.current = window.open( + url, + "_blank", + "noopener,noreferrer", + ); + }; + + const startUserAuth = ( + browserWindow = browserWindowRef.current, + request = authRequestRef.current, + ) => { + startAuth.mutate(request, { + onSuccess: (result) => { + setPendingFlow({ kind: "auth", ...result }); + openAuthorizationUrl(result.verification_url, browserWindow); + authToastIdRef.current = toast.info( + t.settings.integrations.lark.authStarted, + ); + startAutomaticAuthorizationCheck(result); + }, + onError: (err) => { + closePendingBrowserWindow(browserWindow); + toast.error(err instanceof Error ? err.message : String(err)); + }, + }); + }; + + const handleContinueConnection = () => { + if (!pendingFlow || pendingFlow.kind !== "config") return; + completeConfig.mutate( + { + device_code: pendingFlow.device_code, + brand: pendingFlow.brand, + interval: pendingFlow.interval, + expires_in: pendingFlow.expires_in, + }, + { + onSuccess: () => { + toast.success(t.settings.integrations.lark.connectionReady); + setPendingFlow(null); + startUserAuth(browserWindowRef.current); + }, + onError: (err) => { + setPendingFlow(null); + toast.error(err instanceof Error ? err.message : String(err)); + }, + }, + ); + }; + + const startConnectionFlow = ( + status: LarkIntegrationStatus, + browserWindow: Window | null, + ) => { + if (!status.app_configured) { + startConfig.mutate( + { brand: "feishu" }, + { + onSuccess: (result) => { + setPendingFlow({ kind: "config", ...result }); + openAuthorizationUrl(result.verification_url, browserWindow); + toast.success(t.settings.integrations.lark.connectionStarted); + }, + onError: (err) => { + closePendingBrowserWindow(browserWindow); + toast.error(err instanceof Error ? err.message : String(err)); + }, + }, + ); + return; + } + startUserAuth(browserWindow); + }; + + const buildAuthRequest = (): LarkAuthStartRequest => { + const domains = selectedAuthDomains.includes("all") + ? ["all"] + : uniqueScopes(selectedAuthDomains); + const scopes = uniqueScopes(splitScopes(trimmedCustomAuthScope)); + return { + recommend: false, + domains, + scope: scopes.length > 0 ? scopes.join(" ") : null, + }; + }; + + const toggleAuthDomain = (domain: LarkAuthDomain) => { + setSelectedAuthDomains((current) => { + if (domain === "all") { + return current.includes("all") ? [] : ["all"]; + } + const withoutAll = current.filter((item) => item !== "all"); + if (withoutAll.includes(domain)) { + return withoutAll.filter((item) => item !== domain); + } + return [...withoutAll, domain]; + }); + }; + + const handleConnect = async () => { + if (!data) return; + authRequestRef.current = buildAuthRequest(); + // Pre-open the blank tab synchronously inside the click gesture. We cannot + // trust the cached auth status here: an `authenticated` cache can be stale + // (session expired server-side), and if we skipped the pre-open and then + // discovered that only after `await refetch()`, the later `window.open` + // would run outside the user gesture and be blocked by the browser. Opening + // now and closing below when it turns out unneeded keeps the popup reliable. + const browserWindow = openPendingBrowserWindow(); + setIsCheckingConnection(true); + try { + const refreshed = await refetch(); + const latestStatus = refreshed.data ?? data; + startConnectionFlow(latestStatus, browserWindow); + } catch (err) { + closePendingBrowserWindow(browserWindow); + toast.error(err instanceof Error ? err.message : String(err)); + } finally { + setIsCheckingConnection(false); + } + }; + + const completeAuthorization = ( + deviceCode: string, + { automatic, attemptId }: { automatic: boolean; attemptId?: number }, + ) => { + const toastOptions = + authToastIdRef.current == null + ? undefined + : { id: authToastIdRef.current }; + completeAuth.mutate( + { + device_code: deviceCode, + ...(automatic + ? { wait_timeout_seconds: AUTOMATIC_LARK_AUTH_WAIT_SECONDS } + : {}), + }, + { + onSuccess: (result) => { + // react-query still fires this after the dialog unmounts; bail so we + // don't toast, setState, refetch, or reschedule a retry timer on a + // component that is gone. + if (!isMountedRef.current) { + return; + } + if (automatic && attemptId !== authAttemptIdRef.current) { + return; + } + if (result.success) { + clearAuthRetryTimer(); + toast.success(result.message, toastOptions); + authToastIdRef.current = null; + setPendingFlow(null); + browserWindowRef.current = null; + return; + } + toast.info( + result.message || + t.settings.integrations.lark.authorizationStillPending, + toastOptions, + ); + if (automatic && attemptId != null) { + scheduleAuthorizationRetry(deviceCode, attemptId); + } + }, + onError: (err) => { + if (!isMountedRef.current) { + return; + } + if (automatic && attemptId !== authAttemptIdRef.current) { + return; + } + if ( + automatic && + err instanceof LarkIntegrationRequestError && + err.status === 504 + ) { + toast.info( + t.settings.integrations.lark.authorizationStillPending, + toastOptions, + ); + if (attemptId != null) { + scheduleAuthorizationRetry(deviceCode, attemptId); + } + return; + } + toast.error( + err instanceof Error ? err.message : String(err), + toastOptions, + ); + authToastIdRef.current = null; + }, + }, + ); + }; + + const scheduleAuthorizationRetry = ( + deviceCode: string, + attemptId: number, + ) => { + clearAuthRetryTimer(); + if (!isMountedRef.current) { + return; + } + if (Date.now() >= authDeadlineRef.current) { + toast.info(t.settings.integrations.lark.authorizationStillPending); + return; + } + authRetryTimeoutRef.current = setTimeout(() => { + completeAuthorization(deviceCode, { automatic: true, attemptId }); + }, 1500); + }; + + const startAutomaticAuthorizationCheck = (result: LarkAuthStartResponse) => { + clearAuthRetryTimer(); + const attemptId = authAttemptIdRef.current + 1; + authAttemptIdRef.current = attemptId; + authDeadlineRef.current = + Date.now() + Math.max(result.expires_in ?? 300, 30) * 1000; + completeAuthorization(result.device_code, { automatic: true, attemptId }); + }; + + const handleCompleteAuth = () => { + if (!pendingFlow) return; + if (pendingFlow.kind !== "auth") { + return; + } + clearAuthRetryTimer(); + authAttemptIdRef.current += 1; + completeAuthorization(pendingFlow.device_code, { automatic: false }); + }; + + const handleCopyAuthLink = async () => { + if (!pendingFlow) return; + try { + await navigator.clipboard.writeText(pendingFlow.verification_url); + toast.success(t.clipboard.copiedToClipboard); + } catch { + toast.error(t.clipboard.failedToCopyToClipboard); + } + }; + + const installDisabled = + env.NEXT_PUBLIC_STATIC_WEBSITE_ONLY === "true" || + !isAdmin || + install.isPending; + const authDisabled = + env.NEXT_PUBLIC_STATIC_WEBSITE_ONLY === "true" || + !data?.installed || + !data?.cli.available || + connectActionBusy; + + const connectButtonLabel = isCheckingConnection + ? t.settings.integrations.lark.checkingConnection + : connectBusy + ? t.settings.integrations.lark.authStarting + : credentialsConfigured && hasAdditionalPermissionRequest + ? t.settings.integrations.lark.requestPermissions + : credentialsConfigured + ? t.settings.integrations.lark.connectedAction + : t.settings.integrations.lark.connect; + + const permissionDomains = LARK_AUTH_DOMAINS.map((id) => ({ + id, + label: t.settings.integrations.lark.authDomains[id].label, + description: t.settings.integrations.lark.authDomains[id].description, + })); + + return ( + + +
+
+ +
+
+ {t.settings.integrations.lark.title} + + {t.settings.integrations.lark.description} + +
+
+ + + + + + {isLoading ? ( +
+ {t.common.loading} +
+ ) : error ? ( + + + {t.settings.integrations.loadFailed} + + {error instanceof Error ? error.message : String(error)} + + + ) : data ? ( + <> +
+ + + + {showSandboxRuntime && ( + + )} +
+ {data.installed && ( +
+ + {t.settings.integrations.lark.installedVersion( + data.manifest_version ?? data.version, + )} + + {data.latest_available_version && + data.latest_available_version !== + (data.manifest_version ?? data.version) && ( + + {t.settings.integrations.lark.updateAvailable( + data.latest_available_version, + )} + + )} + {data.runtime_version_mismatch && ( + + {t.settings.integrations.lark.runtimeVersionMismatch} + + )} +
+ )} + + {data.installed && data.cli.available && ( +
+
+
+ {t.settings.integrations.lark.permissionTitle} +
+

+ {t.settings.integrations.lark.permissionDescription} +

+
+
+ {permissionDomains.map((domain) => { + const selected = selectedAuthDomains.includes(domain.id); + return ( + + ); + })} +
+
+ + setCustomAuthScope(event.currentTarget.value) + } + disabled={connectActionBusy} + placeholder={ + t.settings.integrations.lark.customScopePlaceholder + } + aria-label={t.settings.integrations.lark.customScopeLabel} + /> +

+ {t.settings.integrations.lark.customScopeDescription} +

+
+
+ )} +
+ + + {!isAdmin && ( + + {t.settings.integrations.adminRequired} + + )} +
+ {install.isPending && ( + + + + {t.settings.integrations.lark.installingTitle} + + + {t.settings.integrations.lark.installingDescription} + + + )} + {pendingFlow && ( + + + + {pendingFlow.kind === "config" + ? t.settings.integrations.lark.openConnectionLinkTitle + : completeAuth.isPending + ? t.settings.integrations.lark.waitingAuthTitle + : t.settings.integrations.lark.openAuthLinkTitle} + + +
+

+ {pendingFlow.kind === "config" + ? t.settings.integrations.lark + .openConnectionLinkDescription + : completeAuth.isPending + ? t.settings.integrations.lark.waitingAuthDescription + : t.settings.integrations.lark + .openAuthLinkDescription} +

+
+ {pendingFlow.verification_url} +
+
+ + + {pendingFlow.kind === "config" ? ( + + ) : ( + + )} +
+ {pendingFlow.expires_in != null && ( +

+ {t.settings.integrations.lark.authExpiresIn( + pendingFlow.expires_in, + )} +

+ )} +
+ + + )} + + ) : null} + + + ); +} + +function StatusItem({ + label, + ok, + value, +}: { + label: string; + ok: boolean; + value: string; +}) { + const { t } = useI18n(); + return ( +
+
+
{label}
+ + {ok ? ( + + ) : ( + + )} + {ok ? t.settings.integrations.ready : t.settings.integrations.pending} + +
+
{value}
+
+ ); +} + +function IntegrationNextStep({ + installed, + cliReady, + connected, + credentialsConfigured, +}: { + installed: boolean; + cliReady: boolean; + connected: boolean; + credentialsConfigured: boolean; +}) { + const { t } = useI18n(); + if (!installed) { + return ( + + {t.settings.integrations.lark.installNextTitle} + + {t.settings.integrations.lark.installNextDescription} + + + ); + } + if (!cliReady) { + return ( + + {t.settings.integrations.lark.cliNextTitle} + + {t.settings.integrations.lark.cliNextDescription} + + + ); + } + if (connected) { + return ( + + + {t.settings.integrations.lark.connectedTitle} + + {t.settings.integrations.lark.connectedDescription} + + + ); + } + if (credentialsConfigured) { + return ( + + + {t.settings.integrations.lark.configuredTitle} + + {t.settings.integrations.lark.configuredDescription} + + + ); + } + return ( + + + {t.settings.integrations.lark.authNextTitle} + + {t.settings.integrations.lark.authNextDescription} + + + ); +} diff --git a/frontend/src/components/workspace/settings/settings-dialog-host.tsx b/frontend/src/components/workspace/settings/settings-dialog-host.tsx new file mode 100644 index 000000000..9c4a03767 --- /dev/null +++ b/frontend/src/components/workspace/settings/settings-dialog-host.tsx @@ -0,0 +1,25 @@ +"use client"; + +import { SettingsDialog } from "./settings-dialog"; +import { + setSettingsDialogOpen, + useSettingsDialog, +} from "./settings-dialog-store"; + +/** + * The single application-wide Settings dialog instance. + * + * Mounted once at the workspace root; every entry point (nav menu, command + * palette, deep link) opens it through the shared store rather than mounting + * its own dialog. + */ +export function SettingsDialogHost() { + const { open, section } = useSettingsDialog(); + return ( + + ); +} diff --git a/frontend/src/components/workspace/settings/settings-dialog-store.ts b/frontend/src/components/workspace/settings/settings-dialog-store.ts new file mode 100644 index 000000000..dcd3f6acd --- /dev/null +++ b/frontend/src/components/workspace/settings/settings-dialog-store.ts @@ -0,0 +1,89 @@ +"use client"; + +import { useCallback, useSyncExternalStore } from "react"; + +import type { SettingsSection } from "./settings-dialog"; + +type Listener = () => void; + +type SettingsDialogState = { + open: boolean; + section: SettingsSection; +}; + +const listeners = new Set(); + +let state: SettingsDialogState = { open: false, section: "appearance" }; + +function emitChange() { + for (const listener of listeners) { + listener(); + } +} + +function setState(next: SettingsDialogState) { + if (next.open === state.open && next.section === state.section) { + return; + } + state = next; + emitChange(); +} + +export function subscribeSettingsDialog(listener: Listener): () => void { + listeners.add(listener); + return () => { + listeners.delete(listener); + }; +} + +export function getSettingsDialogSnapshot(): SettingsDialogState { + return state; +} + +const SERVER_SNAPSHOT: SettingsDialogState = { + open: false, + section: "appearance", +}; + +function getServerSnapshot(): SettingsDialogState { + return SERVER_SNAPSHOT; +} + +export function openSettingsDialog(section: SettingsSection) { + setState({ open: true, section }); +} + +export function setSettingsDialogOpen(open: boolean) { + setState({ open, section: state.section }); +} + +/** + * Shared open/section state for the single application-wide Settings dialog. + * + * Multiple entry points (nav menu, command palette, `?settings=` deep link) + * drive this one store instead of each mounting its own `SettingsDialog`, so + * two dialogs can never be open at once with racing per-instance flows (e.g. + * duplicate Lark auth device-code polling). + */ +export function useSettingsDialog() { + const snapshot = useSyncExternalStore( + subscribeSettingsDialog, + getSettingsDialogSnapshot, + getServerSnapshot, + ); + + const open = useCallback((section: SettingsSection) => { + openSettingsDialog(section); + }, []); + + const setOpen = useCallback((next: boolean) => { + setSettingsDialogOpen(next); + }, []); + + return { + open: snapshot.open, + section: snapshot.section, + openSettings: open, + setSettingsOpen: setOpen, + }; +} diff --git a/frontend/src/components/workspace/settings/settings-dialog.tsx b/frontend/src/components/workspace/settings/settings-dialog.tsx index 4b9e3f373..779b91b0e 100644 --- a/frontend/src/components/workspace/settings/settings-dialog.tsx +++ b/frontend/src/components/workspace/settings/settings-dialog.tsx @@ -6,6 +6,7 @@ import { InfoIcon, BrainIcon, PaletteIcon, + PlugZapIcon, SparklesIcon, UserIcon, WrenchIcon, @@ -23,6 +24,7 @@ import { AboutSettingsPage } from "@/components/workspace/settings/about-setting import { AccountSettingsPage } from "@/components/workspace/settings/account-settings-page"; import { AppearanceSettingsPage } from "@/components/workspace/settings/appearance-settings-page"; import { ChannelsSettingsPage } from "@/components/workspace/settings/channels-settings-page"; +import { IntegrationsSettingsPage } from "@/components/workspace/settings/integrations-settings-page"; import { MemorySettingsPage } from "@/components/workspace/settings/memory-settings-page"; import { NotificationSettingsPage } from "@/components/workspace/settings/notification-settings-page"; import { SkillSettingsPage } from "@/components/workspace/settings/skill-settings-page"; @@ -30,10 +32,11 @@ import { ToolSettingsPage } from "@/components/workspace/settings/tool-settings- import { useI18n } from "@/core/i18n/hooks"; import { cn } from "@/lib/utils"; -type SettingsSection = +export type SettingsSection = | "account" | "appearance" | "channels" + | "integrations" | "memory" | "tools" | "skills" @@ -80,6 +83,11 @@ export function SettingsDialog(props: SettingsDialogProps) { label: t.settings.sections.channels, icon: CableIcon, }, + { + id: "integrations", + label: t.settings.sections.integrations, + icon: PlugZapIcon, + }, { id: "memory", label: t.settings.sections.memory, @@ -93,6 +101,7 @@ export function SettingsDialog(props: SettingsDialogProps) { t.settings.sections.account, t.settings.sections.appearance, t.settings.sections.channels, + t.settings.sections.integrations, t.settings.sections.memory, t.settings.sections.tools, t.settings.sections.skills, @@ -153,6 +162,7 @@ export function SettingsDialog(props: SettingsDialogProps) { )} {activeSection === "notification" && } {activeSection === "channels" && } + {activeSection === "integrations" && } {activeSection === "about" && } diff --git a/frontend/src/components/workspace/workspace-nav-menu.tsx b/frontend/src/components/workspace/workspace-nav-menu.tsx index 8b99be078..98cd2355c 100644 --- a/frontend/src/components/workspace/workspace-nav-menu.tsx +++ b/frontend/src/components/workspace/workspace-nav-menu.tsx @@ -28,7 +28,7 @@ import { import { useI18n } from "@/core/i18n/hooks"; import { GithubIcon } from "./github-icon"; -import { SettingsDialog } from "./settings"; +import { useSettingsDialog } from "./settings"; function NavMenuButtonContent({ isSidebarOpen, @@ -51,10 +51,7 @@ function NavMenuButtonContent({ } export function WorkspaceNavMenu() { - const [settingsOpen, setSettingsOpen] = useState(false); - const [settingsDefaultSection, setSettingsDefaultSection] = useState< - "appearance" | "memory" | "tools" | "skills" | "notification" | "about" - >("appearance"); + const { openSettings } = useSettingsDialog(); const [mounted, setMounted] = useState(false); const { open: isSidebarOpen } = useSidebar(); const { t } = useI18n(); @@ -65,11 +62,6 @@ export function WorkspaceNavMenu() { return ( <> - {mounted ? ( @@ -90,8 +82,7 @@ export function WorkspaceNavMenu() { { - setSettingsDefaultSection("appearance"); - setSettingsOpen(true); + openSettings("appearance"); }} > @@ -139,8 +130,7 @@ export function WorkspaceNavMenu() { { - setSettingsDefaultSection("about"); - setSettingsOpen(true); + openSettings("about"); }} > diff --git a/frontend/src/components/workspace/workspace-settings-deep-link.tsx b/frontend/src/components/workspace/workspace-settings-deep-link.tsx new file mode 100644 index 000000000..2c08b84be --- /dev/null +++ b/frontend/src/components/workspace/workspace-settings-deep-link.tsx @@ -0,0 +1,68 @@ +"use client"; + +import { usePathname, useRouter, useSearchParams } from "next/navigation"; +import { useEffect, useRef } from "react"; + +import { + openSettingsDialog, + type SettingsSection, + useSettingsDialog, +} from "./settings"; + +const SETTINGS_SECTIONS = new Set([ + "account", + "appearance", + "channels", + "integrations", + "memory", + "tools", + "skills", + "notification", + "about", +]); + +function asSettingsSection(value: string | null): SettingsSection | null { + if (!value) return null; + return SETTINGS_SECTIONS.has(value as SettingsSection) + ? (value as SettingsSection) + : null; +} + +/** + * Bridges the `?settings=
` query param to the shared settings dialog + * store. It does not mount its own dialog — a single {@link SettingsDialogHost} + * renders the one dialog — so a deep link can never race a second dialog opened + * from the nav menu or command palette. + */ +export function WorkspaceSettingsDeepLink() { + const router = useRouter(); + const pathname = usePathname(); + const searchParams = useSearchParams(); + const { open } = useSettingsDialog(); + const openedFromDeepLinkRef = useRef(false); + + useEffect(() => { + const nextSection = asSettingsSection(searchParams.get("settings")); + if (nextSection) { + openedFromDeepLinkRef.current = true; + openSettingsDialog(nextSection); + } + }, [searchParams]); + + useEffect(() => { + if (open || !openedFromDeepLinkRef.current) { + return; + } + openedFromDeepLinkRef.current = false; + if (searchParams.has("settings")) { + const next = new URLSearchParams(searchParams); + next.delete("settings"); + const suffix = next.toString(); + router.replace(suffix ? `${pathname}?${suffix}` : pathname, { + scroll: false, + }); + } + }, [open, pathname, router, searchParams]); + + return null; +} diff --git a/frontend/src/core/i18n/locales/en-US.ts b/frontend/src/core/i18n/locales/en-US.ts index 7caec8dcb..d81ef11db 100644 --- a/frontend/src/core/i18n/locales/en-US.ts +++ b/frontend/src/core/i18n/locales/en-US.ts @@ -706,6 +706,7 @@ export const enUS: Translations = { account: "Account", appearance: "Appearance", channels: "Channels", + integrations: "Integrations", memory: "Memory", tools: "Tools", skills: "Skills", @@ -817,6 +818,203 @@ export const enUS: Translations = { disabled: "Channel connections are not enabled on this server. Ask an administrator to enable channel_connections.", }, + integrations: { + title: "Integrations", + description: + "Connect third-party tools and work platforms so agents can use them directly.", + refresh: "Refresh", + install: "Install", + reinstall: "Reinstall", + installing: "Installing...", + ready: "Ready", + pending: "Pending", + available: "Available", + unavailable: "Unavailable", + connected: "Connected", + loadFailed: "Failed to load integration status", + adminRequired: "Admin privileges are required to install integrations.", + lark: { + title: "Lark / Feishu CLI", + description: + "Install the official Lark/Feishu agent skills and let agents use Lark after authorization.", + skillPack: "Skill pack", + gatewayCli: "Gateway CLI", + auth: "Auth", + sandboxRuntime: "Sandbox runtime", + sandboxRuntimeInitContainer: "Provisioned by init container", + sandboxRuntimeGatewayDownload: "Provisioned by Gateway", + sandboxRuntimeNotReady: + "Not ready — lark-cli may be missing at chat time", + notInstalled: "Not installed", + skillsInstalled: (installed, expected) => + `${installed}/${expected} skills installed`, + installedVersion: (version) => `Installed: ${version}`, + updateAvailable: (version) => + `Update available: ${version} — admin reinstall updates the managed Gateway CLI and skill pack`, + runtimeVersionMismatch: + "Skill pack version differs from the Gateway runtime lark-cli; admin reinstall attempts to update the managed Gateway CLI and realign the skill pack", + authNotConfigured: "Not connected", + authConfigured: "Credentials configured (not live-verified)", + authConfiguredFor: (user) => + `${user} · credentials configured (not live-verified)`, + connect: "Connect Lark", + authStarting: "Opening connection link...", + checkingConnection: "Checking connection...", + connectedAction: "Reconnect Lark", + requestPermissions: "Request permissions", + alreadyConnected: + "Lark is already connected. If authorization expires, refresh the status and reconnect.", + connectionStarted: "Connection link opened", + connectionReady: "Connection is ready. Opening authorization...", + authStarted: + "Authorization page opened. DeerFlow will detect completion automatically.", + authorizationStillPending: + 'Authorization is not complete yet. Finish it in the browser; DeerFlow keeps checking automatically. You can click "I completed authorization" if the page does not update.', + permissionTitle: "Authorization scope", + permissionDescription: + "By default, DeerFlow only completes the base sign-in and does not request any business permissions. Select the domains you need here; connected users can re-authorize to add more (scopes accumulate).", + authDomains: { + calendar: { + label: "Calendar", + description: + "Events, free/busy, RSVP, and meeting-room scheduling.", + }, + im: { + label: "Messenger", + description: + "Send/reply messages, manage group chats, search history, download media.", + }, + docs: { + label: "Docs", + description: "Create, read, update, and search documents.", + }, + drive: { + label: "Drive", + description: + "Upload/download files, search docs & wiki, manage comments.", + }, + sheets: { + label: "Sheets", + description: "Read, write, append, find, and export spreadsheets.", + }, + base: { + label: "Base", + description: + "Bitable tables, fields, records, views, dashboards, and workflows.", + }, + wiki: { + label: "Wiki", + description: "Knowledge spaces, nodes, and wiki documents.", + }, + task: { + label: "Tasks", + description: + "Tasks, task lists, subtasks, comments, and reminders.", + }, + mail: { + label: "Mail", + description: + "Browse, search, read, send, reply, forward, and manage drafts.", + }, + vc: { + label: "Meetings", + description: "Meeting records, minutes artifacts, and recordings.", + }, + minutes: { + label: "Minutes", + description: "Meeting minutes content and transcripts.", + }, + note: { + label: "Notes", + description: "Meeting notes and related content.", + }, + slides: { + label: "Slides", + description: "Presentations and slide content.", + }, + markdown: { + label: "Markdown", + description: + "Create, fetch, patch, and overwrite Drive-native .md files.", + }, + mindnotes: { + label: "Mind notes", + description: "Mind notes content.", + }, + contact: { + label: "Contacts", + description: "Look up users by name/email/phone and read profiles.", + }, + approval: { + label: "Approval", + description: + "Query and act on approval tasks; cancel and CC instances.", + }, + attendance: { + label: "Attendance", + description: "Query personal attendance check-in records.", + }, + okr: { + label: "OKR", + description: + "Objectives, key results, alignments, indicators, and progress.", + }, + event: { + label: "Events", + description: "Subscribe to and consume real-time platform events.", + }, + apps: { + label: "Apps", + description: + "Create Spark/Miaoda apps, publish sites, and manage access scope.", + }, + all: { + label: "All", + description: + "Request every business domain supported by lark-cli. Use this only when the missing permission is unclear.", + }, + }, + customScopeLabel: "Exact OAuth scope", + customScopePlaceholder: "For example calendar:calendar.event:read", + customScopeDescription: + "Advanced: if an error reports a missing scope, paste it here. Examples: calendar:calendar.event:read, calendar:calendar.free_busy:read.", + openConnectionLinkTitle: "Continue connecting Lark", + openConnectionLinkDescription: + "The first connection needs one browser confirmation from Lark. Open the link below and finish the prompt, then return here to continue authorization.", + openAuthLinkTitle: "Authorize Lark in your browser", + openAuthLinkDescription: + "Open the link below to authorize. DeerFlow keeps checking automatically and will save the connection after approval.", + waitingAuthTitle: "Waiting for Lark authorization", + waitingAuthDescription: + "Finish authorization in the browser page that just opened. DeerFlow will update this panel automatically; the button below is only a fallback.", + openAuthLink: "Open link", + copyAuthLink: "Copy link", + completeAuth: "I completed authorization", + continueAuth: "I completed browser confirmation, continue", + preparingAuthorization: "Preparing authorization...", + completingAuth: "Checking...", + authExpiresIn: (seconds) => + `This link expires in about ${seconds} seconds.`, + installingTitle: "Installing official skill pack", + installingDescription: + "This usually finishes within 30 seconds; slower networks may take about 1 minute. The status refreshes automatically when installation completes.", + installNextTitle: "Install the official skill pack first", + installNextDescription: + "After installation, /lark-doc, /lark-im, /lark-sheets and related skills appear in the skill index.", + cliNextTitle: "Install Gateway CLI", + cliNextDescription: + "The skill pack is installed, but the Gateway cannot find lark-cli. Admin reinstall attempts to download the managed Gateway CLI; offline deployments can use an image with @larksuite/cli built in.", + configuredTitle: "Lark credentials are configured locally", + configuredDescription: + "Credentials are present, but their current validity has not been checked with Lark. Reconnect to refresh and live-verify authorization.", + connectedTitle: "Lark authorization is live-verified", + connectedDescription: + "The current user's authorization was verified with Lark during this connection flow. Reconnect whenever you need to refresh it or add permissions.", + authNextTitle: "Complete browser authorization next", + authNextDescription: + "Click “Connect Lark”; DeerFlow checks the current status first and opens browser authorization only when disconnected or expired.", + }, + }, skills: { title: "Agent Skills", description: diff --git a/frontend/src/core/i18n/locales/types.ts b/frontend/src/core/i18n/locales/types.ts index 35b3469eb..918d172a0 100644 --- a/frontend/src/core/i18n/locales/types.ts +++ b/frontend/src/core/i18n/locales/types.ts @@ -590,6 +590,7 @@ export interface Translations { account: string; appearance: string; channels: string; + integrations: string; memory: string; tools: string; skills: string; @@ -692,6 +693,105 @@ export interface Translations { description: string; disabled: string; }; + integrations: { + title: string; + description: string; + refresh: string; + install: string; + reinstall: string; + installing: string; + ready: string; + pending: string; + available: string; + unavailable: string; + connected: string; + loadFailed: string; + adminRequired: string; + lark: { + title: string; + description: string; + skillPack: string; + gatewayCli: string; + auth: string; + sandboxRuntime: string; + sandboxRuntimeInitContainer: string; + sandboxRuntimeGatewayDownload: string; + sandboxRuntimeNotReady: string; + notInstalled: string; + skillsInstalled: (installed: number, expected: number) => string; + installedVersion: (version: string) => string; + updateAvailable: (version: string) => string; + runtimeVersionMismatch: string; + authNotConfigured: string; + authConfigured: string; + authConfiguredFor: (user: string) => string; + connect: string; + authStarting: string; + checkingConnection: string; + connectedAction: string; + requestPermissions: string; + alreadyConnected: string; + connectionStarted: string; + connectionReady: string; + authStarted: string; + authorizationStillPending: string; + permissionTitle: string; + permissionDescription: string; + authDomains: Record< + | "approval" + | "apps" + | "attendance" + | "base" + | "calendar" + | "contact" + | "docs" + | "drive" + | "event" + | "im" + | "mail" + | "markdown" + | "mindnotes" + | "minutes" + | "note" + | "okr" + | "sheets" + | "slides" + | "task" + | "vc" + | "wiki" + | "all", + { label: string; description: string } + >; + customScopeLabel: string; + customScopePlaceholder: string; + customScopeDescription: string; + openConnectionLinkTitle: string; + openConnectionLinkDescription: string; + openAuthLinkTitle: string; + openAuthLinkDescription: string; + waitingAuthTitle: string; + waitingAuthDescription: string; + openAuthLink: string; + copyAuthLink: string; + completeAuth: string; + continueAuth: string; + preparingAuthorization: string; + completingAuth: string; + authExpiresIn: (seconds: number) => string; + installingTitle: string; + installingDescription: string; + installNextTitle: string; + installNextDescription: string; + cliNextTitle: string; + cliNextDescription: string; + configuredTitle: string; + configuredDescription: string; + connectedTitle: string; + connectedDescription: string; + authNextTitle: string; + authNextDescription: string; + }; + }; skills: { title: string; description: string; diff --git a/frontend/src/core/i18n/locales/zh-CN.ts b/frontend/src/core/i18n/locales/zh-CN.ts index d49d1119b..eb996cabf 100644 --- a/frontend/src/core/i18n/locales/zh-CN.ts +++ b/frontend/src/core/i18n/locales/zh-CN.ts @@ -679,6 +679,7 @@ export const zhCN: Translations = { account: "账号", appearance: "外观", channels: "渠道", + integrations: "集成", memory: "记忆", tools: "工具", skills: "技能", @@ -786,6 +787,188 @@ export const zhCN: Translations = { disabled: "当前服务器未启用渠道连接。请联系管理员开启 channel_connections。", }, + integrations: { + title: "集成", + description: "连接第三方工具和办公生态，让 Agent 能直接使用对应能力。", + refresh: "刷新", + install: "安装", + reinstall: "重新安装", + installing: "安装中...", + ready: "就绪", + pending: "待处理", + available: "可用", + unavailable: "不可用", + connected: "已连接", + loadFailed: "加载集成状态失败", + adminRequired: "需要管理员权限才能安装集成。", + lark: { + title: "Lark / 飞书 CLI", + description: + "安装官方 Lark/Feishu Agent Skills，并在授权后让 Agent 直接使用飞书能力。", + skillPack: "技能包", + gatewayCli: "Gateway CLI", + auth: "授权", + sandboxRuntime: "沙箱运行时", + sandboxRuntimeInitContainer: "由 init container 提供", + sandboxRuntimeGatewayDownload: "由 Gateway 提供", + sandboxRuntimeNotReady: "未就绪 —— 对话时 lark-cli 可能不可用", + notInstalled: "尚未安装", + skillsInstalled: (installed, expected) => + `已安装 ${installed}/${expected} 个技能`, + installedVersion: (version) => `已安装版本：${version}`, + updateAvailable: (version) => + `有新版本：${version} —— 管理员重新安装会更新 managed Gateway CLI 和技能包`, + runtimeVersionMismatch: + "技能包版本与 Gateway 运行时 lark-cli 不一致；管理员重新安装会尝试更新 managed Gateway CLI 并重新对齐技能包", + authNotConfigured: "尚未连接", + authConfigured: "凭证已配置（未实时验证）", + authConfiguredFor: (user) => `${user} · 凭证已配置（未实时验证）`, + connect: "连接飞书", + authStarting: "正在打开连接链接...", + checkingConnection: "正在检查连接状态...", + connectedAction: "重新连接飞书", + requestPermissions: "申请新权限", + alreadyConnected: + "飞书已连接，无需重复授权。如果授权已过期，刷新状态后可重新连接。", + connectionStarted: "连接链接已打开", + connectionReady: "连接准备已完成，正在打开授权链接", + authStarted: "授权页已打开，DeerFlow 会自动检测授权结果。", + authorizationStillPending: + "还没有检测到授权完成。请在浏览器完成授权；DeerFlow 会继续自动检测。如果页面没有更新，可点击“我已完成授权”。", + permissionTitle: "授权范围", + permissionDescription: + "默认只完成基础登录，不会申请任何业务权限。按需在这里勾选要授权的业务域；已连接用户可以重新授权继续追加（scope 会累积）。", + authDomains: { + calendar: { + label: "日历", + description: "日程、忙闲、日程回复与会议室预定。", + }, + im: { + label: "消息", + description: "收发/回复消息、管理群聊、搜索记录、下载媒体。", + }, + docs: { + label: "文档", + description: "创建、读取、编辑和搜索云文档。", + }, + drive: { + label: "云空间", + description: "上传/下载文件、搜索文档与知识库、管理评论。", + }, + sheets: { + label: "电子表格", + description: "读取、写入、追加、查找和导出电子表格。", + }, + base: { + label: "多维表格", + description: "多维表格的表、字段、记录、视图、仪表盘与工作流。", + }, + wiki: { + label: "知识库", + description: "知识空间、节点与知识库文档。", + }, + task: { + label: "任务", + description: "任务、清单、子任务、评论与提醒。", + }, + mail: { + label: "邮件", + description: "浏览、搜索、阅读、发送、回复、转发与管理草稿。", + }, + vc: { + label: "视频会议", + description: "会议记录、纪要产物与录制。", + }, + minutes: { + label: "妙记", + description: "会议纪要内容与逐字稿。", + }, + note: { + label: "笔记", + description: "会议笔记及相关内容。", + }, + slides: { + label: "幻灯片", + description: "演示文稿与幻灯片内容。", + }, + markdown: { + label: "Markdown", + description: "创建、获取、局部修改和覆盖云盘原生 .md 文件。", + }, + mindnotes: { + label: "思维笔记", + description: "思维笔记内容。", + }, + contact: { + label: "通讯录", + description: "按姓名/邮箱/电话查用户并读取资料。", + }, + approval: { + label: "审批", + description: "查询和处理审批任务、撤销与抄送实例。", + }, + attendance: { + label: "考勤", + description: "查询个人考勤打卡记录。", + }, + okr: { + label: "OKR", + description: "目标、关键结果、对齐、指标与进展。", + }, + event: { + label: "实时事件", + description: "订阅并消费平台实时事件。", + }, + apps: { + label: "妙搭应用", + description: "创建 Spark/妙搭应用、发布站点并管理可见范围。", + }, + all: { + label: "全部", + description: + "申请 lark-cli 支持的全部业务域权限。仅在不确定缺哪个权限时使用。", + }, + }, + customScopeLabel: "具体 OAuth scope", + customScopePlaceholder: "例如 calendar:calendar.event:read", + customScopeDescription: + "高级用法：如果错误里给出了缺失 scope，可直接填在这里。例如 calendar:calendar.event:read、calendar:calendar.free_busy:read。", + openConnectionLinkTitle: "继续完成飞书连接", + openConnectionLinkDescription: + "首次连接需要在浏览器里完成一次飞书确认。打开下面的链接按提示完成；完成后回到这里继续授权。", + openAuthLinkTitle: "在浏览器中完成飞书授权", + openAuthLinkDescription: + "打开下面的链接完成授权。DeerFlow 会持续自动检测，并在授权通过后保存连接状态。", + waitingAuthTitle: "等待飞书授权完成", + waitingAuthDescription: + "请在刚打开的浏览器页面完成授权。DeerFlow 会自动更新这里的状态；下方按钮只是兜底操作。", + openAuthLink: "打开链接", + copyAuthLink: "复制链接", + completeAuth: "我已完成授权", + continueAuth: "我已完成浏览器确认，继续授权", + preparingAuthorization: "正在准备授权...", + completingAuth: "确认中...", + authExpiresIn: (seconds) => `链接将在约 ${seconds} 秒后过期。`, + installingTitle: "正在安装官方技能包", + installingDescription: + "通常 30 秒内完成，网络较慢时可能需要约 1 分钟。安装完成后会自动刷新状态。", + installNextTitle: "先安装官方技能包", + installNextDescription: + "安装后，/lark-doc、/lark-im、/lark-sheets 等技能会出现在技能索引中。", + cliNextTitle: "需要安装 Gateway CLI", + cliNextDescription: + "技能包已安装，但 Gateway 找不到 lark-cli。管理员重新安装集成会尝试下载 managed Gateway CLI；离线部署可使用内置 @larksuite/cli 的镜像。", + configuredTitle: "飞书凭证已在本地配置", + configuredDescription: + "当前只确认本地存在凭证，尚未向飞书实时验证有效性。重新连接可刷新并实时验证授权。", + connectedTitle: "飞书授权已实时验证", + connectedDescription: + "本次连接流程已向飞书验证当前用户授权。需要刷新授权或追加权限时，可重新连接。", + authNextTitle: "下一步完成浏览器授权", + authNextDescription: + "点击“连接飞书”后，DeerFlow 会先检查当前状态；未连接或授权过期时会拉起浏览器授权。", + }, + }, skills: { title: "技能", description: "管理 Agent Skill 配置和启用状态。", diff --git a/frontend/src/core/integrations/lark/api.ts b/frontend/src/core/integrations/lark/api.ts new file mode 100644 index 000000000..dac796523 --- /dev/null +++ b/frontend/src/core/integrations/lark/api.ts @@ -0,0 +1,153 @@ +import { fetch } from "@/core/api/fetcher"; +import { getBackendBaseURL } from "@/core/config"; + +import type { + LarkAuthCompleteRequest, + LarkAuthCompleteResponse, + LarkAuthStartRequest, + LarkAuthStartResponse, + LarkConfigCompleteRequest, + LarkConfigCompleteResponse, + LarkConfigStartRequest, + LarkConfigStartResponse, + LarkInstallResponse, + LarkIntegrationStatus, +} from "./types"; + +export class LarkIntegrationRequestError extends Error { + readonly status: number; + + constructor(status: number, message: string) { + super(message); + this.name = "LarkIntegrationRequestError"; + this.status = status; + } + + get isAdminRequired(): boolean { + return this.status === 403; + } +} + +async function readErrorDetail(response: Response): Promise { + const data = (await response.json().catch(() => ({}))) as { + detail?: string; + }; + return data.detail ?? `HTTP ${response.status}: ${response.statusText}`; +} + +export async function loadLarkIntegrationStatus(): Promise { + const response = await fetch( + `${getBackendBaseURL()}/api/integrations/lark/status`, + ); + if (!response.ok) { + throw new LarkIntegrationRequestError( + response.status, + await readErrorDetail(response), + ); + } + return response.json(); +} + +export async function installLarkIntegration(): Promise { + const response = await fetch( + `${getBackendBaseURL()}/api/integrations/lark/install`, + { + method: "POST", + }, + ); + if (!response.ok) { + throw new LarkIntegrationRequestError( + response.status, + await readErrorDetail(response), + ); + } + return response.json(); +} + +export async function startLarkAuthorization( + request: LarkAuthStartRequest = {}, +): Promise { + const response = await fetch( + `${getBackendBaseURL()}/api/integrations/lark/auth/start`, + { + method: "POST", + headers: { + "Content-Type": "application/json", + }, + body: JSON.stringify(request), + }, + ); + if (!response.ok) { + throw new LarkIntegrationRequestError( + response.status, + await readErrorDetail(response), + ); + } + return response.json(); +} + +export async function startLarkConfiguration( + request: LarkConfigStartRequest = {}, +): Promise { + const response = await fetch( + `${getBackendBaseURL()}/api/integrations/lark/config/start`, + { + method: "POST", + headers: { + "Content-Type": "application/json", + }, + body: JSON.stringify(request), + }, + ); + if (!response.ok) { + throw new LarkIntegrationRequestError( + response.status, + await readErrorDetail(response), + ); + } + return response.json(); +} + +export async function completeLarkConfiguration( + request: LarkConfigCompleteRequest, +): Promise { + const response = await fetch( + `${getBackendBaseURL()}/api/integrations/lark/config/complete`, + { + method: "POST", + headers: { + "Content-Type": "application/json", + }, + body: JSON.stringify(request), + }, + ); + if (!response.ok) { + throw new LarkIntegrationRequestError( + response.status, + await readErrorDetail(response), + ); + } + return response.json(); +} + +export async function completeLarkAuthorization( + request: LarkAuthCompleteRequest, +): Promise { + const response = await fetch( + `${getBackendBaseURL()}/api/integrations/lark/auth/complete`, + { + method: "POST", + headers: { + "Content-Type": "application/json", + }, + body: JSON.stringify(request), + }, + ); + if (!response.ok) { + throw new LarkIntegrationRequestError( + response.status, + await readErrorDetail(response), + ); + } + return response.json(); +} diff --git a/frontend/src/core/integrations/lark/hooks.ts b/frontend/src/core/integrations/lark/hooks.ts new file mode 100644 index 000000000..22d57b7f0 --- /dev/null +++ b/frontend/src/core/integrations/lark/hooks.ts @@ -0,0 +1,68 @@ +import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; + +import { + completeLarkAuthorization, + completeLarkConfiguration, + installLarkIntegration, + loadLarkIntegrationStatus, + startLarkAuthorization, + startLarkConfiguration, +} from "./api"; + +export const larkIntegrationQueryKey = ["integrations", "lark"] as const; + +export function useLarkIntegrationStatus() { + return useQuery({ + queryKey: larkIntegrationQueryKey, + queryFn: loadLarkIntegrationStatus, + }); +} + +export function useInstallLarkIntegration() { + const queryClient = useQueryClient(); + return useMutation({ + mutationFn: installLarkIntegration, + onSuccess: async (result) => { + queryClient.setQueryData(larkIntegrationQueryKey, result.status); + await queryClient.invalidateQueries({ + queryKey: larkIntegrationQueryKey, + }); + await queryClient.invalidateQueries({ queryKey: ["skills"] }); + }, + }); +} + +export function useStartLarkAuthorization() { + return useMutation({ + mutationFn: startLarkAuthorization, + }); +} + +export function useStartLarkConfiguration() { + return useMutation({ + mutationFn: startLarkConfiguration, + }); +} + +export function useCompleteLarkConfiguration() { + const queryClient = useQueryClient(); + return useMutation({ + mutationFn: completeLarkConfiguration, + onSuccess: async (result) => { + queryClient.setQueryData(larkIntegrationQueryKey, result.status); + await queryClient.invalidateQueries({ + queryKey: larkIntegrationQueryKey, + }); + }, + }); +} + +export function useCompleteLarkAuthorization() { + const queryClient = useQueryClient(); + return useMutation({ + mutationFn: completeLarkAuthorization, + onSuccess: (result) => { + queryClient.setQueryData(larkIntegrationQueryKey, result.status); + }, + }); +} diff --git a/frontend/src/core/integrations/lark/index.ts b/frontend/src/core/integrations/lark/index.ts new file mode 100644 index 000000000..0733bf1a4 --- /dev/null +++ b/frontend/src/core/integrations/lark/index.ts @@ -0,0 +1,3 @@ +export * from "./api"; +export * from "./hooks"; +export * from "./types"; diff --git a/frontend/src/core/integrations/lark/types.ts b/frontend/src/core/integrations/lark/types.ts new file mode 100644 index 000000000..7e8c107b5 --- /dev/null +++ b/frontend/src/core/integrations/lark/types.ts @@ -0,0 +1,102 @@ +export interface LarkCliProbe { + available: boolean; + path: string | null; + version: string | null; + error: string | null; +} + +export interface LarkAuthProbe { + status: + | "authenticated" + | "not_authorized" + | "not_configured" + | "unavailable" + | "error"; + message: string | null; + user: string | null; + verified: boolean; +} + +export type LarkSandboxRuntimeMode = + | "none" + | "gateway-download" + | "init-container"; + +export interface LarkIntegrationStatus { + installed: boolean; + version: string; + manifest_version: string | null; + latest_available_version: string | null; + runtime_version_mismatch: boolean; + app_configured: boolean; + app_id: string | null; + app_brand: string | null; + skills_expected: number; + skills_installed: number; + installed_skills: string[]; + enabled_skills: string[]; + install_path: string; + cli: LarkCliProbe; + auth: LarkAuthProbe; + sandbox_runtime_mode: LarkSandboxRuntimeMode; + sandbox_runtime_ready: boolean; + sandbox_runtime_detail: string | null; +} + +export interface LarkInstallResponse { + success: boolean; + installed_skills: string[]; + message: string; + status: LarkIntegrationStatus; +} + +export interface LarkAuthStartRequest { + recommend?: boolean; + domains?: string[]; + scope?: string | null; +} + +export interface LarkAuthStartResponse { + verification_url: string; + device_code: string; + expires_in: number | null; + user_code: string | null; + hint: string | null; +} + +export interface LarkConfigStartRequest { + brand?: "feishu" | "lark"; +} + +export interface LarkConfigStartResponse { + verification_url: string; + device_code: string; + expires_in: number | null; + interval: number | null; + user_code: string | null; + brand: "feishu" | "lark"; +} + +export interface LarkConfigCompleteRequest { + device_code: string; + brand: "feishu" | "lark"; + interval: number | null; + expires_in: number | null; +} + +export interface LarkConfigCompleteResponse { + success: boolean; + message: string; + status: LarkIntegrationStatus; +} + +export interface LarkAuthCompleteRequest { + device_code: string; + wait_timeout_seconds?: number; +} + +export interface LarkAuthCompleteResponse { + success: boolean; + message: string; + status: LarkIntegrationStatus; +} diff --git a/frontend/tests/e2e/integrations.spec.ts b/frontend/tests/e2e/integrations.spec.ts new file mode 100644 index 000000000..083d81034 --- /dev/null +++ b/frontend/tests/e2e/integrations.spec.ts @@ -0,0 +1,156 @@ +import { expect, test } from "@playwright/test"; + +import { mockLangGraphAPI } from "./utils/mock-api"; + +test.describe("Integrations settings", () => { + test("opens integrations settings from a query-string deep link", async ({ + page, + }) => { + mockLangGraphAPI(page); + + await page.goto("/workspace/chats/new?settings=integrations"); + + const dialog = page.getByRole("dialog", { name: "Settings" }); + await expect(dialog).toBeVisible(); + await expect(dialog.getByText("Lark / Feishu CLI")).toBeVisible(); + }); + + test("keeps a single settings dialog across deep link and nav menu openings", async ({ + page, + }) => { + mockLangGraphAPI(page); + + // Deep link opens the shared dialog on Integrations. + await page.goto("/workspace/chats/new?settings=integrations"); + const dialog = page.getByRole("dialog", { name: "Settings" }); + await expect(dialog).toBeVisible(); + await expect(dialog.getByText("Lark / Feishu CLI")).toBeVisible(); + await expect(page.getByRole("dialog", { name: "Settings" })).toHaveCount(1); + + // Close the modal before using the sidebar. While the modal is open, the + // background is intentionally inert and Playwright should not be able to + // click sidebar controls there. + await page.keyboard.press("Escape"); + await expect(page.getByRole("dialog", { name: "Settings" })).toHaveCount(0); + + // Opening again from the nav menu must still use the same shared host, not + // mount a second SettingsDialog instance. + const sidebar = page.locator("[data-sidebar='sidebar']"); + await sidebar.getByRole("button", { name: /Settings and more/ }).click(); + await page.getByRole("menuitem", { name: "Settings" }).click(); + + // Exactly one Settings dialog is mounted/visible at any time. + await expect(page.getByRole("dialog", { name: "Settings" })).toHaveCount(1); + }); + + test("can install the Lark integration skill pack from settings", async ({ + page, + }) => { + mockLangGraphAPI(page); + let authStartRequest: unknown; + const authCompleteRequests: unknown[] = []; + await page.route( + "**/api/integrations/lark/auth/complete", + async (route) => { + authCompleteRequests.push(route.request().postDataJSON()); + await route.fallback(); + }, + ); + await page.route("**/api/integrations/lark/config/start", async (route) => { + await route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify({ + verification_url: "about:blank", + device_code: "mock-config-device-code", + expires_in: 600, + interval: 5, + user_code: "config", + brand: "feishu", + }), + }); + }); + await page.route("**/api/integrations/lark/auth/start", async (route) => { + authStartRequest = route.request().postDataJSON(); + await route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify({ + verification_url: "https://open.feishu.cn/auth/mock-device", + device_code: "mock-device-code", + expires_in: 600, + user_code: null, + hint: null, + }), + }); + }); + + await page.goto("/workspace/chats/new"); + + const sidebar = page.locator("[data-sidebar='sidebar']"); + await sidebar.getByRole("button", { name: /Settings and more/ }).click(); + await page.getByRole("menuitem", { name: "Settings" }).click(); + + const dialog = page.getByRole("dialog", { name: "Settings" }); + await expect(dialog).toBeVisible(); + await dialog.getByRole("button", { name: "Integrations" }).click(); + + await expect(dialog.getByText("Lark / Feishu CLI")).toBeVisible(); + await expect( + dialog.getByText("Install the official skill pack first"), + ).toBeVisible(); + + await dialog.getByRole("button", { name: "Install" }).click(); + await expect( + page.getByText("Installed 3 Lark/Feishu skills."), + ).toBeVisible(); + + // Sandbox-runtime readiness row surfaces once the init-container runtime is + // reported ready, so a green UI can't hide a chat-time command-not-found. + await expect(dialog.getByText("Sandbox runtime")).toBeVisible(); + await expect( + dialog.getByText("Provisioned by init container"), + ).toBeVisible(); + + await dialog.getByRole("button", { name: "Calendar" }).click(); + await dialog + .getByLabel("Exact OAuth scope") + .fill("calendar:calendar.event:read"); + await dialog.getByRole("button", { name: "Connect Lark" }).click(); + await expect(dialog.getByText("about:blank")).toBeVisible(); + await expect(dialog.getByText(/app configuration/i)).toHaveCount(0); + + await dialog + .getByRole("button", { + name: "I completed browser confirmation, continue", + }) + .click(); + await expect + .poll(() => authStartRequest) + .toMatchObject({ + recommend: false, + domains: ["calendar"], + scope: "calendar:calendar.event:read", + }); + + await expect + .poll(() => authCompleteRequests) + .toContainEqual({ + device_code: "mock-device-code", + wait_timeout_seconds: 8, + }); + await expect( + dialog.getByText("Lark authorization is live-verified"), + ).toBeVisible(); + await expect( + page.getByText("Authorization page opened. Waiting for completion..."), + ).toHaveCount(0); + + await dialog.getByRole("button", { name: "Calendar" }).click(); + await dialog.getByLabel("Exact OAuth scope").fill(""); + await dialog.getByRole("button", { name: "Reconnect Lark" }).click(); + await expect( + dialog.getByText("https://open.feishu.cn/auth/mock-device"), + ).toBeVisible(); + }); +}); diff --git a/frontend/tests/e2e/utils/mock-api.ts b/frontend/tests/e2e/utils/mock-api.ts index 95cec6651..300935787 100644 --- a/frontend/tests/e2e/utils/mock-api.ts +++ b/frontend/tests/e2e/utils/mock-api.ts @@ -258,6 +258,42 @@ export function mockLangGraphAPI(page: Page, options?: MockAPIOptions) { max_file_size: 50 * 1024 * 1024, max_total_size: 100 * 1024 * 1024, }; + let larkIntegrationStatus = { + installed: false, + version: "v1.0.65", + manifest_version: null as string | null, + latest_available_version: "v1.0.65" as string | null, + runtime_version_mismatch: false, + app_configured: false, + app_id: null as string | null, + app_brand: null as string | null, + skills_expected: 27, + skills_installed: 0, + installed_skills: [] as string[], + enabled_skills: [] as string[], + install_path: "/tmp/deer-flow/integrations/skills/lark-cli", + cli: { + available: false, + path: null as string | null, + version: null as string | null, + error: "lark-cli is not on PATH" as string | null, + }, + auth: { + status: "unavailable", + message: "lark-cli is not installed on the Gateway" as string | null, + user: null as string | null, + verified: false, + }, + sandbox_runtime_mode: "init-container" as + | "none" + | "gateway-download" + | "init-container", + sandbox_runtime_ready: false, + sandbox_runtime_detail: + "The provisioner has no lark-cli init image configured (LARK_CLI_INIT_IMAGE)." as + | string + | null, + }; const featureFlags = { agentsApiEnabled: options?.features?.agentsApiEnabled ?? true, browserControlEnabled: options?.features?.browserControlEnabled ?? true, @@ -1066,6 +1102,149 @@ export function mockLangGraphAPI(page: Page, options?: MockAPIOptions) { return route.fallback(); }); + void page.route("**/api/integrations/lark/status", (route) => { + if (route.request().method() === "GET") { + return route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify(larkIntegrationStatus), + }); + } + return route.fallback(); + }); + + void page.route("**/api/integrations/lark/install", (route) => { + if (route.request().method() === "POST") { + larkIntegrationStatus = { + installed: true, + version: "v1.0.65", + manifest_version: "v1.0.65", + latest_available_version: "v1.0.65", + runtime_version_mismatch: false, + app_configured: false, + app_id: null, + app_brand: null, + skills_expected: 27, + skills_installed: 3, + installed_skills: ["lark-doc", "lark-im", "lark-shared"], + enabled_skills: ["lark-doc", "lark-im", "lark-shared"], + install_path: "/tmp/deer-flow/integrations/skills/lark-cli", + cli: { + available: true, + path: "/usr/bin/lark-cli", + version: "lark-cli version v1.0.65", + error: null, + }, + auth: { + status: "not_configured", + message: "lark-cli auth is not configured", + user: null, + verified: false, + }, + sandbox_runtime_mode: "init-container", + sandbox_runtime_ready: true, + sandbox_runtime_detail: null, + }; + return route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify({ + success: true, + installed_skills: ["lark-doc", "lark-im", "lark-shared"], + message: "Installed 3 Lark/Feishu skills.", + status: larkIntegrationStatus, + }), + }); + } + return route.fallback(); + }); + + void page.route("**/api/integrations/lark/config/start", (route) => { + if (route.request().method() === "POST") { + return route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify({ + verification_url: "https://open.feishu.cn/page/cli?user_code=config", + device_code: "mock-config-device-code", + expires_in: 600, + interval: 5, + user_code: "config", + brand: "feishu", + }), + }); + } + return route.fallback(); + }); + + void page.route("**/api/integrations/lark/config/complete", (route) => { + if (route.request().method() === "POST") { + larkIntegrationStatus = { + ...larkIntegrationStatus, + app_configured: true, + app_id: "cli_mock", + app_brand: "feishu", + auth: { + status: "not_authorized", + message: "Lark user authorization is not configured", + user: null, + verified: false, + }, + }; + return route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify({ + success: true, + message: "Lark/Feishu connection setup completed.", + status: larkIntegrationStatus, + }), + }); + } + return route.fallback(); + }); + + void page.route("**/api/integrations/lark/auth/start", (route) => { + if (route.request().method() === "POST") { + return route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify({ + verification_url: "https://open.feishu.cn/auth/mock-device", + device_code: "mock-device-code", + expires_in: 600, + user_code: null, + hint: null, + }), + }); + } + return route.fallback(); + }); + + void page.route("**/api/integrations/lark/auth/complete", (route) => { + if (route.request().method() === "POST") { + larkIntegrationStatus = { + ...larkIntegrationStatus, + auth: { + status: "authenticated", + message: "Lark/Feishu authorization is live-verified.", + user: "Alice", + verified: true, + }, + }; + return route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify({ + success: true, + message: "Lark/Feishu authorization completed.", + status: larkIntegrationStatus, + }), + }); + } + return route.fallback(); + }); + // Follow-up suggestions — input box auto-suggest after AI response void page.route("**/api/threads/*/suggestions", (route) => { if (route.request().method() === "POST") { diff --git a/frontend/tests/unit/components/workspace/settings/settings-dialog-store.test.ts b/frontend/tests/unit/components/workspace/settings/settings-dialog-store.test.ts new file mode 100644 index 000000000..b76215b4a --- /dev/null +++ b/frontend/tests/unit/components/workspace/settings/settings-dialog-store.test.ts @@ -0,0 +1,57 @@ +import { afterEach, expect, test } from "@rstest/core"; + +import { + getSettingsDialogSnapshot, + openSettingsDialog, + setSettingsDialogOpen, + subscribeSettingsDialog, +} from "@/components/workspace/settings/settings-dialog-store"; + +afterEach(() => { + // Reset shared module state between tests. + setSettingsDialogOpen(false); +}); + +test("starts closed on the default section", () => { + expect(getSettingsDialogSnapshot()).toEqual({ + open: false, + section: "appearance", + }); +}); + +test("openSettingsDialog opens on the requested section", () => { + openSettingsDialog("integrations"); + expect(getSettingsDialogSnapshot()).toEqual({ + open: true, + section: "integrations", + }); +}); + +test("setSettingsDialogOpen(false) keeps the last section", () => { + openSettingsDialog("channels"); + setSettingsDialogOpen(false); + expect(getSettingsDialogSnapshot()).toEqual({ + open: false, + section: "channels", + }); +}); + +test("notifies subscribers only on real state changes", () => { + let notifications = 0; + const unsubscribe = subscribeSettingsDialog(() => { + notifications += 1; + }); + + openSettingsDialog("integrations"); + // Opening again on the same section is a no-op and must not re-notify. + openSettingsDialog("integrations"); + expect(notifications).toBe(1); + + openSettingsDialog("memory"); + expect(notifications).toBe(2); + + unsubscribe(); + openSettingsDialog("about"); + // No further notifications after unsubscribe. + expect(notifications).toBe(2); +}); diff --git a/frontend/tests/unit/core/integrations/lark/api.test.ts b/frontend/tests/unit/core/integrations/lark/api.test.ts new file mode 100644 index 000000000..816e05981 --- /dev/null +++ b/frontend/tests/unit/core/integrations/lark/api.test.ts @@ -0,0 +1,309 @@ +import { beforeEach, describe, expect, rs, test } from "@rstest/core"; + +rs.mock("@/core/api/fetcher", () => ({ + fetch: rs.fn(), +})); + +rs.mock("@/core/config", () => ({ + getBackendBaseURL: () => "/backend", +})); + +import { fetch as fetcher } from "@/core/api/fetcher"; +import { + completeLarkAuthorization, + completeLarkConfiguration, + installLarkIntegration, + LarkIntegrationRequestError, + loadLarkIntegrationStatus, + startLarkAuthorization, + startLarkConfiguration, +} from "@/core/integrations/lark/api"; + +const mockedFetch = rs.mocked(fetcher); + +function jsonResponse(status: number, body: unknown): Response { + return new Response(JSON.stringify(body), { + status, + statusText: status >= 400 ? "Bad Request" : "OK", + headers: { "Content-Type": "application/json" }, + }); +} + +beforeEach(() => { + mockedFetch.mockReset(); +}); + +describe("lark integration api", () => { + test("loads status", async () => { + mockedFetch.mockResolvedValueOnce( + jsonResponse(200, { + installed: false, + version: "v1.0.65", + manifest_version: null, + latest_available_version: "v1.0.65", + runtime_version_mismatch: false, + app_configured: false, + app_id: null, + app_brand: null, + skills_expected: 27, + skills_installed: 0, + installed_skills: [], + enabled_skills: [], + install_path: "/tmp/lark-cli", + cli: { available: false, path: null, version: null, error: "missing" }, + auth: { status: "unavailable", message: "missing", user: null }, + sandbox_runtime_mode: "init-container", + sandbox_runtime_ready: false, + sandbox_runtime_detail: "init image not configured", + }), + ); + + await expect(loadLarkIntegrationStatus()).resolves.toMatchObject({ + installed: false, + version: "v1.0.65", + sandbox_runtime_mode: "init-container", + sandbox_runtime_ready: false, + }); + expect(mockedFetch).toHaveBeenCalledWith( + "/backend/api/integrations/lark/status", + ); + }); + + test("installs integration", async () => { + mockedFetch.mockResolvedValueOnce( + jsonResponse(200, { + success: true, + installed_skills: ["lark-doc"], + message: "Installed 1 Lark/Feishu skills.", + status: { + installed: true, + version: "v1.0.65", + manifest_version: "v1.0.65", + latest_available_version: "v1.0.65", + runtime_version_mismatch: false, + app_configured: false, + app_id: null, + app_brand: null, + skills_expected: 27, + skills_installed: 1, + installed_skills: ["lark-doc"], + enabled_skills: ["lark-doc"], + install_path: "/tmp/lark-cli", + cli: { + available: true, + path: "/usr/bin/lark-cli", + version: "v1.0.65", + error: null, + }, + auth: { + status: "not_configured", + message: "not configured", + user: null, + }, + }, + }), + ); + + await expect(installLarkIntegration()).resolves.toMatchObject({ + success: true, + installed_skills: ["lark-doc"], + }); + expect(mockedFetch).toHaveBeenCalledWith( + "/backend/api/integrations/lark/install", + { method: "POST" }, + ); + }); + + test("surfaces admin-required install errors", async () => { + mockedFetch.mockResolvedValueOnce( + jsonResponse(403, { detail: "Admin privileges required." }), + ); + + const promise = installLarkIntegration(); + await expect(promise).rejects.toMatchObject({ + name: "LarkIntegrationRequestError", + status: 403, + isAdminRequired: true, + message: "Admin privileges required.", + }); + await expect(promise).rejects.toBeInstanceOf(LarkIntegrationRequestError); + }); + + test("starts browser authorization", async () => { + mockedFetch.mockResolvedValueOnce( + jsonResponse(200, { + verification_url: "https://open.feishu.cn/auth/mock", + device_code: "device-code", + expires_in: 600, + user_code: null, + hint: null, + }), + ); + + await expect( + startLarkAuthorization({ + recommend: true, + domains: ["calendar"], + scope: "calendar:calendar.event:read", + }), + ).resolves.toEqual({ + verification_url: "https://open.feishu.cn/auth/mock", + device_code: "device-code", + expires_in: 600, + user_code: null, + hint: null, + }); + expect(mockedFetch).toHaveBeenCalledWith( + "/backend/api/integrations/lark/auth/start", + { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + recommend: true, + domains: ["calendar"], + scope: "calendar:calendar.event:read", + }), + }, + ); + }); + + test("starts connection setup", async () => { + mockedFetch.mockResolvedValueOnce( + jsonResponse(200, { + verification_url: "https://open.feishu.cn/page/cli?user_code=config", + device_code: "config-device-code", + expires_in: 600, + interval: 5, + user_code: "config", + brand: "feishu", + }), + ); + + await expect(startLarkConfiguration({ brand: "feishu" })).resolves.toEqual({ + verification_url: "https://open.feishu.cn/page/cli?user_code=config", + device_code: "config-device-code", + expires_in: 600, + interval: 5, + user_code: "config", + brand: "feishu", + }); + expect(mockedFetch).toHaveBeenCalledWith( + "/backend/api/integrations/lark/config/start", + { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ brand: "feishu" }), + }, + ); + }); + + test("completes connection setup", async () => { + mockedFetch.mockResolvedValueOnce( + jsonResponse(200, { + success: true, + message: "Lark/Feishu connection setup completed.", + status: { + installed: true, + version: "v1.0.65", + manifest_version: "v1.0.65", + latest_available_version: "v1.0.65", + runtime_version_mismatch: false, + app_configured: true, + app_id: "cli_mock", + app_brand: "feishu", + skills_expected: 27, + skills_installed: 1, + installed_skills: ["lark-doc"], + enabled_skills: ["lark-doc"], + install_path: "/tmp/lark-cli", + cli: { + available: true, + path: "/usr/bin/lark-cli", + version: "v1.0.65", + error: null, + }, + auth: { + status: "not_authorized", + message: "not authorized", + user: null, + }, + }, + }), + ); + + await expect( + completeLarkConfiguration({ + device_code: "config-device-code", + brand: "feishu", + interval: 5, + expires_in: 600, + }), + ).resolves.toMatchObject({ + success: true, + status: { app_configured: true, app_id: "cli_mock" }, + }); + expect(mockedFetch).toHaveBeenCalledWith( + "/backend/api/integrations/lark/config/complete", + { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + device_code: "config-device-code", + brand: "feishu", + interval: 5, + expires_in: 600, + }), + }, + ); + }); + + test("completes browser authorization", async () => { + mockedFetch.mockResolvedValueOnce( + jsonResponse(200, { + success: true, + message: "Lark/Feishu authorization completed.", + status: { + installed: true, + version: "v1.0.65", + manifest_version: "v1.0.65", + latest_available_version: "v1.0.65", + runtime_version_mismatch: false, + app_configured: true, + app_id: "cli_mock", + app_brand: "feishu", + skills_expected: 27, + skills_installed: 1, + installed_skills: ["lark-doc"], + enabled_skills: ["lark-doc"], + install_path: "/tmp/lark-cli", + cli: { + available: true, + path: "/usr/bin/lark-cli", + version: "v1.0.65", + error: null, + }, + auth: { + status: "authenticated", + message: "ok", + user: "Alice", + }, + }, + }), + ); + + await expect( + completeLarkAuthorization({ device_code: "device-code" }), + ).resolves.toMatchObject({ + success: true, + status: { auth: { status: "authenticated", user: "Alice" } }, + }); + expect(mockedFetch).toHaveBeenCalledWith( + "/backend/api/integrations/lark/auth/complete", + { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ device_code: "device-code" }), + }, + ); + }); +});