Zeren Wang a58ab484a6
feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash (#5443)
* feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash

Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md
(issue #5160, tracker #5129) in the slice order of the spec's §16.

Slices:
- A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY
  admission pinning (both server-owned sets + worker hoist); latest-only
  request-scoped <project> block via DynamicContextMiddleware
  wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID
  prefix + marker + provenance, never persisted); journal audit
  fingerprints; Instructions tab.
- B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with
  locked check-and-set; hash-qualified immutable shelf storage with
  Paths helpers; upload/list/content/delete-to-trash routes; project
  delete trashes the shelf in-transaction; request-scoped bounded
  <documents> index with honest count/shown + actionable overflow note;
  list_project_documents/read_project_document tools registered only on
  pinned runs; PAT allowlist + drift guards; blocking-IO anchors.
- C: shared thread-upload ingestion service (uploads router refactored to
  parity); POST from-thread with provenance; attach-to-thread with
  lock-staged copy (archived source allowed); read-only thread-files
  view with per-group truncation reporting.
- D: restore (restored/merged/not_found/no_target/content_missing; no
  file moves), purge (continuous row lock across unlink/delete/commit,
  retryable on FS errors), retention sweep (lazy + startup, 24h orphan
  guard, row-side reconciliation never deletes).
- E: Documents tab (shelf + conversation-files browser, provenance,
  archived banner, content-missing rows), /workspace/trash route,
  sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks
  + specs.

Review hardening folded in (10 rounds, all with tests):
- force active shelf content (HTML/XML family) to download; nosniff on
  artifact + content responses; unified unsandboxed-iframe PDF preview
  (fixes the pre-existing Chromium sandbox blank in the artifact viewer)
- scope document trash to the URL project under the document lock
- atomic no-overwrite filename reservation for ALL ingestion (seeded
  claims + os.link commit with suffix retry; same-name re-upload now
  unique-names instead of replacing); hidden staging only, no visible
  placeholders; lease cleanup on setup failure
- serialize conversion under the document lock with post-lock active
  revalidation; drain locked filesystem work on cancellation; preserve
  bytes when an insert's commit state is uncertain (including trashed
  rows)
- original-integrity checks before serving text or cached conversions;
  content_missing surfaced in list responses (UI reads the flag, no
  409-probe); downloads always serve original bytes
- bounded streaming document reads with cached char counts; shelf limits
  declared in middleware release identity
- thread-root confinement for from-thread sources; config fallback
  rejects fractional/infinite values; composer counts staged
  attachments; pending attachments persist until submission or removal;
  in-flight instruction/rename edits survive save refetches; shelf and
  trash pagination; conversation-file and thread-files pages stay
  subscribed to refetches

Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md
contracts, config.example.yaml projects block.

Review follow-ups (head b4807477 → this revision):
- The trash retention sweep is split so repeated lazy triggers stay
  bounded: the indexed expiry purge still runs on every trigger
  (GET /api/trash/documents, POST /api/trash/purge) while the
  O(all rows + all files) reconciliation is throttled to one run per
  user per 15 minutes (process-local, per-user window). The startup
  sweep now runs as a background task instead of blocking gateway
  readiness, and shutdown awaits it (bounded).
- The export scrub (stripInternalMarkers) is fence- and indentation-aware
  like the render path, so a pasted, fenced <project>/<documents> snippet
  survives markdown export while real injected blocks (never fenced) are
  still removed. Fence regexes moved to a dependency-free leaf module to
  avoid the messages↔streamdown import cycle.
- The artifact viewer's PDF iframe no longer carries an added title
  attribute (the upstream e2e contract locates it via :not([title])), and
  the upstream artifact-preview spec now pins the new contract: PDFs
  render unsandboxed, images keep sandbox="".

* fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title

- Shutdown cancelled only the shield around the background startup sweep,
  so an all-users reconciliation that outlived the 5s budget kept walking
  rows and files while the document repo and DB engine were disposed
  underneath it. The wait now lives in `_shutdown_startup_trash_sweep`,
  which cancels the task and drains it before worker exit: the shield
  keeps the wait bounded, the cancel makes it final (CancelledError lands
  at the sweep's next await, and `_run_startup_trash_sweep` only catches
  `Exception`, so nothing swallows it).
- The browser-preview iframe lost `title={getFileName(filepath)}` in the
  previous fix round, leaving the PDF frame without an accessible name
  while its siblings keep theirs. Restore it (WCAG frame titles), assert
  it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]`
  instead of `iframe:not([title])`.

* fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel

`Task.cancel()` returns False when the sweep already finished inside the
window between the deadline firing and the cancel, so the shutdown log
claimed a cancellation that never happened. Branch on that outcome: the
warning stays for a real cancel, a late finish is logged at info, and both
paths still reap the task before worker exit.

* fix(projects): round-4 review — make Empty trash delete what it confirms

`POST /api/trash/purge` only ran the retention sweep, and the sweep's
candidate selection is age-gated, so a freshly trashed document survived
"Empty trash" even though the confirmation promises that every listed
document is permanently deleted. With one trashed row the route answered
`{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps
expired rows before listing, so the visible rows were normally ineligible
for the action by construction.

Empty trash now drives `purge_all_trashed`: the caller's trashed rows
(`list_all_trashed`, no age filter) each go through the same guarded,
row-locked `purge` as the single-document delete — bytes first, then the
row, in one transaction — so a row restored mid-flight is skipped instead of
force-deleted, and an unlink failure rolls that row back and answers 500 with
a retryable message. Retention expiry stays where it was: the sweep's
`purge_candidates` is now the only age-gated selection, and the lazy
retention sweep still runs on the listing and at startup.

Tests: the router suite replaces the retention-gated expectation with the
reviewer's repro (fresh row purged, bytes unlinked, shelf and other users'
trash untouched, a failing unlink stays retryable and 500); a blocking-I/O
anchor drives the new entry point through the offload; the mocked e2e covers
the action end to end; a new real-backend spec performs it against the real
gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE
and the phase-2 design docs (en+zh) state the age-independent contract.
2026-09-16 18:46:18 +08:00

404 lines
16 KiB
YAML

# DeerFlow Helm chart values
#
# Copy to my-values.yaml, edit, and install with:
# helm install deer-flow deploy/helm/deer-flow -n deer-flow --create-namespace -f my-values.yaml
# -- Target namespace (also used as the K8s sandbox namespace for provisioner-spawned Pods).
namespace: deer-flow
# -- Image registry & tag for the three DeerFlow images you build+push.
# Required: set `registry` to your registry (e.g. ghcr.io/yourorg).
image:
registry: "" # REQUIRED, e.g. ghcr.io/yourorg
tag: "latest"
pullPolicy: IfNotPresent
# -- Existing image pull secrets, e.g. [{ name: regcred }]
pullSecrets: []
# Image names match what .github/workflows/container.yaml publishes on GHCR
# as ${repository}-<name> (e.g. ghcr.io/<owner>/deer-flow-backend). Set
# `registry` to the owner prefix (e.g. ghcr.io/<owner>) and `tag` to consume
# the published images.
gatewayImage: deer-flow-backend
frontendImage: deer-flow-frontend
provisionerImage: deer-flow-provisioner
# -- Gateway (backend) deployment.
gateway:
replicas: 1 # Safe default. Postgres + the Redis stream bridge are
# wired, but multi-replica needs issue #3948's run-control
# work (cancel/dedup/reconcile). Scheduler multi_instance
# recovery is safe only when its config prerequisites are met;
# max_concurrent_runs is shared across Pods in that mode.
# Pod termination grace period (seconds). MUST exceed the Gateway's graceful
# shutdown work: channel stop (~5s) + memory queue drain
# (memory.shutdown_flush_timeout_seconds, default 30s) + buffer. K8s defaults
# to 30s, which SIGKILLs the memory drain mid-flight and silently re-introduces
# the memory loss that shutdown_flush is fixing. Raise this when you raise the
# memory timeout. See gateway-deployment.yaml + README "Graceful shutdown".
terminationGracePeriodSeconds: 45
# preStop hook sleep (seconds) before SIGTERM, so the Service/ingress
# deregisters the pod and in-flight HTTP requests drain before graceful
# shutdown begins (which runs the memory queue drain). 0 disables the hook.
# Counts against terminationGracePeriodSeconds.
preStopSleepSeconds: 5
resources:
requests:
cpu: 200m
memory: 512Mi
limits:
cpu: "2"
memory: 2Gi
# -- Frontend (Next.js) deployment.
frontend:
replicas: 1
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 512Mi
# -- nginx reverse-proxy deployment (preserves compose routing).
nginx:
replicas: 1
image:
repository: nginx
tag: alpine
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 500m
memory: 256Mi
service:
# -- Service type fronting nginx. `LoadBalancer` on Docker Desktop / kind /
# OrbStack routes to localhost (no Ingress controller needed). `ClusterIP`
# pairs with the Ingress resource. `NodePort` exposes on a node port.
type: ClusterIP
port: 2026
loadBalancerIP: ""
nodePort: ""
# -- Sandbox provisioner (K8s-native code execution). Creates sandbox Pods in
# this namespace via a ServiceAccount + RBAC.
provisioner:
enabled: true
resources:
requests:
cpu: 50m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
# -- AIO-compatible sandbox container image used for sandbox Pods.
sandboxImage: "enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest"
# -- Service type for per-sandbox Services. `ClusterIP` (default) exposes
# sandboxes only inside the cluster via Service DNS - correct for the
# in-cluster gateway and avoids binding the code-execution surface on
# every node's interfaces. `NodePort` is the Docker-Compose/hybrid escape
# hatch (used when the gateway is not in K8s); pair with `nodeHost`.
sandboxServiceType: "ClusterIP"
# -- Host the gateway uses to reach sandbox NodePorts. Only consulted when
# `sandboxServiceType: NodePort`. Empty (default) falls back to the
# provisioner pod's node IP via the Kubernetes downward API, which routes
# on most clusters because a NodePort is exposed on every node. Set
# explicitly only when pod->node-IP traffic is blocked by your
# CNI/network policy. In Docker Compose this is `host.docker.internal`.
nodeHost: ""
# -- Sandbox container port (must match the sandboxImage's listening port).
sandboxPort: 8080
# -- PostgreSQL database. Bundled mode (default) deploys a single-instance
# postgres StatefulSet; set `enabled: false` to use an external managed DB.
# The gateway reads DATABASE_URL from the resolved Secret and config.yaml
# references it as $DATABASE_URL in database.postgres_url.
postgresql:
# -- Deploy a bundled postgres StatefulSet. Disable to use an external DB.
enabled: true
image:
repository: postgres
tag: "16"
auth:
database: deerflow
username: deerflow
# -- Password auto-generated (32 chars, persisted across upgrades) when
# empty. Ignored if existingSecret is set.
password: ""
# -- Use an existing Secret (key `database-url`, plus `postgres-password`
# in bundled mode) instead of generating one. Skips Secret creation.
existingSecret: ""
primary:
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi
persistence:
enabled: true
storageClass: "" # "" = cluster default
accessMode: ReadWriteOnce # RWX (NFS) only needed if postgres itself is HA
size: 20Gi
# -- External postgres (used when `enabled: false`). Provide either a full
# `databaseUrl` (chart wraps it into a Secret for you) or an
# `existingSecret` you manage (key `database-url`).
external:
# -- Full DSN, e.g. postgresql://user:pass@host:5432/deerflow. Chart
# writes it into the Secret so it never touches the gateway env directly.
databaseUrl: ""
# -- Secret you manage (key `database-url`). Use this with External Secrets
# / Vault / Sealed Secrets so the DSN never appears in values files.
existingSecret: ""
# -- Redis stream bridge. The gateway stores per-run SSE events in Redis Streams
# so live run events reach a client connected to any gateway pod (cross-pod
# SSE delivery + reconnect replay, PR #3191). Bundled mode (default) deploys
# a single-instance redis StatefulSet; set `enabled: false` to use an external
# managed Redis. The gateway reads DEER_FLOW_STREAM_BRIDGE_REDIS_URL from the
# resolved Secret; config.yaml's stream_bridge.type is `redis` by default.
redis:
# -- Deploy a bundled redis StatefulSet. Disable to use an external Redis.
enabled: true
image:
repository: redis
tag: "7-alpine"
auth:
# -- Empty (default) = no auth, matching the compose deployment (ClusterIP
# isolation only). Set a password to enable AUTH; the DSN becomes
# redis://:<password>@host:6379/0. Ignored if existingSecret is set.
password: ""
# -- Use an existing Secret (key `redis-url`, plus `redis-password` if auth)
# instead of generating one. Skips Secret creation.
existingSecret: ""
primary:
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: "1"
memory: 512Mi
persistence:
enabled: true
storageClass: "" # "" = cluster default
accessMode: ReadWriteOnce # RWX (NFS) only needed if redis itself is HA
size: 5Gi
# -- External redis (used when `enabled: false`). Provide either a full
# `redisUrl` (chart wraps it into a Secret for you) or an `existingSecret`
# you manage (key `redis-url`).
external:
# -- Full URL, e.g. redis://:pass@myredis.example:6379/0. Chart writes it
# into the Secret so it never touches the gateway env directly.
redisUrl: ""
# -- Secret you manage (key `redis-url`). Use with External Secrets / Vault
# / Sealed Secrets so the URL never appears in values files.
existingSecret: ""
# -- Persistent volume for runtime state (.deer-flow): sqlite DB, memory,
# custom agents, mutable extensions_config.json, and per-thread user-data.
# Also mounted (PVC mode) into provisioner-spawned sandbox Pods. When
# disabled, Gateway runtime state uses a pod-local emptyDir instead.
persistence:
home:
enabled: true
storageClass: "" # "" = cluster default
accessMode: ReadWriteOnce # Use ReadWriteMany (NFS/etc.) for multi-node
size: 10Gi
# -- Skills library mounted at /app/skills. Default: emptyDir (skills disabled).
# Populate via an existing PVC or ConfigMap, or bake skills into a custom
# gateway image. Provisioner PVC mode references this same claim when set.
skills:
enabled: false
existingClaim: ""
configMap: ""
# -- Provider/channel/search secrets injected as env vars into the gateway.
# Reference them from config.yaml as $VAR. Example:
# secrets:
# OPENAI_API_KEY: "sk-..."
# FEISHU_APP_ID: "cli_xxx"
# FEISHU_APP_SECRET: "xxx"
# GITHUB_TOKEN: "ghp_xxx"
secrets: {}
# -- Use an existing Secret instead of creating one from `secrets` above.
existingSecret: ""
# -- Ingress in front of nginx (port 2026). nginx preserves all internal routing.
ingress:
enabled: true
className: "nginx"
host: "deer-flow.example.com"
annotations:
# Allows a 100 MiB .skill archive plus multipart framing at the outer ingress.
nginx.ingress.kubernetes.io/proxy-body-size: "101m"
# Streams request bodies to nginx instead of spooling them at the outer ingress.
nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
# Skill validation may perform multiple sequential LLM calls.
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
tls:
enabled: false
secretName: ""
# hosts: [] # defaults to [ingress.host]
# -- DeerFlow config.yaml content. Secrets MUST stay as $VAR references — never
# inline literal secret values here. The default enables provisioner sandbox.
config: |
config_version: 45
log_level: info
recursion_limit: 100
max_recursion_limit: 1000
models: []
# Example (uncomment & set the matching secret in `secrets`):
# - name: gpt-4
# display_name: GPT-4
# use: langchain_openai:ChatOpenAI
# model: gpt-4
# api_key: $OPENAI_API_KEY
# request_timeout: 600.0
sandbox:
use: deerflow.community.aio_sandbox:AioSandboxProvider
provisioner_url: http://provisioner:8002
image: enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest
port: 8080
replicas: 3
database:
# PostgreSQL is the default backend (bundled StatefulSet, or external).
# DATABASE_URL is injected from the postgres Secret; $DATABASE_URL is
# resolved by the harness before the config is instantiated.
backend: postgres
postgres_url: $DATABASE_URL
pool_recycle: 300
command_timeout: 30
# The LangGraph Store (cross-thread memory + thread list) reads this legacy
# `checkpointer:` section — it does NOT fall back to `database:` the way the
# checkpointer does. Point it at the same postgres so the Store is shared
# across gateway pods (required for multi-replica operation).
checkpointer:
type: postgres
connection_string: $DATABASE_URL
# Stream bridge: `redis` (default) stores per-run SSE events in Redis Streams
# so live run events reach a client on any gateway pod (cross-pod SSE delivery
# + reconnect replay, PR #3191). The URL is read from the
# DEER_FLOW_STREAM_BRIDGE_REDIS_URL env var injected from the redis Secret.
# Set `type: memory` (and disable the redis chart) for single-pod-only mode.
stream_bridge:
type: redis
memory:
storage_path: memory.json
verification:
receipts_enabled: true
receipts_render_mode: "delegation_only"
judge_enabled: false
judge_model_name: null
# -- Tools configuration. The agent gets NO tools unless they're listed here
# (BUILTIN_TOOLS only adds present_file + ask_clarification). The file/bash
# tools run inside the AIO sandbox configured above. The web tools
# (web_search, web_fetch, image_search) need no API key but require outbound
# internet from the gateway pod - swap backends or remove entries for
# air-gapped clusters (see config.example.yaml).
tool_groups:
- name: web
- name: file:read
- name: file:write
- name: bash
# - name: knowledge # Enable with the RAGFlow tool below.
tools:
# Optional tenant-shared, read-only RAGFlow retrieval. Put RAGFLOW_API_KEY
# in `secrets`. `datasets` is an optional stable-ID allowlist; omit it to
# search all tenant-visible datasets. An explicit empty list is invalid.
# Empty datasets are skipped; remaining
# datasets are grouped by embedding model with up to four parallel retrieval
# requests and one global `page_size` limit. Multi-group scores are omitted
# because they are not comparable. The Agent cannot see the IDs.
# - name: knowledge_search
# group: knowledge
# use: deerflow.community.ragflow.tools:knowledge_search_tool
# base_url: http://ragflow:9380
# api_key: $RAGFLOW_API_KEY
# datasets: # Optional operator-controlled allowlist
# - 0123456789abcdef0123456789abcdef
# timeout: 30
# page_size: 8
# similarity_threshold: 0.2
# vector_similarity_weight: 0.3
# top_k: 256
# max_chars_per_chunk: 800
# max_total_chars: 8000
# Optional tenant-shared, read-only LightRAG retrieval; alternative
# provider for the same knowledge_search tool (duplicate names keep the
# first entry, so configure exactly one). Requires LightRAG v1.4.9+. Put
# LIGHTRAG_API_KEY in `secrets` when the server enables API-key auth;
# omit `api_key` only for unauthenticated trusted-network deployments.
# Searches the single indexed workspace through the data-retrieval
# endpoint with no LLM generation inside LightRAG. Internal chunk and
# reference identifiers stay hidden from the Agent.
# - name: knowledge_search
# group: knowledge
# use: deerflow.community.lightrag.tools:knowledge_search_tool
# base_url: http://lightrag:9621
# api_key: $LIGHTRAG_API_KEY
# mode: mix
# timeout: 30
# top_k: 60
# chunk_top_k: 8 # Optional; omit to use the server default
# max_chars_per_chunk: 800
# max_total_chars: 8000
- name: web_search
group: web
use: deerflow.community.ddg_search.tools:web_search_tool
max_results: 5
- name: web_fetch
group: web
use: deerflow.community.jina_ai.tools:web_fetch_tool
timeout: 10
- name: image_search
group: web
use: deerflow.community.image_search.tools:image_search_tool
max_results: 5
- name: ls
group: file:read
use: deerflow.sandbox.tools:ls_tool
- name: read_file
group: file:read
use: deerflow.sandbox.tools:read_file_tool
- name: glob
group: file:read
use: deerflow.sandbox.tools:glob_tool
max_results: 200
- name: grep
group: file:read
use: deerflow.sandbox.tools:grep_tool
max_results: 100
- name: write_file
group: file:write
use: deerflow.sandbox.tools:write_file_tool
- name: str_replace
group: file:write
use: deerflow.sandbox.tools:str_replace_tool
- name: bash
group: bash
use: deerflow.sandbox.tools:bash_tool
# -- Initial DeerFlow extensions_config.json content (MCP servers + skill
# state). The Gateway copies this ConfigMap seed into its writable home
# volume only when no runtime file exists. With persistence enabled, API
# updates and the original seed survive pod replacement and Helm upgrades;
# remove the runtime file explicitly if a later seed should replace it.
extensionsConfig: |
{"mcpServers":{},"skills":{}}