mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-26 21:26:39 +00:00
* feat(workflows): add leased multi-Gateway coordination and fenced dispatch * test(workflows): validate deployed failover and canonicalize config revisions * fix(workflows): share namespace across registration and lease fencing * fix(workflows): recover interrupted cancellation without admission caps * fix(workflows): retain execution lease through child cleanup * docs: trim guidance padding to fit merged instruction budget * fix(workflows): release rejected admission slots and normalize gateway routing --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
519 lines
31 KiB
Markdown
519 lines
31 KiB
Markdown
# Script workflows (Phase 4)
|
||
|
||
An opt-in packaged extension that runs inspectable JavaScript orchestration in the
|
||
background. Scripts call full DeerFlow Agents through the public `AgentRuns`
|
||
capability. A chat tool or authenticated plugin action returns a workflow run ID
|
||
promptly; **Script workflows** shows the script, argument hashes, execution limits,
|
||
progress, child conversation/run identities, and final result.
|
||
|
||
Versioned definitions, retained output snapshots, explicit same-process pause/resume,
|
||
and authenticated human checkpoints support staged inspection reports. Agent success,
|
||
script verdicts, tool permission approval and human output approval remain separate facts.
|
||
Compatible workflows can be recovered after restart or, in PostgreSQL mode, on another Gateway after lease expiry.
|
||
No arbitrary JavaScript continuation serialization or hard monetary budget is provided.
|
||
|
||
## Install and enable
|
||
|
||
Requires a POSIX Gateway (Linux/macOS), Python 3.12+, the public extension API 0.2.9,
|
||
and a **single Gateway process per SQLite storage file**, or the shared PostgreSQL
|
||
mode described below. Windows execution is unsupported. SQLite holds an exclusive process lock; another process using that same file
|
||
fails service startup rather than recovering another worker's live workflows.
|
||
Use only a trusted operator-installed package.
|
||
|
||
Pinned QuickJS 1.19.4 provides CPython 3.12 wheels for macOS universal2 and Linux
|
||
x86_64. Python 3.13/3.14 and Linux arm64 installations build its source distribution:
|
||
install a C compiler and matching Python development headers first (for example,
|
||
Xcode Command Line Tools on macOS, or `build-essential` plus the matching
|
||
`python3-dev` package on Debian/Ubuntu). Source-build compatibility depends on your
|
||
Python/toolchain; use CPython 3.12 on a wheel-supported platform if the build fails.
|
||
This also applies to backend development dependency installation and arm64 containers.
|
||
|
||
From the repository root:
|
||
|
||
```sh
|
||
make extension-install SOURCE="$(pwd)/examples/deerflow-extension-workflows"
|
||
```
|
||
|
||
Set the installed plugin's operator-owned `config.yaml` entry:
|
||
|
||
```yaml
|
||
plugins:
|
||
- name: workflows
|
||
use: deerflow_extension_workflows:install
|
||
enabled: true
|
||
config:
|
||
enabled: true
|
||
storage_path: /absolute/deployment-owned/workflows.sqlite
|
||
agents: [lead_agent] # Add explicitly authorized existing Custom Agent names.
|
||
concurrency: 2
|
||
global_concurrency: 4
|
||
max_calls: 16
|
||
max_workflows: 4
|
||
max_workflows_per_owner: 2
|
||
timeout_seconds: 300
|
||
admission_timeout_seconds: 10
|
||
pause_timeout_seconds: 86400 # Separate bound for each human/pause wait, maximum 7 days.
|
||
cpu_seconds: 5
|
||
memory_bytes: 16777216
|
||
```
|
||
|
||
Find **Script workflows** in the workspace extension directory for the installation
|
||
guide. Its catalog listing does not indicate installation or activation; the operator
|
||
configuration above and Gateway runtime registration determine availability.
|
||
|
||
Restart Gateway. The namespace is `community.workflows`; the page is
|
||
`/workspace/extensions/community.workflows/runs`. The UI and tools are disabled by
|
||
default. `agents` restricts target selection in addition to normal host permissions;
|
||
it does not grant access to otherwise unauthorized models, tools or Agents.
|
||
Only supported host session/auth-disabled credentials supply a retained AgentRuns
|
||
handle; PAT/internal/channel callers cannot acquire one.
|
||
|
||
## Run an example
|
||
|
||
Open **Script workflows**, paste
|
||
[`examples/checks.workflow.js`](examples/checks.workflow.js), and enter:
|
||
|
||
```json
|
||
{"product":"Synthetic sample","formula":["ingredient A"],"claims":["sample claim"]}
|
||
```
|
||
|
||
Click **Launch workflow**. An accepted launch shows its run ID immediately;
|
||
if refreshing the list or details fails, use **Retry inspection**. Inspection retries
|
||
do not launch another workflow. Resubmitting unchanged script/arguments retains the
|
||
same request ID, including after a lost launch response; changing either creates a
|
||
new launch identity. The example sequences evidence collection, fans out two
|
||
checks, performs at most two review calls, and branches on a structured verdict.
|
||
It returns findings for human review, never a product-pass or approval claim.
|
||
This invokes your configured real Agents and can incur provider costs.
|
||
|
||
In chat, ask the lead to write an inspectable workflow script and launch it with
|
||
`launch_workflow`, retaining a stable `request_id`. Host tool names are derived
|
||
from the namespace. Use `inspect_workflow` on demand to inspect outcomes and
|
||
`cancel_workflow` to request cancellation; do not ask the lead to poll continuously.
|
||
The plugin page refreshes active progress every two seconds.
|
||
|
||
## Script API
|
||
|
||
A script is an async function **body**, not a Node module. No imports, module
|
||
loader, `process`, shell, filesystem, network, credentials, or Python callbacks
|
||
are exposed. Frozen `args` is a JSON object.
|
||
|
||
```js
|
||
const first = await agent({stepId:'first', agentRef:'lead_agent', task:args.task});
|
||
const checks = await parallel([
|
||
() => agent({stepId:'a',agentRef:'lead_agent',task:{evidence:first.output}}),
|
||
() => agent({stepId:'b',agentRef:'lead_agent',task:{evidence:first.output}})
|
||
]);
|
||
log('Two checks finished');
|
||
return checks.map(check => check.output);
|
||
```
|
||
|
||
`agent()` resolves to `{output, threadId, runId, status: 'success', artifactVersion}`. `output` is the
|
||
last attributable non-tool assistant answer: parsed JSON if valid, otherwise text
|
||
or content blocks. There is no provider-enforced structured output schema in this
|
||
API; validate verdicts in the script before branching. Every Agent call must be
|
||
awaited, including fan-out calls. Returning while child promises are outstanding
|
||
fails the workflow and interrupts tracked children.
|
||
|
||
Agent task inputs allow 16,000 serialized JSON bytes, including wrapping fields
|
||
and file references; step outputs allow 12,000. This leaves room to pass a saved
|
||
output to the next Agent with a short task instruction. JSON escaping counts
|
||
toward these limits. Larger material must use the file workflow described below.
|
||
|
||
`parallel()` takes lazy thunks and preserves order. The service, independently of
|
||
JavaScript promises, bounds admitted children per workflow and globally. A rejected
|
||
call rejects its promise; failures are not silently converted to null. Scripts may
|
||
catch ordinary failures and use explicit bounded revisions with **new** step IDs.
|
||
Limit faults, human-input waits, and uncertain admissions stop the supervisor even
|
||
if script code catches their promise rejection.
|
||
|
||
Step IDs contain 1–80 ASCII letters, digits, underscores or hyphens and cannot
|
||
repeat within a workflow. Attempts are currently always 1. `task` is JSON and
|
||
becomes an ordinary human message on a distinct, stably named owned child thread.
|
||
The host binds that thread's target Agent, checks current authorization on each
|
||
operation, and uses its normal agent assembly, models, tools, skills, memory and
|
||
checkpoints. The ledger records the requested agent reference and child IDs;
|
||
it does **not** claim to record a per-run effective assembly fingerprint.
|
||
|
||
## Saved versions, live resume and human review
|
||
|
||
Use **Workflow name → Save workflow version**, then select an exact saved version.
|
||
The immutable version is the SHA-256 of the script; identical saves are idempotent.
|
||
Definitions are owner-scoped (100 versions per owner). Launching the same saved version
|
||
with new arguments requires a new request ID and records both argument and script hashes.
|
||
Saved scripts never silently follow the latest version. Editing the script detaches
|
||
it from the selected definition. Chat tools `save_workflow` and `list_workflows`
|
||
provide the same library; `launch_workflow` accepts either `script` or
|
||
`definition: {name, version}` with `args` and `request_id`.
|
||
|
||
```js
|
||
const evidence = await agent({stepId:'evidence',agentRef:'lead_agent',task:args.product});
|
||
await humanCheckpoint({
|
||
stepId:'confirm-evidence', sourceStepId:'evidence', artifactVersion:evidence.artifactVersion
|
||
});
|
||
await pause({stepId:'before-report'});
|
||
return await agent({stepId:'report',agentRef:'lead_agent',task:{evidence:evidence.output}});
|
||
```
|
||
|
||
Checkpoints must be **awaited alone after all outstanding calls**. They share the
|
||
unique step-ID namespace; each attempt is 1 and has at most one decision. Up to
|
||
`max_calls` checkpoints are allowed in addition to the Agent call limit.
|
||
A pause retains the same isolated worker and its live JS variables. **Resume this
|
||
checkpoint** continues there, reusing completed values without reading newer thread
|
||
state or replaying earlier script code. It cannot retry a failed/cancelled workflow,
|
||
answer a child's graph interrupt or change script/arguments. After restart use
|
||
**Recover workflow** first, then resume or decide the retained checkpoint.
|
||
An old resume request names its exact step/attempt and cannot release a later pause.
|
||
Paused workers consume normal active workflow slots and memory; cancel unused waits.
|
||
The execution wall budget is suspended while waiting; each wait has its own
|
||
`pause_timeout_seconds` deadline (default 24 hours, maximum 7 days). CPU, call and
|
||
log limits remain cumulative; worker CPU limits apply per worker incarnation. Expiry fails closed and never approves anything.
|
||
|
||
The host records an opaque durable **declarative configuration revision**:
|
||
app configuration, extension configuration, and allowed Custom Agent config/SOUL.
|
||
It rechecks this revision before each new child and when continuing a checkpoint;
|
||
changes require a new workflow. Operator limits/allowlist changes also invalidate
|
||
continuation. This is deliberately conservative (even unrelated declarative config
|
||
changes invalidate), and **not** a full effective-assembly fingerprint: mutable
|
||
memory, referenced Skill/tool file contents, remote provider state and external
|
||
side effects are not attested. Completed results are historical snapshots, not a
|
||
claim that rerunning against changed external data would produce the same result.
|
||
Each new child still goes through ordinary current authorization and Agent assembly.
|
||
|
||
**Review exact output** displays the captured source-step output. Only then are
|
||
**Approve this version / Reject this version** offered. The host decision action
|
||
requires an authenticated browser session for the owner, exact checkpoint ID,
|
||
attempt and server-derived output hash. Model tools, internal/PAT callers and
|
||
auth-disabled sessions cannot approve. Auth-disabled deployments can run workflows
|
||
and ordinary pauses, but must enable login to use human checkpoints. Approvals
|
||
cannot be submitted through script messages; model-written “approved” has no effect.
|
||
Matching duplicate decisions return the saved receipt without duplicate dispatch;
|
||
a contradictory decision conflicts. Rejection stops downstream execution.
|
||
|
||
For files, use `captureArtifact({stepId, sourceStepId, path})` after the source
|
||
Agent completes. This stores independent bytes with SHA-256 and an immutable
|
||
version, rather than approving a mutable path. Pass the capture step/version to
|
||
`humanCheckpoint`. The browser reads that version, verifies its hash, and offers
|
||
safe text/image/PDF preview or an exact-version download before approval.
|
||
Completed checkpoint records retain this review/download entry after approval
|
||
and when reopened; only pending checkpoints offer decision controls.
|
||
A failed saved-definition lookup blocks launch until a version loads or the user
|
||
explicitly selects a new script or edits the editor.
|
||
`agent({..., artifacts:[version]})` admits only approved versions and includes
|
||
version/hash references in the child input. The child reads bytes through
|
||
`read_workflow_artifact`; scripts can use `readArtifact({artifactVersion, offset})`.
|
||
Both return base64 chunks with `next_offset` (null at EOF). Neither reopens the
|
||
source path. Limits: 2 MiB/file, 16 files/workflow, 64 MiB retained bytes/owner,
|
||
and 16 KiB per read. Deleting a retained workflow also deletes its snapshots.
|
||
See [`examples/file-review.workflow.js`](examples/file-review.workflow.js).
|
||
JSON/text output checkpoints remain supported; links inside them require explicit
|
||
file capture. No Agent success or reviewer verdict implicitly approves a version.
|
||
|
||
Each completed child also retains **host assembly evidence**, bound to its run ID.
|
||
The host records it before graph execution from resolved instructions, actual
|
||
model/parameters, authorized tools, Skills and middleware/policies. Only safe
|
||
numeric parameters and opaque keyed component versions are exposed, never raw
|
||
prompts, provider kwargs or credentials. `get_step` exposes this historical proof;
|
||
missing proof fails the workflow step. This is separate from the declarative
|
||
preflight guard above. Versions compare within one host lifetime; stored proof
|
||
survives restart. It describes assembly-time configuration, not later external
|
||
service state, runtime tool results or deterministic replay.
|
||
|
||
The UI exposes retained outputs one at a time to keep responses bounded. Checkpoint
|
||
receipts retain who decided, when, source attempt and output version. On restart
|
||
these records remain inspectable. Compatible runs become `recoverable`; a fresh
|
||
authenticated browser session must explicitly recover them. No credentials or
|
||
capability handles are serialized. Start new work only after considering prior effects.
|
||
|
||
## Limits and status
|
||
|
||
Limits are deployment-controlled, not writable by generated script or browser input.
|
||
Defaults are in the configuration above; ceilings are concurrency 4, global
|
||
concurrency 8, calls 32, active workflows 8 (4 per owner), wall time 600 seconds,
|
||
admission 20 seconds, cumulative worker CPU 10 seconds, JS memory 32 MiB.
|
||
|
||
Each JS engine entry has a 50 ms CPU interruption limit; stack is 256 KiB. The
|
||
isolated worker additionally has a cumulative POSIX CPU limit, no core dumps,
|
||
zero file-write size allowance, a temporary working directory and a minimal
|
||
credential-free environment. The parent kills/reaps it on termination. This is a
|
||
capability-restricted JS interpreter in a separate process, not a general OS
|
||
sandbox or a guarantee against native engine vulnerabilities.
|
||
|
||
Wall time (`timeout_seconds`) is the only clock bound on a run. A separate cap
|
||
of 10,000 worker turns stops scripts that spin the JS job queue; idle waits on
|
||
pending Agent calls do not exchange frames with the worker and do not consume
|
||
turns, so long child runs are limited only by the wall deadline.
|
||
The local wait checks worker exits every 20 ms alongside cancellation and faults;
|
||
an exit triggers prompt failure handling and cleanup of pending child runs.
|
||
|
||
IPC frames are bounded to 48 KiB. Script JSON encoding is at most 16,000 bytes,
|
||
arguments 4,000 bytes, task 8,000 bytes, each captured Agent output/final result
|
||
12,000 bytes. There are at most 20 progress messages of 160 JSON bytes each and
|
||
100 retained workflows per owner. Inspect responses are bounded to 60 KiB. Unicode
|
||
limits apply to the JSON encoding too, including escapes. Oversized results fail
|
||
explicitly instead of truncating evidence silently. Latest list returns 20 records;
|
||
keep a workflow run ID to inspect older retained records.
|
||
|
||
There is **no hard dollar-cost limit**. Call/time/CPU limits and ordinary host token
|
||
policies do not constitute a shared workflow currency budget. Unknown configuration
|
||
keys such as `max_cost_usd` are rejected. Add a generic, authoritative host budget
|
||
reservation/enforcement contract before advertising that guarantee.
|
||
|
||
| State | Meaning |
|
||
| --- | --- |
|
||
| queued / running | Supervised execution on the current execution owner |
|
||
| completed | Script and awaited child calls returned; output is unapproved task data |
|
||
| failed | Script, resource limit (including wall deadline), attribution, child execution or observation failed |
|
||
| cancelled | Stop requested and known active children interrupted or already terminal |
|
||
| paused / waiting_approval | Live worker retained at an explicit script checkpoint; occupies an active workflow slot |
|
||
| waiting_input | Child has a graph interrupt or human-input artifact; downstream stopped |
|
||
| needs_reconciliation | Admission/cancellation outcome uncertain; inspect child identities before new work |
|
||
| recoverable | Restart/shutdown retained a compatible execution; sign in and select Recover workflow |
|
||
| interrupted | Legacy record, cancelled startup record, or externally interrupted supervisor; inspect child effects |
|
||
|
||
Wall-clock expiry reports `failed` with `Execution deadline exceeded` and cleans
|
||
up tracked children and the worker. A timeout after child admission reports child
|
||
observation failure and interrupts/observes the known run; only uncertain cleanup
|
||
escalates it to `needs_reconciliation`. Admission timeouts without a known run ID
|
||
retain the stable child thread identity for reconciliation.
|
||
|
||
Allowlist order does not change request identity; adding or removing allowed Agent
|
||
references does. Matching retries still return the retained record after restart.
|
||
|
||
Cancel is asynchronous and never rolls back side effects. The service retains
|
||
in-flight admission until its bounded response returns, records the actual child
|
||
run ID, then explicitly interrupts and observes that child. If admission times
|
||
out or authorization is revoked during cleanup, it retains `needs_reconciliation`
|
||
with stable thread/input/idempotency identity. It never automatically retries an
|
||
uncertain admission. Stable request IDs return an existing workflow record; they
|
||
never themselves resume it or reexecute completed work; use the explicit checkpoint action. Changed scripts/args/limits with the
|
||
same request ID conflict. Definition versions are pinned; editing a definition creates a new version.
|
||
|
||
Caller cancellation during launch does not abandon a committed record: the service
|
||
owns persistence through supervisor publication, including shutdown coordination.
|
||
A matching retry returns that same supervised record rather than an unowned queued
|
||
launch. This does not add execution replay or resume.
|
||
Cancellation of a recoverable workflow is also service-owned: disconnecting the
|
||
caller does not abandon child cleanup or its final status write. Shutdown drains
|
||
that cleanup before releasing the store's process lock.
|
||
If a background action fails after disconnection, the Gateway logs its exception
|
||
type at error level so operators can inspect retained workflow and child records.
|
||
|
||
Result collection anchors on the visible human input, including the host's canonical
|
||
`__user` rewrite after dynamic context injection. Hidden system/memory reminders
|
||
cannot become the anchor; genuinely later human input still conflicts.
|
||
Result collection checks input message/run attribution and rejects an intervening
|
||
human turn, because `get_state()` is the thread's latest state, not immutable run
|
||
output. Results are copied before further script calls. Per-step inputs remain private. Owner-scoped `get_step` returns one immutable retained
|
||
output with its attempt and artifact version; run inspection returns bounded metadata. Child conversations remain accessible through normal host authorization.
|
||
|
||
At startup, compatible unfinished records become `recoverable`; legacy or cancelling
|
||
records become `interrupted`. Retained IDs do not recreate a capability by themselves. Gateway revokes handles during shutdown,
|
||
so cleanup may require reconciliation. Paused/approval records become recoverable; committed decisions retain their exact version and can continue through the recovery contract below. Neither graph resume nor tool permission
|
||
approval is interpreted as authenticated approval of an artifact version.
|
||
|
||
## Verify
|
||
|
||
Backend CI installs QuickJS as a POSIX development dependency; the extension declares
|
||
its own pinned runtime dependency. No provider keys are required for these checks.
|
||
|
||
```sh
|
||
cd backend
|
||
uv run pytest tests/test_workflow_plugin.py -q
|
||
make test
|
||
make test-blocking-io
|
||
make format
|
||
```
|
||
|
||
Browser checks use a synthetic loopback fixture, not a deployed Gateway:
|
||
|
||
```sh
|
||
node examples/deerflow-extension-workflows/verify_browser.mjs
|
||
# If using an installed Chrome instead of downloaded Playwright Chromium:
|
||
PLAYWRIGHT_CHANNEL=chrome node examples/deerflow-extension-workflows/verify_browser.mjs
|
||
```
|
||
|
||
Install frontend dependencies first, or set `PLAYWRIGHT_MODULE` to an absolute
|
||
Playwright ES module path. The suite exercises launch, inspect, child navigation,
|
||
cancel, escaped output, deletion and disposal. Backend tests include the real
|
||
Gateway/LangGraph bridge with deterministic test nodes, plus controlled admission
|
||
races, limits, owner scope, service shutdown and restart records.
|
||
|
||
The implementation uses an original small JSON bridge. `deer-workflow` was evaluated
|
||
for API shapes but is not imported or copied: its dynamic Node module loader,
|
||
unbounded fan-out and external CLI Agent adapters are not used here.
|
||
|
||
|
||
## Phase 3 recovery contract
|
||
|
||
Recovery is explicit and owner-scoped. In **Script workflows**, select a
|
||
`recoverable` run and **Recover workflow**. This is a browser action, not a model
|
||
tool. It obtains a fresh session-bound AgentRuns capability and checks current
|
||
permissions, declarative configuration, operator limits, script/arguments, and
|
||
runtime compatibility before starting a replacement worker. Recovery itself does
|
||
not approve pending human output; review the same retained version and approve
|
||
or reject it separately. Duplicate recovery requests share one active worker.
|
||
|
||
The extension persists a versioned bridge journal: each emitted request frame is
|
||
committed before host operations; each response batch is committed before it is
|
||
returned to JavaScript. Recovery evaluates the unchanged script from its start,
|
||
verifies its request frames against that journal, and feeds back recorded responses
|
||
in their original order. Confirmed Agent results are reused rather than dispatched
|
||
again. It reconstructs script variables without serializing a JS heap. `Date` and
|
||
`Math.random` are unavailable; pass timestamps/random values in `args`. A divergent
|
||
history stops before issuing new effects. The journal is bounded at 4 MiB and the complete record at
|
||
8 MiB per run; the worker has a 10,000-round limit.
|
||
|
||
Crash boundaries have explicit outcomes:
|
||
|
||
| Boundary | Recovery behavior |
|
||
| --- | --- |
|
||
| Before child admission intent | A queued step may be admitted with its original identity/input/key |
|
||
| After admission intent, before run ID is stored | `needs_reconciliation`; never resubmit an uncertain Agent call |
|
||
| Known child run, before result is stored | Observe that run without starting it again; validate success, message attribution and host evidence before retaining the result |
|
||
| Child interrupted/failed or external outcome uncertain | Stop; inspect child identities and reconcile effects before starting new work |
|
||
| Approval persisted, before downstream dispatch | Reuse that exact decision and continue through the recorded boundary |
|
||
| File captured, before workflow metadata is stored | Recover the immutable stored bytes by capture step; never overwrite with a newer source |
|
||
|
||
There is no automatic retry or operator override of uncertain external effects.
|
||
Cancellation requests interruption of known children and cannot undo committed
|
||
business effects. Recovery does not resume a child's interrupted graph. Legacy
|
||
Phase 1/2 records have no replay journal and remain inspection-only after restart.
|
||
Normal shutdown preserves compatible pending waits; interrupted active children
|
||
may require reconciliation.
|
||
|
||
Execution time remaining is retained. A crash during active execution consumes
|
||
elapsed wall time including downtime; a persisted human/pause wait suspends the
|
||
execution budget, but its original absolute wait expiry still applies. Replay
|
||
also consumes the remaining execution budget. The CPU limit is per worker;
|
||
Agent-call and log quotas are retained across recovery.
|
||
|
||
The workflow runtime source and QuickJS version must match. Incompatible code,
|
||
script/arguments, configuration or operator-limit changes reject recovery rather
|
||
than silently migrating execution. The durable configuration digest uses a
|
||
purpose-separated HMAC derived from the host's persisted signing secret. Preserve
|
||
that secret and the Gateway's durable run/thread/checkpoint data across restart;
|
||
secret rotation invalidates recovery. Stored assembly evidence remains historical
|
||
and is distinct from this configuration preflight check.
|
||
|
||
Phase 3 and Phase 4 share the design: stable step/attempt and idempotency identities,
|
||
conditional record revisions, a durable intent/result journal, and fresh authority
|
||
support the leases and fenced dispatch in Phase 4 below. SQLite retains the Phase 3
|
||
single-owner contract and exclusive process lock. Both modes require explicit
|
||
session authorization for recovery; neither reconstructs credentials from records.
|
||
|
||
|
||
## Phase 4: shared PostgreSQL deployment
|
||
|
||
Install the extension with its `postgres` extra. Configure every Gateway with the
|
||
same `storage_url` (a psycopg PostgreSQL DSN), `lease_seconds` (5–60, default 15),
|
||
Agent allowlist and limits. Keep `storage_path` configured for the default SQLite
|
||
mode; `storage_url` opts into PostgreSQL and disables the file lock. Existing
|
||
SQLite histories are not automatically migrated or merged.
|
||
|
||
The DSN must resolve to the **same database and schema as the Gateway**. The host
|
||
migration installs `extension_execution_leases`; the extension creates its own
|
||
workflow, artifact, definition, control and quota tables. Startup checks database
|
||
identity and rejects differing cluster quota configurations. Use the normal
|
||
multi-instance Gateway prerequisites: `deployment.multi_instance: true`, shared
|
||
PostgreSQL and signing secret, Redis stream bridge, run ownership heartbeats,
|
||
DB run events, and consistent shared user/config/artifact storage. Credentials
|
||
stay in operator configuration; records contain only opaque configuration hashes.
|
||
|
||
A launch through A can be inspected, approved, resumed or cancelled through B
|
||
without sticky sessions. Controls are committed durably and normally observed
|
||
within one second; a successful action acknowledges persistence, not completed
|
||
child cleanup. Exact attempt/artifact checks and immutable decision keys prevent
|
||
conflicting approvals. The owner revalidates configuration and permissions before
|
||
continuing. No model tool can approve output.
|
||
|
||
Each execution has a database-clock lease and increasing epoch. Writes, captured
|
||
files, child admissions and child cancellation are fenced. Admission validates
|
||
the lease in the **same transaction** that creates the Agent run. A lease check
|
||
before a network call is not sufficient. Lease loss stops the local JS worker;
|
||
previously admitted Agents may have external effects and are reconciled rather
|
||
than blindly retried. An unreachable database stops progress instead of granting
|
||
local ownership. Temporary DB failure can therefore require explicit recovery.
|
||
|
||
After expiry, any compatible Gateway exposes the workflow as recoverable. A user
|
||
signs in and chooses **Recover workflow** to supply a fresh capability. This is
|
||
session-authorized failover, not unattended failover: no raw credential or durable
|
||
user impersonation grant is stored. The same Phase 3 replay checks apply. Unknown
|
||
admissions require reconciliation; completed results and exact file versions are
|
||
reused. A pending cancellation prevents recovery from dispatching new work; cancel
|
||
the expired workflow to finish observing known children instead.
|
||
|
||
Workflow and per-owner limits are shared; active child reservations enforce the
|
||
cluster Agent concurrency limit. An unknown child outcome keeps its reservation,
|
||
even after lease expiry. Operators must inspect the recorded thread/run before
|
||
releasing that reservation; the extension does not automatically reclaim capacity
|
||
from potentially running external effects. Reconciliation is manual in this release.
|
||
Cluster limit changes require a coordinated stop and update of the stored cluster
|
||
policy; rolling replicas cannot silently enforce different quotas.
|
||
|
||
Rolling upgrades may continue live runs on their existing owner. Takeover requires
|
||
the exact runtime signature and matching operator/Agent configuration. An
|
||
incompatible replica rejects recovery; retain a compatible replica to drain or
|
||
cancel old workflows. This does not support arbitrary continuation conversion.
|
||
|
||
Validation uses `DEERFLOW_TEST_WORKFLOW_POSTGRES` with a dedicated PostgreSQL
|
||
instance: `test_workflow_cluster.py` exercises independent services and process
|
||
kill, while `test_extension_execution_fence.py` checks a real PostgreSQL lock-wait
|
||
race at host admission. Agent responses are synthetic; tests make no guarantee
|
||
about undoing arbitrary business side effects.
|
||
|
||
### Deployed acceptance and delivery boundary
|
||
|
||
Run the opt-in HTTP/fault suite from the repository root with the backend dev
|
||
and PostgreSQL dependencies and this example's `quickjs` dependency installed:
|
||
|
||
```sh
|
||
DEERFLOW_WORKFLOW_ACCEPTANCE=1 backend/.venv/bin/python -m pytest \
|
||
backend/tests/test_workflow_gateway_acceptance.py -v --tb=short
|
||
```
|
||
|
||
Docker must be running (OrbStack works); set `DEERFLOW_TEST_DOCKER` when the CLI
|
||
is not on PATH. The suite creates disposable PostgreSQL 17, Redis 7 and nginx
|
||
containers, two real Gateway processes and a shared temporary home. It uses
|
||
normal login cookies/CSRF, actual host AgentRuns/RunManager/checkpoints and a
|
||
round-robin proxy without affinity. Only the model answers and fault barriers
|
||
are test fixtures. No paid model is called. Logs remain in pytest's temporary
|
||
directory; teardown kills only the suite's processes and removes only its named
|
||
containers/volumes. This is an opt-in deployment test, not part of offline CI.
|
||
|
||
Coverage includes concurrent duplicate launch/approval, peer controls, shared
|
||
workflow/child limits, revoked sessions, changed configuration, SIGKILL around
|
||
admission/result/approval persistence, SIGSTOP/lease takeover/stale dispatch and
|
||
artifact writes, and immutable-file approval/consumption after source mutation
|
||
and owner death. The rolling test runs two distinct runtime source trees, with
|
||
a changed worker IPC limit, verifies live old-owner draining and rejects
|
||
incompatible takeover. It is a compatibility-guard test, not certification of
|
||
all historical releases or Kubernetes/network-partition topologies.
|
||
|
||
**Delivered authority boundary:** lease expiry exposes recovery but never grants
|
||
permission to execute as a user. Recovery requires a fresh valid browser-session
|
||
capability and current permission/config/runtime checks. Automatic unattended
|
||
takeover requires a separately designed durable authorization/delegation contract.
|
||
|
||
**Manual reconciliation boundary:** the current product retains stable child
|
||
identities, uncertain status and occupied child capacity. Operators must compare
|
||
the recorded conversation/run and external system effects before deciding what
|
||
happened. There is currently **no supported reconciliation API/UI to attest an
|
||
outcome, audit that decision, and release the reservation atomically**. Do not
|
||
treat direct database edits, deleting a workflow, or retrying it under a new ID
|
||
as that missing closure. This follow-up remains open. Cancellation requests do
|
||
not roll back effects; if the host consumes a cancellation after the extension's
|
||
bounded observation window, the workflow may retain `needs_reconciliation` even
|
||
when the child subsequently becomes interrupted.
|
||
|
||
An expired owner in the middle of cancellation is exposed as recoverable for
|
||
**Cancel workflow** cleanup. Recovery rejects a persisted cancellation even if no
|
||
control-inbox row remains. Cancellation takeover bypasses workflow admission caps,
|
||
but retains lease fencing and permission/config checks; uncertain child effects
|
||
still retain reservations. A completed cancellation can be deleted normally.
|
||
|
||
PostgreSQL workflow inspection/listing uses read-only transactions without the
|
||
cluster budget lock; listing reads records and lease visibility in one query.
|
||
Mutations still take the conservative global advisory lock, and connections are
|
||
not pooled. Those remain throughput limits of this bounded deployment mode.
|