mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-27 13:40:37 +00:00
cfe2bec371
* feat(workflows): add leased multi-Gateway coordination and fenced dispatch * test(workflows): validate deployed failover and canonicalize config revisions * fix(workflows): share namespace across registration and lease fencing * fix(workflows): recover interrupted cancellation without admission caps * fix(workflows): retain execution lease through child cleanup * docs: trim guidance padding to fit merged instruction budget * fix(workflows): release rejected admission slots and normalize gateway routing --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
123 lines
8.7 KiB
Markdown
123 lines
8.7 KiB
Markdown
# Phase 4 implementation contract
|
|
|
|
- Opt-in PostgreSQL workflow store; default SQLite/exclusive flock remains unchanged.
|
|
- Shared workflow rows/artifacts/definitions, durable control inbox, database-clock execution leases and epoch fencing.
|
|
- Generic host ExecutionFence checked in same RunStore transaction as child admission. A preflight-only lease check is insufficient.
|
|
- Lease loss stops local worker; stale owner cannot save, dispatch, approve, capture artifacts or cancel a successor's children.
|
|
- Remote browser actions revalidate current permission/config, commit exact-version decisions, and owner consumes durable controls. Duplicates are idempotent.
|
|
- Lease expiry exposes recoverable/reconciliation state. Takeover requires fresh authenticated browser capability; no credentials serialized and no autonomous execution under a reconstructed user ID.
|
|
- Cluster quotas are transactionally reserved; uncertain child effects retain capacity until reconciled, never automatically redispatched.
|
|
- Runtime/config incompatibility blocks takeover; old compatible replicas may continue their own live runs during rolling upgrade.
|
|
- Tests: opt-in deployed two-Gateway HTTP/auth/PostgreSQL/Redis/nginx acceptance, real process faults, and complementary PostgreSQL transaction/fencing regressions.
|
|
|
|
|
|
## Validation boundaries
|
|
|
|
`backend/tests/test_workflow_gateway_acceptance.py` starts two real Gateways with
|
|
normal login/CSRF, shared PostgreSQL 17, Redis 7, and nginx round-robin routing.
|
|
The deterministic model replaces provider answers, not AgentRuns, admission,
|
|
LangGraph/checkpoints, storage or authorization. Test-only barriers pause real
|
|
operations before SIGKILL/SIGSTOP; no production fault hook or auth bypass exists.
|
|
|
|
| Scenario | Deployed assertion |
|
|
| --- | --- |
|
|
| A launch; B inspection/resume/approval/cancel | Same session, real workflow reads through both proxy upstreams without affinity |
|
|
| Duplicate launch and approval | Concurrent HTTP requests to A/B produce one workflow, one decision, one child per step |
|
|
| Admission receipt lost | Actual child committed, Gateway killed before workflow receipt; recovery retains uncertain slot and never redispatches |
|
|
| Completed child, result not yet persisted | Kill before result write; survivor reads the known completed child without starting another |
|
|
| Result persisted, response not delivered | Kill after result commit; replay reuses retained output |
|
|
| Approval persisted, downstream not dispatched | Kill after decision commit; survivor preserves approval and dispatches the next step once |
|
|
| SIGSTOP old owner and restore after takeover | Actual stale host admission rejected; old workflow/artifact writes cannot change successor state |
|
|
| File approval after owner death | Modify original after capture; B reviews/approves and downstream receives original immutable bytes |
|
|
| Shared quotas | Per-owner workflow limit and cluster child reservations constrain real HTTP launches/host admissions |
|
|
| Revocation/config drift | Revoked cookie is rejected; fresh login still cannot bypass changed configuration; rejected takeover does not advance epoch |
|
|
| Rolling runtime change | Separate source trees with different worker IPC limits execute; old owner drains, incompatible survivor rejects takeover |
|
|
| Active-child cancellation | Peer request reaches actual child owner; delayed observation remains an explicit reconciliation outcome |
|
|
|
|
The database-level suite additionally tests stale cancellation and admission while
|
|
waiting for a real PostgreSQL row lock. It complements the process faults; it is
|
|
not a claim that every transaction boundary has been exercised over HTTP.
|
|
The local deployment uses OrbStack containers and native Gateway processes with
|
|
shared local storage. This does not certify Kubernetes, arbitrary network partitions,
|
|
all published-version upgrade pairs, external business-effect rollback, or model
|
|
quality. The alternate runtime is a test source revision, not a historical release.
|
|
|
|
### Authorization and reconciliation delivery boundary
|
|
|
|
Lease expiry alone grants no user authority. Takeover is an explicit action using a
|
|
fresh valid browser-session capability; current permissions, configuration and exact
|
|
runtime compatibility are rechecked. No serialized credentials or reconstructed user
|
|
ID substitutes for that capability. Unattended automatic takeover needs a separate
|
|
durable delegation/authorization design and is not delivered here.
|
|
|
|
Unknown effects retain the workflow/step evidence and child reservation. Manual
|
|
inspection can establish the business outcome, but the product currently has no
|
|
supported API/UI that records an operator attestation, audits it, and atomically
|
|
releases that reservation. That closure remains open; do not equate direct SQL,
|
|
workflow deletion, or a new request ID with a supported safe retry. Cancellation is
|
|
not rollback. Host cancellation may be observed after the extension's bounded wait,
|
|
leaving `needs_reconciliation` even when the child later becomes interrupted.
|
|
|
|
### Defect found by deployed acceptance
|
|
|
|
Pydantic JSON dumps converted `tool_progress.exempt_tools` from a set into a
|
|
process-dependent list order. Identical configuration on A and B could therefore
|
|
produce different durable HMACs and reject peer controls. Revision encoding now
|
|
retains Python set identity and canonicalizes sets while preserving ordered lists.
|
|
A focused regression and Gateways with different `PYTHONHASHSEED` values exercise
|
|
this fix. Existing pre-fix fingerprints may fail compatibility checks; drain them
|
|
on their original compatible runtime rather than rewriting stored fingerprints.
|
|
|
|
## Earlier validation (before deployed acceptance)
|
|
|
|
- Full offline backend suite with PostgreSQL enabled: 34,527 passed, 686 skipped.
|
|
- Final extension/recovery/API subset: 144 passed, including 14 cluster cases.
|
|
- Host fencing, AgentRuns, run ownership and SQL repository subset: 213 passed.
|
|
- Blocking-I/O suite: 262 passed, 1 skipped.
|
|
- Backend and extension lint/format, frontend lint/typecheck, browser fixture and
|
|
guidance checks passed. PostgreSQL 17 ran on loopback with disposable schemas.
|
|
- The final targeted runs include the owner predicate on fenced cancellation and
|
|
file-integrity validation for remote approval added during the full-suite run.
|
|
|
|
## Deployed acceptance follow-up (2026-10-19)
|
|
|
|
- OrbStack PostgreSQL 17 / Redis 7 / nginx, native Python 3.12 Gateways:
|
|
**14 deployed acceptance cases passed** (157 seconds).
|
|
- Final strengthened proxy/readiness and active-successor stale-artifact checks:
|
|
**1 passed each**, after the full deployed run.
|
|
- Full backend offline target with PostgreSQL regressions enabled:
|
|
**34,573 passed, 656 skipped** across four shards. The deployed suite is explicitly
|
|
opt-in and skipped in that target; its result is reported separately above.
|
|
- Blocking-I/O target: **262 passed, 1 skipped**.
|
|
- PostgreSQL coordination/fencing and AgentRuns subset: **48 passed**.
|
|
- AgentRuns and guidance checks: **38 passed**. Changed Python files passed Ruff
|
|
lint/format checks; `git diff --check` passed.
|
|
- Acceptance teardown removed its Gateway processes and disposable containers;
|
|
pytest temporary directories retain diagnostic logs. No production/user stack
|
|
was reused or modified. The standalone PostgreSQL regression server was stopped.
|
|
|
|
## Review follow-up (2026-10-20)
|
|
|
|
Expired `cancelling` records now project as recoverable so authenticated cancellation
|
|
can reclaim the lease and finish cleanup. Ordinary replay rejects stored cancelling,
|
|
even without a cancel inbox entry; cancel claims themselves persist cancelling so a
|
|
second crash cannot lose the cancellation intent. Cancel takeover bypasses workflow
|
|
admission caps while retaining fencing and current permission/configuration checks.
|
|
Uncertain child reservations still require reconciliation before deletion.
|
|
|
|
Workflow get/list and database-identity reads use read-only transactions without
|
|
the global budget advisory lock. Listing joins workflow records and lease visibility
|
|
in one query/connection. Writes retain conservative global serialization; pooling
|
|
and finer write-lock granularity remain deferred for this bounded mode.
|
|
|
|
Regressions reproduce expired cancellation with and without saturated limits and
|
|
inspection while another connection holds the budget lock. Deployed acceptance adds
|
|
SIGKILL during cancellation takeover, then cleanup and deletion through a survivor.
|
|
|
|
Review validation: 17 PostgreSQL cluster cases passed (the three new regressions
|
|
failed before the fix); 15 real deployed acceptance cases passed. Full backend
|
|
regression: 34,576 passed, 657 skipped; blocking-I/O: 262 passed, 1 skipped.
|
|
Backend format/lint, frontend lint/typecheck, extension browser checks and diff
|
|
whitespace checks passed. The acceptance processes/containers and the dedicated
|
|
PostgreSQL regression server were cleaned up after validation.
|