mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-10-27 13:40:37 +00:00
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Nan Gao
|
a9917e986b
|
docs(channels): align leader-lease failover and runtime-config notes with the merged code (#7420)
The two-Gateway harness checklist in CONFIGURATION.md said a crashed leader hands its providers over within lease_seconds + grace_seconds; a peer only tries to acquire on its fixed lease_seconds // 3 tick, so the worst case is one renewal interval more (50 s with the defaults), as the leader-lease section of the same file already says. The grace_seconds field description had the same shorter bound. The channels.leader_lease comment in config.example.yaml still said web UI credential changes reach the holder only when it next starts the channel; since #6988 the holder applies them within one renewal interval. The harness `check` footer still listed the IM channel gates as waiting for leader election, which landed in #6843; it now points at the manual channel item in CONFIGURATION.md, which also gains the runtime-config step. |
||
|
Aling33
|
15a31e30eb
|
fix(scripts): reject a non-file --out path in dev_multi_instance (#7289)
`render-config`, `secrets` and `nginx-conf` write `--out` through `os.replace`, so an existing directory surfaced as a raw `PermissionError` traceback, and a file ancestor as `OSError` from `mkdir`. Validate the first existing ancestor once in `main()` before any writer runs, and state the file contract in `--help` for all three commands. |
||
|
Nan Gao
|
e8c7df9042
|
feat(channels): arbitrate IM transports with a per-provider leader lease (#6843)
* feat(channels): arbitrate IM transports with a per-provider leader lease Every Gateway instance connected every enabled IM platform with the same bot credentials, so channels could not run on more than one instance (Telegram 409 storms, Discord answering twice, scattered per-process state). channels.leader_lease was reserved by #6746; this implements it. - channel_leases table (migration 0041): one row per provider. Acquire is one dialect-native upsert decided against the database clock and bumps a fencing epoch; renew extends only a live lease at the holder's epoch; release sets the expiry to 0 so a peer takes over on its next tick. - app/channels/leadership.py: the holder's local monotonic deadline (send time of the last confirmed renewal + lease_seconds); holds() turns false one renewal interval early, so the transport stops at least grace_seconds before a peer may connect. DB calls are bounded. - ChannelService: start fence in _start_channel (no DB call) behind every start path; fixed-rate supervisor renews/acquires and runs transport starts/stops as per-provider tasks; takeover reloads the provider's saved config; failed starts back off; shutdown releases only leases whose transport stopped. get_status reports leader/leader_here and the holder's transport state on every instance. - Notification worker leaves rows of providers another instance holds. - Gate: the enabled=false refusal now recommends auto/true; an explicit true with database.backend memory is refused when a channel is enabled. - Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated. * docs(changelog): link the channel leader lease entry to #6843 * fix(channels): see peer-held leases for providers absent from local config Review on #6843: a provider connected from the web UI on a peer never enters this instance's channel config, so the notification worker here still claimed its rows (and parked them for 900s) and GET /api/channels/ reported no leader for it. Every supervision round, and startup, now also reads the whole lease table once (bounded), and notification exclusions and status use those observations: a provider whose live lease a peer holds is left to it even without local config. Running that read concurrently with renewals exposed that, through SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on another connection can hand back one of its own bind values instead of the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do not show it). A lease must never take such a row for a confirmed renewal, so acquire/renew/release now decide by affected-row count and acquire reads its epoch back in the same transaction. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
d18d2937e7
|
feat(channels): add channels.leader_lease and gate an explicitly disabled lease under multi-process signals (#6746)
* feat(channels): add channels.leader_lease and gate it under multi-process signals IM channels other than GitHub connect outbound to their platform. With more than one Gateway instance each one opens that connection with the same bot credentials: Telegram rejects the second poller with 409, Discord answers every message twice and per-process ChannelManager state diverges. The multi-process startup gate said nothing about channels; the only guard was a comment in the Helm values. Add the `channels.leader_lease` section (`enabled: auto|true|false`, `lease_seconds`, `grace_seconds`; startup-only, registered in the reload boundary) that the per-provider leader lease will read, validated by a Pydantic model from the non-schema `channels` mapping. The gate now refuses `enabled: false` under a multi-process signal while a non-GitHub provider is enabled under `channels.*` or connectable through `channel_connections`, naming the providers and the rollback; a malformed block is refused on every topology. `auto` follows the multi-process signal, so a single instance is unchanged. ChannelService strips the block before its per-provider loops and exposes it as `leader_lease`; `ensure_ready_channels` only visits registered providers. The two-Gateway harness overlay leaves the block alone. config_version 60 in the example, the Helm chart and its README; docs replace the comment-only "keep 1 replica" guidance with the gate. * docs(changelog): link #6746 * fix(channels): stop recommending the unimplemented leader lease as a remedy Review on #6746: the refusal told operators to switch channels.leader_lease from false to auto/true "so one instance owns each provider's connection", but the transport arbitration ships separately, so that change only bypassed the clause and every worker still started the same bot. The refusal now offers only the remedies that work today (a single instance, or disabling the channels) and states that auto/true are reserved for the lease that ships separately. A regression helper in the gate tests asserts the remedies start with the rollback and that auto/true are never offered. The config example, Helm values/README, CONFIGURATION.md, README, the deployment guide (en/zh), the model docstrings and both changelogs describe auto/true as reserved settings rather than an ownership guarantee. * fix(channels): warn on an enabled unknown channel key; reword the changelog entry Review on #6746: the registry guard in ensure_ready_channels skipped a misspelled provider key (channels.telegam) silently, where it previously reached _start_channel and logged 'Unknown channel type'. An enabled dict block that is neither a registered provider nor leader_lease now logs a warning that lists the registry. The changelog entry claimed the chart and docs drop the "keep 1 replica" guidance; they keep it (auto/true do not make several replicas safe yet) and only stop relying on the comment alone. * fix(dev): disable UI channel providers in multi-instance overlay --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan Gao
|
35f6a98cd9
|
feat(dev): add a two-Gateway multi-instance harness script (#6613)
* feat(dev): add a two-Gateway multi-instance harness script There was no way to run the deployment.multi_instance topology locally: scripts/serve.sh pins one Gateway to port 8001, the local nginx config has a single upstream, and getting two processes past the multi-instance startup gate needs Postgres, Redis, a shared DEER_FLOW_HOME, shared auth secrets and a per-process DeerMem retrieval index by hand. scripts/dev_multi_instance.sh (make dev-multi / dev-multi-check / dev-multi-down) starts throwaway Postgres and Redis containers on loopback, renders a config from the developer's config.yaml (or config.example.yaml), and runs Gateway A on 8001 and Gateway B on 8011 with one shared home, one copy of extensions_config.json and generated AUTH_JWT_SECRET, DEER_FLOW_INTERNAL_AUTH_TOKEN and DEER_FLOW_CREDENTIALS_KEY values, plus an optional round-robin nginx on 2027. B starts after A answers /health: Alembic migrations are advisory-locked, but the LangGraph checkpointer/store setup() that follows is not. scripts/dev_multi_instance.py holds the testable parts: the config overlay (neutralizing and reporting settings the gate refuses), the secrets file, the nginx config, and `check`, which automates the scriptable cross-instance acceptance checks: readiness with stream_bridge ok, A-issued sessions and the internal token accepted by B, thread/upload/artifact visibility, a skill toggle seen by B, nginx balancing, and SSE join on B with Last-Event-ID resume on A. Tests pin that the rendered config passes the real _enforce_postgres_for_multi_worker gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): link the two-Gateway harness entries to #6613 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dev): keep the multi-instance harness inside its own state Address review findings on the two-Gateway harness (#6613): - Containers carry a deerflow.harness.state-dir label; up/down remove only containers of their own state dir and refuse those of another checkout or DEERFLOW_MI_STATE_DIR instead of force-removing them. - Containers are removed with `docker rm -f -v`, so the anonymous volumes the postgres and redis images declare no longer leak on every up/down. - `up` installs an EXIT trap once it starts containers, so any later failure (including B's port being taken after preflight) tears down A and both containers; start_gateway returns failures instead of exiting the script, and the Gateway is disowned so stopping it prints no job notice. - The overlay removes any checkpointer section (a postgres one would write checkpoints and the Store into the developer's database), points an explicit sandbox.ownership and a Redis checkpoint_cache at the harness Redis (keeping ownership timing), drops DeerMem storage_path (both spellings) and the blob_storage root so data lands in the harness home, gives a local AIO sandbox a state-dir-specific container_prefix, and reports a non-DeerMem memory backend it keeps sharing. - nginx paths are rendered as quoted string literals ('$' and control characters are refused), so a state dir with spaces works. - `check` skips the run-based SSE check when model discovery did not succeed, unless --with-llm, and enforces the SSE time budget per raw line so heartbeat-only traffic cannot hold it open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(scripts): keep the harness guidance under the AGENTS.md soft limit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |