2 Commits

Author	SHA1	Message	Date
Nan Gao	e8c7df9042	feat(channels): arbitrate IM transports with a per-provider leader lease (#6843) ... * feat(channels): arbitrate IM transports with a per-provider leader lease Every Gateway instance connected every enabled IM platform with the same bot credentials, so channels could not run on more than one instance (Telegram 409 storms, Discord answering twice, scattered per-process state). channels.leader_lease was reserved by #6746; this implements it. - channel_leases table (migration 0041): one row per provider. Acquire is one dialect-native upsert decided against the database clock and bumps a fencing epoch; renew extends only a live lease at the holder's epoch; release sets the expiry to 0 so a peer takes over on its next tick. - app/channels/leadership.py: the holder's local monotonic deadline (send time of the last confirmed renewal + lease_seconds); holds() turns false one renewal interval early, so the transport stops at least grace_seconds before a peer may connect. DB calls are bounded. - ChannelService: start fence in _start_channel (no DB call) behind every start path; fixed-rate supervisor renews/acquires and runs transport starts/stops as per-provider tasks; takeover reloads the provider's saved config; failed starts back off; shutdown releases only leases whose transport stopped. get_status reports leader/leader_here and the holder's transport state on every instance. - Notification worker leaves rows of providers another instance holds. - Gate: the enabled=false refusal now recommends auto/true; an explicit true with database.backend memory is refused when a channel is enabled. - Docs, Helm, deployment guides, AGENTS.md and CHANGELOG updated. * docs(changelog): link the channel leader lease entry to #6843 * fix(channels): see peer-held leases for providers absent from local config Review on #6843: a provider connected from the web UI on a peer never enters this instance's channel config, so the notification worker here still claimed its rows (and parked them for 900s) and GET /api/channels/ reported no leader for it. Every supervision round, and startup, now also reads the whole lease table once (bounded), and notification exclusions and status use those observations: a provider whose live lease a peer holds is left to it even without local config. Running that read concurrently with renewals exposed that, through SQLAlchemy's aiosqlite adapter, an UPDATE ... RETURNING racing a read on another connection can hand back one of its own bind values instead of the column (stored rows stay correct; plain sqlite3 and raw aiosqlite do not show it). A lease must never take such a row for a confirmed renewal, so acquire/renew/release now decide by affected-row count and acquire reads its epoch back in the same transaction. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-14 13:51:42 +08:00
Nan Gao	db7340d2fa	fix(security): add DEER_FLOW_CREDENTIALS_KEY and wire the channel credentials cipher (#6611) ... * fix(security): add DEER_FLOW_CREDENTIALS_KEY for at-rest credential encryption There was no at-rest encryption key. ChannelCredentialCipher had no production caller: the channel service, the scheduled-task notification outbox and the channel-connections router all built ChannelConnectionRepository without a cipher, so store_credentials refused (RuntimeError) and get_credentials always returned None. - New harness module deerflow.config.credentials_key: env-only DEER_FLOW_CREDENTIALS_KEY (comma-separated Fernet keys; first encrypts, all decrypt via MultiFernet), CredentialsCipher with encrypt_text / decrypt_text / rotate_text, a fernet:v2: prefix with legacy fernet:v1: values still readable, and get_credentials_cipher() as the process-wide entry point. Malformed keys are refused without echoing key material. Without the variable a single instance generates {base_dir}/.credentials_key. Never derived from AUTH_JWT_SECRET. - New deerflow.config.secret_file.read_or_create_secret_file: O_EXCL create (0600) plus read-back with a bounded settle window, shared by .credentials_key, .jwt_secret (fixes two replicas cold-starting on a shared volume keeping different JWT secrets) and the managed-model key. - ChannelCredentialCipher now builds on CredentialsCipher (same constructors); the Gateway loads the cipher off the event loop at startup when channel_connections is enabled and passes it to every ChannelConnectionRepository. Undecryptable values stay "missing + warn". - Startup gate: a malformed key stops startup in any topology; a declared multi-instance deployment with channel_connections enabled and no key is refused. Plain multi-worker shares the generated file and is allowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(deploy): generate and inject DEER_FLOW_CREDENTIALS_KEY in Helm, compose and deploy.sh - Helm: the app Secret generates a Fernet key (randBytes 32 translated to the urlsafe alphabet) and preserves it across upgrades via lookup like AUTH_JWT_SECRET; the gateway reads it through an optional secretKeyRef so a user-managed existingAppSecret without it still boots (the Gateway gate refuses only multi-instance deployments that store credentials). README, NOTES and values document backup, rotation and existingAppSecret. - scripts/deploy.sh resolves the key shell -> .env -> persisted file -> generated, persisting it as $DEER_FLOW_HOME/.credentials_key (0600, noclobber/O_EXCL with read-back): the same file the Gateway generates in its runtime home, so either side's key is the one both keep. - Both compose files pass DEER_FLOW_CREDENTIALS_KEY=${...:-} to the gateway. - .env.example entry with a generation hint; README and CHANGELOG notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(security): make abandoned secret-file replacement single-winner A rename is last-writer-wins, so replicas that restart together and all find the same abandoned empty secret file could each replace it and keep a different value. The process that exclusively creates <name>.replacing now owns the replacement, re-checks the file is still empty, and publishes by renaming the claim over it; peers wait for that value, bounded so a crashed replacer cannot block startup forever. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): link DEER_FLOW_CREDENTIALS_KEY entries to #6611 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(security): name the setting that turns off a credentials-key consumer The multi-instance refusal ended with "or disable channel_connections.enabled=true"; it now says "or set channel_connections.enabled=false". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(security): never expose a secret file before its value is written O_EXCL published the name before its content, so a creator that stalled mid-write left an empty file a peer replaced after the settle window; the resumed creator then returned a value nobody persisted, or deleted the peer's valid file when its own write failed. Creators now write and sync a private temporary file and hard-link it to the name, which fails if the name exists; a failed creator only removes its own temporary file. Filesystems without hard links publish through the same single-winner <name>.replacing claim that replaces abandoned empty files, so creation and replacement are serialized there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(models): recover an empty managed-model key when no catalog exists An interrupted first save could leave an empty key file; every later save then passed empty bytes to Fernet and the endpoint returned 503 until the file was removed by hand. An empty key with no catalog encrypted nothing, so it now goes through the secret-file recovery path; next to an existing catalog it is still refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(docker): pass the dotenv DEER_FLOW_CREDENTIALS_KEY through make docker-start scripts/docker.sh runs Compose from docker/ without --env-file, so the dev gateway's DEER_FLOW_CREDENTIALS_KEY=${DEER_FLOW_CREDENTIALS_KEY:-} entry interpolated to empty and overrode the key the checkout .env supplied via env_file. Export it from the dotenv like AUTH_TRUSTED_PROXIES. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-10 20:51:09 +08:00