* fix(channels): stream-cap and validate WeChat/WeCom inbound media downloads, fixes #5223 * fix(channels): address WeCom APPID, decompression, and log-sanitization review findings (#5223) Round-4 review follow-ups on the inbound-media download cap: - The COS bucket numeric suffix is the owner's Tencent Cloud APPID and bucket names are user-chosen, so any Tencent Cloud account could register a matching ww-aibot-img-* bucket and pass the shape gate. The built-in rule now admits only the APPID observed in Tencent's published aibot callback examples (1258476243), across regions; any other account (including a future WeCom rotation) goes through channels.wecom.allowed_media_hosts. - aiter_bytes() transparently decodes Content-Encoding, and the decoder allocates the full decompressed body before the byte cap sees a chunk (an ~8 KB gzip wire chunk decoding to 8 MiB reproduces it). Both URL readers now send Accept-Encoding: identity, refuse a response with a residual Content-Encoding before reading, and iterate aiter_raw(). - httpx.HTTPStatusError formats the signed URL (path + query credentials) into its message, so _ingest_inbound_files' reader-failure branch logs a sanitized summary (class + status) instead of logger.exception, and the WeChat extract paths catch httpx.HTTPError so the polling loop's per-message logger.exception can never render a media URL. Every change ships with a red/green regression: the reviewer's 403 mock-transport repro asserted against caplog.text (fully formatted logs), the reviewer's different-APPID bucket host, and gzip bombs driven through real httpx mock transports in both readers. Docs (channels AGENTS.md, README, config.example.yaml) updated for the APPID pinning and encoding gate. * docs(channels): document why the inbound-media cap is 50 MB, not WeCom's 100 MB ceiling * fix(logging): redact URLs in httpx request logs down to scheme + host, fixes #5223 httpx emits 'HTTP Request: GET <full URL>' at the Gateway's INFO level before any response handling runs, so even successful signed-media downloads leaked their credentials. HttpxUrlQueryRedactionFilter (installed by configure_logging) rewrites those records in place — path and query become /<redacted>, method/status/duration observability is preserved — which also keeps Telegram's token-bearing Bot API paths out of the logs. Reader-level regression tests run at production INFO level with a real MockTransport, success paths included. * fix(logging): blank userinfo credentials in httpx request-log redaction * fix(logging): redact authority-only URLs and cover urllib3 redirect logs Two follow-ups from the review plus one extrapolation of the same class: - rest is now optional in _URL_REDACT_RE, so an authority-only URL (scheme://user:pass@host, no path) is rewritten too — userinfo had nowhere else to hide and previously passed through verbatim. A bare credential-free origin still passes through unchanged. - Renamed to UrlRedactionFilter / install_url_log_redaction and attached to the urllib3 logger as well: urllib3 logs 'Redirecting <url> -> <url>' at INFO with full URLs on both sides, the same leak class on a different library logger. No gateway path today both uses requests and redirects a signed URL, but the class stays closed instead of dormant. - Unit tests now build records with the real httpx 0.28.1 format string ('HTTP Request: %s %s "%s %d %s"', 5 args) and httpx.URL args, per the nit, instead of a synthetic shape httpx never emits. * fix(logging): install URL redaction at handler level so propagated records are covered A logging.Filter on a logger only runs for records emitted through that exact logger — child loggers neither inherit it nor trigger it on propagation — so the previous attachment to the bare urllib3 logger was dead code: urllib3 emits Redirecting via urllib3.poolmanager at INFO and urllib3.connectionpool at DEBUG. The filter is now attached to every root handler (mirroring _install_trace_filter, which already iterates root handlers; handler-level filters see propagated records) in addition to the httpx logger (httpx emits via the bare name, and emission-point coverage survives handlers added later). The wiring is pinned by tests that emit through the real urllib3 child loggers — a mutation removing the handler-level install turns them red. Comments, docstrings, and AGENTS.md now state the actual emitter names and levels. * fix(logging): redact urllib3 DEBUG request lines, whose split shape evaded the URL regex urllib3's per-request line (connectionpool.py:545 on 2.7.0) renders as `scheme://host:port "METHOD /path?query HTTP/x.x" status len` — the authority ends at a space so _URL_REDACT_RE's bare-origin early return applies, and the quoted origin-form target has no scheme, so neither half was rewritten. UrlRedactionFilter now runs a dedicated request-line shape first (collapsing the target to /<redacted>, keeping scheme+host+method+ version), then the absolute-URL pass. Regressions pin the exact format string both at unit level and through the real urllib3.connectionpool DEBUG emit path; AGENTS.md wording now names both covered DEBUG shapes. * fix(logging): redact urllib3 retry lines and linearize scheme scanning Closes the two open review threads on the inbound-media log hardening: Retry/redirect targets: urllib3 logs the request target with no scheme in five shapes the generic absolute-URL pass cannot see - `Retry: <target>` (connectionpool.py:954 DEBUG), `Incremented Retry for (url='<target>')` (util/retry.py:545 DEBUG, absolute on the redirect path), `Retrying (...) after connection broken by '<err>': <target>` (connectionpool.py:869 WARNING, above the INFO root), and origin-form halves of both Redirecting emitters (poolmanager.py:500 INFO / connectionpool.py:922 DEBUG). Each gets a rewrite anchored to the exact urllib3 format, collapsing the target to /<redacted>; the generic pass's rest now stops at quote characters so a quoted URL keeps its closing punctuation (previously the absolute-form increment line was mangled), and the request-line method class accepts any case. The emitter enumeration in channels AGENTS.md is closed against the installed urllib3 2.7.0 source. Quadratic scanning: both scheme-bearing patterns start with a character class, so re.sub retried every suffix of a long token - 64K paths cost ~1.8s and URL-free 64K error bodies ~3.1s per record, synchronously in every root handler. The two passes are now driven from literal "://" occurrences: _scheme_starts walks back over the scheme charset to each run's first letter and the pattern is attempted only there, reproducing re.sub's leftmost-non-overlapping result in linear time (256K path: 5.6ms; worst adversarial shapes <= 28ms). Long-input regressions pin the URL-bearing and URL-free cases with mutation-verified bounds, plus nested-scheme and digit-headed-run equivalence cases. Validation: tests/test_logging_config.py 12/12; scheme-pass equivalence against the old re.sub pipeline verified by two independent 30k+ case fuzz runs; full-suite A/B against HEAD shows zero tests that pass on HEAD and fail with this diff. * fix(logging): boundary-aware quote stops and whole-message Redirecting anchor Two follow-ups on the urllib3 redaction shapes: Embedded quotes: `rest` treated ANY quote as a closing mark, so a URL with an apostrophe in the path kept everything after it verbatim (`https://h/path'quoted'?token=Q` rendered the credential suffix in full) while the class docstring claimed path/query/fragment are replaced. A quote now closes `rest` only at a boundary - followed by whitespace, a closing parenthesis, or end of string - so urllib3's Incremented Retry (url='...') scaffolding keeps its ') closer while an embedded quote stays consumed. The increment line's url capture gets the same rule narrowed to its fixed ')' closer. Redirecting anchoring: the origin-half pass matched `(? <=-> )/path` as a substring, and an `-> /path` arrow is not urllib3-owned shape - the sandbox provider's actionable mount error (`sandbox.mounts entry <host> -> /mnt/knowledge ignored: ...`) had its container path rewritten to /<redacted>, failing test_setup_path_mappings_logs_actionable_error_for_missing_host_path on CI (backend-unit-tests shard 3). The pass is now anchored to the whole `Redirecting <t> -> <t>` message, which is exactly urllib3's record; origin slots collapse, absolute slots stay for the generic pass. Regression tests pin the embedded-quote shapes and the sandbox error's byte-for-byte passthrough; both mutations verified red. Validation: tests/test_logging_config.py 14/14; the CI-failing sandbox test green locally; every test file asserting redaction/arrow log content passes (attachments, support bundle, run metadata, skill secrets, ragflow, skillscan, sandbox provider); full offline backend suite 14084 passed / 164 failed with the failure set matching this machine's documented Windows-environment baseline (NTFS chmod/symlink, docker/lark/langfuse absences) - no failure involves redaction output. * fix(logging): redact redirects with spaced locations * fix(logging): grammar-complete Redirecting anchor; neutral WeChat guard labels Round-13 P3 (Redirecting anchor strictness): the whole-message anchor kept the ^Redirecting prefix (the urllib3-owned literal that stops the sandbox false positive) but required BOTH slots whitespace-free, so a Location header with an interior space voided the pass and leaked the origin-form request target in the first slot - redirect_location is the raw header string and interior spaces are legal field syntax. The tail is now loose (\S.*$) and the first slot gets the same grammar treatment (\S.*?): the recursive urlopen frame passes the previous raw Location as its url, so t1 can carry interior spaces too, lazy-split at the first arrow the way the line is constructed. A space-carrying slot collapses whole when it starts with /; the sandbox mount error keeps passing through untouched. Round-14 nit (None conflation): _download_cdn_bytes returns None for two reasons (in-flight cap abort, Content-Encoding refusal) but both image and file callers labeled it "exceeds size limit (N bytes)" - contradicting the accurate encoding line right above it, and reporting the plaintext limit for a ciphertext-cap decision. Callers now log a neutral "skipped by download guard" line (the manager reader callers' shape); the accurate reason stays inside the download function. The same sweep also logs _stage_downloaded_file's silent None (no state dir configured), which made an attachment vanish with no log line at all. Also anchors the emitter-enumeration closure to its urllib3 version: the closure reopens if an upgrade changes these format strings, so the comment now says so explicitly. Validation: logging 15/15 and attachments 60/62 (the two pre-existing Windows symlink-privilege failures documented in the PR body); three mutations verified red (old wording, strict t1, silent staging None); ruff clean. Full offline suite run before push (per round-11 lesson). --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
41 KiB
IM Channels System (app/channels/)
Bridges external messaging platforms (Feishu, Slack, Telegram, Discord, DingTalk, GitHub) to the DeerFlow agent via Gateway's LangGraph-compatible API.
Architecture: Channels communicate with Gateway through the langgraph-sdk HTTP client (same as the frontend), ensuring threads are created and managed server-side. The internal SDK client injects process-local internal auth plus a matching CSRF cookie/header pair so Gateway accepts state-changing thread/run requests from channel workers without relying on browser session cookies.
Components:
message_bus.py- Async pub/sub hub (InboundMessage→ queue → dispatcher;OutboundMessage→ callbacks → channels)store.py- JSON-file persistence mappingchannel_name:chat_id[:topic_id]→thread_id(keys arechannel:chatfor root conversations andchannel:chat:topicfor threaded conversations). Every access to_datamust be protected by_lock;list_entries()snapshots keys and copied entries under the lock, then formats the result after releasing it so concurrent channel threads cannot resize the dictionary during iteration without extending the critical section.manager.py- Core dispatcher: creates threads viaclient.threads.create(), routes commands including/goal(setting a goal persists it through Gateway and then routes the objective as a chat turn) and/agent(listis owner-scoped;usecreates a fresh thread and persists the Custom Agent selection under both the channel restart key and Web's canonical routing metadata so existing checkpoint lineage never changes agent across IM/Web), keeps Slack/Discord onclient.runs.wait(), usesclient.runs.stream(["messages-tuple", "values"])for Feishu/Telegram incremental outbound updates, serializes same-thread Feishu turns in-manager when the channel'sChannelRunPolicy.serialize_thread_runs=Trueso rapid follow-ups queue instead of tripping the runtime busy reply, and switches toclient.runs.create()(fire-and-forget, returns once the run ispending) for channels whoseChannelRunPolicy.fire_and_forget=Trueso long autonomous runs do not hit the SDK default 300shttpx.ReadTimeoutA swallowed streaming failure publishes its final outbound before releasing the inbound dedupe key, so a provider redelivery can retry without overtaking the terminal reply. What may be published from the stream is an allowlist, not a denylist (_accumulate_stream_text): only assistant message types — LangChain serializesAIMessage.typeas"ai"andAIMessageChunk.typeas"AIMessageChunk", plus the OpenAI-style"assistant"spelling for foreign runtimes — become displayable text. The previous rule rejected only payloads whosetypecontained"tool"and therefore published everything else, which leaked DeerFlow's hidden model context to every streaming IM channel:DynamicContextMiddlewareinjects recalled memory as a hiddenHumanMessage(type == "human") and rewrites the user's own turn into a newHumanMessage,DurableContextMiddlewareinjects a hidden<durable_context_data>HumanMessage, and LangGraph fans those state writes out on themessages-tuplestream. Proved live on a Buzz relay, which published a<memory>fact block and, in another run, a verbatim echo of the user's own message as the assistant's reply. Matching is by prefix (ai/assistant), never substring, because ordinary words contain"ai"(chain,domain). The message type is resolved by_stream_payload_type, which handles both themodel_dump()shape DeerFlow's own gateway emits and LangChain'sto_json()constructor shape (whose top-leveltypeis the literal"constructor", with the class name at the tail of theidpath). A barestrpayload is no longer accepted at all: it carries no type information, so it cannot be attributed to the assistant, and nothing in DeerFlow produces one (runtime/serialization.py::serialize_messages_tuplealways emits[message_dict, metadata]).base.py- AbstractChannelbase class (start/stop/send lifecycle). Provider callbacks that submit coroutines from SDK threads must use_submit_threadsafe_coroutine(): it creates and retains the realasyncio.Taskon the owner loop instead of treatingrun_coroutine_threadsafe()'s proxy Future as a completion signal. Submission is closed atomically with shutdown, andstop()must call_close_and_drain_threadsafe_futures()before tearing down SDK resources.service.py- Manages lifecycle of all configured channels fromconfig.yaml. Shutdown closes manager admission first and keeps transports alive until every manager worker/follow-up watcher has exited. A successful manager stop therefore owns no live handler; if the Gateway's outer timeout cancels shutdown, the service retains its channel objects and global singleton so unfinished resources are not detached and cleanup can be retried.slack.py/feishu.py/telegram.py/discord.py/dingtalk.py- Platform-specific implementations (feishu.pytracks the running cardmessage_idin memory and patches the same card in place;telegram.pyaccepts inbound text/photos/documents, preserves media captions, hands token-free attachment bytes to the shared upload pipeline, edits the "Working on it..." stream target in place viaeditMessageText, and can optionally send final Markdown replies as Rich Messages throughchannels.telegram.rich_messages;discord.pyregisters typing loops before inbound handling yields and_start_typing()refuses work once_runningis false; becausestop()runs on the main loop while typing tasks belong to_discord_loop, normal cross-thread cancellation, awaiting, and map cleanup are scheduled there with a bounded wait, while_run_client()drains tasks in itsfinallyblock before an exception/disconnect can make that loop unusable; an already-stopped foreign loop must never have its tasks awaited from the main loop; to that end every outbound cross-loop call (send/send_file/_get_channel_or_thread) goes through_run_on_discord_loop, which bounds the await (DISCORD_OUTBOUND_TIMEOUT_SECONDS30s;DISCORD_UPLOAD_TIMEOUT_SECONDS120s for file uploads, whose unbounded-size payload needs room for a slow uplink plus 429 retry-after) and fails fast with aRuntimeErrorwhen the client loop is missing or not running — a dead client becomes a logged send failure instead of a permanently hungChannelManagerworker — andis_runningreports client-thread aliveness (likefeishu.py) soensure_channel_readycan restart the channel after_run_client()exits on a fatal error;dingtalk.pyoptionally uses AI Card streaming for in-place updates whencard_template_idis configured, and overridesreceive_fileto download inbound images (picture/richText) and documents (file) bydownloadCodeinto the thread uploads bucket, mirroringfeishu.py)buzz.py- Buzz (Nostr relay) implementation: one NIP-42-authenticated websocket, pubkey-allowlist + mention/DM/thread-follow gating, streaming replies via in-place kind-40003 edits; requires thebuzzdependency extra. Its durable seen-event replay guard uses asyncaseen()/arecord()/aflush()boundaries: initial JSON reads and coalesced atomic writes run off the Gateway event loop, andstop()awaits the final flush before returning. Subscription model (operator-facing version in IM_CHANNEL_CONNECTIONS.md): the relay fans kind-9 chat events out only to channel-scoped subscriptions, proved against a live relay —REQ {"kinds":[9]}is accepted and answered withEOSEbut never receives an event (the connector authenticated and then sat silent forever),REQ {"kinds":[9],"#h":[uuid]}works, and a multi-value#hmatches nothing, so it is strictly one REQ per channel (the same shape as Buzz's ownbuzz-acpharness). Every connection therefore rebuilds three kinds of subscription after NIP-42 auth:buzz-discovery({"kinds":[39000]}, a historical query returning exactly the channels this identity belongs to, one stored event each thenEOSE— adding#preturns zero, do not "narrow" it),buzz-membership({"kinds":[44100,44101],"#p":[us],"since":<connect time −MEMBERSHIP_LOOKBACK_SECONDS>}, the relay-signed member-added/member-removed notifications whoseptag names the affected member andhtag the channel), and onebuzz-chat-<uuid>per discovered channel. Chat subscriptions open as each kind-39000 arrives; the discoveryEOSEis the completeness barrier that retries any that failed and warns when discovery found nothing. A kind-44100 for our pubkey subscribes to the new channel live (then re-issues discovery so its name/type reach the DM-detection cache, but only when that channel's metadata is actually missing — an unconditional refresh is how a burst of 44100s multiplied into one discovery pass each); a kind-44101 issuesbuzz_nostr.close_framefor exactly that channel's subscription and drops its metadata. The membership subscription is scoped to LIVE events and this is load-bearing: buzz-relay stores 44100/44101 and serves history newest-first (default limit 2000), so an unscoped filter replayed the whole membership history on every connect — every stored add read as live, re-running discovery once each (M+1 discovery passes × N stored kind-39000 events per connect, observed live as twochannel discovery completelines and one channel logged<unnamed>because the 44100 path subscribed before its metadata arrived), re-subscribing channels we have since been removed from, and letting a stored 44101 transiently unsubscribe a channel we are still in.sinceis anchored at the moment the socket opened (_session_started_at) minusMEMBERSHIP_LOOKBACK_SECONDS(60) of slack, which covers both relay clock skew and a membership change published during the connect/auth handshake; the slack can only cost an idempotent replay of the last minute. A relayCLOSEDframe is recovered, not merely forgotten — every subscription on the socket fails silently when dropped, so_handle_closedre-issues it, bounded byMAX_RESUBSCRIBE_ATTEMPTS(3) per subscription id per connection and per auth epoch (_resubscribe_attemptsis reset on session start, on re-auth, and bystop(), so pre-auth rejections — which the auth branch already recovers wholesale — never spend the authenticated session's budget). The first retry is immediate (the common case is a one-off hiccup); later ones back off 1s then 2s, awaited inline in the read loop rather than in a background task that could outlive its own socket.auth-required:before this socket has completed its NIP-42 handshake is the expected bootstrap sequence, not a refusal, and_handle_closedshort-circuits it ahead of every recovery path: the connector opens its control REQs immediately in case the relay serves unauthenticated reads, a closed relay answersauth-required:plus anAUTHchallenge, and the auth branch re-opens everything. That case is logged at DEBUG and consumes neither the permanent-refusal branch nor the retry budget (a chat subscription is still dropped from_chat_subscriptions, since it genuinely is not subscribed and discovery is what re-opens it). Treating it as permanent produced an operator-facing warning claiming discovery/membership tracking was DOWN in the same run where discovery then completed, every channel was subscribed, and a brand-new channel's kind-44100 was picked up one second later. The boundary is the per-socket_auth_completedflag, set once the signed AUTH event has been sent and cleared on session entry, session exit, andstop(); the same reason after that stays loud, and says the subscription is down until the relay's next AUTH challenge or the next reconnect rather than borrowing the non-auth wording. Then_is_transient_closedecides whether to retry at all: NIP-01/NIP-42auth-required:/restricted:/blocked:/mute:/invalid:/pow:prefixes and buzz-relay's own removal/revocation prose are permanent (do not fight the relay over a channel that is no longer ours; a post-authauth-required:is recovered by the AUTH branch, not by re-issuing),rate-limited:/error:/no-reason-at-all and anything unrecognized are transient — the default resolves toward "keep listening" because going silently deaf is the failure this exists to remove, and the attempt budget bounds a wrong guess. A chatCLOSEDis only ever recovered for a channel already in_chat_subscriptions: aCLOSEDis relay-supplied, so acting on an unknown one would let a relay induce a subscription just by naming a channel. Every subscription that goes unlistened is logged at WARNING, never INFO. Subscription ids are deterministic per channel precisely so one can be replaced or closed without disturbing the others on the socket, and_chat_subscriptionsis per-socket state cleared on session end, on re-auth (a pre-auth REQ may have been rejected), and bystop(). Three bounds on remote-fed state:MAX_CACHED_CHANNELS(512) caps the kind-39000 metadata cache, the watermark map, and the resubscribe-attempt map;MAX_CHANNEL_SUBSCRIPTIONS(256, well under buzz-relay's own 1024-per-connection ceiling) caps live chat subscriptions — at the cap new channels are refused and named in a warning rather than evicting a working subscription. Known bound (documented, not fixed): the relay caps historical delivery at 2000 events per subscription, newest-first, even with asince, so >2000 unread messages in a single channel across a disconnect loses the oldest — the relay never sends them and the watermark advances past them. That is the one remaining path that can skip; everything else fails toward replay. Trust model (operator-facing version in IM_CHANNEL_CONNECTIONS.md): every inboundEVENTis authenticated at the singlehandle_relay_framechoke point — the NIP-01 id is recomputed from the delivered payload and the BIP-340 Schnorr signature verified against the claimedpubkey(buzz_nostr.verify_event, pure and total: malformed input returnsFalse, never raises) — soev["pubkey"], the authorization principal for both the allowlist and the/connectbind, cannot be forged by a relay the DeerFlow operator does not run. What remains trusted is the authorship of kind-39000 channel metadata: any member can sign one, and because per-channel subscriptions are now driven by discovery, a forged kind-39000 has two effects rather than one — it can mark a channeltype: "dm"(relaxingrequire_mentionfor that channel) and it can induce a chat subscription for a channel of the forger's choosing, since the channels we listen to are exactly the channels we hold metadata for. Neither makes anything be acted on:allowed_usersand per-event signature verification are independent gates, so an induced subscription only means the relay reads its own traffic back to a subscriber that drops it, bounded byMAX_CHANNEL_SUBSCRIPTIONS(which refuses rather than evicts, so it cannot displace a real channel). Same for a forged kind-44100, except itsptag is re-checked locally so it must at least name us. Closing this needs a configured trusted relay pubkey, whichrelay_urlis not.allowed_usersis deny-by-default (empty = nobody, unlike siblings' empty = everyone), sostart()logs a WARNING when it is empty and each drop logs at DEBUG. The resubscribe cursor (since) is per channel, advances only for events that were actually processed, and never pastnow + MAX_FUTURE_SKEW_SECONDS, because it is peer-supplied (created_at) and a single future-dated event otherwise made the connector permanently deaf. Per channel rather than global is the safety-critical half: subscriptions are per channel, so one shared cursor is the newest event seen in any channel and a busy channel would drag it past a quiet channel's unread messages, skipping them on the next reconnect — measured on a live relay, three channels of one identity sat ~28h apart. Per-channel cursors can only ever cost duplicate delivery (absorbed by the manager'sevent_iddedupe), and an evicted cursor degrades to "nosince", i.e. the relay's default backlog — both fail toward replay, never toward a miss. Streaming tracks every oversize chunk index (_stream_targetsfor chunk 0,_stream_tailsfor the rest), since the manager republishes cumulative text and repostingchunks[1:]per update flooded the channel; all of it is per-connection state cleared bystop(), and the remote-fed kind-39000 cache is capped atMAX_CACHED_CHANNELS.send()refuses outright to publish text carrying a hidden model-context wrapper (<memory>,<durable_context_data>,<system-reminder>—_HIDDEN_CONTEXT_MARKERS), logging at ERROR and clearing the stream bookkeeping on a blockedis_final. This is defense in depth behind the manager's allowlist, and it lives here rather than in a sibling connector because on Buzz a leak is permanent: every streaming update is an immutable public Nostr event, so a corrective edit only changes what clients render while the original leaked event stays on the relay. Matching is on the literal opening tag, so a reply that merely talks about memory is still published Buzz seen-event shutdown stays bounded and retryable:aflush()awaits any in-flight write, attempts at most one final snapshot, leaves a still-changing generation dirty for fail-open replay, and returns without a live persistence timer. A Gateway cancellation does not cancel the underlying worker-thread write;BuzzChannel.stop()tracks cleanup completion separately from transport admission so ChannelService can retry the retained channel without racing a second snapshot against that write. The store is quiesced before stop (including the already-stopped guard), so a timed-out relay task that records after stop only marks data dirty and cannot schedule detached file work; a repeatedstop()still drains that dirty state, whileBuzzChannel.start()explicitly resumes scheduling and flushes it automatically.github.py- Webhook-driven GitHub channel. Inbound messages come fromPOST /api/webhooks/github; outbound is log-only because GitHub agents post explicitly withghfrom their sandbox when they choose to comment or create a PRapp/gateway/routers/channel_connections.py- Browser-facing user connection and disconnect APIsdeerflow.persistence.channel_connections- SQL-backed user-owned connection, optional credential, connect state, and conversation store
Message Flow:
- External platform -> Channel impl ->
MessageBus.publish_inbound()- For GitHub, the webhook router verifies the delivery then calls
fanout_event(bus, ...); matching agent bindings publish oneInboundMessageeach instead of a long-polling channel worker. - Telegram photo/document updates use the largest photo size or document metadata, preserve
message.caption, enforce the hosted Bot API's 20,000,000-byte download ceiling before and after download, and never expose the token-bearing Bot API file URL. Downloaded bytes cross the adapter/manager boundary only throughmessage_bus.INBOUND_FILE_CONTENT_KEY; the manager consumes that transient field before persisting safe upload metadata. - Feishu/Lark inbound image/file downloads read at most 20,000,001 bytes and reject anything above 20,000,000 bytes before persistence or sandbox sync. Oversize, unsafe-path, and path-resolution failures rewrite only that attachment placeholder to
Failed to obtain the [type]so later attachments in the same message can still load. - WeChat iLink inbound image/file downloads stream with an in-flight cap derived from
channels.wechat.max_inbound_image_bytes/max_inbound_file_bytes(20 MB / 50 MB defaults) that aborts before the full read, and the payload-suppliedmedia.full_urlmust pass a scheme + dot-boundary host-suffix allowlist (channels.wechat.allowed_media_hosts, defaulting to theqq.comfamily plus the configuredcdn_base_urlhost) before anything is fetched. The limits are PLAINTEXT limits while the stream measures CIPHERTEXT, so the in-flight cap is the PKCS#7-padded size of exactly-limit plaintext (WechatChannel._stream_cap_for); the exact post-decryption check stays as the authority. There is deliberately no URL re-fetch fallback:_read_wechat_inbound_file(manager) only reads the locally stagedpath— the channel always stages one before publishing — so nothing fetches a WeChat URL outside the channel-side gate. WeCom media URLs (frame-suppliedurl, no per-channel size config) get the same style of host gate —qq.comsuffixes plus the COS shapeww-aibot-img-<APPID>.cos.<region>.myqcloud.comWeCom serves signed links from, where the numeric suffix must be one of the verified WeCom-owned Tencent Cloud APPIDs (_WECOM_MEDIA_COS_APPIDS, currently only the1258476243observed in Tencent's published callback examples) because bucket names are user-chosen and any Tencent Cloud account can register a lookalikeww-aibot-img-*bucket; other accounts go through operator suffixes fromchannels.wecom.allowed_media_hostsresolved from the live channel at read time — and the manager-levelMAX_INBOUND_URL_FILE_BYTES(50 MB) streaming cap. Both URL downloads and_download_cdn_bytesrequestAccept-Encoding: identity, refuse a response with any residualContent-Encodingbefore reading, and iterateaiter_raw(): httpx's transparent decoding would allocate the full decompressed body before the byte cap sees a chunk. No URL-based inbound-media log line may contain any part of the URL (signed links carry credentials in path and query): failure labels use the attachment filename or host only (manager._inbound_file_label), and reader failures log a sanitized exception summary — class name plus HTTP status (manager._reader_error_summary,wechat._media_download_error_summary), never the URL-bearing message or traceback. The rule extends to the HTTP client libraries' own records: httpx emitsHTTP Request: GET <full URL>at INFO via the barehttpxlogger, and urllib3 — enumerated against the installed source; every remaining emitter logs host:port only (connection establishment/reset) or one absolute URL (the header-parse warning) that the generic pattern rewrites — renders a request target in five shapes, each needing a dedicated redaction pattern because the generic absolute-URL pass cannot see a target with no scheme:Redirecting <target> -> <target>at INFO viaurllib3.poolmanagerand DEBUG viaurllib3.connectionpool(either slot may be an origin-form relative Location, which pool/connection callers pass or the Location header carries; the shape keeps the^Redirectingprefix anchor because-> /patharrows also appear in non-URL logs — sandbox mount mappings — and must pass through untouched, while both slots consume through end-of-line: the raw Location header may carry interior spaces, and the recursive urlopen frame can pass it on as the next request target), the per-requestscheme://host:port "METHOD /path?query HTTP/x.x"DEBUG line,Retry: <target>DEBUG andRetrying (…) after connection broken by '<error>': <target>WARNING — above the INFO root, so reachable at the default log level — viaurllib3.connectionpool, andIncremented Retry for (url='<target>')DEBUG viaurllib3.util.retry(origin-form on the request path, absolute on the redirect path). The generic pass'sresttreats a quote as a closing mark only at a boundary — followed by whitespace, a closing parenthesis, or end of string — so urllib3'sIncremented Retry for (url='…')keeps its')scaffolding while a quote embedded in a URL stays consumed and redacted. Logger filters only see records emitted through that exact logger — child loggers neither inherit them nor trigger them on propagation — soUrlRedactionFilteris attached to thehttpxlogger AND to every root handler byconfigure_logging(handler-level filters see propagated records), which also keeps Telegram's token-bearing Bot API paths (api.telegram.org/bot<token>/..., emitted by python-telegram-bot's httpx client) out of the logs.
- For GitHub, the webhook router verifies the delivery then calls
ChannelManager._dispatch_loop()consumes from queue- For user-owned channel connections, incoming messages carry
connection_id,owner_user_id, andworkspace_id;owner_user_idbecomes the DeerFlow runuser_id, while the raw platform user id remainschannel_user_id. The Gateway acceptschannel_user_idonly from an internally authenticated channel caller's top-levelbody.context, clears it from both free-formbody.configsections, and writes it into runtime context only (neverconfigurable, which is checkpointed).bash_toolexposes it to sandbox commands as the fixed env varDEERFLOW_CHANNEL_USER_ID— via a shell-quoted command-string prefix, NOT theexecute_command(env=...)channel, which is reserved for request-scoped secrets and would switchAioSandboxonto thebash.execpath (image >= 1.9.3, fresh session per call). Per-call injection keeps group-chat identity correct (one thread/sandbox, many senders) without depending on the AIO shell's session semantics: every IM-channel command carries an explicitexport VAR=<id>;(valid id) orunset VAR;(empty / non-str / over the 256-char cap). The AIO no-env path reuses a persistent shell session (the reason for the class lock, #1433), so a bare command could otherwise resolve a stale id an earlier sender exported; theunsetcloses the window the length/type guard would open (a dropped id would inherit the previous sender's value). Non-IM runs (nochannel_user_idin context) are left untouched. Not injected on the Windows local sandbox (its PowerShell/cmd.exe fallback has noexport/unset). Propagates acrosstaskdelegation:task_toolcaptures the dispatching turn's id and the subagent executor forwards it into the subagent's runtime context, same as the guardrail attribution fields. The runtime-context value is authorization-grade at the Gateway/guardrail boundary, but the exported shell variable remains informational because any bash command can overwrite its own environment; skills must not treat the shell variable itself as authenticated identity. Tests:tests/test_gateway_services.py,tests/test_channel_user_id_env.py - For chat: look up/create thread through Gateway's LangGraph-compatible API
- Feishu/Telegram chat:
runs.stream()→ accumulate AI text → publish multiple outbound updates (is_final=False) → publish final outbound (is_final=True) - Slack/Discord chat:
runs.wait()→ extract final response → publish outbound 6b. GitHub chat (ChannelRunPolicy.fire_and_forget=True):runs.create()returns once the run ispending; the manager does not wait for the final state and does not publish an outbound. The agent posts its own reply mid-run viaghfrom the sandbox.ConflictErroron a busy thread still trips the standardTHREAD_BUSY_MESSAGEpath (log-only on GitHub); when the channel's policy also setsbuffer_followups_on_busy=True(GitHub's default — see "Follow-up buffering while busy" below), the triggering message is additionally captured into a per-thread buffer instead of only logged, so a concurrent comment is not silently dropped. - Feishu channel sends one running reply card up front, then patches the same card for each outbound update (card JSON sets
config.update_multi=truefor Feishu's patch API requirement). Messages already sent inside an existing Feishu topic carry a compact source-message preview in that card, and queued same-thread follow-ups patch their own source message's card from queued → running → final without falling back to the generic busy reply. - Telegram streaming: the "Working on it..." placeholder message is registered as the stream target; non-final updates
editMessageTextit in place (channel-side throttle: 1s in private chats, 3s in groups due to Telegram's 20 msg/min group cap; 4096-char truncation; rate-limited updates dropped); the final update performs the last edit and splits >4096 texts into follow-up messages - DingTalk AI Card mode (when
card_template_idconfigured):runs.stream()→ create card with initial text → stream updates viaPUT /v1.0/card/streaming→ finalize onis_final=True. Falls back tosampleMarkdownif card creation or streaming fails - For commands (
/new,/status,/models,/memory,/goal,/agent,/help): handle locally or query Gateway API./agent listreads only the effective owner's Custom Agents;/agent use <name>validates in the same owner bucket, creates a new thread, and persistschannel_agent_namein its metadata. A Custom Agent also writes canonicalmetadata.agent_name, which makes thread-search results route Web continuation through/workspace/agents/<name>/chats/<thread_id>;lead_agentdeliberately omits that canonical key and stays on the ordinary chat route. The manager cacheschannel_agent_namefor the hot path and reloads it after restart before routing a resumed turn. An explicit selection also normalizesagent_nameacross top-level run context plus the existing RunnableConfigcontextandconfigurablecarriers (or clears all three forlead_agent) before Gateway'ssetdefaultcompatibility merge, so stale channel defaults cannot silently win. - Outbound → channel callbacks → platform reply
- GitHub is the exception: the channel logs the final assistant message and does not auto-post it to GitHub. Agents use the sandbox
ghCLI (gh issue comment,gh pr comment,gh pr create, etc.) for intentional writeback, so silence is cheap when several agents fan out on the same event.
- GitHub is the exception: the channel logs the final assistant message and does not auto-post it to GitHub. Agents use the sandbox
Owner-scoped file storage: inbound files, uploads, and output artifacts are staged under the DeerFlow owner's bucket so they land where the agent run reads/writes (users/{user_id}/threads/{thread_id}/user-data/{uploads,outputs}). ChannelManager._handle_chat resolves the storage owner once via _channel_storage_user_id(msg) (sanitized owner id, falling back to safe(msg.user_id) for unbound auth-enabled channels — mirroring _resolve_run_params's run identity; None only when no identity is available) and threads it as the user_id= kwarg through the file pipeline:
Channel.receive_file(msg, thread_id, user_id=...)— owner-bound channels persist downloaded files under the owner's bucket instead of the default bucketFeishuChannel._receive_single_file(...)/DingTalkChannel._receive_single_file(...)— normalize provider filenames, claim a collision-free basename and write it throughwrite_upload_file_no_symlinkunder the same channel lock; the returned basename drives both the agent-visible virtual path and non-local sandbox syncsandbox_files.py— non-mounted Feishu/DingTalk syncs acquire unique non-releasing execution holders, drain blockingupdate_fileworkers across repeated cancellation, and release only after the last sandbox operation, so a parallel run cannot close the shared client mid-upload_ingest_inbound_files(...)and the underlyingensure_uploads_dir/get_uploads_dir— owner-scoped via the same kwarg_resolve_attachments/_prepare_artifact_delivery— resolve output artifacts from the bound owner's bucket throughapp.gateway.path_utils.resolve_outputs_confined_path, the same outputs-only rule the artifact editor uses, so a siblinguploads//workspace/path or a symlink planted inoutputs/is skipped with a warning The cached value is reused for both the blocking (runs.wait) and streaming (_handle_streaming_chat) paths, so uploads and artifact delivery always target the same bucket even if a channel returns a rewrittenInboundMessagefromreceive_file. The bucket id matches the memory bucket resolved by_resolve_memory_user_id(both normalize throughmake_safe_user_id).
Configuration (config.yaml -> channels):
langgraph_url- LangGraph-compatible Gateway API base URL (default:http://localhost:8001/api)gateway_url- Gateway API URL for auxiliary commands (default:http://localhost:8001)- In Docker Compose, IM channels run inside the
gatewaycontainer, solocalhostpoints back to that container. Usehttp://gateway:8001/apiforlanggraph_urlandhttp://gateway:8001forgateway_url, or setDEER_FLOW_CHANNELS_LANGGRAPH_URL/DEER_FLOW_CHANNELS_GATEWAY_URL. - Per-channel configs:
feishu(app_id, app_secret),slack(bot_token, app_token),telegram(bot_token, optionalrich_messagesfor final Markdown Rich Messages),dingtalk(client_id, client_secret, optionalcard_template_idfor AI Card streaming),github(operator kill-switchenabled, plusdefault_mention_loginfor mention-required GitHub triggers),buzz(relay_url, private_key)
User-owned channel connections (config.yaml -> channel_connections):
- Disabled by default. It is a user-binding layer on top of the existing
channels.*runtime config, not a replacement for provider bot credentials. - No public IP, OAuth callback URL, or provider webhook route is required by the current implementation.
- Telegram uses a deep-link
/start <code>flow over the existing long-polling worker. Slack, Discord, Feishu/Lark, DingTalk, WeChat, and WeCom use/connect <code>over their existing outbound channel workers. - WeChat timing settings (
polling_timeout,polling_retry_delay,qrcode_poll_interval,qrcode_poll_timeout) accept only positive finite seconds; invalid values fall back to their defaults so polling cannot enter a hot loop or sleep forever. - WeCom serializes
start()andstop()for each channel instance. The SDKconnect()task covers connection setup only; after the handshake, the SDK owns a separate receive task. Shutdown cancels an in-progress connection attempt and awaits the SDK's actual asynchronous receive-task/socket cleanup before releasing lifecycle state or allowing a restart. Cancellation ofstop()still propagates, but only after owned cleanup finishes and lifecycle references are cleared; real connection failures remain reported by_on_ws_task_done. - WeCom outbound content is capped at the protocol's 20480 UTF-8 bytes: stream replies clip on a character boundary with a truncation marker (one stream carries the whole reply and cannot split mid-way), while proactive pushes split into at most 10 sequential markdown messages per push, with the remaining tail clipped and marker-terminated. Both paths measure bytes, not characters. A per-chat send lock serializes each split batch end to end, because manager workers run concurrently and two long pushes to the same chat would otherwise interleave chunks; locks are reference-counted and reclaimed once no sender holds or waits on them, so the registry does not grow for the life of the Gateway.
- Frontend APIs:
GET /api/channels/providers,GET /api/channels/connections,POST /api/channels/{provider}/connect, andDELETE /api/channels/connections/{connection_id}. - Browser APIs remain protected by normal Gateway auth/CSRF. Provider messages arrive through the already-configured channel workers.
- Provider-level
connection_statusreflects the user's newest connection row. With no binding it isnot_connected, except in auth-disabled local mode where a configured running channel reportsconnectedbecause all channel messages already route to the default user. - Slack replies use the configured operator bot token from
channels.slackunless per-connection credentials are present; unreadable or corrupt stored credentials are treated as unavailable. - Telegram, Slack, Discord, Feishu/Lark, DingTalk, WeChat, and WeCom workers resolve incoming platform identities to connection records before reaching
ChannelManager. - Connect-code ordering vs
allowed_users: inbound workers consume a valid/connect <code>(or Telegram/start <code>) before applying theallowed_usersfilter, so a newly allowlisted-but-unbound user can bootstrap their first bind via the browser flow. Consequence:allowed_usersis not a bind-time defense — any sender who possesses a valid code can consume it (not only allowlisted users). The bind security model rests on the code's confidentiality:secrets.token_urlsafe(16), 600 s TTL, one-timeconsume_oauth_state, and codes surfaced only in the initiating browser (never echoed to chat).allowed_usersstill gates ordinary (non-bind) messages. - Single-active-owner transfer semantics: an external identity is keyed by
(provider, external_account_id, workspace_id). The latest successful bind wins —upsert_connectionrevokes other owners' active rows for the same identity (ownership transfer). This invariant is enforced at the DB layer by the partial unique indexuq_channel_connection_active_identity(WHERE status != 'revoked'), so concurrent connects from different owners cannot both endconnected; the losing writer retries against the now-visible state.find_connection_by_external_identitytherefore resolves deterministically. - See
backend/docs/IM_CHANNEL_CONNECTIONS.mdfor provider setup, operational notes, and the architecture diagrams (connect-code flow, single-active-owner transfer, sync vs streaming dispatch, owner-scoped file storage pipeline).
GitHub event-driven agents (webhook-driven IM channel):
- Custom agents declare a
github:block in theirconfig.yamlto bind to repos and event triggers; the webhook route is fail-closed by default (mounted only whenGITHUB_WEBHOOK_SECRETis set) and exempt from auth/CSRF because authenticity is enforced by HMAC. - Registry caching is keyed by the configured agent store's opaque signature: file storage uses agent config mtimes, while database storage hashes the ordered owner/name/config/soul contents so same-timestamp writes still invalidate webhook routing.
- Outbound is log-only by design: each agent posts its own reply mid-run via the
ghCLI from its sandbox, so the manager usesfire_and_forget=Trueandruns.create()returns once pending. - Follow-up buffering while busy (issue #4121): because outbound is log-only, the pre-existing
THREAD_BUSY_MESSAGEreply on aConflictErrorwas invisible to the commenter — a comment posted while a run was already active looked like it had been silently ignored. WhenChannelRunPolicy.buffer_followups_on_busy=True(GitHub's default), aConflictErroronruns.create()now also appends the triggering message to a per-thread, in-memory buffer (ChannelManager._followup_buffers) — deduped by GitHub delivery id, capped atFOLLOWUP_BUFFER_MAX_PER_THREAD(20, oldest dropped with a WARNING log on overflow). The first successfulruns.create()on a thread now captures itsrun_idand spawns a background watcher that subscribes to that run'sStreamBridgestream; once it observesEND_SENTINEL, the watcher drains up toFOLLOWUP_DRAIN_BATCH_SIZE(10) buffered entries into one<followups-while-busy>-wrapped input and fires a follow-upruns.create()— itself watched the same way, so a backlog deeper than one batch chains into further drain cycles instead of growing one unbounded input. If that follow-upruns.create()itself hitsConflictError(e.g. a manual Web UI turn or a scheduled run raced onto the same thread), the batch is requeued rather than lost, and is retried whenever this manager next successfully creates and watches a run on that thread. Reactions/acknowledgment (e.g. GitHub'seyes/confusedreaction API) on buffered comments are intentionally out of scope for this mechanism and left to a follow-up — comments are coalesced silently. Plumbing: the watcher needs the Gateway'sStreamBridgesingleton, whichChannelManagerdid not previously have access to; it is threaded fromapp.py's lifespan (whereapp.state.stream_bridgeis already set bylanggraph_runtime) throughstart_channel_service(get_stream_bridge=...)→ChannelService.__init__→ChannelManager.__init__, as a zero-arg closure mirroring the existinglaunch_run=lambda **kwargs: launch_scheduled_thread_run(app=app, **kwargs)pattern used forScheduledTaskServicein the same lifespan function. AChannelManagerconstructed without it (e.g. directly in a test) still buffers safely — it just has no watcher to auto-drain. Scope limitation: the buffer and watcher state are per-process, in-memory. UnderGATEWAY_WORKERS>1or multi-pod, a follow-up comment routed to a different worker process than the one running the busy thread's agent will not see that buffer. This is a known, deliberately deferred limitation with the same shape as the cross-pod gap described for issue #4120 (a shared buffer store or IM-leader election would be needed to close it) — single-process/single-pod deployments, the safe default, see no correctness issue from this, only the documented per-process scope. - See backend/docs/GITHUB_AGENTS.md for the architecture diagrams: webhook → fan-out →
InboundMessagedispatch,preferred_thread_id = UUID5(repo, number, agent_name)thread determinism, mention-handle precedence chain, GH token lifecycle viaGH_TOKEN/GITHUB_TOKENper-callextra_env, and the narrowConflictError(HTTP 409) thread-create race recovery.