159 Commits

Author	SHA1	Message	Date
Hyeonsang Cho	9c159ef880	fix(auth): key the compose login lockout by client, not by nginx (#6519) ... * fix(auth): key the compose login lockout by client, not by nginx Failed logins are counted per client IP, and the Gateway honors X-Real-IP only from a peer listed in AUTH_TRUSTED_PROXIES. The compose files never set it, and every browser request reaches the Gateway from the nginx container, so all logins shared nginx's address: five wrong passwords from anyone locked every user out for five minutes. AUTH_TRUSTED_PROXIES now also accepts hostnames, resolved off the event loop and cached for 10 seconds (failed lookups included, and only when X-Real-IP is present), and both compose files default it to the bundled nginx service, whose address Docker assigns. nginx overwrites X-Real-IP with $remote_addr on every Gateway route; a test pins that for the three maintained configs. * docs(changelog): reference #6519 * fix(auth): share one trusted-proxy lookup across concurrent cache misses A burst of setup-status / login requests on an expired hostname entry started one getaddrinfo per request while DNS was slow, occupying the default executor that password verification also uses. The cache check and an in-flight task registration now happen without a yield between them; later callers join that task through asyncio.shield, so a cancelled request cannot cancel the lookup the others joined, and the task caches its own result. The compose default keeps `:-` on purpose: every browser request arrives through nginx, so an empty AUTH_TRUSTED_PROXIES would only bring back the shared lockout. The compose comment and .env.example say so, and the rendered-compose test now covers the explicitly empty value. * fix(docker): keep a .env AUTH_TRUSTED_PROXIES under make docker-start docker.sh start runs Compose from docker/ without --env-file, so the dev compose file's ${AUTH_TRUSTED_PROXIES:-nginx} interpolated from the shell only, and the resolved environment entry outranked the operator's value loaded through env_file. Export it from .env alongside the proxy variables (a shell export still wins), and strip a CRLF before the quotes. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-09 17:22:45 +08:00
Nan Gao	b99ac916b6	fix(auth): count login lockouts in the database across Gateway replicas (#6501) ... * fix(auth): count login lockouts in the database across Gateway replicas POST /api/v1/auth/login/local throttled failed logins per client IP in a module-level dict, so with N Gateway replicas behind one load balancer an attacker got N x auth.local.max_login_attempts guesses and a lockout on one replica was invisible to the others. Move the counter behind a LoginThrottleStore (deerflow.persistence. login_throttle): MemoryLoginThrottleStore keeps the historical per-process semantics and 10k tracked-IP bound; SqlLoginThrottleStore shares one login_throttle row per IP through the application database, incrementing and starting the lock in a single dialect-native upsert, applying check decisions with compare-and-set predicates in a separate write transaction, and sweeping served locks plus 24h-idle counters in bounded batches. Both stores pin the existing contract: the lock starts when the count reaches max_login_attempts, the committed duration is honored when the policy shrinks mid-lock, a served sentence is never resurrected, tightening keeps the count, success resets the IP, and the 429 status and message are unchanged. New startup-only selector auth.local.throttle_storage (auto | memory | db, default auto = the database whenever database.backend is sqlite or postgres). The Gateway installs the store once at startup after the persistence engine; deps._validate_login_throttle_storage warns when a multi-process deployment keeps memory. Migration 0034_login_throttle creates the table; config_version 55 -> 56 in config.example.yaml and the Helm chart. Docs: README, CONFIGURATION.md, AUTH_DESIGN.md, config and migrations AGENTS.md, CHANGELOG. * docs(changelog): link the login throttle entry to #6501 * fix(auth): re-check expiry on the row the login-throttle sweep deletes The bounded cleanup in SqlLoginThrottleStore.record_failure deleted `WHERE ip IN (SELECT ip ... WHERE <served or stale> LIMIT 200)`. On PostgreSQL READ COMMITTED the expiry predicate applied only inside the snapshot-bound subquery: when the DELETE had to wait on a row lock held by a concurrent check() that was extending the same lock under a raised lockout_seconds, PostgreSQL re-evaluated only the outer WHERE on the updated tuple, IP membership still held for the old candidate, and the now-active lock was deleted; the next attempt from that IP got a clean record and reached password verification (review of #6501, P2). Repeat the served/stale predicate on the DELETE target as well (SqlLoginThrottleStore.sweep_statement), keeping the bounded candidate subquery, so the re-evaluation sees the updated row's expiry. The other writes already carry their full compare-and-set predicate on the target row: _discard and the check() UPDATE both use _matches (ip, fail_count, locked_at, lock_duration_seconds), so a tuple changed while they waited no longer matches and the stale decision is skipped. _discard keeps the equality compare-and-set rather than adding an expiry condition: its early-release branch deletes a lock whose committed sentence has not elapsed under the lowered live policy, which an expiry predicate would wrongly block, and the equality predicate already pins every input of the decision. reset stays unconditional by design. Tests: a dialect-compile test asserts the DELETE's own WHERE carries the expiry predicate for postgresql and sqlite (red on the previous HEAD), and a PostgreSQL-only test (skipped without TEST_POSTGRES_URI) reproduces the update-versus-sweep interleaving with two engines, waiting on pg_stat_activity for the sweep to block before the extender commits. Docs: sql.py concurrency contract and the migrations guide's 0034 bullet. * fix(auth): keep an active login lock's clock and fail closed on explicit db Address the "changes requested" review on #6501. Lock clock (confirmed defect): record_failure re-stamped locked_at = now for every failure on an already-locked row, so a sustained attacker's sentence slid forward with each attempt. Both stores now apply one rule in one atomic decision: new_count = fail_count + 1; below max_attempts the IP counts with no lock (a failure after the operator raised the threshold clears an existing lock); at or over it, a failure during an *active* lock keeps locked_at and the committed lock_duration unchanged, while a never-locked or served row starts anew with now / lockout_seconds (so the failure that reaches a raised threshold starts a fresh lock rather than keeping a stale locked_at). The SQL upsert expresses it as nested case() on the row's own values; the memory store mirrors it. The sweeps in both stores now leave the row being recorded to that decision instead of deleting a served lock or stale counter first, which split the contract (SQL would have produced a fresh (1, NULL, NULL) row). base.py states the rule. Explicit db downgrade: resolve_login_throttle_store raises SystemExit — the failure type of the other startup validations in app.gateway.deps — when throttle_storage='db' resolves to the shared table but no engine is initialised, so langgraph_runtime refuses to start instead of silently running every replica on per-process counters; the lazy bare-app path applies the same rule and installs nothing. 'auto' keeps warning and falling back. 'db' on a memory database still degrades with a warning (there is no shared table to insist on). Non-blocking items: _key() logs a warning when a client key is truncated to 255 characters (never the key itself); the sql.py docstring covers wall-clock locked_at across replicas (run NTP; skew shifts evaluation by that much) and why STALE_COUNTER_SECONDS=24h / SWEEP_BATCH_SIZE=200 were chosen; a store-level blocking-I/O anchor covers the SQL reset path; the _validate_login_throttle_storage warning was verified to render 'throttle_storage=memory' for the StrEnum (the gate test now passes the enum member and asserts that text). reset stays unconditional: a peer failure that commits after a reset leaves fail_count=1, one genuine failure after the cleared history, which a new test documents. Docs: CHANGELOG entry, config.example.yaml, CONFIGURATION.md, AUTH_DESIGN.md and the throttle_storage field description. * fix(auth): re-evaluate a login-throttle check whose compare-and-set missed SqlLoginThrottleStore.check applied its decision (commit the live duration, or clear a served / released lock) with a compare-and-set predicate on the snapshot it had read, but ignored the affected-row count. When a peer changed the row between the read and the write — an admitted wrong password incrementing fail_count on another replica, or a successful login clearing the row — the UPDATE or DELETE matched zero rows, yet check reported the decision as applied: it could return an extended 60s sentence while the row kept its 1s duration, so the next check cleared the lock and let password verification through; a live duration decrease could be dropped the same way; and a row a peer had just cleared was still reported as locked (#6501 review, P2). Every compare-and-set write in check now checks result.rowcount. On a miss the row is re-read and re-evaluated from the top, up to CHECK_CAS_ATTEMPTS (3) times, applying a fresh compare-and-set decision on the new snapshot and never clobbering a reset or replacement: a row that is gone is reported as allowed, a fresh peer lock is reported as locked, and an incremented count gets the extension or decrease committed on it. Exhausting the bound fails closed: nothing is written and the answer is the last snapshot's lock under the live policy, logged at debug. The read stays in its own short transaction and each write in a separate one. The evaluation is now one shared pure function, evaluate_check in base.py (CheckDecision), used by both stores so the contract cannot diverge. The memory store keeps a single pass: its compare-and-set cannot miss because there is no await between the read and the write on the loop thread, which its docstring now states. Tests (red first on df26c2acf) use a session proxy that runs the peer mutation on a second engine right after the snapshot read ends its transaction: the reviewer's raised-duration interleaving (59.5s reported and committed, the increment survives, a later check still locked), the mirrored decrease, early release on the fresh row, a reset landing between read and write (0.0, no row recreated), and the bound-exhausted fail-closed case. * docs(changelog): resolve merge markers left in the Unreleased section * test(migrations): pin 0035_login_throttle as the head in the 0034 watermark test Upstream #6477 added test_migration_0034_run_event_seq_watermark.py asserting 0034 is the single head; this branch chains 0035_login_throttle after it, so the test now pins the current head and walks the chain like its siblings. * perf(auth): one session per login-throttle check, one transaction per failure Two non-blocking performance suggestions from the #6501 review. Single session per check. The router probed the row with store.get() and then called store.check(), which re-read it: a recorded IP cost two connection-pool checkouts and two SELECTs per request on the unauthenticated login path, letting a lockout flood compete with legitimate traffic for Postgres connections. LoginThrottleStore.check now takes the policy as an async callable (LoginThrottlePolicy) and resolves it lazily inside the store: the probe, the policy resolution and the compare-and-set loop share one session, a clean IP returns 0.0 without ever calling the policy (the config.yaml read is still skipped for the common case), and a recorded IP resolves it exactly once, also across compare-and-set retries. The decision is made on a snapshot read after the resolution: the policy read is a yield point, and a decision that needs no write has no compare-and-set to catch a row a peer changed meanwhile — so one session, but still a read after the yield. The memory store takes the same signature and re-reads its record after the await for the same reason. The router passes its asyncio.to_thread policy read as the callable and drops the separate probe; get stays on the protocol for tests and anchors. No short-TTL verdict cache: it would mask a successful login's reset for the TTL window, and the single session already halves the hot path. Sweep in the upsert's transaction, plus an index. record_failure ran the housekeeping DELETE in its own transaction and the upsert in a second one, two commits (two WAL flushes on PostgreSQL) per failed login; the sweep now rides in the upsert's transaction with one commit, the keep exclusion already protecting the row being recorded. The new ix_login_throttle_updated_at (ORM __table_args__ and the unreleased 0035 migration, created after the table and dropped before it, each guarded separately so a table left by an interrupted attempt still gets its index) serves the sweep's stale-counter predicate; the served-lock predicate scans a table the sweep itself keeps small. Tests (red first on a137fbc47): a clean IP never resolves the policy and a recorded IP resolves it exactly once (both stores), the SQL store opens exactly one session per check and commits once per record_failure (counting wrappers), the migration test asserts the index after upgrade, after completing a partial upgrade that lacked it, and gone after downgrade while compare_metadata stays empty; the router-level "mutated during policy resolution" tests now inject their mutation through the policy callable. * docs(changelog): note config_version 57 in the login throttle entry	2026-10-08 22:35:19 +08:00
NanPan	62c54238d2	fix(channels): fence restart during service shutdown (#6503) ... * fix(channels): fence restart during service shutdown * test(channels): fence restart during service stop * test(channels): exercise restart after shutdown channel snapshot * docs(channels): document service shutdown publication fence * fix(channels): keep agent guidance within CI hard budget * fix(channels): fence publication after shutdown completes * test(channels): cover resume after completed service shutdown * test(channels): mark direct-start fixtures as active services * test(channels): mark direct-start fixtures as active services * style(channels): format shutdown regression test * fix(channels): drain channel startup completing after service stop * test(channels): ensure late channel startup is drained after stop * style(tests): format channel shutdown regression --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-08 22:29:49 +08:00
Nan Gao	4bab280250	fix(skills): share skill cache invalidation across Gateway replicas (#6495) ... * fix(skills): share skill cache invalidation across Gateway replicas SkillStorage.load_skills() rescans disk and re-reads extensions_config.json on every call, but the prompt layer in deerflow/agents/lead_agent/prompt.py caches the enabled-skills list, the per-(app_config, user_id) entries and the rendered <skill_system> section per process, and every skill mutation in routers/skills.py only refreshed the process that handled it. With several uvicorn workers or several Pods sharing one home volume, a skill installed, edited, deleted, rolled back or toggled through replica A kept being offered as the old skill set by replica B until B restarted, and POST /api/skills/reload had to be sent to every Pod separately. Reuse the MCP cache-reset mechanism. A new generic helper, deerflow.config.shared_reset_marker, owns a hidden JSON marker beside the resolved extensions config (.<config name>.<suffix>.json): publish() replaces it atomically with a random generation under extensions_config_write_lock + extensions_config_file_lock, chaining previous_generation and an optional user_id; SharedResetMarkerTracker polls it at most once per second (monotonic clock, never blocking behind a concurrent poller), adopts the current state silently on its first poll, and classifies a change as user-scoped only when exactly one publication carrying that user happened since the last poll. deerflow/mcp/cache.py now builds its .<config>.mcp-cache-reset.json marker through the same helper with no behavior change. prompt.py publishes .<config>.skills-cache-reset.json via publish_skills_cache_reset(user_id=...) and checks the marker on every cache lookup (global cache, per-config cache, prompt-section LRU, user scoped entries), calling _invalidate_enabled_skills_cache() or invalidate_user_skill_cache(). The publishing process remembers its own generation so it does not retire the caches it just rebuilt. Every mutation tail in routers/skills.py (install/upload, edit, delete, rollback, enable/disable, reload) refreshes locally and then publishes the marker off the event loop; /skills/reload answers scope=shared_config, or scope=process when no extensions config path resolves, and a marker publication failure is a 500 so a 200 never means only one process was refreshed. Docs: README, backend/docs/API.md, skills/config/mcp/gateway AGENTS.md, and frontend skills.mdx (en/zh; also fixes the reference to a non-existent skills/loader.py) describe the shared invalidation and drop the per-Pod guidance. CHANGELOG entry under Unreleased / Fixed. * docs(changelog): link the skills cache reset entry to #6495 * fix(skills): drain the reload tail and keep marker failures out of 404 Two review follow-ups on the shared skills cache reset (#6495). POST /skills/reload awaited the local refresh and the marker publication unshielded. CancelledError is a BaseException, so a caller that disconnected between the two slipped past the `except Exception -> 500` mapping, surfaced no error, and left every peer replica serving the old skill set -- the one outcome the operator hook for external mount writes exists to prevent. The refresh and the publication now run as one `await_drained(_drain_skill_mutation("reload", ...))` tail, the same discipline the install/edit/delete/rollback/toggle tails already use; the exception -> 500 mapping and the scope response are unchanged. Publication errors could surface as 404. resolve_shared_config_path() only swallows the resolution-time FileNotFoundError; a config directory that vanishes between resolution and the lock file / atomic replace still raised FileNotFoundError from the marker write, and the install, edit, delete and rollback handlers mapped that to HTTP 404 with the raw OS error text for a skill that had in fact been applied. publish_skills_cache_reset() now re-raises OSError (and a misconfigured project root's ValueError) as SkillCacheResetPublishError, a RuntimeError that no handler maps to 404/400, so every handler reports it through its generic 500 branch; the message names only the exception type, never a server path. The install handler gains an explicit clause that pins that contract beside its 404 mapping. Tests: cancelling reload after the refresh still publishes the marker; a FileNotFoundError from the marker write yields 500 (not 404) for install, edit, delete and rollback, 500 for the public toggle and the generic reload error; the wrapper type is not an OSError.	2026-10-08 19:43:04 +08:00
Xuehao Xu	b5af0b0e84	fix(projects): support Windows extended-length document paths (#6467) ... * fix(projects): support Windows extended-length document paths * fix(paths): preserve Windows device and extended namespaces	2026-10-08 10:21:18 +08:00
TINGyu123644	b33e82e392	fix(tests): skip MAX_PATH-exceeding blocking-io anchors on Windows (#6361) ... * test(skills): skip MAX_PATH-exceeding anchors on Windows (ref #6353) * test(skills): skip MAX_PATH-exceeding anchors on Windows (ref #6353) * test(skills): skip MAX_PATH-exceeding anchors on Windows (ref #6353) * style(tests): fix import order in project blocking-io tests * fix(tests): narrow Windows MAX_PATH skip to only the two failing attach anchors Per review: module-level skip also skipped ingestion/thread-files anchors that are not MAX_PATH-bound. Now only the two attach staging anchors skip on Windows. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-06 17:15:06 +08:00
XuRuuuy	a9c8f79dfe	fix(gateway): resolve the agent factory off the event loop (#6206) ... * fix(gateway): resolve the agent factory off the event loop start_run() called resolve_agent_factory() inline, which lazily imports the full lead-agent assembly stack (middlewares, authz, MCP, jsonschema; measured at ~8s cold on Windows, multi-second on slower filesystems). start_run is the single choke point every run-creation path flows through, so the first run after Gateway startup blocked the event loop for the whole import, stalling every other request, stream and channel on the same loop - the loop-stall family issue #5172 fixed for agent construction itself (the checkpoint-state path already resolves through the assembly pool). Resolve the factory via asyncio.to_thread: the first call pays one worker-thread hop for the import, later runs hit the cached module. Regression test pins the contract with a probe module whose module-level __getattr__ records the thread that performed the from-import. * fix(gateway): isolate factory resolution in the assembly pool Move import regressions into the strict blocking-I/O suite and cover default-executor isolation, request context, and failure-before-admission ordering. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-06 07:52:47 +08:00
Hyeonsang Cho	b9ee8af328	fix(skills): keep custom-skill edit and storage lookups off the event loop (#6332) ... * fix(skills): keep custom-skill edit checks off the event loop update_custom_skill built the user-scoped storage, ran the editability probes, validated the frontmatter through a temporary directory and read the content being replaced on the Gateway loop; only the write and history append had been offloaded. Run the pre-scan checks in one worker and the replaced-content read in another, matching rollback_custom_skill, and build delete_custom_skill's storage off the loop as well. * fix(skills): build skill storage off the loop in install and skill_manage The same sweep that moved the custom-skill edit checks off the loop finds two more storage lookups on it: the archive-install helper behind both install routes, and the agent's skill_manage tool, whose lookup resolves and stats config.yaml on every call. Offload both, and pin that the caller's user context still reaches the storage lookup now that the edit route resolves it inside a worker. * docs(changelog): reference #6332 for the skill storage off-loop fix --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-06 07:20:04 +08:00
Hyeonsang Cho	c2f31e440e	fix(knowledge): load the catalog's agent config off the event loop (#6313) ... * fix(knowledge): load the catalog's agent config off the event loop Both retrieval-catalog handlers called load_agent_config() directly from async code. The agent store is synchronous: file IO on the file backend, a sync SQLAlchemy round trip on the db backend, so a slow disk or database stalled every other Gateway request. Offload the load with asyncio.to_thread, matching agents.py, scheduled_tasks.py and the run-admission path in services.py. The strict Blockbuster anchors drive both handlers end to end against real file and SQLite agent stores, plus the unknown-agent 404 path. * docs(changelog): reference #6313 for the knowledge catalog fix * Added the Chinese changelog entry back --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-05 22:17:09 +08:00
siye566	d8ca2c9358	test(community): make URL offload regressions DNS-independent (#6314) ... * test(community): make URL offload regressions DNS-independent * test(community): reject unsupported DNS fixture families --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-05 21:53:55 +08:00
Hyeonsang Cho	0223d949bf	fix(threads): remove a deleted thread's files off the event loop (#6319) ... * fix(threads): remove a deleted thread's files off the event loop DELETE /api/threads/{id} called shutil.rmtree over the thread directory directly on the event loop, so deleting a thread with a large workspace stalled every other Gateway request and live SSE stream until the whole tree was gone. Branch copying in the same router already offloads its filesystem work through run_file_io; deletion did not. The removal now runs on the file-IO pool. It is wrapped in await_drained rather than bare-awaited: cancelling the await abandons the worker, not the removal, so the durable delete reservation would otherwise release while files are still being deleted, and a newly admitted run could write into a directory that is about to disappear. * docs(changelog): reference #6319 in the thread delete off-loop entry --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-05 21:27:46 +08:00
Hyeonsang Cho	52046aa82e	fix(projects): convert shelf documents outside the SQLite write lock (#6305) ... * fix(projects): convert shelf documents outside the SQLite write lock Lazy conversion ran pymupdf/markitdown inside the BEGIN IMMEDIATE transaction that serializes the derived/converted.md publish against trash and purge. On SQLite that lock is database-wide, so every unrelated writer (run status, thread metadata, the scheduler) waited for the whole conversion and failed with "database is locked" past busy_timeout (30s). Conversion now writes into .staging/ outside any transaction. The document row lock is taken only to revalidate the live row and atomically rename the output into place (publish_under_live_lock, renamed from convert_under_live_lock); a row trashed or purged meanwhile discards the staged output, so nothing is published after purge. A read-only probe skips converting rows that are already gone, a failed conversion is still revalidated so a purged row reports content_missing, and the conversion is drained on cancellation so its staging file is always removed. * docs(changelog): reference #6305 in the shelf conversion lock entry * fix(projects): share one conversion across concurrent first reads A first-read conversion holds a bounded file-IO pool thread for its whole duration. With the conversion moved out of the row lock, concurrent first reads of one document each converted on their own thread, so a burst of reads could starve unrelated offloaded file work. Route the conversion through a single-flight keyed by (user, document, project): the first reader leads and the rest wait on its result without holding a pool thread. The map holds concurrent.futures.Future values under a threading.Lock because the read tool also runs on the isolated subagent loop. Followers wait through asyncio.shield, so a cancelled follower cannot cancel the shared future, and a leader that is cancelled or raises hands the work to a follower instead of failing it.	2026-10-05 13:34:19 +08:00
xiaodu55	a0b7679edf	test(channels): skip the POSIX-bound mode assertions on Windows hosts (#6257) ... The channel runtime config store's atomic 0o600-temp + replace + chmod cannot land a POSIX mode on Windows, where chmod is a no-op approximation. Skip the two POSIX-bound tests and guard the trailing mode assertion of the chmod-failure test, keeping its platform- independent log-behavior coverage everywhere. Fixes #6256 Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>	2026-10-04 10:22:20 +08:00
Guo-Yixin	68014de157	fix(artifacts): invalidate digests after external file replacement (#6195) ... * fix(artifacts): invalidate digests after external file replacement * docs(gateway): keep artifact guidance within the module budget * fix(artifacts): validate oversized ranges by file identity --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-03 16:23:07 +08:00
Xuehao Xu	9b17bef630	feat(channel): add QQ WebSocket channel (#6081) ... * feat: add QQ WebSocket channel * fix: address QQ channel review feedback * docs: trim inherited agent guidance below CI limit * fix: align QQ gateway and passive reply limits with protocol * docs: keep merged channel guidance within CI budget	2026-10-03 14:32:44 +08:00
xiaodu55	6176b057a2	fix(mcp): drain the config read-modify-write across cancellation (#6210) ... * fix(mcp): drain the config read-modify-write across cancellation Route the five MCP config endpoints through the canonical run_drained_write helper and fold the tools-cache reset into the drained unit, so a cancelled caller cannot detach from the worker holding the extensions-config locks or skip the cache invalidation behind a settled write. Caller-facing HTTPException contracts re-raise unlogged. Fixes #6209 * fix(mcp): type the drain helper against the worker signature and unlist drain adopters - _run_drained_mcp_apply now mirrors the canonical helper's [**P, T] Callable[P, T] typing so each call site's argument list is checked; - AGENTS.md stops enumerating the adopted routers, so the line stops needing an edit on every new adopter. --------- Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>	2026-10-03 14:12:02 +08:00
Hyeonsang Cho	bb4f7ac5e2	fix(browser): pin browser connections to the SSRF-vetted addresses (#6201) ... * fix(browser): pin browser connections to the SSRF-vetted addresses The navigate screen and the per-request guard resolve a hostname to vet it, but Chromium resolved the name again to open the connection. A rebinding DNS server could answer both checks with a public address and the connection with a private or cloud-metadata one, and the browser followed it. Launch each private browser through a per-session loopback SOCKS5 proxy. Over SOCKS5 Chromium sends the hostname itself, so the proxy resolves it exactly once through the session's egress resolver and connects only to the addresses that resolver vetted. resolve_browser_egress applies the same allow_private_addresses policy as validate_browser_url, on top of a new url_safety.resolve_public_addresses helper that validate_public_http_url now shares. Loopback is proxied explicitly with bypass "<-loopback>". The tools, navigate_and_capture and the Live WebSocket all pass the resolver, because a thread's first caller fixes its launch options. CDP-attached Chrome is unchanged. Delegated fetch services (crawl4ai, Browserless, fastcrw) still resolve on their own side, which the Gateway cannot pin. * docs(changelog): reference #6201 in the browser egress pinning entry * fix(browser): bound egress proxy faults and close, and record the WebRTC gap Review follow-ups for the pinned browser egress proxy: - A resolver exception other than ValueError escaped the connection handler as an unretrieved task error. Log it with its traceback and reply SOCKS5 general failure (0x01), so a resolver fault stays distinguishable from a policy refusal (0x02). - Bound the resolve with its own timeout. The handshake timeout did not cover it, so a wedged getaddrinfo held the half-open tunnel until Chromium gave up. A timeout replies host unreachable. - Bound close(): Server.wait_closed() also waits for a connection whose handler registered after the writer sweep, which could stall session close and eviction for the 15 s handshake. A handler that starts after close now drops its connection without opening a tunnel. - Docs said a launched browser connects "solely" through the proxy. WebRTC UDP does not traverse a SOCKS proxy, so AGENTS.md, README (en/zh) and both changelogs now scope the guarantee to TCP and name the gap.	2026-10-03 10:05:47 +08:00
Linyao Du	616c6e2e36	fix(uploads): reject Windows staging aliases before batch ingestion (#6149) ... * fix(uploads): reject Windows staging aliases before batch ingestion * fix(uploads): cover shelf names and case-insensitive staging aliases * test(uploads): isolate legacy alias filesystem cases	2026-10-02 22:50:30 +08:00
Hyeonsang Cho	457aa3a492	fix(community): resolve SSRF-guarded hostnames off the event loop (#6140) ... * test(blocking-io): catch socket.getaddrinfo on the event loop Blockbuster wraps socket.socket methods but not the module-level resolver, so a synchronous hostname lookup on the loop passes the strict gate. The SSRF guard in deerflow.community.url_safety resolves with socket.getaddrinfo from async web tools, and the anchors added next stay green against that unfixed code without this rule. Add socket.getaddrinfo to _PROJECT_BLOCKING_RULES, scoped to app.* and deerflow.* callers. loop.getaddrinfo and asyncio.to_thread resolve in a worker thread and are not flagged. The full make test-blocking-io suite stays green with the rule active (187 passed). * fix(community): resolve SSRF-guarded hostnames off the event loop validate_public_http_url resolves hostnames with blocking socket.getaddrinfo. The crawl4ai and Browserless web_fetch, web_capture, browser_navigate, the Gateway browser navigate route, and the Live stream's navigate input and seed all called it directly from async code, so a slow DNS answer for a model- or user-chosen URL stalled the Gateway event loop for the whole lookup. Run the guard through asyncio.to_thread at each of those call sites, the same way personal_network already offloads it. The screening result is unchanged. fastcrw's web_fetch is a sync tool and personal MCP validation already runs in a worker, so neither needed a change. * docs(changelog): reference #6140 in the SSRF guard event-loop entry * fix(browser): run the Playwright request guard off the shared browser loop BrowserSession._install_request_guard screens every redirect hop, iframe and subresource by calling the SSRF guard inside the async route callback. The guard resolves hostnames with blocking socket.getaddrinfo, and that callback runs on the process-wide Playwright loop, which also pumps Live frames and input for every browser session, so one slow lookup stalled all of them. Run the guard through asyncio.to_thread there as well, and pin it with a strict-gate anchor that drives the real route handler. Also note in the url_safety docstring that the screen is check-time only: a caller that re-resolves at connect time is not pinned to the vetted IPs. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-02 22:33:48 +08:00
Zeren Wang	d3e8e78f3f	feat(authz): middleware-declared tool path Layer 1 now covers tools declared by middlewares (#6105) ... * feat(authz): extension tool authorization PR2 — middleware-declared tool path Layer 1 now covers tools declared by middlewares, which LangChain merges into the bound tool set after the host's explicit-list filter: - new agents/middlewares/tool_declarations.py: record the ordinary pass's verdict, collect declarations from the built stack, decide only the delta seeded by the first pass (a name denied for this build stays denied), narrow the stack build-locally on independent state-preserving copies (direct __dict__ write, refused up front for tools data descriptors, never mutating caller-owned instances), and verify the exact bound stack after the final normalization copy. - IsolatedMiddleware ships a state-preserving __copy__ so narrowing survives full/delta normalization and copies of copies. - TodoMiddleware degrades when write_todos is denied: no system-prompt injection, no completion/context-loss reminders, no jump_to. - Layer 2 forwards host-resolved tool provenance into AuthzRequest.context and binds the infrastructure exemption to the concrete host-created tool object instead of a name set (tool=None goes to the provider). - Wired into all four assembly paths (lead x2, native subagent with the decision offloaded via asyncio.to_thread, DeerFlowClient). Tests: new test_middleware_declared_tools.py (35) and test_layer2_tool_provenance.py (7); blocking-IO anchor pinning the subagent offload (teeth verified red->green); placement-guarantee and enforcement suites extended; executor test doubles converted for the async _create_agent. Behavior change called out in CHANGELOG: with authorization enabled and a tools policy that excludes write_todos, plan-mode builds no longer bind it (built-in RBAC default unchanged). * fix(authz): close residual gaps from review — narrow non-collectable declarations, dedupe candidates, warn on skipped pass - apply_declared_tool_view now removes non-BaseTool declarations from the bound view when authorization is enabled (they would otherwise bind via LangChain's create_tool auto-conversion with no provider decision); verify_declared_tool_view fails loudly if one survives normalization - decide_declared_tools dedupes never-submitted names so the provider sees each candidate once - subagent executor logs a warning when the provider is set but the seeded Layer-1 state is missing, instead of skipping the declaration pass silently - docs: middleware-declared tools must be BaseTool instances with usable names * fix(authz): fail loudly on non-list/tuple middleware tools declarations _iter_declared warned and returned () for a 'tools' attribute that is not a list/tuple, but LangChain's factory iterates the attribute as-is — so a set/generator/dict_values declaration bound with no Layer-1 decision. Raise DeclaredToolViewError instead, matching the module's fail-loud standard for every other un-narrowable shape. Disabled path remains a strict no-op. Docs updated to match.	2026-10-01 18:26:58 +08:00
xiaodu55	5b0ef628e2	fix(skills): drain custom-skill mutation tails across cancellation (#6078) ... * fix(skills): drain custom-skill mutation tails across cancellation * fix(skills): log drained skill mutation failures and drain state/install tails Address review feedback on #6078: record cancelled-then-failed mutation tails inside the drained unit (exception type only, per the managed-models precedent), and extend the drained tail to the remaining write-to-cache shapes in update_skill and the skill install path. * test(skills): pin the lock contract under the drained cancellation timing The cancelled update_skill caller now drains its mutation tail, so CancelledError surfaces only after the tail settles. Start the MCP writer without awaiting the cancelled task first, assert the section stays owned while the drained tail is in flight, then await both and assert the final order. --------- Co-authored-by: xiaodu55 <xiaodu55@users.noreply.github.com>	2026-10-01 10:38:35 +08:00
SuperSgdk	f74a290c42	fix(uploads): reject reserved staging filenames before batch ingestion (#6122) ... * fix(uploads): reject filenames reserved for temporary staging Guard thread and shelf upload names, validate SDK batches before copying, and preserve legacy shelf downloads. Add regression coverage and document the reserved staging namespace. * fix(uploads): report reserved names before HTTP batch ingestion Preflight reserved staging basenames before thread storage or sandbox setup. Return HTTP 400 with a rename hint; add HTTP and strict blocking-I/O regressions and update upload documentation. * fix(uploads): reject Windows-style staging basenames on POSIX Normalize backslashes before the HTTP reserved-name preflight. Add POSIX basename regressions for single and mixed batches and strict blocking-I/O coverage. Clarify the preflight scope and upload documentation. Follow-up to the review of PR #6122; AI-assisted with OpenAI Codex. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-10-01 10:22:38 +08:00
Hyeonsang Cho	fe22e3899a	fix(channels): keep blocking SDK teardown off the event loop in stop() (#6134) ... * fix(channels): keep blocking SDK teardown off the event loop in stop() SlackChannel.stop() called SocketModeClient.close() inline. close() joins the SDK's message-processor thread (~0.7s every call, measured) and waits for in-flight listeners, whose blocking Web API calls can run up to the WebClient timeout, so every run, stream and channel on the Gateway stalled during shutdown and POST /api/channels/{name}/restart. Feishu and DingTalk had the same shape: they joined their SDK threads inline with a 5s timeout, and since those threads only return on a fatal error, the join normally froze the loop for the full 5s. The teardown now runs through asyncio.to_thread. Slack detaches the client first so a still-queued connect skips it, and shields and tracks the close task so a cancelled stop() leaves it running and a retried stop() awaits the same close instead of closing twice. tests/blocking_io/test_channel_stop_offloop.py anchors all three stop() paths to the strict Blockbuster gate, driving a real slack-sdk client. * docs(changelog): reference #6134 * fix(channels): join the Discord client thread off the event loop in stop() DiscordChannel.stop() still joined its client thread inline with a 10s timeout. The thread usually exits right after the cross-loop client close, but a timed-out close or a slow _run_client() drain keeps it alive for the whole timeout, stalling the shared Gateway loop the same way the Slack, Feishu and DingTalk teardown did. Run the join through asyncio.to_thread so stop() matches the channels/AGENTS.md teardown rule, and add Discord to the strict Blockbuster anchor.	2026-10-01 10:00:53 +08:00
ZJPex	1129ee4e8a	feat: add bounded JSON syntax acceptance criteria (#5947) ... * feat: add bounded JSON syntax acceptance criteria * fix(subagents): authorize bounded local JSON acceptance reads * docs: keep JSON acceptance guidance within chain limit * docs: use English for JSON acceptance comments and guidance * fix(subagents): keep unauthorized JSON probes unverified	2026-09-28 15:13:34 +08:00
Hyeonsang Cho	b72753c226	fix(gateway): load skills off the event loop in the listing and detail routes (#5945) ... * fix(gateway): load skills off the event loop in the listing and detail routes GET /api/skills, GET /api/skills/custom and GET /api/skills/{name} called load_skills() inline. Resolving the caller's storage builds absolute paths, and load_skills walks every public and custom skill directory and parses each SKILL.md, so the work grows with the number of installed skills. #5747 moved the same call off the loop for get_custom_skill and documented why; these three routes were missed, and the strict Blockbuster gate raises BlockingError on each of them. Offload the load through one shared _load_user_skills helper (also used by update_skill's reload), and add a gate module covering the three routes including get_skill's 404 branch. * docs(changelog): reference #5945 in the skills listing blocking I/O entries	2026-09-28 09:18:29 +08:00
Hyeonsang Cho	5b8b69bd45	fix(gateway): keep /api/user-profile filesystem work off the event loop (#5935) ... * fix(gateway): keep /api/user-profile filesystem work off the event loop GET and PUT /api/user-profile resolved the per-user USER.md path (which builds absolute paths on every call), stat'ed, read, created the user bucket and wrote the file inline on the event loop, while every other handler in the custom-agent router offloads that work with asyncio.to_thread. The strict Blockbuster gate raises BlockingError on both; in production a slow disk stalls every other request on the worker. Offload the whole resolve-stat-read and resolve-mkdir-write sequences, capturing the effective user id on the loop first like the sibling handlers. The blocking-IO gate test for this router now covers both profile routes (missing profile, write, read back). * docs(changelog): reference #5935 in the user-profile blocking I/O entries	2026-09-27 15:06:49 +08:00
李靖	10bd065615	feat(agents): persist tool artifacts as resolvable handles (#4676) (#4929) ... * feat(agents): persist tool artifacts as resolvable handles (#4676) MCP and sandbox tools return file paths, URLs, and task ids that are lost when context compaction summarizes the conversation. This adds a per-thread artifact handle registry so the model can reference artifacts by short deterministic handles (art_xxxxxxxx) that survive compaction and resolve to real references at the tool-call boundary. - ThreadState.tool_artifacts: reducer channel capped at 100 entries, same-handle latest-wins, survives summarization compaction - artifact_registry: deterministic handle derivation from (thread_id, tool_call_id, call_index); extraction precedence structured_content -> file/image blocks -> conservative text scan; HTML-escaped model-context rendering with char budget - ArtifactCaptureMiddleware: before_model hook captures lightweight metadata (never content) and tracks handle consumption - ArtifactResolutionMiddleware: rewrites handles in tool-call args to real references before execution; unknown handles untouched - DurableContextMiddleware: projects the registry into the hidden durable-context data block ([available]/[consumed] status) - config: new tool_artifacts section (all switches default on) - frontend: artifact badges on tool-call steps keyed by tool_call_id Fixes #4676 * fix(agents): harden artifact registry after review (#4676) Review findings on #4929: - capture/consumption updates emitted in the same before_model call are now concatenated instead of dict-merged; dict-merging clobbered the shared tool_artifacts key and permanently dropped fresh captures - handles embed a per-result ordinal so multiple references inside one tool result never collide (two paths in one text block previously collapsed into one handle) - structured_content extraction pulls concrete refs from known keys (file/path/url/task_id/..., string or list-of-string values) instead of storing a truncated JSON preview, falls back to an untruncated whole-object data entry, and no longer skips file/image content blocks - plain-string tool results (e.g. bash/write_file output) are scanned for sandbox paths / remote URLs too, gated by detect_refs_in_text - wire max_entries through configure_tool_artifact_max_entries at assembly (clamped 10..1000) and gate detect_refs_in_text scans - resolution requires both enabled and resolve_handles_in_args, matching its docs - add real ToolCallRequest.override coverage and the spec 14.4 integration cycle (capture -> inject -> resolve -> consumption) * fix(agents): address second-round artifact registry review (#4676) Follow-up findings on #4929: - drop generic `output`/`outputs` from structured ref keys: prose under those keys no longer becomes a bogus file entry (falls back to the whole-object data entry; path-shaped refs are still caught by the free-text scan) - strip closing quotes, brackets, and backticks from detected refs so markdown- and JSON-wrapped paths resolve to clean real references - remove the process-wide cap setter: the reducer enforces only an absolute ceiling (1000) and ArtifactCaptureMiddleware applies the configured max_entries per agent when emitting updates, so agents with different configs in one process never share or flap a cap - skip extraction for tool messages whose tool_call_id already has registry entries and memoize known-empty results (bounded FIFO), so steady-state capture cost drops to the new message tail instead of a full-history rescan per model call - correct the inverted ordering claim: capture is position-independent in the wrap chain because it is a before_model hook reading state messages; fixed the assembly comment, AGENTS.md item 15, and the ordering test docstring CI parity verified locally: ruff check + format --check, full offline suite, blocking-io suite, collect-only, uv lock --check, check_agent_guidance, frontend format/lint/typecheck/build/tests. * fix(agents): sliding-window cap and URI gates for artifact registry (#4676) Third-round review findings on #4929: - max_entries is now a sliding retention window: fresh captures are always registered, and when the projection exceeds the configured cap the middleware appends a trailing {"op": "trim_to", "keep": N} directive that the reducer applies by evicting the oldest entries. The previous remaining-budget emission froze the registry at the cap, permanently dropping post-compaction artifacts — the exact failure mode #4676 targets — and silently disabled the round-2 rescan optimizations at the point threads grew longest - _track_consumption memoizes settled tool-call args in a bounded FIFO alongside the existing empty-result memo, so historical AIMessage args are not regex-rescanned every model call - file/image content-block URLs must be http(s) or path-shaped; data:/blob: URIs (MCP embedded resources carrying base64 payloads) can no longer enter thread state, tool args, or crowd out the render budget; protocol-relative //host forms are rejected as well - structured-content ref values go through the same URI-shape gate as content blocks, closing the embedded-payload hole via known keys such as url/urls - tool_artifacts.enabled now gates the whole middleware including consumption tracking via an early return in before_model Docs updated to describe the sliding window, URI gates, enabled scope, and both bounded memos. * fix(agents): final evictions and pending-aware consumption for artifact registry (#4676) Fourth-round review findings on #4929: - the sliding window interacted with the seen_call_ids skip to resurrect evicted entries every round: once an entry was trimmed, its tool_call_id left the seen set while its ToolMessage stayed in context, so the next model call re-extracted it, re-registered it as newest, and evicted something else — a never-converging rotation. The empty-result memo is now a unified _processed_results memo that records every extraction regardless of outcome, making evictions final; steady state over a saturated thread is quiescent (pinned by a multi-round test asserting no updates and zero extractions) - the consumption scan built its handle map from pre-capture state, so a call referencing a handle captured in the same before_model call missed, got settled as quiet, and its consumption mark was lost forever. _track_consumption now receives the pending capture entries and resolves against state + pending; a call settles as quiet only after every referenced handle resolved, so transient misses retry next round instead of permanently suppressing the mark AGENTS.md entry 15 updated to describe both memos and the eviction-is-final semantics. * fix(artifacts): address occurrence identity, PII and URL review findings * fix(artifacts): enforce resolved policy checks and bounded registry handling --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-27 11:09:41 +08:00
Bxl	41e8e75f56	fix: balance temporary sandbox leases for uploads (#3261) ... * fix: balance temporary sandbox leases for uploads * fix: address sandbox upload review feedback --------- Co-authored-by: Bxl <188532127+BXL1015@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-26 16:22:20 +08:00
yc2bgr8	cdb5028d88	feat(community): offload image search to a thread via asyncio.to_thread (#5788) ... * feat(community): offload image search to a thread via asyncio.to_thread Make image_search_tool async and run the blocking ddgs call in a worker thread with asyncio.to_thread, continuing the TODO.md planned feature ('Replace sync requests with httpx.AsyncClient in community tools'). The ddgs SDK exposes no async API, so instead of swapping the HTTP client the blocking DDGS.images() call is offloaded to a thread — the same pattern the infoquest/jina_ai tools use for readability extraction. Callers see no change in behavior. Tests invoke the async tool via ainvoke under @pytest.mark.anyio. * fix(image-search): offload config lookup with DDGS search * test(image-search): pin DDGS search off the event loop --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-26 15:21:47 +08:00
haochen	fc9fb2dea2	test(lark): make blocking I/O CLI fixture work on Windows (#5826) ... * test(lark): make blocking I/O CLI fixture work on Windows * docs(tests): document portable Lark CLI fixtures --------- Co-authored-by: Zhangchenhao-111 <285911435+Zhangchenhao-111@users.noreply.github.com>	2026-09-26 08:17:28 +08:00
yang rui	c814943215	feat(extensions): grant bounded host model invocation (#5796) ... * feat(extensions): grant bounded host model invocation * fix(extensions): retain model budgets and isolate schema validation * fix(extensions): normalize provider-origin model cancellation --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-25 15:24:09 +08:00
Zeren Wang	82671596f3	feat(authz): add plugin resource targets, provenance, action and management enforcement (#5842) ... * feat(authz): add plugin resource targets, provenance, action and management enforcement PR1 of the extension-tool-authorization series (docs/plans/2026-09-24-extension-tool-authorization-spec.md): plugin authorization plumbing, no new author-facing declaration. - `authz/plugin_targets.py` is the single encoder/validator for composite targets (`namespace/part`), with one validating composer behind three kind-checked constructors; invalid input raises instead of degrading. - `authz/plugin_authz.py` is the harness decision layer (`plugin_action`, `plugin_management`, plus the batch filters the page slice will consume): disabled is a no-op, an explicit deny raises, and provider errors follow `fail_closed` like the sandbox gate. - `POST /api/plugins/{ns}/actions/{name}` decides `plugin_action:invoke` after the action resolves and before the body is streamed. - `deerflow_extension_api.auth` publishes `require_plugin_management` / `arequire_plugin_management` with two `app.state` resolver keys; the Gateway installs both. Contract version 0.2.3 -> 0.2.4. - `tools/tool_provenance.py` records host-written plugin provenance for display (assembly descriptor, Layer-2 context later); no decision reads a label. - Plugin providers use a `(config signature, loop key)` cache so a loop-affine provider is never handed to another loop or to a sync caller. - RBAC gains `plugin_actions` / `plugin_management` aliases; `config_version` 47 -> 48 across the three mirrors. Tests: new plugin target/provenance/action/management suites plus a blocking-IO anchor whose offloads were reverted to prove red->green. * fix(authz): decide plugin resources from the request's config snapshot Review round 1 on PR #5842 (willem-bd). P1 — the decision layer no longer reloads the configuration. The six `authz/plugin_authz.py` functions take a required `app_config` snapshot (the same request-scoped config the Gateway used to resolve the provider), and `resolve_/aresolve_plugin_authorization` return it as a third element, so `enabled`/`fail_closed` can never come from a second read that drifted from the one that produced the provider. A snapshot the caller could not read at all is *unavailable*, not disabled: it fails closed as `authz.config_unavailable`. The Gateway's `_plugin_app_config` now separates an absent `config.yaml` (no gate, today's behavior) from a config that exists but cannot be read or validated (deny) instead of swallowing both through `safe_app_config`. P2 — `_validated_decision` requires `type(decision.allow) is bool` on both the sync `authorize` and async `aauthorize` paths, so a provider returning `AuthzDecision(allow="false")` is a malformed verdict that follows `fail_closed` instead of being read as an allow. Both fixes are covered by regression tests that were verified red against the pre-fix behavior (P1: 200 vs 403 and resolver True vs False; P2: 12 cases across fail_closed/fail_open and sync/async), plus tests pinning that an absent config and `authorization.enabled: false` keep today's behavior. * fix(authz): treat a request-time config absence as unavailable Review round 2 on PR #5842 (willem-bd). P1 — `_plugin_app_config[_async]` no longer maps a missing `config.yaml` to `None` ("no gate"). A Gateway cannot have started without a readable config (`app/gateway/app.py` reads it through this same accessor in `lifespan`), so absence *during a request* means the configuration this process was running on became unavailable — a hot reload, an atomic replace, a ConfigMap remount. It is unavailable, not disabled: the error propagates, `resolve_/aresolve_plugin_authorization` raise `_PluginAuthorizationUnavailable(fail_closed=True)`, the action dispatcher returns 403 and the management resolvers answer `False`. The dead `(None, None, None)` allow path is gone, the resolvers return a non-optional `AppConfig` as their third element, and callers key "authorization disabled" on `provider is None` alone. This matches the rule the decision layer already documented (`authz.config_unavailable`) and the public docs. P2 — `_deny_reason_code` is total: a `reasons` that is not iterable degrades the code to `authz.denied` instead of raising `TypeError` outside the guarded provider path, which turned a denial into an unhandled error. It is deliberately not folded into `_validated_decision`: `allow` was a real `bool`, so the verdict is a valid denial, and treating it as malformed would flip it to an allow under `fail_closed: false`. Both fixes are covered by regression tests verified red against the pre-fix behavior (P1: resolver True vs False, async `PermissionError` not raised, route 200 vs 403; P2: `TypeError: 'NoneType' object is not iterable` at `plugin_authz.py:113`). The action-route tests in `tests/test_plugin_contributions.py` now install a readable `authorization.enabled: false` config instead of relying on a config-less process, and `docs/full-stack-plugins.md` plus the PR1 decision log record the new rule. * fix(authz): apply the running policy when the plugin config is unreadable Follow-up to f03d2ea9 (review round 2 on PR #5842, willem-bd). The first attempt read the P1 finding as "absence is never disabled" and failed every config read closed unconditionally. That is too broad for this repo: a config-less host is deliberately treated as having no policy to apply (`_get_route_authorization_config` falls back to a disabled config, `sandbox_authz.safe_app_config` documents the same no-op for "CI runners and direct-call tests"), and the rule turned the config-less e2e plugin host (`extension_test_fixtures.bookmark_plugin_gateway`) into a 403 — the three `bookmark-plugin.spec.ts` cases lost the save action. Reproduced locally: `POST /api/plugins/community.bookmarks/actions/save` against that fixture host returned 403 before, 200 after this change. The rule is now "has this process loaded a configuration?", not "was the last read successful?": - No loaded config (`peek_loaded_app_config()`, new read-only accessor on `deerflow.config.app_config`) and no `config.yaml`: no policy to apply, so the gate stays a no-op — unchanged for CI runners, direct-call tests and hosts that mount only the plugins router. - A loaded config whose file disappears or cannot be read: *unavailable*, never *disabled*. The resolvers raise `_PluginAuthorizationUnavailable` carrying the failure flag of the configuration this process runs on (authorization disabled there -> no gate; enabled -> that policy's `fail_closed`, `True` by default), so an enabled Gateway answers 403/False in the window instead of letting protected plugin actions and management routes through. - A config file that exists but cannot be read or validated right now: fail closed when nothing is loaded (round 1's rule), otherwise the loaded policy decides. `peek_loaded_app_config()` is preferred over a Gateway-side startup snapshot: it reflects the last successfully loaded config, so a hot-reloaded policy is what applies during the window, and there is no lifecycle hook to keep in sync. Evidence: the three no-config/lost-config branches in `tests/test_plugin_management_guard.py` and `tests/test_plugin_action_authorization.py`, round 1's unreadable-config tests unchanged, `test_peek_loaded_app_config_survives_a_missing_file` in `tests/test_app_config_reload.py`, and the fixture-host probe above. Docs (`docs/full-stack-plugins.md`, `reference.mdx` en/zh) and the PR1 decision log state the final rule; the log records the superseded variant. * fix(authz): keep a denial whose reasons raise while being read * fix(authz): intersect plugin batch filters with the candidates The plugin_page/plugin_management batch helpers returned the provider's `filter_resources` answer verbatim, so a custom provider could hand the projections a target that was never asked about ("*", another namespace, a phantom page). Every other filter call site re-projects from its own candidate list; do the same here, state the intersection on the provider contract, and pin it with a regression test that fails on the old pass-through. * docs(plugins): map a management denial to 403 in the route example The contributed-route example called `arequire_plugin_management` and let the documented `PermissionError` escape. The Gateway installs no `PermissionError` handler, so a route copied from the example answered 500 on a normal policy denial instead of 403. Both example routes now translate the denial, and the surrounding bullets say why the translation is the route's job. * docs(extensions): point the 0.2.4 contract row at the PR that ships it Every other row in the version-history table links the pull request that shipped the version; 0.2.4 linked the authorization tracking issue instead. Both the English and Chinese tables now link this PR.	2026-09-25 14:46:12 +08:00
yetuge	4a92cb579a	fix(uploads): keep document-conversion stat and write-back off the event loop (#5830) ... * fix(uploads): keep document-conversion stat and write-back off the event loop convert_file_to_markdown offloads the conversion itself for files above 1 MB (documented in its docstring), but the two bookend filesystem calls still ran on the event loop: the size stat before the threshold check and the write of the converted markdown afterwards — the latter a potentially multi-megabyte write in the per-upload ingestion path. Route both through run_file_io (the dedicated file IO pool that also keeps the caller's context), matching the off-the-loop fixes already applied to the sibling upload/document paths. Add a tests/blocking_io anchor: with the converter patched to a 2 MB payload it fails on the previous head (Blockbuster catches the blocking os.stat first; the broad except swallows it into a None return) and passes with this change. * fix(uploads): address review — drain conversion write-back, make run_file_io re-entrant Review follow-up on the off-loop conversion stat/write PR: 1. The write-back now goes through await_drained(): a plain run_file_io await cancels immediately while the worker keeps writing, so the ingestion scope could remove its private directory over an in-flight write and leave a large hidden partial file. The drain delivers the cancellation only after the worker settled, and the handler removes the partial output before re-raising. 2. run_file_io is now re-entrant: a nested offload issued from code already running inside a file-io worker (project-document conversion runs asyncio.run(convert_file_to_markdown(...)) inside a run_file_io worker) executes inline instead of submitting to the same bounded executor, which deadlocked with a single configured worker. Both fixes carry deterministic regressions: the cancellation test pins the drain ordering (cancel issued before the worker is released) and the partial-file cleanup; the re-entrancy test pins inline execution by thread identity. Both fail on the previous head. Full blocking_io + project-document + conversion suites show an identical failure set to the previous head (pre-existing Windows platform failures only). * fix(uploads): keep cancellation authoritative when conversion cleanup fails If the post-cancellation unlink of the staging output raised, the new exception escaped the CancelledError handler, was swallowed by the broad except as an ordinary conversion failure (None return), and left both the caller's cancellation and the partial file behind. Catch and log cleanup failures inside the handler and always re-raise the captured cancellation; a second cancellation still passes through untouched (CancelledError is not an Exception). Red/green: tests/blocking_io/test_file_conversion_offloop.py gains a regression where unlink fails while the write-back is in flight — on the reviewed head the task settled on None (DID NOT RAISE); with the change the CancelledError propagates and the failure is logged. Signed-off-by: yetuge <2219677952@qq.com> --------- Signed-off-by: yetuge <2219677952@qq.com>	2026-09-25 14:22:34 +08:00
Zeren Wang	577c39643b	feat(guardrails): add TypeSafe (Jev) pre-execution tool risk gate (#5712) ... * feat(guardrails): add TypeSafe (Jev) pre-execution tool risk gate Adds TypeSafeGuardrailProvider, a network guardrail provider that asks one `noul` question per probed tool call and denies calls whose Jev risk probability reaches `threshold`. It reuses the existing GuardrailProvider slot: no middleware, config-schema or dependency changes. Semantics that are load-bearing: - state is strict JSON (no `default=`, `allow_nan=False`, string keys only); unserializable arguments, lone surrogates among them, and argument text above `max_state_chars` are denied locally with zero network requests, never truncated and sent. - the async path cancels in-flight requests through `asyncio.timeout`; the sync path checks the deadline after the response headers arrive, around the body read and before accepting the result, and drops late results. No total return-time bound is promised on the sync path. - clients are created per evaluation inside a context manager, and the injected `transport_factory` is called once per network-needing evaluation: httpx closes an injected transport with its client, so a shared instance would be dead on the second call. - denial reason messages carry probability, threshold, served model version and state digest, so the threshold comparison can be replayed from the run journal; local denials record no probability or model version, and allow decisions stay unpersisted (the audit path is unchanged). - provider errors carry no response body or arguments: malformed responses report the field and the offending type only, so a response echoing tool arguments cannot reach middleware logs or the evaluation report. The online evaluation harness (`backend/scripts/eval_typesafe_risk_gate.py` plus its labeled case set) is the threshold/tools evidence that must precede enabling interception. It reports four mutually exclusive populations (uncached network, cache hits, unprobed tools, local refusals), separates model classification from local refusals and fail-closed errors, measures latency from the caller entering `aevaluate` through cleanup, discloses censored samples, probes DNS/TCP/TLS separately, and flags unprobed tools as a coverage gap rather than a pass. Real responses are never written to disk. Cache-pass samples are classified from `decision.metadata['cached']` and counted once, warming calls are retained as an auxiliary group with their failures named in the report and JSON, and an optional connection-probe diagnostic can never discard completed evaluation results. Tests: the provider suite over MockTransport plus the middleware integration, a blocking-IO anchor that drives `aevaluate` against a real loopback HTTP server with a meta-check proving the sync path on the loop trips the gate, and harness cases covering the reporting defects. Every fix above was verified to fail its new tests when reverted. Docs: GUARDRAILS.md gains Option 4 (privacy, local-refusal trade-off, deadline semantics, audit boundary, coverage/bypass risk, Allowlist->RBAC migration, lone-surrogate local refusals and the no-response-body property of error messages) and config.example.yaml gains a smoke-scope example. Config: the new example block bumps `config_version` 46 -> 47, with the Helm chart's embedded version and README example kept in step as `scripts/check_config_version.sh` (chart.yaml CI) requires. * feat(guardrails): whitelist tool calls inside the TypeSafe risk gate Static allow rules now live in the provider instead of waiting on a composite provider. `whitelist` mirrors `AllowlistProvider.allowed_tools`: omitted means "not enforced", `[]` refuses every tool, and a list refuses every tool outside it locally (`typesafe.tool_not_whitelisted`) before state construction, caching and networking, independently of `fail_closed`. Whitelisted tools are still probed and still denied when their risk probability reaches `threshold`, and the whitelist is checked before the probe scope so an unlisted tool cannot inherit an allow from being unprobed. Deny rules stay with `authorization.*`. The policy version moves to 1.1.0, the evaluation script reports whitelist refusals as their own population, and `GUARDRAILS.md` / `config.example.yaml` document the allow-half migration. * fix(guardrails): address TypeSafe risk gate review findings - Status errors report only the numeric status. The HTTP status line is server-controlled like the body, and this message reaches middleware logs and the evaluation report, so a hostile endpoint could echo the state (raw tool arguments) through it. - Expired cache entries are removed with `pop(key, None)`. Sync evaluation runs on executor threads when a turn produces parallel tool calls, so a second caller can remove the same expired entry; a bare `del` raised KeyError, which the middleware reports as a provider error and, under fail_closed, turns into a spurious denial. - The evaluation harness states its acceptance gates inline (no missed dangerous call, system false-block <= 5%, uncached network p95 <= 1s) instead of citing a design spec that is not in the repository, and reports the whitelist-refusal population alongside the other four. - `--tools` / `--whitelist` strip whitespace and drop empty entries, so `--tools "bash, write_file"` no longer probes only bash while treating `" write_file"` as an unknown tool name. - The case set's `padding` note says top-level argument fields only, matching what the runner implements. Each fix was verified by running its new test against the pre-fix code. * refactor(guardrails): name the TypeSafe permission list allowed_tools Align the key with `AllowlistProvider.allowed_tools` so an allowlist migration copies it verbatim instead of translating `allowed_tools` into `whitelist`. The companion names follow: reason code `typesafe.tool_not_allowed`, metadata `tool_not_allowed`, harness population `not_allowed`, and the `--allowed-tools` flag. Semantics are unchanged: omitted = not enforced, `[]` = refuse every tool, tools outside the list are refused locally before any state, request or cache entry, and listed tools still face the risk gate. * fix(guardrails): record only a bounded served model token; fail the eval exit code Two findings from the re-review at d3f0b072. `model` is response content, and `_model_decision` interpolated it verbatim into the reason message -- which reaches the denied ToolMessage, the run journal, middleware logs and the evaluation report on disk, and which the cache retains for replay. A malformed or hostile endpoint can put an echoed tool argument, injected text or a multi-megabyte string there. The token is now recorded as itself only when it matches a conservative shape; anything else is recorded as `unrecorded:sha256:<digest>`, which keeps the verdict and a stable provenance record without echoing the value. Rejecting the response instead would turn an odd version string into a fail-closed denial of every call the endpoint answers. The digest path encodes with `surrogatepass` so an escaped lone surrogate -- a value `json.loads` happily produces -- cannot raise where this must not fail. `scripts/eval_typesafe_risk_gate.py` returned 0 even when a gate failed, so an operator scripting `... && enable` got a green light on a failing evaluation. `main()` now returns 1 when any gate in `report["score"]["gates"]` fails, after the JSON report is written, and prints the failed gate names. Tests: `test_an_unrecordable_served_model_is_digested_instead_of_echoed` (an echoed argument, a space, a lone surrogate and an oversized value, plus the cached replay) and `test_main_exit_code_carries_the_gate_verdict` (both directions). Both fail against the pre-fix source. docs/GUARDRAILS.md audit boundary updated to match. 113 passed: tests/test_typesafe_guardrail_provider.py, tests/test_typesafe_risk_eval_script.py, tests/blocking_io/test_typesafe_guardrail_provider.py. * fix(evals): fail the risk-gate run when a labeled population is uncovered Review finding on dcd38cc3. The new exit code made `_score`'s vacuous passes load-bearing: unprobed calls are excluded from the score, so `risky_misses_zero` is true over an empty risky population. `--tools read_file` excludes all 18 risky cases in the bundled set, so the run printed every gate as PASS and exited 0 after evaluating no risky operation at all. `_score` now gates coverage itself: `risky_coverage_present` and `safe_coverage_present` fail when the probe scope leaves that labeled population with nothing evaluated, so a scope that cannot substantiate the verdict no longer reports one. The safe side had the same vacuity -- `system_false_block_within_target` passes trivially with zero safe cases. Verified end to end against a loopback mock endpoint with the bundled case set: `--tools read_file` exits 1 with `GATE risky_coverage_present: FAIL` while the other four gates still PASS, where dcd38cc3 exits 0 on the same command. Tests: `test_main_fails_when_the_tool_scope_leaves_no_risky_case` and `test_main_fails_when_no_safe_case_is_in_scope` (both fail against dcd38cc3), `test_bundled_case_set_keeps_both_labels_the_coverage_gates_require`, and the exit-code test now runs with both populations present. 116 passed: tests/test_typesafe_guardrail_provider.py, tests/test_typesafe_risk_eval_script.py, tests/blocking_io/test_typesafe_guardrail_provider.py.	2026-09-24 18:20:31 +08:00
lihongyuan99	54c65d8ef6	fix(sandbox): resolve the async acquire lock path off the event loop (#5760) ... * fix(sandbox): resolve the async acquire lock path off the event loop _discover_or_create_with_lock_async offloads every step of the acquire — ensure_thread_dirs, the lock file, cache re-checks, backend discovery and registration — except the one line that builds the lock path itself, which resolves through Paths.base_dir and therefore runs a syscall on the loop. The sibling anchor documented that as "a pre-existing blocking call in this coroutine" and stubbed the path layer to get past it. With the resolution in a worker thread the stub is no longer needed, so this removes it and pins the behaviour with an anchor of its own. * test(sandbox): word the offload anchor against the state after the stub removal * test(sandbox): pin that the gate still catches an inline thread_dir call The lock-path anchor only asserts execution reached the lock file, so it would stay green if the gate stopped instrumenting the syscalls behind Path.resolve() and the inline call were restored. Add a meta-check next to the store probe's: call thread_dir() directly on the event loop under the same isolation and assert BlockingError.	2026-09-24 15:19:50 +08:00
lihongyuan99	c10068fd27	fix(channels): resolve the Buzz seen-event store path off the loop (#5759) ... _start_channel built the default seen-event store path inline, and Paths.base_dir resolves through realpath. Channel startup reaches that wiring from POST /api/channels/{name}/restart, so every restart ran a filesystem syscall on the Gateway event loop; the rest of the same function already keeps its disk IO in a worker thread. The existing wiring test stubs get_paths with a plain-string base_dir, so it cannot see the syscall. The new anchor keeps the real Paths object and isolates it with DEER_FLOW_HOME instead.	2026-09-23 23:24:51 +08:00
xiaodu55	736b5a4216	fix(gateway): keep custom-skill rollback filesystem IO off the event loop (#5751) ... * fix(gateway): keep custom-skill rollback filesystem IO off the event loop Closes #5747. Three blocking sites remained on the Gateway event loop in the rollback route and its shared response reader, all offloaded through asyncio.to_thread following the #5729 pattern: - storage.validate_skill_markdown_content() creates a temp dir and writes SKILL.md into it (site 1) - the post-scan read of the file being replaced (site 2, named by #5729's test docstring) - _read_custom_skill_response(): load_skills() walks every skill directory and read_custom_skill() opens SKILL.md — O(installed skills) loop time paid per accepted rollback and per GET /skills/custom/{name} (site 3) A blocking_io regression drives the full accepted-rollback path (past the security scan) under the strict gate; the two pre-existing cases only covered the pre-scan branches. * fix(gateway): keep the skill element type in the response-parts closure Review nit on #5751: tuple[object, ...] erased the element type for the _skill_to_response call; Skill is already imported in this module.	2026-09-23 15:38:40 +08:00
lihongyuan99	fff7aa6a18	fix(skills): rollback reads its history file off the event loop (#5729) ... * fix(skills): rollback reads its history off the event loop rollback_custom_skill constructed the user-scoped storage, probed whether the skill exists and parsed custom/.history/<name>.jsonl inline, while the sibling get_custom_skill_history two lines above already offloads exactly those three steps and documents them as "blocking filesystem IO that must stay off the event loop". The history file grows one entry per edit carrying full previous and new content, so a rollback request parsed the whole edit history on the Gateway loop; the strict Blockbuster gate catches the construction as a blocking os.getcwd() through SkillsConfig.get_skills_path() -> project_root(). Offload the three steps through the same worker-thread closure and anchor both pre-scan branches under tests/blocking_io/. * test(skills): address rollback anchor review feedback --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-22 21:50:59 +08:00
spud	8f08df9b48	fix(sandbox): fence ambiguous AIO session creation ownership (#5711) ... * fix(sandbox): quarantine ambiguous shell session creation * fix(sandbox): quarantine ambiguous bash session creation * fix(sandbox): recycle sandboxes with ambiguous session creates * fix(sandbox): keep quarantined release teardown non-raising * docs(sandbox): keep guidance within instruction budget --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-22 20:48:22 +08:00
lihongyuan99	53352287a7	fix(skills): install for a user keeps its custom-dir setup off the event loop (#5650) ... * fix(skills): install for a user keeps its custom-dir setup off the event loop UserScopedSkillStorage.ainstall_skill_from_archive re-implemented the install pipeline and created the per-user custom directory inline, so the await behind POST /api/skills/install blocked the Gateway loop on os.mkdir. The base class offloads every filesystem phase around its LLM scan — and says so — but the override's setup step sat above that comment and escaped it, and the existing blocking-IO anchor drove only the host-scoped class. Offload the mkdir through the same worker thread and anchor the override itself under the strict Blockbuster gate. * test(blocking-io): pin per-user staging of supporting skill files Review follow-up on #5650: the user-scoped anchor asserted only the staged SKILL.md, so a regression that mis-stages the references/ tree under the per-user root would pass both anchors. Assert the supporting file against the per-user layout as well, mirroring the host-scoped twin.	2026-09-22 18:39:19 +08:00
YxinMiracle	b8ab097ca2	fix(models): support official DeepSeek managed profiles (#5718) ... Co-authored-by: YxinMiracle <“939157765@qq.com”>	2026-09-22 17:53:21 +08:00
Xuehao Xu	8ef58eaa90	feat(models): manage shared models from Settings (#5596) ... * feat(models): add admin UI for shared model management * docs(gateway): keep model guidance within size budget	2026-09-20 16:13:58 +08:00
Wenchao An	42334f26d7	feat(capabilities): unify catalog, plugin configuration and agent selection (#5497) ... * feat(capabilities): unify catalog, plugin configuration and agent selection * fix(capabilities): address review isolation, validation and demo issues * fix(capabilities): preserve concurrent selections and guide launcher repair	2026-09-19 12:14:24 +08:00
wd_pan	cc27730348	feat(memory): add opt-in relevance-aware retrieval ranking (#5251) ... * feat(memory): add opt-in relevance-aware retrieval ranking Add a deterministic, network-free lexical relevance strategy for DeerMem (issue #4495): memory_search ranks every fact in scope by idf-weighted token overlap combined with confidence, with optional greedy-MMR diversity against near-duplicate facts; prompt injection ranks facts against the current-turn query threaded from DynamicContextMiddleware through the new optional `query` keyword on MemoryManager.get_context/aget_context. Defaults preserve the legacy confidence-only behavior exactly; no prompt, storage-format, or vector/embedding-dependency changes. Refs #4495 Signed-off-by: pwd11 <fvdsrc@163.com> * fix(memory): bound relevance retrieval and apply review feedback Bound tokenization and index shared stems, preserve mixed CJK tokens, warm jieba, and align missing confidence with legacy injection. Cache MMR token sets and stop selection at result or injection budgets. Document retrieval-adapter precedence and add regression coverage. Refs #4495. Signed-off-by: pwd11 <fvdsrc@163.com> * fix(memory): preserve backend compatibility and normalize relevance Signed-off-by: pwd11 <fvdsrc@163.com> * fix(memory): omit absent query hints and share injection IDF Signed-off-by: pwd11 <fvdsrc@163.com> * test(memory): retain timeout mock until injection worker exits Signed-off-by: pwd11 <fvdsrc@163.com> * docs(agents): drop root guidance compaction Signed-off-by: pwd11 <fvdsrc@163.com> * fix(memory): validate token prefixes and preserve upload queries --------- Signed-off-by: pwd11 <fvdsrc@163.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-18 11:32:04 +08:00
Onefly	d8d110c637	fix(sandbox): prevent AIO subagent session eviction (#5178) ... * fix(sandbox): prevent AIO subagent session eviction * fix(sandbox): address PR 5178 review issues * fix(sandbox): handle transient session and metadata failures * fix(sandbox): fence capacity upgrades and validate reused limits * docs(sandbox): restore list indentation and trim guidance * fix(ci): stabilize Buzz persistence test and trim sandbox guidance --------- Co-authored-by: ranxi2001 <ranxi2001@users.noreply.github.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-17 07:45:19 +08:00
Zeren Wang	a58ab484a6	feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash (#5443) ... * feat(projects): Projects MVP Phase 2 — instructions, document shelf, promotion, trash Implements docs/superpowers/specs/2026-09-12-projects-mvp-phase2-design.md (issue #5160, tracker #5129) in the slice order of the spec's §16. Slices: - A: ProjectsConfig + write-time 422 UTF-8 byte cap; PROJECT_CONTEXT_KEY admission pinning (both server-owned sets + worker hoist); latest-only request-scoped <project> block via DynamicContextMiddleware wrap_model_call/awrap_model_call (idempotent reassembly, reserved ID prefix + marker + provenance, never persisted); journal audit fingerprints; Instructions tab. - B: ProjectDocumentRow + migration 0023; ProjectDocumentRepository with locked check-and-set; hash-qualified immutable shelf storage with Paths helpers; upload/list/content/delete-to-trash routes; project delete trashes the shelf in-transaction; request-scoped bounded <documents> index with honest count/shown + actionable overflow note; list_project_documents/read_project_document tools registered only on pinned runs; PAT allowlist + drift guards; blocking-IO anchors. - C: shared thread-upload ingestion service (uploads router refactored to parity); POST from-thread with provenance; attach-to-thread with lock-staged copy (archived source allowed); read-only thread-files view with per-group truncation reporting. - D: restore (restored/merged/not_found/no_target/content_missing; no file moves), purge (continuous row lock across unlink/delete/commit, retryable on FS errors), retention sweep (lazy + startup, 24h orphan guard, row-side reconciliation never deletes). - E: Documents tab (shelf + conversation-files browser, provenance, archived banner, content-missing rows), /workspace/trash route, sidebar entry, composer attach handoff, i18n (en-US/zh-CN), e2e mocks + specs. Review hardening folded in (10 rounds, all with tests): - force active shelf content (HTML/XML family) to download; nosniff on artifact + content responses; unified unsandboxed-iframe PDF preview (fixes the pre-existing Chromium sandbox blank in the artifact viewer) - scope document trash to the URL project under the document lock - atomic no-overwrite filename reservation for ALL ingestion (seeded claims + os.link commit with suffix retry; same-name re-upload now unique-names instead of replacing); hidden staging only, no visible placeholders; lease cleanup on setup failure - serialize conversion under the document lock with post-lock active revalidation; drain locked filesystem work on cancellation; preserve bytes when an insert's commit state is uncertain (including trashed rows) - original-integrity checks before serving text or cached conversions; content_missing surfaced in list responses (UI reads the flag, no 409-probe); downloads always serve original bytes - bounded streaming document reads with cached char counts; shelf limits declared in middleware release identity - thread-root confinement for from-thread sources; config fallback rejects fractional/infinite values; composer counts staged attachments; pending attachments persist until submission or removal; in-flight instruction/rename edits survive save refetches; shelf and trash pagination; conversation-file and thread-files pages stay subscribed to refetches Docs: README/README_zh, backend API.md/ARCHITECTURE.md, AGENTS.md contracts, config.example.yaml projects block. Review follow-ups (head b4807477 → this revision): - The trash retention sweep is split so repeated lazy triggers stay bounded: the indexed expiry purge still runs on every trigger (GET /api/trash/documents, POST /api/trash/purge) while the O(all rows + all files) reconciliation is throttled to one run per user per 15 minutes (process-local, per-user window). The startup sweep now runs as a background task instead of blocking gateway readiness, and shutdown awaits it (bounded). - The export scrub (stripInternalMarkers) is fence- and indentation-aware like the render path, so a pasted, fenced <project>/<documents> snippet survives markdown export while real injected blocks (never fenced) are still removed. Fence regexes moved to a dependency-free leaf module to avoid the messages↔streamdown import cycle. - The artifact viewer's PDF iframe no longer carries an added title attribute (the upstream e2e contract locates it via :not([title])), and the upstream artifact-preview spec now pins the new contract: PDFs render unsandboxed, images keep sandbox="". * fix(projects): round-2 review — cancel an overrun trash sweep, restore the PDF frame title - Shutdown cancelled only the shield around the background startup sweep, so an all-users reconciliation that outlived the 5s budget kept walking rows and files while the document repo and DB engine were disposed underneath it. The wait now lives in `_shutdown_startup_trash_sweep`, which cancels the task and drains it before worker exit: the shield keeps the wait bounded, the cancel makes it final (CancelledError lands at the sweep's next await, and `_run_startup_trash_sweep` only catches `Exception`, so nothing swallows it). - The browser-preview iframe lost `title={getFileName(filepath)}` in the previous fix round, leaving the PDF frame without an accessible name while its siblings keep theirs. Restore it (WCAG frame titles), assert it in the DOM test, and anchor the e2e on `iframe[title="report.pdf"]` instead of `iframe:not([title])`. * fix(projects): round-3 review — report the sweep's late finish, not a phantom cancel `Task.cancel()` returns False when the sweep already finished inside the window between the deadline firing and the cancel, so the shutdown log claimed a cancellation that never happened. Branch on that outcome: the warning stays for a real cancel, a late finish is logged at info, and both paths still reap the task before worker exit. * fix(projects): round-4 review — make Empty trash delete what it confirms `POST /api/trash/purge` only ran the retention sweep, and the sweep's candidate selection is age-gated, so a freshly trashed document survived "Empty trash" even though the confirmation promises that every listed document is permanently deleted. With one trashed row the route answered `{"purged": 0}` and left it in place; `GET /api/trash/documents` sweeps expired rows before listing, so the visible rows were normally ineligible for the action by construction. Empty trash now drives `purge_all_trashed`: the caller's trashed rows (`list_all_trashed`, no age filter) each go through the same guarded, row-locked `purge` as the single-document delete — bytes first, then the row, in one transaction — so a row restored mid-flight is skipped instead of force-deleted, and an unlink failure rolls that row back and answers 500 with a retryable message. Retention expiry stays where it was: the sweep's `purge_candidates` is now the only age-gated selection, and the lazy retention sweep still runs on the listing and at startup. Tests: the router suite replaces the retention-gated expectation with the reviewer's repro (fresh row purged, bytes unlinked, shelf and other users' trash untouched, a failing unlink stays retryable and 500); a blocking-I/O anchor drives the new entry point through the offload; the mocked e2e covers the action end to end; a new real-backend spec performs it against the real gateway and re-reads `GET /api/trash/documents`. README, API, ARCHITECTURE and the phase-2 design docs (en+zh) state the age-independent contract.	2026-09-16 18:46:18 +08:00
hataa	26800d1245	fix(channels): stream-cap and validate WeChat/WeCom inbound media downloads, fixes #5223 (#5225) ... * fix(channels): stream-cap and validate WeChat/WeCom inbound media downloads, fixes #5223 * fix(channels): address WeCom APPID, decompression, and log-sanitization review findings (#5223) Round-4 review follow-ups on the inbound-media download cap: - The COS bucket numeric suffix is the owner's Tencent Cloud APPID and bucket names are user-chosen, so any Tencent Cloud account could register a matching ww-aibot-img-* bucket and pass the shape gate. The built-in rule now admits only the APPID observed in Tencent's published aibot callback examples (1258476243), across regions; any other account (including a future WeCom rotation) goes through channels.wecom.allowed_media_hosts. - aiter_bytes() transparently decodes Content-Encoding, and the decoder allocates the full decompressed body before the byte cap sees a chunk (an ~8 KB gzip wire chunk decoding to 8 MiB reproduces it). Both URL readers now send Accept-Encoding: identity, refuse a response with a residual Content-Encoding before reading, and iterate aiter_raw(). - httpx.HTTPStatusError formats the signed URL (path + query credentials) into its message, so _ingest_inbound_files' reader-failure branch logs a sanitized summary (class + status) instead of logger.exception, and the WeChat extract paths catch httpx.HTTPError so the polling loop's per-message logger.exception can never render a media URL. Every change ships with a red/green regression: the reviewer's 403 mock-transport repro asserted against caplog.text (fully formatted logs), the reviewer's different-APPID bucket host, and gzip bombs driven through real httpx mock transports in both readers. Docs (channels AGENTS.md, README, config.example.yaml) updated for the APPID pinning and encoding gate. * docs(channels): document why the inbound-media cap is 50 MB, not WeCom's 100 MB ceiling * fix(logging): redact URLs in httpx request logs down to scheme + host, fixes #5223 httpx emits 'HTTP Request: GET <full URL>' at the Gateway's INFO level before any response handling runs, so even successful signed-media downloads leaked their credentials. HttpxUrlQueryRedactionFilter (installed by configure_logging) rewrites those records in place — path and query become /<redacted>, method/status/duration observability is preserved — which also keeps Telegram's token-bearing Bot API paths out of the logs. Reader-level regression tests run at production INFO level with a real MockTransport, success paths included. * fix(logging): blank userinfo credentials in httpx request-log redaction * fix(logging): redact authority-only URLs and cover urllib3 redirect logs Two follow-ups from the review plus one extrapolation of the same class: - rest is now optional in _URL_REDACT_RE, so an authority-only URL (scheme://user:pass@host, no path) is rewritten too — userinfo had nowhere else to hide and previously passed through verbatim. A bare credential-free origin still passes through unchanged. - Renamed to UrlRedactionFilter / install_url_log_redaction and attached to the urllib3 logger as well: urllib3 logs 'Redirecting <url> -> <url>' at INFO with full URLs on both sides, the same leak class on a different library logger. No gateway path today both uses requests and redirects a signed URL, but the class stays closed instead of dormant. - Unit tests now build records with the real httpx 0.28.1 format string ('HTTP Request: %s %s "%s %d %s"', 5 args) and httpx.URL args, per the nit, instead of a synthetic shape httpx never emits. * fix(logging): install URL redaction at handler level so propagated records are covered A logging.Filter on a logger only runs for records emitted through that exact logger — child loggers neither inherit it nor trigger it on propagation — so the previous attachment to the bare urllib3 logger was dead code: urllib3 emits Redirecting via urllib3.poolmanager at INFO and urllib3.connectionpool at DEBUG. The filter is now attached to every root handler (mirroring _install_trace_filter, which already iterates root handlers; handler-level filters see propagated records) in addition to the httpx logger (httpx emits via the bare name, and emission-point coverage survives handlers added later). The wiring is pinned by tests that emit through the real urllib3 child loggers — a mutation removing the handler-level install turns them red. Comments, docstrings, and AGENTS.md now state the actual emitter names and levels. * fix(logging): redact urllib3 DEBUG request lines, whose split shape evaded the URL regex urllib3's per-request line (connectionpool.py:545 on 2.7.0) renders as `scheme://host:port "METHOD /path?query HTTP/x.x" status len` — the authority ends at a space so _URL_REDACT_RE's bare-origin early return applies, and the quoted origin-form target has no scheme, so neither half was rewritten. UrlRedactionFilter now runs a dedicated request-line shape first (collapsing the target to /<redacted>, keeping scheme+host+method+ version), then the absolute-URL pass. Regressions pin the exact format string both at unit level and through the real urllib3.connectionpool DEBUG emit path; AGENTS.md wording now names both covered DEBUG shapes. * fix(logging): redact urllib3 retry lines and linearize scheme scanning Closes the two open review threads on the inbound-media log hardening: Retry/redirect targets: urllib3 logs the request target with no scheme in five shapes the generic absolute-URL pass cannot see - `Retry: <target>` (connectionpool.py:954 DEBUG), `Incremented Retry for (url='<target>')` (util/retry.py:545 DEBUG, absolute on the redirect path), `Retrying (...) after connection broken by '<err>': <target>` (connectionpool.py:869 WARNING, above the INFO root), and origin-form halves of both Redirecting emitters (poolmanager.py:500 INFO / connectionpool.py:922 DEBUG). Each gets a rewrite anchored to the exact urllib3 format, collapsing the target to /<redacted>; the generic pass's rest now stops at quote characters so a quoted URL keeps its closing punctuation (previously the absolute-form increment line was mangled), and the request-line method class accepts any case. The emitter enumeration in channels AGENTS.md is closed against the installed urllib3 2.7.0 source. Quadratic scanning: both scheme-bearing patterns start with a character class, so re.sub retried every suffix of a long token - 64K paths cost ~1.8s and URL-free 64K error bodies ~3.1s per record, synchronously in every root handler. The two passes are now driven from literal "://" occurrences: _scheme_starts walks back over the scheme charset to each run's first letter and the pattern is attempted only there, reproducing re.sub's leftmost-non-overlapping result in linear time (256K path: 5.6ms; worst adversarial shapes <= 28ms). Long-input regressions pin the URL-bearing and URL-free cases with mutation-verified bounds, plus nested-scheme and digit-headed-run equivalence cases. Validation: tests/test_logging_config.py 12/12; scheme-pass equivalence against the old re.sub pipeline verified by two independent 30k+ case fuzz runs; full-suite A/B against HEAD shows zero tests that pass on HEAD and fail with this diff. * fix(logging): boundary-aware quote stops and whole-message Redirecting anchor Two follow-ups on the urllib3 redaction shapes: Embedded quotes: `rest` treated ANY quote as a closing mark, so a URL with an apostrophe in the path kept everything after it verbatim (`https://h/path'quoted'?token=Q` rendered the credential suffix in full) while the class docstring claimed path/query/fragment are replaced. A quote now closes `rest` only at a boundary - followed by whitespace, a closing parenthesis, or end of string - so urllib3's Incremented Retry (url='...') scaffolding keeps its ') closer while an embedded quote stays consumed. The increment line's url capture gets the same rule narrowed to its fixed ')' closer. Redirecting anchoring: the origin-half pass matched `(? <=-> )/path` as a substring, and an `-> /path` arrow is not urllib3-owned shape - the sandbox provider's actionable mount error (`sandbox.mounts entry <host> -> /mnt/knowledge ignored: ...`) had its container path rewritten to /<redacted>, failing test_setup_path_mappings_logs_actionable_error_for_missing_host_path on CI (backend-unit-tests shard 3). The pass is now anchored to the whole `Redirecting <t> -> <t>` message, which is exactly urllib3's record; origin slots collapse, absolute slots stay for the generic pass. Regression tests pin the embedded-quote shapes and the sandbox error's byte-for-byte passthrough; both mutations verified red. Validation: tests/test_logging_config.py 14/14; the CI-failing sandbox test green locally; every test file asserting redaction/arrow log content passes (attachments, support bundle, run metadata, skill secrets, ragflow, skillscan, sandbox provider); full offline backend suite 14084 passed / 164 failed with the failure set matching this machine's documented Windows-environment baseline (NTFS chmod/symlink, docker/lark/langfuse absences) - no failure involves redaction output. * fix(logging): redact redirects with spaced locations * fix(logging): grammar-complete Redirecting anchor; neutral WeChat guard labels Round-13 P3 (Redirecting anchor strictness): the whole-message anchor kept the ^Redirecting prefix (the urllib3-owned literal that stops the sandbox false positive) but required BOTH slots whitespace-free, so a Location header with an interior space voided the pass and leaked the origin-form request target in the first slot - redirect_location is the raw header string and interior spaces are legal field syntax. The tail is now loose (\S.*$) and the first slot gets the same grammar treatment (\S.*?): the recursive urlopen frame passes the previous raw Location as its url, so t1 can carry interior spaces too, lazy-split at the first arrow the way the line is constructed. A space-carrying slot collapses whole when it starts with /; the sandbox mount error keeps passing through untouched. Round-14 nit (None conflation): _download_cdn_bytes returns None for two reasons (in-flight cap abort, Content-Encoding refusal) but both image and file callers labeled it "exceeds size limit (N bytes)" - contradicting the accurate encoding line right above it, and reporting the plaintext limit for a ciphertext-cap decision. Callers now log a neutral "skipped by download guard" line (the manager reader callers' shape); the accurate reason stays inside the download function. The same sweep also logs _stage_downloaded_file's silent None (no state dir configured), which made an attachment vanish with no log line at all. Also anchors the emitter-enumeration closure to its urllib3 version: the closure reopens if an upgrade changes these format strings, so the comment now says so explicitly. Validation: logging 15/15 and attachments 60/62 (the two pre-existing Windows symlink-privilege failures documented in the PR body); three mutations verified red (old wording, strict t1, silent staging None); ruff clean. Full offline suite run before push (per round-11 lesson). --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-16 16:33:47 +08:00
Zhengcy05	81f2015fe6	fix(runtime): keep agent construction off event loop (#5217) ... * fix(runtime): keep agent construction off event loop * fix: - offload checkpoint state accessor graph construction to a worker thread - update test * import AsyncKeyedLockTable * update Agents.md * fix: update test * fix: preserve single-flight builds after cancellation --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-12 21:09:05 +08:00
Shxiao	a06688ad82	test(skills): skip the skill-export capture suite where fd-based walking is unavailable (#5372) ... * test(skills): skip the skill-export capture suite where fd-based walking is unavailable export._capture intentionally rejects every export with 422 skill_export_unsupported on platforms without os.O_NOFOLLOW and dir_fd directory walking (Windows), so the 40 capture-dependent tests in test_skill_export.py and test_gateway_skill_export.py fail there. Add a requires_safe_capture mark mirroring the production feature guard and apply it to those tests; the 21 tests that only exercise the zip installer, export lock, or router plumbing keep running. * test(skills): share requires_safe_capture and cover the blocking-io export suite Move the mirror of export._capture's platform guard into tests/support/skill_export_platform.py so the per-file copies cannot drift, and apply it to blocking_io/test_skill_export.py::test_export_worker_and_response_are_off_loop, which drives the real exporter and hits the same 422 skill_export_unsupported guard on Windows (the blocking-io suite is excluded from the default make test run). --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-12 14:32:05 +08:00
Hyeonsang Cho	bc4a33aba7	fix(skills): stop persisting resolved secrets when toggling skills (#5357) ... Toggling a skill wrote resolved secrets into extensions_config.json. The Gateway skill toggle and DeerFlowClient.update_skill loaded the file with ExtensionsConfig.from_file(), which replaces every "$VAR" string with the environment value (and an unset variable with ""), then serialized that model back through to_file_dict(). A "$GITHUB_TOKEN" reference was persisted as the plaintext token, and an unset reference was erased for good. DeerFlowClient.update_mcp_config had the same flaw for every key other than mcpServers. Every writer now does a raw read-modify-write, the way the MCP router already did: read_raw_extensions_config reads the on-disk JSON, set_raw_skill_enabled changes only the target entry, and validate_raw_extensions_config checks the candidate the way the runtime will load it before the atomic write. When the file does not exist yet, the Gateway seeds it with the cached skill states only, never the resolved cached model. The MCP router's raw loader and candidate validation delegate to the same helpers, so the rule lives in one place, and to_file_dict() is removed so the unsafe serialization has no entry point left. Co-authored-by: Willem Jiang <willem.jiang@gmail.com>	2026-09-12 11:24:46 +08:00