6f24bc7c36
Thread title generation runs inside the agent graph (aafter_model), and it built its configured model inline. create_chat_model can read local CLI credentials while it is constructed (Codex auth.json, Claude Code credential files), so the first turn of every new thread read them on the shared loop and other requests waited for it. Build the provider through run_assembly, the dedicated assembly pool the one-shot helpers, skill moderation, the goal evaluator and start_run already use. Inference still awaits on the loop, and the sync path is unchanged. A Blockbuster regression builds the real Codex provider: on main the gate rejects the credential read, the broad fallback swallows it, and the local fallback title replaces the model's title; with the change the provider is constructed on a worker thread and the model's title is kept.
14 KiB
Readability Fallback
readability.py caches the optional Readability.js availability probe. If the
Node extractor is unavailable, its probe raises, or an attempted extraction
cannot launch Node, subsequent calls in that process use readabilipy's Python
fallback without retrying Node. A non-zero Node exit can be document-specific,
so it falls back only for that document and later documents retry Node. If a
future readabilipy version removes the private availability probe, log that
once at import and attempt extraction instead of silently disabling Node. Keep
successful Node extraction unchanged and allow unexpected parser/programming
exceptions to propagate. The resolved-body fallback preserves article links
when Python extraction returns the original document wrapper. Regression
coverage lives in tests/test_readability.py and
tests/test_web_fetch_relative_links.py.
Atomic Byte Publication
atomic_write.publish_bytes_atomically owns browser screenshot and Browserless
capture/fetch publication. It writes and fsyncs a hidden exclusive sibling before
os.replace, cleaning it on failure. New files use 0666 filtered by the host
umask; replacements preserve existing permission bits before publication. Never
read the process umask by changing it in this threaded runtime. Behavioral tests
for both tool imports live in tests/test_browser_outputs_atomic.py.
cleanup_failed_atomic_write shares best-effort cleanup across this publisher,
skills/projection.py's private manifest writer, project document staging,
extension snapshot rollback (including removal of a file that did not exist),
the managed plugins-block rewrite, custom-skill file writes, per-user
skill-state writes, Gateway upload staging, and the Gateway skill-archive
upload copy and its post-install discard. The upload route discards that
archive before closing the multipart form and suppresses close errors.
Use it on publication failure or after deciding to discard an owned
temporary; pass a raw descriptor only until fdopen succeeds, or an explicitly
transferred duplicate descriptor owned by the cleanup caller. Shelf discards
must preserve both a rejected project's 404 and a dedup hit's 200 response.
Callers retain their own serialization, permissions and publication policy.
Shelf stage_document_bytes drains preparation, writes, close and failure
cleanup before propagating cancellation. Retain the opened handle across its
handoff, but leave waiting for incoming chunks cancellable; regressions in
tests/test_project_document_staging_cancellation.py cover both shelf entry points.
Shelf conversion also drains its final staging discard before releasing
single-flight ownership. Cleanup failures warn without replacing the conversion
result, original error or cancellation; the retention sweep owns later retries.
Coverage lives in tests/test_project_document_tools.py::TestConversionCleanup.
Local Linux/macOS text revisions opt into a short temporary_prefix and
preserve_metadata_from (a caller-owned descriptor). Staging starts at 0600;
file_revision_metadata transfers ownership, mode, ACLs and xattrs before data.
macOS uses native descriptor ACL APIs and fcopyfile(COPYFILE_XATTR); Linux uses
descriptor xattrs, including its access ACL, removing inherited extras. Errors
abort; never ignore metadata permission failures. Refuse set-id bits, macOS file
flags and content-bound privilege attributes rather than reattach stale privileges to new
content. Other publisher callers retain their existing mode-only contract.
Do not restore old timestamps on a content revision, including an empty save.
Source-descriptor close failures are best-effort warnings, not save failures;
guard diagnostics so the publication result or original interruption survives.
Failure cleanup must preserve the original publication exception, including
interruptions. Descriptor close and sibling unlink are separate best-effort
operations: a close failure must not prevent the unlink attempt.
Attempt warning logs for cleanup OSErrors with the temporary path and exception
details. Guard logging-handler exceptions so diagnostics neither replace the
publication exception (including interruptions) nor skip remaining cleanup.
Both tools take collision names from output_names.output_name_candidates
(name, -1..-1000, five timestamps); claim each atomically, never pre-check.
Model Retry Hints
retry_after.py::bounded_retry_after_ms shares the 24-hour provider-hint ceiling
between LLM middleware (numeric/date hints) and Claude backoff. Non-finite or
over-limit delays return None for local backoff; finite negatives clamp to zero.
Validate before integer truncation and preserve valid hints above the jitter cap.
Port Allocation Bounds
network.py::PortAllocator.allocate caps its exclusive search endpoint at
65536. A valid start near 65535 must report exhaustion with RuntimeError
when all remaining ports are reserved or occupied, without probing invalid
TCP ports. Keep the last valid port allocatable and preserve max_range.
Regression coverage lives in tests/test_port_allocator_bounds.py.
Goal Objective Validation
goal_objective.py owns the dependency-free normalized 4000-character objective
validation shared by runtime and scheduled-task capability/repository admission.
runtime.goal re-exports the helper for compatibility. Validation does not replace
the saved original text; blank or over-limit text is rejected before persistence.
Message Text Extraction
message_content_to_text takes raw message content and treats None as empty
text so downstream empty-result, error-detail, and archive-skip fallbacks work.
Keep its other conversions unchanged, including newline-separated list blocks
and literal strings such as "None". Do not replace it with message_to_text,
which takes a whole message and uses different list/mapping semantics.
Regression coverage lives in tests/test_utils_messages.py,
tests/test_subagent_executor.py, and tests/test_task_continuity.py.
llm_text.py::extract_response_text ignores content blocks with non-string
type values, including lists and mappings, while retaining adjacent text and
output-text blocks. Tests cover extraction and the one-shot input-polish path
with real LangChain messages in tests/test_utils_llm_text.py and
tests/test_input_polish_router.py.
llm_text.py::strip_think_blocks scans opening and closing tags forward so
repeated unclosed <think> prefixes in model output do not rescan the same
suffix. Keep its case-insensitive tag handling, optional whitespace before
the closing >, and the truncate_unclosed behavior. Regression coverage
lives in tests/test_utils_llm_text.py.
Opening reasoning tags require whitespace, >, or /> after the exact think
name; punctuated names such as <think-tank>, <think:note>, and <think/other>
are ordinary content. The shared prefix regex owns this boundary rule; do not
duplicate it in a manual re-check. Self-closing <think/> and <think /> tags
are empty reasoning blocks: remove the tag without consuming subsequent text.
Both cleanup functions find opening delimiters outside quoted attributes; a
quoted > cannot terminate the tag or hide an answer after a self-closing tag.
Keep exact unfinished <think prefixes hidden in leading summaries. Coverage:
tests/test_think_tag_boundaries.py.
Display summaries use strip_leading_think_blocks before limiting text;
test_run_journal_visible_summary.py preserves literal tags in the answer.
Agent / Tool Assembly Off-Load
One-shot input-polish, follow-up-suggestion, and project-file-summary helpers,
skill moderation (scan_skill_content), the goal evaluator, and thread title
generation build providers through run_assembly before awaiting inference;
the run worker calls its cached evaluator factory the same way. Provider
construction can read local credentials; keep it off the event loop and the
default executor while preserving request context. Cancellation during
construction must not start inference. A scan without app_config (in-graph
skill_manage) reads the live config in the same pool, and its fail-closed
fallback reuses that config rather than reading it again on the loop. Other
model construction is separate and not covered here: manual compaction
(compact_thread_context builds the summarization middleware and its models
inline) and, inside graph runs, summarization and subagent construction.
Coverage: tests/blocking_io/test_oneshot_llm_offloop.py,
tests/blocking_io/test_system_model_offloop.py and
tests/blocking_io/test_title_model_offloop.py.
start_run also resolves the factory through run_assembly before admission:
cold imports cannot occupy the loop or its default pool, and import errors
cannot admit a run. tests/blocking_io/test_agent_factory_resolve_offloop.py
pins offload, executor isolation, request context, and failure ordering.
Tool and agent assembly re-enters get_available_tools() and may block on MCP discovery, so the four async assembly entry points — run_agent's agent_factory call, task_tool, durable batch _execute_item, and abuild_checkpoint_state_accessor — dispatch through deerflow.utils.assembly_io.run_assembly, a dedicated ContextVar-preserving bounded executor (DEER_FLOW_ASSEMBLY_WORKERS, default 8) rather than the loop's default executor; a hung MCP server therefore parks an assembly worker instead of queueing unrelated default-executor work, and the pool logs a warning when pending assemblies exceed the worker count. tests/blocking_io/test_tool_assembly_offloop.py pins all four offloads plus the ContextVar propagation.
Document Conversion
file_conversion.convert_file_to_markdown resolves the live PDF converter in
its drained parsing worker. Config cache hits still stat/hash files; keep them
off-loop with parsing. Cold/cache-hit coverage uses real temporary configs in
tests/blocking_io/test_file_conversion_offloop.py.
Uploaded Document Summaries
Outlines skip root-level HTML comment blocks (<!-- with up to three leading
spaces through the first --> line). Comment contents cannot open code fences;
comment markers inside fenced code cannot open comment blocks. Keep line numbers
and limits unchanged. Coverage: tests/test_file_outline_html_comments.py.
file_outline.py reads outlines and fallback previews as utf-8-sig so an
optional leading UTF-8 BOM cannot hide a first-line heading or code fence, or
occupy a preview line. Preserve physical line numbers, embedded U+FEFF
characters, and the original file bytes.
All outline heading styles skip root-level indented code (four leading spaces
or a tab after up to three spaces), including PDF-style single and split bold
headings. Keep legitimate bold headings with up to three spaces and the shared
conversion-companion path working. Regression coverage lives in
tests/test_file_outline_indented_bold.py.
Split-bold headings reject numeric/punctuation/currency-only blocks after the
section number, including parentheses, signs, leading whitespace, and $€£¥.
Apply the check to the second through fourth blocks; preserve the four-block
limit, punctuated titles containing text, and non-ASCII titles. Coverage lives
in tests/test_file_outline_split_bold.py.
Active Content MIME Types
text_detection.py::_is_active_content_mime_type is the shared download-safety
boundary for artifacts and project documents. Keep platform aliases such as
Windows' image/svg aligned with their standard active type (image/svg+xml),
and preserve the generic +xml rule.
Conversion Worker Ownership
convert_file_to_markdown offloads parsing for every source size: compressed
Office files can require seconds of work below 1 MiB. It drains conversion workers
across caller cancellation before returning control to source-file cleanup. Keep conversion
inside await_drained, including when the worker fails or cancellation repeats;
upload ingestion owns a temporary source that it removes in finally. Coverage
lives in tests/test_file_conversion_cancellation.py.
Host Path Portability
host_paths.extended_length_path() provides lexical Windows filesystem
spelling (absolute \\?\ drive paths or \\?\UNC\ shares); it is a no-op
on POSIX. It does not replace symlink resolution or confinement checks and
must not be used for persisted relative paths or Docker mount sources.
Existing extended (\\?\) and device (\\.\, including named pipes)
namespaces pass through unchanged before normalization.
host_paths.py rejects Windows device names for host-visible creation paths on
every platform, including the COM/LPT aliases with superscript ¹, ² and ³.
The console aliases CONIN$ and CONOUT$ are reserved too; match them
case-insensitively before the first dot, without rejecting longer ordinary
names such as CONIN$notes.txt.
Device-name comparison ignores ASCII spaces before the first dot, so
NUL .txt and COM1 .log remain reserved. Trim the comparison stem only,
not the supplied filename; ordinary names such as report .txt and
non-ASCII whitespace remain unchanged.
Do not normalize arbitrary Unicode digits into device numbers: names such as
COM⁴.txt and COM¹notes.txt are ordinary portable names. Read/removal callers
retain their existing portability exemptions.
Advisory File Lock Retries
file_lock.retry_until_locked owns contention-only acquisition retries for the
Windows Lark, extension-manager and channel runtime-config locks and bounded
skill-projection reads.
Retry only EACCES/EAGAIN/EDEADLK; other errors propagate without sleeping.
Keep per-caller policies: Lark has a 15-minute Windows file-lock deadline,
extension management and channel runtime config wait indefinitely, and skill
export retains its existing deadline, 0.02-second cadence and cancellation
check. Clocks and sleeps are injectable; tests must not patch the shared stdlib
time module.
The channel caller wraps its injected sleep to warn once after 50 contention
retries; the helper remains responsible for error classification and cadence.
parse_retry_after_date_ms(raw, now=...) owns HTTP-date parsing for Claude, Codex
and LLM error middleware. Interpret timezone-less asctime as UTC and apply the
shared delay bound. Callers retain numeric/header precedence and local backoff.