fix(agents): preserve complete remote artifact URLs (#6574)

* fix(agents): preserve complete remote artifact URLs

* fix(agents): handle CJK punctuation and mixed-case artifact suffixes

* fix(agents): preserve balanced CJK artifact references

---------

Co-authored-by: kuku <186267668+xbzz1018@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>

backend/packages/harness/deerflow/agents/middlewares/TOOL_ARTIFACTS.md

@@ -13,6 +13,16 @@ message history and checkpoints remain unchanged.
 
 *(optional, if `tool_artifacts.enabled`, default on)* Captures lightweight artifact references from tool results into `ThreadState.tool_artifacts` (a reducer channel, so entries survive summarization compaction). `tool_artifacts.enabled` gates the whole middleware (capture and consumption tracking); when off, `before_model` returns nothing regardless of registry state. Runs as a `before_model` hook (never wraps a `ToolMessage` in a `Command`). `ThreadState.tool_artifact_processed` stores deduplicated hashes of processed result and settled consuming-call occurrences through an additive reducer. It records empty results too. The ledger is independent of registry retention and survives checkpoint reload, graph recreation and compaction; it is not trimmed with artifacts and grows by a small identity record per processed occurrence. Never replace it with a process-local memo or infer it from currently retained handles. A missing handle gets one retry before its consuming-call occurrence settles, so permanently unknown or evicted handles are scanned at most twice, including across restarts; same-round consumption resolves against state plus pending captures. Capture and consumption updates concatenate list channels so neither is lost. Position-independent in the tool-execution wrap chain: it is a `before_model` hook reading state messages, so it sees the normalized results stored in state regardless of its wrap position (error results are skipped at extraction via `status == "error"`); it is appended after ToolErrorHandlingMiddleware for readability only. Handles are `art_` + 8 hex chars derived deterministically from (thread_id, durable ToolMessage.id, provider tool_call_id, per-result ordinal). Provider call IDs may repeat across turns; graph-assigned message IDs distinguish occurrences and remain stable after compaction. Standalone ID-less hook callers must supply distinct call indices. These short hashes are identifiers, not ownership credentials. Extraction combines all of: structured-content string values under known keys (`file`/`path`/`url`/`task_id`/...) as concrete refs (unknown or non-reference payloads fall back to complete JSON `data` only when they fit 4096 UTF-8 bytes, 1024 nodes and 32 levels; empty and oversized payloads are skipped before unbounded serialization), `file`/`image` content blocks with HTTP(S), Unix-absolute or drive-absolute references (`data:`/`blob:` URIs are rejected so embedded-resource payloads never enter state or tool args), and conservative free-text scanning of both text blocks and plain-string results such as built-in sandbox tool output (sandbox paths + remote file URLs including query/fragment suffixes, gated by `detect_refs_in_text`). The reducer enforces only an absolute ceiling (1000); the configured `tool_artifacts.max_entries` is a per-agent sliding retention window: when a capture round would exceed it, the middleware appends a trailing `{"op": "trim_to", "keep": N}` directive and the reducer evicts the oldest entries (fresh captures are always registered — nothing freezes at the cap), so agents with different configs in one process do not share a cap. Reference entries contain metadata (path/URL, name, type, MIME). The JSON compatibility fallback is an explicitly bounded exception. Concrete path/URL keys reject prose and relative paths; remote task IDs use a separate non-empty, whitespace-free, 256-character token gate. File references on all extraction paths are bounded to 4096 UTF-8 bytes. Structured traversal is bounded before fallback encoding, and valid content-block references are still captured if structured data is rejected.
 
+Free-text remote file detection checks the complete URL path for a supported
+file suffix. Query strings and fragments are retained in the reference and do
+not determine its file type; a file-like prefix inside a longer path is not
+registered as a shortened URL. Supported file suffixes are matched without
+case sensitivity. Both URL and sandbox-path text scans strip common trailing
+ASCII/CJK prose punctuation and closing quotes or brackets. CJK closers paired
+with openers inside the detected reference are preserved, so a numbered glob
+result such as `/mnt/user-data/workspace/项目（归档）` retains its complete name
+through capture, checkpoint persistence and subsequent handle resolution.
+
 ## Durable projection
 
 Artifact handles are projected with working notes and summary history in the hidden HumanMessage data block when `tool_artifacts.enabled` and `inject_model_context` are enabled. Every model-visible artifact label (display name, type, tool name, MIME type) is redacted with the configured PII policy before HTML escaping. Generated handles remain usable and internal `real_ref` values remain unchanged for resolution. Values remain untrusted data. Artifact capture precedes compaction; task continuity and PII redaction retain independent configuration. `inject_tool_artifacts` participates in the middleware release policy fingerprint.

backend/packages/harness/deerflow/tools/AGENTS.md

@@ -1,5 +1,12 @@
 ### Tool System (`packages/harness/deerflow/tools/`)
 
+`artifact_registry.py` checks remote file suffixes against the complete parsed
+URL path without case sensitivity. Free-text URL and sandbox-path extraction
+strips common trailing ASCII/CJK prose punctuation while preserving CJK
+closers paired with openers inside the detected reference (including literal
+glob directory names). Structured references remain literal. See
+[tool artifact middleware](../agents/middlewares/TOOL_ARTIFACTS.md).
+
 `task` and `batch_task` opt into JSON checks with `file:<path> json-valid`.
 See [subagents/AGENTS.md](../subagents/AGENTS.md) for read limits and UNVERIFIED semantics.
 Batch readers bound by assembly use its AppConfig output budget. Fit the complete escaped response under 10K and active per-tool/fallback limits; too-small envelopes stop reading. Numeric reader inputs are strict integers. Escape `<` in outer JSON to preserve untrusted data without exposing framework tags; decoding restores the document unchanged.

backend/packages/harness/deerflow/tools/artifact_registry.py

@@ -30,8 +30,10 @@ _HANDLE_LENGTH = 8
 # inside the mounted user-data tree).
 _SANDBOX_PATH_PATTERN = re.compile(r"/mnt/user-data/\S+")
 
-# Conservative URL-with-file-extension match for remote references.
-_REMOTE_FILE_URL_PATTERN = re.compile(r"https?://[^\s\"'`<>]+\.(?:png|jpg|jpeg|gif|html|pdf|csv|json|txt|log|md|xlsx?|docx?|zip)(?:[?#][^\s\"'`<>]*)?")
+# Scan complete URLs before checking the path, so extensions in query strings
+# cannot truncate a reference or turn a non-file URL into a file artifact.
+_REMOTE_URL_PATTERN = re.compile(r"https?://[^\s\"'`<>]+")
+_REMOTE_FILE_EXTENSION_PATTERN = re.compile(r"\.(?:png|jpg|jpeg|gif|html|pdf|csv|json|txt|log|md|xlsx?|docx?|zip)\Z", re.IGNORECASE)
 
 # Structured-content keys whose string values are treated as concrete
 # references (paths, URLs, remote task ids) rather than opaque payload.
@@ -43,8 +45,10 @@ _STRUCTURED_TASK_KEYS = frozenset({"task_id", "job_id"})
 
 # Characters stripped from detected refs: prose punctuation plus the closing
 # quotes/brackets/backticks that markdown- and JSON-formatted tool output
-# commonly wraps paths in. `\S+` would otherwise consume them into `real_ref`.
-_REF_TRAILING_NOISE_CHARS = ".,;:)]}\"'`"
+# commonly wraps paths in, including CJK punctuation. `\S+` would otherwise
+# consume them into `real_ref`.
+_REF_TRAILING_NOISE_CHARS = ".,;:)]}\"'`\u3002\uff0c\uff1b\uff1a\u3001\uff09\u3011\u300b\u201d\u2019"
+_CJK_CLOSING_DELIMITERS = {"）": "（", "】": "【", "》": "《", "”": "“", "’": "‘"}
 
 # Content-block and structured-key refs are trusted only in these shapes.
 # `data:`/`blob:` URIs can carry arbitrarily large embedded payloads (MCP
@@ -126,12 +130,30 @@ def _make_entry(
     return entry
 
 
+def _strip_ref_trailing_noise(raw: str) -> str:
+    """Strip prose closers while preserving paired CJK characters inside a ref."""
+    end = len(raw.rstrip(_REF_TRAILING_NOISE_CHARS))
+    if end == len(raw):
+        return raw
+
+    unmatched_openers = dict.fromkeys(_CJK_CLOSING_DELIMITERS.values(), 0)
+    for index, char in enumerate(raw):
+        if char in unmatched_openers:
+            unmatched_openers[char] += 1
+        elif (opener := _CJK_CLOSING_DELIMITERS.get(char)) and unmatched_openers[opener]:
+            unmatched_openers[opener] -= 1
+            # An opener outside the detected token is prose; an opener inside
+            # it makes this closer part of a literal directory name or URL.
+            end = max(end, index + 1)
+    return raw[:end]
+
+
 def _detect_refs_in_text(text: str) -> list[dict[str, str]]:
     """Conservatively detect file paths and remote file URLs in free text."""
     refs: list[dict[str, str]] = []
     seen: set[str] = set()
     for match in _SANDBOX_PATH_PATTERN.finditer(text):
-        raw = match.group(0).rstrip(_REF_TRAILING_NOISE_CHARS)
+        raw = _strip_ref_trailing_noise(match.group(0))
         if raw in seen or not _is_referenceable_url(raw):
             continue
         seen.add(raw)
@@ -142,10 +164,12 @@ def _detect_refs_in_text(text: str) -> list[dict[str, str]]:
                 "display": raw.split("/")[-1],
             }
         )
-    for match in _REMOTE_FILE_URL_PATTERN.finditer(text):
-        raw = match.group(0).rstrip(_REF_TRAILING_NOISE_CHARS)
+    for match in _REMOTE_URL_PATTERN.finditer(text):
+        raw = _strip_ref_trailing_noise(match.group(0))
         if raw in seen or not _is_referenceable_url(raw):
             continue
+        if not _REMOTE_FILE_EXTENSION_PATTERN.search(urlsplit(raw).path):
+            continue
         seen.add(raw)
         refs.append(
             {

backend/tests/test_artifact_integration.py

@@ -8,6 +8,8 @@ rewrites it to the real reference at the tool-call boundary.
 import re
 from typing import Any
 
+import httpx
+import pytest
 from _agent_e2e_helpers import FakeToolCallingModel
 from langchain.agents import create_agent
 from langchain_core.messages import AIMessage, HumanMessage
@@ -122,3 +124,100 @@ def test_handle_projected_into_model_context():
     durable_blocks = [message.content for message in round3 if getattr(message, "additional_kwargs", {}).get("durable_context_data")]
     handle = next(e["handle"] for e in result["tool_artifacts"] if e["tool_call_id"] == "call_make")
     assert any(handle in content for content in durable_blocks), "captured handle never reached the model-facing durable context block"
+
+
+@pytest.mark.parametrize(
+    ("url", "closing_punctuation"),
+    [
+        ("https://files.example/report.pdf?token=part.csvX", "]"),
+        ("https://files.example/report.pdf?token=abc&download=copy.csvX#page=2", "]"),
+        ("https://files.example/report.pdf?token=abc", "]"),
+        pytest.param("https://files.example/report.pdf?token=part.csvX#page=2", "\u3002\u201d\uff09", id="cjk-punctuation"),
+        pytest.param("https://files.example/report.PdF?token=part.csvX#page=2", "]", id="mixed-case-extension"),
+    ],
+)
+def test_remote_url_survives_capture_checkpoint_and_resolved_download(url, closing_punctuation):
+    """Exercise the real agent graph with a fake model and offline HTTP transport."""
+    downloaded = []
+
+    @tool("make_file")
+    def remote_report(name: str) -> str:
+        """Create a remotely hosted report."""
+        return f"Download [{url}{closing_punctuation}"
+
+    def serve_report(request: httpx.Request) -> httpx.Response:
+        downloaded.append(request.url)
+        return httpx.Response(200 if request.url.query == httpx.URL(url).query else 403, content=b"report")
+
+    @tool("read_file")
+    def download_report(path: str) -> str:
+        """Download a report by its URL or artifact handle."""
+        assert path == url
+        with httpx.Client(transport=httpx.MockTransport(serve_report)) as client:
+            response = client.get(path)
+            response.raise_for_status()
+        return response.text
+
+    agent = create_agent(
+        model=_cycle_model(),
+        tools=[remote_report, download_report],
+        middleware=[DurableContextMiddleware(), ArtifactCaptureMiddleware(), ArtifactResolutionMiddleware()],
+        state_schema=ThreadState,
+        checkpointer=InMemorySaver(),
+    )
+    config = {"configurable": {"thread_id": THREAD_ID}}
+
+    result = agent.invoke({"messages": [HumanMessage(content="make then download")]}, config, context={"thread_id": THREAD_ID})
+
+    assert downloaded == [httpx.URL(url)]
+    made = next(entry for entry in result["tool_artifacts"] if entry["tool_call_id"] == "call_make")
+    assert made["real_ref"] == url
+    assert made["consumed_by"] == ["call_read"]
+    persisted = next(entry for entry in agent.get_state(config).values["tool_artifacts"] if entry["tool_call_id"] == "call_make")
+    assert persisted["real_ref"] == url
+
+
+def test_glob_directory_survives_capture_checkpoint_and_resolved_listing(tmp_path):
+    from deerflow.sandbox.tools import _format_glob_results
+
+    directory_name = "项目（归档）"
+    real_ref = f"/mnt/user-data/workspace/{directory_name}"
+    directory = tmp_path / directory_name
+    directory.mkdir()
+    (directory / "report.txt").write_text("report", encoding="utf-8")
+    listed = []
+
+    @tool("glob")
+    def glob_directory(path: str, include_dirs: bool) -> str:
+        """Find a directory using the sandbox's numbered glob result format."""
+        assert path == "/mnt"
+        assert include_dirs is True
+        return _format_glob_results(path, [real_ref], truncated=False)
+
+    @tool("ls")
+    def list_directory(path: str) -> str:
+        """List a directory by its sandbox path or artifact handle."""
+        listed.append(path)
+        return "\n".join(sorted(child.name for child in (tmp_path / path.rsplit("/", 1)[-1]).iterdir()))
+
+    model = _cycle_model()
+    model.responses[0].tool_calls[0].update(name="glob", args={"path": "/mnt", "include_dirs": True}, id="call_glob")
+    model.responses[1].tool_calls[0].update(name="ls", id="call_ls")
+    agent = create_agent(
+        model=model,
+        tools=[glob_directory, list_directory],
+        middleware=[DurableContextMiddleware(), ArtifactCaptureMiddleware(), ArtifactResolutionMiddleware()],
+        state_schema=ThreadState,
+        checkpointer=InMemorySaver(),
+    )
+    config = {"configurable": {"thread_id": THREAD_ID}}
+
+    result = agent.invoke({"messages": [HumanMessage(content="find then list the archive directory")]}, config, context={"thread_id": THREAD_ID})
+
+    assert listed == [real_ref]
+    assert next(message.content for message in result["messages"] if getattr(message, "tool_call_id", None) == "call_ls") == "report.txt"
+    found = next(entry for entry in result["tool_artifacts"] if entry["tool_call_id"] == "call_glob")
+    assert found["real_ref"] == real_ref
+    assert found["consumed_by"] == ["call_ls"]
+    persisted = next(entry for entry in agent.get_state(config).values["tool_artifacts"] if entry["tool_call_id"] == "call_glob")
+    assert persisted["real_ref"] == real_ref

backend/tests/test_artifact_registry.py

@@ -2,6 +2,7 @@
 
 import json
 
+import pytest
 from langchain_core.messages import ToolMessage
 
 from deerflow.agents.thread_state import merge_tool_artifacts
@@ -41,6 +42,117 @@ def test_generate_handle_format():
     assert len(handle) == len("art_") + 8
 
 
+@pytest.mark.parametrize(
+    "url",
+    [
+        "https://files.example/report.pdf?token=part.csvX",
+        "https://files.example/report.pdf#section.md-more",
+        "https://files.example/report.pdf?token=abc&download=copy.csvX#page=2",
+        "https://files.example/report.pdf?token=abc",
+        "https://files.example/report.pdf#page=2",
+        "https://files.example/report.pdf?token=part%EF%BC%89#section%EF%BC%8C",
+    ],
+)
+@pytest.mark.parametrize("content_block", [False, True])
+def test_text_file_urls_preserve_complete_query_and_fragment(url, content_block):
+    text = f"Report: [{url}]"
+    result = ToolMessage(content=[{"type": "text", "text": text}] if content_block else text, tool_call_id="call_url", name="remote_report")
+
+    entries = extract_artifacts_from_result(result, thread_id="thread-url")
+
+    assert [entry["real_ref"] for entry in entries] == [url]
+
+
+@pytest.mark.parametrize(
+    "suffix",
+    [
+        pytest.param("\u3002", id="cjk-period"),
+        pytest.param("\uff0c", id="fullwidth-comma"),
+        pytest.param("\uff1b", id="fullwidth-semicolon"),
+        pytest.param("\uff1a", id="fullwidth-colon"),
+        pytest.param("\u3001", id="ideographic-comma"),
+        pytest.param("\uff09", id="fullwidth-parenthesis"),
+        pytest.param("\u3011", id="cjk-bracket"),
+        pytest.param("\u300b", id="cjk-angle-bracket"),
+        pytest.param("\u201d", id="closing-double-quote"),
+        pytest.param("\u2019", id="closing-single-quote"),
+        pytest.param("\u3002\u201d\uff09", id="combined-closers"),
+    ],
+)
+@pytest.mark.parametrize(
+    "ref",
+    [
+        "https://files.example/report.pdf",
+        "https://files.example/report.pdf?token=part.csvX#page=2",
+        "/mnt/user-data/outputs/report.pdf",
+    ],
+)
+def test_text_refs_followed_by_cjk_punctuation(ref, suffix):
+    assert [entry["ref"] for entry in _detect_refs_in_text(f"Report: {ref}{suffix}")] == [ref]
+
+
+@pytest.mark.parametrize("content_block", [False, True])
+@pytest.mark.parametrize("directory_name", ["项目（归档）", "项目【归档】", "项目《归档》", "项目“归档”", "项目‘归档’", "项目（归档【旧版】）", "项目）归档（完成）"])
+def test_glob_result_preserves_balanced_cjk_directory_names(directory_name, content_block):
+    from deerflow.sandbox.tools import _format_glob_results
+
+    root = "/mnt/user-data/workspace"
+    path = f"{root}/{directory_name}"
+    text = _format_glob_results(root, [path], truncated=False)
+    result = ToolMessage(content=[{"type": "text", "text": text}] if content_block else text, tool_call_id="call_glob", name="glob")
+
+    entries = extract_artifacts_from_result(result, thread_id="thread-glob")
+
+    assert [entry["real_ref"] for entry in entries] == [root, path]
+    assert entries[-1]["display_name"] == directory_name
+
+
+@pytest.mark.parametrize("suffix", ["", "。", "。）", "]"])
+@pytest.mark.parametrize("ref", ["/mnt/user-data/workspace/项目（归档）", "https://files.example/report.pdf?label=项目（归档）"])
+def test_text_refs_preserve_balanced_cjk_closers_while_stripping_prose(ref, suffix):
+    assert [entry["ref"] for entry in _detect_refs_in_text(f"Report: （{ref}{suffix}")] == [ref]
+
+
+@pytest.mark.parametrize(
+    "url",
+    [
+        "https://files.example/report.PDF",
+        "https://files.example/image.PNG",
+        "https://files.example/report.PdF?token=part.csvX#section.md-more",
+        "https://files.example/image.PnG?token=encoded%EF%BC%89",
+    ],
+)
+def test_text_file_url_extension_case_insensitive_preserves_reference(url):
+    assert [entry["ref"] for entry in _detect_refs_in_text(f"Download [{url}]")] == [url]
+
+
+def test_structured_url_keeps_literal_cjk_punctuation():
+    url = "https://files.example/report.pdf?token=literal\u3011"
+    result = ToolMessage(content="done", tool_call_id="call_url", name="remote_report", artifact={"structured_content": {"url": url}})
+
+    entries = extract_artifacts_from_result(result, thread_id="thread-url")
+
+    assert [entry["real_ref"] for entry in entries] == [url]
+
+
+@pytest.mark.parametrize(
+    "url",
+    [
+        "https://files.example/report.pdf/download",
+        "https://files.example/report.pdf.gz",
+        "https://files.example/report.pdfx",
+        "https://files.example/download?name=report.pdf",
+        "https://files.example/download#report.pdf",
+        "https://reports.pdf",
+        "https://files.example/report.PDF/download",
+        "https://files.example/report.PDFx",
+        "https://REPORT.PDF",
+    ],
+)
+def test_text_urls_require_a_file_extension_in_the_complete_path(url):
+    assert _detect_refs_in_text(f"Result: {url}") == []
+
+
 def test_extract_from_file_block():
     result = ToolMessage(
         content=[