fix(skills): preserve Unicode separators in JSONL history (#6615)

README.md

@@ -1288,6 +1288,8 @@ Discovery follows operator-managed directory symlinks, but skips links back to a
 
 Skill Markdown and bundled text resources use UTF-8. Skill-creator CLI and review utilities read and write text explicitly as UTF-8 so localized skills behave consistently across operating systems.
 
+Custom skill history preserves Unicode line separators inside saved content and metadata, keeping those revisions readable for history inspection and rollback. Malformed JSON history records are still rejected.
+
 Users can explicitly activate an enabled skill for a single turn by starting the request with `/skill-name`, for example `/data-analysis analyze uploads/foo.csv`. DeerFlow loads that skill's `SKILL.md` as hidden current-turn context while leaving the base prompt limited to skill metadata. Slash activation respects disabled skills, custom-agent skill whitelists, and existing channel commands such as `/new` and `/help`.
 
 After an answer loads skills, its toolbar includes **Skills used**. Hover over or click the icon to see the skills and their sources; hovering a skill name underlines it. Select a skill to inspect its `SKILL.md` in the resizable side panel (a drawer on mobile). The view uses snapshots captured during successful configured read-tool loads or explicit slash activation, so later edits or removal of a skill do not rewrite its history. Repeated loads appear once per run, in first-load order. Range reads and bounded snapshots are labeled as partial; older conversations without captured evidence do not show this menu. Copy returns the captured Markdown, including YAML frontmatter. Package-relative links and images remain readable references rather than navigating away from the conversation. Skill loads performed through other tools, such as shell commands, are not inferred from answer text.

backend/packages/harness/deerflow/skills/AGENTS.md

@@ -1,5 +1,7 @@
 ### Skills System (`packages/harness/deerflow/skills/`)
 
+- **History JSONL**: split records on LF, not Unicode text separators; preserve U+0085/U+2028/U+2029 in saved fields. Keep malformed JSON rejected. Tests: `tests/test_skill_history_jsonl.py`.
+
 - **SkillScan API tokens**: `secret-cloud-token` accepts `-` and `_` in the `sk-` family, matching the character class used by PII redaction. Legacy, OpenAI project/service-account/admin, and Anthropic keys must trigger `CRITICAL` even in a Bearer header without a secret assignment. Skip placeholder matches until a real credential is found, retaining at most one cloud-token finding per file. Keep evidence fully redacted and preserve the minimum length and word boundaries; synthetic regressions live in `tests/test_skillscan_native.py`. Keep the mirrored `sk-` pattern in `app/scheduler/notification_text.py` aligned so scheduled-run summaries scrub the same key formats before IM delivery.
 
 - **SkillScan shell pipes**: resolve `env` launchers through interpreter paths and `command`/`exec`/`sudo` wrappers, including value-taking sudo flags. Preserve direct-pipe coverage of common shell interpreter paths/names and real backslash-newline continuations; keep repeated regex alternatives disjoint so failed matches on literal backslash text cannot stall scanning. Tokenize only the current simple command, in place, under a shared per-file work budget; only its stdin redirections override the pipe. Keep quoted words distinct from operators. Parse GNU `env -S` escapes separately from shell syntax, with iterative expansion capped at 256 splits/65,536 characters. Unknown runtime expansions remain unresolved; never read the scanner environment. Limit exhaustion retains warnings and already collected critical findings. Tests: `test_skillscan_env_review_regressions.py`, `test_skillscan_env_launcher.py`, and the existing native/shell suites.

backend/packages/harness/deerflow/skills/storage/local_skill_storage.py

@@ -268,7 +268,8 @@ class LocalSkillStorage(SkillStorage):
         if not history_path.exists():
             return []
         records: list[dict] = []
-        for line in history_path.read_text(encoding="utf-8").splitlines():
+        # JSONL boundaries are newlines, not Unicode separators inside JSON strings.
+        for line in history_path.read_text(encoding="utf-8").split("\n"):
             if not line.strip():
                 continue
             records.append(json.loads(line))

backend/tests/test_skill_history_jsonl.py

@@ -0,0 +1,56 @@
+"""Skill history records use physical JSONL boundaries, not Unicode text lines."""
+
+import json
+
+import pytest
+
+
+@pytest.fixture(params=["local", "user"])
+def storage(request, monkeypatch, tmp_path):
+    from deerflow.config.paths import Paths
+    from deerflow.skills.storage.local_skill_storage import LocalSkillStorage
+    from deerflow.skills.storage.user_scoped_skill_storage import UserScopedSkillStorage
+
+    root = tmp_path / "skills"
+    if request.param == "local":
+        return LocalSkillStorage(host_path=str(root))
+    monkeypatch.setattr("deerflow.config.paths.get_paths", lambda: Paths(base_dir=tmp_path))
+    return UserScopedSkillStorage("history-owner", host_path=str(root))
+
+
+@pytest.mark.parametrize("separator", ["\u0085", "\u2028", "\u2029"])
+@pytest.mark.parametrize("field", ["prev_content", "new_content", "reason"])
+def test_append_read_round_trip_preserves_unicode_separators(storage, separator, field):
+    records = [{"action": "create", "new_content": "first"}, {"action": "human_edit", field: f"before{separator}after"}, {"action": "human_edit", "reason": "last"}]
+    for record in records:
+        storage.append_history("demo-skill", record)
+
+    # The real writer leaves the separator inside a valid JSON string.
+    raw = storage.get_skill_history_file("demo-skill").read_bytes().decode("utf-8")
+    assert separator in raw
+    assert len([json.loads(line) for line in raw.split("\n") if line.strip()]) == 3
+    actual = storage.read_history("demo-skill")
+    assert [{key: value for key, value in record.items() if key != "ts"} for record in actual] == records
+    assert all(record["ts"] for record in actual)
+
+
+@pytest.mark.parametrize("ending", ["\n", "\r\n"])
+def test_physical_lines_keep_blank_lines_and_unterminated_final_record(storage, ending):
+    path = storage.get_skill_history_file("demo-skill")
+    path.parent.mkdir(parents=True)
+    records = [{"new_content": "first\nsecond"}, {"reason": "last"}]
+    content = ending + " \t" + ending + ending.join(json.dumps(record) for record in records)
+    path.write_bytes(content.encode("utf-8"))
+    assert storage.read_history("demo-skill") == records
+
+
+def test_missing_history_keeps_empty_result(storage):
+    assert storage.read_history("demo-skill") == []
+
+
+def test_malformed_json_is_still_rejected(storage):
+    path = storage.get_skill_history_file("demo-skill")
+    path.parent.mkdir(parents=True)
+    path.write_bytes(b'{"reason":"valid"}\n{"reason":')
+    with pytest.raises(json.JSONDecodeError):
+        storage.read_history("demo-skill")