deer-flow/backend/tests/test_lark_credential_tree_churn.py
Storm c277c1572e
fix(harness): complete Lark device auth via a single-flight login run (#7182)
* fix(harness): complete Lark device auth via a single-flight login run

The web authorization polling loop reused the request wait budget as the
lark-cli auth login subprocess lifetime: each /lark/auth/complete poll ran
the CLI with subprocess timeout=wait_timeout_seconds (8s from the frontend)
and killed it on timeout. The login transaction is not atomic — poll, redeem
the single-use device code, fetch user info, then persist credentials — so a
kill after redemption but before persistence stranded the flow, and the next
poll with the same code failed with 'device_code is invalid'.

Own the login process by the flow generation instead: the first poll spawns
one background run per (user_id, generation) and later polls attach to it;
wait_timeout_seconds only bounds the caller's wait (unchanged 504 contract)
and never terminates the CLI. Advancing the generation or re-issuing a device
code abandons and kills the orphaned process, and a finished run's outcome
stays attachable for 120s so late polls return the same result instead of
re-consuming the redeemed code.

* fix(lark): tolerate vanished credential-tree entries and bound owner-alive grace

Publication could kill the login worker with an unhandled FileNotFoundError
when a concurrently-written temp file vanished between iterdir() and lstat(),
leaving the marker live but no result file ever published; every foreign poll
then blocked 504 until the 600 s attempt bound, and could respawn an
already-redeemed code.  Also, a bare-pid owner_alive check trusted for the
full 600 s let a recycled-pid or crashed-thread marker outlive any process
that could still publish.

- _credential_tree_path_kind returns "gone" on FileNotFoundError so the
  walk skips vanished entries instead of aborting; the same tolerance is
  applied to iterdir().
- _write_lark_login_result_once treats the re-harden as best-effort: a
  filesystem hiccup is logged and publication continues so the marker sweep
  still runs.
- The marker now records owner_pid_started_at and
  _read_live_lark_login_marker_locked only keeps an owner-alive marker
  within a 10 s publication grace, so a stale marker is swept and the flow
  respawns instead of wedging every foreign poll.

Regression: test_walk_and_harden_tolerates_a_vanished_entry,
test_write_result_once_continues_when_hardening_fails,
test_abandoned_marker_with_dead_owner_is_swept_within_grace.
Focused suites: 139 passed, 12 skipped.

* fix(lark): harden surviving credentials through filesystem churn

Atomic login publication can remove a descendant after lstat but before
chmod. Catch FileNotFoundError around the entire per-entry walk so that
lstat, chmod and directory enumeration tolerate those disappearances.
A vanished root, permission error or symlink remains a hard failure.

Exercise real removals at each boundary and concurrent result publication;
assert that injection ran and surviving directories/files reach 0700/0600.
The regressions detect a skipped walk, swallowed permission failures and
loss of directory-churn tolerance in isolated mutation runs.

* fix(lark): retain login ownership through durable completion

Browser approval can take longer than the publication grace. Start that
grace at the first observed CLI exit, then reject abandoned attempts
without retrying a device code that may already have been redeemed.

Serialize atomic result publication under the credential lock. Preserve
published results through cleanup failures and retain terminal failures
on the marker when result storage fails. Never bypass credential hardening.
Reserve storage before spawning and roll back failed marker, process or
thread startup by reaping the CLI, clearing the registry and waking waiters.

Add process/filesystem failure regressions, a long-login two-worker case,
and strict unhandled-thread gates. Fix invocation logging in both workers
and reap forked test CLIs after the parent observes their launch. Document
the authorization lifecycle and required regression assertions.

Validated the reproduced failures and ten isolated mutations; final
focused, blocking-I/O, full-suite and independent review details are in PR #7182.
2026-10-21 21:00:35 +08:00

178 lines
6.7 KiB
Python

"""Real filesystem churn must preserve credential hardening and publication."""
from __future__ import annotations
import json
import os
import stat
import threading
from pathlib import Path
import pytest
from deerflow.config import paths as paths_module
from deerflow.config.paths import Paths
from deerflow.integrations import lark_cli
pytestmark = [
pytest.mark.skipif(os.name == "nt", reason="POSIX credential-tree semantics"),
pytest.mark.filterwarnings("error::pytest.PytestUnhandledThreadExceptionWarning"),
]
def _seed(base_dir: Path, user_id: str) -> dict[str, Path]:
root = Paths(base_dir=base_dir).user_dir(user_id) / "integrations" / lark_cli.INTEGRATION_ID
paths = {"root": root, "config": root / "config", "locks": root / "config" / "locks", "data": root / "data"}
for directory in paths.values():
directory.mkdir(parents=True, exist_ok=True)
directory.chmod(0o755)
for name in ("victim", "survivor"):
paths[name] = paths["data"] / f"{name}.json"
paths[name].write_text("{}", encoding="utf-8")
paths[name].chmod(0o644)
return paths
def _mode(path: Path) -> int:
return stat.S_IMODE(path.lstat().st_mode)
@pytest.mark.parametrize("boundary", ["lstat", "chmod", "iterdir"])
def test_vanished_descendant_is_skipped_and_survivors_are_hardened(monkeypatch, tmp_path, boundary):
paths = _seed(tmp_path, "alice")
victim = paths["victim"]
if boundary == "iterdir":
victim = paths["root"] / "transient"
victim.mkdir()
injected = []
original = getattr(Path, boundary)
def remove_at_boundary(path, *args, **kwargs):
if path == victim and not injected:
injected.append(path)
if boundary == "iterdir":
path.rmdir()
else:
path.unlink()
return original(path, *args, **kwargs)
monkeypatch.setattr(Path, boundary, remove_at_boundary)
lark_cli.ensure_lark_cli_credential_tree("alice", paths=Paths(base_dir=tmp_path))
assert injected == [victim], "the walk never reached the injected boundary"
assert not victim.exists(), "the test must actually remove the candidate"
for name in ("root", "config", "locks", "data"):
assert _mode(paths[name]) == 0o700, name
assert _mode(paths["survivor"]) == 0o600
def test_root_renamed_at_walk_lstat_propagates(monkeypatch, tmp_path):
root = _seed(tmp_path, "alice")["root"]
moved = tmp_path / "moved-root"
injected = []
armed = threading.Event()
original_lstat, original_mkdir = Path.lstat, Path.mkdir
def mkdir(path, *args, **kwargs):
result = original_mkdir(path, *args, **kwargs)
if path == root:
armed.set()
return result
def lstat(path):
if path == root and armed.is_set() and not injected:
injected.append(path)
root.rename(moved)
return original_lstat(path)
monkeypatch.setattr(Path, "mkdir", mkdir)
monkeypatch.setattr(Path, "lstat", lstat)
with pytest.raises(FileNotFoundError):
lark_cli.ensure_lark_cli_credential_tree("alice", paths=Paths(base_dir=tmp_path))
assert injected == [root]
assert moved.is_dir() and not root.exists()
@pytest.mark.parametrize("target", ["root", "victim"])
def test_permission_errors_still_propagate(monkeypatch, tmp_path, target):
paths = _seed(tmp_path, "alice")
injected = []
original = Path.chmod
def chmod(path, *args, **kwargs):
if path == paths[target]:
injected.append(path)
raise PermissionError("synthetic chmod denial")
return original(path, *args, **kwargs)
monkeypatch.setattr(Path, "chmod", chmod)
with pytest.raises(PermissionError, match="synthetic chmod denial"):
lark_cli.ensure_lark_cli_credential_tree("alice", paths=Paths(base_dir=tmp_path))
assert injected == [paths[target]]
def test_symlink_descendant_is_still_rejected(tmp_path):
paths = _seed(tmp_path, "alice")
outside = tmp_path / "outside"
outside.mkdir()
(paths["data"] / "link").symlink_to(outside, target_is_directory=True)
with pytest.raises(ValueError, match="symlink"):
lark_cli.ensure_lark_cli_credential_tree("alice", paths=Paths(base_dir=tmp_path))
def test_login_result_publication_survives_real_unlink_churn(monkeypatch, tmp_path):
monkeypatch.setattr(paths_module, "_paths", Paths(base_dir=tmp_path))
generation = "gen-pub"
digest = lark_cli._lark_login_code_digest("code-A")
listed, removed = threading.Event(), threading.Event()
original_iterdir, original_lstat = Path.iterdir, Path.lstat
missing_candidates = set()
# Acquire before seeding loose permissions, so publication's real walk is
# responsible for fixing them while another thread removes listed files.
with lark_cli._lark_credential_lock("alice"):
paths = _seed(tmp_path, "alice")
victims = {paths["victim"], paths["data"] / "victim-b.json"}
(paths["data"] / "victim-b.json").write_text("{}", encoding="utf-8")
def iterdir(path):
entries = list(original_iterdir(path))
if path == paths["data"] and not listed.is_set():
assert victims <= set(entries)
listed.set()
assert removed.wait(10), "churn thread did not remove listed files"
return iter(entries)
def lstat(path):
if path in victims and removed.is_set():
missing_candidates.add(path)
return original_lstat(path)
def churn():
assert listed.wait(10), "publication never enumerated the data directory"
for victim in victims:
victim.unlink()
removed.set()
monkeypatch.setattr(Path, "iterdir", iterdir)
monkeypatch.setattr(Path, "lstat", lstat)
thread = threading.Thread(target=churn, name="lark-credential-churn")
thread.start()
try:
assert lark_cli._write_lark_login_result_once("alice", generation, digest, {"ok": True})
finally:
thread.join(10)
assert not thread.is_alive()
assert listed.is_set() and removed.is_set()
assert missing_candidates == victims, "the walk must inspect the removed candidates"
assert all(not victim.exists() for victim in victims)
result = lark_cli._lark_login_result_path("alice", generation, digest)
assert json.loads(result.read_text(encoding="utf-8")) == {"ok": True}
assert list(paths["root"].glob(".login-result-*.json")) == [result]
assert not list(paths["root"].glob(f".{result.name}.*"))
assert _mode(paths["survivor"]) == 0o600
assert _mode(result) == 0o600
for name in ("root", "config", "locks", "data"):
assert _mode(paths[name]) == 0o700, name