mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-06-09 17:12:01 +00:00
88759015e4
* test(e2e): record/replay front-back contract verification Guards the front-back contract with a deterministic, key-free record/replay harness (mirrors open-design's golden-trace approach): - ReplayChatModel (tests/replay_provider.py): replays recorded LLM turns by a normalized hash of the model input. Strips <system-reminder>/date/uuid/tmp-path so one fixture replays across days and from both the browser and direct-POST paths; a miss raises loudly (no silent divergence). - Recording is record-through-browser (scripts/record_gateway.py + build_fixture_from_jsonl.py + frontend/tests/e2e-record): a real run is driven through the real frontend so captured inputs match exactly what the browser sends; fixtures contain no API key. - Layer 1 — backend golden (tests/test_replay_golden.py): replay through the real gateway, assert the SSE event sequence == committed golden. - Layer 2 — full-stack render (frontend/tests/e2e-real-backend): real Next.js + real gateway (replay model) + Chromium; assert the replayed auto-title and follow-up suggestions render. DOM assertions are the gate; visual regression is a local dev gate (CI uploads the render as an artifact). - CI (.github/workflows/replay-e2e.yml): both layers, triggered on EITHER side of the contract (frontend/** or backend gateway/harness/fixtures). * test(e2e): multi-run render-order cross-stack scenario (#3352) Guards the dangerous front-back class where a backend ordering change silently breaks a frontend assumption while both sides' unit tests stay green. Reproduces issue #3352: backend list_by_thread returns runs newest-first (#2932) and the frontend prepended per-run pages, inverting chronological order once the checkpoint no longer held the older messages. - tests/seed_runs_router.py: test-only seeder, mounted on the replay gateway only when DEERFLOW_ENABLE_TEST_SEED=1 (never in the production app). Seeds a thread with >=2 runs + per-run message events and no checkpoint -- the #3352 precondition -- so the frontend per-run reload path is the sole source of truth and the prepend inversion is observable. - frontend/tests/e2e-real-backend/multi-run-order.spec.ts: drives the real frontend against the real gateway, asserts the first run renders above the second. Reverting the #3354 fix turns it red. - replay-e2e.yml: trigger on the new replay test-infra paths. - docs: REPLAY_E2E.md cross-stack scenario section. * test(e2e): address Copilot review on the replay harness - Fix stale recorder references (scripts/record_traces.py -> scripts/record_gateway.py + scripts/build_fixture_from_jsonl.py) in replay_provider.py, test_replay_golden.py, _replay_fixture.py. - MODE_CONTEXT['ultra']: thinking_enabled False -> True, mirroring the frontend's `context.mode !== 'flash'` (hooks.ts). It did not affect the hashed input (Layer 1 golden still green), but the table now matches the real frontend context it claims to mirror. - replay_provider.py docstring: stop claiming memory is recorded-enabled; the replay config disables memory/summarization for determinism (title stays, as an in-graph deterministic call). - record_gateway.py / run_replay_gateway.py: override DEER_FLOW_HOME instead of setdefault, so an outer value can't leak into the hermetic harness. - record_gateway.py: clear error when DEERFLOW_RECORD_OUT is unset (was a bare KeyError). - playwright.record.config.ts: forward OPENAI_*/DEERFLOW_RECORD_OUT only when set, so the gateway raises a clear 'missing env' error instead of getting ''. * test(e2e): address Copilot review round 2 - seed_runs_router.py: constrain SeedMessage.role to Literal['human','ai'] so a bad value is a clean 422 at the boundary instead of a 500 (KeyError on _EVENT_TYPE). - record-write-read-file.spec.ts: waitForCaptureStable now throws on timeout instead of returning the last count, so a truncated/partial recording can't pass silently. - real-backend-render.spec.ts: guard the suggestions JSON.parse; a bracket-prefixed non-JSON turn falls back to '' so the existing not.toBe('') assertion fails clearly instead of a generic parse throw.
61 lines
2.1 KiB
TypeScript
61 lines
2.1 KiB
TypeScript
import { defineConfig, devices } from "@playwright/test";
|
|
|
|
/**
|
|
* Layer 2 of the record/replay e2e: the REAL Next.js frontend rendering data
|
|
* from a REAL gateway whose LLM is the deterministic `ReplayChatModel` (no API
|
|
* key). This is separate from `playwright.config.ts` (which mocks the backend)
|
|
* so the mock-based suite is untouched.
|
|
*
|
|
* Two webServers are started: the replay gateway (:8011) and the frontend
|
|
* (:3000, pointed at the gateway). Auth uses a throwaway test account the spec
|
|
* registers at runtime — no secrets.
|
|
*/
|
|
export default defineConfig({
|
|
testDir: "./tests/e2e-real-backend",
|
|
fullyParallel: false,
|
|
forbidOnly: !!process.env.CI,
|
|
retries: process.env.CI ? 1 : 0,
|
|
workers: 1,
|
|
reporter: process.env.CI ? "github" : "html",
|
|
timeout: 90_000,
|
|
|
|
use: {
|
|
baseURL: "http://localhost:3000",
|
|
trace: "on-first-retry",
|
|
},
|
|
|
|
projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }],
|
|
|
|
webServer: [
|
|
{
|
|
command: "uv run python scripts/run_replay_gateway.py --port 8011",
|
|
cwd: "../backend",
|
|
url: "http://localhost:8011/health",
|
|
reuseExistingServer: !process.env.CI,
|
|
timeout: 180_000,
|
|
stdout: "pipe",
|
|
stderr: "pipe",
|
|
// Mount the test-only run/message seeder used by multi-run-order.spec.ts
|
|
// (#3352). The endpoint exists only on this replay gateway, never in the
|
|
// production app.
|
|
env: { DEERFLOW_ENABLE_TEST_SEED: "1" },
|
|
},
|
|
{
|
|
command: "pnpm build && pnpm start",
|
|
url: "http://localhost:3000",
|
|
reuseExistingServer: !process.env.CI,
|
|
timeout: 240_000,
|
|
env: {
|
|
SKIP_ENV_VALIDATION: "1",
|
|
DEER_FLOW_AUTH_DISABLED: "1",
|
|
BETTER_AUTH_SECRET: "local-dev-secret",
|
|
// Leave NEXT_PUBLIC_* unset so the frontend uses its built-in
|
|
// next.config rewrites (same-origin proxy) instead of talking to the
|
|
// gateway cross-origin — cross-origin fetches drop the auth cookies.
|
|
// Just point that proxy at the replay gateway.
|
|
DEER_FLOW_INTERNAL_GATEWAY_BASE_URL: "http://127.0.0.1:8011",
|
|
},
|
|
},
|
|
],
|
|
});
|