* fix(frontend): support standalone demo APIs and runtime GitHub stars
* Update API origin URL to use environment variables
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* test(frontend): align static demo tests with runtime origin
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* feat(artifacts): preview CSV and TSV files as bounded tables
* chore: keep preview screenshots out of the PR file diff
* fix(artifacts): detect record newlines outside quoted fields
* test(auth): include project permissions in me contract expectations
* feat(projects): project workspaces with scoped chats and thread membership
Backend:
- projects table model and migration; fail-closed ProjectRepository with
ownership checks, CRUD/archive/restore/delete router, and atomic thread
move between projects
- threads_meta.project_id column exposed as reserved deerflow_project_id
metadata; project-aware thread create/search with pagination bounds and
membership echoed in create responses
- first-run admission assigns the project only at genuine first run, seeded
at write time and dropped when invalid; serialized against project
deletion and thread assignment
- branch creation inherits the source thread's project membership (an
archived/deleted project degrades the branch to unassigned instead of
failing the request)
Frontend:
- projects data layer, thread move API, and sidebar projects section with
flat/grouped modes, archived-project threads, and stable virtual-list
offsets
- project detail page with project-scoped new chat
(/workspace/chats/new?project=) and paginated thread list
- move-to-project thread menu, new-project dialog, archived-project gates
- project-scoped new chats pre-create the thread with membership before the
first submit or /goal set, so runs never proceed outside the project
- goal-set preparation is fenced against conversation switches: a stale
continuation is dropped instead of saving the goal or launching the
abandoned submission on the newly opened conversation
- project thread lists join thread lifecycle invalidations (stop, pin) so
an open project page never keeps stale titles, recency, or pagination
* fix(chats): keep archive undo toast when the sidebar row unmounts
The archive success toast was fired from per-mutate callbacks passed to
mutation.mutate. React Query drops those handlers when the observer
component unmounts before the mutation settles; archiving the open chat
removes its sidebar row mid-flight, so the undo toast never appeared and
the e2e archive-undo test timed out waiting for it.
Move the success/error handlers to the mutation level (useArchiveThread
options, same pattern as useMoveThreadToProject) where callbacks are
delivered even after the originating row unmounts.
* fix(projects): pin project thread listing contract and exclude archived chats
GET /api/projects/{id}/threads returned the thread store row verbatim
(list[dict], no response_model): user_id/assistant_id leaked, any future
ThreadMetaRow column would auto-leak, and the OpenAPI schema was empty.
Return a narrow ProjectThreadResponse (the exact fields ProjectThread
declares) with the same metadata secret redaction the surrounding thread
endpoints get from _MetadataRedactingResponse.
The listing also ran search() without the archived filter, so a retired
chat rendered as a normal row on the project page while the sidebar hid
it. Search archived=False to mirror the sidebar's archived:false lists;
restore stays on the global Archived tab.
Both regressions pinned by new router tests: wire-shape allowlist and
archived-member exclusion.
* docs(migrations): record the 0019/0020 chain against the bootstrap reservation
The tree now chains 0018 -> 0019_projects -> 0020_threads_meta_project_id,
so migrations/AGENTS.md was stale twice over: the revision index stopped at
0018 and the rolling-forward section still claimed the tree 'deliberately
remains at 0018'.
Document the new head and record the intentional numeric-prefix reuse of
0019: 0019_projects is in-chain while 0019_thread_incarnations stays the
reserved, allowlisted out-of-tree rollout id. The owning rollout revision
must re-parent onto this tree's head when it merges so alembic never sees
two heads off 0018; bootstrap.py now cross-references that note next to
_FORWARD_COMPATIBLE_REVISION.
* fix(chats): invalidate project thread lists on archive/restore
useArchiveThread refreshed the infinite sidebar cache, threads/search and
the per-thread metadata cache but not the project-scoped list
([...PROJECTS_QUERY_KEY, 'threads', id]) this PR adds — the one thread
mutation not wired to that key, after usePinThread, useRenameThread,
useDeleteThread, useMoveThreadToProject and invalidateStoppedThreadCaches.
An archive from a sidebar row while a project page is open therefore left
the archived chat rendered as a normal row until remount (and undo left it
missing). Invalidate the prefix in the mutation-level success handler.
Regression test asserts the project-list prefix is invalidated on success.
* fix(projects): fetch project discovery only in grouped sidebar mode
RecentChatList mounted two useProjects queries per sidebar render, but
knownProjectIds is consumed only by the grouped-mode exclusion filter; in
the default flat mode every page load paid two GET /api/projects?status=
round trips for data nothing read. Gate both queries on grouped mode —
GroupedProjectList fetches the same keys when the toggle is on and
TanStack dedupes the observers.
Also set retry: false on useProject: a deleted or foreign project 404s
deterministically, and the page renders a dedicated not-found state for
it, so the default 1s/2s/4s retry backoff kept deep links in 'loading'
for ~7s before that state appeared. Matches useThreadMetadata /
useThreadTokenUsage.
* fix(threads): fail closed on project-scoped create in memory mode
MemoryThreadMetaStore.create accepted project_id and silently ignored it,
making memory mode the one membership path that fails open: POST
/api/threads with a project id returned 200 and the run started
unassigned, violating the invariant that a run never proceeds outside the
selected project (the SQL store raises ProjectNotAssignableError inside
the insert transaction for the same request).
Raise ProjectNotAssignableError whenever project_id is present so the
router's existing 404 mapping applies, the frontend keeps the composer
text for a retry, and memory mode behaves exactly like SQL mode.
set_project already reports rejection; create now matches it.
Store-level test (raises, nothing persisted, project filter stays empty,
unscoped creates still work) plus a router-level test asserting the 404
and that no row is left behind.
* fix(projects): window the project page thread list
ProjectThreadsSection rendered every loaded page as a plain Link row, so a
long-lived project accumulated unbounded DOM on the page's scroll surface:
each load-more appended another 100 rows and every formatTimeAgo tick
re-rendered the whole list.
Reuse VirtualThreadList (now generic over any row shape with a
thread_id), pointing its scroll parent at this page's ScrollArea viewport
via the shared [data-slot="scroll-area-viewport"] selector used by
/workspace/chats; under the 60-row threshold it falls back to the plain
render, so small projects are unchanged.
* fix(projects): restore row dividers and pin them with a render test
The row class template literal concatenated transition-colors directly
with the conditional border-b token, so non-final rows rendered the
invalid class 'transition-colorsborder-b' and lost both the divider and
the transition. Compose the row classes with cn() and a boolean guard
instead.
The section moved out of page.tsx into a testable component so the row
markup finally has coverage: a DOM test asserts every row except the
final data row carries border-b (index-based, not last: — correct under
virtualization where the last mounted row is not the last data row), and
the untitled fallback plus load-more button render for a partial page.
* fix(projects): validate forward schemas and fence membership reads
## Why
The composer in input-box.tsx carried a commented-out legacy <PromptInputActionMenu> attachments block (TODO: Add more connectors here) left over from before the AddAttachmentsButton component replaced it. Dead commented UI misleads maintainers into thinking the old path is live or half-migrated, and it has no runtime effect.
## What changed
- Deleted the 9-line commented-out JSX block between <PromptInputTools> and <AddAttachmentsButton>.
- No component, import, or i18n key changed: PromptInputActionMenu* are still used by the live menus below, and AddAttachmentsButton already provides the attachments entry point.
## Surface area
- [x] Frontend UI - composer tool row, comment-only change
- [ ] Backend API / Agents / Sandbox / Skills / Dependencies / Default behavior change
## Validation
- Comment-only deletion: no behavior change; the surrounding JSX is byte-identical outside the removed lines.
- Full pnpm check requires node_modules install on this host; diff is limited to dead comments so lint/typecheck risk is nil.
## AI assistance
**Tool(s) used:** Codex (coding agent)
**How you used it:** located and removed the dead block with AI assistance; reviewed before commit.
- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
* fix(frontend): truncate long subtask card titles to a single line
The subtask card header rendered task.description without any width
constraint; when a provider omits the optional description, the full
task prompt becomes the title and overflows the card.
Wrap the title in a truncating span (full text remains available via
the title tooltip and the expanded card body), give the step min-w-0
flex-1, and pin the status cluster with shrink-0 so overflow resolves
at the title.
Add an e2e test asserting a long prompt renders with the truncate
class, a real ellipsis (scrollWidth > clientWidth), and single-line
height.
* fix(frontend): keep subtask card status cluster shrinkable on narrow viewports
The shrink-0 status cluster could not shrink below its max-content (model
label + usage + status pill, up to ~456px with a long tool-call
description), so on narrow viewports it overflowed the header row while the
title collapsed to zero. Drop shrink-0 and add min-w-0 to both the cluster
and the pill (the pill's min-content is the status text's longest
unbreakable word, so one min-w-0 was not enough), and floor the title at
min-w-24 so it stays visible.
Also extend the e2e spec per review: an in_progress shimmer truncation test
(held-open SSE stream keeps the card running), a 375px no-overflow
assertion for both the resting and running card, and a pixel-budget
single-line check instead of parseFloat(lineHeight) which NaNs on the
'normal' keyword.
* test(frontend): honest fixture text and explicit visibility timeout in subtask spec
Review nits: the long-title fixture lifted the stopped test's human turn
whose text narrates the stop scenario; give it its own LONG_TASK_USER_TEXT
and override content alongside id and tool_calls. Add the missing 15s
timeout on the running-375px title visibility assertion so a future
reorder doesn't turn the 5s default into a cold-start flake.
* fix(messages): drop legacy <uploaded_files> tag handling (#4212)
PR #4174 unified upload-context injection on <current_uploads> (IM and web
both flow through UploadsMiddleware), and #4632 documented the current
path. This removes the remaining backward-compat parsing of the
pre-#4174 <uploaded_files> tag, the final cleanup item tracked by the
issue:
- deermem: only <current_uploads> is stripped from human turns before
memory persistence, and the upload-sentence scrubber drops the legacy
tag alternative.
- mem0: the mirrored message filter recognises only <current_uploads>.
- InputSanitizationMiddleware: remove the legacy tag from the blocked-tag
denylist (it existed only because deermem parsed the old tag).
- frontend: stripUploadedFilesTag / stripInternalMarkers /
parseUploadedFiles and the message-list fallback parse only
<current_uploads>; demo thread fixtures are migrated to the current tag.
Scope decision: a <uploaded_files> block in pre-#4174 history is now
treated as ordinary user content (pinned by tests in both layers) instead
of being silently dropped or stripped.
* style: apply prettier formatting to stripUploadedFilesTag
* fix(uploads): keep legacy <uploaded_files> stripping for display/export only
Addresses review feedback on #4826: removing the legacy tag from the
frontend display layer made pre-#4174 threads render raw <uploaded_files>
XML (with server-side upload paths) in chat, copy data, and JSON exports.
The backend cleanup stands — memory pipelines and the sanitization denylist
treat only <current_uploads> as an internal marker. The frontend keeps the
legacy spelling in its display/export-only utilities
(stripUploadedFilesTag / INTERNAL_MARKER_TAGS / parseUploadedFiles and the
message-list fallback) so old history renders cleanly without leaking
internal paths, while the memory/sanitization scope-decision tests remain
unchanged.
Frontend tests now pin both spellings: <current_uploads> and legacy
<uploaded_files> are stripped from copy data, markdown leak-stripping, and
JSON exports.
* docs(ui): record accepted display-spoof tradeoff for legacy upload tag
Review note (willem-bd): since <uploaded_files> is off the sanitization
denylist, a live user can type the legacy spelling and fabricate file
chips / hide their own message text in display. Display-only and
self-inflicted with no backend semantics, so it is accepted for now;
documented at both the message-list fallback and stripUploadedFilesTag.
Age-gating the legacy spelling remains a possible follow-up.
---------
Co-authored-by: betterkite <313258397+betterkite@users.noreply.github.com>
* perf(frontend): cache settled copy-data derivation across streaming chunks
Every SSE values chunk re-renders MessageList, and the re-render re-derived
copy/toolbar text for every settled row: getAssistantTurnCopyData re-ran the
O(turn bytes) content extraction per settled group, and MessageListItem's
toolbar recomputed getMessageCopyData per message. Settled group arrays keep
their identity across chunks (deriveStableMessageGroups), so both derivations
now cache on that stable reference: a WeakMap keyed on the messages array for
turn copy data, and a useMemo on message identity for the toolbar copy text.
Fixes#5094
* fix(frontend): gate row copy-data memo and correct cache win claim
Address review: derive one memoized copy value only when
isHuman || (!isLoading && showCopyButton) and reuse it for both editing
and the toolbar, so settled assistant rows (whose toolbar never renders)
skip the derivation and human rows derive once, not twice; correct the
assistantTurnCopyDataCache comment — the regex/trim split is already
cached per message, the cache's win is the traversal/allocations for
string turns and the uncached O(bytes) map/join/trim for array-content
turns (benchmarked: 5.1x / 17.2x per settled history sweep).
* style(frontend): expand single-line messages array for Prettier
* feat(frontend): render markdown artifacts in the new window
The artifacts panel's "open in new window" action handed the browser the
raw Gateway response. For markdown that is a `text/markdown` body the
browser can only show as source, so the new window was a text dump rather
than a reader.
Route markdown artifacts to a new `/artifacts/view` page that renders them
with the same components the panel uses (SafeStreamdown + the artifact
rehype chain + citation links/panel), including the truncated-preview
banner and its "load full file" action. Everything else keeps the raw
Gateway URL — notably HTML/SVG, which the Gateway deliberately serves as a
download so active content never executes in the application origin.
- `core/artifacts/viewer.ts` centralizes which stored artifacts are
markdown (`.skill` archives included, since they hold a SKILL.md), so
the panel and the viewer route cannot drift.
- `ArtifactFilePreview` and its siblings move out of
`artifact-file-detail.tsx` into `artifact-file-preview.tsx`; otherwise
the standalone route would pull the CodeMirror editor into its bundle.
- The window title comes from the route's `generateMetadata`, not
`document.title`, which the App Router overwrites after hydration.
- The viewer reads content through `useStandaloneArtifactContent`, which
shares `useArtifactContent`'s query key but not its `useThread`
dependency, since a detached window has no thread context.
Claude-Session: https://claude.ai/code/session_013AiCrC5SBc3HdFYNxsp1EC
* fix(frontend): keep the artifact target across re-authentication
Review found the standalone viewer unrecoverable from an expired session.
The window's target lives entirely in `?path=...&thread_id=...`, and both
auth paths dropped it:
- The layout guard redirected to `/login` with no `next` at all. A layout
cannot read `searchParams`, so the guard moves into the page, which can
— and rebuilds the full viewer address for `next`. The layout loses its
AuthProvider along the way: nothing under this route reads `useAuth`,
and the guard now makes a single `getServerSideUser` call per request.
- The shared fetch wrapper built `next` from `window.location.pathname`,
which silently truncated the query string. It now carries `search` too,
so any route holding state in the query survives a 401, not just this
one. `validateAuthNextPath` already accepts a query string.
`buildArtifactViewerURL` is split out of `resolveArtifactOpenURL`: the
guard needs the route itself, never the Gateway fallback that the latter
takes for non-markdown targets.
Tests: the login round trip (unit — the rebuilt URL survives
`validateAuthNextPath` and parses back to the same target), the fetch
wrapper preserving the query on 401 (unit), and the expired-session
window reaching `/login` with the artifact intact (E2E). The E2E asserts
on the popup's navigation *requests*, since `(auth)/layout` answers
`/login` with a server redirect under DEER_FLOW_AUTH_DISABLED and no
navigation commits.
`tests/unit/core/models/api.test.ts` stubbed `window.location` without
`search`; a real Location always has it.
Claude-Session: https://claude.ai/code/session_013AiCrC5SBc3HdFYNxsp1EC
* fix(frontend): keep public showcase artifacts out of the auth gate
Review found that the viewer's access check regressed `/showcase`. Those
pages render with `isMock`, their artifacts are served by the
unauthenticated demo route, and the raw artifact URL this window replaced
stayed public — so gating the window unconditionally bounced every
logged-out showcase visitor to /login for a document that is already
public.
`requiresAuthenticatedViewer` exempts a mock target only when
`resolveStaticDemoArtifact` would actually serve it. The allowlist is the
authority rather than the flag: `mock=true` is caller-supplied, so a
target the demo route answers with 404 — a non-allowlisted path, or a
thread that is not a demo thread — still needs a session.
Covered in `tests/e2e-auth/`, since the default E2E config disables auth
and cannot see this: a public showcase artifact renders without a
session, while a non-allowlisted path and a missing mock flag both land
on /login. Verified the positive case goes red without the exemption.
Claude-Session: https://claude.ai/code/session_013AiCrC5SBc3HdFYNxsp1EC
* fix(frontend): restore sanitization in custom streamdown rehype chains
Streamdown 2.5 replaces its entire default rehype chain
[rehype-raw, rehype-sanitize, rehype-harden] with whatever array the
caller passes via the rehypePlugins prop. Every custom chain in this
repo therefore rendered LLM/stored markdown without any sanitization:
- Artifact markdown previews (markdown-preview-plugins.ts +
artifact-file-detail.tsx) parse raw HTML via rehypeRaw, so a
generated .md artifact could inject <style>/<iframe>/on* handlers
into the workspace DOM (stored XSS; only javascript: anchors were
blocked by the ArtifactLink component).
- The memory settings summary (memory-settings-page.tsx) spread the
shared preset without component overrides, so a hostile
<a href="javascript:..."> in stored memory content rendered as a
clickable anchor.
Fix strategy:
- Add rehype-sanitize (already resolved in the lockfile via streamdown)
as a direct dependency and re-insert a [rehypeSanitize, schema] step
in the shared preset (core/streamdown/plugins.ts). It runs after
rehypeRaw (raw HTML must be parsed into hast before it can be
cleaned) and before rehypeKatex/rehypeSlug (their output is trusted
and would otherwise be filtered or clobbered) - the same
raw -> sanitize -> math ordering streamdown itself uses.
- The schema extends rehype-sanitize's GitHub-style defaultSchema (the
base of streamdown's own sanitize schema) so legitimate authored
artifact HTML (tables, details, images, alignment/size attributes)
keeps working while script/iframe/style, on* handlers and
non-allow-listed URL schemes (javascript:, data:, ...) are dropped.
The only extensions are tel: hrefs and the math-inline/math-display
class markers remark-math emits and rehype-katex detects.
- Position rehypeSlug after the sanitize step in the artifact chain so
sanitize's id clobbering (id="x" -> id="user-content-x") cannot break
the heading anchors it creates.
- Pass a: createMarkdownLinkComponent() on the memory settings page as
defense in depth, matching the chat rendering path.
Unit tests feed a hostile payload (<a href="javascript:...">,
<img onerror>, <script>, <iframe>, <style>, ontoggle) through both
render paths and assert no executable/clickable equivalent survives,
plus regression guards for heading anchors, legitimate HTML and KaTeX
math rendering.
* fix(frontend): keep the sanitize clobber prefix on heading anchors; minimal lockfile
Review follow-ups on the sanitization change:
- Heading anchors: rehypeScopedSlug replaces rehype-slug in the artifact
chain. It runs after the sanitize step (so raw-HTML headings are also
anchored) but keeps rehype-sanitize's user-content- id prefix — an
untrusted heading like "## current" cannot mint an unprefixed
id="current" (the DOM-clobbering shape the sanitizer guards against).
In-page fragment links are translated to the prefixed anchors so they
still resolve; external URLs, bare "#", already-prefixed fragments and
sanitize-prefixed raw-HTML ids are left untouched.
- Lockfile: regenerated as a minimal diff — only the two direct-dependency
importer entries (rehype-sanitize, github-slugger for the scoped slug)
are added; the libc platform selectors on the 64 native package records
are preserved byte-for-byte instead of being dropped by lockfile
normalization.
Full frontend suite: 1034 tests passing; tsc and prettier clean.
* style: reorder github-slugger import ahead of the hast type import
* test(e2e): expect the clobber-prefixed heading anchor in artifact preview
The scoped slug plugin gives generated heading ids rehype-sanitize's
user-content- prefix and translates fragment links to match, so the
anchor-scroll test must locate the prefixed id.
* fix(frontend): reset the scoped slugger per tree; keep footnote anchors single-prefixed
Review follow-ups:
- The scoped slug attacher holds one GithubSlugger, but streamdown
caches the unified processor by plugin name, so the instance survived
across parses and repeated renders of the same heading grew -1/-2
suffixes (the artifact-anchor e2e could not find the id on re-render).
The transformer now resets the slugger per tree, as rehype-slug does;
a regression test renders identical artifact markdown twice.
- remark-rehype emits GFM footnote anchors already clobber-prefixed
(user-content-fn-1); the sanitize step prefixed those ids again while
their hrefs stayed single-prefixed, breaking footnote navigation in
every chain built on the shared preset. A new rehypeClobberFragments
step runs right after sanitize: double-prefixed ids are normalized
back to one prefix, and unprefixed fragment hrefs are translated to
the prefixed form (already-prefixed and external links untouched).
The artifact slug now inserts after this step; covered by a footnote
regression test on the shared render path.
Unit suite 1036 passing; artifact-preview e2e verified locally
(9/9, including the heading-anchor scroll test).
* fix: DeerFlow brand casing in disclaimer, 'What's New' title, setup-sandbox .PHONY
- en-US disclaimer said 'Deerflow'; every other user-facing string
brands it 'DeerFlow' (rendered under the chat composer)
- landing section title 'Whats New in DeerFlow 2.0' -> 'What's New'
- setup-sandbox was the only Makefile target missing from .PHONY
* test: align disclaimer assertions with DeerFlow brand casing
The unit and e2e suites still expected the old "Deerflow" spelling in
inputBox.disclaimer; update both to the corrected locale string so the
i18n load test and the chat footer assertion pass again.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(artifacts): serve SHA-256 via ETag so preview/edit work on non-secure contexts
crypto.subtle is only available in secure contexts (HTTPS or localhost). The frontend fell back to it to compute an artifact's SHA-256 when the Gateway did not return one, which threw on http://<lan-ip>:<port> and broke both artifact preview and inline editing (issue #4864).
- Gateway now returns the real SHA-256 as an ETag header for inline text and active-content artifact responses (and skill-archive members).
- Frontend prefers the ETag and only computes a hash as a last resort, falling back gracefully (FNV-1a) instead of throwing when crypto.subtle is missing.
* fix(artifacts): address PR review feedback for #4864
- Cache SHA-256 digests by (path, mtime_ns, size) so the many small Range
requests a browser issues while scrubbing/paginating a preview do not each
re-hash a potentially huge artifact from scratch (performance).
- Gate inline editing on a real 64-hex revision: hasRevision requires
sha256.length === 64, so the FNV-1a fallback on non-secure origins keeps
preview working but no longer 422s on save (contract).
- Anchor and lowercase the ETag regex and accept the weak W/ prefix gzip
emits, so uppercase hex and longer digests (sha-384/512) can't masquerade
as sha-256.
- Cover the forced-download ETag on the backend and add frontend tests for
weak-ETag parsing and the non-secure-context FNV fallback.
Feedback from reviewer willem-bd on PR #4865.
* style: fix ruff format and prettier issues
- test_artifacts_router.py: collapse two over-split client.get() calls to
satisfy ruff format (line-length 240)
- loader.ts / artifact-file-detail.tsx / loader.test.ts: apply prettier
formatting and restore LF line endings
* style: fix ruff format and prettier issues
- test_artifacts_router.py: collapse two over-split client.get() calls to
satisfy ruff format (line-length 240)
- loader.ts / artifact-file-detail.tsx / loader.test.ts: apply prettier
formatting and restore LF line endings
* style: fix ruff format and prettier issues
- test_artifacts_router.py: collapse two over-split client.get() calls to
satisfy ruff format (line-length 240)
- loader.ts / artifact-file-detail.tsx / loader.test.ts: apply prettier
formatting and restore LF line endings
* style: fix ruff format and prettier issues
- test_artifacts_router.py: collapse two over-split client.get() calls to
satisfy ruff format (line-length 240)
- loader.ts / artifact-file-detail.tsx / loader.test.ts: apply prettier
formatting and restore LF line endings
* style: reformat artifact-file-detail.tsx for prettier with tailwind class ordering
* fix: invalidate SHA-256 cache after artifact edit
Clear the LRU cache after os.replace() so the next preview request
computes the new digest. Edits are rare, so clearing the whole
256-entry cache costs nothing (addressing PR review comment #5).
* fix(artifacts): skip ETag for oversized files + CRLF->LF + cache invalidation (#4865)
* fix(loader): use real empty-content SHA-256 for empty 416 range (#4865)
* test(artifacts): assert oversized artifacts carry no SHA-256 ETag (#4865)
* style(frontend): format long sha256 constant (prettier)
* test(backend): fix oversized-artifact ETag assertions and formatting (ruff)
* test(backend): keep oversized-payload line within ruff 240-col config
* feat(frontend): reopen the skill list after a skill is selected
Selecting a skill closed the composer's skill list for good: `/` no longer
reopened it, so a skill could not be looked up or swapped without deleting
the chip first.
The list now reopens from the editable text beside the chip, and picking an
entry swaps the chip rather than stacking a second activation, since the wire
format carries exactly one leading /skill. Builtin commands are withheld in
that state because they own the whole composer line, and Enter navigates the
list before submitting except while an IME is composing.
The trigger is unchanged: a slash still opens the list only at the start of
the input.
* fix(frontend): keep builtin names reserved in the reopened skill list
Withholding the builtin list from getMatchingSkillSuggestions in chip mode
also disabled the reserved-name filter it drives, so a custom skill named
after a builtin command became selectable there. Nothing rejects such a name
at install time, and submitting the resulting chip runs the command instead
of the skill.
Pass the builtin list as before and drop the builtin entries from the result
instead. The new regression covers both sides of the reservation, and the
reopen test now waits for the list before pressing Enter.
* fix(frontend): hide skills the slash parsers refuse from the picker
The composer picker reserved only the two builtin command names, while both
slash parsers refuse the seven names in the shared contract. A skill named
bootstrap, help, memory, models, new or status was therefore offered, could
be selected into a chip, and submitted — and then activated nothing, because
parse_slash_skill_reference drops the name on the way in. The turn reached
the model as literal text with no skill loaded and no error anywhere.
Reserve the contract names alongside the builtin ones, so the picker cannot
offer what the parsers will not honour.
* fix(frontend): hide stale follow-up chips while a turn is streaming (#3395)
Follow-up suggestion chips are generated when a turn finishes streaming,
but `showFollowups` did not exclude the streaming state. If a user sent a
new message before the previous response finished, the old chips (and the
lone close button) stayed mounted and overlapped the To-dos panel and the
input box.
Gate `showFollowups` on `status !== "streaming"` so stale chips are never
shown while a response is in progress.
* fix(frontend): suppress follow-up suggestions for user-interrupted turns (#3395)
Gating showFollowups on status alone was not enough: stopping a streaming
turn (or sending a new message mid-stream, which also stops it) flips
status back to a non-streaming state and triggers the follow-up generation
effect on that streaming->ready transition, producing chips for a
half-finished, interrupted response.
Track user interruption with a ref set in the stop path, and have the
generation effect skip that transition (and clear/hide any pending chips),
so follow-ups are only generated for turns that finished on their own.
* feat(artifacts): inline editing for text artifacts in the panel
Add a PUT /api/threads/{id}/artifacts/{path} endpoint that atomically
replaces an existing UTF-8 text file under /mnt/user-data/outputs after
verifying its SHA-256 revision. Active runs conflict (409); binary,
symlink, oversized, and non-output paths are rejected.
Frontend: edit/save/discard buttons, draft state with conflict detection,
CodeEditor onChange/onSave, loader SHA-256 from ETag, i18n, beforeunload guard.
Backend: PUT endpoint with thread reservation, atomic temp-file replacement,
sandbox sync for non-mounted providers, rollback on failure, ETag on GET.
Tests: 8 backend + 1 blocking-IO + 3 frontend test files.
* fix(artifacts): scope replacement permissions and release sandboxes
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: show real-time context window usage in chat UI (#3125)
Adds a `context_usage` block to `GET /api/threads/{id}/token-usage`
(token count from the live checkpoint, the thread model's
`context_window`, and a percentage), introduces a new
`ModelConfig.context_window` distinct from the per-call `max_tokens`
output cap, and surfaces the percentage in the chat header — inside
`TokenUsageIndicator` when token-usage tracking is on, or as a
standalone badge when it's off so context capacity stays visible
independent of cost tracking.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat: per-category breakdown for context window usage
Replace the single-number context_usage payload with a Claude-Code-style
breakdown — messages, system prompt, skills, system/MCP tools (active +
deferred), custom agents, memory injection, autocompact buffer, and free
space — and surface it in the chat UI with a segmented progress bar and
per-row table.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(config): document context_window across model examples
Add `context_window` to every example model in config.example.yaml so the
new chat-UI "% context used" indicator works out of the box for whichever
example a user adopts. Each value is the published default at the time of
writing; users are pointed at the official model spec to verify. Bumps
config_version to 11 so `make config-upgrade` flags outdated user configs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* style: ruff format (line-length 240)
No behavior change — collapses two multi-line expressions that fit on
one line under the project's 240-char limit. Picked up by `make format`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* review: address Copilot bot comments on #3183
- token-usage-indicator: switch `{contextPercentage && (...)}` to an
explicit `!= null` check. (The string `"0"` is actually truthy in JS so
the original code wasn't buggy, but the explicit check is clearer.)
- context-usage-breakdown: drop the `useMemo` around segments/totals — the
computation is O(n) over a handful of rows and the previous memo deps
omitted `t.contextUsage.categories`, so the bar's tooltips/aria-labels
could stay in the old language after a locale switch.
- context_usage._split_tools: snapshot MCP names from
`get_cached_mcp_tools()` directly instead of re-reading
`extensions_config.json` after `get_available_tools()` already loaded
it. Removes redundant file I/O on every `/token-usage` poll.
(`get_available_tools()` still emits its own INFO logs — silencing
those is out of scope here.)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* style(frontend): prettier --write context-usage-breakdown
CI's `pnpm format` (prettier --check) caught two lines previously
formatted by hand. Collapses one comma to fit on one line; no behavior
change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(gateway): correct context-usage breakdown + add exact token counting
The context-usage indicator shipped two bugs that silently zeroed whole
breakdown rows (both caught by try/except, so the feature looked alive but
produced wrong numbers):
1. _count_system_prompt passed app_config= to get_deferred_tools_prompt_section,
which only accepts deferred_names -> TypeError swallowed -> system_prompt
row always 0, and used_tokens/percentage undercounted by the full prompt.
Also subtracted the deferred section twice (the rendered prompt already
excluded it). Fix: derive deferred names deterministically and pass them to
apply_prompt_template; drop the redundant subtraction.
2. _split_tools imported a non-existent get_deferred_registry -> ImportError
swallowed -> all four tool-category rows always 0. Fix: classify via the
public is_mcp_tool predicate + tool_search.enabled (mirrors
build_deferred_tool_setup); the MCP tag is set by get_available_tools.
Added token_usage.counting (approximate|exact). 'exact' routes text/schema/
message counting through the model tokenizer (tiktoken cl100k_base) via the
existing memory-module machinery (lazy load + cache + cooldown + CJK-aware
fallback), so CJK-heavy threads stop being undercounted by chars//4.
Regression + e2e tests added; 6621 backend tests pass.
* fix(gateway): harden context usage accounting
* fix(gateway): count promoted MCP tools as active in context usage
Promoted tools (deferred MCP tools the thread has fetched via tool_search)
have their full schema bound on every subsequent turn by
DeferredToolFilterMiddleware, so they consume context like any active tool.
The breakdown previously left them in the reserved *_deferred rows, under-
counting the thread's used_tokens.
Classification now treats a tool as deferred only when tool_search is enabled,
it is MCP-sourced, AND it has not been promoted. The promoted set is read from
the checkpoint's channel_values and scoped by catalog hash — matching the
runtime middleware, so a stale promotion from MCP-config drift cannot inflate
the active count.
The static system prompt still lists all deferred tool names (promotions only
affect schema binding, not the prompt), so _count_system_prompt's deferred
rendering is intentionally left unchanged.
8 new tests cover classification, catalog-hash scoping (match / drift /
compute-failure / malformed), and checkpoint extraction.
* fix(context): address review feedback
* fix(context): count structured message payloads
* fix(context): harden usage accounting
* fix(config): bump schema for context usage fields
* refactor: narrow context usage to core indicator
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
A reasoning model's turn showed its thinking below its answer while
streaming, then flipped to thinking-above-answer once the turn settled.
One message is rendered by two components with opposite ordering rules:
while streaming, an AI message with content and reasoning but no tool
calls yet is deliberately held out of the terminal bubble (#4304) and
rendered by MessageGroup's chain-of-thought panel, which pinned the
trailing reasoning disclosure to the bottom; the settled bubble paints
its <Reasoning> disclosure above the content.
Render the trailing reasoning disclosure before the assistant text that
follows it, and emit a message's reasoning step before its content step
in convertToSteps -- the step list was content-first, so ordering by
step position alone could not fix it. Assistant text emitted before that
reasoning keeps its earlier position.
This also covers two cases the report does not mention: tool-using turns
reversed the same way, and expanding "N more steps" showed a message's
answer above its own thinking.
* fix(frontend): render one workspace-change card per run
The workspace-change card is resolved from (threadId, runId) alone, so
every AI message in a run fetches the identical summary. It was rendered
per AI message.
getMessageGroups() opens a separate terminal assistant group for every AI
message that has content and no tool calls, so a run ends in more than one
bubble whenever the model emits answer text mid-run that never gains a
tool call. Each bubble then painted a byte-identical "Edited N files"
card.
Fold the card onto a single position per run, matching how run duration
already anchors its own run-scoped display after the run's last visible
group. Anchoring keys on group index rather than message id because a
terminal assistant group holds exactly one message whose id may be absent.
Fixes#4555
* docs(frontend): explain the workspace-change anchor's group-type restriction
The helper narrows anchor candidates to terminal assistant groups while
getRunDurationDisplaysByGroupIndex accepts a run's last group of any type.
That asymmetry is load-bearing: run duration is emitted by MessageList
around every group, but the workspace-change card comes from
MessageListItem, which MessageList invokes only for human/assistant
groups. Anchoring a run that ends in an assistant:processing group would
pick a position that never renders and silently drop the card.
Record the rule in both places that could invite a future "unification":
the helper's docstring, and the frontend AGENTS.md paragraph that tells
maintainers where run-scoped displays belong.
The shared right panel is collapsible with collapsedSize="0%", so dragging
the divider past minSize makes the library collapse it to zero without
going through the state that owns the panel. The panel disappears while
that state still reads open, leaving the divider draggable but inert and
the panel's trigger needing two clicks to bring it back.
Mirror a zero-width resize back into the owning state so a drag-collapse
closes the panel the same way its trigger does, and keep recording
non-zero widths there as the size to reopen at.
* feat(lark): sidecar credential broker for sandbox lark-cli (Pattern B)
Removes the plaintext Lark credential mounts (appSecret + OAuth tokens)
from the sandbox container. A long-running broker sidecar owns lark-cli
and the per-user config/data dirs and serves the command surface over
Pod loopback; the sandbox gets only a forwarding shim on PATH, so the
raw credential files never exist in the sandbox filesystem.
- lark_broker.py: stdlib-only loopback broker (argv passthrough with
shell=False, server-injected credential env, bounded I/O) + shim
script constant + install-shim mode.
- docker/lark-cli-broker: init(install-shim) + serve image.
- provisioner: LARK_CLI_BROKER_IMAGE + provision_lark_cli_broker →
shim init container + lark-cli-broker sidecar (config/data mounted
sidecar-only); credentials dropped from the sandbox container;
/api/capabilities reports lark_cli_broker_image. Broker supersedes
the Pattern A init-container binary when both are configured.
- gateway: lark_cli_env_overlay(broker=True) omits config/data env;
sandbox_lark_broker_active() TTL-cached mode resolver; broker added
to sandbox_runtime_mode / readiness and the settings UI.
Opt-in and off by default (empty LARK_CLI_BROKER_IMAGE ⇒ no change).
Closes#4338
* fix(lark): address Pattern B broker review findings (#4501)
Follow-up to the sidecar credential broker addressing the PR #4501 review:
- shim: split the on-PATH lark-cli into a /bin/sh launcher + Python shim body
so broker mode fails loudly (exit 127, actionable message) instead of ENOEXEC
when the sandbox image ships no python3; interpreter pinnable via
DEERFLOW_LARK_BROKER_PYTHON. Launcher bakes in the shim's absolute path since
$0 is the bare command name when run off PATH.
- broker: drop the dead cwd payload field (broker can't see the sandbox FS) and
document the command-surface-only / no-file-IO limitation.
- broker: return a structured 500 JSON on unexpected exec errors so the shim
gets a meaningful message, not an opaque transport failure; set a handler
socket timeout to bound slow/stuck connections.
- broker: add an opt-in DEERFLOW_LARK_BROKER_DENY_SUBCOMMANDS denylist that
refuses secret-dumping subcommands before spawning the binary, forwarded from
the provisioner sidecar.
- gateway: tighten the per-bash-call broker probe timeout (1.5s) and cache
negatives longer (300s) so non-broker remote-provisioner users don't pay a
latency hit; guard the mode cache with a lock; drop the dead
_probe_provisioner_lark_cli_init_image wrapper.
- docs: remove the broken design-doc link from the broker README.
Adds tests for launcher python resolution, cwd omission, denylist enforcement,
500-on-error, hot-path probe timeout + negative caching, and provisioner
denylist-env wiring.
* feat(frontend): allow chat replies during clarification
* fix(frontend): unlock input polish during clarification
Remove hasOpenHumanInputCard from inputPolishDisabled so the polish
button stays available when a clarification card is open, matching
the composer unlock behavior. Clean up the now-unused useMemo and
import.
