Yusuf Gürdoğan 23bd76046a
feat(community): add Sofya web search provider (#5239)
* feat(community): add Sofya web search provider

Add a community provider backed by Sofya (https://sofya.co). Its search
endpoint returns the content of the result pages, not only their snippets,
and its fetch endpoint returns a page as markdown. Both are plain JSON over
HTTP, so this needs no extra Python package (uses httpx, already a
dependency).

Changes:
- backend/packages/harness/deerflow/community/sofya/__init__.py
- backend/packages/harness/deerflow/community/sofya/tools.py
  Implements web_search_tool and web_fetch_tool using httpx.
  API key is read from the config.yaml `api_key` field or the SOFYA_API_KEY
  env var. Follows the same interface and output shape as the existing
  ddg_search and serper providers, including the max_results parameter with
  config override and the structured "No results found" error.
- backend/tests/test_sofya_tools.py
  Unit tests covering API key resolution, config overrides, result mapping,
  time range, HTTP errors, empty results, and fetch failures.
- config.example.yaml: add commented-out Sofya web_search and web_fetch
  examples alongside the other providers
- .env.example: add SOFYA_API_KEY placeholder
- backend/docs/CONFIGURATION.md: list Sofya under web_search, web_fetch and
  the environment variables

* fix(sofya): honor caller max_results, validate search_depth, join time_range contract test

- Caller-supplied max_results now wins; config is used only when the
  argument is omitted, matching GroundRoute.
- search_depth is clamped to basic/snippets; an unsupported value logs a
  warning and falls back to basic.
- Sofya added to the shared time_range schema contract test.

* fix(sofya): cap per-result content so a search stays inline

An unbounded search payload (up to 20 read pages) crossed the tool output
budget middleware's externalize_min_chars threshold, which replaces the
result list with a file reference. Cap each result's content at
contents_max_characters (default 2000, 0 disables), matching Exa's config
key. Five capped results stay under the 12000 char threshold.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* fix(sofya): list Sofya in the recency contract, coerce non-string content

_clip subscripted its input, so a non-string content or description from
the API raised TypeError instead of degrading. Coerce to text first, the
way _sofya_post and _response_results guard the shapes around it. Also add
Sofya to the Web Search Recency section in backend/AGENTS.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* fix(sofya): coerce web_fetch content, list sofya in the tools guide, add changelog

web_fetch sliced its content the same way web_search did before the last
push: a truthy non-string from the API passed the falsiness guard and then
raised TypeError. Reuse _clip, keeping the `or ""` so empty content still
reports "No content found".

Also add sofya to the community provider inventory in
packages/harness/deerflow/tools/AGENTS.md and an [Unreleased] changelog entry.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

* docs(zh): add the missing InfoQuest and Firecrawl web_fetch tabs

The ZH web_fetch tab list named five providers where EN names seven. Both
tabs mirror their EN counterparts, so the two locales list the same
web_fetch providers again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016TZhyPNCX2GYyBPkvTgJV5

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-09-08 10:50:03 +08:00
..
2026-01-14 09:58:53 +08:00
2026-02-10 22:07:25 +08:00

DeerFlow Frontend

Like the original DeerFlow 1.0, we would love to give the community a minimalistic and easy-to-use web interface with a more modern and flexible architecture.

Tech Stack

Quick Start

Prerequisites

  • Node.js 22+
  • pnpm 10.26.2+

Installation

# Install dependencies
pnpm install

# Copy environment variables
cp .env.example .env
# Edit .env with your configuration

Development

# Start development server
pnpm dev

# The app will be available at http://localhost:3000

Build & Test

# Type check
pnpm typecheck

# Check formatting
pnpm format

# Apply formatting
pnpm format:write

# Lint
pnpm lint

# Run unit tests
pnpm test

# One-time setup: install Playwright Chromium browser
pnpm exec playwright install chromium

# Run E2E tests (builds and starts production server automatically)
pnpm test:e2e

# Build for production
pnpm build

# Start production server
pnpm start

Site Map

├── /                    # Landing page
├── /chats               # Chat list
├── /chats/new           # New chat page
└── /chats/[thread_id]   # A specific chat page

Configuration

Environment Variables

Key environment variables (see .env.example for full list):

# Backend API URL (optional, uses local Next.js/nginx proxy by default)
NEXT_PUBLIC_BACKEND_BASE_URL="http://localhost:8001"
# LangGraph-compatible API URL (optional, uses local Next.js/nginx proxy by default)
NEXT_PUBLIC_LANGGRAPH_BASE_URL="http://localhost:8001/api"

Project Structure

tests/
├── e2e/                    # E2E tests (Playwright, Chromium, mocked backend)
└── unit/                   # Unit tests (mirrors src/ layout)
src/
├── app/                    # Next.js App Router pages
│   ├── api/                # API routes
│   ├── showcase/           # Allowlisted public read-only demos
│   ├── workspace/          # Main workspace pages
│   └── mock/               # Mock/demo pages
├── components/             # React components
│   ├── ui/                 # Reusable UI components
│   ├── workspace/          # Workspace-specific components
│   ├── landing/            # Landing page components
│   └── ai-elements/        # AI-related UI elements
├── core/                   # Core business logic
│   ├── api/                # API client & data fetching
│   ├── artifacts/          # Artifact management
│   ├── config/              # App configuration
│   ├── i18n/               # Internationalization
│   ├── mcp/                # MCP integration
│   ├── messages/           # Message handling
│   ├── models/             # Data models & types
│   ├── settings/           # User settings
│   ├── skills/             # Skills system
│   ├── threads/            # Thread management
│   ├── todos/              # Todo system
│   └── utils/              # Utility functions
├── hooks/                  # Custom React hooks
├── lib/                    # Shared libraries & utilities
├── server/                 # Server-side code
│   └── better-auth/        # Authentication setup and session helpers
└── styles/                 # Global styles

Scripts

Command Description
pnpm dev Start development server with Webpack
pnpm build Build for production
pnpm start Start production server
pnpm test Run unit tests with Rstest
pnpm test:e2e Run E2E tests with Playwright
pnpm format Check formatting with Prettier
pnpm format:write Apply formatting with Prettier
pnpm lint Run ESLint
pnpm lint:fix Fix ESLint issues
pnpm typecheck Run TypeScript type checking
pnpm check Run both lint and typecheck

Development Notes

  • Uses pnpm workspaces (see packageManager in package.json)
  • Webpack is the default development bundler until the upstream Turbopack PostCSS worker leak is fixed in a stable Next.js release (#5132). Set DEER_FLOW_DEV_BUNDLER=turbo to opt in to Turbopack for local diagnosis, or DEER_FLOW_DEV_BUNDLER=webpack to select Webpack explicitly. Reconsider the default after the stable fix is verified on macOS arm64 and Linux.
  • Environment validation can be skipped with SKIP_ENV_VALIDATION=1 (useful for Docker)
  • Backend API URLs are optional; nginx proxy is used by default in development

License

MIT License. See LICENSE for details.