scholar-ai

A Go, API-first research system (NotebookLLM-like). Each Project (notebook) holds uploaded Sources (text/PDF/images) plus many chat Threads. The LLM is grounded in project sources via RAG, discovers external papers via arXiv and Exa, shares knowledge across threads via project-wide memories + RAG over thread summaries + a project search tool, and routes chat models through OpenRouter (Cohere or Anthropic models only). A non-interactive CLI drives management/testing; the HTTP API is the integration point for future UIs.

Architecture

cmd/
  api/         HTTP API server (REST + SSE)
  cli/         non-interactive management/testing CLI (cobra)
internal/
  config/      env load + validation + chat-model allow-list
  shared/      core types, ULID ids, error types
  llm/         OpenRouter chat client (go-openai), tool registry, bounded tool loop
  storage/
    blob/      BlobStore iface + local FS + S3 (content-addressed by sha256)
    sqlite/    DB open (+ sqlite_vec.Auto()), goose migrations, repositories
    vector/    VectorStore iface + sqlite-vec impl (per-(project,dim) vec0 tables)
               + optional Qdrant gRPC backend (collection-per-dim, tenant-partitioned)
  ingest/      TextExtractor (ledongthuc/pdf + optional pdftotext), chunker,
               Cohere embed-v4 embedder, ingest pipeline
  search/      arXiv Atom client + Exa client
  memory/      memory store CRUD, thread-summary generator, retrieval/injection
  rag/         hybrid retrieval, grounding prompt, citation assembly
  service/     composition root + orchestration (sources, search, threads, chat)
  httpserver/  chi router, handlers, SSE, multipart, bearer guard
migrations/    embedded SQL migrations (goose)

All metadata and vectors live in a single SQLite file (mattn/go-sqlite3) with the sqlite-vec extension loaded via sqlite3_auto_extension. Vectors are partitioned into one vec0 virtual table per (project_id, dim), so projects can use different embedding dimensions; changing a project's dimension triggers a full reindex.

The metadata backend is selectable via DATABASE_TYPE (sqlite, the default, or postgres for PostgreSQL major version 18+). When DATABASE_TYPE=postgres, the metadata repos (projects, threads, messages, assets, sources, chunks, memories, links, citations) are stored in Postgres. With the default sqlite vector backend, vectors then live in a separate local file DATA_DIR/vectors.db (hybrid model); because they are node-local, running multiple app instances against one Postgres database yields divergent/incomplete vector search (each instance keeps its own index) — use a single instance, or give each instance its own DATA_DIR.

Vector store backends

Vector storage is pluggable via VECTOR_BACKEND (process-global switch):

• sqlite (default): sqlite-vec vec0 tables, one per (project_id, dim). Under the sqlite metadata backend these live inside the app DB; under postgres they live in the separate DATA_DIR/vectors.db. No extra services required.
• qdrant: a remote Qdrant server over gRPC (QDRANT_HOST, QDRANT_PORT=6334, optional QDRANT_API_KEY/QDRANT_USE_TLS). Vectors live in one collection per embedding dimension, named <QDRANT_COLLECTION_PREFIX>_dim_<dim> (default prefix scholar); projects are partitioned by a tenant-indexed project_id payload field. Point ids are deterministic UUIDv5 over project_id + "/" + ref_id, so upsert is idempotent and deletes are project-scoped.

Metadata (chunks, sources, memories) always stays in the configured metadata backend (SQLite or Postgres) regardless of vector backend. The backend switch is a fresh start: there is no automatic vector copy between backends — when moving to Qdrant, run a per-project reindex to repopulate it. Startup fails fast: VECTOR_BACKEND=qdrant health-checks Qdrant and exits on a connection error.

Prerequisites

• Go 1.23+ (built/tested on Go 1.26)
• CGO enabled with a C compiler (gcc/clang): CGO_ENABLED=1 (default). Required because mattn/go-sqlite3 and sqlite-vec are CGO. The first build compiles sqlite-vec from source (slow; cached afterwards).
• Optional: pdftotext (poppler) on PATH for higher-quality academic PDF text extraction (the pure-Go ledongthuc/pdf is the default fallback).
• API keys: OPENROUTER_API_KEY (required), COHERE_API_KEY (required). EXA_API_KEY optional. Set EMBED_BACKEND=fake to run fully offline with deterministic vectors (useful for tests/local experiments; chat still needs a real OpenRouter key).

Configuration

Loaded from .env (optional, never overrides real env) then environment variables (caarlos0/env), validated at startup. Chat models are restricted by an allow-list: must be cohere/* or anthropic/*.

Variable	Default	Description
OPENROUTER_API_KEY	— (required)	OpenRouter API key
OPENROUTER_BASE_URL	https://openrouter.ai/api/v1	OpenRouter base URL
DEFAULT_CHAT_MODEL	anthropic/claude-sonnet-4	default chat model (cohere/* or anthropic/*)
COHERE_API_KEY	— (required)	Cohere API key (embed-v4)
DEFAULT_EMBED_MODEL	embed-v4.0	embedding model
DEFAULT_EMBED_DIM	1024	embedding dimension (256/512/1024/1536)
EMBED_BACKEND	cohere	cohere or fake
EXA_API_KEY	—	Exa API key (optional)
DATA_DIR	./data	SQLite DB + local blobs (+ vectors.db under Postgres)
DATABASE_TYPE	sqlite	sqlite or postgres (requires PG major 18+)
DATABASE_PATH	DATA_DIR/app.db	SQLite metadata file (sqlite only)
DATABASE_HOST	localhost:5432	Postgres <host>[:<port>] or /socket/dir
DATABASE_NAME	— (required for postgres)	Postgres database name
DATABASE_USER	—	Postgres role
DATABASE_PASSWD	—	Postgres password
DATABASE_SCHEMA	public	Postgres schema (auto-created; applied via search_path)
DATABASE_SSLMODE	prefer	Postgres TLS mode (ignored for socket hosts)
BLOB_BACKEND	local	local or s3
S3_*	—	S3 bucket/prefix/region/endpoint/credentials/path-style
VECTOR_BACKEND	sqlite	sqlite (sqlite-vec in the app DB) or qdrant
QDRANT_HOST	—	Qdrant host (required when VECTOR_BACKEND=qdrant)
QDRANT_PORT	6334	Qdrant gRPC port
QDRANT_API_KEY	—	optional Qdrant API key
QDRANT_USE_TLS	false	use TLS for the Qdrant gRPC connection
QDRANT_COLLECTION_PREFIX	scholar	collection-name prefix (<prefix>_dim_<dim>)
HTTP_ADDR	:8080	API listen address
API_BEARER_TOKEN	—	optional bearer guard for all routes

See .env.example.

Build & run

# API server
go build -o scholar-ai-api ./cmd/api
./scholar-ai-api          # serves http://localhost:8080

# CLI
go build -o scholar-ai-cli ./cmd/cli
./scholar-ai-cli health

CGO + a C compiler are required. If your shell disables CGO, prefix builds with CGO_ENABLED=1.

API surface (REST + SSE)

• Projects: POST/GET /projects, GET/PATCH/DELETE /projects/:id, POST /projects/:id/reindex.
• Sources: POST /projects/:id/sources (multipart file or JSON {title,text} / {title,url}), POST /projects/:id/sources/arxiv ({query} search or {arxiv_id,title} fetch+ingest), POST /projects/:id/sources/exa ({query} search or {exa_id,url,title} fetch+ingest), GET /projects/:id/sources, DELETE /sources/:id.
• Threads: POST/GET /projects/:id/threads, GET /threads/:id (messages + citations), POST /threads/:id/messages {content} → SSE stream of content / tool_call / tool_result / done events (final data: [DONE]).
• Memories: GET/POST /projects/:id/memories, PATCH/DELETE /memories/:id.
• Admin: GET /health.

Examples

# create a project
curl -s localhost:8080/projects -d '{"name":"RAG Study","embed_dim":1024}' | jq

# add a text source (ingested + indexed synchronously)
curl -s localhost:8080/projects/<id>/sources \
  -d '{"title":"RAG Doc","text":"Retrieval-augmented generation grounds models..."}' | jq

# create a thread
curl -s localhost:8080/projects/<id>/threads -d '{"title":"T1"}' | jq

# chat (SSE)
curl -N localhost:8080/threads/<id>/messages -d '{"content":"How does RAG reduce hallucination?"}'

# arXiv search then fetch+ingest
curl -s localhost:8080/projects/<id>/sources/arxiv -d '{"query":"cat:cs.AI","max":3}' | jq
curl -s localhost:8080/projects/<id>/sources/arxiv -d '{"arxiv_id":"2301.00001","title":"..."}' | jq

CLI examples

scholar-ai-cli project create --name "RAG Study" --embed-dim 1024
scholar-ai-cli source add --project <id> --title "Doc" --text "..."
scholar-ai-cli source add --project <id> --title "Paper" --file paper.pdf
scholar-ai-cli thread create --project <id> --title "T1"
scholar-ai-cli thread chat <thread-id> "How does RAG reduce hallucination?"
scholar-ai-cli source arxiv --query "cat:cs.AI" --max 3
scholar-ai-cli source arxiv --project <id> --arxiv-id 2301.00001
scholar-ai-cli memory save --project <id> --kind fact --content "..." --importance 8
scholar-ai-cli reindex <project-id>
scholar-ai-cli health

Global flags mirror env vars for local use without a .env: --data-dir, --database-type, --database-path, --embed-backend, --chat-model, --openrouter-key, --cohere-key, --exa-key.

LLM agentic tools

The chat loop (bounded steps) gives the model these tools, wired to the service layer: search_arxiv, fetch_arxiv_paper, search_exa, fetch_exa_contents, save_source, search_project, recall_memory, save_memory, update_memory, delete_memory, get_thread, link_thread. Streaming tool-call argument deltas are accumulated across SSE chunks (go-openai) and re-issued as follow-up requests.

Cross-thread knowledge uses three mechanisms:

1. Project-wide memories (tool-driven CRUD) + per-turn injection of the project profile and top-k relevant memories, plus recall_memory/search_project.
2. RAG over thread summaries: each thread auto-generates a summary, embedded as a thread source kind; get_thread retrieves full transcripts.
3. Project search: unified search_project over sources + thread summaries + memories.

Chunking & ingestion

Recursive text chunker (~800 tokens, ~100 overlap; tokens approximated by chars/4). PDFs chunk page-aware with page locators; images embed whole via embed-v4 multimodal. Pipeline: extract → chunk → embed (Cohere embed-v4, batched, input_type=search_document) → upsert chunks + vectors + asset. Idempotent on content hash; sources carry a status lifecycle (pending → extracting → embedding → ready | error | needs_ocr). PDFs with no extractable text layer are flagged needs_ocr (OCR is out of scope for v1).

arXiv usage & ToS

The arXiv client enforces a ≥3s interval between requests and sends a descriptive User-Agent. Fetched PDFs are deduped by content hash as assets. Keep rate limits in mind when scripting bulk fetches.

Testing

go test ./...                                   # unit + integration (offline)
SCHOLAR_AI_NETWORK_TESTS=1 go test ./internal/search/...   # live arXiv e2e (gated)

# Vector backend conformance against a real Qdrant (gated):
QDRANT_TEST_URL=localhost:6334 go test ./internal/storage/vector/...
SCHOLAR_AI_POSTGRES_DSN="host=/tmp/pg.sock ..." go test ./internal/storage/postgres/... # live PG repo e2e (gated)

Tests use a deterministic FakeEmbedder and in-memory SQLite (:memory: / temp dirs) so the full ingestion → RAG → chat → citation flow runs without network. Recorded HTTP fixtures cover the arXiv Atom/Exa clients. The vector Store contract is covered by a shared conformance suite run against sqlite-vec always and against Qdrant when QDRANT_TEST_URL is set (e.g. docker run -p 6334:6334 qdrant/qdrant). The Postgres repository suite is gated behind SCHOLAR_AI_POSTGRES_DSN and skipped when unset.