nashsu/llm_wiki：可自建的个人知识库工具

# LLM Wiki A personal knowledge base that builds itself. LLM reads your documents, builds a structured wiki, and keeps it current. What is this? • Features • Tech Stack • Installation • Credits • License English | 中文 | 日本語 | 한국어 --- ## Features - **Two-Step Chain-of-Thought Ingest** — LLM analyzes first, then generates wiki pages with source traceability and incremental cache - **Multimodal Image Ingestion** — extract embedded images from PDFs, generate factual captions with a vision LLM, surface them in image-aware search results with lightbox preview and jump-to-source - **Multi-format Document Parsing** — ingest PDF, Office documents, EPUB/MOBI, Org mode, images, media, web clips, and batches of URLs, with built-in, cloud, or local MinerU PDF processing - **Flexible Model Configuration** — configure models per project, route Chat and Ingest independently, and manage custom providers, headers, and streaming output - **Source-grounded Retrieval** — use Read Sources Only mode to answer exclusively from original imported material - **Project Management & Migration** — export and import complete project archives across devices, and rebuild the Wiki index from existing pages - **4-Signal Knowledge Graph** — relevance model with direct links, source overlap, Adamic-Adar, and type affinity - **Louvain Community Detection** — automatic knowledge cluster discovery with cohesion scoring - **Graph Insights** — surprising connections and knowledge gaps with one-click Deep Research - **Vector Semantic Search** — optional embedding-based retrieval via LanceDB, supports any OpenAI-compatible endpoint - **Persistent Ingest Queue** — serial processing with crash recovery, cancel, retry, and progress visualization - **Folder Import** — recursive folder import preserving directory structure, folder context as LLM classification hint - **Source Folder Auto-Watch** — detects external changes in `raw/sources/` and keeps ingest/delete cleanup in sync - **Deep Research** — LLM-optimized search topics, multi-query web search via Tavily, SerpApi, or SearXNG, auto-ingest results into wiki - **Rust Backend Chat Agent** — tool-using chat runtime with wiki/source/graph/web retrieval, workspace file generation, shell approval, cancellation, and streaming tool events - **Agent Skills** — scan and enable local `SKILL.md` folders, select skills with `/skill`, and let the Agent read skill instructions on demand - **Generated Outputs Preview** — Agent-created Markdown, HTML, images, and other workspace files appear as outputs with preview and quick folder access - **Mermaid Diagram Rendering** — render Mermaid code blocks directly in chat and preview, with compact syntax-error cards instead of raw parser output - **Async Review System** — LLM flags items for human judgment, predefined actions, pre-generated search queries - **Chrome Web Clipper** — one-click web page capture with auto-ingest into knowledge base - **Local HTTP API + MCP Server + AI Agent Skill** — built-in `127.0.0.1:19828` JSON API and bundled MCP server for hybrid search, file read, graph traversal, and source rescan; ready-made [agent skill](https://github.com/nashsu/llm_wiki_skill) installs into Claude Code / Codex with one command (`npx skills add …`) ## What is this? LLM Wiki is a cross-platform desktop application that turns your documents into an organized, interlinked knowledge base — automatically. Instead of traditional RAG (retrieve-and-answer from scratch every time), the LLM **incrementally builds and maintains a persistent wiki** from your sources. Knowledge is compiled once and kept current, not re-derived on every query. This project is based on [Karpathy's LLM Wiki pattern](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) — a methodology for building personal knowledge bases using LLMs. llm_wiki is created and maintained by [nash_su](https://x.com/nash_su), who implemented the core ideas as a full desktop application with significant enhancements. ## Credits The foundational methodology comes from **Andrej Karpathy**'s [llm-wiki.md](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f), which describes the pattern of using LLMs to incrementally build and maintain a personal wiki. The original document is an abstract design pattern; this project is a concrete implementation with substantial extensions. ## What We Kept from the Original The core architecture follows Karpathy's design faithfully: - **Three-layer architecture**: Raw Sources (immutable) → Wiki (LLM-generated) → Schema (rules & config) - **Three core operations**: Ingest, Query, Lint - **index.md** as the content catalog and LLM navigation entry point - **log.md** as the chronological operation record with parseable format - **[[wikilink]]** syntax for cross-references - **YAML frontmatter** on every wiki page - **Obsidian compatibility** — the wiki directory works as an Obsidian vault - **Human curates, LLM maintains** — the fundamental role division ## What We Changed & Added ### 1. From CLI to Desktop Application The original is an abstract pattern document designed to be copy-pasted to an LLM agent. We built it into a **full cross-platform desktop application** with: - **Three-column layout**: Knowledge Tree / File Tree (left) + Chat (center) + Preview (right) - **Icon sidebar** for switching between Wiki, Sources, Search, Graph, Lint, Review, Deep Research, Settings - **Custom resizable panels** — drag-to-resize left and right panels with min/max constraints - **Activity panel** — real-time processing status showing file-by-file ingest progress - **All state persisted** — conversations, settings, review items, project config survive restarts - **Scenario templates** — Research, Reading, Personal Growth, Business, General — each pre-configures purpose.md and schema.md ### 2. Purpose.md — The Wiki's Soul The original has Schema (how the wiki works) but no formal place for **why** the wiki exists. We added `purpose.md`: - Defines goals, key questions, research scope, evolving thesis - LLM reads it during every ingest and query for context - LLM can suggest updates based on usage patterns - Different from schema — schema is structural rules, purpose is directional intent ### 3. Two-Step Chain-of-Thought Ingest The original describes a single-step ingest where the LLM reads and writes simultaneously. We split it into **two sequential LLM calls** for significantly better quality: ``` Step 1 (Analysis): LLM reads source → structured analysis - Key entities, concepts, arguments - Connections to existing wiki content - Contradictions & tensions with existing knowledge - Recommendations for wiki structure Step 2 (Generation): LLM takes analysis → generates wiki files - Source summary with frontmatter (type, title, sources[]) - Entity pages, concept pages with cross-references - Updated index.md, log.md, overview.md - Review items for human judgment - Search queries for Deep Research ``` Additional ingest enhancements beyond the original: - **SHA256 incremental cache** — source file content is hashed before ingest; unchanged files are skipped automatically, saving LLM tokens and time - **Persistent ingest queue** — serial processing prevents concurrent LLM calls; queue persisted to disk, survives app restart; failed tasks auto-retry up to 3 times - **Folder import** — recursive folder import preserving directory structure; folder path passed to LLM as classification context (e.g., "papers > energy" helps categorize content) - **Source folder auto-watch** — files added, edited, or deleted in `raw/sources/` outside the app are picked up automatically and reuse the same ingest/delete lifecycle as in-app actions - **Queue visualization** — Activity Panel shows progress bar, pending/processing/failed tasks with cancel and retry buttons - **Auto-embedding** — when vector search is enabled, new pages are automatically embedded after ingest - **Source traceability** — every generated wiki page includes a `sources: []` field in YAML frontmatter, linking back to the raw source files that contributed to it - **overview.md auto-update** — global summary page regenerated on every ingest to reflect the latest state of the wiki - **Guaranteed source summary** — fallback ensures a source summary page is always created, even if the LLM omits it - **Language-aware generation** — LLM responds in the user's configured language (English or Chinese) - **Progressive Sources view** — large source folders render progressively while scrolling, keeping big source collections responsive ### 4. Knowledge Graph with Relevance Model The original mentions `[[wikilinks]]` for cross-references but has no graph analysis. We built a **full knowledge graph visualization and relevance engine**: **4-Signal Relevance Model:** | Signal | Weight | Description | |--------|--------|-------------| | Direct link | ×3.0 | Pages linked via `[[wikilinks]]` | | Source overlap | ×4.0 | Pages sharing the same raw source (via frontmatter `sources[]`) | | Adamic-Adar | ×1.5 | Pages sharing common neighbors (weighted by neighbor degree) | | Type affinity | ×1.0 | Bonus for same page type (entity↔entity, concept↔concept) | **Graph Visualization (sigma.js + graphology + ForceAtlas2):** - Node colors by page type or community, sizes scaled by link count (√ scaling) - Edge thickness and color by relevance weight (green=strong, gray=weak) - Hover interaction: neighbors stay visible, non-neighbors dim, edges highlight with relevance score label - Zoom controls (ZoomIn, ZoomOut, Fit-to-screen) - Position caching prevents layout jumps when data updates - Legend switches between type counts and community info based on coloring mode ### 5. Louvain Community Detection Not in the original. Automatic discovery of knowledge clusters using the **Louvain algorithm** (graphology-communities-louvain): - **Auto-clustering** — discovers which pages naturally group together based on link topology, independent of predefined page types - **Type / Community toggle** — switch between coloring nodes by page type (entity, concept, source...) or by discovered knowledge cluster - **Cohesion scoring** — each community scored by intra-edge density (actual edges / possible edges); low-cohesion clusters ( MinerU is optional. Use MinerU Cloud, an official Local API endpoint, or Local Pipeline mode for complex PDFs. Local modes keep processing on your machine, and extracted images are stored in the project-managed `wiki/media` directory. If MinerU fails, LLM Wiki falls back to the built-in parser. ### 16. File Deletion with Cascade Cleanup The original has no deletion mechanism. We added **intelligent cascade deletion**: - Deleting a source file removes its wiki summary page - **3-method matching** finds related wiki pages: frontmatter `sources[]` field, source summary page name, frontmatter section references - **Shared entity preservation** — entity/concept pages linked to multiple sources only have the deleted source removed from their `sources[]` array, not deleted entirely - **Index cleanup** — removed pages are purged from index.md - **Wikilink cleanup** — dead `[[wikilinks]]` to deleted pages are removed from remaining wiki pages ### 17. Configurable Context Window Not in the original. Users can configure how much context the LLM receives: - **Slider from 4K to 1M tokens** — adapts to different LLM capabilities - **Proportional budget allocation** — larger windows get proportionally more wiki content - **60/20/5/15 split** — wiki pages / chat history / index / system prompt ### 18. Cross-Platform Compatibility The original is platform-agnostic (abstract pattern). We handle concrete cross-platform concerns: - **Path normalization** — unified `normalizePath()` used across 22+ files, backslash → forward slash - **Unicode-safe string handling** — char-based slicing instead of byte-based (prevents crashes on CJK filenames) - **macOS close-to-hide** — close button hides window (app stays running in background), click dock icon to restore, Cmd+Q to quit - **Windows/Linux close confirmation** — confirmation dialog before quitting to prevent accidental data loss - **Tauri v2** — native desktop on macOS, Windows, Linux - **GitHub Actions CI/CD** — automated builds for macOS (ARM + Intel), Windows (.msi), Linux (.deb / .AppImage) ### 19. Other Additions - **i18n** — English + Chinese interface (react-i18next) - **Settings persistence** — LLM provider, API key, model, context size, language saved via Tauri Store - **Obsidian config** — auto-generated `.obsidian/` directory with recommended settings - **Markdown rendering** — GFM tables with borders, proper code blocks, wikilink processing in chat and preview - **Multi-provider LLM support** — OpenAI, Anthropic, Google, Ollama, Custom — each with provider-specific streaming and headers - **Configurable LLM timeout** — adjust request timeouts for slow local models and long-running operations - **Configurable Firecrawl** — optional API key and custom Base URL for hosted or self-hosted services - **Collapsible file sidebar** — collapse Knowledge/Files navigation while preserving its state - **Project maintenance** — ZIP export/import for mi