RAG Knowledge Assistant
A retrieval-augmented assistant that answers questions grounded in private documents - the engine behind this site's hero chatbot.
RAG
Grounded answers
Always works
Cold-start
Streaming
Latency
Overview
A retrieval-augmented generation (RAG) assistant that grounds LLM answers in a curated knowledge base - the same pattern powering the AI chatbot in this site's hero.
The business problem
LLMs hallucinate when asked about private, specific information. Recruiters and teams want accurate answers grounded in real source material, not generic guesses.
The solution
I built a RAG pipeline: content is chunked and embedded, semantically retrieved per query, and passed as grounded context to the model, which streams a cited, on-topic answer - with a graceful canned-answer fallback when no API key is configured.
Architecture
- Knowledge base chunked into semantically coherent passages.
- Embeddings generated and cached for fast cosine similarity retrieval.
- Top-k retrieval injected as grounded context into the prompt.
- Streaming responses via the Vercel AI SDK.
- Deterministic fallback answers when the model is unavailable.
Engineering decisions
- In-process embedding cache to avoid recompute and cost.
- Strict system prompt that restricts answers to retrieved context.
- Fallback layer so the feature always works on a fresh deploy.
Performance optimizations
- Cached embeddings for sub-100ms retrieval.
- Token-aware context windows to control cost.
- Streaming UX for instant perceived response.
Challenges
- Chunking content for high-precision retrieval.
- Keeping answers grounded and refusing out-of-scope questions.
- Graceful degradation without an API key or on rate limits.
Lessons learned
- Retrieval quality matters more than model size for grounded Q&A.
- Always design AI features to degrade gracefully.