RAGOps — Production-Style RAG Platform
Personal Applied-AI / Retrieval Engineering Project
Problem
A RAG system is difficult to improve when ingestion, retrieval quality, latency, citations, cache behavior, and failures are hidden behind one answer endpoint. The goal was to build an operations-oriented platform where those components could be inspected, measured, and reproduced.
Constraints
- Keep ingestion asynchronous and observable without blocking the API
- Compare sparse, dense, fused, and reranked retrieval on the same held-out query set
- Prevent training leakage between SciFact's training and test splits
- Retain dataset, model, run, checksum, and per-query evidence for reproducibility
- Separate local deterministic-provider load evidence from claims about real production traffic
Approach
Built a FastAPI service backed by PostgreSQL/pgvector and a Redis worker for asynchronous ingestion. Retrieval combines BM25 and all-MiniLM-L6-v2 dense search through reciprocal rank fusion, with optional cross-encoder reranking. Prometheus and Grafana expose service behavior while a React console makes ingestion, retrieval, citations, caches, and failures inspectable. The evaluation pipeline versions SciFact data, verifies checksums, records model provenance and run configuration, emits per-query traces, and enforces quality gates in CI.
Architecture
Metrics
| Metric | Baseline | Achieved |
|---|---|---|
| Held-out retrieval quality | BM25: 0.7740 Recall@10 | RRF: 0.8059 Recall@10 / 0.6816 nDCG@10 |
| Evaluation scope | Versioned BEIR SciFact | 5,183 documents / 300 held-out queries |
| Local-provider load test | 30 s / concurrency 8 | 9,234 successes; 24.56 ms p50 / 31.75 ms p95 |
| Cross-encoder result | Train split only; zero test overlap | Negative result: slower and lower held-out quality than RRF |
Product Impact
RRF was the strongest tested configuration. The small train-split-only cross-encoder was much slower and reduced held-out quality, so the project retains it as a negative result instead of implying that reranking helped. The latency figures come from a 30-second, concurrency-8 local Docker Compose workload with deterministic local providers; they are reproducible engineering evidence, not a production SLO or externally served traffic.
Tech Stack
Links
Evidence is pinned to commit e63df6444a8e411c7b187027dfa230c0d99f0546. Production-style describes the architecture and operational discipline, not an enterprise production deployment.