An enterprise-grade, self-hosted benchmarking platform engineered for multi-model parallel inference, real-time WebSocket token streaming, non-blocking PyTorch BERTScore scoring, tripartite RAG evaluation, sentence-level NLI hallucination detection, and native FastMCP agent integration.
Engineering teams deploying LLMs to production lack rigorous, automated benchmarking infrastructure to compare frontier models, track prompt regressions, and detect hallucinations before shipping. Manual qualitative testing is subjective, unscalable, and slow, while naive single-prompt evaluation fails to measure critical production failure modes such as semantic drift, context unfaithfulness in RAG pipelines, latency spikes, and cost-per-token efficiency.
A self-hosted, full-stack LLM Evaluation & Benchmarking Platform. It orchestrates parallel inference across frontier models (Gemini 3.5/3.8 Flash, Groq, Qwen, Ollama) via LiteLLM with real-time WebSocket token dispatching to Next.js 15. The evaluation core executes asynchronous PyTorch BERTScore and ROUGE-L calculations, tripartite RAG scoring, sentence-level NLI hallucination detection, Gemini G-Eval rubrics, distributed Celery batch processing, and a native FastMCP server exposing tools directly to Cursor IDE and Claude Desktop.
Streams responses concurrently from multiple frontier models (Gemini, Groq, Ollama) using LiteLLM with sub-millisecond WebSocket streaming over a Redis pub/sub backbone.
Computes BERTScore F1 (bert-base-uncased / all-MiniLM) in dedicated async thread pools alongside ROUGE-L, latency tracking, token counts, and cost estimation.
Automatically evaluates retrieval quality and generation fidelity across three dimensions: Context Relevance, Faithfulness (via NLI grounding), and Answer Relevance.
Decomposes generated answers into individual claims, verifies each claim against source context for contradictions or ungrounded facts, and applies penalizing winner weights.
Exposes 5 standardized Model Context Protocol tools over stdio and sse transports, enabling Cursor IDE and Claude Desktop agents to run evaluations and query model recommendations.
Automated semantic grading powered by Gemini 3.5 Flash across 9 pre-configured domain rubrics (Code, Summarization, QA, Data Extraction, RAG) with custom weight balancing.
backend/services/eval_runner.py — Multi-model parallel inference orchestrator with LiteLLM gatewaybackend/services/scoring.py — PyTorch BERTScore F1, ROUGE-L, latency, and token cost estimatorsbackend/services/rag_eval.py — Tripartite RAG metrics (Context Relevance, Faithfulness, Answer Relevance)backend/services/hallucination.py — Sentence-level NLI contradiction & claim ungroundedness scorerbackend/services/llm_judge.py — Gemini G-Eval rubric evaluator with weighted multi-criteria gradingmcp/eval_server.py — FastMCP server exposing 5 tools over stdio/sse for Cursor & Claude Desktopfrontend/app/ — Next.js 15 App Router dashboard with React 19, Turbopack, and Tailwind CSSbackend/tasks/eval_tasks.py — Celery distributed worker queues (eval_batch) for dataset benchmarkingbackend/core/database.py — Async PostgreSQL engine (SQLAlchemy 2.0 + AsyncPG) with Alembicbackend/services/visualizer.py — Multi-dimensional projection (Semantic MDS) and similarity matrixbackend/services/report_service.py — Jinja2 and WeasyPrint print-styled executive PDF/HTML report exportbackend/tests/ — Comprehensive test suite with 93%+ coverage across 120 automated testsPython 3.12, Pydantic v2, Uvicorn, SlowAPI rate limiting, unified gateway with retry backoff across frontier models.
App Router, React 19, Tailwind CSS, Recharts, plus Model Context Protocol tools over stdio & SSE for Cursor IDE.
Asynchronous BERTScore F1, ROUGE-L, sentence-level NLI claim verification, and Gemini G-Eval domain rubrics.
PostgreSQL 16 with AsyncPG & SQLAlchemy 2.0, Redis 7 Pub/Sub & caching, and Celery distributed batch queues.