Opik by Comet

Opik by Comet is an open-source platform designed for debugging, evaluating, and monitoring Large Language Model (LLM) applications, Retrieval Augmented Generation (RAG) systems, and agentic workflows. It provides comprehensive tools for logging traces and spans, defining and computing evaluation metrics, scoring LLM outputs, and comparing performance across different application versions. The platform also includes automated prompt and agent optimization capabilities, utilizing various optimizers like Few-shot Bayesian, MIPRO, evolutionary, and LLM-powered MetaPrompt. Beyond evaluation, Opik focuses on maximizing trust and safety with built-in guardrails that screen user inputs and LLM outputs to prevent unwanted content, detect and redact PII, and manage off-topic discussions. It supports end-to-end LLM observability, allowing users to log traces during both development and production. Developers can confidently test their LLM pipelines within CI/CD using LLM unit tests built on PyTest and monitor production data to identify issues and generate datasets for new iterations. Opik is built for developers and integrates with popular LLM frameworks and services like OpenAI, LangChain, and LlamaIndex.