Best Open-Source LLMOps & Evaluation | open-source.storeAll categoriesLLMOps & Evaluation
Evaluation, monitoring, prompt management, and observability for LLM-based applications.
20 tools found 
Mem0
Mem0 provides self-hosted long-term memory for AI agents, replacing manual vector pipelines with automated fact extraction and cross-session persistence.
Apache-2.0LLMOps & Evaluation

LiteLLM
LiteLLM is a self-hosted AI gateway replacing OpenRouter with a unified OpenAI-compatible API, spend tracking, and private VPC deployment.
LLMOps & Evaluation

Langfuse
Langfuse is an open-source LLM observability platform and LangSmith alternative that delivers fast trace ingestion and self-hosted ClickHouse analytics.
MITLLMOps & Evaluation

Supermemory
Supermemory provides a self-hosted AI memory engine that replaces manual vector search and LangChain pipelines with automated user fact extraction.
MITLLMOps & Evaluation

FastGPT
FastGPT is an open-source AI agent and RAG platform with visual workflow orchestration, hybrid retrieval, and an OpenAI-compatible API for developers.
LLMOps & Evaluation

DeepEval
DeepEval replaces commercial LLM evaluation dashboards with an open-source Python framework that runs unit tests, G-Eval metrics, and traces inside CI/CD.
19.3kLinuxmacOSWindowstoday Apache-2.0LLMOps & Evaluation
Go & gRPC engineer · founder of EasyP
PromotedGo, gRPC and Protobuf done right, LLM agents in production, and the hard parts of shipping a first product.
GogRPCProtobufLLM agents

Portkey AI Gateway
Portkey AI Gateway replaces proprietary routing proxies with an open-source, self-hosted control plane for over 1,600 models and automated guardrails.
MITLLMOps & Evaluation

Phoenix
Phoenix is an open-source alternative to Arize AI and LangSmith providing self-hosted OpenTelemetry LLM tracing and evaluation without cloud data egress.
12.1kLinuxmacOSWindowstoday Apache-2.0LLMOps & Evaluation

Promptflow
Promptflow is an open-source alternative to LangSmith for building, tracing, and evaluating LLM pipelines directly within private infrastructure.
11.3kLinuxmacOSWindows3d ago MITLLMOps & Evaluation

Bifrost
Bifrost is a self-hosted Go AI gateway replacing hosted proxies with private multi-provider routing, virtual keys, and sub-100-microsecond overhead.
Apache-2.0LLMOps & Evaluation

Activeloop
Activeloop Deep Lake is an open-source multimodal data lake for deep learning and vector retrieval, replacing Pinecone with unified object storage.
Apache-2.0LLMOps & Evaluation

OpenLLMetry
OpenLLMetry replaces proprietary APM agents with open-source OpenTelemetry instrumentation for LLMs, routing traces and token metrics to any collector.
Apache-2.0LLMOps & Evaluation
Apache-2.0LLMOps & Evaluation
Web
today
Apache-2.0LLMOps & Evaluation