Test AI with Langwatch
Evaluate LLMs and AI agents with Langwatch, a platform for founders working with AI technologies, backed by 3.3k+ GitHub stars
3,360 stars328 forksTypeScriptQuality 9/10Updated 7/10/2026100% free ยท open source Langwatch evaluates and tests Large Language Models (LLMs) and AI agents to help founders assess their performance and capabilities.
git clone https://github.com/langwatch/langwatch.git
- โขWhen selecting an LLM for a specific task or project
- โขWhen comparing the performance of different LLMs or AI agents
- โขWhen testing and fine-tuning LLMs or AI agents for a particular use case
- 1Run yarn install to set up the project
- 2Edit the config file to specify the LLMs or AI agents to evaluate
- 3Use the langwatch command to start the evaluation process
- 4Review the results in the output directory
- 5Use the langwatch --help command to explore additional options and configurations
Ready-to-paste prompt langwatch --llm gpt-3 --task text-classification --dataset my-dataset.json
Heads up: Langwatch requires a GitHub personal access token to fetch LLM models, which must be stored in a .env file at the root of the project
Saves to your device
Topics
ai
analytics
datasets
dspy
evaluation
gpt
llm
llm-ops
llmops
low-code
observability
openai
prompt-engineering
What's inside โ free to inspect
No purchase needed
Read the entire source before you build โ unlike paid marketplaces that hide it behind a buy button.