Get fast and compatible LLM inference with openinfer, a Rust-based engine for startup founders, serving models from Qwen3 to Kimi-K2.
517 stars75 forksRustQuality 9/10Updated 7/7/2026100% free ยท open source
What it does
OpenInfer provides fast and compatible large language model (LLM) inference for startup founders, serving models from Qwen3 to Kimi-K2 using a Rust-based engine
Install / run
cargo build --release
When to use it
โขWhen you need to deploy LLM models without relying on PyTorch
โขWhen you require OpenAI-compatible model serving for your startup
โขWhen you prefer a Rust-based solution for performance and compatibility
Quick start
1Clone the OpenInfer repository from GitHub using the command `git clone https://github.com/openinfer-project/openinfer.git`
2Change into the OpenInfer directory with `cd openinfer`
3Build the OpenInfer engine with `cargo build --release`
4Run the OpenInfer server with `target/release/openinfer-server`
5Test the OpenInfer API using a tool like `curl` or a REST client
Ready-to-paste prompt
curl -X POST -H 'Content-Type: application/json' -d '{"prompt": "Write a short story about a character who discovers a hidden world."}' http://localhost:8000/infer
Heads up: Make sure you have Rust and CUDA installed on your system, as OpenInfer relies on these dependencies to function correctly
Saves to your device
Topics
cuda
cuda-kernels
deepseek
gpu
inference
inference-engine
kimi
kimi-k2
kv-cache
llm
llm-inference
llm-serving
model-serving
moe
openai-api
paged-attention
qwen
qwen3
rust
vllm
What's inside โ free to inspect
No purchase needed
Read the entire source before you build โ unlike paid marketplaces that hide it behind a buy button.