Show HN: Janus – Go binary that runs GGUF models via Vulkan on AMD/Intel/Nvidia

Janus — Local LLM Server & OpenAI-Compatible API Janus is a single Go binary that runs .gguf models on your machine (GPU or CPU) and exposes an OpenAI-compatible API. No Python, no Docker, no Ollama required. Use it your way: call it from the command line (curl, PowerShell, scripts), wire it into Cursor / Cline / any OpenAI client — same local models, whatever workflow fits you. What you get Local inference — llama.cpp via Vulkan (AMD / Intel / NVIDIA) or CPU fallback OpenAI-compatible API — /v1/chat/completions, /v1/models Hot-swap models — change .gguf without restarting Thinking model support — reasoning split into reasoning_content Chat template auto-detection — uses the template from GGUF metadata Zero dependencies — one .exe on Windows, no Python, no Docker Requirements Disk: plan for the model size (often 2–8 GB per model) plus ~50 MB for Janus + llama.dll. Quick start (Windows) 1. Clone and build git clone https://github.com/Vibra-Ingenn/Janus.git cd Janus .\build.ps1 build.ps1 downloads pre-built llama.cpp Vulkan DLLs and compiles dist\janus.exe. 2. Download a model Put a .gguf file in the models\ folder. Use the included downloader: go build -o dist\modelget.exe .\cmd\modelget .\dist\modelget.exe -repo meta-llama/Llama-3.2-3B-Instruct -file Llama-3.2-3B-Instruct-Q8_0.gguf -out .\models\ Or download any GGUF from Hugging Face. 3. Configure copy .env.example .env Edit .env: INFERENCE_BACKEND=vulkan JANUS_MODEL_PATH=./models/Llama-3.2-3B-Instruct-Q8_0.gguf JANUS_MAX_TOKENS=4096 4. Run .\dist\janus.exe Opens http://127.0.0.1:8990 in your browser. 5. Verify curl http://127.0.0.1:8990/health Quick start (Linux / macOS) git clone https://github.com/Vibra-Ingenn/Janus.git cd Janus go mod tidy go build -o dist/janus ./cmd/janus cp .env.example .env # edit .env — set JANUS_MODEL_PATH and INFERENCE_BACKEND=cpu if no Vulkan ./dist/janus On Linux you need libllama.so next to the binary or on LD_LIBRARY_PATH. OpenAI-compatible API curl http://127.0.0.1:8990/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "local", "messages": [{"role": "user", "content": "Hello!"}] }' Base URL: http://127.0.0.1:8990/v1 Endpoints Streaming curl http://127.0.0.1:8990/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"local","stream":true,"messages":[{"role":"user","content":"Tell me a joke"}]}' Connect to Cursor / Cline / other clients Base URL: http://127.0.0.1:8990/v1 API Key: (leave blank) Pitfalls (learned the hard way) Project layout cmd/janus/ Main server (OpenAI-compatible API) cmd/modelget/ Hugging Face model downloader internal/engine/ llama.cpp Vulkan/CPU backend internal/bridge/ DLL loader and FFI bindings internal/singleton/ Single-instance guard models/ Put .gguf files here (not committed) dist/ janus.exe + llama.dll after build Development Get-Process -Name "janus" -ErrorAction SilentlyContinue | Stop-Process -Force go test ./... go build -o dist\janus.exe .\cmd\janus .\dist\janus.exe Or just .\run.ps1. For a full build including llama DLLs, use .\build.ps1. Contributions welcome — see CONTRIBUTING.md. License MIT — see LICENSE.