Bring your own model: run RAG on local or cloud LLMs
DocsGPT is model-agnostic: it runs RAG on local LLMs through Ollama, vLLM, llama.cpp, TGI, SGLang or any OpenAI-compatible engine, or on OpenAI, Anthropic, Google Gemini, Azure OpenAI, DeepSeek, Qwen, GLM and more. Pick the model per chat or per agent, add your own endpoints as Custom Models, and fall back automatically when a model fails.
Last updated:
Model stack — inside your network or cloud
- LLM
localOllama · vLLM · llama.cpp server · TGI · SGLang · LMDeploy · Aphrodite · FriendliAI
cloudOpenAI · Anthropic · Google Gemini · Azure OpenAI · Groq · OpenRouter · Novita · Hugging Face · DeepSeek · Qwen · GLM
- Embeddings
localFastEmbed on ONNX Runtime (granite-311m default)
cloudOpenAI · Azure OpenAI · remote /v1/embeddings
- Vector store
localFAISS · pgvector · Qdrant · Milvus · Elasticsearch
cloudMongoDB Atlas · a managed endpoint for any of the above
- Object storage
localLocal disk · a self-hosted S3-compatible store
cloudAWS S3 · Cloudflare R2 · Backblaze B2 · DigitalOcean Spaces
A fully local stack is a local LLM, local embeddings, a self-hosted vector store and local disk. Object storage is one setting either way: local disk or any S3-compatible endpoint.
Local and self-hosted engines
Eight documented engines, and any other that speaks the OpenAI API, all on your own hardware.
DocsGPT talks to a local inference engine through its OpenAI-compatible endpoint: set the base URL and the model name, and the engine is used like any other provider. Ollama and llama.cpp's server suit a single workstation or a small server; vLLM, TGI, SGLang, LMDeploy and Aphrodite serve larger models on GPUs for many users, and FriendliAI runs behind the same interface.
A fully local stack pairs the engine with local embeddings (FastEmbed on ONNX Runtime), a self-hosted vector store and local disk; docsgpt prefetch-models downloads the embedding and parser models ahead of time for an offline install. That is the combination the air-gapped posture requires. For GPU classes and VRAM per model size, see hardware fit and sizing.
local LLM + local embeddings + self-hosted store + local disk
- Ollama:11434/v1
- llama.cppllama-server · :8000/v1
- vLLM:8000/v1
- TGI:8080/v1
- SGLang:30000/v1
- LMDeploy:23333/v1
- Aphrodite:2242/v1
- FriendliAI:8997/v1
- Any OpenAI-compatible engine/v1/chat/completions
Cloud providers
Every cloud provider sends prompts and retrieved context to the vendor. Cloud and Managed do this by design; on-premises may (hybrid); air-gapped never does.
| Provider id | Where inference runs | External egress | |
|---|---|---|---|
| OpenAIresponses api · reasoning effort | openai | Vendor cloud | Yes |
| Anthropicclaude models | anthropic | Vendor cloud | Yes |
| Google Geminiai studio api key | Vendor cloud | Yes | |
| Azure OpenAIazure deployment name · api version | openai + Azure base URL | Vendor cloud | Yes |
| Groq | groq | Vendor cloud | Yes |
| OpenRouteraggregator | openrouter | Vendor cloud | Yes |
| Novitaaggregator | novita | Vendor cloud | Yes |
| Hugging Faceinference api | huggingface | Vendor cloud | Yes |
| DeepSeek · Qwen · GLMin the model catalog | their own keys | Vendor cloud | Yes |
| Any OpenAI-compatible endpointserver-wide, or per user as a Custom Model | openai_compatible | Provider or your own | Depends on where it runs |
| DocsGPT Public APIfree; for evaluation | docsgpt | Arc53 | Yes |
The in-app catalog lists ready-to-pick models for each provider once its key is set.
Pick a model, add your own, fall back automatically
Change the model behind a chat or an agent without a restart.
People choose a model in the chat, and an agent can offer several models with a default. In Settings › Custom Models anyone can add an OpenAI-compatible endpoint, such as Mistral, Together or a hosted vLLM, with their own key: test the connection, choose Chat Completions or the Responses API, set the reasoning effort and declare tools, structured output and image support. The key is stored encrypted, and only public endpoints are accepted.
Fallback runs in two layers. When the model answering fails, the agent's other models are tried in order, then the instance-wide fallback model. Answers through the OpenAI-compatible API report which model answered and whether it was a fallback.
In workflows each AI step can use a different model, so a small model can classify and a larger one can write.
model picker · agent models · custom models · fallback · per-step model
Embeddings and vector stores
Local embeddings on ONNX Runtime; six vector stores, and pgvector unlocks hybrid search and GraphRAG.
Embeddings run locally through FastEmbed on ONNX Runtime. A new installation is pinned to ibm-granite/granite-embedding-311m-multilingual-r2: multilingual, with a 32k-token context and 768 dimensions. An installation that already has an index stays on the model that index was built with, all-mpnet-base-v2, until you re-embed; DocsGPT prints the migration steps. OpenAI and Azure OpenAI embeddings and any remote /v1/embeddings endpoint work too.
Changing the embedding model means rebuilding the index. Queries and stored vectors have to come from the same model, and because granite-311m and all-mpnet-base-v2 are both 768-dimensional nothing fails loudly. Retrieval just gets worse. Set EMBEDDINGS_NAME, then run the re-embed script; DocsGPT reports any source whose vectors came from a different model.
FAISS is the default vector store and needs no server. pgvector turns your PostgreSQL into the store and is required for hybrid BM25 + vector search and GraphRAG. Qdrant, Milvus, Elasticsearch and MongoDB Atlas are supported as vector-only stores.
fastembed · onnx runtime · ibm-granite/granite-embedding-311m-multilingual-r2 · reembed after a swap
| Role | Hybrid BM25 + vector | GraphRAG | |
|---|---|---|---|
| FAISS | Default; file-based, no server | No | No |
| pgvector | PostgreSQL extension; required for hybrid search and GraphRAG | Yes | Yes |
| Qdrant | Alternative vector store | No | No |
| Milvus | Alternative vector store | No | No |
| Elasticsearch | Alternative vector store | No | No |
| MongoDB Atlas | Alternative vector store, with Atlas Vector Search | No | No |
Hybrid search and GraphRAG need pgvector. Every other store is vector-only: the base store returns no keyword matches, so hybrid retrieval degrades to vector search.
How it runs privately
Model choice decides the posture.
| Cloud | Managed | On-premises | Air-gapped | |
|---|---|---|---|---|
| Who operates | Arc53, multi-tenant | Arc53, dedicated instance | Customer, with Arc53 support | Customer, with Arc53 support |
| LLM | Cloud provider via Arc53 | Cloud provider, local engine or your own key | Local engine, or your cloud LLM (hybrid) | Local engine only |
| External network | Yes | Yes | Optional | None |
Hybrid, meaning an on-premises platform calling a cloud LLM, is on-premises, not air-gapped. Air-gapped means zero external network.
The platform is the same in every posture; what changes is the model layer. Air-gapped means a local engine, local embeddings and a self-hosted vector store with no external network. On-premises may add a cloud LLM (hybrid). Cloud and Managed use a provider by default, or your own key.
Compare all four on deployment options; model choice does not change the price. Test a candidate model on your own cases before switching with benchmarks and evals.
Frequently asked questions
Can I run the LLM locally?
- Yes. Point DocsGPT at Ollama, llama.cpp, vLLM, TGI, SGLang, LMDeploy, Aphrodite, FriendliAI or any OpenAI-compatible engine running on your own hardware, and pair it with local embeddings and a self-hosted vector store for a fully local stack.
Which local LLM engines work with RAG in DocsGPT?
- Any engine that serves an OpenAI-compatible API. Ollama, llama.cpp's server, vLLM, Text Generation Inference (TGI), SGLang, LMDeploy, Aphrodite and FriendliAI are documented with their default ports; set the base URL and the model name and DocsGPT treats the engine like any other provider.
Can I switch models without downtime?
- Yes. People pick a model in the chat, an agent can offer several models with a default, and Custom Models adds new endpoints from the settings without a restart. If a model fails, the agent's other models are tried in order, then the instance-wide fallback model.
Can users bring their own model key?
- Yes. In Settings › Custom Models anyone can add an OpenAI-compatible endpoint with their own API key, test the connection and set its capabilities. The key is stored encrypted, the model is private to that user, and only public endpoints are accepted, so nobody can point the platform at an internal address.
Which vector databases are supported?
- Six: FAISS (the default), pgvector, Qdrant, Milvus, Elasticsearch and MongoDB Atlas. Hybrid BM25 + vector search and GraphRAG require pgvector; the others are vector-only.
Do I have to re-index when I change the embedding model?
- Yes. Stored vectors and queries must come from the same embedding model, so an existing index has to be rebuilt after a change. The catch is that it does not fail loudly: the current default and the legacy model are both 768-dimensional, so a swap without a re-embed silently returns worse answers. Set the new model, run the re-embed script, and DocsGPT will name any source still holding vectors from another model.
See DocsGPT on your documents
A 30-minute demo on your model of choice, or self-host with Ollama in five minutes.