Local LLM Setup

RTILA X can run AI models 100% locally on your own computer. No internet connection is needed after the model is downloaded. No data ever leaves your machine. Your scraping targets, your project configurations, your extracted data — everything stays private.

Changed in v8.3.2: The Local LLM provider has been completely redesigned. The previous finetuned models (RTILA Lite 1.5 Assistant) have been removed in favor of supporting any GGUF model from Hugging Face and any OpenAI-compatible API endpoint. See Release Notes for details.

---

Two Modes

The Local LLM provider supports two distinct modes. Choose the one that fits your setup:

Mode	Description	Best For
GGUF Model (Hugging Face)	Downloads and runs a GGUF model via the bundled llama.cpp server. RTILA handles everything automatically.	Maximum privacy, offline use, simplest setup
Custom API Endpoint	Connects to any OpenAI-compatible API server you already have running (Ollama, LM Studio, vLLM, etc.)	Users who already run a local AI server and want to reuse it

To switch between modes:

1. Go to Preferences → AI Assistant tab.
2. Set AI Provider to Local LLM.
3. Select your desired mode from the mode toggle.

---

Mode 1: GGUF Model (Hugging Face)

This is the recommended mode for most users. RTILA X bundles a high-performance llama.cpp server and handles model downloading, loading, and inference automatically.

Model Identifier Format

To use a model, you need its Hugging Face identifier in the format:

user/repo/file.gguf

This identifier has exactly three parts separated by exactly two forward slashes:

Part	Description	Example
user	The Hugging Face user or organization name	bartowski
repo	The repository name	Qwen2.5-7B-Instruct-GGUF
file.gguf	The specific GGUF file (must end in .gguf)	Qwen2.5-7B-Instruct-Q4_K_M.gguf

Validation Rules

RTILA X validates the model identifier before attempting to download:

Rule	Valid Example	Invalid Example	Reason
Must have exactly 2 slashes	bartowski/Qwen2.5-7B-Instruct-GGUF/Qwen2.5-7B-Instruct-Q4_K_M.gguf	Qwen2.5-7B-Instruct-Q4_K_M.gguf	Missing user/repo
Must end in .gguf	...Q4_K_M.gguf	bartowski/Qwen2.5-7B-Instruct-GGUF	Missing filename
All three parts must be non-empty	bartowski/repo/file.gguf	bartowski//file.gguf	Empty repo part

If the identifier is invalid, you’ll see a validation hint below the input field: “Expected format: user/repo/file.gguf”

Recommended Models

Here are tested and recommended models for use with RTILA X:

Model	Hugging Face Identifier	VRAM Needed	Best For
Qwen 2.5 7B Instruct Q4_K_M	bartowski/Qwen2.5-7B-Instruct-GGUF/Qwen2.5-7B-Instruct-Q4_K_M.gguf	~5 GB	General automation tasks (recommended default)
Llama 3.1 8B Instruct Q4_K_M	bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf	~5.5 GB	Balanced quality and speed
Mistral 7B Instruct v0.3 Q4_K_M	bartowski/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf	~4.5 GB	Fast responses, lower resource usage
Qwen 2.5 14B Instruct Q4_K_M	bartowski/Qwen2.5-14B-Instruct-GGUF/Qwen2.5-14B-Instruct-Q4_K_M.gguf	~9 GB	Complex multi-step tasks
DeepSeek R1 Distill Qwen 7B Q4_K_M	bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF/DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf	~5 GB	Reasoning-heavy tasks
Phi-3.5 Mini Instruct Q4_K_M	bartowski/Phi-3.5-mini-instruct-GGUF/Phi-3.5-mini-instruct-Q4_K_M.gguf	~2.5 GB	Very low-resource machines

Tip: Browse all available GGUF models at huggingface.co/models?library=gguf. Any GGUF model will work — the table above is just a starting point.

Configuration Steps

1. Go to Preferences → AI Assistant tab.
2. Set AI Provider to Local LLM.
3. Select GGUF Model (Hugging Face) mode.
4. Enter the model identifier in the format user/repo/file.gguf.
5. The model is downloaded automatically on first use (see Model Download Process below).

---

Mode 2: Custom API Endpoint

If you already run a local AI server (Ollama, LM Studio, vLLM, etc.), you can connect RTILA X to it directly without downloading a separate model.

Supported Servers

Server	Default Base URL	API Key Needed?	Notes
Ollama	http://localhost:11434/v1	No	Most popular local AI runner
LM Studio	http://localhost:1234/v1	No	GUI-based model manager
vLLM	http://localhost:8000/v1	No	High-performance inference server
text-generation-webui	http://localhost:5000/v1	No	Flexible web-based interface
llama-server	http://localhost:8080/v1	No	Bundled with RTILA X (used in GGUF mode)
Custom	Any OpenAI-compatible URL	Depends	Any server implementing the OpenAI chat completions API

Configuration Steps

1. Go to Preferences → AI Assistant tab.
2. Set AI Provider to Local LLM.
3. Select Custom API Endpoint mode.
4. Enter the API Base URL (e.g., http://localhost:11434/v1).
5. Optionally enter an API Key (most local servers don’t require one).
6. Optionally enter a Model Name (auto-detected if left empty).

Tip: If you’re using Ollama, make sure the Ollama service is running before starting RTILA X. You can start it with ollama serve in a terminal.

---

Server Configuration (GGUF Mode Only)

When using GGUF mode, you can fine-tune how the llama.cpp server runs. These settings are found under Preferences → AI Assistant → Server Configuration.

Context Size

The context size determines how much information the AI can “remember” in a single conversation. Higher values allow more context but use more RAM.

Option	Value	Best For
4K	4,096 tokens	Very low-RAM machines (4 GB)
8K	8,192 tokens	Basic tasks on limited hardware
16K	16,384 tokens	Standard use on 8 GB RAM
32K (Default)	32,768 tokens	Recommended for most users
64K	65,536 tokens	Complex multi-page tasks
128K	131,072 tokens	Very complex tasks, 16+ GB RAM
256K	262,144 tokens	Large context needs, 32+ GB RAM
512K	524,288 tokens	Extreme context, 64+ GB RAM
1M	1,048,576 tokens	Maximum context, 128+ GB RAM

Changed in v8.3.2: The default context size has been increased from 8K to 32K. This is required to fit the full system prompt and complete command reference that RTILA X injects into every conversation.

GPU Layers

Setting	Default	Description
GPU Layers	99	Number of model layers to offload to your GPU. Set to 99 to offload all layers. Lower this value if you experience crashes or out-of-memory errors. Set to 0 for CPU-only inference.

Threads

Setting	Default	Description
CPU Threads	4	Number of CPU threads used for inference. Higher values can speed up generation but may slow down other applications.

Batch Size

Setting	Default	Description
Batch Size	512	Number of tokens processed per batch during prompt processing. Higher values speed up initial prompt processing but use more memory.

Additional Flags

Setting	Default	Description
Additional Flags	-fa on	Advanced command-line options passed directly to the llama.cpp server. Use this for expert-level tuning. Separate multiple flags with spaces.

---

KV Cache Configuration (New in v8.3.2)

The KV (Key-Value) cache stores the attention state of previously processed tokens. By tuning the cache, you can significantly reduce memory usage and improve response speed.

New in v8.3.2: Full KV cache control has been added for optimal local performance.

KV Cache Settings

Setting	Options	Default	Description
KV Cache Type K	F16, Q8_0, Q4_0, F32	F16	Quantization type for the Key cache. Lower quantization uses less memory.
KV Cache Type V	F16, Q8_0, Q4_0, F32	F16	Quantization type for the Value cache. Same options as Key cache.
Flash Attention	On / Off	Off	Enables flash attention kernels for reduced memory usage during generation.
Parallel Slots	1–8	1	Number of concurrent conversation slots sharing the same loaded model.
No MMap	On / Off	Off	Disables memory-mapped file loading. Uses more RAM but avoids disk I/O stalls.
MLock	On / Off	Off	Locks model weights in RAM to prevent the operating system from swapping them to disk.

KV Cache Type Guide

Type	Memory Usage	Quality Loss	Best For
F32	Highest (4x F16)	None	Maximum precision (debugging only)
F16	High (baseline)	None	Default, best quality for most users
Q8_0	~50% of F16	Negligible	Recommended for most users — halves memory with virtually no quality impact
Q4_0	~25% of F16	Slight	Very low-RAM machines where every MB counts

Important: Quantized KV cache types (Q8_0, Q4_0) require Flash Attention to be enabled on GPU backends. If you set KV Cache Type to Q8_0 or Q4_0, also enable Flash Attention.

Flash Attention

Flash Attention is a memory-efficient attention algorithm that:

• Reduces peak memory during text generation
• Is required for quantized KV cache on GPU (NVIDIA Ampere+, Apple M1+, AMD RDNA2+)
• Slightly increases prefill speed for long contexts

Enable Flash Attention if your GPU supports it. It is automatically beneficial and has no downsides on supported hardware.

Parallel Slots

Parallel slots allow multiple chat conversations to share the same loaded model simultaneously:

• Each slot maintains its own KV cache segment
• All slots share the model weights (no additional model memory)
• Each slot caches the system prompt and command reference independently
• Switching between chat tabs doesn’t require model reloading

Memory note: Each additional slot consumes extra KV cache memory. Start with 1 slot and increase only if you frequently use multiple chat tabs simultaneously.

Token Optimization Tips

These tips are displayed in the Local LLM settings panel:

1. KV Cache: The system prompt and reference document are loaded once per conversation slot and cached by llama-server. Subsequent messages reuse this cache, dramatically reducing response time.

2. Q8_0 KV cache halves memory usage with negligible quality impact. Recommended for systems with less than 16 GB RAM.

3. Flash Attention reduces peak memory during generation. Enable it if your GPU supports it. It is required when using quantized KV cache types.

4. Parallel slots allow multiple chat tabs to share the same loaded model without reloading.

5. A 32K context window is the recommended default. Increase only if your hardware has sufficient RAM (rule of thumb: ~1 GB RAM per 8K context for 7B–9B models at Q4 quantization).

---

Hardware Requirements

Model Size	Minimum RAM	Recommended RAM	GPU
7B–8B parameters (Q4)	6 GB	8 GB	Optional (speeds up generation)
13B–14B parameters (Q4)	10 GB	16 GB	Recommended
30B+ parameters (Q4)	20 GB	32 GB+	Required

VRAM Estimation Example

For a 7B Q4_K_M model at 32K context:

Component	Memory
Model weights (Q4_K_M)	~4.5 GB
KV cache (F16)	~2 GB
KV cache (Q8_0)	~1 GB
Total VRAM needed	~5.5–6.5 GB

Tip: If you don’t have enough VRAM, lower GPU Layers to offload some layers to CPU. Set GPU Layers to 0 for CPU-only inference (slower but works on any machine).

---

Monitoring the Local AI Engine

The sidebar shows the Local AI Engine status when the Local LLM provider is active:

Status	Icon	Description
Stopped	⏹️	Engine is not running. Click Pre-load Engine to start it.
Starting…	⏳	Engine is loading the model into memory. This can take 10–60 seconds depending on model size.
Downloading… (X%)	⬇️	Model is being downloaded from Hugging Face. Shows progress percentage.
Ready	✅	Engine is ready. Shows the active model name and a Stop button.

Pre-loading the Engine

You can pre-load the model before starting a conversation:

1. Look at the Local AI Engine section in the sidebar.
2. If the status shows Stopped, click Pre-load Engine.
3. Wait for the status to change to Ready.

This is useful when you want to avoid the initial loading delay when starting your first conversation.

Stopping the Engine

To free up memory when you’re done using the AI:

1. Click the Stop button (power icon) next to the model name in the sidebar.
2. The engine unloads the model and frees the memory.
3. The status changes to Stopped.

---

Model Download Process

When you first use a GGUF model, RTILA X downloads it automatically:

1. You enter the model identifier and send your first message (or click Pre-load Engine).
2. RTILA X checks if the model file already exists locally.
3. If not found, the download begins automatically.
4. The sidebar shows Downloading… (X%) with real-time progress.
5. Once downloaded, the model is loaded into memory.
6. The status changes to Ready.

Note: Model files can be large (4–10 GB for 7B–14B models). Ensure you have sufficient disk space and a stable internet connection for the initial download.

Cancelling a Download

If a download is in progress and you want to cancel it:

1. Click the Cancel button (X icon) next to the download progress in the sidebar.
2. The download is cancelled.
3. The status returns to Stopped.

---

Error Handling (New in v8.3.2)

RTILA X now provides hardware-specific error diagnostics when the local AI engine fails to start:

Error Type	What You’ll See	Recommended Action
Out of Memory	“Your hardware lacks sufficient RAM/VRAM for the requested context size”	Reduce Context Size or GPU Layers, or switch to OpenRouter
GPU Driver Error	“GPU driver may be outdated or incompatible”	Update GPU drivers, or set GPU Layers to 0 for CPU-only
Model Load Error	“GGUF file may be corrupted or incompatible”	Re-download the model by deleting the file and trying again

All error messages include the guidance: “This is a hardware compatibility issue, not a software bug. You can switch to OpenRouter or a Custom API endpoint in Preferences → AI Assistant to use cloud-based models instead.”

Common Error Scenarios

Symptom	Likely Cause	Fix
Engine crashes immediately after starting	Not enough RAM for the model + context	Reduce Context Size to 8K or 4K, or use a smaller model
Very slow generation (< 1 token/second)	All layers running on CPU	Increase GPU Layers if you have a GPU
“CUDA out of memory”	Model too large for GPU VRAM	Reduce GPU Layers to offload some layers to CPU
Download fails at 99%	Disk full or network interruption	Free up disk space and retry
“Failed to spawn llama-server”	Antivirus blocking the executable	Add an exclusion for the RTILA installation directory

---

Switching Models

If you change the model identifier while the engine is running:

1. The current engine is stopped automatically.
2. The new model is downloaded (if not already cached locally).
3. The engine restarts with the new model.
4. All active conversations continue seamlessly.

Tip: Models are cached locally after download. Switching back to a previously downloaded model is instant — no re-download needed.

---

Quick Start Checklist

1. ☐ Go to Preferences → AI Assistant
2. ☐ Set AI Provider to Local LLM
3. ☐ Select GGUF Model (Hugging Face) mode
4. ☐ Enter a model identifier (e.g., bartowski/Qwen2.5-7B-Instruct-GGUF/Qwen2.5-7B-Instruct-Q4_K_M.gguf)
5. ☐ Set Context Size to 32K (default)
6. ☐ Set KV Cache Type K and V to Q8_0 (recommended)
7. ☐ Enable Flash Attention
8. ☐ Click Pre-load Engine in the sidebar
9. ☐ Wait for status to show Ready
10. ☐ Start chatting with the AI Assistant!