Welcome, fellow tech enthusiasts and homelab adventurers! The Raspberry Pi has long been a staple for countless DIY projects, self-hosting endeavors, and educational pursuits. Today, we’re taking its capabilities to the next level by transforming it into a robust platform for Local Large Language Model (LLM) automation. This isn’t just about running an LLM; it’s about building a comprehensive, self-contained AI and automation system right in your homelab.
In this in-depth guide, we’ll walk through the process of setting up Ollama for local LLM inference, integrating Open Web UI for a user-friendly chat interface, and finally, orchestrating powerful automated workflows using n8n. Whether you’re a developer, a technical architect, or simply passionate about self-hosting and AI, prepare to unlock a new realm of possibilities with your Raspberry Pi.
Before we dive in, here are links to the products used in this article.
– Raspberry Pi 5 16gb: https://t4g.link/raspberrypi
– 128gb MicroSD Card: https://t4g.link/128gbmsd
– Argon ONE V5 Case: https://t4g.link/argonv5
– USB C Card Reader: https://t4g.link/usbccr
🎥 Video Overview
Prerequisites and Initial Setup
Our journey into Raspberry Pi LLM automation begins with ensuring we have the right foundation. For this demonstration, we’re leveraging a Raspberry Pi 5 with 8GB of RAM, running directly from its SD card. While NVMe offers performance benefits, an SD card is perfectly viable for getting started.
Essential Tools & System Preparation
Before we dive into the core applications, confirm you have the following ready:
- Raspberry Pi 5 (8GB): Our primary hardware.
- Docker and Docker Compose: These are crucial for containerizing our applications. If you haven’t installed them, refer to dedicated guides on installing Docker and Portainer on your Raspberry Pi.
- Terminal Access: Connect to your Raspberry Pi via SSH for command-line operations.
First, ensure your system is fully updated to avoid any compatibility issues. Open your terminal and execute:
sudo apt update && sudo apt upgrade -yDeploying Ollama: Your Local LLM Runtime
Ollama is a fantastic tool that allows you to run large language models locally. It simplifies the process of downloading, configuring, and interacting with various LLMs, making it perfect for our Raspberry Pi homelab setup.
Ollama Installation
Installing Ollama is straightforward. Execute the following command in your terminal. This script will download and set up the Ollama service on your Raspberry Pi. This process may take some time depending on your internet connection.
curl -fsSL https://ollama.com/install.sh | shOnce the installation completes, you might see a warning about no NVIDIA or AMD GPU detected. This is expected on a Raspberry Pi and can be safely ignored as we’ll be utilizing the CPU. Verify the installation by checking the Ollama version:
ollama –versionSelecting and Running Your First LLM
The key to a successful local LLM experience on a Raspberry Pi is choosing lightweight models. While powerful, Raspberry Pis aren’t designed for heavy GPU computation, so smaller parameter models offer a smoother experience. Here are a few recommended models suitable for your Raspberry Pi:
- Gemma 2 (2 billion parameters)
- Qwen 2.5 (1-3 billion parameters)
- Llama 3.2 (smaller variants)
- Numotron Mini
For our example, let’s download and run Qwen 2.5. This command will first download the model (if not already present) and then initiate an interactive chat session with it.
ollama run qwen:2.5bAfter the model downloads and loads, you’ll see a prompt to send a message. You can now interact directly with your local LLM! Ask it questions, request jokes, or query for information. To exit the interactive session, type `/bye`.
Integrating Open Web UI: A User-Friendly Interface for Ollama
While interacting with Ollama via the command line is functional, a graphical user interface significantly enhances usability. Open Web UI provides a beautiful and intuitive web-based chat interface for your local LLMs. We’ll deploy it using Docker Compose.
Setting Up Open Web UI with Docker Compose
First, create a dedicated directory for Open Web UI and navigate into it:
mkdir /opt/stacks/openwebui
cd /opt/stacks/openwebui
mkdir dataNext, create your `docker-compose.yaml` file. Use `sudo nano docker-compose.yaml` and paste the following content:
sudo nano docker-compose.yamlPaste the following content into the docker-compose.yaml file:
services:
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
volumes:
- ./data:/app/backend/data
ports:
- "3000:8080"
extra_hosts:
- "host.docker.internal:host-gateway"
restart: unless-stopped**Important Docker Compose Notes:**
- ports: The left `3000` is the external port you’ll use to access the UI (e.g., `http://your-pi-ip:3000`). Change it if port 3000 is already in use.
- volumes: `data` directory inside your `openwebui` folder will persist application data.
- restart: unless-stopped: Ensures the container restarts automatically if the Pi reboots or the container crashes.
Save the file (`Ctrl+X`, `Y`, `Enter`). Then, deploy the container:
sudo docker compose up -dConfiguring Ollama for Docker Communication
Initially, when you try to connect Open Web UI to Ollama, you might encounter a “network problem.” This happens because Open Web UI is in a Docker container, and by default, your host-installed Ollama isn’t configured to listen for connections from external sources (even from another container on the same host).
To fix this, we need to tell Ollama to listen on all network interfaces. Edit the Ollama service file:
sudo systemctl edit ollama.serviceAdd the following line under the `[Service]` section:
Environment=”OLLAMA_HOST=0.0.0.0″Save the file. Then, restart the Ollama service for the changes to take effect:
sudo systemctl daemon-reload
sudo systemctl restart ollamaNow, navigate to your Raspberry Pi’s IP address on port 3000. You’ll be prompted to create your admin account. Once logged in, go to `Settings` -> `Admin Settings` -> `Connections`. You should now see Ollama connected successfully. You can then select your Qwen model (or any other downloaded models) and start chatting!
Managing Models with Open Web UI
Open Web UI also simplifies model management. From the `Admin Settings` -> `Models` section, you can click “Manage” and pull new models directly from `ollama.com`. Simply copy the model name (e.g., `gemma2`) and paste it into the “Pull a model” field, then click the download icon. This allows you to effortlessly expand your local LLM collection.
Automating Workflows with n8n
n8n is a powerful open-source workflow automation tool that lets you connect APIs, services, and devices with a visual editor. It’s the perfect companion for our local LLM setup, enabling us to build custom automations triggered by our AI.
n8n Installation with Docker Compose
Similar to Open Web UI, we’ll deploy n8n using Docker Compose. Create its dedicated directories:
mkdir n8n-compose
cd n8n-compose
mkdir local-filesCreate your `docker-compose.yaml`:
sudo nano docker-compose.yamlPaste the following into the Docker Compose file and save it:
services:
n8n:
image: docker.n8n.io/n8nio/n8n
restart: always
ports:
- "5678:5678"
environment:
- N8N_ENFORCE_SETTINGS_FILE_PERMISSIONS=true
- GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
volumes:
- n8n_data:/home/node/.n8n
- ./local-files:/files
volumes:
n8n_data:Next we need to create our Environment file:
nano .envPaste the following into the .env file (get a list of Linux Time Zones here: https://t4g.gg/linux-time-zone-list/):
# Get the time zone list at https://t4g.gg/linux-time-zone-list/
GENERIC_TIMEZONE=America/Los_Angeles
# The top level domain to serve from (only needed if exposing to the internet)
DOMAIN_NAME=example.com
SUBDOMAIN=n8n
SSL_EMAIL=user@example.comSave the file and deploy n8n:
sudo docker compose up -dBypassing Initial Security for n8n
Upon navigating to `http://your-pi-ip:5678`, you might initially encounter a security error if `N8N_SECURE_COOKIE` is not set to `false` (or if your `docker-compose.yaml` was modified later). You can edit this directly via Portainer (if installed):
- Go to Portainer, find your `n8n` container
- Select “Duplicate/Edit”
- Scroll down to “Environment Variables”
- Add a new variable: `N8N_SECURE_COOKIE` with the value `false`
- Click “Deploy the container” to apply changes
After successfully deploying, refresh your browser. You’ll now be able to set up your n8n admin account. It’s recommended to enter your email and request a free license key to unlock additional features like workflow folders, which are invaluable for organizing your automations.
Connecting n8n to Ollama
To allow n8n to interact with your local LLMs, you need to create an Ollama credential within n8n.
- In n8n, click the `+` icon for “Credentials.”
- Search for “Ollama” and select it.
- For the “Base URL,” enter `http://YOUR_PI_IP:11434` (replace `YOUR_PI_IP` with your Raspberry Pi’s actual IP address). `localhost` often won’t work in this context from inside the n8n container.
- Click “Save.” If the connection is successful, you’re ready to build workflows!
Practical Application: Your First n8n Workflow (Ollama to Discord)
Now for the exciting part: building an automation that leverages our local LLM and integrates with Discord! We’ll create a “Jokebot” that takes a prompt, asks Ollama for a joke, and posts it to a Discord channel.
Setting Up Discord Webhooks
First, you need a Discord Webhook URL. This allows n8n to send messages to a specific channel without needing a full bot.
- In Discord, go to your server settings, then `Integrations`
- Select `Webhooks` and click `New Webhook`
- Give it a name (e.g., `n8n-jokebot`) and select the channel where messages will be posted
- Click `Copy Webhook URL`. Keep this URL secure; anyone with it can post to your channel.
Building the Workflow in n8n
- Create a New Workflow: In n8n, click `Create Workflow`.
- Chat Message Trigger: Add a “Chat Message” node. This will trigger our workflow when a message is sent in n8n’s chat interface. Select “On a new chat event.”
- Ollama Node: Click the + and search for Ollama
- Select your Ollama credential.
- Select your desired model (e.g., `qwen:2.5b`).
- For the “Content” field, drag and drop the `Chat Input` variable from the “Chat Message” node’s output. This sends your chat message to Ollama.
- Discord Node: Click the + and search for Discord
- Choose “Send a Message” via “Webhook.”
- Create a new credential and paste your Discord Webhook URL.
- For the “Message” field, drag and drop the `Content` variable from the “Ollama” node’s output. This is Ollama’s response.
**Testing Your Jokebot:**
With the workflow active, go to the n8n chatbox (bottom left of the dashboard). Type something like “Tell me a joke about space.” You’ll see the workflow execute, and after a short processing time on your Raspberry Pi, the joke will appear in your designated Discord channel!
This simple jokebot demonstrates the power of integrating local LLMs with automation. The possibilities are truly endless, transforming your Raspberry Pi into an intelligent assistant for various tasks. You might notice your Raspberry Pi’s CPU utilization spike during these operations, a clear indicator that your local AI is hard at work!
Conclusion
We’ve journeyed through setting up a sophisticated local AI and automation system on a Raspberry Pi. From deploying Ollama for local LLMs, integrating Open Web UI for seamless interaction, to orchestrating intelligent workflows with n8n, you’ve equipped your homelab with powerful new capabilities. While running LLMs on a Raspberry Pi might be more of a hobbyist’s delight than a high-performance solution, the educational value and sheer fun of controlling your own AI infrastructure are immense.
This setup provides a fantastic sandbox for experimentation, allowing developers and IT professionals to explore the nuances of local AI deployment without relying on cloud services. The robust combination of these open-source tools on a compact Raspberry Pi is a testament to the versatility of modern homelab environments.
We encourage you to experiment with different models, build more complex workflows, and discover how this powerful trio can solve unique challenges in your projects. If you’ve built cool workflows with n8n or have insights on optimizing LLMs on Raspberry Pi, we’d love to hear about it!
Connect with the HAVOK community and share your creations on our Discord server: https://havok.link/discord-wp
*If you buy something through any of the links featured in this article I may get a small share of the sale.