Your Local Private AI Assistant

This post details how to deploy your own private local AI assist on your own PC using Ollama and open-webui without inernet connection.

- - - Example – AI Assistant like this - - - - - - - ->

- - - Details for each steps - - - - - - - ->

1. Go to the offical website in Ollama and click “Download” to chose the right version to download according the system of your PC.

2. Using following Terminal Command to check if you’ve installed successfully, my is windows so I’m using Command Prompt as example.

  • if it shows a version then you installed successfully.
1
2
3
>>ollama --version

ollama version is 0.1.32

3. Typing following command to check what kinds of operations you can do with ollama now

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
>>ollama --help

Large language model runner

Usage:
  ollama [flags]
  ollama [command]

Available Commands:
  serve       Start ollama
  create      Create a model from a Modelfile
  show        Show information for a model
  run         Run a model
  pull        Pull a model from a registry
  push        Push a model to a registry
  list        List models
  cp          Copy a model
  rm          Remove a model
  help        Help about any command

Flags:
  -h, --help      help for ollama
  -v, --version   Show version information

Use "ollama [command] --help" for more information about a command.

4. Now you shouldn’t have any LLM (large language model) installed yet.
You can check what LLM models you want to download from this link Ollama Models. Thus you can download an example model first by typing following command for instance to download this gemma:2b to your local PC (gemma is latest model series released by Google and 2B means 2 billion parameters, which is lightweight anyway)

1
2
3
4
>>ollama pull gemma:2b

pulling manifest
pulling c1864a5eb193...  98% ▕███████████████████████████████████████████████████████ ▏ 1.6 GB/1.7 GB   77 KB/s   6m19s

5. Run following code to check if you’ve downloaded successfully
I’ve already downloaded a number of models

1
2
3
4
5
6
7
>>ollama list
NAME                    ID              SIZE    MODIFIED
deepseek-coder:6.7b     ce298d984115    3.8 GB  10 days ago
gemma:2b                b50d6c999e59    1.7 GB  33 minutes ago
gemma:7b                a72c7f4d0a15    5.0 GB  11 days ago
llama3:8b               71a106a91016    4.7 GB  10 days ago
openchat:7b             537a4e03b649    4.1 GB  10 days ago

6. In the terminal window you can use “ollama run gemma:2b” command to start using this model.

1
2
3
4
5
6
7
8
9
>>ollama run gemma:2b

>>>
>>> Hi, please introduce yourself.
Hello! My name is AI, and I'm a large language model trained by Google. I'm designed to be informative and
helpful, and I'm always happy to assist with your queries.

How can I help you today?
>>> Send a message (/? for help)

7. However I believe you wouldn’t like to using Terminal window to chat with AI, thus we shuold use somthing like this, which is called open-webui

8. Click the link of open-webui for installation guide, however you need to use Docker to install, pls using this linke to install Docker first Docker. There are tons of online tutorial to install docker, pls check youtube

9. Once installed docker you can use following command to install this “open-webui” docker image

1
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

10. Then open the docker desktop to check you’ve already installed open-webui

11. Open your web browser to this address http://localhost:3000/, then you are in your own private chat
- Click following red area to change to another LLM model if like.

12. One imporant note for windows user if you want to download the models to another drive rather than C: Drive because Ollama defaults to download to C:

  • You need to create new system variable in the “Environment Variables”.

  • Add new variable with following name and path you want to store the models to and exit Ollama then restart Ollama server.