H. Yaghi

How to setup a Local LLM – part #2

, ,

By: Husam Yaghi

In a previous post, I stated that as an AI enthusiast and developer, I embarked on an exciting journey to install and fine-tune a local Large Language Model (LLM) on my Windows PC. This project, which took several weeks to perfect, was both challenging and incredibly rewarding. In this blog post, I’ll share my experience #2, hoping to help others who wish to set up their own local LLM.  This time, I am setting up a local LLM in a much simpler way, using a Docker service; of course there are advantages as well as disadvantages for this approach.  I will leave it to you to decide which approach suites you best.

Install Docker on Windows

Go to docker.com and install the software and follow the instructions.

Once installation is complete, open a command window and type the following command:

docker run -d -p 3000:8080 –add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data –name open-webui –restart always ghcr.io/open-webui/open-webui:main

This command pulls the Open WebUI container insider Docker.

 

Install Ollama on Windows

  1. Download the installer from the Ollama website (https://ollama.com/).
  2. Once the installation is complete, ensure that the latest version has been installed by checking the version in the terminal:

                            ollama  –version

  1. Select and install the desired LLM models:

Visit Ollama’s Library (https://ollama.com/library?sort=popular ) to view and choose models to download.  Then, in the command window, run the following command to pull the model of your choice (i.e. mistral 7b):

                           ollama pull mistral:7b

 

Step 3: Use Open WebUI

Useful link: https://docs.openwebui.com/getting-started/

  1. When Open WebUI has been installed as a container, you can click on the link under ‘Port’.

  1. Alternatively, you can access Open WebUI by typing http://localhost:3000 in your browser directly.
  2. If an account has not been created, you will be prompted to make a new account.
  3. Select a model:  Click on “Select a Model” on the top to choose from your pulled LLMs.

  1. Upload documents (your dataset) to Open WebUI:  Click on Workspace > Documents > ‘+’  to add documents.

        I recommend adding tags that mention the key topics of the document.

  1. Toggle ‘ON’ for web search:

          Click on your account at the bottom left of the screen

           > Settings > Admin Settings > Web Search.

       I recommend using duckduckgo for the web search engine.

Once you’ve done this, press ‘SAVE’ and go back to your chat by clicking ‘New Chat’ in the top left corner. When you’re in this screen, click the More button ‘+’ to turn on Web Search for your chat.

  1. Referencing specific files when chatting with LLMs:

Use the ‘#’ in the chat bar to select ‘All Documents’ or specific collections to have the model answer queries using your uploaded files. This is where tags come in handy.

 

Manually updating Open WebUI

As you’ve installed Open WebUI using Docker Compose, follow these steps to update:

  1. Pull the latest images:

This command fetches the latest versions of the images specified in your docker-compose.yml files.

                              docker compose pull

  1. Recreate the Containers with the latest images:

This command recreates the containers based on the newly pulled images, ensuring your installation is up-to-date. No build step is required for updates.

                            docker compose up -d

This method ensures your Docker Compose-based installation of Open WebUI (and any associated services, like Ollama) is updated efficiently and without the need for manual container management.

 

Additional information to note

  • When installing Open WebUI through Docker, there are different options based on if you have GPU support (Nvidia or AMD), Linux, CPU only. Learn more here.
  • In order to run Open WebUI on the browser, you need to keep Docker and Ollama running. They can run in the background, but both have to be running.
  • Unless you opt to search the web, your local LLM on Open WebUI does not require Internet connection.

 

Link the local dataset to Docker.

Create a dockerfile:

Using an available simple editor,  create a dockerfile with the following content:

      # Use an official Ubuntu as the base image
      FROM ubuntu:latest
      # Set the working directory inside the container
      WORKDIR /app
      # Copy the contents of your local dataset to /app inside the container
      COPY . /app
      # Install any dependencies (if needed)
      RUN apt-get update && apt-get install -y python3 python3-pip
       # Default command to run when the container starts
       CMD ["bash"]

and save as “dockerfile” without an extension it in a reachable location (in my case:  d:\docker), then change directory to that location.  

                         cd  d:\docker

then on a command prompt type:

docker build -t yaghidataset -f  d:\docker\dockerfile  .

This command explicitly tells Docker to use the Dockerfile located at d:\docker\dockerfile.

 

Verify the Image Was Built Successfully

After the build completes, verify that the image was created by listing all available images:

                     docker images

You should see your newly built image in the list, with the REPOSITORY name as yaghidataset (or whatever name you used in the -t flag during the build).

REPOSITORY      TAG       IMAGE ID       CREATED          SIZE

yaghidataset    latest    <IMAGE_ID>     <time_created>   <image_size>

 

Run a Container Using the Image

Now that your image has been built, you can create and run a container based on this image. You use the docker run command to do this.  For example, if you want to run a container using the yaghidataset image, you would run:

                   docker run -it yaghidataset

  • -it: This flag allows you to run the container interactively with a terminal attached (useful if you want to interact with the container, e.g., using a shell).
  • yaghidataset: This is the name of the image that you built earlier.

If your Dockerfile specifies a command like CMD [“bash”], this will provide you with a terminal inside the running container.

 

Mount a Local Directory (Optional)

If you want to mount a directory from your host machine (e.g., D:\yaghiDataSet) into the running container, use the -v flag to specify the volume mapping:

     docker run   -it   -v   /d/yaghiDataSet:/data   yaghidataset

This command will:

  • Mount the D:\yaghiDataSet directory from your local machine to /data inside the container.
  • Start the container interactively with a terminal.

Once the container is running, you can navigate to the /data directory inside the container to see the mounted files:

       cd   /data     

       ls             

 

Run a Background Container (Optional)

If your container is meant to run in the background (e.g., a web server or another service), you can use the -d flag to run the container in detached mode:

               docker  run  -d  yaghidataset

This will start the container in the background, and it will continue to run until stopped.

  

Check Running Containers

You can check the status of your running containers with:

                   docker   ps

This will list all running containers. If you want to see all containers (including stopped ones), use:

                 docker   ps   -a

 

Stop a Running Container

If you need to stop a running container, use the docker stop command, followed by the container ID or name:

             docker   stop   <container_id_or_name>

To get the container ID or name, you can use docker ps to list running containers.

            docker  ps

Once the docker build command completes successfully, you’ve built a Docker image that can be used to run containers. The next step depends on what you want to do with the image, but typically, you will want to run a container using the image you just built.

 

Remove a Container

If you want to remove a container after you stop it, use:

                    docker   rm   <container_id_or_name>

 

Tag and Push the Image to a Registry (Optional)

If you want to share your image with others or deploy it to a production environment, you can tag the image and push it to a Docker registry (such as Docker Hub, AWS ECR, or a private registry).

  1. Tag the image (if you want to use a different tag or push to a specific registry):

             docker   tag   yaghidataset  username/repository:tag

For example, to tag it for Docker Hub:

            docker  tag  yaghidataset  your-dockerhub-username/yaghidataset:latest

Log in to Docker Hub (or another registry):

          docker login

Push the image:

        docker push your-dockerhub-username/yaghidataset:latest

This will upload your image to Docker Hub, where it can be pulled or shared with others.

 

Inspect the Image (Optional)

 You can use the docker inspect command to get detailed information about the image you built:

                   docker  inspect  yaghidataset

This will display details about the image, including metadata, layers, and configuration. 

Summary of What to Do After the Build:

  1. Check the image: Use docker images to verify the image was built.
  2. Run a container: Use docker run -it yaghidataset to start an interactive container.
    • Optionally, mount a local directory using -v (e.g., docker run -it -v /d/yaghiDataSet:/data yaghidataset).
  3. Background container: Use docker run -d yaghidataset to run the container in detached mode.
  4. Check running containers: Use docker ps to see running containers.
  5. Stop a container: Use docker stop <container_id_or_name> to stop a running container.
  6. Remove a container: Use docker rm <container_id_or_name> to remove a stopped container.
  7. Push the image: Optionally, tag and push your image to a Docker registry.

 

Reminder:

In order to use your LLMs, you must have both applications are running:

  1. Ollama
  2. Docker

From your Windows “Start”, run both of these applications.

Let me know if you find these posts of value.  hyaghi at gmail.com