Search for a command to run...
I’m a backend developer based in Beijing, passionate about crafting efficient and scalable solutions. Amidst my coding endeavors, I’ve embarked on an exciting journey as an active English learner. Exploring the intricacies of language allows me to broaden my horizons and connect with diverse communities worldwide. Join me as I navigate the realms of technology and language, striving for excellence in both.
No comments yet. Be the first to comment.
Introduction Recently, I attended a training session where the host used many filler words and repeated examples, causing it to last longer than it should have. Fortunately, I recorded the video using the Tencent Meeting software. I had an idea to ex...
Background I am using a QNAP ARM CPU NAS, so its power efficiency is supposed to be a great advantage. I want to integrate the accessories, especially the IR remote, with HomeKit for a better user experience when controlling the old air conditioner...
#include <future> #include <iostream> #include <thread> int main() { // future from a packaged_task std::packaged_task<int()> task([]{ return 7; }); // wrap the function std::future<int> f1 = task.get_future(); // get a future std::t...
Q & A CLANG basic setting Break point is not accurate optimize_list = [ "-O0", "-g", ] Envs export CXX=clang compile_flags.txt # Treat all files as C++ files -xc++ -std=c++17 BUILD If use blade, the generated build command with CLANG will in...
Introduction Recently, I attended a training session where the host used many filler words and repeated examples, causing it to last longer than it should have. Fortunately, I recorded the video using the Tencent Meeting software. I had an idea to ex...
As you know, Ollama is a very easy-to-use tool for local LLM. It lets people run and experience the latest models with little effort, using simple commands such as:
ollama run llama3
ollama ps / ollama list
ollama serve
What's more groundbreaking is that it's now available on Docker. Here is the news.
And now you can use Docker to combine the popular open-webui and Ollama to build an easy-to-use, modern, data-controlled chatbot just for you.
Install Ollama
Ensure good network conditions; the model file is usually large, so 100Mbps bandwidth is necessary
[Optional] Install Docker
Installing Ollama and running it through the command line on macOS is very easy.
# Install via Homebrew on macOS or use the setup package for Windows
brew install ollama
ollama run llama3
Running Ollama on Docker with Open-WebUI
# Run ollama
docker run -d --gpus=all -v 'D:\repos\ollama:/root/.ollama/models' -p 11434:11434 --name ollama ollama/ollama
# Run open-webui
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v D:\repos\open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main
On the Windows platform, you can choose the model storage path by specifying the path in the system environment variables, called OLLAMA_MODELS.
Please note, it seems less efficient when running on Docker. While it does enable and accelerate with GPU(nVidia 4070Ti 12GB), switching and initializing the model is very slow (running the model on a host machine is super fast on my machine with a 7b llama3 model).
And, GPU acceleration running on Docker is not available on Apple Silicon.