Ollama

Ollama is informally called Docker for LLMs. It is an inference program to download and run LLM models on your local computer. It supports many backends, but is primarily used with the llama.cpp backend.

Download and run

• Download and install:

$ curl -fsSL https://ollama.com/install.sh | sh

• Pull a model and run it:

$ ./ollama pull smollm:135m

$ ./ollama run smollm:135m

>>> Is cat an animal?
Yes

>>> /bye

Build and run

• Install golang if you do not have it already:

$ sudo apt golang-go

• Download the code:

$ git clone https://github.com/ollama/ollama.git

$ cd ollama

• Build it:

$ go generate .

$ go build .

• Pull a model and run it:

$ ./ollama pull smollm:135m

$ ./ollama run smollm:135m

>>> Is cat an animal?
Yes

>>> /bye

Ready to rock!