Running Dramatron locally using Koboldcpp + LLM

There aren’t many user interfaces which break out of the now ubiquitous chat prompt for interating with an LLM.

Dramatron is a really interesting guided process and user interface. LLM interfaces really need more experimentations, if you’re interesetd in trying some other alternatives there are a few notable ones appearing in the open source LLM interfaces, https://github.com/FartyPants Playground, Twinbook and BlockWriter and https://novelai.net/ is also worth a look.

If you don’t have the capability to run Dramatron, you can take a look at the process in the video at the end of this article.

• remove the guardrails; there are an incredible selection of characterful LLMs, which have been trained on specific styles or content, offering a much richer experience than some of the more corporate, business focused models.

• data leakage; you keep control of your data your prompts and results won’t end up being fed back into any model without your knowledge
  a lot of hosted LLMs have strict guardrails which depending on what type of dialog you’re authoring can quickly put a stop to your project.

• No fees using a local; you can experiment without racking up any costs

The process to get this up and running is a little complex, I’ll outline the main process below:

Use my Dramatron fork, it has a connector for koboldcpp’s API.

You’ll need to be running https://github.com/LostRuins/koboldcpp and have it running an appropriate model, a couple of recommendations would be
https://huggingface.co/TheBloke/dolphin-2.7-mixtral-8x7b-GGUF
https://huggingface.co/mlabonne/NeuralBeagle14-7B-GGUF

Clone my dramatron repository at https://github.com/makeplayhappy/dramatron

I've added a new Jupyter notebook in the colob folder (dramalocal-kobold.ipynb) which uses Koboldcpp API so you can run this using a local LLM server.

If you don’t have jupyter installed then install using pip

pip install jupyterlab

Locally you can run Jupyter notebooks with this command

jupyter notebook

or use the no-browser and ip flags if you want to run it on a local server (change 192.168.1.10 to whatever your local server's IP is)

jupyter notebook --no-browser --ip="192.168.1.10" --port=7080

then you can open up http://192.168.1.10:7080 in a browser connect to it, go into the collab folder and open up the dramalocal-kobold.ipynb notebook.

Then update the LLM_API_GENERATE_URL setting to use your local kobold api IP address (localhost or “http://192.168.1.10:5000” in my example)