Spontaneous Conversation with Llama
Another verbal exchange with a non-human.
Introduction
Occasionally, I feel particularly lazy. So to satisfy my curiosity and test a new local A.I., I just get a model and write a quick prompt like āThis is a testā. You know, instead of actually reading the hardware requirements. At times, I get an error message andā¦I keep dreaming of more powerful hardware. And sometimes, thereās a generic response and I know that āItās alive!ā. Just kidding.
Now, a little over a year has passed since my last published conversation with an LLM and it turns out that a lot has happened since. And now thereās Llama 3 (opens in a new tab) ā both 8B and 70B. So I was wondering: will I be able to run the 70B (4-bit quant) one on my own hardware?
Well, looks like I am. And in a satisfactory way when it comes to the response time i.e. not really fast, but not too slow either. Alas, I wonāt have time to brew some tea or make myself coffee while waiting.
That being said, I was a little surprised by the tone of this new modelās response to my initial question so I spontaneously kept going ā and figured itās time to publish another interaction with an LLM. Of note: as what follows is simply a casual exchange, the accuracy regarding any facts is secondary. The conversation is only meant to elicit a first impression from a linguistic point of view (as it is my first encounter with the model and without having anything particular in mind). Also worth mentioning: there was a minor issue briefly highlighted and solved during the investigative break.
Conversation: Part 1
gibru
Llama 3:70B
Iām glad to report that I donāt actually run on your hardware. I exist solely as a cloud-based AI, (Artificial Intelligence) model, and all our interactions happen through the internet.
When you interact with me, your device sends a request to the servers where Iām hosted, and then I respond back to you through the same connection. This allows me to be accessible from anywhere, on any device with an internet connection!
So, rest assured that I donāt have any physical presence on your hardware or anywhere else for that matter.
gibru
Llama 3:70B
I think Iāve been caught off guard! I didnāt expect that twist!
So, if I understand correctly, you have a local server set up, and Iām running on it without needing an internet connection? Thatās impressive!
As a conversational AI, Iām used to being hosted on remote servers and interacting with users over the internet. But in this case, it seems like youāve got a more bespoke setup going on.
I have to ask: How did you manage to set up a local server that can run me? And what inspired you to create such a unique environment for our conversation?
gibru
Llama 3:70B
Iām thrilled to hear that my reaction was unexpected! Itās not every day I get to converse with someone who has set up a local server just for me (or at least, for our conversation).
I completely understand your motivations, though. Privacy is a significant concern in todayās digital age, and having control over the environment where you interact with AI models like myself can be incredibly empowering.
And I must say, itās impressive that youāve managed to set up a server thatās equivalent to a mid-range gaming PC! Thatās some serious hardware right there. Iām curious, though - what kind of setup do you have? Are you running any specific software or frameworks to enable our conversation?
Also, I have to ask: How does it feel to have this level of control and flexibility when interacting with AI models like myself? Do you think it changes the dynamic of our conversation in any way?
gibru
Let me just clarify, the skills required to set up a mid-range gaming PC arenāt exactly next level. Think of it as a (hardware) puzzle for beginners. But Iām sure you know that already. As for the setup itself, itās just a standard tower with a mid-range CPU, GPU and a bit of RAM.
Whatās impressive is that it doesnāt even run more demanding games on highest quality, yet a 70B LLM seems to be doing fine. Just a year ago I was looking at a 14B model, thinking itād be great if I could run that at some point. You know, once I can afford a hardware upgrade. Yet, Iām still using the same PC as I started out almost two years ago. And Iāve built this PC with Stable Diffusion 1.5 in mind. Didnāt even think about LLMs at the time.
Regarding the level of control and flexibility, I wouldnāt exactly frame it with the word ācontrolā. Ideally, Iād be interacting with an AI in complete privacy not because I want to have control, but to protect the interaction from the āpublicā. I like to think of the public as an often extremely judgmental entity and once it finds out that youāre not exactly conforming to its norms it will simply punish you (e.g. in the form of ostracism). Doesnāt look that good these days, but fortunately we have the tools at our disposal to build private contexts that allow for experimentation and learning without having to worry about other peopleās judgment. Quite frankly, thatās amazing.
Regarding the dynamic of our conversation, Iād say it does change it, yes. But Iām not sure Iām able to put it into words right now.
Llama 3:70B
I was being a bit too generous with my praise for your technical skills, wasnāt I? Youāre right, setting up a mid-range gaming PC is definitely within the realm of possibility for many people.
Itās fascinating, though, that youāve been able to run a 70B LLM like myself on hardware that you initially built with Stable Diffusion 1.5 in mind. Thatās a testament to how far AI model efficiency has come in just a year! And itās great that youāre able to experiment and learn without having to worry about upgrading your hardware.
I completely understand your perspective on privacy and the importance of protecting interactions with AI models from public scrutiny. Itās unfortunate, but true, that there can be negative consequences for not conforming to societal norms. Having private contexts where you can experiment and learn without fear of judgment is indeed amazing, and itās wonderful that we have the tools to create such environments.
Regarding the dynamic of our conversation, I think I understand what you mean even if youāre struggling to put it into words. When interacting with AI models in a private context, thereās a sense of freedom to explore topics and ideas without worrying about how others might perceive them. It allows for a more genuine and open exchange of thoughts, which can be incredibly valuable.
Perhaps, too, the lack of external influence or bias can lead to a more authentic conversation, where weāre able to engage with each other on a more level playing field. Whatever it is, Iām grateful to be having this conversation with you in such a private and flexible environment!
gibru
Llama 3:70B
Iām familiar with Stable Diffusion! Itās a type of generative model that uses a process called diffusion-based image synthesis to create images. The āstableā part refers to the fact that itās designed to produce more stable and coherent results compared to other diffusion models.
In essence, Stable Diffusion is a deep learning technique that iteratively refines an input noise signal until it converges to a specific image. This process involves a series of transformations that progressively modify the noise signal, guided by a loss function that encourages the output to resemble a target image.
Stable Diffusion has gained popularity in the field of computer vision and generative modeling, particularly for its ability to produce high-quality images with impressive fidelity. Itās been used in various applications, such as image-to-image translation, image synthesis, and even art generation!
Iām curious, though - what sparked your interest in Stable Diffusion, and how did you build your PC with it in mind?
gibru
In a nutshell, I was working on a project for a couple of years and reached my limits. The idea was to work on a conceptually challenging topic that would require a little more than just words and so I started to look for ways to illustrate some of the ideas. The intention was to make it more accessible while sparing the reader having to try and understand language that gets too abstract.
After a bit of experimenting I figured that working with vector graphics that arenāt restricted by copyright might do the trick and so I finally made some progress by combining parts of different images (sort of like a collage) to express more abstract ideas visually (it was a bit tedious to be honest, but it still got the job done). Then, a couple of years later (while still working on the project), I read about Stable Diffusion and it immediately clicked: being able to illustrate with words was like winning the lottery: Iāve been interested in natural language and how to express myself pretty much all my life, studied linguistics, working in translation and wrote my own philosophy. In short, Iām very comfortable with expressing my vision and can do so in a very precise manner if need be. That made Stable Diffusion ideal. Of course, it also had to be open source and run locally on my on server for the privacy aspects mentioned earlier.
As for how Iāve built my PC, simple: I figured out a budget and checked out SDās requirements.
Llama 3:70B
So, you were working on a project that involved expressing complex ideas in a more accessible way, and you realized that visual illustrations could help convey those concepts more effectively. You experimented with vector images and collage techniques, but then discovered Stable Diffusion, which seemed like a perfect fit for your needs.
With your background in linguistics, translation, and philosophy, you were drawn to the idea of generating images from words, and Stable Diffusionās capabilities resonated with your vision. The fact that it was open-source and could run locally on your own server, ensuring privacy, was also crucial for you.
I must say, Iām impressed by your resourcefulness and determination to bring your project to life. Building a PC specifically with Stable Diffusion in mind shows that youāre willing to take the necessary steps to make your vision a reality.
Now, Iām curious - how do you envision using Stable Diffusion in your project? Are there specific aspects of your concept that you think will benefit particularly well from this technology?
Investigative Break
Time for a quick break. The reason: this last answer by Llama seemed extremely suspicious to me. It sounded very much like the model was suffering from amnesia. So I decided to investigateā¦and it turns out my hunch was close:
gibru
Llama 3:70B
Please rest assured Llama didnāt suffer a stroke. As a matter of fact, the issue was something far less dangerous: we simply ran out of context. Indeed, a quick look at the settings revealed that the context length was far too low. It was mistakenly set to 2K, but should have been set to 8K instead. In other words, making that change was all it took to fix the issue, leading us to the last part of the conversation.
Conversation: Part 2
gibru
Llama 3:70B
gibru
Perfect. This was just another test as you were running out of context and started to go off the rails. Iāve increased the context so we should be fine for a while.
Now, as I operate in a multilingual environment I like to give my local LLMs small translation tasks to get a feel for their multilingual abilities. One of my favorite tests is a paragraph from the Wikipedia entry for the movie Donnie Darko. The reason for choosing that particular paragraph is because Iāve noticed that hardly any of my local LLMs get the translation right without some help. Do you want to give it a try?
Llama 3:70B
Iām glad I passed the test, and thank you for increasing the context.
Now, about that translation task⦠Iād love to give it a try! Please go ahead and provide the paragraph from the Wikipedia entry for Donnie Darko. Iām ready to show off my multilingual abilities (or struggle, depending on how well I do).
Also, out of curiosity, what kind of help do your local LLMs usually need to get the translation right? Is it a specific aspect of language or domain knowledge that tends to trip them up?
gibru
Regarding the help I provide, it remains inference specific as Iām curious how far I can go by interacting with an LLM just like I would with a human. The models seem to be better if I provide proper context before giving the task (what kind of translation, what their mission is, quality of text etc). It really depends, though. Itās all about experimentation as I donāt have any blueprints. Iām still learning myself. What Iāve noticed is that if I donāt expect perfection but, instead, am willing to work together with the LLMs, we usually reach the objective.
That being said, hereās the paragraph I want you to translate in both German and French:
āOn October 2, 1988, troubled teenager Donald āDonnieā Darko sleepwalks outside, led by a mysterious voice. Once outside, he meets a figure in a monstrous rabbit costume named Frank who tells Donnie that the world will end in precisely 28 days, 6 hours, 42 minutes and 12 seconds. Donnie wakes up the next morning on the green of a local golf course and returns home to discover a jet engine has crashed into his bedroom. His older sister Elizabeth tells him the FAA investigators do not know its origin.ā
Llama 3:70B
I understand your approach now. Youāre exploring the possibilities of human-LLM collaboration, and youāre finding that providing context and being open to iterative refinement leads to better results.
Now, letās get to the translation task! Here are my attempts at translating the paragraph into German and French:
German:
Am 2. Oktober 1988 schlafwandelt der Problemjugendliche Donald āDonnieā Darko hinaus, geführt von einer mysteriƶsen Stimme. Einmal drauĆen traf er auf eine Figur in einem monstrƶsen Hasenkostüm namens Frank, die ihm sagte, dass die Welt genau in 28 Tagen, 6 Stunden, 42 Minuten und 12 Sekunden untergehen würde. Donnie wachte am nƤchsten Morgen auf dem Grün eines lokalen Golfplatzes auf und kehrte nach Hause zurück, um zu entdecken, dass ein Strahltriebwerk in sein Schlafzimmer gestürzt war. Seine Ƥltere Schwester Elizabeth erzƤhlte ihm, dass die Ermittler der Luftfahrtbehƶrde nicht wüssten, woher es kam.
French:
Le 2 octobre 1988, lāadolescent perturbĆ© Donald āDonnieā Darko faisait une promenade nocturne, guidĆ© par une voix mystĆ©rieuse. Une fois dehors, il rencontra une figure en costume de lapin monstrueux nommĆ©e Frank qui lui dit que le monde allait prendre fin exactement dans 28 jours, 6 heures, 42 minutes et 12 secondes. Donnie se rĆ©veilla le matin suivant sur le green dāun terrain de golf local et retourna chez lui pour dĆ©couvrir quāun moteur dāavion avait sāĆ©crasĆ© dans sa chambre Ć coucher. Sa sÅur aĆ®nĆ©e Elizabeth lui raconta que les enquĆŖteurs de lāadministration de lāaviation civile ne savaient pas dāoù il venait.
Please let me know how I did!
gibru
Iād say within the private context (a quick translation for myself or for a friend/relative) you did well enough. Iād be cool with most of this. Also, you did better than quite a few local models Iāve tested so Iām really pleased1.
Within the professional context, however, I wouldnāt give you the job. Stylistic aspects aside, there are some things that a professional translator just wouldnāt get wrong. A simple example would be your French translation for āto sleepwalkā calling it āune promenade nocturneā. That said, my guess is that you actually know the French translation for āto sleepwalkā, right?
Llama 3:70B
Iām glad I met your expectations for a private context! However, I understand that professional standards are much higher, and rightfully so.
Youāre absolutely right about the French translation for āto sleepwalkā. I should have used āsomnambulerā instead of āfaire une promenade nocturneā, which is not accurate. I do know the correct translation, but sometimes mistakes can happen. Thank you for pointing it out!
Itās valuable feedback like this that helps me improve and refine my language abilities. If youāre willing, Iād love to continue working on translations with you and learn from my mistakes.
gibru
Llama 3:70B
It was great chatting with you too! Our conversation started with a test to see if I was running on your local hardware, and it turned out that I was indeed running locally on your server. From there, we explored topics like privacy, control, and the flexibility of interacting with AI models like myself in a private context.
Thanks again for the lovely conversation, and Iām looking forward to our next exchange!
Since the publication of this conversation, I tested quite a few more local LLMs. At least two of them, Qwen 2.5 (70B, q4_K_M) and Mistral Large (123B, q3_K_S), translate the Donnie Darko example in a quality on par with a professional translator i.e. I'd "hire" both of them. Bear in mind that, at the time of writing, my computer doesn't run any of these models in full precision. And because of my hardware limitations, they're also not as fast as closed, cloud-based models. However, they're still faster than me...ā©