The LLM prototyping process

Much has been made of the fact that early returns on generative AI applications have been meager. A recent Goldman Sachs report summarizes the overriding concern: that LLMs cost too much, that the use cases aren’t clear, and that contribution to companies’ revenue will likely not justify the expense.

Some of the issue like stems from the fact that working with LLMs is still very new (ChatGPT launched in November 2022). People are still getting used to AI workflows, to the strengths and weaknesses of (rapidly changing) LLMs, to the weirdness of prompt engineering and probabilistic outputs. It will take time for people and organizations to figure out what they’re “supposed” to do with these tools, and to build out systems and processes for those use cases.

But another issue might be that these models are just too broad for the use cases we're trying to develop. Software products do specific, well-defined things that help an end user achieve their goals as quickly and effectively as possible. Chatbots, in their current form, don’t really fit the bill - they’re capable of a lot, but there’s a shorter path to each individual task within a purpose-built system. And there are some indications that genAI is moving toward narrower applications - GitHub Copilot uses an LLM for sophisticated autocomplete, Apple Intelligence is breaking its genAI out into a range of integrated features across the OS. The LLM vanishes into the background, becoming one feature in service of a cohesive product.

(As a side benefit - more specific applications should lead to smaller models, helping to reduce the cost of incorporating LLMs into products.)

So, we can think of LLMs in a broad-to-narrow development cycle. The giant frontier model with free chat interface - GPT, Claude - is like a wide open space of compute. You can try out different configurations and potential designs quickly, without committing to any one of them. As the optimal approach starts to become clear, you can solidify your pipeline. Then, you can invest in a purpose-built model.

This is a huge boon to natural language processing tasks like entity recognition, classification, and summarization. Before widely-available LLMs, NLP often required specifically trained models. That meant labeled training data, tuning architectures and parameters, and expending the compute to train the model. All this had to be done, at least on a sample of data, before you could fully know whether a given task was feasible. This was a higher barrier to entry, limiting what’s worth testing to only those ideas with a high probability of success.

But with an LLM, you can take those gambles! There’s still cost associated with spinning up a prototype, but not nearly as much - you can quickly slot in a model that’s okay to pretty good at a wide range of NLP tasks, and assess whether it’s worth pursuing a full implementation. You start with a quick but inefficient gut check, then move on to a more involved, more efficient long-term system.

So while we might not have revolutionary models that upend the entire economy, we do have a powerful prototyping interface, a general-use block of compute upon which we can sketch out any number of product ideas.