THE HIGH COST OF LLM FLAB

The world has been taken by the power of generative AI. ChatGPT and others have come forward with technology that opens the door to “common” people having access to awesome computing abilities with a natural interface. Much of the old ways of analytical processing have died with the advent of generative AI.

At the foundation of generative AI is technology known as LLM – large language models. Simply put, for generative AI technology to work, raw text is read, reduced to tokens and placed in the LLM. The LLM then uses the tokens in order to conduct an analysis of the text that is presented to it for analysis.

DIFFERENT TYPES OF LLM’S

There are two different kinds of LLM’s – one LLM is serving the universe. That kind of LLM is what is found in ChatGPT. It applies to the large body of users that can ask any sort of question.

The second kind of LLM is the one that serves the corporation. It is a corporate LLM or a private LLM.

The universal LLM can contain a very wide variety of data. The corporate LLM applies only to private corporate data, such as proposals, call center conversations, legal contracts, corporate projections of sales, and so forth.

There are many good reasons why private corporate data should never find its way into a universal LLM.

The conversations in this paper apply to the corporate LLM, not the universal LLM.

THE RANDOMNESS OF RAW TEXT

Raw text that is fed to the LLM can come from anywhere – voice recordings, newspapers, spreadsheets, the Internet, emails, and more. Furthermore, there is no order to raw text. A person can say or write whatever they want.

Raw text is simply random.

As stated, at the heart of generative AI is the LLM – the large language model. In many ways the LLM is analogous to the brain of the human body. Information is stored, organized and used there in order to control and manage the other functions of the technology.

INGESTING RAW TEXT

The LLM is created from the ingestion and structuring of raw text. The contents of the LLM and the volume of the LLM all depend on the raw text that is used to form the contents of the LLM’s vector database.

What is not understood by most is that the massive volume of text shaping the LLM has a profound effect on the cost and speed of the analytical processing that will occur after the LLM is built. But it is a fact, the volume of text fed to the LLM for ingestion does have a profound effect on performance and resources consumed once the analytical processing of the LLM starts to occur.

SEPARATING RAW TEXT

Let’s step back and examine what the contents of the raw text entering the LLM might look like. In particular let’s look at how much of the raw text is devoted to the actual business of the organization versus text in the document that is extraneous to the business of the organization.

In the diagram that is shown, ingested text that is relevant to the business of the organization is shown in black and ingested text that is irrelevant to the running of the corporation is shown in red.

It is seen that the LLM can be likewise classified. The irrelevant raw text turns into the part of the LLM that contains irrelevant contents. And by the same token, the relevant raw text turns into a portion of the LLM that is relevant to the business of the corporation.

There are many ways to characterize this division of relevancy. One way to characterize it is to say that the black portion of the diagram is the muscle of the LLM and the red portion of the LLM is the flab.

Business analytical processing can proceed nicely with no reference to or mention of the flab. The flab just gets in the way and makes analytical processing slower and more expensive.

FLAB

Another way of thinking of flab is that the flab is unnecessary and a costly drag on operations.

It is absolutely normal for the organization to gather flab in its interior when raw text is just fed to the LLM. Some of the raw text is useful, some of the raw text is not.

SO WHY SHOULD I CARE?

So why should an organization pay attention to the flab in the raw text feeding the LLM? Is it really important? Does it really matter

A CASE STUDY

In order to answer this question in the context of reality, a case study was done. A banking procedures document was read into the systems in order to build an LLM using Retrieval Augmented Generation or RAG, which is a technique to improving LLM’s accuracy and reducing hallucinations. The banking procedures document was divided into business relevant text and business irrelevant text.

In the diagram shown, the highlighted text is text that is relevant to banking business and the text that is not highlighted is text that is extraneous to the conduct of business.

A CASE STUDY

In order to answer this question in the context of reality, a case study was done. A banking procedures document was read into the systems in order to build an LLM using Retrieval Augmented Generation or RAG, which is a technique to improving LLM’s accuracy and reducing hallucinations. The banking procedures document was divided into business relevant text and business irrelevant text.

In the diagram shown, the highlighted text is text that is relevant to banking business and the text that is not highlighted is text that is extraneous to the conduct of business.

When a word that is relevant to business is encountered, the text surrounding the word – a chunk or a snippet – is collected. The chunk or snippet is then collected into a data base where the word, the classification or context of the word is determined, and the source of the word is stored.

The data base then is used to preprocess raw text before the text is included into the LLM

.SO, WHAT ARE THE COSTS?

An interesting question then arises. The question is this – how much does it actually cost you to include extraneous text along with business relevant text for inclusion into the LLM?

You are going to be shocked at what was found in the case study that was conducted.

In order to be clear about the calculation that will be presented, we need to be crystal clear about what is being compared. The first part of the calculation refers to the inclusion of only the business relevant text. The second part of the calculation refers to the inclusion of all the text – both flab and business-related text – into the building of the LLM.

In the case study there were 440 words that were found that were deemed to be business relevant. The document in its entirety encompassed 1600 words. 1160 of those words were flab, or words that were not deemed to be directly relevant to the business of the corporation

.CORELATIVE ANALYSIS

In order to make the calculation you need to know that when a query is done, that a corelative analysis against all tokens in the LLM is done. The more tokens you have, the more comparisons that have to be made. And each comparison costs a tiny bit of energy in the computer.

The simple concept for the comparisons of tokens that need to be made is this –

Lower number of tokens x number of tokens = Lower total energy required to do each analysis.

Using the relevant text, with the Textual ETL supported context, the LLM reduced it token usage, and thus energy costs, and thus real costs by 92%! The amount of hallucinations were reduced by almost 20%! So the results were better and the costs were drastically reduced.

THE TRUE COST OF FLAB

By adding flab to the mix, the organization has raised the amount of energy in the computer by a factor of almost 10. In other words, the computer had to expend almost 10 times the amount of energy to process the entire document as opposed to the amount required to simply process the business relevant data.

The organization has consumed over an order of magnitude of energy by adding irrelevant data to the LLM. Furthermore, there’s no business payback for the irrelevant text that has been added. And even more – the addition of the irrelevant text increased the hallucinations produced by the LLM in it analytical processing.

If there were any business value that had been added by including extraneous text, then it might be useful to add the irrelevant text. But there is no business value that has been added. Instead, the organization has placed an anchor on its speedboat

.

ON A QUERY-BY-QUERY BASIS

Not only is the energy factor interesting, the waste of computer energy becomes really interesting when it is understood that EACH new query triggers this waste of energy. Every time anyone does any query, a huge amount of computer energy is wasted. Furthermore, the answers that include the FLAB were misleading or just plain wrong more than 20% of the time. Result quality increased and costs decreased by using Textual ETL as part of the preprocessing of your LLM

.A SECRET (AND KEEP IT A SECRET!)

So, there is a secret. (Don’t tell this to anyone.)

If you want to save a LOT of money and streamline your processing, just load business relevant text into your LLM.

A LOT OF MONEY!!

LLM MANAGEMENT

So how can you do this? Just give LLM Management organization a call and we will show you just how easy it is to separate business relevant text from extraneous text.

Give Carol Renne a call at – crenne54@gmail.com