Are you forcing every problem into an LLM-shaped hole?

All Roads Lead To LLMs

It is extremely tempting to think that you could solve every problem by just throwing all of your data at an LLM, then asking it questions about the data. Surely this is what it was built for? And when you do this, you actually receive answers, brilliant! So what do you have to watch out for? 

LLM Problem 1: Data privacy

Every time you use an LLM, where are your prompts going? If you’re uploading data or images, where are they going? Depending on which type of LLM solution you’re using, it could be that the provider is taking, saving and using that data to retrain their models. Further, if you look to try and turn that ability off, it can cost extra money. By default, almost all providers use your prompts and data for training purposes with their free version.

We will be exploring the different types of LLM solutions your firm could be using in a future article.

LLM Problem 2: Lack of levers to pull

Great, so you’ve uploaded the data you want to use, and you’re satisfied that you’re ok with the privacy set up. But now you’ve entered a prompt and you’re clearly not getting the right answer. So you change things slightly with your prompt, and you’re still not getting the right answer. Where do you now turn?

Here’s your issue: LLMs only offer one lever to pull. If you don’t like what you’re getting, it’s extremely difficult to fine tune for the right answers, set the right guardrails in place, and ensure deterministic answers. In fact, you can waste many an hour in LLMs to get them to output something that would take you seconds in excel. They are extremely good at some things, terrible at others.

LLM Problem 3: Limited Context Window

Again, you’ve uploaded the data you want to use, and now you’re asking questions of the LLM. You start to notice that the answers being output bear no relation to what you’ve uploaded. In fact, it seems like the LLM is using whatever it wants, yet it’s not letting you know your data source isn’t being used. What’s happening?

As mentioned previously, one of the features of an LLM is its context window. Think of this like its working memory, which is the number of tokens (or words) it can retain from the current input and previous responses. Increasing this window adds extensively to costs. At some point there is a cut off for this number of words, at which point even files you’ve uploaded get “forgotten”. At this point, you’ll have to reupload the data in order for it to get used again. Rather frustratingly, the LLM won’t even tell you it’s not using that data anymore, and will confidently tell you it’s happy with the output.

LLM Problem 4: Black Box

Ok, you’ve got the output from the LLM. You’re about to use the answer it’s given you, however you want to be sure that the answer is correct. How do you do this? It is very, very difficult to see exactly how an LLM has arrived at the answer it has, and if you’re using a commercial solution nearly impossible. You are effectively outsourcing your intelligence to a solution you don’t have control of.

What to do: Pick the right tool for the job

In a previous article, we attempted to define AI as “[helping] to estimate approximate solutions to problems which are otherwise difficult (or otherwise impossible) to solve.” If we were to then expand into how you should actually be using AI, it is critical that we then understand exactly what problem it is we’re trying to solve. No matter what line of business you’re in, you can analyse your use cases to a level of detail where it’s understood what the problem is (what is the question you’re trying to answer?) and the system into which the solution might fit. 

For example, I might be a content creator, looking to ensure that I send relevant content only to people who are interested in the subject matter. 

We can split this larger problem into several smaller steps, each with their own question to answer:

By breaking down the problem into these questions, we can then start to understand the data needed and how we could use AI to answer each question

Potential Solutions

Using LLMs only

Consider if we just used an LLM for all of this. We could end up with:

Using “other” ML solutions only

Similarly, using purely only “other” ML solutions, we could end up with:

Human only

Or, we could fall back on the human content creator doing all of the above:

Clearly, just choosing one option in the above leads to a sub-optimal solution in one way or another. 

Why not pick the best of all worlds? Humans write the content, LLMs tag it, and other ML solutions (like a neural network or other interest model) for selecting who should receive it. Then, you’ve got the right type of solution solving each problem, and every problem no longer looks like a nail!