Solving the business context problem for LLMs

A way to infuse live context into LLMs for business applications

Alluding the the famouse phrase in the LLM land, “attention is all you need”, I would like to propose a solution to the problem of context in LLMs for business applications.

How about an API endpoint with a simple interface that allows you to send a context object to the LLM before sending your prompt?

Say, for example, you need to project revenue for a business unit given the constraints such as spend and overhead costs. You could send a context object like the following:

1
2
3
4
5
{
  "business_unit": "Sales",
  "spend": 100000,
  "overhead_costs": 50000
}

Then, when you send your prompt to the LLM, it would have access to this context and could generate a more accurate response.

Given the fact that this value could changes in the next few days, the context object would always be at par with what’s in your business systems.


The caveat is that you would have to worry about security given that the LLM would always be hitting your production API endpoints.

A way to avert the above would be, for example, creating a dedicated service for the LLM while relying on the production service as just a data source.

an illustration of a service set up

This way, you are always in control of what data gets saved or served without worrying about the potentioal security implications of exposing your production endpoints to the LLM.

I hope this helps you in any way as you integrate LLMs into your business applications without relying on stale data that’s outdated and not relevant to your current business context.

I believe that this approach is better way managing Markdown files that contain business context that you want to use for LLM ingestion.