LLM use inside enterprises is at an all time high, but when you look closely at how it is being used, most of it leans on general knowledge from the model’s training data and a bit of internet search, perhaps supplemented with MCP servers (where your security team allows them), connectors, and a collection of skills and .md files. The problem, however, is that a model without true knowledge of your business, your people, your customers, your products and your systems gives general advice, nothing specific, which severely limits its institutional value.
Think of it as hiring the most talented consultant on the planet and sitting them at a desk on day one. They have a ton of experience, but zero knowledge about your business. They do not know why you discounted that deal last quarter, which product line quietly loses money, what your customers complain about on the phone, or how your fulfillment process really runs (as opposed to how the process document says it runs).
So where does that knowledge live? It sits in your people’s heads, their email, your SharePoint, Teams/Slack discussions, your file servers, your process documentation, your sales & marketing plans and your old proposals. Much of it is unstructured, scattered across systems that were never designed to talk to each other, and some of it was never written down at all.
The obvious shortcut is to use LLM connectors, via MCP, and we are all told that if you pay them enough, your data remains private to you. Maybe so, but two problems remain. You must still disclose to your customers that their information is being processed by a third party, and data that sits with a provider can still be obtained by law enforcement through legal process served on that provider. I am not a lawyer, and the exact exposure will depend on your jurisdiction and your contracts, so have yours reviewed; either way, the material you would be entrusting with the LLM provider is pricing, customer detail, strategy and intellectual property, which is the most concentrated version of your crown jewels you could assemble in one place.
This is why running a local RAG (retrieval augmented generation) engine is critical. The mechanism is simple enough to explain; the model does not need to be trained on your data, because at the moment someone asks a question the engine retrieves the handful of passages most relevant to it and places them in front of a local LLM, which then answers from your material and can point back to the source documents it used. Getting there means ingesting everything, chunking it into pieces small enough to retrieve precisely, and making it searchable.
Permissions deserve their own sentence, because retrieval must respect the access controls you already have; otherwise you have built a very efficient way for the intern to find the board paper. This is probably the most overlooked concept in the race to AI enable everything!!
On top of the retrieval layer sits the LLM / conversation router, which is the piece your people actually touch. They hit one endpoint, and the router decides where the question is best answered; a question about your pricing, your customers or your internal process goes to a local LLM with local retrieval, whilst a general question about a public programming language or a published standard can go to a cloud LLM provider, if your policy allows, with nothing from the crown jewels attached. The decision is driven by data classification and policy, not by the user remembering which tool is safe for which question (and I would not bet on any of us remembering, right!!).
I also know what the pressure looks like from the other side of the desk. The board and the CEO wants an AI story, and somewhere a senior executive is saying “we have this under control, we have licensed the enterprise tier of a leading provider, and the contract says our data is private”. The CIO and CISO are the people who then have to be comfortable that this is actually true, and nobody wants to be the one who signed off on the crown jewels leaving the building. A local RAG engine and a router give them an answer they can defend, and give your people an assistant that knows your business.
Now the sales pitch :) Our own answer to the ingestion problem is Portainer-AiGrid, a self-hosted appliance that ingests and retrieves your document corpus, using the idle GPUs on your workstations to do the heavy lifting. No need for you to spend $$ with a cloud RAG provider, or to try and deploy an army of MCP servers to directly connect cloud LLMs to your internal data. Internal data stays internal at all times, where it should be.