Grounding AI agents in enterprise data has been one of our main focuses lately. One of the most successful ways to do that is building an AI ontology, as discussed on our latest roundtable on operationalizing AI ontologies. Even earlier, we covered what an ontology is, and our Genie Ontology test covers Databricks’ solution. But an ontology won’t be the first or best choice for everyone, so this guide covers how to implement an enterprise context layer of any kind at the right size: how to scope it, check whether it’s working, and measure its impact.
Quick recap on ontologies for AI agents
As Mihály Darányi, our very own Director of Data & AI Solutions here at Hiflylabs, succinctly put it at the latest roundtable: an ontology used to be documentation for people, and now it should be designed to be read by AI agents.
And that kind of reader needs an ontology that’s:
- Versioned and timestamped, so the data lineage of each result can be pinned to the definitions and sources in force when it was given
- Trust-scored, because source systems can disagree: the model has to know which dashboard people actually use, or which table is the single source of truth and not just an archived copy of it
- Retrieved at inference time, inside the agent’s context pipeline (graph RAG, for example)
- Self-updating from query patterns, dashboards and metadata
- Often action-linked, so an agent can propose a purchase order or reroute a shipment (usually with a human-in-the-loop mechanism in place)

Put together, published results point to 61-77% more reliable answers, 55-99% fewer hallucinations, 40-50% faster responses, and 50-60% lower cost per agent call for successfully implemented AI ontologies. (The last two are results achieved by shortening the LLM’s reasoning paths while finding and checking data sources, as correct paths will already be implied by the ontology.)
An illustrative example: this 2026 clinical Q&A study reached 98% accuracy on 60 questions with an ontology-grounded knowledge graph, against 37%.
What’s the difference between a taxonomy, an ontology, and a semantic layer?
People often conflate ontologies, taxonomies, semantic layers, and knowledge graphs. Understandably, because they are related concepts, and not exclusive with each other. But before exploring how they might overlap, let’s clear up their differences:
- Ontology = a formal, machine-readable description of the types of entities in your business, how they relate, and the rules they obey. When it’s action-linked, it also covers the processes those entities take part in.
- Knowledge graph = the entities and their connections stored as nodes and edges. (You don’t need a graph to have an ontology, though. And not all knowledge graphs use ontology-based schemas. But because they do pair well, they’re often mentioned almost synonymously.)
- Taxonomy = hierarchical classification. Which category each thing belongs to. It only holds parent-child relations, with no rules or other links between entities.
- Semantic layer = a formal mapping from business terms to the data in your warehouse. For analytics and BI purposes, the functionality of a traditional semantic layer and an AI ontology will be similar: business stakeholders can ask stuff and get grounded answers. But the ontology reaches further, understanding operational rules as well as unstructured data.
So a well-designed data model is useful, but doesn't ground AI agents properly just by itself. Gartner also says schema-based models lack the business context and meaning agents need.
As Ross Leher, CEO and Chairman of WAND Taxonomies, put it at our latest roundtable:
“One of the problems that we see with agentic AI is that processes within the enterprise are not well defined, if defined at all.”
You need to spell out how processes run at your place. Only then will unstructured data really mean something to your agents, and ground their answers in how your business actually works.
How to right-size the scope before you implement an enterprise context layer
However, you won’t always need a full AI ontology. A context layer (the tier between the data layer and the intelligence layer) and its complexity should follow that of your operations. Several things can fill the role:
| Option | What you get | Who builds it | Typical build time | Good fit when |
| Context repo (“LLM wiki” or “second brain”) | A folder or Git repo: relevant documents, an index file, glossary, ideally some metric definitions, process notes and examples. | One analyst or engineer, or a domain owner | Hours to days | One team, one platform, question answering only |
| Semantic layer | Business terms mapped to metrics, dimensions and joins | Domain owners, a sponsor, analytics engineers | Weeks to a couple of months | Several teams sharing metrics from one warehouse |
| Single-domain ontology | Entities, relationships and rules for one function | Domain owners, a sponsor, AI and analytics engineers | Within a quarter | AI answers or automations must follow business rules |
| Full organizational AI ontology | A company-wide model with action-linking, multi-agent routing, versioning and feedback loops | Cross-functional, up to department heads and an executive sponsor | Quarters, company-wide effort | Teams are writing to the same data, and agents are working across domains |
The cheapest on-ramp is a lightweight context repo: Markdown files an agent reads directly (what Andrej Karpathy calls an LLM wiki). It’s what keeps our agentic BI projects consistent. This approach stops being enough when different teams need different access rights, volume becomes enterprise-scale, and definitions have to stay in sync with live data. Sometimes you can scale a lighter repo up to become a formal context layer. Other times, the lightest option is all a team will ever need.
Alex Woolford, Senior Field Engineer at Neo4j, proposed this rule of thumb at the roundtable:
If you have one platform, one warehouse, one stakeholder in mind, then you probably don't need an organization-wide ontology. When there are multiple teams consuming the same data sources, or data stored in different places, then you’ll need discipline about meaning and access.
As Ross put it, one of the most important points of an AI ontology is:
“Governance, governance, governance, governance, governance!”
Any AI search or GraphRAG project can be approached similarly. If the job is finding and connecting facts across documents, a taxonomy or lightweight graph schema will do. Level up to an ontology when answers have to obey definitions and rules, or when several teams write to the same graph, or when agents will act as well as answer. In fields with established standards like healthcare, you might want to start from those (SNOMED CT, ICD-10).
Start at either end of the data pipeline when building a context layer for AI
The context layer is the cheapest piece to build, so it isn’t a bottleneck in itself. Your entry point should be coupled with solving a core bottleneck at either end of the data pipeline. Instead of starting in the middle, choose either of these: the data foundations and governance problem, or a very concrete business use case.

The data governance end
At the data end, you consolidate and federate sources, fix quality and lineage, and align definitions. A lot of this is organizational work. If departments run the same process in their own way, the model won't match reality, and you may have to design a new process and onboard people onto it. Ross said it:
"If I automate an inefficient process, what I have is an inefficient automated process."
Get subject matter experts to optimize it first.
The use case end
At the application end, you take one dashboard, agent, app or automation people want, and build only the formal context that one use case requires.
Pick one department and start from either end. Work toward the other, then expand horizontally. Shared definitions go company-wide, and team-specific ones stay local.
The knowledge graph work we've done started the same way, with one purpose each: mapping a telco's legacy codebase and dependencies for incremental modernization, or tracking rooms and beds at a hospital.
If you're adding an ontology to an AI product that already exists, the plumbing is really the small part. One new step in retrieval, plus evaluation. The real work is in surfacing every definition the product will have to use, and tuning them until eval scores hold up.
Prove the impact of the context layer
You might wonder: ‘How do I evaluate if the context layer is really helping my AI agents?’ Alex Woolford had this recommendation (he was talking about ontologies, but his advice is applicable to any type of context scaffold):
“The first thing that has to happen is having an evaluation set… You create a battery of tests that give you a “with ontology” / “without ontology” score. That’s the fastest way to determine whether what you build is useful.”
How to make an evaluation set for your first AI context layer project
- Collect around 50 questions your team asks daily, with known correct answers. Don’t generate synthetic questions, use real queries.
- Score the AI with and without the added context grounding, on the same set of questions every time.
- For every miss, investigate why. Was context missing? Was the definition too loose? Did the agent get more of the context than the question needed?
- Fix it, version the context layer, and rerun. When the same mistake keeps coming back, add the corrective pattern itself to your context.
As Alex put it, the context an agent gets in the moment decides whether it works or not. Each call should see only the slice its question needs. In our previous post on preparing your organization for Genie Ontology, we explained how that working memory differs from the full knowledge graph.
Our rule of thumb is at least 90% entity resolution accuracy: records that describe the same real-world thing, like one customer in CRM, billing and support, get matched correctly.
A warning sign: Fewer than 15% of enterprises pass the pilot stage on ontology implementation, usually stuck on stakeholder alignment and defining business processes, which results in insufficient entity resolution accuracy.
That means, if you can define something in your business, go ahead and do it instead of waiting for the agents to infer the correct knowledge. The more you can normalize business domains, processes and definitions yourself, the more reliably agents can fill in any remaining gaps. If you have multiple documents, reports, dashboards lying around that carry information about the same topic, and serve similar purposes, your team will have to decide:
Which one should the AI use when someone asks about that given topic?
Define the single source of truth your agents should trust for each domain and functionality.
What to measure to determine the ROI of an enterprise AI context layer
Track these on your eval set, then keep tracking them in production:
- Reliability: first-attempt accuracy and error rate on the same real questions.
- Speed: latency on multi-hop questions, and how long analysts spend pulling context from several systems.
- Cost: tokens per resolved query, plus retry and escalation rate. You want to measure the difference between the wrong first answers that turn into follow-up calls, vs. how much cheaper the short path is when you get answers right the first time.
- Traceability: the share of answers you can trace to source data, transformation and definition, and how a number can be explained.
- Coverage: did the model look at available sources that should be relevant to your question? How thoroughly does it read through them? Did it miss anything obviously relevant?
Your benchmark should vary with task complexity, domain and how rigorous your baseline is. Clinical, legal and finance work tends to show the biggest improvement in quality and performance.
Gartner predicts that by 2027, organizations that prioritize semantics will see up to 80% better agentic AI accuracy and 60% lower costs. That opens the budget conversation, and your own before-and-after numbers close it. As Alex put it at the roundtable, the metadata work is an investment that pays off over time in the quality of the results.
How to update your AI’s context layer when your business logic changes
Your definitions won’t last forever, of course. And when a model of your business stops matching reality, the usefulness of agents will quickly drift.

If you introduce a new process (or overhaul an old one), you’ll have to rework the schema and definitions in the engineered context to onboard your agents, just as you would rewrite the documentation for onboarding new colleagues.
What does that look like in practice? Say, the definition of “customer” switches from buyer to household. Treat that change like code:
- Get the definition’s owner to relay the organizational change into the context layer.
- They’ll add a new version of the definition with an effective date, and keep the old one queryable.
- Rerun the evaluation set. This way, the context layer becomes a versioned component you can objectively test.
- Stamp every answer with the version it used. Past answers stay reproducible, and you can tell the results of bad data from the results of a changed definition.
How often should you review? You can update anything on the fly, but Ross suggested subject matter experts go through it maybe once a year. (Of course, if you’ve got an action-linked ontology, certain state updates can be automated. But the logic of those automations should be reviewed regularly all the same.)
Where AI context engineering projects often stall
Mihály’s point at the roundtable was that AI pushes organizations to change faster, and mechanisms like an ontology that documents changes let agents keep up. Where real life projects stall is people: unaligned stakeholders, missing owners, and processes remaining undefined. Change management and cross-functional alignment are the main work.
So what should you avoid to improve the chances of success on your first MVP?
- Too much at once: a dozen data domains in week one.
- No dedicated owners: four definitions of "active customer", but no single named semantic owner.
- Automating an undefined process: not describing domains and processes, just building pipelines.
- Modeling the org chart: describing how things should run on paper instead of how the business actually operates.
- A mixed-vintage document pile: a corpus with one version each from 2023, 2024, 2025 and 2026. Which one does the AI use?
- No evaluation set: nobody can tell whether a change helped.
- Static handovers: the business moves, the model doesn't, and all the automations and answer accuracy drift with it.
If you’d like a second opinion on how to implement an enterprise context layer at the right size, our ontology engineering team runs scoping sessions. "No full ontology needed" is a result we're happy to hand over, in which case data analysts can help spring up a lightweight context repo.


