I am in an enterprise which experiments with ontologies and knowledge graphs since its hype heydays ca. 15 years ago. It never took off, good ol’ relational databases and its modern day implementations in the cloud still rule the data landscape. Many smart consultants tried their best and we spend many millions over time into it. I would love to read more success stories, but they seem to be rare beyond academic exercises.
95% of sparql, cypher, and gremlin can be lowered to sql anyway. I rolled my own sql lowering system for duckdb over iceberg which lets me do everything in that blog post but with no bespoke data layer.
Without actual examples of it working to reduce latency, this is purely academic and not engineering. What happened to the 1.5s problem in the introduction?
Agreed that centralizing identity is critical (which implies strong reconciliation systems, that can both do well on merges/splits/identity-shifts, but also need to enable backtracing to sources for each triple), but then I don't think it makes sense anymore to force ontology across all systems in the pipeline.
As long as you give AI ability to do toolcalls/skills for each individual system and let it build up memory, then AI does much better job piecing together the actual practical ontology. You can give the observability AI to ask each system about it's state for the given piece of information and the AI should be free to make any toolcalls (e.g. call the specific system appropriately via CLI) it needs to gather the precise state.
I think human establishing the ontology gets in a way here, just let the AI poke the systems on its own via clean small interfaces.
It's at the end of this Netflix post: but the end game is to connect whatever your observability ontology is with whatever system the business people and data engineering teams use to be able to answer previously hard questions like "how much money did this operational or engineering change make or lose?"
In theory, at least.
Observability vendors have been beating the drum for 10+ years that there's so much untapped business value in operational data. Seems like we might be getting close to seeing how real that is.
The problem here is we're going to let the machines dictate the ontology, and we will lose expressiveness by reverting to the mean. All in order to communicate more cheaply with the machines - but on their terms.
Forgive the anthropomorphizing, it was necessary - "terms" has double meaning there, and each of us can understand that without an ontology - because we can access nuance. I can read schema, structure, and shape, and understand; but only one of those triggered some readers.
Anyway, the machines will choose terms which are most well-represented in the training data. This will normalize communication by chopping off the top-end where nuance lies. This is token-efficient.
Ontologies, in their implementation in the Linked Data ecosystem, do offer the ability to relate one understanding-system to another, but this is not token-efficient. I would love to see each domain, each business even, define their view of the world in the most expressive and culture matching owl representation, but I don't see it happening. "Shape" it is.
If the preference is for books, I can recommend "Semantic web for the working ontologist: effective modeling for linked data, RDFS, and OWL". https://search.worldcat.org/title/1193030282
If the preference is for papers, two papers from Deborah McGuinness et al, from early 2000s are still my favorites:
Dig into LinkML; I've been eye-deep in defining an ontology for my life (which I struggle to grasp), and just getting into the tools forced me to learn ... outward. There's so much cool stuff, and now I have a mental model as well as a machine-based thing I can talk to (obv an LLM) having a "shared understanding ".
It seems to be nearly impossible to find good resources about the subject as they have been crowded out by AI-related SEO-sites. But have you started with W3C’s specs on the subject? Ontologies, linked data, RDF, etc. In Norway we have a government agency publishing local adaption and guidance for these concepts and standards, so that is at least two sources of information on the general concepts and syntax at least. What are you looking for?
Besides AI/LLM the linked data ecosystem struggles with a deep adversion to the technology by the general developer population, dating back from the time when the Semantic Web hype cycle fizzled out. You can see that for instance very clearly on the Fediverse, where the ActivityPub [0] protocol is a W3C linked data based open standard. But ActivityPub can also be plain JSON, and most Fediverse software does not follow real JSON-LD approach. There's a lot of resistance to going the linked data route, being perceived as overly complex. Which is probably a fair assessment too, as the whole set of standards around linked data are large and intricate. As a result of Semantic Web and Linked Data failing to find good adoption, there's a chicken/egg that there's no good tooling to work with the technology and little incentive to create it.
I found the same - it's definitely studied in academia, some people do seem to practice it, and there do seem to be some standards or frameworks in some places.
I made OrchidDB to let people express ontology layers over an existing iceberg data lake. It's just a SQL lowering system with a datafusion core for everything else.
Netflix must have the most overengineered tech stack out there. Ppl hired with such high cash comp must feel the obligation to live upto that comp by creating these sort monsters.
Tech industry is overstaffed to the gills . No one really knows how to create value for customers( only for themselves through these overegineered slop for career advancement) . Tech jobs are really sad place to be now.
> This agreement has a name: an ontology. Ontology is a formal specification of types, the properties they carry, and the relationships they may have. Its unit is the triple, one fact expressed as subject, predicate, object.
Er... nope.
In the limited context of OWL and RDF, that's what an ontology is. It has a much larger meaning outside of that context. As M-W puts it, ontology is "the branch of metaphysics concerned with the nature and relations of being."
Projected into computer science, an ontology is a formal model of concepts in a domain, their properties and relationships, along with axioms or constraints that apply to those concepts.
The "triple" described in the quote is an implementation detail, how a particular formalism (OWL and RDF) chooses to represent a rather pale shadow of what an ontology is conceptually.
My local knowledge graph dropped my token usage spend on reads from being 30% of total spend to 2% and I haven't suffered a regex false positive deleting 300 lines of code randomly since. As of this weekend it also now feeds my QA metrics, replacing what was multiple tool calls taking multiple seconds down to about 25ms because it's a single cached graph traversal now rather than multiple tools traversing the same ground repeatedly.
I am in an enterprise which experiments with ontologies and knowledge graphs since its hype heydays ca. 15 years ago. It never took off, good ol’ relational databases and its modern day implementations in the cloud still rule the data landscape. Many smart consultants tried their best and we spend many millions over time into it. I would love to read more success stories, but they seem to be rare beyond academic exercises.
95% of sparql, cypher, and gremlin can be lowered to sql anyway. I rolled my own sql lowering system for duckdb over iceberg which lets me do everything in that blog post but with no bespoke data layer.
Without actual examples of it working to reduce latency, this is purely academic and not engineering. What happened to the 1.5s problem in the introduction?
I noticed this with a lot of blog posts from eng teams on AI.
Talk about what they built, not the business problem it solved and how it created value somewhere.
This seems somewhat outdated approach.
Agreed that centralizing identity is critical (which implies strong reconciliation systems, that can both do well on merges/splits/identity-shifts, but also need to enable backtracing to sources for each triple), but then I don't think it makes sense anymore to force ontology across all systems in the pipeline. As long as you give AI ability to do toolcalls/skills for each individual system and let it build up memory, then AI does much better job piecing together the actual practical ontology. You can give the observability AI to ask each system about it's state for the given piece of information and the AI should be free to make any toolcalls (e.g. call the specific system appropriately via CLI) it needs to gather the precise state.
I think human establishing the ontology gets in a way here, just let the AI poke the systems on its own via clean small interfaces.
It's at the end of this Netflix post: but the end game is to connect whatever your observability ontology is with whatever system the business people and data engineering teams use to be able to answer previously hard questions like "how much money did this operational or engineering change make or lose?"
In theory, at least.
Observability vendors have been beating the drum for 10+ years that there's so much untapped business value in operational data. Seems like we might be getting close to seeing how real that is.
The problem here is we're going to let the machines dictate the ontology, and we will lose expressiveness by reverting to the mean. All in order to communicate more cheaply with the machines - but on their terms.
Forgive the anthropomorphizing, it was necessary - "terms" has double meaning there, and each of us can understand that without an ontology - because we can access nuance. I can read schema, structure, and shape, and understand; but only one of those triggered some readers.
Anyway, the machines will choose terms which are most well-represented in the training data. This will normalize communication by chopping off the top-end where nuance lies. This is token-efficient.
Ontologies, in their implementation in the Linked Data ecosystem, do offer the ability to relate one understanding-system to another, but this is not token-efficient. I would love to see each domain, each business even, define their view of the world in the most expressive and culture matching owl representation, but I don't see it happening. "Shape" it is.
I would welcome any and all links or resources about learning more about ontologies, including anyone's personal practice of/with them.
SEP entry on "Ontology and Information Systems" is a good start: https://plato.stanford.edu/entries/ontology-is/
If the preference is for books, I can recommend "Semantic web for the working ontologist: effective modeling for linked data, RDFS, and OWL". https://search.worldcat.org/title/1193030282
If the preference is for papers, two papers from Deborah McGuinness et al, from early 2000s are still my favorites:
- "Ontology development 101: A guide to creating your first ontology": https://scholar.google.com/citations?view_op=view_citation&h...
- "Ontologies come of age": https://scholar.google.com/citations?view_op=view_citation&h...
Thanks so much. I'm open to any information in the space where/how people have seen or experienced it.
Dig into LinkML; I've been eye-deep in defining an ontology for my life (which I struggle to grasp), and just getting into the tools forced me to learn ... outward. There's so much cool stuff, and now I have a mental model as well as a machine-based thing I can talk to (obv an LLM) having a "shared understanding ".
Appreciate it, added to the list.
It seems to be nearly impossible to find good resources about the subject as they have been crowded out by AI-related SEO-sites. But have you started with W3C’s specs on the subject? Ontologies, linked data, RDF, etc. In Norway we have a government agency publishing local adaption and guidance for these concepts and standards, so that is at least two sources of information on the general concepts and syntax at least. What are you looking for?
Besides AI/LLM the linked data ecosystem struggles with a deep adversion to the technology by the general developer population, dating back from the time when the Semantic Web hype cycle fizzled out. You can see that for instance very clearly on the Fediverse, where the ActivityPub [0] protocol is a W3C linked data based open standard. But ActivityPub can also be plain JSON, and most Fediverse software does not follow real JSON-LD approach. There's a lot of resistance to going the linked data route, being perceived as overly complex. Which is probably a fair assessment too, as the whole set of standards around linked data are large and intricate. As a result of Semantic Web and Linked Data failing to find good adoption, there's a chicken/egg that there's no good tooling to work with the technology and little incentive to create it.
[0] https://www.w3.org/TR/activitypub/
I found the same - it's definitely studied in academia, some people do seem to practice it, and there do seem to be some standards or frameworks in some places.
I made OrchidDB to let people express ontology layers over an existing iceberg data lake. It's just a SQL lowering system with a datafusion core for everything else.
Is this like XML and UML all over again?
Netflix must have the most overengineered tech stack out there. Ppl hired with such high cash comp must feel the obligation to live upto that comp by creating these sort monsters.
relevant hn
https://news.ycombinator.com/item?id=49878857 - A Staff Engineer's Guide to Inventing Work
Tech industry is overstaffed to the gills . No one really knows how to create value for customers( only for themselves through these overegineered slop for career advancement) . Tech jobs are really sad place to be now.
> This agreement has a name: an ontology. Ontology is a formal specification of types, the properties they carry, and the relationships they may have. Its unit is the triple, one fact expressed as subject, predicate, object.
Er... nope.
In the limited context of OWL and RDF, that's what an ontology is. It has a much larger meaning outside of that context. As M-W puts it, ontology is "the branch of metaphysics concerned with the nature and relations of being."
Projected into computer science, an ontology is a formal model of concepts in a domain, their properties and relationships, along with axioms or constraints that apply to those concepts.
The "triple" described in the quote is an implementation detail, how a particular formalism (OWL and RDF) chooses to represent a rather pale shadow of what an ontology is conceptually.
Are knowledge graphs still relevant? Agents can grep and fetch on a unix operating system with higher accuracy
My local knowledge graph dropped my token usage spend on reads from being 30% of total spend to 2% and I haven't suffered a regex false positive deleting 300 lines of code randomly since. As of this weekend it also now feeds my QA metrics, replacing what was multiple tool calls taking multiple seconds down to about 25ms because it's a single cached graph traversal now rather than multiple tools traversing the same ground repeatedly.