Very cool. I recall an HN submission (which I can't find offhand unfortunately) that did something similar -- it used an LLM to decompose articles into a set of statements which were used to construct an entity-relationship graph of facts and events. It then queried that using conventional graph query methods, much like DataLog / Lemmalog is doing here. I remember it was particularly effective at answering timeline-based queries that LLMs (back then) sucked at.
I think approaches like this are going to be (or maybe already are?) the basis of effective grounding of LLM responses in authoritative data sources. It should be possible to pinpoint any error to an incorrect traversal or an incorrect "fact." This would work best for concrete, unambiguous facts, however; fuzzy, ambiguous or opinion-based information will probably remain the purview of LLMs.
My problem with memory is that it goes stale, and updates to facts are often not changing all locations of that fact. Such a system should make it easier to maintain a single source of truth, and versioning, no?
Yes you need a VCS and to recalculate relationships at certain thresholds. It adds a lot of complexity and computarional overhead, which is why there's no uniform widespread graph use
Oh, interesting, is this based on your experience, or is there a reference I could look up? My layman intuition is better grounding should result in more accurate results, is that not the same as better reasoning?
Comments
Very cool. I recall an HN submission (which I can't find offhand unfortunately) that did something similar -- it used an LLM to decompose articles into a set of statements which were used to construct an entity-relationship graph of facts and events. It then queried that using conventional graph query methods, much like DataLog / Lemmalog is doing here. I remember it was particularly effective at answering timeline-based queries that LLMs (back then) sucked at.
(See also Cyc: https://en.wikipedia.org/wiki/Cyc)
I think approaches like this are going to be (or maybe already are?) the basis of effective grounding of LLM responses in authoritative data sources. It should be possible to pinpoint any error to an incorrect traversal or an incorrect "fact." This would work best for concrete, unambiguous facts, however; fuzzy, ambiguous or opinion-based information will probably remain the purview of LLMs.
Is it this one:
https://news.ycombinator.com/item?id=41445445
Yes, thank you!
This is great for evidence grounding, but doesn't produce a large memory/reasoning improvement (in most cases)
My problem with memory is that it goes stale, and updates to facts are often not changing all locations of that fact. Such a system should make it easier to maintain a single source of truth, and versioning, no?
Yes you need a VCS and to recalculate relationships at certain thresholds. It adds a lot of complexity and computarional overhead, which is why there's no uniform widespread graph use
Oh, interesting, is this based on your experience, or is there a reference I could look up? My layman intuition is better grounding should result in more accurate results, is that not the same as better reasoning?