I encountered this with trying to have LLMs populate facts about electoral campaigns. Like when a candidate drops out, when endorsements happen, but also if a candidate is un-endorsed or drops and rejoins. It also needed to handle if any of these facts were incorrect.
I settled on a knowledge graph in Postgres and downloading/storing the source documents so it could iterate on past results without more scraping or network calls.
This blog post helped me understand security analysis in this context! A lot of the important systems around malware analysis or large scale system security (the parts people really care about) clicked for me. So thanks for writing it.
Anyways I hope we can find some pattern to converge on with this wrt "facts management" since I feel this is currently something a lot of people and LLMs are struggling with. In practice current LLMs working with episodic memory feels similar to a grandparent with dementia scrawling things down in notebooks, crossing things out, and getting very confused.
I had similar problems to solve for trading advice, complex project reasoning etc.
I settled on simplicity, extended it to serve clear, useful purpose.
Started with markdown database, single fact per line, structured/parseable (subject VERB object). This can be easily diffed/reviewed etc in git. Then added optional metadata (confidence, tags, persisted comments for natural language, uncertainty for numeric values, context, timestamps/spans), querying, alternative sqlite3 backend, self describing VERBS, z3 solver etc.
Those kind of graph information systems are great to quickly structure knowledge in a way that LLMs and humans can use/act on/loop on. Creating ontology, linking, some rules and actions and kicking it so it all munches and spits out results that feed back in so it self evolves. It's very natural for llms to query/update/restructure those graphs (also good for humans because it's very terse, essential information only).
There is no need to create k8s style complexity/services/what-not, it all works well from single sqlite db file or bunch of markdown (.cave in my case) files.
It's also interesting to see how well local open weight models are dealing with information arranged this way.
Comments
I encountered this with trying to have LLMs populate facts about electoral campaigns. Like when a candidate drops out, when endorsements happen, but also if a candidate is un-endorsed or drops and rejoins. It also needed to handle if any of these facts were incorrect.
I settled on a knowledge graph in Postgres and downloading/storing the source documents so it could iterate on past results without more scraping or network calls.
This blog post helped me understand security analysis in this context! A lot of the important systems around malware analysis or large scale system security (the parts people really care about) clicked for me. So thanks for writing it.
Anyways I hope we can find some pattern to converge on with this wrt "facts management" since I feel this is currently something a lot of people and LLMs are struggling with. In practice current LLMs working with episodic memory feels similar to a grandparent with dementia scrawling things down in notebooks, crossing things out, and getting very confused.
I had similar problems to solve for trading advice, complex project reasoning etc.
I settled on simplicity, extended it to serve clear, useful purpose.
Started with markdown database, single fact per line, structured/parseable (subject VERB object). This can be easily diffed/reviewed etc in git. Then added optional metadata (confidence, tags, persisted comments for natural language, uncertainty for numeric values, context, timestamps/spans), querying, alternative sqlite3 backend, self describing VERBS, z3 solver etc.
Those kind of graph information systems are great to quickly structure knowledge in a way that LLMs and humans can use/act on/loop on. Creating ontology, linking, some rules and actions and kicking it so it all munches and spits out results that feed back in so it self evolves. It's very natural for llms to query/update/restructure those graphs (also good for humans because it's very terse, essential information only).
There is no need to create k8s style complexity/services/what-not, it all works well from single sqlite db file or bunch of markdown (.cave in my case) files.
It's also interesting to see how well local open weight models are dealing with information arranged this way.