Ask HN: How to manage docs for LLM RAG app?
I'm building a LLM RAG QA bot for my company, a financial institution. Right now I know the 'basic' building blocks, e.g. prompt engineering, RAG, vector db, eval, etc. Funny enough the first challenge I encounter is to curate and manage all types of docs, e.g.: * email chains * teams recording transcripts * confluence pages * pdf manuscripts
These can be ever-evolving and may hook up with periodic delta updates, manual sync, add/remove, etc. And I'm trying to figure out if there's a way to manage these docs/texts properly. Basically, I think I would need a system to store these files, their metadata, etc, and provide a web UI for people to manage them. Then these blob of texts will go through frameworks like langchain/LlamaIndex and be cleaned/chunked into vector db, and different chunking strategies can be A/B tested while other people maintain this ever-growing docs system.
Any suggestions are welcomed. I've tried some all-in-one frameworks but so far my experience are lackluster. Also, my company due to compliance constraints cannot use cloud-based solutions, so it has to be either open-source local-deployed, or developed locally.
Comments
Did you try dify? I found it was a good beginning for me.
https://dify.ai/
Yes. There're a couple of problems I encountered during my trial and ended up choose to build my own: 1. the version I was using requires the configuration of a 'reranking model', which AFAIK has to be from cohere. Funny enough, I can later opt to not use this reranking model, but I cannot get any functionality working if I don't preconfigure the cohere API key for this model. The problem with this is that, I need to request my company's infosec to allow API call onto cohere, for a functionality that I didn't intend to use.
And I think this results in my observation of the startup llm app world vs the heavily-regulated finance industry world: that the startup llm world is taking a default opt-in, default saas mode; however the regulated clients cannot benefit from these practices due to arcane regulation requirements. In this case, dify requires the setup of an API key for a service that I don't intend to use, and I have to go through the hurdles just to get the hello world going.
My analysis is that, there are 2 ways to make the heavily-regulated clients happy: 1. major cloud provider integrating everything into the myriad of its service catalogs and offer a full suite of services with the sales and compliance completed as a whole. 2. startup go down a curated, opinionated, battery-included oss approach with the least of sass fuss.
2. for some reason when I test using the simplest QA template, it seems dify cannot retrieve the document of any kind that I uploaded. I don't know why it happened but debugging the simplest offering of an app that boasts ease of use is not my thing.
I didn't go further from there. I wish my experience is just an extremely rare case since dify definitely seems to be the most polished all-in-one provider out there. If my experience turned out to be more positive I'll definitely stick with it, now I just feel picking a all-in-one is not the right choice.
you own it dude? Damn crazy good UI