Is there theoretically a method to have an LLM to not hallucinate and just say “I don’t know” or just answer questions based on a certain domain of knowledge which it is well trained on?

That is a very good question. What Google returns for "LLM hallucination" mostly describes post-processing hacks to detect the problem, or pre-processing hacks to guide the LLM into using data directly relevant to the question asked. Not doing something to the core LLM system to get some measure of confidence out.

Anyone working on this?