I'd bet that still to this day, most interview questions test recall (of XYZ) and recall is what LLMs are best at. "Explain the difference between X and Y" or "what would you do in scenario Z" are search queries now. A lot more streamlined than individual google searches people would patch together from their search journey...
If I'd ever be doing a hiring spree, I'd give them a real problem. Put them in a position to think out loud. Not "what's the answer" but "walk me through how you'd approach this." Probe the reasoning. When they hit a fork, ask why they'd go left instead of right. Ask about tradeoffs they see. The best/real candidates will naturally say "I'd need to know more about X before deciding" because they're actually thinking through it.
IMO acknowledging the lack of information (without coming from a place of uninformed/non-experienced) almost impossible to fake with an overlay. LLMs always have an answer. Good engineers know when they don't.
Comments
I'd bet that still to this day, most interview questions test recall (of XYZ) and recall is what LLMs are best at. "Explain the difference between X and Y" or "what would you do in scenario Z" are search queries now. A lot more streamlined than individual google searches people would patch together from their search journey...
If I'd ever be doing a hiring spree, I'd give them a real problem. Put them in a position to think out loud. Not "what's the answer" but "walk me through how you'd approach this." Probe the reasoning. When they hit a fork, ask why they'd go left instead of right. Ask about tradeoffs they see. The best/real candidates will naturally say "I'd need to know more about X before deciding" because they're actually thinking through it.
IMO acknowledging the lack of information (without coming from a place of uninformed/non-experienced) almost impossible to fake with an overlay. LLMs always have an answer. Good engineers know when they don't.