Skip to content

Comment on Some critical issues with the SWE-bench datasetparent

Comments

You might not be getting gaslit.

I’ve sat through some interviews recently with candidates who started their careers in the last 6 years or so… during the boom cycle. Some were quite good but a troubling amount were clearly over-leveled at their current/previous employers.

For example, last month we interviewed someone for a Staff Engineering role (current role: L5 Senior II engineer), for Python. This person was unable to explain what a set was in Python, didn’t seem to grok the basic HTTP request/response pattern etc. This wasn’t a leetcode interview; it was an engineering conversation. It was the same questions we’d given dozens and dozens engineers in the past. It wasn’t a language barrier issue (guy was American, interviewer was American). Dude just seemed to have a very very narrow set of skills.

For people like this I imagine AI feels like a superpower.

I'm pretty sure that's what's going on too. The quality of junior -> midlevel engineers has plummeted and these AI tools have been a major crutch to help them appear productive/competent again.

Problem is they don't know enough to really assess if what the LLM is spitting out is any good or not so they claim amazing wins.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.