I get the premise , but part of me asks : Well, the candidate was able to come up with an answer, why does it matter how they did it?
Presumably these questions are rooted in some kind of developer reality, they're asked to gauge technical expertise and suitability.. what's the real difference between someone with a magic box that gives them the right answers versus someone who can derive the answers themselves if the questions suitably emulate real life scenarios?
why should a company care about 'natural' problem solvers aside from the context of IP ownership and so on?
It sounds like it really just turns application questions into a voight-kampf test for no real good reason when the real point is to ascertain whether or not a candidate can get the job done.
I think this kind of stuff is just a gut reaction from a humanity that realizes that it's not the cleverness of the interview questions that are the problem, it's that machine tools are now at near human levels in the majority of mundane stuff a developer does every day -- the reactions are born from a panicked realization that it no longer makes sense to employ the lower end of the developer skill spectrum.
Presumably these questions are rooted in some kind of developer reality
Have you ever taken one of these interviews? One of the primary criticisms of the practice is how disconnected they are from the realities of the job itself, requiring studying specifically for the interview process to actually pass.
After avoiding similar companies for years, a couple of years ago, out of curiosity, I took a call from Meta, and they wanted to send me a reading list of books before the first interview.
I absolutely hate any homework a firm wants to impose on me. I’m usually juggling multiple interviews and don’t have time for that (especially when they’ll just ghost you at the end).
But I get it. It’s so expensive for everyone involved if you hire the wrong person and need to correct it. Asking a candidate to put more work up front seems reasonable when the offer can easily be worth $250k or more for what is essentially a day of zoom calls.
What’s a better way? Hire as a contractor and convert to full time after 6mo of performance evals? That’s also problematic.
I get the premise , but part of me asks : Well, the candidate was able to come up with an answer, why does it matter how they did it?
I might be able to pass the bar / medical exams with ChatGPT 4. And even if not, I will be able to with ChatGPT 5/6 etc. I have no knowledge at all of medicine or law would you want me as your doctor/lawyer?
I've interviewed people who were employed in senior developer positions in fairly large companies who could not write code. I don't mean they wrote bad code, I mean that I could not find a way of promoting them to plexplain even basic programming concepts or write a single line of code.
They're not near the developers you want to hire. But they are better at what should be done than some people who have developer jobs.
Some people freeze during interviews, like complete mental shutdown. And it's a downward spiral, once you start freezing it gets worse and then even simple questions become impossible.
That, or you were dealing with professional liars.
For what it's worth, I probably have given off this vibe in interviews in the past, despite being (IMO) a decent programmer.
My brain just shuts off during some interviews and I can't recall even the simplest of things. I forgot the name for ternaries in one of my interviews, despite these being things I use more or less daily.
I'm not sure what it is really. I do fine in exams/tests. I don't have any kind of anxiety on the job or otherwise. I don't even really feel anxious during interviews either, but my brain just goes poof and I can barely form a coherent sentence anymore out of the blue, never understood it.
When people shut down and struggle to answer, that is very noticeable. If that happened, I'd be asking about their wellbeing and see if there's alternative ways for us to assess.
My problem is candidates that keep being able to talk in ways that to a non-technical manager would sound as if they plausibly know their stuff, but then struggle to offer up any kind of detail when you dig into specifics, while still being articulate yet not giving me any reasons as their answers remain superficially coherent yet descends into technobabble.
I've had candidates telling me they've gone blank. That's fine - I'll find something else to ask about, dial it back, slowly circle back to the problem from another angle, and if I'm fine with everything else we'll discuss e.g. giving them a problem to solve without me there, outside the interview setting. I've had people similarly go blank in front of a whiteboard. That's fine - I ask them to forget about that, and talk through the problem instead.
The candidates I consider to be unable to code are not the ones who freeze up or go incoherent but can pass some other assessment, but the ones that are "confidently wrong" from a fairly high level, and you drill down until they can't explain a simple if statement, while often still talking with apparent confidence.
Maybe I'm sometimes wrong and they're just too anxious to admit to finding it challenging. But if so, that's a much bigger problem to me than if they're struggling with the interview setting more generally.
Measuring soft skills and knowing how to hire the people who have them when your organization needs them is a different problem imo. Obviously there are problems with giving guys with decades of experience graph theory problems when the problems you want to solve are more abstract and organizational.
We're not talking complex problems here, because I firmly believe those kinds of problems do not belong in interviews as coding problems. Maybe spoken about as higher level problem solving. We're talking checking very basic stuff and stressing to them I couldn't care less about whether the syntax is correct. Usually we'd work our way "down" to those kinds of problems once I started getting suspicious about their abilities based on higher level answers.
I think you're actually pointing towards a bigger issue here, which is that chatgpt can help poor candidates get over the interview process and into your org despite not knowing what they're doing. There's also a deeper commentary where we discuss how the industry is completely unwilling to train eager people and how training costs are largely absorbed by the candidate before they start working.
Two thoughts on this. The first is honesty - I don’t mind if candidates google the question if they tell me about it. Information retrieval is a vital part of the software engineering job. What I take issue with is dishonesty, a trait that would not be something I’d accept in my teams.
The other thought is this: interview questions aim to collect evidence for skills needed for the job in a setting where you can’t observe these skills directly. I had a candidate for a senior engineering position once who clearly used an LLM. They had a vague understanding about what latency is, had no idea about SLOs, P99 or circuit breakers. The LLM then helped them to parrot this knowledge to me. But I didn’t want to hire them because of their theoretical knowledge - they need to monitor systems, act on alerting, build and improve existing monitoring and maybe join the 24x7 rotation. These are all scenarios in which it’s insufficient to know where to look for information given a keyword. The work reality doesn’t provide them always with something they can easily put into an LLM. Even if it did, progress would be too slow if for any incident they’d need to consult their llm first.
Could even have more than one mistake per question.
As an aside, I really hate the 'art' of US essay writing. The AI has now learned to emulate this equivocating waffle that sits somewhere between not wrong and not even wrong. I do wonder now - with better tools available could we instead teach people to first act as an editor for other peoples content, perhaps even AI content. In effect have the people act as the discriminator in the GAN. I think it might be a faster and more thorough way of learning that embraces the help of AI while ditching the awful equivocating waffle that is currently being taught.
Comments
I get the premise , but part of me asks : Well, the candidate was able to come up with an answer, why does it matter how they did it?
Presumably these questions are rooted in some kind of developer reality, they're asked to gauge technical expertise and suitability.. what's the real difference between someone with a magic box that gives them the right answers versus someone who can derive the answers themselves if the questions suitably emulate real life scenarios?
why should a company care about 'natural' problem solvers aside from the context of IP ownership and so on?
It sounds like it really just turns application questions into a voight-kampf test for no real good reason when the real point is to ascertain whether or not a candidate can get the job done.
I think this kind of stuff is just a gut reaction from a humanity that realizes that it's not the cleverness of the interview questions that are the problem, it's that machine tools are now at near human levels in the majority of mundane stuff a developer does every day -- the reactions are born from a panicked realization that it no longer makes sense to employ the lower end of the developer skill spectrum.
Have you ever taken one of these interviews? One of the primary criticisms of the practice is how disconnected they are from the realities of the job itself, requiring studying specifically for the interview process to actually pass.
After avoiding similar companies for years, a couple of years ago, out of curiosity, I took a call from Meta, and they wanted to send me a reading list of books before the first interview.
I absolutely hate any homework a firm wants to impose on me. I’m usually juggling multiple interviews and don’t have time for that (especially when they’ll just ghost you at the end).
But I get it. It’s so expensive for everyone involved if you hire the wrong person and need to correct it. Asking a candidate to put more work up front seems reasonable when the offer can easily be worth $250k or more for what is essentially a day of zoom calls.
What’s a better way? Hire as a contractor and convert to full time after 6mo of performance evals? That’s also problematic.
I might be able to pass the bar / medical exams with ChatGPT 4. And even if not, I will be able to with ChatGPT 5/6 etc. I have no knowledge at all of medicine or law would you want me as your doctor/lawyer?
possibly in some cases companies think it is better to employ programmers (or managers, doctors, lawyers, accountants, etc.) than prompt engineers
If you think the machine tools are near human level, you're why technical interviews are still in use.
I've interviewed people who were employed in senior developer positions in fairly large companies who could not write code. I don't mean they wrote bad code, I mean that I could not find a way of promoting them to plexplain even basic programming concepts or write a single line of code.
They're not near the developers you want to hire. But they are better at what should be done than some people who have developer jobs.
Some people freeze during interviews, like complete mental shutdown. And it's a downward spiral, once you start freezing it gets worse and then even simple questions become impossible. That, or you were dealing with professional liars.
For what it's worth, I probably have given off this vibe in interviews in the past, despite being (IMO) a decent programmer.
My brain just shuts off during some interviews and I can't recall even the simplest of things. I forgot the name for ternaries in one of my interviews, despite these being things I use more or less daily.
I'm not sure what it is really. I do fine in exams/tests. I don't have any kind of anxiety on the job or otherwise. I don't even really feel anxious during interviews either, but my brain just goes poof and I can barely form a coherent sentence anymore out of the blue, never understood it.
When people shut down and struggle to answer, that is very noticeable. If that happened, I'd be asking about their wellbeing and see if there's alternative ways for us to assess.
My problem is candidates that keep being able to talk in ways that to a non-technical manager would sound as if they plausibly know their stuff, but then struggle to offer up any kind of detail when you dig into specifics, while still being articulate yet not giving me any reasons as their answers remain superficially coherent yet descends into technobabble.
I've had candidates telling me they've gone blank. That's fine - I'll find something else to ask about, dial it back, slowly circle back to the problem from another angle, and if I'm fine with everything else we'll discuss e.g. giving them a problem to solve without me there, outside the interview setting. I've had people similarly go blank in front of a whiteboard. That's fine - I ask them to forget about that, and talk through the problem instead.
The candidates I consider to be unable to code are not the ones who freeze up or go incoherent but can pass some other assessment, but the ones that are "confidently wrong" from a fairly high level, and you drill down until they can't explain a simple if statement, while often still talking with apparent confidence.
Maybe I'm sometimes wrong and they're just too anxious to admit to finding it challenging. But if so, that's a much bigger problem to me than if they're struggling with the interview setting more generally.
Measuring soft skills and knowing how to hire the people who have them when your organization needs them is a different problem imo. Obviously there are problems with giving guys with decades of experience graph theory problems when the problems you want to solve are more abstract and organizational.
We're not talking complex problems here, because I firmly believe those kinds of problems do not belong in interviews as coding problems. Maybe spoken about as higher level problem solving. We're talking checking very basic stuff and stressing to them I couldn't care less about whether the syntax is correct. Usually we'd work our way "down" to those kinds of problems once I started getting suspicious about their abilities based on higher level answers.
How can it be that some seniors cannot write code and have worked at some large companies? It seems so weird but I have hard similar stuff before
I've had the misfortune of working with someone who wasn't even at the level of the original ChatGPT release, a decade before it came out.
And more recently some interns, also (but more forgivably) below that level.
It's certainly raised the minimum necessary standard beyond some humans.
I think you're actually pointing towards a bigger issue here, which is that chatgpt can help poor candidates get over the interview process and into your org despite not knowing what they're doing. There's also a deeper commentary where we discuss how the industry is completely unwilling to train eager people and how training costs are largely absorbed by the candidate before they start working.
Two thoughts on this. The first is honesty - I don’t mind if candidates google the question if they tell me about it. Information retrieval is a vital part of the software engineering job. What I take issue with is dishonesty, a trait that would not be something I’d accept in my teams.
The other thought is this: interview questions aim to collect evidence for skills needed for the job in a setting where you can’t observe these skills directly. I had a candidate for a senior engineering position once who clearly used an LLM. They had a vague understanding about what latency is, had no idea about SLOs, P99 or circuit breakers. The LLM then helped them to parrot this knowledge to me. But I didn’t want to hire them because of their theoretical knowledge - they need to monitor systems, act on alerting, build and improve existing monitoring and maybe join the 24x7 rotation. These are all scenarios in which it’s insufficient to know where to look for information given a keyword. The work reality doesn’t provide them always with something they can easily put into an LLM. Even if it did, progress would be too slow if for any incident they’d need to consult their llm first.
Need to know that the dev knows when the AI is wrong.
A more direct solution would be to curate a list of mistakes made by the AI and see if the dev can correct them.
Even better would be a mixed bag of AI mistakes and correct code with no indication of which is which.
Could even have more than one mistake per question.
As an aside, I really hate the 'art' of US essay writing. The AI has now learned to emulate this equivocating waffle that sits somewhere between not wrong and not even wrong. I do wonder now - with better tools available could we instead teach people to first act as an editor for other peoples content, perhaps even AI content. In effect have the people act as the discriminator in the GAN. I think it might be a faster and more thorough way of learning that embraces the help of AI while ditching the awful equivocating waffle that is currently being taught.