I did (as an interviewer) a lot of interviews last year, and I was always pretty explicit about allowing the candidate to use “whatever tools you'd use in real life, including copilot and ChatGPT”. This has two big advantages:
- if someone not using them is less performant than someone using them, I should definitely hire the later, because mastering the use of these tools is an important skill, like knowing your key bindings or using a proper IDE and debugger.
- it forces you to design the interview in a way that isn't trivially solved by an AI, and challenges the actual skills of the interviewees. Of course it means the interview cannot be given to brainless hiring managers or HR, but must be handled by software team leaders/managers themselves, or at least senior developers.
You can do point 2 alone, skipping the nonsense of point 1. Please respect the time of the people that are interviewing and don't push your "innovative" ideas about interviewing on them. You are interviewing, not doing experiments.
skipping the nonsense of point 1. Please respect the time of the people that are interviewing and don't push your "innovative" ideas about interviewing on them. You are interviewing, not doing experiments.
I have to admit I have no idea what you're complaining about. I've never treated the interviewees like they were experiment subjects and I don't see where you are reading that in “point 1”…
All I'm saying in the first bullet point is that weather you like them or not, these tools are here to stay if they improve developers productivity and it would make no sense not to take the mastery of a tool as an asset when evaluating a candidate (like if someone is showing that they are proficient with git or a debugger during the interview question, that's a good thing, if they show they are proficient with Copilot or whatever AI-based tool then it's also a valuable skill that you should have in mind when making your evaluation. That's it. There's nothing here about making people lose their time, experimenting with them or whatever.
because mastering the use of these tools is an important skill like
Personal opinion.
if they show they are proficient with Copilot or whatever AI-based tool then it's also a valuable skill that you should have in mind when making your evaluation.
Again, this is your personal opinion. I don't even consider this being a skill. Even less than "knowing IDE bindings". What is important during interviews is verifying that someone is at least able to deliver, troubleshoot and will not be a nuisance to the rest of the team (loosely speaking). Can they use AI tools effectively to speed up development fixing undesirable artifacts? Good, but not something that matter to evaluate if someone should be hired or not. Also, if you want to evaluate how they solve problems, using AI during interview will introduce a huge amount of noise.
> because mastering the use of these tools is an important skill like
Personal opinion.
If those tools improve your productivity (it's highly dependent on what kind of job you're doing tbh) then it's not a matter of personal opinion: all things equal your employer will always value a more productive employee over a least productive one. Refusing for personal reasons to use a particular kind of tools that would increase your productivity is fine, but then don't be shocked if employers favor other candidates!
What is important during interviews is verifying that someone is at least able to deliver, troubleshoot
Yes, and if his use of such a tool improves his ability to do so, then it is actually an asset!
and will not be a nuisance to the rest of the team (loosely speaking)
Indeed, and I discussed that [0].
Can they use AI tools effectively to speed up development fixing undesirable artifacts? Good, but not something that matter to evaluate if someone should be hired or not.
It's not the use of AI itself that's worth being hired, but the overall productivity: if he is more productive with the AI tool than you without it, I'll hire him every time (assuming his non-tech skills are at least equivalent to yours, indeed).
Also, if you want to evaluate how they solve problems, using AI during interview will introduce a huge amount of noise.
In your job you either have problems that are out of reach of current LLM tech (very likely today, as they aren't that smart) in which case exhibiting such a problem to ask in interview should not be hard[1], or you don't and then you don't need to have employees capable of solving such problems.
Keep in mind that for most part your company and the job you're offering are very likely boring anyway, and you don't need to have a team of genius onboard. You just need decent human being that are able to cooperate, with just good enough technical abilities to solve the problem they are facing. And it's actually good if an IA can lower the technical skill cap, so that you can focus on the human part (we're clearly not there yet, as IMHO the current tech mostly improve the productivity of people solid enough to catch all the bullshit they spit out every other answer, hidden among helpful stuff).
[1] My interview was built around a fairly basic concurrent program problem, which is something we were using all the time and at the same time something ChatGPT 4 really struggles at without being heavily nudged, and when you make it correct its mistake it introduces bugs that where there before.
This + show me tools you are using. Like show me how you use copilot or even what you would search on stackoverflow for.
A surprising number of candidates would fail even the "I can use my own IDE" part of this exercise. It is still the same thing after all - show me what you have in your toolbox and how well you can use those tools.
So in general it means giving harder and harder interviews as the AI gets better. Can you give me an example of what you ask that ChatGPT is unable to solve?
No, it has nothing to do with giving harder interviews, but more about giving more realistic ones, because as you might have noticed, ChatGPT is still quite far from being an autonomous agent able to solve realistic tasks end to end.
And if you manage to solve realistic problems using ChatGPT or equivalent, then you certainly are a good enough candidate for the job, which consists of solving real world problems no matter how you do it.
But at this point it's very clear that most people using ChatGPT cannot, so it's a good sign that the candidate in themselves have the added value the company is looking for.
Then again there are other aspects to being a good candidate besides the ability to solve problems, like how you behave in a group and how you communicate, but these are also things you can see in an interview and for which an LLM isn't going to help (And god I wish it could help them on that too, because the number of antisocial people or sociopath having harmful effect on organizations is too damn high).
Comments
I did (as an interviewer) a lot of interviews last year, and I was always pretty explicit about allowing the candidate to use “whatever tools you'd use in real life, including copilot and ChatGPT”. This has two big advantages:
- if someone not using them is less performant than someone using them, I should definitely hire the later, because mastering the use of these tools is an important skill, like knowing your key bindings or using a proper IDE and debugger.
- it forces you to design the interview in a way that isn't trivially solved by an AI, and challenges the actual skills of the interviewees. Of course it means the interview cannot be given to brainless hiring managers or HR, but must be handled by software team leaders/managers themselves, or at least senior developers.
You can do point 2 alone, skipping the nonsense of point 1. Please respect the time of the people that are interviewing and don't push your "innovative" ideas about interviewing on them. You are interviewing, not doing experiments.
I have to admit I have no idea what you're complaining about. I've never treated the interviewees like they were experiment subjects and I don't see where you are reading that in “point 1”…
All I'm saying in the first bullet point is that weather you like them or not, these tools are here to stay if they improve developers productivity and it would make no sense not to take the mastery of a tool as an asset when evaluating a candidate (like if someone is showing that they are proficient with git or a debugger during the interview question, that's a good thing, if they show they are proficient with Copilot or whatever AI-based tool then it's also a valuable skill that you should have in mind when making your evaluation. That's it. There's nothing here about making people lose their time, experimenting with them or whatever.
Personal opinion.
Again, this is your personal opinion. I don't even consider this being a skill. Even less than "knowing IDE bindings". What is important during interviews is verifying that someone is at least able to deliver, troubleshoot and will not be a nuisance to the rest of the team (loosely speaking). Can they use AI tools effectively to speed up development fixing undesirable artifacts? Good, but not something that matter to evaluate if someone should be hired or not. Also, if you want to evaluate how they solve problems, using AI during interview will introduce a huge amount of noise.
If those tools improve your productivity (it's highly dependent on what kind of job you're doing tbh) then it's not a matter of personal opinion: all things equal your employer will always value a more productive employee over a least productive one. Refusing for personal reasons to use a particular kind of tools that would increase your productivity is fine, but then don't be shocked if employers favor other candidates!
Yes, and if his use of such a tool improves his ability to do so, then it is actually an asset!
Indeed, and I discussed that [0].
It's not the use of AI itself that's worth being hired, but the overall productivity: if he is more productive with the AI tool than you without it, I'll hire him every time (assuming his non-tech skills are at least equivalent to yours, indeed).
In your job you either have problems that are out of reach of current LLM tech (very likely today, as they aren't that smart) in which case exhibiting such a problem to ask in interview should not be hard[1], or you don't and then you don't need to have employees capable of solving such problems.
Keep in mind that for most part your company and the job you're offering are very likely boring anyway, and you don't need to have a team of genius onboard. You just need decent human being that are able to cooperate, with just good enough technical abilities to solve the problem they are facing. And it's actually good if an IA can lower the technical skill cap, so that you can focus on the human part (we're clearly not there yet, as IMHO the current tech mostly improve the productivity of people solid enough to catch all the bullshit they spit out every other answer, hidden among helpful stuff).
[0] here https://news.ycombinator.com/item?id=40364095
[1] My interview was built around a fairly basic concurrent program problem, which is something we were using all the time and at the same time something ChatGPT 4 really struggles at without being heavily nudged, and when you make it correct its mistake it introduces bugs that where there before.
This + show me tools you are using. Like show me how you use copilot or even what you would search on stackoverflow for.
A surprising number of candidates would fail even the "I can use my own IDE" part of this exercise. It is still the same thing after all - show me what you have in your toolbox and how well you can use those tools.
This, exactly. Thank you for wording it much better than I did.
So in general it means giving harder and harder interviews as the AI gets better. Can you give me an example of what you ask that ChatGPT is unable to solve?
No, it has nothing to do with giving harder interviews, but more about giving more realistic ones, because as you might have noticed, ChatGPT is still quite far from being an autonomous agent able to solve realistic tasks end to end.
And if you manage to solve realistic problems using ChatGPT or equivalent, then you certainly are a good enough candidate for the job, which consists of solving real world problems no matter how you do it.
But at this point it's very clear that most people using ChatGPT cannot, so it's a good sign that the candidate in themselves have the added value the company is looking for.
Then again there are other aspects to being a good candidate besides the ability to solve problems, like how you behave in a group and how you communicate, but these are also things you can see in an interview and for which an LLM isn't going to help (And god I wish it could help them on that too, because the number of antisocial people or sociopath having harmful effect on organizations is too damn high).