I wish more companies would have interviewees conduct code reviews. Code reviews as an interview show a number of things you wouldn’t get from a typical interview—what opinions they have, what are the things they call out vs what they don’t waste time on, how they might communicate with another teammate, and more. And if we’re going to a world where AIs do much of the work and we just need to check that they implemented what we intended, those code review skill will still be highly relevant.
The objective is not finding someone good at reviews, or even that gets a specific review "right". It's having a chance to understand how someone thinks, better evaluate their actual knowledge and test for some important red flags.
gpt-4 is only mediocre at many things, and i've found mistakes in its code reviews (not, to be sure, mistakes i couldn't have made), but one thing it's absolutely superhuman at is avoiding red flags
gpt-4 is only mediocre at many things, and i've found mistakes in its code reviews
> gpt-4 is at least as good as i am at code reviews, so i don't think this solves the problem this post is about
These two comments don't seem consistent.
Honestly, "i've found mistakes in its code reviews" vs "it's absolutely superhuman at is avoiding red flags" is possibly not self-consistent. But I think you mean glaring mistakes?
Which if I understand you correctly, than I'm not sure how you get the first conclusion unless you are saying you're mediocre (which is fine). But really it just makes it seem like you and the machine complement one another, rather than compete, which makes me not understand the original comment in context.
But I mostly agree with Copenjin. Interviews are less about on the spot skill checks so much as learning how someone thinks and problem solves. Honestly, you could play a boardgame/videogame/cardgame with them and it would be an effective interview (asking them relevant questions during the game, and maybe even better if it's a relatively unknown game so they can zero shot it). The reason for this is that in real world work you are more concerned with how someone adapts to changing environments and thinks through situations. To see when they'll ask for help, what they might get stuck on, and how they strategize.
Your employees will always be gaining new skills. And honestly, it is easier to take a lower skilled person who's more adaptable and driven and turn them into a great employee than it is to take someone who's got skills but will stagnate. But ymmv depending on the job and requirements. Sometimes you just need to fill a seat.
code reviews are not one of the things gpt-4 is mediocre at; it's much better at them than it is at, for example, providing reliable information or writing code. its pattern of strengths and weaknesses in code reviews is not the same as human patterns
by 'red flags' i inferred copenjin to be referring to things like getting aggressive or defensive, rather than making dumb mistakes, but i could be wrong about that. i guess there are also some mistakes that are so dumb that they'd be a red flag, and i have to admit that gpt-4 is somewhat subhuman at avoiding those
if you play a board game with someone you can assess their general intelligence and capacity for logic. all else being equal, having more general intelligence and knowing how to think logically do make you a better programmer. (if you just want to assess general intelligence, a much faster pair of tests would be reverse digit span and reaction time.) but those are far from the only things that matter, they're not enough to be a great programmer, and they're not even among the most important factors. other important factors in programming include things like knowing how to program, knowing how to listen, and being willing to ask for help (and accept it), which a board game generally will not test
its pattern of strengths and weaknesses in code reviews is not the same as human patterns
I agree with you. Which is why I say that they complement. But your reply to Baron implies that what they were suggesting wasn't a solution. I agree with the sentiment of the post to do things in person. But what I take from Baron is that it is much harder to fake the process with GPT because the actual part of the code review isn't so much about finding the bugs, it is you watching someone perform the code review (presumably through screen sharing and a video chat). You could have this completely virtual, but I think you're right to imply that the same task could be then optimized.
But at the end of the day, I think the underlying issue is that we're testing the wrong things. If GPT can do sufficient, then what do we need the human for? Well... the actual coding, logic, and nuance. So we need to really see how a human performs in those domains. Your interviewing process should adapt with the times. It is like having a calculus exam where you test someone and ban calculators but also include a lot of rote, mundane, and arduous arithmetic calculations. That isn't testing the material that the course is on and isn't making anyone a better mathematician, because any mathematician in the wild will still use a calculator (mathematicians and physicists are often far more concerned with symbolic manipulation than numerals).
other important factors in programming include things like knowing how to program, knowing how to listen, and being willing to ask for help (and accept it), which a board game generally will not test
I agree the game won't help with the first part. But I thought I didn't need to explicitly state that you should also require a resume and ask for a github if they have one. But I did explicitly say you should ask relevant questions. And I'm not entirely convinced on the latter, which are difficult skills to check for under any setting. There's a large set of collaborative games in which do require working and thinking as a team. I was really just throwing a game out there as a joke, being more a stand-in for an arbitrary setting.
At the end of the day, interviewing is a hard process and there are no clear cut solutions. But hey, we had this conversation literally a week ago: https://news.ycombinator.com/item?id=40291828
I'm not sure what this comment is supposed to imply?
Are you bad at code reviews? Is the code you're reviewing fairly standard?
GPT misses nuance. It can't reason while you still can. It certainly can do certain tasks better than you but certainly humans can do better at other tasks (specifically in the creativity side, logic, and when it comes to nuanced thinking). But if you're always focused on being quick (quantity over quality) then yeah, I think GPT could replace you. Otherwise, I don't know how anyone comes to this conclusion.
I've seen it notice logic errors in proprietary non-standard code that a human missed. It may not be able to literally "reason" through your code but it can follow the logic pretty well. I've even been able to have it pretty accurately comment the "intent" (vs. function) or spaghetti code with some reasonable accuracy.
i think i'm better than the average senior developer at code reviews, but of course your reference point for the average senior developer may be different from mine. if you'd like to find out, we can schedule a jitsi call where i review some of your code, and you can post your assessment in this thread afterwards
gpt-4 makes stupid logic errors a lot, and i agree that sometimes it's bad at nuance, inappropriately applying heuristics in a context where they're inapplicable. it's much more creative than people are, though, and people also have those same flaws
I just recently did an interview with Gitlab and they do this. They have a codebase that is obviously not something they monetize or use themselves for anything other than interview purposes (for the ones worried about stolen labor). They make a PR for the codebase to add a new small feature, and assign you as a reviewer.
I quite liked it, though I wasn't a fan of how bad the PR is itself, since I ended up having way too many comments, some of them being massive "wtf is this and why would you ever do it this way?" Types of things that IRL I would've refused to review without a proper rewrite.
Had a job interview yesterday and in one part of it they showed me an existing class in their project and had me tell them what I think of it. What I thought was good and not so good about it
Yup. When I was doing a lot of engineering interviews I often used to give candidates a choice. Either a standard type of interview or if they had any public code (eg on their github) we could review together. It was a pleasant experience and gave really great signal.
Yeah - It's always a really interesting conversation. "Why did you do x rather than y?" etc. As an interviewer you have to be much more on your game to make sure you get the signal you need, but you're just talking one engineer to another and getting a sense for what they are like, how they deal with annoying/hard aspects of problems, what they find interesting to do etc. It's much better for those things than a standard programming interview.
The best interview I've ever had was when the guy just asked me to compare two technologies I had worked with. Obviously, the interviewer was also familiar with them. He was able to gauge my knowledge, my ability to choose the right tool for the job, my communication skills. Looking back, it really felt like one of those "we have to make a decision" meetings.
I did a code review style interview at a startup a few years ago (loved them; got an offer, but my then-current employer countered surprisingly well). I really enjoyed it.
This is great. I'm in DevOps and when hiring, the tech interview is a round we've been off and on over time. Doing code reviews instead makes a lot more sense.
I don't like code reviews. If need be I might try to bs my way (paychecks come quicker than permission to fire) into a coding job and those would be a showstopper.
Comments
I wish more companies would have interviewees conduct code reviews. Code reviews as an interview show a number of things you wouldn’t get from a typical interview—what opinions they have, what are the things they call out vs what they don’t waste time on, how they might communicate with another teammate, and more. And if we’re going to a world where AIs do much of the work and we just need to check that they implemented what we intended, those code review skill will still be highly relevant.
gpt-4 is at least as good as i am at code reviews, so i don't think this solves the problem this post is about
The objective is not finding someone good at reviews, or even that gets a specific review "right". It's having a chance to understand how someone thinks, better evaluate their actual knowledge and test for some important red flags.
gpt-4 is only mediocre at many things, and i've found mistakes in its code reviews (not, to be sure, mistakes i couldn't have made), but one thing it's absolutely superhuman at is avoiding red flags
These two comments don't seem consistent.
Honestly, "i've found mistakes in its code reviews" vs "it's absolutely superhuman at is avoiding red flags" is possibly not self-consistent. But I think you mean glaring mistakes?
Which if I understand you correctly, than I'm not sure how you get the first conclusion unless you are saying you're mediocre (which is fine). But really it just makes it seem like you and the machine complement one another, rather than compete, which makes me not understand the original comment in context.
But I mostly agree with Copenjin. Interviews are less about on the spot skill checks so much as learning how someone thinks and problem solves. Honestly, you could play a boardgame/videogame/cardgame with them and it would be an effective interview (asking them relevant questions during the game, and maybe even better if it's a relatively unknown game so they can zero shot it). The reason for this is that in real world work you are more concerned with how someone adapts to changing environments and thinks through situations. To see when they'll ask for help, what they might get stuck on, and how they strategize.
Your employees will always be gaining new skills. And honestly, it is easier to take a lower skilled person who's more adaptable and driven and turn them into a great employee than it is to take someone who's got skills but will stagnate. But ymmv depending on the job and requirements. Sometimes you just need to fill a seat.
code reviews are not one of the things gpt-4 is mediocre at; it's much better at them than it is at, for example, providing reliable information or writing code. its pattern of strengths and weaknesses in code reviews is not the same as human patterns
by 'red flags' i inferred copenjin to be referring to things like getting aggressive or defensive, rather than making dumb mistakes, but i could be wrong about that. i guess there are also some mistakes that are so dumb that they'd be a red flag, and i have to admit that gpt-4 is somewhat subhuman at avoiding those
if you play a board game with someone you can assess their general intelligence and capacity for logic. all else being equal, having more general intelligence and knowing how to think logically do make you a better programmer. (if you just want to assess general intelligence, a much faster pair of tests would be reverse digit span and reaction time.) but those are far from the only things that matter, they're not enough to be a great programmer, and they're not even among the most important factors. other important factors in programming include things like knowing how to program, knowing how to listen, and being willing to ask for help (and accept it), which a board game generally will not test
I agree with you. Which is why I say that they complement. But your reply to Baron implies that what they were suggesting wasn't a solution. I agree with the sentiment of the post to do things in person. But what I take from Baron is that it is much harder to fake the process with GPT because the actual part of the code review isn't so much about finding the bugs, it is you watching someone perform the code review (presumably through screen sharing and a video chat). You could have this completely virtual, but I think you're right to imply that the same task could be then optimized.
But at the end of the day, I think the underlying issue is that we're testing the wrong things. If GPT can do sufficient, then what do we need the human for? Well... the actual coding, logic, and nuance. So we need to really see how a human performs in those domains. Your interviewing process should adapt with the times. It is like having a calculus exam where you test someone and ban calculators but also include a lot of rote, mundane, and arduous arithmetic calculations. That isn't testing the material that the course is on and isn't making anyone a better mathematician, because any mathematician in the wild will still use a calculator (mathematicians and physicists are often far more concerned with symbolic manipulation than numerals).
I agree the game won't help with the first part. But I thought I didn't need to explicitly state that you should also require a resume and ask for a github if they have one. But I did explicitly say you should ask relevant questions. And I'm not entirely convinced on the latter, which are difficult skills to check for under any setting. There's a large set of collaborative games in which do require working and thinking as a team. I was really just throwing a game out there as a joke, being more a stand-in for an arbitrary setting.
At the end of the day, interviewing is a hard process and there are no clear cut solutions. But hey, we had this conversation literally a week ago: https://news.ycombinator.com/item?id=40291828
I'm not sure what this comment is supposed to imply?
Are you bad at code reviews? Is the code you're reviewing fairly standard?
GPT misses nuance. It can't reason while you still can. It certainly can do certain tasks better than you but certainly humans can do better at other tasks (specifically in the creativity side, logic, and when it comes to nuanced thinking). But if you're always focused on being quick (quantity over quality) then yeah, I think GPT could replace you. Otherwise, I don't know how anyone comes to this conclusion.
I think you may be discounting GTP.
I've seen it notice logic errors in proprietary non-standard code that a human missed. It may not be able to literally "reason" through your code but it can follow the logic pretty well. I've even been able to have it pretty accurately comment the "intent" (vs. function) or spaghetti code with some reasonable accuracy.
i think i'm better than the average senior developer at code reviews, but of course your reference point for the average senior developer may be different from mine. if you'd like to find out, we can schedule a jitsi call where i review some of your code, and you can post your assessment in this thread afterwards
gpt-4 makes stupid logic errors a lot, and i agree that sometimes it's bad at nuance, inappropriately applying heuristics in a context where they're inapplicable. it's much more creative than people are, though, and people also have those same flaws
For now, you can easily get around this by showing the code via screen share.
You mean, the interviewer does the screen share, and the candidate should review whatever they see on the screen?
"Can you scroll back a bit? Now, can you show me the docstrings for Foo.detachBar and Bar.attachFoo?"
Yeah. Not a great experience.
can you? couldn't the candidate just use ocr to feed the code to gpt-4?
“Before I answer, let me scroll through the entire pull request to make sure all the code has been made visible.”
I just recently did an interview with Gitlab and they do this. They have a codebase that is obviously not something they monetize or use themselves for anything other than interview purposes (for the ones worried about stolen labor). They make a PR for the codebase to add a new small feature, and assign you as a reviewer.
I quite liked it, though I wasn't a fan of how bad the PR is itself, since I ended up having way too many comments, some of them being massive "wtf is this and why would you ever do it this way?" Types of things that IRL I would've refused to review without a proper rewrite.
Had a job interview yesterday and in one part of it they showed me an existing class in their project and had me tell them what I think of it. What I thought was good and not so good about it
I've been doing this for more than a decade, can confirm that it gives very high quality signals.
Yup. When I was doing a lot of engineering interviews I often used to give candidates a choice. Either a standard type of interview or if they had any public code (eg on their github) we could review together. It was a pleasant experience and gave really great signal.
I would LOVE someone to roast my code in real-time!
Yeah - It's always a really interesting conversation. "Why did you do x rather than y?" etc. As an interviewer you have to be much more on your game to make sure you get the signal you need, but you're just talking one engineer to another and getting a sense for what they are like, how they deal with annoying/hard aspects of problems, what they find interesting to do etc. It's much better for those things than a standard programming interview.
The best interview I've ever had was when the guy just asked me to compare two technologies I had worked with. Obviously, the interviewer was also familiar with them. He was able to gauge my knowledge, my ability to choose the right tool for the job, my communication skills. Looking back, it really felt like one of those "we have to make a decision" meetings.
I did a code review style interview at a startup a few years ago (loved them; got an offer, but my then-current employer countered surprisingly well). I really enjoyed it.
This is great. I'm in DevOps and when hiring, the tech interview is a round we've been off and on over time. Doing code reviews instead makes a lot more sense.
I don't like code reviews. If need be I might try to bs my way (paychecks come quicker than permission to fire) into a coding job and those would be a showstopper.