/sigh interview techniques really are a perennial topic on HN.
> [a coding test] won't tell you squat about how good they are in a year long project with 10 other people,
The author misses the point of a coding test. It's a negative rather than positive filter. Someone who does amazing on FizzBuzz is not by definition an amazing programmer. Someone who can't solve FizzBuzz however almost certainly is not a good programmer.
Simple coding tests are a very effective filter in terms of the time spent.
> So why spend an inordinate amount of time on relatively minor parts of a programmer's skill set?
I wouldn't call half an hour "an inordinate amount of time".
The author then opines about how "can just tell" if someone is a good programmer or not. I can sympathize because frankly so can I. I went through a period of taking 10 interviewees to lunch. I asked them nothing technical and basically just answered their questions. From the first 10 minutes I could tell:
- 1 would probably get an offer (he did);
- 8 would not (they didn't); and
- 1 I was unsure about (no offer).
Looking at the author's profile [1] I believe I can see the problem: it doesn't appear he's ever worked for a large engineering organization. This is fairly obvious from the content of this post because none of his solutions scale.
Let's say you have a large organization with 5 Andrews. Each of them says to hire this guy they just interviewed. What do you do if you're looking for less than 5 people? Are they going to work on the respective teams of those giving the recommendation? If not, how do you know they're a good fit? You need to consider company-wise culture and expectations. How do you calibrate between them? Do they have the necessary foundation to do not just the job they'll be starting on but to grow with the organization, adapt and perhaps work on other projects?
The other problem I have is the "war stories" aspect. This is a very definite bias. Take the way human memory works. Imagine you have a conversation with someone that's memorable in some way. At first you can remember word-for-word what happened. That quickly fades and you remember the gist of what was said. You may even think you remember exactly what was said and how it was said but usually you're wrong (try it by writing down what was exactly said and going back to it after months or having two or more people recount the same event). After awhile even that made fade and you may just be left with an emotion about the event.
Some things I can remember very well but more often than not, I've learnt my lesson and the exact circumstances or even the origin entirely are lost. This is largely because it's useless information.
But here's the biggest problem of all with the post:
> My favorite idea is still contract to hire everyone after your (hopefully reasonable) interview;
Okay, you've excluded anyone really good because they're not going to jump through that hoop. I don't consider myself a "rockstar" and even I won't jump through that.
Maybe the real problem is the OP doesn't even know what good programmers are because this is a recipe for mediocrity.
Lastly there's a story about team bonding (probably distorted if not made up outright at a guess). The author doesn't seem to realize that his hiring process is pretty much about hiring people like himself, which is fine, but that's not the definition of a good programmer.
I've seen teams work well who:
- all work closely together and socialize together;
- never socialize together;
- work remotely;
- work on different schedules;
- and so on.
Don't confuse programming ability with culture fit and work style. Those are three different things all important.
In a large organization you're going to have some aspect of a common culture but very different team cultures (and even site cultures). Part of hiring in a large organization is recognizing that you're not trying to hire "you" and then working out how you can best use someone not like you such that they flourish.
EDIT: oh and if he thinks Google, as one example, hasn't done extensive data analysis on interview techniques he's nuts.
Interviewing is a perennial topic because it doesn't make much sense.
Case in point - over the past week I've interviewed for four positions. Three gave me offers within a day, with very little effort on my part. On the other hand, one told me that I had little idea of what I was doing and told me to study up and apply again next year. And, that was after wasting my time with 3.5 hours of interviews. Of the offers I received, two were well over 120k plus great bonus. The other came in under 90k but told me that I could work anywhere in the world, whenever I wanted, as long as I got my work done.
Over that same time, I flat out rejected three recruiters from large engineering companies because their recruitment processes would have cost me $700-1500 just in opportunity costs. And, they weren't even willing to give me the info I needed to help me determine my chance of success or even my expected pay.
Everyone these days seems to believe that their company is so important that they deserve or need a team of top 1000 engineers. When in fact a top 20-percent team who bonds well together would likely be sufficient.
The topic of interviewing is fascinating because there seems to be very little science behind it. Who knows what works?
The opportunity cost is the most frustrating aspect. I've solved several "sample" problems from companies, and been involved in full day interviews writing code on the whiteboard. The worse was spending a day writing some sample code, then doing a panel interview where I was asked to make a change to the code to cover a non-sensical edge case. I was rejected because I "hesitated".
People make several mistakes when analyzing interview techniques:
1. The interviewer and interviewee are after similar and complementary things.
Wrong. The interviewee is trying to find out if the work is interesting, whether or not this would be a good place to work, whether or not he or she could work in that environment with that team and so on.
The interviewer (or rather the employer) is trying to fill a position. This is really important.
Some make the mistake of thinking that if an interview process doesn't evaluate the interviewee accurately it has failed. This is grossly inaccurate. The point of an interview process shouldn't be to look at a single candidate but to look at the process of filling a position, which may well span interviewing many candidates.
A false positive (someone who looks qualified but isn't) for an employer can be incredibly costly. The cost of a false negative is essentially zero as long as the employer can otherwise adequately fill the position.
To spell it out: if an employer gets 50 applications, has phone interviews for 8, brings in 4 for face-to-face interviews and makes two offers, one of which is accepted, the employer has gained the desired result. The fact that a qualified person was rejected along the way (false negative) is essentially irrelevant.
2. Your worth is constant.
Wrong. You said it yourself. You got several different offers. You're implying that if you had been accurately gauged market conditions would have you valued roughly similarly.
Company A might be desperate. Company B less so. You may fill a niche far better at one employer than another. One employer may simply be more cashed up and able to pay a better salary. A given employer may simply suck at negotiating or be under a misconception about market value. The list goes on.
Likewise, your desirability to the employer factors into this and it goes beyond technical skills. If the employer thinks you'll be a great fit and they'd really like to work with you, that improves your worth (to them).
3. Companies are looking for the same thing.
Clearly this is wrong but I do see this attitude come up, typically being implied by showing mismatches in offers as "proof" or similar.
There are many examples of this. For example, all other things being equal I've found that an MIT graduate is much more likely to hire other MIT graduates. The same is true for Stanford, CMU or [insert school here].
Part of this is the "social proof" element (going to a great school and/or working for a top-tier employer can be a huge advantage). But more than that it comes down to cultural similarity, common background and being a known quantity (to some degree).
This is of course different for every employer.
The real problem with interviewing, particularly at larger organizations, is that people who are bad at it are doing it. Interviewing and assessing potential colleagues is a skill and a talent. Some people have it. Some don't.
I've seen another comment here that said you need great engineers to do interviewing. I disagree. Many great engineers seem to be essentially savants who are often ill-equipped for the social discourse entailed in interviewing.
To interview an engineer I firmly believe you need to be an engineer (the same goes for managing engineers) but you don't need to be a rock star. You just have the right additional skills.
The cost of a false negative is essentially zero as long as the employer can otherwise adequately fill the position.
I couldn't quantify them exactly, but I don't think the costs are zero. For one, any candidate that you bring in and reject consumes (in the case of my company) about 4 man hours of developer time. And that's mostly senior/lead developer time. And that's not counting the time we spend discussing the candidate and the general cost of a distraction/context-switch. And then there's the opportunity cost of not having a position filled and work being started. Sometimes the short run matters.
That said, I'd definitely agree that the scale should be tipped in favor of false negatives.
It's better to think in terms of risks not just costs. Interviewers tend to higher people just like them, which means the team is often filled with people that think the same way and come up with the same types of solutions. On the other hand when you get someone in that can 'do the job' but has a radically different perspective they are more likely to bring something new to the table. To be overly simplistic in a team of world class programmers adding someone with a great UI background can be worth a lot more than yet another world class developer. The advantage being diversity does not require a larger paycheck.
Interviewing skills and engineering skills are completely different. You should be an engineer if you want to interview other engineers, but it's likely that a great engineer isn't a great interviewer unless they've been specifically trained and coached on how to interview.
FizzBuzz is OK for quick filtering, but asking to come up with an efficient substring search algorithm ([1] or similar) on the spot, assuming that one doesn't know this algorithm a priori (and many good programmers don't), for which, scientists like Knuth, Morris and Pratt spent months, is just ridiculous.
Frankly, your comment bewilders me. Sure, you might not be able to replicate KMP in the middle of an interview if you didn't already know it, but I'd certainly expect any competent programmer to be able to come up with some form of solution (and I'd probably expect them to get reasonably close to KMP) reasonably quickly and then to be able to discuss their solution and identify it's flaws. A moderately intelligent version ought to be fairly trivial for someone with a solid comp-sci grounding.
Even then, it's not the coded solution that I'm particularly interested in - it's the explanation that follows about how you got to that solution and the discussion about the pros and cons of your particular implementation. I'm looking for signs that you can solve problems and that you can look at your solutions with a critical eye and understand their weaknesses as well.
For what it's worth, the nastiest interview question I've ever had involved building and searching interval trees with up to a billion intervals stored in it where they were looking for solutions which provided the fastest posible search times. It was less than fun.
I have several points/cases regarding inability to come up with a KMP like algorithm:
1. Fresh CS graduate from the top university (MIT, ETH, Stanford, etc..) - I agree, it is a bad sign.
2. Mid/late career professional - I disagree.
3. Self-taught professional - I disagree.
In latter two cases, some might just never needed/seen similar reasoning/algorithm implementation in their life, but that doesn't make them bad programmers.
Another point - many interviewers have beforehand written down exact fixed algorithm, and if your solution diverges at some point with that single variant (although your solution might be correct as well) - this is the red flag for them - and they interrupt you in the middle. I've experienced this even with an open ended design questions - this is really frustrating.
And my last point - remember that when KMP were coming up with this algorithm, no one were breathing in their back/neck.
Agree on the 2 & 3... I've probably been out of school (CS) for longer than most on here have been programming.
Not being able to come up with KMP on a whiteboard in an interview hasn't kept me from playing no small part in producing millions in revenue for companies.
Especially outrageous is all of the companies that think they need to ask these questions, when the day to day work is essentially mundane CRUD type stuff.
What exactly do you (or for that matter Google hiring in general) consider fizzbuzz?....Is Pascals's triangle fizzbuzz?...how about an algorithm for calculating the Levenshtein Distance?
It seems like everyone has a pet favorite interview problem that they like to throw at the candidate and in my experience some of them were definitely not fizzbuzz .And then if one interviewer doesn't like you then you dont get hired.All the work you have done is irrelevant if that pet problem is not solved optimally on the whiteboard.But then of course I am probably a little incompetent too.
One reason that interviewers ask the same question to many candidates (i.e., seem to have a "pet" problem) is because they have calibrated the question. A strong interviewer has asked that question to respected colleagues at various levels, and many tens or even hundreds of candidates. He knows all the ins-and-outs of the question, and more importantly, knows how to judge someone's effectiveness in that topic area based on their answers.
If you asked a different question to every candidate you interviewed, it would be very difficult to get a sense of how they compare to other candidates, or to employees.
Maybe every company is unhappy with built-in string searching operations... :-)
It's a rite of passage I think. Programming interviews remind me of rushing for fraternities in college. And once you get to the on-site, the hazing begins...
They probably ask questions similar to the ones that they were asked... so over time they select people more and more similar to themselves and their culture becomes homogenous and inbred.
What would happen if you just refused to answer the question and asked them for something more practical that would actually demonstrate your skill & experience level?
I've understood that FizzBuzz is the lowest possible requirement for any programming job. It's a trivial programming exercise that doesn't require experience in algorithmic theory or mathematics. It's like going to the driving test and being asked to start the car: anyone who wouldn't know how to do that is likely highly incapable of passing the actual test.
Now, the astonishing issue, for me, is that there apparently are huge loads of people applying for programming positions who actually fail FizzBuzz. In effect, it's akin to applying for the position of a bus driver "coz I once travelled on a bus". I don't get that but apparently it does happen often enough to warrant FizzBuzz.
What's even more astonishing to me than the fact that programming applicants mess it up is the fact that when the problem is introduced to a programming forum or blog there will inevitably be commenters commenting with a solution, and that solution will be wrong. The only car analogy I came up with was a person hearing about someone putting on their spare tire, so they go out to do it on their own car from memory of the account, and they do it wrong, when the instructions were right in the glovebox.
Comments
/sigh interview techniques really are a perennial topic on HN.
> [a coding test] won't tell you squat about how good they are in a year long project with 10 other people,
The author misses the point of a coding test. It's a negative rather than positive filter. Someone who does amazing on FizzBuzz is not by definition an amazing programmer. Someone who can't solve FizzBuzz however almost certainly is not a good programmer.
Simple coding tests are a very effective filter in terms of the time spent.
> So why spend an inordinate amount of time on relatively minor parts of a programmer's skill set?
I wouldn't call half an hour "an inordinate amount of time".
The author then opines about how "can just tell" if someone is a good programmer or not. I can sympathize because frankly so can I. I went through a period of taking 10 interviewees to lunch. I asked them nothing technical and basically just answered their questions. From the first 10 minutes I could tell:
- 1 would probably get an offer (he did);
- 8 would not (they didn't); and
- 1 I was unsure about (no offer).
Looking at the author's profile [1] I believe I can see the problem: it doesn't appear he's ever worked for a large engineering organization. This is fairly obvious from the content of this post because none of his solutions scale.
Let's say you have a large organization with 5 Andrews. Each of them says to hire this guy they just interviewed. What do you do if you're looking for less than 5 people? Are they going to work on the respective teams of those giving the recommendation? If not, how do you know they're a good fit? You need to consider company-wise culture and expectations. How do you calibrate between them? Do they have the necessary foundation to do not just the job they'll be starting on but to grow with the organization, adapt and perhaps work on other projects?
The other problem I have is the "war stories" aspect. This is a very definite bias. Take the way human memory works. Imagine you have a conversation with someone that's memorable in some way. At first you can remember word-for-word what happened. That quickly fades and you remember the gist of what was said. You may even think you remember exactly what was said and how it was said but usually you're wrong (try it by writing down what was exactly said and going back to it after months or having two or more people recount the same event). After awhile even that made fade and you may just be left with an emotion about the event.
Some things I can remember very well but more often than not, I've learnt my lesson and the exact circumstances or even the origin entirely are lost. This is largely because it's useless information.
But here's the biggest problem of all with the post:
> My favorite idea is still contract to hire everyone after your (hopefully reasonable) interview;
Okay, you've excluded anyone really good because they're not going to jump through that hoop. I don't consider myself a "rockstar" and even I won't jump through that.
Maybe the real problem is the OP doesn't even know what good programmers are because this is a recipe for mediocrity.
Lastly there's a story about team bonding (probably distorted if not made up outright at a guess). The author doesn't seem to realize that his hiring process is pretty much about hiring people like himself, which is fine, but that's not the definition of a good programmer.
I've seen teams work well who:
- all work closely together and socialize together;
- never socialize together;
- work remotely;
- work on different schedules;
- and so on.
Don't confuse programming ability with culture fit and work style. Those are three different things all important.
In a large organization you're going to have some aspect of a common culture but very different team cultures (and even site cultures). Part of hiring in a large organization is recognizing that you're not trying to hire "you" and then working out how you can best use someone not like you such that they flourish.
EDIT: oh and if he thinks Google, as one example, hasn't done extensive data analysis on interview techniques he's nuts.
[1]: http://www.linkedin.com/in/andrewwulf
Interviewing is a perennial topic because it doesn't make much sense.
Case in point - over the past week I've interviewed for four positions. Three gave me offers within a day, with very little effort on my part. On the other hand, one told me that I had little idea of what I was doing and told me to study up and apply again next year. And, that was after wasting my time with 3.5 hours of interviews. Of the offers I received, two were well over 120k plus great bonus. The other came in under 90k but told me that I could work anywhere in the world, whenever I wanted, as long as I got my work done.
Over that same time, I flat out rejected three recruiters from large engineering companies because their recruitment processes would have cost me $700-1500 just in opportunity costs. And, they weren't even willing to give me the info I needed to help me determine my chance of success or even my expected pay.
Everyone these days seems to believe that their company is so important that they deserve or need a team of top 1000 engineers. When in fact a top 20-percent team who bonds well together would likely be sufficient.
The topic of interviewing is fascinating because there seems to be very little science behind it. Who knows what works?
The opportunity cost is the most frustrating aspect. I've solved several "sample" problems from companies, and been involved in full day interviews writing code on the whiteboard. The worse was spending a day writing some sample code, then doing a panel interview where I was asked to make a change to the code to cover a non-sensical edge case. I was rejected because I "hesitated".
People make several mistakes when analyzing interview techniques:
1. The interviewer and interviewee are after similar and complementary things.
Wrong. The interviewee is trying to find out if the work is interesting, whether or not this would be a good place to work, whether or not he or she could work in that environment with that team and so on.
The interviewer (or rather the employer) is trying to fill a position. This is really important.
Some make the mistake of thinking that if an interview process doesn't evaluate the interviewee accurately it has failed. This is grossly inaccurate. The point of an interview process shouldn't be to look at a single candidate but to look at the process of filling a position, which may well span interviewing many candidates.
A false positive (someone who looks qualified but isn't) for an employer can be incredibly costly. The cost of a false negative is essentially zero as long as the employer can otherwise adequately fill the position.
To spell it out: if an employer gets 50 applications, has phone interviews for 8, brings in 4 for face-to-face interviews and makes two offers, one of which is accepted, the employer has gained the desired result. The fact that a qualified person was rejected along the way (false negative) is essentially irrelevant.
2. Your worth is constant.
Wrong. You said it yourself. You got several different offers. You're implying that if you had been accurately gauged market conditions would have you valued roughly similarly.
Company A might be desperate. Company B less so. You may fill a niche far better at one employer than another. One employer may simply be more cashed up and able to pay a better salary. A given employer may simply suck at negotiating or be under a misconception about market value. The list goes on.
Likewise, your desirability to the employer factors into this and it goes beyond technical skills. If the employer thinks you'll be a great fit and they'd really like to work with you, that improves your worth (to them).
3. Companies are looking for the same thing.
Clearly this is wrong but I do see this attitude come up, typically being implied by showing mismatches in offers as "proof" or similar.
There are many examples of this. For example, all other things being equal I've found that an MIT graduate is much more likely to hire other MIT graduates. The same is true for Stanford, CMU or [insert school here].
Part of this is the "social proof" element (going to a great school and/or working for a top-tier employer can be a huge advantage). But more than that it comes down to cultural similarity, common background and being a known quantity (to some degree).
This is of course different for every employer.
The real problem with interviewing, particularly at larger organizations, is that people who are bad at it are doing it. Interviewing and assessing potential colleagues is a skill and a talent. Some people have it. Some don't.
I've seen another comment here that said you need great engineers to do interviewing. I disagree. Many great engineers seem to be essentially savants who are often ill-equipped for the social discourse entailed in interviewing.
To interview an engineer I firmly believe you need to be an engineer (the same goes for managing engineers) but you don't need to be a rock star. You just have the right additional skills.
The cost of a false negative is essentially zero as long as the employer can otherwise adequately fill the position.
I couldn't quantify them exactly, but I don't think the costs are zero. For one, any candidate that you bring in and reject consumes (in the case of my company) about 4 man hours of developer time. And that's mostly senior/lead developer time. And that's not counting the time we spend discussing the candidate and the general cost of a distraction/context-switch. And then there's the opportunity cost of not having a position filled and work being started. Sometimes the short run matters.
That said, I'd definitely agree that the scale should be tipped in favor of false negatives.
adequately fill the position
It's better to think in terms of risks not just costs. Interviewers tend to higher people just like them, which means the team is often filled with people that think the same way and come up with the same types of solutions. On the other hand when you get someone in that can 'do the job' but has a radically different perspective they are more likely to bring something new to the table. To be overly simplistic in a team of world class programmers adding someone with a great UI background can be worth a lot more than yet another world class developer. The advantage being diversity does not require a larger paycheck.
Interviewing skills and engineering skills are completely different. You should be an engineer if you want to interview other engineers, but it's likely that a great engineer isn't a great interviewer unless they've been specifically trained and coached on how to interview.
FizzBuzz is OK for quick filtering, but asking to come up with an efficient substring search algorithm ([1] or similar) on the spot, assuming that one doesn't know this algorithm a priori (and many good programmers don't), for which, scientists like Knuth, Morris and Pratt spent months, is just ridiculous.
[1] http://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pr...
Frankly, your comment bewilders me. Sure, you might not be able to replicate KMP in the middle of an interview if you didn't already know it, but I'd certainly expect any competent programmer to be able to come up with some form of solution (and I'd probably expect them to get reasonably close to KMP) reasonably quickly and then to be able to discuss their solution and identify it's flaws. A moderately intelligent version ought to be fairly trivial for someone with a solid comp-sci grounding.
Even then, it's not the coded solution that I'm particularly interested in - it's the explanation that follows about how you got to that solution and the discussion about the pros and cons of your particular implementation. I'm looking for signs that you can solve problems and that you can look at your solutions with a critical eye and understand their weaknesses as well.
For what it's worth, the nastiest interview question I've ever had involved building and searching interval trees with up to a billion intervals stored in it where they were looking for solutions which provided the fastest posible search times. It was less than fun.
I have several points/cases regarding inability to come up with a KMP like algorithm:
1. Fresh CS graduate from the top university (MIT, ETH, Stanford, etc..) - I agree, it is a bad sign.
2. Mid/late career professional - I disagree.
3. Self-taught professional - I disagree.
In latter two cases, some might just never needed/seen similar reasoning/algorithm implementation in their life, but that doesn't make them bad programmers.
Another point - many interviewers have beforehand written down exact fixed algorithm, and if your solution diverges at some point with that single variant (although your solution might be correct as well) - this is the red flag for them - and they interrupt you in the middle. I've experienced this even with an open ended design questions - this is really frustrating.
And my last point - remember that when KMP were coming up with this algorithm, no one were breathing in their back/neck.
Agree on the 2 & 3... I've probably been out of school (CS) for longer than most on here have been programming.
Not being able to come up with KMP on a whiteboard in an interview hasn't kept me from playing no small part in producing millions in revenue for companies.
Especially outrageous is all of the companies that think they need to ask these questions, when the day to day work is essentially mundane CRUD type stuff.
What exactly do you (or for that matter Google hiring in general) consider fizzbuzz?....Is Pascals's triangle fizzbuzz?...how about an algorithm for calculating the Levenshtein Distance?
It seems like everyone has a pet favorite interview problem that they like to throw at the candidate and in my experience some of them were definitely not fizzbuzz .And then if one interviewer doesn't like you then you dont get hired.All the work you have done is irrelevant if that pet problem is not solved optimally on the whiteboard.But then of course I am probably a little incompetent too.
One reason that interviewers ask the same question to many candidates (i.e., seem to have a "pet" problem) is because they have calibrated the question. A strong interviewer has asked that question to respected colleagues at various levels, and many tens or even hundreds of candidates. He knows all the ins-and-outs of the question, and more importantly, knows how to judge someone's effectiveness in that topic area based on their answers.
If you asked a different question to every candidate you interviewed, it would be very difficult to get a sense of how they compare to other candidates, or to employees.
Congratulations on marrying the interview process to a single question. I hope that question is very close to the core competency you are hiring for.
Maybe every company is unhappy with built-in string searching operations... :-)
It's a rite of passage I think. Programming interviews remind me of rushing for fraternities in college. And once you get to the on-site, the hazing begins...
They probably ask questions similar to the ones that they were asked... so over time they select people more and more similar to themselves and their culture becomes homogenous and inbred.
What would happen if you just refused to answer the question and asked them for something more practical that would actually demonstrate your skill & experience level?
I've understood that FizzBuzz is the lowest possible requirement for any programming job. It's a trivial programming exercise that doesn't require experience in algorithmic theory or mathematics. It's like going to the driving test and being asked to start the car: anyone who wouldn't know how to do that is likely highly incapable of passing the actual test.
Now, the astonishing issue, for me, is that there apparently are huge loads of people applying for programming positions who actually fail FizzBuzz. In effect, it's akin to applying for the position of a bus driver "coz I once travelled on a bus". I don't get that but apparently it does happen often enough to warrant FizzBuzz.
What's even more astonishing to me than the fact that programming applicants mess it up is the fact that when the problem is introduced to a programming forum or blog there will inevitably be commenters commenting with a solution, and that solution will be wrong. The only car analogy I came up with was a person hearing about someone putting on their spare tire, so they go out to do it on their own car from memory of the account, and they do it wrong, when the instructions were right in the glovebox.