"As part of our hiring process at Mixer Labs, we would often give people a half day coding exercise. "
I (and many people who I've hired) don't have time to waste on your startup. I can tell within two minutes if someone can code or not. I've never understood the need to have someone write code (and then nitpickingly critique the code because it doesn't look like my code) when I can just ask them and know if they are lying or not.
And I've hired a LOT of super fantastic programmers.
Well, recently I interviewed at a place where they called me in 18 rounds of interviews over almost 3.5 full days.
The only reason I was putting up with them was because of friend inside who had referred me. The thing is they made me right endless code rounds after rounds and I did code pretty well. I solved close to 90% of the problems without the internet. But most of them were coding problems.
During the final rounds, I don't know what got into them. They started going into Core CS areas. And damn then they started the Algo pop quiz. Some how during the last 3-4 rounds they started giving out the perception that they were not happy with some one who did not know the algorithms from the book memorized by heart. And that even good coding skills can't compensate for that.
After that they called me back and told, that some other department from the same company would like to have an another round. I just politely denied and, cut the call.
I worked in a company that did that a couple of times, but not to that extent. Usually the job description wasn't well defined and the candidate didn't have any champions within the company. We would keep on bringing the candidate in over and over in the hope that some new perspective would make the choice obvious. (it never did)
Oddly enough, when you decline to continue interviewing they often immediately send you a job offer.
Frankly speaking if you can't decide about a candidate after 3.5 days of interviewing there's probably something really badly broken about your interviewing process. Or just that its so inefficient.
I wouldn't have accepted the offer from the company, from the very outlook it looked like bureaucratic jungle where no one is capable of taking a decision/risk until they run through some 20 meetings to decide what to do. And then toss it on to some one else.
For that if this is what you make candidate run through, you are sure to get CareerCup interview questions ebook masters but not good programmers.
Phone interviews or in person? I think after 3 such I would have decided these people just couldn't make decisions or were such a political organization they would have been toxic to work for. Did your contact give any insight into this inefficiency?
Hiring people who know how to program is actually not hard. I also can screen for this in an hour, no problem.
It's everything else that is the problem. Are they lazy? Do disappear for days at a time? Will they not fix bugs in someone else's code? Will they complain NIH at the drop of a hat. Will they care to write all the ugly code we need to ship with, and not just the first 80% of the project?
These are the things that are hard to assess. But just knowing if they can code -- that's pretty easy.
I've never had a problem hiring great programmers, and it's not difficult at all really. It's insanely difficult though for common companies when they are not a good or interesting place to work, or they don't want to pay competitively for the people they need, or they have incorrect ideas about what sort of talent they really need. There is no shortage of great developers out there. The shortage is in companies that have management with any clue. These companies obviously are unable to understand or correct this though.
It's kind of a two-way thing. If they're investing enough time into telling me about them, what projects they have and what's exciting about them, what the environment is like, what hardware they use, what they do after work, whether they have any employees using Dvorak †, then sure, if what I'm hearing sounds good, I can devote half a day to them. But on the other hand, I've had a company administer a test and then a coding exercise with contact via email from their secretary, and maybe I'm spoiled but I'm not doing a multi-hour task for you to earn the privilege of talking to a person involved in development.
One of the fastest ways is to ask about something they had to "fix" (use the word "fix") at their last job, or some problem in a pet project that they just figured out. This will sound like poetic license, but really good programmers have a light that flares when they start talking about that. They immediately are back in the trenches, telling about how they couldn't figure out some nasty recursion error or some server problem that was untraceable. Usually their hands start getting animated; they loosen up.
I couldn't care less about the problem OR the eventual solution. I couldn't care less if he used an open source module to write some new code last week or if he thinks CouchDB is super amazing.
What I really want to know is how they think.
When you have a programmer who lives in code, he (or she) swim through problems like they are physical objects. They find passion in killing bugs and making code more efficient. I can hire a hundred people who can recite what the Gang of Four is or write a bubble sort. I can't hire guys who care about whether their own code can be improved.
But this is just one way. There are lots of ways to tell.
I 100% completely agree. I think asking programming trivia questions in this day and age is ridiculous because everyone now has memorized every single CareerCup question out there.
When I interviewed for previous copmanies, this is actually one of the questions I would use. "What is the hardest bug you've ever worked on?" If you take this question, and keep pressing for more and more details, and expand on that question, you can get a surprising amount of information about someone's experience and understanding. If they can't adequately answer this question, it means they haven't really been deep in code, or don't have a lot of experience, so that sets your expectations relative to the position that you're hiring for. If they claim to have a lot of experience, but can't go into too much depth, it means they're not very excited about getting their hands dirty or they don't understand what really happened, which are both bad signs to me.
I don't care if they can write up a function that can spit out the Fibonacci sequence. I can that they are smart enough to figure out relatively complex issues and can communicate this intelligently.
There's no way to figure this out from a one on one interview with that person?
I mean, a half-day programming test feels excessive. Are you worried about loosing good candidates that just don't want to deal with the headaches?
I've heard people try to justify insane hiring processes by saying "we want to filter out the people that don't want to work here. We only want people that really want it." But the best programmers -- the ones you want to hire -- are going to be in high demand by companies that look at their github, have a three minute interview, and pull the trigger.
We would only do the half day test with a subset of the candidates. A lot of people actually welcomed it - we would explain it was a way for them to also get to know us, see what we were like to work with etc.
A lot of people about to make a long term commitment to a company also want to make sure the company is right for them. So this was a good way to get to know one another.
This stuff just seems easy to figure out from a resume and asking questions. If they can talk competently in detail about cool stuff on their resume, and you yourself are smart enough to spot BS, then you know they get stuff done.
As for culture fit, again, a socially competent interviewer can size these things up reasonably quickly.
There are real diminishing returns on technical interviews. If you've got well thought out, well structured questions that cover the key areas for your organisation, unless you're doing something massively complex or specific, you can work out whether someone has the level of competence you're looking for in a couple of hours.
I've only been interviewing for a few years so I don't have a large statistical sample, but I can tell you that within a few minutes I can usually only tell if they think like me or not. The rest of the interview determines if they can code regardless of my biases, and can actually get stuff done.
> within a few minutes I can usually only tell if they think like me or not. The rest of the interview determines if they can code regardless of my biases
If they don't ask for relevant details, etc, they won't be coming up with a solution no matter how they do it.
You can usually establish that someone can't code inside one minute, which happens surprisingly often. You ask: Explain virtual methods to me. If they have C or C++ on their resume you write a three line function that returns a pointer to a stack variable and ask them to explain what's wrong.
If you feel like you even have to ask these questions your candidate selection process is badly broken, but the same approach scales up to all levels.
I disagree. These days people just memorize the CareerCup e-book, and know the answers to almost every major question out there. I've had plenty of people who knew what a virtual function is, and not know how to code. People have gamed the interview system, it doesn't work anymore.
"Explain virtual methods to me" - thereby ignoring anyone who can code in Python or Ruby, or anyone from VB.NET and several other languages where Overridable/Overrides is the terminology used.
Comments
"As part of our hiring process at Mixer Labs, we would often give people a half day coding exercise. "
I (and many people who I've hired) don't have time to waste on your startup. I can tell within two minutes if someone can code or not. I've never understood the need to have someone write code (and then nitpickingly critique the code because it doesn't look like my code) when I can just ask them and know if they are lying or not.
And I've hired a LOT of super fantastic programmers.
Well, recently I interviewed at a place where they called me in 18 rounds of interviews over almost 3.5 full days.
The only reason I was putting up with them was because of friend inside who had referred me. The thing is they made me right endless code rounds after rounds and I did code pretty well. I solved close to 90% of the problems without the internet. But most of them were coding problems.
During the final rounds, I don't know what got into them. They started going into Core CS areas. And damn then they started the Algo pop quiz. Some how during the last 3-4 rounds they started giving out the perception that they were not happy with some one who did not know the algorithms from the book memorized by heart. And that even good coding skills can't compensate for that.
After that they called me back and told, that some other department from the same company would like to have an another round. I just politely denied and, cut the call.
I worked in a company that did that a couple of times, but not to that extent. Usually the job description wasn't well defined and the candidate didn't have any champions within the company. We would keep on bringing the candidate in over and over in the hope that some new perspective would make the choice obvious. (it never did)
Oddly enough, when you decline to continue interviewing they often immediately send you a job offer.
Frankly speaking if you can't decide about a candidate after 3.5 days of interviewing there's probably something really badly broken about your interviewing process. Or just that its so inefficient.
I wouldn't have accepted the offer from the company, from the very outlook it looked like bureaucratic jungle where no one is capable of taking a decision/risk until they run through some 20 meetings to decide what to do. And then toss it on to some one else.
For that if this is what you make candidate run through, you are sure to get CareerCup interview questions ebook masters but not good programmers.
Absolutely, any company that does this has a totally broken interviewing process.
Though in our case there wasn't too much structure day-to-day, there was total lack of structure.
Phone interviews or in person? I think after 3 such I would have decided these people just couldn't make decisions or were such a political organization they would have been toxic to work for. Did your contact give any insight into this inefficiency?
If you can do this reliably, you should start a business.
Common knowledge is that hiring programmers is insanely difficult. If you have a gift for it, you could make craptonnes of money!
Hiring people who know how to program is actually not hard. I also can screen for this in an hour, no problem.
It's everything else that is the problem. Are they lazy? Do disappear for days at a time? Will they not fix bugs in someone else's code? Will they complain NIH at the drop of a hat. Will they care to write all the ugly code we need to ship with, and not just the first 80% of the project?
These are the things that are hard to assess. But just knowing if they can code -- that's pretty easy.
I've never had a problem hiring great programmers, and it's not difficult at all really. It's insanely difficult though for common companies when they are not a good or interesting place to work, or they don't want to pay competitively for the people they need, or they have incorrect ideas about what sort of talent they really need. There is no shortage of great developers out there. The shortage is in companies that have management with any clue. These companies obviously are unable to understand or correct this though.
I did start a business. But I should state that I learned how to interview and hire for years working for other people.
Your parent probably means you should start a business of helping other people hire good programmers.
I did. It does quite well. But I am no longer involved in it day-to-day.
In my experience, hiring programmers is difficult because it's difficult to find the right people ... but once they're in a room with you, it's easy.
It's kind of a two-way thing. If they're investing enough time into telling me about them, what projects they have and what's exciting about them, what the environment is like, what hardware they use, what they do after work, whether they have any employees using Dvorak †, then sure, if what I'm hearing sounds good, I can devote half a day to them. But on the other hand, I've had a company administer a test and then a coding exercise with contact via email from their secretary, and maybe I'm spoiled but I'm not doing a multi-hour task for you to earn the privilege of talking to a person involved in development.
† List exaggerated
Yeah, we would put the coding exercise at the end: http://blog.eladgil.com/2011/06/our-10-step-engineering-hiri...
It was also a burden on our team to spend all this time with a candidate, so this would often be our last step.
We would next expect someone to invest in us if we did not also invest time on our end.
What are the key things in your experience that let you know if someone is good or not as a long term fit with your company?
Our test was not primarily just code quality.
When we brought people in we would look at:
1. Approach. Did they use e.g. an open source pre-existing module or write something themselves? Did they come up with a creative solution?
2. Culture fit. While they were in working with us, how did they interact with the team? Did they enjoy working with us and vice versa?
3. Productivity. How much did they actually get done?
One of the fastest ways is to ask about something they had to "fix" (use the word "fix") at their last job, or some problem in a pet project that they just figured out. This will sound like poetic license, but really good programmers have a light that flares when they start talking about that. They immediately are back in the trenches, telling about how they couldn't figure out some nasty recursion error or some server problem that was untraceable. Usually their hands start getting animated; they loosen up.
I couldn't care less about the problem OR the eventual solution. I couldn't care less if he used an open source module to write some new code last week or if he thinks CouchDB is super amazing.
What I really want to know is how they think.
When you have a programmer who lives in code, he (or she) swim through problems like they are physical objects. They find passion in killing bugs and making code more efficient. I can hire a hundred people who can recite what the Gang of Four is or write a bubble sort. I can't hire guys who care about whether their own code can be improved.
But this is just one way. There are lots of ways to tell.
I 100% completely agree. I think asking programming trivia questions in this day and age is ridiculous because everyone now has memorized every single CareerCup question out there.
When I interviewed for previous copmanies, this is actually one of the questions I would use. "What is the hardest bug you've ever worked on?" If you take this question, and keep pressing for more and more details, and expand on that question, you can get a surprising amount of information about someone's experience and understanding. If they can't adequately answer this question, it means they haven't really been deep in code, or don't have a lot of experience, so that sets your expectations relative to the position that you're hiring for. If they claim to have a lot of experience, but can't go into too much depth, it means they're not very excited about getting their hands dirty or they don't understand what really happened, which are both bad signs to me.
I don't care if they can write up a function that can spit out the Fibonacci sequence. I can that they are smart enough to figure out relatively complex issues and can communicate this intelligently.
There's no way to figure this out from a one on one interview with that person?
I mean, a half-day programming test feels excessive. Are you worried about loosing good candidates that just don't want to deal with the headaches?
I've heard people try to justify insane hiring processes by saying "we want to filter out the people that don't want to work here. We only want people that really want it." But the best programmers -- the ones you want to hire -- are going to be in high demand by companies that look at their github, have a three minute interview, and pull the trigger.
We would only do the half day test with a subset of the candidates. A lot of people actually welcomed it - we would explain it was a way for them to also get to know us, see what we were like to work with etc.
A lot of people about to make a long term commitment to a company also want to make sure the company is right for them. So this was a good way to get to know one another.
Which subset, and why only those people?
This stuff just seems easy to figure out from a resume and asking questions. If they can talk competently in detail about cool stuff on their resume, and you yourself are smart enough to spot BS, then you know they get stuff done.
As for culture fit, again, a socially competent interviewer can size these things up reasonably quickly.
I agree with this.
There are real diminishing returns on technical interviews. If you've got well thought out, well structured questions that cover the key areas for your organisation, unless you're doing something massively complex or specific, you can work out whether someone has the level of competence you're looking for in a couple of hours.
> I can tell within two minutes if someone can code or not.
How?
Have you heard of the Dunning-Kruger effect?
Yes. I know everything one could possibly know about it and more.
Nice. I wish I could say I know nothing about it and understand it worse than anyone else.
I have not but now that I have I will use in conversation. It will make me look smarter than I am, exacerbating my own Dunning Kruger effect
Do you raise this in the context of asking people to self-asses their productivity or something else?
Probably in the context of the grandparent so assured that he knows how to assess programmers' abilities within 5 minutes.
Not five minutes. Two minutes. Sometimes less. And you should, too.
I've only been interviewing for a few years so I don't have a large statistical sample, but I can tell you that within a few minutes I can usually only tell if they think like me or not. The rest of the interview determines if they can code regardless of my biases, and can actually get stuff done.
> within a few minutes I can usually only tell if they think like me or not. The rest of the interview determines if they can code regardless of my biases
If they don't ask for relevant details, etc, they won't be coming up with a solution no matter how they do it.
Really!! in 2 mins you would determine if someone can code or note how would you do that.
You can usually establish that someone can't code inside one minute, which happens surprisingly often. You ask: Explain virtual methods to me. If they have C or C++ on their resume you write a three line function that returns a pointer to a stack variable and ask them to explain what's wrong.
If you feel like you even have to ask these questions your candidate selection process is badly broken, but the same approach scales up to all levels.
I disagree. These days people just memorize the CareerCup e-book, and know the answers to almost every major question out there. I've had plenty of people who knew what a virtual function is, and not know how to code. People have gamed the interview system, it doesn't work anymore.
"Explain virtual methods to me" - thereby ignoring anyone who can code in Python or Ruby, or anyone from VB.NET and several other languages where Overridable/Overrides is the terminology used.