The hiring process
A big part of my previous job was to take part in the hiring process and conduct technical interviews. That process was quite forward:
1/ An interview conducted by HR determined if the candidate is a serial killer / psychopath
2/ An interview conducted by technical experts determined if the candidate is a good programmer
3/ An interview conducted by big boss determined how low the candidate is willing to get paid
I interviewed 2 types of people: interns and future employees. Interns only went through #2 while the others went through all three steps. In the span of 2+ years working at that company I must have performed 200+ technical interviews. It was an enriching learning process for me, and I ended up figuring it out one step at a time. Now it is important for you to note that this occured in France, where you simply cannot fire people. Hire the wrong guy and you will be stuck with that person forever. It is critical to filter out the best candidates and not make any mistakes about it. It was a tedious process, and I loved every single part of it.
The very-specific lottery quiz
I conducted my first tech interview in 2008. At that time, the company already had a working process that I followed: interviews were 1 hour. The candidates would have 30 minutes to answer a 15 questions quiz. Then we would spend 15 minutes talking about their answers plus an additional 15 minutes answering questions about the job. I quickly realized how terrible that questionnaire was. I mean, I think that even if you tried, you wouldn’t be able to come up with something that terrible. About 50% of the company’s projects were in java, so the quiz was very java-centric. It contained 5 trivial questions and then 10 very hard questions specific to the java frameworks we were mostly using :
It went from
- What is the difference between a class and an object?
to
- What is the purpose of the execAndwait interceptor in
the struts 2 framework?
Heck even I couldn’t explain or expand on half the stuff that was asked... Every single time I would pray so that they wouldn’t ask questions about the questions! Pretty ironic for the interviewer... Anyways I would usually skim really fast (2-5 mins) through their answers and spend the rest of the time talking about their resume. It sucked big time and I wanted to improve it. So I went online and compared hundreds of interview questions. At that time I believed in the quiz format. It just had to contain the right questions in order to reveal how good people were. The right quiz for the right people.
The very generic quiz
After about a month of research, I had come up with the best 50 questions I could find online. I felt that they were good questions because they could be answered in any language, and they were in a smooth crescendo of difficulty. I scattered the 50 and ended up with 5 sets of 10 great questions that I would hand out randomly.
Example:
What is a singleton and when would you use it / not use it?
This was much better. or so I thought... The questions were good and I would also usually get good answers in return. I went on for a few weeks but somehow didn't feel completely right. I couldn’t shake off the feeling that even if what I did was good, it just wasn’t great. Yes it did test whether or not the person knew programming theories, but in the end it left me clueless as to whether or not the person could code. In the end I’m not sure that we ended up hiring anyone better using that method than with the caveman struts 2 questionnaire. The more I thought about it, the more I realized that there were 2 big issues with those 5 questionnaires:
1/ The questions were too generic. by not going into language specifics, I couldn’t talk about SQL, front-end specifics, etc.
2/ The quiz was too short. 10 generic questions just didn’t cut it. There was no way I could know if they were good programmers or not.
What I really needed was a lot more questions, and questions specific to the job the person was applying to.
Quiz manager 3000
This is where things got a little out of hand. I went ahead and created (with the help of an intern) a fully automated quiz tool: quiz manager (QM). The tool made the hiring process perfect: after the first interview, HR would select 3 topics related to the job description. The tool would then automatically create a multiple choice question quiz with 3 x 20 = 60 random but specific questions with a difficulty matching the person’s experience.
Example:
(javascript)
var i = 0;
function a(){
var i = 2;
i++;
}
a();
alert(i); => 0 ? 2 ? 3 ?
It would then draw little graphs, generate and email reports to HR, displaying the results compared to the average along with a bunch of other useless metrics. Man was I proud of that tool! I was looking forward to having candidates take the test! I would sit next door with HR and watch the candidate’s live score on the intranet app as the answers were selected. QM made all of our lives so much easier, it seemed perfect… Until we tested it on our own developers!
Well... Turns out that a lot of our great developers were getting the same score as some of the people I had refused. That's right, QM was proven useless! I had spent so much time building the tool that it took me a long time to realize the big mistake I had made: our desire to automate the results had constrained me to only ask multiple-choice questions. The user could only select one answer, and the questions ended up being mostly trick questions. The outcome was that we were not testing software development skills at all! It was tough for me to swallow my pride, but in the end I admitted that the tool was counter-productive, reflecting the wrong impressions about developers more than anything.
Just let ’em code
8 months had passed since I had the job. I did some more research and checked at how some US companies did their screening process. This is when I decided to go for another method: just have them code. That’s the reason they get paid, so why not show me right now how they do it. Quite logical when you think about it... Having learned some lessons with the first months, the test became quite simple: I would give out 3 algorithms to be solved in 30 minutes. Candidates could pick the language of their choice and have access to a machine (disconnected from the internet). Those were classic problems found online: One algorithm usually dealt with string operations (such as reverse words in a sentence), another with recurrence (such as calculate a term in fibonacci’s suite), and the last one with collections (such as ordering a list).
Example:
print out digits 1-100.
for multiples of 3, print out foo.
for multiples of 5, print out bar.
for multiples of both 3 and 5, print out foobar.
Everything got much clearer and better. I could directly see those who was indenting, commenting, using conventions, finding the solutions, etc. It gave me a pretty good sense of how much programming the person had done in the past. moreover, discussing about their solutions was also very informative. I like to think that candidates were comfortable with those tests because I had tried to take off all the pressure out of them. They could take their time, choose the language they wanted, ask for advice, etc.
I was initially happy with the results and did that for a couple of months. But then again I started feeling that I was missing something… Something that just wasn’t right… It's true that I could easily spot the ones who could solve algorithms from the others. But were they really the great programmers I was looking for? When you think about it, is the quality of a programmer defined by how well he/she can solve a math problem, or whether or not he is able to sort a list in O(n log n) and not O(n^2)?
The one question to rule them all
I can remember exactly when I first started programming. QBasic was shipped with MSDOS 5.0 way before windows 3.1 came out. It contained its own help screen with all of the functions and keywords of the language, like the perfect offline man page. To this day I distinctively remember the feeling that grasped me every time I hit F5 and saw my programs execute before my eyes. A single printed line, a prompt for a name, some colors, a puzzle... I was in heaven. I remember putting line numbers before each command, filling my code with horrible GOTOs, learning with excitement and fascination something new everyday. I loved programming. I would spend hour after hour creating games, solving problems, showing stuff to my parents and friends. Years went by, I went from qbasic to pascal to vb, wrote games for our BBS "Atomic BBS” that we ran from our home phone line through a 2400bps modem. I wasn’t really good. Well in fact I really sucked and my code was pretty horrible! But man did I love it!! I just couldn’t let it go... I guess some people feel that type of adrenaline the first time they fly a plane, sail a boat, smoke weed, eat at in n out... For me it was programming, compiling, executing. I gained that feeling 25 years ago, and it has never left me since. I was born for this. I’ve always been a programmer.
I have always been convinced that those who love code do not restrict their coding activities to their work. They take home that love and continue to create for fun as a hobby. How many times have I felt frustrated at work because of a struggling eclipse, only to find relief and joy when writing ruby on rails code back home!
And so it was, that after 1 year of trial and error, I completely stopped handing out technical tests. I would sit down with the candidate, read and comment his resume without asking him any questions for a good 5-10 minutes. And then I would flip over the resume, look at the candidate in the eyes and ask: “we have about 30 minutes left. Will you please tell me about the best project that you’ve ever created?”
That simple, unique and nonjudmental question was the key. Some answered vaguely about their previous work or school project. And then some others became suddenly alive and excited, even those who appeared to be the shyest. They would talk passionately about the game they were creating, the website they had made, the open source projects they had contributed to, the utilities they made after being stuck in the middle of nowhere without any internet access. They were proud to show me. I was always fascinated by what I heard and would ask about all the details of the project they had treasured. They opened up and talked about the technical difficulties that they had overcome, about the little personal touch they added. It was their baby. And as they talked it was impossible to miss: I could see that light in their eyes, the excitement of a child that compiles and runs his first hello world. I would know right then that we had something in common. They were programmers too.
Most of them didn't have a clue about struts or some other specific framework we were using. Yet once they got the job, they always ended up being golden developpers. They learned faster, they produced better code, they inspired others with their creativity and positivism. They were coders at heart.
And in the end that’s all that matters.
[Update 07/27/14]
I never thought that this would escalate so let me clarify 2 things:
#1: The question I would ask would include all types of projects, and I would put work projects at the exact same level as hobby projects. It was great (but rare since they were changing jobs) to meet people who were passionate about their work project. There is no perfect interview method, no method that will work perfectly for every single candidate out there. But I came to find out that even the best could freeze in front of a simple FizzBuzz problem, just because of the interview context pressure. So basically instead of asking the candidate if they knew "this" or "that", this question allowed candidates to bring me to their world, tell me who they were and what they knew best.
#2: The first 5-10 minutes weren't minutes of terror and silence. I just didn't ask deep questions about the resume. I used those first minutes to chit-chat and make the candidate feel as comfortable as possible, by reading and making positive comments about the experience he had written down. After all, it's easy to put anything on a piece of paper. My main objective was to spend most of the interview time listening to what he had to say on the thing he was most passionate about. I never looked down at candidates because each of them was unique and you never knew how good they were just by reading their resume. Also i've always been surrounded by amazing programmers so I've learned to listen more than to talk.
