How I ended up conducting the most successful technical interviews with a single question

The hiring process
A big part of my previous job was to take part in the hiring process and conduct technical interviews. That process was quite forward:

1/ An interview conducted by HR determined if the candidate is a serial killer / psychopath

2/ An interview conducted by technical experts determined if the candidate is a good programmer

3/ An interview conducted by big boss determined how low the candidate is willing to get paid

I interviewed 2 types of people: interns and future employees. Interns only went through #2 while the others went through all three steps. In the span of 2+ years working at that company I must have performed 200+ technical interviews. It was an enriching learning process for me, and I ended up figuring it out one step at a time. Now it is important for you to note that this occured in France, where you simply cannot fire people. Hire the wrong guy and you will be stuck with that person forever. It is critical to filter out the best candidates and not make any mistakes about it. It was a tedious process, and I loved every single part of it.

The very-specific lottery quiz
I conducted my first tech interview in 2008. At that time, the company already had a working process that I followed: interviews were 1 hour. The candidates would have 30 minutes to answer a 15 questions quiz. Then we would spend 15 minutes talking about their answers plus an additional 15 minutes answering questions about the job. I quickly realized how terrible that questionnaire was. I mean, I think that even if you tried, you wouldn’t be able to come up with something that terrible. About 50% of the company’s projects were in java, so the quiz was very java-centric. It contained 5 trivial questions and then 10 very hard questions specific to the java frameworks we were mostly using :

It went from

- What is the difference between a class and an object?

to

- What is the purpose of the execAndwait interceptor in 
  the struts 2 framework?

Heck even I couldn’t explain or expand on half the stuff that was asked... Every single time I would pray so that they wouldn’t ask questions about the questions! Pretty ironic for the interviewer... Anyways I would usually skim really fast (2-5 mins) through their answers and spend the rest of the time talking about their resume. It sucked big time and I wanted to improve it. So I went online and compared hundreds of interview questions. At that time I believed in the quiz format. It just had to contain the right questions in order to reveal how good people were. The right quiz for the right people.

The very generic quiz
After about a month of research, I had come up with the best 50 questions I could find online. I felt that they were good questions because they could be answered in any language, and they were in a smooth crescendo of difficulty. I scattered the 50 and ended up with 5 sets of 10 great questions that I would hand out randomly.

Example:

What is a singleton and when would you use it / not use it?

This was much better. or so I thought... The questions were good and I would also usually get good answers in return. I went on for a few weeks but somehow didn't feel completely right. I couldn’t shake off the feeling that even if what I did was good, it just wasn’t great. Yes it did test whether or not the person knew programming theories, but in the end it left me clueless as to whether or not the person could code. In the end I’m not sure that we ended up hiring anyone better using that method than with the caveman struts 2 questionnaire. The more I thought about it, the more I realized that there were 2 big issues with those 5 questionnaires:

1/ The questions were too generic. by not going into language specifics, I couldn’t talk about SQL, front-end specifics, etc.

2/ The quiz was too short. 10 generic questions just didn’t cut it. There was no way I could know if they were good programmers or not.

What I really needed was a lot more questions, and questions specific to the job the person was applying to.

Quiz manager 3000
This is where things got a little out of hand. I went ahead and created (with the help of an intern) a fully automated quiz tool: quiz manager (QM). The tool made the hiring process perfect: after the first interview, HR would select 3 topics related to the job description. The tool would then automatically create a multiple choice question quiz with 3 x 20 = 60 random but specific questions with a difficulty matching the person’s experience.

Example:

(javascript)
var i = 0;
function a(){
  var i = 2; 
  i++;
} 
a(); 
alert(i);    =>    0 ? 2 ? 3 ?

It would then draw little graphs, generate and email reports to HR, displaying the results compared to the average along with a bunch of other useless metrics. Man was I proud of that tool! I was looking forward to having candidates take the test! I would sit next door with HR and watch the candidate’s live score on the intranet app as the answers were selected. QM made all of our lives so much easier, it seemed perfect… Until we tested it on our own developers!

Well... Turns out that a lot of our great developers were getting the same score as some of the people I had refused. That's right, QM was proven useless! I had spent so much time building the tool that it took me a long time to realize the big mistake I had made: our desire to automate the results had constrained me to only ask multiple-choice questions. The user could only select one answer, and the questions ended up being mostly trick questions. The outcome was that we were not testing software development skills at all! It was tough for me to swallow my pride, but in the end I admitted that the tool was counter-productive, reflecting the wrong impressions about developers more than anything.

Just let ’em code
8 months had passed since I had the job. I did some more research and checked at how some US companies did their screening process. This is when I decided to go for another method: just have them code. That’s the reason they get paid, so why not show me right now how they do it. Quite logical when you think about it... Having learned some lessons with the first months, the test became quite simple: I would give out 3 algorithms to be solved in 30 minutes. Candidates could pick the language of their choice and have access to a machine (disconnected from the internet). Those were classic problems found online: One algorithm usually dealt with string operations (such as reverse words in a sentence), another with recurrence (such as calculate a term in fibonacci’s suite), and the last one with collections (such as ordering a list).

Example:

print out digits 1-100.
for multiples of 3, print out foo.
for multiples of 5, print out bar.
for multiples of both 3 and 5, print out foobar.

Everything got much clearer and better. I could directly see those who was indenting, commenting, using conventions, finding the solutions, etc. It gave me a pretty good sense of how much programming the person had done in the past. moreover, discussing about their solutions was also very informative. I like to think that candidates were comfortable with those tests because I had tried to take off all the pressure out of them. They could take their time, choose the language they wanted, ask for advice, etc.

I was initially happy with the results and did that for a couple of months. But then again I started feeling that I was missing something… Something that just wasn’t right… It's true that I could easily spot the ones who could solve algorithms from the others. But were they really the great programmers I was looking for? When you think about it, is the quality of a programmer defined by how well he/she can solve a math problem, or whether or not he is able to sort a list in O(n log n) and not O(n^2)?

The one question to rule them all
I can remember exactly when I first started programming. QBasic was shipped with MSDOS 5.0 way before windows 3.1 came out. It contained its own help screen with all of the functions and keywords of the language, like the perfect offline man page. To this day I distinctively remember the feeling that grasped me every time I hit F5 and saw my programs execute before my eyes. A single printed line, a prompt for a name, some colors, a puzzle... I was in heaven. I remember putting line numbers before each command, filling my code with horrible GOTOs, learning with excitement and fascination something new everyday. I loved programming. I would spend hour after hour creating games, solving problems, showing stuff to my parents and friends. Years went by, I went from qbasic to pascal to vb, wrote games for our BBS "Atomic BBS” that we ran from our home phone line through a 2400bps modem. I wasn’t really good. Well in fact I really sucked and my code was pretty horrible! But man did I love it!! I just couldn’t let it go... I guess some people feel that type of adrenaline the first time they fly a plane, sail a boat, smoke weed, eat at in n out... For me it was programming, compiling, executing. I gained that feeling 25 years ago, and it has never left me since. I was born for this. I’ve always been a programmer.

I have always been convinced that those who love code do not restrict their coding activities to their work. They take home that love and continue to create for fun as a hobby. How many times have I felt frustrated at work because of a struggling eclipse, only to find relief and joy when writing ruby on rails code back home!

And so it was, that after 1 year of trial and error, I completely stopped handing out technical tests. I would sit down with the candidate, read and comment his resume without asking him any questions for a good 5-10 minutes. And then I would flip over the resume, look at the candidate in the eyes and ask: “we have about 30 minutes left. Will you please tell me about the best project that you’ve ever created?”

That simple, unique and nonjudmental question was the key. Some answered vaguely about their previous work or school project. And then some others became suddenly alive and excited, even those who appeared to be the shyest. They would talk passionately about the game they were creating, the website they had made, the open source projects they had contributed to, the utilities they made after being stuck in the middle of nowhere without any internet access. They were proud to show me. I was always fascinated by what I heard and would ask about all the details of the project they had treasured. They opened up and talked about the technical difficulties that they had overcome, about the little personal touch they added. It was their baby. And as they talked it was impossible to miss: I could see that light in their eyes, the excitement of a child that compiles and runs his first hello world. I would know right then that we had something in common. They were programmers too.

Most of them didn't have a clue about struts or some other specific framework we were using. Yet once they got the job, they always ended up being golden developpers. They learned faster, they produced better code, they inspired others with their creativity and positivism. They were coders at heart.

And in the end that’s all that matters.


[Update 07/27/14]
I never thought that this would escalate so let me clarify 2 things:
#1: The question I would ask would include all types of projects, and I would put work projects at the exact same level as hobby projects. It was great (but rare since they were changing jobs) to meet people who were passionate about their work project. There is no perfect interview method, no method that will work perfectly for every single candidate out there. But I came to find out that even the best could freeze in front of a simple FizzBuzz problem, just because of the interview context pressure. So basically instead of asking the candidate if they knew "this" or "that", this question allowed candidates to bring me to their world, tell me who they were and what they knew best.
#2: The first 5-10 minutes weren't minutes of terror and silence. I just didn't ask deep questions about the resume. I used those first minutes to chit-chat and make the candidate feel as comfortable as possible, by reading and making positive comments about the experience he had written down. After all, it's easy to put anything on a piece of paper. My main objective was to spend most of the interview time listening to what he had to say on the thing he was most passionate about. I never looked down at candidates because each of them was unique and you never knew how good they were just by reading their resume. Also i've always been surrounded by amazing programmers so I've learned to listen more than to talk.

43 thoughts on “How I ended up conducting the most successful technical interviews with a single question

  1. Great article – I completely agree with your thoughts and will use this question in the future – if I am ever on the other side of the table.

  2. We’re discussing this on Hacker News, and the big question seems to be: Are you only interested in personal/non-work projects, as your examples seem to imply? Or are you willing to accept any exceptional work that the applicant is proud of and excited about?

    There’s lots of really excellent programmers who don’t have a lot of time for exciting personal projects, but still get really excellent work done on the job. When I read your question, the first thing that popped to mind was a data-processing script I wrote in three twelve-hour days that saved many hundreds of person-hours in manual work. The data set came to us with a lot of errors that needed to be corrected from context; two of my coworkers, better general programmers than me, had said it was impossible without human judgment; but my particular perspective let me see the cracks in the problem and get in done in one long push. I’m proud of that, and excited about it, and I could happily talk your ear off about how I did it; but I did it for money, and after it was done I went home and played video games.

    How would you evaluate that response?

    • I’m in my car right now. I didn’t expect this thread to escalate, so I’ll answer in depth later. The question included all type of projects. It led to a lot more questions about specific details on the project architecture and design. It wasn’t perfect but if one could speak with passion about his experience as a programmer it was always an excellent sign.

      • Also the main advantage is that I was setting myself in the candidate’s ground. No surprises, nothing he wouldn’t be able to answer because it was his project, and by taking 30 minutes to go in depth into it, I could grasp a good sense of the person’s knowledge as well as his excitement for code.

  3. I’d strongly recommend not asking “best”, but “proud of”. This is important because quite a few people have projects that are extremely important to them, do amazing things, use amazing techniques, but have shitty code you wouldn’t inflict on anyone else.

  4. Nicolas, this article is priceless and should be shared without moderation.
    I have been thinking about it for a long time also and recently read an article on Quora about having people talk about what they do but you nailed it.

  5. You are absolutely correct. I have been programming since 1965. I was hooked at 16, and never lost the thrill. This month I’m learning Python.

    Did Picasso retire? Did Casals cease playing? You never stop programming.

    The best employees I ever hired all had taught themselves programming in high school.

  6. I just left school two years ago in hope to find a company that appreciate exactly the things you describe. However I ended up somewhere where it is more important to talk about things and buy every IT-stuff you need. I guess I would feel much more comfortable in your company. Thank you for this inspiring post!
    PS: Recently I sat in a training and wanted to tell the other (mostly non IT) guys why I love what I do. I tried to describe my feelings when I make my git push to the release version, but they were not able to understand. “i guess some people feel that type of adrenaline the first time they fly a plane, sail a boat, smoke weed, eat at in n out…” is a nice metaphor for this.

  7. I totally agree with you. I performed technical interviews for 3 years for a consulting company I was working for at the time. The interviews were done after hours on own time. I really can’t add anything to what you’ve already said. I can only say that are exactly right. That question alone will help you find the real enthusiastic programmers you want.

  8. Nice article – thanks!

    FWIW, my approach is similar to yours, with just a few questions, but the intent is the same I think: to see if someone has a “fire in the belly”.

    Anyone who cares about their work is golden.

  9. +1

    That is my favorite question too.

    In addition to bringing out their enthusiasm, you find out quickly if they are motivated to do useful things. If the thing they are make proud of brought value is key.

  10. great post and good night lecture but I don’t understand why did you say this:

    “now it is important for you to note that this occured in france, where you simply cannot fire people.”

  11. @auraham: over there you don’t have the flexibility to fire someone who isn’t doing his job. More than anywhere in the world, the screening process in france has to minimize the risk of hiring the wrong people.

  12. @TML: I had read about it! That’s a great question IMO. The only downside I would see is that some good programmers might be blocked (brain fart) by the whole interview context even though they would be able to create great answers given a more relaxed context.

  13. As an electronic engineer I’ve been part of the hiring process of quite a few technicians. What astonished me is that some of the technicians I interviewed had BSEE degrees, and had never worked in their field since graduating. In a couple of cases for 5-7 years!

    My go-to question was, “How have you used your knowledge of electronics for a fun project?”

    Blank stares. One person told me that he plays video games for fun. Two of the engineering graduates had never used their education after graduation, with their Senior project being the last thing they ever worked on. One of them worked in my company as a forklift driver, and told me he was never given the chance to move to Engineering (but we have in-company job openings all the time! This was the first time Engineering ever saw a resume from this guy – I checked!)

    One guy built a transistor radio several years back.

    My best hire was a woman who had built and programmed a robot based upon the Basic Stamp. She made the bump sensors out of push buttons and wire, she used a light sensor as an edge detector, and she modified off-the-shelf servos for the wheels. She did use a couple of off the shelf parts for robots, but that stuff was expensive, so she tried to find cheap alternatives.

    That little robot worked great. And she worked very well for us for several years before she moved out of state.

    I have had one objection to my preference that technicians (and engineers!) build something for fun – even on their own time. The objection is that specialized engineering tools, (like oscilloscopes) are expensive – so they don’t have the ability to try this sort of thing.

    Which is a bunch of BS.

  14. It is only *half* of the perfect interview questions.

    The other half is for the interviewed, after taking half the time to answer your question, should then sit back and state: “Will you take the rest of the time to show me what you would do for me”.

    Unfortunately the young can be naive; the young who recognise that they can be paid for doing their hobby: doubly so. (But maybe less-so in the future after reading this answer).

  15. I recognize a lot of the process you’ve been through! I’d like to quickly share my approach with you, as I believe it to be a mixture of some of your approaches and worked so well that it quickly became the standard where I as working. First I would ask some technical questions any engineer should know, like what is the ASCII table? This was only meant to weed out the candidates that really weren’t worthy and would only take 15 minutes. Next we would take half an hour ‘peer programming’ with some code the candidate brought along. In advance we sent an e-mail to the candidate asking to bring code that makes him or her proud. Then I would suggest a setting in which we pretended to be collegues and would peer program to improve or extend upon the code. With some candidates this would trigger passionate debates that lasted well beyond the half hour I’d set and with some we would just get into a defensive argument. During this process the level of knowledge, passion and coorporation of the candidate always became crystal clear. I’ve never mishired since using this process.

  16. Having done a TON of interviews as well, the 2 most valuable discussions in all of them are..

    “Tell me about your favorite project”.

    “Tell me about your least favorite project”.

    I ask that early, sometimes they become the whole interview, and that’s ok. If they are too fast with these, I’m likely not interested in continuing anyway.

  17. While asking what is your favorite project is good, what if he don’t have one? Some people are good programmers despite the fact that, they don’t work out of office on programming projects. So, for sure, it is not a silver bullet to screen for good programmers. But what if these passionate programmers cannot show the same level passion in their official work.

    How many people crib about their professional work which don’t give a challenging work. So, looking for passionate programmers, for sure have negative effects as well. So, in my opinion ( I have about 10 years experience in interviewing) I agree no questionnaire is perfect – as you mentioned they have serious problems. But, we could ask him how he would solve a particular problem. We might ask him what was his challenging time being a programmer. Also, throw some design questions and see how he responds with out worrying about the answer. You may be able to find some traits of a good programmer from their responses. But the biggest drawback is we can’t test is his learn-ability. Most of the job gets done if he could learn new things on the job. This is something often neglected as well as most difficult thing to test. I think a guy with good attitude with a programmers mind and willing to learn new things has a good probability being a good hire.

  18. A resume can give us a general idea of technical exposure if not some level of capability, but there is no quiz to measure passion. I made a hiring mistake early on by weighting a resume too heavily in my process. The candidate we hired was, on paper, excellent, but in reality we struggled with communication and productivity issues and I never was able to find a way to make the relationship work. The next hire, I almost did not look at the resume at all after it had been vetted for minimum qualifications. We were successful on take two and the contrast between the processes has left an enduring impact in how I will evaluate new hires in the future. Our second attempt was similar to your approach and worked out very well.

  19. What a beauty! Wow.
    Simplicity and genius on so many levels.

    Can’t stop rewinding my 20 years in java development and realizing how different it would feel to create or work in teams created by this genius principle.
    IT IS the spark in the eyes that we had to look for!
    If I only hired those with the sparks in their eyes and never hired the brainy guys with no passion, the ones that end up as HR stats at big trendy billion dollar tech corps…
    Imagine a team of 5 or 10 devs fully built by this principle? This is how success is born.
    This is how you end up loving your work because you are surrounded by people who love it as much as you do.
    So many levels…

    Thank you Nicolas.

    P.S.: Now that “Bize theorem” is out in the open beware of fake sparks. Spark is a vector :) Need to recognize the direction and length, filter out meaningless from real.

  20. Thanks for the article, I agree that it is one of the more important questions. I ask that question too when interviewing.

    When I was a graduate student, I guided students through their final projects. I always used this to motivate them telling them that in interviews people will ask them about a project they are proud of or enjoyed and they better have something good to say.

  21. An excellent article. I recall my first interview and the question asked was, “Tell me about your tech report”. I hire ACE’s – Attitude, Character, and Enthusiasm.

    What you know is taught, what you understand is learned. A personal “best” can bring out the passion in anyone – well … almost.

  22. This is probably the oldest method to check any skill, if one is passionate enough on what he does, then almost all the time they would do better than those who does that just as work, but same time Its not easy to master any programming language in say 2 month or 3 month or even 6 month. It takes years of experience to build that expertise which is required by real world applications, and of-course you cannot afford someone trial and error coding your mission critical application. So, I like to go for an hybrid approach of all the things you mentioned. Remember, their is no silver bullet :)

  23. Great article. I’ve been working in IT for about 20 years, and I started using this approach a while ago.

    It *doesn’t* automatically get you the ‘star programmer’ (I think there’s all sorts of stuff that needs to line up for stars to shine), but it really does separate those that have “the spark” from those that don’t.

    Where I work, we still use a technical test to try and identify the skills that we need, because having enthusiasm without any experience is nearly as bad as having experience with no enthusiasm…

  24. Tried this, didn’t work very well. Once that half hour was up and I got them to explain some code to me I had printed out and they would stumble. Think I will start with the code next time…

    • Hey Scott! What did you feel about the person once the half hour was up? Were you surprised to see them stumble with some code because after the first 30 minutes part you thought they were really good? Talking deep about a project should usually get you a very good idea of how someone handles code.

  25. Hey Nicolas, first I want to say I’m really glad you mentioned your starting point (QBasic) because we have that in common. Ohhh those were the days…
    You know even today I remember that feeling when I wrote my first QBasic commands, it printed olympic rings and I felt so accomplished :D
    Also in that first try I also learned the meaning of “bug”, I used incorrect color order :D

    Anyway back to an article, some time ago we also used the approach that you mentioned in “Just let ’em code” section, but I found that a problem with this approach is that you are presenting a commonly known coding questions and candidates who invested even a few hours in interview preparation would find this or a similar questions on the web and would prepare them self for it.
    So the problem is that although it seems that a candidate demonstrated a good problem solving skills, or algorithmic thinking, or coding conventions… actually he just demonstrated his memorisation capabilities without us even realizing it, until it was too late.
    That is why we currently use various small and straightforward coding tests for a elimination process (for example something like the programming tests from TestDome), followed by an interview and last the pair programming is conducted.

    Now regarding the single question approach, I do like the idea and it does sound logical. I mean when a person is passionate about his work then he is not confined with just a work experience but with a personal experience as well. Also I need to point out that when I encounter a candidate that has contributed to an open source project I immediately put him as one of the favourites.
    But I can’t help myself to feel that something is missing here, to me this is not enough for the evaluation process…
    Also few of my acquaintances come to my mind that would just fail at something like this, even though they are a great developers they are just not passionate about it. They treat it as a work, nothing more and nothing less. Yes it is true that they do not inspire anyone, but then again they do not have to.

    I would really like to believe that every passionate programmer is great at it, but I’m afraid that is not true. I have a friend who just had a bad luck at his early job positions. He ended up learning a very poor coding conventions and terrible practices and this kind of knowledge at the early stage sticks with you for a very long time… Even though he loves programming the problem is that because he had a series of bad mentors they left a nasty scar in his knowledge. This also resulted in decreasing of his learning capabilities for new technologies, because his learning curve consisted of correcting the previous understanding and adding an enchantments to it…

    • Hey Nicholas,

      Thanks for your feedback! I did find the similar issue with the coding interview. Also, I’m sure some great developers don’t think we take them seriously when presented with problems like FizzBuzz.

      Regarding passion, it’s definetely not a 100% success method, otherwise noone would try to perform interviews otherwise. But I found it to be more successful than the other methods I had tried, even though I’m sure I missed on some great developers out there using it.

  26. Your article makes me want to tell you all about my favorite project! Does it matter that I wrote it in 1975 on a time-sharing computer?

    My experience has shown that good programmers can learn whatever they need to perform well in a job. The point of the interview is to find a good programmer. I don’t care if he doesn’t know the language and tools that we use.

  27. Many years ago, I was in a similar position at a small company, interviewing and hiring programmers for my team. What’s interesting is that I basically used the same format and asked the same question. An interview with me was spent going over their resume and then asking them to tell me about their projects, what their role and what their contribution had been.

    It was interesting to see people with “good” resumes flounder when asked what they’d actually done, and to watch others come to life – even over what might be considered “small” or “side” projects.

    So – Here’s a hearty +1 and confirmation from someone who came to the same conclusions you did.

  28. Thanks for sharing great insights, Nicolas.
    I also like balancing it out with “tell me about a project you worked on but least enjoyed” to bring to surface some other traits the candidate may have.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>