danbo finds a lightbulb — Credit: Flickr / Andrés Nieto Porras

What Makes a Good Technical Interview Question?

Three criteria of a good interview question:

Gives a sense of problem-solving and understanding.
Explorable and extendable.
Deeply understood by the interviewer.

Problem Solving and Understanding

What does this look like?

The problem presented needs to be decomposed into smaller problems in order to be solved.
There are a variety of different answers.

This criteria eliminates:

Knowledge questions and factoids. If a question requires memory of a minor detail that rarely arises in day to day programming, it is a bad question.
Answers that are easily searchable, any question where there might be a specific answer on StackOverflow.

Explorable and Extendable

Assuming people’s performance on this question falls on a normal distribution:

A few people will really struggle to come up with an answer.
A few people will quickly produce a perfect answer.
Most people will fall somewhere in the middle.

A good question needs to cater to this range of scenarios.

One approach we could take here is to ask a huge question up front, and just expect hardly anyone to get to the end of it. I’m not a big fan about this approach because it means almost every candidate leaves berating themselves for not getting to the end of it… that’s not an experience I want to give people.

Alternatives:

Build up.
- The first question constitutes part of the problem.
- Then situate it within a bigger problem.
  - Ask how does this change things?
- … and repeat.
Change constraints.
- The question is manageable in size, candidate comes up with a solution.
- Interviewer changes…
  - … constraints.
  - … requirements.
- … and repeat.

Whichever approach is taken here, some things I like to see are:

Reasoning about what changes and what doesn’t as the situation changes.
Intentional consideration of when code can be reused… and when it shouldn’t be.

Deeply Understood by the Interviewer

It’s easy to think that if two people ask the same question, they give close to the same interview. I don’t believe this is true. Technical interviews are not standardised tests, they are a system involving the interviewer, the candidate, the environment, the programming language, and the question.

Even when the interviewer, the question, and the environment are constant, with a question of any complexity, it’s going to be different each time, because different candidates will see the problem differently (possibly influenced by programming language, more on which later), and take a different route through it.

This is why the interviewers understanding of the question is so important. You have to be able to follow fundamentally different implementations, often also in a variety of languages.

The question is an island the candidate landed on for the first time. The interviewer needs to have a map, to help guide them through.

This brings us to one of the most important skills of being a good interviewer.

Interview Time Management

Think about:

How much do you expect a “good” candidate to get through.
How long should each section take.

Goals:

Time is spent in places proportional to what you learn from it.
The interview keeps moving along.

What does this mean? When the candidate gets stuck, you need to move them on from whatever it is that they are stuck on. How long you spend on that should be proportional to how important that thing is. Is the mistake a fundamental misunderstanding of the problem? That deserves some time to explore. Is it a typo? Who cares – move on. (NB: the examples below are for remote pair programming).

Fundamental misunderstanding of the problem:

Ask them what they understand the problem to be.
- If they have misunderstood, explain again.
- Ideally use a different way to express it than the first time.
- Write it down.
Ask them if their code will work.
Suggest a test case that demonstrates the problem.
Highlight the section of code where the mistake is.
Tell them explicitly what is wrong.
Fix it yourself and move on.

Off by one:

“I think you have an off by one”.
Suggest a test case that demonstrates the problem.
Highlight the section of code where the mistake is.
Fix it yourself and move on.

Typo:

“You have a typo, here I’ve got it.”

On Programming Languages

It’s important to understand that the programming language used can fundamentally change the problem.

It’s obvious that solving a problem in Haskell is fundamentally different from solving it in C. But solving a problem in Python is different from solving it in Java and both of these are very different from solving it in C. Languages don’t just differ in syntax but also in the library methods available, the ease of using them (see: sorting numbers in Javascript), and the choice of data-structures available in core libraries.

This means that you can’t compare progress. If someone writing Python got through 80% of the question and someone writing C got through 70% you can’t conclude that the Python programmer was better than the C programmer. They were doing different things.

One solution to this is that you constrain everyone to the same language. But if someone is writing code in the language they use every day, and the other person is not, then once again… they are doing different things.

My solution:

People code in whatever language they want.
I’m familiar enough with the most common choices (Python / Java / C / Javascript) that I understand how they differ (what’s harder, how problems are generally approached).
I ask questions about things I’m unfamiliar with (bonus: I get to learn things).

Takeaways

Finding a good technical interview question is hard. Expect to spend some time on it.
Deep understanding of the question is key to interview time management.
Two people asking the same question is not the same.
Two people answering the same question in different languages is not the same.

My previous writing on interviews: