Selection Factors

A major study and meta-analysis has some useful ideas for what works best for interviews:

• General Mental Ability: while IQ tests are not legal to administer due to bias, a good lesson here is to focus on how candidates solve problems instead of whether they know the "password" to a brainteaser.
• Structured Interviewing: Use consistent questions and pre-determined clear criteria with detailed feedback to reduce the propensity for interviewer bias, and create the opportunity to assess question performance over time.

Phases

The rest of this talk will orient around the three phases of an interview:

• Before
• During
• After

Before: Prerequisites

A hiring manager should pre-commit and articulate the following:

• What does our ideal candidate look like? What skills should they have? What problems should they be able to solve? What is the comparative advantage of our organization and given tradeoffs between different skills, which ones are higher priorities? Everyone wants to hire the unicorn who is good at everything, but who is the non-unicorn who would excel in the role we have open?
• What is the role of the interviewer? Given a hiring loop with multiple interviewers, who is evaluating which dimensions and attempting to acquire what signal?

Before: Scripted Question

If there's one thing you should do, it's to use a scripted question and rubric - while this won't eliminate bias, it does help to reduce and compartmentalize it - after all, your evaluations shouldn't depend on whether you're grumpy right before lunch.

Walk into your interview with the following in hand:

1. A scripted statement of the problem.
2. A reference solution of the problem.
3. Common mistakes, errors, and paths people take through the problem.
4. A scoring rubric for the problem.

Before: Practice

The second most important thing is to practice. This will help you smooth out things like confusing problem statements that may impact candidate performance without providing any signal.

• Run your question on a few people before you go live.
• Have someone run it on you.
• Have someone shadow you running the question.

Interviews tend to be pretty noisy, which can make practice awkward, so make sure to establish a rapport with your partner.

2-4-6 Game

2-4-6 Game

It turns out that you want to ensure your interviewing properly classifies both positive and negative samples.

You can achieve this and mitigate some of the awkwardness of practicing by having the candidate flip a coin and then use an artificial limitation to hinder performance.

During

You now have (most likely) 60 minutes to arrive at a concrete decision, which means you want to have the candidate spend as much time as possible on the interesting parts of the interview.

Imagine the candidates walking in and facing a completely blank map - you have an opportunity to fill in some of it, so that the candidate will spend more time in the sections that you are actually trying to assess for.

During: Structure Preamble

I like to start by eliminating as much meta-ambiguity as I possibly can, by explicitly describing the structure of the entire interview: it eliminates guessing about remaining time, shifting requirements, and surprise anxiety.

1. Describe the overall structure of your interview (major phases, time allocated), and the high level of what you'll be assessing for.
2. Give a very short background description of yourself, to help the candidate understand their audience (and to also help them think of questions to ask later).

"I'll spend two minutes telling you about myself, 45 minutes on a technical evaluation of your programming skills with a laptop during which I'll be taking notes, and I'll leave 10 minutes at the end for you to ask me any questions you might have."

During: Evaluation Preamble

Before starting a particular evaluative question, I also like to eliminate as much ambiguity as possible by describing the evaluation structure:

• Are there multiple parts, and how long is each part expected to take?
• What matters and what doesn't? (E.g. - "Attempt to write working code, but you won't fail if you miss a semicolon on the whiteboard, and don't worry about degenerate inputs")

During: Evaluation

During the evaluation, you should be focused on bifurcating the candidate pool as well as possible with the time you have remaining.

Is the candidate currently trending towards "yes" or "no"? In the time that remains, what do you need to hear in order to reach a concrete decision one way or another?

After: Feedback

When writing your feedback, cover a few major areas:

1. Use your rubric to assist you with a structured decision.
2. Provide top-level relevant insights ("strong problem solver") backed up with useful concrete details.
3. Include general notes after light curation to remove excess noise.

After: Logging

The one final step I recommend after interviews is to log your decisions. If you keep track of:

• What you recommended
• What you predict about the candidate

Then for some candidates, you'll be able to revisit the issue 6 months later and get some concrete data and how your interview performs.