Reader,

Wow, are we halfway through February already? That was fast. Hope you had a great V Day over the weekend 🫶

Let's talk about another interview I had back in December.
This was with a facilities management SaaS company for an SDET role.

A friend from a past job referred me, so I thought it would be a walk in the park.
(This is a terrible posture to take when getting referred, by the way. Great recruiters often speak with referrals so make sure you educate your candidate on mindset before hooking them up with your managers 😉)

I'll spoil the ending for you -- I didn't get an offer, but I did make it through all the interviews.

So what happened?

The Req

The role asked for these skills (I've passed it through Gemini to conceal the firm):

Some insider knowledge the job description didn't mention but which I found out during the interview:

• The hiring manager already built the "bones" of the mobile automation framework -- he was just hiring a steward to autonomously maintain and scale coverage for the mobile app
• The hire would be expected to occasionally contribute to the Playwright suite for the web app

So, in a nutshell, they needed someone who could...

1. write automated test code
2. work independently

But they didn't need an architect or a mentor. Just someone to crank out tests and fix the framework when it broke.

A Mid-Level QA Engineer, basically.

So how did they interview for this req?

The Interview Process

To their credit, it truly was as streamlined as it gets (both a good and bad thing, btw):

1. Phone Screen -- explain the strange comp situation, the required on-site onboarding for Week 1, and get a gauge on the candidate's QA experience
2. Hiring Manager Conversation -- HM asks about past work experience to get a sense of what kinds of problems the candidate has solved
3. Technical Conversation -- HM, Software Engineer, and SDET all ask the candidate questions about the test automation topics to understand the candidate's technical opinions

Did you see code in there? I didn't.

At the time, that was a bonus for me. Personally, I don't love coding exercises. I don't know many who do. Especially when it feels high-pressure or some contrived Leetcode problem that none of the SDETs on the team know how to solve.

The Decision

When they ultimately passed on me, they told me there was another candidate who slightly beat me but they wouldn't give a reason.

So I ended up pulling my friend aside who referred me and asked what happened.

They told me something so shocking that it changed my entire view of the hiring process:

Let's break down the problems this feedback reveals.

"Vibes-based" hiring

They made a $130,000/year decision based on a personality type.

Planner vs. Adapter.

They wanted a Planner. But can the Planner code? They won't know until Month 1.

The problem with conversational interviews -- as much as I do love them, I'm a yapper -- is that it's tougher to learn the candidates true technical abilities.

Anyone can learn enough lingo to "talk the talk" about testing and test automation. It's even possible to make up convincing work experience stories.

Who's gonna know you didn't actually increase test coverage at your last job by 50% using Playwright or cut test runtime by 70% by parallelizing jobs in GitHub Actions?

AI is great at helping people cheat in conversational interviews nowadays.

A quick aside here about personality type, too:

• Planners plan to succeed
• Adapters plan to learn

If you have "ambiguity" in a project, you don't want someone who needs to calculate their every move. You want someone who understands that clarity comes from investigating and iterating.

But hey, I'm a little biased 😉

The "years of experience" trap

There's no way of knowing what the "density" of that candidate's years of experience is.

I use the term "density" to refer to the amount of learning and achievement baked into each year.

(Maybe it's a thing already, haven't googled it.)

Let's take 2 imaginary candidates with 6 years of experience, for example:

Candidate A

• 3 jobs x 2 years at each job
• Job 1 asked for mobile automation and API testing scripts, Azure shop
• Job 2 asked for a web automation framework built from scratch with Playwright/TypeScript
• Job 3 asked for CI/CD optimization in Jenkins for a desktop test automation framework

Candidate B

• 1 job x 6 years
• Job has the candidate becoming a master of Cypress / JavaScript and GitHub Actions for CI/CD
• Scaled coverage and modified CI/CD pipeline

Candidate A has proven their ability to adapt to new challenges and build stuff from scratch.

Candidate B has more depth of experience with certain tools but is mostly writing test scripts.

Drastically different skill profiles with the same years of experience. It's like what mom always used to say:

Recruiters!

You’re responsible for ensuring the best cultural and technical fit.

In this case, the process wasn’t structured around clearly defining what success looked like, and the recruiter waited an extra week while the team "came to a decision".

And after all that, it ultimately came down to “who is stronger” with a gut-based choice.

Hopefully that role doesn’t end up back on the recruiter’s desk in six months. Make sure you’re partnering with hiring managers early to clearly define success for the role so you can maintain control of the wheel.

In Part 2, we'll talk about how this interview process could be improved so the hiring manager doesn't have to hire based on "vibes" and calendar years.

-Steven