Review of a Continuation of a Review

Zach Holman’s 2016 article, Startup Interviewing Is Fucked, is an opinionated piece which questions the effectiveness of startup interviews from one professional’s experience. Nearly 20 days later after Holman’s article, Aline Lerner measured the effectiveness of technical reviews and published a post on sharing his results.

Today, I will review Lerner’s continuation of Holman’s review (review-ception), bringing to light errors in methodology to discredit Lerner’s blog post. This includes an erroneous assumption and conflict of interest in Lerner’s interviewer sample.

I would like to start off by pointing out that there is no bias with Lerner’s association to his site when it comes to interpreting the data. In fact, its rather telling that a business built on coding interviews is criticizing their product.

Overview of Lerner’s Post

Now that you know how data was retrieved, lets talk about the results. I can’t elaborate on every detail but what you need to know is Lerner found that individuals who are on average capable of giving a strong interview (avg. of 3 or more in interviews) may still fail up to 22% of the time. Lerner ends by criticizing the interview process for being unreliable.

Criticizing the blog post

A secondary problem with Lerner’s argument is the sample of interviewers which provide data for him. Lerner did all the hard work of making sure that scores were not inconsistent based on the interviewer and created a whole algorithm to properly grade interviewee’s. However, he fails to take into account that requires you pay 100$ per interview (as of today). Anyone using the site would be looking to consistently have good interviews. This is going into conspiracy theory territory but bare with me. If applicants want consistency and they are getting inconsistencies, they will obviously try again and again, paying 100$ per interview. Unlike the real world in which companies actually lose $$ giving inconsistent interviews (they need to hire someone to fill vacancies), here the interviewer benefits. This means that score variability is actually independent of the hiring process, making technical interviews innocent.


This is going to be my final blog post for who knows how long. Perhaps I’ll eventually start writing my own blog posts. If you have taken the time to read even one of those six, I want to thank you. This isn’t something I do because I want to, but knowing someone will take their time to look over something that I have done is satisfying.

Until next time

~ Saiem



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store