A few weeks ago, I posted an ops job listing to Twitter and LinkedIn. I was worried that we weren’t going to get any applicants, but I turned out to be very wrong.

We received over 1,000 applications for this one job posting.

Since then, we’ve made one hire. So we’ve had to reject 99.9% of people. Many of the applicants reached out to me asking for feedback and I haven’t had the time to get back to everyone. If you applied and are reading this, hopefully this gives you some insight into our imperfect hiring process.

The Initial Application Applicants had to fill out an Airtable form with some basic information and then some questions specific to Luma.

Here are the more interesting questions:

  1. Share an Event — We asked them to list an event that they thought was well run. This question was pretty low signal. It gave some insight into their thinking and writing quality, but it was tough to screen on.
  2. Tests / Awards — I wanted to let people share something they have acheived, but I realized if this was too open ended I would get things like “won Carver University’s top undergraduate” and I would have no idea how to rank that. So people could submit tests scores or awards. This helped a few people with high test scores stand out.
  3. Example Support Response — This was by far the most helpful question. I gave an example support question from a user and I asked applicants to respond as if they were on the Luma support team. Most of the applicants who filled this out got the answer wrong. And I suspect a large portion of those were using AI to write their answer.

We got a ton of applicants, so I set up a script that would go through and check their application for two basic things — did they complete the application and did they get the support answer question correct.

Let’s get to the numbers:

  • 1,260 People Applied — This is the number of people who filled out their application and hit submit.
  • 878 People Completed the Application — Over 30% of people left off a required question on the application.
  • 262 Answered the Support Question Correctly — Only 30% of people took the time to understand the product and answer the support question correctly. This was the biggest filter step in the entire process.

Scoring -> Demo Videos OK, now we have 262 people. It’s more manageable, but it’s still a ton of people.

We need to get more data. So I created another step of the pipeline. Applicants would create a Luma calendar for an event host they admire and then recorded a one minute Loom walking through the calendar.

This tests their ability to create a nice looking account on Luma and to present it over video in a clear and compelling way.

But I didn’t want to ask 262 people to do this. I don’t want to watch 4+ hours of Loom videos and I didn’t want to ask people to take on a project if I wasn’t going to be able to review it.

In order to whittle down the list, I created an AI prompt that ranked each person on a level of 1-10 based on the writing in their application, their background, and other traits we looked for. I don’t think the AI did a particularly good job here, but I had to have some way of reducing the list to a manageable number.

We ended up sending the Loom video assignment to about 30 people and 20 of them filled it out. This was super high signal! Many people didn’t follow the instructions or felt uncomfortable on video.

Work Trial → Hire As a next step, I asked a couple of the most promising candidates to do a paid work trial. In this week, they did the job we were hiring for but with more support and training.

We had two candidates take on work trials and hired the candidate from the second work trial.

Recap The job market seems hard right now. This is not a particularly selective role and yet we only hired 1 in 1,200.

I think a few things are happening here:

  • There are many people who are applying to a ton of jobs. Some of these must be fake or automated or otherwise garbage applications.
  • People are using AI to turn in “reasonable” job applications. It’s so much easier to fill out free form fields with proper English these days. The support question helped filter out a lot of these people.
  • It’s hard for candidates to know what steps of an application to take seriously. I think a lot of people were demoralized and didn’t take the time to really care about the job application - probably because many job applications aren’t read. It’s an understandable feeling, but it’s a negative loop.