Recently I wrote about how I used Hundred5 to hire six great programmers with very little effort. In that post, I discussed my general hiring workflow, as well my general rules for creating an automated skill test with Hundred5.

Today I will go further in depth and show you the reasoning behind some of the questions I use to find good candidates for our team of mobile developers right now.

Why automate testing

The biggest argument for automated testing is to reduce the effort it takes to filter through the hundreds or thousands of applicants we get for a typical opening.

To achieve this, each question on our short test has to test something clear and meaningful that gives us real and objective knowledge about the candidate.

The objectivity of the questions is vital. Any question where the answer depends on interpretation, or where the answer has to be checked by hand, at best will cause a lot of manual effort, and at worst randomizes test results, making the test essentially useless.

What we cannot test automatically

As such, we have to realise that not everything can be easily tested for. Skills that I consider almost impossible to select for automatically are: cultural and team fit, personality, work ethic, and other “soft skills”.

All questions I have seen that try to test for these fail for the same reason: They are too easy to guess ‘correctly’ if you know the least bit about the company you are applying for.

For example, should I ask candidates whether they agree with the statement ‘Done is better than perfect’, it takes only a five-second Google search to find the “correct” answer.

Lying on these kinds of questions is so easy that they tilt the scoring of the test in favour of liars while disadvantaging honest applicants – and I would much rather interview an honest person and judge them in person than talking to a dishonest one, no matter the latter’s qualifications.

These kinds of traits and skills are much better looked after during interviews and paid trial work (we call these “test weeks”).

What we can test

All is not lost however – quite the opposite in fact.

Almost any technical position requires reasonably well definable experience and skill. But there are some skills that come in handy for any kind of work and can thus be tested to filter candidates for almost any position.

For us developers an understanding of the ecosystem we work in, the programming languages we use and the ability for highly analytical reasoning are a must.

Similarly, attention to detail, general logical thinking ability and the willingness to tackle problems that might appear unapproachable are not only useful skills for programmers, but are indeed indicators of job performance for any profession.

Most of these skills fall under the category of “intelligence”. We could simplify and say that we are looking to find “smart” people, but while intelligence is an important measure, it is not the only thing that matters, and breaking things down helps us create questions aimed at specific skills and expertise.

What I ask candidates in our automated Hundred5 test, and why

It can be difficult to test the skills outlined above without context. Proper clinical IQ tests can be very successful at this, but using those can be a hassle and, depending on legislation, legally questionable. Further, they only speak for intelligence as a whole and do not consider experience in the field in question.

Therefore, it is better to design questions within the context of the specific position, requiring both good thinking and domain knowledge. I will do exactly that for my case of hiring programmers below.

Understanding algorithms

Reading and understanding code is probably the skill most developers use the most during a typical day. It is essential that a candidate can understand how a given piece of code behaves, even if it is a bit trickier.

To test this, I present short snippets of code and ask the applicant what they expect the output or the state of a variable to be after the code has run.

For example:

 

list = [14, 32, 54, 3, 249, 67, 128, 5]
x = 0
foreach y in list
  if y > 100
    continue
  if y > x
    x = y

What is the value of x after this code has run?

 

This piece of code is not very complicated, but it is not something that will be easy to answer for a non-developer. To determine that right answer, the applicant has to correctly understand the nature of loops, conditional statements and the continue keyword. They have to put the different parts together to understand that the algorithm searches for the largest number smaller than or equal to 100. All that remains is picking out that number from the list afterwards.

Short term memory

For those that think that such a question is too simple, note how easily we can modify it to require significantly more mental effort.

 

list = [14, 32, 54, 3, 249, 67, 128, 5]
x = 0
foreach y in list
  if y > 100 - x
    continue
  if y > x
    x = y

What is the value of x after this code has run?

 

The only change is the addition of `- x` in the first condition. All of a sudden, determining the correct answer requires stepping through the algorithm in your head with the given input numbers since the modified condition now depends on both x and y.

Doing so requires a much higher attention span than the previous question. The applicant has to be able to keep several pieces of information in their mind while stepping through the code at the same time.

Attention span

Being able to concentrate on a complex problem involving many parts is an important skill. This can be tested by presenting a complicated question where, under examination, most details turn out to be irrelevant. The candidate has to be able to determine what parts of the question matter to them, and which do not, and then draw the right conclusions.

 

Let a1 to a100 be the first 100 Fibonacci numbers (1, 1, 2, 3,
5, 8, 13, 21, 34, 55, ..). Let b1 to b100 be the squares of 
those numbers. Let A = a1 * a2 * .. * a100.

What is the numerical value of the following expression?

(A * A) / (b1 * b2 * .. * b100)

 

At a first glance, this question seems impossible to solve in the short time available. It seems to require calculations with large numbers, and perhaps an understanding of the Fibonacci sequence. In principle, any programmer can write code to solve this question, but none of these things are required.

Instead, solving this question only requires applying primary school level mathematics – something that every developer should be proficient in. Given the commutativity and associativity of multiplications, the numerator and denominator of the fraction in question are exactly equal, so the result is simply 1.

 

Read more: How to hire Backend Developers.

 

Coming to this conclusion, however, is not trivial and requires careful attention to the question, understanding what is asked, thinking it through and taking it apart before realising the solution.

Logical thinking.

There are many ways to test for pure logical thinking. Among my favourites are boolean satisfaction problems. Simply put, I present a list of statements that refer to themselves and each other, and ask the applicant to check these statements so that all true statements are checked, and all false statements remain unchecked.

 

  • [ ] Exactly one statement is checked
  • [ ] The statement below this is not checked
  • [ ] The statement above this is checked
  • [ ] At least one of the statements above or below this is checked
  • [ ] The first statement is checked
  • [ ] There is a solution to this question
  • [ ] There is no solution to this question

 

Answering this question correctly requires attention to detail, a good attention span, and abstract thinking – all skills I would like new hires to have.

Domain knowledge

Testing domain knowledge can be tricky since questions quickly turn into a trivia quiz with answers that are easy to search for online. If we also include some analytical thinking however, we can minimize that risk.

For example, one thing I like to ask about is the performance characteristics of a piece of code. While one can be a decent developer without a deep understanding of this area, understanding the performance of both existing algorithms, as well as code we write ourselves is important in making sure our applications process data quickly and do not waste time, power, and money.

 

f(n)
{
  if n < 1000
    return 1
  sum = 0
  for i = 0 to n
    sum += i * i
  return sum + f(n - 13)
}

What is the runtime complexity of this function?

  • ( ) Θ(n)
  • ( ) Θ(n^2)
  • ( ) Θ(log(n))
  • ( ) Θ(n * log(n))
  • ( ) Θ(n + log(n))
  • ( ) none of the above

 

By presenting a small piece of code that includes both loops and recursion as well as several mathematical operations and constants I test the candidates understanding of these concepts, and how they affect performance. They need to have an understanding of all the individual components and be able to put them together, realising which parts are important in this example, and which are not. This also tests their attention to detail.

Putting it all together

All of these questions have objectively correct answers and follow my rules as lined out in my previous article. They test real experience and skills relevant for the position in question.

Using these questions and Hundred5, I can automatically select the top 15%, or even top 5% of all candidates and move on to further stages of our hiring process with them. All this requires only the effort of setting up a test once and can yield many great candidates and hires for several months.

In addition to the questions above, I would usually also include a few more questions that relate more closely to the technologies and ecosystems we are working with, but I will leave that up to your specific case.

I hope these examples have given you some idea of how to approach automatic applicant filtering with Hundred5. Please feel free to share your own thoughts and ideas in the comments below!