From The Joel Test to The Intopalo Test From The Joel Test to The Intopalo Test

By Juanjo Diaz


On 10 August 2000, Joel Spolsky, Stack Overflow’s CEO published his famous “Joel Test”, a simple list of 12 question to rate the quality of a software team.

Here they are:

The Joel Test

  1. Do you use source control?
  2. Can you make a build in one step?
  3. Do you make daily builds?
  4. Do you have a bug database?
  5. Do you fix bugs before writing new code?
  6. Do you have an up-to-date schedule?
  7. Do you have a spec?
  8. Do programmers have quiet working conditions?
  9. Do you use the best tools money can buy?
  10. Do you have testers?
  11. Do new candidates write code during their interview?
  12. Do you do hallway usability testing?

It’s now 10 August 2017. Things have changed quite a bit during the last 17 years. I don’t live with my parents anymore and could grow a beard if I wanted to. So much has changed that even Stack Overflow employees are wondering if their beloved leader should come up with a revised version of the test. Here at Intopalo we always brag of our modern ways and our high-performing teams, so we have come up with a renewed set of questions to cover what we consider fundamental to assess the quality of a software team.

The Intopalo Test

  1. Do you use an effective source control process?
  2. Can you set up the development and production environment in one step?
  3. Do you build, test, and deploy (somewhere) on every commit?
  4. Do you have an issue tracker?
  5. Do you fix bugs before writing new code?
  6. Do you have a prioritized and estimated backlog?
  7. Do you have a clear vision?
  8. Do programmers have the working conditions they want?
  9. Do you always use the best tools for the job?
  10. Is QA an integral part in every step of the development process?
  11. Do you have a comprehensive recruitment process covering technical criteria and social aspects?
  12. Is UX an integral part at every step of the development process?

1. Do you use an effective source control process?

We all have spent a whole day doing a horrendous merge (that most probably broke the build and had to be fixed several times). We all have seen distributed teams overriding each other’s changes because it’s too much to wait the 7 hour time difference to discuss about it. We all have seen a commit with the message “Improvements” that changed 340 files and introduced 27 bugs. We all have seen that dark commit done directly on the master branch the night before the release, resulting in a huge bug in production. We all should agree that having source control is a must, but it doesn’t guarantee your team’s success.

What guarantees the team’s efficiency is a predictable and reliable production branch, a tagging system that allows you to rollback to previously released versions easily, an effective flow so merges don’t become a problem, and development branches that go through a thorough code review process. In conclusion, having a standard process that makes everyone happy and productive.

Here at Intopalo we use the GitFlow process, which, combined with agile methodologies, offers great flexibility and adaptability while enabling easy collaboration and ensuring predictable branch behaviour. We improve GitFlow with a strict pull request and code review process, standard commits and semantic versioning tagging for releases.

2. Can you set up the development and production environment in one step?

“Can you make a build in one step?” What build? Which one of my 345235 microservices build? Does that include the runtime dependencies?

I still remember my first job where it took me two whole weeks to get the system running, and it was a single service with a single database. And I didn’t even get to see the system in production. Ridiculous!
I also remember working for a company where half of the servers we had (MANY!!) were unknown, but no one dared to touch them because god knows what those servers contained or were there for (The well known snowflake server taken to the extreme).

Now, I simply run sh or docker-compose up and I have my NGINX reverse proxy, my backend, my frontend, my logging system, my database, etc. running. First meaningful view of the application happens 30 seconds after pulling the code using Git. SCORE!
Our whole infrastructure and servers are scripted and source controlled. I could tear down the whole network with all its servers and services in one click and restart them with another click (The phoenix server killed the snowflake one….).

3. Do you build, test, and deploy (somewhere) on every commit?

“Do you make daily builds?” Of course not! Why would I wait a whole day for that? If I break something I want to know immediately and fix it within 10 minutes.

In any case, the fact that a solution builds means nothing. Software is all about confidence. The confidence that your product works as it should so you don’t lose sleep over it. I want unit tests, integration test, e2e tests, exploratory tests, load tests, penetration tests, fuzzy tests, … Every possible test to keep my confidence as high as possible.

And, since I’m so confident that my software works, what’s stopping me from updating it continuously? I can spin up the whole system with one click at any time. Why would I wait until Sunday night to do a very stressful deployment, just to find out on Monday that there is still a bug and have to pink-sombrero it. I want everything deployed to test environments as soon as there is new code ready to ship and I want to promote that to production as soon as I’m confident that everything is ready to go.

4. Do you have an issue tracker?

It’s important to keep track of your bugs. However, it’s equally essential to keep track of your pending tasks so no one run out of work, to keep track of the tasks in progress so everyone can collaborate, and to be able to match old commits with the issue they tackled so a bug fix doesn’t become a regression bug and you end up with developers reverting each other’s fixes.

Each issue should be self contained, as small as possible, and as standard as possible. So anyone in the team can work on any issue with minimal dependencies to the person who wrote it.

Unfortunately, regardless of how self-contained your tasks are, there are always dependencies. A good issue tracker also helps everyone to have a clear view of tasks’ dependencies, blockers, etc.

5. Do you fix bugs before writing new code?

No changes here. You must invest in testing based on risk rating and fix all the bugs that are worth fixing before writing new code. The zero defects methodology has been around since the sixties and it’s now more important than ever. Bug costs grow over time so the sooner you fix them the better. Having a bug free code base at all times allows you to deploy any time with confidence. Also, fixing bugs is a good opportunity to familiarize everyone with parts of the codebase they might not be familiar with. It improves the team morale and increases productivity.

6. Do you have a prioritized and estimated backlog?

“Do you have an up-to-date schedule?” Schedule?? What schedule? In today’s crazy fast-moving world you can’t predict what your competitors will release tomorrow, that could force you to change your whole strategy.

So the key is not schedule. The key is the organization. Everyone should know at all times what’s the priority and what are the short, mid, and long term goals.

Alright, you have an issue tracker. You have good tasks, small and self-contained. But which task is more important? We already agreed that fixing bugs is the first thing to do. A nasty bug that brings down your whole database is different from a typo in a contact form. What about that task to integrate a new payment system in our platform? Market studies show that it’s not relevant to our user base so it’s not top priority anymore. Our competitor released feature a last week, so implementing it and doing it better is our current priority.

Of course, that’s overly idealistic. In most cases, you will be working within a budget and your customers will have strict deadlines. For that reason, it’s essential to know how long each task in the backlog is going to take, i.e. your backlog tasks need to be estimated. Tasks to be done in the short term need much more accurate estimation than the long term ones. Thus, just looking at the backlog, anyone should be able to say what can be achieved within a given resource limit (budget / time). This also allows better re-prioritization by making it clear which task needs to be deferred in order to prioritize a another one.

The larger and more distributed the team is, the more essential it becomes to have a well prioritized and estimated backlog. There’s nothing worse than having a very talented developer working on features that will never be released. Or even worse, waiting idle for someone to give him something to do.

7. Do you have a clear vision?

“Do you have a spec?” Did I mention the crazy fast-moving world already? I love companies that spend a whole year creating the spec of a product or feature, and by the time they are ready to code that feature is simply old and irrelevant (actually I don’t, it makes me sad…).

What I believe to be fundamental is to have a clear vision. A vision of what you want to have and how you want it to be. A vision that is not rigid and forcing, but a vision that’s flexible and convincing. A vision that allows to keep the backlog, to layout the tasks correctly and to overcome the issues created by the fast-moving environment we live and the uncertainties in the way.

8. Do programmers have the working conditions they want?

“Do programmers have quiet working conditions?” But I like to work with Metallica up to 11. Why would I work in an oppressively quiet environment? Are you also going to force me to drink coffee when I prefer tea??

Some developers work better in quiet environments and some in more collaborative ones, some developers work better from home and some need a very nice office, some work better during the mornings and some during the evenings, some prefer to work weekends and some prefer the weekdays, … the list is endless.

If someone is smart enough to code your mission critical software, they are surely smart enough to decide how they like to work. Give that person maximum freedom and you’ll see the increase in productivity.

9. Do you always use the best tools for the job?

Mr. Spolsky really focuses on the importance of hardware. And it’s true. My productivity significantly increases by having 2 extra monitors apart from my laptop screen and having a phone with reliable internet that I can use even when I’m abroad. It helps me enormously during business trips.

But what about software? I like working on a Mac and use Sublime Text, but the guy sitting by me likes his self-compiled Arch Linux and uses PhpStorm. You don’t want to burden your developers with unproductive on-premise tools when it’s much easier to share documents and collaborate using modern cloud services. As with hardware, software has a high impact on productivity.

And what about the actual technologies, languages, frameworks, services, and so on? Your developers should be empowered to choose whatever technologies they think works better for the job. It might be Node.js for APIs, Python for exploratory data science or Scala for big data jobs. It might be better to use .Net for that customer who is 100% Microsoft-based and deploys to Azure, but better to use Java for that other customer who uses Java Spring across all his services. I’m surprised when people ask what our tech stack is at Intopalo. There is no tech stack. We always choose the best tool for the job.

I would also like to mention that the world is moving towards open-source and standardization. NPM and other package managers have revolutionized the way software is built. Why wouldn’t you leverage the collective effort done by the open-source community? Don’t reinvent the wheel! Focus on what matters: creating value for your customers.

I was surprised when I started working at Intopalo and one of the first things I got was a company credit card. However, it has proven extremely useful and has eliminated “My mouse broke so I’ve been waiting 2 days for IT to get me a new one” type of problem. When I need something I simply go and buy it.

I was also very satisfied to being able to choose the technologies and tools for each project. I still remember the horror of working in a company that only allowed .Net technologies and explicitly forbade open-source libraries and services.

Once again, if someone is smart enough to code your mission critical software, they are surely smart enough to decide how they like to work. Give that person maximum freedom and you’ll see the increase in productivity.

10. Is quality assurance (QA) an integral part in every step of the development process?

According to Joel “If your team doesn’t have dedicated testers, at least one for every two or three programmers, you are either shipping buggy products, or you’re wasting money by having $100/hour programmers do work that can be done by $30/hour testers.”

Maybe that’s true if you pay someone with low education to look at your running product. However QA is much more than that. QA should be built-in to the product development, from the initial design all the way to the release. QA is code reviews, it is automated testing (on every commit, remember??), it is reliable operations, … QA is what lets you sleep at night knowing that you won’t be pulled out of bed at 3am because your servers are down, or the support phones are on fire because of your last release. It’s naive to assume that a cheap person looking at your product is comparable to your experts creating a comprehensive suite of tests, automating deployments and server provisioning, and so on.

Dedicated testers are good and necessary in many cases. However, QA shouldn’t be the responsibility of a few members of the team. It should be the responsibility of the whole team and should be part of your development process.

11. Do you have a comprehensive recruitment process covering technical criteria and social aspects?

Did you see my previous post about our recruitment process? You should! It explains the importance of evaluating all aspects of a candidate.

You don’t want a nice guy that can’t do his job and becomes a burden to everyone else. You don’t want a person who is an excellent coder but creates a toxic environment and ruins your team’s morale. You want people who can code, get along, and bring out the best in each other instead of trying to outshine each other.

This becomes even more essential when you move away from a traditional hierarchical organization to a more self-organized one. It’s preferable to lose a potentially suitable candidate than to hire a wrong one. As explained in the previous post, at Intopalo we have a four step recruitment process that allows us to ensure that every relevant aspect of the candidates is evaluated (motivation, technical skills, social aspects) and only the right candidates are hired.

12. Is user experience (UX) an integral part at every step of the development process?

Joel focused on hallway usability testing. However, unless you have an initial design you are increasing unnecessarily the number of iterations needed to get the UX right. Using professional UX and service designers throughout the whole development process minimizes the UX issues and reduces significantly the time spent fixing usability issues.

Competition is fierce nowadays. A product that does its job is not enough anymore. There is so much competition that a product not only has to do its job, but also be attractive and easier to use than the competition. Usability is more important than ever.