Bugs can be discovered during the development process, and other times it’s discovered after the product has been launched to the market. When we encounter a bug, we are faced with the issue of prioritization. How important is it that this bug gets fixed now, versus later? In this article, I will go over some of the ways that Product and Development Teams handle bug prioritization. (Disclaimer: not all teams are the same, and every company will deal with bugs in their own unique way. This is my personal experience from working at my last company and does not reflect Square’s internal process.)
Let’s start with the definition of what a “bug” really is. To loosely put it, a bug is when the software behaves in an unintended or unexpected manner. When you’re using a mobile app, if you tap on a button that is in the top right corner, but the button in the top left corner responds instead, that would be a bug.
When a bug is identified, a few things happen. We work to ensure that the bug is reproducible (Steps to Reproduce), we identify what the current behaviour is, and then we identify what the user’s expected behaviour is. In an ideal world, if we cannot reproduce the bug, then the issue is closed, and the user is notified as such. In many cases, users will not be notified when their reported bug has been looked at, and what the status of that bug report is. It’s great that bugs that are reported here in the Beta Community have a status attached to them, so that you are kept in the loop. When we can replicate the bug, we move on to prioritizing it within our backlog.
In the real world, a product may have multiple bugs. Each bug is important to the person who reported it, so how do we decide if this bug needs to be resolved now, or if we can resolve it later? Teams don’t want to only solve bugs that are brought up by 1 user (even if this user is responsible for 90% of the revenue of the company); rather they want to be fair, and resolve bugs that affect a high amount of users.
At my last company, I ran the bug backlog grooming sessions. When I took over, we had a ton of bugs, and some of them had been around since the beginning of time. I sat down with a few stakeholders, and we reevaluated all of our existing bugs to ensure that 1) they were reproducible (we had gone through quite a few version changes for some of these bugs), and 2) that the current priority it was given was still accurate and if not, we could change it.
Key stakeholders for me were the Customer Support Team, the Onboarding Team, the Product Managers, and a Lead Developer. We needed Seller-facing employees to tell us if they were hearing about this issue today and the general customer sentiments towards this bug, and we needed Product Managers & Developers to tell us how the feature was supposed to behave.
This was the general prioritization scoring sheet we used. We (myself and the stakeholders) gave each of our products (or features) a score. I can’t disclose how we came to each score, but generally speaking, revenue to the company, and number of overall users were considered. Next was the Product Impact. What kind of impact did the bug have to the product experience? If the bug caused a crash, the impact to the user was a 10, the highest impact. If the bug caused a text field to not be centre-aligned, but was otherwise usable, the impact to the user was negligible and could be scored at a 1 or 2.
Next came the Customer Impact. We wanted to keep track of who was discovering these bugs, and how many potential users would be affected. The distinction between an internal vs external discovery was important for us to measure, and we added tagging in our tickets to help us generate reports that highlighted these things. The more users it could potentially affect, the higher its score. If it was a bug in a screen that most people saw (for example, the Sign In page for a website), that meant the possibility of people encountering it was fairly high, and we would score it at a 5. We would add a point for every case reported through our Customer Success team (that’s the Customer Score), and because we were a smaller company, we had VIP ranked customers and they automatically bumped the score up by 5 points.
Once you have a score, the spreadsheet can automatically tell you its priority. The numbers that you plug into the formula will differ from team to team, and from company to company. What’s important is that all of your stakeholders agree on the numbers and the priority levels.
So how did we get the priority levels of each bug? We used the following formula:
IF F2 (the cell that displays the sum of the scores) is greater than or equal to 30 points, it is a P0. If F2 is greater than or equal to 24 points, it is a P1. If F2 is greater than or equal to 18 points, it is a P2. If F2 is greater than or equal to 12 points, it is a P3. If F2 is less than or equal to 11, it is a P4.
SLAs are discussed and agreed upon, and each bug is given the First In, First Out (FIFO) treatment. What that meant was that even if you had 5 different bugs, all scored at 18, the first bug to be reported would be first on the backlog, and the first bug to be worked on by a developer.
Making sure to include Steps to Reproduce, Current Behaviour, and Expected Behaviour meant that any developer that was familiar with the product (and had time/effort within their current sprint) could take on a scored bug. I scored the bugs weekly with my stakeholders, and developers took on bugs from the top of the backlog by themselves, without needing to be involved in any additional meetings. Meetings cost money, and the less time developers spent in meetings, the more time they could spend shipping features, and fixing bugs.
Encountering a bug sucks. Developers never intend on creating bugs, but when we go through our QA process, we may not cover every single possible scenario our users do when they’re using our products. Product Managers and Product Designers envision a “happy path” for their products but users don’t always use our products in the way we envision, which is good but can create situations where unintended bugs appear. When you report a bug to us, it is crucial to include steps to reproduce the bug, what device(s) you are using, the OS version and some photos or videos. The more detail you give us, the less back and forth we’ll have, which means the sooner a developer can look at the issue, and the sooner the issue can be resolved.