Building software modules within a team requires communicating, at a minimum,  method or class signatures and a general idea of the functionality.  This may include a detailed design document clearly specifying method signature and a pseudo-code implementation or method stubs and comments about what the method should do right in the source file.  One thing that should never be used for communicating a signature and functionality is dummy code.

Dummy code looks like real code.  It is real code but it rarely works.  Even if it compiles it doesn’t produce the correct result when run.  Nor are there any tests that prove it (in)correct.  Dummy code exists only for the sake of its author’s gratification.

Rather than talk about the desired functionality through some more productive means the dummy code author thinks he is helpfully providing a guesstimate of correct implementation.  What he has actually done is provide a jumbled mess of non-working code cleverly disguised as a working part of the project when he checks it in to the SCM.  The dummy code author thinks someone else will come along and provide the correct implementation “later”.

What he doesn’t realize is that his dummy code is almost indistinguishable from code that should be part of the project.  The next person to try to reuse this code is in for hours of WTF moments trying to figure out WTF the code is supposed to do and why it’s got code branching five levels deep.

At my first post-college job I had a wonderful mentor.  Thys (pronounced Tays, like “taste” without the second ‘t’) and I were talking one day when he asked me a question.  “Anthony, if you were given a task of writing software that would only be run once would you do it quick-and-dirty or with a well-measured, engineered approach?”

How bad could “quick-and-diry” be if the software would run only once?  I thought about it for a moment before answering.  Quick and dirty.

Thys said he would never write quick and dirty software.  Doing so figuratively creates a  monkey on the developerment team’s back.  The software that only needs to run once inevitably needs to run a second time and a third time and eventually run regularly.  Thys helped me realize that software evolves and escapes.  Useful software will run indefinitely.  Any time and effort saved on the initial write will be lost 10 times (I made that number up.  It is actually higher.  Any want to provide real data or examples?) over on the maintenance, bug fixes and rewrites.

Writing dummy code is writing software quick and dirty without any idea of the implementation details.  Bad code and bad algorithms.  The worst of both worlds.  It’s been a long time since Thys and I had that conversation but the years have proven him right again and again.  Any code worth checking in to SCM is worth writing well.  That includes unit tests.

Dummy code has no place in SCM.  If you’re not going to correctly implement a method you stub out then don’t provide an implementation at all.  Leave some comments around the method stub if you have to.  Open a ticket in the issue tracking system to let someone know the method needs an implementation and describe the input and output.  Let the implementer determine which algorithms and data structures to use.

Dummy code is a hazard to software projects.  It violates the principle of least astonishment.  Next time you think about writing some remember that the next person that uses the code will waste time figuring out why it doesn’t work as expected.

If you check code in to your SCM where other people can pull it, take a look at what you’re checking in.  Does it work?  Does it work well?  Are there tests that prove it?  Helping someone by coding less is sometimes the best help of all.

I took on the project of integrating the start-up application with an application made by one of their customers.  The start-up application was running an instance in production for another one of their customers.  Part of the project was defining and building a RESTful API that both my customer, the start-up, and their customer could use to communicate between applications.  The API had to include comments made to different event types.  When someone makes a comment on the start-up app it gets propagated to their customer app.  When someone makes a comment on their customer’s app it would be sent to the start-up’s app through a RESTful API call.  It would be similar to having Facebook and MySpace synchronized on your status updates.  Update your status on Facebook and MySpace reflects the change too.

As I built the API I found that their comment model didn’t work.  I looked back a few months in the SCM and found the methods in the model hadn’t worked in months.  Comparing that version with the previous version showed that everything except the class definition was new.  The comment model never worked.  The comment model had no unit tests to verify its functionality is correct nor did QE raise any questions or file any bugs on it.  The developer never actually ran this code.  The mistake would have been obvious if they took the time to run it.

Asking if anyone knew that commenting didn’t work was met with blank stares.  “No, it works.”  I asked them to demonstrate on any environment they had set up.  I hoped I was wrong and I didn’t have my environment set up right.  Leaving a comment failed every time on every environment.  I found out someone checked in a fix “last week”, but that developer never bothered to run the code either.  I don’t know how he verified his fix.  No one could leave comments anywhere, not even in production.  Yep, this code was broken for months in production for another customer and no one noticed.

I finally found out someone was “cleaning up” the comment functionality.  I asked about the precautions the developer is taking with this bug and was told something I already knew – how to “temporarily” fix it while the developer worked on the “clean up”.

Fixing the bug was beside the point.  This company needed to figure out a way to prioritize development.  Priorities save developers from wasting time building software that no one will use for months, years or ever.  Part of the start-up mentality is “throw stuff against the wall and see what sticks” but at the cost of basic functionality?  That’s a poor implementation and a poor business decision.

Getting off to the right start in a project is critically important.  Automated test frameworks are available for every major software development platform.  Not using them right at the start of a project makes it impossible for meaningful tests to be added later.  Not difficult – impossible.  We don’t use the strict  “waterfall” methodology anymore because testing after development is complete doesn’t work.  As a community we have known this for a long time.  Short, tactical releases are the way to go.  These short iterations are the perfect time to write tests with or even before writing the software.  Test-driven development and behavior-driven development are hot for a reason.

A long time ago I picked up some work for a shop that had a test-first mentality as part of the culture.  Before day one at this company they decided a test-first approach was the way to go.  Every developer was held to unit-testing their own software.  They also didn’t build any software that wasn’t needed for the current iteration.  Looking to build functionality needed “in the future”, they learned, was a waste of time.  That future never came.  Instead they used that time to test the software that was needed for that iteration.  They typically had well-scoped projects and were customer driven.  They had a lot of customers.  Somehow they found a way to please everyone.  Short, planned, targetted releases made those guys a lot of money and it was a pleasure to work with them.

I was doing some work for a startup where I got to see the CEO warm up for his series B funding pitch.  He wanted to do the actual presentation he would give investors with an audience.  He invited me, the developers and some people in marketing to the practice talk.  He wanted us to think like the investors and ask some difficult questions about the company, product and business plan.  When asked about the kind of people he needed to hire for his plan to work the CEO quoted General Erich von Manstein to illustrate his point.

There are only four types of officer. First, there are the lazy stupid ones. Leave them alone, they do no harm. Second, there are the hard-working intelligent ones. They make excellent staff officers, ensuring that every detail is properly considered. Third, there are the hard-working, stupid ones. These people are a menace and must be fired at once. They create irrelevant work for everyone. Finally, there are the intelligent lazy ones. They are suited for the highest office.

The CEO claimed to the the fourth type of person, intelligent enough to make the big decisions and lazy enough to delegate the actions to someone else.  I’d like to think I work with reasonably bright people so I didn’t have a problem accepting his claim.  Moving on to the dumb and stupid, the CEO said he would never work with someone he classified as such.  They are a waste of capital and management.  They produce no results.  I tend to agree.  I never accept work from someone I deem waste of my time.  The stupid and hard-working produce results but leave so many problems in their wake that the intelligent and hard-working have to clean up after them.

The intelligent and hard-working are the people the CEO needs to bring his business plan to life.  His argument, though, was poor.  He claimed the intelligent and hard-working aren’t actually intelligent.  If they were intelligent they would be intelligent and lazy making the big decisions.  The intelligent and hard-working, he argued, are just smart enough to not create a mess of problems while they work.  They’re nothing more than worker bees who know enough to listen to his direction and not smart enough to make a decision, develop an idea or even communicate with other people like he could.  He only wants to work with the intelligent and hard-working because they were the ones that would do his bidding but they are not smart enough to question him.  He said this in a room full of people he worked with, his employees.

I looked around at their faces.  Some were shocked he would so openly insult them.  Some were disgusted by what he said.  Others showed disappointment.  Disappointment in I don’t know what.  Maybe they were disappointed in themselves for coming to work for a man that thought so poorly of them.  Either way the company morale plummited.  No one missed the implication of the CEO’s words.

Perhaps he is a very bold man.  Or he might not have realized what he said until it was too late.  Either way he seemed unfazed by the fact he deeply insulted everyone that worked for him.

The company failed.  Series B never materialized and he laid off all of his worker bees a few weeks after the warm-up talk.

Here are some of the topics I want to talk about:

• Data growth and access – What kind of DBMS topologies allow maximum scalability without breaking the budget? What if the budget wasn’t a problem? How do you migrate your data model from hundreds of users to millions of users? Should you partition user data across shards or keep it in a central database? How do you profile data access paterns? Is your application mostly-read or mostly-write? When should you use an LDAP directory for data storage? When should you use MySQL and when should you use Oracle?
• Application server scaling – app server clustering, web server integration, load balancing, application session management, data caching
• Web Tier – load balancing reverse proxies, data caching, working with HTTP, considerations for exposing functionality via REST
• Language conisderations and platform choices – Dynamic languages vs. Static languages, Linux vs. Windows, Solaris vs. Linux, RedHat vs. Oracle, IBM vs. Oracle – How do you make these choices? What evaluation critera are most important? Which ones are misleading?
• Framework scalability – How can you scale if your framework can’t? Does Rails really scale better than Spring Web MVC? Where do PHP frameworks fit in? What are the alternatives? This is where we will investigate what you gain and lose by binding yourself to a particular framework, how to keep the coupling to a minimum and how to use your framework as a solid foundation instead viewing it as a cage.
• Emerging technology – Should you become familiar with Map Reduce and Hadoop? What kind of impact do object databases have on your application? Should you buy your own Sun or Dell boxes or use Amazon’s EC2 and S3?
• Application architecture – How do you write an application that scales? What does your application look like as it grows from servicing hundreds to millions of users? How does it handle session management? How does it access datastores? Are there any design patterns that are helpful? What are the anti-patterns to be aware of?

Like I said, I haven’t found a group dedicated to discussing these topics. If you know of one in the Boston area please let me know. Assuming there isn’t one I am willing to start one. I’d like to start off small and meet at coffee shops around Burlington or Lowell. I have no delusions that this is going to start off or even become as big as NEJUG is now. If it starts off as a few people getting together to talk about scalability trends, cool caching solutions and specific products then I’d call it a good start.

The meeting location is still TBD and will be based on how many people are interested in attending. It will probably be somewhere in Burlington, MA. There is no planned presentation at this time. Instead we will have a meet and greet and then discuss scalability trends and news, what approaches to scalability are commonly used now and what are the plans for the future.

If you are interested in coming or in finding out more information please email me at <my first name>@<my domain name>.<my tld> or leave a comment below. Please make sure to include your email address when you fill out the form so that I can get in touch with you – your email address will not be displayed on my site.

To some people it might seem that a lot of what I’ve written about here is common sense. The condensed version of this text might be “go talk to QE”. But surprisingly there are exceedingly few people that actually do it. QE, or any other group for that matter, is a mysterious team with bad intentions that lives on the other side of the fence we through our software over. Working with QE over Bugzilla and email is like a 2400 bps modem. You just don’t get what you need over it. Software developers are typically even more prone to this reclusive behavior due to the kind of people attracted to the field. I struggled with the same social problems for years, usually with less than satisfactory results. Eventually I found a way to work with others that works well for everyone. I had to struggle against my strongly introspective personality and force myself to communicate verbally, in person, with people around me and affected by my work. Why? Because it didn’t seem like anyone else was doing it and there had to be a better way to deliver high quality software that people wanted to use.

By paying attention to these things the quality of my technical work improved and my visibility to people both on and outside of my teams skyrocketed. I’ve made an effort to improve how I work with others and in doing so I’ve made a name for myself. Possessing superior technical skill is of little use if no one recognizes it. Think about that for a minute. You can be the greatest programmer of your generation and you’re not going anywhere if you can’t convince others you’re worthwhile.

Oh yeah, what effect does this have on the bug triage meeting? If you’ve read this far then you know you’re showing up loaded with ammo. Every bug you talk about is going to be important because you’ve eliminated all the small ones along the way. Imagine being in a room with your peers and a cross functional team where every word you say is inherently more valuable because you’ve done your homework. You’ve put in the effort to make this meeting more valuable for everyone else that attends. You’re not wasting their time by even looking at bug 34987 that only had a minor impact on documentation. You’re in front of an audience that wants to discuss things that matter, business or technical.

No one wants to get up in the morning and think, “I want to sit in a bug triage session all afternoon today and again tomorrow”. In this meeting you’re able to talk about every bug as it comes up. You know the issue, you know the root cause and you know what kind of scope it takes to fix the problem. That’s the kind of information that’s needed at bug triage sessions. That information helps to determine the severity and priority. It let’s people decide whether or not the bug needs to be fixed right now and that is exactly the point of bug triage meetings. People will notice that you’ve done your homework and they will appreciate it. It will get you noticed. People will realize you have in-depth technical skill and you can work well with others which almost guarantees a position as a successful technical lead and it will open up a lot of other doors in your career. What, you just wanted to make the bug triage meeting shorter?

Well, like software this paper isn’t perfect. I’m certain there are missing topics, incorrect statements and gaping holes in logic and arguments. There are entire books written on this subject and I don’t have any delusion that I’ve covered even one one-hundredth of 1% of the social issues that hold our industry back from consistently releasing high quality software, on time, on budget. But, like I do for my software, I keep a list of known risks for this paper and I’ve decided that it’s finally time to ship it, bugs and all. Congratulations on reading the entire thing and bigger congratulations for just skipping to the end if that’s how you got here. If you’ve found this to be worthwhile please submit it to Digg, Slashdot, StumbleUpon, del.icio.us or whatever other social network you use. Also please leave your comments and discuss what I’ve written. Feel free to add topics you think would be interesting.

There’s the old saying “when you’re in a hole stop digging”. So far we’ve talked about how to stop digging but we haven’t talked about how to get out of the hole yet. A bug list is a bit like Pandora’s box in that once the bugs are open they’re like all the sorrows let out of the box. You just can’t put the sorrows back in the box and you just can’t get rid of all the bugs. What happens to you when you’ve got a bug list a mile long? You try to avoid looking at it for one. But there are always people in management that subscribe to the mange-by-numbers theory who take bug metrics a little too seriously. Eventually you’re going to make it to the top of the naughty list when they’re looking at who has the most open bugs. This can be interpreted in a few ways and none of them are good. They could think you’re a terrible developer if you write so many bugs. They could think you’re lazy if you’ve got such a huge backlog. They could think you can’t prioritize well if they see something related to their pet topic is on your bug list and you’re working on something different. They could think any number of different things and whatever they come up with definitely will not benefit you one bit. You need to massage your bug list to stay off the slacker radar.

Start by picking off some bugs that are related somehow. They can be related on feature, topology specificity, platform specificity. Pick a few that you don’t understand well. Pick a few that you think are configuration error, user error, documentation error. Make up a list with these bugs and a summary of each one or print out each bug report. I hope you can guess what’s coming next. That’s right – go get your QE person or people and ask when they will have time to sit down with you to address your concerns. What you’re doing right here is proving to QE that they are valuable and their work is not in vain. You’re validating their presence in the organization by asking them to talk about their work. They need someone to listen to what they’re saying and it’s your job as a developer to do it. You’re making them feel good by making them feel important. This is a great way to start off a meeting or meeting request.

Let them know what bugs you want to discuss before the meeting starts to give them time to prepare too. This meeting is going to go very smoothly if everyone comes prepared. Your goal here is to find out enough information to resolve all of the bugs on your list you prepared. Every one of them must have some resolution before you leave this meeting. This is not a bug triage meeting because you know going into it that all the bugs must be resolved. This is a technical meeting that results in action items from all parties.

Start the meeting by saying “I understand you have some concerns about… <fill in the software module blank>. I was wondering if you could give me some insight into what you’re looking for from me”. Begin the discussion on each bug by asking them how they encountered it even if it’s already logged in the bug report. Ask questions about their process. A lot of extra information that wasn’t captured in Bugzilla will come out of this conversation.

Sometimes enough information will come out that you will all see that the bug is clearly due to incorrect expectations or incomplete documentation. These are easy bugs to resolve. Sometimes you’ll get that a bug is clearly a bug and through your conversation you will find out exactly what you need to do to fix it. These are also pretty easy to deal with. Another kind of bug that can end up here is one that turns out to be an enhancement request. I’ll talk in-depth about how to deal with enhancement requests another time, but if they come up in the meeting it’s best to just move on to the next bug. If you don’t you’ll end up at a mini-triage meeting trying to decide the priority, whether or not it’s an enhancement request and so on.

At the end of the meeting there should be a list of developer action items for you to take care of and a list of QE action items for them to take care of. Have these meetings as often as you can in order to drive down your list of bugs. Remember you’re trying to show up at the bug triage meeting with the shortest bug list possible and this will drive down your list very quickly.

I can derive another relevant example from the first iteration I wrote about in part two. Remember that QE had opened a huge number of bugs in a very short time period on my deliverable. I was overwhelmed by the incoming load and I couldn’t just mark all the bugs “won’t fix”. If QE wasn’t riled up already that would be the thing to do it. Most of the bugs came from two people in QE so I had to get them into a room with me, if not to talk to them then just to keep them away from their keyboards so they couldn’t open any more bugs for me.

I went in to the meeting with a print out of every bug I wanted to discuss. Each one was related to the feature that I had just delivered. I started the meeting by saying, “John, I understand you have some concerns about this feature. It seems like there are some things that are preventing you from testing further into the feature and I would like to know what I can do to help you get further along”. Notice how I didn’t say anything about “Please stop opening bugs or I’m going to quit my job” or “I hate you QE guys so much that I’ve written a thousand more bugs that you’ll never discover” or even something nicer like “There are an awful lot of bugs against this feature and I’m not sure some of them are valid”. Everything I said was phrased in a way that conveys my want to help QE. And I really did want to help them. Helping them helps me. There is no rule that says developers can’t help QE and if there is a rule that says that it’s wrong. Dev./QE is a two way street.

After that we went through each bug. We discovered that some of them were duplicates of each other. That gave QE the action item to close each duplicate bug. Some of the bugs were discovered to be due documentation that wasn’t explicit or detailed enough. Those bugs came into my action item list to fix the documentation. I asked the QE guys what documentation details would have helped them navigate through the steps safely so that I could use it in my resolution.

Some of the bugs were because the user tried to set up the software on an unsupported platform. You can’t use SuSE 7 and Perl 4 when the supported platform is Red Hat ES 4 and Perl 5. Those bugs become action items for QE; retest those bugs on supported platforms.

The most interesting bugs were real bugs caused by unexpected QE needs, but needs vital to their testing. Getting a firm grasp of what is causing these bugs is important and hopefully you can gain enough information that you can propose a fix in a relatively short period of time. Those bugs became action items for me to resolve.

In creating these action items it’s also important to provide a time table as to when QE can expect to see the resolution. Talking about the bug is one thing, but the goal is to drive down the bug list and the only way to do that is to actually fix the bugs. For my action items I told the QE guys that they could expect to see a fix in the next patch release they get from development and I made sure to deliver. Setting joint expectations and then driving toward them is a recurring pattern here and your consistency in doing so is important to your success.

Come back again tomorrow for the conclusion!

By eliminating the setup/config/curiosity class of bugs you should notice an immediate drop off in the amount of incoming bugs. It’s easy to file bugs on things that aren’t quite clear at first. By helping others get on board with what you’ve produced you’ve started setting expectations on where to test to get the most value out of the process. This isn’t to say that you as a software developer tell QE what to test and how to test it. That would defeat the purpose of having QE. They know how to test software and they’re good at it. Rather, during your ongoing dialogue with QE (it is ongoing, right?) there should be a discussion where you provide them with a list of the implementation topics that you’re most concerned about. You should address both business concerns and technical concerns.

The business concern is directly traceable to the business requirement. This is really just asking the question, “does this software fulfill this requirement?”. The answer should be fairly obvious but sometimes the requirement a software deliverable fulfills is not immediately apparent. Sometimes the software address multiple business requirements and there should be a discussion as to how they depend on each other and just as importantly how they do not depend on each other. Functionality of one requirement should not interfere with the functionality of a different requirement. It’s important that QE have some idea about the desired dependent and independent behavior in order to write tests to prove the software is correct.

The technical concerns you discuss arise from how you feel about the way you implemented the business requirement. Pointing out areas where the software is very flexible is always a good idea. This gives QE the hint that the software accepts a wide range of input and they should spend a little more time in validating what constitutes “correct input” and that the software only operates on correct input. If there is a point of extensibility in the software QE should take that as a directive to test that part as a high-risk to software security.

You shouldn’t limit yourself to talking about just flexibility or extensibility, these are just examples of the topics you should bring to QE. Remember, you’re taking the initiative to put a stop to brutal bug triage meetings and if the quality of your software improves as a result then just accept it as a positive side effect. Discussing these concerns with QE is a sure way to get some high-quality bugs out of them by allowing them to focus on areas of high-risk and core features. If you ever want to stop dealing with junk bugs then you have to do this. If there is ever a question on whether or not to talk to QE then always err on the side of over communicating. Too much is never enough.

Check back tomorrow for part four and the conclusion on Wednesday. Until then please forward this on to anyone that would find it interesting or helpful. And of course, please click the Digg This button below!

The reason we have bug triage sessions in the first place is no one has bothered to talk about these bugs before dealing with the entire list became a high priority. Everyone pays the high price of brutally long bug triage sessions because the low price of communicating on a consistent basis hasn’t been a priority right along. In order to make triage sessions more valuable you have to make it your top priority to get to the meeting with the fewest, highest quality bugs possible. This doesn’t mean close all the small bugs before the meeting and hope no one notices or to assign them all to someone else. It means effectively dealing with bugs while they’re incoming. In order to do this right you’ll need to build a relationship with someone in QE. Yes I know QE and developers are supposed to be like cats and dogs or water and oil. In this case the relationship needs to be more like peanut butter and jelly. Or peanut butter and chocolate. Or peanut butter and whatever else you like since it goes well with just about everything. You need to be like peanut butter.

Part of the beauty in doing this is that your chosen QE co-conspirator doesn’t even need to know they are a co-conspirator, though eventually they will realize their job has become much, much easier to do effectively. One of your goals is to stop bugs from coming in by stopping them at the source, which means you need to get QE to reduce the number of bugs filed on your software. There are a few ways to go about this and not all of them are good. Some can even get you in trouble with HR. If you want to reduce the number of incoming bugs you have to first find out what software QE is testing. If you’re late to the game then you should start just after the code is released to QE, though starting any time is better than not starting at all. In order to find out what’s going on in QE you have to, gasp, talk to someone in QE! It’s true, they know how to communicate using spoken language, not just Bugzilla. You can ask your QE person, “hey, how’s it going?” and they will tell you everything is going fine, testing is going fine, day to day tasks are going fine and then move on to filing more bug reports on your software. Or you could ask, “hey <QE person>, have you had a chance to look at <my software module in question>? If you have a chance I’d like to go over the work I’ve done with you in order to answer any questions you have on it.” The second way always gets better results than the first and it’s surprising how very few software engineers are capable of doing this. Asking the question in this way sets up a discussion by showing the QE person that you are open to questions and feedback. It gives them assurance that their input is valuable because it is! If they’re testing software and they don’t understand what it’s doing they’re not going to be responsive to the first question of “how’s it going” unless they have an ax to grind. You’re not going to get anything useful out of that question because it gives no indication that questions or constructive criticism are welcome even if they are. Do you really want to reduce your bug count? Spend a half hour going over your work with them. Answer every question they have and go over the implementation. Draw diagrams, play charades, do everything you can to help them understand what you’ve unleashed. All of those bugs that are really just setup and configuration questions will be vaporized if you do this. Ditto for the “what does this software actually do” bugs that would have been filed. You have just taken the first step to reducing the number of low quality bugs you have to deal with.

I remember an iteration where I had a fairly high profile deliverable. I felt like I nailed the business requirement. All the features were complete and robust. The implementation was impeccable. This deliverable was a feature that had been lacking for a long time and I had a feeling that I’d be heralded as a hero for delivering just what everyone was begging to have for such a long time. I didn’t think anyone would be able to open even a single bug on my project. How could they? They would be too busy drooling over the amazing quality of what I just delivered.

Imagine my surprise when QE started opening up bug after bug after bug on what most people in development considered perfection. I watched in panic all day long as I saw my bug list get longer and longer. And almost everyone one of the bugs was shallow. The QE person didn’t dig very far at all for a reason to file a bug. If they didn’t understand what was supposed to happen they filed a bug. If they thought the software was supposed to do something and it didn’t then there was a bug. If they thought the software should have additional features they considered nice to have there was a bug. Needless to say that iteration’s acceptance process was very stressful for me. That’s when I realized the value in stopping bugs at the source.

Almost all of the bugs were in the class that would have gone away if I just communicated what was going to happen before the release. Fast forward to the next iteration. Once again I was working on high profile features that should be a slam dunk as far as quality goes. The week before it was released to QE I asked around to find out who would be testing my features. I scheduled a meeting with them to talk about my deliverable and give them a demo. In my meeting invite I sent a brief description of what I wanted to talk about and encouraged them to show up with questions or relevant topics to discuss. During the meeting we went over just about every facet of every feature in great detail. We went over how to configure the operating system environment, the topology and the application. I wanted to do everything in my power to make sure they had all the information they needed to meaningfully test my software. Meaningless bugs are a waste of everyone’s time. Once the software was officially released to QE I didn’t see a single bug filed on the first day of testing. One was filed on the second day of testing. None on the third. I had only a handful of bugs for that iteration’s release because I made sure to discuss my concerns with QE before they started testing it. I also had more time to concentrate on the meaningful bugs.

On Monday we’ll talk about what to do with the bugs that are already in your queue. If you found this worthwhile please forward it on to anyone else you think would find it helpful.

Right off the bat you have to read and comprehend the defect report. Not all reports are clear and concise Sometimes the problem still isn’t obvious just from reading the report and you’ve got to wait on clarification. Worse, sometimes the bug report is perfectly clear and you can’t determine whether the issue in question is a feature or a defect. You can expect to spend a lot of time going back and forth with dev. management, QE, customer support and product management discussing whether or not the functionality is correct, especially if you work at a large company. So let’s say you know that some functionality is not correct or at least not desired. Now you’ve got to think about the steps to reproduce it and what the root cause could be. Still, you’re just thinking about it; you haven’t yet dug into the unit tests searching for a missed failure condition. Despite not knowing the root cause for certain there has to be a business decision about this bug: is it worth our time to fix it? Is there a business case that says that this functionality must be corrected before it is released into the wild? Sometimes yes, sometimes no. More than five minutes has gone by in either case and you’re still not done. Someone in management is going to want to know the delivery time table. Now repeat this process for 74 more bugs. If you’re triaging by yourself and you work at near-100% efficiency you’re looking forward to at least one full work day of triaging and the bug reports are still incoming.

Realistically you’re not working by yourself and you don’t set aside an entire day to triage your bug list. In real life the situation is a lot worse because a long time ago somebody in management invented the bug triage meeting. Bug triage meetings can last for hours or even days where usually too many people get into a room, close the door and stay caged in there until they triage some percentage of the product or service’s entire bug list. Instead of spending one person-day on bug triage, these meetings make it possible to burn entire person-weeks in the blink of an eye by turning lots of little technical problems into one large social problem. The technical problems don’t matter here because, easy or hard, there is a fix for (almost) every bug. The social problem is a lot bigger: getting people to agree on the priority and severity of each bug in the list.

Remember the process to triage each bug? Now everyone in the room has to go through that process for each and every bug that comes up in the meeting. Unless your company is the Borg Collective (no, not Microsoft. I actually mean the Borg from ST:TNG) there are going to be questions that require in-depth answers and those answers will foster discussion to get everyone up to speed on the history of the bug, why someone thinks it’s high priority, why someone else thinks it’s low priority, until everyone agrees and the meeting moves on to the next bug. The bug triage meeting also introduces the notion that you’re not going to be dealing with only your own bugs. Other developers are going to be in the meeting and their bugs have to be triaged too. This means that a significant portion of the meeting time is going to be spent NOT talking about your own bugs which is a recipe for boredom even if you have the best intentions.

It would seem that I don’t have a very high opinion of bug triage meetings but that’s actually not the case. Bug triage meetings are vitally important to a project at the right time, with the right people and the right material to work with. Unless you’re into fooling yourself no product or service ever ships with zero bugs. Sure you can get all kinds of management involved to do requirements-lawyering and invalidate bugs in bulk but then the zero bug metric loses its meaning. I don’t know about you but I’d rather have a list of known risks going into release instead of sweeping them under the rug because company policy dictates that everything must ship with zero bugs and is completely perfect, absolutely flawless. One of them seems just a bit more realistic than the other too.

Anyway, because there are always bugs we need the triage meeting to decide which ones to fix right now. These are usually the bugs which will cause the most damage in the wild either to the customer or the company’s reputation. The rest should be put in the parking lot for consideration in the next release cycle. If you’ve done your engineering cycle right then a vast majority of the bugs will be put in the parking lot. If development is complete and you’re working on driving your bug list down for the release then 75% of the bugs in the list should not be classified as “must be fixed”. Maybe development isn’t complete after all. So if you know that most of the meeting discussion is about bugs that don’t need to be fixed right away then why spend so much time talking about them? It’s because people lack initiative. Do you want to be a rock star team player adored by your peers and management? There are a few things you can do to make bug triage more valuable to everyone involved and give your career a big boost by addressing the social issues before they become social problems.

Tomorrow we’ll start taking a look at specific things you can do to reduce your bug count, help QE file real bugs and make your contributions more valuable and more visible.