(This article is borrowed from a section in Eric Mumford’s upcoming book, which will be published, at current progress, in 1,043 years.)
In my opinion, the most basic unit of measure of the quality of a piece of software is the defect. Without measuring defects, there can be no currency of exchange in determining the value of the software itself and the underlying process that built the product. And yet, teams and managers seem to struggle with exactly how a defect is defined, how it differs from a bug, and what to do with the information.
Defects might be the most misunderstood notion in software Quality Assurance. As a result of a somewhat bare cupboard of Quality Assurance education in a formal sense, certain terms are confused with one another or just outright unexplored. Doing a Google search on Defect vs Bug will turn up a variety of opinions but very few articles that precisely define and demonstrate the use of defects vs. bugs. (This article finally does.)
Rather than tell you the answer, I’m going to make you suffer through a learning process so you can play with why you currently think the way you do. More important than simply adopting my answer because it’s the best fit (which it may not be) is knowing why you use the words you use to describe software quality. Let’s begin by looking at human nature in terms of how we put 2 and 2 together to make 4.
Human nature is to fill in a pattern where none exists, so let us put into existence a consistent set of definitions for some words so that we can play with their meaning and build up a foundation of understanding. As is good practice when discussing any topic or idea that people may have preconceived notions or opinions on, it is best to start as far away from the idea as possible and sneak in on it. This is best done by gently discussing the star furthest from the center of the galaxy, rather than fathoming the source of life itself. So, we shall start by thinking about the human affect (this is not a misspelling.)
Some people have a flat affect – you can’t tell what they’re thinking or what emotional state they are in. These are your good poker players – games requiring the silencing of all facial expression. There are people who have a flat affect but actually feel quite happy inside. These are the people who you ask “Oh my God what’s wrong?” and they’re like “What are you talking about? I’m fine.”
Others have a rather lively affect – they wear their heart on their sleeve, the states of joy and sadness and confusion are obvious and often entertaining.
Some hypothesize that one’s affect is an effect of their mind’s connection with their body. Having an extensive collection of expressions in one’s repertoire of relaying one’s experience is believed by some to stem from a deeper sense of one’s self.
Infants and toddlers naturally play with their face – it is humorous to watch adults who have long neglected their ability to play interface with a baby within whom or a small pet within which they see their inner child inside. Adults who play with children this young are fun to watch; the huskiest construction worker becomes a teddy bear, the most ferocious corporate lawyer a sap.
Sometimes you go for the cause and fail, but your intention was consistent throughout the attempt.
People who have a value of helping those who cannot help themselves and have a value or natural draw toward shelters and civil engineering might choose to build homes for the poor, or may choose to volunteer their money to a cause that promotes these activities.
To cause something is to utilize time, effort, and resources to create a situation or thing where there was no situation or thing before your time and effort created it. A cause is like a goal — it is personal to you, and exists only in your own mind before you materialize it out into the world.
Our human understanding of science specifies an assumption and belief that all things that exist came about by means of one or more causal processes.
Sunlight hitting the earth is the result of a chemical process on a huge ball of burning gas around which our home planet revolves. We have hypothesized and tested theories on how this gas burns, what is burned up in the chemical process that creates the light, and limitations on the source of the light. Our science validates that we’re probably mostly right.
All science begins as magic, if magic is an effect that cannot be initially partnered with one or more causes. When humans sense magic, some revel in the magic, and some seek to explain the magic. Good software quality assurance people love magic because it is a sense of humor — and also a lie — to them. Good QA people seek to create a wide variety of effects from the software under test so that a full menu of causes can be found within the software that were laying undiscovered. All effects, our science believes, are a product of a cause. All causes are effects of the materials and forces that combined to create them. In systems theory, all causes are effects, and all effects are causes: the question is not do they exist, the question is do you see their relationships.
The verb defect as used in defecting from a team or group of people means to leave, perhaps to uphold a higher purpose or value that the team or group of people either no longer shares or never shared. Defection is an effect of a single entity breaking from a larger group. This sense of defection is related to the noun defect as we use it in software quality assurance. A defect is a single entity that has broken from the pack, or package of software, that stands out in that it is not behaving the way it is expected.
In software development, a defect is a measurable effect caused by a broken piece of code.
A defect is an effect, usually caused by human error, of writing correct code. Literally, code was typed in and compiled and the compilation succeeded – the code was syntactically correct, and may have even passed unit tests built into the code as flags along the way to indicate the understanding of a micro-requirement in the class or method under construction – but despite these safety harnesses, erroneous code was compiled into the build and sent to QA for testing. This human error in writing code resulted in a specified operation or set of operations calculating an eventual unexpected value. These operations caused undesired system response that strayed from specification which may have been caught by a regression test, functional test, automated test, or a performance test.
Regardless of how the error was discovered, the nature of a defect is that it is simply an erroneous piece of code. Regardless of whether the transgression in the code was purposeful or mistaken (Yes, there are software QA techniques that involve purposeful defect injection by developers as a statistical tool of measuring defect discovery rates), the definition of defect still holds as above.
If you look up “bug” in Webster’s Dictionary, you’ll find that Webster includes as a second option (the first option is our exo-skeletory friends) the computer software definition, “an unexpected defect, fault, flaw, or imperfection.” According to Webster, defect and bug are “sort of the same thing.”
As QA professionals, this fuzzy definition does not suffice for our needs, so let’s analyze this definition and then propose a much simpler working definition that we’ll leverage throughout this series of articles. This is an important topic to think through, because there are a tremendous amount of people who think that defects and bugs are the same thing. Bugs and defects are not the same thing. Let’s go through Webster’s definition and see if it clarifies the meaning of the word “bug” for us in a software development sense.
“Unexpected defect” – to think about an expected defect and an unexpected defect is a bit perplexing. Nobody really considers an expected defect, or “hidden feature,” something that they look forward to with each release of software. It’s true that some unanticipated functionality can be positive and beneficial to the business, but these easter eggs are rare. The use of the word defect in defining a bug introduces even more confusion and supports the widespread misunderstanding and confusion between the two terms in the first place. No, “unexpected defect” is entirely too confusing and in fact knocks the level of understanding down a few notches in terms of really getting into software quality assurance and trying to grasp what it’s all about. We’ll put this one to the side for now… since the phrase expected defect sounds more like poor planning, its converse of unexpected defect really sounds more like a circular definition than anything else.
“Fault” – to say that something is someone’s fault is to point blame. Blame is a shrugging off of one’s participation in a situation by declaring another wholly responsible. Blame is an insidious beast – a person can invoke blame without using any words at all. Have you ever been in a meeting where you ask why a project is late and someone almost imperceptibly shrugs and looks to their left? A well-timed knowing glance can throw daggers. Blaming is a minimizing – and ultimately an eradication – of one’s participation; it also tears away at the blame thrower’s self-esteem because it erodes one’s very sense of self. To blame is to erase yourself. Blaming is a lie: it is a lie because you wouldn’t need to blame if you didn’t know you were somehow participating in the problem to begin with. Blaming is vastly different than performing a root cause analysis in the spirit of fixing a problem. To find fault is to minimize one’s role and empowerment; this is the opposite of the goal we are building toward. We are building toward a state of empowerment, of noble effort and envisioning a software product to be greater than it is and to increase our role in its creation. No, blame won’t help us in this sense, so we’ll set aside “fault” for now.
“Flaw” – a flaw is a marker of a missing piece of a physical form. Flaws aside, a physical form or visionary piece would be complete and whole. A flaw is the negative energy physical measure of completeness. A software package or build is a collection of well-formed code consisting of instructions that together:
- perform carefully architected actions on data input to the code,
- that perform actions on the logical processes themselves,
- that perform actions on the data generated, and
- stores generated data as various forms of output, whether these forms be stored in memory, set aside in cache, stored to disk, displayed to screen, or sent across a network.
Thinking about a running software product as a system, it is difficult to project forward into a physical form what an integrous whole would like like, because data is constantly being input, processed, operated upon, recursively processed again, compared, stored, and output. A flaw can be easily imagined upon a static scene when there is a component of expectant predictability (fly in the ointment, wine on the white rug, a missing tooth in a child’s grin) but the concept of a flaw grows difficult to imagine and impossible to isolate within the complex dynamics of a software system. This description of “flaw,” I submit to you, is too vague for software development.
“Imperfection” – This one is easy. Every algorithm is perfect in that, once compiled, it will, all other things being equal, execute perfectly according to its design. Humans are imperfect in the sense that we are constantly changing – our organic chemistry, the physics of our biology – and this dynamicism prevents us from being exactly the same from one microsecond to the next. Perfection is quite literally physically and metaphorically impossible for human beings… this is why the Olympic games are so amazing. Our bodies are constantly taking something in, processing something, and passing stuff out, all the time, with no time taken for rest. Our cellular metabolism and our mental neurons, we believe, are firing at all times, even while we sleep or daydream. Our intellect and sense of self is deeper than we could ever imagine. Science even today struggles to measure and describe human potential for love, compassion, forgiveness, and tolerance. Our ideas and observations often cloud our memory of what we actually did versus what we intend to do. To say that a software program is imperfect is a falsehood – if it compiles, it is a perfect compilation. The instructions, when called with parameters that match the algorithm’s input requirements, will be executed in accordance with the rules around the class and method. No, there are more human factors at play here, and this definition seems to be clearly unfit for a computer software system. I think that a simple definition of bug will be necessary to clear the air and proceed forward with building our definitions and meaning around what it is to build software.
A bug is an emergent property of a software system that brings attention to a human limitation of understanding.
A bug is not a mistake in coding. A bug is the system doing something that isn’t incorrect per se… but it wasn’t purposefully designed in and you didn’t see it coming. The system is doing something odd or unplanned – it isn’t that the system is incorrect, but because it wasn’t expected, the end behavior wasn’t explicitly called out in the requirements to avoid or work around.
We phrase the definition of a bug in terms of human limitation because this avoids blame and rests on simple truths. The truth of a bug is that we didn’t see it coming and we didn’t plan for it. The truth of a bug is that there it is, it exists, and we didn’t realize our requirements would “mix together” in quite this way. We could picture the 99% of what we were asking for – the 1% of “reality” we couldn’t see until we built it, and here it is right in front of us in the form of a bug.
Part of our brain is used for computation, logic, and building– it is geared toward linear thought. Linear thought can connect what we want, with the path to get there, to the expectation we imagine we’ll get. This expectation seems to be by and large built on past experience, intelligent analysis of the present-day reality of the software architecture, the present-day reality of the business model, and knowledge of the development and product team. Given these data points, a linear thinker can sketch out a well-formed and well-thought out product vision and probably a decent project plan that pushes out a 3-6 month agenda and milestones. Linear thinking is very useful, but it has its limits – we hypothesize the human brain can’t handle more than 3 consecutive tasks at once at a deep focus level. Little bits of data and reality will be missed as plans are put together and requirements are written – this is not a “problem” with humanity or a “delta” in evaluating one’s skills, this is simply a limitation of the human ability to process data and make accurate decisions. Computer systems, on the other hand, always do what they are told, at least within the constraints of their underlying hardware systems having full power, a consistent operating environment, and full component function.
Measuring the rate of discovery of bugs in addition to [new defects] opened and closed and [regression defects] opened and closed will enable a view into how well we are doing in specifying what we want and also developing what we think we want. Code quality going up while bug discovery going down indicates a requirements management machine that is improving its communication to the development team. Code quality going down (defect discovery rate going up or flat, failed test cases maintaining a jagged sideways graph) while bug discovery going up tells just the opposite story: the development team is struggling to hammer together a technology solution and the business isn’t doing a very good job in writing accurate user stories or accurate requirements.
By differentiating between defects and bugs, the team can get a measure of both code quality and requirements quality. A good business systems team will demand that bugs and defects be tracked separately so that the team can get a read on the quality of the requirements as well as the quality of the code. The QA metrics of bug discovery rate and defect discovery rate tells a useful story in any software development lifecycle. Agile software development in particular can use this metric in concert with a burn-down or burn-up chart to indicate how ready a software product is for prime time.