Unit testing and integration testing: the X-ray scan for your software

Adarsh is a young chocolate maker whose chocolate products, especially  chocolate sauce, are extremely popular in town. His chocolate sauce is loved by retails shoppers and restaurant kitchens alike. Restaurants use his chocolate products as part of their own recipes such as sundaes. It’s a wonder, because Adarsh has been making chocolates for only two years. Food connoisseurs are impressed with Adarsh. They have been to his shop floor and they attribute his success to his meticulous attention to detail. One particular connoisseur, Bindia, writes for the ‘Foodies’ section of the town newspaper and is excited at the chance to interview Adarsh about his success. Here is how their interview goes.

Bindia: Adarsh, it’s great to get to talk to someone who is so talked about in this town. The first thing on everyone’s mind, readers and foodies alike, is — how have you been able to succeed as a chocolate maker? You had no experience at chocolate making before this. So what is your secret?

Adarsh: This may sound strange. Before being a chocolate maker, I was a software tester at a company named Star Software Services. That job taught me many things. Quality assurance has become a way of life for me, be it in chocolate making or even while I brush my teeth.

Bindia: That is indeed strange. What does being a software tester make possible for you?

Adarsh: Two things. First, I limit the number of ways in which my products can be used. I make those limitations publicly known. Let’s call this scope limitation. Second, when I advertise that something is good, it is damn good. I have tested it several times myself. This is quality assurance.

Look at the label on this chocolate sauce bottle, for instance. It says, “Suitable for use with vanilla, butterscotch, strawberry, black currant and mango”. I have limited the scope of the product. Why? Because given the limited time I have to produce and ship this sauce to the market, that is only testing I am able to fit in. But I have made sure that my sauce works really well on those five flavours.

So my previous job helps me promise less and limit the scope. Then it helps me truly deliver on the promises I make.

“In quality assurance, you promise less, then make sure that what you promise is delivered without flaws.”

Adarsh, chocolate maker

Bindia: How do you test?

Adarsh: If I were to speak like a software engineer, I do unit testing and integration testing on all my chocolate products.

Bindia: Ehh? Now what are those?

Adarsh: It’s very simple actually. Software professionals enjoy using jargon and watching the rest of the world look bewildered. I don’t know why!

Anyway, in unit testing, you take a single part of a system and test if that part works exactly as promised. In integrating testing, you test if those individual parts work together as promised.

If you were to take my chocolate sauce for instance, you need to ask yourself: what is chocolate sauce supposed to do? You devise a set of tests based on that question. Those form the unit tests. Next, you ask: before chocolate sauce ends up inside the tummy of a foodie, how does it interact with the world? It gets stored in pet bottles or cartons and then inside a fridge. It gets dumped on an ice cream sundae, and so on. Initially you make assumptions based on the current market for a product, but as your product gets used, you get to hear new conditions from its users. It is upto you to listen to those users and put their use cases as part of your unit and integration testing.

Unit testing: If someone has promised that he/she that will perform a task, are the results delivered exactly as promised?

Integration testing: If someone has promised that he/she will work with another person to perform a task, are the results delivered exactly as promised?

Bindia: What are the unit tests that you do?

Adarsh: Chocolate sauce is supposed to taste sweet. Every 200 gram batch of our sauce is tested for the same level of sweetness. But that’s too vague a description. So we have measurements in place. A batch of our chocolate sauce shouldn’t be sweeter than honey. So we use a spoon of honey as a benchmark. It shouldn’t be less sweet than a digestive biscuit. Anything in between will pass the test.

The colour is supposed to be dark brown. But that’s too vague a specification as well. So I have a colour sheet on the wall for every product. I compare the colour on the sheet with that of the actual product after it’s done.

The sauce is supposed to solidify if it is dripped on ice cream. So I test that the sauce freezes at -12 degrees C and thaws at 1 degree C. I use a refrigerator with temperature settings to test this. It is a special lab refrigerator, since a household refrigerator does not let you control temperature exactly. You MUST invest in the right testing tools if you want reliable test results.

The sauce is supposed to be of a certain thickness and density. 200 grams of our sauce is supposed to take up 100 ml in volume.

If even a single property fails the test, I reject the batch and set it aside for adjustments. In software sense, we’d say that the sauce has bugs!

You MUST invest in the right testing tools if you want reliable test results.

Adarsh, chocolate maker

Bindia: And what after you make adjustments? Do you perform just the test that failed or all of them?

Adarsh: One must ALWAYS repeat every test meant for an item. It is possible that an adjustment introduced for passing one test may interfere with another.

E.g. I see that a batch of sauce of 100 ml weighs 215 grams. That’s thicker than usual by 7.5%. Huge difference. Then I add a little bit of milk to the sauce to make it thinner and make sure that it passes the weight test. But what if it doesn’t freeze at -12 degrees C anymore? We’ll see unhappy customers with liquid sauce dripping all over their plates, while they expect it to freeze over on top of their ice cream.

Sometimes, a test failure is indicative of a larger problem and small adjustments will further break the system. Sometimes the solution may need to be from the ground up with a lot of rework. But you should catch the problem yourself by doing all the tests after every tiny adjustment or major rework. A customer shouldn’t have to see those problems. There should be no defect among the features you already promised.

Bindia: What about integration testing?

Adarsh: If there is anything in the world that our sauce touches, we test with it.

E.g. The sauce is stored in 100 ml pet bottles and shipped. So first things first, no bottle should leak. Before we ship, we inspect every bottle.

We guarantee that our sauce is usable for 2 weeks. From every batch of shipment, we retain one bottle at the shop and consume it for our own use. We gradually deplete that test bottle, making sure that some sauce left in the bottle at the end of two weeks. If we notice that the sauce is still good for use, then we consider that the sauce is not reacting to the pet bottle or to external elements. That integration is a success. Otherwise we may have to recall that batch and give free replacements.

How should sauce behave when scooped out with a steel spoon? Our promise is that when your pour it back, it will fall in a smooth syrupy stream. It shouldn’t fall in chunks, nor as a watery liquid. A 10th of the weight of the sauce you have scooped out should still be sticking to the spoon after you finish pouring.

Finally, we promise that our sauce works well with five flavours. We have samples of ice creams in those flavours. The sauce should freeze on contact with ice cream and the end result should taste good.

Bindia: This sounds to me like a display of a lot of patience.

Adarsh: It is. Display of patience is one of the points that a quality assurance professional should take care of.

Bindia: What are the others?

Adarsh: #1 Preparation. Quality assurance starts before the product is produced. Everyone has to agree upon standards and behaviour. Too often we see that production and quality assurance teams are out of sync, causing arguments. This is because the expectations of the two are so out of sync. Without a gold standard, the production team is working towards something that just works. Quality assurance is looking for perfection. It is the responsibility of the product / project manager to bring everyone to the table and set standards on what will be delivered by the production team and certified by the quality assurance team. Too often, quality assurance is an after-thought. Imagine if I brought someone off the streets and simply asked him/her to test our chocolate sauce. That person is going to judge based on his/her taste and not according to the standards we set. What if he/she likes chocolate sauce sweeter than honey or a bitter one. Naturally, he/she will consider the test a failure while the sauce chef is bewildered. This is what will happen if you thrust an app into the hands of a software tester without telling him/her the expectations. They will use their judgement and not the ones agreed upon by the production team or the project manager.

#2 The ability to communicate in fine details. It is not enough to say that the ‘sauce doesn’t taste good’. Be more specific. Say, “Here is batch #23, box number #8 of chocolate sauce. The sauce in this box tastes sweeter than honey. Here is the bottle of honey I used as reference.” Often we have testing teams coming up with complaints like “this isn’t working”. You have to be very, very specific about what isn’t working and under what conditions. It is your job as a tester to make the production team’s job easy to narrow down on the problem. Don’t use a vague sentence like, “Doesn’t work” and leave the production team searching for a needle in the haystack all day to figure out what doesn’t work.

#3 Honesty. Mainly about the product’s capabilities. You are explaining to either the product manager or to a client that your sauce tastes good with vanilla, butterscotch and strawberry. They may ask you if it tastes good with carrot too. You shouldn’t say, “Of course”, unless you have really tested your sauce with carrot flavour. Rather, your answer should be, “I haven’t tested that. This wasn’t agreed upon or thought out. So you won’t see it on the product label either. I cannot guarantee that it will taste good. Either you may try at your own discretion or we may open a new discussion on tweaking the product as per your wishes.” This takes a lot of integrity, even some courage. Dialling down the expectation of a client or a superior. Sometimes your own team may question your intelligence when you reveal that something is unverified and so you won’t vouch for it. But you need to stick to it. The last thing you need are the bad consequences of a false promise to a client. Testing is the only way to guarantee that expectations are met. Wishful thinking isn’t.

“Testing is the only way to guarantee that expectations are met. Wishful thinking isn’t.”

Adarsh, chocolate maker

Bindia: Adarsh, It was such a pleasure talking to you. Thanks for sharing the secret sauce for your chocolate sauce!

Adarsh: Aptly put! My pleasure.