In today’s article, we shall see why we should toss aside peripheral arguments, like the programming language to use and which cloud service to opt for, to build long-lasting solutions. Rather, we should bring a large whiteboard and attempt to solve problems in a time-tested way: diagrams and formulae. Why? So that we determine the most efficient solution by using the basic building blocks of programming: data structures and algorithms.
What seperates the best from the ordinary
Let’s play a small rapid-fire game.
- You need the recipe of a truffle cake. Which website do you head for?
- You need a taxi from the airport to your home. Which app do you reach for?
- Where do you upload the photos from your recent visit to Thailand?
Probably, you didn’t even think before answering. The answers were perhaps at the tip of your tongue before you even finished reading the question.
What makes solutions like Google, YouTube, Uber, Facebook and Instagram so good at their game? It is very easy to conclude that superior, cutting edge technology is THE answer to such seamless and lightning-fast solutions. However, technology is only a means to an end. The same technology that is available to the top 10 apps/websites in the world is available to everyone.
In the age of the Internet and open-source software, what seperates the best from the ordinary is definitely not the technology used, because technology is commoditised and easily downloadable.
Instead, these companies are good at drawing diagrams on a whiteboard, making paper models and simulating the entire solution on paper with a pencil. Long before they enter the world of apps, web services, SaaS, and finally the market. But what are they drawing on whiteboards and modelling on paper before writing programs? And how does that make a difference to their solution? They are figuring out something very fundamental to the world of computing.
Regardless of whether you use Android, iOS, Java, Python or NodeJS, computing devices understand only two building blocks in programming: data structures and algorithms.
Data structures and algorithms: What they are
Computing devices work on data. Engineers use mathematics to study the various forms in which data can be represented, e.g. sets, lists, matrices, etc. Different types of data need to be stored in different forms. There is an entire subject for the study on how to store data efficiently so that it can be accessed fast when required later. The form in which computing devices store data is called a data structure.
In order to store data and fetch it for later use, a computer must go through a set of instructions, one set for storing and another for fetching. Carefully planned instructions ensure that there are no bottlenecks while storing or retrieving data and that the computation is as quick as possible. The instructions are based on mathematical principles. Such a set of instructions is called an algorithm.
A well-planned combination of data structures and algorithms can mean the difference between a smooth experience for the user and a frustratingly slow one.
Planning a good combination requires engineers to abandon a computer for a week, or even several weeks. Instead, they need to draw diagrams and write formulae on a whiteboard before writing the first line of code.
Using data structures and algorithms: A real-life example
To understand how the choice of data structures and algorithms affects user experience, let us take a real-life example.
- Eddie has just moved into a new neighbourhood and is invited for a get-together at the local club.
- He is excited to meet the people from his neighbourhood and get to know them. He takes a sheet of paper and a pen with him. At the club, he meets the people from the neighbourhood and notes their names and phone numbers. He writes the names in the order in which he meets the persons. So the list is chronological.
This order works for him right now. Even if he meets three persons together and all of them rapidly dictate their names and numbers to him, he can keep up. Because he is simply adding new names to the end of the list. His hand can quickly reach beyond the end of the list as soon as he hears a new name and number. - The chronological list on the sheet of paper doesn’t make searching for names easy. So Eddie goes home and fishes out his phone book.
The phone book is structured such that it is easy to find names in a snap. The names are arranged in alphabetic order, with pages having markers for alphabets. Names that start with a letter go to the corresponding page on the phone book. To find Carlos’s phone number, Eddie simply goes to the page marked with C, where he can find all the contacts with names starting with C. Carlos will be right up there. - Eddie wants to wish his friends on their birthdays and anniversaries. Let’s say the day is 5th March. Combing through each contact’s record on the phone book to find out if this day is special isn’t a very attractive process.
But Eddie is smart. He has a good calendar, where he marks the name of the person and the occasion related to that person, against the date on the calendar. Eddie just looks up the date on the calendar and calls the person to make his / her day. - It is the football world cup final and Eddie wants to invite over all his football-loving friends to his home to watch the match on his big-screen smart TV at night. However, not all his friends are into sports and not everyone likes to stay up at night to watch TV. Going through his phone book is not the best option to gather a list of people who like sports and do not mind staying up late.
Luckily, Eddie has been fairly inquisitive while talking to his friends and has smartly made categorised lists on who likes football and who are night owls. He just needs to look at his categorised lists to find friends who would fit the occasion. He can make a list of the names under the categories and then use the phone book to find the phone number and then call each friend. - Eddie decides to throw a barbecue party at his home’s lawn. He decides to visit each house to invite his friends in person. He decides to ride his bicycle around the neighbourhood and distribute hand-drawn invitation cards to each family. But what route should he take to cover each family’s home and do it in the shortest route possible? Will his phone book or calendar help here? Of course not.
Perhaps a map of the neighbourhood will. And Eddie does have one. So he sets down to work, drawing a good route with a pencil over the map. Before long, the cards have been distributed and Eddie is set to have an awesome barbecue party.
Why data structures and algorithms are important
Eddie pays attention to how he uses his list of contacts for different purposes. By augmenting his contact list with a phone book, calendar, categorised lists and a map, he has designed his data such that he can find information almost immediately.
Had he missed out on any of the tools mentioned, it would have been a pain for him to get the right information quickly.
A phone book isn’t good to look up anniversaries, a calendar isn’t good to look up routes and a neighbourhood map isn’t a good solution to find football lovers.
The stationary Eddie uses are synonymous to data structures in software engineering. He also uses methods relevant to the data’s structure so that he can find the desired information easily, e.g. turn to letter C on the phone book, find the square labelled ‘5th March’ on the calendar, draw a route on the map threading all the houses on the best route, etc. These methods are synonymous to algorithms. Each algorithm is suitable for a corresponding data structure.
Eddie’s example shows how having different representations of the same small neighbourhood and its people can drastically improve or destroy the experience when using that information.
Imagine what it would be for Google or Uber not to use the right data structures and algorithms, especially with the humongous amount of data that they have.
Conclusion
Using the right data structures and algorithms seems to be common sense, but software companies get caught up in the lure of cutting edge technology. They fail to pay attention to the basic building blocks and the core foundations of computing, i.e. data structures and algorithms.
Most companies brag about having used cutting edge technology such as NoSQL, web sockets, machine learning or edge computing. But very few companies boast that their match-finding algorithm can service 10000 users at once, using just 4 GB of RAM.
Since the inception of computing, data structures and algorithms have been unsung heroes of the world’s best solutions. You’ll benefit from treating them as the most important piece in your computing solution.
Further reading
- Another one of my blog posts:Â 4 elementary must-have skills for a software engineer of which one skill is mastering data structures and algorithms.
- Blog: Introduction to data structures and algorithms by Study Tonight
- Book: Common–Sense Guide to Data Structures and Algorithms (Buy on Amazon.in | Amazon.com)
One thought on “Data structures and algorithms: The most important factor in the best software solutions”