SCRUM: The Story of an Agile Team
Scrum is one of the most heavily used agile techniques. It's not about coding; instead, it focuses on organization and project management. If you have a few moments, let me tell you about the team I work with, and how we adopted Scrum techniques.
A Little History
Scrum's roots actually extend beyond the Agile era.
Scrum's roots actually extend beyond the Agile era. The first mention of this technique can be found in 1986, by Hirotaka Takeuchi and Ikujiro Nonaka, for commercial product development. The first official paper defining Scrum, written by Jeff Sutherland and Ken Schwaber, was presented in 1995.
A Few Facts
Scrum defines a set of recommendations, which teams are encouraged to follow. It also defines several actors - or roles, if you prefer that terminology - together with an iterative process of production and periodical planning. There are several tools, which accommodate the Scrum process. I will reference a few in this article, but the most powerful tools are the white board and sticky notes.
There is not, and never will be, a list of "Scrum Best Practices," because team and project context trumps all other considerations. -- Mike Cohn
Everything starts with the pig and the chicken. The chicken asks the pig if he is interested in jointly opening a restaurant. The chicken says they could call it, "Ham-and-Eggs." The pig answers, "No thanks. I'd be committed, but you'd only be involved!"
That's Scrum! It specifies a concrete set of roles, which are divided into two groups:
- Committed - those directly responsible for production and delivery of the final product. These roles include the team as a whole, its members, the scrum master, and the product owner.
- Involved - represents the other people interested in the project, but who aren't taking an active or direct part in the production and delivery processes. These roles are typically stakeholders and managers.
This is How We Started
Everything depends on dedication and good will. If you want your team to be efficient, productive, and deliver on time, you need someone to embrace some form of Agile techniques. Scrum may or may not be ideal for you, but it is surely one of the best places to start. Find that someone on your team who is willing to help the others, or you, yourself, can take on the responsibility of introducing Scrum.
You may ask why you should care how another team, like mine, does Scrum. You should care because we all learn how to do Scrum better by hearing stories of how it has been done by others - especially those who are doing it well. - Mike Cohn
The talented team I work with already knew a lot about Agile. We switched from Waterfall development to a more agile process, and released quite frequently. We successfully managed to release every three to six months, having a decently low number of bugs after each release.
But, still, we were far from what we can achieve today. We missed the process, or rules, that would force us to change our perspective on the product and process. That was the moment when our team manager introduced us to Scrum, a term we, at that time, had never heard of.
This person took the role of the Scrum Master.
The Scrum Master
The Scrum Master is easily one of the most important roles. This person is responsible for creating a bridge between the Product Owner (defined below) and the Team (also defined later). This person usually possesses a strong technical knowledge, and actively participates in the development process. He or she also communicates with the Product Owner about the User Stories, and how to organize the Backlog.
The Scrum Master coordinates development processes, but he does not micro-manage (the team is self-organized). At the beginning of the process, however, the Scrum Master might micro-manage part of the team, in order to improve their team interaction and self-organization techniques.
Scrum Masters have more responsibilities, and I'll cover them as we discuss this process.
Introducing the Term "Sprint"
Personally, we don't have any problem with three to six month releases, but I originally couldn't imagine such a frequent release schedule. I thought it was too fast, and didn't provide us with the necessary time to integrate and debug our products. But then, our Scrum Master introduced us to the term, sprint.
Sprint: a basic unit of development in Scrum. It can take between one week and one month, and the product is in a stable state after each sprint.
That sounds outrageous! Stable every week> Impossible! But, in actuality, it's quite possible. First, we reduced our production cycles from three months to one-and-a-half months, and, finally, to a single month. All of this was accomplished without changing our style. However, you'll need to introduce some rules for a sprint less than thirty days. There's no magic rule-set here; the rules must benefit your project.
If I recall correctly, the first significant change in our development process came by the introduction of sprint planning.
This is one of the several meetings that Scrum recommends. Before each new sprint, the team, product owner, and scrum master plan the next sprint. This meeting can take a whole day, but shorter sprints likely only need a couple hours or so.
Our process is mostly reviewing the product backlog, and deciding upon a subset of user stories that will be included in the next sprint. These decisions are made by direct negotiations between the team, represented by the scrum master, and the product owner.
The Product Owner
Setting the direction of a product by guessing which small feature will provide the most value may be the most difficult task.
This person is responsible for defining the User Stories and maintaining the Product Backlog. He or she is also a bridge between the team and higher management. The product owner evaluates the requests from stakeholders, higher management, users, and other feedback (like bug reports, user surveys, etc).
He or she prioritizes this backlog, providing the maximum value to the stakeholders in the shortest possible time. The product owner achieves this by planning the most valuable user stories that can be completed in a timely manner. This may sound sophisticated - it is! In fact, setting the direction of a product by guessing which small feature will provide the most value may be the most difficult task of the whole process. On the other hand, sometimes it's rather easy. You may have a thousand users asking for a specific feature. In these cases, the correct choice is obvious.
If those users represent a large portion of you user base, providing that specific feature increases loyalty.
But again, this is a difficult choice. What if you could increase your user base by 100% by implementing a different feature? So, you can either increase your current users' loyalty, or increase the user base. What is the correct choice? It all depends on the current direction of the business, and the product owner must decide where to take the product.
In the company I work for, these decisions propagate to the team. It's not a requirement of the Scrum process, but it is especially useful with new teams. Sharing information goes a long way in helping everyone understand why some decisions are made, and why seemingly obvious features may be delayed or dropped.
The Planning Board
I remember the morning it happened: I arrived at the office, only to find our scrum master preparing a makeshift planning board with A4 paper and transparent tape. I had no idea what he was doing. As with every morning, I prepared a pot of coffee, and waited to see.
When finished, a white board was placed on the wall. It had several columns, and morphed into a rectangular shape. Several colored sticky notes peppered the "board." That was two years ago.
The board now accommodates the Lean Development Process that we use today. Remember, Agile is all about change and adapting to change. Never blindly follow the rules.
So, what's on that board?
Columns for the Development Process
There are four main columns:
- Release Backlog - Where all the stories reside for the current release. Yes, the product is ready for release after each sprint, but that doesn't necessarily mean that we actually release it. We typically have five-day sprints.
- Sprint Backlog - When we plan, we negotiate what the product owner wants to complete in the sprint. How do we decide what we can and cannot complete? By estimating the difficulty of each story (see below - Estimating stories). The end result is a set of stories moved from the release backlog to the sprint backlog. The team concentrates on finishing those stories in the upcoming week.
- Working On - This one is simple. When team members take a story, they add it to this column, to show everyone what they are working on. This is meant for employee control, but rather for communicating with team members. Everyone should always know what their teammates are working on. In the above image, the small bookmarks stuck on the sticky notes contain my team members' names.
- Done - Complete all the things! Yes, this is where completed stories go. However, it's important to define "what is done." At the end of an ideal sprint, all stories and bugs from the sprint backlog should be contained within this column.
Tip: Many teams like to split the Working On column into several others to better define different stages of work. We split ours into Design, Development and Testing. Feel free to invent your own steps, according to your process.
The Definition of Done
What is done? When can you confidently state that a story is complete? How do you know?
"Done" must be clearly and precisely defined, so that everyone knows when a story is complete. The definition of "done" can differ from team to team, and even from project to project. There is no exact rule. I recommend that you raise this issue at a team meeting, and decide what determines a complete story. Here are some ideas that you might want to consider:
- Create a minimalistic design.
- Create a GUI mockup.
- Use TDD or ensure that there are unit tests.
- Write the code.
- Let another team member manually test your story.
- The whole system can be built and compiled with the new code.
- Functional or acceptance tests pass as expected, after the new code is integrated into the system.
There are multiple ideas that can be included in the definition of done. Take what you consider to be necessary for your team and use it. Also, don't forget to adapt this definition over time. If you notice that your definition is becoming outdated, you may consider removing some elements or adding necessary, but frequently forgotten, ideas.
In the picture above, the green sticky notes describe what we considered to be done, for each part of the process.
Populating the Board
That was the question we asked ourselves. Until this point, we did not use sticky notes for planning. We used software to keep track of user stories and bugs, but, other than that, we used nothing. After lunch, our scrum master presented us with a mountain of colored sticky notes. After preparing a dozen notes, he explained them to us.
The User Stories
A user story is a short, one sentence definition of a feature or functionality.
These represent the main features that we want to implement. A user story is a short, one sentence definition of a feature or functionality. It is referred to as a user story, because it is presented from the perspective of a user. Naturally, the term user is the person using our application. This person can be in one or more different categories: a system administrator, restricted user, manager, etc.
An example of such a story might sound something like, "As a user, I want to share a folder with my teammates."
At that point, we did not have a product owner defined, so our scrum master invented these stories. This is acceptable at the beginning, but I highly recommended that you separate these two roles. Otherwise, how can the scrum master negotiate the sprint backlog with the product owner?
You may think to yourself, "Why negotiate? Isn't it actually better for a single person to decide what to do and when?" Not quite. The two roles would be influenced by a single person's views of the system or project. Two people, on the other hand, have a better chance of providing a more objective path for the team, ensuring the end goal (better valuable software) is achieved.
The product owner should define user stories; the team should negotiate their execution, and the scrum master represents them.
User stories define everything new that needs to be done; they're represented by yellow sticky notes on our board.
Bugs, Bugs, Bugs
Tracking your bugs is incredibly important.
We also list bugs on the board. Do you see those red sticky notes? Those are the bugs that we need to fix for the next release.
Different teams treat bugs in different ways. Our team mixes the bugs with the stories, but we always begin a sprint by fixing the bugs.
I know of other teams who pile up bugs for a period of three sprints, and spend every fourth sprint fixing them. Others split sprints into 25/75 for bugs/stories. Further, other teams may work on stories in the morning, and bugs after lunch; it simply depends on the company.
It's up to you to find the best solution for your team, and of course, keep track of your bugs. Write them on sticky notes so that you can track your system's issues and the fixes for those issues. Tracking your bugs is incredibly important.
Tasks or Sub-Stories
Tasks are written as simple sentences from the developer's point of view.
Ideally, each story should be short enough to be completed with relative ease, but splitting stories into other stories can prove difficult. Some projects simply don't afford this possibility. Still, you'll find large stories that several team members can work on. It's important to divide huge chunks of work into smaller, easier to manage pieces.
One approach splits big stories into Tasks. Tasks are written as simple sentences from the developer's point of view. For example, the previous folder sharing story might be divided into several tasks, such as: "Create UI for sharing", "Implement public sharing mechanism", "Implement access control functionality", "Add Access Control checkboxes to the UI", and so on. The point is that you have to think more about the story every time you break it into smaller tasks. Your team will have much greater control over a story, when you analyze each piece.
We use light-blue sticky notes for tasks on our board. Look in the last column (the "Done" column), and you'll find our tasks under the user story stickies. That particular story was broken into around four tasks.
Certain activities must be completed in order to finish a task, story, or the sprint as a whole. These tasks are usually infrastructure related, but you may find that the tasks require changes to the system. This process may or may not be part of a story. For example, you may find that a third party application must be installed, in order to implement your application's sharing capability. Is this part of our user story? Possibly, but maybe not.
We determined that these tasks should be separated from the actual story. This helped us to better track these extra tasks. On our board, you can find these tasks on green sticky notes. There is one on the sprint backlog, and about three in testing.
The Technical Backlog
For a young team with little experience with Agile and Scrum, it's helpful to highlight these tasks with a mini-backlog.
This backlog is for infrastructural tasks, such as updating the automated testing system, configuring a new server, and other things, which make our everyday development work easier. These are things that must be completed at some point, but are not directly related to development.
You don't have to put these items into a separate backlog. In fact, I know of teams who don't separate them. Our team dropped our technical backlog a few months ago; we decided that infrastructural tasks are as important as other tasks. However, for a young team with little experience in Agile and Scrum, it's helpful to highlight these tasks with a mini-backlog. So, I recommend that you should give it a try. It may work for you, and, if not, then just put Infrastructural Tasks on you planning board, possibly with a different color.
The Big Challenge: Estimation
During a planning meeting, we decide which user stories and bugs from the product backlog (or release backlog in our case) to include in the next sprint. This may sound simple, but, in reality, it's quite complicated.
The product owner comes forward with a set of stories to work on. This list typically contains more work than what can be accomplished in the sprint. The scrum master, together with the team, has to convince the product owner of what can be done during a sprint. Over time, this process becomes easier, as the product owner learns the approximate velocity of the team. Then again, the team may become more productive with each sprint, thus allowing more stories to be finished. The trick is to have a team who really wants to exceed expectations!
Now, the product owner wants to complete more stories than we can do in a sprint. We need to estimate the amount of work we can do in relation to the stories submitted by the product owner, and we can do this in a variety of ways.
Story Points are one of the most common methods for estimating stories, bugs or tasks. They are not necessarily the best approach, but they still are a good way to start.
So what is a story point? At the beginning of the process, the team looks for the simplest story they can find on the board. It doesn't matter how difficult it is, or how long it takes to complete. When they find that story, they set it as the reference story of having one point. In some projects, such a story can be as simple as fixing UI elements in ten minutes; whereas, the simplest story for more complex systems may take two hours for three people to complete. Now that you have a baseline, evaluate the other stories and bugs and assign them points.
This can be more difficult than it seems, and there are several point techniques to better help estimate stories. Here are two of them:
- Use Fobinacci Numbers - 1,2,3,5,8 (and maybe 13, but a 13-sized story smells too big to me).
- Use Powers of 2 - 1,2,4,8 (and maybe 16, but this number should be avoided).
You are free to choose whatever you feel most comfortable with. Be agile! Maybe you want to use two points increments, because your tasks are best estimated that way. Bravo to you!
Numbers are great, and many technical people love them. You may find, however, that, at some point, programmers begin to associate story points with time. They will think, "It takes me two days to do this. Let's give it five points". This is wrong. Estimates go from bad to worse when this happens. Story points should never relate to time.
Using semaphore colors may help alleviate this problem. Instead of assigning numbers to stories, the team can associate those stories with colors. Our team made this change a few months ago, and it greatly helped change our point of view.
Naturally, each color has a different meaning.
- Green signifies an easy task that can be completed in the next sprint.
- Yellow refers to a more difficult task - one that requires more effort from several team members. It also has a high chance of completion in the next sprint.
- Red labels are assigned to stories, which are extremely difficult and may not be finished in a single sprint. There should be little to no such stories, but if you adopt one week sprints, five days is a short time.
Numbers may be ugly to you, colors too artistic. It's time for t-shirt sizes! This technique suggests giving up on comparing story complexity with time of completion. Simply put, instead of numbers, you use sizes like S, M, L, XL, XXL for your stories.
I personally never felt attracted to this kind of estimation, but, hey, some feel that it's the best way to go. Try it out, if you feel comfortable with the idea.
The product owner, stake holders, and managers have to know what to expect from the end of a sprint. They must decide if the stories that were worked on should be released, and they have to know when features are ready. It's not a good solution to release every new feature at the end of a product's development cycle; releasing the most valuable features on a more frequent basis is a considerably better way to go. To achieve this, they must know what will be available in the short term, and their information should stem from the team. Therefore, the team should estimate, as best as possible, the work they can do in a sprint.
Measuring Speed of Development
So you want to see how well you perform in the current sprint? A frequently-used method is the burndown chart:
In this chart, we have a five day sprint, and we assumed that we could complete ten points in the sprint. Each value point represents the remaining story points at the end of each day. The green line reveals the ideal path: a steady two points per day. The red line shows our actual performance, or the true speed of development.
It's not on the planning board picture, but my team used to have an A4 paper sheet positioned above the planning board, with the burndown chart for each sprint. At the conclusion of each day, one of the team members was responsible for calculating the points completed for that day. It's simple: if programmers move the stories from column to column as they work, finding the remaining unfinished stories is easy.
There are no half-done stories. If a story is done, it is done. If it is not complete, then it is not counted on the burndown chart.
Of course, you will fail - BIG TIME - at estimating! This is especially true at the beginning. There is no way to avoid this, unfortunately. There is no way to know how many points you can to deliver. Your first chart may very well look like:
Our first chart surely looked similar to that. I think we did not even complete half of what we committed to. Why? Well, because estimation is hard. No matter what you do, or how good are you, when someone asks you how complicated something you have never done before is, you will have a hard time providing an accurate estimation. Don't panic! Try your best. With time, these things become easier. You may become able to estimate with a 70% accuracy at some point for short sprints. If the sprints are longer, your accuracy will likely be less, because there will be more stories to estimate and more variables that can go wrong.
When this happens, you adjust. For the next sprint, take four points. Like this:
This is bad again. You were too conservative, and finished early. It's a natural reaction for the team to adjust, based on the failure of the previous estimation. Still, this is a failure again, but on the other side of the road.
The problem? What do you do after you've finished what you were planning for? Another story? How do you put that on the chart? You can't redraw the original line.
When working with burndown charts, it's recommended to always make an average of the last 3-5 charts, in order to specify the points for the upcoming sprint. Of course, at the beginning, you don't have such information available, so you won't be as accurate as in the future. That's okay.
After some time, your charts will begin to resemble the first example more and more. You will, most of the time, finish all the stories and have a sustained velocity.
This term refers to your speed of development. It relates to how much you can do in a sprint. One of the most important concepts in Agile is to have a consistent velocity. Make the team deliver at a constant pace. With traditional project management, velocity decreases as a project ages. Complexity and rigidity forces the speed of development down.
Agile methodologies and techniques target to maintain a constant pace. Deliver fast now, and deliver faster later. How do they do it? Scrum is one of the elements of the puzzle. Other important pieces include the techniques that programmers can use to make their code and development process better. For example, XP (eXtreme Programming), Pair Programming, TDD (Test Driven Development), etc. All these, together, can make a team really great.
So we measure this velocity, but what do we actually do with it?
Tip: Measuring velocity is for making better predictions; not for judging a team or its members.
Use velocity to know what your team can do. Use velocity to know what to expect. Use velocity to make your team want more and be better. Never use velocity to judge your team or evaluate the productivity of your programmers.
Always Look Back and Improve
Following the first few sprints, our scrum master gathered the whole team. He began asking us about the good and bad things from the past week. This might be uncomfortable at the beginning, but it's still incredibly important. Describing what you felt went wrong in the past week will create awareness. And, of course, it is also helpful to highlight what went well!
These meetings are typically referred to as Retrospectives. They offer the scope to highlight what was good and what went wrong. Here are some examples from my own retrospectives.
- Team members were fighting too much
- Team member X or Y was not collaborative when pair programming
- We lost too much time with small things, like X or Y
- We did not pair program all the time
- We did not write unit tests for module M
When discussing problems, try to put aside your personal feelings; speak about what's bothering you. This is the only way for the team to resolve the issue. The most important thing is to immediately propose a solution for each problem. Don't simply let the list be written and forgotten; stay for a few minutes, and have the whole team think about what can be done to avoid them next week.
- We finished on time
- We were able to talk without fighting
- Some of us became more receptive to suggestions and ideas
- We wrote all the code with TDD
Again, highlight as many good things as possible - especially at the beginning or with junior programmers. For example, having all the code written with TDD may be a big achievement for a junior team. Make them feel really good about it, make them want to do it more and more. The same is true for a senior team; they simply have other things to highlight (TDD is done by reflex).
I Want to See What You Did This Sprint
The demo is for showing stake holders (and the product owner) the progress of the project.
This heading comes from the words of my scrum master. At that point, he also was the product owner. Before the end of a sprint, he'd ask us to present him with what we accomplished. We prepared a Demo, or a working example in a controlled environment.
Scrum proposes these demos at the end of each sprint. These should be done before the retrospective meeting that we discussed above. The team should prepare a special environment, and ensure that the product is capable of showcasing the features done in this sprint. The demo is for showing stake holders (and the product owner) the progress of the project.
You may ask yourself why I mentioned a controlled environment, when our product should be production-ready at the end of each sprint. Yes, the product should be as close to production-ready as possible, but that does not mean that the feature, itself, is ready. Often, there will be features which are too big to fit within a single sprint. The product will remain stable, but the feature will not quite be ready. When stakeholder see the demo, they want to review the feature and what it can do. In many cases, to showcase some functionalities for unfinished features, special environments must first be prepared.
Additionally, based on these demos, the product owner may determine that a bigger feature is good enough, and a new version of the product should be published and sent to the users. Most of the time, even if a feature is not quite complete, a release will help the project gain valuable user feedback, and concentrate the completion of the feature in a way that will satisfy as many users as possible.
Well, this sounds quite simple. You're an agile team, you keep your tests always on green, and your product is in a stable state. How difficult can it be to prepare a quick demo? It's more difficult than you might think!
Our team needed, if I remember correctly, more than five attempts before we managed to correctly prepare the demo. Luckily for us, the stake holders were not included in these first failed demos!
Still, We Need More Guidance
In these meetings, each team member must answer three questions.
It was the moment when our scrum master proposed to make a meeting each day. Yes! Every day, every morning, at an exact hour!
I find this to be a very good thing for new teams - for people who are not yet comfortable with one another, or with the project. These daily meetings, called Daily Scrum, are kept with the team, at a specified time every day. This should be kept before any work is done for the respective day. In my team, we set the time to 10AM each morning, which was difficult to do. Nonetheless, it was the correct decision.
The daily scrum is a short and simple meeting (not more than fifteen minutes). The scope of it is to help team members see who is doing what, and determine where the problems and bottlenecks in the development project are.
In these meetings, each team member must answer three questions:
- What did you do yesterday? - A short answer is expected - max 2-3 sentences.
- What are you planning to do today? - Same type of short answer, things like "I will work on this story today."
- Are there any problems with your process? If yes, what? Can they be quickly solved? This should be an answer highlighting the problems and solutions, if known. No detailed discussions should be taken, while this meeting goes on. The scrum master should take note of the problem, and work toward solving it together with the team, after the meeting is adjourned.
Solving the problems and impediments in the way of developers should be high priority for the team, so that they can continue with their development as soon as possible. Often, the person who had the problem is capable of solving it in a timely manner. Other times, he or she requires the help of a teammate. And other times, the problem can be so serious that the team will have to stop development and concentrate exclusively on solving the one thing that prevents them from continuing their work.
I remember my team encountering these huge road-blocks on several different occasions. There were tasks and stories, which seemed to be quite obvious at first site, but, after a pair or a single programmer had the chance to dig into the problem, the obvious became confusing and wrong. We discovered several times that a third party library could not provide us with the necessary functionality, and ended up concentrating all of our efforts into finding another, more capable library - or even implementing a solution ourselves.
The majority of our project is written in PHP. At some point, we had to interface our project with VMWare. We reviewed the official libraries for VMWare API, and found out there are Java and Perl versions. There's also an unofficial Ruby option. We were sure that we could use one of them, and simply do some
exec() calls from PHP to capture the output as a string. As we thought, parsing from there should be piece of cake.
It turned out that this was next to impossible. Neither API library worked quite as we expected it to. Some were abandoned or incomplete, and they had nearly impossible to parse outputs. Ultimately, we were forced to do something that nobody had ever done before: implement a VMWare API library in PHP. Unfortunately, there was no other reasonably acceptable way to do it.
This problem was massive; it set back the initial plans by weeks! Of course, our product owner was immediately notified, and, together with him, we planned a new schedule, and developed new stories, which included the creation of this API library.
More often than not, your problems will be much smaller. People might get stuck with some more sophisticated logic, but, many times, by the following morning, they already have ideas and solutions. Other times, they will simply be going on the wrong road, and a teammate will need to help get them back on track. These represent your typical issues.
Here we are at the conclusion. At least, this is how my team got started with Scrum. Some rules were very useful; others less so. Further, some rules were only useful for a short time, while others are still respected religiously by our team.
I can only recommend that you embrace the agile process, try out Scrum, and form your own conclusions. I'm sure that you'll find bits and pieces to adopt for your team. Be agile, adapt it for your style of work, for you projects and your personalities, and don't be afraid of adding your own custom rules.
After all, Agile refers to adaptation, not blindly following a set of pre-determined rules!