Primer: What is a Software Development Process, and why do I need one?

I’m sitting here in the sun at McMenamin’s Imbrie Hall after work, enjoying a nice 85 degree sunny day, and sipping a pint of Terminator Stout. If you’ve worked in the tech industry in the Portland area, you know what I’m talking about.

I’ve been pondering how to describe the various flavors of software development processes, in a short and understandable manner. We used to be an agile shop (Scrum), and lately we’ve been required to follow an iterative process (RUP). This has really brought many of the differences into stark contrast. We were a small shop, we delivered applications about every half-year, we all were located in the same office and everyone knew everybody else. Now we’re part of a large company, where you might not even meet some of the members of a project, and some developers are on the other coast or in India. Our success on delivering projects has dropped, and the project timelines have doubled or tripled. However, the requirements are also more extensive than they used to be.

The software development process seems to be the eternal question in our industry. How do we efficiently manage defining the objectives of a project, defining the specific requirements, managing the efficient creation of the software, and managing the deployment of the software to produce the greatest benefit?

Although I now work for a large company, I’m addressing this post, as always, to the reader from a small software development organization, particularly start-ups and high-growth companies. I note that it’s always possible for small teams within a large company to act like a small company, but it’s really hard to pull off.

I also presume that, as part of the Primer, you’re looking for basic understanding of software development practices. Some of the statements here might be considered  by the expert practitioner as over-simplified, which I freely acknowledge.

As such, I’ll try to keep this post at a high level, and point you to resources which can give you more depth.

There are three general types of processes that have evolved over the last three decades – planned, iterative, and adaptive.

For the purposes of this discussion, let’s assume you’re creating a customer support application. You know that you’re going to need the following capabilities:

1. Support Rep can sign on
2. Support Rep can look up an order
3. Support Rep can look up a user profile
4. Support Rep can update an order (change payment method, shipping address, cancel)

Planned Process

A planned process is based on traditional project management techniques, which assumes the work is completely known before the project starts, the staff and resources are available, everything can be reliably estimated, and a schedule can be created.

Following a planned process, we would proceed as follows in our example:

1. Assign the requirements and development team.
2. Write a requirement document for all features.
3. Developers create the application, conforming exactly to the requirements.
4. QA tests the application for conformance to the requirements. Re-work is done as necessary.
5. The business owner reviews the delivered application, identifies the differences between what’s needed and the application, and creates a list for re-work. The re-work is either done, or the project is canceled.
6. The application is rolled into production. User feedback creates another re-work list. Defects are identified and patches are released.
7. Plan the next release for next year.

In the software world, this is commonly denigrated as a “waterfall” process, but that is not entirely accurate. By waterfall they mean that each step must fully complete before you can move onto the next step. This is an extreme and unsupported view.

A planned process is very useful in many corporate and IT projects, such as building a data center, introducing a new product to the market, and installing and configuring a new application such as a farm of web servers. This work is known, can be broken down to steps, and can be reliably tracked.

Generally these processes were called Software Development Lifecycles (SDLC), and followed traditional Project Management Institute (PMI) standards. There were various tools to evaluate your success, such as Six-Sigma, Capability Maturity Model (CMM), and ISO-9001.

However, for creative work, such as developing a new product, these processes have been shown to create failure and increase costs more often than they foster success.

Iterative Process

An iterative process involves working your way through the project in moderate size steps. A key feature of an iterative process is that it provides regular opportunities for adjustment of project goals and requirements, as you learn about the problem you’re solving. It also provides opportunities for regular review of progress and project goals by the management team.

Some of the approaches are Rapid Prototyping, which involved creating a working system early to evaluate requirements and design decisions, the Spiral Model, and the Unified Process, which is a planned but iterative process for discovery of the requirements and construction of the application. Unified Process also has an IBM version called the Rational Unified Process.

Generally, an iterative approach goes through several phases:

1. Problem Definition
2. Requirements Capture
3. Investigation
4. Construction
5. Deployment

You proceed by completing step 1, and then partially executing steps 2-4, perhaps on a monthly or quarterly basis, until the system is complete. At no time do you attempt to capture all of the requirements upfront and at a single point in time, or to construct the entire system in one go. You build the system one subsystem at a time. Finally, when the product is complete, you move to step 5 to deploy the product. You then return to step 1 for the next release cycle of the product.

Following again our Customer Support application example, and using an iterative process, you might proceed as follows:

1. Understand the requirements for feature 2 first
2. Fully implement the “look up an order” function. The remaining functionality (sign on, user profile, manage order) can be mocked up.
3. Test the developed features.
4. Show the application to the business owner, who would then identify differences between what is needed, and what was built, creating a small re-work list.
5. Defects are identified and fixes are incorporated in the next iteration.
6. Now understand the requirements for feature 3,and implement feature 3.
7. Similarly for feature 4
8. Similarly for feature 1
9. Ship the product. Gather user feedback and defects, and plan the next release for next year.

Note that there is no specific order in which the features need to be completed – but you should roughly complete a feature before you move on to the next.

Iterative processes can be very effective if managed closely. They typically require a substantial amount of training, and work best on projects where you have a pretty good idea of what you’re building.

If you are working on a project where a defect in the system design or construction could cause death, injury, catastrophic business interruption, or extreme embarrassment, you should consider using an iterative development process. It provides a pretty good balance between gathering complete requirements, efficient construction and testing at several stages of construction. However, these projects are less common than you might believe.

If you are managing a project that can be fully described, similar to building a house, you should probably choose a planned process instead.

Iterative processes can degrade into a planned process if managed by persons unfamiliar with software development, and who do not understand creative work. The projects can become subject to the dreaded “analysis paralysis”, where months are spent “capturing the requirements”, but which is usually a sign of unempowered or absent leadership and weak decision-making. Iterative processes require exceptional leadership from management and active participation from the development team.

Iterative process also fall apart when the development team starts working first on whatever they find interesting, and never really completes anything; or when each team member or group picks a different feature and starts work in parallel, creating applications silos that never really integrate well.

Adaptive Process

Adaptive processes generally fall under the label of agile processes. Some of the common variants are Scrum, Extreme Programming (XP), and Lean Development.  They are generally considered the current best practice for creating new products, perhaps for the first three or so product release cycles. Agile processes are designed to allow a team to work through the project in very small steps of understanding the requirements for a feature, building the feature, and releasing working software for internal or production use.

The benefits of agile processes are that they usually produce a product quite close to the business needs, they are efficient in cost and calendar time, and they build a very strong development organization.

Our example application for Customer Support might be developed like this:

1. The Product Owner meets with the Business Owner and Software Architect to define the product backlog (features).
2. The Product Owner writes short narratives (stories) that describes how a user uses the application.
3. The Product Owner chooses a story to implement, gathers up the development, test and tech writer team.
4. The team breaks the story down to tasks, picks a group of tasks that can be completed in a short sprint (2 weeks or 4 weeks).
5. The team works only on those tasks for the sprint.
6. The team completes each task before moving on (code, etst and documentation).
7. The team meets daily for 15 minutes, and the Product Owner answers any questions from the team, fills in any gaps in the application definition, and re-iteratates how the user will use the application.
8. At the end of the sprint, the team delivers a working feature, which the Business Owner and real user can review and test.
9. Feedback on the application is incorporated into the next sprint.
10. The team continues working in short sprints, completing a feature in each sprint, until the application is judged “ready to ship” by the Product Owner and Business Owner.
11. Ship the application, and continue to release additional updates every couple of months to closely follow the market demand.

The problems with agile processes are that they require a re-design of your organization, with very high participation by business folks in the product development process, require business folks to have substantial technical understanding, require a development team with above average skills and smaller egos than average, and require the team to be completely dedicated to the project and co-located. The agile community and the software product development community would argue that these are in fact benefits. I tend to agree, but I have rarely seen the organization willing to embrace the process.

Poorly managed agile projects can go wildly astray, as there are few controls and opportunities for reviews. Costs and schedule can be unknown, and product development might say that the “project will get done when it gets done”, which is a challenge for business planning. I emphasize poorly managed here. Well-managed agile projects have very close coordination with business folks and managers, but require full participation from all stakeholders, which is rare.

Some of the key features are:

• Clearly defined roles of product owner, stakeholders, developers, and project managers.
• Working from a product backlog of features or “user stories”, rather than requirements or a schedule.
• Estimation of backlog items and tasks, but not the whole project.
• Daily formal and informal interaction between the product owner and developers.
• A co-located, completely dedicated, cross-functional team.
• Frequent product deliveries, typically monthly or less.
• Working in sprints (monthly increments or less), where the entire team is working on the same deliverables.
• Frequent re-planning (at the start of each sprint)
• Frequent reviews and continuous improvement (at the end of each sprint)

Agile will only be successful if you have a very strong product owner, and a supportive organization. If you have these, it will be the most efficient, produce high quality product, and build a more capable organization.

Agile closely mirrors how humans would divide up the work for any unknown task.

Recommendations

This Primer is about software product development, therefore, I believe that most of my readers should consider the adaptive processes, such as agile, and in particular, I like the Scrum methodology. There are many good books available and good industry knowledge of Scrum.

Those who are managing IT infrastructure projects should look at planned processes, and those who are managing IT software development projects in a high constraint environment, with many stakeholders and high risks, should consider an iterative process.

In adopting Scrum or other agile processes, please consider the following:

• Agile is a holistic company approach to delivering software.
• You will need to identify and empower Product Owners, who must be both business and technically savvy.
• You will need to dedicate the team. A team that has multiple project responsibilities will fail.
• The management team will need to understand the process, participate in the training, and participate in the project.

Will your Software Development Process make you successful?

You will be successful if your team is highly skilled, has low egos, knows what the product is supposed to do, and they all follow the same process. Ultimately, studies have shown that success is not that dependent on the process, although the mode of failure does tend to correlate to the process. Success is dependent on your team. The right team,with good leadership, will produce a great product.

Aphorism: When you come across a turtle sitting on a fence post …

Heard this on NPR – “When you come across a turtle sitting on a fence post, it didn’t get there by itself.”

Now I’m not advocating abusing our turtles. I do think this is a picturesque American farm aphorism.

I think this aphorism does capture the support we all get to be where we are. We shouldn’t forget that when we start feeling “important”, or when we look at someone else and wonder what special talents they had to succeed. I’ve also been thinking about this in context of Malcolm Gladwell’s new book, “Outliers”, which explores how skill, opportunity, support and long hours of practice combine, often by chance, to create successful careers.

I think this is particularly appropos of software development. I’ve had a number of people who were mentors for me, often as not unknowingly. They were just the type of guys that loved engineering, and wanted to help a young guy who also loved engineering.

Right out of school, I was working at General Dynamics, and sitting right across from a fellow by the name of Don Romary. I really didn’t know diddly, having just graduated from Michigan Tech. Don would call me over to show me something interesting he was working on, or to ask me about what I was working on. Don embodied quiet confidence. Among many things I learned, I clearly remember Don telling me “If you pick a good man for the job, he’ll learn what ever he needs to get it done.”

I’ve remembered that throughout the years. I’ve always tried to be less interested in specific skills when hiring engineers, and more interested in their desire, creativity and ability to learn.  We all need to keep learning, and we learn best on projects.

Give somebody a break, and throw them into a project over their head. Hire that Java developer to do C#, or a C++ developer to do AJAX work. Look at the person, not just the skills. They might not be as productive in the first month or two, but you’ll be helping a fellow engineer and building a stronger organization.

Be the change you want to see happen.

Aligning IT and Business

Catchy headline. “Aligning IT and Business”. Hey, everybody wants that! It’s like Mom and apple pie in our industry.

I don’t want it.

From the perspective of PRODUCT development, in contrast to writing internal applications for business, you are starting with a definitional problem.

First, if you produce software products or services, the folks who create and deliver those products are not IT.  They are the Product Developers. The IT group manages the company’s computing infrastructure.

Second, who are these Business folks? If you’re talking about sales and marketing communications, then yes, those are Business folks. But if you’re talking about the Product Managers, then they also are part of the Product Development team.

You should not be aligning two things that should not have been separate in the first place.

A central tenet of agile product development is that all the involved parties are working together from a common plan to produce the product. The Product Managers are designing and producing a software product, and they should know how it works, because they are sitting every day with the development teams (whether coders, testers, UX design, writers, etc). The software developers should understand the cost of adding “that one more cool feature”, in real money terms. Everyone should understand the customers, how they will use the product, what problems are being solved for the customer, and how the product compares to competitor’s products.

If you are still separating your software developers from your product managers, then you’re thinking like Chrysler.

If you have removed this artificial separation, you’re thinking like Toyota.

I’d rather be part of Toyota.

Aphorism – Malice vs Ignorance

I’m a big fan of aphorisms. One that I find myself pulling out in many situations, and which has served me well, is:

“Never attribute to malice that which can be adequately explained by stupidity.“

I do prefer to use the word “ignorance”, though, in place of stupidity, at least in the context of software product development.

According to Wikipedia, this is know variously as “Hanlon’s Razor” or “Heinlein’s Razor”.

I see from Wikipedia that there is a Brit version, which I quite like:

“Cock-up before conspiracy“

Colliding Worlds

I work for a Fortune 500 company, which recently bought the company I worked for, which in turn had recently bought the company I worked for. Just before these couple of acquisitions, I was part of the team managing the acquisition of another company.

I’ll keep the names of the companies out of this to protect the guilty and the innocent …

So in the classic fish swallowing fish picture, I worked for the second fish from the right!

Each of these companies had a different software development process, at various stages of maturity.

The big company had a lot of divisions which had never been integrated, so there wasn’t any company-wide process. Their processes ranged from some pretty respectable agile shops to totally ad hoc.

The medium company had traditionally been a mainframe shop, and traditionally had followed a waterfall process (I hesitate to call it planned – more passive-aggressive, really). They started the roll-out of their flavor of the Rational Unified Process, complete with a squad of IBM consultants and a bunch of IBM tools.

Now, I like a lot of things about RUP. And for this company, rolling out RUP and getting it adopted was an important step in their evolution as an organization. Going directly to agile might very well have been ‘a bridge too far’. I do however view RUP as incorporating ideas that were current when Philippe Kruchten wrote his book “The Rational Unified Process” – still a great book by the way – but ideas which might not be the best anymore for many organizations.

The world has moved on, and there is a half-life to all ideas in software development, as Philippe notes.

The small company that I worked for had been pretty darn productive. We had followed process ideas that evolved from my work at HP and some from Intel, and we were doing a flavor of agile before there was a name for it. As the company grew, and different folks joined as managers, our process decayed.

Then about 2004 we adopted XP and then Scrum, as the whole agile movement became formalized and there were books to which we could refer the managers and developers. After several years and many projects, we were improving continuously and had a pretty fair process in place. Nothing to write books about, mind you.

So, to cut a long story short, we get bought, the acquiring company rolls out their new process, which is really about 15 years old, and requires us to go backwards.

We lost several key developers as a result, and I feel that our product delivery was slowed to a crawl for several years. This was only one of many factors, of course. I’ll write about some of the others over time.

I see the company viewpoint as well – some groups within the company had a strong and current SDLC, others had a weak and old SDLC. We also had others with a current SDLC, like agile, but weakly followed, or with an old SDLC, well-executed. We were all over the map.

They didn’t single us out. It was important for integrating the acquisitions to have a standard process so different organizations could collaborate on projects, and so that we had a common model for planning and reporting within the company.

But, you bought the company because they were successful at delivering products. Why kill off what they were doing right, whatever it was? IMO, the real value in a software company is the culture, the ideas, the management team and the developers, in roughly that order.

What would I do differently, having lived this, and if I was in charge of the do-over?

I believe that corporations, like governments and schools, should set minimal standards and then govern to bring any groups that lag up to that standard. Groups who already exceed the standards, and who are able and willing to collaborate with the other groups, should be allowed to continue to develop and innovate, and identified as the process leaders “out in front”. I see this failing to occur in our schools, in our government, and in our companies.

The greatest under-achievers are not those who fail to meet the standard, but those who are not allowed to exceed the standard.

Welcome.

Hi, thanks for reading my blog.

My name is Darrell, and I’ve been a software architect and software developer for the past 25 years. I’m a veteran of two successful start-ups, one of which went public, and have worked at several large companies in the past, including Hewlett-Packard and Intel. I have also been involved on a limited basis with 5 other start-ups.

Besides being a software architect, I have been involved in engineering management, product management, product marketing, sales, and consulting.

I have a particular interest in the process by which an idea is identified, and a group of folks take that idea from Concept to Product Delivery to Growth Phase to Support Life.

Some years ago, in the mid-90′s, I wanted to write a book on the software development process for small companies. I never did complete that book, as I got too busy with a start-up, where we created one of the premier internet banking solutions (Voyager).

Consider this that book I never completed, with another decade of contemplation and experiences thrown in.