Primer: What is a Software Development Process, and why do I need one?

I’m sitting here in the sun at McMenamin’s Imbrie Hall after work, enjoying a nice 85 degree sunny day, and sipping a pint of Terminator Stout. If you’ve worked in the tech industry in the Portland area, you know what I’m talking about.

I’ve been pondering how to describe the various flavors of software development processes, in a short and understandable manner. We used to be an agile shop (Scrum), and lately we’ve been required to follow an iterative process (RUP). This has really brought many of the differences into stark contrast. We were a small shop, we delivered applications about every half-year, we all were located in the same office and everyone knew everybody else. Now we’re part of a large company, where you might not even meet some of the members of a project, and some developers are on the other coast or in India. Our success on delivering projects has dropped, and the project timelines have doubled or tripled. However, the requirements are also more extensive than they used to be.

The software development process seems to be the eternal question in our industry. How do we efficiently manage defining the objectives of a project, defining the specific requirements, managing the efficient creation of the software, and managing the deployment of the software to produce the greatest benefit?

Although I now work for a large company, I’m addressing this post, as always, to the reader from a small software development organization, particularly start-ups and high-growth companies. I note that it’s always possible for small teams within a large company to act like a small company, but it’s really hard to pull off.

I also presume that, as part of the Primer, you’re looking for basic understanding of software development practices. Some of the statements here might be considered  by the expert practitioner as over-simplified, which I freely acknowledge.

As such, I’ll try to keep this post at a high level, and point you to resources which can give you more depth.

There are three general types of processes that have evolved over the last three decades – planned, iterative, and adaptive.

For the purposes of this discussion, let’s assume you’re creating a customer support application. You know that you’re going to need the following capabilities:

1. Support Rep can sign on
2. Support Rep can look up an order
3. Support Rep can look up a user profile
4. Support Rep can update an order (change payment method, shipping address, cancel)

Planned Process

A planned process is based on traditional project management techniques, which assumes the work is completely known before the project starts, the staff and resources are available, everything can be reliably estimated, and a schedule can be created.

Following a planned process, we would proceed as follows in our example:

1. Assign the requirements and development team.
2. Write a requirement document for all features.
3. Developers create the application, conforming exactly to the requirements.
4. QA tests the application for conformance to the requirements. Re-work is done as necessary.
5. The business owner reviews the delivered application, identifies the differences between what’s needed and the application, and creates a list for re-work. The re-work is either done, or the project is canceled.
6. The application is rolled into production. User feedback creates another re-work list. Defects are identified and patches are released.
7. Plan the next release for next year.

In the software world, this is commonly denigrated as a “waterfall” process, but that is not entirely accurate. By waterfall they mean that each step must fully complete before you can move onto the next step. This is an extreme and unsupported view.

A planned process is very useful in many corporate and IT projects, such as building a data center, introducing a new product to the market, and installing and configuring a new application such as a farm of web servers. This work is known, can be broken down to steps, and can be reliably tracked.

Generally these processes were called Software Development Lifecycles (SDLC), and followed traditional Project Management Institute (PMI) standards. There were various tools to evaluate your success, such as Six-Sigma, Capability Maturity Model (CMM), and ISO-9001.

However, for creative work, such as developing a new product, these processes have been shown to create failure and increase costs more often than they foster success.

Iterative Process

An iterative process involves working your way through the project in moderate size steps. A key feature of an iterative process is that it provides regular opportunities for adjustment of project goals and requirements, as you learn about the problem you’re solving. It also provides opportunities for regular review of progress and project goals by the management team.

Some of the approaches are Rapid Prototyping, which involved creating a working system early to evaluate requirements and design decisions, the Spiral Model, and the Unified Process, which is a planned but iterative process for discovery of the requirements and construction of the application. Unified Process also has an IBM version called the Rational Unified Process.

Generally, an iterative approach goes through several phases:

1. Problem Definition
2. Requirements Capture
3. Investigation
4. Construction
5. Deployment

You proceed by completing step 1, and then partially executing steps 2-4, perhaps on a monthly or quarterly basis, until the system is complete. At no time do you attempt to capture all of the requirements upfront and at a single point in time, or to construct the entire system in one go. You build the system one subsystem at a time. Finally, when the product is complete, you move to step 5 to deploy the product. You then return to step 1 for the next release cycle of the product.

Following again our Customer Support application example, and using an iterative process, you might proceed as follows:

1. Understand the requirements for feature 2 first
2. Fully implement the “look up an order” function. The remaining functionality (sign on, user profile, manage order) can be mocked up.
3. Test the developed features.
4. Show the application to the business owner, who would then identify differences between what is needed, and what was built, creating a small re-work list.
5. Defects are identified and fixes are incorporated in the next iteration.
6. Now understand the requirements for feature 3,and implement feature 3.
7. Similarly for feature 4
8. Similarly for feature 1
9. Ship the product. Gather user feedback and defects, and plan the next release for next year.

Note that there is no specific order in which the features need to be completed – but you should roughly complete a feature before you move on to the next.

Iterative processes can be very effective if managed closely. They typically require a substantial amount of training, and work best on projects where you have a pretty good idea of what you’re building.

If you are working on a project where a defect in the system design or construction could cause death, injury, catastrophic business interruption, or extreme embarrassment, you should consider using an iterative development process. It provides a pretty good balance between gathering complete requirements, efficient construction and testing at several stages of construction. However, these projects are less common than you might believe.

If you are managing a project that can be fully described, similar to building a house, you should probably choose a planned process instead.

Iterative processes can degrade into a planned process if managed by persons unfamiliar with software development, and who do not understand creative work. The projects can become subject to the dreaded “analysis paralysis”, where months are spent “capturing the requirements”, but which is usually a sign of unempowered or absent leadership and weak decision-making. Iterative processes require exceptional leadership from management and active participation from the development team.

Iterative process also fall apart when the development team starts working first on whatever they find interesting, and never really completes anything; or when each team member or group picks a different feature and starts work in parallel, creating applications silos that never really integrate well.

Adaptive Process

Adaptive processes generally fall under the label of agile processes. Some of the common variants are Scrum, Extreme Programming (XP), and Lean Development.  They are generally considered the current best practice for creating new products, perhaps for the first three or so product release cycles. Agile processes are designed to allow a team to work through the project in very small steps of understanding the requirements for a feature, building the feature, and releasing working software for internal or production use.

The benefits of agile processes are that they usually produce a product quite close to the business needs, they are efficient in cost and calendar time, and they build a very strong development organization.

Our example application for Customer Support might be developed like this:

1. The Product Owner meets with the Business Owner and Software Architect to define the product backlog (features).
2. The Product Owner writes short narratives (stories) that describes how a user uses the application.
3. The Product Owner chooses a story to implement, gathers up the development, test and tech writer team.
4. The team breaks the story down to tasks, picks a group of tasks that can be completed in a short sprint (2 weeks or 4 weeks).
5. The team works only on those tasks for the sprint.
6. The team completes each task before moving on (code, etst and documentation).
7. The team meets daily for 15 minutes, and the Product Owner answers any questions from the team, fills in any gaps in the application definition, and re-iteratates how the user will use the application.
8. At the end of the sprint, the team delivers a working feature, which the Business Owner and real user can review and test.
9. Feedback on the application is incorporated into the next sprint.
10. The team continues working in short sprints, completing a feature in each sprint, until the application is judged “ready to ship” by the Product Owner and Business Owner.
11. Ship the application, and continue to release additional updates every couple of months to closely follow the market demand.

The problems with agile processes are that they require a re-design of your organization, with very high participation by business folks in the product development process, require business folks to have substantial technical understanding, require a development team with above average skills and smaller egos than average, and require the team to be completely dedicated to the project and co-located. The agile community and the software product development community would argue that these are in fact benefits. I tend to agree, but I have rarely seen the organization willing to embrace the process.

Poorly managed agile projects can go wildly astray, as there are few controls and opportunities for reviews. Costs and schedule can be unknown, and product development might say that the “project will get done when it gets done”, which is a challenge for business planning. I emphasize poorly managed here. Well-managed agile projects have very close coordination with business folks and managers, but require full participation from all stakeholders, which is rare.

Some of the key features are:

• Clearly defined roles of product owner, stakeholders, developers, and project managers.
• Working from a product backlog of features or “user stories”, rather than requirements or a schedule.
• Estimation of backlog items and tasks, but not the whole project.
• Daily formal and informal interaction between the product owner and developers.
• A co-located, completely dedicated, cross-functional team.
• Frequent product deliveries, typically monthly or less.
• Working in sprints (monthly increments or less), where the entire team is working on the same deliverables.
• Frequent re-planning (at the start of each sprint)
• Frequent reviews and continuous improvement (at the end of each sprint)

Agile will only be successful if you have a very strong product owner, and a supportive organization. If you have these, it will be the most efficient, produce high quality product, and build a more capable organization.

Agile closely mirrors how humans would divide up the work for any unknown task.

Recommendations

This Primer is about software product development, therefore, I believe that most of my readers should consider the adaptive processes, such as agile, and in particular, I like the Scrum methodology. There are many good books available and good industry knowledge of Scrum.

Those who are managing IT infrastructure projects should look at planned processes, and those who are managing IT software development projects in a high constraint environment, with many stakeholders and high risks, should consider an iterative process.

In adopting Scrum or other agile processes, please consider the following:

• Agile is a holistic company approach to delivering software.
• You will need to identify and empower Product Owners, who must be both business and technically savvy.
• You will need to dedicate the team. A team that has multiple project responsibilities will fail.
• The management team will need to understand the process, participate in the training, and participate in the project.

Will your Software Development Process make you successful?

You will be successful if your team is highly skilled, has low egos, knows what the product is supposed to do, and they all follow the same process. Ultimately, studies have shown that success is not that dependent on the process, although the mode of failure does tend to correlate to the process. Success is dependent on your team. The right team,with good leadership, will produce a great product.