On Managing [Complexity in] Systems Development

Modern society invariably tends to need complex¹ systems. (If a system is too 'simple', there may not be any market for it). Consider, for example, some of the successful products in present times:

1. Smartphones such as the iPhone, the Samsung Galaxy
2. Tablets such as the iPad, Google Nexus 7
3. Electric cars such as the Tesla

Each one of the foregoing has, in addition to software, a collection of parts, in several levels of hierarchy, in creating the end user product. (A blog post makes an interesting observation that mature products tend to have not more than 4 levels of hierarchy in their bill of materials).

Even software-only products exemplified by today's social networks, software-as-a-service (SaaS), etc., have, behind them, a huge cloud computing environment ².

Now, why this elaborate preface? What methodology can be successfully employed in managing the development of such complex systems? A methodology that helps in conquering the system complexity. Hierarchy, it turns out, is a method to conquer the complexity of software. While the concept of hierarchy is not new, its effective utilization in managing software (or software-centric) projects tends to be.

In order to manage a development a software (or a software-centric) project effectively, a hierarchical set of requirements, at whose leaf nodes are the user stories ^3,4, needs to be created. A user story, when fully implemented, i.e., done, is ready for end user consumption. User stories provide a unit of development, and measurement of the value, with a end user in mind.

Agile practitioners can use several concepts of hierarchy in managing the software development that suits their particular environment:

1. The Difference Between Agile Themes, Epics and User Stories.
2. Rally: Story Hierarchy
3. Microsoft: Working with Linked Hierarchies (Tree of Work Items) 

It is my hope that this blog post can be useful in prompting engineering managers to establish the user story hierarchy early on during the project.

---

¹ It is important in this blog post to distinguish between the words 'complex' and 'complicated'. 'Complex' implies a collection of interacting entities ending up serving a meaningful purpose. 'Complicated' is a collection that either does not produce a meaningful purpose or, if it did, it is very difficult to comprehend.

² Google provides a map of their data centers on their company web site.

³ The concept of user stories is now well entrenched in Agile environments as the Agile methodology experiences its inexorable growth in software development. As a matter of fact, it is even making inroads in hardware-oriented projects.

⁴ A presentation on user stories, titled Invest in Good User Stories, is a good and succinct introduction.