Unlearn SDLC and TNF: the death of requirement definition

Everyone who has learned software engineering has seen this picture. This was the most important message learned in young days. If customer requirements are not clear and precise, the project cannot be delivered. It was an axiom to be kept to heart. Another most important axiom in IT comes from databases — that all data in the store can be perfectly preserved in Third Normal Form. Third Normal Form was critical to building modern database systems which made sure that there was no data loss. All data was perfect, all relationships were accounted for. A true picture of information existed. And the truth was unique and identifiable.

These two axioms supported each other and created a foundation for Information Technology growth in large organizations. They influenced how projects were delivered, how organizations were structured and how interactions happened between IT and other departments. It worked flawlessly for decades, as the bulk of work was in conversion projects, where requirements were easy to define — e.g. manual to IT process conversion, offline to online, etc. With this separation of structure, IT departments took a servicing role and IT professionals focused on learning new technologies, new systems, and releases.

This flawless structure is now under threat. Slowly the “needed” role of IT in the value chain began to move from servicing to strategic. There are two major needs which are part of this change:

Compressed time frames to deliver.
Discovery-driven requirements.

These changes are breaking the SDLC-influenced transaction model between IT and business.

While IT organizations have responded well in application development with methodologies like Agile and DevOps, they lack clear methodological guidance in data management and analytics. Early projects in data management were influenced by SDLC and TNF thoughts and tried to create a “single point of truth” data warehouse. Most of these data warehouse projects are unable to achieve the objectives, have high costs and tremendous delays. Attempts to further force-fit the SDLC approach in data management will lead to more wasted spending — data management projects need to follow a discovery ladder.

There are two major characteristics of data projects:

Data models and functional mappings are much more fluid — conceptual artefacts, not strict truths.
The data system design is endogenously dependent on the inter-relationships existing inside the data. Hence system design cannot be fully pre-discovered and needs to be constructed progressively using small discoveries.

It is very critical for IT professionals to unlearn the SDLC and TNF axioms. If not, there will be many failed attempts before new paths are found.

This 2016 piece set the foundation for what we later named the single source of truthful intelligence — and for how Softsensor builds data and AI systems through discovery, not rigid requirements. Originally published on LinkedIn →

← All insights

Unlearn SDLC and TNF. The death of requirement definition is real.