The Secret to Big Data Analytics Success Comes Down to One Word

Those of you who follow me know I am always evangelizing on the “Sentient Enterprise” capability maturity model for analytics that I co-developed with the Kellogg School of Management’s Dr. Mohanbir Sawhney. I tell anyone who will listen how one word – agility – is key to getting an enterprise to the “sentient” point where it can analyze data and make autonomous decisions at massive scale in real-time.

A company must take many steps on the journey to sentience, but most have to do with agility. The word is even embedded in the first of the Sentient Enterprise’s five stages – building the Agile Data Platform – proof of how front-and-center agility needs to be for anyone looking to survive and compete in today’s data-driven world.

Webster’s dictionary defines agility as “marked by ready ability to move with quick easy grace” or “having a quick resourceful and adaptable character.” In the corporate world, “business agility” is usually defined as a company’s ability to rapidly respond and adjust to change or adapt goods and services to meet customer demands. I’d suggest, however, that we also entertain a more basic definition, one that can help us identify and borrow lessons from agility in many different business settings.

Agility is the ability to “decompose” or break big problems or systems into smaller ones, so they’re easier to solve and collaborate around.

That’s the gist behind a comparison I often make between tomorrow’s data architectures and the Open Systems Interconnection (OSI) model that the telecommunications industry developed in the 1970’s to segment complicated infrastructure (wiring; relay circuits; software, etc.) into manageable chunks for better collaboration among various specialists (the Sentient Enterprise’s Layered Data Architecture is the way we do this with data). I like this example because, even though OSI was developed four decades ago, the technique of segmenting big systems (or data sets) into overlapping but distinct and manageable elements is a timeless ingredient for agility – one that we continue to see in some cutting edge settings today.

Check out a company called Docker to see what I mean. Docker lets you break down the app building process into a series of manageable steps. Through a simple “Docker Engine” and cloud-based “Docker Hub,” the company lets you assemble apps from modular components in a way that can reduce delays and friction between development, QA and production environments. By breaking things down into smaller components, Docker claims the app-building process becomes more manageable and reliable, allowing IT to build and ship faster, and run the same app – unchanged – on laptops, data center VMs and in the cloud.

Another example is the entire “microservices” approach to building software architectures. Unlike more traditional service-oriented architectures (SOAs) that integrate various business applications together, microservice architectures involve complex applications built from small, independent processes. These processes communicate with each other freely using application programming interfaces (APIs) that are language agnostic. You’re still building powerful architectures, but it happens more efficiently with modular elements broken down to focus on discreet small tasks. As a result, microservices architectures can be tremendously agile. They facilitate continuous-delivery software development and let you easily update or improve services organized around distinct capabilities like user interfacing, logistics, billing and other tasks.

As decomposition becomes part of a trend towards increased agility, it is however also import to call out the need for even more governance — governance that is put into the platform as a foundational element and not an afterthought. In fact, it sometime needs to be hidden from the individual developer. Combining the likes of DevOps principles and capabilities with microservices architecture are extremely crucial to the success of such strategies.

These examples show how we are moving away from monolithic applications and approaches that can be unwieldy and anything but agile. This is the same line of thinking when I talk about agile access to data in the Sentient Enterprise: You have to structure the agility with the right kind of governance, so you don’t create more problems than you’re solving. I’ve talked about the wild, wild west pitfalls of data anarchy and error. Even advocates of microservices caution against potential problems with things like network latency, load balancing and fault tolerance. Tackling new problems, however, is a small price to pay for the agility you can unlock by breaking things down in manageable, often profitable, ways.