Introduction
Conversational Insights.
If you’re one of those readers who are keenly following what was constantly being discussed in the past few weeks, this topic definitely would’ve caught your attention. And no, we are not talking about taking customer conversations and analyzing them for behavior (Though interestingly that also falls under the broader definition that we are trying to communicate.)
According to us conversational insights is using natural language to derive actionable insights at a moment’s notice, literally! The input can be text or voice based on your preference.
But before we deep dive into the story of understanding in detail conversational insights, we are going to take a step back and scan the lay of the land. We’re going to look at the BIG picture of BIG Data to see how and where data analytics will fit into the scheme of BIG things.
So what are you waiting for? Hop on for the ride of a lifetime to a modern space where data, engineering, and technology come together to fuel the future of this entire world.
Looking at the BIG Picture – BIG Data Value Chain
The compound annual growth rate (CAGR) for global Big Data Analytics spending over 2021-2025 will be 12.8% – IDC
A google search of the term “big data value chain” will show you what it is. Tucked neatly away in the archives of Springer publications is a journal that subtly explains what the big data value chain is. “The Big Data Value Chain is introduced to describe the information flow within a big data system as a series of steps needed to generate value and useful insights from data.” It does sit well within the definition of the value chain in the business lingo.
A more effective way for the reader to understand what big data value chain is to put it into a pictographic format and The European Parliamentary Research Service has done an excellent job there. A simple yet elegant way of representing big data flow in an organization.
As outlined in the image above, there are four major stages in the value chain, namely
- Generation and Acquisition – This stage deals with data (both structured and unstructured) capture and collection that happens at so many levels and channels. To give example, cookies from the website, QR codes for payments, website heatmaps, and RFID scanning at warehouses, are all some of the seamless digital ways in which data is captured.
- Analysis and Processing – This is the stage where data transformations happen before loading them into the storage devices. Some common ETL or ELT transactions can be quoted as general examples. Data discovery and cross-sectorial analysis also happen at this stage.
- Storage and Curation – After processing the data, the next step in the journey is to store it. This stage comes up with the most effective solutions for storing data like data warehouses, data lakes, etc. based on the requirements. Basic data quality checks, data validation, and other data curation exercises also form part of this stage.
- Services and Visualization – This is the layer where businesses or end users like you interact with the data. Various data analytics tools and dashboarding software interact with the stored data to process relevant information from it and present the insights visually in the format required by the end user.
However, the key point to note here is that each stage in the value chain has its evolutionary curve. Data capture, data processing, and data storage are pretty advanced in the evolutionary curve and have either attained maturity or are in the process of attaining maturity except for data visualization and information access.
Why so much Hype around Conversational Insights?
Indeed. There’s no dearth of analytics tools. A free spreadsheet tool can help in data analytics efforts. Still conversational insights or conversational AI-powered business intelligence is slowly gaining traction amongst business leaders.
Why? There are two reasons for the same.
Maturity Stage of Modern Data Analytics
Data visualization and information access have not matured enough to be inclusive. By inclusive, I mean the democratization of information access. A good part of the blame for this will fall on technical data access which is still too complex for mainstream business users. This in turn creates a great level of dependency on specialized teams for getting business insights. Leave the cost factor, the time delays to get one business dashboard is preposterous.
Maybe that’s one of the reasons that Gartner has gone ahead and predicted that
Through 2025, the majority of CDOs will fail to foster the necessary data literacy within the workforce to achieve their stated strategic data-driven business goals
To shift this trend and introduce data as a culture amongst employees, there is a renewed focus on simplifying information access.
Context-Driven Data Storytelling
Businesses are not looking at data as a standalone entity. There’s a story associated with data. Data storytelling is a framework to communicate information in a way that is easy to understand. It can be used by many professionals, across the length and breadth of the organization. It’s a useful methodology for everyone because it helps them communicate their findings in a way that makes sense to experts and laymen alike.
The same study from Gartner also points out that,
By 2025, context-driven analytics and AI models will replace 60% of existing models built on traditional data.
Thus the data lifecycle has come a full circle where information access is simplified and is presented with a story that makes sense of the metrics being tracked.
Behind the Scenes of Conversational Insights
Let’s now discuss the third most important area in the entire conversational insights ecosystem. What’s happening behind the scenes? All you can see is you’re texting or speaking to the tool and it displays the desired result on your screen.
A simple representation of how conversational insights is displayed in the below diagram.
As you can see there are three levels after the information seeker inputs his data question.
- Layer 1 – NLP+NLU Layer: This is where the first level of magic happens. It’s a given fact that machines don’t speak the way we speak. It’s a world of zeroes and ones. However, Artificial Intelligence is so advanced that a small tool from its arsenal called Natural Language Processing (coupled with Natural Language Understanding) converts human language into boolean values that a machine can understand. Once the data analytics layer presents the answer, this is the same layer that converts the boolean values into visual and text formats. Now off to the next stage.
- Layer 2 – Data Analytics Engine: The next stage is where the machine gets to work. It goes through the available data and like Santa’s little elves, picks relevant data points and processes them into required answers. We always have to keep in mind as explained in the previous blogs that a human psychologically hunts only for momentary insights. Now that the tool has the answer to your momentary question, it feeds it back to the previous layer that presents the answer to you, the information seeker.
- Layer 3 – BIG DATA: This is self-explanatory. This layer deals with the storage of data both structured and unstructured using data lakes, data lakehouses, or cloud data storage. The little bots from the data analytics engine come down into this layer to pick the required data points.
Of course, this is a rudimentary explanation of how a conversational insights platform is structured. Just like rocket science, it’s a complex process and one that’s continuously learning thanks to machine learning algorithms at work. There are complicated gears turning behind the curtains to bring the data insight you require.
Stay Tuned for the Next One
Now you’ve understood the three key areas one must know about conversational insights. Let me refresh your memory once and reiterate them for driving the points home.
- How the big data value chain has matured but data analytics has lagged behind.
- How conversational insights is riding the wave of the next generation of context-driven data analytics
- How conversational insights work behind the scenes
If you’re looking for an analogy, I’ve just given the simplest explanation for how a car works. I’ve pointed out to you that this is the steering wheel for direction control, those are the throttle, breaks and on your right side is the paddle shifter. I’ve not spoken about the engine, its capacity, how many pistons are there, and what’s the fuel injection system at work. But wait, the fun is not over yet! I will be speaking in detail about the engine shortly.
More blogs will follow suit that will speak in intricate detail about the technology followed, so keep your enthusiasm up. But until the next one, maybe you can read about
