The Threat of Invisible Data to Retailers


There are two forms of invisible data:  

  • The data about a brand’s customers from sources within the company
  • The data about a brand’s customers from sources outside the company

What is invisible data?  Let me explain…

A retail brand that has grown over time has many sources of data to manage. There are systems to manage customers, products, online and offline sales, inventory, shipping, finance, etc. These sources are separated by technical and organizational boundaries. Looked at in isolation, you’ll often hear these referred to as data silos. 

Any silo is easy to analyze – but only on a single area of the business. Decision makers need a comprehensive view of all the lines of the brand. And since data is not stored and managed in a single, central location, to easily ask business questions, extraordinary time and effort is spent to bring disparate sources together into a unified view. That’s why a solution – the process of transforming and mapping data from one form into another – is required.

Additionally, data must be made available in a single repository and data format. This data unification process has taken on a name of its own, “data engineering.” Over time, the behavior and practice of manipulating data has been normalized to the point where a new job category of “data engineer” has recently emerged. For retailers having multiple complex systems with complex requirements for data integration, governance, security and reporting, data engineering goes well beyond manipulating data into the infrastructures and pipelines that are required to unify the data, as well. Data engineering can be so complex it requires a computer science degree.

However, even the most skilled data engineers can’t keep up with the volume of data at the speed a business requires. So manual manipulation of data by multiple non- or less-technical individuals within a company is regularly performed to normalize and rationalize the overlaps and inconsistencies in data from across systems.  It’s not uncommon to have multiple people working with data in excel and sharing spreadsheets via email, cutting and pasting to generate the reports a company requires. This process is fraught with risk and by the time analysis is “complete,” it’s irrelevant to where the business is today and more relevant to a time that existed at some point in the past.

But what’s even more frightening, is at the end of all this effort, once a quarter, somehow these numbers that “answer” very important questions on the state of the business, actually make it into the boardroom. If one person in that meeting were to ask, “Can we rerun this report?” It can’t be done. The data that was used to create the report is floating around in excel sheets, access databases, emails … and people’s heads. There is no central, unified source to refer back to. It’s invisible data.

And that’s just the data within the company.

For retailers that can successfully pull all relevant customer data into a single view of the customer, they can then tell you all about their customers’ buying habits and relationships to the brand … and other customers. However, what that retailer doesn’t see is where that customer may be going in the future to make purchases instead.  The next Amazon Prime or Zappos isn’t discovered until after the customer is lost.

GDPR compliant data may be available that can be connected with a company’s internal sources into a Single View of the Customer. This allows the company to intersect cohorts from internal and external data sources to provide a more complete picture of their customer’s journey.  But if that external, third-party data isn’t incorporated, it remains invisible as well, and the company will continue to have blind spots that can threaten its continued success.

The future of retail belongs to those who can master their data at the speed of business and extract insights automatically. This must be done with a minimum of data engineering. This is why digital transformation and the ability to express key business concepts in a business view of data is so relevant now. To keep up and illuminate the important data insights a business requires to operate successfully, the traditional tools of ETL and relational databases are no longer sufficient.

Pete Aven is the Director of Sales Engineering for FactGem.





This ad will auto-close in 10 seconds