Are you frustrated on your unmapped data journey?

It’s going to take how much longer?! It’s going to cost how much more?!!

If those sound like all too familiar expressions of frustration, in relation to your data journey (projects), you’re in good company.

It seems most corporations these days struggle to make the progress they plan, with regards to building a Single Customer View (SCV), or providing the data needed by their analysts.

An article on MyCustomer.com, by Adrian Kingwell, cited a recent Experian survey that found 72% of businesses understood the advantages of an SCV, but only 16% had one in place. Following that, on CustomerThink.com, Adrian Swinscoe makes an interesting case for it being more time/cost-effective to build one directly from asking the customer.

That approach could work for some businesses (especially SMEs) and can be combined with visible data transparency, but it is much harder for large established businesses to justify troubling the customer for data they should already have. So the challenge remains.

Results of our Data Model survey

Our survey results touch on another reason for problems in ‘data project land’. In summary you shared that:

• 100% of you disagree or strongly disagree with the statement that you have a Conceptual Data Model in place;
• 50% of you disagreed (rest were undecided) with the statement that you have a Logical Data Model in place;
• Only 50% agreed (rest disagreed) with statement that you have a Physical Data Model in place.

These results did not surprise me, as they echo my experience of working in large corporates. Most appear to lack, especially the conceptual, data models. Given the need to be flexible in implementation and respond to the data quality or data mapping issues that always arise on such projects, this is concerning.

With so much focus on technology these days, I fear the importance of a model/plan/map has been lost. Without a technology-independent view of the data entities, relationships & data items that a team needs to do their job, businesses will continue to be at the mercy of changing technology solutions.

A metadata gap in your models

Your later answers also point to a related problem that can plague customer insight analysts seeking to understand customer behaviour:

• All of you strongly disagreed with the statement that all 3 types of data models are updated when your business changes;
• 100% of you also disagreed with the statement that you have effective meta data (e.g. up-to-date data dictionary) in place.

Without the work to keep models reflecting reality and meta data sources guiding users/analysts on the meaning of fields and which can be trusted, both can ‘wither on the vine‘. Isn’t it short-sighted investment to spend perhaps millions of pounds on a technology solution but then balk at the cost of data specialists to manage these precious knowledge management elements?

Data speakers need to share the data journey reality

Perhaps those of us speaking about insight, Data Science, Big Data etc also carry a responsibility. If it has always been true that data tends to be viewed as a boring topic compared to analytics, it is doubly true that we tend to avoid the topics of data management & data modelling. But voices need to cry out in the wilderness for these disciplines.

Despite the ways Hadoop, NoSQL or other solutions can help overcome potential technology barriers – no-one gets data solutions for their business ‘out of the box’. It takes hard work & diligent management to ensure data is used & understood effectively.

I hope, in a very small way, these survey results act as a bit of a wake-up call. Over the coming weeks, I will be attending or speaking at various events. So, I’ll also reflect how I can speak out more effectively for this neglected but vital skill.

Even more challenges on your data journey to an SCV

On that challenge, of why businesses fail to build the SCVs they need, another cause has become apparent to me over the years. Too often, requirements are too ambitious in the first place. During my time working on both sides of the “IT fence“, it’s common to hear expressed by analytical teams that they want all the data available (at least from feeds they can get).

Without more effective prioritization of which data feeds, or specifically which variables within those feeds, are worth the effort – projects get bogged down in excessive data mapping work.

Could a Data Lab/playpen approach help you?

Have you seen the value of a “data labs” approach?

Finding a way to enable you analysts to manually get hold of an example data extract, so they can try analysing data and building models, can help massively. At least 80% of the time, they will find that only a few of the variables are actually useful in practice. This enables more pragmatic requirements and a leaner IT build which is much more likely to deliver (sometimes even within time & budget).

Here’s that article from Adrian Swinscoe with links to Adrian Kingwell too:

How are you progressing on your data journey?

What’s your experience?

If you recognise the results of this survey, how do you cope with the lack of data models or up-to-date metadata? Are you suffering data project lethargy as a result? Care to share with our readers?