Is your data project really a software project? Or vice versa?

Do you recognise that your top priority project is both a software project & a data project? In this guest post, we learn about the intersection between data & software, plus why seeing projects this way matters.

Richard Pugh, Chief Data Scientist at Mango Solutions has a wealth of experience to draw upon. Both through his own work and his active engagement in the EARL community. Richard has seen plenty of projects succeed or fail by neglecting either sufficient focus on data or software.

He has shared with us before on Data Science readiness and why Santa needs Big Data Analytics. In this post, Richard helpfully focusses on why the intersection of data & software engineering matters for digital projects. Over to Richard to show us his perspective…

Value is at the intersection of data & software

For the last 18 years, Mango has been helping customers deliver on the potential of data and analytics.

When we started Mango back in 2002, the wider world of data and analytics was mostly reactive, with workflows conducted by individuals who produced reports as ‘one-time’ outputs. As such, while data professionals wrote code, it could largely be considered a by-product of what they did. The advent of data science, together with the increasing need for just-in-time intelligence, has driven more proactive analytic workflows underpinned by open-source technologies such as Python and R.

Working at the forefront of data science, Mango understands the vital role of technology; to allow data to be transformed into wisdom in a repeatable way and deployed to business users at the right time, to support informed decision making. There is clear learning here for modern technology initiatives:

To realise business value, it is vital that we balance both data and software elements of technical projects around a common and clear purpose.

Every data project is a software project.

Back in 2012, Josh Wills described a data scientist as someone who is “better at statistics than any software engineer and better at software engineering than any statistician”. While modern data science incorporates a broader range of analytic approaches than statistical modelling alone, Josh’s description of data science at the intersection of analytics and software engineering still holds today.

The changing role of data and analytics from a reactive practice to a strategic approach has driven the need for advanced analytics to be combined effectively with software engineering. If analytics is now an always-on capability, we need to codify the intelligence in systems that can be properly deployed and scaled within a business.

A ‘local’ alternative is just not practical – you can’t become a truly data-driven business if analytics is run by experts on their laptops. We can’t stop making intelligent decisions if a data scientist is on leave. If a consumer purchases a product on Amazon, they will not wait hours or days until a statistician crunches the data to come up with other recommended products.

To positively impact a business with data, an end-to-end analytic workflow needs to be implemented using software engineering approaches. This encompasses everything from the creation of data pipelines, the deployment of models, and the creation of user interfaces and applications that can convey insight in the right way, linked directly to operational systems to action and process outcomes.

Every software project is a data project.

Increasingly digitalisation and regulation have driven more focus on requirements regarding the role of data in software systems. We can consider 3 types of requirement regarding the treatment of data:

• User – requirements relating to users and preferences to provide a more personalised experience
• Governance – requirements relating to the way in which data is managed in a secure fashion to confirm with data regulations and protect confidential data
• Provenance – requirements relating to historical system actions to provide an audit trail, or to enable rollout back to, or understanding of, previous actions
• Beyond this, the most important consideration in the design of modern systems is the ability to leverage advances in data and analytics to create richer, more useful experiences and applications. A growing understanding of the possibilities offered by analytics allows us to strive to ask better questions – to build software tools that are truly aligned to users’ objectives.

For example, imagine we are building a software application to be used by call centre staff when speaking with customers. Traditionally, we may have built a system that combined data from various sources to give the user a single view of the customer. Perhaps this included data on previous orders, previous interactions, demographic data etc.

With data science, we could extend the functionality for the user – perhaps to include an understanding of likely customer churn linked to suggested retention actions, or a suggested ‘next best offer’ for the customer, or suggestions around the ways in which to talk to the user. Perhaps when the customer calls the call centre they can be allocated to exactly the right person to talk to, as opposed to being randomly allocated to the next available agent.

The use of data and analytics in software can have a transformative effect on the quality and usefulness of our software systems.

How can you improve your customer journey with such a perspective?

Helping customers build capabilities at the intersection of data and software is the most effective way to unlock value in an increasingly digital economy. Technology businesses like ours who want to be part of that customer journey need to be ambidextrous in their approach to data and software, agile in their execution and above all empathetic to each customer’s unique context.

We’re excited to apply our passion for data science to a wider market as we join forces with Ascent – increasing our combined ability to design and deliver ‘the big picture’ for customers that helps them compete and flourish.

Thanks to Rich for sharing that perspective from Mango Solutions. What about you? Do you routinely consider both the data & software perspectives of all your projects? What would you do differently if you did? What’s the first step you could take to get there?