Dark data, Dirty data

Written by
Mark Bands

Partner

Published on

14 April 2022

Under the category

CLM Experts, Let's Talk

Does your organisation know the value hidden in dark data and, equally, the risks lurking in your dirty data?

It’s the 23rd March 2003 and the Academy Award for Best Animated Feature has just been won by an animated Japanese fantasy film, Spirited Away. In the imaginary story there are small, unfamiliar characters called Makkuro Kurosuke. In the English dub of the film, they are referred to as ‘Dust Bunnies’.  These elusive, sooty creatures are tennis ball-sized, pitch-black, and fuzzy-haired beings with two large eyes and long, thin limbs. They move by hovering around and can extend stick-like limbs from their bodies to carry out specific tasks or lift ultra-heavy objects. 

Hold that thought, park the fantastical imagery - and the 2003 nostalgia - and let's shift our focus for a moment. Looking into the domain of organisational data management, we note that in recent years two prevalent and important concepts have arisen relating to what are called ‘dark’ data and ‘dirty’ data. 

Dark data is defined as the data that organisations collect, process and store as an outcome of business-as-usual activities, but fail to identify, categorise, or even use for actionable business insights.  Conversely to this, dirty data is data that you already know you have, but that is clearly incomplete, incorrect, contains unreadable content or is duplicated in multiple data stores, in multiple forms. In short, the kind of data you don’t really want.

Let’s revert back to Spirited Away for a minuteAs the narrative events of the story unfold, the lead character Chihiro befriends several dust bunnies after learning that if they are not given a job to do, they turn back into soot. It can only be analogous then that like the dust bunnies, dark data should be given a job to do and dirty data should be fixed, and fixed fast. 

While it is logistically expensive to store large amounts of dark data, lost revenue opportunity is by far the larger issue.  Experience has proven that if this data is analysed, categorised, and mined, it contains hidden treasures in the form of insights that can be turned into actionable and revenue generating business initiatives.  

Dirty data is staggeringly risky for a financial services organisation to use in day-to-day operations because it inherently creates transactional, operational and compliance risks. Research has indicated that inaccuracies could be costing firms between 15% and 25% of annual revenue. With global banking revenues of over US$2.2 trillion, this means that dirty data may be costing the global banking sector over US$400 billion a year. Enough said, right?

In our Client Lifecycle Management (CLM) advisory work, Aurora SDE often encounters situations where the CLM journey can be markedly improved, both tactically and strategically, by understanding and then utilising dark data and/or fixing dirty data.  With this pursuit of happier banking customers in mind plus increased shareholder revenues, better utilised staff effort and a massive reduction in compliance and regulatory risk, we strongly recommend an immediate focus on your neglected data dust bunnies.     

We also really recommend watching Spirited Away if you haven’t already seen it.  

Keep Reading

Aurora Live »

🌈 Embracing Allyship 🌟

As we celebrate LGBTQ+ history month, Malcolm Mackenzie, author of Queerbook joins our CEO, Sean Vickers to talk about the importance of the workplace ally and how it's less 'us and them', and more just one big 'us'.

Continue Reading »
Sean Vickers

Chief Executive Officer

21 February 2024

Women in Tech

When it comes to celebrating women in technology- have we been prioritising visibility over accessibility? Aurora’s Global Business Manager Rebecca Grant, Chief Operating Officer Jiwan Laly, Associate Kamile Survilaite and Partner Sally Taylor reflect on the term Women in Tech, the roots of its cause and how it could adapt for the future.

Continue Reading »
Rebecca Grant

Delivery Lead

Jiwan Laly

Chief Operating Officer

Sally Wyllie

Partner

Kamile Survilaite

Associate

25 October 2023

Kickstarting a Career at Aurora

Aurora Business Analyst Saihej Mangat and Delivery Lead Joshua Dent discuss the importance of flexibility and training when transitioning from education into full-time employment.

Continue Reading »
Saihej Mangat

Business Analyst

25 July 2023

Still haven't
found what you're
looking for?

Get in Touch

Be the early bird and get in touch today if you have any questions, thoughts or ideas. We’d love to hear from you.

Book an appointment today

Go to Calendly