In this high-speed, interactive landscape, data analytics is already making significant and beneficial changes to our world. If we can make the most out of its huge capacity for good, data can drastically improve outcomes and drive stronger relationships between businesses and their customers along the way

Data is changing nearly every aspect of our lives, whether we’re aware of it or not. The way we purchase goods, run our businesses, treat medical patients, and manage our finances are all increasingly shaped by data. It’s the key to solving some of society’s biggest problems; famine, disease, poverty and ineffective education.

In business, we need to make sure we are ahead of the curve as this data revolution takes shape. With connected technology putting our customers firmly in the driving seat, there are clear challenges which will need to be overcome.

In order to deliver on these burgeoning expectations, there is a lot of pressure on businesses to achieve and work with a more complete understanding of their customers’ lives

However, within this environment lies an interesting paradox: people expect a round-the-clock, ‘always-on’ approach from brands, yet won’t accept anything that appears to be too intrusive or with the wrong message. As organisations of all shapes and sizes increasingly find themselves in possession of more customer information than ever before, there also comes greater responsibility. Evolving data protection regulation means that businesses will need to look for guidance and support to meet new regulatory requirements. Moreover, cybersecurity is an ever present concern for organisations of all sizes. The future is packed with opportunity and we have to make sure we are willing and ready to move with the times.

Whether it’s via laptops, mobile phones or any other connected device, our customers will increasingly live their lives online. The world is becoming more connected every day. If we are serious about keeping up with consumer-driven change we need a truly holistic approach, one which protects our customers and our products from cyber threats, whilst ensuring the customer journey is as seamless and fluid as it needs to be. This type of joined-up thinking will need to become the norm, as the datafication of our world continues.

We are all living through a period of exciting, widespread technological change. We need to be prepared to take on the challenges and embrace the many opportunities that the data revolution brings.

The evolution of Big Data - the '6 Vs'

Let’s address the elephant in the server room – ‘Big Data’. While many may focus their attention on the idea of Big Data, it’s not new and the term can be misleading. Large data sets have been in existence for many years, built and used by major companies for decades. In fact, many commentators have argued that ‘Big Data’ really just means more data.

At Experian we process more than 1,151 billion records annually, with a global segmentation of more than 2.3 billion consumers in more than 30 countries, and demographic data on 700 million individuals and 270 million households combined – that’s pretty big data.

There is no fully agreed definition of Big Data. Simon Rogers at The Guardian took a tongue in cheek view: “Big Data is one byte more than you are comfortable dealing with.” But actually, this is rather close to the truth as Big Data is data that is so big it won’t fit on a single machine. It has to be spread over many machines.

And it can come from anywhere, so it might be in strange and exotic formats. And it’s added to and updated all the time.

The Four V’s – a term which IBM coined in 2012 to describe the phenomenon of Big Data, expanding an earlier version of Gartner’s Three V’s – are still a very useful starting point for thinking about data.

Volume

The vast amount of data generated every second, by organisations and consumers, is hard to fathom. Many data sets are too large to store or analyse using traditional database technologies, and are also continuously being added to or updated. This volume is set to increase exponentially as the Internet of Things matures, though this eruption could also bring a greater volume and depth of insights into customer behaviour, if harnessed effectively. Either way, the challenge of increasing volume is clear.

Velocity

While data volumes grow, the speed of data creation and use is increasing too. This means processing, storage, and analysis must accelerate in tandem – business advantage lies in having and acting on the most up-todate information, which means receiving data and insight as soon as possible, then acting with equal speed. This high velocity also means that data is becoming more and more perishable, as it’s updated or made obsolete faster than ever before. Real-time analysis helps to deal with velocity, allowing businesses to make decisions based on the most up-to-date information.

Variety

Data comes in numerous shapes and forms, from geospatial data to website logs, from tweets to visual data like photos and videos. Though often overlooked compared with the well-publicised issue of ‘volume’, the variety of data out there is likely to be a bigger problem for most businesses. The ability to harness the different forms, together, can be the key stepping stone to the unified insight that many businesses are crying out for. A ‘holistic view’ is what companies should strive for, and dealing with data variety is the key.

Veracity

Veracity is about ensuring the reliability and validity of the insights derived from data. Inaccurate data is virtually worthless, even damaging in some cases. The flipside is that chasing veracity can lead to over-cautiousness, as organisations and individuals wait for perfect, clean data before making any decisions – something which is very often impractical. Veracity often depends on individual users, meaning engagement with data and a commitment to cleaning it up – and keeping it clean – are critical on a person by person basis.

While these four V’s represent significant data-related challenges, we’ve reached a point where they no longer encapsulate all of the major issues. We would argue that there are two further V’s to add to the list, to help provide a more complete picture. ‘Vulnerability’ and ‘Value’ should be considered by organisations when assessing their data landscapes.

Vulnerability

The proliferation of data has left many people feeling exposed and vulnerable to the way their data is being used.

As people think about the issues and learn more they don’t always become more reassured – in fact, it makes some more worried because they were not aware of how much data is being collected and used.

Conversely, a growing number of ever more tech-savvy consumers are willing to sacrifice some privacy as a tradeoff to the benefits of digital, personalised technology, but under their own preferences and conditions. Increasingly, people want to be informed about data use and have the ability to easily opt-in or out at any point in time.

The challenge for organisations is to find a way of addressing their customers’ concerns and exceeding their expectations. In order to alleviate confusion and apprehension, a growing number of companies are moving towards privacy by design and becoming far more transparent around data usage and value.

In this context the provenance of the data, right back to its origin, will be key. Organisations know that data is exploding all around them; mobile data generation, realtime connectivity and digital business have changed the scene entirely and made things more difficult in many respects. Analytics have an increasingly important role to play in data security and are already transforming intrusion detection, differential privacy, digital watermarking and malware countermeasures.

However, security is also about building brand reputation and trust. Strong security practices, including the use of advanced analytics capabilities to manage privacy and security challenges, can set businesses apart from the competition and create comfort and confidence with the public.

Value

At the most simplistic level, data has no intrinsic value. It only becomes useful when you’re able to extract the insight needed to solve a particular problem or meet a specific need. Once you can do this, the data acquires value through the business impact and consumer value this insight delivers.

Consumers are looking for value in terms of convenience, better products and better service. Organisations are seeking value via more engaged customers, lower costs and reduced business risk. For both parties to be successful there has to be a fair exchange – each needs to feel satisfied.

This means that data – and data analytics – is only valuable if it generates some form of payback. Thankfully, advances in analytics are helping businesses to achieve this more consistently, combatting the challenges of data volume, variety, and velocity, and delivering the allimportant value. This is being done by:

• Tying insights more closely to business decisions
• Drawing on, integrating, and analysing new data sources
• Moving beyond simpler business intelligence and analysis, towards diagnostic, predictive, and prescriptive analytics
• Developing data strategies that relate closely to clearly defined business goals, outcomes and use cases, rather than simply deciding a ‘Big Data Strategy’ is required, and proceeding in a business vacuum

Value exists in a two-way model between both parties. The organisation provides added value in the form of superior, more relevant content, products, and user experiences, and the recipient engages with that content, consuming it, buying products and signing up for offers and promotions.

To garner the full value that data can offer, organisations, governments and regulators will need to invest time and money in educating both the general public and businesses about how to manage the vulnerabilities and value opportunities that data presents

What companies should focus on

• Invest time to clearly understand the value buried in data and develop a clear data strategy
• Monitor regulatory change carefully, particularly with regards to security requirements.
• Consider how the use of data could be more transparent to customers.
• Proceed cautiously and ensure that you develop an appropriate business model.

The trust factor

The opportunities presented by data are universal, but alongside the extensive upsides, concerns about privacy have to be addressed. The impact of data sharing on the notion of privacy is still in flux. What is clear is that organisations must build trust with consumers, by demonstrating their integrity through better data stewardship, transparency and veracity.

There is a need for more conversation and more openness about how data is collected and used for the benefit of the end users. We are all very aware of the great advantages that new data-driven technology can bring, from fighting disease and tackling environmental challenges, to powering fitness apps and satellite navigation systems, and yet the way data is harnessed to make these things work isn’t something that has been part of the ‘data narrative’, to date.

A general lack of understanding has caused understandable scepticism about the increasingly substantial role that data is playing in our daily lives. Only 14% of people say they have a good understanding of how personal information and data is used by businesses. Genuine concerns persist around whether some people will be discriminated against unfairly as a result of our shift towards a data-driven culture.

Privacy is also a major concern for many people and these issues need to be debated openly and frankly, so that we can all agree how we can utilise the power of data in a way which is fair, transparent and for the benefit of everyone.

Only 14% of people say they have a good understanding of how data is used by businesses

New attitudes, same standards

However, in the digitalised world there is evidence that value and convenience are in fact superseding trust concerns. But that doesn’t mean that organisations can get away with anything less than the highest standards when it comes to using personal data. Quite the opposite, in fact. The problems arise when something goes wrong – reputation is damaged and trust is lost.

It is worth considering how attitudes are changing by generation. 50% of the working population will be Millennials or younger by 2020, a demographic group which has demonstrated a much more relaxed attitude toward sharing personal information with brands in a fair value exchange than older age groups.

However, there is plenty of evidence that convenience is in fact the primary driver for all generations already. In a recent survey looking at attitudes toward secure realtime exchange of bank statements, which consumers consent to provide in support of a credit application, 85% of respondents said that they believe banks should offer the service. Furthermore, 75% said they are happy for automatic updates of their data to be used to help avoid any repayment issues.

One thing is certain: today’s consumers are far more aware of the value of their data. They recognise that they own their data and accept the responsibilities that come with this ownership. That said, this increased awareness has not meant that more consumers are shying away from sharing their data, nor has it therefore hampered the growth of the data market.

In fact, people seem to be increasingly comfortable with sharing their data, on their own terms. A recent Experian survey found that 69% were happy for brands to use their personal information to send them discounts on products and services that they really want.

Trust from transparency

Transparency is essential. Customers are far more willing to share data with brands they trust and are already loyal to. This means respecting and protecting privacy. Omnichannel marketing and sales are now a reality, and they truly put the consumer at the heart of business. This means bigger data volumes, which, in turn, mean more focus on security. Consumers need to know that their data is safe, and the more they share, the more important that becomes.

Moreover, people need to feel that they can have a channel for challenging accuracy with data holders. One holistic model for this could be “ethical business regulation” as positioned by Professor Chris Hodges – a clear measure and message that reassures consumers that all businesses must meet a certain, agreed ethical standard. There needs to be more encouragement for businesses to act ethically, but businesses should seek to resolve complaints from data subjects directly in the first place.

Where a consumer and a business cannot resolve a data accuracy complaint, the consumer should be able to go to a free-of-cost ombudsman service – an informal route that leads to quick resolution (as works with financial services in the UK for example). The ombudsman can take an overview of the market and help keep regulators informed about issues and trends. Regulators would then support businesses that act ethically and clamp down hard on the ones that don’t meet their legal requirements.

Rather than having to confront hordes of resolute fundamentalists, businesses are increasingly faced with entrepreneurial pragmatists. Most people are more than willing to engage with the data economy, provided it’s worth their while. This has been clearly demonstrated by supermarket loyalty card schemes for the last two decades, where purchasing data is used to provide more targeted, personalised shopping experiences and promotions.

What we can take away from this is that there is an opportunity to develop a culture of data exchange fit for the 21st century economy. Organisations across sectors need to lead that dialogue, stepping up efforts to raise awareness about the way in which data is created, collected and used for the benefit of all.

What companies should focus on

• Make transparency a core part of your strategy and daily operations.
• Always act in the best interest of the customer, don’t just talk about it.
• Ensure frequent and open communication
• Question if what you’re doing is delivering value to the customer

Data freedom

We are witnessing one particularly disruptive data trend. The opening up of data, and its democratisation through open source solutions and Application Programming Interfaces (APIs), brings new kinds of opportunities for organisations and their customers. But what’s actually happening in this arena and what does more open, freer data mean?

As data volumes and types increased and evolved in recent years, we’ve seen a growing group of organisations and individuals campaigning for more freedom and openness of data. Rather than cloistered data, hidden away, these groups want data to be opened up to the digital landscape.

‘Data Freedom’ is manifesting in numerous guises through various channels, with the potential to radically change how data is used, who controls its use, and where it is used. Here are some of the most important types.

Open Data

As the name would suggest, Open Data is data that is made available for anyone to access, use, and share, whether they’re a business, government, organisation, or individual. Many of us have always used Open Data, it just hasn’t had that label before. Now we have a brand name for this kind of information.

The Open Data trend is becoming significantly more prevalent in the public sector, but some private sector organisations have begun dipping their toes into the seas of Open Data too.

Many commentators see a potential gold rush from the rise of Open Data. The land grab has certainly materialised, as Open Data sets are proliferating fast, with millions of data sets available through a handful of dedicated sites. There are numerous formal initiatives aimed at promoting data sharing, with a view to driving greater innovation and competition. While these formal initiatives are still developing, Open Data is very much here to stay, and businesses will soon have to consider their stance and offerings within the context of Open Data.

Data Portability

The UK government has a stated ambition to have a leading digital economy. They fundamentally believe that a strong digital economy is good for consumers, good for businesses and ultimately good for the country. This has been a developing philosophy since 2012, when they launched the miData initiative, and the sentiment has become increasingly visible in both UK and EU legislation and regulation.

More recently the focus has switched to banking, where the Competition and Markets Authority has mandated that bank product and transaction information is made available in the market through Open APIs. The first product information in this space is expected early next year and transaction level information is due by Q1 2018. This data will be made available to anyone, whether they’re the individual consumer or the small business that decides to use it. The stated intention is that more data will drive more innovation and more competition, and ultimately deliver better outcomes for customers.

This is a blueprint that we could now see being applied to other sectors, such as the pensions industry, and one which will continue to evolve over the next few years. Giving increasing control to the consumer is something that organisations of all shapes and size will have to adjust to.

Freer data

The freeing up of data has meant it’s no longer something being discussed solely in boardrooms and technical laboratories. Every day, people get out of bed and look at the data collected on their sleep patterns, investigate what they are spending money on through apps, or look at the possession and running stats from their favourite sports teams.

Data is now everywhere in our society, which means that the general population is becoming increasingly switchedon to how it is used and how to use it. This isn’t to say that people are going to suddenly become data scientists overnight, but it means that the general discourse around data can become more sophisticated as the base level of understanding across the population increases. However, when discussing these matters it’s crucial that we avoid going too far and veering towards unfounded prediction and conjecture, so that informed and open discussions about using data can take place and we can move forwards with confidence.

What companies should focus on

• Frequently review how key competitors are working with or incorporating Open Data
• Monitor regulation and legislation that relates to Open Data standards.
• Ensure that data strategy is regularly reviewed to incorporate any new developments or changes

New data

Data sits at the heart of the digital revolution. The huge volume of digital data being generated by organisations and individuals has played a pivotal role in the exponential growth in the amount of data as a whole … And that’s only going to increase.

New data coming from sources like social media, connected cities, cars (90% of cars will be connected by 2020) and wearables has the potential to make profound and beneficial changes to the way we all live. These are transformative times and we should all start to look forward to the opportunities that lie ahead and think big.

Imagine a future scenario where your NHS health data is connected to your wearable technology, so that your doctor can monitor your health in realtime. That’s the kind of data–driven change that could not only improve your life, but potentially save it.

Some key themes have emerged in the new digital world that many businesses are still trying to come to terms with.

Data is driven by digital

Today, ‘digital’ and huge amounts of data exist in a symbiotic relationship. The smart use of data is already improving customer experience. Consider online commerce where digitalisation has changed everything. Personalisation of digital experiences is no longer an extra, but an expectation. If a customer has visited your site once, they expect you to recognise them on subsequent visits and they expect a tailored experience from businesses that have their data. Online retailers have all responded to this expectation.

But, we are only at the beginning of this change, as the smarter use of more data has the potential to transform customers’ experiences far more. It is data that is at the heart of these user experiences. It is the fuel that in an increasingly cluttered digital world, delivers a superior online user experience - across advertising impressions, published news and content, e-commerce, banking and financial platforms and other touchpoints.

In order to provide a personalised experience, businesses must leverage data from multiple sources, and apply that data to serve customers with relevant content, product recommendations and services based on intent, location and interests

The Internet of Things

As the Internet of Things develops, digital data will allow brands to collect permission-based, real-time data about how, when, and where consumers use their products, creating data ecosystems.

Think of going to a shopping centre and being directed via your watch to the stores with new items in stock that atch your personal tastes. Today the growth of almost all organisations is tied to their ability to operate successfully in the new digital landscape. So, where would this ‘Internet of Things’ be without data?

A series of objects, connected to each other across a network, is not automatically ‘intelligent’. The interconnection is merely the starting point for building an intelligent system, and predictive analytics harnesses data to provide that intelligence. By analysing data flowing across the network and applying machine learning, raw data can be converted into clearly defined outcomes that will allow the network to learn how to govern itself over time. Predictive analytics therefore uses data to enable Artificial Intelligence (AI), as the network will gradually become self-aware, self-regulating, and potentially self sufficient.

As products themselves create data in an Internet of Things world, brands will be able to understand how a product or service is actually being used. They’ll then be able to deliver a better service experience, using the product as an interface for data-driven support. Faults will be detected and even remedied before the customer notices them, and product upgrades will be automatically delivered.

There are some cautionary notes, however, associated with this digital data deluge. We need to be careful that the ability to track a consumer’s movements in the digital landscape doesn’t make them feel ‘creeped out’ or uncomfortable. And we still need to focus on data quality

Customer centricity enabled by data

For most organisations, much of the value of collating and analysing data lies in acquiring a more sophisticated understanding of their customers – developing a 360˚ view of the people they are trying to provide products and services for. This is key to improving and perfecting the service and customer experience, but it’s also becoming increasingly complex as new digital channels emerge. First-party data – provided by the customers themselves – has been the primary contributor to the single customer view, as consumers share their data in the expectation of a better experience, but now businesses must look to second- and third- party data as well.

Second-party data

The newest form of data, second-party data, comes from a partner who’s given permission for you to use it. You still require consumer consent, so this is – in effect – first-party data from another source, as it’s collected in the same manner but then shared with another organisation. A major benefit is that it allows you to reach users who show they’re a good match for your product or service, but aren’t already customers.

Second-party data is more scalable too, as organisations begin to form strategic partnerships and affiliations to share information. The idea of sharing the load like this can alleviate some of the concerns around gathering large volumes of first-party data, making it a growing area for the future. In the digital advertising sector, partnerships like this are particularly popular, as they allow advertisers to identify and reach audiences with similar qualities to existing customers.

Third-party data

Aggregated from multiple sources via indirect channels, third-party data is usually drawn on to create generalised segments for targeting. Data companies aggregate customer consented information from multiple websites, showing interests in particular topics based on behaviour, before categorising people into defined groups, such as ‘sports enthusiasts’. It can also include information such as smartphone location or purchasing transactions.

Bringing it all together

First-party data’s power is augmented when it’s enriched by the other two categories. High performers take advantage of all three types, as this approach moves them closer to the ‘Holy Grail’ of a truly 360˚holistic view of the consumer, that benefits business and customer alike. According to Experian research, 97% of organisations are looking to achieve a single view of their customers.

Data must paint a full and nuanced picture of individual customers. Achieving this means drawing on multiple data sources, creating a rich blend of insight that cuts to the heart of the customer’s needs and wants. As far as possible, organisations should therefore be looking at all the data they can.

Collecting and interpreting a brand’s first-party data is the first and most important step that organisations can take to achieving a customer-first approach. This makes it a top priority for any successful business strategy, as study after study has shown that there’s no better predictor of future customer actions than past customer actions.

Data quality must also be a central part of any business’ planning around data, as poor data means poor targeting, poor efficiency, poor results, and wasted money – potentially even reputational damage. Misuse of consumers’ personal data will erode trust and belief in the value that brands and media owners provide. Businesses have a collective responsibility not to undermine its power. If used responsibly, and effectively, data stands to enrich the digital experience for brands and individuals alike.

What companies should focus on

• Maintain a laser focus on creating a ‘single customer view’ – make it a strategic priority for the business.
• Explore emerging data types and formats.
• Focus on data quality, rather than simply volume.

More analytics – speed and agility through prediction

Data only becomes useful if it provides relevant and timely insights, and analysis is what extracts insights from data. These insights are only useful if they generate payback for the business or the consumer, preferably both.

All told, this means that when people talk about data – ‘Big’ or ‘Small’ – they should always be talking about data analytics as well. Once large datasets are analysed, trends can be revealed, correlations can be determined, and businesses can run more efficiently.

The 6 V’s of data are also driving an increasingly pressing need for smarter and more automated analytics, more commonly referred to as ‘advanced analytics’.

A new analytics market is evolving, with specific characteristics, including:

• Fast, automated analytics that informs operational decision-making in real-time, or near real-time
• New predictive methodologies, such as machine learning, which drive recommendations that can keep pace with market behaviour
• Analysis of huge data sets, from a variety of sources, through self-service tools
• • Findings delivered in an easily visualised format, which are consumed through a single inter-connected technology platform

Prediction, not reaction

The ‘science’ of predictive analytics is at the forefront of making sense of large data sets. It provides a set of techniques that deal with extreme complexity, at speed and with agility, extracting meaningful information from present as well as historical data sets. It spots patterns in data and predicts unknowns, future outcomes, and trends.

Traditional analytics insights and descriptions (what happened and why?) are moving towards predictive and prescriptive applications (what will happen and how can we make it happen?). This means far more tangible return on investment for organisations.

Cognitive analytics

Focusing on one branch of predictive analytics, the advent of cognitive analytics means we can now automate analytical thinking and self-learning through machine learning. Learnings are fed back into the analytics ecosystem to be applied in future situations, to answer new or related questions. Every time the mechanism becomes smarter.

Needless to say, analytics at this level has a vast range of applications. Fraud agencies will be able to develop risk engines that flag candidates for further examination. Retailers will be able to optimise decision processes such as monitoring and addressing inventory levels, or adjusting in-store pricing in real-time for flash sales.

Cognitive analytics will also make other analytics better by analysing huge data sets in their entirety, rather than the far smaller samples which users can cope with.

Learning algorithms are increasingly anticipating what we want, delivering recommendations and offers to the consumer based on what they’re about to do, and not what they’ve previously done. This anticipatory analytics is enabled by access to vast new stores of data, from customer behaviours and external sources, which enable organisations to deploy ‘situational selling’ based on the specific individual and their situation at that moment in time.

Some of the world’s top companies continue to invest heavily in Big Data, machine learning, and artificial intelligence capabilities. Over the next few years, we can expect to see significant advances in machine learning, reducing the time to insight and manual data wrangling required. This will then free up significant chunks of time for employees to focus on driving efficiencies and work on directly revenue-generating activities.

Data visualisation

Advanced analytics may be powerful and impressive, but the data and findings must be presented in an accessible, readable format which makes them clear. Developments in data visualisation technology now allow users to view and manipulate hundreds of variables and parameters, and the impact these are having has led some commentators to suggest that ‘Visualisation’ should be included as another of the data V’s.

New opportunities

The powerful combination of behavioural data and predictive analysis is opening up new areas of opportunity for analytics businesses:

• • Dynamic pricing – from ride-sharing services to sporting events, this real-time situational pricing capability allows companies to balance supply and demand. Research has shown that consumers are becoming more accepting of these practices, as long as the variation remains reasonable.
• • Pre-emptive safety – the ability to predict events such as fires or failing machine parts, or even priority lists for food hygiene in restaurants
• Targeted intervention – the ability to predict drop-out students and identify which at-risk students require extra support. Rehabilitation assignment decisions on an individual offender basis, drawing on predictions of future repeat offenses.

In addition, complex event processing and streaming analytics software can detect poor user experiences, enabling brands to offer proactive customer service. A customer who presses the start button of a new device five times in a row would indicate they’re having trouble operating the device, and could be offered real-time support or an instructional video via their mobile device.

While this may all sound like the human element of analytics will be circumvented, this won’t be the case at all. Models must be built and tested, designed to accommodate incomplete information, and developed to adapt to mistakes and failures. Actual and expected performance must be monitored and compared, and any disparity addressed.

Above all, transparency of advanced analytics will still be demanded for the foreseeable future so that organisations can track what is happening and why.

With people in every area of business often struggling with data and analytics, the opportunity for analytics companies is huge. As data sets grow, we’ll see predictive algorithms used more and more as a methodology for mining data where traditional techniques can’t be used. Predictive analytics will continue to evolve from an innovative extra capability to must-have functionality. We’ll also experience the integration of advanced analytics, predictive analytics and machine learning.

Future developments will see analytics simplified and outsourced. New automation and visualisation software is already breaking down barriers between the datainitiated and uninitiated, and through a continuation of this trend we’re going to see analysis of data sets becoming considerably simpler. Drag and drop analysis has already arrived on tablets, allowing almost anyone to conduct an increasingly sophisticated level of analytics.

What companies should focus on

• Ensure the company as a whole takes a ‘datadriven’ approach, leaning on insights from analytics to guide decision making.
• Make sure that analytics is employed in all departments in the business, not just those where it’s already playing a role.
• Provide training for any staff who are not comfortable with the fundamentals of analysing data.
• Examine the day-to-day operations of the company for opportunities where real-time analytics or predictive analytics could play a role.

New identities and linkage

The proliferation of data, and the variety of channels from which it comes, present a growing challenge in consistently identifying individuals across channels. How do organisations establish and track this identity?

We’ve already seen in the discussion of data variety that a key characteristic of Big Data is that it comes from numerous different silos and channels, both offline and online. In pre-digital days, when we only had consumer data records that related to a physical consumer address or postcode, it was simpler to bring separate data elements about the same consumer together.

Today, however, linking data which an individual generates across their offline and online journey is challenging. The proliferation of channels has had a profound effect on the range and volume of data available, as well as hampering the ability of organisations to connect with their customers.

There are arguably more opportunities for engagement, but meaningful connection and consistent personalised experiences have become far more difficult, and the challenge is only growing.

Link for success

‘Linkage’ is growing to become a fundamental idea. This concept involves identifying users by their device or internet connection, and linking together their identities across each interaction in order to create a single view of the customer and understand how to engage with them across a variety of channels, both digital and ‘traditional’.

This enables us to derive a holistic view across multiple data sources and, importantly, to enrich customer records with additional data. As customer engagement activity shifts to the digital realm, and digital channels continue to grow and diversify, linkage data will become a highly valued asset.

The 360 Customer View

In a digital world, a truly unified, cross-channel view of customers is something that no modern business can do without, although many still struggle to achieve it. As people move from social networks to websites, physical stores, and traditional media, they expect brands to keep up. They need to be treated consistently, regardless of the channel they’re using, and the challenge is for businesses to really understand what they want and who they are at all stages of the customer journey

This has various implications for businesses, but fundamentally it means that businesses must be able to recognise a specific customer, wherever they are and no matter what device they’re using. A single customer view means achieving a deeper understanding of a consumer’s life, throughout their customer journey, and regardless of where they are interacting and transacting. Harnessing the power of data is the key to achieving this.

From there, they must then be able to adapt their marketing materials to factor in previous encounters and preferences, ensuring that the customer’s experience is seamless and tailored. And all of this must be done flexibly and at scale.

Perhaps unsurprisingly, 81% of businesses still report challenges in achieving a single customer view. It seems that technology is the main obstacle to achieving this, from the need to work in real-time, to using data from multiple sources, and accessing data that sits elsewhere within the business. This represents everything from storage, through data blending, to accessibility and short and long-term storage.

Looking at the business as a whole, linkage and the single customer view will address a growing market need across all sectors, and all business divisions. Change won’t be rapid but it will come, and when it does then it will completely transform the relationship between business and customer.

What companies should focus on

• Support clear and open communication between any departments which interact with customers, from sales to marketing to finance.
• Ensure there is a central database of customer interaction histories and account data, which all departments can access, draw on, and update.
• Review data management practices on a regular basis, to ensure opportunities for greater efficiency aren’t missed.

Technology - making sense of the data explosion

Unlocking value in Big Data goes hand in glove with the technology developments which allow us to more efficiently store, manage, and process it. What are the major developments and why do they matter?

If data isn’t new, what’s held us back from better harnessing it in the past? The answer to this lies largely in technology and its ability to ingest, process, and make sense of the volume and variety of data that consumers and businesses create. Our ability to unlock the value in data has therefore progressed alongside with technological developments, their affordability and their accessibility.

Data proliferation has hastened and accentuated the demand for technologies that can create value, particularly from unstructured data, which constitutes 80% of all data. These constantly improving technologies are also helping to put structured data into context by mapping it to the vast amount of unstructured data.

The fundamental technologies which help us to examine and interpret raw information in increasingly sophisticated ways fall into three categories – storage, computing, and analytical software. Each of these are deployed by organisations in order to realise the value in the growing torrent of data at their disposal.

Distributed computing

Big Data has driven significant developments in storage and processing, all of which have been well documented. In the last few years we’ve seen surging adoption of open solutions which enable distributed storage and processing. These solutions allow organisations to work with very large data sets via computer clusters built from commodity hardware.

The success of distributed computing is clear to see, and its power is simple to understand, as organisations unite and co-ordinate the power of a network of computers in the cloud for their compute needs. Search engines, for example, use hundreds of computers, all working in concert, to deliver answers to search queries every day – all in a fraction of a second.

Cloud computing

The options for ‘The Cloud’ seem to increase almost daily, as more and more businesses transfer to the cloud and stay there. It’s predicted that by 2020, at least a third of all data will pass through the cloud.

Cloud-centred data processing and analysis has drastically improved the cost efficacy of these capabilities, allowing organisations without the budget for traditional Big Data analytics to embrace the idea of better managing their structured and unstructured data sources. In short, The Cloud continues to offer organisations a wider range of services, at more affordable prices than ever before.

Data Management Platforms

The explosion in first, second, and third-party data has given rise to a surge in popularity for data management platforms “(DMPs)”, particularly in the marketing department. As organisations struggle with the challenge of simply managing disparate and varied data, let alone analysing it, a data management platform enables consolidation into a centralised platform.

These platforms can unite both online and offline data. This could include – for example – first-party data from your email system and web interactions, second-party data from a partner, and third-party data from a social media platform’s demographic data sets. This can then be combined with personal, channel, and device type information from user IDs for a more complete customer view. Once it’s all in one place, it can then be activated across all of your marketing and advertising systems.

Consolidating data from various user channels means ‘DMPs’ can help publishers gain more precise audience information. While this can translate into selling more tailored, effective advertising, it also delivers greater insight into the actual needs and interests of users.

As a result, technological development has been driving towards real-time analysis of huge volumes of data. Various current solutions include using in-memory databases to speed up analytic processing; real-time data access; decisionmaking and optimization engines; analytics software platforms; and data visualisation packages. In fact, progress has been so marked in recent years that we are looking at a future where analytics software is able to incorporate predictive capabilities into a usable package for non-specialists. Data is being democratised, and the combination of accessible analytical tools and their substantial predictive power will speed the change.

The focus for businesses must be to commit investment to the infrastructure needed to support growing volumes of data. This means continuing hardware improvements, finding software innovations, bolstering security policies, and combining them all to enhance the customer experience.

What companies should focus on

• Review the core infrastructure currently in place for handling data, and schedule regular reviews to evaluate whether any changes or upgrades are needed for managing the exponentially increasing volumes of data.
• Monitor developments in cloud computing and distributed computing on an ongoing basis, in order to identify possible efficiencies and improvements.
• Consider setting up an internal working group, comprised of different department heads, to jointly review new technological propositions and how they could improve the business.

Faster distribution

With data velocity increasing all the time, our data also becomes more perishable. Aside from making analytics software faster, how are businesses taking on the challenge of harnessing data effectively?

Whilst faster data can be much more valuable than slower, ‘after-the-fact’ data, the core challenge is how to create rapid insights from it and then act on them. This means rapid analysis is critical, but this comes after another fundamental step – ensuring rapid distribution.

Without speedy distribution of data, fast insights and actions aren’t possible, and this is a challenge for organisations of all sizes.

Real-time vs batch processing

Historically organisations and their computing systems have been set up to work over cycles in days, weeks and even months. The world works entirely differently today, and things need to work in an instant. Take online banking as an example. We can now view our statements via our mobile phones or make instant payments with a few clicks. These are things that would have historically taken several days.

In the financial markets things move even faster, sometimes thousands of transactions are taking place in a fraction of a second, and all this digital, data–driven innovation and speed is changing our expectations about how things should work. This is a real challenge presented for all large and well-established organisations, which need to move their infrastructure to a point which can meet and exceed those consumer expectations.

The API economy

To meet this challenge, we have seen more mainstream growth of interconnected Application Programming Interfaces (APIs). These are tools and technical protocols that allow third-party software programs to interact more easily with an application – vital in a digital world with an ever-growing variety of technical solutions and software vendors.

For example, the numerous APIs for the major online video services allow web developers to integrate videos directly into websites, amongst other things. Taking a more general view, these innovations are designed to allow organisations to more easily and quickly access the data and processes of a piece of software, by making communication smoother.

APIs have already started to change how companies share information and reach new customers, and consequently they’re being positioned as a key driver for the digital economy. As a result, there’s a growing pool of businesses devoting time to this emerging area.

Speedy data delivers fast returns

APIs make data more accessible across a given ecosystem, which then allows many organisations to open up new ways of generating revenue. Even governments are making concerted efforts to promote APIs, such as requiring public sector agencies to make their data available in open, machine-readable APIs.

Rather than re-inventing the wheel, API innovators are honing and perfecting existing services in order to create an exceptional user experience. This in turn allows smaller companies to disrupt large competitors with minimal investment in resources. By spending only a few thousand pounds, start-ups can now create a product that, from the customer’s perspective, is indistinguishable from a multi-million dollar project that took years to build. You no longer have to be a software giant to make a sizeable impact, thanks to the API economy.

By using APIs, companies can go to market quickly, and by providing APIs and establishing an ecosystem they can scale up to be competitive.

Blockchain

Used to underpin the cryptocurrency Bitcoin, blockchain technology effectively creates a distributed database that maintains a continuously-growing list of data records across a global network. Each new block contains a certain amount of data, and refers back to previous items in the chain, giving the blockchain a linear, chronological order. Once a given block has been completed, it becomes a permanent database and a new block is automatically generated, but blocks can only be updated if the majority of participants in the network consent.

Crucially, because data is distributed between many parties, there’s no need for a central server. Advocates therefore argue this provides greater security – as the ‘digital ledger’ is shared between many thousands of terminals, making it far harder to tamper with – and improved resilience, as the chain isn’t reliant on any one node in particular.

The European Securities and Markets Authority recently launched an independent consultation team to explore blockchain technology, which predicted that using distributed ledgers would improve the verification of transactions.

Currently, while identity is being debated widely, innovations for blockchain technology remain focused in the financial sector, with multiple organisations envisioning a key role for blockchains in areas from remittances to securities and exchanges.

In the business world, competitive advantage relies on fast insights. Fast insights rely on fast data. Data speed depends on the distribution technology that carries it. In a digital environment where speed is everything, and microseconds matter, APIs and blockchain technology have the potential to distribute data at the pace needed for the next generation of businesses and innovators. Organisations need to challenge themselves by looking beyond their immediate needs for new solutions and opportunities to improve.

What companies should focus on

• Review your legacy data distribution technologies now to identify any which urgently need to be replaced.
• Monitor new innovations in the API space and experiment to see which have the greatest effect on data distribution speed.
• Agree and safeguard a budget for regularly refreshing your technologies and systems.

Regulation and governance in a data rich world

While data has exploded, a move towards a more appropriate regulatory landscape has emerged alongside it. On the one hand this regulatory drive aims to ensure greater protection for individuals and to impose greater restrictions on organisations around the use of data. On the other hand, regulation is also driving free movement of data, in order to support greater competition in markets deemed to be failing the consumer.

New regulatory initiatives about data are not just being adopted in Europe, but also across Asia, South America and North America, helping both individuals and businesses alike. While the regulatory landscape may be complex, the central idea of keeping consumer interests front of mind at all times mirrors sound fundamental advice for all companies. Customer-first business practices remain the way forward.

UK regulation - current key issues

There are several upcoming developments of which all organisations should be aware. Many initiatives are already impacting UK business – or will do in the near future – but the most prominent regulatory and governance developments currently on the table relate mainly to consumer privacy, data protection and data portability

• EU General Data Protection Regulation – legally enforceable in May 2018, the changes resulting from this major reform of data protection legislation across the EU include higher levels of control around consumer consent, increased accountability and transparency requirements, and significantly higher fines and rights of redress. This will ultimately mean greater responsibility for all data-holding companies.
• Competition Markets Authority (CMA) Retail Banking Investigation – requires the sharing of banking transactional data through an Open API (and always with the consumer’s consent) as a primary remedy for perceived lack of competition in the retail banking market.
• • EU Payment Services Directive 2 (PSD2) – due to be enacted into English law by early 2018, PSD2 will provide consumers with the ability to access their payment data and transmit it to certain third parties. This will allow them to aggregate their account information and make payments from their accounts from outside of their online banking.
• • Digital Economy Bill – an important piece of proposed UK legislation designed to help consolidate and preserve the UK’s position as a world leader in the digital economy.
• EU Digital Single Market – a framework policy initiative launched by the European Commission in 2015, designed to improve access to digital goods and services, create the right environment for digital networks and services to flourish, and promote digital as a driver for growth.

Of course, political influences on the regulatory landscape will have a bearing on exactly how some of this regulation emerges and how businesses will need to respond, with ‘Brexit’ in particular introducing an additional layer of uncertainty in relation to EU legislation. However, the overall direction is clear and amongst this uncertainty lies opportunity for smart brands. Those businesses which can anticipate the inevitable changes, and act voluntarily before regulation is enacted – such as improving transparency around how they use consumer data – stand to benefit from a significant ‘halo effect’.

The underlying factors

Navigating regulation may well be a significant challenge, but appreciating the underlying drivers of new and existing regulations will help organisations to understand the rationale behind most developments, and perhaps even anticipate them. So what are the market-based and societal changes that are driving regulatory evolution?

1. Data Security

Perhaps the most prominent issue on the regulatory agenda – how do we protect competitively sensitive data, or any other data that should be kept private? High profile data breaches have shown the world that security issues can expose confidential information, making this a high profile focus for technological innovation and policy

2. Multiple use cases for the same data

Data privacy used to be simple. You received people’s data, then once you had used it, you deleted it. Recently this has been less clear cut, as the use of data is likely to go well beyond the initial use case. For instance, using historical data to predict future outcomes is the entire basis of predictive analytics. However, this requires the data to essentially lie dormant for an indeterminate amount of time, which may well clash with future guidelines and laws. These rules will therefore have a significant impact on the way that companies and governments gather, store, and use data.

3. The power of one!

Data privacy issues are now prominent enough that we actually have private individuals bringing legal test cases against some of the biggest global data organisations, focused on the use of their personal data. What’s more, some high profile rulings have been made in favour of these individuals.

Whilst data ‘knows no boundaries’, major international organisations are now setting up physical data hubs in Europe to comply with national data regulation laws. As a result, pressure is being applied to others to follow suit.

4. Digital content challenges

Large enterprises engaging in digital business models and activities are increasingly hiring a Digital Risk Officer or an equivalent. This is the result of ongoing grappling with enormous banks of digital content, which will in all likelihood be spread across a number of regions and multiple borders. Regulatory and legislative bodies will expect organisations to recognise the importance and vulnerability of their digital assets, but the reality is that digital risk management is still in its infancy.

5. Automation and algorithm

As more and more data becomes available, and technologies become increasingly sophisticated, the trend towards automation of decision making and algorithmic “learning” will continue. This trend is already bringing up a range of new challenges, as concerns surface around the consequences of automating results generation. By their very nature, algorithms could amplify inadvertent or historical biases on the part of businesses, as well as making these biases harder to identify and rectify.

6. The Internet of Things

The Internet of Things involves an intricate mesh of people, devices, systems, and network connections, as well as different data storage and transport locations. To ensure privacy and increase consumer trust, it’s essential that each discrete part of the system is only able to access, manage, or share data that it’s allowed to.

Multi-level security and privacy controls are required, which means policies built into the very architecture of any platform. Picturing a basic consumer example, if you visit someone’s house, you shouldn’t be able to detect their devices on your smartphone. Permissions or secure keys must be given

7. Data minimisation

As businesses and other organisations store increasing volumes of data, questions have arisen about the actual purpose of these ‘data lakes’. Companies seemingly waste millions on the storage of what appears to be useless, redundant data. According to Veritas’s Databerg Report, released in October 2015, 54% of data retained by companies is unidentified. Of the remaining 46% – data which has been tagged and classified – a third (32%) is redundant, obsolete, or trivial. This means that only 14% of corporate data could currently be classified as business-critical.,

Apparently 76% of the UK public sector’s data is not yet identified – this level of ‘dark data’ is the highest proportion in Europe. We’re beginning to see governance officers talk about ‘data minimisation by design’, whereby organisations only collect data that’s necessary for the business to operate. This is particularly true of those businesses which harvest large volumes of customer data, where a ‘just in case’ mentality prevails when it comes to deciding which data to collect and store. As the amount of regulation increases, this will be an interesting development to monitor

8. Where does the IP in data reside?

The growing economic importance of data raises a number of legal questions, particularly when coupled with the fact that data differs from other assets. A unit of data can be copied perfectly and combined with other data, or used simultaneously by more than one person. These characteristics make data unique when compared with physical assets, and raise questions of intellectual property rights. Who ‘owns’ a piece of data? What rights come attached with a data set? What defines ‘fair use’ of data? There are questions of liability too, such as who is responsible when inaccurate data leads to negative consequences? Each of these legal issues and many more, need clarification over the coming months and years, in order to capture the full potential of data

Ethical data code

As the focus moves away from technology to governance, security, and infrastructure integration, businesses will face tough decisions. Whether its new deployments, investment choices, or transparency around data and analytics, the path won’t be easy to navigate. The trustworthiness of the way analytics algorithms handle and manipulate data, and thereby derive insight from it, will remain an issue for the majority of organisations.

The good news is that ethical guidelines are beginning to emerge, helping business analytics professionals to do the right thing and avoid unintended consequences. Clear laws surrounding data privacy and collection will be critical, but also data storage, distribution, and security, as these will be key to the overall future success of data use in business.

What companies should focus on

• Review upcoming changes to legislation and regulation, and ensure the business is compliant BEFORE they come into effect.
• Ensure clear governance and compliance models are set up within the company, and that department heads and managers fully understand the expectations surrounding them.
• Where possible, take a visible leadership position on regulatory changes.

Data driving new skill requirements

Current data trends are not only changing how organisations operate on a day-to-day basis, they’re also affecting the kind of people who are most in demand. A new collection of skills and business roles are needed to manage and unlock data, as well as minimise any risks.

Today’s organisations must look beyond the latest technologies when it comes to data. They must pay attention to a variety of new and developing skills, working out how these can be combined with technology to deliver the insights and decisions they need. This expanding skillset can’t be combined in only one person, so this means we’ll soon see a growing number of crossfunctional teams, across both business and technology groups.

In amongst this upheaval there are particular skills resourcing gaps, and these are where organisations will need to focus their attention.

Data scientists

The era of data has created a talent gap for people who can pull usable insight out of raw data. Several commentators have pointed to data scientists as a key force in influencing modern business practices, holding them up as ‘the new superheroes’.

That said, news about the data science talent gap persists, and it seems that gap isn’t closing as rapidly as many would like. The core issue isn’t finding technically capable people, as many analysts have coding skills. Instead, the difficulty lies in finding people who can code while keeping an analytical perspective in mind. They need to ensure the results they discover and produce are not only reliable but also relevant to the specific commercial aims they’ve been set. This is where the gap is. There simply aren’t enough people with the skills to analyse and interpret data, transforming it into useable, focused, business-relevant insight – the ultimate aim of any data-driven initiative.

In-house training

Instead of focusing solely on new talent, numerous organisations are growing their data science capabilities through developing internal talent. Many companies, of various sizes, make their first steps towards embracing data science through training up current employees and helping them to experiment with cloud-based tools.

If this isn’t possible, and you aren’t able to hire permanent team members, then there are a growing number of data science agencies that can provide the support that businesses need. However, the skillsets of most agencies focus on the pure data science element, rather than the commercial awareness we’ve already identified as crucial. To accommodate these data science secondments, organisations must already have the right infrastructure, business organisation, and supervisory skills in place, so a certain amount of in-house training will be inevitable.,

Open source analytical and data packages

Off-the-shelf data analytics algorithms and visualisation capabilities have exploded, thanks to open source. There is a possibility that in time we may not need data scientists to produce any complex coding for analytics or visualisations at all. Instead, the latest generation of packages and solutions will allow organisations to conduct analytics and data visualisation without the need for technical specialists. If this transpires, and the complex technical elements are taken care of, then there could also be a renewed emphasis on solving business problems.

Data-driven leaders and the Chief Data Officer

With people throughout organisations waking up to the strategic value of data, the skills gap amongst business managers is becoming more and more obvious. Management – and senior management – almost invariably aren’t equipped to translate results from data science teams into meaningful business implications. Moreover, they need to develop a broad understanding of the data-driven world overall.

We’ve seen the rise of the Chief Information Officer within the boardroom, but businesses need to do far more. There’s a worrying lack of capability at a senior level when it comes to seeing and understanding the benefits and risks of developing a data strategy, whether for driving results for the business or value for customers, or both.

What has become apparent is the need for a Chief Data Officer to support the work of the CIO. We can add Chief Digital Officer, alongside Chief Data Officers and Director of Insights, as emerging new roles which have come about in response to the pressure and opportunity presented by Big Data.

Organisations without strong data leadership are going to fall significantly behind their competitors in all areas, including their ability to win, serve, and retain customers. Intuition has little place in understanding the customer and the customer experience any more. There’s more than enough data to tell you everything you need to know.

As with data scientists, these data professionals are in high demand. Experian’s 2015 report into the ‘Dawn of the Chief Data Officer’ found that around 90% of businesses feel data is transforming the way they do business.

Data Protection Officers, security experts, data management, and legal skills

Any data strategy must include an appropriate roadmap to meet the criteria of meeting the challenges presented by data regulation, legislation, and also security issues. As a result, many data-related roles are emerging to address various skills gaps in these closely related areas.

In many organisations, data governance functions still aren’t able to cope with growing data requirements and increased regulation, let alone the legal and IP issues associated with data. What is required is data leaders, with the right skills for the job. Supported by the Government, companies need to work quickly to fill these skills gaps in time to address increasing levels of EU and domestic legislation. This isn’t a quick issue to fix.

All senior business leaders will be expected to understand and articulate a company’s data strategy, as well as their position on data security, privacy and ethical use of data. Articulating each of these to shareholders and customers will require a particular blend of expertise. In the short term, working with data specialists – both individuals and consultancies – will be vital. In the long term, all companies should be looking to up-skill and upgrade their levels of data expertise.

What companies should focus on

• Ensure all employees are familiar with basic data analysis and working with data on a daily basis.
• Demonstrate a data-driven approach to decision making at the management level, in order to encourage this way of thinking throughout the organisation.
• Encourage and reward those who drive a culture of data within the organisation.

Conclusion

Across each of the trends we’ve highlighted, numerous possibilities for businesses exist. From improving operational efficiency to building a more trusted brand, data brings challenges but also unlocks enormous opportunities.

We’re living in the midst of a data and analytics revolution, driven by the six V’s of data – volume, variety, veracity, velocity, vulnerability and value. Previous assumptions are being challenged and ‘data’ is set to enjoy a highprofile position on the agendas of governments, NGOs, regulators and businesses for many years to come.

Against this burgeoning backdrop, consumer attitudes towards personal data and how it’s used are evolving, as the public becomes more engaged in the debate around data use and the many benefits it can bring.

Every organisation must remember that it doesn’t have carte blanche to use data as it sees fit. Consumer privacy, data protection and data security must be considered at all times. Savvy businesses will take a clear, proactive stance on these issues, ensuring that they permeate every department and process.

Getting these factors right as soon as possible, rather than waiting for direction from regulators or industry bodies, will allow organisations to drive value for everyone and build that all-important trust. The world of data has always been one of constant change, but trust must be the constant that sits at the heart of every approach.