The future of big data and analytics, by Jacobs Edo



In this age of digital transformation, we are nearly an inflexion point as emerging technologies especially in big data analytics, machine learning, artificial intelligence, etc. are converging, making possible the idea of actionable insights in support of better strategic decision making and possibly increasing our lifetime here on earth(Edo, 2016). Data is becoming a priced commodity, and its significance is on a gradual rise. In the words of Minelli et al., big data now represents a transition-in-kind for both storage and analysis; it is not only about size but also about insights. (Ambiga Dhiraj, 2012). As digital disruption transforms lives, communication and interaction channels and as digitally enabled businesses to renovate their customer experiences across multiple devices, social media platform and the natural face-to-face interactions—multistructured data will continue to evolve.

Interestingly, the power of data to improve healthcare and living conditions of city residents were first demonstrated in combating cholera epidemics in London, during the middle of the 19th century. While the society and the medical profession were still unsure on the true nature of epidemic, transmission and causative agents, Dr John Snow was not just busy collecting associated patients’ data about the dying residents but also analysed the resultant data, where he discovered with amazement the direct correlation between higher risk buildings and their connection to contaminated water supply lines. With this early discovery, the solution to the problem became immediately apparent: better pipe insulations, better handling and treatment of wastewaters. And thus, future epidemics were averted in London(Richard Barkham, 2018).


Historically, data was something one owned and was generally seen as structured and human-generated. However, technology trends over the past decade have broadened the definition, which now includes data that is unstructured or conventional and machine-generated, as well as data that resides outside of organisational boundaries and control(Young, 2015). Big data is a collection of data from traditional and digital sources inside and outside the organisation that represents a source for ongoing discovery and analysis(Arthur, 2015). Compared to conventional data, Vassakis et al. characterised big data by seven ‘V’s – volume, variety, veracity, velocity, variability, visualisation and value(Vassakis, Petrakis, & Kopanakis, 2018). Where big data volume refers to its sheer size, variety represents increasing diversity, variability often mistaken for variety relates with rapid contextual changes, velocity represents the rate of generation, and veracity connotes data reliability and accuracy. Visualisation deals with the quantitative and qualitative schematic representation of data indicating patterns, trends, anomalies, constancy, variation, in ways that cannot be presented in other forms like text and tables(Friendly, 2008).

While enough has been said about the importance of big data and its associated analytics but the challenge of industry focus continues to per exist. This challenge is hitherto associated with the notion of “collect all available data and sort it later” mentality which have created huge data lakes across numerous enterprises as data continues to grow year-on-year. Many of this organisations or institutions are now sinking in their data as their ability to swim is being challenged every day. It is now increasingly clear to many observers, businesses and IT leaders that a change in mindset as it relates to digital transformation within the enterprise is needed to guarantee a broad focus on both operational and transformative outcomes to uncover the real value proposition of big data analytics. To engender the necessary mindset shift, a two-prong approach is required for assessing the real value of big data analytics. The first involves a business process optimisation approach –finding a clear path to automation through gradual drawdown on all manual process steps while the second approach involves those focusing on innovation – creating new business models, which is substantially more transformative and data-intensive (Araujo, 2018). Raw data is valueless unless used in context, this lack of context or clear action-oriented business focus remains the primary challenge of the existing traditional data-first focus approach to big data analytics.

As old habits die hard, this is a similar situation with the challenge with moving away from historical, retrospective analysis of value proposition to big data analytics centred on action driven by insights. However, this same paradigm change will completely transform the game and raises the stake as more real-time data-driven action becomes ubiquitous within the enterprise –the higher the risks and the higher the rewards. As this data-driven-action world evolves, the veracity and value proposition needed for a real-time decision or to execute an action becomes a strategic imperative. So the future cognitive system will not only require data but the requisite action based model necessary to stimulate strategic value-based decisions making(Araujo, 2018).

The strategic value proposition of big data is not only necessary for organisational innovation, but also can lead to changes in business operations that will increase productivity and build enduring competitive advantages. This could be achieved through the use of powerful big data predictive analytics tools. These tools can provide detailed reporting and identify market trends that allow companies to accelerate new business ideas and generate creative thinking. In addition to using big data analytics to answer obvious questions, managers should encourage users to leverage outputs such as reports, alerting, KPIs, and interactive visualisations, to discover new ideas and market opportunities, and assess the feasibility of ideas(Wang, Kung, & Byrd, 2018).

Although big data analytics is considered generally at its infancy, the next 15-20 years promises to be an exciting time with data. Industry pundits are already estimating that by 2020, every individual in the world will be creating at least seven megabytes (7 MB) of data per second. We have today created more data in the past couple of years than in the entire history of humanity (Ahmed, 2017). While you wonder where the big data industry is going in the next year? Here are ten big data analytics predictions that will change the world as we know it today:

Big data analytics is the future of healthcare

Big data in healthcare would not only profit every industry player, allowing for superior care and providing access to low-cost healthcare in the future but would also most importantly benefit the patient by providing the right treatment, based on a sustainable pricing model. It will soon be possible to predict precisely how every segment of the healthcare industry would be impacted by big data, enabling us to fully comprehend how it should encourage desirable behaviours and minimise less desirable behaviours. But it will also require close collaboration and innovation between stakeholders – caregivers, payers, pharmaceutical producers, government and policymakers, and the academia – to re-invent their existing business models and increase the performance of their systems. They must build the technological infrastructure to accommodate and converge the massive volumes of data, which industry analysts estimate will grow far above the 2,314 Exabyte by 2020(catalyst, 2018).

Let’s suppose a graduate student presents with dyspnea and dizziness. The doctor immediately orders blood tests, an echocardiogram, an electrocardiogram, and documents the patient’s historical narratives into an Electronic Health Record (EHR) system. A big data analytic algorithm immediately finds the patient’s prior test results, medical history, genetic profile and demographic characteristics, and links in his wearable biosensor data. The algorithm creates a unique phenotype by processing and analysing all the data sources, compares results with tens of million other patients with similar phenotype within minutes, and then suggest to the doctor that the patient has hypertrophic cardiomyopathy, with over 80% predicted probability of sudden cardiac death in the next ten years. This recommendation supports an accurate and efficient diagnosis and provides individualised risk assessment to inform shared decision making for a potential implantable cardioverter defibrillator – this is the future of healthcare(Shah & Rumsfeld, 2017).

Big data processing moves to the edge

One way to handle the increasingly growing data generation and consumption challenge especially those related to the Internet of Things (IoT) will be a gradual movement of big data processing activity to the network edge. With edge computing, much of the heavy big data lifting will be offloaded to network devices instead of pushing everything to the centre or multi-tenancy cloud platform for processing. For most enterprises, this approach will not only save money but support faster analysis process, allowing decision makers to take action via actionable insights much more than before. While edge computing promises to bring computing power to the edge of the network, it presupposes the enchantment of network edge devices through digitalisation. Edge computing is therefore the necessary ingredient for the actualization of our number 4 prediction – Internet of Everything (IoE).

Data-as-a-Service (DaaS) is the new fuel for business success

Data as a Service (DaaS) is an information provision and distribution model in which data files (including text, images, sounds, and videos) are made available to customers over a network, typically the Internet. Increasingly, organisations are coming to terms with the value proposition of (big) data. This strategic understanding that a company’s data should be its greatest asset will soon become widespread. Technically, the initiative of data as a service involves the consolidation and reorganisation of existing enterprise data in one place, then making it available to serve new and existing digital exploration purposes. Access to such data can be commoditised via commercial subscriptions. DaaS, therefore, becomes a system of innovation, exposing data as an asset, for gaining additional revenue(Guney, 2018). Today, organisations are growing their ability to create, derive and administer high-value data internally to gain more competitive advantages while a few are increasingly gaining financial leverage by repackaging some of these data for the commercial marketplace.

An interesting example is Workday – a cloud-based enterprise relations planning system vendor’s benchmarking data as a service offer. The benchmarking service lets customers directly select data that they wish to contribute to the service while gaining access to benchmarking data on the same metrics across industry or peer groups. The service helps in providing a clear understanding of areas of improvement, resources allocation and opportunities for capitalisation, with supporting industry-leading results. According to IDC, 90% of large enterprises will generate revenue from Data-as-a-Service in 2020. External data monetisation and consumption of third-party data are rising in strategic importance for enterprises across many industries. A plethora of new data providers are emerging to coexist with more traditional vendors in the information industry(IDC, 2018). This trend will accelerate geometrically as better data governance and privacy barrier are lessen.

Internet of Everything (IoE) wins a popularity contest

It is now common place in this age of digital transformation that object enchantment is another name for digitalisation. To make an object, process or anything smart today means adding a digital interface. Thereby transforming a dumb object into an intelligent edge device that can be further leveraged for computing power. We now also live in a world of ubiquitous reliance on data and mobility, coupled with an accelerating need to excess networks, devices, and external data sources; interconnectivity is now the new key for building a cohesive society(Silva, 2018). Artificial intelligence powered essentially by big data is accelerating the potential for digital transformation which will soon create an Internet of Everything (Edo, 2016).

Smart analytics, the future of business intelligence

“By 2021, the number of users of modern BI and analytics platforms that are differentiated by smart data discovery capabilities will grow at twice the rate of those that are not, and will deliver twice the business value(Cindi Howson, 2017).” In the coming years the maturity of smart analytics will be completed, with business intelligence and analytic reporting as we know it today going obsolete. Smart analytics will natively incorporate real-time automated data discovery, meta-data extraction, and artificial intelligence analysis enabled algorithms designed to spot real-time influencers, key drivers, relationship, and exception in data. With the continuous evolution of artificial intelligence, it won’t be long before natural language query processing, and contextual data triangulation evolves into the smart analytics domain commonplace in business analytics.

The age of algorithms

As the rise and performance of data-driven leadership and corporate strategy formulation increases, businesses and institutions will be challenged to acquire the most productivity and performance enhancing algorithms instead of software. The expectation will be for companies to continuously iterate their go-to-market strategies though own data and algorithmic enabled options customisation. Necessarily, an algorithmic platform will have the ultimate goal of replacing objective judgements by objective measurement and known fact through data. “For example, a recruiting software as a service (SaaS) platform will learn that if X million companies experience a certain scenario, a probable outcome will occur — essentially checking against specific values while asking itself what has been learned over a set amount of time. It knows that if you do this, then that will happen and, therefore, can suggest how best to work towards a positive result.”(Griffiths, 2017)

The awakening of cloud 2.0

The cloud will continue to win and lead the way for on-going emerging technological convergence and innovation. Cloud 1.0 has provided an exciting pedestal and an extremely stable foundation for many businesses to build their cloud adoption strategies. The increasing rate of data production and consumption is already impacting existing cloud infrastructure providers, enterprises and existing cloud adopters, pushing them to use this enormous resource to evolve an innately data-driven intelligence infrastructure that improves significantly on agility, security and data analysis.  Also, some ongoing technology service trends and standards within various industries are further exacerbating the situation, prominent amongst which are:

  • Cloud backend as a service based on the serverless architecture
  • Software appliance as a service
  • Multi-cloud tenancy with open source as portability enabler
  • One size fits all data storage provisioning
  • Automated machine learning algorithmic enabled data analytics
  • Self-healing infrastructure provisioning
  • Cloud vulnerabilities and security

Cloud 2.0 is therefore expected to provide and support an intelligence new infrastructural platforms that are flexibly driven by the customers. The consumer not the cloud service provider now holds the driver’s seat in this emerging scenario with incredible new features being delivered and explored. The emerging scenario includes increasingly moving many businesses and organisations from cloud first to a cloud the only strategy that enhances digital transformation everywhere. Cloud 2.0 implementation and services are expected to support natively multiple tenancy scenarios (private, public, mini/micro-cloud, etc.) with hosted services expected to double to over $530 billion according to IDC estimates in 2021. It should be able to leverage the diversifying cloud environment that is expected to be about 20% at the network edge, over 15% specialised compute and over 80% multi-cloud consumption. Other planned features will include:

  1. Bring intelligence to cloud data
  2. Micro-services becoming the de-facto information technology management standards through DevOps, Kubernetes (container orchestration)
  3. Cloud as the foundation for digital trust at scale using the blockchain

While many people continue to make the erroneous conclusion of the cloud being an object of the economics of scale, cloud 2.0 promises to correct this impression and make it more about data and its value proposition.

Personalised pricing is the future of competitive pricing

Dynamic pricing or flexible pricing isn’t new. Before the 19th century, customers haggled for discounts, and loyal patrons were often given a “friendly price.” Mass marketing introduced a little more science by using demographic research, and more recently big data and analytics, but lost the shopkeeper’s ability to negotiate with a customer directly. Prices now are mostly set according to market segment and supply-demand, not based on first-hand buyer knowledge. Dynamic pricing today is human-driven, assisted by big data and analytics. Airlines change their pricing based on peak times. Uber charges more for high-traffic hours, during driver shortages or for travelling in certain neighbourhoods(Chemko, 2018). In the emerging digital arena, price personalisation is the ultimate goal for digital shoppers. It will involve taking existing dynamic pricing models and strategies a step further and focus more on the interest of the potential shopper, their purchasing power, product interaction history, market and competitor’s prepositions, etc. While big data analytics remains key to competitor price monitoring an intuitive approach to dynamic pricing, its relevance to personalised pricing is even more substantial. Some research regarding personalised pricing have demonstrated the possibility of pricing freedom that will eventually make certain goods and services more accessible to those who have lower purchasing power or value the product less; this is an emerging market democratisation and an opening made possible by data(minderest, 2018). Hence tomorrow’s markets will not only be driven by the price consumers are willing to pay alone. They will place significant value on the overall buying experience the seller provides – customer service, shipping speed, return policies, discounts, and many other factors. Although the products and services are the same, the quality of the buying experience will invariably vary across and between sellers.

The rise of data-driven strategy and leadership

Big data importance is on the rise as the attention of most industry are now laser-focused on it. Senior management and top executives across the globe are currently seeking expert guidance for big data analytics best practices and greater understanding of its role in strategic decision making. The increasing power afforded this group of managers by big data analytics results within their various organisations will have a significant effect on corporate strategy. Although the importance of data is not in doubt and, often considerable investments in data collection, storage and analytical tools, many executives are unsure of how to handle the growing influx of data and how to properly use them internally for competitive value. The predominant questions are those around, what to collect, how best to manage, codify, store this data; how to analyse and interpret the same; and how actionable insights can be utilised effectively.  The ability to answer these questions will most definitely support organisational management in the deployment and future usage of its big data analytic investment for greater success(Mazzei & Noble, 2017).

As data production and organisational ability to capture same grows in leaps and bounds, digital transformation promises a steady shift of the strategic landscape of businesses and evolution of innovative new business models. While big data approaches and innovations are gradually becoming mainstream in business environments, the management field to say the least completely ignores the serious competitive advantages and strategic business value implications. According to Mazzei et al. (2017), while the dialogue in management literature revolves around the effect of big data on management research, the discussions continues to centre around new modes of data collection and analysis. Unfortunately, management scholars have failed to see the bigger picture of the capabilities of big data and the potential shifting of organisational decision making and strategic posture. “The uses for data are shifting as collected data helps to determine what markets to explore and how consumer trends are changing, and the data can drive these determinations in real time. We are seeing firms take on non-traditional markets, leveraging their data and analytic resources–—in conjuncture with massive amounts of human and financial capital–—to upend traditional barriers to entry. The ultimate goal of big data movers and innovators is to build greater knowledge and dynamic capabilities and to apply the benefits of big data analytics in a way that creates unique and sustainable competitive advantage through the development of diverse ecosystems and data flows. Through these advances in the consumption and application of big data, competition and competitive forces are being redefined. We further this conversation through the introduction of a new framework, which outlines three distinct approaches for how organisations can embrace and use big data(Mazzei & Noble, 2017, p. 3).” The data-driven leadership concept is fast becoming the norm and motivation for many industry leaders to leverage the value proposition of big data. However, as always the case, a data-driven leadership style does not suffice but by encouraging a cultural shift within the enterprise that priorities a top-down and bottom-up culture driven by data.

Privacy takes a new life

The amount of data that we are creating and, the volume of data to be generated in the future and their associated importance and use cases makes data privacy and security of most importance whether now or in the future. In most of the developed world, privacy is considered a fundamental human right, and it’s protected by law. However, the concept of privacy is now in a flux — the European Union’s General Data Protection Regulation (GDPR) which is designed to enable individuals to control their data better. It is hoped that these modernised and unified rules will allow businesses to make the most of the opportunities of the Digital Single Market by reducing regulation and benefiting from reinforced consumer trust. It is expected that data security and privacy concerns will be the most significant barrier for big data analytics and if it fails to cope with it effectively and satisfactorily, the ensuring conflict maybe is catastrophic.


Please enter your comment!
Please enter your name here