6 Missing Drivers of Failed Big Data Projects in Australia

Given Contexti’s 50+ Big Data Solutions engagements and having provided Big Data Training to over 1,000 professional from over 200 enterprises across Australia, we’ve seen, heard and been involved with enough projects to draw some insights on what drives big data project success and failure.

At the same time, talking with our international business partners and counterparts, we find our observations, while made in Australia, are not unique to Australian enterprises and businesses across the globe will benefit by understanding the importance of the following drivers.

To help illustrate each driver below, we’ve provided an insight of how Contexti client Seven West Media positively addressed each driver, resulting in the delivery of one of the most successful Big Data Analytics projects in Australia.

#1 – Commercial Strategy

What’s the purpose of your big data project? Are you responding to business disruption? Are you leading with innovation? Which revenue lines have been impacted and which revenue lines do you intend to impact? Having a clear commercial strategy with the right narrative will help to align team members, limit initial scope and set the bar for getting a return on investment. No clear commercial strategy, a poorly defined strategy or the common ‘proof of concept’ approach of ‘let’s throw all our data in a data lake and see what we find’ will deliver a tyre kicking exercise, resulting in wasted time, money and missed opportunities.

Contexti Client Example: Seven West Media recognised ‘television was no longer just in the living room’ and they were now in the business of ‘quality content on any device, anywhere, any time’. This headline narrative guided the commercial strategy for Seven West Media’s big data project which was to find innovative and effective ways to engage with audiences on any platform. Engaged audiences directly correlate to increased revenue.

#2 – Defined Actions

Aligned to your commercial strategy, you need clarity on how you will take action with your new-found insights. Worse than not having the capability to find new insights from your data is when you have the insights but you’re unable to take actions on those insights. This require to step away from the data and think about systems, people, processes, customers and partners that will be impacted when you take action with your new insights.

Contexti Client Example: Seven West Media during the 2016 Rio Olympic Games leveraged their big data platform to find new audience insights and to personalise the viewer experience. Leveraging the insights they executed on a marketing program that delivered 2.7 million emails across 108 targeted campaigns. By defining and acting on their insights they achieved an ROI of a 29% lift in the average minutes streamed by users who were part of this targeted program.

#3 – Executive Engagement

All projects need executive sponsorship and support to get basic budget sign off, however some projects significantly benefit by having executives more actively engaged with the project. Big data projects fall into this category. With the right engagement, executives will provide headline messaging across the organisation of why this project exists, what are the benefits and importantly why departmental colleagues should pro-actively enable and support this project. There will come a time when requests will be made to departmental colleagues to make data available, make people available and make their expertise available and without the right mandate supported by an engaged executive your project is on-hold in the best case or dead in the extreme case.

Contexti Client Example: Seven West Media’s big data project was sponsored by its Chief Commercial Officer who was aware, active, vocal and visible across the project. With an engaged executive sponsor, the Director of Data and Business Intelligence who was driving the project had mandate to align internal departments and external services providers, ultimately delivering a successful project.

#4 – Team Composition

When considering the team composition for big data projects, many will be able to quickly identify the Data Scientist role, but will struggle to name other roles. Many project failures have been a result of unallocated or misallocated resources. A Data Scientist is a Data Scientist, they are not Platform Architects, Platform Engineers, Platform Administrators, Data Architects, Data Engineers, Commercial Strategist, Change Managers, Trainers or industry subject matter experts… to name just a few of the roles that need to be filed in a successful project. If you start with getting clarity on your commercial strategy (Driver #1) and work through how you will action your insights (Driver #2) then that should be a good start to thinking through your team composition. Of course depending on the size of your project and complexity of your mandate and budget you will take a different approach to composition of your team. Some roles may be internally filled, some will be full time roles, others will be part time roles, you might supplement your core team with contractors or you might outsource some functions to third party service providers. Key point here is don’t just hire a Data Scientist then wonder why the project failed.

Contexti Client Example: Seven West Media’s big data project team was comprised of many internally filled roles (Director of Data and BI, Data Architect, Data Scientist, Data Analyst), they also also engaged with other internal departments (e.g. marketing who executed on the targeted email campaigns) as well as third party providers such as Contexti who took care of the design, build and management of their big data platform (meaning Platform Architecture, Platform Engineering, Platform Administration, Data Engineering and Support was outsourced).

#5 – Technology Choice

Your big data solution needs to be powered by an appropriate technology that supports your commercial objectives and use cases (Driver #1). Further, the technology needs to be supported by a appropriately skilled, qualified and enabled resources (Driver #2). Too often we’ve seen technology choices being made in the absence of commercial objectives, use cases and clarity of how the technology will be supported in a production environment. What’s worse is we’ve also seen technology choices and decisions being made because of internal politics, pre-existing vendor relationships and by people who are not close enough to the issues of the project. This not only sets up the project for failure from a technology perspective, it also causes the project team to lose faith in the leadership and decision making process.

Contexti Client Example: Seven West Media had clarity on their technology choices. Leveraging the project leadership team’s prior experience and with clarity of their commercial objectives they identified there were two key challenges that they would focus on. Firstly, the variety and complexity of the data sources drove their decision towards selecting a mature, secure and scalable enterprise-grade data management platform at the core of their data solution (they selected Cloudera). Secondly, given the highly variable nature of the events they were covering (such as the Australian Open, Wimbledon, Rio Olympic Games etc) with variable amounts of traffic and data, they needed the ability to scale up and down in a flexible manner, driving their decision for a cloud-based solution (they selected Amazon Web Services). Further looking at the team composition, their skill-sets and familiarity with existing tools, the overall solution also integrated with Teradata on AWS, Tableau for visualisation and R for advanced analytics.

#6 – Agile Approach

With the first five drivers in place (Commercial Strategy, Defined Actions, Executive Engagement, Team Composition and Technology Choice) you want to execute effectively and demonstrate quick wins. Adopting an agile approach will enable to you test and learn, share the lessons-learned and move to your next iteration to meet your commercial objectives. Without these first five drivers in place, it’s difficult to be agile and we’ve seen many organisations fail as they have talked themselves into building what they perceived to be ‘the biggest, most flexible, most secure big data platform that could solve any big data use case’ but in reality solved none. These non-agile projects lost momentum and ultimately failed having spent so much time waiting for hardware to arrive, waiting for integrators to spend months building and securing their big data platforms, waiting for their internal departments to make data available, waiting for the newly hired data scientists to start and the list goes on. Without all six drivers in place all involved with the project had no sense of urgency, purpose or accountability.

Contexti Client Example: Seven West Media adopted an agile approach from day one. Within six weeks of Contexti being engaged (to design, build and manage a big data solution) an initial pilot program was run to support the coverage of the 2016 Australian Open Tennis event. Following that the solution was expanded to support the 2016 Wimbledon Tennis event, then more work was done to scale up the solution for Seven West Media’s coverage of the 2016 Rio Olympic Games in Australia.


By: Sidney Minassian – Founder & CEO, Contexti – Big Data Analytics


Subscribe for updates on the latest big data analytics training courses, industry events and career insights.