Dr. Sean Gustafson explains the necessity of cultural changes during migration from a Data Warehouse to a Cloud Data Platform

Dr. Sean Gustafson, Product Owner – Data Platform at Scout24, presented his opinion on ‘Necessary Cultural Change during Migration from Data Warehouse to Cloud Data Platform’ during the Data Festival 2018. He provides insights into technological and cultural changes at Scout24 and explains why he considers cultural change to be both a crucial and challenging step in the successful remodeling of a company’s data landscape.

Initial data management at Scout24 in 2007

Scout24 basically operates in two (online) business areas: sales of (used) cars and procurement of apartments and properties. The company has gone through several technological and cultural changes.

The last of these changes, the ‘Road to Microservice Architecture’, started in 2007. By then, Scout24 was organizing its data in a state-of-the-art three-tier architecture consisting of a web tier, a middle tier, and a core database.

After realizing the importance of data, it built a Data & Analytics team that was introduced into the BI-Tool Micro Strategy. In that context, an Oracle-based Data Warehouse was implemented.

Things got complicated in 2011

By 2011, the core operational database had become disadvantageous in many ways:

  • costs exploded
  • strange access patterns occurred
  • management of the database became cumbersome

The technological shift to a Microservice Architecture starts in 2013

In 2013, Scout24 began to slice up its Monolithic Architecture in order to make the product teams design their own (Micro)services. The goal of this step was the proliferation of data across the entire company.

Once the technical part of the migration was finalized, the company faced chaos throughout its entire organization: the central Data Team’s capacity was overloaded in its function as a bottleneck, which slowed down the entire organization.

  A Data Lake is part of the remodeling of the data landscape

Following the advice of the time, Scout24 implemented a Data Lake in an AWS cloud that was used as the main source for reporting within the Micro Strategy. However, this technological change requires an organizational and – most importantly – a cultural change.

On an organizational level, the Data Engineering and Business Intelligence teams were merged in order to take care of the Data Platform together. Besides other organizational measures taken, a Data Lab was implemented to incentivize employees to generate innovative, data-driven ideas.

  Last, but most important: the cultural change

After the technical and organizational implementation of the new Data Architecture, Dr. Sean Gustafson points out the importance of a cultural change.

Scout24 wanted to become a truly data-driven company – without inflating its Data & Analytics team. In order to achieve this goal and allow the team to grow in size, a cultural change was necessary: the active participation of all members of the organization and a change of mindset are vital in order to realize this data-driven approach.

  A manifesto transfers cultural change to the entire organization

Scout24 released its Data Landscape Manifesto as a framework to set roles, values, and responsibilities and promote cultural change:

‘Preamble: Data is a key asset of our company’

This understanding is the basis of a data-driven company.

‘#2 Our Responsibility
We, Data & Analytics, are responsible for providing a solid Data Platform as well as clear guidelines and training how to participate in the Data Landscape’

This paragraph specifies the responsibility of the Data & Analytics team: After the cultural change, they are responsible to supply the Data Platform as a baseline to the organization; however, they are not in charge of producing or consuming the data.’

‘#3 Data Autonomy Not Anarchy
Data autonomy puts data producers & data consumers in control of their data & of their metrics and thereby allows us to be data-driven at scale, but this comes with responsibility’

‘#4 Producer’s Responsibility
Data producers are responsible for publishing data to the central Data Lake, for the data’s quality, and for publishing metadata that makes it easy to find and consume the data’

‘#5 Consumer’s Responsibility
Data Consumers are responsible for the definition & visualization of metrics and for driving the implementation and maintenance of these metrics’

Paragraphs 4 and 5 outline the responsibilities of data consumers and producers.’

‘#6 Exception: Core KPIs
We, Data & Analytics, take the full ownership and responsibility of the few top company-wide core KPIs’

In order to ensure consistency of the core KPIs such as revenue and its allocation, the Data & Analytics team is responsible for those data.

‘#7 Transparency Over Continuity 

We value data transparency over data continuity, which means we may break metric comparability if it is for the cause of enabling better insights’

When everyone contributes to the Data Platform, there is a tendency to continuity. Therefore, it is important to be aware of this tendency and communicate mistakes as soon as they happen.

The ultimate goal of the entire manifesto is to establish a Federal Data Landscape that supplies enough rules to maintain fruitful cooperation without limiting autonomy too much.

Summary of the migration from Data Warehouse to Cloud Data Platform at Scout24

A quick comparison between the features of a Data Warehouse and a Cloud Data Platform reveals the main developments and cultural changes at Scout24:

  • The company evolved from a centralized to a federated system.
  • Instead of control, the company supports (data) autonomy.
  • Instead of perfection, the Data & Analytics team achieves scale.
  • Instead of pulling data from the database, it is now pushed into the Data Lake.
  • Data used to be a product at Scout24, now it is a platform.

All these developments have one major consequence: the product teams bear much greater responsibility.

How can the product teams be incentivized to get onboard with the new development?

Dr. Sean Gustafson points out three options that help the product teams to accept the new development.

  • Active promotion of the cultural change.
  • Data & Analytics refuses to take in new use cases in the Data Warehouse.
  • Nudging the organization into using the new system.

Nudging means that the company makes it easy for employees to participate in the new system and hard to refuse participation. At Scout24, several nudges, such as automatic table publishing or backup and damage control, were implemented.

Lessons and Learnings

One key learning of the migration project at Scout24 is that change has to be technical, organizational, and cultural – it cannot be realized on only one of these levels.

Furthermore, measures need to be implemented to counteract resistance within other parts of the organization (nudging). Finally, communication is a key method in promoting cultural change to an organization.

 

Dr. Sean Gustafson’s complete presentation about “cultural change” is available here..