Eighty-nine percent of organizations agree that the rate of change has accelerated in the past two years, and it’s not likely to slow down anytime soon. In an age of relentless change, companies must have a firm grasp on the data and insights that can help them evolve their customer, workforce, and operational strategies accordingly. At the core, this requires a readily scalable approach to data architecture, best enabled by public cloud architectures. Yet, the movement away from on-premise data storage and towards the public cloud is a tall order. It can be challenging to pick an ideal public cloud platform that meets the needs of your business, and that best utilizes the existing skillsets within your workforce. Using an expertise-guided approach to inform your selection is the best way forward. In this blog series, we distill the data pipeline architectures of the top cloud platforms available today. From there, we equip you with some practical criteria to inform the right selection for your organization.
Business Intelligence (BI) and Data Warehouse (DW) managers have an elevated charge: to establish a data architecture foundation that enables the business to readily scale and adapt in response to a growing set of disruptors. Underscoring this imperative, business leaders report that “data and technology” is their top transformation area for 2020.1 In this climate, selecting the right set of extract, transform, and load (ETL) tools is more important than ever. The significant increase in data consumption over the past decade has driven an explosion of public cloud architecture choices. Many organizations host their ETL infrastructure on-premise, and they have come to realize it is ill-equipped to support the significant spikes in data usage caused by operational disruption—with COVID-19 being the latest example. This can put organizations in a tough spot, requiring a substantial upfront investment in solutions that may later become redundant.
Cloud architecture offers a solution. This type of ETL architecture is more flexible, scalable, and cost-efficient, helping IT departments stay nimble and tackle any surprises that come their way.
Why you should make the switch
Below is an example of what a generic, on-premise ETL architecture might look like:
For this blog we will be discussing only structured data, but these platforms easily support unstructured data without significant changes to the architecture.
This approach often encounters a few bottlenecks:
- Underpowered servers or small slices of virtual machines (VMs) with limited CPU and memory.
- Storage devices that are undersized or databases that continuously hit their capacity.
- Networking devices that are not compatible or have low throughput.
Moving your ETL solution to the cloud addresses these challenges and offers scalability and additional features with a pay-as-you-go pricing model. Cloud technologies provide flexibility and instant access, two vital features that have become the foundation of quality tools needed during the COVID-19 pandemic. Also, organizations that adopt cloud architecture often have an edge in rapidly testing and implementing solutions. They achieve higher productivity, decrease the cost of operations, can triple the number of features delivered, and reduce unplanned infrastructure downtime.
With cloud the clear choice for ETL architecture, the main question is: Which cloud provider best suits your needs? The answer to this question can be complicated. Several business factors come into play, including company partnerships, your industry, and your customers' preferred provider. Organizations may not want to become dependent on a single cloud provider. In fact, using a mix of cloud vendors and shifting workloads as needed can help you unlock new sources of efficiency. According to a recent study, 69 percent of organizations use (or plan to use) two or more providers. Our clients often choose a hybrid multi-cloud solution (such as the sample below) encompassing different services of three cloud providers.
A hybrid cloud solution can vary based on organizational needs:
Organizations, now more than ever, need scalable, adaptive digital infrastructure to help their business navigate uncertain times. With restrictions brought by COVID-19, there is an even greater motivation to accelerate the adoption to cloud and capitalize on the value that cloud delivers. By making full use of cloud technologies, companies can more readily adapt to the changing world and an unknowable future.
When it comes to selecting the best cloud platform to fit your needs, we recommend starting with an expertise-based perspective, grounded in an understanding of the attributes of each option. In the upcoming blogs of this series, we’ll show you how to put this expertise into practice. We’ll walk you through examples of cloud-based ETL architectures across the three major platforms (Amazon Web Services, Google Cloud Platform, and Azure) and show you how to scale all of them.
Click here to read parts two, three, and four in our blog series.
[1] North Highland-sponsored survey of > 700 business leaders from organizations across industries that had 2018 revenues > $1B and that are headquartered in the U.S. or U.K.
