Must-Read Books for High-Performance System Design (Sponsored)

Access 4 PDFs on building and optimizing data-intensive applications.

Read experts’ proven strategies for optimizing data-intensive applications, database performance, and latency:

Designing Data-Intensive Applications by Martin Kleppmann:Discover new ways of thinking about your distributed data system challenges – with actionable insights for building scalable, high-performance solutions.
Latency by Pekka Enberg:Learn how to expertly diagnose latency problems and master the low-latency techniques that have been predominantly “tribal knowledge” until now.
Database Performance at Scale: A Practical Guide:Discover new ways to optimize database performance — and avoid common pitfalls – based on learnings from thousands of real-world database use cases.
ScyllaDB in Action by Bo Ingram:A practical guide to everything you need to know about ScyllaDB, from your very first queries to running it in a production environment.

Whether you’re working on large-scale systems or designing distributed data architectures, these books will prepare you for what’s next.

Access Books for Free

Disclaimer: The details in this post have been derived from the details shared online by the Spotify Engineering Team. All credit for the technical details goes to the Spotify Engineering Team. The links to the original articles and sources are present in the references section at the end of the post. We’ve attempted to analyze the details and provide our input about them. If you find any inaccuracies or omissions, please leave a comment, and we will do our best to fix them.

Every day, Spotify processes around 1.4 trillion data points. These data points come from millions of users around the world listening to music, creating playlists, and interacting with the app in different ways. Handling this volume of information is not something a few ad hoc systems can manage. It requires a robust, well-designed data platform that can reliably collect, process, and make sense of all this information.

From payments to personalized recommendations to product experiments, almost every decision Spotify makes depends on data. This makes its data platform one of the most critical parts of the company’s overall technology stack.

Spotify did not build this platform overnight. In the early years, data systems were more improvised. As the company grew, the number of users increased, and the complexity of business decisions became greater. Different teams began collecting data for their own needs. Over time, this led to a growing need for a centralized, structured, and productized platform that could support the entire company, not just individual teams.

The shift toward a formal data platform was driven by both business and technical factors:

Business drivers:Spotify needed high-quality, reliable data to support its core functions, such as financial reporting, advertising, product experimentation, and personalized recommendations. Data had to be consistent and trustworthy so that teams could make important decisions with confidence.
Technical drivers:The sheer scale of the data meant the company needed strong infrastructure that could collect events from millions of devices, process them quickly, and make them available in usable form. It also required clear data ownership, easy searchability, built-in quality checks, and compliance with privacy regulations.

This evolution happened organically as Spotify’s products matured. With each new feature and business need, new data requirements emerged. Over time, this learning process shaped a platform that now supports everything from real-time streaming analytics to large-scale experimentation and machine learning applications.

In this article, we will look at how Spotify built its data platform and the challenges it faced along the way.

State of Trust: AI-driven attacks are getting more sophisticated (Sponsored)

AI-driven attacks are getting bigger, faster, and more sophisticated—making risk much more difficult to contain. Without automation to respond quickly to AI threats, teams are forced to react without a plan in place.

This is according to Vanta’s newest State of Trust report, which surveyed 3,500 business and IT leaders across the globe.

One big change since last year’s report? Teams falling behind AI risks—and spending way more time and energy proving trust than building it.

61% of leaders spend more time proving security rather than improving it
59% note that AI risks outpace their expertise
But 95% say AI is making their security teams more effective

Get the full report to learn how organizations are navigating these changes, and what early adopters are doing to stay ahead.

Download the report

Platform Composition and Evolution

In its early years, Spotify’s data operations were run by a single team. At one point, this team was managing Europe’s largest Hadoop cluster. For reference, Hadoop is an open-source framework used to store and process very large amounts of data across many computers.

At that stage, most data work was still centralized, and many processes were built manually or handled through shared infrastructure.

As Spotify grew, this model became too limited. The company needed more specialized tools and teams to handle the increasing scale and complexity. Over time, Spotify moved from that single Hadoop cluster to a multi-product data platform team. This new structure allowed them to separate the platform into clear functional areas, each responsible for a specific part of the data journey.

At the core of Spotify’s platform are three main building blocks that work together:

Data Collection:This part of the platform focuses on how data is gathered from millions of clients around the world. These “clients” include mobile apps, desktop apps, web browsers, and backend services. Every time a user plays a song, skips a track, adjusts the volume, or interacts with the app, an event is recorded. Spotify uses specialized tools and event delivery systems to collect these events in real time. This ensures that the data entering the platform is structured, consistent, and ready to be processed further.
Data Processing:Once the data is collected, it must be cleaned, transformed, and organized so it can be used effectively. This happens through data pipelines, which are automated workflows that process large amounts of information on a fixed schedule or in real time. These pipelines might do things like aggregating how many times a track was played in a day, linking a user’s activity to recommendation systems, or preparing data for financial reports. By scheduling and running thousands of pipelines, Spotify can provide up-to-date and accurate data to every team that needs it.
Data Management:This part ensures that the data is secure, private, and trustworthy. It includes data attribution, privacy controls, and security mechanisms to comply with regulations and internal governance rules. Data management also deals with data integrity, which means making sure the data is correct, consistent, and not corrupted during collection or processing.

These three areas are not isolated. They are deeply interconnected, forming a platform that is reliable, searchable, and easy to build on.

See the diagram below that shows the key components of the data platform:

For example, once data is collected and processed, it becomes searchable and can be used directly in other systems. One of the most important systems it powers is Spotify’s experimentation platform, called Confidence. This platform allows teams to run A/B tests and other experiments at scale, ensuring new product features are backed by real data before being fully launched.

Data Collection - Event Delivery at Scale

One of the most impressive parts of Spotify’s data platform is its data collection system.

Every time a user interacts with the app, whether by hitting play, searching for a song, skipping a track, or creating a playlist, an “event” is generated. Spotify’s data collection system is responsible for capturing and delivering all of these events reliably and at a massive scale.

Spotify collects more than one trillion events every day. This is an extraordinary amount of data flowing in constantly from mobile apps, desktop clients, web players, and other connected devices. To handle this, the architecture of the data collection system has evolved through multiple iterations over the years. Early versions were much simpler, but as the user base and product features expanded, the system had to be redesigned to keep up with growing scale and complexity.

How Developers Work with Event Data

At Spotify, product teams don’t need to build custom infrastructure to collect events. Instead, they use client SDKs that make it easy to define what events should be collected.

Here’s how the workflow looks:

A team defines a new event schema, which describes what kind of data will be collected and in what format. For example, a schema might specify that each “play” event should include the user ID (or an anonymous identifier), the song ID, the timestamp, and the device type.
Once the schema is defined, the platform automatically deploys all the infrastructure needed to handle that event. This includes:
- Pub/Sub queues to reliably stream data as it arrives
- Anonymization pipelines to remove or protect sensitive user information
- Streaming jobs to process and route the data further down the platform
If a schema changes (for example, if a team adds a new field like “playlist ID”), the system automatically redeploys the affected components so the infrastructure stays in sync.

See the diagram below:

This level of automation is made possible through Kubernetes Operators. An operator is a special kind of software that manages complex applications running on Kubernetes, which is the container orchestration system Spotify uses to run its services.

In simple terms, operators allow Spotify to treat data infrastructure as code, so changes are applied quickly and reliably without manual work.

Built-in Privacy and Security

Handling user data at this scale comes with serious privacy responsibilities. Spotify builds anonymization directly into its pipelines to ensure that sensitive information is protected before it ever reaches downstream systems.

They also use internal key-handling systems, which help control how and when certain pieces of data can be accessed or decrypted. This is essential for compliance with privacy regulations like GDPR and for maintaining user trust.

Centralization and Self-Service

A key strength of Spotify’s data collection system is its ownership model. Instead of making the central infrastructure team responsible for every change, Spotify has designed the platform so that product teams can manage most of their own event data.

This means a team can:

Add or modify event schemas
Deploy event pipelines
Make small adjustments to how their data is processed

They can do all this without depending on the central platform team. This balance between centralization and self-service helps the platform scale to thousands of active users inside the company while keeping operational overhead low.

Breadth of Event Types

The platform currently handles around 1,800 different event types, each capturing different kinds of user interactions and system signals.

There are dedicated teams responsible for:

Maintaining the event delivery infrastructure
Managing the client SDKs that developers use
Building “journey datasets”, which combine multiple event streams into structured, meaningful timelines
Supporting the underlying infrastructure that keeps the system running smoothly

This massive, well-structured data collection layer forms the foundation of Spotify’s entire data platform. Without it, the rest of the platform (processing, management, analytics, and experimentation) would not be possible. It ensures that the right data is captured, secured, and made available at the right time for everything from recommendations to business decisions.

Data Management and Data Processing

Once data is collected, Spotify needs to transform it into something meaningful and trustworthy. This is where data processing and management come into play. At Spotify’s scale, this is a massive and complex operation that must run reliably every hour of every day.

Spotify runs more than 38,000 active data pipelines on a regular schedule. Some run hourly, while others run daily, depending on the business need. A data pipeline is essentially an automated workflow that moves and transforms data from one place to another.

For example:

A pipeline might take raw event data from user streams and aggregate it into daily summaries of how many times each song was played.
Another pipeline might prepare datasets that support recommendation algorithms.
Yet another might generate financial reports.

Operating this many pipelines requires a strong focus on scalability (handling growth efficiently), traceability (understanding where data comes from and how it changes), searchability (finding the right datasets quickly), and regulatory compliance (meeting privacy and data retention requirements).

The Execution Stack

To run these pipelines, Spotify uses a scheduler that automatically triggers workflows at the right times. These workflows are executed on:

BigQuery, Google’s cloud data warehouse, allows fast analysis of very large datasets.
Flink or Dataflow are frameworks for processing data streams or large batches of data in parallel.

Most pipelines are written using Scio, a Scala library built on top of Apache Beam. Apache Beam is a framework that allows developers to write data processing jobs once and then run them on different underlying engines like Flink or Dataflow. This gives Spotify flexibility and helps standardize development across teams.

Each pipeline in Spotify’s platform produces a data endpoint. An endpoint is essentially the final dataset that other teams and systems can use. These endpoints are treated like products with well-defined characteristics:

Explicit schemas so everyone knows exactly what data is included and in what format.
Multi-partitioning to organize data efficiently, often by time or other logical dimensions.
Retention policies specify how long the data is stored.
Access Control Lists (ACLs) are used to define who can view or modify the data.
Lineage tracking that records where the data came from and what transformations were applied.
Quality checks to catch errors early and ensure the data is trustworthy.

Platform as Code

Spotify has built the platform in a way that allows engineers to define their pipelines and endpoints as code. This means all configuration and resource definitions are stored in the same place as the pipeline’s source code.

As mentioned, the system uses custom Kubernetes Operators to manage this. When developers push updates to their code repositories, the operators automatically deploy the necessary resources. This code ownership model ensures that the team responsible for a pipeline also controls its configuration and lifecycle, reducing dependency on centralized teams.

With tens of thousands of pipelines, keeping everything healthy and efficient is a major priority. Spotify has a strong operations and observability layer that includes:

Alerts for late pipelines, long-running jobs, or failures.
Endpoint health monitoring to make sure datasets are fresh and accurate.
Backstage integration to provide a single interface where teams can view and manage their data resources.

Backstage is an internal developer portal that Spotify open-sourced. It brings together tools for monitoring, cost analysis, quality assurance, and documentation. Instead of searching across many systems, engineers can manage everything from one central place.

Conclusion

Spotify’s data platform is a great example of how a company can evolve its infrastructure to meet growing business and technical demands. What began as a small team managing an on-premises Hadoop cluster has grown into a platform run by more than a hundred engineers, operating entirely on Google Cloud.

This transformation did not happen overnight. Spotify aligned its organizational needs with its technical investments, defined clear goals, and built strong feedback channels with its internal users. By starting small, iterating, and learning from each stage of growth, they created a platform that balances centralized infrastructure with self-service capabilities for product teams. Other organizations can take valuable lessons from this journey.

A dedicated data platform becomes essential when:

Teams need searchable and democratized data across business and engineering functions.
Financial reporting and operational metrics require predictable, reportable pipelines.
Data quality and trust are critical for decision-making.
Experimentation and development need efficient workflows with strong tooling.
Machine learning initiatives depend on well-organized and structured datasets.

References:

SPONSOR US

Get your product in front of more than 1,000,000 tech professionals.

Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.

Space Fills Up Fast - Reserve Today

Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailingsponsorship@bytebytego.com.

","contentEncoded","freeContentEncoded","textContent","Must-Read Books for High-Performance System Design (Sponsored)Access 4 PDFs on building and optimizing data-intensive applications.Read experts’ proven strategies for optimizing data-intensive applications, database performance, and latency:Designing Data-Intensive Applications by Martin Kleppmann: Discover new ways of thinking about your distributed data system challenges – with actionable insights for building scalable, high-performance solutions.Latency by Pekka Enberg: Learn how to expertly diagnose latency problems and master the low-latency techniques that have been predominantly “tribal knowledge” until now.Database Performance at Scale: A Practical Guide: Discover new ways to optimize database performance — and avoid common pitfalls – based on learnings from thousands of real-world database use cases.ScyllaDB in Action by Bo Ingram: A practical guide to everything you need to know about ScyllaDB, from your very first queries to running it in a production environment.Whether you’re working on large-scale systems or designing distributed data architectures, these books will prepare you for what’s next.Access Books for FreeDisclaimer: The details in this post have been derived from the details shared online by the Spotify Engineering Team. All credit for the technical details goes to the Spotify Engineering Team. The links to the original articles and sources are present in the references section at the end of the post. We’ve attempted to analyze the details and provide our input about them. If you find any inaccuracies or omissions, please leave a comment, and we will do our best to fix them.Every day, Spotify processes around 1.4 trillion data points. These data points come from millions of users around the world listening to music, creating playlists, and interacting with the app in different ways. Handling this volume of information is not something a few ad hoc systems can manage. It requires a robust, well-designed data platform that can reliably collect, process, and make sense of all this information.From payments to personalized recommendations to product experiments, almost every decision Spotify makes depends on data. This makes its data platform one of the most critical parts of the company’s overall technology stack.Spotify did not build this platform overnight. In the early years, data systems were more improvised. As the company grew, the number of users increased, and the complexity of business decisions became greater. Different teams began collecting data for their own needs. Over time, this led to a growing need for a centralized, structured, and productized platform that could support the entire company, not just individual teams.The shift toward a formal data platform was driven by both business and technical factors:Business drivers: Spotify needed high-quality, reliable data to support its core functions, such as financial reporting, advertising, product experimentation, and personalized recommendations. Data had to be consistent and trustworthy so that teams could make important decisions with confidence.Technical drivers: The sheer scale of the data meant the company needed strong infrastructure that could collect events from millions of devices, process them quickly, and make them available in usable form. It also required clear data ownership, easy searchability, built-in quality checks, and compliance with privacy regulations.This evolution happened organically as Spotify’s products matured. With each new feature and business need, new data requirements emerged. Over time, this learning process shaped a platform that now supports everything from real-time streaming analytics to large-scale experimentation and machine learning applications.In this article, we will look at how Spotify built its data platform and the challenges it faced along the way.State of Trust: AI-driven attacks are getting more sophisticated (Sponsored)AI-driven attacks are getting bigger, faster, and more sophisticated—making risk much more difficult to contain. Without automation to respond quickly to AI threats, teams are forced to react without a plan in place.This is according to Vanta’s newest State of Trust report, which surveyed 3,500 business and IT leaders across the globe.One big change since last year’s report? Teams falling behind AI risks—and spending way more time and energy proving trust than building it.61% of leaders spend more time proving security rather than improving it59% note that AI risks outpace their expertiseBut 95% say AI is making their security teams more effectiveGet the full report to learn how organizations are navigating these changes, and what early adopters are doing to stay ahead.Download the reportPlatform Composition and EvolutionIn its early years, Spotify’s data operations were run by a single team. At one point, this team was managing Europe’s largest Hadoop cluster. For reference, Hadoop is an open-source framework used to store and process very large amounts of data across many computers.At that stage, most data work was still centralized, and many processes were built manually or handled through shared infrastructure.As Spotify grew, this model became too limited. The company needed more specialized tools and teams to handle the increasing scale and complexity. Over time, Spotify moved from that single Hadoop cluster to a multi-product data platform team. This new structure allowed them to separate the platform into clear functional areas, each responsible for a specific part of the data journey.At the core of Spotify’s platform are three main building blocks that work together:Data Collection: This part of the platform focuses on how data is gathered from millions of clients around the world. These “clients” include mobile apps, desktop apps, web browsers, and backend services. Every time a user plays a song, skips a track, adjusts the volume, or interacts with the app, an event is recorded. Spotify uses specialized tools and event delivery systems to collect these events in real time. This ensures that the data entering the platform is structured, consistent, and ready to be processed further.Data Processing: Once the data is collected, it must be cleaned, transformed, and organized so it can be used effectively. This happens through data pipelines, which are automated workflows that process large amounts of information on a fixed schedule or in real time. These pipelines might do things like aggregating how many times a track was played in a day, linking a user’s activity to recommendation systems, or preparing data for financial reports. By scheduling and running thousands of pipelines, Spotify can provide up-to-date and accurate data to every team that needs it.Data Management: This part ensures that the data is secure, private, and trustworthy. It includes data attribution, privacy controls, and security mechanisms to comply with regulations and internal governance rules. Data management also deals with data integrity, which means making sure the data is correct, consistent, and not corrupted during collection or processing.These three areas are not isolated. They are deeply interconnected, forming a platform that is reliable, searchable, and easy to build on.See the diagram below that shows the key components of the data platform:For example, once data is collected and processed, it becomes searchable and can be used directly in other systems. One of the most important systems it powers is Spotify’s experimentation platform, called Confidence. This platform allows teams to run A/B tests and other experiments at scale, ensuring new product features are backed by real data before being fully launched.Data Collection - Event Delivery at ScaleOne of the most impressive parts of Spotify’s data platform is its data collection system.Every time a user interacts with the app, whether by hitting play, searching for a song, skipping a track, or creating a playlist, an “event” is generated. Spotify’s data collection system is responsible for capturing and delivering all of these events reliably and at a massive scale.Spotify collects more than one trillion events every day. This is an extraordinary amount of data flowing in constantly from mobile apps, desktop clients, web players, and other connected devices. To handle this, the architecture of the data collection system has evolved through multiple iterations over the years. Early versions were much simpler, but as the user base and product features expanded, the system had to be redesigned to keep up with growing scale and complexity.How Developers Work with Event DataAt Spotify, product teams don’t need to build custom infrastructure to collect events. Instead, they use client SDKs that make it easy to define what events should be collected.Here’s how the workflow looks:A team defines a new event schema, which describes what kind of data will be collected and in what format. For example, a schema might specify that each “play” event should include the user ID (or an anonymous identifier), the song ID, the timestamp, and the device type.Once the schema is defined, the platform automatically deploys all the infrastructure needed to handle that event. This includes:Pub/Sub queues to reliably stream data as it arrivesAnonymization pipelines to remove or protect sensitive user informationStreaming jobs to process and route the data further down the platformIf a schema changes (for example, if a team adds a new field like “playlist ID”), the system automatically redeploys the affected components so the infrastructure stays in sync.See the diagram below:This level of automation is made possible through Kubernetes Operators. An operator is a special kind of software that manages complex applications running on Kubernetes, which is the container orchestration system Spotify uses to run its services.In simple terms, operators allow Spotify to treat data infrastructure as code, so changes are applied quickly and reliably without manual work.Built-in Privacy and SecurityHandling user data at this scale comes with serious privacy responsibilities. Spotify builds anonymization directly into its pipelines to ensure that sensitive information is protected before it ever reaches downstream systems.They also use internal key-handling systems, which help control how and when certain pieces of data can be accessed or decrypted. This is essential for compliance with privacy regulations like GDPR and for maintaining user trust.Centralization and Self-ServiceA key strength of Spotify’s data collection system is its ownership model. Instead of making the central infrastructure team responsible for every change, Spotify has designed the platform so that product teams can manage most of their own event data.This means a team can:Add or modify event schemasDeploy event pipelinesMake small adjustments to how their data is processedThey can do all this without depending on the central platform team. This balance between centralization and self-service helps the platform scale to thousands of active users inside the company while keeping operational overhead low.Breadth of Event TypesThe platform currently handles around 1,800 different event types, each capturing different kinds of user interactions and system signals.There are dedicated teams responsible for:Maintaining the event delivery infrastructureManaging the client SDKs that developers useBuilding “journey datasets”, which combine multiple event streams into structured, meaningful timelinesSupporting the underlying infrastructure that keeps the system running smoothlyThis massive, well-structured data collection layer forms the foundation of Spotify’s entire data platform. Without it, the rest of the platform (processing, management, analytics, and experimentation) would not be possible. It ensures that the right data is captured, secured, and made available at the right time for everything from recommendations to business decisions.Data Management and Data ProcessingOnce data is collected, Spotify needs to transform it into something meaningful and trustworthy. This is where data processing and management come into play. At Spotify’s scale, this is a massive and complex operation that must run reliably every hour of every day.Spotify runs more than 38,000 active data pipelines on a regular schedule. Some run hourly, while others run daily, depending on the business need. A data pipeline is essentially an automated workflow that moves and transforms data from one place to another.For example:A pipeline might take raw event data from user streams and aggregate it into daily summaries of how many times each song was played.Another pipeline might prepare datasets that support recommendation algorithms.Yet another might generate financial reports.Operating this many pipelines requires a strong focus on scalability (handling growth efficiently), traceability (understanding where data comes from and how it changes), searchability (finding the right datasets quickly), and regulatory compliance (meeting privacy and data retention requirements).The Execution StackTo run these pipelines, Spotify uses a scheduler that automatically triggers workflows at the right times. These workflows are executed on:BigQuery, Google’s cloud data warehouse, allows fast analysis of very large datasets.Flink or Dataflow are frameworks for processing data streams or large batches of data in parallel.Most pipelines are written using Scio, a Scala library built on top of Apache Beam. Apache Beam is a framework that allows developers to write data processing jobs once and then run them on different underlying engines like Flink or Dataflow. This gives Spotify flexibility and helps standardize development across teams.Each pipeline in Spotify’s platform produces a data endpoint. An endpoint is essentially the final dataset that other teams and systems can use. These endpoints are treated like products with well-defined characteristics:Explicit schemas so everyone knows exactly what data is included and in what format.Multi-partitioning to organize data efficiently, often by time or other logical dimensions.Retention policies specify how long the data is stored.Access Control Lists (ACLs) are used to define who can view or modify the data.Lineage tracking that records where the data came from and what transformations were applied.Quality checks to catch errors early and ensure the data is trustworthy.Platform as CodeSpotify has built the platform in a way that allows engineers to define their pipelines and endpoints as code. This means all configuration and resource definitions are stored in the same place as the pipeline’s source code.As mentioned, the system uses custom Kubernetes Operators to manage this. When developers push updates to their code repositories, the operators automatically deploy the necessary resources. This code ownership model ensures that the team responsible for a pipeline also controls its configuration and lifecycle, reducing dependency on centralized teams.With tens of thousands of pipelines, keeping everything healthy and efficient is a major priority. Spotify has a strong operations and observability layer that includes:Alerts for late pipelines, long-running jobs, or failures.Endpoint health monitoring to make sure datasets are fresh and accurate.Backstage integration to provide a single interface where teams can view and manage their data resources.Backstage is an internal developer portal that Spotify open-sourced. It brings together tools for monitoring, cost analysis, quality assurance, and documentation. Instead of searching across many systems, engineers can manage everything from one central place.ConclusionSpotify’s data platform is a great example of how a company can evolve its infrastructure to meet growing business and technical demands. What began as a small team managing an on-premises Hadoop cluster has grown into a platform run by more than a hundred engineers, operating entirely on Google Cloud.This transformation did not happen overnight. Spotify aligned its organizational needs with its technical investments, defined clear goals, and built strong feedback channels with its internal users. By starting small, iterating, and learning from each stage of growth, they created a platform that balances centralized infrastructure with self-service capabilities for product teams. Other organizations can take valuable lessons from this journey.A dedicated data platform becomes essential when:Teams need searchable and democratized data across business and engineering functions.Financial reporting and operational metrics require predictable, reportable pipelines.Data quality and trust are critical for decision-making.Experimentation and development need efficient workflows with strong tooling.Machine learning initiatives depend on well-organized and structured datasets.References:Data Platform Explained Part 1Data Platform Explained Part 2SPONSOR USGet your product in front of more than 1,000,000 tech professionals.Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.Space Fills Up Fast - Reserve TodayAd spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com.","published","strategy","content-ready","locked","deletedAt","isSyncedFromOwnBlog","original","duplicatesCount","duplicateVersion","originalArticleId","errorsCount","appearance","color","backgroundColor","shortSummary","summary","bullets","persons","places","words","sentiment","subject","meta","eventId","aee41ed0d53ec67c240339e0da75473c69fdd1080e744015f69fd56f5cf979f4","nostrRootEvent","event",{"_19":188,"_192":193,"_194":195,"_67":196,"_208":26,"_158":209,"_210":211},"sig","f8a7155c2f4e358e60ff2e0dc97fbabb1dc9699a63bc403547e79499664841fea3367d2e5ea0f5178c39e8257db40405fa83119d0958ca221c17e5e4d8318659","kind",1,[197,198,199,203,206],[185,144],[34,150],[200,201,31,202],"a","30023:cd5870305a2d73e3fe5b103df964a49da5ddcc026dc2556ffc8800172b7afc85:3e425cb9-f416-4464-a719-87a4e3b24dd7","mention",[204,205],"client","Layer3.press",[207,26,31],"p","pubkey","How Spotify Built Its Data Platform To Understand 1.4 Trillion Data Points\nThis evolution happened organically as Spotify’s products matured. With each new feature and business need, new data requirements emerged.\n\n\nnostr:naddr1qvzqqqr4gupzpn2cwqc95ttnu0l9kypal9j2f8d9mhxqymwz24hlezqqzu4h4ly9qqjrxef5xg6kxc3e94nrgvfk956rgd3594snwvfe95urwcf5v5ekyv35v3jrwz29zqh ","created_at",1763983791,"authorNPub","markdown","## [Must-Read Books for High-Performance System Design (Sponsored)](https://bit.ly/ScyllaDB_111125)\n\n_Access 4 PDFs on building and optimizing data-intensive applications._\n\n[](https://bit.ly/ScyllaDB_111125)\n\nRead experts’ proven strategies for optimizing data-intensive applications, database performance, and latency:\n\n- **Designing Data-Intensive Applications by Martin Kleppmann:**Discover new ways of thinking about your distributed data system challenges – with actionable insights for building scalable, high-performance solutions.\n \n- **Latency by Pekka Enberg:**Learn how to expertly diagnose latency problems and master the low-latency techniques that have been predominantly “tribal knowledge” until now.\n \n- **Database Performance at Scale: A Practical Guide:**Discover new ways to optimize database performance — and avoid common pitfalls – based on learnings from thousands of real-world database use cases.\n \n- **ScyllaDB in Action by Bo Ingram:**A practical guide to everything you need to know about ScyllaDB, from your very first queries to running it in a production environment.\n \n\nWhether you’re working on large-scale systems or designing distributed data architectures, these books will prepare you for what’s next.\n\n[Access Books for Free](https://bit.ly/ScyllaDB_111125)\n\n* * *\n\n_Disclaimer: The details in this post have been derived from the details shared online by the Spotify Engineering Team. All credit for the technical details goes to the Spotify Engineering Team. The links to the original articles and sources are present in the references section at the end of the post. We’ve attempted to analyze the details and provide our input about them. If you find any inaccuracies or omissions, please leave a comment, and we will do our best to fix them._\n\nEvery day, Spotify processes around 1.4 trillion data points. These data points come from millions of users around the world listening to music, creating playlists, and interacting with the app in different ways. Handling this volume of information is not something a few ad hoc systems can manage. It requires a robust, well-designed data platform that can reliably collect, process, and make sense of all this information.\n\nFrom payments to personalized recommendations to product experiments, almost every decision Spotify makes depends on data. This makes its data platform one of the most critical parts of the company’s overall technology stack.\n\nSpotify did not build this platform overnight. In the early years, data systems were more improvised. As the company grew, the number of users increased, and the complexity of business decisions became greater. Different teams began collecting data for their own needs. Over time, this led to a growing need for a centralized, structured, and productized platform that could support the entire company, not just individual teams.\n\nThe shift toward a formal data platform was driven by both business and technical factors:\n\n- **Business drivers:**Spotify needed high-quality, reliable data to support its core functions, such as financial reporting, advertising, product experimentation, and personalized recommendations. Data had to be consistent and trustworthy so that teams could make important decisions with confidence.\n \n- **Technical drivers:**The sheer scale of the data meant the company needed strong infrastructure that could collect events from millions of devices, process them quickly, and make them available in usable form. It also required clear data ownership, easy searchability, built-in quality checks, and compliance with privacy regulations.\n \n\nThis evolution happened organically as Spotify’s products matured. With each new feature and business need, new data requirements emerged. Over time, this learning process shaped a platform that now supports everything from real-time streaming analytics to large-scale experimentation and machine learning applications.\n\nIn this article, we will look at how Spotify built its data platform and the challenges it faced along the way.\n\n![](https://substackcdn.com/image/fetch/$s_!YDbp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0acc24c-ac0f-4f13-b497-5a28290ee334_1600x977.png)\n\n* * *\n\n## **[State of Trust: AI-driven attacks are getting more sophisticated (Sponsored)](https://bit.ly/Vanta_111125)**\n\n![](https://substackcdn.com/image/fetch/$s_!X5R9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5656d50-b8be-44b5-af7d-4bac17e597eb_1200x628.png)\n\nAI-driven attacks are getting bigger, faster, and more sophisticated—making risk much more difficult to contain. Without automation to respond quickly to AI threats, teams are forced to react without a plan in place.\n\nThis is according to Vanta’s newest State of Trust report, which surveyed 3,500 business and IT leaders across the globe.\n\nOne big change since last year’s report? Teams falling behind AI risks—and spending way more time and energy proving trust than building it.\n\n- 61% of leaders spend more time proving security rather than improving it\n \n- 59% note that AI risks outpace their expertise\n \n- But 95% say AI is making their security teams more effective\n \n\nGet the full report to learn how organizations are navigating these changes, and what early adopters are doing to stay ahead.\n\n[Download the report](https://bit.ly/Vanta_111125)\n\n* * *\n\n## Platform Composition and Evolution\n\nIn its early years, Spotify’s data operations were run by a single team. At one point, this team was managing Europe’s largest Hadoop cluster. For reference, Hadoop is an open-source framework used to store and process very large amounts of data across many computers.\n\nAt that stage, most data work was still centralized, and many processes were built manually or handled through shared infrastructure.\n\nAs Spotify grew, this model became too limited. The company needed more specialized tools and teams to handle the increasing scale and complexity. Over time, Spotify moved from that single Hadoop cluster to a multi-product data platform team. This new structure allowed them to separate the platform into clear functional areas, each responsible for a specific part of the data journey.\n\nAt the core of Spotify’s platform are three main building blocks that work together:\n\n- **Data Collection:**This part of the platform focuses on how data is gathered from millions of clients around the world. These “clients” include mobile apps, desktop apps, web browsers, and backend services. Every time a user plays a song, skips a track, adjusts the volume, or interacts with the app, an event is recorded. Spotify uses specialized tools and event delivery systems to collect these events in real time. This ensures that the data entering the platform is structured, consistent, and ready to be processed further.\n \n- **Data Processing:**Once the data is collected, it must be cleaned, transformed, and organized so it can be used effectively. This happens through data pipelines, which are automated workflows that process large amounts of information on a fixed schedule or in real time. These pipelines might do things like aggregating how many times a track was played in a day, linking a user’s activity to recommendation systems, or preparing data for financial reports. By scheduling and running thousands of pipelines, Spotify can provide up-to-date and accurate data to every team that needs it.\n \n- **Data Management:**This part ensures that the data is secure, private, and trustworthy. It includes data attribution, privacy controls, and security mechanisms to comply with regulations and internal governance rules. Data management also deals with data integrity, which means making sure the data is correct, consistent, and not corrupted during collection or processing.\n \n\nThese three areas are not isolated. They are deeply interconnected, forming a platform that is reliable, searchable, and easy to build on.\n\nSee the diagram below that shows the key components of the data platform:\n\n![](https://substackcdn.com/image/fetch/$s_!46FY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8539cc40-9072-4aee-a136-6d0e5aa5cc2a_1600x977.png)\n\nFor example, once data is collected and processed, it becomes searchable and can be used directly in other systems. One of the most important systems it powers is Spotify’s experimentation platform, called Confidence. This platform allows teams to run A/B tests and other experiments at scale, ensuring new product features are backed by real data before being fully launched.\n\n## Data Collection - Event Delivery at Scale\n\nOne of the most impressive parts of Spotify’s data platform is its data collection system.\n\nEvery time a user interacts with the app, whether by hitting play, searching for a song, skipping a track, or creating a playlist, an “event” is generated. Spotify’s data collection system is responsible for capturing and delivering all of these events reliably and at a massive scale.\n\nSpotify collects more than one trillion events every day. This is an extraordinary amount of data flowing in constantly from mobile apps, desktop clients, web players, and other connected devices. To handle this, the architecture of the data collection system has evolved through multiple iterations over the years. Early versions were much simpler, but as the user base and product features expanded, the system had to be redesigned to keep up with growing scale and complexity.\n\n### How Developers Work with Event Data\n\nAt Spotify, product teams don’t need to build custom infrastructure to collect events. Instead, they use client SDKs that make it easy to define what events should be collected.\n\nHere’s how the workflow looks:\n\n- A team defines a new event schema, which describes what kind of data will be collected and in what format. For example, a schema might specify that each “play” event should include the user ID (or an anonymous identifier), the song ID, the timestamp, and the device type.\n \n- Once the schema is defined, the platform automatically deploys all the infrastructure needed to handle that event. This includes:\n \n - Pub/Sub queues to reliably stream data as it arrives\n \n - Anonymization pipelines to remove or protect sensitive user information\n \n - Streaming jobs to process and route the data further down the platform\n \n- If a schema changes (for example, if a team adds a new field like “playlist ID”), the system automatically redeploys the affected components so the infrastructure stays in sync.\n \n\nSee the diagram below:\n\n![](https://substackcdn.com/image/fetch/$s_!aiSP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91b6c056-1476-43f6-85af-cec0ca8b633b_1600x882.png)\n\nThis level of automation is made possible through Kubernetes Operators. An operator is a special kind of software that manages complex applications running on Kubernetes, which is the container orchestration system Spotify uses to run its services.\n\nIn simple terms, operators allow Spotify to treat data infrastructure as code, so changes are applied quickly and reliably without manual work.\n\n### Built-in Privacy and Security\n\nHandling user data at this scale comes with serious privacy responsibilities. Spotify builds anonymization directly into its pipelines to ensure that sensitive information is protected before it ever reaches downstream systems.\n\nThey also use internal key-handling systems, which help control how and when certain pieces of data can be accessed or decrypted. This is essential for compliance with privacy regulations like GDPR and for maintaining user trust.\n\n### Centralization and Self-Service\n\nA key strength of Spotify’s data collection system is its ownership model. Instead of making the central infrastructure team responsible for every change, Spotify has designed the platform so that product teams can manage most of their own event data.\n\nThis means a team can:\n\n- Add or modify event schemas\n \n- Deploy event pipelines\n \n- Make small adjustments to how their data is processed\n \n\nThey can do all this without depending on the central platform team. This balance between centralization and self-service helps the platform scale to thousands of active users inside the company while keeping operational overhead low.\n\n### Breadth of Event Types\n\nThe platform currently handles around 1,800 different event types, each capturing different kinds of user interactions and system signals.\n\nThere are dedicated teams responsible for:\n\n- Maintaining the event delivery infrastructure\n \n- Managing the client SDKs that developers use\n \n- Building “journey datasets”, which combine multiple event streams into structured, meaningful timelines\n \n- Supporting the underlying infrastructure that keeps the system running smoothly\n \n\nThis massive, well-structured data collection layer forms the foundation of Spotify’s entire data platform. Without it, the rest of the platform (processing, management, analytics, and experimentation) would not be possible. It ensures that the right data is captured, secured, and made available at the right time for everything from recommendations to business decisions.\n\n## Data Management and Data Processing\n\nOnce data is collected, Spotify needs to transform it into something meaningful and trustworthy. This is where data processing and management come into play. At Spotify’s scale, this is a massive and complex operation that must run reliably every hour of every day.\n\nSpotify runs more than 38,000 active data pipelines on a regular schedule. Some run hourly, while others run daily, depending on the business need. A data pipeline is essentially an automated workflow that moves and transforms data from one place to another.\n\nFor example:\n\n- A pipeline might take raw event data from user streams and aggregate it into daily summaries of how many times each song was played.\n \n- Another pipeline might prepare datasets that support recommendation algorithms.\n \n- Yet another might generate financial reports.\n \n\nOperating this many pipelines requires a strong focus on scalability (handling growth efficiently), traceability (understanding where data comes from and how it changes), searchability (finding the right datasets quickly), and regulatory compliance (meeting privacy and data retention requirements).\n\n![](https://substackcdn.com/image/fetch/$s_!iVSR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6960ea09-1368-4ccb-b143-d53f048739c6_1600x1020.png)\n\n### The Execution Stack\n\nTo run these pipelines, Spotify uses a scheduler that automatically triggers workflows at the right times. These workflows are executed on:\n\n- BigQuery, Google’s cloud data warehouse, allows fast analysis of very large datasets.\n \n- Flink or Dataflow are frameworks for processing data streams or large batches of data in parallel.\n \n\nMost pipelines are written using Scio, a Scala library built on top of Apache Beam. Apache Beam is a framework that allows developers to write data processing jobs once and then run them on different underlying engines like Flink or Dataflow. This gives Spotify flexibility and helps standardize development across teams.\n\nEach pipeline in Spotify’s platform produces a data endpoint. An endpoint is essentially the final dataset that other teams and systems can use. These endpoints are treated like products with well-defined characteristics:\n\n- Explicit schemas so everyone knows exactly what data is included and in what format.\n \n- Multi-partitioning to organize data efficiently, often by time or other logical dimensions.\n \n- Retention policies specify how long the data is stored.\n \n- Access Control Lists (ACLs) are used to define who can view or modify the data.\n \n- Lineage tracking that records where the data came from and what transformations were applied.\n \n- Quality checks to catch errors early and ensure the data is trustworthy.\n \n\n### Platform as Code\n\nSpotify has built the platform in a way that allows engineers to define their pipelines and endpoints as code. This means all configuration and resource definitions are stored in the same place as the pipeline’s source code.\n\nAs mentioned, the system uses custom Kubernetes Operators to manage this. When developers push updates to their code repositories, the operators automatically deploy the necessary resources. This code ownership model ensures that the team responsible for a pipeline also controls its configuration and lifecycle, reducing dependency on centralized teams.\n\nWith tens of thousands of pipelines, keeping everything healthy and efficient is a major priority. Spotify has a strong operations and observability layer that includes:\n\n- Alerts for late pipelines, long-running jobs, or failures.\n \n- Endpoint health monitoring to make sure datasets are fresh and accurate.\n \n- Backstage integration to provide a single interface where teams can view and manage their data resources.\n \n\nBackstage is an internal developer portal that Spotify open-sourced. It brings together tools for monitoring, cost analysis, quality assurance, and documentation. Instead of searching across many systems, engineers can manage everything from one central place.\n\n## Conclusion\n\nSpotify’s data platform is a great example of how a company can evolve its infrastructure to meet growing business and technical demands. What began as a small team managing an on-premises Hadoop cluster has grown into a platform run by more than a hundred engineers, operating entirely on Google Cloud.\n\nThis transformation did not happen overnight. Spotify aligned its organizational needs with its technical investments, defined clear goals, and built strong feedback channels with its internal users. By starting small, iterating, and learning from each stage of growth, they created a platform that balances centralized infrastructure with self-service capabilities for product teams. Other organizations can take valuable lessons from this journey.\n\nA dedicated data platform becomes essential when:\n\n- Teams need searchable and democratized data across business and engineering functions.\n \n- Financial reporting and operational metrics require predictable, reportable pipelines.\n \n- Data quality and trust are critical for decision-making.\n \n- Experimentation and development need efficient workflows with strong tooling.\n \n- Machine learning initiatives depend on well-organized and structured datasets.\n \n\n**References:**\n\n- [Data Platform Explained Part 1](https://engineering.atspotify.com/2024/04/data-platform-explained)\n \n- [Data Platform Explained Part 2](https://engineering.atspotify.com/2024/5/data-platform-explained-part-ii)\n \n\n* * *\n\n## **SPONSOR US**\n\nGet your product in front of more than 1,000,000 tech professionals.\n\nOur newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.\n\nSpace Fills Up Fast - Reserve Today\n\nAd spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing**sponsorship@bytebytego.com.**","freeMarkdown","kind1EventId","kind30023EventId","8d743df39dfbaa2cd2bf39411f002736a0306cb737bdc475dd1139f7a184e0a1","kind30023",{"_19":218,"_192":221,"_194":222,"_67":223,"_208":26,"_158":237,"_210":238},"13cdf7b7ee8a8b0d8398f0a96f427d195c3bcd10428bc7ba9a65c5d3cc79db90d63ab5fa17a1a25791bdcedf0d54503df7690f44924603302dc4bc6428aa79c7",30023,[224,227,228,231,232,234,235,236],[225,226],"r","https://magda.syndichain.com/articles/3e425cb9-f416-4464-a719-87a4e3b24dd7",[32,144],[229,230],"published_at","1762878692",[34,150],[233,139],"d",[179,145],[204,205],[207,26,31],"How Spotify Built Its Data Platform To Understand 1.4 Trillion Data Points\n## [Must-Read Books for High-Performance System Design (Sponsored)](https://bit.ly/ScyllaDB_111125)\n\n_Access 4 PDFs on building and optimizing data-intensive applications._\n\n[](https://bit.ly/ScyllaDB_111125)\n\nRead experts’ proven strategies for optimizing data-intensive applications, database performance, and latency:\n\n- **Designing Data-Intensive Applications by Martin Kleppmann:**Discover new ways of thinking about your distributed data system challenges – with actionable insights for building scalable, high-performance solutions.\n \n- **Latency by Pekka Enberg:**Learn how to expertly diagnose latency problems and master the low-latency techniques that have been predominantly “tribal knowledge” until now.\n \n- **Database Performance at Scale: A Practical Guide:**Discover new ways to optimize database performance — and avoid common pitfalls – based on learnings from thousands of real-world database use cases.\n \n- **ScyllaDB in Action by Bo Ingram:**A practical guide to everything you need to know about ScyllaDB, from your very first queries to running it in a production environment.\n \n\nWhether you’re working on large-scale systems or designing distributed data architectures, these books will prepare you for what’s next.\n\n[Access Books for Free](https://bit.ly/ScyllaDB_111125)\n\n* * *\n\n_Disclaimer: The details in this post have been derived from the details shared online by the Spotify Engineering Team. All credit for the technical details goes to the Spotify Engineering Team. The links to the original articles and sources are present in the references section at the end of the post. We’ve attempted to analyze the details and provide our input about them. If you find any inaccuracies or omissions, please leave a comment, and we will do our best to fix them._\n\nEvery day, Spotify processes around 1.4 trillion data points. These data points come from millions of users around the world listening to music, creating playlists, and interacting with the app in different ways. Handling this volume of information is not something a few ad hoc systems can manage. It requires a robust, well-designed data platform that can reliably collect, process, and make sense of all this information.\n\nFrom payments to personalized recommendations to product experiments, almost every decision Spotify makes depends on data. This makes its data platform one of the most critical parts of the company’s overall technology stack.\n\nSpotify did not build this platform overnight. In the early years, data systems were more improvised. As the company grew, the number of users increased, and the complexity of business decisions became greater. Different teams began collecting data for their own needs. Over time, this led to a growing need for a centralized, structured, and productized platform that could support the entire company, not just individual teams.\n\nThe shift toward a formal data platform was driven by both business and technical factors:\n\n- **Business drivers:**Spotify needed high-quality, reliable data to support its core functions, such as financial reporting, advertising, product experimentation, and personalized recommendations. Data had to be consistent and trustworthy so that teams could make important decisions with confidence.\n \n- **Technical drivers:**The sheer scale of the data meant the company needed strong infrastructure that could collect events from millions of devices, process them quickly, and make them available in usable form. It also required clear data ownership, easy searchability, built-in quality checks, and compliance with privacy regulations.\n \n\nThis evolution happened organically as Spotify’s products matured. With each new feature and business need, new data requirements emerged. Over time, this learning process shaped a platform that now supports everything from real-time streaming analytics to large-scale experimentation and machine learning applications.\n\nIn this article, we will look at how Spotify built its data platform and the challenges it faced along the way.\n\n![](https://substackcdn.com/image/fetch/$s_!YDbp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb0acc24c-ac0f-4f13-b497-5a28290ee334_1600x977.png)\n\n* * *\n\n## **[State of Trust: AI-driven attacks are getting more sophisticated (Sponsored)](https://bit.ly/Vanta_111125)**\n\n![](https://substackcdn.com/image/fetch/$s_!X5R9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff5656d50-b8be-44b5-af7d-4bac17e597eb_1200x628.png)\n\nAI-driven attacks are getting bigger, faster, and more sophisticated—making risk much more difficult to contain. Without automation to respond quickly to AI threats, teams are forced to react without a plan in place.\n\nThis is according to Vanta’s newest State of Trust report, which surveyed 3,500 business and IT leaders across the globe.\n\nOne big change since last year’s report? Teams falling behind AI risks—and spending way more time and energy proving trust than building it.\n\n- 61% of leaders spend more time proving security rather than improving it\n \n- 59% note that AI risks outpace their expertise\n \n- But 95% say AI is making their security teams more effective\n \n\nGet the full report to learn how organizations are navigating these changes, and what early adopters are doing to stay ahead.\n\n[Download the report](https://bit.ly/Vanta_111125)\n\n* * *\n\n## Platform Composition and Evolution\n\nIn its early years, Spotify’s data operations were run by a single team. At one point, this team was managing Europe’s largest Hadoop cluster. For reference, Hadoop is an open-source framework used to store and process very large amounts of data across many computers.\n\nAt that stage, most data work was still centralized, and many processes were built manually or handled through shared infrastructure.\n\nAs Spotify grew, this model became too limited. The company needed more specialized tools and teams to handle the increasing scale and complexity. Over time, Spotify moved from that single Hadoop cluster to a multi-product data platform team. This new structure allowed them to separate the platform into clear functional areas, each responsible for a specific part of the data journey.\n\nAt the core of Spotify’s platform are three main building blocks that work together:\n\n- **Data Collection:**This part of the platform focuses on how data is gathered from millions of clients around the world. These “clients” include mobile apps, desktop apps, web browsers, and backend services. Every time a user plays a song, skips a track, adjusts the volume, or interacts with the app, an event is recorded. Spotify uses specialized tools and event delivery systems to collect these events in real time. This ensures that the data entering the platform is structured, consistent, and ready to be processed further.\n \n- **Data Processing:**Once the data is collected, it must be cleaned, transformed, and organized so it can be used effectively. This happens through data pipelines, which are automated workflows that process large amounts of information on a fixed schedule or in real time. These pipelines might do things like aggregating how many times a track was played in a day, linking a user’s activity to recommendation systems, or preparing data for financial reports. By scheduling and running thousands of pipelines, Spotify can provide up-to-date and accurate data to every team that needs it.\n \n- **Data Management:**This part ensures that the data is secure, private, and trustworthy. It includes data attribution, privacy controls, and security mechanisms to comply with regulations and internal governance rules. Data management also deals with data integrity, which means making sure the data is correct, consistent, and not corrupted during collection or processing.\n \n\nThese three areas are not isolated. They are deeply interconnected, forming a platform that is reliable, searchable, and easy to build on.\n\nSee the diagram below that shows the key components of the data platform:\n\n![](https://substackcdn.com/image/fetch/$s_!46FY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8539cc40-9072-4aee-a136-6d0e5aa5cc2a_1600x977.png)\n\nFor example, once data is collected and processed, it becomes searchable and can be used directly in other systems. One of the most important systems it powers is Spotify’s experimentation platform, called Confidence. This platform allows teams to run A/B tests and other experiments at scale, ensuring new product features are backed by real data before being fully launched.\n\n## Data Collection - Event Delivery at Scale\n\nOne of the most impressive parts of Spotify’s data platform is its data collection system.\n\nEvery time a user interacts with the app, whether by hitting play, searching for a song, skipping a track, or creating a playlist, an “event” is generated. Spotify’s data collection system is responsible for capturing and delivering all of these events reliably and at a massive scale.\n\nSpotify collects more than one trillion events every day. This is an extraordinary amount of data flowing in constantly from mobile apps, desktop clients, web players, and other connected devices. To handle this, the architecture of the data collection system has evolved through multiple iterations over the years. Early versions were much simpler, but as the user base and product features expanded, the system had to be redesigned to keep up with growing scale and complexity.\n\n### How Developers Work with Event Data\n\nAt Spotify, product teams don’t need to build custom infrastructure to collect events. Instead, they use client SDKs that make it easy to define what events should be collected.\n\nHere’s how the workflow looks:\n\n- A team defines a new event schema, which describes what kind of data will be collected and in what format. For example, a schema might specify that each “play” event should include the user ID (or an anonymous identifier), the song ID, the timestamp, and the device type.\n \n- Once the schema is defined, the platform automatically deploys all the infrastructure needed to handle that event. This includes:\n \n - Pub/Sub queues to reliably stream data as it arrives\n \n - Anonymization pipelines to remove or protect sensitive user information\n \n - Streaming jobs to process and route the data further down the platform\n \n- If a schema changes (for example, if a team adds a new field like “playlist ID”), the system automatically redeploys the affected components so the infrastructure stays in sync.\n \n\nSee the diagram below:\n\n![](https://substackcdn.com/image/fetch/$s_!aiSP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91b6c056-1476-43f6-85af-cec0ca8b633b_1600x882.png)\n\nThis level of automation is made possible through Kubernetes Operators. An operator is a special kind of software that manages complex applications running on Kubernetes, which is the container orchestration system Spotify uses to run its services.\n\nIn simple terms, operators allow Spotify to treat data infrastructure as code, so changes are applied quickly and reliably without manual work.\n\n### Built-in Privacy and Security\n\nHandling user data at this scale comes with serious privacy responsibilities. Spotify builds anonymization directly into its pipelines to ensure that sensitive information is protected before it ever reaches downstream systems.\n\nThey also use internal key-handling systems, which help control how and when certain pieces of data can be accessed or decrypted. This is essential for compliance with privacy regulations like GDPR and for maintaining user trust.\n\n### Centralization and Self-Service\n\nA key strength of Spotify’s data collection system is its ownership model. Instead of making the central infrastructure team responsible for every change, Spotify has designed the platform so that product teams can manage most of their own event data.\n\nThis means a team can:\n\n- Add or modify event schemas\n \n- Deploy event pipelines\n \n- Make small adjustments to how their data is processed\n \n\nThey can do all this without depending on the central platform team. This balance between centralization and self-service helps the platform scale to thousands of active users inside the company while keeping operational overhead low.\n\n### Breadth of Event Types\n\nThe platform currently handles around 1,800 different event types, each capturing different kinds of user interactions and system signals.\n\nThere are dedicated teams responsible for:\n\n- Maintaining the event delivery infrastructure\n \n- Managing the client SDKs that developers use\n \n- Building “journey datasets”, which combine multiple event streams into structured, meaningful timelines\n \n- Supporting the underlying infrastructure that keeps the system running smoothly\n \n\nThis massive, well-structured data collection layer forms the foundation of Spotify’s entire data platform. Without it, the rest of the platform (processing, management, analytics, and experimentation) would not be possible. It ensures that the right data is captured, secured, and made available at the right time for everything from recommendations to business decisions.\n\n## Data Management and Data Processing\n\nOnce data is collected, Spotify needs to transform it into something meaningful and trustworthy. This is where data processing and management come into play. At Spotify’s scale, this is a massive and complex operation that must run reliably every hour of every day.\n\nSpotify runs more than 38,000 active data pipelines on a regular schedule. Some run hourly, while others run daily, depending on the business need. A data pipeline is essentially an automated workflow that moves and transforms data from one place to another.\n\nFor example:\n\n- A pipeline might take raw event data from user streams and aggregate it into daily summaries of how many times each song was played.\n \n- Another pipeline might prepare datasets that support recommendation algorithms.\n \n- Yet another might generate financial reports.\n \n\nOperating this many pipelines requires a strong focus on scalability (handling growth efficiently), traceability (understanding where data comes from and how it changes), searchability (finding the right datasets quickly), and regulatory compliance (meeting privacy and data retention requirements).\n\n![](https://substackcdn.com/image/fetch/$s_!iVSR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6960ea09-1368-4ccb-b143-d53f048739c6_1600x1020.png)\n\n### The Execution Stack\n\nTo run these pipelines, Spotify uses a scheduler that automatically triggers workflows at the right times. These workflows are executed on:\n\n- BigQuery, Google’s cloud data warehouse, allows fast analysis of very large datasets.\n \n- Flink or Dataflow are frameworks for processing data streams or large batches of data in parallel.\n \n\nMost pipelines are written using Scio, a Scala library built on top of Apache Beam. Apache Beam is a framework that allows developers to write data processing jobs once and then run them on different underlying engines like Flink or Dataflow. This gives Spotify flexibility and helps standardize development across teams.\n\nEach pipeline in Spotify’s platform produces a data endpoint. An endpoint is essentially the final dataset that other teams and systems can use. These endpoints are treated like products with well-defined characteristics:\n\n- Explicit schemas so everyone knows exactly what data is included and in what format.\n \n- Multi-partitioning to organize data efficiently, often by time or other logical dimensions.\n \n- Retention policies specify how long the data is stored.\n \n- Access Control Lists (ACLs) are used to define who can view or modify the data.\n \n- Lineage tracking that records where the data came from and what transformations were applied.\n \n- Quality checks to catch errors early and ensure the data is trustworthy.\n \n\n### Platform as Code\n\nSpotify has built the platform in a way that allows engineers to define their pipelines and endpoints as code. This means all configuration and resource definitions are stored in the same place as the pipeline’s source code.\n\nAs mentioned, the system uses custom Kubernetes Operators to manage this. When developers push updates to their code repositories, the operators automatically deploy the necessary resources. This code ownership model ensures that the team responsible for a pipeline also controls its configuration and lifecycle, reducing dependency on centralized teams.\n\nWith tens of thousands of pipelines, keeping everything healthy and efficient is a major priority. Spotify has a strong operations and observability layer that includes:\n\n- Alerts for late pipelines, long-running jobs, or failures.\n \n- Endpoint health monitoring to make sure datasets are fresh and accurate.\n \n- Backstage integration to provide a single interface where teams can view and manage their data resources.\n \n\nBackstage is an internal developer portal that Spotify open-sourced. It brings together tools for monitoring, cost analysis, quality assurance, and documentation. Instead of searching across many systems, engineers can manage everything from one central place.\n\n## Conclusion\n\nSpotify’s data platform is a great example of how a company can evolve its infrastructure to meet growing business and technical demands. What began as a small team managing an on-premises Hadoop cluster has grown into a platform run by more than a hundred engineers, operating entirely on Google Cloud.\n\nThis transformation did not happen overnight. Spotify aligned its organizational needs with its technical investments, defined clear goals, and built strong feedback channels with its internal users. By starting small, iterating, and learning from each stage of growth, they created a platform that balances centralized infrastructure with self-service capabilities for product teams. Other organizations can take valuable lessons from this journey.\n\nA dedicated data platform becomes essential when:\n\n- Teams need searchable and democratized data across business and engineering functions.\n \n- Financial reporting and operational metrics require predictable, reportable pipelines.\n \n- Data quality and trust are critical for decision-making.\n \n- Experimentation and development need efficient workflows with strong tooling.\n \n- Machine learning initiatives depend on well-organized and structured datasets.\n \n\n**References:**\n\n- [Data Platform Explained Part 1](https://engineering.atspotify.com/2024/04/data-platform-explained)\n \n- [Data Platform Explained Part 2](https://engineering.atspotify.com/2024/5/data-platform-explained-part-ii)\n \n\n* * *\n\n## **SPONSOR US**\n\nGet your product in front of more than 1,000,000 tech professionals.\n\nOur newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.\n\nSpace Fills Up Fast - Reserve Today\n\nAd spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing**sponsorship@bytebytego.com.**\n https://magda.syndichain.com/articles/3e425cb9-f416-4464-a719-87a4e3b24dd7 magda magda magda magda",1762878692,"publishedToNostr","htmlContentKind30023","

Must-Read Books for High-Performance System Design (Sponsored)

Access 4 PDFs on building and optimizing data-intensive applications.

Read experts’ proven strategies for optimizing data-intensive applications, database performance, and latency:

\n
**Designing Data-Intensive Applications by Martin Kleppmann:**Discover new ways of thinking about your distributed data system challenges – with actionable insights for building scalable, high-performance solutions.
\n
\n
**Latency by Pekka Enberg:**Learn how to expertly diagnose latency problems and master the low-latency techniques that have been predominantly “tribal knowledge” until now.
\n
\n
**Database Performance at Scale: A Practical Guide:**Discover new ways to optimize database performance — and avoid common pitfalls – based on learnings from thousands of real-world database use cases.
\n
\n
**ScyllaDB in Action by Bo Ingram:**A practical guide to everything you need to know about ScyllaDB, from your very first queries to running it in a production environment.
\n

Whether you’re working on large-scale systems or designing distributed data architectures, these books will prepare you for what’s next.

Access Books for Free

The shift toward a formal data platform was driven by both business and technical factors:

\n
**Business drivers:**Spotify needed high-quality, reliable data to support its core functions, such as financial reporting, advertising, product experimentation, and personalized recommendations. Data had to be consistent and trustworthy so that teams could make important decisions with confidence.
\n
\n
**Technical drivers:**The sheer scale of the data meant the company needed strong infrastructure that could collect events from millions of devices, process them quickly, and make them available in usable form. It also required clear data ownership, easy searchability, built-in quality checks, and compliance with privacy regulations.
\n

In this article, we will look at how Spotify built its data platform and the challenges it faced along the way.

$\"\"$

State of Trust: AI-driven attacks are getting more sophisticated (Sponsored)

$\"\"$

This is according to Vanta’s newest State of Trust report, which surveyed 3,500 business and IT leaders across the globe.

One big change since last year’s report? Teams falling behind AI risks—and spending way more time and energy proving trust than building it.

\n
61% of leaders spend more time proving security rather than improving it
\n
\n
59% note that AI risks outpace their expertise
\n
\n
But 95% say AI is making their security teams more effective
\n

Get the full report to learn how organizations are navigating these changes, and what early adopters are doing to stay ahead.

Download the report

Platform Composition and Evolution

At that stage, most data work was still centralized, and many processes were built manually or handled through shared infrastructure.

At the core of Spotify’s platform are three main building blocks that work together:

\n
**Data Collection:**This part of the platform focuses on how data is gathered from millions of clients around the world. These “clients” include mobile apps, desktop apps, web browsers, and backend services. Every time a user plays a song, skips a track, adjusts the volume, or interacts with the app, an event is recorded. Spotify uses specialized tools and event delivery systems to collect these events in real time. This ensures that the data entering the platform is structured, consistent, and ready to be processed further.
\n
\n
**Data Processing:**Once the data is collected, it must be cleaned, transformed, and organized so it can be used effectively. This happens through data pipelines, which are automated workflows that process large amounts of information on a fixed schedule or in real time. These pipelines might do things like aggregating how many times a track was played in a day, linking a user’s activity to recommendation systems, or preparing data for financial reports. By scheduling and running thousands of pipelines, Spotify can provide up-to-date and accurate data to every team that needs it.
\n
\n
**Data Management:**This part ensures that the data is secure, private, and trustworthy. It includes data attribution, privacy controls, and security mechanisms to comply with regulations and internal governance rules. Data management also deals with data integrity, which means making sure the data is correct, consistent, and not corrupted during collection or processing.
\n

These three areas are not isolated. They are deeply interconnected, forming a platform that is reliable, searchable, and easy to build on.

See the diagram below that shows the key components of the data platform:

$\"\"$

Data Collection - Event Delivery at Scale

One of the most impressive parts of Spotify’s data platform is its data collection system.

How Developers Work with Event Data

At Spotify, product teams don’t need to build custom infrastructure to collect events. Instead, they use client SDKs that make it easy to define what events should be collected.

Here’s how the workflow looks:

\n
A team defines a new event schema, which describes what kind of data will be collected and in what format. For example, a schema might specify that each “play” event should include the user ID (or an anonymous identifier), the song ID, the timestamp, and the device type.
\n
\n
Once the schema is defined, the platform automatically deploys all the infrastructure needed to handle that event. This includes:
\n
- \n
  Pub/Sub queues to reliably stream data as it arrives
  \n
- \n
  Anonymization pipelines to remove or protect sensitive user information
  \n
- \n
  Streaming jobs to process and route the data further down the platform
  \n
\n
\n
If a schema changes (for example, if a team adds a new field like “playlist ID”), the system automatically redeploys the affected components so the infrastructure stays in sync.
\n

See the diagram below:

$\"\"$

In simple terms, operators allow Spotify to treat data infrastructure as code, so changes are applied quickly and reliably without manual work.

Built-in Privacy and Security

Centralization and Self-Service

This means a team can:

\n
Add or modify event schemas
\n
\n
Deploy event pipelines
\n
\n
Make small adjustments to how their data is processed
\n

Breadth of Event Types

The platform currently handles around 1,800 different event types, each capturing different kinds of user interactions and system signals.

There are dedicated teams responsible for:

\n
Maintaining the event delivery infrastructure
\n
\n
Managing the client SDKs that developers use
\n
\n
Building “journey datasets”, which combine multiple event streams into structured, meaningful timelines
\n
\n
Supporting the underlying infrastructure that keeps the system running smoothly
\n

Data Management and Data Processing

For example:

\n
A pipeline might take raw event data from user streams and aggregate it into daily summaries of how many times each song was played.
\n
\n
Another pipeline might prepare datasets that support recommendation algorithms.
\n
\n
Yet another might generate financial reports.
\n

$\"\"$

The Execution Stack

To run these pipelines, Spotify uses a scheduler that automatically triggers workflows at the right times. These workflows are executed on:

\n
BigQuery, Google’s cloud data warehouse, allows fast analysis of very large datasets.
\n
\n
Flink or Dataflow are frameworks for processing data streams or large batches of data in parallel.
\n

\n
Explicit schemas so everyone knows exactly what data is included and in what format.
\n
\n
Multi-partitioning to organize data efficiently, often by time or other logical dimensions.
\n
\n
Retention policies specify how long the data is stored.
\n
\n
Access Control Lists (ACLs) are used to define who can view or modify the data.
\n
\n
Lineage tracking that records where the data came from and what transformations were applied.
\n
\n
Quality checks to catch errors early and ensure the data is trustworthy.
\n

Platform as Code

With tens of thousands of pipelines, keeping everything healthy and efficient is a major priority. Spotify has a strong operations and observability layer that includes:

\n
Alerts for late pipelines, long-running jobs, or failures.
\n
\n
Endpoint health monitoring to make sure datasets are fresh and accurate.
\n
\n
Backstage integration to provide a single interface where teams can view and manage their data resources.
\n

Conclusion

A dedicated data platform becomes essential when:

\n
Teams need searchable and democratized data across business and engineering functions.
\n
\n
Financial reporting and operational metrics require predictable, reportable pipelines.
\n
\n
Data quality and trust are critical for decision-making.
\n
\n
Experimentation and development need efficient workflows with strong tooling.
\n
\n
Machine learning initiatives depend on well-organized and structured datasets.
\n

References:

\n
Data Platform Explained Part 1
\n
\n
Data Platform Explained Part 2
\n

SPONSOR US

Get your product in front of more than 1,000,000 tech professionals.

Space Fills Up Fast - Reserve Today

Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailingsponsorship@bytebytego.com.

","pubDate","2025-11-11T16:31:32.000Z","modifiedAt","2025-11-24T11:29:12.594Z","storyId","likes","zaps","reposts","comments","tweets",[],"relatedArticles",[],"actionData","errors"]