Terraform Module, Used for Running Kubeflow ML Pipelines, Open-Sourced by Spotify

Spotify’s Terraform module, which is used for creating Google Kubernetes Engine (GKE) clusters to run Kubeflow machine learning (ML) pipelines, has been open-sourced by the company. This would help Spotify to do 7x more experiments in developing end-to-end machine learning solutions and deploying these solutions to the market faster than ever before.

Terraform Module, Used for Running Kubeflow ML Pipelines, Open-Sourced by Spotify

What is Spotify’s Paved Road Concept?

Spotify’s Discover Weekly service uses machine learning to provide music recommendations to users. The company initially used Scala language and other frameworks, most of which have been open-sourced by Spotify. The music streaming company faced several drawbacks for choosing these tools and frameworks as they didn’t scale well.

It started the “Paved Road” concept to address the data interface problems in their ML workflow. They started using TFRecord and tf.Example formats by Google’s TensorFlow Extended (TFX) and Tensorflow Data Validation (TFDV). However, these components and tools lacked the orchestration framework. Spotify then moved to Kubeflow Pipelines in which the clusters are configured using the Terraform module. Now, the company is using Kubeflow Pipelines for managing its entire machine learning lifecycle.

Benefits of Open-Sourcing The Terraform Module

Open-sourcing the Terraform module would enable Spotify to develop and release machine learning-based services with faster time-to-market delivery.

If you want to leverage the power of emerging technologies like machine learning, artificial intelligence, big data, cloud computing, and IoT for the growth of your business, then contact our experts to discuss your needs.

To know how Source Soft can help your business grow!

Contact Us






Copyright Disclaimer: Unauthorized use, publication, and/or duplication of our website and blog content without express and written permission from this site’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Source Soft Solutions with appropriate and specific direction to the original content.