[Edit 1] This article has been edited after initial release for clarity. Both the tl;dr and the end section have added information.

[Edit 2] This hypothesis was actually wrong and thank you to everyone who commented!

Here’s a full explanation of where I went wrong. I want to address this mistake as I realize it might have a meaningful impact on someone's financial position.

I’m not editing the actual article except where absolutely necessary so it doesn’t look like I’m covering up the mistake—I want to address it. Instead, I’ve included the important information below.

There is one takeaway this article provides that definitely holds true:

• Local models are far more capable than they’re given credit for, even for coding.

It also explains the process of setting up a local coding model and technical information about doing so which is helpful for anyone wanting to set up a local coding model. I would still recommend doing so.

But do I want someone reading this to immediately drop their coding subscription and buy a maxed out MacBook Pro? No, and for that reason I need to correct my hypothesis from ‘Yes, with caveats’ to ‘No’.

This article was not an empirical assessment, but should have been to make these claims. Here’s where I went wrong:

• While local models can likely complete ~90% of the software development tasks that something like Claude Code can, the last 10% is the most important. When it comes to your job, that last 10% is worth paying more for to get that last bit of performance.

• I realized I looked at this more from the angle of a hobbiest paying for these coding tools. Someone doing little side projects—not someone in a production setting. I did this because I see a lot of people signing up for $100/mo or $200/mo coding subscriptions for personal projects when they likely don’t need to. I would not recommend running local models as a company instead of giving employees access to a tool like Claude Code.

• While larger local models are very capable, as soon as you run other development tools (Docker, etc.) that also eat into your RAM, your model needs to be much smaller and becomes a lot less capable. I didn’t factor this in in my experiment.

So, really, the takeaway should be that these are incredible supplemental models to frontier models when coding and could potentially save you on your subscription by dropping it down a tier, but practically they’re not worth the effort in situations that might affect your livelihood.

Exactly a month ago, I made a hypothesis: Instead of paying $100/mo+ for an AI coding subscription, my money would be better spent upgrading my hardware so I can run local coding models at a fraction of the price (and have better hardware too!).

So, to create by far the most expensive article I’ve ever written, I put my money where my mouth is and bought a MacBook Pro with 128 GB of RAM to get to work. My idea was simple: Over the life of the MacBook I’d recoup the costs of it by not paying for an AI coding subscription.

After weeks of experimenting and setting up local AI models and coding tools, I’ve come to the conclusion that my hypothesis was correct, with nuance, not correct [see edit 2 above] which I’ll get into later in this article.

In this article, we cover:

• Why local models matter and the benefits they provide.

• How to view memory usage and make estimates for which models can run on your machine and the RAM demands for coding applications.

• Walk through setting up your own local coding model and tool step-by-step.

Don’t worry if you don’t have a high-RAM machine! You can still follow this guide. I’ve included some models to try out with a lower memory allotment. I think you’ll be surprised at how performant even the smallest of models is. In fact, there hasn’t really been a time during this experiment that I’ve been disappointed with model performance.

If you’re only here for the local coding tool setup, skip to the section at the bottom. I’ve even included a link to my modelfiles in that section to make setup even easier for you. Otherwise, let’s get into what you need to know.

tl;dr:

• Local coding models are very capable. Using the right model and the right tooling feels only half a generation behind the frontier cloud tools. I would say that for about 90% of developer work local models are more than sufficient. Even small 7B parameter models can be very capable. [Edited to add in this next part] Local models won’t compete with frontier models at the peak of performance, but can complete many coding tasks just as well for a fraction of the cost. They’re worth running to bring costs down on plenty of tasks but potentially not worth using if there’s a free tier available that performs better.

• Tools matter a lot. This is where I experienced the most disappointment. I tried many different tools with many different models and spent a lot of time tinkering. I ran into situations where the models wouldn’t call tools properly or their thinking traces wouldn’t close. Both of these rendered the tool essentially useless. Currently, tooling seems very finicky and if there’s anything developers need to be successful, it’s good tools.

• There’s a lot to consider when you’re actually working within hardware constraints. We take the tooling set up for us in the cloud for granted. When setting up local models, I had to think a lot about trade-offs in performance versus memory usage, how different tools compared and affected performance, nuances in types of models, how to quantize, and other user-facing factors such as time-to-first-token and tokens per second.

• Google threw a wrench into my hypothesis. The local setup is almost a no-brainer when compared to a $100/mo+ subscription. Compared to free or nearly-free tooling (such as Gemini CLI, Jules, or Antigravity) there isn’t quite as strong of a monetary justification to spend more on hardware. There are benefits to local models outside of code, though, and I discuss those below.

If the tl;dr was helpful, don’t forget to subscribe to get more in your inbox.

Subscribe now

Why local models?

You might wonder why local models are worth investing in at all. The obvious answer is cost. By using your own hardware, you don’t need to pay a subscription fee to a cloud provider for your tool. There are also a few less obvious and underrated reasons that make local models useful.

First: Reliability. Each week there seems to be complaints about performance regression within AI coding tools. Many speculate companies are pulling tricks to save resources that hurt model performance. With cloud providers, you’re at the mercy of the provider for when this happens. With local models, this only happens when you cause it to.

Second: Local models can apply to far more applications. Just the other day I was having a discussion with my dad about AI tooling he could use to streamline his work. His job requires studying a lot of data—a perfect application for an LLM-based tool—but his company blocks tools like Gemini and ChatGPT because a lot of this analysis is done on intellectual property. Unfortunately, he isn’t provided a suitable alternative to use.

With a local model, he wouldn’t have to worry about these IP issues. He could run his analyses without data ever leaving his machine. Of course, any tool calling would also need to ensure data never leaves the machine, but local models get around one of the largest hurdles for useful enterprise AI adoption. Running models on a local machine opens up an entire world of privacy- and security-centric AI applications that are expensive for cloud providers to provide.

Finally: Availability. Local models are available to you as long as your machine is. This means no worrying about your provider being down or rate limiting you due to high traffic. It also means using AI coding tools on planes or in other situations where internet access is locked down (think highly secure networks).

While local models do provide significant cost savings, the flexibility and reliability they provide can be even more valuable.

Understanding memory

To get going with local models you must understand the memory needed to run them on your machine. Obviously, if you have more memory you’ll be able to run better models, but understanding the nuances of that memory management will help you pick out the right model for your use case.

Local AI has two parts that eat up your memory: The model itself and the model’s context window.

The actual model has billions of parameters and all those parameters need to fit into your memory at once. Excellent local coding models start at around 30 billion (30B, for short) parameters in size. By default, these models use 16 bits to represent parameters. At 16 bits with 30B parameters, a model will take 60 GB of space in RAM (16 bits = 2 bytes per parameter, 30 billion parameters = 60 billion bytes which equals about 60 GB).

The second (and potentially larger) memory consuming part of local AI is the model’s context window. This is the model inputs and outputs that are stored so the model can reference them in future requests. This gives the model memory.

When coding with AI, we prefer this window to be as large as it can because we need to fit our codebase (or pieces of it) within our context window. This means we target a context window of 64,000 tokens or larger. All of these tokens will also be stored in RAM.

The important thing to understand about context windows is that the memory requirement per-token for a model depends on the size of that model. Models with more parameters tend to have large architectures (more hidden layers and larger dimensions to those layers). Larger architectures mean the model must store more information for each token within its key-value cache (context window) because it stores information for each token for each layer.

This means choosing an 80B parameter model over a 30B parameter model requires more memory for the model itself and also more memory for the same size context window. For example, a 30B parameter model might have a hidden dimension of 5120 with 64 layers while an 80B model has a hidden dimension of 8192 with 80 layers. Doing some back-of-the-napkin math shows us that the larger model requires approximately 2x more RAM to maintain the same context window as the 30B parameter model (see formula below).

Luckily, there are tricks to better manage memory. First, there are architectural changes that can be made to make model inference more efficient so it requires less memory. The model we set up at the end of this article uses Hybrid Attention which enables a much smaller KV cache enabling us to fit our model and context window in less memory. I won’t get into more detail in this article, but you can read more about that model and how it works here.

The second trick is quantizing the values you’re working with. Quantization means converting a continuous set of values into a smaller amount of distinct values. In our case, that means taking a set of numbers represented by a certain number of bits (16, for example) and reducing it to a set of numbers represented by fewer bits (8, for example). To put it simply, in our case we’re converting the numbers representing our model to a smaller bit representation to save memory while keeping the value representations within the model relatively equal.

You can quantize both your model weights and the values stored in your context window. When you quantize your model weights, you “remove intelligence” from the model because it’s less precise in its representation of innate information. I’ve also found the performance hit when going from 16 to 8 bits within the model to be much less than 8 to 4.

We can also quantize the values in our context window to reduce its memory requirement. This means we’re less precisely representing the model’s memory. Generally speaking, KV cache (context window) quantization is considered more destructive to model performance than weight quantization because it causes the model to forget details in long reasoning traces. Thus, you should test quantizing the KV cache to ensure it doesn’t degrade model performance for your specific task.

In reality, like the rest of machine learning, optimizing local model performance is an experimentation process and real-world machine learning requires understanding the practical limitations and capabilities of models when applied to specific applications.

Here are a few more factors to understand when setting up a local coding model on your hardware:

Instruct versus non-instruct

Instruct models are post-trained to be well-suited for chat-based interactions. They’re given chat pairings in their training to be optimized for excellent back-and-forth chat output. Non-instruct models are still trained LLMs, but focus on next-token prediction instead of chatting with a user. For our case, when using a chat-based coding tool (CLI or chat agent in your IDE) we need to use an instruct model. If you’re setting up an autocomplete model, you’ll want to find a model specifically post-trained for it (such as Qwen2.5-Coder-Base or DeepSeek-Coder-V2).

Serving tools

You need a tool to serve your local LLM for your coding tool to send it requests. On a MacBook, there are two primary options: MLX and Ollama.

Ollama is the industry standard and works on non-Mac hardware. It’s a great serving setup on top of llama.cpp that makes model serving almost plug-and-play. Users can download model weights from Ollama easily and can configure modelfiles with custom parameters for serving. Ollama can also serve a model once and make it available to multiple tools.

MLX is a Mac-specific framework for machine learning that is optimized specifically for Mac hardware. It also retrieves models for the user from a community collection. I’ve found Ollama to be very reliable in its model catalog, while MLX’s catalog is community sourced and can sometimes be missing specific models. Models are sourced from the community so a user can convert a model to MLX format themselves. MLX requires a bit more setup on the user’s end, but serves models faster because it doesn’t have a layer providing the niceties of Ollama on top of it.

Either of these is great, but I chose MLX to maximize what I can get with my RAM, but Ollama is probably the more beginner-friendly tool here.

Time-to-first-token and tokens per second

In real-world LLM applications it’s important that the model is able to serve its first token for a request in a reasonable amount of time and continue serving tokens at a speed that enables the user to use the model for its given purpose. If we have a high-performance model running locally, but it only serves a few tokens per second, it wouldn’t be useful for coding.

This is something taken for granted with cloud-hosted models that is a real consideration when working locally on constrained hardware. Another reason I chose MLX as my serving platform is because it served tokens up to 20% faster than Ollama. In reality, Ollama served tokens fast enough so I don’t think using MLX is necessary specifically for this reason for the models I tried.

Performance trade-offs

There are many ways to optimize local models and save RAM. It’s difficult to know which optimization method works best and the impact each has on a model especially when using them in tandem with other methods.

The right optimization method also depends on the application. In my experience, I find it best to prioritize larger models with more aggressive model quantization over smaller models with more precise model weights. Since our application is coding, I would also prioritize a less-quantized KV cache and using a smaller model to ensure reasoning works properly while not sacrificing the size of our context window.

Coding tools

There are many tools to code with local models and I suggest trying until you find one you like. Some top recommendations are OpenCode, Aider, Qwen Code, Roo Code, and Continue. Make sure to use a tool compatible with OpenAI’s API standard. While this should be most tools, this ensures a consistent model/tool connection. This makes it easier to switch between tools and models as needed.

Getting set up

I’ll spare you the trial and error I experienced getting this set up. The one thing I learned is that tooling matters a lot. Not all coding tools are created equal and not all of the models interact with tools equally. I experienced many times where tool calling or even running a tool at all was broken. I also had to tinker quite a bit with many of them to get them to work.

If you’re a PC enthusiast, an apt comparison to setting up local coding tools versus using the cloud offerings available is the difference between setting up a MacBook versus a Linux Laptop. With the Linux laptop, you might get well through the distro installation only to find that the drivers for your trackpad aren’t yet supported. Sometimes it felt like that with local models and hooking them to coding tools.

For my tool, I ended up going with Qwen Code. It was pretty plug-and-play as it’s a fork of Gemini CLI. It supports the OpenAI compatibility standard so I can easily sub in different models and affords me all of the niceties built into Gemini CLI that I’m familiar with using. I also know it’ll be supported because both the Qwen team and Google DeepMind are behind the tool. The tool is also open source so anyone can support it as needed.

For models, I focused on GPT-OSS and Qwen3 models since they were around the size I was looking for and had great reviews for coding. I ended up deciding to use Qwen3-Coder models because I found it performed best and because GPT-OSS frequently gave me “I cannot fulfill this request” responses when I asked it to build features.

I decided to serve my local models on MLX, but if you’re using a non-Mac device give Ollama a shot. A MacBook is an excellent machine for serving local models because of its unified memory architecture. This means the RAM can be allotted to the CPU or GPU as needed. MacBooks can also be configured with a ton of RAM. For serving local coding models, more is always better.

I’ve shared my modelfiles repo for you to reference and use as needed. I’ve got a script set up that automates much of the below process. Feel free to fork it and create your own modelfiles or star it to come back later.

1. Install MLX or download Ollama (the rest of this guide will continue with MLX but details for serving on Ollama can be found here).

2. Increase the VRAM limitation on your MacBook. macOS will automatically limit VRAM to 75% of the total RAM. We want to use more than that. Run sudo sysctl iogpu.wired_limit_mb=110000 in your terminal to set this up (adjust the mb setting according to the RAM on your MacBook). This needs to be set each time you restart your MacBook.

3. Run pip install -U mlx-lm to install MLX for serving community models.

4. Serve the model as an OpenAI compatible API using python -m mlx_lm.server --model mlx-community/Qwen3-Next-80B-A3B-Instruct-8bit. This command both runs the server and downloads the model for you if you haven’t yet. This particular model is what I’m using with 128GB of RAM. If you have less RAM, check out smaller models such as mlx-community/Qwen3-4B-Instruct-2507-4bit (8 GB RAM), mlx-community/Qwen2.5-14B-Instruct-4bit (16 GB RAM), mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit (32 GB RAM), or mlx-community/Qwen3-Next-80B-A3B-Instruct-4bit (64-96 GB RAM).

5. Download Qwen Code. You might need to install Node Package Manager for this. I recommend using Node Version Manager (nvm) for managing your npm version.

6. Set up your tool to access an OpenAI compatible API by entering the following settings:

   1. Base URL: http://localhost:8080/v1 (should be the default MLX serves your model at)

   2. API Key: mlx

   3. Model Name: mlx-community/Qwen3-Next-80B-A3B-Instruct-8bit (or whichever model you chose).

7. Voila! Your coding model tool should be working with your local coding model.

I recommend opening Activity Monitor on your Mac to monitor memory usage. I’ve had cases where I thought a model should fit within my memory allotment but it didn’t and I ended up using a lot of swap memory. When this happens your model will run very slowly.

One tip I have for using local coding models: Focus on managing your context. This is a great skill even with cloud-based models. People tend to YOLO their chats and fill their context window, but I’ve found greater performance by ensuring that just what my model needs is sitting in my context window. This is even more important with local models that may need an extra boost in performance and are limited in their context.

Was my hypothesis correct?

My original hypothesis was: Instead of paying $100/mo+ for an AI coding subscription, my money would be better spent upgrading my hardware so I can run local coding models at a fraction of the price.

I would argue that—yes!—no [see edit 2 above], it is correct. If we crunch the numbers, a MacBook with 128 GB is $4700 plus tax. If I spend $100/mo for 5 years, a coding subscription would cost $6000 in that same amount of time. Not only do I save money, but I also get a much more capable machine for anything else I want to do with it.

[This paragraph was added in after initial release of this article] It’s important to note that local models will not reach the peak performance of frontier models; however, they will likely be able to do most tasks just as well. The value of using a local model doesn’t come from raw performance, but from supplementing the cost of higher performance models. A local model could very well let you drop your subscription tier for a frontier coding tool or utilize a free tier as needed for better performance and run the rest of your tasks for free.

It’s also important to note that local models are only going to get better and smaller. This is the worst your local coding model will perform. I also wouldn’t be surprised if cloud-based AI coding tools get more expensive. If you figure you’re using greater than the $100/mo tier right now or that the $100/mo tier will cost $200/mo in the future, the purchase is a no-brainer. It’s just difficult to stomach the upfront cost.

From a performance standpoint, I would say the maximum model running on my 128 GB RAM MacBook right now feels about half a generation behind the frontier coding tools. That’s excellent, but something to keep in mind as that half a generation might matter to you.

One wrench thrown into my experiment is how much free quota Google hands out with their different AI coding tools. It’s easy to purchase expensive hardware when it saves you money in the long run. It’s much more difficult when the alternative is free.

Initially, I considered my local coding setup to be a great pair to Google’s free tier. It definitely performs better than Gemini 2.5 Flash and makes a great companion to Gemini 3 Pro. Gemini 3 Pro can solve more complex tasks with the local model doing everything else. This not only saves quota on 3 Pro but also provides a very capable fallback for when quota is hit.

However, this is foiled a bit now that Gemini 3 Flash was just announced a few days ago. It shows benchmark numbers much more capable than Gemini 2.5 Flash (and even 2.5 Pro!) and I’ve been very impressed with its performance. If that’s the free tier Google offers, it makes local coding models less fiscally reasonable. The jury is still out on how well Gemini 3 Flash will perform and how quota will be structured, but we’ll have to see if local models can keep up.

I’m very curious to hear what you think! Tell me about your local coding setup or ask any questions below.

Thanks for reading!

Always be (machine) learning,

Logan

Share

Software engineers should understand recommendation systems because any company serving content to users is using a system similar to this. Collaborative filtering is simple, intuitive, and very effective.

This article is a follow-on to a previous case study detailing how collaborative filtering is used by Spotify and how a system like this has impacted the music industry. If you haven’t read that article, do that first. It puts everything below into context.

When you’re finished with this article, you will have trained your own collaborative filtering model using matrix factorization and be able to visualize it in a UI that shows how interacting with content and retraining a model changes recommendations over time.

Housekeeping & Things You Should Know

• The complete code for building this collaborative filtering system can be found in the recommendation systems repo. Please star it to support ML for SWEs and stay updated when new tutorials are added. This is the first of many ML system tutorials I’ll be putting out.

• Don’t forget about our Machine Learning Roadmap. It’s a guide to ML fundamentals that can be completed entirely for free. I spent some time in 2024 curating it and confidently say it’s the best free ML roadmap available.

• I’m going to start including this Housekeeping and What You Should Know sections in each article and make each article about something. I felt the roundups were too shallow and I wasn’t having fun or learning enough spending my time on them. Instead, each article will have a little roundup section included.

• ML for SWEs is looking for sponsors! If you have a job opportunity, developer tool, or want to share anything else that would be beneficial for software engineers working in AI, reach out to me to get it in front of over 10,000 developers. I reserve the right to deny anything I don’t think is helpful. There’s a high bar for what I share with my audience to ensure it’s a good fit for both readers and sponsors.

• I’ll be posting more frequent jobs updates/who’s hiring/the skills you should acquire for paid subscribers in the ML for SWEs Substack chat. Upgrade to paid if you want those.

  Subscribe now

Part of Machine Learning for Software Engineers is keeping you abreast of the happenings in AI that are actually important. Here are the most important items since our last article:

• OpenAI and Amazon announced a multi-year, $38B partnership for AWS to provide large-scale compute infrastructure, including NVIDIA GB200s and GB300s.

• Apple is reportedly nearing a deal to pay Google ~$1B annually to use a custom 1.2T parameter Gemini model to power a major Siri update.

• OpenAI announced it now has over 1 million paying business customers and 7 million ChatGPT for Work seats.

• Moonshot AI’s Kimi K2 Thinking is a new 1T parameter Mixture-of-Experts model (32B active) that uses native INT4 inference for a ~2x speedup.

• NVIDIA reports achieving 4x faster inference for math problem solving using FP8 quantization and kernel optimizations.

• Researchers propose Continuous Autoregressive Language Models (CALM), which compress tokens into continuous vectors to cut training FLOPs by 44%.

• Terminal-Bench 2.0 was released alongside Harbor, a new framework for testing AI agents in containerized developer environments.

• OpenAI published a post on understanding prompt injections, which it calls a frontier security challenge requiring multi-layered defenses.

• Wikipedia is urging AI companies to stop scraping and use its paid Enterprise API to support the nonprofit’s servers and mission.

• A new report details cybersecurity in the era of AI and quantum, highlighting threats from AI-automated attacks and quantum decryption.

• A Stanford study found that 22-25 year-olds in AI-exposed roles, like software development, experienced a 13% employment decline since ChatGPT’s launch. [Credit: ]

• Platforms like Anthropic’s Claude Code are pushing a shift toward agentic coding, where developers orchestrate agent fleets rather than coding line-by-line. [Credit: ]

• An article explains how models like Qwen3-Next and Kimi Linear are using hybrid attention mechanisms to achieve O(n) scaling for long contexts. [Credit: ]

• OpenAI is offering a free year of ChatGPT Plus to transitioning U.S. servicemembers and veterans.

If you’re particularly interested in one of these things and would like a deep dive, leave a comment and I’ll see what I can do.

Leave a comment

Now onto our collaborative filtering system!

Step 1: Retrieve the Data

For this specific project, we’ll use the HetRec 2011 Last.fm 2k dataset to initially train our model and enable retraining based on simulated interactions between users and artists. HetRec 2011 Last.fm 2k is a great example of implicit feedback which is perfect for a recommendation system. It contains a mapping of listening counts between user and artist IDs, each with a given weight to infer preference (i.e. a higher listen count means a user likes an artist).

First, create a file named last_fm_loader.py. This will be used to load our dataset and prepare it for training. Include our imports at the top of the file. We’ll get into how we use each in a later section.

import requests
import zipfile
import io
import pandas as pd
import os

Define a class LastFmLoader to encapsulate all the data-loading logic. In that class, create two variables: One with the url for downloading our dataset and one naming the directory of the folder we’ll store the dataset in. The __init__ function initializes placeholders for our dataframes and defines the file paths we expect to find inside the extracted zip.

class LastFmLoader:

  _ZIP_URL = “[https://files.grouplens.org/datasets/hetrec2011/hetrec2011-lastfm-2k.zip](https://files.grouplens.org/datasets/hetrec2011/hetrec2011-lastfm-2k.zip)”
  _DATA_DIR = “lastfm-2k”

  def __init__(self):
    self.interactions = None
    self.artists = None
    self._interactions_file = os.path.join(self._DATA_DIR, ‘user_artists.dat’)
    self._artists_file = os.path.join(self._DATA_DIR, ‘artists.dat’)

Add a private method _download_data to this class. This method downloads the zip file from the URL, and extracts its contents into the _DATA_DIR skipping this process if the folder for the data already exists. The print statements are niceties for debugging.

  def _download_data(self):
    
    if os.path.exists(self._DATA_DIR):
      print(f”Directory {self._DATA_DIR} already exists. Skipping download.”)
      return
    
    os.makedirs(self._DATA_DIR, exist_ok=True)
    
    print(f”Downloading data from {self._ZIP_URL}...”)
    try:
      response = requests.get(self._ZIP_URL)
      response.raise_for_status() 

      print(’Extracting data...’)
      with zipfile.ZipFile(io.BytesIO(response.content)) as z:
        z.extractall(self._DATA_DIR)

      print(’Download and extraction complete.’)

    except requests.exceptions.RequestException as e:
      print(f”Error downloading file: {e}”)
      raise
    except zipfile.BadZipFile as e:
      print(f”Error extracting file: {e}”)
      raise
    except Exception as e:
      print(f”An error occurred during download/extraction: {e}”)

Add a public load_data method. This is the function we’ll call from our training script. It runs _download_data to ensure the data is present. Then, it uses pandas.read_csv to load the two files we care about to train our model: user_artists.dat (which contains userID, artistID, weight) and artists.dat (which contains id, name).

  def load_data(self):

    self._download_data() 

    try:
      print(’Loading interactions data...’)
      self.interactions = pd.read_csv(
          self._interactions_file, 
          sep=’	‘, 
          header=0, 
          encoding=’utf-8’
      )

      print(’Loading artists data...’)
      self.artists = pd.read_csv(
          self._artists_file,
          sep=’	‘,
          header=0, 
          encoding=’utf-8’,
          usecols=[’id’, ‘name’] 
      )
      print(’Data loading complete.’)
    
    except FileNotFoundError as e:
      print(f”Error loading data: {e}”)
      raise
    except Exception as e:
      print(f”An error occurred during data loading: {e}”)

Lastly, add a test block at the end of last_fm_loader.py. This block runs a simple test showing the columns present in our data if you execute python last_fm_loader.py directly. We won’t run our training or serving system from this file, but this is great for testing its functionality.

if __name__ == “__main__”:
  loader = LastFmLoader()
  loader.load_data()
  if loader.interactions is not None:
    print(loader.interactions.head())
  
  if loader.artists is not None:
    print(loader.artists.head())

Step 2: Define the Model

Create model.py. This will define our MatrixFactorization class. Start with the imports from torch.

import torch
import torch.nn as nn

Define the MatrixFactorization class, inheriting from torch.nn.Module. The __init__ method sets up our learnable parameters. These are the two embedding matrices our model will learn. nn.Embedding is a PyTorch layer that acts as a lookup table. self.user_embedding will store learned user representations and self.artist_embedding will do the same for artists.

embedding_dim is the size we choose for those representations. In __init__, we also define values for our embedding matrices.

class MatrixFactorization(nn.Module):

  def __init__(self, num_users, num_artists, embedding_dim=500):
    super(MatrixFactorization, self).__init__()

    self.user_embedding = nn.Embedding(num_users, embedding_dim)
    self.artist_embedding = nn.Embedding(num_artists, embedding_dim)

    self.user_embedding.weight.data.uniform_(0, 0.05)
    self.artist_embedding.weight.data.uniform_(0, 0.05)

Now, we define the forward method for the class. This is what PyTorch runs when the model is called. It takes a batch of user indices and artist indices, looks up their corresponding embedding vectors, and then computes the dot product between them as described in our overview of collaborative filtering systems. The .sum(dim=1) is how we perform a batched dot product by computing the dot product over a specified dimension. This resulting “score” is our model’s prediction of how much the user likes the artist.

  def forward(self, user, artist):

    user_vector = self.user_embedding(user)
    artist_vector = self.artist_embedding(artist)

    score = (user_vector * artist_vector).sum(dim=1)

    return score

Again, we add a test block at the end of model.py. This is a great way to perform a quick test via python model.py to make sure our model’s input and output shapes are correct.

if __name__ == “__main__”:

  print(”Testing model.py”)

  test_num_users = 100
  test_num_artists = 50
  test_emb_size = 10

  model = MatrixFactorization(test_num_users, test_num_artists, test_emb_size)
  print(”Model created.”)

  test_user_ids = torch.LongTensor([1, 5, 20, 99])
  test_artist_ids = torch.LongTensor([4, 10, 30, 45])

  predictions = model(test_user_ids, test_artist_ids)
  
  print(f”\nInput user tensor shape: {test_user_ids.shape}”)
  print(f”Input artist tensor shape: {test_artist_ids.shape}”)
  print(f”Output predictions shape: {predictions.shape}”)

  assert predictions.shape == (4,)

  print(”\nModel test passed!”)
  print(”Example predictions (randomly initialized):”)
  print(predictions)

Step 3: Train the Model

Create your third file, train.py. This script will use the LastFmLoader and MatrixFactorization classes to train and save our model.

Start with all the necessary imports. Notice that we’re importing our custom classes here.

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader
import pandas as pd
from sklearn.model_selection import train_test_split
import numpy as np
import os

from last_fm_loader import LastFmLoader
from model import MatrixFactorization

We define a custom LastFmDataset class. DataLoader will use this class to retrieve our training data. The __init__ takes our data (as numpy arrays) and stores them as Tensors. The __len__ method returns the total number of samples. The __getitem__ method returns a single sample (one user, artist, and weight) at a given index. These will be used further down.

class LastFmDataset(Dataset):

  def __init__(self, users, artists, weights):
    self.users = torch.LongTensor(users)
    self.artists = torch.LongTensor(artists)
    self.weights = torch.FloatTensor(weights)

  def __len__(self):
    return len(self.weights)

  def __getitem__(self, idx):
    return self.users[idx], self.artists[idx], self.weights[idx]

When training a model, it’s possible to overfit. This is when the model “memorizes” the training data but gets worse at handling new, unseen data. You know you’re overfitting when your training loss goes down but your validation loss stays higher.

Early Stopping is a technique to prevent this. We monitor the validation loss at each epoch. If the loss stops improving for a set number of epochs (our patience), we stop the training, since continuing would only make the model worse.

We’ll build this logic into a class. The __init__ method sets up our tracking parameters:

• patience: How many epochs to wait for improvement before stopping.

• delta: A small amount the loss must improve by to be considered an “improvement”.

• The other variables (counter, best_score, etc.) are for internal tracking.

class EarlyStopping:

  def __init__(self, patience=5, verbose=False, delta=0, path=’checkpoint.pt’):
    self.patience = patience
    self.verbose = verbose
    self.counter = 0
    self.best_score = None
    self.early_stop = False
    self.val_loss_min = np.inf
    self.delta = delta
    self.path = path

The __call__ method makes the class instance callable (like a function). We’ll call it at the end of each epoch, passing in the current val_loss.

• It checks if this is the best score it has seen.

• If not, it increments a counter.

• If the counter exceeds our patience, it sets the early_stop flag to True.

• If the score is better, it resets the counter and calls save_checkpoint.

  def __call__(self, val_loss, model):

    score = -val_loss
    if self.best_score is None:
      self.best_score = score
      self.save_checkpoint(val_loss, model)
    elif score < self.best_score + self.delta:
      self.counter += 1
      if self.verbose:
        print(f’EarlyStopping counter: {self.counter} out of {self.patience}’)
      if self.counter >= self.patience:
        self.early_stop = True
    else:
      self.best_score = score
      self.save_checkpoint(val_loss, model)
      self.counter = 0

The save_checkpoint method is a helper called by __call__. It’s only triggered when a new best validation loss is found. It saves the model’s current weights to the specified path. This ensures that when training stops, the file at path contains the weights from the best performing epoch.

  def save_checkpoint(self, val_loss, model):

    if self.verbose:
      print(f’Validation loss decreased ({self.val_loss_min:.6f} --> {val_loss:.6f}).  Saving model ...’)
    torch.save(model.state_dict(), self.path)
    self.val_loss_min = val_loss

PyTorch nn.Embedding layers need sequential integer indices. The IDs in our data aren’t sequential. Thus, we write a helper function to create two dictionaries: one to map from the original ID to sequential indices, and an inverse mapping to go back.

def create_id_mapping(df):

  user_id_mapping = {original_id: i for i, original_id in enumerate(df[’userID’].unique())}
  artist_id_mapping = {original_id: i for i, original_id in enumerate(df[’artistID’].unique())}

  user_inv_map = {i: original_id for original_id, i in user_id_mapping.items()}
  artist_inv_map = {i: original_id for original_id, i in artist_id_mapping.items()}

  return user_id_mapping, artist_id_mapping, user_inv_map, artist_inv_map

Now we define the main train_model function. This first part sets up hyperparameters, creates the model_store directory, and loads our data using the LastFmLoader. If we were writing a production system, we would run experiments to optimize the hyperparameters. This can be a lengthy process so we’re sticking with guesses and pushing forward.

def train_model(epochs=20, batch_size=1024, emb_size=50, learning_rate=0.001, model_save_path=”model_store/model.pt”):

  os.makedirs(os.path.dirname(model_save_path), exist_ok=True)

  loader = LastFmLoader()
  loader.load_data()
  df = loader.interactions

  if df is None:
    print(”Failed to load data.”)
    return

Still inside train_model, we preprocess our data. First, we call create_id_mapping to get our dictionaries. Then, we use the .map() method to replace the original userID and artistID columns with their new sequential indices.

  print(”Create ID mappings...”)
  user_id_mapping, artist_id_mapping, user_inv_map, artist_inv_map = create_id_mapping(df)

  df[’userID’] = df[’userID’].map(user_id_mapping)
  df[’artistID’] = df[’artistID’].map(artist_id_mapping)

Next, we apply np.log1p to the weight column. This log(1 + x) transform is useful because it scales down massive listen counts, so a user who listened 100,000 times doesn’t dominate the loss function. We also get the total count of unique users and artists for our model.

  df[’weight_log’] = np.log1p(df[’weight’])

  num_users = len(user_id_mapping)
  num_artists = len(artist_id_mapping)

  print(f”Number of users: {num_users}”)
  print(f”Number of artists: {num_artists}”)

We split our data into an 80% training set and a 20% validation set using train_test_split.

  train_df, valid_df = train_test_split(df, test_size=0.2, random_state=42)

We create LastFmDataset instances for both the training and validation dataframes.

  train_dataset = LastFmDataset(train_df[’userID’].values, train_df[’artistID’].values, train_df[’weight_log’].values)
  valid_dataset = LastFmDataset(valid_df[’userID’].values, valid_df[’artistID’].values, valid_df[’weight_log’].values)

Then we wrap our Dataset instances in DataLoader. The DataLoader is a PyTorch utility that handles batching, shuffling, and multi-process data loading for us.

  train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, num_workers=4)
  valid_loader = DataLoader(valid_dataset, batch_size=batch_size, shuffle=False, num_workers=4)

We initialize our MatrixFactorization model with the number of users and artists.

  print(”Initializing model...”)
  model = MatrixFactorization(num_users, num_artists, embedding_dim=emb_size)

We check for a GPU (CUDA for NVIDIA, MPS for Apple) and move the model to that device for faster training if available. The .to(device) call moves all of the model’s parameters (the embedding matrices) onto the GPU’s memory. The model and data must be on the same device.

  if torch.cuda.is_available():
    device = torch.device(”cuda”)
  elif torch.backends.mps.is_available():
      device = torch.device(”mps”)
  else:
      device = torch.device(”cpu”)
      
  print(f”Using device: {device}”)
  model.to(device)

Then, we define our loss function. MSE is a standard loss function for regression that works by calculating the average squared difference between the model’s prediction and the actual weight_log. It heavily penalizes large errors, which is good for this kind of system.

For the optimizer, we choose optim.Adam (Adaptive Moment Estimation). Adam is a highly effective and popular optimizer that works well “out of the box” for most problems. It combines the benefits of other optimizers by adapting the learning rate for each model parameter individually, which often leads to faster convergence than standard optimizers like SGD.

We also initialize our EarlyStopping class, telling it to save the best model to model_save_path.

  loss_fn = nn.MSELoss()  
  optimizer = optim.Adam(model.parameters(), lr=learning_rate, weight_decay=1e-5)
  early_stopper = EarlyStopping(patience=3, verbose=True, path=model_save_path)

This is the core of the training. We loop for epochs times. First, we set the model to model.train() mode.

  print(”Training model...”)
  for epoch in range(epochs):
    model.train()
    total_train_loss = 0.0

Inside the epoch loop, we loop over every batch in our train_loader. For each batch, we move the data to our device. This is the second half of the device equation: the model lives on the GPU, so every batch of data we feed it must also be moved to the GPU. This user.to(device), artist.to(device), etc. call does that.

    for user, artist, weight in train_loader:
      user, artist, weight = user.to(device), artist.to(device), weight.to(device)

We then get into a standard 5-step PyTorch training process for a batch:

1. optimizer.zero_grad(): Clear old gradients.

2. prediction = model(...): Get the model’s prediction.

3. loss = loss_fn(...): Calculate the loss.

4. loss.backward(): Compute new gradients.

5. optimizer.step(): Update the model’s weights.

We also compute the model’s total training loss as we go along.

      optimizer.zero_grad()
      prediction = model(user, artist)
      loss = loss_fn(prediction, weight)
      loss.backward()
      optimizer.step()
      total_train_loss += loss.item()

After training on all batches, we switch to model.eval() mode and use with torch.no_grad() to turn off gradient calculations for validation.

    model.eval()
    total_val_loss = 0.0
    with torch.no_grad():

We loop over the valid_loader to get the predictions and calculate the total validation loss.

      for users, artists, weights in valid_loader:
        users, artists, weights = users.to(device), artists.to(device), weights.to(device)
        predictions = model(users, artists)
        val_loss = loss_fn(predictions, weights)
        total_val_loss += val_loss.item()

At the end of each epoch, we calculate and print the average training and validation losses.

    avg_train_loss = total_train_loss / len(train_loader)
    avg_val_loss = total_val_loss / len(valid_loader)

    print(f”Epoch {epoch+1}/{epochs} - Train Loss: {avg_train_loss:.4f} - Val Loss: {avg_val_loss:.4f}”)

Finally, we call our early_stopper with the validation loss. It will run its internal logic and if the early_stop flag has been set to True, we break the training loop.

    early_stopper(avg_val_loss, model)
    if early_stopper.early_stop:
      print(”Early stopping triggered.”)
      break

After the loop, the model_save_path will hold the best version of our model, thanks to our EarlyStopping class. We also must save our ID mappings. Without them, we have no way to connect userID 1002 to user_index 5.

  print(f”\nTraining complete. Best model saved to {model_save_path}”)

  mapping_path = “model_store/mappings.pth”
  torch.save({
      ‘user_id_mapping’: user_id_mapping,
      ‘artist_id_mapping’: artist_id_mapping,
      ‘user_inv_map’: user_inv_map,
      ‘artist_inv_map’: artist_inv_map
  }, mapping_path)

  print(f”Mappings saved to {mapping_path}”)

Finally, add the if __name__ == “__main__”: block to train.py so we can run it as a script using python train.py.

if __name__ == “__main__”:
  train_model()

You should now be able to run the full training loop. It will download the data, train the model, and save model.pt and mappings.pth in the model_store directory.

Step 4: Serve the Recommendations

Create the final file, app.py. We’ll use Streamlit to build a simple web UI.

Import Streamlit, PyTorch, pandas, numpy, and our custom classes. We also import LastFmDataset because we’ll need it for retraining. We also define constants for our saved paths.

import streamlit as st
import torch
import pandas as pd
import os
import numpy as np
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader

from model import MatrixFactorization
from last_fm_loader import LastFmLoader
from train import LastFmDataset

MODEL_PATH = os.path.join(”model_store”, “model.pt”)
MAPPINGS_PATH = os.path.join(”model_store”, “mappings.pth”)
SIMULATIONS = 5000

Create a function load_assets to load our model, mappings, and artist data. We use Streamlit’s @st.cache_resource decorator. This tells Streamlit to run this function once and cache the result, so our app is fast and doesn’t reload the model on every interaction.

Inside load_assets, we load the mappings. Note: torch.load(MAPPINGS_PATH, weights_only=False) is important. PyTorch’s security features default to weights_only=True, but our mappings file is a dictionary, not model weights.

@st.cache_resource
def load_assets():

  try:
    mappings = torch.load(MAPPINGS_PATH, weights_only=False)
    user_map = mappings[’user_id_mapping’]
    artist_map = mappings[’artist_id_mapping’]
    
    num_users = len(user_map)
    num_artists = len(artist_map)

Now, we initialize a new MatrixFactorization model instance and load the saved weights from our model.pt file.

    model = MatrixFactorization(num_users, num_artists, embedding_dim=50)
    model.load_state_dict(torch.load(MODEL_PATH))
    model.eval()

Finally, we load the artist names using our LastFmLoader so we can display them later, and we return all the loaded assets.

    loader = LastFmLoader()
    loader.load_data()
    artists_df = loader.artists.set_index(’id’)

    return model, mappings, artists_df
  
  except FileNotFoundError as e:
    print(f”Error loading assets: {e}”)
    st.stop()
  except Exception as e:
    print(f”An error occurred during asset loading: {e}”)
    st.stop()

Now we create the get_recommendations function. This is the core of our app’s logic. We use @st.cache_data to cache the results for a given user.

It maps the selected_user_id to its user_idx, then gets the user_vector from the model’s embedding layer for the current user and all artist vectors from the artist embedding layer.

@st.cache_data(show_spinner=”Generating recommendations...”)
def get_recommendations(selected_user_id, _model, _mappings, _artists_df, num_recs=10):
  user_idx = _mappings[’user_id_mapping’][selected_user_id]
  if user_idx is None:
    st.error(f”User ID {selected_user_id} not found in the mapping.”)
    return pd.DataFrame(columns=[’Artist’, ‘Predicted Score’])
  
  user_tensor = torch.LongTensor([user_idx])
  user_vector = _model.user_embedding(user_tensor)

  all_artist_vectors = _model.artist_embedding.weight

We perform a single matrix multiplication between the one user vector and the entire matrix of artist vectors. This gets us all our predictions for a given user at once.

  with torch.no_grad():
    scores = torch.matmul(user_vector, all_artist_vectors.T). squeeze()

We sort the scores using torch.argsort to get the top N, then loop over them, mapping the artist_model_idx back to the original artist ID and then to the artist’s name.

  top_indices = torch.argsort(scores, descending=True)[:num_recs]

  rec_data = []
  for idx in top_indices:
    artist_model_idx = idx.item()
    original_artist_id = _mappings[’artist_inv_map’].get(artist_model_idx)
    if original_artist_id:
      artist_name = _artists_df.loc[original_artist_id, ‘name’]
      rec_data.append((artist_name, scores[idx].item()))
  
  return pd.DataFrame(rec_data, columns=[’Artist’, ‘Score’])

To make the visual we want to really understand collaborative filtering, we’ll add functions to simulate new data and retrain the model live. simulate_new_listen creates a new dataframe of random user-artist interactions.

def simulate_new_listen(_mappings, num_simulations=100):
  st.write(f”Simulating {num_simulations} new listens...”)
  all_user_indices = list(_mappings[’user_inv_map’].keys())
  all_artist_indices = list(_mappings[’artist_inv_map’].keys())

  sim_users = np.random.choice(all_user_indices, num_simulations)
  sim_artists = np.random.choice(all_artist_indices, num_simulations)
                                 
  sim_weights = np.random.randint(50, 500, num_simulations)

  sim_df = pd.DataFrame({
      ‘user_idx’: sim_users,
      ‘artist_idx’: sim_artists,
      ‘weight’: sim_weights
  })

  return sim_df

Now we create the retrain_model function. It first checks Streamlit’s st.session_state to see if a ‘retrained_model’ already exists. If it does, we use that one. If not, then this is the first retraining so we start from the original load_assets() model. This ensures that clicking the button multiple times keeps improving the same “live” model.

def retrain_model(new_data):  
  st.sidebar.write(”Retraining model...”)

  if ‘retrained_model’ in st.session_state:
    model_to_retrain = st.session_state.retrained_model
    st.sidebar.write(”Starting from *previously* retrained model.”)
  else:
    model, _, _ = load_assets()
    model_to_retrain = model
    st.sidebar.write(”Starting from *original* loaded model.”)

Next, we prepare the new data. Just like in train.py, we apply the log1p transform and load the data into a LastFmDataset and a DataLoader.

  new_data[’weight_log’] = np.log1p(new_data[’weight’])
  new_dataset = LastFmDataset(new_data[’user_idx’].values, new_data[’artist_idx’].values, new_data[’weight_log’].values)
  new_loader = DataLoader(new_dataset, batch_size=32, shuffle=True)

We also need to define our optimizer and loss function again, pointing them at the model_to_retrain‘s parameters.

  optimizer = optim.Adam(model_to_retrain.parameters(), lr=0.001)
  loss_fn = nn.MSELoss()

We run a smaller training loop, just on the new data. We set the model to train() mode and loop over our new_loader, applying the same 5-step PyTorch training process as before.

  model_to_retrain.train()

  for users, artists, weights, in new_loader:
    optimizer.zero_grad()
    predictions = model_to_retrain(users, artists)
    loss = loss_fn(predictions, weights)
    loss.backward()
    optimizer.step()

Finally, we set the model back to eval() mode and save the updated model back into st.session_state[’retrained_model’]. This replaces the old “live” model with the new, retrained one with the more up to date weights.

  model_to_retrain.eval()
  st.session_state.retrained_model = model_to_retrain
  st.sidebar.success(”Retraining complete!”)

Now we create a simple app to visualize all the calculations that are happen. We’re using Streamlit to keep things simple and build it entirely in Python.

First, the UI loads our assets. Then it checks st.session_state to see if a retrained model exists. If so, we use it; otherwise, we use the original model we loaded.

st.set_page_config(page_title=”Music Recommender”, layout=”wide”)
st.title(”Interactive Music Recommender”)

model, mappings, artists_df = load_assets()

if ‘retrained_model’ in st.session_state:
  model_to_use = st.session_state.retrained_model
else:
  model_to_use = model

We create a st.selectbox dropdown for the user to pick a user ID.

original_user_ids = list(mappings[’user_inv_map’].values())
st.subheader(”Select a user to see their recommendations:”)
selected_user_id = st.selectbox(”Select a user”, original_user_ids)

If a user is selected, we call get_recommendations and display the results in a st.table.

if selected_user_id:
  st.write(f”Top Recommendations for user: **{selected_user_id}**”)
  recs_df = get_recommendations(selected_user_id, model_to_use, mappings, artists_df)
  st.table(recs_df.set_index(’Artist’))

Finally, we add a sidebar with a button that, when clicked, runs the simulation and retraining. It then clears the recommendation cache and calls st.rerun() to refresh the app and show the new recommendations.

st.sidebar.title(”Retraining Simulation”)
st.sidebar.write(”Simulate new user activity and retrain.”)

if st.sidebar.button(f”Simulate {SIMULATIONS} listens and retrain”):

  new_data = simulate_new_listen(mappings, num_simulations=SIMULATIONS)
  retrain_model(new_data)

  get_recommendations.clear()
  st.rerun()

And that’s it! You’ve built a complete, end-to-end recommendation system with four files.

To see it in action, run the following command in your terminal:

streamlit run app.py

You’ll now be able to select any user, see their initial recommendations, and use the sidebar to simulate new data and retrain the model live to watch how its predictions change over time.

Always be (machine) learning,

Logan

Share

I'm certain you've heard the phrase: "Music is terrible these days." This was likely from someone who grew up in the 1980s or earlier remarking about the style of music the 'youngins' listen to and what's been playing on the radio recently. Most of us roll our eyes because every generation seems to think the next generation's music is garbage, but the truth is that music has changed drastically over the past decade.

Generative AI has caused more people to be conscious of how AI impacts everyday life. This is easy to notice when a person frequently has to determine if images and videos are real or fake. This is much more difficult to notice when AI is being used to feed you recommendations instead of generating the content itself. I would argue this can be even more impactful because of how difficult it is to notice.

To understand this, we're going to look at Spotify's music recommendation algorithm. We'll walk through it from the problem statement (what Spotify is trying to accomplish) through to the algorithms they use to accomplish that goal and all the way to the impact their methodology has on their users and the industry.

Spotify has made much more music available to many more people. The purpose of sharing this is to walk through the considerations that go into making a machine learning system, many of which go beyond choosing a model and building software.

All machine learning engineers need to understand the tradeoffs that come with approaching problems using machine learning. Machine learning is fundamentally an optimization problem, and optimizing for specific metrics always has trade-offs.

In this case study, we're going to:

1. Start from Spotify's problem. What are they trying to solve?

2. Identify how Spotify is solving that problem.

3. Understand the side effects their approach has.

4. Realize the impact those side effects have on users and the music industry as a whole.

My goal is to help you not only understand Spotify's systems, but also have a better understanding of why case studies like this are important to understanding impact.

The Problem to Solve

Like all companies, Spotify is trying to be profitable. As a music streaming service, they need to increase subscriptions. They do this by optimizing the user experience for long-term listener satisfaction, creating a personalized experience the user enjoys so they'll continue to be a subscriber.

Spotify needs to connect users with music they already enjoy and facilitate them finding new music they also enjoy. To do this, Spotify needs to create a recommendation system that feeds users the music they want. This guide will focus on how that recommendation system works and what it means for users.

This guide won't focus on other, more complex problems Spotify also has to solve, such as:

• Spotify's relationship with musical artists: Spotify needs to create a reason for musical artists to include their songs on the platform so users have the songs they want, but they need to do so without compromising the user experience.

• How recommendations work in different mediums: Spotify has to recommend music on the home page and as a continuity to the user's current listening. There are considerations for how each differs, but that's out of scope for this article.

• Other problems such as explicit content filtering, spam identification, anything having to do with ads (recommendations, filtering, etc.), and more.

I’ll also be including technical details of Spotify’s primary recommendation algorithm, but I don’t have the space, time, or knowledge (not all information is made publicly available!) to include technical details about everything.

Let's get into it.

The Data

All machine learning problems start with the data. Spotify uses four categories: 

• Implicit user feedback like songs played, skips, and saves

• Explicit signals like demographics and follows

• Audio analysis of the tracks themselves

• Textual data from across the web to understand cultural context.

All of these data points are events that are used to understand and improve the listener's experience via machine learning. Spotify processes an estimated half a trillion events daily. They need a system that can keep up with this volume.

The System

Spotify's main goal is to increase user retention by making excellent recommendations at scale. This goal comes with a number of problems that Spotify needs to solve, and we'll walk through the systems Spotify is using to solve them problem-by-problem.

This system started simply and evolved over time as Spotify's user base evolved and a more complex system was required to meet user needs. The following will give you a great understanding of how Spotify's system works based on publicly available sources and provide enough information to better understand the impact those systems have had on the music industry.

Collaborative Filtering: The Foundation Behind Recommendations

If you pay attention to one section of this article, make it this one. This is the section that part 2 of this article will focus on.

To increase user retention, the first problem Spotify needed to solve was how to recommend songs to users to keep them listening. This means using data about what a user has already listened to in order to accurately recommend their next song.

The solution Spotify used for this is called Collaborative Filtering with Implicit Feedback. Using implicit feedback is more practical for both Spotify and the listener because it doesn't require any explicit work on the listener's part (like rating songs). Instead, it uses data the user provides just by using the platform to train a machine learning model.

The Original Problem: A Massive, Mostly Empty Matrix

Put simply, imagine a giant table where every row is a user and every column is a song. Each cell contains how many times that user listened to that song (using a preference-confidence framework where listening once = low confidence, repeated listening = high confidence).

This matrix is massive. One row for each user and one column for each song means a matrix with billions (or more!) entries. The problem with this matrix is that it's sparse, or mostly empty. Since most users and songs won’t have any interaction data, the majority of the matrix won’t have any meaningful values.

Using the sparse matrix for predictions has three primary holdups:

1. Any computations done on this matrix would be computationally expensive. With billions of entries, calculations get expensive and slow very quickly.

2. Predictions are difficult. We want to find music and recommend music a user hasn’t listened to.The sparse nature of this matrix means it doesn’t help at all with this.

3. Adding new songs and users is impossible. New columns and rows will contain all zeroes meaning we have zero meaningful information to use for predictions.

The Solution: Matrix Factorization

Matrix factorization solves this by breaking our giant sparse matrix into two much smaller, dense matrices (meaning they're filled with actual numbers, not zeroes):

1. A User Matrix: Each user gets a row of hidden features (like "how much does this user like rock music?" or "does this user prefer upbeat songs?")

2. A Song Matrix: Each song gets a column of the same hidden features (like "how rock-influenced is this song?" or "how upbeat is this song?")

When you multiply these two smaller matrices together, you get back a complete version of the original matrix—but now every cell has a predicted value, even for user-song combinations that never happened.

The exact features Spotify uses in these matrices is unknown, but the beauty of this approach is that Spotify can add and remove features as necessary since the algorithm learns their relevancy for itself.

Why These Smaller Matrices Are So Much Better

The magic is in those hidden features. The algorithm automatically learns meaningful patterns like:

• User 1 has high values for "rock" and "energetic" features

• Song A also has high values for "rock" and "energetic" features

• Therefore, User 1 will probably like Song A (even if they've never heard it)

This is why matrix factorization works for recommendations: it discovers hidden connections between users and songs that aren't obvious from the raw listening data.

Fun fact: Factorized matrices separated into songs and users with different features for each column make it easier to add items (songs or users) to the matrices with meaningful values. Instead of randomly initializing values because implicit information isn’t known, explicit information can be used to estimate starting feature values.

I haven’t found any sources explicitly stating Spotify is doing this or how they’re doing this (they seem to solve the cold start problem differently—see ‘The Cold Start’ section below), but this is a benefit of the dense matrices worth mentioning.

How The Algorithm Learns: Gradient Descent

The system learns these hidden features through gradient descent:

1. Start with random numbers in both the User Matrix and Song Matrix

2. Make a prediction by multiplying a user's features with a song's features

3. Calculate the error between the prediction and the actual listening data

4. Update both matrices by nudging the numbers in the direction that reduces the error

5. Repeat millions of times until the predictions become accurate

This process automatically discovers what those hidden features should represent to best explain the listening patterns in the data.

Making Predictions: How Recommendations Actually Work

Once the matrices are trained, generating recommendations for a user follows a straightforward process:

1. Calculate prediction scores: Take the user's feature vector and multiply it with every song's feature vector to get a prediction score for each song

2. Sort by highest scores: Rank all songs by their prediction scores for that user (highest scores = most likely to be enjoyed)

3. Apply filters: Remove songs the user has already heard, songs not available in their region, explicit content if filtered, etc.

4. Recommend the top results: Serve the highest-scoring remaining songs

So if User 123 gets prediction scores like Song A: 0.94, Song B: 0.87, Song C: 0.82, Song D: 0.31, and they've already heard Song A, then Spotify would recommend Song B first, then Song C, and so on.

What Spotify Actually Does (The Reality)

In practice, Spotify doesn't calculate predictions for every song for every user in real-time—that would require billions of calculations per second. Instead, they likely:

• Pre-calculate predictions for popular songs and store them

• Use other algorithms to narrow down candidates first (maybe focusing on recently released songs or songs similar to what you've been listening to lately)

• Calculate predictions on-demand for only a smaller subset of songs

But the core concept remains the same: matrix multiplication gives you prediction scores, you sort by those scores, filter out what doesn't make sense, and recommend what's left. The beauty is that this simple mathematical approach can work at Spotify's massive scale once you get clever about which predictions to calculate when.

The Cold Start

Collaborative filtering works really well for recommendation systems, but like any machine learning model, it struggles when there isn't any data to train on. This is called The Cold Start Problem. How do we make recommendations when the user hasn't interacted with the platform yet?

A popular solution to the cold start is recommending the content that is most popular or preferred by everyone. This works well in visual recommendation feeds where many of the most popular items with a bit of mix in categories can be shown on a screen at once. But this doesn't work with a music recommendation system where one song is recommended at a time. Just recommending the most popular music is sure to turn away listeners who aren't at all interested in hip hop.

Spotify takes a different approach to this problem by leveraging popular streams with some variance on the home page menu and then using convolutional neural networks (CNNs) to analyze spectrograms (visual representations of sound) to recommend songs where the audio is similar to a song the user is already listening to.

A new user will be shown recommendations on their home page or by searching. Once they select a song, it will play and Spotify will queue recommendations after it based on how that song sounds. Once the user has listened to more songs, Spotify will create a user profile to start recommending items via collaborative filtering.

Going Past Sound

The next question Spotify engineers had to answer was how to suggest music by more than just sound. Collaborative filtering and analyzing spectrograms via CNNs lean heavily into analyzing the music preference of a user via sound. This doesn't work when the user wants music for a specific occasion, and this goes beyond what the music sounds like.

The solution was most surprising to me because it wasn't something I thought would be effective until I read more about it. Spotify uses "cultural vectorization."

Spotify's systems will use natural language processing (NLP) to search the internet and find text relating to a song. This includes blog posts, song descriptions, and song lyrics. This processing checks the language others are using to describe songs, and Spotify's system will tag tracks based on that. These descriptors are then used for context-based recommendations.

Filling in the Gaps

Spotify uses the above methods to create a hybrid recommendation system that works for users in multiple scenarios. There are also further methods to boost music discovery via reinforcement learning (RL) and ensure less popular music is recommended to users. This is important for lesser-known artists to benefit from posting their music on Spotify's platform.

Spotify further supplements the above algorithms by having editors (human experts) curate playlists using their understanding of cultural trends. This is especially important when targeting playlists toward local markets, as it requires an understanding of cultural trends within the area to ensure the best listening experience.

I'm certain there are other problems Spotify works to solve and other algorithms used to solve them, but they're outside the scope of this article. Many likely aren't made available to the public, and all of Spotify's solutions are adapting over time.

Music is a dynamic space. Trends are changing constantly. Taste in music develops over time. Thus, all the solutions Spotify creates have to be constantly evolving with the problem they solve. Some examples of the evolution Spotify sees are:

• A user's musical taste changing. All the recommendation algorithms need to be constantly trained and updated to ensure an up-to-date user profile within the recommendation system.

• Overall song and musical tastes changing. Algorithms must be updated to account for changing trends in music. Human editors need to be aware of cultural shifts when curating playlists.

• Technology evolves. Spotify can integrate more advanced technology to improve their algorithms and make them more efficient. This also might come in the form of new machine learning technology, which I'm sure Spotify is integrating into their platform.

• The platform develops. The introduction of audiobooks required Spotify to develop an entirely new machine learning system that can leverage user music and podcast preferences to make audiobook recommendations. Audiobook data is much more sparse than musical data, requiring Spotify to leverage other data for recommendations.

All of these require constant training of models and development of new solutions.

Impact

This is where the best machine learning engineers are separated from the rest. Technical knowledge is a requirement for the role, but understanding the interplay between machine learning and the problem space in which it's creating solutions is heavily understated.

Be familiar with Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure". When a metric is identified as a key metric for success, systems will start optimizing specifically for that metric, which results in behaviors that optimize for that metric but undermine the actual, broader objective.

Machine learning algorithms are trained by optimizing them for a certain measure. Successful training means an algorithm was successfully optimized for that specific metric. It doesn't necessarily mean the model achieves the objective we're hoping for. This creates a system where the tested metric can be optimized to game the system and this almost always leads to adverse outcomes.

Here's an example: Machine learning algorithms power search and are optimized for specific metrics. Those creating websites can game SEO by ensuring their website fits into what that metric decides is best to serve to a user. This creates a cat-and-mouse game where web page creators game SEO, then it changes, then they game it again. When SEO is gamed, it's a poor experience for the user.

A similar thing has happened with music recommendations. Over one-third of people get their music via a streaming service in 2025. Record labels and artists make money when people listen to their music, so knowing this, record labels will optimize music to work well with these algorithms to drive listening and increase profits.

This has had a very real impact on the song creation process. Instead of artists leading their creative process with what they want to create, they have to heavily factor in how a song will be recommended while they're creating music.

This can be noticed in modern-day songs. The first 30 seconds of a song are weighed very heavily within music recommendation systems. If a user skips a song within the first 30 seconds, the recommendation algorithm won't recommend it as much. Songs are created so the beginning of the song is engaging, meaning getting to a hook faster and having fewer drawn-out introductions.

Another way artists get around negative weighting when songs are skipped is by making songs shorter. Shorter songs are also favored by Spotify's pay-per-stream model. Since Spotify pays artists per stream instead of duration of streams, shorter songs mean listeners can stream an artist's songs more times in a shorter amount of time.

Pay-per-stream has had the biggest impact on small artists. This favors large artists considerably because they are often prominently placed in curated playlists. The algorithm has an easier time recommending those artists because they already have a large catalog of similar music that is frequently listened to. Thus, smaller artists have a hard time reaching the streaming volume required for any significant income.

When people figure out what works or what essentially games the algorithm to make it recommend a certain type of content, they'll continue to produce that type of content. This is a huge reason why we're seeing greater homogeneity in the music that's released today as opposed to the music that was released two decades ago.

Don't hate the player, hate the game

Spotify has made music far more accessible for the consumer by solving a very complex problem at scale using machine learning. The outcome is an algorithmically-driven feed that can be gamed as soon as artists understand how to do so.

While a major problem is solved, others are introduced. Spotify has continued iterating on their solution to mitigate the unintended problems that arise due to machine learning systems optimizing for a specific measure.

Streaming services have fundamentally changed the music industry and will continue to do so as algorithms evolve and change over time to make music even more accessible to the consumer.

As a machine learning engineer, studying Spotify should show you that machine learning algorithms solve complex problems, and solving those problems leads to those systems having massive impact. Any side effects of those solutions can also be largely impactful.

So when you're building machine learning algorithms, make sure you:

1. Understand your problem space and the right metric to target.

2. Study the side effects of optimizing for that metric and pay attention to how your algorithm can be gamed.

3. Continue iteration on the algorithm to more optimally solve the problem.

If you keep these things in mind, you can solve very complex problems elegantly.

Stay tuned for part 2, where we’ll implement our own simple collaborative filtering solution.

Thanks for reading!

Always be (machine) learning,

Logan

Share