How Data Lakes Are Changing the Way Swimmers Train and Race

From Playlists to Pools

When Spotify engineers talk about indexing Parquet files, most swimmers would tune out. But the same data problems that plague music streaming are starting to hit competitive swimming head-on. Coaches, teams, and even age-group clubs are drowning in data—lap times, stroke rates, heart-rate zones, stroke counts, turn splits, and more. The challenge is pulling that data out fast enough to actually use it.

Spotify recently unveiled a storage architecture called Random Access Parquet (RAP). It lets them grab a single user's record from a massive data lake without copying everything into a separate database. That's a big deal for music, but it's just as relevant for a swim coach who wants to check a swimmer's last 50-meter split during a meet, not after a 20-minute query.

Think of your typical practice data. You might have thousands of files—from GPS trackers, manual stopwatch entries, or even video analysis. If you want to find one swimmer's 100m freestyle time from last Tuesday, the old way means scanning thousands of files. That's like swimming a 200 fly when you just needed a 50 sprint.

The Problem with Traditional Data Lakes

Most swim teams aren't using BigQuery or Trino, but the underlying issue is the same. Data lakes are great for storing everything, but they're built for broad scans, not quick lookups. If you want to analyze a whole season's worth of data to find patterns, a scan works fine. But if you need to pull one athlete's race splits during a meet, the overhead of query planning and file discovery can make it painfully slow.

Spotify noted that cloud object storage now offers millisecond-level access, but the query planning, metadata traversal, and file discovery still add significant overhead. In swimming terms, it's like having a fast pool but spending all your time walking from the locker room to the blocks. The actual swim is quick, but the trip kills you.

That's why Spotify built RAP. It adds an external index layer on top of Apache Parquet files. The index maps query keys—like a user ID—directly to the file and row position. So instead of scanning thousands of files, the query looks up the key, then does a targeted read. It's like having a meet program that tells you exactly which heat and lane your swimmer is in, instead of making you watch every race.

Indexing Your Swimmers

For competitive swimming, think of a swimmer's ID as the key. You want to pull up their fastest 200m backstroke time, their stroke rate at the 100m mark, or their turn efficiency. With an index, you can find that data in milliseconds, even if it's stored in a massive data lake.

Spotify's RAP also supports secondary indexes. That means you can query on different dimensions without rewriting your data files. For swimming, that could be querying by event type, by meet date, or by coach. If you want to compare all your swimmers' 100m breaststroke times from a specific meet, a secondary index lets you do that quickly, without copying data around.

This is a game-changer for teams that rely on cloud storage but also need fast access for live meets or training feedback. Instead of maintaining two separate systems—one for analysis and one for online queries—you can use the same dataset for both. That means less duplication, lower costs, and less data sync headaches.

Storage Layout: Sorting Your Lane Lines

Spotify didn't just add an index. They also optimized how data is laid out in files. One trick is sorting data by query key. If you sort your swimmer data by athlete ID, you'll access fewer files when you want a specific swimmer. It's like organizing your swim bag so your goggles are always on top.

They also interleave value columns. Instead of storing all lap times in one column and all stroke counts in another, they mix them together in the same file segment. That way, when you read one chunk, you get multiple attributes at once. For a swim coach, that means one read can give you split times, stroke rate, and breathing pattern without pulling from multiple files.

Covering indexes are another trick. Sometimes you can answer a query without even reading the Parquet file. If you just need a swimmer's best time, and that's stored in the index, you don't need to open the file at all. That saves even more time and money.

Real-World Benefits for Swim Teams

So what does this mean for a competitive swim program? First, you can store all your practice and meet data in a data lake—cheap and scalable—and still get low-latency queries for live feedback. During a meet, a coach can pull up a swimmer's split times immediately after a race, compare it to their season best, and adjust strategy for the next event.

Second, you can run ML pipelines and AI models on the same data without copying it. Want to predict a swimmer's fatigue based on historical training loads? You can train a model on the same data lake, then use that model to make real-time recommendations during practice. No more duplicate storage.

Third, it reduces costs. Spotify had petabytes of online data in Bigtable and exabytes in data lakes. Copying that data to a separate service database was getting expensive. Swim teams, even smaller ones, face similar cost pressures when they try to maintain separate systems for analysis and online access. With an index-based approach, you can avoid that duplication.

The Community Weighs In

The data engineering community has taken notice. Andrew Lamb sees RAP as an example of expanding open data formats to support interactive workloads. Vikas Singh points out that as cloud storage gets faster, the bottleneck shifts to query planning and metadata access. RAP tackles that with precomputed indexes.

For swimming, this is a reminder that the tools we use for music streaming or e-commerce can be adapted to our sport. The same principles of indexing, sorting, and column interleaving apply to any data that needs fast, focused queries.

A Fresh Approach to Open Data Lakes

Spotify's RAP is part of a larger push to make data lakes handle more than just analytics. Google Cloud recently showed a lakehouse architecture based on Apache Iceberg for AI applications. But RAP goes further by adding a dedicated external index for point queries, while staying compatible with existing Parquet files and Iceberg tables.

For a swim team, that means you can keep using your current data formats and tools, but add a fast index layer for the queries that matter. You don't have to rip and replace your whole system. You just add an index and start getting faster answers.

Practical Takeaways for Swim Coaches

If you're a coach or a swim tech enthusiast, here are a few things to consider:

• Think about your query patterns. What questions do you ask most? Swimmer-specific? Event-specific? Time-range-specific? Build your index around those.
• Sort your data by your most common key. If you always look up swimmers by name, sort by that.
• Use interleaved columns to get multiple metrics in one read. You'll cut down on I/O.
• Consider covering indexes for high-frequency queries. If you just need a best time, store it in the index.
• Don't be afraid to add secondary indexes for different dimensions, like event type or meet date.

These techniques aren't just for tech giants. They're for anyone who wants to make data work faster, including a swim club trying to get an edge.

The Bottom Line

Spotify's RAP isn't about music. It's about making data lakes work for interactive, low-latency applications. And that's exactly what competitive swimming needs as it becomes more data-driven. Whether you're tracking a 10-year-old's progress or a national team's Olympic prep, the ability to query your data instantly can make a real difference.

So next time you hear about a tech company's data architecture, don't tune out. Think about how you could apply the same ideas to your swim data. You might just find a way to shave a few milliseconds off your next race.