Vortex：从S3到GPU的一次拷贝，革新ML训练数据加载

InfoQ Homepage Presentations From S3 to GPU in One Copy: Rethinking Data Loading for ML Training AI, ML & Data Engineering From S3 to GPU in One Copy: Rethinking Data Loading for ML Training View Presentation Vertical Horizontal Full Speed: 1x 1.25x 1.5x 2x Download Slides 49:37 Summary Onur Satici explains how Vortex, an open-source columnar file format under the Linux Foundation, revolutionizes high-throughput data loading. He details how cascading lightweight encodings, layout-based segment pruning, and zero-copy memory pipelines eliminate CPU/NVMe bottlenecks to stream S3 data straight to GPUs at speeds up to 60 Gbps without requiring upfront data reprocessing. Bio Onur Satici is a Staff Engineer at SpiralDB and a core maintainer of Vortex, an open source columnar file format now part of the Linux Foundation (LF AI & Data). He focuses on high-performance data systems, GPU acceleration, and making analytical workloads faster at every layer of the stack. About the conference Software is changing the world. QCon London empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams. Transcript Onur Satici: Before I get into anything, I want to show you this video. Not specifically like what's playing in the video, but how we produce this video. Because this is not backed by a regular MP4 file in some main disk, but this is actually a visualization of a file scan. What I mean by this is we have a Vortex file, Vortex being a columnar file format. It has three columns, one per color. Each column would then have a row per frame of this video. Each frame would include around 8 million pixels, because it's a 4K video. Then we then process this data, which is three columns, around 8 million pixels per frame, 60 frames per second, so around 13 gigabits per second, from S3, through the network card, over the CPU, and to the GPU. The GPU is seeing this data this fast, but we want to see what's happening in the GPU, we want to visualize that. We can't downlink that much data to our laptops, so because it's already in the GPU, I'm using the GPU itself to re-encode it into H.264, wrap it into TCP buffers, send it over my SSH tunnel, and we can display and see what's happening. Now have a look at this. This is the same video, but this doesn't have any post-processing on it. This is actually a visualization of a column pruning, or projection pruning, where we only select, in our projection expression, one column, and only the bytes that are associated with that column are flowing through the same pipeline, keeping the same bandwidth, and then ending up with the video that you were seeing before. Today I want to talk about how this is possible, and the technologies that enable us to do this. I'm Onur. I work at SpiralDB. I help maintain Vortex, which is an open-source file format that is under the Linux Foundation. I want to talk about what makes Vortex special. I'll start with why we need this speed in the first place. I'll then move on to Vortex itself, explain some of the main topics around Vortex, layouts and arrays and the scan itself, and then end with what is possible. Problems in Data Loading to GPUs To start with the problem, why we need the speed. Because you have some GPUs that you hired and you want to use them. GPUs are expensive, and then each time they're idling, it's expensive, and it's an opportunity cost of not doing valuable work. I want to tell this like we have two types of taxes that we pay when utilizing GPUs. The first one is a movement tax. Getting the data from some source that you have into the GPU is not trivial, and it's not really cheap. If you use web dataset for training, or for example, MosaicML's streaming dataset, then they go more or less these steps. For example, let's say your data is in S3. You fetch from S3 into the disk, into NVMe. From NVMe, you read compressed bytes into your RAM. You decompress on the CPU, and then send those decompressed bytes over the host of device link, PCIe, to the GPU. This not only has many steps that you have to follow through. If you zoom in into the supported bandwidth per connection, you can see that we are bottlenecked mainly by CPU decompression, and the NVMe throughput. The reason why they all persist into NVMe is the main consumer of this pipeline is PyTorch. PyTorch is Python. Python doesn't like multi-threading, so therefore we use multi-processing. Each worker doesn't want to step into each other's toes, so they want to deduplicate the downloads, so they persist the shards on the disk. Ideally, we want this pipeline to look more like that, where you fetch from S3, eliminating the bandwidth bottlenecks through the network card speeds and the PCIe speeds to the GPU. Then, further, you want to use the right compression that enables you to decompress in the GPU, the parallel compressions, and GPU can decompress much faster than CPUs. You're not only eliminating the bottlenecks you had in NVMe, and you're streaming much faster now, you're streaming maybe 50 gigabits per second, but those bytes are compressed. If you pick the right compression, effectively, this pipeline runs through 10 times faster than the actual compressed speed. That was movement. There's another tax I want to talk about, which is decision. It is when you want to iterate on your training workload. You've loaded from your source into the GPU, and maybe you did some preprocessing to your training, then you want to evaluate the results. What you want to then try, for example, is a new data mix, or you want to change the curriculum. You want some new filter on this data. With the current tools we have, you have to reprocess the entire data, probably re-tokenize, reshuffle to get the right order, and then load again to your source, which might be S3, might be somewhere else, but this is slow. If you look at the hypothetical pie chart of your percentage of time spent, you want to maximize the gray, where you're thinking, and the green, where your GPU's working. Everything else is overhead. You want to turn this into that, where you load from your source into the GPU without wasting much time. Then once you evaluate and decide you want to change something in your input source, then you just change your scan. You just give a different query, a different filter. You don't need to reprocess. Then you immediately can start training for your next iteration. Vortex - Overview How does Vortex, using a different file format, enable all this? Vortex, to give a quick overview of what it is, it's a columnar file format. It is similar to Parquet, but it's also much different than Parquet. It has an extensible spec, a minimal spec that you can plug in into various points. It has a separation of logical types versus physical types. That's one of the main differences from Parquet, is that Vortex doesn't have a notion of tying the types themselves, like integers, into how they're encoded on disk. It's up to you, it's decoupled. It does as little as possible, as late as possible, saving a lot of memcopies for materializing intermediate results, and leveraging from query optimizations. Then it can push through most of the projections, filters, and aggregation from your compute engine into the file format itself. Also, the way it does compression is different. It doesn't use block compression that loses the meaning of the underlying data. It does use lightweight and cascading encodings that can also give a similar compression ratio, but more importantly, they allow you to do compute on the compressed data. You can do random access, you can do aggregations. Some of these computes can be even faster than doing it on the decompressed data itself. To give some concrete numbers versus Parquet, this S3 to GPU scans that we are showing is around 30 times faster. Thanks to the encodings and the compression Vortex has, it's around 100 times or more faster than Parquet in random access. What does the file look like? We will approach this from a file reader's perspective. There's a Postscript that's fixed size, you read it, it has a lot of pointers into other sections of the file. There's a footer. The footer itself is a legend, it's a bill of materials of what you expect to see in this file. You have a list of arrays and you have a list of layouts. Arrays are how you encode the data itself in Vortex, and layouts are like the logical plan. They are the I/O layer of pruning. We will dive into that later. These are showing you the main plug-in points. If you have a new array of a new layout, you just add it here, and you have the logic in your reader, then Vortex will delegate to your reader to do what you want to do. Thirdly, and maybe most importantly, it's the list of segments that you expect to see in this file, and the segments are the majority of the file. They have the array data themselves. They come with triplets, so offsets, length, and alignment. They're pointers, but they also have alignment. Alignment is important to bake in the file, we think, because if you have the alignment, you know that if you want to use SIMD operations or GPU kernels when you load this, if you know the alignment beforehand, in your receiving buffer, you allocate that buffer with the right alignment, so you don't need to copy, just to dispatch the compute on top. I've talked about arrays and layouts, but what are layouts? Layouts enable you to prune those segments as quickly as possible, given a filter and projection. They are like the logical plan. They are also very minimal. This is the entire flat buffer definition. The only two maybe important points I want to touch is the last two fields. It has the children, which is an array of itself, so it's a tree, and each node in this tree has segments. Segments is a list of integers, and those integers point to the offsets of segments. We know from the footer where they are in terms of pointer, like exact byte ranges. Each node in the layout tells, I am responsible for reading these segments, and give me a filter, and I'll tell you which of these segments I'm responsible for, you have to read. Vortex Layouts To give an example on how layouts work, let's start with a schema, for example. We have the root layout. It has two columns, a name, string, an age, integer. The StructLayout only tells it, I have a child per column, so you expect it to have two different children, one per column. Each of these is a ZonedLayout itself. ZonedLayout is interesting because it not only contains the data as segments, it also contains another segment that is a summary statistics of the data that you expect to read. That's also a segment itself. When you read that segment, it's a zone-map. A zone-map is a Vortex array by itself, but it does have a per row range in the actual data, and has columns of statistics, like min, max. The data is then just a concatenation of multiple segments. It's a way of breaking it down into smaller arrays when you want to read them. How does this work, then? Say you have this query. You want to get the name column, but you only want to get the rows where the age column is greater than 30. You start from the root. Root knows it has a projection and a filter in each column, so it delegates into each ZonedLayout through this. The age will receive a filter, and it will then initially load the zone-map, check the zone-map, check the filter, find which segments in this data passes the filter. Let's say the last segment passes the filter, so then it returns the last segment. We also get the name column to return the last segment, because that's the actual data we need. Do the actual filter on the actual rows, and return the rows that we need. By reading the zone-map and only 30% of the segments, you can satisfy this query. Maybe this is a more detailed view. The center is the zone-map. That was a segment. We read it. We deserialized it. It's an array. It has one row per row range, and we have the filter. By just looking at the max value, we can prove that only the last row here can have any rows that match this filter. All the others you don't need. You read the entire chunk. That's the Vortex atomic unit of reading. It's a segment inside. Then, what's neat about Vortex is the compression we have is sliceable. You don't need to decompress everything. You slice it first, and then you only decompress the rows that match the row range we have it. The last 8k rows is decompressed. That is like doing as little as possible and as late as possible. The difference between chunk granularity and row granularity is also, I think, important to mention, because these serve different purposes. Chunk granularity is an I/O concern. It deals with what type of I/O source you have, what is bandwidth and latency characteristics, so you can tune them in Vortex. The zone granularity itself is a compute concern because, depending on your architecture, you might have different wider SIMD registers that you want to use, and those play well with different lengths of data that you have. You can tune them as to your liking with Vortex. Vortex Arrays We talked about reading the segments and layout is helping us prune to the segments that we want, but what do each segment contain? They contain a serialized, most likely compressed Vortex array. A Vortex array, I want to also explain with an example. Let's say we have this array. It's an array of arrays, so the element is a list itself. The elements inside the inner lists are strings. We want to encode them and compress them as much as possible while keeping the meaning. We can start with a ListArray that's similar to Arrow. It basically flattens the values and saves the flattened values as a values child, and then it keeps the inner brace locations because it removes them while flattening, it loses the information, so it keeps that information into the offsets child. Any two consecutive offsets would point to the inner braces of an inner list location. You can see this is also a tree. Now we have the ListArray, but two children are also arrays themselves. We can go on further, we can compress further. For example, the values child has only three unique elements so we dictionary encode them. Dictionary encoding gets the unique values and stores them as children, so like in a values child has one city appearing once only, but you then encode the repetition of the original array as the indices that point to the values child. Then you decoupled the values themselves and the repetition into separate children. Then you can continue further because the codes now have contiguous repeated values, which are called runs. You can run length encode them, or in Vortex, more specifically, run end encode them, meaning you get the values that repeat, save them as a child, 0, 1, and 2 are the values of each run here, and then you also encode which offsets in the original array these runs end, so the actual indices that end. With this, we have the string values as a child, but we also have multiple integer values that we have, and we can see that all of these values are really small. The values child of run end, they're all smaller than four. Four is second power of two, so you can encode each of them in two bits. The ends and the offsets are all less than eight, so you can use three bits to encode them. You bitpack all of this integer data you have and end up with the actual bits on disk like that. That means this is total 30 bits or so, so this is less than one integer. It's less than 4 bytes. Then using this information, you are storing all the repetition and nested structure of your input array, and you only store the unique elements once. We can further compress this. We have FSST, for example, is an algorithm that extracts the common substrings and then further divides into arrays. For this example, I think this gets the point. I want to separate these two things into two ways. You have the arrays, which is a structure, which is how you encode this segment. Then, it is important because this is what you would use to dispatch compute. On the right-hand side, you have the data. I want to think of these two as the control plane and the data plane, and that will be important in the scan, which we'll go into later, but before, I want to mention one more thing. Having this array tree on hand and having this data as pointers somewhere in some location, you can dispatch compute without decompressing. For example, I want to get a random read. I want to get the second element of the first list. If you get that query into the ListArray, ListArray knows that it can binary search the offset child to find the actual location, then does propagate that to the flattened values child because it now knows the actual location. DictArray can do the same for code, and then substitute with the right value before returning. Also, I mentioned some compute you can run on the compressed data might end up being faster than running the compute on decompressed arrays, like summing, for example. Summing requires, on a decompressed array, iterating through each element and summing the values. If you have a run end array, that's just multiplying with the lengths of each run, and then per run, you are summing the total result of the multiplication. It's a lot faster to run the compute on compressed data. Vortex Scan/Byte Copies Back to the separation. The control plane shows you the amount of information you need in a compute engine so you can dispatch work on the data itself. The data plane is the information you have to transfer to your compute engine. These serve different purposes, and we don't want them to step in each other's toes. How do we do this? Vortex has a scan that tries to maximize this throughout. The way it works is it tries to maximize the I/O engine's work and the compute engine's work to be busy all the time. Then, scan is the orchestrating layer. The orchestrating itself also wastes compute and has to dispatch work, reconstruct the array trees, and then follow the array tree logic all the way to the end. The I/O layer then gets some bytes to the compute source, returns the pointers, the scan then dispatch work on those pointers. A small detail, but compute also can request more segments because, for example, in the query that we initially did, select name where age is greater than 30, you don't know if you need name column before executing the age filter. It could just be pruned all together so you don't end up getting those segments. Say we have a query in the scan, and then we have the layout in the scan so we know which segments we need to fetch. Those are transferred into byte ranges, and these are byte ranges that you want to get from your source. Each individual byte range might have a latency, and depending on what source you're reading, you don't want to dispatch each individual byte ranges by itself. We have a coalescing layer on top, and coalescing means if two byte ranges are close enough sufficiently, you merge them and then have a single byte range. You get extra bytes in between, but because you pay the latency once and get the entire throughput, it works on your benefit. This is source-dependent. NVMe is happy with some coalescing configuration, but S3, for example, might require a lot more. S3, you can do 16-megabyte segments because it's a huge throughput with huge latency. Once you get the segment request and then have the byte ranges, you then dispatch work on them. It's a CPU scan. This work is done when you have the bytes on RAM. Then the work you dispatch is regular Rust functions like slice, projection, filter, any compute that the input query wants you to do. That was CPU. Most of the things stay the same when you are scanning to the GPU. The scan orchestration stays the same, the coalescing layer stays the same because we still need those byte ranges, but now the I/O plane has to ship these buffers all the way to the GPU and return pointers to the GPU. That's when its job is done. The compute, now instead of dispatching Rust functions, it does dispatch CUDA kernels, but everything else is the same. We have a buffer pool in the middle, and I want to explain why we have a buffer pool in the middle. This is the lifetime of a byte range through a scan from S3 to the GPU. Initially, when you request bytes from S3, it lands on the kernel, receive buffer. This is a direct memory copy, so it's not an actual copy, it's the first place that these bytes land from the network card, is this kernel, receive buffer. It immediately copies the ciphertext, still in a TLS encrypted form, into the userland, one-to-one. Then we are in our HTTP libraries land where we use Rust TLS and request. Rust TLS decrypts it into plaintext copies, that's one copy, and then it's still TCP, we want to get the HTTP body out of it, so the request layer deframes it, gets the body into its own buffer, then finally it arrives to our own Vortex land. Because these yellow buffers are all streaming, because they're multi-part GET request, we have to aggregate them because we want the segment, we want a contiguous byte range. We aggregate them, so we have to copy one more time, but that's the only copy we do. We receive these chunks as a streaming fashion, and then we accumulate them until the request is done. Immediately after this is done, we dispatch the CUDA. We do CUDA async copy and expect it to land on GPU, but unbeknownst to us, internally, CUDA does another copy. It does another copy because it requires some alignment rules from us. Basically, the buffers you can copy to the GPU from the host has to be pinned and page locked, meaning it should be outside the operating system's virtual page system, so then the copy, if the buffer gets paged out during the copy, GPU doesn't segfault and stays happy. Before diving into how we can reduce the copies, I want to mention one more thing. The memcpy cost is what's important to us. The unit is a core per gigabit of bandwidth you can sustain. It's important because this core budget is also used by the orchestrating layer in scan, and also compute if you're doing the compute on the CPU. It has the opportunity cost, because you have limited CPU, if you're just wasting it on memcopies all around, then you slow down your own bandwidth. It's like a double-edged damage, so you want to reduce this as much as possible. How can we reduce this? Initially, we start with the CUDA bounce buffer because we know the requirements. We can satisfy those requirements on our own buffer that we accumulate the HTTP buffers into. We have a pinned buffer that we use, and then we then call CUDA async copy on it. It doesn't use a bounce buffer, and this goes directly into the GPU. Further, because allocating these are expensive, we keep a buffer pool around to amortize the allocation cost. The way we reuse these buffers is when you dispatch the CUDA copy, you append the CUDA event after that CUDA stream. You append an event. Once that event fires, you know everything that comes before it is complete, so then you can safely reclaim the buffer because that means the transfer is complete, and you can reuse that same buffer for next copies in your pipeline. This reduces the cost a bit, but we need to then look to the left, to the HTTP land, to get rid of some of the copies. One thing you can do here, if you have a sufficiently high Linux version, you can use a kTLS, kernel-side TLS decryption. That pushes the decryption into the kernel, and because the kernel was immediately copying into the userland, then it doesn't need to do it anymore because now it's decrypted, and then the decrypted can be in place. The decryption can be in place in the receive buffer itself, so it saves one copy. There's one caveat that the Linux ciphers are mostly not really that up-to-date in kTLS, so they might not end up using the widest SIMD registers your machine has, but still, the memcpy saves here because we reduce one copy, offsets that deficiency, in a way, so we still keep this kTLS in this land. What else can we improve? We can improve more things on the HTTP side, but for this, we need to do bigger changes because there is no low-hanging things we can do here. If you do bigger changes, namely writing your own HTTP client, then it pays off well, because if you write your HTTP client, you control which buffers you give to kernel. If you do it, you can use the already pinned and page locked Vortex buffer you have to accumulate and deframe on the fly in the same buffer. Once you get the decrypted bytes as a stream in the multi-part GET request, you keep allocating them into the same buffer you already had to be ready to be sent to CUDA. Then when the multi-part GET request is complete, you dispatch to work, and you cycle these buffers, so with one copy, you go from S3 to GPU, and you have very little CPU overhead during the entire scan. There's one more iteration of this, but you have to get rid of TCP. If you have a very expensive and tedious setup, but really performant and cool RDMA object store in your hands, then you can use RDMA protocol, which is different from TCP, that it doesn't have to go to the kernel sockets layer. It can bypass the CPU entirely because it works as two peers connected via some sort of fiber, can be PCIe, can have some network connection, can be in between, but they can read and write from each other's memory regions. If you have this ability, then the GPU itself can dispatch the I/O request on your behalf. If it does it, it not only bypasses all CPU, it even bypasses the PCIe hub of your CPU, because these boxes come with multiple GPUs and come with many network cards. Most often, each GPU has an associated network card, so the RDMA would flow through from the GPU into the nearest network card, bypassing your CPU PCIe hub all together, so it's basically free. The CPU doesn't even know what's happening. For this, the network card also needs to be smart enough, but most boxes that come with high-end GPUs have that network card in place already. With all that, let's revert back to the video. Now, you can see there is some byte range coalescing, so we have a filter or projection. In this case, we have no filter, and we SELECT *, essentially, all columns, red, green, and blue. They all go through this coalescing layer. Then we fetch these bytes from S3, and then we cycle the byte buffer that we read them through, and they go only one copy into the GPU, and then the GPU processes them as 15 gigabits per second, 4K, 60 hertz, and then we stream it back to see what's happening. What's also cool about this, because while we have it, we can change the filters and projections. We can do, for example, this. This is not a post projection. This is actually a combination of a projection and a custom Vortex expression. I have written this custom expression called quantise, and I've written a GPU kernel for it, which is very basic. Quantise just gets the colors, each pixel value is 0 to 55, and then it snaps it to three levels, basically, depending on which level it's closest to. You get this effect of having only three columns. We also pruned the blue column out of the way, just for fun, so it's more yellowy, but also this discretized footage we have. This is, importantly, mostly handled by Vortex orchestration. You don't tie this kernel at the end yourself. You just say that I want this query, and the scan orchestration takes care of appending your kernel, and then dispatching work onto your kernel, and then you can keep the same throughput. Or you can do something like this. This is a bit jarring to watch, because the video skips all over. The way this works, remember I told you when we have this file with three columns? This file has four columns, and the fourth column is whether the frame has a cyclist or not. I've run this object recognition algorithm per frame, and then we now have this has_bike Boolean column, as the fourth column, and I'm filtering to only the frames that has a bike in it. The way this works is because each frame is huge. Each frame we have is 8 megabytes. We can afford to have a zone per frame. In the layout level, we have a zone-map that is as granular as it gets. We have one row per frame, and so we can prune each individual frame that doesn't have a bicyclist on it. Maybe what's more important to talk about here is all of these changes don't affect throughput. You can come up with an arbitrary filter. You can come up with your own projection. You can combine them, but you can keep the same throughput, and you only read the data that you need. Remember I told you the videos were 60 frames per second because I was intentionally slowing them down. Normally, when you let them loose and then write it to like a file, then we can stream 240 frames per second, because I was pacing the video so we can see it at 60 frames per second, but we can sustain 60 gigabits per second end-to-end with Vortex. What is Possible? I want to talk very briefly on what you can do with Vortex if you want to customize it. Remember I told you, you can have your own arrays and you can have your own layouts. A layout is responsible for I/O and then logical plan and then pruning and all that stuff, but it's also responsible for placement. There's nothing in Vortex file spec that enforces you to be in the same physical file. You can separate your segments into two. You can get the pruning segments and store them in Redis, and you can get all the array segments in S3. Redis has one-hundredth of the latency of S3, so you get pruning nearly immediately compared to the numbers you would get with S3, and then you get the S3 throughput on streaming the data to the end. Or, say you have this giant instance with multiple high-end GPUs but a single CPU, and then you have this GPU analytics engine that you shard the data into individual GPUs, and then you run a scan, and then dispatch the scan into individual shards. What if the query has an aggregation? Then you need to aggregate all of the results into a single GPU. You can model that aggregation as an RDMA scan, because GPUs are connected to each other with an extremely fast fiber called NVLink. It can sustain tremendous speeds. You can scan from one GPU to your central GPU to aggregate, and you can scan in terabits per second for the segregation. Takeaways That was Vortex. I wanted to talk about how it allows you to change your filter, to change your data mix and not reprocess everything over and again. I also want to mention that you can read from S3 into GPU around network card speeds. It can be 60 gigabits per second, can be more if you have a stronger network card. You can read from RDMA sources to your GPU at whatever PCIe connection you have, and that can be 500 gigabits or terabits with the newest PCIe. Questions and Answers Participant 1: What I try to understand about the video example, you seem to not keep the video in S3, but RGB frames that are completely separate, and you don't apply any video codec that would compress data, and then there would be I-frames, P-frames. You mentioned explicitly that's like 8 megabytes per frame? Onur Satici: Yes. Participant 1: Is this synthetical example of how great is Vortex, or there are really use cases to fetch attributes from S3? Onur Satici: That was a hypothetical example. Vortex is not a video codec, and it was just to demonstrate that this throughput we can sustain end-to-end, and we can visualize a scan in a video format, happens to be a video, but Vortex can be used like this. It's mostly catered towards analytical or training input data that you normally use. Participant 1: Is there a use for Vortex with compressed codecs, then? Onur Satici: You mean block compression, like zlib? Participant 1: If you happen to have a video stream, or even a JPEG, separate but compressed data, is there a value in passing it into Vortex column to get other benefits of it? Onur Satici: Yes, you can. With compressed JPEGs, you won't be able to run arbitrary compute on them, but you can model a sensor stream for a camera, for example, as a Vortex file. We support a lot of wide columns, so you can have a column that is responsible for these JPEG images. What would be cooler to have is some sort of transparent compression, so you can, for example, crop, and then get only the byte ranges in that crop, for example. The layouts allow you to do that, but you need to use the right image compression. Participant 2: Are database engines adopting this? Onur Satici: We support DataFusion, DuckDB, and Polars. We are also in the works of getting an Iceberg support for Vortex. It's in the works, yes. DuckDB and DataFusion is stable. You can use it today. Participant 3: You just commented on the Iceberg. You have some more details when we can expect it? Onur Satici: Yes. We are working with Iceberg to have a compatible official Vortex support. Iceberg is abstracting the file from it, that's backing Iceberg away from Iceberg itself, so we are iterating with them to ship this as fast as possible, but it's still in the works. Participant 3: I read about it a year ago in your blog post. Onur Satici: It's ongoing, yes. Participant 3: It's ongoing. Onur Satici: Yes. Participant 2: I had a slightly different question. I happened to bring this up when I was talking to some folks at LinkedIn, and then their response was, does your company let you use GPUs for anything other than pure ML workloads? This seems like a great speed-up for analytic data loads, but it seems companies tend to allocate their budgets for ML workloads specifically on GPUs. Onur Satici: I think it might be even better for other analytics use cases, because ML workloads, you save from reprocessing your data, but the speeds that Vortex can sustain, the consuming model, if it's a pre-training model, is really slow, so it doesn't need that much of a bandwidth. If you have some analytics work that's happening in the GPU, that would benefit a lot more from that higher throughput, I think. Participant 4: First, the infamous interoperability question, I suppose, in that I pick a columnar format, like Parquet, for instance, it gives me relatively wide interop. In a way, if I had to pitch a convincing narrative as to why I should pick an alternative columnar format that may not have the same level of interop, what would your top arguments be? Also, my second question would be, you talked about the network penalty and the CPU penalty when moving data. Isn't the network penalty almost a function of how S3 and S3 APIs work. If you had a better source, wouldn't you, in a way, avoid some of those network penalties? The CPU penalty, I understand, you're almost always going to get it anyway. Onur Satici: For interop, yes, we have official support for DuckDB, DataFusion, and Polars. We are working to integrate more. With Iceberg, I think you would unlock that the transition would be a lot easier, because Iceberg abstracts the file format away from you. That's the main idea. If you want to try Vortex, it's really easy to converge from Parquet, and just test it yourself to see. The other question was about network limits. S3, surprisingly needs a lot of tuning as well. You can change the byte request, even the order of those byte range requests that you send to S3, change a lot of the throughput that you can get from S3. Even you have to tune the network cards, you have to bind the right number of ENIs to the network card, that you have in the instance, so on and so forth, to maximize that throughput. Yes, for most of the time, when tuned right, S3 itself would not be your bottleneck in this pipeline. Participant 5: Most of the pruning stuff that you showed seemed like it was reliant on the data being sorted in the first place. Biggest problem that we've ended up having with a lot of this stuff is that lots of different users need the same data sorted in 10 different ways. Could you use something like the layouts to basically keep all the metadata that you need so that you could effectively have the same data sorted 10 different ways for people to read? I'm not sure it would be as efficient. You'd still have to pick a most efficient option. Onur Satici: Yes, it'd be an option. We mostly delegate sorting itself to the compute engine that we're using, so DataFusion or like DuckDB, that would handle it. Yes, I think it's an option. Participant 6: I have a question about the metadata we store in the layout. We talked about min, max of a particular field. How do we decide what metadata to expose? In the case of age, like the query was designed and the min, max was there. In general, how do you write Vortex files knowing what might make it better? Onur Satici: This actually is transparent to you, so you don't get to pick which zone-maps will end up being in the file. The statistics that we support, I showed just a subset of them, like anything that you can do to calculate before and as statistics when writing that you can use to prune afterwards, we will sort it. Like null count, for example, like min, max, for example. You can have even cardinality estimation that you have in your file, so you can help up with aggregations and stuff. Anything that can help with pruning, we try to integrate it. Adding a new one is really easy in Vortex. Participant 2: In one of your slides, really early on, you had the dictionary encoding, the three colors. I think you said there's like 4 bytes for the data, but then you didn't count the bytes in the dictionary itself? Onur Satici: That's right. Just the overhead was around 30 bits or so. Yes, so this is like 30 bits, and that's plus this. You can compress this further if you have the right encoding. Maybe to finish on the S3 question, actually, like maybe counterintuitively, but S3 bandwidth can surpass the NVMe bandwidth if you tune it right. For example, in those scans that I was showing, you can scan from S3 faster than you can scan from your local NVMe. The initial latency is higher, but the throughput is also higher from S3. Participant 2: I remember when Parquet first came out. This was when the big data world was trying to compete with the established MPP databases. One of the first things they did was they said, like row-major is not great, let's go to this columnar format. The next thing they said is, if we're using JDBC, we're taking what was columnar and going back to row-major. Then we're not getting any of the benefit with pipelining this through memory, through all the layers to registers. That's where Arrow came in. How does Arrow figure into this? Onur Satici: All of the compressed Vortex, like we call them the canonical encodings, for each logical type, we have a canonical encoding. They are one-to-one with Arrow, and they're zero-copy from and to Arrow. That's how we plug into DataFusion as well. DataFusion needs Arrow batches. We just decompress Vortex arrays, get the Arrow batches, and then forward with that. If you have, for example, DuckDB, DuckDB has multiple types of vectors. There's a constant vector, and we also have a constant array. It's already a constant array. There's no need to decompress and then re-encode it by DuckDB as a constant array. We pipe that through. Participant 2: Let's say I take a Parquet file and convert to Vortex. I think it had stats on how much faster. Onur Satici: Yes, I have numbers. Participant 2: I think it was before, you had something like 30x faster or something. Onur Satici: Yes, I have numbers for GPU scans, but I also have numbers for CPU scans and writes and reads and random access. This was telling that the GPU scans were 30 times faster, 100 times faster for random access. The CPU scans are around 20 times faster. CPU writes, writing a Parquet file is around five times faster than Parquet, in that ballpark. Participant 2: I think, to the other question, which is like Avro was established for a long time. I was at LinkedIn for many years, and Avro was the standard. Then it was a huge amount of data, but they converted it to Parquet because it made sense. It would also make sense, eventually, to move all Parquet to Vortex, if it's faster on CPU and GPU. Is there any size difference in the actual file? Onur Satici: Size difference? The compression ratio is very similar to Parquet. It's like 10% more, 10% less in that same ballpark. Participant 2: There's like the compression of Parquet, and then there's compression, like Parquet ZSTD, which is even further. Are you converting Parquet to Vortex? Onur Satici: ZSTD Parquet to Vortex. Participant 2: Parquet ZSTD to Vortex? Onur Satici: Yes. Participant 2: You don't need to apply ZSTD on Vortex? Onur Satici: No. Participant 2: Would there be any benefit to doing that, or would you basically not get much compression? Onur Satici: The layouts, they also have writers and readers. We have a custom layout writer that you can use called a compact writer that does zstandard for some data, but, for example, uses Pcodec for floats. If you want an even smaller file, you can tune that in, but then you would lose the computation advantages that you would get. It's up to you. Vortex is completely configurable. You can have any writer or reader and combine them as you wish. Participant 2: This is V1 or V sub-1? Onur Satici: It's V1. It's stable as of six months ago, I think. Participant 2: You're a committer, maintainer on this project. Onur Satici: Yes. Participant 2: If we look at Protobuf, and Protobuf 3 over 20-something years. Every 7 years something had changed, it became incompatible with the previous. What is your view on Vortex? One thing is if we look at GPUs, they have this virtualization layer. You code to that virtualization layer. The GPUs change very often, but the CUDA coding doesn't change at all. How do you view your format? What do you have to keep up with? Because if you're coding to that layer, versus to the GPU hardware itself, then you're standard. You have to change this to make the most use of the hardware itself. The hardware changes very fast. How would you see the backward compatibility of Vortex over time? Onur Satici: The file spec is standard, and it's intentionally minimal. We don't need to change the file spec to change most of the things that you would need a file change for other formats, for example. Like Parquet spent around 7 years incorporating f16s, because the committee, with a lot of big corporations behind it, have to agree on one form or another. Or ALP, for example, is a cool floating-point compression. It's been open for a couple of years now, and it's nowhere near to be merged. For Vortex, it's just a crate. You just write a new Rust crate, and then that's your new layout. The core part of the file would not break, because it's very simple. What you can have is you can have some new encodings that other people have written that you don't have, so you cannot read their files, but you can just add the library and then read them. Participant 2: Do you support non-Rust Serde? Onur Satici: Yes, we have an FFI binding. We even have a Java binding. Yes, any languages, you can plug in with Vortex. Participant 7: You gave an example with arrays and dictionaries. Ultimately, it's still you keep nested data structures inside a single column. You optimize them inside the column, but you still don't attempt to expand them on the shredding of nested structures, build columns, or achieve better compressions there. Is there a step improvement in Vortex versus Parquet for maps or JSON, these kinds of data structures? Onur Satici: We currently don't have that in the core, but it can be easily added as a custom layout, because shredding is just like a new layout that you would write yourself. It would treat the JSON blobs as binary, and then extract that, and then store the fields that you want as a separate column. It's a great extension point. It's just not in the core yet. Participant 7: Is it yet not in the core, or it's by design, and not what you're trying to solve? Onur Satici: We think the compression ratio and then the query patterns that, for example, in TPC-H, TPC-DS, and ClickBench, like this compression works well so far. If we need some use case that would benefit greatly from shredding, then it's not that difficult to integrate in with a new type of layout, essentially. Participant 8: I have a question regarding consistency and error correction about the Vortex file. Which mechanism are you using, and how, when you read the file, you can identify a file is corrupted, and so you need to re-download it or recreate it? Onur Satici: Currently, by default, we rely on the network stack, I think. You can add Merkle trees and parity on the footer as metadata, and you can get your layout to check them before deserializing the files, if you want. See more presentations with transcripts Recorded at: Sep 04, 2026 by Onur Satici Related Sponsors This content is in the AI, ML & Data Engineering topic Related Topics: Development Architecture & Design AI, ML & Data Engineering S3 Columnar Databases QCon London 2026 Rust AWS System Programming Amazon Database QCon Software Development Conference Transcripts NoSQL GPU Streaming Data Lake Data Pipelines Performance CUDA Machine Learning Architecture InfoQ Cloud Computing Related Editorial Popular across InfoQ Kubernetes Promotes KYAML as a Safer, More Consistent Way to Work with Manifests Airbnb Cuts Authentication Code by 60% with Server Driven Architecture OpenClaw 2.0 Releases with Simplified Setup and Collaborative Agents Google Mantis: an Agentic Vulnerability Scanning Harness for Reducing False Positives Java News Roundup: TornadoVM 6, JReleaser, LangChain4j, Java Operator SDK, JHipster, Yupiik Fusion Personality Over Skillset: How Adam Wachtel Builds Engineering Teams