Post History

Pipeline for zipping multiple file chanel into a single output channel

I am in need of downloading some files (from dynamic list of URLs) and archive them into a single zip file. Since these files can be arbitrarily sized (totaling hundreds of MB), I want to avoid downloading each entry individually, creating the entire archive in-memory and then uploading it, but rather make a sort of streamed pipeline for it.  
In short, I want to have something going through the list of URLs, streaming each file into a `ZipOutputStream` (adding a new entry per file), and another thing consuming that stream (ktor multipart request, for example).

I have tried creating a [`ByteChannel`](https://api.ktor.io/ktor-io/io.ktor.utils.io/-byte-channel.html) and passing [`.toOutputStream()`](https://api.ktor.io/ktor-io/io.ktor.utils.io.jvm.javaio/to-output-stream.html) to a [`ZipOutputStream`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/zip/ZipOutputStream.html). Then, spawn a background job to grab each URL, create a zip entry and write packets into the stream, and finally return the channel wrapper in a [`ChannelProvider`](https://api.ktor.io/ktor-client/ktor-client-core/io.ktor.client.request.forms/-channel-provider/index.html) to pass into [`FormBuilder#append`](https://api.ktor.io/ktor-client/ktor-client-core/io.ktor.client.request.forms/-form-builder/append.html).  
However, sometimes ktor times out, sometimes it doesn't and I manage to upload the archive as a multipart request, but it is corrupt.

Has anyone tried anything similar? If so, how have you tackled it?

Here's the code I have:
```kotlin
ChannelProvider {
    val fileChannel = ByteChannel()
    val zos = ZipOutputStream(fileChannel.toOutputStream())
    coroScope.launch {
        files.forEach { f ->
            httpClient.get(f.url).reader {
                while (!channel.isClosedForRead) {
                    channel.readRemaining(DEFAULT_BUFFER_SIZE.toLong()).let {
                        zos.writePacket(it)
                    }
                }
            }
        }
    }
    fileChannel
}
```