🏋️🏽 Uploading Video and other Large File to S3 Compatible Storage
This post on SvelteKit S3 multipart upload follows on from the earlier post on uploading small files to S3 compatible storage. We will see how to upload large video files to cloud storage. In that earlier post we saw using an S3 compatible API (even while using Backblaze, Cloudflare R2, Supabase or another cloud storage provider) makes your code more flexible than using the provider's native API. We also saw the benefits of using presigned URLs for file upload and download. We level up the code from that tutorial here and introduce multipart uploads with presigned URLs. Sticking with an S3 compatible API, we will still leverage the flexibility benefits that brings. I hope you find this a useful and interesting extension to the previous tutorial.
⚙️ SvelteKit S3 Multipart Upload: Getting Started
Instead of building everything from scratch, we will use the previous tutorial on SvelteKit S3 Compatible Storage Uploads as a starting point. You can start here and check out the other tutorial another day, although multipart S3 uploads might make more sense if you start with the other tutorial. If you did work through the presigned URL upload tutorial, you can create a new branch in your repo and carry on from your existing code. Otherwise, clone the following repo to get going:
We won't need to add any extra packages beyond the ones we used last time.
🔨 Utility Functions
With multipart uploads, the presigned URL part works much as it did for a single upload. The workflow is a little different though. We will still keep the single file upload code and only use this when the file is small. With a multipart upload, we need to create a signed URL for each part we need to upload. Another difference is that once we have uploaded all of the parts to their respective URLs, we then we need to tell the provider we are done. This is so that they can combine the pieces at their end. For this to work we need to add a few more utility functions to our
src/lib/utilities.js file. On top we will be restructuring our app slightly, so need to export some of the existing functions.
To get going let us import a few extra functions from the S3 SDK. Remember, although we are using the S3 SDK, we can expect our code to work with any S3 compatible provider (recalling only the initial authorisation step will vary from provider to provider).
Continuing, in line
18, export the
authoriseAccount function because we will want to access it from our SvelteKit endpoint:
Multipart Upload Functions
Next we have to create the function which tells the provider we are done uploading. Add this code to the same file:
authoriseAccount, we will need to export
Next we want a function to generate presigned URLs. This works just like the function we had for single file upload presigned URLs:
Talking of the single upload, the
generatePresignedUrls function needs exporting too:
Lastly, we will create a function to initiate a multipart upload using the S3 SDK:
That was a lot of pasting! Do not worry if it is not 100% clear what we are doing yet, We will start to pull everything together in the next section where we call these functions from our endpoint.
📹 Multipart Presigned Upload Endpoint
You might remember from our SvelteKit frontend, we called an endpoint to tell us the presigned URL to upload the file to. Once we had that URL back, we proceeded with the upload directly from the frontend to the cloud provider. With multipart uploads, our ambition is again to upload directly from the frontend to our provider. For this to work we will change the logic in the endpoint.
We will pass the file size to the endpoint when we request the presigned upload URLs. Based on the file size, our logic will decide whether we will do a single file or multipart upload. When we create an S3 client object, we get back some parameters from the provider which give us minimum, maximum and recommended file part size. So to look at a concrete example. Let's say we want to upload a 16 MB video and the recommended part size is 5 MB. In this case we will need four parts: the first 3 parts will be 5 MB and the final one, 1 MB. Typically, the minimum part size is not enforced by the provider for the final part in a multipart upload.
Now we know what we are doing let's get coding!
SvelteKit S3 Multipart Upload: presigned-urls.json Endpoint Code
This is a substantial refactor on the previous code for the file at
At the top of the file, you can see we now import the functions we have just exported from the utilities file. In line
13, we get the file size parameters we talked about. We use them in line
16 to work out if we will do a multipart upload or single. For a single upload we jump to line
50 and the code is not too different to what we had last time. We just add a
partCount field in the response, to let the front end code know we only have one part (line
For multipart uploads, we work out how big each of the parts is based on the
recommendedPartSize provided by our authorisation response. Once we have that it is just a case of generating the presigned URLs and returning these to the frontend with some extra meta we will find handy.
🚛 Complete Multipart Upload Endpoint
Once the the parts have been uploaded, we need to let the provider know so they can piece the parts together. We will have a separate endpoint for this. Let's create the file now at
src/routes/api/complete-multipart-upload.json.js, pasting in the content below:
That's all the endpoint code in place now. Let's move on to the client page next.
🧑🏽 Client Homepage Svelte Code
There's not too much to change vs. the single file upload code. We'll start by adding a
completeMultipartUpload function which calls that last endpoint we created. Add this block to
Next we need to check in
handleSubmit whether we have a single or multipart upload. If you are using this code in your own new project, you will probably want to refactor the block into separate functions, possibly in different files. Anyway, for now paste in this block:
Notice in line
49 we now get the file size, so we can pass that to the presigned URL endpoint. The value we have is in bytes. For single part uploads, nothing really changes. So let's jump to the
reader.onloadend block for multipart uploads starting at line
The code in lines
92 splits the file into chunks of the right size. We compute the part length and send it in the
Content-Length header in line
Multipart Upload Completion
When we complete the multipart upload, to help piece together the parts, we send an ID to identify each part. That ID comes in the form an ETag which is included in the multipart upload response header sent from our provider. We collate this data in lines
103 into the
parts object is passed to our
completeMultipartUpload in this file and subsequently passed to the endpoint and the utility function.
Allowing Video Upload
The final change is to update the user interface to accept video as well as image files:
Remember you can change this to be more restrictive or, in fact, allow other types based your own needs.
⛔ CORS Update
Because we want to look at a new header (the ETag header) from the client browser, we will need to update the bucket CORS policy. Check how to do this with your storage provider. If you are using Backblaze, you can update the
backblaze-bucket-cors-rules.json file we introduced in the previous tutorial and submit this to Backblaze using the CLI.
🙌🏽 SvelteKit S3 Multipart Upload: What we Learned
In this post we looked at:
how you can upload larger files to S3 compatible cloud storage,
generating presigned URLs for multipart upload,
how you can determine whether to use single or multipart upload and also calculate part size when choosing multipart upload.
I do hope there is at least one thing in this article which you can use in your work or a side project. As an extension you might consider throttling uploads, especially when uploading very large files with many parts. You can also extend the UI to show existing uploads in the bucket and even generate download presigned links with custom parameters, like link validity. On top consider adding code to abandon failed multipart uploads. This can potentially reduce costs.
🏁 SvelteKit S3 Multipart Upload: Summary
When should you use an S3 multipart upload?
- Multipart uploads make sense for larger files. Typically this would be when uploading videos and files of similar sizes. Generally for images, small PDFs and so on, single part uploads are fine. As a rule of thumb, for files larger than 5 MB you will consider multipart uploads. Using multipart uploads allows your file to upload quicker. This is because the file can be split into parts uploaded simultaneously. Consider a 16 MB file; instead of uploading 16 MB from start to finish, you can split the file into four chunks and start uploading all of them at the same time.
Can you upload large files to S3 with a presigned URL?
- Yes. Much like you can upload single files using presigned URLS, you can upload multipart files too. As an initial step you need to work out how many parts you will use and get a presigned URL for each of these parts. You then upload the parts looking out for the ETag header included in the server response when each part finishes uploading. You will use those ETags in a final call to let the provider know the upload is complete. The tags will help combine the parts into a single object (or file) in the cloud.
Is it possible to upload large files to S3 compatible storage from a SvelteKit app?
- Yes! We look at SvelteKit S3 multipart uploads in this tutorial. On top we sort out determining when a multipart or single part upload makes most sense and how to upload directly from the client browser using presigned links for S3 compatible cloud storage providers.
🙏🏽 SvelteKit S3 Multipart Upload: Feedback
Have you found the post useful? Would you prefer to see posts on another topic instead? Get in touch with ideas for new posts. Also if you like my writing style, get in touch if I can write some posts for your company site on a consultancy basis. Read on to find ways to get in touch, further below. If you want to support posts similar to this one and can spare a few dollars, euros or pounds, please consider supporting me through Buy me a Coffee.
🚚 how you can upload large files like video to S3 compatible storage with SvelteKit.— Rodney (@askRodney) November 19, 2021
You can use this to let users upload content using your ❤️ Svelte app or to push your own media to the cloud.
Hope you find it useful!
https://t.co/kLFO52d8SG#askRodney #sveltekit @backblaze
Finally, feel free to share the post on your social media accounts for all your followers who will find it useful. As well as leaving a comment below, you can get in touch via @askRodney on Twitter and also askRodney on Telegram . Also, see further ways to get in touch with Rodney Lab. I post regularly on SvelteKit as well as other topics. Also subscribe to the newsletter to keep up-to-date with our latest projects.