Multipart uploads

Large objects go up in parts that are uploaded separately (in parallel, and retried one by one if the network fails), then assembled. The AWS CLI, the AWS SDKs and rclone do this for you above a size threshold, so most of the time you don't need to think about it.

Sizes

Re-uploading a part number replaces it.

Completing

Completing an upload checks the parts' order, ETags, sizes and checksums, and then turns them into the object without copying any data, so it takes milliseconds whatever the size. The object's ETag is S3's multipart form (…-N, for N parts). If the upload declared a checksum algorithm, SwiftFS computes the whole object's checksum as S3 does.

Abandoned uploads

An upload that is never completed keeps its parts, and they count as stored data (as on AWS). Abort it to delete them:

aws --endpoint-url "$S3_ENDPOINT" s3api list-multipart-uploads --bucket acme-backups
aws --endpoint-url "$S3_ENDPOINT" s3api abort-multipart-upload --bucket acme-backups --key db.dump --upload-id …

SwiftFS aborts uploads that are still incomplete after 7 days (configurable when self-hosting with SWIFTFS_MULTIPART_EXPIRY). Aborting is free.

Copying parts

UploadPartCopy builds a part from an existing object, or a byte range of one (x-amz-copy-source-range), so large objects can be copied or spliced on the server.

In code

The JavaScript SDK's Upload helper (@aws-sdk/lib-storage) streams a file in parts:

import { Upload } from "@aws-sdk/lib-storage";

await new Upload({
  client: s3,
  params: { Bucket: "acme-backups", Key: "db.dump", Body: fileStream },
  partSize: 16 * 1024 * 1024,
  queueSize: 4,
}).done();