datopian/r2-bucket-uploader

Cloudflare R2 bucket File Uploader with multipart upload enabled. Tested with files up to 10 GB size.

Cloudflare R2 bucket File Uploader

A minimal and flexible uploader component tailored for Cloudflare R2 bucket. Fully compatible with PortalJS.

The intent of this repo is to provide simple to use and simple to copy and paste file uploader component for Next.js.

The file uploader components use Uppy under the hood and are accompanied by the necessary API routes.

The components were written mainly to be used with R2 but any blob storage with a S3-compatible API should work with just a few tweaks.

Good for simple projects or for bootstrapping more complex file-uploading workflows when customized.

Features

• R2 blob storage support
• File content hashing
• Drag and drop
• URL pre-signing
• Multipart upload support
• Customizable

Setup

This repo provides two different sets of component and API routes, one for simple uploads and the other for multipart uploads. If your application is meant to handle large files, multipart upload is the recommended approach. Choose one and follow the instructions below.

Install the dependencies

On your project, run:

npm i @uppy/core @uppy/react @uppy/aws-s3 @uppy/dashboard @uppy/drag-drop @uppy/progress-bar @uppy/file-input crypto-hash @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

If you want to use the multipart upload component, one more dependency is required:

npm i @uppy/aws-s3-multipart

Copy the component and the API routes into your project

Simple upload

Copy the following file to the components folder of your project:

https://github.com/datopian/r2-bucket-uploader/blob/main/components/FileUploader.tsx

Next, copy the following file into your project's pages/api/ folder:

https://github.com/datopian/r2-bucket-uploader/blob/main/pages/api/upload.ts

Multipart upload

Copy the following file to the components folder of your project:

https://github.com/datopian/r2-bucket-uploader/blob/main/components/MultipartFileUploader.tsx

Next, copy the following folder into your project's pages/api/ folder:

https://github.com/datopian/r2-bucket-uploader/blob/main/pages/api/multipart-upload

Set the environment variables

In your .env file, set the following environment variables:

R2_ACCOUNT_ID=
R2_ACCESS_KEY_ID=
R2_SECRET_KEY_ID=
R2_BUCKET_NAME=

The values should be available from the R2 management dashboard on Cloudflare.

Set the CORS settings for the R2 bucket

Create the following CORS settings in order to make the upload components work with your bucket:

[
  {
    "AllowedOrigins": ["*"],
    "AllowedMethods": ["GET", "PUT"],
    "AllowedHeaders": ["Authorization", "content-type"],
    "ExposeHeaders": ["ETag", "Location"],
    "MaxAgeSeconds": 3000
  },
  {
    "AllowedOrigins": ["*"],
    "AllowedMethods": ["GET"],
    "AllowedHeaders": ["*"]
  }
]

Optionally, to increase security you can also customize the AllowedOrigins properties.

Use the component

Now you can import <FileUploader /> or <MultipartFileUploader /> anywhere in your project. E.g.:

// pages/index.js
import FileUploader from "@components/FileUploader";
import MultipartFileUploader from "@components/MultipartFileUploader";

export default function Home() {
  return (
    <div>
      <h1>Upload an image:</h1>

      <FileUploader
        onUploadSuccess={(result) => alert(JSON.stringify(result))}
      />

      {/* OR */}

      <MultipartFileUploader
        onUploadSuccess={(result) => alert(JSON.stringify(result))}
      />
    </div>
  );
}

r2-bucket-uploader's Issues

Allow only upload specific file type

The issue is more related to the R2 itself, but perhaps someone has advice on how to validate the file type in a secure environment?

Passing ContentType in PutObjectCommand doesn't work.

Currently, anyone from the client side can send a malware file with any header, and it will be correctly uploaded via presigned url.

At the moment, presigned URLs don't support POST methods, and R2 doesn't have Bucket Policies.

Add UML diagram for documentation purposes

I was pretty taken aback at how complex uploading files actually is so I made it into a uml diagram to help understand it better

Seems like client based uploading including larger files is actually fairly difficult

This doesn't even include logic for accessing the files after they've been uploaded haha

Return url to a file after upload

When I upload file to R2 bucket, I want to get URL to the uploaded file so that I can share it or use it in my app.

Nextjs 14 issue with multipart upload?

Not sure if it is an issue with nextjs 14 but I am struggling using the multipart upload procedure.
The completeMultipartUpload does not run somehow.

Not sure if someone already tested it with nextjs 14 or if it is just a mistake which I am doing (the single file upload work fine though).

This is the component:

import React, { useMemo } from "react";
import Uppy, { type UploadResult } from "@uppy/core";
import { Dashboard } from "@uppy/react";
import { sha256 } from "crypto-hash";
import AwsS3Multipart from "@uppy/aws-s3-multipart";

// Uppy styles
import "@uppy/core/dist/style.min.css";
import "@uppy/dashboard/dist/style.min.css";

const fetchUploadApiEndpoint = async (endpoint: string, data: any) => {
  const res = await fetch(`/api/multipart-upload/${endpoint}`, {
    method: "POST",
    body: JSON.stringify(data),
    headers: {
      accept: "application/json",
      "Content-Type": "application/json",
    },
  });

  return res.json();
};

export function MultipartFileUploader({
  onUploadSuccess,
}: {
  onUploadSuccess: (result: UploadResult) => void;
}) {
  const uppy = useMemo(() => {
    const uppy = new Uppy({
      autoProceed: true,
    }).use(AwsS3Multipart, {
      createMultipartUpload: async (file) => {
        const arrayBuffer = await new Response(file.data).arrayBuffer();
        const fileHash = await sha256(arrayBuffer);
        const contentType = file.type;
        console.log("createMultipartUpload", file, fileHash, contentType);

        return fetchUploadApiEndpoint("create-multipart-upload", {
          file,
          fileHash,
          contentType,
        });
      },
      prepareUploadParts: async (file, partData) => {
        const response = await fetchUploadApiEndpoint("prepare-upload-parts", {
          file,
          partData,
        });

        return {
          presignedUrls: response.presignedUrls,
        };
      },
      completeMultipartUpload: async (file, props) => {
        const response = await fetchUploadApiEndpoint(
          "complete-multipart-upload",
          {
            file,
            ...props,
          }
        );

        console.log("completeMultipartUpload", response);

        return response;
      },
      listParts: async (file, props) => {
        console.log("inside listParts", file, props);
        const response = await fetchUploadApiEndpoint("list-parts", {
          file,
          ...props,
        });

        return response;
      },
      abortMultipartUpload: async (file, props) => {
        console.log("inside abortMultipartUpload", file, props);
        const response = await fetchUploadApiEndpoint(
          "abort-multipart-upload",
          {
            file,
            ...props,
          }
        );

        return response;
      },
    });
    return uppy;
  }, []);

  uppy.on("complete", (result) => {
    onUploadSuccess(result);
  });

  uppy.on("upload-success", (file, response) => {
    console.log("upload-success location", response.body.Location);
    uppy.setFileState(file.id, {
      progress: uppy.getState().files[file.id].progress,
      uploadURL: response.body.Location,
      response: response,
      isPaused: false,
    });
  });

  return <Dashboard uppy={uppy} showLinkToFileUploadResult={true} />;
}

This is how my API routes look like:

This is the prepareUploadParts method, which runs before the complete:

import {
    UploadPartCommand,
} from "@aws-sdk/client-s3";
import { type NextRequest, type NextResponse } from 'next/server'
import { R2, R2_BUCKET_NAME } from "@/lib/s3/util";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

export async function POST(
    req: NextRequest,
    res: NextResponse
) {
    const { partData } = await req.json();
    const parts = partData.parts;

    const response: { presignedUrls: { [key: string]: string } } = {
        presignedUrls: {},
    };

    for (let part of parts) {
        try {
            const params = {
                Bucket: R2_BUCKET_NAME,
                Key: partData.key,
                PartNumber: part.number,
                UploadId: partData.uploadId,
            };
            const command = new UploadPartCommand({ ...params });
            const url = await getSignedUrl(R2, command, { expiresIn: 3600 });

            response.presignedUrls[part.number] = url;
        } catch (err) {
            console.log("Error", err);
            return Response.json({
                status: 500,
                err
            });
        }
    }

    return Response.json({
        status: 200,
        response
    });
}

Upload and Multipart Upload Not working

I cloned this repo and followed all instructions,

but I am unable to upload files to my R2 bucket. I have not seen any errors in my VS Code. However, I am seeing three errors in my browser console.

1. Uncaught (in promise) SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON
2. app-index.js:35 [Uppy] [15:24:34] Unexpected token '<', "<!DOCTYPE "... is not valid JSON
3. POST http://localhost:3000/api/multipart-upload/abort-multipart-upload 500 (Internal Server Error)

Multipart upload fails for large files

Tested with 5GB and 8GB files.
Works fine for a 2GB file.

Missing License

Can you please upload a license file that explains what license this code uses?