File Upload Application for Buckets in Oracle Cloud Infrastructure

README - Português-BR
About

About

The upload-bucket-service project is an automatic file¹ upload application for Buckets in the Oracle Cloud Infrastructure (OCI) Object Storage service.

With few configurations it is possible to map multiple directories² and upload to Oracle Object Storage, allowing you to configure file overwriting, scheduling and webhook notification.

You will be able to:

Upload files from multiple directories;
Assist in the migration of extensive data;
Save and replicate backup files in a secure and automated way.
Configure webhooks to receive notification of each run with job status and details.

See all samples on Wiki.

Caution: this app does not sync deleted files, that is, it does not delete the file in the Bucket, as the intention is to provide a secure way of syncing files, especially for backups. For file updates, it will only update modified files if the service.folders[*].overwriteExistingFile parameter is true.

Note 1: file size limit is 50GB.

Note 2: when used for the purpose of transferring backups, you must remove the OBJECT_READ permission in the OCI permissions policy and disable file overwriting for the directory. It is recommended to disable the OBJECT_DELETE permission, also in the OCI permissions policy for the user in which it is used to communicate with the service's API, thus ensuring greater security and integrity of the backup files. More info see Securing Object Storage

What you will need

OCI Client API key file and key file - see more details on About OCI API key - .oci;
Configure directories in application.properties file;
Only that!

See more details on Settings below.

Service

See On-demand and service compatible below.

System Requirements

Java: Java 1.8, Java 11 or later;
Operational System: Windows, Unix/Linux, MacOS (not tested);
RAM: ~5MB (may vary when file upload is running);
Disk space: ~35MB.

Features

On-demand and service compatible

The application can be run on demand or configured as a service on Windows and Linux.

Multiple directories

The application allows the configuration of one or more directories, in which it is possible to enable or disable it and enable the overwriting of files. Directory analysis is performed in parallel by multiple threads and asynchronously, thus ensuring better performance and without spending resources for a long time.

Multiple buckets

You can choose to send to the global bucket or to a different bucket per directory.

service.oci.bucket for global configuration;
service.folders[*].oci.bucket for specific folder.

If the bucket does not exist, use the 'createBucketIfNotExists' option to create a new bucket.

service.oci.createBucketIfNotExists for global configuration;
service.folders[*].oci.createBucketIfNotExists=true for a specific folder.

Job scheduling

The application uses Cron expression notation, in which it is possible to specify an interval or time for the execution of the job. You can set a default value for all jobs (service.cron) or set specific values for each folder (service.folders[*].cron).

Samples:

Expression	Meaning
0 0 12 * * ?	Execute at 12:00 every day
0 15 10 ? * *	Execute at 10:15 every day
0/10 * * * * ?	Execute every 10 seconds
0 0/30 * * * ?	Execute every 30 minutes

Attempts failure

Allows you to set execution attempts when a failure occurs during job processing.

Webhook notification

You can configure an API URL that you will be notified when the job finishes running (either to failure or success). Your API must accept the POST method.

The body can be JSON or XML, just configure the service.hookContentType with one of the values:

application/json
application/xml

Sample JSON success:

On details field, each directory is separated by the '¢' character and the directory properties are separated by the ';' character.

{
	"jobName": "DEFAULT_CRON_JOB",
	"jobStatus": "COMPLETED",
	"details": "DIRECTORY=C:/temp;CRON=0/10 * * * * ?;BUCKET=teste¢DIRECTORY=C:/temp2;CRON=0/10 * * * * ?;BUCKET=teste",
	"createdTime": "2022-10-12T22:26:05+0000",
	"endTime": "2022-10-12T22:26:08+0000",
	"exceptions": []
}

Sample JSON error:

{
  "jobName": "DEFAULT_CRON_JOB",
  "jobStatus": "FAILED",
  "details": "DIRECTORY=C:/temp;CRON=0/10 * * * * ?;BUCKET=teste",
  "createdTime": "2022-10-12T22:57:54+0000",
  "endTime": "2022-10-12T22:57:57+0000",
  "exceptions": [
    {
      "cause": null,
      "stackTrace": [
        {
          "classLoaderName": null,
          "moduleName": null,
          "moduleVersion": null,
          "methodName": "lambda$parseJobExecution$0",
          "fileName": "Hook.java",
          "lineNumber": 68,
          "className": "dev.alexmaycon.bucketservice.hook.model.Hook",
          "nativeMethod": false
        }
      ],
      "message": "It's a test!!!!",
      "suppressed": [],
      "localizedMessage": "It's a test!!!!"
    }
  ]
}

Settings

Installation folder structure:

root
│   upload-bucket-service.jar
│   .oci
|   application.properties

application.properties

Attention: You must have at least one directory using the global service.cron.

Property	Description	Required	Default Value	Type
service.cron	Default cron expression to all jobs execution	No	"0/10 * * * * ?"	String
service.hook	API endpoint URL at which to be notified (POST) at the end of the JOB execution.	No		String
service.hookContentType	Media type (json/xml)	No	"application/json"	String
service.attemptsFailure	Number of attempts when a failure occurs	No	1	int
service.oci.profile	Profile session of .oci configuration	No	"DEFAULT"	String
service.oci.bucket	OCI Bucket name	Yes		String
service.oci.compartmentOcid	Compartment OCID - if you wanted to create the bucket in a specific compartment.	No		String
service.folders[*]	Folders configuration	Yes		List
service.folders[*].directory	Folder path (need to include escape character for \ on Windows)	Yes		String
service.folders[*].cron	Cron expression specifies to the folder	No	Value from service.cron	String
service.folders[*].overwriteExistingFile	Enable/disable file overwriting	No	false	boolean
service.folders[*].enabled	Enables/disables folder processing.	No	true	boolean
service.folders[*].mapToBucketDir	Set the directory to use in bucket. Leave empty to use root.	No		String
service.folders[*].oci.profile	Profile session of .oci configuration (apply only to folder)	No	"DEFAULT"	String
service.folders[*].oci.bucket	OCI Bucket name (apply only to folder)	No		String
service.folders[*].oci.compartmentOcid	Compartment OCID - if you wanted to create the bucket in a specific compartment. (apply only to folder)	No		String

About OCI API key - .oci

After getting the file, rename the file to .oci and paste it in the same folder as the application.

Sample .oci file:

[DEFAULT]
user=
fingerprint=
tenancy=
region=
key_file=

Sample application.properties file

service.cron=0 0/30 * * * ?
service.oci.profile=DEFAULT
service.oci.bucket=bkp_bd
service.folders[0].directory=C:/temp
service.folders[0].cron=0 0/10 * * * ?
service.folders[1].directory=C:/temp2
service.folders[1].overwriteExistingFile=false
service.folders[1].enabled=true
service.folders[2].directory=C:/temp3
service.folders[2].overwriteExistingFile=false
service.folders[2].enabled=true
service.folders[3].directory=C:/temp4
service.folders[3].overwriteExistingFile=false
service.folders[3].enabled=true
service.folders[4].directory=C:/temp4
service.folders[4].overwriteExistingFile=false
service.folders[4].enabled=true
service.attemptsFailure=5

Building

This project was written in Java with Spring Batch and Oracle Cloud Infrastructure Client SDK.

To build we use Maven.

Running DEV profile

The DEV profile should be used for debugging and testing. You will need to change the value of the --oci property.

Go to the project root directory and run:

mvn clean package install
mvn spring-boot:run -D"spring-boot.run.profiles=dev" -D"spring-boot.run.arguments"="--oci=/path/to/.oci"

The project has configuration files for IntelliJ IDEA, allowing these configurations to be performed in the IDE:

Maven
- [clean, install]
- [clean, package]
Spring Boot
- upload-bucket-service (DEV)

Running

On-demand

java -jar upload-bucket-service.jar

Code Of Conduct

We encourage community participation, but be aware of the rules in our Code of Conduct.

See CODE OF CONDUCT for details.

Contributing

upload-bucket-service is an open source project, stay free to participate with us.

See CONTRIBUTING for details.

License

See LICENSE for more details.

395784985 / upload-bucket-service Goto Github PK

upload-bucket-service's Introduction