guitarrapc/s3sync

Amazon S3 Content Synchronization with .NET

S3Sync

Amazon S3 Content Synchronization with .NET.

S3Sync synchronize a directory to a S3 Bucket. It meakes bucket identical to the LocalRoot (source).

Note: Remote files that are not in the LocalRoot are removed.

How to use

You can download latest version from Release Page.

Action	Full.NET	.NETCore 2.0	Docker
Requirement	.NETFreamework 4.7 or higher	.NETCore 2.0 or higher	Docker
Download	s3sync_netfull.zip	s3sync_netcore.tar.gz	guitarrapc/s3sync
Run	Extract zip and run S3Sync.exe	Extract zip and run dotnet S3Sync.dll	docker run guitarrapc/s3sync

Configuration

You can pass parameter to S3Sync with Arguments or Environment Variable.

Arguments	Environment Variable	Required? Optional?	Description
BucketName="string"	S3Sync_BucketName	Required	Specify S3 BucketName to sync.
LocalRoot="string"	S3Sync_LocalRoot	Required	Specify Local File Path to Sync.
KeyPrefix="string"	S3Sync_KeyPrefix	Optional	Specify KeyPrefix to add to localfile when Sync.
IgnoreKeyPrefix="string"	S3Sync_IgnoreKeyPrefix	Optional	Specify KeyPrefix to ignore on S3.
ExcludeFiles="string","string"	S3Sync_ExcludeFiles	Optional	Specify local file names you want to exclude. (use , for multiple.)
ExcludeDirectories="string","string"	S3Sync_ExcludeDirectories	Optional	Specify local directory names you want to exclude. (use , for multiple.)
CredentialProfile="string"	S3Sync_CredentialProfile	Optional	Specify Credential Profile name.
Silent=bool	S3Sync_Silent	Optional	Set true when you want to supress upload progress. (Default : false)
DryRun=bool	S3Sync_DryRun	Optional	Set true will not change s3 but see estimate plan. Set false to execute synchronization. (Default : true)
ContentType="string"	S3Sync_ContentType	Optional	Specify ContentType for object. (default null and will be application/octet-stream
Region="string"	S3Sync_Region	Optional	Specify region for the bucket. (default null and will be ap-northeast-1)

Sample

You can use dotnet to run as .NETCore.

$ dotnet S3Sync.dll BucketName=your-awesome-bucket LocalRoot=/Home/User/HogeMoge ExcludeFiles=.gitignore,.gitattributes ExcludeDirectories=.git,test

No .NETCore? You can use Full.NET as a ConsoleApp.

S3Sync.exe BucketName=your-fantastic-bucket KeyPrefix=hoge LocalRoot=C:/Users/User/HomeMoge DryRun=false

Docker Support

You can run with docker.

Run with IAM Role is recommended.

docker run --rm -v <YOUR_SYNC_DIR>:/app/sync/ -e S3Sync_BucketName=<YOUR_BUCKET_NAME> S3Sync_DryRun=false guitarrapc/s3sync

Local run without IAM Role, use AWS Credentials.

$ docker run --rm -v <YOUR_SYNC_DIR>:/app/sync/ -e S3Sync_BucketName=<YOUR_BUCKET_NAME> -e AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY> -e AWS_SECRET_ACCESS_KEY=<YOUR_SECRET> S3Sync_DryRun=false guitarrapc/s3sync

Build s3sync within docker

Build S3Sync with docker-compose. This enable you not to think about .NETCore2.0 sdk installed on your host.

docker-compose -f docker-compose.ci.build.yml up

Build artifacts will be generated in following path.

S3Sync\source\S3Sync\obj\Docker\publish

Clean up build docker container resource with down.

docker-compose -f docker-compose.ci.build.yml down

Docker Image Build

Create docker image with docker-compose.

docker-compose -f docker-compose.yml build

Credential handling

Synchronization operation requires read, write and delete objects permission.

It is recommended that you use IAM Policy and Profile to handle appropriate access right.

Configure IAM Policy

Here's some sample IAM Policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1446117060000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ]
        }
    ]
}

If you want to restrict access to certain Bucket, then replace * with desired bucketName.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1446117060000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::PutYourBucketName"
            ]
        }
    ]
}

Configure Profile

There are several way to set profile.

If you run S3Sync on AWS Resource, you should AWS managed profile like IAM Instance Profile. If you run S3Sync on Local Environment, you can configure your machine with "aws cli" or otheres.

aws cli onfigure sample

You can create AWS Credential Profile with AWS CLI.

aws configure --profile sample

Other options?

You can create Profile with other tools.

• AWS Toolkit for Visual Studio .
• AWS Tools for Windows PowerShell

Or you can use following method.

public static void RegisterProfile(string profileName, string accessKey, string accessSecret)
{
    var option = new CredentialProfileOptions
    {
        AccessKey = accessKey,
        SecretKey = accessSecret
    };
    new NetSDKCredentialsFile().RegisterProfile(new CredentialProfile(profileName, option));
}

License

The MIT License (MIT)

s3sync's Issues

ContentType Support

DryRun Support

No File Change but notify what change will be shown.

Docker Support

https://hub.docker.com/r/guitarrapc/s3sync/

Use latest AWS.SDK 3.7.x

S3Sync/source/S3Sync/S3Sync.csproj

Line 10 in 21e0333

<PackageReference Include="AWSSDK.S3" Version="3.3.111.33" />

Use a command line parsing library

S3Sync/source/S3Sync/Program.cs

Lines 137 to 251 in 21e0333

	static void EvaluateArguments(string[] args)
	{
	// BucketName=nantokakantokabucket
	BucketName = args.Where(x => x.StartsWith(ArgumentType.BucketName.ToString(), StringComparison.InvariantCultureIgnoreCase))
	.SelectMany(x => x.SplitEx("="))
	.LastOrDefault()
	?? GetEnvValueString(ArgumentType.BucketName, EnvType.S3Sync_BucketName);
	// LocalRoot=c:\hogemoge
	LocalRoot = args.Where(x => x.StartsWith(ArgumentType.LocalRoot.ToString(), StringComparison.InvariantCultureIgnoreCase))
	.SelectMany(x => x.SplitEx("="))
	.LastOrDefault()
	?? GetEnvValueString(ArgumentType.LocalRoot, EnvType.S3Sync_LocalRoot);
	// KeyPrefix=image
	KeyPrefix = args.Where(x => x.StartsWith(ArgumentType.KeyPrefix.ToString(), StringComparison.InvariantCultureIgnoreCase))
	.SelectMany(x => x.SplitEx("="))
	.LastOrDefault()
	?.TrimEnd('/')
	?? GetEnvValueString(ArgumentType.KeyPrefix, EnvType.S3Sync_KeyPrefix);
	// IgnoreKeyPrefix=image
	IgnoreKeyPrefix = args.Where(x => x.StartsWith(ArgumentType.IgnoreKeyPrefix.ToString(), StringComparison.InvariantCultureIgnoreCase))
	.SelectMany(x => x.SplitEx("="))
	.LastOrDefault()
	?.TrimEnd('/')
	?? GetEnvValueString(ArgumentType.IgnoreKeyPrefix, EnvType.S3Sync_IgnoreKeyPrefix);
	// ExcludeFiles=hogemoge,fugafuga
	ExcludeFiles = args.Where(x => x.StartsWith(ArgumentType.ExcludeFiles.ToString(), StringComparison.InvariantCultureIgnoreCase))
	.SelectMany(x => x.SplitEx("="))
	.LastOrDefault()
	?.SplitEx(",")
	.Select(x => x.Trim())
	.ToArray()
	?? GetEnvValueString(ArgumentType.ExcludeFiles, EnvType.S3Sync_ExcludeFiles)
	?.SplitEx(",");
	// ExcludeDirectories=hogemoge,fugafuga
	ExcludeDirectories = args.Where(x => x.StartsWith(ArgumentType.ExcludeDirectories.ToString(), StringComparison.InvariantCultureIgnoreCase))
	.SelectMany(x => x.SplitEx("="))
	.LastOrDefault()
	?.SplitEx(",")
	?.Select(x => x.Trim())
	.ToArray()
	?? GetEnvValueString(ArgumentType.ExcludeDirectories, EnvType.S3Sync_ExcludeDirectories)
	?.SplitEx(",");
	// Silent=false
	Silent = bool.Parse(args.Where(x => x.StartsWith(ArgumentType.Silent.ToString(), StringComparison.InvariantCultureIgnoreCase))
	.SelectMany(x => x.SplitEx("="))
	.Where(x => string.Equals(x, "true", StringComparison.InvariantCultureIgnoreCase) || string.Equals(x, "false", StringComparison.InvariantCultureIgnoreCase))
	.LastOrDefault()
	?.Trim()
	?? GetEnvValueString(ArgumentType.Silent, EnvType.S3Sync_Silent)
	?? "false");
	// CredentialProfile=ProfileName
	CredentialProfile = args.Where(x => x.StartsWith(ArgumentType.CredentialProfile.ToString(), StringComparison.InvariantCultureIgnoreCase))
	.SelectMany(x => x.SplitEx("="))
	.LastOrDefault()
	?.Trim()
	?? GetEnvValueString(ArgumentType.CredentialProfile, EnvType.S3Sync_CredentialProfile);
	// DryRun=true
	Option.DryRun = bool.Parse(args.Where(x => x.StartsWith(ArgumentType.DryRun.ToString(), StringComparison.InvariantCultureIgnoreCase))
	.SelectMany(x => x.SplitEx("="))
	.Where(x => string.Equals(x, "true", StringComparison.InvariantCultureIgnoreCase) || string.Equals(x, "false", StringComparison.InvariantCultureIgnoreCase))
	.LastOrDefault()
	?.Trim()
	?? GetEnvValueString(ArgumentType.DryRun, EnvType.S3Sync_DryRun)
	?? "true");
	// ContentType=application/json
	Option.ContentType = args.Where(x => x.StartsWith(ArgumentType.ContentType.ToString(), StringComparison.InvariantCultureIgnoreCase))
	.SelectMany(x => x.SplitEx("="))
	.LastOrDefault()
	?.TrimEnd('/')
	?? GetEnvValueString(ArgumentType.ContentType, EnvType.S3Sync_ContentType);
	// Region=us-east-1
	Option.Region = args.Where(x => x.StartsWith(ArgumentType.Region.ToString(), StringComparison.InvariantCultureIgnoreCase))
	.SelectMany(x => x.SplitEx("="))
	.LastOrDefault()
	?.TrimEnd('/')
	?? GetEnvValueString(ArgumentType.Region, EnvType.S3Sync_Region);
	// Show Arguments
	Log($"{nameof(BucketName)} : {BucketName}");
	Log($"{nameof(LocalRoot)} : {LocalRoot}");
	Log($"{nameof(KeyPrefix)} : {KeyPrefix}");
	Log($"{nameof(IgnoreKeyPrefix)} : {IgnoreKeyPrefix}");
	Log($"{nameof(ExcludeFiles)} : {ExcludeFiles?.ToJoinedString(",")}");
	Log($"{nameof(ExcludeDirectories)} : {ExcludeDirectories?.ToJoinedString(",")}");
	Log($"{nameof(Silent)} : {Silent}");
	Log($"{nameof(CredentialProfile)} : {CredentialProfile}");
	Log($"{nameof(Option.DryRun)} : {Option.DryRun}");
	Log($"{nameof(Option.ContentType)} : {Option.ContentType}");
	Log($"{nameof(Option.Region)} : {Option.Region}");
	// Validate Required arguments
	if (string.IsNullOrWhiteSpace(BucketName))
	{
	Error("Please pass arguments. See detail followings.");
	PrintHelp();
	throw new NullReferenceException(nameof(BucketName));
	}
	if (string.IsNullOrWhiteSpace(LocalRoot))
	{
	Error("Please pass arguments. See detail followings.");
	PrintHelp();
	throw new NullReferenceException(nameof(LocalRoot));
	}
	}

What do these ServicePointManager changes accomplish?

S3Sync/source/S3Sync/Program.cs

Lines 129 to 135 in 21e0333

	static void ApplyInitialConfiguration()
	{
	// Web周りの設定
	ServicePointManager.UseNagleAlgorithm = false;
	ServicePointManager.Expect100Continue = false;
	ServicePointManager.DefaultConnectionLimit = 96;
	}

.NET Core 3.1 support

Array dimensions exceeded supported range

Hi, bug found Array dimensions exceeded supported range., System.OverflowException, at S3Manager.Core.LocalFiles.FileHashHelper.GetFileBinary(String fileName) at S3Manager.Core.S3Client.<>c__DisplayClass44_0.<GetSysncStatus>b__4(KeyValuePair2 x)
at System.Linq.Enumerable.SelectEnumerableIterator2.ToArray() at System.Linq.Enumerable.ToArray[TSource](IEnumerable1 source)
at S3Manager.Core.S3Client.GetSysncStatus(IEnumerable1 localFiles, IEnumerable1 s3Objects, String prefix)
at S3Manager.Core.S3Client.SyncWithLocal(SlimFileInfo[] localFileInfos, String bucketName, String prefix, String ignorePrefix, Action1 uploadCallback) at MTBackup.Program.MainCoreAsync(String[] args) at MTBackup.Program.Main(String[] args)

Once you try to sync files larger than 2GB