alanedwardes/estranged.lfs

A Git LFS server implementation in C# designed to run in a serverless environment.

Estranged.Lfs

A Git LFS backend which provides pluggable authentication and blob store adapters. It is designed to run in a serverless environment to be used in conjunction with a Git provider such as GitHub or BitBucket, or self hosted Git.

Basic Usage

1. Add the Git LFS services to your application:

services.AddLfs();

2. Register an implementation for IBlobAdapter and IAuthenticator. Amazon AWS S3 and Azure Blob Storage are provided out of the box:

var s3BlobConfig = new S3BlobAdapterConfig
{
    Bucket = "estranged-lfs-test"
};
services.AddLfsS3Adapter(s3BlobConfig, new AmazonS3Client());
services.AddLfsDictionaryAuthenticator(new Dictionary<string, string>{{"username","password"}});

Or use the following example for Azure Blob Storage.

Note: Keep the naming rules for Azure Blob Storage in account, review them here.

var blobServiceClient = new Azure.Storage.Blobs.BlobServiceClient("<your connection string here>");
var blobConfig = new AzureBlobAdapterConfig
{
    ContainerName = "estranged-lfs-test"
};
services.AddLfsAzureBlobAdapter(blobConfig, blobServiceClient);
services.AddLfsDictionaryAuthenticator(new Dictionary<string, string> {{"username","password"}});

GitHub Authenticator

A GitHub authenticator implementation is provided out of the box. This authenticator takes the supplied username and password and makes a "get repository" call against the GitHub API. If the result is that the user has access, the LFS call succeeds, if the user does not have access, the LFS call fails with a 401 error.

To configure the GitHub authenticator, you need to register it with the IServiceProvider:

// services.AddLfsDictionaryAuthenticator(new Dictionary<string, string>{{"username","password"}});
var ghAuthConfig = new GitHubAuthenticatorConfig
{
    Organisation = "alanedwardes",
    Repository = "Estranged.Lfs"
};
services.AddLfsGitHubAuthenticator(ghAuthConfig);

When LFS prompts you for credentials, enter your GitHub username, and a personal access token to authenticate. Your token should have the "repository read" scope.

BitBucket Authenticator

A BitBucket authenticator implementation is provided out of the box. This authenticator takes the supplied username and password and makes a "get repository" call against the BitBucket API. If the result is that the user has access, the LFS call succeeds, if the user does not have access, the LFS call fails with a 401 error.

To configure the BitBucket authenticator, you need to register it with the IServiceProvider:

// services.AddLfsDictionaryAuthenticator(new Dictionary<string, string>{{"username","password"}});
var bbAuthConfig = new BitBucketAuthenticatorConfig
{
    Workspace = "alanedwardes",
    Repository = "Estranged.Lfs"
};
services.AddLfsBitBucketAuthenticator(bbAuthConfig);

When LFS prompts you for credentials, enter your BitBucket username, and a personal access token to authenticate. Your token should have the "repository read" scope.

Extensibility

Blob Adapter

Any blob store which generates pre-signed URLs can be used by implementing the interface IBlobAdapter:

public interface IBlobAdapter
{
    Task<SignedBlob> UriForUpload(string oid, long size, CancellationToken token);
    Task<SignedBlob> UriForDownload(string oid, CancellationToken token);
}

An S3 implementation is included, which generates pre-signed GET and PUT requests. This can be used out of the box if desired.

Authentication Adapter

Git LFS supports HTTP Basic authentication, the mechanics of which the library deals with but the authentication portion is exposed behind the IAuthenticator interface.

public interface IAuthenticator
{
    Task Authenticate(string username, string password, LfsPermission requiredPermission, CancellationToken token);
}

A sample implementation exposing a dictionary of username => password is included as a reference.

Example Deployables

There are currently two hosting examples:

• Estranged.Lfs.Hosting.AspNet
• Estranged.Lfs.Hosting.Lambda

The former is a simple example using only Asp.NET components, and the latter is an Asp.NET Lambda function which can be deployed directly to AWS Lambda, behind API Gateway.

Deploying to Lambda

1. Head over to the Estranged.Lfs.Hosting.Lambda project in the hosting folder.
2. Install the dotnet-lambda global tool from AWS: https://github.com/aws/aws-extensions-for-dotnet-cli
3. Edit the aws-lambda-tools-defaults.json file to suit your environment setup:

{
    "profile": "default",
    "configuration": "Release",
    "framework": "net6.0",
    "function-handler": "Estranged.Lfs.Hosting.Lambda::Estranged.Lfs.Hosting.Lambda.LambdaEntryPoint::FunctionHandlerAsync",
    "function-memory-size": 256,
    "function-timeout": 30,
    "function-runtime": "dotnet6",
    "region": "<aws region>",
    "s3-bucket": "<s3 bucket to upload the lambda to>",
    "s3-prefix": "<path in s3 to upload the lambda to>",
    "function-name": "<lambda name to deploy or update>",
    // Set other variables required by the Lambda function
    "environment-variables": "LFS_BUCKET=<lfs s3 bucket>;<key>=<value>"
}

4. Run dotnet-lambda deploy-serverless to deploy the Lambda function

estranged.lfs's Issues

Regarding the magic in git-lfs.yaml

I previously tried using the magic link from your blog to set up the stack. Since I'm in us-west I thought changing the console region would do the trick, but not quite.

After getting the manual setup to work (thanks for the git lfs version hint!), I looked at the git-lfs.yaml and am seeing that the line:

Code: S3Bucket: !Sub 'ae-infrastructure-${AWS::Region}' S3Key: git-lfs/git-lfs-lambda.zip

Should actually hardcode the region in this case (as its pointing to the packaged version of this codebase you've kindly uploaded to eu-west-1). So instead the code should be:

Code: S3Bucket: !Sub 'ae-infrastructure-eu-west-1.s3.amazonaws.com' S3Key: git-lfs/git-lfs-lambda.zip

Support for Azure blob storage

Hi,

Thanks, I was looking for a self-hosted alternative to GitHub's LFS storage because I'm currently blasting through my LFS storage limit's. I came across your library and blog posts which is exactly what I was looking for.

The thing is I would like to store my LFS files in the Azure cloud, so I decided to code support for azure blob storage (basically the Azure equivalent of S3).

I'll open a PR once I'm done, so you can take a look.

Let me know what you think.

Stephanvs

[Feature Request]: Convert Estranged.Lfs.Api to utilize minimal APIs instead of controllers

This may be an overkill suggestion, but as it doesn't seem possible (or easy) to add route prefixes to third-party controllers, I decided to convert them to minimal API methods. This may allow more user flexibility, if for some reason a user may require more configuration beyond customizing the route prefix.
Example for POST /objects/batch:

public static void MapGitHubLfsBatch(this IEndpointRouteBuilder endpoints, string pathBase = "")
{
    endpoints.MapPost($"{pathBase}/objects/batch", async (IObjectManager objectManager, [FromBody] BatchRequest request) =>
    {
        return request.Operation switch
        {
            LfsOperation.Upload => new BatchResponse
            {
                Transfer = request.Transfers.First(),
                Objects = await objectManager.UploadObjects(request.Objects, GenerateTimeoutToken())
                    .ConfigureAwait(false)
            },
            LfsOperation.Download => new BatchResponse
            {
                Transfer = request.Transfers.First(),
                Objects = await objectManager.DownloadObjects(request.Objects, GenerateTimeoutToken())
                    .ConfigureAwait(false)
            },
            _ => throw new Exception("Method not implemented")
        };
    }).AddEndpointFilter<BasicAuthEndpointFilter>();
    
    static CancellationToken GenerateTimeoutToken() => new CancellationTokenSource(TimeSpan.FromSeconds(25)).Token;
}

*Note: BasicAuthEndpointFilter is just a copy of the already existing BasicAuthFilter, converted to implement IEndpointFilter instead of IAsyncActionFilter.

I would be happy to submit a PR, but always like asking in Issues before bothering to do so.

git lfs prune --verify-remote behave unexpectedly

git lfs prune --verify-remote behave unexpectedly when combined with Estranged.Lfs .
It seems Estranged.Lfs always responds to inquiries with a positive answer that is as if "the file in question does exists" unconditionally.

GitLab authenticator

Estranged.Lfs only supports authentication via GitHub and BitBucket. A GitLab authentication method would be highly desired. Estranged.Lfs uses Octokit for connecting with the Github API (to authenticate user). Maybe GitLabApiClient could be used to implement a similar authentication method for GitLab.

Make API endpoints return the correct status code when returning an error

Hi,

I am using this library with the Estranged.Lfs.Api and Estranged.Lfs.Authenticator.GitHub addons.

Not sure if this is an issue on my end, but looking at source code and other examples, I have built my own IBlobAdapter, which will return errors in certain circumstances. Testing this in Postman locally, my error object is populated and code is set, but the status code returned by the server is still 200 OK. It would be nice to set the status code in the response when handled internally by the library, as as far as I'm aware I do not have control over this.

Support for OVH Object Storage

Hello Alan,

Huge thanks to you to provide a working out of the box solution!

The ideal solution (at least for me) would be to use OVH Object Storage (hosted in France and cheaper) instead than AWS S3.

But I am new to cloud and it looks confusing, so maybe you see clearly what should be done to configure Estranged.Lfs.

OVH say that they are S3-compatible, here is their guide.
Does it mean that is is possible to use your solution as is ? (only changing the S3 URL somehow)

Or is it needed to implement a IBlobAdapter? By using AmazonS3Client too?

I am a bit lost...

LFS: Authorization error from Cloudfront template

Using the Amazon S3 template results in an LFS: Authorization error. "Check that you have proper access to the repository"

Everything was generated via the Cloudfront template so I'm not sure where access is being denied.