Errors loading config larger than 1MB

[palantir]

Errors loading config larger than 1MB

[bluekeyes]

Hey @SoulPancake, thanks for the suggestion. I think we can simplify this further by making one request with the application/vnd.github.raw content type, checking for the file size error, and then using the getLargeFileContents method in that case. The raw content type is supposed to work for all files up to 100MB, although I haven't tested it yet to make sure it still returns the expected error for files larger than 100MB.

Also, it looks like the Media property is not a field in the RepositoryContentGetOptions, at least in go-github v50.2.0. Are you looking at a different version of the library? I think we'll probably have to fall back to the lower-level methods provided by go-github for this request.

[SoulPancake]

@bluekeyes
Can we do something like

func getFileContents(ctx context.Context, client *github.Client, owner, repo, ref, path string) ([]byte, bool, error) {
    file, _, _, err := client.Repositories.GetContents(ctx, owner, repo, path, &github.RepositoryContentGetOptions{
        Ref: ref,
    })
    if err != nil {
        switch {
        case isNotFound(err):
            return nil, false, nil
        case isTooLargeError(err):
            b, err := getLargeFileContents(ctx, client, owner, repo, ref, path)
            return b, true, err
        }
        return nil, false, errors.Wrap(err, "failed to read file")
    }

    // file will be nil if the path exists but is a directory
    if file == nil {
        return nil, false, nil
    }

    if file.GetSize() > 100*1024*1024 {
        b, err := getLargeFileContents(ctx, client, owner, repo, ref, path)
        return b, true, err
    } else if file.GetSize() > 1024*1024 {
        rawFile, _, _, err := client.Repositories.GetContents(ctx, owner, repo, path, &github.RepositoryContentGetOptions{
            Ref: ref,
            Media: "application/octet-stream",
        })
        if err != nil {
            return nil, false, errors.Wrap(err, "failed to read file content")
        }
        content, err := rawFile.GetContent()
        if err != nil {
            return nil, true, errors.Wrap(err, "failed to decode file content")
        }
        return []byte(content), true, nil
    }

    content, err := file.GetContent()
    if err != nil {
        return nil, true, errors.Wrap(err, "failed to decode file content")
    }

    return []byte(content), true, nil
}