A polycloud .NET cloud storage abstraction layer. Provides Blob storage (AWS S3, GCP, FTP, SFTP, Azure Blob/File/Event Hub/Data Lake) and Messaging (AWS SQS, Azure Queue/ServiceBus). Supports .NET 5+ and .NET Standard 2.0+. Pure C#.
License: MIT License
Hi,
We have a Problem with the AwsS3BlobStorage. We have a read-only S3-Bucket. After creating a BlobStorage from our ConnectionString we get an Access Denied Exception cause of the PutBucketRequest in the GetClientAsync method from AwsS3BlobStorage. Would it be possible to set the PutBucketRequest optional?
This issue was already requested in Storage.NET (Issue) and a pull request (PullRequest)
When using the zip-backed module, Windows explorer claims the archive is empty despite being > 0 size.
Opening the archive in 7-Zip shows a folder at the top level, displayed as '_', and when browsing to it the data I'm expecting at root is there, and the path in the UI shows as:
my_zip.zip\\
I think what might be happening is that a leading '/' is being provided when getting an entry in the archive, but I'm not 100% on this.
Currently the fileprefix is applied client side. If there are many files in a path then this can be very slow.
A small change to AWSS3DirectoryBrowser would allow the filtering to be applied on the server.
E.g.
Task ListFolderAsync(List<Blob> container, string path, ListOptions options, CancellationToken cancellationToken)
{
var request = new ListObjectsV2Request()
{
BucketName = _bucketName,
Prefix = FormatFolderPrefix(path),
Delimiter = "/" //this tells S3 not to go into the folder recursively
};
// if we have provided a file prefix, then append it on.
// as S3 can perform the filtering at source
if (!string.IsNullOrEmpty(options.FilePrefix))
{
request.Prefix += options.FilePrefix;
}
The current implementation of IBlobStorage.WriteAsyncdoes not honor the append hint value in FluentStorage.FTP implementation
public async Task WriteAsync(string fullPath, Stream dataStream,
bool append = false, CancellationToken cancellationToken = default) {
AsyncFtpClient client = await GetClientAsync().ConfigureAwait(false);
await retryPolicy.ExecuteAsync(async () => {
using (Stream dest = await client.OpenWrite(fullPath, FtpDataType.Binary, true).ConfigureAwait(false)) {
await dataStream.CopyToAsync(dest).ConfigureAwait(false);
}
}).ConfigureAwait(false);
}I think the stream should be created using client.OpenAppend when the parameter append is true and client.OpenWrite otherwise.
Is it possible to have a connection string to Azure Blob storage that will use DefaultAzureCredential? I don't want to pass any keys to access since it is less secure.
Will it support minio later?
Are there plans to support MongoDB GridFS
A great addition when debugging would be to be able to step into the code of the library.
Enabling SourceLink for open source libraries is almost a standard nowadays and greatly help reporting bugs (if any) but more important it's an opportunity to better understand how the library works
I want to read an object from an S3 storage with ReadBytesAsync. However, I get an exception saying that I am not authorized to perform create-bucket. Why do I need that permission for reading?
Edit: I found the issue in the implementation. The method GetClientAsync in AwsS3BlobStorage makes a PutBucket request:
Why does it do that? It should at least check if the bucket already exists.
You can fix it like so:
private async Task<AmazonS3Client> GetClientAsync() {
if (!_initialised) {
var bucketExists = await AmazonS3Util.DoesS3BucketExistV2Async(_client, _bucketName);
if (!bucketExists) {
var request = new PutBucketRequest { BucketName = _bucketName };
await _client.PutBucketAsync(request).ConfigureAwait(false);
}
_initialised = true;
}
return _client;
}I can do a PR if you want?
The append argument does not appear to do anything in WriteAsync for FluentStorage.SFTP.
await remoteStorage.WriteAsync("/home/test/test_document_2", new byte[] { (byte)'1' }, true);
If I run this multiple times I'd expect 1 to be appended to the file, but it is not.
PR #57 was updated to solve this.
Both the AWS and Azure interfaces for blobs, IAwsS3BlobStorage and IAzureBlobStorage have a definition for pre-signed/SAS URLs (GetUploadUrlAsync for AWS and GetBlobSasAsync for AWS).
Can we please:
Hi Guys, do you have plan to upgrade the library to support .NET 8. Thank you
Sometimes when doing the following:
using var stream = ... // a Stream here
await blobStorage.WriteAsync("path_here", stream);OR:
var stream = ... // a Stream here
await blobStorage.WriteAsync("path_here", stream);
await stream.DisposeAsync();OR:
var bytes = ... // a byte array
await blobStorage.WriteAsync("path_here", bytes);Occasionally a System.ObjectDisposedException: 'Cannot access a closed file.' will be raised.
This code path is being called by a background processor that utilises a worker pool, the idea being that files for writing are enqueued as tasks, and the pool will wait for all tasks to complete; so I'm wondering if there's a threading / async issue occurring here?
after CreateFolderAsync why .empty create Automatically is it necessary? how skip to create it?
Settings
public class KomodoSettings : BlobSettings
{
#region Public-Members
/// <summary>
/// Komodo endpoint URL, of the form http://[hostname]:[port]/.
/// </summary>
public string Endpoint { get; set; } = null;
/// <summary>
/// Komodo index GUID.
/// </summary>
public string IndexGUID { get; set; } = null;
/// <summary>
/// Komodo API key.
/// </summary>
public string ApiKey { get; set; } = null;
/// <summary>
/// Initialize the object.
/// </summary>
/// <param name="endpoint">Komodo endpoint, i.e. http://localhost:8000/</param>
/// <param name="indexGuid">GUID of the index.</param>
/// <param name="apiKey">API key with read, write, and delete permissions.</param>
public KomodoSettings(string endpoint, string indexGuid, string apiKey)
{
if (String.IsNullOrEmpty(endpoint)) throw new ArgumentNullException(nameof(endpoint));
if (String.IsNullOrEmpty(indexGuid)) throw new ArgumentNullException(nameof(indexGuid));
if (String.IsNullOrEmpty(apiKey)) throw new ArgumentNullException(nameof(apiKey));
Endpoint = endpoint;
IndexGUID = indexGuid;
ApiKey = apiKey;
if (!Endpoint.EndsWith("/")) Endpoint += "/";
}
}
Init
_Komodo = new KomodoSdk(_KomodoSettings.Endpoint, _KomodoSettings.ApiKey);
API
private async Task KomodoDelete(string key, CancellationToken token)
{
await _Komodo.DeleteDocument(_KomodoSettings.IndexGUID, key, token).ConfigureAwait(false);
}
private async Task<byte[]> KomodoGet(string key, CancellationToken token)
{
DocumentData data = await _Komodo.GetSourceDocument(_KomodoSettings.IndexGUID, key, token).ConfigureAwait(false);
return data.Data;
}
private async Task<BlobData> KomodoGetStream(string key, CancellationToken token)
{
BlobData ret = new BlobData();
DocumentData data = await _Komodo.GetSourceDocument(_KomodoSettings.IndexGUID, key, token).ConfigureAwait(false);
ret.ContentLength = data.ContentLength;
ret.Data = data.DataStream;
return ret;
}
private async Task<bool> KomodoExists(string key, CancellationToken token)
{
try
{
DocumentMetadata md = await _Komodo.GetDocumentMetadata(_KomodoSettings.IndexGUID, key, token).ConfigureAwait(false);
return true;
}
catch (KomodoException)
{
return false;
}
}
private async Task KomodoWrite(string key, string contentType, byte[] data, CancellationToken token)
{
await _Komodo.AddDocument(_KomodoSettings.IndexGUID, key, key, null, key, DocType.Unknown, data, null, token).ConfigureAwait(false);
}
private async Task KomodoWrite(string key, string contentType, long contentLength, Stream stream, CancellationToken token)
{
byte[] data = Common.StreamToBytes(stream);
await KomodoWrite(key, contentType, data, token).ConfigureAwait(false);
}
private async Task KomodoWriteMany(List<WriteRequest> objects, CancellationToken token)
{
foreach (WriteRequest obj in objects)
{
if (obj.Data != null)
{
await KomodoWrite(obj.Key, obj.ContentType, obj.Data, token).ConfigureAwait(false);
}
else
{
await KomodoWrite(obj.Key, obj.ContentType, obj.ContentLength, obj.DataStream, token).ConfigureAwait(false);
}
}
}
private async Task<BlobMetadata> KomodoGetMetadata(string key, CancellationToken token)
{
DocumentMetadata dm = await _Komodo.GetDocumentMetadata(_KomodoSettings.IndexGUID, key, token).ConfigureAwait(false);
BlobMetadata md = new BlobMetadata();
md.ContentLength = dm.SourceRecord.ContentLength;
md.ContentType = dm.SourceRecord.ContentType;
md.CreatedUtc = dm.SourceRecord.Created;
md.ETag = dm.SourceRecord.Md5;
md.Key = dm.SourceRecord.GUID;
md.LastAccessUtc = null;
md.LastUpdateUtc = null;
return md;
}
private async Task<EnumerationResult> KomodoEnumerate(string prefix, string continuationToken, CancellationToken token)
{
int startIndex = 0;
int count = 1000;
if (!String.IsNullOrEmpty(continuationToken))
{
if (!KomodoParseContinuationToken(continuationToken, out startIndex, out count))
{
throw new ArgumentException("Unable to parse continuation token.");
}
}
EnumerationQuery eq = new EnumerationQuery();
eq.StartIndex = startIndex;
eq.MaxResults = count;
if (!String.IsNullOrEmpty(prefix))
{
SearchFilter sf = new SearchFilter("GUID", SearchCondition.StartsWith, prefix);
eq.Filters.Add(sf);
}
Komodo.Sdk.Classes.EnumerationResult ker = await _Komodo.Enumerate(_KomodoSettings.IndexGUID, eq, token).ConfigureAwait(false);
EnumerationResult ret = new EnumerationResult();
ret.NextContinuationToken = KomodoBuildContinuationToken(startIndex + count, count);
if (ker.Matches != null && ker.Matches.Count > 0)
{
foreach (SourceDocument curr in ker.Matches)
{
BlobMetadata md = new BlobMetadata();
md.ContentLength = curr.ContentLength;
md.ContentType = curr.ContentType;
md.CreatedUtc = curr.Created;
md.ETag = curr.Md5;
md.Key = curr.GUID;
ret.Blobs.Add(md);
}
}
return ret;
}
private async Task<EmptyResult> KomodoEmpty(CancellationToken token)
{
EmptyResult er = new EmptyResult();
string continuationToken = null;
while (true)
{
EnumerationResult result = await KomodoEnumerate(null, null, token).ConfigureAwait(false);
continuationToken = result.NextContinuationToken;
if (result.Blobs != null && result.Blobs.Count > 0)
{
foreach (BlobMetadata md in result.Blobs)
{
await KomodoDelete(md.Key, token).ConfigureAwait(false);
er.Blobs.Add(md);
}
}
else
{
break;
}
}
return er;
}
private string BuildContinuationToken(long start, int count)
{
string ret = start.ToString() + " " + count.ToString();
byte[] retBytes = Encoding.UTF8.GetBytes(ret);
return Convert.ToBase64String(retBytes);
}
private bool KomodoParseContinuationToken(string continuationToken, out int start, out int count)
{
return KvpbaseParseContinuationToken(continuationToken, out start, out count);
}
private string KomodoBuildContinuationToken(long start, int count)
{
if (start >= count) return null;
return BuildContinuationToken(start, count);
}
private string KomodoGenerateUrl(string key)
{
if (!_KomodoSettings.Endpoint.EndsWith("/")) _KomodoSettings.Endpoint += "/";
string ret =
_KomodoSettings.Endpoint +
_KomodoSettings.IndexGUID + "/" +
key;
return ret;
}
public static byte[] StreamToBytes(Stream input)
{
if (input == null) throw new ArgumentNullException(nameof(input));
if (!input.CanRead) throw new InvalidOperationException("Input stream is not readable");
byte[] buffer = new byte[16 * 1024];
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
return ms.ToArray();
}
}
Ok - this is an issue I had with Storage.Net and recently with Fluent.Storage
My use case is that I need to store sensitive files in an encrypted state, even though the blob storage I use will be completely private, just for a 2nd level encryption to satisfy the customers Information Governance. Anyway not being an encryption guru, it's taken me a bit of time to work out the issue, and it's a long standing one going back to the original implementation.
Since most of the time we will be using Dependency Injection (DI) to instantiate our objects, each time the SymmetricEncryptionSink (or in my case the AesSymmetricEncryptionSink) is created, the IV is set
public AesSymmetricEncryptionSink(string key) {
_cryptoAlgorithm = Aes.Create();
_cryptoAlgorithm.Key = Convert.FromBase64String(key);
_cryptoAlgorithm.GenerateIV();
}
This means that if a blob is stored, then it can only be unencrypted using the same instance of the Sink or with an instance of the Sink with the same IV (secret key)
So if you have the case (and this looks to be quite important given that we are pushing to AWS/Azure storage, then if you try and retrieve the blob at another time, you can't read the blob even though the original key might be the same, the secret generated at the time of encryption is no longer the secret that the sink is trying to use the decrypt.
Almost every instance of the use of Storage.Net has got this wrong and has used a pattern like this, which creates a transient (I think) and not a singleton, this one is from Elsa that doesn't use encryption
builder.Services.Configure<BlobStorageWorkflowProviderOptions>(options => options.BlobStorageFactory = () => StorageFactory.Blobs.DirectoryFiles(Path.Combine(builder.Environment.ContentRootPath, "Workflows")));
So there are previous examples see that I found, and in that instances its basically the issue described above.
There are two solutions I think.
for my use case (which is in fact 2) above
builder.Services.AddSingleton<IBlobStorage>(StorageFactory.Blobs
.DirectoryFiles(Path.Combine(builder.Environment.ContentRootPath, "Cache"))
.WithAesSymmetricEncryption(CryptoSeed.GenerateNewKey()));
As rijndael has become obsolete in .net6 and above, are there any plans to change the default from Rijndael to Aes in SymmetricEncryptionSink.cs
I know I can write my own transform, to implement, however I think this should be at least considered
When a call to DeleteAsync() is provided with an invalid path, it will fall back to recursively deleting from the root of the bucket.
There appears to be an rm -rf /* type of issue with the way the recursive folder deletion is handled. It looks like this List call was supposed to be a ListAt call rather than listing the contents of the bucket.
Calling ListAsync on a FluentStorage.FTP with Recurse set to true in ListOptions does not do anything.
ListOptions options = new ListOptions();
options.Recurse = true;
options.FolderPath = "/some/test/";
IBlobStorage storage = StorageFactory.Blobs.Ftp("myhost.com", new NetworkCredential("username", "password"));
var results = await storage.ListAsync(options);Problem:
Try to create a blob connection azure blobstorage gen2 with FromConnectionString
did the StorageFactory.Modules.UseAzureBlobStorage() to register the azure blob storage extention
throws a NullReferenceException
Cause:
ExtendedSdk.GetHttpPipeline(BlobServiceClient sdkClient) is using internals of Azure BlobServiceClient and they have changed I assume.
This ugly hack did the trick:
var accountName = <<StorageAccountName>>;
var credential = new StorageSharedKeyCredential(accountName ,<<accessToken>>);
var uri = new Uri($"https://{accountName}.blob.core.windows.net/");
var client = new BlobServiceClient(uri, credential);
FieldInfo BlobClientConfigurationField =
typeof(BlobServiceClient).GetField("_clientConfiguration", BindingFlags.NonPublic | BindingFlags.Instance);
var clientConfig = BlobClientConfigurationField.GetValue(client);
var clientConfigType = clientConfig.GetType();
PropertyInfo httpPipelineProperty =
clientConfigType.GetProperty("Pipeline", BindingFlags.Public | BindingFlags.Instance);
Azure.Core.Pipeline.HttpPipeline httpPipeline = httpPipelineProperty.GetValue(clientConfig) as Azure.Core.Pipeline.HttpPipeline;
Type AzureDataLakeStorageType = typeof(IBlobStorage)
.Assembly.GetType("FluentStorage.Azure.Blobs.AzureDataLakeStorage");
// ConstructorInfo method = t.GetConstructor(new Type[] { typeof(BlobServiceClient), typeof(string), typeof(StorageSharedKeyCredential), typeof(string )} );
//# Object o = method.Invoke(new Object[] {client, accountName, credential, null});
Type extendedSdkType = typeof(IBlobStorage)
.Assembly.GetType("Blobs.ExtendedSdk");
var dlStorage = FormatterServices.GetUninitializedObject(AzureDataLakeStorageType);
var extendedSdk = FormatterServices.GetUninitializedObject(extendedSdkType);
var sdkClientField =
extendedSdkType.GetField("_sdkClient", BindingFlags.Instance | BindingFlags.NonPublic);
var pipelineField = extendedSdkType.GetField("_httpPipeline", BindingFlags.Instance | BindingFlags.NonPublic);
var dsfBaseAddress = extendedSdkType.GetField("_dfsBaseAddress", BindingFlags.Instance | BindingFlags.NonPublic);
sdkClientField.SetValue(extendedSdk, client);
pipelineField.SetValue(extendedSdk, httpPipeline);
dsfBaseAddress.SetValue(extendedSdk, $"https://{accountName}.dfs.core.windows.net/");
var extendedField = AzureDataLakeStorageType.GetField("_extended", BindingFlags.Instance | BindingFlags.NonPublic);
extendedField.SetValue(dlStorage, extendedSdk);
var clientField2 = AzureDataLakeStorageType.BaseType.GetField("_client", BindingFlags.Instance | BindingFlags.NonPublic);
clientField2.SetValue(dlStorage, client);
var sasCredField = AzureDataLakeStorageType.BaseType.GetField("_sasSigningCredentials", BindingFlags.Instance | BindingFlags.NonPublic);
sasCredField.SetValue(dlStorage, credential);
var containerField = AzureDataLakeStorageType.BaseType.GetField("_containerNameToContainerClient", BindingFlags.Instance | BindingFlags.NonPublic);
containerField.SetValue(dlStorage, new ConcurrentDictionary<string, BlobContainerClient>());
Is ExistsAsync just looking for Files? how about Directories?
When uploading a jpg file as an Azure Blob using WriteAsync, the file is created in Azure with a Content-Type set to 'application/octet-stream'. There doesn't seem be any support for setting the Content-Type to 'application/image/jpeg'. Or is there?
The Newtonsoft.Json v12.0.3 package used in the library has a known vulnerability
that was fixed in 13.0.1+
This could help in the adoption of this library (which is really handy by the way)
The nuget package Renci.SshNet.Async appears rather out of date. On its own it has a dependency of SSH.NET v2016.1.0 which indicates that it has a vulnerability with moderate severity. There is an issue created 12 months ago which asked to bump the dependency but nothing came of it. Consdering FluentStorage.SFTP uses later versions (see below) It appears that the package just needs csporj text edit and no other major changes.
This isn't as much of an issue for FluentStorage.SFTP as it currently has a dependency of SSH.NET (>= 2020.0.2) across all target platforms. So by default people using FluentStorage.SFTP do not have this problem to worry about.
However if you attempt to manually update SSH.NET to 2023.0.0 or even 2023.0.1 you will get an error while attempting to call ListDirectories
Method not found: 'System.Collections.Generic.IEnumerable`1<Renci.SshNet.Sftp.SftpFile> Renci.SshNet.SftpClient.EndListDirectory(System.IAsyncResult)'.
at Renci.SshNet.Async.SshNetExtensions.ListDirectoryAsync(SftpClient client, String path, Action`1 listCallback, TaskFactory`1 factory, TaskCreationOptions creationOptions, TaskScheduler scheduler)
I don't know if this is due to SSH.NET having targets of .NET 6 and higher (and also .NET framework, and other things) while Renci.SshNet.Async package only has netstandard1.3;net40;uap10.0.
Playing around locally if I remove using Renci.SshNet.Async from SshNetSftpBlobStorage I get 2 errors. Both are calls to client.ListDirectoryAsync( which are here and here.
It looks like the people working on SSH.NET took on a dependency of Microsoft.Bcl.AsyncInterfaces to allow IAsyncEnumerable on frameworks that don't support it (here). I don't know how that would work here but it looks similar as this project also targets netstandard2.0 and up.
As for code changes, I have never used IAsyncEnumerable but I hacked this together which hopefully is a starting off point for someone who knows how to use this better.
I first changed
IEnumerable<SftpFile> directoryContents = await client.ListDirectoryAsync(fullPathGrouping.Key);
List<Blob> blobCollection = directoryContents
.Where(f => (f.IsDirectory || f.IsRegularFile) && f.FullName == fullPath)
.Select(ConvertSftpFileToBlob).ToList();to
List<Blob> blobCollection = new List<Blob>();
await foreach (SftpFile file in client.ListDirectoryAsync(fullPathGrouping.Key, cancellationToken))
{
if ((file.IsDirectory || file.IsRegularFile) && file.FullName == fullPath)
{
blobCollection.Add(ConvertSftpFileToBlob(file));
}
}and then later on
IEnumerable<SftpFile> directoryContents = await client.ListDirectoryAsync(folderToList, cancellationToken);
var tempBlobCollection = directoryContents
.Where(dc => (options.FilePrefix == null || dc.Name.StartsWith(options.FilePrefix))
&& (dc.IsDirectory || dc.IsRegularFile || dc.OwnerCanRead)
&& !cancellationToken.IsCancellationRequested
&& dc.Name != "."
&& dc.Name != "..")
.Take(options.MaxResults.Value)
.Select(ConvertSftpFileToBlob)
.Where(options.BrowseFilter).ToListo
var tempBlobCollection = new List<Blob>();
await foreach (SftpFile dc in client.ListDirectoryAsync(folderToList, cancellationToken))
{
if ((options.FilePrefix == null || dc.Name.StartsWith(options.FilePrefix))
&& (dc.IsDirectory || dc.IsRegularFile || dc.OwnerCanRead)
&& !cancellationToken.IsCancellationRequested
&& dc.Name != "."
&& dc.Name != "..")
{
tempBlobCollection.Add(ConvertSftpFileToBlob(dc));
}
}
// TODO: options.BrowseFilter()(note: this doesn't use options.BrowseFilter, nor does it really make any use of options.MaxResults.Value).
With this I am able to push SSH.NET to 2023.0.1 in my test project which is a .NET MAUI app targeting .NET 8). My test project only ever calls ListAsync and never GetBlobsAsync so I don't know if that first half of the code even works correctly. I just know that it compiles and works for my use case here.
When trying to create an azure blob storage instance from a connection string, if the key contains a '+' this fails with the cryptic "not a valid base64 value".
After digging around and creating an instance of StorageConnectionString myself and looking at the code, I can see the value of each part gets url decoded.
The documentation does not mention this at all, so either the code should leave the value as-is, or the documentation should be very clear that the secret value needs to be url-encoded.
Might be helpful having an exception in there too to catch this scenario and warn the user that they need to encode the value.
Installing the latest version of FluentStorage.Azure.Blobs (5.2.2) shows that there are two dependencies with vulnerabilities as per the VS Nuget analysis
Mitigation
Resolution
Some aws s3 compatible services return content-type header with charset field and AWSSDK.S3 version less then 3.7.202 incorrect handle response if error occur.
See here aws/aws-sdk-net#3009
Create mr to fix this.
Hello,
the AzureServicebus plugin for FluentStorage is implemented with the package Microsoft.Azure.ServiceBus wich is deprecated since 2021. I cannot use it in any of my projects cause conflicts with outated references:
Version conflict detected for System.IdentityModel.Tokens.Jwt. I
Also there are missing functionalities, in the wiki we have this:
IMessagePublisher queuePublisher = StorageFactory.Messages.AzureServiceBusQueuePublisher(
connectionString,
queueName);
IMessagePublisher topicPublisher = StorageFactory.Messages.AzureServiceBusTopicPublisher(
connectionString,
topicName);
IMessageReceiver queueReceiver = StorageFactory.Messages.AzureServiceBusQueueReceiver(
connectionString,
queueName,
peekLock);
IMessageReceiver topicReceiver = StorageFactory.Messages.AzureServiceBusTopicReceiver(
connectionString,
topicName,
subscriptionName,
peekLock);
But in realty only these methods are currently implemented:
Plus i'm not 100% sure that sending directly to subiscriptions will work with the actual code.
I'm implementing a PR wich fixes everything and use the current, supported package.
Should my PR replace the current plugin (there would be breaking changes, is inevitable) or should i make a new package FluentStorage.Azure.Messaging.ServiceBus (wich is similar to the name of the new Microsoft package) e maintain the older for retrocompatibility?
I'm currently reworking a system that was using Azure blob storage to use FluentStorage so we can target different providers / deploy on-prem and make use of disk storage.
Data for the application is split into containers, and as per the docs the first folder in the path is used as the container for Azure, but otherwise should just be treated as a folder otherwise.
This behaviour works just fine for the in-memory module, calling GetBlobAsync returns a blob that is a folder, and I can check the full path to ensure it sits under root so I can classify it as a "container".
For the local-disk module however, creating one of these produces a directory with a .empty file under it, and having a look at the code of the disk implementation, it doesn't appear to have handling for directories - I'd expect it to be happy enough to create empty directories, and just check whether a path is for a directory or a file and return a blob with the relevant properties set.
The ZIP module I can sort of understand the behaviour for because you have to create the ZipEntry objects and it infers hierarchy based on the paths as opposed to supporting folders within the archive as a first-class concept.
I guess what I'm asking is whether there's a consistent way I can support the arbitrary modules and still have everything work fine? I guess I could enforce the creation of a '.empty' file and explicitly check for its existence, but that feels redundant for the Azure module.
My first guess would be to check whether the storage implementation is IHierarchicalBlobStorage but the azure one doesn't appear to be.
Any suggestions would be appreciated.
If you have a file locally that contains the following text
12345678
and you upload that file with await myRemoteStorage.WriteFileAsync(myRemoteFile, myLocalFile) you will create a file at myRemoteFile and it will be 8 bytes long.
If you now change the file locally to be
0000
and call await myRemoteStorage.WriteFileAsync(myRemoteFile, myLocalFile) then the result file on the server will not change its filesize and it will actually contain the text
00005678
I believe this comes from the WriteAsync method which opens a local and remote stream (see FluentFtpBlobStorage and SshNetSftpBlobStorage) and calls CopyAsync. This does exactly that, copies the stream of data across. But after the rest of the file is not removed.
This is very similar to the behaviour which casued people to be able to uncrop screenshots due to a bug in the image cropper.
Using UploadFile from the respective clients should solve this problem, but it prevents being able to append to a file.
I'll try it out and see how it goes, if I find a solution I'll create a PR.
| Provider | Impacted (may not be accurate) |
|---|---|
| FluentStorage | ✅ |
| FluentStorage.AWS | ❌ |
| FluentStorage.GCP | ❌ |
| FluentStorage.Databricks | ❌ |
| FluentStorage.FTP | ❌ |
| FluentStorage.SFTP | ✅ |
| FluentStorage.Azure.Blobs | ❌ |
| FluentStorage.Azure.Files | ❌ |
| FluentStorage.Azure.EventHub | ❓ |
| FluentStorage.Azure.ServiceBus | ❓ |
| FluentStorage.Azure.KeyVault | ❌ |
| FluentStorage.Azure.ServiceFabric | ❌ |
| FluentStorage.Azure.Queues | ❓ |
| FluentStorage.Azure.DataLake | ✅ |
Skimming through code to see what is and isn't impacted. May not be 100% accurate. (EDIT: Tested FluentStorage.FTP thismroning and it does not have this problem)
Eyeballing the test here (without actuallying knowing how these tests work), changing the attributes to have something smaller after the longer test should find it? (assuming upload to the same location)
[InlineData("sample")]
[InlineData("123")]
[InlineData("123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123")]
[InlineData("456")]
Hi !
Is there any plan on adding support of Span<byte> / Memory<byte> where byte[] is currently required ?
(given that the introduction of these types help enhance memory usage and are spreading across the SDK)
Here a example where I think having overloads that accepts Span or Memory depending on the context could help
Memory<byte> buffer = GetBuffer();
Guid id = Guid.NewGuid();
string fileName = section.FileName;
string fullPath = $"upload/{id}";
Stream stream = section.Section.Body;
int bytesRead = 0;
// Streaming of the file
do
{
bytesRead = await stream.ReadAsync(buffer.Memory, ct);
await _storage.WriteAsync(fullPath, buffer.Memory.ToArray(), true, cancellationToken: ct);
}
while (bytesRead > 0);which could be rewritten
Memory<byte> buffer = GetBuffer();
Guid id = Guid.NewGuid();
string fileName = section.FileName;
string fullPath = $"upload/{id}";
Stream stream = section.Section.Body;
int bytesRead = 0;
// Streaming of the file
do
{
bytesRead = await stream.ReadAsync(buffer.Memory, ct);
await _storage.WriteAsync(fullPath, buffer.Memory, true, cancellationToken: ct);
}
while (bytesRead > 0);From what i've seen in the codebase, it would require to either :
IBlobStorage implementations to add support for this but with a default interface implementation, the change could be as minimal as adding default methodTask WriteAsync(string fullPath, Memory<byte> data, bool append = false, CancellationToken cancellationToken = default) => WriteAsync(fullPath, new MemoryStream(data.ToArray()), append, cancellationToken);This has the advantage of allowing implementors to provide a more memory efficient approach without breaking them (net6.0 upwards)
IBlobStorage.WriteAsync extension method that would take a Memory<byte> instead of byte[]Is that something that make sense to you ?
Pretty much the same issue as #53, creating it for a second PR.
Calling ListAsync on a FluentStorage.SFTP with Recurse set to true in ListOptions does not do anything.
ListOptions options = new ListOptions();
options.Recurse = true;
options.FolderPath = "/some/test/";
IBlobStorage storage = StorageFactory.Blobs.Sftp("myhost.com", "username", "password");
var results = await storage.ListAsync(options);I assume this will be the same for ZipFileBlobStorage and will need a change to include it since it targets the Blob objects and not the content streams.
SinkedBlobStorage which is used when sinks are in use, needs to implement a new interface to cater for the retrieval of Blob objects when the sink is transformed as follows:
Task<IReadOnlyCollection<Blob>> GetBlobsAsync(IEnumerable<string> fullPaths, CancellationToken cancellationToken = default)
Task<IReadOnlyCollection<Blob>> ListAsync(ListOptions options = null, CancellationToken cancellationToken = default)
Task SetBlobsAsync(IEnumerable<Blob> blobs, CancellationToken cancellationToken = default)
As the documentation states, the FileTransferUtility.UploadAsync will cache everything in memory.
The Multipart upload allows to at least upload the stream in chunks.
The TwentyTwenty library implents this: https://github.com/2020IP/TwentyTwenty.Storage/blob/dev/src/TwentyTwenty.Storage.Amazon/AmazonFileUploader.cs
When attempting to add FluentStorage to a .NET 7.0 project (which has apparently has been computed to be compatible) I get the following error:
@ Installing FluentStorage in project.Server finished (0.052 sec)
[Notification][Install] Install failed (project: project.Server, package: FluentStorage v5.0.0)
The package is missing the required nuspec file. Path: C:\Users\person\.nuget\packages\system.runtime\4.3.0
Package version : 5.0.0
Looking at the code, seems like this method is indeed not implemented (yet ?).
Are there any plan on implementing this method ?
Is there any workaround in the meantime ?
The Connection string parser will call UrlDecode() on the parsed output
https://github.com/robinrodricks/FluentStorage/blob/develop/FluentStorage/ConnectionString/StorageConnectionString.cs#L131
Which calls Netbox's UrlDecode, and strips the plus sign from the string
https://github.com/aloneguid/netbox/blob/master/src/NetBox/WebUtility.cs#L146
Currently Azure Blob Storage key's have plus signs and it makes it impossible connect to when this happens.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.