Unable to respond with a Stream

[cartercommunity]

Unable to respond with a Stream

[jchannon]

Might not be able to, just a brain dump for now 😄

[bazwilliams]

This works perfectly for streams which can be processed in memory, but unfortunately in my case, this doesn't fix the issue. I'm using a producer/consumer pattern to minimise the memory requirements of my service - the data coming from the database could be many hundreds of MB and I'd rather stream it directly to the client.

This means, I'm not likely to have the entire contents of the stream in memory at once and is the cause of the block - StreamCopyOperation.CopyToAsync won't start copying until the source stream has completed.

The solution (with and without hacky delay) doesn't work for streams larger than my producer/consumer buffer size. :(

[JoeStead]

There's a SendFileAsync extension on HttpResponse that looks like it may be what you want: https://github.com/aspnet/HttpAbstractions/blob/dev/src/Microsoft.AspNetCore.Http.Extensions/SendFileResponseExtensions.cs#L25

I haven't got time at the moment to check 100%, will try to in a few hours

[bazwilliams]

I had a look at that, but I need an IFileInfo object to use that. I'm streaming data straight from a database so it isn't hitting the filesystem.

[JoeStead]

Ah ok.

In which case, you might be able to do something similar to the internal operation here: https://github.com/aspnet/HttpAbstractions/blob/dev/src/Microsoft.AspNetCore.Http.Extensions/SendFileResponseExtensions.cs#L164

e.g.:
await StreamCopyOperation.CopyToAsync(fs, res.Body, null, CancellationToken.None);

If this does the job, then I think Botwin should probably have a nicer wrapper around it.

[jchannon]

I've not looked at that link and won't until later but you might need to do something similar to this which is from Nancy ``` MemoryStream ms = new MemoryStream(documentBody); var response = new Response(); response.ContentType = "application/pdf"; response.Contents = stream => { ms.WriteTo(stream); }; return response; ```

[bazwilliams]

That is very similar to what I had before in Nancy - I didn't realise I could return a Response in a handler.

[bazwilliams]

@JoeStead Your suggestion is promising. My module now ends with:

var cd = new ContentDisposition { FileName = "journal-{type}.csv" };
res.Headers.Add("Content-Disposition", cd.ToString());
res.ContentType = "application/csv";

await StreamCopyOperation.CopyToAsync(CreateExportStream(models, requestLog, ct), res.Body, null, res.HttpContext.RequestAborted);

Still needs work as it looks like something is blocking the stream from completing.

[bazwilliams]

Looks like the response blocks because my stream has not yet been written to before passing to CopyToAsync. Hacking in a forced delay of 1 second fixes the issue, this gives time for at least the CSV headers to have been written. Not ideal. but this has got me moving forward! Thanks for the help.

[jchannon]

Just having a quick look in the depths of MVC and the FileResultExecutorBase calls:

Stream outputStream = context.Response.Body;
Stream stream = fileStream;
await StreamCopyOperation.CopyToAsync(fileStream, outputStream, new long?(), 65536, context.RequestAborted);
stream.Dispose();
stream = null;

Do you want to see if you can do what you had before the hacky timeout and see if it works

[jchannon]

OK this works:

  this.Get("/actors/download", async (request, response, routeData) =>
            {
                using (var mystream = new MemoryStream(Encoding.ASCII.GetBytes("hi")))
                {
                    var cd = new ContentDisposition { FileName = "journal.csv" };
                    response.Headers.Add("Content-Disposition", cd.ToString());
                    response.ContentType = "application/csv";
                    await StreamCopyOperation.CopyToAsync(mystream, response.Body, new long?(), 65536, request.HttpContext.RequestAborted);
                }
            });

[jchannon]

@bazwilliams you will have to create your stream first then call CopyToAsync as it will copy what's in your stream into the response body so it makes sense that it hangs if there's nothing inside it.

However that code above is ugly as hell so I think Botwin needs an extension on Response to write a memorystream to the response like the MVC FileStreamResult. Something like FromStream(Stream myStream, string contentType, ContentDisposition contentDisposition = null) and used like:

            this.Get("/actors/download", async (request, response, routeData) =>
            {
                using (var mystream = new MemoryStream(Encoding.ASCII.GetBytes("hi")))
                {
                    var cd = new ContentDisposition { FileName = "journal.csv" };
                    await response.FromStream(mystream, "application/csv", cd);
                }
            });

[jchannon]

Fancy sending a PR 🤔 👍

[bazwilliams]

The Nancy version of this code looks like:

var models = this.journalReader.Read(start, end, type, ct);

return Task.FromResult(new StreamResponse(() => CreateExportStream(models, requestLog, ct), "application/csv")
    .AsAttachment($"journal-{type}.csv")
    .WithStatusCode(200));

I'll see if I can solve this using pure asp.net first and report back :)

[bazwilliams]

Hmm - the StreamResponse() in Nancy just uses CopyTo() https://github.com/NancyFx/Nancy/blob/master/src/Nancy/Responses/StreamResponse.cs#L32

There must be something else I need to do here.

[jchannon]

MVC's FileResult can handle partial responses. I was going to add that to my PR at a later date but sounds like that's what you need now. Can take a look later and/or over the weekend

…

On 23 March 2018 at 09:35, Barry Williams ***@***.***> wrote: Hmm - the StreamResponse() in Nancy just uses CopyTo() ( https://github.com/NancyFx/Nancy/blob/master/src/Nancy/ Responses/StreamResponse.cs#L32) There must be something else I need to here. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#89 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGapnESCm0955u-wDkQX-ZfV6L_wm4qks5thMHWgaJpZM4S2vLW> .

[bazwilliams]

I'll check it out thanks. If I get it working I'll submit a PR!

[JoeStead]

I reckon @danbarua might have some suggestions

[danbarua]

I'm using a producer/consumer pattern to minimise the memory requirements of my service - the data coming from the database could be many hundreds of MB and I'd rather stream it directly to the client.

Are there any constraints preventing you from writing out the data to file and then returning that file via the web api? That's how most reporting/CSV exports I've seen work.

[bazwilliams]

Technically no, but it is a service I'd like to run via lambda or something like that and just not have to worry about having an S3 bucket or local file system constraints.

It is currently deployed to AWS ECS, so I could gain access to the local file system for the duration of the request, but then I have to manage the available space and remove older files. If I were going to do this, I'd consider rewriting the whole thing and that's not something I want to do at the moment.

It's in the same solution as 5 other Ex Nancy, now Botwin projects (all working perfectly), I thought I'd try my luck at this more tricky one :)

[jchannon]

What db is it? Postgres/MSSQL?

…

On 23 March 2018 at 13:00, Barry Williams ***@***.***> wrote: Technically no, but it is a service I'd like to run via lambda or something like that and just not have to worry about having an S3 bucket or local file system constraints. It is currently deployed to AWS ECS, so I could gain access to the local file system for the duration of the request, but then I have to manage the available space and remove older files. If I were going to do this, I'd consider rewriting the whole thing and that's not something I want to do at the moment. It's in the same solution as 5 other Ex Nancy, now Botwin projects (all working perfectly), I thought I'd try my luck at this more tricky one :) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#89 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGapltu6aUt96qRlWKgfG3lpC35Da87ks5thPHVgaJpZM4S2vLW> .

[bazwilliams]

DynamoDb

[bazwilliams]

This is definitely not a Botwin issue; I've rewritten the module as an MVC controller and I see the same problem.

Thanks for all the help so far! Looks like this is the right approach, especially after reading an article about streaming video (linked below) and the asp.net code on Github so look forward to the extension suggested above being merged.

However, I think the ProducerConsumerStream I'm using is introducing some kind of deadlock with asp.net. The way I'm using Nancy to do this seems to work, so will revert my work and park for now. I'll be considering a rewrite instead.

http://anthonygiretti.com/2018/01/16/streaming-video-asynchronously-in-asp-net-core-2-with-web-api/

Cheers!