Python traceback when sending output to a pipe

[saketkc]

Python traceback when sending output to a pipe

[saketkc]

Hi @nimitbhardwaj, thanks for looking into this. If you think the fix could potentially work, please do submit a pull request. The test cases for cli.py are currently disabled, but if you submit a PR, it would be good to reinstate them along with this new test case.

[saketkc]

Thanks Simon!

I can reproduce this at my end. I think I have a fix for this that I'll update soon.

[jarrodscott]

is this still an issue? I ask because I received the same error when running:

pysradb search '"ribosome profiling"' | head

[VangelisTheodorakis]

I also get the broken pipe error.

[saketkc]

This is still an issue. Sorry, but I don't have a fix yet. As a temporary fix, you can save the output to a tsv by appending --saveto myfile.tsv and then do a head on the tsv.

[nimitbhardwaj]

I would like to work this issue. I did some research why this issue is happening, I came to know that in file cli.py, in function _print_save_df, line number 33, we are printing the output what is needed to be printed. By referring stackoverflow, the Broken Pipe Error occurs because when one end of the pipe is blocked, and for our case the I/O redirection by | head is blocking the output pipe, so the buffer of the stdout pipe overflows and is broken, this is because the head or output redirection only reads when the preceding process is completed.
Following the same question thread, I came to know that ignoring the exception is best way to do it, as at some point when the buffer is full it wont take any other byte in buffer, so it can be done by exception handling.
For this approach it is easy to code, but I want to confirm will this approach be good for our case.

[nimitbhardwaj]

@saketkc ok cool I make the PR for it today and will re initiate the testcase for the same.

[DaasDaham]

Is this issue still open? I was thinking on the same lines as nimit, basically, this error is thrown due to the large size of the file which is being redirected through stdout to pipe. The pipe typically has a limit of ~60KB as mentioned in this answer's comment. So to make it user-friendly a hard-coded limit on the size of to_print list (line 43 in cli.py) can be put to restrict it from printing anything out to the terminal if it crosses the threshold. When it is greater than the limit the output can be saved to a unique .tsv file and an error message output is too large may be displayed along with the name of tsv file.
This approach does not exactly solve the Broken pipe error but is merely a workaround.
@saketkc If this issue is still open, is the approach mentioned above fine? Also should I write a few test cases to test it?