Progress bars for steps (integration of tqdm)

[netflix]

Progress bars for steps (integration of tqdm)

[crk-codaio]

Sorry for not having an update on this. We are looking into it internally and seeing if there is a way we can make this work. From the screencast you've pasted, clearly the log lines from tqdm are:

1. Getting buffered (and hence seeing it later)
2. Not seeing updates of progress - this is the one I need to spend more time debugging if there is a way to make it work with our log buffering + multiprocessing solution.

[crk-codaio]

Thank you for the very elaborate description (with screencasts :))
Will take a look and get back to you.

[crk-codaio]

I don't have any new updates about this. Few related comments though -

1. We are reworking our log streaming (and tailing) solution internally; and we hope to announce it externally as well after converging the codebases. We hope to make log tailing using the metaflow client so issues w.r.t buffering and tailing should be better after that.
2. Specifically for observing progress of foreach tasks and making it work with tqdm, have you considered doing the progress tracking outside of Metaflow - probably in a separate wrapper script; by outputting special lines perhaps after each iteration like print("<SHOW PROGRESS 1%>").
   I don't know tqdm well enough to know if you can use special log lines to bump up the progress.

Curious to hear to your response? I also apologize for extremely slow response rate from my front on this issue.

[crypdick]

@jonaschn I came up with a solution which is a little less intrusive: https://stackoverflow.com/questions/68225881/how-to-show-tqdm-progress-in-metaflow

@tuulos Is there a way to make MetaFlow show stderr output instead of hiding it

[jonaschn]

@seeravikiran Do you have any update on this?

[leftys]

The buffering issue happened to me with lightgbm training output. The training results are not printed every iteration, but at the end.

[crk-codaio]

Unfortunately I don't have an update on this yet. Apologies for that.

[jonaschn]

@seeravikiran Do you have any update on this?

[mattlbeck]

I have also stumbled into this issue. Have tried various methods for forceably releasing the buffer on each iteration but I clearly don't understand enough about how metaflow is capturing logs to identify a workaround.

[mattlbeck]

I think tqdm might be messing with buffer flushing in unexpected ways. Running the below flow with and without tqdm set in the for loop def shows that all print output is witheld until the end of the step only when tqdm is used:

from metaflow import FlowSpec, step
import tqdm
from time import sleep

class TQDMFlow(FlowSpec):

    @step
    def start(self):
        print('Training...')
        for t in tqdm.tqdm(range(100)):
            print(t)
            sleep(0.004)
        sleep(5)
        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == '__main__':
    TQDMFlow()

[tuulos]

more discussions here https://outerbounds-community.slack.com/archives/C020U025QJK/p1624953736201700

here's a workaround https://share.getcloudapp.com/4guneg9E

https://gist.github.com/tuulos/ca8029e3d2ce8141226e4b7a474f4750