BedTool.sequence hangs on a BedTool made with generator?

[daler]

BedTool.sequence hangs on a BedTool made with generator?

[brentp]

calling len() will cause the iterator to exhaust itself and then be empty by the time it gets to the .sequence() call.

if you really want to iterate over it multiple times, then use:

sampled_cluster_intervals = list(get_shuffled_clusters_intervals(...))

maybe if you used "raise StopIteration", the error would be more helpful.

[yarden]

Sure about len, didn't meant to imply that it fixes the issue, but I just found it surprising that len can exhaust the iterator (thus ending in an empty sequence output as you write) whereas sequence cannot. I definitely don't want to load it in to memory so I avoided list - there's no need for me to iterate through it multiple times, it's only made into a BedTool as a generator so that it can be written to a file line by line.

I tried using raise StopIteration at the end of the function but it gets stuck prior to that. Where were you thinking of placing it? Thanks.

[daler]

I think this boils down to a question about whether .sequence() can handle an iterator. Here's a standalone example using the pybedtools example files:

import pybedtools

def generator():
    seqsize = 3
    for i in range(100):
        strand = '+'
        if i % 2 == 0:
            strand = '-'
        yield pybedtools.create_interval_from_list([
            'chr1',
            str(seqsize * i),
            str(seqsize * i + seqsize),
            'region_%s' % i,
            '.',
            strand
        ])
    return

x = pybedtools.BedTool(generator())

x.sequence(
    fi=pybedtools.example_filename('test.fa'),
    fo='example',
    name=True,
    s=True)

print open('example').read()

This works fine. Are you able to modify this (perhaps by using your bedtools_utils.sample_intervals) so that it hangs?

It's possible that if bedtools_utils.sample_intervals is using subprocess.Popen and reading from subprocess.PIPE, you may be running into something like what's described in #49. If this is the case, the solution is simply to write to an intermediate file that can be deleted at the end of the generator function.