iromeo · GitHub

Hi,
I'm using metaseq (https://github.com/daler/metaseq) and it fails with Too many open files error while parallel process large *.bed files. This issue is due to bedtool.tabix_intervals() method:

  1. metaseq splits input in lots of chunks for parallel processing
  2. each chunk is processed with metaseq.filetype_adapters.__getitem__() which gets intervals iterator using pybedtools
  3. bedtool.tabix_intervals() opens tbx file, but never close it

I see two possible fixes:

  1. Close tbx file in tabix_intervals()
def tabix_intervals(self, interval_or_string):
        #...
        tbx = pysam.TabixFile(self.fn)
        results = list(tbx.fetch(str(interval.chrom), interval.start, interval.stop))
        tbx.close()
        return BedTool(results)

But in this case BedTool will create intermediate tmp file for results (see #73) and metaseq should do not forget to clean up loads of such tmp files

  1. The other way is return bedtool object and tbx file descriptor pair and let user to close tbx file after handling bedtool
def tabix_intervals(self, interval_or_string):
        #...
       return (BedTool(results), tbx)

It breaks code compatibility but doesn't create intermediate tmp files. For my own purposes I've quick fixed pybedtools and metaseq using this way

I think my both suggested ways have disadvantages and some other way should be found.

Read the original on github.com ↗