Hi,
I'm using metaseq (https://github.com/daler/metaseq) and it fails with Too many open files error while parallel process large *.bed files. This issue is due to bedtool.tabix_intervals() method:
metaseqsplits input in lots of chunks for parallel processing- each chunk is processed with
metaseq.filetype_adapters.__getitem__()which gets intervals iterator using pybedtools bedtool.tabix_intervals()openstbxfile, but never close it
I see two possible fixes:
- Close tbx file in
tabix_intervals()
def tabix_intervals(self, interval_or_string):
#...
tbx = pysam.TabixFile(self.fn)
results = list(tbx.fetch(str(interval.chrom), interval.start, interval.stop))
tbx.close()
return BedTool(results)
But in this case BedTool will create intermediate tmp file for results (see #73) and metaseq should do not forget to clean up loads of such tmp files
- The other way is return bedtool object and tbx file descriptor pair and let user to close tbx file after handling bedtool
def tabix_intervals(self, interval_or_string):
#...
return (BedTool(results), tbx)
It breaks code compatibility but doesn't create intermediate tmp files. For my own purposes I've quick fixed pybedtools and metaseq using this way
I think my both suggested ways have disadvantages and some other way should be found.