Error in BaseRecalibrator while using NIO
closed | Created 2017-07-20 | Last updated 2017-08-01| Posted by meganshand | See in Github


NIO PRIORITY_HIGH bug


I got the following error and log file from BaseRecalibrator:

Using GATK jar /usr/gitc/gatk4/gatk-package-4.beta.1-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=1 -Dsnappy.disable=true -XX:GCTimeLimit=50 -XX:GCHeapFreeLimit=10 -XX:+PrintFlagsFinal -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:gc_log.log -Xms4000m -jar /usr/gitc/gatk4/gatk-package-4.beta.1-local.jar BaseRecalibrator -R /cromwell_root/broad-references/hg38/v0/Homo_sapiens_assembly38.fasta -I gs://broad-gotc-dev-cromwell-execution/PairedEndSingleSampleWorkflow/66442def-ad3f-4c6c-960e-17578f6b382c/call-SortAndFixSampleBam/CHMI_CHMI3_WGS2.aligned.duplicate_marked.sorted.bam --useOriginalQualities -O CHMI_CHMI3_WGS2.recal_data.csv -knownSites gs://broad-references/hg38/v0/Homo_sapiens_assembly38.dbsnp138.vcf -knownSites /cromwell_root/broad-references/hg38/v0/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz -knownSites /cromwell_root/broad-references/hg38/v0/Homo_sapiens_assembly38.known_indels.vcf.gz -L chr11:1+
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/cromwell_root/tmp.WXYB31
[July 20, 2017 2:18:26 PM UTC] BaseRecalibrator  --useOriginalQualities true --knownSites gs://broad-references/hg38/v0/Homo_sapiens_assembly38.dbsnp138.vcf --knownSites /cromwell_root/broad-references/hg38/v0/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz --knownSites /cromwell_root/broad-references/hg38/v0/Homo_sapiens_assembly38.known_indels.vcf.gz --output CHMI_CHMI3_WGS2.recal_data.csv --intervals chr11:1+ --input gs://broad-gotc-dev-cromwell-execution/PairedEndSingleSampleWorkflow/66442def-ad3f-4c6c-960e-17578f6b382c/call-SortAndFixSampleBam/CHMI_CHMI3_WGS2.aligned.duplicate_marked.sorted.bam --reference /cromwell_root/broad-references/hg38/v0/Homo_sapiens_assembly38.fasta  --mismatches_context_size 2 --indels_context_size 3 --maximum_cycle_value 500 --mismatches_default_quality -1 --insertions_default_quality 45 --deletions_default_quality 45 --low_quality_tail 2 --quantizing_levels 16 --bqsrBAQGapOpenPenalty 40.0 --preserve_qscores_less_than 6 --enableBAQ false --computeIndelBQSRTables false --defaultBaseQualities -1 --interval_set_rule UNION --interval_padding 0 --interval_exclusion_padding 0 --readValidationStringency SILENT --secondsBetweenProgressUpdates 10.0 --disableSequenceDictionaryValidation false --createOutputBamIndex true --createOutputBamMD5 false --createOutputVariantIndex true --createOutputVariantMD5 false --lenient false --addOutputSAMProgramRecord true --addOutputVCFCommandLine true --cloudPrefetchBuffer 40 --cloudIndexPrefetchBuffer -1 --disableBamIndexCaching false --help false --version false --showHidden false --verbosity INFO --QUIET false --use_jdk_deflater false --use_jdk_inflater false --disableToolDefaultReadFilters false
[July 20, 2017 2:18:26 PM UTC] Executing as root@ab23aa1f82e3 on Linux 4.9.0-0.bpo.3-amd64 amd64; OpenJDK 64-Bit Server VM 1.8.0_111-8u111-b14-2~bpo8+1-b14; Version: 4.beta.1
[July 20, 2017 2:22:06 PM UTC] org.broadinstitute.hellbender.tools.walkers.bqsr.BaseRecalibrator done. Elapsed time: 3.65 minutes.
Runtime.totalMemory()=4188012544
htsjdk.samtools.SAMFormatException: Did not inflate expected amount
	at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:147)
	at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:96)
	at htsjdk.samtools.util.BlockCompressedInputStream.inflateBlock(BlockCompressedInputStream.java:537)
	at htsjdk.samtools.util.BlockCompressedInputStream.processNextBlock(BlockCompressedInputStream.java:519)
	at htsjdk.samtools.util.BlockCompressedInputStream.nextBlock(BlockCompressedInputStream.java:455)
	at htsjdk.samtools.util.BlockCompressedInputStream.readBlock(BlockCompressedInputStream.java:445)
	at htsjdk.samtools.util.BlockCompressedInputStream.available(BlockCompressedInputStream.java:194)
	at htsjdk.samtools.util.BlockCompressedInputStream.read(BlockCompressedInputStream.java:326)
	at java.io.DataInputStream.read(DataInputStream.java:149)
	at htsjdk.samtools.util.BinaryCodec.readBytesOrFewer(BinaryCodec.java:404)
	at htsjdk.samtools.util.BinaryCodec.readBytes(BinaryCodec.java:380)
	at htsjdk.samtools.util.BinaryCodec.readBytes(BinaryCodec.java:366)
	at htsjdk.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:209)
	at htsjdk.samtools.BAMFileReader$BAMFileIterator.getNextRecord(BAMFileReader.java:829)
	at htsjdk.samtools.BAMFileReader$BAMFileIndexIterator.getNextRecord(BAMFileReader.java:981)
	at htsjdk.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:803)
	at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:797)
	at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:765)
	at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.advance(BAMFileReader.java:1034)
	at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:1024)
	at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:988)
	at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:576)
	at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:548)
	at org.broadinstitute.hellbender.utils.iterators.SamReaderQueryingIterator.loadNextRecord(SamReaderQueryingIterator.java:114)
	at org.broadinstitute.hellbender.utils.iterators.SamReaderQueryingIterator.next(SamReaderQueryingIterator.java:151)
	at org.broadinstitute.hellbender.utils.iterators.SamReaderQueryingIterator.next(SamReaderQueryingIterator.java:29)
	at org.broadinstitute.hellbender.utils.iterators.SAMRecordToReadIterator.next(SAMRecordToReadIterator.java:29)
	at org.broadinstitute.hellbender.utils.iterators.SAMRecordToReadIterator.next(SAMRecordToReadIterator.java:15)
	at java.util.Iterator.forEachRemaining(Iterator.java:116)
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at org.broadinstitute.hellbender.engine.ReadWalker.traverse(ReadWalker.java:94)
	at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:838)
	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:115)
	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:170)
	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:189)
	at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:131)
	at org.broadinstitute.hellbender.Main.mainEntry(Main.java:152)
	at org.broadinstitute.hellbender.Main.main(Main.java:230)

I ran the same command again from my computer (not in the cloud) still using the NIO paths and it ran successfully. I've also seen it run successfully when running the same pipeline in the cloud. The only thing I think I've changed is the disk size I'm asking for. I'm in the process of validating the input bam right now.


Return to top