Import VCFs to GenomicsDB
The GATK4 Best Practice Workflow for SNP and Indel calling uses GenomicsDBImport to merge GVCFs from multiple samples. GenomicsDBImport offers the same functionality as CombineGVCFs and initially came from the Intel-Broad Center for Genomics. The datastore transposes sample-centric variant information across genomic loci to make data more accessible to tools.
To query the contents of the GenomicsDB datastore, use SelectVariants. See Tutorial#11813 to get started.
Details on GenomicsDB are at https://github.com/GenomicsDB/GenomicsDB/wiki. In brief, GenomicsDB utilises a data storage system optimized for storing/querying sparse arrays. Genomics data is typically sparse in that each sample has few variants with respect to the entire reference genome. GenomicsDB contains specialized code for genomics applications, such as VCF parsing and INFO field annotation calculation.
One or more GVCFs produced by in HaplotypeCaller with the `-ERC GVCF` or `-ERC BP_RESOLUTION` settings, containing the samples to joint-genotype.
A GenomicsDB workspace
gatk --java-options "-Xmx4g -Xms4g" GenomicsDBImport \
-V data/gvcfs/mother.g.vcf.gz \
-V data/gvcfs/father.g.vcf.gz \
-V data/gvcfs/son.g.vcf.gz \
--genomicsdb-workspace-path my_database \
--tmp-dir=/path/to/large/tmp \
-L 20
Provide sample GVCFs in a map file.
gatk --java-options "-Xmx4g -Xms4g" \
GenomicsDBImport \
--genomicsdb-workspace-path my_database \
--batch-size 50 \
-L chr1:1000-10000 \
--sample-name-map cohort.sample_map \
--tmp-dir=/path/to/large/tmp \
--reader-threads 5
The sample map is a tab-delimited text file with sample_name--tab--path_to_sample_vcf per line. Using a sample map
saves the tool from having to download the GVCF headers in order to determine the sample names. Sample names in
the sample name map file may have non-tab whitespace, but may not begin or end with whitespace.
sample1 sample1.vcf.gz sample2 sample2.vcf.gz sample3 sample3.vcf.gz
This Read Filter is automatically applied to the data by the Engine before processing by GenomicsDBImport.
This table summarizes the command-line arguments that are specific to this tool. For more details on each argument, see the list further down below the table or click on an argument name to jump directly to that entry in the list.
| Argument name(s) | Default value | Summary | |
|---|---|---|---|
| Required Arguments | |||
| --genomicsdb-workspace-path |
null | Workspace for GenomicsDB. Must be a POSIX file system path, but can be a relative path. Must be an empty or non-existent directory. | |
| --intervals -L |
[] | One or more genomic intervals over which to operate | |
| Optional Tool Arguments | |||
| --arguments_file |
[] | read one or more arguments files and add them to the command line | |
| --batch-size |
0 | Batch size controls the number of samples for which readers are open at once and therefore provides a way to minimize memory consumption. However, it can take longer to complete. Use the consolidate flag if more than a hundred batches were used. This will improve feature read time. batchSize=0 means no batching (i.e. readers for all samples will be opened at once) Defaults to 0 | |
| --cloud-index-prefetch-buffer -CIPB |
0 | Size of the cloud-only prefetch buffer (in MB; 0 to disable). Defaults to cloudPrefetchBuffer if unset. | |
| --cloud-prefetch-buffer -CPB |
0 | Size of the cloud-only prefetch buffer (in MB; 0 to disable). | |
| --consolidate |
false | Boolean flag to enable consolidation. If importing data in batches, a new fragment is created for each batch. In case thousands of fragments are created, GenomicsDB feature readers will try to open ~20x as many files. Also, internally GenomicsDB would consume more memory to maintain bookkeeping data from all fragments. Use this flag to merge all fragments into one. Merging can potentially improve read performance, however overall benefit might not be noticeable as the top Java layers have significantly higher overheads. This flag has no effect if only one batch is used. Defaults to false | |
| --disable-bam-index-caching -DBIC |
false | If true, don't cache bam indexes, this will reduce memory requirements but may harm performance if many intervals are specified. Caching is automatically disabled if there are no intervals specified. | |
| --disable-sequence-dictionary-validation |
false | If specified, do not check the sequence dictionaries from our inputs for compatibility. Use at your own risk! | |
| --gcs-max-retries -gcs-retries |
20 | If the GCS bucket channel errors out, how many times it will attempt to re-initiate the connection | |
| --gcs-project-for-requester-pays |
"" | Project to bill when accessing "requester pays" buckets. If unset, these buckets cannot be accessed. | |
| --genomicsdb-segment-size |
1048576 | Buffer size in bytes allocated for GenomicsDB attributes during import. Should be large enough to hold data from one site. Defaults to 1048576 | |
| --genomicsdb-vcf-buffer-size |
16384 | Buffer size in bytes to store variant contexts. Larger values are better as smaller values cause frequent disk writes. Defaults to 16384 which was empirically determined to work well for many inputs. | |
| --help -h |
false | display the help message | |
| --interval-merging-rule -imr |
ALL | Interval merging rule for abutting intervals | |
| --merge-input-intervals |
false | Boolean flag to import all data in between intervals. Improves performance using large lists of intervals, as in exome sequencing, especially if GVCF data only exists for specified intervals. | |
| --overwrite-existing-genomicsdb-workspace |
false | Will overwrite given workspace if it exists. Otherwise a new workspace is created. Defaults to false | |
| --reference -R |
null | Reference sequence | |
| --sites-only-vcf-output |
false | If true, don't emit genotype fields when writing vcf file output. | |
| --validate-sample-name-map |
false | Boolean flag to enable checks on the sampleNameMap file. If true, tool checks whetherfeature readers are valid and shows a warning if sample names do not match with the headers. Defaults to false | |
| --variant -V |
[] | GVCF files to be imported to GenomicsDB. Each file must containdata for only a single sample. Either this or sample-name-map must be specified. | |
| --version |
false | display the version number for this tool | |
| Optional Common Arguments | |||
| --add-output-sam-program-record |
true | If true, adds a PG tag to created SAM/BAM/CRAM files. | |
| --add-output-vcf-command-line |
true | If true, adds a command line header line to created VCF files. | |
| --create-output-bam-index -OBI |
true | If true, create a BAM/CRAM index when writing a coordinate-sorted BAM/CRAM file. | |
| --create-output-bam-md5 -OBM |
false | If true, create a MD5 digest for any BAM/SAM/CRAM file created | |
| --create-output-variant-index -OVI |
true | If true, create a VCF index when writing a coordinate-sorted VCF file. | |
| --create-output-variant-md5 -OVM |
false | If true, create a a MD5 digest any VCF file created. | |
| --disable-read-filter -DF |
[] | Read filters to be disabled before analysis | |
| --disable-tool-default-read-filters |
false | Disable all tool default read filters (WARNING: many tools will not function correctly without their default read filters on) | |
| --exclude-intervals -XL |
[] | One or more genomic intervals to exclude from processing | |
| --gatk-config-file |
null | A configuration file to use with the GATK. | |
| --input -I |
[] | BAM/SAM/CRAM file containing reads | |
| --interval-exclusion-padding -ixp |
0 | Amount of padding (in bp) to add to each interval you are excluding. | |
| --interval-padding -ip |
0 | Amount of padding (in bp) to add to each interval you are including. | |
| --interval-set-rule -isr |
UNION | Set merging approach to use for combining interval inputs | |
| --lenient -LE |
false | Lenient processing of VCF files | |
| --QUIET |
false | Whether to suppress job-summary info on System.err. | |
| --read-filter -RF |
[] | Read filters to be applied before analysis | |
| --read-index |
[] | Indices to use for the read inputs. If specified, an index must be provided for every read input and in the same order as the read inputs. If this argument is not specified, the path to the index for each input will be inferred automatically. | |
| --read-validation-stringency -VS |
SILENT | Validation stringency for all SAM/BAM/CRAM/SRA files read by this program. The default stringency value SILENT can improve performance when processing a BAM file in which variable-length data (read, qualities, tags) do not otherwise need to be decoded. | |
| --seconds-between-progress-updates |
10.0 | Output traversal statistics every time this many seconds elapse | |
| --sequence-dictionary |
null | Use the given sequence dictionary as the master/canonical sequence dictionary. Must be a .dict file. | |
| --tmp-dir |
null | Temp directory to use. | |
| --use-jdk-deflater -jdk-deflater |
false | Whether to use the JdkDeflater (as opposed to IntelDeflater) | |
| --use-jdk-inflater -jdk-inflater |
false | Whether to use the JdkInflater (as opposed to IntelInflater) | |
| --verbosity |
INFO | Control verbosity of logging. | |
| Advanced Arguments | |||
| --max-num-intervals-to-import-in-parallel |
1 | Max number of intervals to import in parallel; higher values may improve performance, but require more memory and a higher number of file descriptors open at the same time | |
| --reader-threads |
1 | How many simultaneous threads to use when opening VCFs in batches; higher values may improve performance when network latency is an issue. Multiple reader threads are not supported when running with multiple intervals. | |
| --sample-name-map |
null | Path to file containing a mapping of sample name to file uri in tab delimited format. If this is specified then the header from the first sample will be treated as the merged header rather than merging the headers, and the sample names will be taken from this file. This may be used to rename input samples. This is a performance optimization that relaxes the normal checks for consistent headers. Using vcfs with incompatible headers may result in silent data corruption. | |
| --showHidden |
false | display hidden arguments | |
Arguments in this list are specific to this tool. Keep in mind that other arguments are available that are shared with other tools (e.g. command-line GATK arguments); see Inherited arguments above.
If true, adds a PG tag to created SAM/BAM/CRAM files.
boolean true
If true, adds a command line header line to created VCF files.
boolean true
read one or more arguments files and add them to the command line
List[File] []
Batch size controls the number of samples for which readers are open at once and therefore provides a way to minimize memory consumption. However, it can take longer to complete. Use the consolidate flag if more than a hundred batches were used. This will improve feature read time. batchSize=0 means no batching (i.e. readers for all samples will be opened at once) Defaults to 0
int 0 [ [ -∞ ∞ ] ]
Size of the cloud-only prefetch buffer (in MB; 0 to disable). Defaults to cloudPrefetchBuffer if unset.
int 0 [ [ -∞ ∞ ] ]
Size of the cloud-only prefetch buffer (in MB; 0 to disable).
int 0 [ [ -∞ ∞ ] ]
Boolean flag to enable consolidation. If importing data in batches, a new fragment is created for each batch. In case thousands of fragments are created, GenomicsDB feature readers will try to open ~20x as many files. Also, internally GenomicsDB would consume more memory to maintain bookkeeping data from all fragments. Use this flag to merge all fragments into one. Merging can potentially improve read performance, however overall benefit might not be noticeable as the top Java layers have significantly higher overheads. This flag has no effect if only one batch is used. Defaults to false
Boolean false
If true, create a BAM/CRAM index when writing a coordinate-sorted BAM/CRAM file.
boolean true
If true, create a MD5 digest for any BAM/SAM/CRAM file created
boolean false
If true, create a VCF index when writing a coordinate-sorted VCF file.
boolean true
If true, create a a MD5 digest any VCF file created.
boolean false
If true, don't cache bam indexes, this will reduce memory requirements but may harm performance if many intervals are specified. Caching is automatically disabled if there are no intervals specified.
boolean false
Read filters to be disabled before analysis
List[String] []
If specified, do not check the sequence dictionaries from our inputs for compatibility. Use at your own risk!
boolean false
Disable all tool default read filters (WARNING: many tools will not function correctly without their default read filters on)
boolean false
One or more genomic intervals to exclude from processing
Use this argument to exclude certain parts of the genome from the analysis (like -L, but the opposite).
This argument can be specified multiple times. You can use samtools-style intervals either explicitly on the
command line (e.g. -XL 1 or -XL 1:100-200) or by loading in a file containing a list of intervals
(e.g. -XL myFile.intervals).
List[String] []
A configuration file to use with the GATK.
String null
If the GCS bucket channel errors out, how many times it will attempt to re-initiate the connection
int 20 [ [ -∞ ∞ ] ]
Project to bill when accessing "requester pays" buckets. If unset, these buckets cannot be accessed.
String ""
Buffer size in bytes allocated for GenomicsDB attributes during import. Should be large enough to hold data from one site. Defaults to 1048576
long 1048576 [ [ -∞ ∞ ] ]
Buffer size in bytes to store variant contexts. Larger values are better as smaller values cause frequent disk writes. Defaults to 16384 which was empirically determined to work well for many inputs.
long 16384 [ [ 1,024 [ 10,240 ∞ ] ]
Workspace for GenomicsDB. Must be a POSIX file system path, but can be a relative path. Must be an empty or non-existent directory.
R String null
display the help message
boolean false
BAM/SAM/CRAM file containing reads
List[String] []
Amount of padding (in bp) to add to each interval you are excluding.
Use this to add padding to the intervals specified using -XL. For example, '-XL 1:100' with a
padding value of 20 would turn into '-XL 1:80-120'. This is typically used to add padding around targets when
analyzing exomes.
int 0 [ [ -∞ ∞ ] ]
Interval merging rule for abutting intervals
By default, the program merges abutting intervals (i.e. intervals that are directly side-by-side but do not
actually overlap) into a single continuous interval. However you can change this behavior if you want them to be
treated as separate intervals instead.
The --interval-merging-rule argument is an enumerated type (IntervalMergingRule), which can have one of the following values:
IntervalMergingRule ALL
Amount of padding (in bp) to add to each interval you are including.
Use this to add padding to the intervals specified using -L. For example, '-L 1:100' with a
padding value of 20 would turn into '-L 1:80-120'. This is typically used to add padding around targets when
analyzing exomes.
int 0 [ [ -∞ ∞ ] ]
Set merging approach to use for combining interval inputs
By default, the program will take the UNION of all intervals specified using -L and/or -XL. However, you can
change this setting for -L, for example if you want to take the INTERSECTION of the sets instead. E.g. to
perform the analysis only on chromosome 1 exomes, you could specify -L exomes.intervals -L 1 --interval-set-rule
INTERSECTION. However, it is not possible to modify the merging approach for intervals passed using -XL (they will
always be merged using UNION).
Note that if you specify both -L and -XL, the -XL interval set will be subtracted from the -L interval set.
The --interval-set-rule argument is an enumerated type (IntervalSetRule), which can have one of the following values:
IntervalSetRule UNION
One or more genomic intervals over which to operate
R List[String] []
Lenient processing of VCF files
boolean false
Max number of intervals to import in parallel; higher values may improve performance, but require more memory and a higher number of file descriptors open at the same time
int 1 [ [ 1 ∞ ] ]
Boolean flag to import all data in between intervals. Improves performance using large lists of intervals, as in exome sequencing, especially if GVCF data only exists for specified intervals.
boolean false
Will overwrite given workspace if it exists. Otherwise a new workspace is created. Defaults to false
Boolean false
Whether to suppress job-summary info on System.err.
Boolean false
Read filters to be applied before analysis
List[String] []
Indices to use for the read inputs. If specified, an index must be provided for every read input and in the same order as the read inputs. If this argument is not specified, the path to the index for each input will be inferred automatically.
List[String] []
Validation stringency for all SAM/BAM/CRAM/SRA files read by this program. The default stringency value SILENT can improve performance when processing a BAM file in which variable-length data (read, qualities, tags) do not otherwise need to be decoded.
The --read-validation-stringency argument is an enumerated type (ValidationStringency), which can have one of the following values:
ValidationStringency SILENT
How many simultaneous threads to use when opening VCFs in batches; higher values may improve performance when network latency is an issue. Multiple reader threads are not supported when running with multiple intervals.
int 1 [ [ 1 ∞ ] ]
Reference sequence
String null
Path to file containing a mapping of sample name to file uri in tab delimited format. If this is specified then the header from the first sample will be treated as the merged header rather than merging the headers, and the sample names will be taken from this file. This may be used to rename input samples. This is a performance optimization that relaxes the normal checks for consistent headers. Using vcfs with incompatible headers may result in silent data corruption.
Exclusion: This argument cannot be used at the same time as variant.
String null
Output traversal statistics every time this many seconds elapse
double 10.0 [ [ -∞ ∞ ] ]
Use the given sequence dictionary as the master/canonical sequence dictionary. Must be a .dict file.
String null
display hidden arguments
boolean false
If true, don't emit genotype fields when writing vcf file output.
boolean false
Temp directory to use.
String null
Whether to use the JdkDeflater (as opposed to IntelDeflater)
boolean false
Whether to use the JdkInflater (as opposed to IntelInflater)
boolean false
Boolean flag to enable checks on the sampleNameMap file. If true, tool checks whetherfeature readers are valid and shows a warning if sample names do not match with the headers. Defaults to false
Boolean false
GVCF files to be imported to GenomicsDB. Each file must containdata for only a single sample. Either this or sample-name-map must be specified.
Exclusion: This argument cannot be used at the same time as sample-name-map.
List[String] []
Control verbosity of logging.
The --verbosity argument is an enumerated type (LogLevel), which can have one of the following values:
LogLevel INFO
display the version number for this tool
boolean false
See also General Documentation | Tool Docs Index Tool Documentation Index | Support Forum
GATK version 4.1.0.0 built at Wed, 30 Jan 2019 10:21:04 +0530.