GATK on the Cloud
Options for running GATK on public cloud platforms


We are working with cloud service providers to provide ready-to-use services for running GATK pipelines on the cloud. For some background information on this effort, please see this announcement from last year.

At this time we are able to offer two services for running WDL workflows on Google Cloud using the Broad's Cromwell execution engine (described in the Pipelining Options section) and the Google Pipelines API. Note that while access to both of these services is free, compute and storage are subject to Google Cloud pricing, charged directly by Google (Broad does not apply any markup).


Google Pipelines API + WDL_Runner

These open-source command-line utilities were developed by Google Cloud to enable execution of analysis pipelines on the Google Cloud platform. The Google Pipelines API is a pipeline execution service that runs jobs on the Google Compute Engine (GCE). WDL_Runner is a package that provides support for running workflows written in WDL through the Pipelines API, via a Cromwell server that is spun up and torn down ad-hoc by WDL_Runner itself. See the GATK on Google Genomics tutorial for a hands-on demonstration of how to run one of our Best Practices production workflows on this service.


FireCloud: Broad's cloud-based analysis platform

This web-based application is provided as a freely accessible service by the Broad Institute's Data Sciences Platform, where GATK itself is also developed. FireCloud provides both GUI (point-and-click) and API access to a persistent Cromwell execution server that manages submissions to the Google Pipelines API. In addition to the core pipeline execution service, the FireCloud platform also includes functionality for data management, a data library of published datasets (including TCGA data) and a method repository for managing and sharing workflows. The platform as a whole is designed to empower analysts, tool developers and production managers to perform large-scale analysis, engage in data curation, and store or publish results. You can find out more about the FireCloud platform at Firecloud.org.

Note that while the documentation describes FireCloud as a cancer-focused analysis platform, it can in fact be used for germline variant analysis, or any analysis that can be scripted as a workflow, really.

We are currently working with the FireCloud team to provide ready-to-run GATK Best Practices pipelines within FireCloud. Stay tuned to the GATK blog or follow @gatk_dev on Twitter for progress updates and announcements.