Developed in the Data Sciences Platform at the Broad Institute, this pipelining solution features Cromwell, a flexible workflow management system that supports multiple computing platforms, from popular public clouds to classic HPC schedulers. Cromwell can run both languages adopted by the GA4GH driver projects: the user-friendly Workflow Description Language (WDL) and the Common Workflow Language (CWL).
Cromwell supports multiple public clouds and HPC schedulers natively through pluggable backends. It is routinely used at the Broad Institute to run thousands of whole genome analysis workflows on the Google Cloud Platform, and is increasingly adopted by other groups to run workloads at various scales. It can be run in standalone mode (one command, one workflow) or as a server that accepts REST queries, and includes efficiency features such as call caching for smart recovery after interruptions.
Originally designed to run the user-friendly Workflow Description Language (WDL), Cromwell is now also capable of running workflows written in the Common Workflow Language (CWL), making it the most flexible system for GA4GH compliant task execution to date.
The Workflow Description Language (WDL) is a way to specify data processing workflows with a human-readable and -writeable syntax. WDL makes it straightforward to define analysis tasks, chain them together in workflows, and parallelize their execution. The language makes common patterns simple to express, while also admitting uncommon or complicated behavior; and strives to achieve portability not only across execution platforms, but also different types of users. Whether you are an analyst, a programmer, an operator of a production system, or any other sort of user, WDL should be accessible and understandable.
WDL was originally developed for genome analysis pipelines by the Broad Institute. As its community grew to include many individuals and organizations outside of the Broad, it became clear that there was a need to allow WDL to become a true community-driven standard. The OpenWDL community was thus formed to serve as an independent steward of the WDL language specification and to advocate for its adoption.
This is a permissive license similar to the BSD 2-Clause License, but with a 3rd clause that prohibits others from using the name of the project or its contributors to promote derived products without written consent.
The full text of the license can be viewed here.
Running pipelines doesn't have to be a nightmare. We provide various resources to help you learn to write and run WDL through Cromwell, as well as getting started with Cromwell itself, whether you're planning to run standalone commands, operate your own server or use a Cromwell server operated by someone else (such as our very own FireCloud service.
The WDL User Guide provides introductory materials for getting started with reading and writing workflows in WDL, along with frequently asked questions and solutions to common problems, and tutorials that explain step by step how to use basic features of the language. The Cromwell Docs, currently hosted separately, provide technical information for configuring and operating Cromwell on various platforms.
Be sure to check out the Presentations from our recurring workshop series. In addition to the slide decks, we provide recordings of the workshops that we hold at the Broad; you can view them on the Broad website or on the Broad education channels on YouTube and iTunesU.
Finally, if you've exhausted all these avenues and still haven't found the answer to your question, check out the forum! You may find that others have run into the same problem and that the solution has already been posted. If not, let us know and we'll do our best to address your problems quickly and accurately. If something's not clearly documented, we'll answer your question and improve the docs accordingly. If you think you found a bug, we'll track it down and fix it. Just ask our team.