By Ruchi Munshi, Senior Software Product Manager, Data Sciences Platform

For a long time now, the Cromwell workflow engine has supported popular high-performance computing (HPC) platforms such as SGE, LSF and SLURM. But when it comes to clouds, Cromwell only had native support for the Google Cloud Platform (GCP), which is done via the Google Pipelines API.

Today, I’m thrilled to announce that over the span of this year, we’ve expanded Cromwell's native cloud support to a suite of several cloud backends to parallel our suite of supported HPC backends. The newly released Cromwell version 35 will support three cloud platforms natively: GCP with an upgraded backend to support the new Pipelines API v2; Amazon Web Services (AWS) through a brand new backend that connects to the Batch API (in beta status); and Alibaba Cloud, whose Cromwell backend was introduced in a previous version earlier this year.


The new AWS backend orchestrates workflows through the AWS Batch API. This enables us to deliver Cromwell's effortless workflow experience to the many users who are already working within the Amazon cloud. Want to learn more? We recently recorded a joint webinar with the AWS team that developed most of the backend code; the webinar covers all you need to know to get started with the new backend, including basic configuration and core capabilities, such as managing data in S3 buckets, taking advantage of cheap computing via the Amazon EC2 Spot market and more. You can find the recording here.

On the GCP front, we have a major update revolving around the adoption of the Google Pipelines API v2, which enables us to provide some eagerly awaited features. For example, many of our users have asked for the ability to request custom machine types, which can be more cost-effective than computing on standard machines. With the new version, you'll be able to request custom machines by default. We have made sure that upgrading the backend will be easy for all users -- all it takes is a small one-time change to your Cromwell configuration file to use the Pipelines API v2 backend, which is documented in the release notes.


Return to top

Mon 1 Oct 2018
Comment on this article


- Recent posts


- Upcoming events

See Events calendar for full list and dates


- Recent events

See Events calendar for full list and dates



- Follow us on Twitter

WDL Dev Team

@WDL_dev

RT @BlayzeStefaniak: Have Cromwell running on AWS Batch, very easy to work with WDL and get things working. Cool stuff!
11 Nov 18
RT @geoffjentry: Cool to see Cromwell and @WDL_dev on display at #sc18 https://t.co/NRizE3YRdd
11 Nov 18
RT @michaelhoffman: CY: Took 2 years, but today would only take 6 months due to availability of CWL, WDL, Cromwell, Kubernetes, Dockstore,…
17 Oct 18
@awscloud @BroadGenomics Jamie the Cromwell pig runs on AWS Batch! Read the blog post by @RuchMunsh at… https://t.co/n6AK77ljQo
1 Oct 18
@sminot @iprophage Tadaaa Cromwell is now on AWS, see https://t.co/gT0KFnmGC5 and https://t.co/375wew4Alk
1 Oct 18

- Our favorite tweets from others

Cool to see Cromwell and @WDL_dev on display at #sc18 https://t.co/NRizE3YRdd
11 Nov 18
I've wrapped @dnanexus and @sz_genomics excellent parliament2 in @WDL_dev, so if you're using the Broad's FireCloud… https://t.co/SBFdXZCzya
9 Nov 18
Have Cromwell running on AWS Batch, very easy to work with WDL and get things working. Cool stuff!
6 Nov 18
@sminot So true! If the workflow manager side of you is interested, check this out. I’ve been getting more into thi… https://t.co/GmXZKac9b5
21 Sep 18
We had a lot of fun in the last day of the #GATK workshop. A lot to learn and practice on their workflow descriptiv… https://t.co/bTMSyT8W60
20 Sep 18

See more of our favorite tweets...