By Ruchi Munshi, Senior Software Product Manager, Data Sciences Platform

For a long time now, the Cromwell workflow engine has supported popular high-performance computing (HPC) platforms such as SGE, LSF and SLURM. But when it comes to clouds, Cromwell only had native support for the Google Cloud Platform (GCP), which is done via the Google Pipelines API.

Today, I’m thrilled to announce that over the span of this year, we’ve expanded Cromwell's native cloud support to a suite of several cloud backends to parallel our suite of supported HPC backends. The newly released Cromwell version 35 will support three cloud platforms natively: GCP with an upgraded backend to support the new Pipelines API v2; Amazon Web Services (AWS) through a brand new backend that connects to the Batch API (in beta status); and Alibaba Cloud, whose Cromwell backend was introduced in a previous version earlier this year.

The new AWS backend orchestrates workflows through the AWS Batch API. This enables us to deliver Cromwell's effortless workflow experience to the many users who are already working within the Amazon cloud. Want to learn more? We recently recorded a joint webinar with the AWS team that developed most of the backend code; the webinar covers all you need to know to get started with the new backend, including basic configuration and core capabilities, such as managing data in S3 buckets, taking advantage of cheap computing via the Amazon EC2 Spot market and more. You can find the recording here.

On the GCP front, we have a major update revolving around the adoption of the Google Pipelines API v2, which enables us to provide some eagerly awaited features. For example, many of our users have asked for the ability to request custom machine types, which can be more cost-effective than computing on standard machines. With the new version, you'll be able to request custom machines by default. We have made sure that upgrading the backend will be easy for all users -- all it takes is a small one-time change to your Cromwell configuration file to use the Pipelines API v2 backend, which is documented in the release notes.

Return to top

Mon 1 Oct 2018
Comment on this article

- Recent posts

- Upcoming events

See Events calendar for full list and dates

- Recent events

See Events calendar for full list and dates

- Follow us on Twitter

WDL Dev Team


RT @gatk_dev: Would anyone in the Boston/Cambridge area be interested in an informal meetup about GATK? Chat about current tools, go over a…
6 Jun 19
RT @geoffjentry: Win a trip to @OBF_BOSC this summer in Basel!
18 May 19
RT @geoffjentry: We had a hackathon yesterday to make the AWS support in Cromwell. A lot of great progress was made, the next release shoul…
20 Apr 19
RT @xDBio_Inc: Are you a Python developer with experience with @WDL_dev and @commonwl ? If so, get in touch with us, we're looking for you!…
18 Apr 19
RT @geoffjentry: It appears Cromwell has reached "bar graffiti" status. Onwards and upwards, Jamie!
8 Apr 19

- Our favorite tweets from others

Win a trip to @OBF_BOSC this summer in Basel!
17 May 19
Do you have a pet workflow or a favorite notebook? Have you thought about sharing them with the world, but keep pus…
17 May 19
Err that’s “make it better”, not “make it” 🤦🏻‍♂️
19 Apr 19
We had a hackathon yesterday to make the AWS support in Cromwell. A lot of great progress was made, the next releas…
19 Apr 19
@ecerami @WDL_dev @googlecloud This is a great resource you put together @ecerami - let us know if we can be of any…
29 Mar 19

See more of our favorite tweets...