By Ruchi Munshi, Senior Software Product Manager, Data Sciences Platform

For a long time now, the Cromwell workflow engine has supported popular high-performance computing (HPC) platforms such as SGE, LSF and SLURM. But when it comes to clouds, Cromwell only had native support for the Google Cloud Platform (GCP), which is done via the Google Pipelines API.

Today, I’m thrilled to announce that over the span of this year, we’ve expanded Cromwell's native cloud support to a suite of several cloud backends to parallel our suite of supported HPC backends. The newly released Cromwell version 35 will support three cloud platforms natively: GCP with an upgraded backend to support the new Pipelines API v2; Amazon Web Services (AWS) through a brand new backend that connects to the Batch API (in beta status); and Alibaba Cloud, whose Cromwell backend was introduced in a previous version earlier this year.


The new AWS backend orchestrates workflows through the AWS Batch API. This enables us to deliver Cromwell's effortless workflow experience to the many users who are already working within the Amazon cloud. Want to learn more? We recently recorded a joint webinar with the AWS team that developed most of the backend code; the webinar covers all you need to know to get started with the new backend, including basic configuration and core capabilities, such as managing data in S3 buckets, taking advantage of cheap computing via the Amazon EC2 Spot market and more. You can find the recording here.

On the GCP front, we have a major update revolving around the adoption of the Google Pipelines API v2, which enables us to provide some eagerly awaited features. For example, many of our users have asked for the ability to request custom machine types, which can be more cost-effective than computing on standard machines. With the new version, you'll be able to request custom machines by default. We have made sure that upgrading the backend will be easy for all users -- all it takes is a small one-time change to your Cromwell configuration file to use the Pipelines API v2 backend, which is documented in the release notes.


Return to top

Mon 1 Oct 2018
Comment on this article


- Recent posts


- Upcoming events

See Events calendar for full list and dates


- Recent events

See Events calendar for full list and dates



- Follow us on Twitter

WDL Dev Team

@WDL_dev

RT @TristanNaumann: Very cool to see Cromwell on Azure 1.0.0 released on GitHub! https://t.co/ZdQ9glQSZs
6 Nov 19
RT @hcaskey: Earlier this month, we released Cromwell on Azure, an open-source project on GitHub from Microsoft Genomics that provides scie…
6 Nov 19
@dbernick @fdmts java -Dconfig.file=google.conf -jar cromwell-47.jar run hello.wdl -i hello.inputs
26 Oct 19
Featuring WDL and Cromwell on @Azure thanks to @Microsoft @Health_IT -- see also our blog at… https://t.co/gCUAQrcgC1
16 Oct 19
Blue skies ahead for Cromwell on Azure! https://t.co/35jHw7yIqt https://t.co/2OD9w8EyZ9
15 Oct 19

- Our favorite tweets from others

To all the workflow developers out there, we’ve added support for @WDL_dev 1.0. Check out our getting started doc h… https://t.co/eYhN1c84QD
28 Oct 19
@tangming2005 @geoffjentry @dnanexus I was surprised at the cost too, most user group meetings in biotech are free/… https://t.co/GP0Os3t64r
9 Sep 19
Nice overview of bioinformatics workflows definition and execution including discussion of @commonwl @galaxyprojecthttps://t.co/3WfecaEEJH
3 Sep 19
Workflow systems turn raw data into scientific knowledge https://t.co/t5jUjl39RW
2 Sep 19
@DNAstack @DTSupercluster Ok, so now I'm expecting this many more @WDL_dev PRs :P
2 Aug 19

See more of our favorite tweets...