Latest posts
 


29 Release Notes

Breaking Changes

  • Command line
    In preparation for supporting CWL scripts (yes, you read that right!), we have extensively revised the Command Line in Cromwell 29. For more details about the usage changes please see the README. And stay tuned to the WDL/Cromwell blog over the next couple of months for more news about CWL.

  • Request timeouts
    Cromwell now returns more specific 503 Service Unavailable error codes on request timeouts, rather than the more generic 500 Internal Server Error. The response for a request timeout will now be plain text, rather than a JSON format.

  • Metadata endpoint
    The response from the metadata endpoint can be quite large depending on your workflow. You can now opt-in to have Cromwell gzip your metadata file, in order to reduce file size, by sending the Accept-Encoding: gzip header. The default behavior now does not gzip encode responses.

  • Engine endpoints
    Previously the engine endpoints were available under /api/engine but now the endpoints are under /engine so they don't require authentication. Workflow endpoints are still available under /api/workflows. We also deprecated the setting api.routeUnwrapped as a part of this internal consistency effort.

  • Call caching diff
    We updated the response format of the callcaching/diff endpoint.

Other changes

  • Cromwell server
    When running in server mode, Cromwell now attempts to gracefully shutdown after receiving a SIGINT (Ctrl-C) or SIGTERM (kill) signal. This means that Cromwell waits for all pending database writes before exiting, as long as you include application.conf at the top of your config file. You can find detailed information about how to configure this feature in the Cromwell Wiki.

  • Concurrent jobs
    You can now limit the number of concurrent jobs for any backend. Previously this was only possible in some backend implementations. Please see the README for details.

WDL

  • Optional WDL variables
    Empty optional WDL values are now rendered as the null JSON value instead of the JSON string "null" in the metadata and output endpoints. You do not need to migrate previous workflows. Workflows run on Cromwell 28 and prior will still render empty values as "null".

  • Empty WDL variables
    Cromwell now accepts null JSON values in the input file and coerces them as an empty WDL value. WDL variables must be declared optional in order to be supplied with a null JSON value.

input.json

{
    "null_input_values.maybeString": null,
    "null_input_values.arrayOfMaybeInts": [1, 2, null, 4]
}

workflow.wdl

workflow null_input_values {
    String? maybeString
    Array[Int?] arrayOfMaybeInts
}
See comments (0)



They have a DOI and everything! Kate Voss presented this poster at the Bioinformatics Open-Source Conference in Prague last month. See the abstract below. The full citation is:

Voss K, Gentry J and Van der Auwera G. Full-stack genomics pipelining with GATK4 + WDL + Cromwell [version 1; not peer reviewed]. F1000Research 2017, 6(ISCB Comm J):1379 (poster) (doi: 10.7490/f1000research.1114631.1)

In case you're wondering, the flying pig with warp nacelles is Jamie, the mascot of the Cromwell execution engine. And the story behind that... is a story for another time :)


Read the whole post
See comments (0)



Heads up! The upcoming version 29 of Cromwell will include a breaking change in the command line syntax. Full details will be provided in the release notes when Cromwell 29 is ready for primetime; in the meantime the gist is that we're moving away from positional inputs in favor of explicit arguments. So for example this simple execution command:

java -jar Cromwell.jar run myWorkflow.wdl myWorkflow_inputs.json

will change to the more explicit:

java -jar Cromwell.jar run myWorkflow.wdl --inputs myWorkflow_inputs.json

The code changes can be viewed here for those who are into that level of detail.

We plan to update usage examples and relevant instructions in the WDL documentation materials at the time the new version is released, and will add a note regarding versions where appropriate for anyone still using a version anterior to 29. Note that there may be a short lag between the release and the documentation updates; we'll try to keep it to a minimum.

See comments (0)



Last week we released Cromwell 27, the latest and greatest Cromwell to date! Much of our work this quarter has been directed towards improving the general speed and scalability of Cromwell.

But before you do, have a look at the new feature highlights below!


Read the whole post
See comments (0)



Cromwell I/O: Highway to the Cloud

Posted by Thib on 16 May 2017 (0)


Cromwell 26 has a new way to deal with I/O (Input/Output) operations. This new approach reduces network load and provides better control over the resources allocated throughout the system. This blogpost will describe some of the optimizations and how they improve Cromwell stability and reliability.


Read the whole post
See comments (0)



Cromwell 25 was released a few weeks ago but already Cromwell 26 is available! In this release we improved Cromwell's ability to cope with failure, including adding cases where Cromwell will retry jobs that failed due to transient errors. Note that migration from version 25 to 26 is expected to take extra time as explained further below.

But before you do, have a look at the version highlights below!


Read the whole post
See comments (0)



TESting, TESting, 1 2 3...

Posted by jgentry on 20 Mar 2017 (0)


TL;DR :In Cromwell 25, we added a backend named TES to Cromwell’s portfolio, promoting the GA4GH vision of interoperability between genomic analysis tools. We exercise the TES backend using Funnel, a neat piece of software coming out of Kyle Ellrott’s group at OHSU that allows us to dispatch jobs to a variety of platforms using the same API.

The Global Alliance For Genomics & Health (aka GA4GH) is an international coalition formed to enable the sharing of genomic and clinical data in order to help unlock potential advancements in medicine and science. For the most part the GA4GH provides APIs that are implemented by frameworks and tools throughout our field. With these standardized APIs, analysts and software developers are able to take advantage of a much broader and richer ecosystem of tools than they previously were able to. But more on this later. The take home message is that now the scientific community as a whole is able to spend more time working towards bettering humanity instead of just gluing tools together.

Within the GA4GH is the Containers & Workflows (CWF) working group, which focuses on providing APIs that define generic schemes for identifying tools, submitting workflows, and submitting jobs to compute platforms (disclosure: I happen to co-chair this group). At my day job as one of the developers of Cromwell I’m interested in how we can implement these APIs to promote interoperability of different computing platforms in the bioinformatics space.


Read the whole post
See comments (0)



Well, no, there isn't a Cromwell/WDL musical (yet) -- it's just that I gave a talk last night in New York, at a meetup organized by Phosphorus and hosted by FirstMark. We had a great crowd, and I have to give them mad props for listening to me go on about GATK workflows and pipelining strategies for well over the allotted hour. Especially considering I've been getting over a bout of laryngitis and my voice kept oscillating between high-pitched whine and raspy whisper... There will be a video posted on YouTube in the next few days and it would be awesome if they could get someone to do a voice dub! In the meantime, my slide deck is available here.

UPDATE: And here's the video on YouTube.


Read the whole post
See comments (0)



Cromwell 25 release

Posted by KateVoss on 23 Feb 2017 (1)


Happy 25th birthday, Cromwell! You're a quarter of a century old! In this version, we added a major new feature we named "call caching" that could potentially save you tons of money; as well as labels, a TES backend and new WDL functions.

But before you do, have a look at the version highlights below!


Read the whole post
See comments (1)



Announcing the WDL Blog!

Posted by KateVoss on 22 Feb 2017 (1)


WDL-away your day with us on the WDL blog! These blog posts will be geared towards developers who are looking for the nitty-gritty-codey details of WDL and Cromwell, such as new backends, parameters, etc. We'll post about new WDL documents and Cromwell releases, as well as other technical topics.

You can use your favorite RSS reader to subscribe to just the WDL Announcements category of posts. Click here for information on subscribing to WDL Announcements by email.

To subscribe via RSS to blog posts about FireCloud, click here. To subscribe via RSS to blog posts about GATK, click here.

And don't forget to follow us on Twitter!

See comments (1)



Latest posts
 

- Recent posts


- Upcoming events

See Events calendar for full list and dates


- Recent events

See Events calendar for full list and dates



- Follow us on Twitter

WDL Dev Team

@WDL_dev

Neat blog post from @DNAstack highlights the user-friendliness of #WDL and Cromwell for building analysis pipelines https://t.co/3maHuy2i2n
9 Aug 17
Cromwell and WDL are now officially citable in literature! They have a DOI and everything! https://t.co/J5Je9BXWj7 https://t.co/NsUkba1nlZ
8 Aug 17
Heads up! Syntax change in upcoming Cromwell v29 command line; positional inputs replaced by explicit CLI args https://t.co/AXmE5MdLlO
6 Aug 17
New Toolkit page on the #WDL website features useful tools & resources for WDL development & execution. Let us know if we're missing one!
3 Aug 17
Goal of WDL is to empower researchers to read and write workflows + understand scripts so pipelines are not black b… https://t.co/P6AiNnqBew
30 Jul 17

- Our favorite tweets from others

See more of our favorite tweets...