Latest posts
 


New Cromwell Documentation

Posted by KateVoss on 31 Oct 2017 (0)


Previously all of the Cromwell "documentation" was in the ReadMe on the Cromwell Github repo. We knew it, you knew it. The documentation needed a lot of work.

But not anymore!

Visit the new and much-improved Cromwell Documentation Site at Cromwell.readthedocs.io.

Take a look around and check out the new section of Cromwell Tutorials.

If you spot any errors or want to make any additions, the source of the documentation is in Github and pull requests are always welcome :wink: Finally if you have trouble finding anything use the Search in the top left-hand corner.

Thank you for your patience!

See comments (0)



TL;DR: We are handing over governance of WDL to an open community-based organization called OpenWDL that will ensure WDL development is driven by and for the research community.

When WDL was created a few years ago, our mission was ostensibly to produce a pipelining system for the Broad's internal needs. But we knew at the time it would have to be much more than just a Broad-specific tool, because we had already decided we would share our analysis pipelines as widely as possible. The most immediate consequence was that whatever pipelining solution we adopted needed to be portable across not only our method development and production environments, but also across a variety of systems used in the research community. The other was that the scripting language we used needed to be as user-friendly as possible, to be at the very least readable by e.g. biomedical scientists without formal computational training (which happens to be my own background, so I have an especially soft spot for that constituency). Those were huge factors that led to our decision to create a new engine (Cromwell) and domain language (WDL itself, obviously ;)) rather than adopt an existing system. It also motivated us to provide as much help (via the forum) and documentation as we could manage to anyone who expressed interest, whether they were Broadies or not.

Fast-forward to the present day, where we're starting to see individuals and organizations throughout the world adopt WDL (sometimes independently of Cromwell) for their own pipelining needs. This is a incredibly gratifying but also rather daunting, because that means we now bear some degree of responsibility in ensuring that the people who adopt WDL are empowered to be successful in their pipelining endeavors.


Read the whole post
See comments (0)



Heads up! We're participating in an event organized by DNAstack in the context of the GA4GH meetings, just prior to the ASHG meeting in Orlando, FL in a few weeks. It's a two-hour evening event focused on open reproducible workflows, and will feature hands-on demos of WDL and Cromwell, as well as Dockstore and DNAstack's own platform, which uses WDL. The WDL and Cromwell section will be presented by our very own Jeff Gentry, who leads the Cromwell development team, accompanied by Cromwell product owner Kate Voss and GATK's resident scientific writer, Soo Hee Lee. We'll be running some GATK4 pipelines and generally having a good time nerding out about easy breezy reproducible workflows.

So join us if you're in town; register here now, as space is limited.

See comments (0)



29 Release Notes

Breaking Changes

  • Command line
    In preparation for supporting CWL scripts (yes, you read that right!), we have extensively revised the Command Line in Cromwell 29. For more details about the usage changes please see the README. And stay tuned to the WDL/Cromwell blog over the next couple of months for more news about CWL.

  • Request timeouts
    Cromwell now returns more specific 503 Service Unavailable error codes on request timeouts, rather than the more generic 500 Internal Server Error. The response for a request timeout will now be plain text, rather than a JSON format.

  • Metadata endpoint
    The response from the metadata endpoint can be quite large depending on your workflow. You can now opt-in to have Cromwell gzip your metadata file, in order to reduce file size, by sending the Accept-Encoding: gzip header. The default behavior now does not gzip encode responses.

  • Engine endpoints
    Previously the engine endpoints were available under /api/engine but now the endpoints are under /engine so they don't require authentication. Workflow endpoints are still available under /api/workflows. We also deprecated the setting api.routeUnwrapped as a part of this internal consistency effort.

  • Call caching diff
    We updated the response format of the callcaching/diff endpoint.

Other changes

  • Cromwell server
    When running in server mode, Cromwell now attempts to gracefully shutdown after receiving a SIGINT (Ctrl-C) or SIGTERM (kill) signal. This means that Cromwell waits for all pending database writes before exiting, as long as you include application.conf at the top of your config file. You can find detailed information about how to configure this feature in the Cromwell Wiki.

  • Concurrent jobs
    You can now limit the number of concurrent jobs for any backend. Previously this was only possible in some backend implementations. Please see the README for details.

WDL

  • Optional WDL variables
    Empty optional WDL values are now rendered as the null JSON value instead of the JSON string "null" in the metadata and output endpoints. You do not need to migrate previous workflows. Workflows run on Cromwell 28 and prior will still render empty values as "null".

  • Empty WDL variables
    Cromwell now accepts null JSON values in the input file and coerces them as an empty WDL value. WDL variables must be declared optional in order to be supplied with a null JSON value.

input.json

{
    "null_input_values.maybeString": null,
    "null_input_values.arrayOfMaybeInts": [1, 2, null, 4]
}

workflow.wdl

workflow null_input_values {
    String? maybeString
    Array[Int?] arrayOfMaybeInts
}
See comments (0)



They have a DOI and everything! Kate Voss presented this poster at the Bioinformatics Open-Source Conference in Prague last month. See the abstract below. The full citation is:

Voss K, Gentry J and Van der Auwera G. Full-stack genomics pipelining with GATK4 + WDL + Cromwell [version 1; not peer reviewed]. F1000Research 2017, 6(ISCB Comm J):1379 (poster) (doi: 10.7490/f1000research.1114631.1)

In case you're wondering, the flying pig with warp nacelles is Jamie, the mascot of the Cromwell execution engine. And the story behind that... is a story for another time :)


Read the whole post
See comments (0)



Heads up! The upcoming version 29 of Cromwell will include a breaking change in the command line syntax. Full details will be provided in the release notes when Cromwell 29 is ready for primetime; in the meantime the gist is that we're moving away from positional inputs in favor of explicit arguments. So for example this simple execution command:

java -jar Cromwell.jar run myWorkflow.wdl myWorkflow_inputs.json

will change to the more explicit:

java -jar Cromwell.jar run myWorkflow.wdl --inputs myWorkflow_inputs.json

The code changes can be viewed here for those who are into that level of detail.

We plan to update usage examples and relevant instructions in the WDL documentation materials at the time the new version is released, and will add a note regarding versions where appropriate for anyone still using a version anterior to 29. Note that there may be a short lag between the release and the documentation updates; we'll try to keep it to a minimum.

See comments (0)



Last week we released Cromwell 27, the latest and greatest Cromwell to date! Much of our work this quarter has been directed towards improving the general speed and scalability of Cromwell.

But before you do, have a look at the new feature highlights below!


Read the whole post
See comments (0)



Cromwell I/O: Highway to the Cloud

Posted by Thib on 16 May 2017 (0)


Cromwell 26 has a new way to deal with I/O (Input/Output) operations. This new approach reduces network load and provides better control over the resources allocated throughout the system. This blogpost will describe some of the optimizations and how they improve Cromwell stability and reliability.


Read the whole post
See comments (0)



Cromwell 25 was released a few weeks ago but already Cromwell 26 is available! In this release we improved Cromwell's ability to cope with failure, including adding cases where Cromwell will retry jobs that failed due to transient errors. Note that migration from version 25 to 26 is expected to take extra time as explained further below.

But before you do, have a look at the version highlights below!


Read the whole post
See comments (0)



TESting, TESting, 1 2 3...

Posted by jgentry on 20 Mar 2017 (0)


TL;DR :In Cromwell 25, we added a backend named TES to Cromwell’s portfolio, promoting the GA4GH vision of interoperability between genomic analysis tools. We exercise the TES backend using Funnel, a neat piece of software coming out of Kyle Ellrott’s group at OHSU that allows us to dispatch jobs to a variety of platforms using the same API.

The Global Alliance For Genomics & Health (aka GA4GH) is an international coalition formed to enable the sharing of genomic and clinical data in order to help unlock potential advancements in medicine and science. For the most part the GA4GH provides APIs that are implemented by frameworks and tools throughout our field. With these standardized APIs, analysts and software developers are able to take advantage of a much broader and richer ecosystem of tools than they previously were able to. But more on this later. The take home message is that now the scientific community as a whole is able to spend more time working towards bettering humanity instead of just gluing tools together.

Within the GA4GH is the Containers & Workflows (CWF) working group, which focuses on providing APIs that define generic schemes for identifying tools, submitting workflows, and submitting jobs to compute platforms (disclosure: I happen to co-chair this group). At my day job as one of the developers of Cromwell I’m interested in how we can implement these APIs to promote interoperability of different computing platforms in the bioinformatics space.


Read the whole post
See comments (0)



Latest posts
 

- Recent posts


- Upcoming events

See Events calendar for full list and dates


- Recent events

See Events calendar for full list and dates



- Follow us on Twitter

WDL Dev Team

@WDL_dev

RT @seandavis12: @galaxyproject @ProjectJupyter @Docker @galaxy And if you want to invert the concept, look at https://t.co/ZEj1l1uMrc, tha…
3 Nov 17
@github now highlights #WDL syntax! https://t.co/Nj4eFZPXRu https://t.co/URGdqGbKvC
17 Oct 17
RT @BroadGenomics: Join our @WDL_dev & @gatk_dev teams to learn more on reproducible workflows! #ASHG17 https://t.co/qt6nypKSLc
13 Oct 17
Introducing plans for OpenWDL, a public organization for WDL governance https://t.co/wNQUTnoXO2 #openwdl
6 Oct 17
Real-world WDL workflows in action! If you can't make it, the WDLs will be available in Github under… https://t.co/xJcXvTemPR
6 Oct 17

- Our favorite tweets from others

See more of our favorite tweets...