Latest posts
 


In January I reported that Cromwell was expanding workflow language support beyond WDL via a concept we call the Workflow Object Model (WOM). In that post I announced that development towards supporting the Common Workflow Language (CWL) was already underway. As of today, that work is complete and we have released Cromwell version 32 which supports running CWL workflows.


Read the whole post
See comments (0)



By Jose Soto, Software Engineer in the Broad Institute's Data Sciences Platform

Over the past few months, I spent a lot of time optimizing our pipelines for cost (including for the $5 Genome Analysis Pipeline), so I thought I'd share a few of the tricks I find especially effective, starting with what we call dynamic sizing.

The idea is that on a pay-as-you-go cloud like Google Cloud Platform, the rate you pay for compute is based on the technical specs of the machines you use; the beefier the machine, the higher the rate. So you can save yourself a lot of money by making sure you don't use machines that are beefier than you need.

Here's how we do that in our pipelines.


Read the whole post
See comments (0)



Cromwell and WDL started out as a codependent couple, matched by design and tied together by habit. But I like to emphasize that Cromwell and WDL share more than just development history. They also share a fundamental philosophy that prioritizes user-friendliness, with the goal of making simple use cases easy to realize while ensuring that users with more complex needs have a reasonable path forward. I believe that this philosophy plays a key role in making Cromwell and WDL best in class tools for bioinformatics computing. But sometimes user-friendliness means a different thing for one compared to the other. And so it has become clear that in order to maximize their usefulness and satisfy the needs of a wide community, Cromwell and WDL need to be decoupled so that they can both evolve more freely.


Read the whole post
See comments (3)


New Cromwell Documentation

Posted by KateVoss on 31 Oct 2017 (0)


Previously all of the Cromwell "documentation" was in the ReadMe on the Cromwell Github repo. We knew it, you knew it. The documentation needed a lot of work.

But not anymore!

Visit the new and much-improved Cromwell Documentation Site at Cromwell.readthedocs.io.

Take a look around and check out the new section of Cromwell Tutorials.

If you spot any errors or want to make any additions, the source of the documentation is in Github and pull requests are always welcome :wink: Finally if you have trouble finding anything use the Search in the top left-hand corner.

Thank you for your patience!

See comments (0)


Github “speaks” WDL

Posted by KateVoss on 17 Oct 2017 (0)


WDL is now an official GitHub language! This will make developing WDL even easier as you can read, write, and review pipelines on Github with beautiful syntax highlighting.

Check it out in a Broad Production pipeline WDL:

We also have syntax highlighting for authoring your WDLs in Sublime and vim, and full IDE support in all JetBrains IDEs (such as IntelliJ, pyCharm, CLion, etc).

See comments (0)



TL;DR: We are handing over governance of WDL to an open community-based organization called OpenWDL that will ensure WDL development is driven by and for the research community.

When WDL was created a few years ago, our mission was ostensibly to produce a pipelining system for the Broad's internal needs. But we knew at the time it would have to be much more than just a Broad-specific tool, because we had already decided we would share our analysis pipelines as widely as possible. The most immediate consequence was that whatever pipelining solution we adopted needed to be portable across not only our method development and production environments, but also across a variety of systems used in the research community. The other was that the scripting language we used needed to be as user-friendly as possible, to be at the very least readable by e.g. biomedical scientists without formal computational training (which happens to be my own background, so I have an especially soft spot for that constituency). Those were huge factors that led to our decision to create a new engine (Cromwell) and domain language (WDL itself, obviously ;)) rather than adopt an existing system. It also motivated us to provide as much help (via the forum) and documentation as we could manage to anyone who expressed interest, whether they were Broadies or not.

Fast-forward to the present day, where we're starting to see individuals and organizations throughout the world adopt WDL (sometimes independently of Cromwell) for their own pipelining needs. This is a incredibly gratifying but also rather daunting, because that means we now bear some degree of responsibility in ensuring that the people who adopt WDL are empowered to be successful in their pipelining endeavors.


Read the whole post
See comments (0)



Heads up! We're participating in an event organized by DNAstack in the context of the GA4GH meetings, just prior to the ASHG meeting in Orlando, FL in a few weeks. It's a two-hour evening event focused on open reproducible workflows, and will feature hands-on demos of WDL and Cromwell, as well as Dockstore and DNAstack's own platform, which uses WDL. The WDL and Cromwell section will be presented by our very own Jeff Gentry, who leads the Cromwell development team, accompanied by Cromwell product owner Kate Voss and GATK's resident scientific writer, Soo Hee Lee. We'll be running some GATK4 pipelines and generally having a good time nerding out about easy breezy reproducible workflows.

So join us if you're in town; register here now, as space is limited.

See comments (0)



29 Release Notes

Breaking Changes

  • Command line
    In preparation for supporting CWL scripts (yes, you read that right!), we have extensively revised the Command Line in Cromwell 29. For more details about the usage changes please see the README. And stay tuned to the WDL/Cromwell blog over the next couple of months for more news about CWL.

  • Request timeouts
    Cromwell now returns more specific 503 Service Unavailable error codes on request timeouts, rather than the more generic 500 Internal Server Error. The response for a request timeout will now be plain text, rather than a JSON format.

  • Metadata endpoint
    The response from the metadata endpoint can be quite large depending on your workflow. You can now opt-in to have Cromwell gzip your metadata file, in order to reduce file size, by sending the Accept-Encoding: gzip header. The default behavior now does not gzip encode responses.

  • Engine endpoints
    Previously the engine endpoints were available under /api/engine but now the endpoints are under /engine so they don't require authentication. Workflow endpoints are still available under /api/workflows. We also deprecated the setting api.routeUnwrapped as a part of this internal consistency effort.

  • Call caching diff
    We updated the response format of the callcaching/diff endpoint.

Other changes

  • Cromwell server
    When running in server mode, Cromwell now attempts to gracefully shutdown after receiving a SIGINT (Ctrl-C) or SIGTERM (kill) signal. This means that Cromwell waits for all pending database writes before exiting, as long as you include application.conf at the top of your config file. You can find detailed information about how to configure this feature in the Cromwell Wiki.

  • Concurrent jobs
    You can now limit the number of concurrent jobs for any backend. Previously this was only possible in some backend implementations. Please see the README for details.

WDL

  • Optional WDL variables
    Empty optional WDL values are now rendered as the null JSON value instead of the JSON string "null" in the metadata and output endpoints. You do not need to migrate previous workflows. Workflows run on Cromwell 28 and prior will still render empty values as "null".

  • Empty WDL variables
    Cromwell now accepts null JSON values in the input file and coerces them as an empty WDL value. WDL variables must be declared optional in order to be supplied with a null JSON value.

input.json

{
    "null_input_values.maybeString": null,
    "null_input_values.arrayOfMaybeInts": [1, 2, null, 4]
}

workflow.wdl

workflow null_input_values {
    String? maybeString
    Array[Int?] arrayOfMaybeInts
}
See comments (0)



They have a DOI and everything! Kate Voss presented this poster at the Bioinformatics Open-Source Conference in Prague last month. See the abstract below. The full citation is:

Voss K, Gentry J and Van der Auwera G. Full-stack genomics pipelining with GATK4 + WDL + Cromwell [version 1; not peer reviewed]. F1000Research 2017, 6(ISCB Comm J):1379 (poster) (doi: 10.7490/f1000research.1114631.1)

In case you're wondering, the flying pig with warp nacelles is Jamie, the mascot of the Cromwell execution engine. And the story behind that... is a story for another time :)


Read the whole post
See comments (0)



Heads up! The upcoming version 29 of Cromwell will include a breaking change in the command line syntax. Full details will be provided in the release notes when Cromwell 29 is ready for primetime; in the meantime the gist is that we're moving away from positional inputs in favor of explicit arguments. So for example this simple execution command:

java -jar Cromwell.jar run myWorkflow.wdl myWorkflow_inputs.json

will change to the more explicit:

java -jar Cromwell.jar run myWorkflow.wdl --inputs myWorkflow_inputs.json

The code changes can be viewed here for those who are into that level of detail.

We plan to update usage examples and relevant instructions in the WDL documentation materials at the time the new version is released, and will add a note regarding versions where appropriate for anyone still using a version anterior to 29. Note that there may be a short lag between the release and the documentation updates; we'll try to keep it to a minimum.

See comments (0)



Latest posts
 

- Recent posts


- Upcoming events

See Events calendar for full list and dates


- Recent events

See Events calendar for full list and dates



- Follow us on Twitter

WDL Dev Team

@WDL_dev

@jdidion @commonwl Or #OpenWDL — the language is BSD-3 and independent of the engine. And the major engine, Cromwel… https://t.co/nJrOnuhCF1
10 Jul 18
RT @geoffjentry: Was great to talk to everyone at the #OpenWDL BoF today at #GCCBOSC https://t.co/bj0l0emMeI
27 Jun 18
RT @morgantaschuk: @boconnor @OpenApiSpec .@boconnor Analyze, "green box" for @humancellatlas, WDL-based workflows, contribute pipelines, u…
27 Jun 18
RT @stevetsa1: @boconnor describes the Green Box, the analysis component of the Human Cell Atlas Data Coordination Platform using #WDL_dev…
27 Jun 18
RT @geoffjentry: Reminder for #GCCBSOC attendees, there's a BoF for #OpenWDL at 12:30 in PAB 332.
27 Jun 18

- Our favorite tweets from others

Reminder for #GCCBSOC attendees, there's a BoF for #OpenWDL at 12:30 in PAB 332.
27 Jun 18
#usegalaxy, #WDL, #CommonWL, #Snakemake and finally I'm at the session of #nextflow here in #GCCBOSC! Workflow lang… https://t.co/C9EVPJQzdH
26 Jun 18
Saw people poking #commonwl at the #OpenWDL workshop and people poking #OpenWDL at the #comonwl workshop, so I supp… https://t.co/kRDat8qtMC
26 Jun 18
The longest output for "hello world" tutorial I got ever :) #gccbosc #wdl
26 Jun 18
.@chapmanb To do better at sharing workflows, we need better abstractions. More interoperability. #CWL, @WDL_dev,… https://t.co/GzgUrfNrbP
26 Jun 18

See more of our favorite tweets...