In the near future, we will update a number of the libraries and packages that are preloaded on notebook clusters on both FireCloud and Terra (beta). The updates are listed below. We expect all of these updates to be backwards compatible, with the exception of the upgrade from Hail 0.1 to 0.2 which is a breaking change. However it is possible that even the minor version updates could break dependencies in some of your notebooks, so we have developed a rollout plan to minimize disruption as follows.

• For FireCloud, we are taking a conservative approach and soliciting feedback from all users before scheduling the update. Please comment below if you think any of the updates listed below will be disruptive for your work. These updates only affect the creation of new clusters; existing clusters that are paused will not be affected by this change.
  
• For Terra (new beta UI for FireCloud), the updates listed below will be released next week (planned for Dec 4). Because these updates will affect all clusters on Terra, we recommend you save a version of your current notebook and results before that date. After the release, re-run your analysis results using the using the old vs new packages and compare results against your saved version. Let us know if you find any problematic discrepancies that you need help with.
  
• In the future, we plan to make it possible to enable access to prior versions of the notebook kernels to ensure long-term reproducibility and avoid this becoming a recurring issue when we roll out kernel upgrades.

The updates Version changes for libraries and packages:

• hail 0.1 -> 0.2
• python 3.4 -> 3.6
• R 3.4 -> 3.5
• pandas 0.22.0 -> 0.23.4
• numpy 1.15.0 -> 1.15.2
• google-api-core 1.3.0 -> 1.5.0
• google-cloud-bigquery 1.4.0 -> 1.6.0
• bokeh 1.0.0 -> 1.0.1

Hail 0.2 has a new file format and cannot read Hail 0.1. You can learn more about Hail 0.1 -> 0.2 changes here.

If you have any concerns, please comment below as soon as possible. Our goal is to equip you with the tools you need to get your work done, so the more we know, the better we can serve you.

Thank you for your patience as we work to improve the platform!

Upcoming FireCloud API Changes

We are planning to make a few changes to our APIs, which are outlined below. If you do not use the FireCloud APIs, or call only the Swagger UI (api.firecloud.org) your work will not be affected and you do not need to read any further. If you do use the APIs, please keep reading, as these changes may break some of your API calls if you do not modify them accordingly.

Here’s the detail:

• What: We are making changes to these APIs. The list describes the API changes and what modifications you will need to make to your own code if you are currently calling these APIs. If you do not make the relevant modifications to your API calls, they will fail once the changes are in place. You can make the changes in advance.
• When: The changes will go into effect on January 3rd, 2019
• Who: Only users who call the above APIs are affected; the changes will have no other impact on your work or overall experience.
• Why: We have been working on a new identity access management (IAM) service that allows for increased flexibility in permissions and sharing across all of our back-end services as well as front-end applications. We need to make the API changes listed above in order to roll out the new service.

We are happy to schedule separate conversations with you if you would like further detail or assistance to determine what modifications make sense for your specific project. Please email irosenbe@broadinstitute.org.

November 14, 2018

1. The FireCloud UI now has links in a few additional places for improved usability:
   • when viewing the list of method configurations in a workspace, the name of the referenced method is now a link.
   • when viewing a single method configuration in a workspace, the snapshot id now links to the referenced method.
   • when viewing a submission, the method configuration name is now a link.
2. The API to get call-level metadata for a workflow now supports query parameters to control which fields should be returned in the response. This is particularly useful for querying specific data about a workflow where the full response is otherwise very large. Cromwell has offered this functionality for multiple releases but it has not been exposed via the FireCloud API. See the Swagger documentation for specifics.
3. The login process now eliminates an unnecessary Ajax request for the Terms of Service content allowing for a slight incremental performance improvement during login.

October 23, 2018

• Workflow submissions to Cromwell now use the user's pet SA.
• Added the DCF provider on users' Profile page in addition to the DCP provider.
• The setCookie endpoint now returns 204 on success instead of 200.
• Increases the dataproc init script timeout from 10 to 30 minutes to support long-running user scripts. The value is now configurable in Leonardo.
• The Data tab of workspaces is now resilient to unknown entity attribute names listed in a workspace-column-defaults workspace attribute.

By Robert Title, Engineering Manager, Data Sciences Platform at the Broad Institute

We are excited to announce that we have substantially improved the way you interact with Jupyter Notebooks in FireCloud. We hope this will increase your productivity and empower you to collaborate more effectively. These changes are publicly available as of today, Sep 12, 2018, and can be accessed in the Notebooks tab of your FireCloud workspace. Read on for a more detailed description of what is changing, and why it is better.

September 25, 2018

• Removed an infinite spinner triggered by attempting to preview a DOS object, if that DOS object resolved to a text file or log file.
• Fixed an error caused by attempting to populate a method configuration with a json file larger than 4kb.
• Attempting to download a file larger than 2GB with your browser through the FireCloud UI would fail. These downloads now work, though we recommend using gsutil for large file download instead of your browser.
• Cluster creation errors no longer cause clusters to appear to get stuck in Creating status in the UI
• The cluster creation dialog now includes static text informing users of the expected time to create a cluster
• Users should no longer see 404 errors when opening a notebook from FireCloud
• FireCloud now displays appropriate error messages when a user does not have permission to modify notebooks
• Enable auto pause of clusters after 30 minutes of idle time
• Leonardo now better handles situations when clusters or projects are deleted in Google
• Leonardo is now more resilient to timeouts from GCR when pulling the notebook image

August 21, 2018

• Improved performance and stability in uncommon cases when reading from the entity data model, if the data model has extremely wide or long attributes.
• Improved the UX for the NPS survey for new users. Users who have already responded to the survey will not see the survey again.
• Squashed a rarely-occurring bug that caused errors when viewing individual workflows, if those workflows had no immediate calls/tasks
• Added the ability to set a google client ID at cluster creation time. If provided, it allows notebook auth refresh to kick off without cross-tab communication to the notebook.
• Added a v2 version of the createCluster endpoint, documented here: https://notebooks.firecloud.org/#!/cluster/createClusterV2. The new endpoint is faster than the previous version, and not subject to occasional race conditions when creating clusters of the same name in quick succession.

By Moran Cabili, product manager, Data Sciences Platform at the Broad Institute

We heard from many of you --both new FireCloud users and experienced WDL pipeline developers-- that you need to be able to quickly test that a WDL workflow can be run successfully on FireCloud. Until now, you were required to reference an entity in the workspace data model, which took extra effort and tended to confuse newcomers. We are happy to announce that this speed bump has been eliminated; you can now bypass the data model and even upload a JSON of inputs to get your WDL up and running in record time.

Today we bring to you a new facet of our forum: the feature request section!

Within this new section, you will be able to suggest new features, upvote existing ones, and see the status of features we implement. We can better gauge the number of people interested in a particular feature by the vote count, which helps to determine which features we work on first. So, if you represent a group of people who all care about a certain feature, ask everyone to vote on your feature request!

Features can be requested in this category of the forum. Simply click the blue New Idea button to start your thread. Be as clear as possible in your title so other users will be able to see what your feature is about and will be more likely to vote on it. If you see a feature you like, please upvote it so we know you'd like to see it implemented.

We've pre-populated this section with all the feature requests that have been posted in the last two months. You can search for your own feature requests by looking at discussions you've started. If we missed yours, or if there's a feature from further back than May, you can ask us to move it by tagging @KateN in the thread. Or, simply create a new request.

Take a peek at what's been posted and vote for what you want to see!

July 31, 2018

• FireCloud has upgraded to Cromwell 34. See release notes for versions 33 and 34 for new features available to you here.
  • WDL 1.0 support, requester pays, and PAPI v2 are not yet available in FireCloud. Work is underway to implement these specific components.
• Notebooks clusters should no longer auto-pause when a Jupyter notebook or a terminal is open in a user's browser.
• Leo now contains an optional user script that can be used to install GATK on clusters.
• The notebooks docker image now includes the tidyverse R package by default.
• Leo now contains an optional user script to allow downloading notebooks as PDF files. Note this significantly increases the size of the notebooks image which leads to slower cluster creation time, so use with caution. A follow-on change will be made to disallow downloading as PDF if this script is not used.
• FireCloud no longer tracks user behaviors in Google Analytics.
• Minor performance improvement via reducing unnecessary ajax requests when switching tabs inside a workspace.
• Performance improvement via caching of WDL validation. Users may notice improved performance when viewing a method configuration inside a workspace.