FireCloud has so far focused on making it easy to share data and execute pipelines, for all your batch-style --one might even say boring-- data processing and analysis needs. That covers a lot of territory; certainly most of the upstream work that is done under the umbrella of "genomic analysis". But for many of you, the end of the pipeline is just the start of the really interesting part; and whether you're moving on to GWAS or something else, the point is that you're moving on -- specifically, to a phase of analysis where you need to be able to interrogate your data interactively. And until now, that often meant downloading your pipeline's outputs and getting back to traditional, on-premises computational work and its limitations.
So today, we're excited to release a beta preview of a new FireCloud feature called Notebooks that makes it possible to run interactive analyses on your data in the cloud, with the convenience of a Jupyter notebook environment.
We are planning to release an important update that will change how the system identifies user access permissions to Google buckets in order to support a technique that renders certain workflows substantially more cost-effective.
This update is scheduled for next Tuesday, February 6 around 4:00 pm EST. At that time, all running workflows will fail and will be automatically restarted using call-caching once the update is complete. We recommend you hold off on launching any long-running workflows until after the release if you want to avoid having your workflow stopped and restarted.
This update will break some functionality if you have data stored in private buckets that are NOT FireCloud workspace buckets. It also deprecates the use of private docker images in Docker Hub, which remain functional for now but will be disabled in 30+ days (exact date will be communicated separately). If you think either applies to you (or might apply to you in the future), please read on to learn more about what is going to change, why we are making this change and what you need to do to resolve any problems that arise for you.
If you're in a hurry, feel free to zip straight through to the last section, but I think you'll find the instructions will make more sense if you read the full explanation.
The National Cancer Institute (NCI) funded the beginning development of FireCloud and two other genetic analysis cloud platforms. NCI started a pilot program that offered free credits ranging from $300, $1000, and $10,000 to give researchers a risk-free trial to compute on these platforms.
The NCI’s Cloud credit pilot was highly popular with around 164 researchers issued FireCloud credit and has been a great vehicle for getting users into the system. As the pilot comes to an end on September 27th, we are trying to make it easy for you to continue using the system and save your data. We have included instructions and individual contacts who can help you through the transition.