FireCloud is now powered by Terra! -- STARTING MAY 1st, 2019 THIS WEBPAGE WILL NO LONGER BE UPDATED.
From now on, please visit the Terra Help Center for documentation, tutorials, roadmap and feature announcements.
Want to talk to a human? Email the helpdesk, post feature requests or chat with peers in the community forum.
FIRECLOUD | Doc #12640 | Allow determining gsutil cp order using a parameter tag

Allow determining gsutil cp order using a parameter tag
Feature Requests | Created 2018-08-10

Comments (0)

When working with indexed files, it is quite common to have the program reading an index perform a quick check to make sure the index is newer than the file it indexes.

FireCloud uses gsutil cp to copy files back to the data store, but the order in which files are copied is not defined. Because gsutil writes the metadata about file creation time based on when the gsutil cp finishes at the destination, it's pretty common for me to end up with indexes that appear older than their files.

It would be nice to have a queuing or management system to determine the order output files get copied back. I'm sure this prevents the use of the -m gsutil flag for parallel copy, but I for one would be willing to pay the additional time penalty.

I can imagine this as a metadata parameter, or perhaps an outputs attribute (e.g. an ordered list: order : ["${output}", "${outputIndex}"]). It might also make sense to simply copy them back in the order they're created.

Return to top Comment on this article