WDL feature request : autosize drive
open | Created 2017-05-13 | Last updated 2018-01-18| Posted by katevoss | See in Github


Low User Requested Improvement


WDL tasks localize during JES runs using WDL like so:

task myTask {
File myFileToLocalize
Array[File] otherFilesToLocalize

...
command <<<
#process ${myFileToLocalize} ....

#process ${otherFilesToLocalize sep="somesep"} ...

>>>
...


also, WDL tasks can specify disk size for attached disk to help ensure that there is sufficient disk-space for files like so (in the example a user-configurable setting):

task diskTask {

	Int diskGB

.....

runtime {
disks: "local-disk ${diskGB} HDD"

...

It would seem like a good idea to have a special variable to automatic allocation of disk size of localized files. For example instead of the call being like

call diskTask {
input:
...
    diskGB=100
}

or

call diskTask {
input:
...
    diskGB=prepTask.calcDisk
}

or

call diskTask {
input:
...
    diskGB=size(someInputFile)
}

that maybe the variable could be like :

call bigTask {
inputs:
fileOne=someFileOfUnknownSize,
fileTwo=anotherFileOfUnknownSize,
diskGB=autoSize()+autoSize()*0.2
}

where "autoSize" automatically calculates sizes for files localized and adds them and I did some "+*" to add some factor from that (to have space for output files from the run) (in this case 20% of the disk for the output)

I wonder about people's thoughts on this?

This Issue was generated from your [forums]
[forums]: http://gatkforums.broadinstitute.org/firecloud/discussion/9346/idea-for-enhancment-autosize/p1


@lbergelson commented on Wed Apr 05 2017

👍 It would be nice to be able to specify memory requirements as a function of input filesize as well...

It would be nice to have fine grained access to each input size as well as just the total.


Return to top