Can anyone explain how you cope with the 30s shutdown notice? Do you save your network every few iterations to your SSD or is it feasible to save the whole 16 GB from the P100 within the 30s window? I mean, we can freeze a running operating system (from RAM to disk), can do the same from GPU memory?
See pre-emptible as something that should only run jobs that can be re-schedule. You don't save what you're doing; you close all that can be closed and mark all tasks that have to be re-done. If you are 3 minutes in a task that takes 5 minutes, then stop it and put it back on your global queue; The gain of running on preempt will compensate for the loss of rerunning a few tasks twice.
