I’m making this film, as many of you know. An animated cartoon about tentacle space-monsters and the end of the world. It’s taken a long time, and now needs to be polished and re-rendered in high-quality. Which is also going to take a long time.
A really really long time. Like, ten months, on my machine. Running flat out.
When I talk about this with people, and explain how I intend to buy computing power in the cloud on the open markets in order to get it done in a more reasonable amount of time, they mostly seem surprised that such a market even exists. That you can buy computing time by the hour on a machine half way around the world.
It is pretty damned cool.
Anyway, I’ve been running ten machines with around 80 gig of memory and 20 cpu cores with 40 compute units for a few days now.
At first I was foolish enough to just order a couple of on-demand machines, until I figured out what kinda price that would come to for these ten-months or so of computer power. That’d be more than two grand at the on-demand price.
Fuck.
Luckily, it seems that the on-demand price is around ten times more than the “spot price”.
Now Amazon have litterally thousands and thousands of computers ready for people to come and rent any time they want. In order to make sure that there’s always one there when someone wants it, they have to keep a whole bunch of spare capacity ready. And letting the machines sit there idle is a waste, so they flog off the spare machines every hour to the highest bidder during that hour.
You have to be prepared for your machine to be turned off and given to someone else in the middle of what you’re doing, but you generally get it for less than the on demand price. About ten times less, on average, it turns out.
That cut my potential bill here down to more like 200 dollars.
Or so I thought.
After running a setup at the spot-price for a few days I examined my bill so far. It’s about three times higher than I’d expected.
The reason, it turns out, is storage. Hard drive space. Each of those running machines uses a hundred gig hard-drive, and I’m paying twice as much again for that as I am for the cost of the machines themselves. About a third in the compute time, a third in hard-drive provision and a third in data-flow there and back.
Damn. Again.
So I look more closely at the market prices this evening, after I should have gone to bed.
I suspect that by renting machines with twice the RAM, I can cut down the amount of IO needed. The machines are using swap-space, coz swap-space always seems so cheap at home. But in the cloud there, when you have to pay for every byte you send to the hard-drive and back, swap space is *expensive*. Using twice the RAM in each machine should cut swap down to bascially nothing. That high-ram machine costs just a penny an hour or so more than the standard large machine. Where as the swap-space is doubling the processor bill.
Plus, of course, without having to swap out all the time the rendering actually runs faster, meaning it’ll all be done with less compute-time, which should reduce the cost some more.
But this data transfer is still killing me. The last eight hours, with ten machines in the new configuration, have cost about $6.50 and done 450 frames, so I’m STILL paying a US cent and a half for every one of the 30,000 images. Or 450 dollars for the entire thing. About 60% of that is just copying all the needed data onto the machines at the start of each reboot. I think I’m going to have to spend today working out a way to bring that down. Ideally these machines would only pull the smallest amount of data they need I guess rather than downloading all the source-files for every single operation. Some kinda NFS link back to the controller rather than each having an independent copy of the data I guess.
So there you go. These are the things you have to think about when your demands for computing power reach such high levels you find yourself rushing to the cloud to rent more.
vPre has to look forward to all this as his brain expands. And figuring out a way to pay for it after I’m gone.