When it launched Amazon Glacier was applauded for providing a super-cheap long-term storage solution. While there are no surprises when uploading and storing files, retrieving them can get expensive. The pricing reflects the fact that Amazon needs to retrieve your files from tape, which is expensive and takes a long time. Several users reported high charges after retrieving their backups. To its defence, Amazon published a very detailed FAQ on this topic.

The key to getting your files back on the cheap is time. You can retrieve 5% of your total files for free each month, but that amount is calculated per hour, rather than per month or day. Example: You keep 500GB in Glacier. 500GB*5%/30/24=36MB/hour.

That’s great to know, but how can you keep retrieving 36MB for days or months without doing it manually? If you’re on OSX or Linux you can use mt-aws-glacier. It’s a Perl script to track and retrieve your Glacier files. Using mtglacier for slow retrieval isn’t straight forward. There is an issue to improve it, but for now let’s work with what’s available.

The tool has two commands to restore files. One initiates the job on Amazon’s side. After the job completes (takes about 4h), the files can be downloaded. With that in mind, we need to tell mtglacier to only request a few files and then sleep for another hour. There is no way to limit the file size to be requested, but the number of files can be limited. As long as you have many small files, this works great.

while true
do
  mtglacier restore --config vault.cfg --max-number-of-files 3
  mtglacier restore-completed --config vault.cfg
  sleep 3600
done

If your hourly allowance is 35MB and your average file size is 10MB, 3 files per hour is about right with some space for error.

Hope this helps.

Yahoo failed as internet company for a reason. Try sending an email with a link to a bank website. E.g. CIMB (Popular across Asia)

http://www.cimb-bizchannel.com.my/index.php?ch=srvpack

Your email will be rejected by Yahoo. Just awesome…

554 Message not allowed - [PH01] Email not accepted for policy reasons

Workaround: Use a shortlink to hide your URL. E.g. http://goo.gl/tPb19A. Now your phishing emails will arrive safely. 😉

At Quantego, we do most high-level work that supports energy analysts in Jupyter Notebooks. This allows us to pull several Java and Python packages together for a highly productive work environment.

Sample notebooks are hosted on Github and distributed with our Docker images. Of course we prefer for our sample notebooks to work, when people run them. They also uncover potential problems, by running at a very high level and thus using almost all available features.

If you have a similar setup – for example in a data analytics-driven environment – the following could work for you as well:

1. Make sure your notebooks run correctly when running “Run All”. When testing you may try different things and run cells out of order. For automatic testing to work, they should run all in sequence.
2. Test locally with

   jupyter nbconvert --to=html --ExecutePreprocessor.enabled=True my-notebook.ipynb

   This will convert your notebooks to HTML. We’re not intersted in the output, only potential errors during conversion. This only works with Jupyter and iPython >=4. Previous versions simply ignore errors.

3. Next you could just run the same command in an isolated Docker container or in a CI step

   docker run my_container /bin/sh -c \
     "/usr/local/bin/jupyter nbconvert \
         --to=html --ExecutePreprocessor.enabled=True \
         --ExecutePreprocessor.timeout=3600 \
         samples/my-sample.ipynb"

   A full working example for CircleCI can be found in our sample-repo.