HN2new | past | comments | ask | show | jobs | submitlogin

For me the most invaluable command line tool that I have ever used is S3 Tools:

http://s3tools.org/s3tools

It provides a great suite of tools for interacting with S3, and is best used on an EC2 instance you are connected to via SSH. It is also ridiculously fast, much faster than trying to interact with S3 from a local FTP browser, or even from Amazon's own S3 dashboard. For example on my computer using Amazon's own web facing dashboard it will take about 30-45 minutes to make 14000 files on S3 public, versus via the command line tool downloaded and running on one of my EC2 instances it can make those files public within minutes.

I assume this is because it is local network traffic. Anyway, if you are ever in a bind and need to move a bunch of files to S3, I highly recommend S3 Tools. It has saved me many times.

Along those lines wget is the most powerful command line tool I've ever used, with so much capability. It is simply incredible when combined with S3 tools, allowing you to easily grab gigabytes of images off of a personal server or staging location and upload them to S3 very quickly.

And if you need to do more than move files around you can manage even more aspects of AWS including EC2 instances from the command line using this powerful command line tool:

http://www.timkay.com/aws/

Its great for writing automated bash scripts to manage your instances via cron jobs.



My main complaint with s3cmd is you can't easily throttle it, and on a busy machine its fast speeds can cause issues.


Have you tried trickle[1]? It can limit the bandwidth of most programs I've tried it on.

  trickle -u 10 -d 20 ncftp
This runs ncftp capped at 10 KB/s up, 20 KB/s down.

[1] http://monkey.org/~marius/pages/?page=trickle


No -- but I will -- thx! I've tried nice, ionice, cpulimit and cstream.


Good point. I guess in the past I have spun up a micro instance to run it, so I didn't have to run my S3 operations using one of my main production servers, and hence I didn't run into any problems with its unthrottled speed.


Perhaps lowering its priority would help.

nice s3cmd ...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: