S3cmd
s3cmd third party tool
Download
Usage
Here's a typical command I'd run
$ time /usr/bin/s3cmd -c /home/ec2-user/.s3cfg sync --no-check-md5 --delete-removed --no-preserve \\ --verbose --progress /source/ s3://bucket/target/ --dry-run
I'll attach a .s3cfg file (at bottom of page)
There are some things to switch off in it. Might help.
Python might be choking on checking md5 on source side if you have little ram and the files are huge.
Example:
$ date; time /usr/bin/s3cmd -c /home/ec2-user/.s3cfg sync --no-check-md5 --delete-removed --no-preserve --verbose --progress /mnt/tagleo/error_pages/ s3://www-cert.thinkcentral.com/error_pages/
"--no-check-md5" prevents md5 checks on initial upload for a speed bump, but you may not want that really
.s3cfg file
[default] access_key = ZZZZZZZZZZ access_token = add_encoding_exts = add_headers = bucket_location = US cache_file = cloudfront_host = cloudfront.amazonaws.com default_mime_type = binary/octet-stream delay_updates = False delete_after = False delete_after_fetch = False delete_removed = False dry_run = False enable_multipart = False encoding = UTF-8 encrypt = False expiry_date = expiry_days = expiry_prefix = follow_symlinks = False force = False get_continue = False gpg_command = /usr/bin/gpg gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_passphrase = guess_mime_type = True host_base = s3.amazonaws.com host_bucket = %(bucket)s.s3.amazonaws.com human_readable_sizes = False ignore_failed_copy = False invalidate_default_index_on_cf = False invalidate_default_index_root_on_cf = True invalidate_on_cf = False list_md5 = True log_target_prefix = max_delete = -1 mime_type = multipart_chunk_size_mb = 2000 preserve_attrs = True progress_meter = True proxy_host = proxy_port = 0 put_continue = False recursive = False recv_chunk = 4096 reduced_redundancy = False restore_days = 1 secret_key = xxxxxxxxxxxxx send_chunk = 4096 server_side_encryption = False simpledb_host = sdb.amazonaws.com skip_existing = False socket_timeout = 300 urlencoding_mode = normal use_https = False use_mime_magic = True verbosity = WARNING website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/ website_error = website_index = index.html