boto is a Python library that makes it easier to work with services supporting the Amazon Web Services API, such as DreamObjects. Please note that using boto directly requires a little bit of Python-scripting knowledge.
This article describes a few different examples on how you can use boto with DreamHost. If you’re ready to dig into some boto code, the boto getting started guide is a great place to start. DreamHost’s servers already have boto installed so you can skip that part in the documentation.
Creating a .boto file to store your keys
Instead of entering your Access and Secret key into every script, you could just create a .boto file to store these credentials. View the following article for details:
The examples below do not use the .boto file, but you could adjust your scripts to use this file if you like.
Example Scripts
Here are some examples of scripts that can be useful for managing your DreamObjects service.
Checking the size of a bucket
#!/usr/bin/python
# -*- coding: utf-8 -*-
#Programmatically calculate the total size of your bucket, by iterating over all your objects
import boto
c = boto.connect_s3("ACCESS KEY", "SECRET KEY", host="objects-us-west-1.dream.io")
b = c.get_bucket("BUCKET NAME")
s = 0
for o in b.list():
s += o.size
#show you the file size in Megabytes if it's too small to be legible in Gigabytes.
print str(s * 1.0 / 1024 ** 2) + " MB\n"
#Shows you the file size in Gigabytes
print str(s * 1.0 / 1024 ** 3) + " GB"
Purging a directory from a bucket
Below is a script which removes content from a bucket. Proceed with caution as this will permanently delete your data.
The prefix in the script below can be set to “” to delete all data from a bucket, in the case you wish to remove a bucket that contains data.
#!/usr/bin/python
import boto
c = boto.connect_s3("ACCESS KEY", "SECRET KEY", host="objects-us-west-1.dream.io")
b = c.get_bucket("BUCKET NAME", validate=True)
l = [o for o in b.list(prefix="path/to/directory/to/delete")]
while len(l):
s = l[0:1000]
len(l)
rs = b.delete_keys(s)
if len(s) == len(rs.deleted):
l = l[1000:]
Change all object permissions in a bucket
This script changes all permissions in a bucket to either PRIVATE or PUBLIC. Uncomment one of the last two lines for the specific permission you wish to use.
#!/usr/bin/python
import boto
#Connect to S3
c = boto.connect_s3("ACCESS KEY", "SECRET KEY", host="objects-us-west-1.dream.io")
b = c.get_bucket("BUCKET NAME")
for o in b.list():
# o.set_acl('public-read')
# o.set_acl('private')
Uploading to a bucket in chunks
If the file you are attempting to upload is too large (over 5GB), it must be uploaded in “chunks”. You can use a client that supports multi-part uploads, or use the boto script below.
This script requires the “FileChunkIO” library. If you wish to use this on a DreamHost server, you will first need to reference the Python documentation setting up a virtualenv, activating it and then running the following commands:
[server]$ pip install FileChunkIO [server]$ pip install boto
If not on a DreamHost server and you have root access, using a virtualenv is optional.
#!/usr/bin/python
import math, os
import boto
from filechunkio import FileChunkIO
#Connect to S3
c = boto.connect_s3("ACCESS KEY", "SECRET KEY", host="objects-us-west-1.dream.io")
b = c.get_bucket("BUCKET NAME")
#file info
source_path ='PATH TO YOUR FILE'
source_size = os.stat(source_path).st_size
#Create a multipart upload request
mp = b.initiate_multipart_upload(os.path.basename(source_path))
#set a chunk size (feel free to change this)
chunk_size = 100000
chunk_count = int(math.ceil(source_size / float(chunk_size)))
#send the file parts using FileChunkIO to create a file-like object
#that points to a certain byte range within the original file. We set
#bytes to never exceed the original file size.
for i in range(chunk_count):
offset = chunk_size * i
bytes = min(chunk_size, source_size - offset)
with FileChunkIO(source_path, 'r', offset=offset,bytes=bytes) as fp:
mp.upload_part_from_file(fp,part_num=i +1)
#Finish the upload
mp.complete_upload()
Additional Examples
View the following article for a multi-part-upload example.
