How to use rclone with DreamObjects

Rclone is an rsync-like tool for Windows, Mac OS Linux and other operating systems designed for cloud storage such as DreamObjects.

It features:

  • MD5/SHA1 hashes checked at all times for file integrity
  • Timestamps preserved on files
  • Partial syncs supported on a whole file basis
  • Copy mode to just copy new/changed files
  • Sync (one way) mode to make a directory identical
  • Check mode to check for file hash equality
  • Can sync to and from network, e.g., two different cloud accounts
  • Optional encryption (Crypt)
  • Optional FUSE mount (rclone mount)

Configure rclone

  1. After you download and install rclone on your operating system, proceed to configure it for DreamObjects using the interactive configuration tool:
    [server]$ rclone config
    No remotes found - make a new one
    n) New remote
    s) Set configuration password
    q) Quit config
    n/s/q> n
  2. Enter n. 

    name > remote
  3. Pick a name you want to assign to this connection, for example remote

    Type of storage to configure.
    Choose a number from below, or type in your own value
     1 / Amazon Drive
       \ "amazon cloud drive"
     2 / Amazon S3 (also Dreamhost, Ceph, Minio)
       \ "s3"
     3 / Backblaze B2
       \ "b2"
     4 / Dropbox
       \ "dropbox"
     5 / Encrypt/Decrypt a remote
       \ "crypt"
     6 / Google Cloud Storage (this is not Google Drive)
       \ "google cloud storage"
     7 / Google Drive
       \ "drive"
     8 / Hubic
       \ "hubic"
     9 / Local Disk
       \ "local"
    10 / Microsoft OneDrive
       \ "onedrive"
    11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
       \ "swift"
    12 / Yandex Disk
       \ "yandex"
    Storage> 2
    
  4. Next configure the type of storage as Amazon S3, option 2
    Get AWS credentials from runtime (environment variables or EC2 meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
    Choose a number from below, or type in your own value
     1 / Enter AWS credentials in the next step
       \ "false"
     2 / Get AWS credentials from the environment (env vars or IAM)
       \ "true"
    env_auth> 1
    
    AWS Access Key ID - leave blank for anonymous access or runtime
    credentials.
    access_key_id> your_access_key
    AWS Secret Access Key (password) - leave blank for anonymous access or
    runtime credentials.
    secret_access_key> your_secret_key
    
  5. Get ready to insert the Access Key and the corresponding Secret Key you can take from the DreamObjects Control Panel

    Region to connect to.
    Choose a number from below, or type in your own value
       / The default endpoint - a good choice if you are unsure.
     1 | US Region, Northern Virginia or Pacific Northwest.
       | Leave location constraint empty.
       \ "us-east-1"
       / US West (Oregon) Region
     2 | Needs location constraint us-west-2.
       \ "us-west-2"
       / US West (Northern California) Region
     3 | Needs location constraint us-west-1.
       \ "us-west-1"
       / EU (Ireland) Region Region
     4 | Needs location constraint EU or eu-west-1.
       \ "eu-west-1"
       / EU (Frankfurt) Region
     5 | Needs location constraint eu-central-1.
       \ "eu-central-1"
       / Asia Pacific (Singapore) Region
     6 | Needs location constraint ap-southeast-1.
       \ "ap-southeast-1"
       / Asia Pacific (Sydney) Region
     7 | Needs location constraint ap-southeast-2.
       \ "ap-southeast-2"
       / Asia Pacific (Tokyo) Region
     8 | Needs location constraint ap-northeast-1.
       \ "ap-northeast-1"
       / Asia Pacific (Seoul)
     9 | Needs location constraint ap-northeast-2.
       \ "ap-northeast-2"
       / Asia Pacific (Mumbai)
    10 | Needs location constraint ap-south-1.
       \ "ap-south-1"
       / South America (Sao Paulo) Region
    11 | Needs location constraint sa-east-1.
       \ "sa-east-1"
       / If using an S3 clone that only understands v2 signatures
    12 | eg Ceph/Dreamhost
       | set this and make sure you set the endpoint.
       \ "other-v2-signature"
       / If using an S3 clone that understands v4 signatures set this
    13 | and make sure you set the endpoint.
       \ "other-v4-signature"
    region> 12
    
  6. Pick option 12 and specify API access point: 

    Endpoint for S3 API.
    Leave blank if using AWS to use the default endpoint for the region.
    Specify if using an S3 clone such as Ceph.
    endpoint> objects-us-west-1.dream.io
    
  7. Next you need to set DreamObject’s endpoint API: 
    Location constraint - must be set to match the Region. Used when
    creating buckets only.
    Choose a number from below, or type in your own value
     1 / Empty for US Region, Northern Virginia or Pacific Northwest.
       \ ""
     2 / US West (Oregon) Region.
       \ "us-west-2"
     3 / US West (Northern California) Region.
       \ "us-west-1"
     4 / EU (Ireland) Region.
       \ "eu-west-1"
     5 / EU Region.
       \ "EU"
     6 / Asia Pacific (Singapore) Region.
       \ "ap-southeast-1"
     7 / Asia Pacific (Sydney) Region.
       \ "ap-southeast-2"
     8 / Asia Pacific (Tokyo) Region.
       \ "ap-northeast-1"
     9 / Asia Pacific (Seoul)
       \ "ap-northeast-2"
    10 / Asia Pacific (Mumbai)
       \ "ap-south-1"
    11 / South America (Sao Paulo) Region.
       \ "sa-east-1"
    location_constraint> 1
    
  8. Leave the location constraint empty:
    >Canned ACL used when creating buckets and/or storing objects in S3.
    For more info visit
    http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
    Choose a number from below, or type in your own value
     1 / Owner gets FULL_CONTROL. No one else has access rights (default).
       \ "private"
     2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access.
       \ "public-read"
       / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.
     3 | Granting this on a bucket is generally not recommended.
       \ "public-read-write"
     4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access.
       \ "authenticated-read"
       / Object owner gets FULL_CONTROL. Bucket owner gets READ access.
     5 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
       \ "bucket-owner-read"
       / Both the object owner and the bucket owner get FULL_CONTROL over the object.
     6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
       \ "bucket-owner-full-control"
    acl> private
    
  9. Set the canned ACL based on how you want to use rclone:
    The server-side encryption algorithm used when storing this object in S3.
    Choose a number from below, or type in your own value
     1 / None
       \ ""
     2 / AES256
       \ "AES256"
    server_side_encryption> 1
    
  10. Pick 1 for server-side encryption option (DreamObjects doesn’t support it at the moment):
    The storage class to use when storing objects in S3.
    Choose a number from below, or type in your own value
     1 / Default
       \ ""
     2 / Standard storage class
       \ "STANDARD"
     3 / Reduced redundancy storage class
       \ "REDUCED_REDUNDANCY"
     4 / Standard Infrequent Access storage class
       \ "STANDARD_IA"
    storage_class>
    
  11. Leave the storage class empty:
    Remote config
    --------------------
    [remote]
    env_auth = false
    access_key_id = your_access_key
    secret_access_key = your_secret_key
    region = other-v2-signature
    endpoint = objects-us-west-1.dream.io
    location_constraint =
    acl = private
    server_side_encryption =
    --------------------
    y) Yes this is OK
    e) Edit this remote
    d) Delete this remote
    y/e/d> y
    
    Current remotes:
    
    Name                 Type
    ====                 ====
    remote               s3
    
    e) Edit existing remote
    n) New remote
    d) Delete remote
    s) Set configuration password
    q) Quit config
    e/n/d/s/q> q
    
  12. Finally, review the remote just configured, save it, and exit the configuration wizard:

Using rclone

With the remote set, you can list the buckets in it with the command:

[user@localhost]$ rclone lsd remote:
          -1 2016-03-04 02:19:25        -1 samplebucket
          -1 2016-05-16 22:06:53        -1 anotherbucket
          -1 2015-10-15 21:33:25        -1 greatbucket
2016/09/01 09:35:41
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:       100ms

Make a new bucket:

[user@localhost]$ rclone mkdir dho:bucket

List the contents of a bucket:

[user@localhost]$ rclone ls dho:bucket

Sync /home/local/directory to the remote bucket, deleting any excess files in the bucket:

[user@localhost]$ rclone sync /home/local/directory remote:bucket

Check rclone’s official documentation for more examples on how to use the software.

Did this article answer your questions?

Article last updated .