DreamObjects cluster migration
DreamHost is currently in the process of migrating all DreamObjects accounts to a new hardware platform.
Please review this article to identify the cluster your DreamObjects account currently resides on and the features it supports.
Overview
Rclone is a rsync-like tool for Windows, Mac OS, Linux, and other operating systems designed for cloud storage, such as DreamObjects.
It features:
- MD5/SHA1 hashes checked at all times for file integrity
- Timestamps preserved on files
- Partial syncs supported on a whole file basis
- Copy mode to just copy new/changed files
- Sync (one way) mode to make a directory identical
- Check mode to check for file hash equality
- Can sync to and from network, e.g., two different cloud accounts
- Optional encryption (Crypt)
- Optional FUSE mount (rclone mount)
Configure rclone
- Download and install rclone based on your computer's operating system. You can download it from rclone's site here:
- After you download it, navigate to the directory to which the file was downloaded. (NOTE: In Windows you can navigate to the /Downloads directory and extract the rclone files.) Copy the directory in your Explorer bar.
- Open your command prompt and navigate to the directory to which the files were extracted.
- In your terminal or command prompt, run the following to begin configuring rclone using the interactive configuration tool:
[user@localhost]$ rclone config No remotes found - make a new one n) New remote s) Set configuration password q) Quit config n/s/q> n
-
Enter n to set up a new configuration profile (i.e. a remote).
name > dreamobjects -
Pick a name you want to assign to this connection. This example uses the name dreamobjects.
Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / 1Fichier
\ (fichier)
2 / Akamai NetStorage
\ (netstorage)
3 / Alias for an existing remote
\ (alias)
4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Magalu, Minio, Netease, Outscale, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, Selectel, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others
\ (s3)
5 / Backblaze B2
\ (b2)
[...] Storage> 4 - Select option 4 for Amazon S3.
Choose your S3 provider. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Amazon Web Services (AWS) S3
\ (AWS)
2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
\ (Alibaba)
3 / Arvan Cloud Object Storage (AOS)
\ (ArvanCloud)
4 / Ceph Object Storage
\ (Ceph)
5 / China Mobile Ecloud Elastic Object Storage (EOS)
\ (ChinaMobile)
6 / Cloudflare R2 Storage
\ (Cloudflare)
7 / DigitalOcean Spaces
\ (DigitalOcean)
8 / Dreamhost DreamObjects
\ (Dreamhost)
9 / Google Cloud Storage
\ (GCS)
[...] provider> 8 - Select option 8 for DreamHost.
Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank. Enter a boolean value (true or false). Press Enter for the default ("false"). Choose a number from below, or type in your own value 1 / Enter AWS credentials in the next step \ "false" 2 / Get AWS credentials from the environment (env vars or IAM) \ "true" env_auth> 1 - Select option 1 for Enter AWS credentials.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
access_key_id> YOUR_ACCESS_KEY
AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
secret_access_key> YOUR_SECRET_KEY -
Replace YOUR_ACCESS_KEY and YOUR_SECRET_KEY with your actual Access Key and its corresponding Secret Key. These can be located in the DreamObjects page.
Option region.
Region to connect to.
Leave blank if you are using an S3 clone and you don't have a region.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Use this if unsure. Will use v4 signatures and an empty region.
\ ""
2 / Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH.
\ "other-v2-signature"
region> us-east-005 - Enter your DreamObjects region us-east-005 and press Enter to continue.
Endpoint for S3 API.
Required when using an S3 clone.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Dream Objects endpoint
\ "objects-us-west-1.dream.io"
endpoint> s3.us-east-005.dream.io - Enter DreamObjects hostname s3.us-east-005.dream.io and press Enter to continue.
Location constraint - must be set to match the Region.
Leave blank if not sure. Used when creating buckets only.
Enter a string value. Press Enter for the default ("").
location_constraint> - Leave this option blank and press Enter to continue.
Canned ACL used when creating buckets and storing or copying objects. This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
If the acl is an empty string then no X-Amz-Acl: header is added and
the default (private) will be used.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
/ Owner gets FULL_CONTROL.
1 | No one else has access rights (default).
\ (private)
/ Owner gets FULL_CONTROL.
2 | The AllUsers group gets READ access.
\ (public-read)
/ Owner gets FULL_CONTROL.
[...] acl> private - Set the canned ACL based on how you want to use rclone. This example uses private.
Edit advanced config? (y/n) y) Yes n) No y/n> n - Choose if you want to edit the advanced config. This example chooses not to.
Configuration complete.
Options:
- type: s3
- provider: Dreamhost
- access_key_id: YOUR_ACCESS_KEY
- secret_access_key: YOUR_SECRET_KEY
- region: us-east-005
- endpoint: s3.us-east-005.dream.io
- acl: private
Keep this "dreamobjects" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y - Type in y.
Current remotes: Name Type ==== ==== dreamobjects s3 e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q> q - Review the remote you configured. If everything looks correct, save it by entering y and quit the configuration wizard by entering q.
Using rclone
View the following for a list of commands you can use with rclone:
With a remote configured, you can list the buckets in it with this command:
[user@localhost]$ rclone lsd dreamobjects: -1 2024-03-29 02:19:25 -1 samplebucket -1 2024-03-29 22:06:53 -1 anotherbucket -1 2024-03-29 21:33:25 -1 greatbucket
Make a new bucket:
[user@localhost]$ rclone mkdir dreamobjects:mynewbucket
Sync /home/localuser/directory to the remote bucket, deleting any excess files in the bucket:
[user@localhost]$ rclone sync /home/localuser/directory dreamobjects:mynewbucket
List the contents of a bucket:
[user@localhost]$ rclone ls dreamobjects:mynewbucket
For more examples on how to use the software, see rclone's official documentation here: