Sync data between buckets with rclone

rclone copies and synchronizes data between two S3 buckets, even when they are on different sites or with different providers. Use it when you move to FPT Object Storage from another provider, or run two S3 environments in parallel.

tip

To move data from another provider into FPT Object Storage without installing any tool, use a migrate job on the Portal. See Migrate bucket data. For s3sync, see Connect with S3 clients.

The following example copies the data of bucket-site01 on site 1 to bucket-site02 on site 2, on Linux.

Prerequisites

• An Access Key, Secret Key, and endpoint for each site.
• An empty bucket on site 2 to receive the data.

Configure rclone

1. Install rclone:

   sudo apt-get install rclone

2. Create the file ~/.config/rclone/rclone.conf with the details of both sites. Set provider to match each S3 environment, for example AWS for Amazon S3.

   [site01]
   type = s3
   provider = Other
   env_auth = false
   access_key_id = YOUR_SITE01_ACCESS_KEY
   secret_access_key = YOUR_SITE01_SECRET_KEY
   endpoint = https://s3-hcm04.fptcloud.com/
   acl = private
   
   [site02]
   type = s3
   provider = Other
   env_auth = false
   access_key_id = YOUR_SITE02_ACCESS_KEY
   secret_access_key = YOUR_SITE02_SECRET_KEY
   endpoint = https://s3-hfx03.fptcloud.com/
   acl = private

3. Check that rclone can see the buckets and data on both sites:

   rclone lsd site01:
   rclone lsd site02:
   rclone ls site01:bucket-site01

Copy the data

1. Run a dry run to check for errors without copying anything:

   rclone sync -Pv site01:bucket-site01 site02:bucket-site02 --dry-run

2. If the dry run shows no errors, run the copy:

   rclone sync -Pv site01:bucket-site01 site02:bucket-site02

3. Check the buckets and data on both sites again with rclone lsd and rclone ls.

warning

rclone sync makes the destination identical to the source, so it deletes files on site 2 that do not exist on site 1. To copy without deleting, use rclone copy instead.