rsync 3 TB of data to 2x 2TB big external drivesSync two external harddrives?Sync 2 hard drives without being...

Exponential growth/decay formula: what happened to the other constant of integration?

Non-Italian European mafias in USA?

Six real numbers so that product of any five is the sixth one

What is the difference between throw e and throw new Exception(e)?

Where was Karl Mordo in Infinity War?

"Murder!" The knight said

Can chords be played on the flute?

Equivalent to "source" in OpenBSD?

How to count occurrences of Friday 13th

How to properly claim credit for peer review?

Is there a low-level alternative to Animate Objects?

Casually inserting sexuality

Auto Insert date into Notepad

As a new poet, where can I find help from a professional to judge my work?

What if I store 10TB on azure servers and then keep the vm powered off?

Sometimes a banana is just a banana

Is there a German word for “analytics”?

Can you use a beast's innate abilities while polymorphed?

Pure Functions: Does "No Side Effects" Imply "Always Same Output, Given Same Input"?

When was drinking water recognized as crucial in marathon running?

How can atoms be electrically neutral when there is a difference in the positions of the charges?

How to avoid being sexist when trying to employ someone to function in a very sexist environment?

Second-rate spelling

It took me a lot of time to make this, pls like. (YouTube Comments #1)

rsync 3 TB of data to 2x 2TB big external drives

Sync two external harddrives?Sync 2 hard drives without being connected at the same time (Mac)Windows alternative for rsyncDoes rsync require 2 computers?rsync … blocked for more than 120 secondsCan't get rsync to sync two external harddrivesFastest way to duplicate encrypted hard drives?Limit rsync to available space on destinationsynchronizing distributed offline archives with large files on external drivesHow do I keep two external drives synced in two ways?

Is it possible to rsync my 3TB big hard drive to two external 2TB hard drives?

If so what would the command look like?

If not, can you think of another solution to sync 3TB of data to 2x 2TB drives?
The only way I can think of is to manually copy the files... but loosing all of rsync's power this way :(

edited Aug 8 '14 at 12:29

An Dorfer

1,2032713

asked Jan 6 '13 at 18:51

jrn

114110

add a comment |

Is it possible to rsync my 3TB big hard drive to two external 2TB hard drives?

If so what would the command look like?

If not, can you think of another solution to sync 3TB of data to 2x 2TB drives?
The only way I can think of is to manually copy the files... but loosing all of rsync's power this way :(

edited Aug 8 '14 at 12:29

An Dorfer

1,2032713

asked Jan 6 '13 at 18:51

jrn

114110

add a comment |

Is it possible to rsync my 3TB big hard drive to two external 2TB hard drives?

If so what would the command look like?

If not, can you think of another solution to sync 3TB of data to 2x 2TB drives?
The only way I can think of is to manually copy the files... but loosing all of rsync's power this way :(

edited Aug 8 '14 at 12:29

An Dorfer

1,2032713

asked Jan 6 '13 at 18:51

jrn

114110

Is it possible to rsync my 3TB big hard drive to two external 2TB hard drives?

If so what would the command look like?

If not, can you think of another solution to sync 3TB of data to 2x 2TB drives?
The only way I can think of is to manually copy the files... but loosing all of rsync's power this way :(

sync rsync

edited Aug 8 '14 at 12:29

An Dorfer

1,2032713

asked Jan 6 '13 at 18:51

jrn

114110

edited Aug 8 '14 at 12:29

An Dorfer

1,2032713

asked Jan 6 '13 at 18:51

jrn

114110

edited Aug 8 '14 at 12:29

An Dorfer

1,2032713

edited Aug 8 '14 at 12:29

An Dorfer

1,2032713

edited Aug 8 '14 at 12:29

An Dorfer

1,2032713

asked Jan 6 '13 at 18:51

jrn

114110

asked Jan 6 '13 at 18:51

jrn

114110

asked Jan 6 '13 at 18:51

jrn

114110

add a comment |

3 Answers
3

active

oldest

votes

Options include setting up the destination drives as a single volume (RAID, LVM, btrfs...), or running two rsync commands on subsets of the data on the 3TB drive.
The former option makes the job simple (there's only one target) but has the disadvantage that both drives must be connected to access the data.
The latter is a little more effort, since it is necessary to identify a set of folders that is less than or equal to 2TB in size, but allows access to the data on each target drive independently. The best way to construct the rsync commands, if the set of folders is more complex than a few on the root of the 3TB drive, is probably to make use of rsync's --exclude-from option.

answered Jan 6 '13 at 19:04

Darael

482210

Thanks, I think the option with having two separate rsync commands is the one I will be end up using. I'll make a text file excluding unwanted files/folders which will then be synced with the second rsync command. To use patterns in the text file would I be typing A* or 'A*'? Is this command case sensitive?

– jrn
Jan 6 '13 at 19:21

1

It's case sensitive, and you want to list files one per line. The same format can be used as in the du command I suggested above - to match everything starting with a through n, use /[a-nA-N]. To match only directories with that format, make it /[a-nA-N]/. The preceding / is to ensure it only matches at the root of the transfer, so that no subfolders are inadvertently excluded. No quotes are necessary in an exclude file (using --exclude-from), but single quotes are needed if using --exclude= to stop the shell from interpreting the pattern instead of rsync itself.

– Darael
Jan 6 '13 at 19:29

add a comment |

1) Simplest way:

If the 3TB of data is in directories less then 2TB, simply use two rsync commands.

E.g.



3TB  --- Folder1 with 1½TB --> rsync ---> 2TB drive

         Folder1 with 1½TB --> rsync ---> 2TB drive

2) Alternative:

Or mount both 2TB drives as a single volume. E.g. with linear mode or a stripe or storage spaces. (Linear mode would concatenate both drives in a form of RAID0. A stripe would alternate between writing to them.)

3) Alternative:

Or mount the second 2TB drive in a folder in a volume on the first drive.

answered Jan 6 '13 at 19:04

Hennes

59.2k793142

So for option 1) would it make sense to copy all folders starting with A* to N* to drive 1 and all folders O* to Z* to drive 2? I guess I'd be using exclude in this case.

– jrn
Jan 6 '13 at 19:13

If all folders are about equal in size: yes.

– Hennes
Jan 6 '13 at 19:15

I would suggest using du to check that the total size of folders a*-n* is <= 2TB - after all, they may be unbalanced. du -c [a-nA-N]* will do this (reports the total size of all files and folders in the current directory with names starting with a to n or A to N - for some other folder, use /path/to/folder/[a-nA-N]*).

– Darael
Jan 6 '13 at 19:20

add a comment |

The previous answers are good but here is a little more detailed.
1) Using 2 drives separately (annoying but needed sometimes)

Mount the 2 new drives

First need to figure out space constraints. Use df -h to see size available on new drives

in the folder with the data you want to transfer, use the du command to figure out what will go in what drive. easiest and most likely will be alphabetical

du -ch -d0 this will show space per top level folder. the -d0 is how deep to show. so change to -d1 to see top level folders and then sub level folders space.

du -ch -d0 [0-9a-tA-T]*/ This will show you space in top level folders only matching top level folders with the first character matching 0-9 or a-t or A-T. Find the letter that best suits your space requirements.

Use the opposite in your rsync --exclude command. So in this case I would use rsync --dry-run -av --exclude '/[u-zU-Z]*/' source destination. Remove --dry-run if it looks good. The leading / is to signify top level folders only. Notice the 0-9 is gone because we want to match all of that. We are also now matching the opposite letters we searched du with.

2) Raid drives as raid0 with mdadm. Then run a normal rsync command. Just know these drives need to stay together and if separated the data is lost.

answered 8 hours ago

BrinkDaDrink

1095

add a comment |

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f528969%2frsync-3-tb-of-data-to-2x-2tb-big-external-drives%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

3 Answers
3

active

oldest

votes

3 Answers
3

active

oldest

votes

answered Jan 6 '13 at 19:04

Darael

482210

Thanks, I think the option with having two separate rsync commands is the one I will be end up using. I'll make a text file excluding unwanted files/folders which will then be synced with the second rsync command. To use patterns in the text file would I be typing A* or 'A*'? Is this command case sensitive?

– jrn
Jan 6 '13 at 19:21

1

It's case sensitive, and you want to list files one per line. The same format can be used as in the du command I suggested above - to match everything starting with a through n, use /[a-nA-N]. To match only directories with that format, make it /[a-nA-N]/. The preceding / is to ensure it only matches at the root of the transfer, so that no subfolders are inadvertently excluded. No quotes are necessary in an exclude file (using --exclude-from), but single quotes are needed if using --exclude= to stop the shell from interpreting the pattern instead of rsync itself.

– Darael
Jan 6 '13 at 19:29

add a comment |

answered Jan 6 '13 at 19:04

Darael

482210

Thanks, I think the option with having two separate rsync commands is the one I will be end up using. I'll make a text file excluding unwanted files/folders which will then be synced with the second rsync command. To use patterns in the text file would I be typing A* or 'A*'? Is this command case sensitive?

– jrn
Jan 6 '13 at 19:21

1

It's case sensitive, and you want to list files one per line. The same format can be used as in the du command I suggested above - to match everything starting with a through n, use /[a-nA-N]. To match only directories with that format, make it /[a-nA-N]/. The preceding / is to ensure it only matches at the root of the transfer, so that no subfolders are inadvertently excluded. No quotes are necessary in an exclude file (using --exclude-from), but single quotes are needed if using --exclude= to stop the shell from interpreting the pattern instead of rsync itself.

– Darael
Jan 6 '13 at 19:29

add a comment |

answered Jan 6 '13 at 19:04

Darael

482210

answered Jan 6 '13 at 19:04

Darael

482210

answered Jan 6 '13 at 19:04

Darael

482210

answered Jan 6 '13 at 19:04

Darael

482210

answered Jan 6 '13 at 19:04

Darael

482210

Thanks, I think the option with having two separate rsync commands is the one I will be end up using. I'll make a text file excluding unwanted files/folders which will then be synced with the second rsync command. To use patterns in the text file would I be typing A* or 'A*'? Is this command case sensitive?

– jrn
Jan 6 '13 at 19:21

1

It's case sensitive, and you want to list files one per line. The same format can be used as in the du command I suggested above - to match everything starting with a through n, use /[a-nA-N]. To match only directories with that format, make it /[a-nA-N]/. The preceding / is to ensure it only matches at the root of the transfer, so that no subfolders are inadvertently excluded. No quotes are necessary in an exclude file (using --exclude-from), but single quotes are needed if using --exclude= to stop the shell from interpreting the pattern instead of rsync itself.

– Darael
Jan 6 '13 at 19:29

add a comment |

Thanks, I think the option with having two separate rsync commands is the one I will be end up using. I'll make a text file excluding unwanted files/folders which will then be synced with the second rsync command. To use patterns in the text file would I be typing A* or 'A*'? Is this command case sensitive?

– jrn
Jan 6 '13 at 19:21

1

It's case sensitive, and you want to list files one per line. The same format can be used as in the du command I suggested above - to match everything starting with a through n, use /[a-nA-N]. To match only directories with that format, make it /[a-nA-N]/. The preceding / is to ensure it only matches at the root of the transfer, so that no subfolders are inadvertently excluded. No quotes are necessary in an exclude file (using --exclude-from), but single quotes are needed if using --exclude= to stop the shell from interpreting the pattern instead of rsync itself.

– Darael
Jan 6 '13 at 19:29

Thanks, I think the option with having two separate rsync commands is the one I will be end up using. I'll make a text file excluding unwanted files/folders which will then be synced with the second rsync command. To use patterns in the text file would I be typing A* or 'A*'? Is this command case sensitive?

– jrn
Jan 6 '13 at 19:21

It's case sensitive, and you want to list files one per line. The same format can be used as in the du command I suggested above - to match everything starting with a through n, use /[a-nA-N]. To match only directories with that format, make it /[a-nA-N]/. The preceding / is to ensure it only matches at the root of the transfer, so that no subfolders are inadvertently excluded. No quotes are necessary in an exclude file (using --exclude-from), but single quotes are needed if using --exclude= to stop the shell from interpreting the pattern instead of rsync itself.

– Darael
Jan 6 '13 at 19:29

add a comment |

1) Simplest way:

If the 3TB of data is in directories less then 2TB, simply use two rsync commands.

E.g.



3TB  --- Folder1 with 1½TB --> rsync ---> 2TB drive

         Folder1 with 1½TB --> rsync ---> 2TB drive

3) Alternative:

Or mount the second 2TB drive in a folder in a volume on the first drive.

answered Jan 6 '13 at 19:04

Hennes

59.2k793142

So for option 1) would it make sense to copy all folders starting with A* to N* to drive 1 and all folders O* to Z* to drive 2? I guess I'd be using exclude in this case.

– jrn
Jan 6 '13 at 19:13

If all folders are about equal in size: yes.

– Hennes
Jan 6 '13 at 19:15

I would suggest using du to check that the total size of folders a*-n* is <= 2TB - after all, they may be unbalanced. du -c [a-nA-N]* will do this (reports the total size of all files and folders in the current directory with names starting with a to n or A to N - for some other folder, use /path/to/folder/[a-nA-N]*).

– Darael
Jan 6 '13 at 19:20

add a comment |

1) Simplest way:

If the 3TB of data is in directories less then 2TB, simply use two rsync commands.

E.g.



3TB  --- Folder1 with 1½TB --> rsync ---> 2TB drive

         Folder1 with 1½TB --> rsync ---> 2TB drive

3) Alternative:

Or mount the second 2TB drive in a folder in a volume on the first drive.

answered Jan 6 '13 at 19:04

Hennes

59.2k793142

So for option 1) would it make sense to copy all folders starting with A* to N* to drive 1 and all folders O* to Z* to drive 2? I guess I'd be using exclude in this case.

– jrn
Jan 6 '13 at 19:13

If all folders are about equal in size: yes.

– Hennes
Jan 6 '13 at 19:15

I would suggest using du to check that the total size of folders a*-n* is <= 2TB - after all, they may be unbalanced. du -c [a-nA-N]* will do this (reports the total size of all files and folders in the current directory with names starting with a to n or A to N - for some other folder, use /path/to/folder/[a-nA-N]*).

– Darael
Jan 6 '13 at 19:20

add a comment |

1) Simplest way:

If the 3TB of data is in directories less then 2TB, simply use two rsync commands.

E.g.



3TB  --- Folder1 with 1½TB --> rsync ---> 2TB drive

         Folder1 with 1½TB --> rsync ---> 2TB drive

3) Alternative:

Or mount the second 2TB drive in a folder in a volume on the first drive.

answered Jan 6 '13 at 19:04

Hennes

59.2k793142

1) Simplest way:

If the 3TB of data is in directories less then 2TB, simply use two rsync commands.

E.g.



3TB  --- Folder1 with 1½TB --> rsync ---> 2TB drive

         Folder1 with 1½TB --> rsync ---> 2TB drive

3) Alternative:

Or mount the second 2TB drive in a folder in a volume on the first drive.

answered Jan 6 '13 at 19:04

Hennes

59.2k793142

answered Jan 6 '13 at 19:04

Hennes

59.2k793142

answered Jan 6 '13 at 19:04

Hennes

59.2k793142

answered Jan 6 '13 at 19:04

Hennes

59.2k793142

So for option 1) would it make sense to copy all folders starting with A* to N* to drive 1 and all folders O* to Z* to drive 2? I guess I'd be using exclude in this case.

– jrn
Jan 6 '13 at 19:13

If all folders are about equal in size: yes.

– Hennes
Jan 6 '13 at 19:15

I would suggest using du to check that the total size of folders a*-n* is <= 2TB - after all, they may be unbalanced. du -c [a-nA-N]* will do this (reports the total size of all files and folders in the current directory with names starting with a to n or A to N - for some other folder, use /path/to/folder/[a-nA-N]*).

– Darael
Jan 6 '13 at 19:20

add a comment |

So for option 1) would it make sense to copy all folders starting with A* to N* to drive 1 and all folders O* to Z* to drive 2? I guess I'd be using exclude in this case.

– jrn
Jan 6 '13 at 19:13

If all folders are about equal in size: yes.

– Hennes
Jan 6 '13 at 19:15

I would suggest using du to check that the total size of folders a*-n* is <= 2TB - after all, they may be unbalanced. du -c [a-nA-N]* will do this (reports the total size of all files and folders in the current directory with names starting with a to n or A to N - for some other folder, use /path/to/folder/[a-nA-N]*).

– Darael
Jan 6 '13 at 19:20

So for option 1) would it make sense to copy all folders starting with A* to N* to drive 1 and all folders O* to Z* to drive 2? I guess I'd be using exclude in this case.

– jrn
Jan 6 '13 at 19:13

If all folders are about equal in size: yes.

– Hennes
Jan 6 '13 at 19:15

I would suggest using du to check that the total size of folders a*-n* is <= 2TB - after all, they may be unbalanced. du -c [a-nA-N]* will do this (reports the total size of all files and folders in the current directory with names starting with a to n or A to N - for some other folder, use /path/to/folder/[a-nA-N]*).

– Darael
Jan 6 '13 at 19:20

add a comment |

The previous answers are good but here is a little more detailed.
1) Using 2 drives separately (annoying but needed sometimes)

Mount the 2 new drives

First need to figure out space constraints. Use df -h to see size available on new drives

in the folder with the data you want to transfer, use the du command to figure out what will go in what drive. easiest and most likely will be alphabetical

du -ch -d0 this will show space per top level folder. the -d0 is how deep to show. so change to -d1 to see top level folders and then sub level folders space.

du -ch -d0 [0-9a-tA-T]*/ This will show you space in top level folders only matching top level folders with the first character matching 0-9 or a-t or A-T. Find the letter that best suits your space requirements.

Use the opposite in your rsync --exclude command. So in this case I would use rsync --dry-run -av --exclude '/[u-zU-Z]*/' source destination. Remove --dry-run if it looks good. The leading / is to signify top level folders only. Notice the 0-9 is gone because we want to match all of that. We are also now matching the opposite letters we searched du with.

2) Raid drives as raid0 with mdadm. Then run a normal rsync command. Just know these drives need to stay together and if separated the data is lost.

answered 8 hours ago

BrinkDaDrink

1095

add a comment |

The previous answers are good but here is a little more detailed.
1) Using 2 drives separately (annoying but needed sometimes)

Mount the 2 new drives

First need to figure out space constraints. Use df -h to see size available on new drives

in the folder with the data you want to transfer, use the du command to figure out what will go in what drive. easiest and most likely will be alphabetical

du -ch -d0 this will show space per top level folder. the -d0 is how deep to show. so change to -d1 to see top level folders and then sub level folders space.

du -ch -d0 [0-9a-tA-T]*/ This will show you space in top level folders only matching top level folders with the first character matching 0-9 or a-t or A-T. Find the letter that best suits your space requirements.

Use the opposite in your rsync --exclude command. So in this case I would use rsync --dry-run -av --exclude '/[u-zU-Z]*/' source destination. Remove --dry-run if it looks good. The leading / is to signify top level folders only. Notice the 0-9 is gone because we want to match all of that. We are also now matching the opposite letters we searched du with.

2) Raid drives as raid0 with mdadm. Then run a normal rsync command. Just know these drives need to stay together and if separated the data is lost.

answered 8 hours ago

BrinkDaDrink

1095

add a comment |

The previous answers are good but here is a little more detailed.
1) Using 2 drives separately (annoying but needed sometimes)

Mount the 2 new drives

First need to figure out space constraints. Use df -h to see size available on new drives

in the folder with the data you want to transfer, use the du command to figure out what will go in what drive. easiest and most likely will be alphabetical

du -ch -d0 this will show space per top level folder. the -d0 is how deep to show. so change to -d1 to see top level folders and then sub level folders space.

du -ch -d0 [0-9a-tA-T]*/ This will show you space in top level folders only matching top level folders with the first character matching 0-9 or a-t or A-T. Find the letter that best suits your space requirements.

Use the opposite in your rsync --exclude command. So in this case I would use rsync --dry-run -av --exclude '/[u-zU-Z]*/' source destination. Remove --dry-run if it looks good. The leading / is to signify top level folders only. Notice the 0-9 is gone because we want to match all of that. We are also now matching the opposite letters we searched du with.

2) Raid drives as raid0 with mdadm. Then run a normal rsync command. Just know these drives need to stay together and if separated the data is lost.

answered 8 hours ago

BrinkDaDrink

1095

The previous answers are good but here is a little more detailed.
1) Using 2 drives separately (annoying but needed sometimes)

Mount the 2 new drives

First need to figure out space constraints. Use df -h to see size available on new drives

in the folder with the data you want to transfer, use the du command to figure out what will go in what drive. easiest and most likely will be alphabetical

du -ch -d0 this will show space per top level folder. the -d0 is how deep to show. so change to -d1 to see top level folders and then sub level folders space.

du -ch -d0 [0-9a-tA-T]*/ This will show you space in top level folders only matching top level folders with the first character matching 0-9 or a-t or A-T. Find the letter that best suits your space requirements.

Use the opposite in your rsync --exclude command. So in this case I would use rsync --dry-run -av --exclude '/[u-zU-Z]*/' source destination. Remove --dry-run if it looks good. The leading / is to signify top level folders only. Notice the 0-9 is gone because we want to match all of that. We are also now matching the opposite letters we searched du with.

2) Raid drives as raid0 with mdadm. Then run a normal rsync command. Just know these drives need to stay together and if separated the data is lost.

answered 8 hours ago

BrinkDaDrink

1095

answered 8 hours ago

BrinkDaDrink

1095

answered 8 hours ago

BrinkDaDrink

1095

answered 8 hours ago

BrinkDaDrink

1095

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Super User!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Tjyylli