www.bonsaiframework.com
Page tree

Versions Compared

Old Version 45

changes.mady.by.user Tin Pham

Saved on

compared with

New Version Current

changes.mady.by.user Tin Pham

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Besides for standard sync and backup, use rsync to,

  1. Copy over network using the faster rsync protocol and measure speed.
  2. Copy and resume large directory backups.

...

Code Block
languagebash
caffeinate rsync [...] # Mac OS X
? systemd-ihibit rsync [...] # Linux

Backup Interactively

Here is the rsync command for folder to folder example,

Code Block
languagebash
rsync --archive --delete --sparse --verbose --itemize-changes --human-readable --progress /home/tempadmin/source /home/tempadmin/destination

--dry-run = Use this first to ensure to simulate a run. Especially if you use --delete.

Warning

Be very careful not to include a trailing slash in source and destination(maybe slash ok in destination but need to test) or you will end up deleting everything in your target.

The trailing slash (/) on the source directory modifies the behaviour,

  • No trailing slash, the source directory is copied to the destination directory, and then the contents of the directory.
  • With trailing slash, rsync only copies the content of the source without creating an additional directory level.


--archive = which includes,

...

--delete = delete any files not in the source

--sparse = --verbose =  

--itemize-changes = 

--human-readable =

--progress = 

source = see what's happening

source = what you are syncing

destination = destination may be a directory but the most useful is actually to or another system running the rsync service

Be very careful not to include a trailing slash in source and destination(maybe slash ok here but need to test) or you will end up deleting everything in your target.

Backup to rsync Service

Instead of a directory, the destination may be another system running the rsync service. This method is extremely fast compare to the everyday use AFP or SMB share protocols. However, I believe there are some limitations when it comes to special characters at least on Mac OS X. I'm still figuring this out.

You must have on the other side an RSync service listening and created an account (in this example rSyncUser) with access to the appropriate directory. Many modern NAS have this capability, you just need to turn them on.

Code Block
languagebash
rsync --archive --delete --sparse --compress --verbose --itemize-changes --human-readable --progress /home/tempadmin/source rsync://rsyncUser@destSystem/destination

--destination = 

--compress = 

Run rsync Multiple Times

A point of interest is that I find I often need to run rsync more than once, as it often finds discrepancies even right after the first transfer. Keep on running rsync until you stop seeing "files to consider".

...

--password-file = 

--log-file= 

Straight Copy

Simple straight copy and can be used to resume (still need to modify for resume),

Code Block
languagebash
 rsync --archive --sparse --compress --verbose --itemize-changes --human-readable --progress /home/tempadmin/tmp/source /home/tempadmin/tmp/destination

...

More Details Understanding of the Attributes

--archive

Same as -rlptgoD (no -H) 

...

This option is useful for transfer of large files with block-based changes or appended data, and also on systems that are disk bound, not network bound.

Exclude Unnecessary Mac OS

...

Hidden Files

Within a script,

Code Block
languagebash
# rsync can't handle spaces if just a variable https://stackoverflow.com/questions/19219774/bash-rsync-with-options-as-variable
EXCLUDE_ARRAY=("'$RECYCLE.BIN'" "'$Recycle.Bin'" "'.AppleDB'" "'.AppleDesktop'" "'.AppleDouble'" "'.com.apple.timemachine.supported'" "'.dbfseventsd'" "'.DocumentRevisions-V100*'" "'.DS_Store'" "'.fseventsd'" "'.PKInstallSandboxManager'" "'.Spotlight*'" "'.SymAV*'" "'.symSchedScanLockxz'" "'.TemporaryItems'" "'.Trash*'" "'.vol'" "'.VolumeIcon.icns'" "'Desktop DB'" "'Desktop DF'" "'hiberfil.sys'" "'lost+found'" "'Network Trash Folder'" "'pagefile.sys'" "'Recycled'" "'RECYCLER'" "'System Volume Information'" "'Temporary Items'" "'Thumbs.db'" "'DF'" "'Trash'" "'Folder'"  "'Volume'"  "'Information'" "'Items'")

And then add the variable at the beginning of your execution line as follows,

Code Block
languagebash
caffeinate -s rsync "${EXCLUDE_ARRAY[@]/#/--exclude=}" --archive ...

You can also do this from the command line too, but the syntax will be different... (still to document).

Understanding the rsync Progress

...

Panel
YXcstpoguax  path/to/file
|||||||||||
||||||||||╰- x: The extended attribute information changed
|||||||||╰-- a: The ACL information changed
||||||||╰--- u: The u slot is reserved for future use
|||||||╰---- g: Group is different
||||||╰----- o: Owner is different
|||||╰------ p: Permission are different
||||╰------- t: Modification time is different
|||╰-------- s: Size is different
||╰--------- c: Different checksum (for regular files), or
||              changed value (for symlinks, devices, and special files)
|╰---------- the file type:
|            f: for a file,
|            d: for a directory,
|            L: for a symlink,
|            D: for a device,
|            S: for a special file (e.g. named sockets and fifos)
╰----------- the type of update being done::
             <: file is being transferred to the remote host (sent)
             >: file is being transferred to the local host (received)
             c: local change/creation for the item, such as:
                - the creation of a directory
                - the changing of a symlink,
                - etc.
             h: the item is a hard link to another item (requires 
                --hard-links).
             .: the item is not being updated (though it might have
                attributes that are being modified)
             *: means that the rest of the itemized-output area contains
                a message (e.g. "deleting")

Example output (and I'll add more details here),

Panel
>f+++++++++ some/dir/new-file.txt
.f....og..x some/dir/existing-file-with-changed-owner-and-group.txt
.f........x some/dir/existing-file-with-changed-unnamed-attribute.txt
>f...p....x some/dir/existing-file-with-changed-permissions.txt
>f..t..g..x some/dir/existing-file-with-changed-time-and-group.txt
>f.s......x some/dir/existing-file-with-changed-size.txt
>f.st.....x some/dir/existing-file-with-changed-size-and-time-stamp.txt 
cd+++++++++ some/dir/new-directory/
.d....og... some/dir/existing-directory-with-changed-owner-and-group/
.d..t...... some/dir/existing-directory-with-different-time-stamp/


Mac OS X to Errors

Use this only if things don't work on Mac OS X with the error message noted. On certain (I've yet to determine) versions of Mac OS X, restrictions have increased.

Special Character Issues btw UTF-8 Mac and UTF-8

...

Terminal Restriction

Before using rsync you need to lift FDA (full disk access) restrictions in System Preferences > Security & Privacy > Privacy > Full Disk Access and add Terminal otherwise you may see the error messages when trying to rsync.

...

Copy extended attributes and resource forks.

Exclude unnecessary Mac OS X System Files

List from Alan Smith,

Code Block
--exclude='$RECYCLE.BIN' --exclude='$Recycle.Bin' --exclude='.AppleDB' --exclude='.AppleDesktop' --exclude='.AppleDouble' --exclude='.com.apple.timemachine.supported' --exclude='.dbfseventsd' --exclude='.DocumentRevisions-V100*' --exclude='.DS_Store' --exclude='.fseventsd' --exclude='.PKInstallSandboxManager' --exclude='.Spotlight*' --exclude='.SymAV*' --exclude='.symSchedScanLockxz' --exclude='.TemporaryItems' --exclude='.Trash*' --exclude='.vol' --exclude='.VolumeIcon.icns' --exclude='Desktop DB' --exclude='Desktop DF' --exclude='hiberfil.sys' --exclude='lost+found' --exclude='Network Trash Folder' --exclude='pagefile.sys' --exclude='Recycled' --exclude='RECYCLER' --exclude='System Volume Information' --exclude='Temporary Items' --exclude='Thumbs.db'


Over SSH Protocol

rsync ....

...

Solution to FUSE or NFS on MAC - https://stackoverflow.com/questions/667992/rsync-error-failed-to-set-times-on-foo-bar-operation-not-permitted/668049#668049

Understanding what the Progress Bar Looks like - https://stackoverflow.com/questions/4493525/what-does-f-mean-in-rsync-logs

Special Character and Platform Difference Issues - https://askubuntu.com/questions/533690/rsync-with-special-character-files-not-working-between-mac-and-linux https://apple.stackexchange.com/questions/148799/rsync-with-linux-server-special-character-problem