{"id":1213,"date":"2019-04-21T17:13:10","date_gmt":"2019-04-21T15:13:10","guid":{"rendered":"http:\/\/hobbykeller.spdns.de\/?p=1213"},"modified":"2019-04-28T01:09:46","modified_gmt":"2019-04-27T23:09:46","slug":"efficient-backup-of-your-raspberry-pi","status":"publish","type":"post","link":"https:\/\/hobbykeller.spdns.de\/?p=1213","title":{"rendered":"Efficient backup of your Raspberry Pi"},"content":{"rendered":"<p>Anyone who runs critical services such as web servers or private clouds on a Raspberry Pi is well advised to produce regular backups. The usual procedure is to remove the SD card, put it into your notebook and produce a card image with <code>dd<\/code>. This approach is as popular as it is inefficient: removing the SD card means that the system has to shut down completely. It is therefore impossible to run these backups unattended at regular intervals. Furthermore, <code>dd<\/code> also backs up empty spaces on the SD card, i.e. time and disk space are wasted. The <a href=\"https:\/\/www.linux-tips-and-tricks.de\/en\/backup\/\" target=\"_blank\" rel=\"noopener noreferrer\">raspiBackup<\/a> script promises to be a more intelligent solution, allowing &#8220;hot backups&#8221; onto local and remote devices while the system is running.<\/p>\n<p><!--more--><\/p>\n<h1>1. Remote backup destination should be NFS<\/h1>\n<p>Although raspiBackup allows for mounting local media, we would prefer to perform the backup of our Pi over the network. Keep in mind, that our Pi ZeroW does not even come with a grown up USB port to attach a memory stick to.<\/p>\n<p>The protocol used for sharing data over the network should satisfy the following basic requirements:<\/p>\n<ul>\n<li>it should allow symlinks and so called &#8220;hard links&#8221; in order to work efficiently with rsync which acts inside of raspiBackup<\/li>\n<li>it should follow the logic of chmod and chown permissions for files and directories on Linux systems<\/li>\n<\/ul>\n<p>Besides being complicated to set up on the file server, the popular samba protocol fails these requirements. The same goes for the handy sshfs protocol. Neither of these protocols can handle hard links. We therefore go for the NFS protocol which should suffice our requirements.<\/p>\n<h2>1.1 Prepare NFS directory on the destination machine<\/h2>\n<p>There is a very good description on sharing files over the NFS system on <a href=\"https:\/\/www.howtoforge.com\/tutorial\/how-to-configure-a-nfs-server-and-mount-nfs-shares-on-ubuntu-18.04\/\" target=\"_blank\" rel=\"noopener noreferrer\">howtoforge.com<\/a>. Make sure that the <code>nfs-kernel-server<\/code> package is installed on the destination machine.<\/p>\n<p>We first create the directory which is to be shared. We deviate from the instruction to create such a directory under <code>\/var\/nfs<\/code> because we do not want to have write-intensive operations on the SSD and our <code>\/var<\/code> partition on the server would have insufficient disk space to hold large amounts of backup data from the Raspberry.<\/p>\n<pre class=\"lang:default decode:true\">sudo mkdir \/home\/nfs\r\nsudo chown nobody:nogroup \/home\/nfs<\/pre>\n<p>Next we will have to edit <code>\/etc\/exports<\/code>. As the raspiBackup script will perform <code>chown<\/code> operations in the destination folder, the NFS access must be granted in a way that super user operations can be performed. This is done by the switch <code>no_root_squash<\/code> in the config file.<\/p>\n<pre class=\"lang:default decode:true\">  \/home\/nfs zerow(rw,sync,no_root_squash,no_subtree_check)\r\n<\/pre>\n<p>Note that each client to which the file system needs a line. So if you have another client with host name pi3bp, you have to add another entry to <code>\/etc\/exports<\/code>. Also note, that you can only use the host name if you have associated it to an ip-address in <code>\/etc\/hosts<\/code>. If this is not the case, you have to use the ip-address, e.g.:<\/p>\n<pre class=\"lang:default decode:true\"> \/home\/nfs 192.168.4.63(rw,sync,no_root_squash,no_subtree_check)<\/pre>\n<p>We refresh the current NFS configuration and start the NFS service:<\/p>\n<pre class=\"lang:default decode:true\">sudo exportfs -a\r\nsudo service nfs-kernel-server start<\/pre>\n<h2>1.2 Prepare the Raspberry client to access the NFS drive<\/h2>\n<p>Make sure you have the <code>nfs-common<\/code> package on your Raspberry.<\/p>\n<p>Next we need a point in the Raspberry directory structure to mount the remote NFS on. We therefore create a directory:<\/p>\n<pre class=\"lang:default decode:true\">pi@zerow:\/srv $ sudo mkdir -p \/mnt\/nfs\/backups\r\n<\/pre>\n<p>As we want to access this directory through raspiBackup with super user privileges, there is no need to alter ownership, group membership and access privileges.<\/p>\n<p>Mount the remote file system to that directory on the raspberry by:<\/p>\n<pre class=\"lang:default decode:true\">pi@zerow:\/srv $ sudo mount 192.168.4.41:\/home\/nfs \/mnt\/nfs\/backups\/\r\n<\/pre>\n<p>We are now connected to the remote file system on the server which stores the backup.<\/p>\n<h1>2. Download, Installation and Configuration of raspiBackup<\/h1>\n<p>I tried my luck with the raspiBackup script. Download links and installation instructions can be found at the <a href=\"https:\/\/www.linux-tips-and-tricks.de\/en\/backup\/\" target=\"_blank\" rel=\"noopener noreferrer\">raspiBackup home page<\/a>.<\/p>\n<h2>2.1 Initial Setup<\/h2>\n<p>After downloading the raspiBackup configuration and programm code resides under <code>\/usr\/local\/bin<\/code>. For configuration and updates, call<\/p>\n<pre class=\"lang:default decode:true\">pi@zerow:\/usr\/local\/bin $ sudo raspiBackupInstall.sh \r\n--- RBI0001I: raspiBackupInstall.sh 0.3.7, 2018-07-14\/17:59:44 - 5f9e49b\r\n--- RBI0055I: Using logfile raspiBackupInstall.log.\r\n--- RBI0057I: Checking internet connection.\r\n--- RBI0032I: Checking if there is a beta version available.\r\n--- RBI0014I: Downloading https:\/\/www.linux-tips-and-tricks.de\/downloads\/raspibackup0613-properties\/download...\r\n--- RBI0053I: Installing raspiBackup.sh 0.6.4.2.\r\n--- RBI0041I: Default option is in UPPERCASE.\r\n--- RBI0002I: Message language (de|EN) : \r\n--- RBI0003I: Normal or partition oriented mode (N|p) : \r\n--- RBI0004I: Backuptype (dd|tar|RSYNC) : \r\n--- RBI0006I: Number of backups (1-52) : 3\r\n--- RBI0008I: Verbose messages (Y|n) : \r\n--- RBI0042I: Selected configuration: Message language: en, Backupmode: normal, Backuptype: rsync\r\n--- RBI0048I: Selected configuration: Compress backups: n, Number of backups: 3, Verbose messages: y\r\n--- RBI0009I: Configuration OK (y|N) : y\r\n\r\n--- RBI0014I: Downloading raspiBackup.sh...\r\n--- RBI0037I: Created \/usr\/local\/bin\/raspiBackup.sh.\r\n--- RBI0037I: Created \/usr\/local\/bin\/raspiBackupInstall.sh.\r\n\r\n--- RBI0014I: Downloading raspiBackup.conf...\r\n--- RBI0037I: Created \/usr\/local\/etc\/raspiBackup.conf.\r\n--- RBI0024I: Updating configuration in \/usr\/local\/etc\/raspiBackup.conf.\r\n--- RBI0049I: Creating sample cron file \/etc\/cron.d\/raspiBackup.\r\n--- RBI0023I: Installation of raspiBackup finished successfully.\r\n<\/pre>\n<h2>2.2 Amendments to <code>raspiBackup.conf<\/code><\/h2>\n<h3>Backup path<\/h3>\n<p>We still need to edit <code>\/usr\/local\/etc\/raspiBackup.conf<\/code> and amend our mounted NFS directory as the destination:<\/p>\n<pre class=\"lang:default decode:true\">DEFAULT_BACKUPPATH=\"\/mnt\/nfs\/backups\"<\/pre>\n<h3>Exclusion of directories<\/h3>\n<p>While raspiBackup excludes typical directories that are &#8220;not worth&#8221; being backed up (e.g. \/tmp) automatically, it may make sense to exclude further directories.<\/p>\n<p>A typical example is the local Nextcloud\/ownCloud directory, which is already remotely stored on the Nextcloud server. As that server is also backed up, including the local \/home\/pi\/ownCloud directory would mean duplicate backups.<\/p>\n<p>These exclusions can be defined by the <code>DEFAULT_EXCLUDE_LIST<\/code> parameter. Entries in that list must follow rsync conventions, i.e.<\/p>\n<ul>\n<li>each exclusion must be introduced by <code>--exclude=&lt;relative path&gt;<\/code>,<\/li>\n<li>unlike when specifying source and destination paths the &#8220;trailing slash&#8221; has no effect. A directory in the exclude list will be excluded completely &#8211; with or without trailing slash.<\/li>\n<li>each directory must have a complete serparate exclude entry in the list. If you want to exclude dir1 and dir2, then the entrys must not be<code>--exclude=dir1 dir2<\/code> <strong>but<\/strong> <code>--exclude=dir1 --exclude=dir2<\/code><\/li>\n<\/ul>\n<p>The below is an example path which excludes the ownCloud directory and the directory holding the media for the minidlna server:<\/p>\n<pre class=\"lang:default decode:true \">DEFAULT_EXCLUDE_LIST=\"--exclude \/home\/pi\/owncloud\/ --exclude var\/lib\/minidlna\/\"<\/pre>\n<h3>SUSPENDING services DURING BACKUP<\/h3>\n<p>In order to prevent inconsistent data, it is advisable to stop services like mysqld, nginx, cron etc. After the backup is completed, each service must be restarted again. This is done by the <code>DEFAULT_STOP_SERVICES<\/code> and <code>DEFAULT_START_SERVICES<\/code> parameters.<\/p>\n<p>Keep in mind that the order in which the services are started or stopped can make a difference. Below is an example of what the services entries might look like:<\/p>\n<pre class=\"lang:default decode:true\" title=\"raspiBackup.conf suspension of services\"># commands to stop services before backup separated by &amp;&amp; \r\nDEFAULT_STOPSERVICES=\"service cron stop &amp;&amp; service nginx stop &amp;&amp; service prosody stop &amp;&amp; service minidlna stop &amp;&amp; service mysql stop\" \r\n \r\n# commands to start services after backup separated by &amp;&amp; \r\nDEFAULT_STARTSERVICES=\"service mysql start &amp;&amp; service minidlna start &amp;&amp; service prosody start &amp;&amp; service nginx start &amp;&amp; service cron start\"\r\n<\/pre>\n<h1>3. Performing Backups<\/h1>\n<p>Keep in mind that backups over the network can be extremely time consuming. The following table summarizes the time needed for backing up different Raspbian systems over NFS. &#8220;Initial backup&#8221; implies that each and every file on the source system has to be transferred over the network. After that backup is completed, another backup is relaunched (&#8220;resync&#8221;). As there have been no modifications to the system, it can be assumed that this is an (almost) pure check if each file on the source system has an identical counterpart on the backup system.<\/p>\n<p>The time needed for a full sync can be interpreted as the &#8220;upper bound&#8221; of the backup time needed for a system of the given size while the &#8220;resync&#8221; time is a proxy for the lower bound.<\/p>\n<p>[table id=20 \/]<\/p>\n<h2>3.1 Manual start of the backup<\/h2>\n<p>Manual start of a backup is triggered by:<\/p>\n<pre class=\"lang:default decode:true\">pi@pi3bp:\/usr\/local\/bin $ sudo .\/raspiBackup.sh -g<\/pre>\n<h1>4. Restore<\/h1>\n<p>Restore instructions are based on the procedures laid out on the <a href=\"https:\/\/www.linux-tips-and-tricks.de\/en\/restore\/\" target=\"_blank\" rel=\"noopener noreferrer\" class=\"broken_link\">raspiBackup page<\/a>. We assume that the backup snapshots are on an NFS exported file system on a remote machine that we call the backup server.<\/p>\n<p>Unlike the backup procedure, the restore procedure is not a &#8220;hot restore&#8221;, i.e. we have to remove the SD card from the Raspberry and put it into an Ubuntu machine that has raspiBackup installed and that we call the restore machine.<\/p>\n<ol>\n<li>Put the SD card\u00a0 tho which the backup shall be restored into a Ubuntu machine that has a SD card reader and raspiBackup installed.<\/li>\n<li>Export the backup directory on the remote server via NFS (see section 1.1, make sure your ubuntu restore client holding the SD card is listed in the\u00a0 <code>\/etc\/exports<\/code> entries on the backup server).<\/li>\n<li>On the restore machine, mount the NFS directory\n<pre class=\"lang:default decode:true\">sudo mount 192.168.4.41:\/home\/nfs \/mnt\/nfs\/backups<\/pre>\n<\/li>\n<li>On the restore machine, determine the path to the SD card. You can find that path by running <code>sudo parted -l<\/code>. In our case, it is <code>\/dev\/mmcblk0<\/code>.<\/li>\n<li>Once you know the default path, start the backup procedure, you need to be root:<\/li>\n<\/ol>\n<pre class=\"lang:default decode:true\" title=\"raspibackup restore procedure\">root@nordic:\/usr\/local\/bin# sudo raspiBackup.sh -d \/dev\/mmcblk0 \/mnt\/nfs\/backups\/pi3test\/pi3test-rsync-backup-20190419-181915\/\r\n--- RBK0009I: nordic: raspiBackup.sh V0.6.4.2 (74411c9) started at Mon Apr 22 15:20:29 CEST 2019.\r\n--- RBK0031I: Checking whether a new version of raspiBackup.sh is available.\r\n--- RBK0168I: :-D raspiBackup.sh beta version 0.6.4.3 is available. Any help to test this beta is appreciated. Just upgrade to the new beta version with option -U. Restore to the previous version with option -V\r\n--- RBK0114I: Visit https:\/\/www.linux-tips-and-tricks.de\/en\/versionhistory\/ to read about the changes in the new version.\r\n!!! RBK0018W: Target \/dev\/mmcblk0 with 7.40 GiB is larger than backup source with 7.40 GiB. root partition will be expanded accordingly to use the whole space.\r\n!!! RBK0065W: Device \/dev\/mmcblk0 will be repartitioned and all data will be lost.\r\n--- RBK0067I: Current partitions on \/dev\/mmcblk0:\r\nNumber  Start   End     Size    Type     File system  Flags\r\n 1      4.19MB  49.2MB  45.0MB  primary  fat32        lba\r\n 2      50.3MB  7948MB  7898MB  primary  ext4\r\n!!! RBK0066W: Device \/dev\/mmcblk0 will be overwritten with the saved boot and root partition.\r\n--- RBK0038I: Are you sure? y\/N y\r\n--- RBK0050I: Restoring backup from \/mnt\/nfs\/backups\/pi3test\/pi3test-rsync-backup-20190419-181915.\r\n--- RBK0053I: Restoring first partition (boot partition) to \/dev\/mmcblk0p1.\r\n--- RBK0055I: Restoring second partition (root partition) to \/dev\/mmcblk0p2.\r\n--- RBK0033I: Please wait until cleanup has finished.\r\n--- RBK0076I: Restore finished successfully.\r\n--- RBK0010I: pi3test: raspiBackup.sh V0.6.4.2 (74411c9) stopped at Mon Apr 22 15:54:25 CEST 2019.\r\n<\/pre>\n<p>As can be seen from the output, it takes about 1\/2 hour to restore a 3 GB test system.<\/p>\n<p>Unmount the SD card, put it into your Raspberry Pi and reboot the restored system.<\/p>\n<h1>5. Trouble shooting<\/h1>\n<p>I have had an issue with ACL (access control list) attributes for the <code>\/media\/pi<\/code> directory. The <a href=\"https:\/\/www.linux-tips-and-tricks.de\/en\/faq\/#a24\" target=\"_blank\" rel=\"noopener noreferrer\" class=\"broken_link\">raspiBackup FAQ<\/a> recommends removing the <code>acl<\/code> entry from the <code>\/etc\/mke2fs.conf<\/code> file. It turns out this modification has not helped at all. Whether the <code>acl<\/code> entry is removed on the Raspberry, on the server or on both systems &#8211; raspiSync will always abort the backup process saying it cannot handle ACL entries.<\/p>\n<p>The solution was therefore to <a href=\"https:\/\/unix.stackexchange.com\/questions\/339765\/how-to-remove-acl-from-a-directory-and-back-to-usual-access-control\" target=\"_blank\" rel=\"noopener noreferrer\">remove the ACL entry<\/a> directly from the <code>\/media\/pi<\/code> directory. ACL flagged directories can be recognized by the + in the <code>ls -alh<\/code> output (e.g. <code>drwxrwxr-x+<\/code>).<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Anyone who runs critical services such as web servers or private clouds on a Raspberry Pi is well advised to produce regular backups. The usual procedure is to remove the<span class=\"more-button\"><a href=\"https:\/\/hobbykeller.spdns.de\/?p=1213\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\">Efficient backup of your Raspberry Pi<\/span><\/a><\/span><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[64,237],"tags":[273,269,270,271],"class_list":["post-1213","post","type-post","status-publish","format-standard","hentry","category-linux","category-raspberry","tag-nfs","tag-raspberry-pi","tag-raspibackup","tag-rsync"],"_links":{"self":[{"href":"https:\/\/hobbykeller.spdns.de\/index.php?rest_route=\/wp\/v2\/posts\/1213","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hobbykeller.spdns.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hobbykeller.spdns.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hobbykeller.spdns.de\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/hobbykeller.spdns.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1213"}],"version-history":[{"count":20,"href":"https:\/\/hobbykeller.spdns.de\/index.php?rest_route=\/wp\/v2\/posts\/1213\/revisions"}],"predecessor-version":[{"id":1240,"href":"https:\/\/hobbykeller.spdns.de\/index.php?rest_route=\/wp\/v2\/posts\/1213\/revisions\/1240"}],"wp:attachment":[{"href":"https:\/\/hobbykeller.spdns.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1213"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hobbykeller.spdns.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1213"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hobbykeller.spdns.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1213"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}