Page 1
Status

rsync alternative to scp

For our work I managed to accumulate lots of simulation results. Now I need to download all that data from the University's server to my SSD and do the analysis. Sounds pretty easy: connect via VPN from Germany, and scp the files, right? Well not really... After around 20 minutes, the connection stalls and experiences a packet_write_wait timeout, which causes the SSH pipe to break and my download to fail. Because I named the files in my output directories the same I thought: why not only download the small and important files first, and then download the larger files for backup purposes? Simply call:

scp username@uni.ac.uk:/path/to/dir/output_*/*/data.mat /path/to/ssd

But what this would do is download one data.mat file, for example, and overwrite it when loading the next data.mat file that's stored in another folder, and so on... That's when I stumbled over rsync which seems to download the files I want, by excluding the ones I don't want, and store them in the correct directory, too!

rsync -avr --rsh=ssh --exclude '*bigFile.mat' --progress username@uni.ac.uk:/path/to/dir/output_*/path/to/ssd

And boom I see a nice directory hierarchy being created first and all small data.mat files being stored in their corresponding places. After having acquired all those files I could even use rsync to only download the bigFile.mat data files and ignore the rest, by either changing the --exclude argument or by replacing it with --ignore-existing.

I like this scp alternative quite a bit now 🙂

The end
Aside

"Old" Raspberry Pi as a cloner

This will be a short post, but helpful... After my recent time machine hard drive fiasco I managed to recover my old backup, but could only restore the old hard drive by repartitioning and reformatting it completely. Now it looks as if it works as normal, yet I don't trust the drive to be functioning much longer as it already had a major hiccup. So instead of having it fully powered on, I decided to get my "old" Raspberry Pi (Rev2 Model B, the one wth 512MB) and copy the backup from my new and working drive onto the old one, so it serves some use whilst collecting dust. And in this short post I'll tell you (and remind future me) how to do that.

Continue Reading →

The end
Aside

Fast and awesome backups

There are many ways to save data; copy individual files by hand, copy entire user folders or clone entire partitions. The issue with most simple backup approaches (especially the ones I've tried in Windows), may store all the data I want, but making the backup bootable is often quite tricky. In this little post I'd like to show some awesome and free apps for Mac OS that allow you to save the entire Mac, and on top of that, make your backup bootable, so you can:

  • boot into your account from any Mac and have a mobile version of all your data,
  • start from the external drive, e.g if your internal one dies or you break Mac OS, and carry on working as usual (maybe a bit slower than usual if you're running over USB)
  • swap the internal and external disks if you want to upgrade size, and
  • even store the Windows bootcamp in a bootable form to the same partition.

Continue Reading →

The end
Aside

My time machine died...

My 4TB backup drive doesn't seem to have recovered from yesterday's power surge, and has major issues functioning as my time machine volume. So far I found my setup pretty elegant. It consists of a Synology NAS (the DS-play variant) and my 4TB (external) hard drive.

The NAS is in daily use since it contains all my important files that I do not (and for my Uni's research point of view must not) lose. On the 4TB drive I initially had set up a time machine from my Mac before I bought the NAS, and I did that pretty quickly over USB 3.0. When plugging this USB drive into my NAS it was not only possible to make the time machine volume accessible via the network, but I could launch the NAS's built in time machine legacy support!

Continue Reading →

The end