Photo credit: ardenswayoflife on Flickr, used under a CC-BY-SA 2.0 license.
The Burden of files
Okay, there’s a lot of problems:
- Backing up your files is a pain in the butt.
- Every time you upgrade your system, either in-place or a fresh install, it is a royal hassle to restore your files.
- That file looks great on your laptop, but how do you show it to a colleague not sitting next to you easily? Ughh.
- You upload files to a random directory on some web server you have some space on, quickly to show an idea to someone. Fast-forward some time, and you’ve got disorganized, poorly-named files scattered across multiple shell / other accounts all over the web, and you’re not sure what you have a copy of where, or which ones are being referred to from other places, so you’re terrified to delete any of them.
- Well, crap. You’ve made a mistake. You can’t go back, can you? No version control…
I think we all know these problems pretty well. I’ve built a solution using Fedora and Sparkleshare – completely free and open source software – that over the past week has addressed all of these issues and has substantially improved the quality of my computing life. It backs my work files up to an internal corporate server and it backs my Fedora files up to a Fedora-maintained public server. I’m planning to configure it to back up some personal files to my Dreamhost account and some to my NAS at home.
What? What are you talking about?
Here’s how it works: I have a ‘Sparkleshare’ folder in my home directory. Under that, I have a couple of subfolders:
- design-team (this is for Fedora stuff)
When I start a project, say it’s for work, then I create a new subfolder under the appropriate directory. So I’m starting a new project, let’s say it’s a logo design for project X. I have the following directory tree:
I open up Inkscape, and start working on the logo. I work on it for a while. Now I’ve got a draft I’d like to save and show to a colleague. From within Inkscape, I navigate to the SparkleShare > workstuff > logos folder and save the logo there. Then I can go to my internal server, and automagically, the logo file I just saved is at http://hostname/workstuff/logos/mylogo.svg. I can quickly copy/paste that URL from my browser and send it via IRC or email to my colleague.
No uploading, no fumbling to find a server that has space in an appropriate location, no waiting for the upload to happen, no coming up with some lame variation of ‘temp’, ‘tmp’, ‘foobar’, ‘blah’ to make a fresh folder to upload it to. Nope. It’s just there.
Here’s how to set this up for yourself.
You will need:
- A desktop or laptop computer with Fedora 14 or Fedora 15 installed. You’ll be needing root access.
- A remote server you’re going to use as your main git repository. This can be a server you administer yourself, or it can be a shell account with lots of space on a hosting provider like Dreamhost. It could even be a NAS if you can install git on it like you can on mine.
- An ssh key, preferably with a passphrase. I am assuming you already have your SSH key configured to enable you access to the remote server you’re intending to set your main git repo on. If not, the following resources below might be helpful:
- Files. Lovely little (or big) files.
- I’m assuming you’re using GNOME. Sparkleshare does work on KDE or XFCE or LXDE. What you lose by not using GNOME is the nautilus right-click extension that lets you – from gitorious or github repos – check out older versions of a document or grab a web link to the document. Not a huge deal.
Step 1: Prepare your local system
We’re going to install Sparkleshare first.
- Open up a terminal.
- Go to the yum repositories directory:
- As root, download Alex Hudson’s Sparkleshare Fedora people repo config:
sudo curl -O http://repos.fedorapeople.org/repos/alexh/sparkleshare/fedora-sparkleshare.repo
- Install sparkleshare and openssh-askpass:
sudo yum install -y sparkleshare openssh-askpass
- Remove nautilus-python (I don’t know why, but if you have this installed and install Sparkleshare, Nautilus will segfault on startup. If you remove it, though, everything works fine.)
sudo yum remove -y nautilus-python
You now have Sparkleshare installed.
Step 2: Set up your main git repo
First you have to figure out where you want your main git repo to be. It needs to be in an account that your desktop/laptop can connect to. I set mine up in a remote home directory I own. Here’s how to do it:
- We’re going to create the empty new git repo. In your remote home directory, type
git init --bare repo.git
. (If this command doesn’t seem to work and you have permissions to do so, try installing openssh-server and updating git.)
- Now we’re going to make a dummy clone of the repo to add an initial file and branch. To make things more interesting, we’re going to do this in your public_html directory, but you can do it anywhere you’d like in your home directory:
- Go into your public_html directory.
- Clone it! Type
git clone repo.git
- Go into your new clone with
- Now make a new dummy file,
echo 'Fedora 15 rocks!' >> test.txt
- Now commit it.
git add test.txt; git commit test.txt -m 'initial commit2'
- Push it, making your initial branch.
git push origin master
- Go into your public_html directory.
Step 3: Hook your local system up to your repo
Okay, now we’re going to get your local system hooked up to the repo you just created on your remote system.
On your local laptop/desktop:
- Turn Sparkleshare on. Click on its icon in your Applications menu, or run it from the terminal:
- Right-click on the Sparkleshare icon in your system tray, and select “Add Remote Folder…”
- Where Sparkleshare asks you: “Where is your remote folder?” select “On my own server.” In the field after “On my own server, replace my fake credentials here with your own, and don’t forget the ‘.git’ on the end.
- Where Sparkleshare asks you for the “Folder name”, fill in something like this, replacing my directory structure with your own on the remote server:
- Sparkleshare should say something about ‘coffee o’clock’ and it should start pulling the remote files down.
- Look in your Sparkleshare folder for the repo you just connected. Are the files from the server there?
- Now we’re going to make sure it’s set up right. Create a text file in your new Sparkleshare repo directory (will be somewhere like ‘/home/user/Sparkleshare/repo’).
Okay, that should have pushed up to the server, no issues. Make sure it did; On your remote system where git is:
- Go into your remote clone.
- Pull down the changes!
- Make sure the changes worked. You can either visit the web side of your clone at (replace with the relevant hostname and username) http://hellokitty.ponies.com/~user/repo or you can just cat it, making sure our new panda message is present:
All right, hopefully that wasn’t an issue either and your remote clone was able to pick up the panda PSA you pushed from your local laptop/desktop. Now we’re going to throw Sparkleshare into the mix.
On your local laptop/desktop where Sparkleshare is installed:
- Restart Sparkleshare. You can click on the icon or
- Go to the Sparkleshare repo directory.
- Make a new test file. In honor of towel day today:
echo '42' >> theanswer.txt
Let’s see if it worked!!!!! On your remote system:
- Go into the clone:
- Pull it down!
Did it work??? It worked for me!!
Step 4: Set up automagical web mirroring of your repo’s content
Okay, so I did a sneaky thing. Our remote repo, in the examples above, has been configured in your remote public_html directory on the same system your remote git repo is. Now, you could follow these directions on a third remote system if you’d like; no problem. Just make sure you’ve got your public SSH key configured on that third system and that you have access to write to a web-readable directory. These instructions should not be difficult to modify for that case; simply git clone your repo in a web-readable directory.
We are now going to hook this all up so that when you save a file on your local laptop/desktop, and SparkleShare checks it in, the web-readable clone of it all also automagically pulls down your change and makes the file available in a web browser. Woo!
On your remote system:
- Get into the hooks directory where your main (not clone) git repo is (don’t forget the .git!):
- We’re gonna set up a hook so whenever you push a file to the repo, it tells the web clone to pull. Create a file called ‘post-receive’ in ~/git.repo/hooks with the following in it, replacing ‘user’ with your own username on the remote system:
#!/bin/sh while read oldrev newrev refname do true done cd /home/user/public_html/repo GIT_DIR=/home/user/public_html/repo/.git git pull
Okay, let’s see if it worked.
On your local laptop/desktop where Sparkleshare is installed:
- Make sure Sparkleshare is running. If it’s not, start it.
- Go into the right directory and make another test file.
cd ~/SparkleShare/repo; echo 'Fedora 15' >> latestfedora.txt
Okay, moment of truth. Is your new test file in your web directory? Either visit (replacing the username and hostname of course) http://hellokitty.ponies.com/~user/repo and look for a file named latestfedora.txt (if it’s there, it worked!) or on the remote system:
- Go into the web-readable repo clone:
- Look for the file.
What do you think?
So, what do you think? Pretty awesome, right? Goodbye, dropbox – a worthless solution when your corporate IT policy rightly frowns upon copying internal documents to third-party public servers. You can create as many repos like this as you like. You can host them on a remote server you have shell access to, as this tutorial assumed, or you can set it up on Gitorious (I recommend this one since it’s backed by free & open source sw) or Github. Or a Fedora Hosted repo if you’ve got one. Each repo will have a folder under your ‘SparkleShare’ folder in your home directory, and anything in that directory is basically remotely backed up, instantly, as soon as you save files in those directories (and given that Sparkleshare is running.)
We’re currently experimenting with using this setup for the Fedora Design team to hopefully make it easier for folks to access our files and contribute.
Some things you might want to consider with this setup
- Set Sparkleshare to always run when your machine is booted. For GNOMEies:
cp /usr/share/applications/sparkleshare.desktop ~/.config/autostart
- Write down the remote git URLs of all of the remote repos your stuff is backed up to and keep it in a safe place. If disaster strikes and a panda mistakes your laptop for bamboo, it will make it that much easier to recover.
- Hylke Bons and the rest of the Sparkleshare community for making such an awesome, life-changing tool.
- Alex Hudson for creating and maintaining the Fedora people repos Sparkleshare repository.
- Ray Strode, Kevin Fenzi, and Ricky Zhou for fielding my crazy git questions. Ray wins the prize for figuring out how to get past git complaining about a ‘sideband demultiplexer’.