Tag Archives: TT

TT: Uni-directional file synchronization between different hosts with Unison

When you work with at least two computers on the same project on a daily basis you might have a problem. You need to get changed files from host A to host B and vice versa. The problem getting bigger when you work in addition on different operation systems or use more than two hosts. On UNIX/Linux the preferred tool for such a task is Rsync. Unfortunately Rsync synchronize only in one direction, it doesn’t work very well when more than two hosts are involved (and it isn’t really comfortable to set up on Windows) and can’t use a secure communication channel. Another approach is to check-in changed source files into a version control system, like CVS. On host A you check it in and on host B you check it out afterwards. But this means you always need a more or less stable variant of your code, so that other developer can, at least compile, or much better use it. That is not always the case (especially when you leave the office at 11:00 p.m.) and it also doesn’t cover files which aren’t handled by a version control system. Luckily there is a solution for all the problems mentioned which is called Unison. So here comes the second post in the ToolTips series, which covers an easy and portable way for file synchronization.

Installing Unison

Most modern Linux distributions include Unison in their package manage system. On Mac OS X you can use MacPorts. Alternatively you could download a binary version for Mac OS X or Windows here. To prevent surprises and unnecessary trouble it might be a good idea to make sure that every involved system use the same version of Unison. At least on Linux and Mac OS X it is relatively easy to compile Unison from the sources.

Setting up public/private key authentication for ssh

One advantage of Unison over Rsync is that you can use different communication channels for the file transfers. One is ssh. As I always prefer/demand encrypted communication this is a big plus of course. In the default setup you can just use ssh. But for a little bit more comfort I suggest to create a public/private key pair for the authentication.
The following creates public/private keys without a password. Although this is much more easier to use, it should be only used on hosts which are trusted. If you are in doubt, use the normal password approach or even better create a public/private key pair with a password. Create a new public/private key pair with the following command:

user@host-a ~ $ ssh-keygen -t rsa

When you are asked for a password just hit Enter twice. The command creates the private key in ~/.ssh/id_rsa and the public key in ~/.ssh/id_rsa.pub. Now copy the public key to host B:

user@host-a ~ $ scp ~/.ssh/id_rsa.pub host_B:.ssh/authorized_keys

If you already have some public keys on host B, make sure you append the new key and not overwrite the file by the above command. Make the file accessible by the user only with:

user@host-b ~ $ chmod 600 ~/.ssh/authorized_keys

Now you should be able to connect to host B without any interaction needed.

Configuring Unison

Like in the long UNIX tradition, Unison is configured using text files. The files are located in the ~/.unison directory. You can configure more than one synchronization target by choosing a meaningful name. There exists one default target which is configured in the file default.prf. Because I have more than one target I prefer to split the configuration into several files. You can include other project files with the include statement as shown here:

# directory on host a (this is where Unison will be executed)
root       = /mnt/data/projects
# directory on host b (this is the remote host)
root       = ssh://host-b//mnt/data/projects
# which directories to sync?
include projects_files.prf
# options
include options.prf
ignorecase = false
# unison executable on the server
servercmd  = /usr/local/bin/unison

We setup the root directories on both machines, including the configuration file for the project target and some generic option file. We also overwriting the default unison location, because this is a self compiled version. The file options.prf looks like this:

# No staled nfs and mac store files
ignore  = Name .nfs*
ignore  = Name .DS_Store
# options
log     = true
rsrc    = true
auto    = true
#debug   = verbose
#logfile = ~/.unison/unison.log

This just set some generic options which are valid for all my targets. For the specific target projects the file projects_files.prf contains mainly the directories and files which should be ignored:

# No ISOs
ignore    = Path vms/ISO
# Ignore VBox branches
ignore    = Path vbox-*
# No binary output from the other platforms
ignore    = Path vbox*/out/*
# One exception:
ignorenot = Path vbox/out/linux.amd64.additions
# No wine stuff
ignore    = Path vbox*/wine.*
# Tools
ignore    = Name vbox*/tools/{FetchDir,freebsd*,os2*}

So in general, you configure the directory to synchronize and later define directories or files which should be ignored. As you see, you can include or exclude paths as you like. Even simple bash wildcards are possible. As shown in this example I exclude all binary files of a VirtualBox build, because they are useless on another platform. Understanding how Unison decide which directories or files should be synchronized is sometimes difficult. So I suggest to carefully read the documentation and just use the “try and failure” approach ;). Another reason for splitting up the configuration files is you can synchronize these files as well. I have another target which synchronize several configuration files, e.g. .bashrc, .profile, .vim* and the sub-project files of Unison like the projects_files.prf. You can’t synchronize e.g. default.prf, cause the root directories are different from host to host, but the general configuration is always the same. My home target looks like this:

# Which directories/files to sync?
path = .bashrc
path = .ion3
path = .gdbinit
path = .cgdb
path = .valgrind-vbox.supp
path = .vim
path = .vimrc
path = .gvimrc
path = .Xdefaults
path = .gnupg
path = .unison/options.prf
path = .unison/home_files.prf
path = .unison/projects_files.prf
# Do not sync:
ignore = Path .vim/.netrwhist
ignore = Path .ion3/default-session*
ignore = Path .cgdb/readline_history.txt

One of the strengths over other synchronization tools is, you can do this for others host as well. So if you synchronize between host a and host b you can also synchronize between host c and host b. However, a little bit of discipline is necessary. There should be one host which all other host synchronize again.
If you now execute unison the project target will be used. If you execute unison home the files of the home target will be synchronized.

Conclusion

Unison is a very powerful tool. You can synchronize between more than two hosts (OS independent), in a secure way and uni-directional. Currently there is no better tool and I use it on a daily basis.

TT: Console navigation the easy way with Apparix

Today I will start a new series where I present small tools which I use on a daily basis and considered very useful. These tools haven’t to be killer applications, but doing the task they are written for, very well. Therefor also the name of this series: TT, which stands for ToolTips. Most of this applications are open source, so I will take the opportunity to say “Thank you” to all the people out there, which create such cool stuff in there free time.

We start with a tool called Apparix.

Console?

As you may know, I’m a keyboard addict. My preferred OS is Linux, I use Ion3 as the window manager, rxvt-unicode as terminal application and (g)Vim as the preferred editor. On the other side I have a MacBook Pro as laptop and love the possibilities of Mac OS X. How does this come together? Well, all these tools are available on Mac OS X as well. The X11 port of Mac OS X is getting better with each release and using it in fullscreen is good enough. Mac OS X is based on BSD and together with MacPorts, there is nothing I missed until today. As I have to work really platform independent, there is Windows. Vim is available, but working console based (read: command.com) isn’t really much fun. There is of course mingw, but I never managed it to create a work environment like on Linux/UNIX. Anyway, I work not that much on Windows, so I can live with other tools there as well (a really good free editor is Notepad++ and if you hate Explorer have a look at Altap Salamander). So the question remains: Console? I have the feeling using the keyboard within a console application is much faster than clicking around with the mouse. This doesn’t mean I like to type every single character when starting an application or changing the current working directory. Many modern command interpreters, like the Bash, have built in word expansion. This allows you to use the tabulator key for fast expansion of typed characters for a given set of commands. Of course this makes the console life much more easier. But there is more!

Bookmarking important directories

Usually, I’m working on projects. One project is VirtualBox. Other might be some private stuff, like personal letters or other development I do. As the regular user organize his projects in directory trees it make sense to bookmark them. Most people believe bookmarking is something Internet specific, but this isn’t true. Bookmarking means getting access to an important information very fast, without remembering the exact location of this information. This is exactly what Apparix does. It bookmarks directories and make these bookmarks easy accessible on the console. So I have bookmarks for sub-projects within VirtualBox, like the GUI or Main. Getting there is as easy as writing to vgui. It’s the same as writing cd /bin and it even provide word expansion of the available bookmarks. In this case the bookmark is vgui and it expand to the GUI directory within the VirtualBox source tree.

How does this work?

Installing Apparix is pretty simple. You will find all the information on the website. The important part is the integration within your shell. You need to extent your .bashrc. One way is to add the functions you need into .bashrc. Alternatively you can download the file .bash_apparix and source it in your .bashrc. When you have done this and start a new shell, you should navigate to a directory you like to bookmark. There you type bm to create your first bookmark. Afterwards, regardless what your current working directory is, you could execute to DIR_NAME to get there. Of course you can name the bookmark other than the current directory name, by using bm MY_NAME. By executing als all bookmarks are shown. There is much more possible of course. See the documentation for more information. Anyway, this is the main use case for Apparix, so there is not much more to say. Really?

Using the Apparix database for other applications

The cool thing on many Linux applications is, they are saving their data in simple text files. Also Apparix. The bookmark “database” is saved in a file called .apparixrc. It’s a simple text file, which is easy to parse. It looks like this:

j,vbox,/Users/poetzsch/projects/vbox
j,VirtualBox,/Users/poetzsch/projects/vbox/src/VBox/Frontends/VirtualBox
j,Main,/Users/poetzsch/projects/vbox/src/VBox/Main
j,Runtime,/Users/poetzsch/projects/vbox/src/VBox/Runtime
j,Installer,/Users/poetzsch/projects/vbox/src/VBox/Installer/darwin
j,cv,/Users/poetzsch/projects/private/office/latex/cv
j,dow,/Users/poetzsch/Downloads

This allows the usage of this information in other contexts as well. One example is Vim. As I mentioned earlier, this is my main editor. So it would be cool to use the bookmarks, I already have, there as well. To easy change the current directory in Vim, add the following to your .vimrc:

command! -complete=custom,BmCe -nargs=1 To :call BmTo('<args>')
fun BmCe(A,L,P)
 return system("awk 'BEGIN{FS=","}/j,/{print $2}' ~/.apparixrc")
endfun
fun BmLs(dir)
 return system('apparix '.a:dir.' | tr -d "n"')
endfun
fun BmTo(dir)
 execute ':cd '.BmLs(a:dir)
endfun

After restarting Vim, it is possible to navigate with :To MY_BOOKMARK to any directory known by Apparix.

Conclusion

One simple task, getting to a directory where you are often have to be, with one simple solution. Thats all to say about Apparix. It does this in a very simple way, which is fully integrated into the Bash and the saved information could be used by other applications as well.