Tag Archives: X11

TT: Uni-directional file synchronization between different hosts with Unison

When you work with at least two computers on the same project on a daily basis you might have a problem. You need to get changed files from host A to host B and vice versa. The problem getting bigger when you work in addition on different operation systems or use more than two hosts. On UNIX/Linux the preferred tool for such a task is Rsync. Unfortunately Rsync synchronize only in one direction, it doesn’t work very well when more than two hosts are involved (and it isn’t really comfortable to set up on Windows) and can’t use a secure communication channel. Another approach is to check-in changed source files into a version control system, like CVS. On host A you check it in and on host B you check it out afterwards. But this means you always need a more or less stable variant of your code, so that other developer can, at least compile, or much better use it. That is not always the case (especially when you leave the office at 11:00 p.m.) and it also doesn’t cover files which aren’t handled by a version control system. Luckily there is a solution for all the problems mentioned which is called Unison. So here comes the second post in the ToolTips series, which covers an easy and portable way for file synchronization.

Installing Unison

Most modern Linux distributions include Unison in their package manage system. On Mac OS X you can use MacPorts. Alternatively you could download a binary version for Mac OS X or Windows here. To prevent surprises and unnecessary trouble it might be a good idea to make sure that every involved system use the same version of Unison. At least on Linux and Mac OS X it is relatively easy to compile Unison from the sources.

Setting up public/private key authentication for ssh

One advantage of Unison over Rsync is that you can use different communication channels for the file transfers. One is ssh. As I always prefer/demand encrypted communication this is a big plus of course. In the default setup you can just use ssh. But for a little bit more comfort I suggest to create a public/private key pair for the authentication.
The following creates public/private keys without a password. Although this is much more easier to use, it should be only used on hosts which are trusted. If you are in doubt, use the normal password approach or even better create a public/private key pair with a password. Create a new public/private key pair with the following command:

user@host-a ~ $ ssh-keygen -t rsa

When you are asked for a password just hit Enter twice. The command creates the private key in ~/.ssh/id_rsa and the public key in ~/.ssh/id_rsa.pub. Now copy the public key to host B:

user@host-a ~ $ scp ~/.ssh/id_rsa.pub host_B:.ssh/authorized_keys

If you already have some public keys on host B, make sure you append the new key and not overwrite the file by the above command. Make the file accessible by the user only with:

user@host-b ~ $ chmod 600 ~/.ssh/authorized_keys

Now you should be able to connect to host B without any interaction needed.

Configuring Unison

Like in the long UNIX tradition, Unison is configured using text files. The files are located in the ~/.unison directory. You can configure more than one synchronization target by choosing a meaningful name. There exists one default target which is configured in the file default.prf. Because I have more than one target I prefer to split the configuration into several files. You can include other project files with the include statement as shown here:

# directory on host a (this is where Unison will be executed)
root       = /mnt/data/projects
# directory on host b (this is the remote host)
root       = ssh://host-b//mnt/data/projects
# which directories to sync?
include projects_files.prf
# options
include options.prf
ignorecase = false
# unison executable on the server
servercmd  = /usr/local/bin/unison

We setup the root directories on both machines, including the configuration file for the project target and some generic option file. We also overwriting the default unison location, because this is a self compiled version. The file options.prf looks like this:

# No staled nfs and mac store files
ignore  = Name .nfs*
ignore  = Name .DS_Store
# options
log     = true
rsrc    = true
auto    = true
#debug   = verbose
#logfile = ~/.unison/unison.log

This just set some generic options which are valid for all my targets. For the specific target projects the file projects_files.prf contains mainly the directories and files which should be ignored:

# No ISOs
ignore    = Path vms/ISO
# Ignore VBox branches
ignore    = Path vbox-*
# No binary output from the other platforms
ignore    = Path vbox*/out/*
# One exception:
ignorenot = Path vbox/out/linux.amd64.additions
# No wine stuff
ignore    = Path vbox*/wine.*
# Tools
ignore    = Name vbox*/tools/{FetchDir,freebsd*,os2*}

So in general, you configure the directory to synchronize and later define directories or files which should be ignored. As you see, you can include or exclude paths as you like. Even simple bash wildcards are possible. As shown in this example I exclude all binary files of a VirtualBox build, because they are useless on another platform. Understanding how Unison decide which directories or files should be synchronized is sometimes difficult. So I suggest to carefully read the documentation and just use the “try and failure” approach ;). Another reason for splitting up the configuration files is you can synchronize these files as well. I have another target which synchronize several configuration files, e.g. .bashrc, .profile, .vim* and the sub-project files of Unison like the projects_files.prf. You can’t synchronize e.g. default.prf, cause the root directories are different from host to host, but the general configuration is always the same. My home target looks like this:

# Which directories/files to sync?
path = .bashrc
path = .ion3
path = .gdbinit
path = .cgdb
path = .valgrind-vbox.supp
path = .vim
path = .vimrc
path = .gvimrc
path = .Xdefaults
path = .gnupg
path = .unison/options.prf
path = .unison/home_files.prf
path = .unison/projects_files.prf
# Do not sync:
ignore = Path .vim/.netrwhist
ignore = Path .ion3/default-session*
ignore = Path .cgdb/readline_history.txt

One of the strengths over other synchronization tools is, you can do this for others host as well. So if you synchronize between host a and host b you can also synchronize between host c and host b. However, a little bit of discipline is necessary. There should be one host which all other host synchronize again.
If you now execute unison the project target will be used. If you execute unison home the files of the home target will be synchronized.

Conclusion

Unison is a very powerful tool. You can synchronize between more than two hosts (OS independent), in a secure way and uni-directional. Currently there is no better tool and I use it on a daily basis.

Creating file shortcuts on three different operation systems

As you may know, developing for multiple platforms is one of my strengths. Strictly speaking, it’s a basic requirement if you are involved in such a product like VirtualBox, which runs on every major (and several minor) platform available today. Beside the GUI, which uses Qt and therewith is portable without any additional cost (which isn’t fully true if you want real native look and feel on every platform, especially on Mac OS X), all the rest of VirtualBox is written in a portable way. This is done by using only C/C++ and Assembler when necessary. Everything which needs a different approach, because of the design of the OS (and the API’s which are available there), is implemented in a platform dependent way. In the history of VirtualBox, several modules are created and grown by the time, which makes it really easy to deal with this differences. For stuff like file handling, paths, strings, semaphores or any other basic functionality, you can just use the modules which are available. On the other side it might be necessary, for a new feature we implement, to write it from the ground. In the following post I will show how to create a file shortcut for the three major operation systems available today.

Why do you want to use file shortcuts

On the classical UNIX systems you have hard and soft links. These are implemented by the filesystem and make it possible to link to another file or folder without any trouble. Most of the time soft links are used, but it really depends on the use case. Unfortunately these kind of links are not available on Windows (yes, I know there are also hard links and junctions on NTFS, but they are not common and difficult to handle), these links doesn’t allow any additional attributes. For example one like to add a different icon to the link or provide more information through a comment field. Beside on Mac OS X, shortcuts can also be work as an application launcher, where the link contain the information what application should be started and how. In contrast to filesystem links which are handled by the operation system, these shortcuts are handled by the window system (or shell) running on the host (which doesn’t mean there is no filesystem support for it). On Windows this is the Explorer, on Mac OS X the Finder and on Linux a freedesktop.org conforming file manager.

Creating a Desktop file on Linux

Desktop files on Linux (or any other UNIX system which conforms to freedesktop.org) is easy. It’s a simple text file which implement the Desktop Entry Specification. In version 1.0 there are 18 possible entries, where not all of them are mandatory. In the following example I use Qt to write these files, but it should be no problem to use any other toolkit or plain C.

bool createShortcut(const QString &strSrcFile,
                    const QString &strDstPath,
                    const QString &strName)
{
 QFile link(strDstPath + QDir::separator() + strName + ".desktop");
 if (link.open(QFile::WriteOnly | QFile::Truncate))
 {
  QTextStream out(&link);
  out.setCodec("UTF-8");
  out << "[Desktop Entry]" << endl
      << "Encoding=UTF-8" << endl
      << "Version=1.0" << endl
      << "Type=Link" << endl
      << "Name=" << strName << endl
      << "URL=" << strSrcFile << endl
      << "Icon=icon-name" << endl;
  return true;
 }
 return false;
}

Replace icon-name by a registered icon on the system and you are done.

Creating a Shell link on Windows

Windows provides an interface for IShellLink since Windows XP. The following example shows how to use it:

bool createShortcut(LPCSTR lpszSrcFile,
                    LPCSTR lpszDstPath,
                    LPCSTR lpszName)
{
 IShellLink *pShl = NULL;
 IPersistFile *pPPF = NULL;
 HRESULT rc = CoCreateInstance(CLSID_ShellLink,
                               NULL,
                               CLSCTX_INPROC_SERVER,
                               IID_IShellLink,
                               (void**)(&pShl));
 if (FAILED(rc))
  return false;
 do
 {
  rc = pShl->SetPath(lpszSrcFile);
  if (FAILED(rc))
   break;
  rc = pShl->QueryInterface(IID_IPersistFile, (void**)&pPPF);
  if (FAILED(rc))
   break;
  WORD wsz[MAX_PATH];
  TCHAR path[MAX_PATH] = { 0 };
  lstrcat(path, lpszDstPath);
  lstrcat(path, "\");
  lstrcat(path, lpszName);
  lstrcat(path, ".lnk");
  MultiByteToWideChar(CP_ACP, 0, buf, -1, wsz, MAX_PATH);
  rc = pPPF->Save(wsz, TRUE);
 } while(0);
 if (pPPF)
  pPPF->Release();
 if (pShl)
  pShl->Release();
 return SUCCEEDED(rc);
}

As you may noticed this uses COM. Many API’s on Windows using the COM interface to communicate between processes. If you don’t use COM in your application you have to initialize it first. This is achieved by adding the following call to the front of the function:

 if (FAILED(CoInitialize(NULL))
  return false;

Depending on your application it might be worth to unitialize COM after usage by appending the following to the function:

 CoUninitialize();

The function itself isn’t any magic. It gets a COM interface to the IShellLink interface and then work with it, by setting the source path and adding a target path by using the IPersistFile interface. As I wrote before you could do much more. Providing a path to a specific application or adding your own parameters is no problem. Have a look at the documentation.

Creating an Alias file on Mac OS X

Shortcut files on Mac OS X are a little bit different. At first, they aren’t one. There are the classical filesystem links and Alias files. Alias files are links which targeting a specific file, but they haven’t all the possibilities of shortcuts like on Windows or Linux. As the name suggest they are really only an alias for another file or directory. So specifying an application to start or things like that aren’t possible. Anyway they allow changing the icon and they are more persistent than on Window or Linux cause they are working with several attributes of the target file. Even if you rename or move the target, an Alias file will resolve the target correctly (if it is possible). On the other side, being such special means also being hard to create. In principle there are two possibilities. The first one is, creating a file which is no file at all, but has several resources forks attached. Therefor you need to know exactly how Alias files are built of and make sure with every release of Mac OS X you are following the development. There is a free project which does exactly that: NDAlias. If you are like me and a little bit more lazy, you ask someone who should know how to create Alias files. This is Finder. Although writing the files itself isn’t easy, asking the Finder to do the job is not really easier, cause the information about doing exactly that are really rare. The following code shows how to achieve it:

bool createShortcut(NSString *pstrSrcFile,
                    NSString *pstrDstPath,
                    NSString *pstrName)
{
 /* First of all we need to figure out which process Id the Finder
  * currently has. */
 NSWorkspace *pWS = [NSWorkspace sharedWorkspace];
 NSArray *pApps = [pWS launchedApplications];
 bool fFFound = false;
 ProcessSerialNumber psn;
 for (NSDictionary *pDict in pApps)
 {
  if ([[pDict valueForKey:@"NSApplicationBundleIdentifier"]
         isEqualToString:@"com.apple.finder"])
  {
   psn.highLongOfPSN = [[pDict
                          valueForKey:@"NSApplicationProcessSerialNumberHigh"] intValue];
   psn.lowLongOfPSN  = [[pDict
                          valueForKey:@"NSApplicationProcessSerialNumberLow"] intValue];
   fFFound = true;
   break;
  }
 }
 if (!fFFound)
  return false;
 /* Now the event fun begins. */
 OSErr err = noErr;
 AliasHandle hSrcAlias = 0;
 AliasHandle hDstAlias = 0;
 do
 {
  /* Create a descriptor which contains the target psn. */
  NSAppleEventDescriptor *finderPSNDesc = [NSAppleEventDescriptor
                                            descriptorWithDescriptorType:typeProcessSerialNumber
                                            bytes:&psn
                                            length:sizeof(psn)];
  if (!finderPSNDesc)
   break;
  /* Create the Apple event descriptor which points to the Finder
   * target already. */
  NSAppleEventDescriptor *finderEventDesc = [NSAppleEventDescriptor
                                              appleEventWithEventClass:kAECoreSuite
                                              eventID:kAECreateElement
                                              argetDescriptor:finderPSNDesc
                                              returnID:kAutoGenerateReturnID
                                              transactionID:kAnyTransactionID];
  if (!finderEventDesc)
   break;
  /* Create and add an event type descriptor: Alias */
  NSAppleEventDescriptor *osTypeDesc = [NSAppleEventDescriptor descriptorWithTypeCode:typeAlias];
  if (!osTypeDesc)
   break;
  [finderEventDesc setParamDescriptor:osTypeDesc forKeyword:keyAEObjectClass];
  /* Now create the source Alias, which will be attached to the event. */
  err = FSNewAliasFromPath(nil, [pstrSrcFile fileSystemRepresentation], 0, &hSrcAlias, 0);
  if (err != noErr)
   break;
  char handleState;
  handleState = HGetState((Handle)hSrcAlias);
  HLock((Handle)hSrcAlias);
  NSAppleEventDescriptor *srcAliasDesc = [NSAppleEventDescriptor
                                           descriptorWithDescriptorType:typeAlias
                                           bytes:*hSrcAlias
                                           length:GetAliasSize(hSrcAlias)];
  if (!srcAliasDesc)
   break;
  [finderEventDesc setParamDescriptor:srcAliasDesc
    forKeyword:keyASPrepositionTo];
  HSetState((Handle)hSrcAlias, handleState);
  /* Next create the target Alias and attach it to the event. */
  err = FSNewAliasFromPath(nil, [pstrDstPath fileSystemRepresentation], 0, &hDstAlias, 0);
  if (err != noErr)
   break;
  handleState = HGetState((Handle)hDstAlias);
  HLock((Handle)hDstAlias);
  NSAppleEventDescriptor *dstAliasDesc = [NSAppleEventDescriptor
                                           descriptorWithDescriptorType:t ypeAlias
                                           bytes:*hDstAlias
                                           length:GetAliasSize(hDstAlias)];
  if (!dstAliasDesc)
   break;
  [finderEventDesc setParamDescriptor:dstAliasDesc
    forKeyword:keyAEInsertHere];
  HSetState((Handle)hDstAlias, handleState);
  /* Finally a property descriptor containing the target
   * Alias name. */
  NSAppleEventDescriptor *finderPropDesc = [NSAppleEventDescriptor recordDescriptor];
  if (!finderPropDesc)
   break;
  [finderPropDesc setDescriptor:[NSAppleEventDescriptor descriptorWithString:pstrName]
    forKeyword:keyAEName];
  [finderEventDesc setParamDescriptor:finderPropDesc forKeyword:keyAEPropData];
  /* Now send the event to the Finder. */
  err = AESend([finderEventDesc aeDesc],
               NULL,
               kAENoReply,
               kAENormalPriority,
               kNoTimeOut,
               0,
               nil);
 } while(0);
 /* Cleanup */
 if (hSrcAlias)
  DisposeHandle((Handle)hSrcAlias);
 if (hDstAlias)
  DisposeHandle((Handle)hDstAlias);
 return err == noErr ? true : false;
}

Although the code above looks a little bit scary, it does not much. It fetch the process serial number of the current Finder process, creates an Application event for creating an Alias file and send this event to the Finder.

Conclusion

Beside showing how to create file shortcuts on different platforms, this article also shows which work is necessary to create platform independent code. It’s a simple example. But it also makes clear that one simple solution for platform one, not necessarily mean it’s such simple on platform two.

Making this easy accessible to any developer is the next step. I will leave this exercise to the reader, but have a look at the platform code of the VirtualBox GUI and the corresponding Makefile.

Getting the backtrace from a kernel panic

You may know the following situation. You arrive in the morning in the office, do what you always do and check out the latest changes of the software you are working on. After a little bit of compile time and the first coffee you start the just build application. Bumm, kernel panic. After rebooting and locking through the changes you may have an idea what the reason for this could be. A colleague of you is working on a fancy new feature which needed changes to a kernel module. As you almost know nothing about this code you seek for help and, as it of course not happen on his computer, he is asking for a backtrace of this panic. You have two problems now. First you need to see the panic yourself and second it would be nice to get a copy of the backtrace for sharing this info within a bugtracker. In the following post I will show how both aims could be easily archived.

Let the kernel manage the graphical modes

As most people are working under X11 they don’t see the output of an kernel panic. When a kernel panic happens the kernel prints the reason for the panic and a kernel backtrace to the console window and stops immediately its own execution. It is not written into a log file or somewhere else. In consequence you don’t have the ability to look into the panic text, cause the graphical mode is still on. Historically the mode settings are done by the graphic driver of the X11 system. So the kernel has no idea that or which graphic mode is currently in use. Fortunately the kernel hackers invented a new infrastructure which let the kernel do the mode switch. This subsystem is called Kernel-Mode-Settings (KMS). As the kernel do the mode settings, he can switch back to the console on a panic, regardless which graphical mode is currently configured. Beside this, KMS has other improvements like Fast User Switching or a flicker free switch between text and graphic mode. On the other side is this highly hardware dependent and even if it was introduced with version 2.6.28, not all today available hardware can make use of it. If you are an owner of an Intel graphic card you are in good shape. Radeon and NVidia cards have limited support through the in kernel drivers radeonhd and nouveau. For an Intel i915 card you need to enable the following kernel options:

CONFIG_DRM_I915=y
Location:
-> Device Drivers
-> Graphics support
-> Direct Rendering Manager (XFree86 4.1.0 and higher DRI support) (DRM [=y])
-> Intel 830M, 845G, 852GM, 855GM, 865G ( [=y])

CONFIG_DRM_I915_KMS=y
Location:
-> Device Drivers
-> Graphics support
-> Direct Rendering Manager (XFree86 4.1.0 and higher DRI support) (DRM [=y])
-> Intel 830M, 845G, 852GM, 855GM, 865G ( [=y])
-> i915 driver (DRM_I915 [=y])

The kernel line in your favorite boot loader needs the following additional parameter:

i915.modeset=1

X11 should have this minimal configuration for the device section:

Section "Device"
 Identifier    "i915"
 Driver        "intel"
 Option        "DRI"   "true"
EndSection

Please note that you need of course some recent kernel, X11 version and Intel X11 driver to make this work. After a compile, install and boot of the new kernel, KMS should be in use. You will notice it, cause the boot messages will be printed in a much higher graphical resolution, than the usual text mode provide. The next time a kernel panic occurs, the kernel will switch back to the console before the panic is printed. This allows you to see the info printed and maybe you get a useful hint for the reason of the panic.

Post the panic

If you can’t use KMS or don’t want transcribe the panic text by hand into the bugtracker, it would be nice if the text could be made available on another computer. Kernel hackers usual use the serial port for that. Unfortunately most modern computers doesn’t have such a serial port anymore. Also you need two hosts with a serial port and the setup is complex (you have to know about baud-rates, parity and stuff like this). But there is a simpler solution: netconsole. Netconsole is a kernel module, which sends kernel messages anywhere to the net using UDP. The setup is really simple. In the kernel configuration you need the following setting:

CONFIG_NETCONSOLE=m
Location:
-> Device Drivers
-> Network device support (NETDEVICES [=y])

I prefer to compile it as module, which allows me to turn it on only when I need it. Load it with the following command:

modprobe netconsole netconsole=@/,@192.168.220.10/

The ip has to be replaced by the one of your target computer. You can of course tune it much more, like setting source and target ports or even let netconsole send the text to more than one host. On your client you need a network tool which can read from a socket and print the read text to stdout. Netcat or nc are two tools which are able to do just that. The call for nc looks like the following:

nc -l -u 6666

Now if a kernel panic will happen you will see an output like this:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [] rb_erase+0x15c/0x320
PGD 6942f067 PUD a1e4067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/virtual/block/md1/dev
CPU 3
Modules linked in: vboxnetadp vboxnetflt vboxdrv netconsole ...

Pid: 18887, comm: VirtualBox Tainted: G        W   2.6.36-gentoo #4 DG33TL/
RIP: 0010:[]  [] rb_erase+0x15c/0x320
RSP: 0018:ffff8800b430db58  EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff880069557a68 RCX: 0000000000000001
RDX: ffff880069557a68 RSI: ffff880001d8ed58 RDI: 0000000000000000
RBP: ffff8800b430db68 R08: 0000000000000001 R09: 000000008edcb5d6
R10: 0000000000000000 R11: 0000000000000202 R12: ffff880001d8ed58
R13: 0000000000000000 R14: 000000000000ed00 R15: 0000000000000002
FS:  00007fffde457710(0000) GS:ffff880001d80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000064f9000 CR4: 00000000000026e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process VirtualBox (pid: 18887, threadinfo ffff8800b430c000, task ffff880091e227f0)
Stack:
 ffff88000a03ba18 ffff880001d8ed48 ffff8800b430dba8 ffffffff8105bf06
<0> ffff8800b430dba8 ffffffff8105c97c ffff8800b430dbc8 ffff88000a03ba18
<0> 00004ff8a86ba455 ffff880001d8ed48 ffff8800b430dc48 ffffffff8105ce77
Call Trace:
 [] __remove_hrtimer+0x36/0xb0
 [] ? lock_hrtimer_base+0x2c/0x60
 [] __hrtimer_start_range_ns+0x2b7/0x3c0
 [] ? rtR0SemEventMultiLnxWait+0x250/0x3d0 [vboxdrv]
 [] ? RTLogLoggerExV+0x12f/0x180 [vboxdrv]
 [] hrtimer_start+0x13/0x20
 [] rtTimerLnxStartSubTimer+0x60/0x120 [vboxdrv]
 [] rtTimerLnxStartOnSpecificCpu+0x21/0x30 [vboxdrv]
 [] rtmpLinuxWrapper+0x23/0x30 [vboxdrv]
 [] RTMpOnSpecific+0x99/0xa0 [vboxdrv]
 [] ? rtTimerLnxStartOnSpecificCpu+0x0/0x30 [vboxdrv]
 [] RTTimerStart+0x2a6/0x2e0 [vboxdrv]
 [] ? g_abExecMemory+0x33665/0x180000 [vboxdrv]
 [] g_abExecMemory+0xc678/0x180000 [vboxdrv]
 [] g_abExecMemory+0x328d7/0x180000 [vboxdrv]
 [] supdrvIOCtlFast+0x6a/0x70 [vboxdrv]
 [] VBoxDrvLinuxIOCtl+0x47/0x1e0 [vboxdrv]
 [] ? pick_next_task_fair+0xde/0x150
 [] do_vfs_ioctl+0xa1/0x590
 [] ? sys_futex+0x76/0x170
 [] sys_ioctl+0x4a/0x80
 [] system_call_fastpath+0x16/0x1b
Code: 07 a8 01 75 9d eb 81 0f 1f 84 00 00 00 00 00 48 3b 78 10 0f 84 ...
RIP  [] rb_erase+0x15c/0x320
 RSP
CR2: 0000000000000000
---[ end trace 4eaa2a86a8e2da24 ]---

Normally only kernel panics are sent to the console. You can increase the verbosity level by executing dmesg -n 8 as root.

Conclusion

To continue with the story from the beginning: With the shown methods you can hope your colleague get enough information to find the reason for the kernel panic. To be more helpful, the next step would be to try to debug the problem yourself. Even if the KGDB was merged into the kernel in version 2.6.35, it is not really usable for me. The reason is that it seems kernel hackers usually have really old hardware which either has a serial port, a PS/2 keyboard or both. Otherwise I can’t find a reason why USB keyboards don’t work. I asked on the mailing list of KGDB about the status of USB keyboard support and I can only hope support will be integrated soon.