HOWTO: Setup a Chirp Cluster Filesystem

Multiple Chirp servers can be combined together to form a single filesystem with a very large capacity. This is known in various publications as a "unified cluster" or a "DSFS", or a "multi filesystem". This article describes how to set up such a filesystem.

A Chirp cluster filesystem consists of a directory server and one or more data servers. The filesystem will also have a unique "volume name", which can be any name that you choose. Let us suppose that you wish to create a volume titled bigdata that will be stored on the following machines:

dir.somewhere.edu - the directory server

data01.somewhere.edu - a data server

data02.somewhere.edu - a data server

data03.somewhere.edu - a data server

First make sure that you have installed Parrot and Chirp properly, started a Chirp server on each of the directory and data servers, and are able to connect to each Chirp server remotely. Now, using Parrot, set up the directory server by creating a few files as follows:

    % parrot tcsh
    % mkdir /chirp/dir.somewhere.edu/bigdata
    % cd /chirp/dir.somewhere.edu/bigdata
    % mkdir root
    % echo data01.somewhere.edu >> hosts
    % echo data02.somewhere.edu >> hosts
    % echo data03.somewhere.edu >> hosts
    % echo bigdatafiles > key

Now that the directory server is established, you may can access the filesystem through the path /multi/dir.somewhere.edu@bigvol and use it to read and write files, which will be spread across the data servers. For example:

    % parrot tcsh
    % cd /multi/dir.somwhere.edu@bigvol/
    % df .
    % cp -r /tmp/mydata .