Filesystems for Grid Computing

PI: Douglas Thain, University of Notre Dame

Grid computing systems such as the Open Science Grid and the NSF TeraGrid give users easy access to hundreds or thousands of CPUs at once. However, within computing grids, it is not always easy to access one's data. Traditional filesystems such as NFS and AFS are not usable in most grid computing systems, because they require privileged access to install and use at both client and server side. A user of grid computing rarely has such access.

To remedy this problem, we have designed and implemented a variety of filesystems for grid computing, all based on the Parrot and Chirp software. These user-level tools can be deployed without special privileges into existing grids, and used to access data wherever it may be located. We work directly with users in bioinformatics and high energy physics to design and deploy production filesystem services. You can download and use our software from this page.

Publications

  • Chirp: A Practical Global Filesystem for Cluster and Grid Computing, Douglas Thain, Christopher Moretti, and Jeffrey Hemmes, Journal of Grid Computing, Springer, accepted for publication in 2008. DOI: 10.1007/s10723-008-9100-5. (Preprint Version)
  • Efficient Access to Many Small Files in a Filesystem for Grid Computing, Douglas Thain and Chris Moretti, IEEE Conference on Grid Computing, Austin TX, September 2007.
  • Flexible Object Based Filesystems for Scientific Computing, Christopher M. Moretti, MSCSE Thesis, University of Notre Dame, April 2007.
  • Grid Deployment of Legacy Bioinformatics Applications with Transparent Data Access, Christophe Blanchet, Remi Mollon, Douglas Thain, and Gilbert Deleage, IEEE Conference on Grid Computing, September 2006.
  • Cacheable Decentralized Groups for Grid Resource Access Control, Jeffrey Hemmes and Douglas Thain, IEEE Conference on Grid Computing, September 2006.
  • Operating System Support for Space Allocation in Grid Storage Systems, Douglas Thain, IEEE Conference on Grid Computing, September 2006.
  • Positioning Dynamic Storage Caches for Transient Data, Sudharshan Vazhkudai, Douglas Thain, Xiaosong Ma, and Vincent Freeh, Workshop on High-Performance I/O Techniques and Deployment of Very Large Scale I/O Systems, at IEEE Cluster Computing, September 2006.
  • Using Condor Glide-Ins and Parrot to Move from Dedicated Resources to the Grid, Stefano Belforte, Matthew Normal, Subir Sarkar, Ifor Sfiligoi, Douglas Thain, Frank Wuerthwein, Workshop on Parallel Systems and Algorithms (PASA), Lecture Notes in Informatics (LNI), Volume 81, pages 285-292, 2006.
  • Transparently Distributing CDF Software with Parrot, Douglas Thain, Christopher Moretti, and Igor Sfiligoi, Computing in High Energy Physics, February 2006.
  • The Consequences of Decentralized Security in a Cooperative Storage System, Douglas Thain, Chris Moretti, Paul Madrid, Phil Snowberger, and Jeff Hemmes, IEEE Workshop on Security in Storage (SISW), San Francisco, December 2005.
  • Separating Abstractions from Resources in a Tactical Storage System, Douglas Thain, Sander Klous, Justin Wozniak, Paul Brenner, Aaron Striegel, and Jesus Izaguirre, in Proceedings of IEEE/ACM Supercomputing, Nov 2005.
  • "Parrot: An Application Environment for Data-Intensive Computing", Douglas Thain and Miron Livny, Scalable Computing: Practice and Experience, Volume 6, Number 3, Pages 9--18, 2005.
  • Parrot: Transparent User-Level Middleware for Data-Intensive Computing, Douglas Thain and Miron Livny, Workshop on Adaptive Grid Middleware (AGM), New Orleans, Louisiana, September 2003.
  • "Multiple Bypass: Interposition Agents for Distributed Computing", Douglas Thain and Miron Livny, Journal of Cluster Computing, Volume 4, Pages 39-47, 2001.
  • "Bypass: A tool for building split execution systems", Douglas Thain and Miron Livny, in Proceedings of the Ninth IEEE Symposium on High Performance Distributed Computing, Pittsburg, Pennsylvania, pp 79-85, August 1-4, 2000.
    • Cooperative Computing Lab - CSE Department - Notre Dame