Sep 7, 2006: Attacking Data Intensive Science with Distributed Computing: Capabilities, Challenges, and Opportunities
Filed in: Colloquium
Dr. Douglas Thain, University of Notre Dame
Distributed computing is critical to many forms of science today. Computing clusters of tens or hundreds of CPUs may be found in many departments of a university, working tirelessly on computationally intensive workloads. However, attacking "grand-challenge" science problems requires a new level of scaling: we must find ways of reliabily harnessing thousands of CPUs with high I/O rates. This is not as easy as it should be: today's large scale systems are very challenging to manage, tune, and debug, and thus inaccessible to ordinary users.
The Cooperative Computing Lab (CCL) at Notre Dame is attacking these problems by designing, implementing, and deploying novel software for running data-intensive workloads at large scales. We will discuss how these tools have been applied to problems in high energy physics, biometrics, and bioinformatics. Finally, we will discuss several open problems in distributed computing that have been revealed by our interactions with scientific users.
Note: This talk will be accessible to a general science and engineering audience. Guests from other departments are quite welcome and encouraged to attend.
Abstract
Distributed computing is critical to many forms of science today. Computing clusters of tens or hundreds of CPUs may be found in many departments of a university, working tirelessly on computationally intensive workloads. However, attacking "grand-challenge" science problems requires a new level of scaling: we must find ways of reliabily harnessing thousands of CPUs with high I/O rates. This is not as easy as it should be: today's large scale systems are very challenging to manage, tune, and debug, and thus inaccessible to ordinary users.
The Cooperative Computing Lab (CCL) at Notre Dame is attacking these problems by designing, implementing, and deploying novel software for running data-intensive workloads at large scales. We will discuss how these tools have been applied to problems in high energy physics, biometrics, and bioinformatics. Finally, we will discuss several open problems in distributed computing that have been revealed by our interactions with scientific users.
Note: This talk will be accessible to a general science and engineering audience. Guests from other departments are quite welcome and encouraged to attend.