Work Queue: A Scalable Master/Worker Framework
Work Queue is a framework for building large master-worker applications that span many computers including clusters, clouds, and grids. Work Queue applications are written in C, Perl, or Python using a simple API that allows users to define tasks, submit them to the queue, and wait for completion. Tasks are executed by a standard worker process that can run on any available machine. Each worker calls home to the master process, arranges for data transfer, and executes the tasks. Together, the master and worker handle a wide variety of failures, allowing for dynamically scalable and robust applications.
Work Queue has been used to write many applications that
scale up to hundreds or thousands of processors. These include
the SAND genome assembler, the Makeflow workflow engine,
and the All-Pairs and Wavefront abstractions.
The framework is easy to use, and is currently use to teach parallel programming techniques
in the Parallel Programming, Distributed Systems, and Programming Paradigms class at Notre Dame.
For More Information
Publications
(Showing papers with tag workqueue. See all papers instead.)
- Dinesh Rajan, Andrew Thrasher, Badi Abdul-Wahid, Jesus A Izaguirre, Scott Emrich, and Douglas Thain,
Case Studies in Designing Elastic Applications,
13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), May, 2013. - Christopher Moretti, Andrew Thrasher, Li Yu, Michael Olson, Scott Emrich, and Douglas Thain,
A Framework for Scalable Genome Assembly on Clusters, Clouds, and Grids,
IEEE Transactions on Parallel and Distributed Systems, 23(12), December, 2012. DOI: 10.1109/TPDS.2012.80 - Badi Abdul-Wahid, Li Yu, Dinesh Rajan, Haoyun Feng, Eric Darve, Douglas Thain, Jesus A. Izaguirre,
Folding Proteins at 500 ns/hour with Work Queue,
8th IEEE International Conference on eScience (eScience 2012), October, 2012. - Andrew Thrasher, Zachary Musgrave, Douglas Thain, Scott Emrich,
Shifting the Bioinformatics Computing Paradigm: A Case Study in Parallelizing Genome Annotation Using Maker and Work Queue,
IEEE International Conference on Computational Advances in Bio and Medical Sciences, February, 2012. - Dinesh Rajan, Anthony Canino, Jesus A Izaguirre, and Douglas Thain,
Converting a High Performance Application to an Elastic Cloud Application,
The 3rd IEEE International Conference on Cloud Computing Technology and Science (CloudCom 2011), November, 2011. - Peter Bui, Dinesh Rajan, Badi Abdul-Wahid, Jesus Izaguirre, Douglas Thain,
Work Queue + Python: A Framework For Scalable Scientific Ensemble Applications,
Workshop on Python for High Performance and Scientific Computing (PyHPC) at the ACM/IEEE International Conference for High Performance Computing, Networking, Storage, and Analysis (Supercomputing) , November, 2011. - Irena Lanc, Peter Bui, Douglas Thain, and Scott Emrich,
Adapting Bioinformatics Applications for Heterogeneous Systems: A Case Study,
Emerging Computational Methods for the Life Sciences Workshop at ACM HPDC, pages 7-13, June, 2011. DOI: 10.1145/1996023.1996025 - Li Yu, Christopher Moretti, Andrew Thrasher, Scott Emrich, Kenneth Judd, and Douglas Thain,
Harnessing Parallelism in Multicore Clusters with the All-Pairs, Wavefront, and Makeflow Abstractions,
Journal of Cluster Computing, 13(3), pages 243-256, September, 2010. DOI: 10.1007/s10586-010-0134-7 - Douglas Thain and Christopher Moretti,
Abstractions for Cloud Computing with Condor,
Syed Ahson and Mohammad Ilyas, Cloud Computing and Software Services: Theory and Techniques, pages 153-171, CRC Press, July, 2010. ISBN: 9781439803158 - Peter Bui, Li Yu and Douglas Thain,
Weaver: Integrating Distributed Computing Abstractions into Scientific Workflows using Python,
Challenges of Large Applications in Distributed Environments at ACM HPDC 2010, June, 2010. DOI: 10.1145/1851476.1851570 - Christopher Moretti, Michael Olson, Scott Emrich, and Douglas Thain,
Highly Scalable Genome Assembly on Campus Grids,
Many-Task Computing on Grids and Supercomputers (MTAGS), November, 2009. DOI: 10.1145/1646468.1646480 - Christopher Moretti, Michael Olson, Scott Emrich, and Douglas Thain,
Scalable Modular Genome Assembly on Campus Grids,
University of Notre Dame, Computer Science and Engineering Department, Technical Report 2009-04, July, 2009. - Li Yu, Christopher Moretti, Scott Emrich, Kenneth Judd, and Douglas Thain,
Harnessing Parallelism in Multicore Clusters with the All-Pairs and Wavefront Abstractions,
IEEE High Performance Distributed Computing, pages 1-10, June, 2009. DOI: 10.1145/1551609.1551613