CCL Home

Research

Software Community Operations
Makeflow Logo

Makeflow = Make + Workflow

Makeflow is a workflow engine for executing large complex workflows on clusters, clouds, and grids. Makeflow is very similar to traditional Make, so if you can write a Makefile, then you can write a Makeflow.

A workflow can be just a few commands chained together, or it can be a complex application consisting of thousands of tasks. It can have an arbitrary DAG structure and is not limited to specific patterns. Makeflow is used to represent complex applications in fields such as data mining, high energy physics, image processing, and bioinformatics.

Makeflow is portable. A workflow is written in a technology neutral way, and then can be deployed to a variety of different systems without modification, including local execution on a single multicore machine as well as batch systems like HTCondor, SGE, Torque or the bundled Work Queue system. The same specification works for all systems, so you can easily grow your application from one machine up to thousands.

Makeflow differs from other distributed and parallel make tools in that it does not require a distributed filesystem. You can use it to harness whatever machines you have available, and Makeflow handles the data transfer and caching. In addition, Makeflow is highly fault tolerant: it can crash or be killed, and upon resuming, will reconnect to running jobs and continue where it left off by making use of a transaction log.

For More Information

  • Makeflow User's Manual
  • Makeflow Tutorial Slides
  • Download Makeflow
  • Getting Help with Makeflow
  • Publications

    (Showing papers with tag makeflow. See all papers instead.)