Quickstart

WfCommons is available on PyPI. It requires Python 3.11+ and has been tested on Linux and macOS.

Installation

We recommend installing into a virtual environment to keep dependencies isolated:

$ python3 -m venv .venv
$ source .venv/bin/activate
$ python3 -m pip install wfcommons

Verify the installation

Confirm the wfchef CLI entry point is available:

$ wfchef --help

Or check the Python import:

$ python3 -c "import wfcommons; print(wfcommons.__version__)"

Your first synthetic workflow

WfCommons ships with ready-to-use recipes for ten scientific applications (see Bundled Workflow Recipes). Generating a realistic workflow takes a few lines:

import pathlib
from wfcommons.wfchef.recipes import SeismologyRecipe
from wfcommons import WorkflowGenerator

generator = WorkflowGenerator(SeismologyRecipe.from_num_tasks(250))
workflow = generator.build_workflow()
workflow.write_json(pathlib.Path("seismology-workflow.json"))

The resulting JSON file follows WfFormat and can be consumed by any tool or simulator that supports it.

Your first workflow benchmark

The same recipes can also produce runnable benchmarks with controlled CPU and I/O behavior, translated to the workflow system of your choice:

import pathlib
from wfcommons import BlastRecipe
from wfcommons.wfbench import WorkflowBenchmark, BashTranslator

benchmark = WorkflowBenchmark(recipe=BlastRecipe, num_tasks=50)
benchmark.create_benchmark(pathlib.Path("/tmp/"), cpu_work=100, data=10, percent_cpu=0.6)

translator = BashTranslator(benchmark.workflow)
translator.translate(output_folder=pathlib.Path("./bash-wf/"))

See WfBench: Workflow Benchmarks for the full list of supported workflow systems.

Installing from source (latest)

If you want the latest development version (potentially unstable), clone the repository and install locally:

$ git clone https://github.com/wfcommons/wfcommons
$ cd wfcommons
$ python3 -m pip install .

Optional requirements

Visualization support (drawing workflow task graphs and reading/writing DOT files) is available as an extra:

$ python3 -m pip install wfcommons[viz]

Graphviz

WfCommons uses pygraphviz for generating visualizations of the workflow task graph. Building pygraphviz requires the graphviz package (version 2.16 or later) and its development headers. You can install graphviz easily on Linux with your favorite package manager, for example for Debian-based distributions:

$ sudo apt-get install graphviz libgraphviz-dev

and for RedHat-based distributions:

$ sudo yum install python-devel graphviz-devel

On macOS you can use the brew package manager:

$ brew install graphviz

Then install the visualization extra (or pygraphviz directly):

$ python3 -m pip install wfcommons[viz]

Next steps