WfChef: Workflow Recipes

WfChef is the WfCommons component that automates the construction of synthetic workflow generators — no manual modeling, and no code to write. The input to this component is a set of real workflow instances described in the WfFormat (e.g., instances available in Loading Workflow Instances). WfChef automatically analyzes the set of real workflow instances for two purposes:

  1. It discovers workflow subgraphs that represent fundamental task dependency patterns of the application.

  2. It derives statistical models of the workflow tasks’ performance characteristics (see WfInstances: Workflow Instances).

WfChef then outputs a recipe that is used by WfGen (see WfGen: Generating Workflows) to generate realistic synthetic workflow instances with any arbitrary number of tasks.

Workflow Recipes

A workflow recipe is a data structure that encodes the discovered pattern occurrences as well as the statistical models of workflow task characteristics. More precisely, a recipe embodies results from statistical analysis and distribution fitting performed for each workflow task type so as to characterize task runtime and input/output data sizes. The recipe also incorporates information regarding the graph structure of the workflows (task dependencies and frequency of occurrences), automatically derived from the analysis of the workflow instances.

This Python package already provides several workflow recipes (see Bundled Workflow Recipes) for generating realistic synthetic workflow instances — if one of those covers your application, you can skip straight to WfGen: Generating Workflows.

Creating a Recipe from Real Instances

To create a recipe, WfChef analyzes the real workflow graphs in order to identify subgraphs that represent fundamental task dependency patterns. Based on the identified subgraphs and on measured task type frequencies in the real workflows, WfChef outputs a generator that can produce realistic synthetic workflow instances with an arbitrary number of tasks (see WfGen: Generating Workflows).

Recipes are created with the wfchef command-line tool, which is installed with the package. The example below creates a recipe for the Epigenomics application from a folder of real instances in WfFormat:

$ wfchef create /path/to/real/instances -o ./epigenomics -v --name Epigenomics

The following flags can be used with this command:

  • -o or --out (required): name of the directory to be created that will contain the recipe.

  • -n or --name (required): name of the recipe. Typically, the format used is ApplicationName.

  • -v or --verbose: if set, activates status messages.

  • --no-install: if set, does not install the recipe automatically.

  • -r or --runs: number of runs used to compute the mean RMSE of the fitted models (default: 1).

  • -c or --cutoff: only consider real instances with at most this number of tasks when creating the recipe (default: 4000).

Example: with --cutoff 4000, all real instances considered for the creation of the recipe will have 4000 or fewer tasks. This is a useful flag when there is confidence that all patterns present in the application can already be found in the smaller instances (recipe creation is then faster).

Installing and Managing Recipes

Workflow recipes are automatically installed (as small Python packages) and can then be used throughout the system. WfCommons creates a Python package in the directory specified by the --out flag, in which the setup.py and recipe.py files are stored. If the --no-install flag was set when creating the recipe, you will need to manually install the package before using it:

# installing the recipe package
$ pip install /path/to/the/package

To list the recipes installed in a system (and how to import them):

$ wfchef ls

To uninstall a recipe:

$ wfchef uninstall -n recipe_name

Using a Recipe

Once installed, a recipe is imported like any of the bundled ones and handed to the WorkflowGenerator class:

from wfcommons.wfchef.recipes import EpigenomicsRecipe
from wfcommons import WorkflowGenerator

generator = WorkflowGenerator(EpigenomicsRecipe.from_num_tasks(1000))
workflow = generator.build_workflow()

See WfGen: Generating Workflows for the full generation guide.