wfcommons.wfgen

wfcommons.wfgen.generator

class wfcommons.wfgen.generator.WorkflowGenerator(workflow_recipe: wfcommons.wfgen.abstract_recipe.WorkflowRecipe, logger: Optional[logging.Logger] = None)

Bases: object

A generator of synthetic workflow instances based on workflow recipes obtained from the analysis of real workflow execution instances.

Parameters
  • workflow_recipe (WorkflowRecipe) – The workflow recipe to be used for this generator.

  • logger (Logger) – The logger where to log information/warning or errors (optional).

build_workflow(workflow_name: Optional[str] = None)wfcommons.common.workflow.Workflow

Generate a synthetic workflow instance based on the workflow recipe used to instantiate the generator.

Parameters

workflow_name (str) – The workflow name.

Returns

A synthetic workflow instance object.

Return type

Workflow

build_workflows(num_workflows: int)List[wfcommons.common.workflow.Workflow]

Generate a number of synthetic workflow instances based on the workflow recipe used to instantiate the generator.

Parameters

num_workflows (int) – The number of workflows to be generated.

Returns

A list of synthetic workflow instance objects.

Return type

List[Workflow]

wfcommons.wfgen.abstract_recipe

class wfcommons.wfgen.abstract_recipe.WorkflowRecipe(name: str, data_footprint: Optional[int], num_tasks: Optional[int], exclude_graphs: Set[str] = {}, runtime_factor: Optional[float] = 1.0, input_file_size_factor: Optional[float] = 1.0, output_file_size_factor: Optional[float] = 1.0, logger: Optional[logging.Logger] = None, this_dir=None)

Bases: abc.ABC

An abstract class of workflow recipes for creating synthetic workflow instances.

Parameters
  • name (str) – The workflow recipe name.

  • data_footprint (int) – The upper bound for the workflow total data footprint (in bytes).

  • num_tasks (int) – The upper bound for the total number of tasks in the workflow.

  • runtime_factor (float) – The factor of which tasks runtime will be increased/decreased.

  • input_file_size_factor (float) – The factor of which tasks input files size will be increased/decreased.

  • output_file_size_factor (float) – The factor of which tasks output files size will be increased/decreased.

  • logger (Logger) – The logger where to log information/warning or errors (optional).

_abc_impl = <_abc_data object>
_generate_file(extension: str, recipe: Dict[str, Any], link: wfcommons.common.file.FileLink)wfcommons.common.file.File

Generate a file according to a file recipe.

Parameters
  • extension (str) –

  • recipe (Dict[str, Any]) – Recipe for generating the file.

  • link (FileLink) – Type of file link.

Returns

The generated file.

Return type

File

_generate_files(task_id: str, recipe: Dict[str, Any], link: wfcommons.common.file.FileLink, files_recipe: Optional[Dict[wfcommons.common.file.FileLink, Dict[str, int]]] = None)None

Generate files for a specific task ID.

Parameters
  • task_id (str) – task ID.

  • recipe (Dict[str, Any]) – Recipe for generating the task.

  • link (FileLink) – Type of file link.

  • files_recipe (Dict[FileLink, Dict[str, int]]) – Recipe for generating task files.

_generate_task(task_name: str, task_id: str, input_files: Optional[List[wfcommons.common.file.File]] = None, files_recipe: Optional[Dict[wfcommons.common.file.FileLink, Dict[str, int]]] = None)wfcommons.common.task.Task

Generate a synthetic task.

Parameters
  • task_name (str) – task name.

  • task_id (str) – task ID.

  • input_files (List[File]) – List of input files to be included.

  • files_recipe (Dict[FileLink, Dict[str, int]]) – Recipe for generating task files.

Returns

A task object.

Return type

task

_generate_task_name(prefix: str)str

Generate a task name from a prefix appended with an ID.

Parameters

prefix (str) – task prefix.

Returns

task name from prefix appended with an ID.

Return type

str

_get_files_by_task_and_link(task_id: str, link: wfcommons.common.file.FileLink)List[wfcommons.common.file.File]

Get the list of files for a task ID and link type.

Parameters
  • task_id (str) – task ID.

  • link (FileLink) – Type of file link.

Returns

List of files for a task ID and link type.

Return type

List[File]

_load_base_graph()networkx.classes.digraph.DiGraph
_load_microstructures()Dict
abstract _workflow_recipe()Dict[str, Any]

Recipe for generating synthetic instances for a workflow. Recipes can be generated by using the InstanceAnalyzer.

Returns

A recipe in the form of a dictionary in which keys are task prefixes.

Return type

Dict[str, Any]

build_workflow(workflow_name: Optional[str] = None)wfcommons.common.workflow.Workflow

Generate a synthetic workflow instance.

Parameters

workflow_name (int) – The workflow name

Returns

A synthetic workflow instance object.

Return type

Workflow

abstract classmethod from_num_tasks(num_tasks: int, runtime_factor: Optional[float] = 1.0, input_file_size_factor: Optional[float] = 1.0, output_file_size_factor: Optional[float] = 1.0)wfcommons.wfgen.abstract_recipe.WorkflowRecipe

Instantiate a workflow recipe that will generate synthetic workflows up to the total number of tasks provided.

Parameters
  • num_tasks (int) – The upper bound for the total number of tasks in the workflow.

  • runtime_factor (float) – The factor of which tasks runtime will be increased/decreased.

  • input_file_size_factor (float) – The factor of which tasks input files size will be increased/decreased.

  • output_file_size_factor (float) – The factor of which tasks output files size will be increased/decreased.

Returns

A workflow recipe object that will generate synthetic workflows up to the total number of tasks provided.

Return type

WorkflowRecipe

generate_nx_graph()networkx.classes.digraph.DiGraph