wfcommons.wfgen

wfcommons.wfgen.generator

class wfcommons.wfgen.generator.WorkflowGenerator(workflow_recipe: WorkflowRecipe, logger: Logger | None = None)

Bases: object

A generator of synthetic workflow instances based on workflow recipes obtained from the analysis of real workflow execution instances.

Parameters:

workflow_recipe (WorkflowRecipe) – The workflow recipe to be used for this generator.
logger (Optional[Logger]) – The logger where to log information/warning or errors (optional).

build_workflow(workflow_name: str | None = None) → Workflow

Generate a synthetic workflow instance based on the workflow recipe used to instantiate the generator.

Parameters:: workflow_name (Optional[str]) – The workflow name.
Returns:: A synthetic workflow instance object.
Return type:: Workflow

build_workflows(num_workflows: int) → List[Workflow]

Generate a number of synthetic workflow instances based on the workflow recipe used to instantiate the generator.

Parameters:: num_workflows (int) – The number of workflows to be generated.
Returns:: A list of synthetic workflow instance objects.
Return type:: List[Workflow]

wfcommons.wfgen.abstract_recipe

Bases: ABC

An abstract class of workflow recipes for creating synthetic workflow instances.

Parameters:

name (str) – The workflow recipe name.
data_footprint (Optional[int]) – The upper bound for the workflow total data footprint (in bytes).
num_tasks (Optional[int]) – The upper bound for the total number of tasks in the workflow.
runtime_factor (Optional[float]) – The factor of which tasks runtime will be increased/decreased.
input_file_size_factor (Optional[float]) – The factor of which tasks input files size will be increased/decreased.
output_file_size_factor (Optional[float]) – The factor of which tasks output files size will be increased/decreased.
logger (Optional[Logger]) – The logger where to log information/warning or errors (optional).

_abc_impl = <_abc._abc_data object>

_generate_file(extension: str, recipe: Dict[str, Any], link: FileLink) → File

Generate a file according to a file recipe.

Parameters:

extension (str) –
recipe (Dict[str, Any]) – Recipe for generating the file.
link (FileLink) – Type of file link.

Returns:

The generated file.

Return type:

File

_generate_files(task_id: str, recipe: Dict[str, Any], link: FileLink) → List[File]

Generate files for a specific task ID.

Parameters:

task_id (str) – task ID.
recipe (Dict[str, Any]) – Recipe for generating the task.
link (FileLink) – Type of file link.

Returns:

List of files.

Return type:

List[File]

_generate_task(task_name: str, task_id: str) → Task

Generate a synthetic task.

Parameters:

task_name (str) – task name.
task_id (str) – task ID.

Returns:

A task object.

Return type:

task

_generate_task_files(task: Task) → List[File]

Generate input and output files for a task.

Parameters:: task (Task) – task object.
Returns:: List of files output files.
Return type:: List[File]

_generate_task_name(prefix: str) → str

Generate a task name from a prefix appended with an ID.

Parameters:: prefix (str) – task prefix.
Returns:: task name from prefix appended with an ID.
Return type:: str

_get_files_by_task_and_link(task_id: str, link: FileLink) → List[File]

Get the list of files for a task ID and link type.

Parameters:

task_id (str) – task ID.
link (FileLink) – Type of file link.

Returns:

List of files for a task ID and link type.

Return type:

List[File]

abstract _workflow_recipe() → Dict[str, Any]

Recipe for generating synthetic instances for a workflow. Recipes can be generated by using the InstanceAnalyzer.

Returns:: A recipe in the form of a dictionary in which keys are task prefixes.
Return type:: Dict[str, Any]

abstract build_workflow(workflow_name: str | None = None) → Workflow

Generate a synthetic workflow instance.

Parameters:: workflow_name (Optional[str]) – The workflow name
Returns:: A synthetic workflow instance object.
Return type:: Workflow

abstract classmethod from_num_tasks(num_tasks: int, runtime_factor: float | None = 1.0, input_file_size_factor: float | None = 1.0, output_file_size_factor: float | None = 1.0) → WorkflowRecipe

Instantiate a workflow recipe that will generate synthetic workflows up to the total number of tasks provided.

Parameters:

num_tasks (int) – The upper bound for the total number of tasks in the workflow.
runtime_factor (Optional[float]) – The factor of which tasks runtime will be increased/decreased.
input_file_size_factor (Optional[float]) – The factor of which tasks input files size will be increased/decreased.
output_file_size_factor (Optional[float]) – The factor of which tasks output files size will be increased/decreased.

Returns:

A workflow recipe object that will generate synthetic workflows up to the total number of tasks provided.

Return type:

WorkflowRecipe