wfcommons.wfbench

wfcommons.wfbench.bench

class wfcommons.wfbench.bench.WorkflowBenchmark(recipe: Type[WfChefWorkflowRecipe], num_tasks: int, with_flowcept: bool = False, logger: Logger | None = None)[source]

Bases: object

Generate a workflow benchmark instance based on a workflow recipe (WfChefWorkflowRecipe)

Parameters:

  • recipe (Type[WfChefWorkflowRecipe]) – A workflow recipe.

  • num_tasks (int) – Total number of tasks in the benchmark workflow.

  • with_flowcept (bool)

  • logger (Optional[Logger]) – The logger where to log information/warning or errors.

create_benchmark_from_input_file(save_dir: Path, input_file: Path, lock_files_folder: Path | None = None, rundir: Path | None = None) → Path[source]

Create a workflow benchmark.

Parameters:

  • save_dir (pathlib.Path) – Folder to generate the workflow benchmark JSON instance and input data files.

  • input_file (pathlib.Path)

  • lock_files_folder (Optional[pathlib.Path])

  • rundir (Optional[pathlib.Path]) – If you would like for the files to be created/saved in a different directory.

Returns:

The path to the workflow benchmark JSON instance.

Return type:

pathlib.Path

create_benchmark_from_synthetic_workflow(save_dir: Path, workflow: Workflow, percent_cpu: float | Dict[str, float] = 0.6, cpu_work: int | Dict[str, int] = None, gpu_work: int | Dict[str, int] = None, num_chunks: int | None = 10, time: int | None = None, mem: float | None = None, lock_files_folder: Path | None = None, rundir: Path | None = None) → Path[source]

Create a workflow benchmark from a synthetic workflow

Parameters:

  • save_dir (pathlib.Path) – Folder to generate the workflow benchmark JSON instance and input data files.

  • workflow (Workflow) – The (synthetic) workflow to use as a benchmark.

  • percent_cpu (Union[float, Dict[str, float]]) – The maximum percentage of CPU threads.

  • cpu_work (Union[int, Dict[str, int]]) – Maximum CPU work per workflow task.

  • gpu_work (Union[int, Dict[str, int]]) – Maximum GPU work per workflow task.

  • num_chunks (Optional[int]) – Number of chunks for pipelining I/O and computation for each task execution.

  • time (Optional[int]) – Time limit for running each task (in seconds).

  • mem (Optional[float]) – Maximum amount of memory consumption per task (in MB).

  • lock_files_folder (Optional[pathlib.Path])

  • rundir (Optional[pathlib.Path]) – If you would like for the files to be created/saved in a different directory.

Returns:

The path to the workflow benchmark JSON instance.

Return type:

pathlib.Path

create_benchmark(save_dir: Path, percent_cpu: float | Dict[str, float] = 0.6, cpu_work: int | Dict[str, int] = None, gpu_work: int | Dict[str, int] = None, num_chunks: int | None = 10, time: int | None = None, data: int | None = 0, mem: float | None = None, lock_files_folder: Path | None = None, regenerate: bool | None = True, rundir: Path | None = None) → Path[source]

Create a workflow benchmark.

Parameters:

  • save_dir (pathlib.Path) – Folder to generate the workflow benchmark JSON instance and input data files.

  • percent_cpu (Union[float, Dict[str, float]]) – The percentage of CPU threads.

  • cpu_work (Union[int, Dict[str, int]]) – CPU work per workflow task.

  • gpu_work (Union[int, Dict[str, int]]) – GPU work per workflow task.

  • num_chunks (Optional[int]) – Number of chunks for pipelining I/O and computation for each task execution.

  • time (Optional[int]) – Time limit for running each task (in seconds).

  • data (Optional[Union[int, Dict[str, str]]]) – Total workflow data footprint (in MB).

  • mem (Optional[float]) – Maximum amount of memory consumption per task (in MB).

  • lock_files_folder (Optional[pathlib.Path])

  • regenerate (Optional[bool]) – Whether to regenerate the workflow tasks

  • rundir (Optional[pathlib.Path]) – If you would like for the files to be created/saved in a different directory.

Returns:

The path to the workflow benchmark JSON instance.

Return type:

pathlib.Path

generate_input_file(path: Path) → None[source]

Generates input file where customization of cpu percentage, cpu work, gpu work, data size

Parameters:

path (pathlib.Path)

wfcommons.wfbench.bench.generate_sys_data(num_files: int, tasks: Dict[str, int], save_dir: Path) → List[str][source]

Generate workflow’s input data

Parameters:

  • num_files (int) – number of each file to be generated.

  • tasks (Dict[str, int]) – Dictionary with the name of the tasks and their data sizes.

  • save_dir (pathlib.Path) – Folder to generate the workflow benchmark’s input data files.

wfcommons.wfbench.bench.cleanup_sys_files() → None[source]

Remove files already used

wfcommons.wfbench.bench.clean_entry(entry)[source]

wfcommons.wfbench.translator.airflow

class wfcommons.wfbench.translator.airflow.AirflowTranslator(workflow: Workflow | Path, logger: Logger | None = None)[source]

Bases: Translator

A WfFormat parser for creating Airflow workflow applications.

Parameters:

  • workflow (Union[Workflow, pathlib.Path],) – Workflow benchmark object or path to the workflow benchmark JSON instance.

  • logger (Logger) – The logger where to log information/warning or errors (optional).

translate(output_folder: Path, name: str | None = None) → None[source]

Translate a workflow benchmark description(WfFormat) into an Airflow workflow application.

Parameters:

  • output_folder (pathlib.Path) – The name of the output folder.

  • name (str) – The name of the workflow in the DAG

wfcommons.wfbench.translator.bash

class wfcommons.wfbench.translator.bash.BashTranslator(workflow: Workflow | Path, logger: Logger | None = None)[source]

Bases: Translator

A WfFormat parser for creating a sequential bash workflow benchmarks.

Parameters:

  • workflow (Union[Workflow, pathlib.Path],) – Workflow benchmark object or path to the workflow benchmark JSON instance.

  • logger (Logger) – The logger where to log information/warning or errors (optional).

translate(output_folder: Path) → None[source]

Translate a workflow benchmark description (WfFormat) into an actual workflow application.

Parameters:

output_folder (pathlib.Path) – The path to the folder in which the workflow benchmark will be generated.

wfcommons.wfbench.translator.cwl

class wfcommons.wfbench.translator.cwl.CWLTranslator(workflow: Workflow | Path, generate_stdout_files: bool | None = True, generate_stderr_files: bool | None = True, logger: Logger | None = None)[source]

Bases: Translator

A WfFormat parser for creating CWL workflow benchmarks.

Parameters:

  • workflow (Union[Workflow, pathlib.Path],) – Workflow benchmark object or path to the workflow benchmark JSON instance.

  • generate_stdout_files (Optional[bool]) – If true, each step will generate a .out file with stdout from the step’s execution

  • generate_stderr_files (Optional[bool]) – If true, each step will generate a .err file with stderr from the step’s execution

  • logger (Logger) – The logger where to log information/warning or errors (optional).

generate_stdout_files: bool
generate_stderr_files: bool
translate(output_folder: Path) → None[source]

Translate a workflow benchmark description (WfFormat) into an actual workflow application.

Parameters:

output_folder (pathlib.Path) – The path to the folder in which the workflow benchmark will be generated.

wfcommons.wfbench.translator.streamflow

class wfcommons.wfbench.translator.streamflow.StreamflowTranslator(workflow: Workflow | Path, generate_stdout_files: bool | None = True, generate_stderr_files: bool | None = True, logger: Logger | None = None)[source]

Bases: Translator

A WfFormat parser for creating Streamflow workflow benchmarks.

Parameters:

  • workflow (Union[Workflow, pathlib.Path],) – Workflow benchmark object or path to the workflow benchmark JSON instance.

  • generate_stdout_files (Optional[bool]) – If true, each CWL step will generate a .out file with stdout from the step’s execution

  • generate_stderr_files (Optional[bool]) – If true, each CWL step will generate a .err file with stderr from the step’s execution

  • logger (Logger) – The logger where to log information/warning or errors (optional).

translate(output_folder: Path) → None[source]

Translate a workflow benchmark description (WfFormat) into an actual workflow application.

Parameters:

output_folder (pathlib.Path) – The path to the folder in which the workflow benchmark will be generated.

wfcommons.wfbench.translator.dask

class wfcommons.wfbench.translator.dask.DaskTranslator(workflow: Workflow | Path, logger: Logger | None = None)[source]

Bases: Translator

A WfFormat parser for creating Dask workflow applications.

Parameters:

  • workflow (Union[Workflow, pathlib.Path],) – Workflow benchmark object or path to the workflow benchmark JSON instance.

  • logger (Logger) – The logger where to log information/warning or errors (optional).

translate(output_folder: Path) → None[source]

Translate a workflow benchmark description (WfFormat) into an actual workflow application.

Parameters:

output_folder (pathlib.Path) – The path to the folder in which the workflow benchmark will be generated.

wfcommons.wfbench.translator.makeflow

class wfcommons.wfbench.translator.makeflow.MakeflowTranslator(workflow: Workflow | Path, logger: Logger | None = None)[source]

Bases: Translator

A WfFormat parser for creating Makeflow workflow applications.

Parameters:

  • workflow (Union[Workflow, pathlib.Path],) – Workflow benchmark object or path to the workflow benchmark JSON instance.

  • logger (Logger) – The logger where to log information/warning or errors (optional).

translate(output_folder: Path) → None[source]

Translate a workflow benchmark description (WfFormat) into an actual workflow application.

Parameters:

output_folder (pathlib.Path) – The path to the folder in which the workflow benchmark will be generated.

wfcommons.wfbench.translator.nextflow

class wfcommons.wfbench.translator.nextflow.NextflowTranslator(workflow: Workflow | Path, max_tasks_per_subworkflow: int = 100, max_parents_threshold: int | None = 100, slurm: bool | None = False, logger: Logger | None = None)[source]

Bases: Translator

A WfFormat parser for creating Nextflow workflow applications.

This translator can generate either a single-file workflow or split the workflow across multiple module files for better scalability with large workflows.

Parameters:

  • workflow (Union[Workflow, pathlib.Path]) – Workflow benchmark object or path to the workflow benchmark JSON instance.

  • max_tasks_per_subworkflow (int) – Maximum number of tasks per module file when using subworkflows (default: 500).

  • max_parents_threshold (int) – Tasks with more parents than this get their own module (default: 100).

  • slurm (bool) – Whether to generate a Slurm template script for workflow submission using sbatch.

  • logger (Logger) – The logger where to log information/warning or errors (optional).

subworkflows: List[List[Task]]
translate(output_folder: Path) → None[source]

Translate a workflow benchmark description (WfFormat) into a Nextflow workflow application.

Parameters:

output_folder (pathlib.Path) – The path to the folder in which the workflow benchmark will be generated.

wfcommons.wfbench.translator.parsl

class wfcommons.wfbench.translator.parsl.ParslTranslator(workflow: Workflow | Path, logger: Logger | None = None)[source]

Bases: Translator

A WfFormat parser for creating Parsl workflow benchmarks.

Parameters:

  • workflow (Union[Workflow, pathlib.Path],) – Workflow benchmark object or path to the workflow benchmark JSON instance.

  • logger (Logger) – The logger where to log information/warning or errors (optional).

translate(output_folder: Path) → None[source]

Translate a workflow benchmark description (WfFormat) into an actual workflow application.

Parameters:

output_folder (pathlib.Path) – The path to the folder in which the workflow benchmark will be generated.

wfcommons.wfbench.translator.pegasus

class wfcommons.wfbench.translator.pegasus.PegasusTranslator(workflow: Workflow | Path, logger: Logger | None = None)[source]

Bases: Translator

A WfFormat parser for creating Pegasus workflow applications.

Parameters:

  • workflow (Union[Workflow, pathlib.Path],) – Workflow benchmark object or path to the workflow benchmark JSON instance.

  • logger (Logger) – The logger where to log information/warning or errors (optional).

translate(output_folder: Path, tasks_priorities: Dict[str, int] | None = None) → None[source]

Translate a workflow benchmark description (WfFormat) into a Pegasus workflow application.

Parameters:

  • output_folder (pathlib.Path) – The path to the folder in which the workflow benchmark will be generated.

  • tasks_priorities (Optional[Dict[str, int]]) – Priorities to be assigned to tasks.

wfcommons.wfbench.translator.pycompss

class wfcommons.wfbench.translator.pycompss.PyCompssTranslator(workflow: Workflow | Path, logger: Logger | None = None)[source]

Bases: Translator

A WfFormat parser for creating PyCOMPSs workflow applications.

Parameters:

  • workflow (Union[Workflow, pathlib.Path],) – Workflow benchmark object or path to the workflow benchmark JSON instance.

  • logger (Logger) – The logger where to log information/warning or errors (optional).

translate(output_folder: Path) → None[source]

Translate a workflow benchmark description (WfFormat) into an actual workflow application.

Parameters:

output_folder (pathlib.Path) – The path to the folder in which the workflow benchmark will be generated.

wfcommons.wfbench.translator.swift_t

class wfcommons.wfbench.translator.swift_t.SwiftTTranslator(workflow: Workflow | Path, stress_path: Path = PosixPath('stress-ng'), logger: Logger | None = None)[source]

Bases: Translator

A WfFormat parser for creating Swift/T workflow applications.

Parameters:

  • workflow (Union[Workflow, pathlib.Path]) – Workflow benchmark object or path to the workflow benchmark JSON instance.

  • stress_path (pathlib.Path) – Path to the stress-ng command.

  • logger (Logger) – The logger where to log information/warning or errors (optional).

translate(output_folder: Path) → None[source]

Translate a workflow benchmark description (WfFormat) into a Swift/T workflow application.

Parameters:

output_folder (pathlib.Path) – The path to the folder in which the workflow benchmark will be generated.

wfcommons.wfbench.translator.taskvine

class wfcommons.wfbench.translator.taskvine.TaskVineTranslator(workflow: Workflow | Path, logger: Logger | None = None)[source]

Bases: Translator

A WfFormat parser for creating TaskVine workflow applications.

Parameters:

  • workflow (Union[Workflow, pathlib.Path],) – Workflow benchmark object or path to the workflow benchmark JSON instance.

  • logger (Logger) – The logger where to log information/warning or errors (optional).

translate(output_folder: Path) → None[source]

Translate a workflow benchmark description (WfFormat) into an actual workflow application.

Parameters:

output_folder (pathlib.Path) – The path to the folder in which the workflow benchmark will be generated.