utils.parcomp

Module Contents

utils.parcomp.run_cmd(cmd)

Run a command using Python subprocess.

Parameters:cmd – The string of the command to run

utils.parcomp.run_cmd_block(cmd)

Run a command using Python subprocess. Block the call until the command has finished.

Parameters:cmd – A tuple containing the string of the command to run

utils.parcomp.parcomp(cmd, cmd_iter_substr, output_dir, cluster="local", runs=1, local_cpus=1, output_msg=False, colnames=list)

Generic function to run a command in parallel

Parameters:

• cmd – the command string to run in parallel
• cmd_iter_substr – the substring of the iteration number. This will be replaced in a number automatically
• output_dir – the output directory
• cluster – the cluster type among local (Python multiprocessing), sge, or lsf
• runs – the number of runs. Ignored if colnames is not empty
• local_cpus – the number of cpus to use at most
• output_msg – print the output messages on screen (available for cluster=’local’ only)
• colnames – the name of the columns to process

Returns:

True if the computation succeeded.

utils.parcomp.progress_bar(it, total)

A minimal CLI progress bar.

Parameters:

• it – current iteration starting from 1
• total – total iterations

utils.parcomp.progress_bar2(it, total)

A CLI progress bar.

Parameters:

• it – current iteration starting from 1
• total – total iterations

utils.parcomp.call_proc(params)

Run a command using Python subprocess.

Parameters:params – A tuple containing (the string of the command to run, the command id)

utils.parcomp.run_jobs_local(cmd, cmd_iter_substr, runs=1, local_cpus=1, output_msg=False, colnames=list)

Run jobs using python multiprocessing locally.

Parameters:

• cmd – the full command to run as a job
• cmd_iter_substr – the substring in command to be replaced with a number
• runs – the number of runs. Ignored if colnames is not empty
• local_cpus – The number of available cpus. If local_cpus <=0, only one core will be used.
• output_msg – print the output messages on screen (available for cluster_type=’local’ only)
• colnames – the name of the columns to process

Returns:

True

utils.parcomp.run_jobs_sge(cmd, cmd_iter_substr, out_dir, err_dir, runs=1, colnames=list)

Run jobs using a Sun Grid Engine (SGE) cluster.

Parameters:

• cmd – the full command to run as a job
• cmd_iter_substr – the substring in command to be replaced with a number
• out_dir – the directory containing the standard output from qsub
• err_dir – the directory containing the standard error from qsub
• runs – the number of runs. Ignored if colnames is not empty
• colnames – the name of the columns to process

Returns:

True if the computation succeeded.

utils.parcomp.run_jobs_lsf(cmd, cmd_iter_substr, out_dir, err_dir, runs=1, colnames=list)

Run jobs using a Load Sharing Facility (LSF) cluster.

Parameters:

• cmd – the full command to run as a job
• cmd_iter_substr – the substring in command to be replaced with a number
• out_dir – the directory containing the standard output from bsub
• err_dir – the directory containing the standard error from bsub
• runs – the number of runs. Ignored if colnames is not empty
• colnames – the name of the columns to process

Returns:

True if the computation succeeded.

utils.parcomp.quick_debug(cmd, out_dir, err_dir)

Look up for error and warning in the standard output and error files. A simple debugging function checking the generated log files. We don’t stop the computation because it happens that these messages are more warnings than real errors.

Parameters:

• cmd – the executed command
• out_dir – the directory containing the standard output files
• err_dir – the directory contining the standard error files

Returns:

True

utils.parcomp.is_output_file_clean(filename, stream_type="standard output")

Check whether a file contains the string ‘error’ or ‘warning’. If so a message is printed.

Parameters:

• filename – a file
• stream_type – ‘stderr’ for standard error, ‘stdout’ for standard output.

Returns:

True