Configuration

On using Balsam, a .balsam configuration directory is created in your home folder. This contains an adjustable settings.json to control how several Balsam components behave. The default should look like the following:

{
    "SCHEDULER_CLASS": "CobaltScheduler",
    "SCHEDULER_SUBMIT_EXE": "/usr/bin/qsub",
    "SCHEDULER_STATUS_EXE": "/usr/bin/qstat",
    "DEFAULT_PROJECT": "datascience",
    "SERVICE_PERIOD": 1,

    "NUM_TRANSITION_THREADS": 5,
    "MAX_CONCURRENT_MPIRUNS": 1000,

    "LOG_HANDLER_LEVEL": "INFO",
    "LOG_BACKUP_COUNT": 5,
    "LOG_FILE_SIZE_LIMIT": 104857600,

    "QUEUE_POLICY": "theta_policy.ini",
    "JOB_TEMPLATE": "job-templates/theta.cobaltscheduler.tmpl"
}

• DEFAULT_PROJECT: If you want to use the Balsam service for automated job submission, you will need to set this to control which allocation your jobs are submitted to.
• NUM_TRANSITION_THREADS: Control how many processes run alongside each launcher to run data staging and pre/post-processing tasks. If these are not relevant to your workflow, you can reduce this value to 1.
• JOB_TEMPLATE: Relative path to the job script template that is used to submit launcher jobs. You may modify this template to insert your own pre-run logic or create a new template and point to it with this field.

Customizing the Job Template

If you want to run some commands at the start of every job or set global environment variables, you can insert this logic in the job template.

An example of copying some data to the SSDs at the beginning of a job on ALCF Theta is shown below.

#!/bin/bash -x
#COBALT -A {{ project }}
#COBALT -n {{ nodes }}
#COBALT -q {{ queue }}
#COBALT -t {{ time_minutes }}
#COBALT --attrs ssds=required:ssd_size=128

export PATH={{ balsam_bin }}:{{ pg_bin }}:$PATH

module unload trackdeps
module unload darshan
module unload xalt
export PMI_NO_FORK=1 # otherwise, mpi4py-enabled Python apps with custom signal handlers do not respond to sigterm

# *** CUSTOM PRE-RUN LOGIC HERE: ***
aprun -n $COBALT_JOBSIZE -N 1 cp -r ~/libs/myLib /local/scratch  

# ***

source balsamactivate {{ balsam_db_path }}
sleep 2
balsam launcher --{{ wf_filter }} --job-mode={{ job_mode }} --time-limit-minutes={{ time_minutes-2 }}

source balsamdeactivate

Warning

Avoid exporting global LD_LIBRARY_PATH or PYTHONPATH variables in your Job Template, because these are inherited by all tasks in the workflow and often clobber how shared libraries or Python packages are loaded. If your codes are properly linked and Python packages are properly installed into isolated environments, it is rarely necessary to set these variables. If you must, it's much better to set them in the Application envscript to prevent polluting the global environment (see the applications guide for more details).