HTCondor Backend
Allows to execute jobs using HTCondor which is a specialized workload management system for compute-intensive jobs created by the Center for High Throughput Computing in the Department of Computer Sciences at the University of Wisconsin-Madison (UW-Madison).
The backend is specified via the actor factory ConfigBackendLifecycleActorFactory
:
backend {
providers {
HtCondor {
config {
actor-factory = "cromwell.backend.impl.sfs.config.ConfigBackendLifecycleActorFactory"
# ... other configuration
}
}
}
}
This backend makes the same assumption about the filesystem that the local backend does: the Cromwell process and the jobs both have read/write access to the CWD of the job.
The CWD will contain a script.sh
file which will contain the same contents as the Local backend:
#!/bin/sh
cd <container_call_root>
<user_command>
echo $? > rc
The job is launched with a configurable script command such as:
chmod 755 ${script}
cat > ${cwd}/execution/submitFile <<EOF
Iwd=${cwd}/execution
requirements=${nativeSpecs}
leave_in_queue=true
request_memory=${memory_mb}
request_disk=${disk_kb}
error=${err}
output=${out}
log_xml=true
request_cpus=${cpu}
executable=${script}
log=${cwd}/execution/execution.log
queue
EOF
condor_submit ${cwd}/execution/submitFile
The HtCondor backend gets the job ID from parsing the submit.stdout
text file.
Since the script.sh
ends with echo $? > rc
, the backend will wait for the existence of this file, parse out the return code and determine success or failure and then subsequently post-process.
The command used to submit the job is specified under the configuration key backend.providers.HtCondor.config.submit
. It uses the same syntax as a command in WDL, and will be provided the variables:
script
- A shell script of the job to be run. This contains the user's command from thecommand
section of the WDL code.cwd
- The path where the script should be run.out
- The path to the stdout.err
- The path to the stderr.job_name
- A unique name for the job.
This backend also supports docker as optional feature. Configuration key backend.providers.HtCondor.config.submit-docker
is specified for this end. When the WDL contains a docker runtime attribute, this command will be provided with two additional variables:
docker
- The docker image name.docker_cwd
- The path wherecwd
should be mounted within the docker container.docker_script
- The path of thescript
within the docker container.docker_out
- The path of theout
within the docker container.docker_err
- The path of theerr
within the docker container.
chmod 755 ${script}
cat > ${cwd}/execution/dockerScript <<EOF
#!/bin/bash
docker run --rm -i -v ${cwd}:${docker_cwd} ${docker} /bin/bash ${docker_script}
EOF
chmod 755 ${cwd}/execution/dockerScript
cat > ${cwd}/execution/submitFile <<EOF
Iwd=${cwd}/execution
requirements=${nativeSpecs}
leave_in_queue=true
request_memory=${memory_mb}
request_disk=${disk_kb}
error=${cwd}/execution/stderr
output=${cwd}/execution/stdout
log_xml=true
request_cpus=${cpu}
executable=${cwd}/execution/dockerScript
log=${cwd}/execution/execution.log
queue
EOF
condor_submit ${cwd}/execution/submitFile
This backend support additional runtime attributes that are specified in the configuration key backend.providers.HtCondor.config.runtime-attributes
. It uses the same syntax as specifying runtime attributes in a task in WDL.
There are five special runtime attribute configurations, cpu
, memory_mb
, disk_kb
, nativeSpecs
, docker
.
Optional values are defined with the prefix ?
attached to the type.
backend {
providers {
HtCondor {
config {
# ... other configuration
runtime-attributes = """
Int cpu = 1
Float memory_mb = 512.0
Float disk_kb = 256000.0
String? nativeSpecs
String? docker
"""
}
}
}
}
Native Specifications
The use of runtime attribute 'nativeSpecs' allows to the user to attach custom HtCondor configuration to tasks. An example of this is when there is a need to work with 'requirements' or 'rank' configuration.
"runtimeAttributes": {
cpu = 2
memory = "1GB"
disk = "1GB"
nativeSpecs: "TARGET.Arch == \"INTEL\" && TARGET.Memory >= 64"
}
nativeSpecs
attribute needs to be specified as String.
Exit code
See also HPC - Exit code timeout