TES Backend
The TES backend submits jobs to a server that complies with the protocol described by the GA4GH schema.
This backend creates three files in the <call_dir>
:
script
- A shell script of the job to be run. This contains the user's command from thecommand
section of the WDL code.stdout
- The standard output of the processstderr
- The standard error of the process
The script
file contains:
#!/bin/sh
cd <container_call_root>
<user_command>
echo $? > rc
<container_call_root>
would be equal to the runtime attribute dockerWorkingDir
or /cromwell-executions/<workflow_uuid>/call-<call_name>/execution
if this attribute is not supplied.
Configuring
Configuring the TES backend is straightforward; one must only provide the TES API endpoint for the service.
backend {
default = "TES"
providers {
TES {
actor-factory = "cromwell.backend.impl.tes.TesBackendLifecycleActorFactory"
config {
endpoint = "https://<some-url>/v1/tasks"
root = "cromwell-executions"
dockerRoot = "/cromwell-executions"
concurrent-job-limit = 1000
}
}
}
}
Cromwell uses an exponential backoff strategy to control the rate of requests sent to the TES server.
Users can override the default settings for this behavior in config, see cromwell.example.backends/TES.conf
for full expected format.
For example, to force a constant polling rate rather than exponential backoff:
backend.providers.TES.config.poll-backoff {
min: "1 minute"
max: "1 minute"
multiplier: 1
randomization-factor: 0
}
Supported File Systems
Currently this backend only works with files on a Local or Shared File System.
Docker
This backend supports the following optional Runtime Attributes and Workflow Options for working with Docker:
docker
: Docker image to use such as "Ubuntu".dockerWorkingDir
: defines the working directory in the container.
CPU, Memory and Disk
This backend supports CPU, memory and disk size configuration through the use of the following Runtime Attributes and Workflow Options:
cpu
defines the amount of CPU to use.- Type: Integer (ex: 4)
memory
defines the amount of memory to use.- Type: String (ex: "4 GB" or "4096 MB")
disk
defines the amount of disk to use.- Type: String (ex: "1 GB" or "1024 MB")
disks
accepts a GCP-style disk declaration and attempts to translate it for use on TES- See table below for supported translations
preemptible
defines whether or not to use preemptible VMs.- Type: Boolean (ex: "true" or "false")
- Integers are accepted and will be converted to boolean (true if > 0)
If they are not set, the TES backend may use default values.
GCP disks
to TES disk
compatibility
GCP disks value |
Supported | TES translation | Remark |
---|---|---|---|
local-disk 25 HDD |
✅ | 25 GB disk | |
local-disk 25 SSD |
✅ | 25 GB disk | Disk type info is dropped |
/some/mnt 25 SSD |
❌ | Custom mount points not supported | |
local-disk 25 HDD, /some/mnt 50 SSD |
❌ | Multiple disks are not supported |
Note: if both
disk
anddisks
attributes are specified, the TES backend will automatically use the value indisk
and not attempt to translatedisks
.
Azure
Azure is an implementation of Cromwell that uses the TES interface for orchestrating the tasks on Azure.
TESK
TESK is an implementation of the TES interface that uses Kubernetes and FTP.
When running Cromwell with a TESK backend, you will want to customize the way Cromwell process globs, as kubernetes will not work well with hard links in a lot of cases which is the default behavior in Cromwell.
By adding this to the config
section of the TES backend in Cromwell, Cromwell will use symlinks instead.
glob-link-command = "ls -L GLOB_PATTERN 2> /dev/null | xargs -I ? ln -s ? GLOB_DIRECTORY"