Cromwell uses Continuous Integration (CI) testing, along with Continuous Delivery (CD) to the Cromwell-as-a-Service (CaaS) DEV environment. Continuous Deployment is not implemented.

CI testing in Travis and Jenkins

Any suite of tests running under 2.5 hours, using 2 cpus, and 6gb of memory executes on Travis. Travis tests every pull request by a trusted contributor. Larger test suites run on Jenkins instances. Examples include the DSP Workbench CI testing (swatomation), a nightly test of develop on (a snapshot of) the $5 Genome WDL, and the Cromwell-Perf tests that call-cache thousands of jobs.

CD to CaaS DEV

CaaS DEV CD

One instance utilizing continuous delivery is the develop branch to CaaS DEV. While manual testing could occur on DEV, users primarily test on PROD.

Manual Deployments

Cromwell Deployment Strategies

The Cromwell developers stage Terra and CaaS PROD deployments. All other deployments are performed by respective groups, who decide when and how to redeploy. Some upgrade Cromwell, while others deploy completely new instances, including a new database schema.

Code Coverage

Only Travis Pull Requests generate maximal code coverage reports. All other CI either doesn't report coverage, or under-reports due to skipped tests.

Vulnerability Scanning

In collaboration with the DSP Information Security team, scans include but are not limited to:

  • Committed Git secrets
  • Vulnerable Java dependencies
  • Penetration testing