Wrangling distributed computing for high-throughput environmental science: An introduction to HTCondor

Area of Science:

• Computational biology
• Environmental science
• Data science

Background:

• Modern biological and environmental research involves complex computational tasks like geospatial data processing, omics analysis, and large-scale simulations.
• Conventional desktop computing and sometimes high-performance computing (HPC) are insufficient or inappropriate for these demanding problems.

Purpose of the Study:

• To introduce high-throughput computing (HTC) as a viable alternative for computationally intensive research in biology and environmental science.
• To demonstrate the application of HTC using the open-source HTCondor system with practical examples.

Main Methods:

• High-throughput computing (HTC) distributes computational tasks across numerous distributed resources (e.g., desktop computers, servers, cloud).
• Contrasts HTC with high-performance computing (HPC), highlighting HTC's distributed nature versus HPC's single supercomputer approach.
• Utilizes HTCondor, an open-source HTC system, for practical examples in biology and environmental science.

Main Results:

• HTC provides a flexible and accessible method for tackling large-scale computational challenges in scientific research.
• Availability of HTC through academic/government facilities, consortia (e.g., Open Science Grid), and commercial cloud solutions facilitates adoption.
• HTCondor serves as an effective open-source tool for implementing HTC strategies.

Conclusions:

• HTC is a powerful and accessible computing paradigm for biologists and environmental scientists facing significant computational demands.
• Researchers can leverage existing infrastructure or readily available solutions to implement HTC for their studies.
• The adoption of HTC, exemplified by HTCondor, empowers scientific discovery through efficient data processing and simulation.