GitHub repository as job scheduling system to orchestrate large data transfer
The ICGC Data Coordination Centre was tasked to transfer an over 700TB dataset into cloud storage systems. We developed a simple and reliable job scheduling system based on GitHub repository, and successfully employed it to orchestrate and track the execution of over 45,000 transfer jobs to complete the task.
