Software Engineering Blog

Posts by: Junjun Zhang
Dec. 12, 2016

GitHub repository as job scheduling system to orchestrate large data transfer

The ICGC Data Coordination Centre was tasked to transfer an over 700TB dataset into cloud storage systems. We developed a simple and reliable job scheduling system based on GitHub repository, and successfully employed it to orchestrate and track the execution of over 45,000 transfer jobs to complete the task.

Full Article »