Automated Workflow For Processing, Distributing and Archiving of Raw Next Generation Sequence Data

Authors

• CE McIntosh
• V Bliskovsky
• EA Conner
• PC FitzGerald

Abstract

With the introduction of the NextSeq platform for DNA sequencing Illumina did away with the on-machine conversion of BCL files into FASTQ files, replacing it with their cloud-based, BaseSpace, solution. For the vast majority of analytical processes the starting point is FASTQ files, and for many NIH researchers an off-site cloud-based solution for the bcl2fastq conversion is inappropriate. We have developed and deployed an automated pipeline to address this problem encountered by the NCI CCR Genomics Core. This solution not only solves the BCL to FASTQ problem, but also generates multiple Quality Control (QC) metrics, gathers the QC-metrics and run-data into a relational database (accessible via a web interface), archives the data onto an object-based file system, with attached metadata, and provides a distribution system of both data and QC metrics via pre-signed URLs. This later step greatly simplifies the task of moving large volumes of data (Gigabytes) between the the core facility and individual investigators without the need for share accounts or common passwords. With the exception of the bcl2fastq step (not required), this system has also been successfully deployed for the distribution of MiSeq data. The key features of the process include: full automation; only uses on-site NIH resources; archives data on an object-based filesystem with associated metadata tags; real-time monitoring and email notification of completion; web-based interface to metadata, QC metrics and usage reports; and simplified distribution of data via pre-signed URLs

Scientific Focus Area: Computational Biology

This page was last updated on Friday, March 26, 2021