illuminaio: An open source IDAT parsing tool for Illumina microarrays

Introduction

The DNA microarray field is dominated by the three manufacturers: Affymetrix, Illumina and Agilent. While the basic premise behind their competing products is the same (i.e. the measurement of hybridisation between sample and immobilised probes on arrays via fluorescence), the formats in which these data are presented to end users are quite different, with each manufacturer electing to use their own proprietary format. The most ubiquitous of these is the CEL file, which has been accepted as a standard format for publishing the raw data generated on the Affymetrix platform. A search of the Gene Expression Omnibus (GEO, http://www.ncbi.nlm.nih.gov/geo/) database finds over 90% of submissions of Affymetrix data include one or more CEL files as supplementary material. The format itself is well documented by the manufacturer, who also provides an open-source software development kit (SDK). As a result, in addition to Affymetrix’s own software suite, a large number of CEL parsing tools exist, including a parser implemented based on the file format documentation: affyio¹ and a parser based on the SDK: affxparser².

The same is not true of the primary IDAT format from Illumina, with only 1.5% (49 out of 3208) of the submissions in GEO that use Illumina BeadArrays including IDAT files as supplementary material. Given that IDATs are the standard file type generated during BeadArray processing, it seems reasonable to assume that the relative dearth of IDAT files in the public domain is due to the lack of widespread support for the format. The development of alternative parsing tools has proven more challenging for IDATs for a number of reasons. The foremost amongst these is a lack of public documentation, leaving tool developers to determine the file structure themselves. A further hurdle has been the encryption of IDAT files generated from expression chips. These barriers initially left researchers reliant on the output from Illumina’s GenomeStudio software to convert the data into a more convenient format. Existing open source tools, particularly those that focus on gene expression analysis such as beadarray³, lumi⁴ and limma⁵, all require that the IDAT files have been processed using GenomeStudio to generate a plain-text ASCII file before any analysis can take place (Figure 1). The GenePattern⁶ software suite includes support for reading expression IDAT files, although it is limited to extracting only a subset of the array information. GenomeStudio output also omits various information that is available from the IDAT, such as control probe intensities (for SNP and methylation platforms), so-called out-of-band probes (methylation 450k)⁷ and meta information including software versions and scan date (all platforms).

Figure 1. A typical BeadArray analysis workflow.

Scanning of BeadChips is performed using the iScan or BeadScan control software, producing IDAT files. Currently, these are read by GenomeStudio where the user has the choice of performing an analysis using that software, or outputting the intensities to a text file for processing by alternative means. illuminaio removes this reliance on GenomeStudio. The intention is for analysis packages to incorporate its routines, effectively merging the dashed region and simplifying the analysis process for end users.

Here we introduce the Bioconductor⁸ package illuminaio that can handle IDAT files from any Illumina BeadArray platform, providing a simple unified interface to various low-level data extraction routines.

Data format

The IDAT file format varies depending upon the array platform (Table 1). IDATs generated during the scanning of genotyping and methylation BeadArrays are binary files (one for each of the red and green channels). The bulk of each file is comprised of four fields: the ID of each bead-type on the array, the mean and standard deviation of their intensities, and the number of beads of each type. Additionally, metadata including the date the array was scanned, specific software versions used and the type of BeadChip are also included. Once the structure of the file is understood these binary values can be read directly.

On the other hand, gene expression IDAT files are produced as encrypted XML files. Once decrypted the majority of the data are found as ten Base64 encoded strings. These ten fields include the ID, mean and standard deviation values as found in genotyping IDATs, as well as median and trimmed-mean intensity values, the mean and standard deviation of local background intensities, and the number of beads both before and after outliers have been excluded.

Each array type is also associated with a manifest file (with file extension BPM or BGX) that provides details of probe sequences, intended genomic targets and whether it is a control probe or not, information that is necessary to correctly interpret the data.

Implementation

illuminaio is an R package⁹. The reading of IDAT files is achieved using the readIDAT function. This routine is able to determine the type of IDAT file that has been passed and calls the appropriate code to read the file and return the data as a R list object (Figure 1). This not only contains intensity data, but also the meta information such as scan date that are not routinely extracted and can be useful for detecting batch effects¹⁰.

Decryption of expression IDATs is performed using the open-source DES decryption routine available in Gnulib¹¹. There is no official documentation of this file format, but illuminaio includes a document describing our findings in detail. Source code for the appropriate routines has been adapted and included in illuminaio, removing any requirement for specific external libraries to be installed on a user’s computer. Thus the package can be built and run on all three major operating systems (Linux, Windows and Mac).

The illuminaio package also supports the parsing of non-encrypted IDAT files compressed by gzip and the reading of manifest files describing the array design (readBGX and readBPM).

The summarised intensity values obtained by illuminaio are nearly identical to those reported using GenomeStudio. Small discrepancies related to rounding performed by GenomeStudio are observed. The package vignette contains a detailed comparison. The time taken to read an IDAT depends on the platform, with encrypted expression arrays taking around 1 second per file (for 50,000 probes), and methylation and SNP platforms between 1 to 6 seconds depending on the chip density (which can range between a few hundred thousand and several million probes).

Discussion

The availability of an open-source IDAT reader through illuminaio that can read files from any of Illumina’s BeadArray technologies will promote greater use of the IDAT file as a primary data format in the analysis and sharing of results from BeadArray based profiling studies. The illuminaio package is intended for use by developers to efficiently extract the content of both IDAT and bead-manifest files, thereby expanding the possibilities for conducting reproducible research with these data.

One exception to the dearth of IDAT files noted in the introduction is the The Cancer Genome Atlas (TCGA, http://cancergenome.nih.gov/). IDAT files from Illumina methylation and genotyping arrays are available in large numbers as Tier 1 data from the TCGA website (https://tcga-data.nci.nih.gov/tcga/). Of particular interest is the Illumina 450k methylation array, for which Triche et al.⁷ has shown improvements in background correction by using out-of-band probes, information that is only available through IDAT files and not the GenomeStudio output. For this work Triche et al. used illuminaio to access the out-of-band probes, which shows the advantage of having access to low-level data.

illuminaio is currently used in the minfi¹², methylumi¹³ and crlmm^14,15 packages for importing IDAT files from the Infinium methylation and genotyping platforms respectively, demonstrating its utility.

Software availability

illuminaio is an R package available from the Bioconductor project (http://www.bioconductor.org) and from 10.5281/zenodo.7588.