Reproducibility is one of the key tenets of the scientific method. But in a recent survey published in Nature, more than 70% of researchers were not able to reproduce another scientist’s experiments, and more than half could not reproduce their own experiments1. While the reasons for this are many, at least some of them stem from issues inherent in data collection.
In the field of proteomics, targeted LC-MS/MS methods based on MRM for data collection are the gold standard for protein quantitation. MRM methods tend to have high reproducibility because they only analyze a relatively small number of pre-selected analytes. Unfortunately the same can’t be said about LC-MS/MS discovery methods. With discovery methods, an attempt is made to create a much more comprehensive picture of the sample by acquiring data on as many analytes as possible. These methods are typically based on data dependent analysis, or DDA, where reproducibility can suffer since the analytes that are selected for analysis depend upon the exact make-up of the sample entering the instrument at any given moment in time, which cannot be predicted precisely from run to run.
SWATH® Acquisition bridges the gap between discovery and targeted proteomics and uses a data collection strategy based on data independent analysis, or DIA. With SWATH, the acquisition attempts to generate a complete picture of the sample by systematically fragmenting every analyte in the sample. It, therefore, has the capability to be extremely reproducible. Indeed, within a lab within a study, SWATH has been proven to provide very reproducible, high-quality quantitative proteomics data. But what about between labs, what about for larger studies? Would the data be comparable for larger studies that span multiple instruments and multiple laboratories? Eleven laboratories around the world decided to find out.
In a new paper published in Nature Communications, researchers collaborated in an inter-laboratory study where samples of known composition and quality were analyzed by each lab across multiple days2. This was very similar in concept to previous studies that investigated MRM and found that a high degree of reproducibility could be achieved across relatively large sample sets. With SWATH, the challenge was to see if this reproducibility could be accomplished using an acquisition strategy that allows much larger numbers of proteins to be quantified.
In the current study, 11 laboratories around the world with nanoflow LC systems coupled to TripleTOF® 5600/5600+ systems were provided a standard sample consisting of 30 stable isotope-labeled standard (SIS) peptides diluted into a complex protein digest from HEK293 cells as a matrix. Five samples were given to each lab, each with a different set of concentrations of SIS peptides, the total of which spanned 6 orders of dynamic range. The samples were run on each of three days, with sample 4 injected in technical triplicate, across the span of a week. This allowed determination of reproducibility and other quantitative figures of merit to be evaluated across one day, one week, and across multiple sites, such as intra- vs. inter-day vs. inter-lab reproducibility, dynamic range, detection consistency, etc. SWATH Acquisition was set to acquire using 64 variable sized Q1 windows, which was the best acquisition strategy at the time (as the study was started from conversations at HUPO, September 2013). In total, a dataset containing 229 SWATH-MS files was generated and was processed centrally to evaluate.
The results were impressive. From the 229 data files collected, ~4500 proteins were detected on average in each sample across all of the laboratories. The important point to note here is that essentially the same set of proteins were detected from every lab and this total was not due to a large cumulative effect as more data files were processed. In fact, the accumulation of new protein identifications over the entire dataset quickly saturated, indicating that the level of data completeness within a single file was very high (Figure 2 in paper2). For this analysis, use of a global error rate was key to achieving this result3.
Next, the reproducibility of quantifying the matrix proteins from the HEK293 digests was evaluated. Peptide areas for the proteins from each sample were determined, and a simple median normalization strategy was implemented that corrected for the variation in instrument response. Post normalization, the intra-day, inter-day and inter-lab reproducibility of the determined protein abundances were computed and evaluated as a function of protein abundance. The overall median CV across the 11 sites (229 data files) and ~4000 proteins was ~22%. As Ben Collins explains4, this “sounds kind-of on the high side. But when you consider what the source of the data is, the completely label-free analysis across 11 different instruments, I think it’s fairly remarkable”.
Finally, the LLOQ, CV, and LDR were evaluated for the synthetic peptides using classical evaluation techniques. Good detection was achieved with LLOQs in the range of mid amol to low fmol on column for the 30 synthetic peptides in matrix, CVs less than 20% (and usually closer to 10%), and linear dynamic range around 4.5 orders of magnitude. This data was also very consistent between all the sites. (Since then, the TripleTOF 6600 was introduced with a different detection system which would have reduced signal saturation at high peptide load and increased linear dynamic range even further.)
The study also enabled the comparison of MS vs. MS/MS (SWATH-MS) data for quantitation. Here, the LLOQ of peptides using MS/MS data was nearly an order of magnitude lower than when using MS scans. Additionally, the reproducibility was greater with lower CVs for the MS/MS data. Although the absolute signal was greater in the MS data, the specificity afforded using the MS/MS data in SWATH acquisition provided better quantitative characteristics overall through the use of cleaner, higher S/N data (see Nature Communications paper Supplementary for details2).
This landmark study demonstrates that sensitive, accurate, and reproducible large-scale protein quantitation (>4500 proteins) is indeed possible across multiple laboratories and multiple days. Using SWATH Acquisition, thousands of proteins can be quantified in a robust and comprehensive manner. This study represents a significant advancement for large-scale quantitative proteomics with SWATH acquisition now firmly established as a highly quantitative methodology for supporting large cohort studies for protein discovery and verification.
Thanks to all the participants in this multi-site study, it was an informative and productive collaboration that has yielded an extremely valuable proof for the use of SWATH in large quantitative proteomics studies!
Please comment below to ask me and Ben questions about the cross lab study.
Watch the complete webinar to learn more about this multi-site study >
Learn about how SWATH acquisition for proteomics has evolved over the past 5 years >
References
- Baker, M., “1,500 scientists lift the lid on reproducibility. Survey sheds light on the ‘crisis’ rocking research”, Nature, 25 May 2016: http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970
- Collins, B. C.*, Hunter, C.*, Liu, Y.*, Stefani, T., Chan, D., Zhang, H., Bader, S., Moritz, R. L., Schilling, B., Gibson, B. W., Krisp, C., Molloy, M. P., Hou, G., Lin, L., Liu, S., Hirayama, M., Ohtsuki, S., Selevsek, N., Schlapback, R., Tzeng, S.-C., Held, J. M., Larsen, B., Gingras, A.-C., and Aebersold, R. “Multi-laboratory assessment of reproducibility, qualitative and quantitative performance of SWATH-mass spectrometry”, Nature Communications (2017), 8.
- Rosenberger, G. et al. “Statistical control of peptide and protein error rates in large-scale targeted data-independent acquisition analyses”, Nature Methods (2017) 14, 921–927.
- Webinar:https://sciex.com/x50781
0 Comments