Fetch Public Sequencing Data and Metadata Using iSeq
A Snakemake 8 workflow to fetch (download) and process public sequencing data and metadata from GSA, SRA, ENA, GEO and DDBJ databases using iSeq.
[!NOTE]
This workflow adheres to the module specifications of MrBiomics, an effort to augment research by modularizing (biomedical) data science. For more details, instructions, and modules check out the project’s repository.⭐️ Star and share modules you find valuable 📤 - help others discover them, and guide our future work!
[!IMPORTANT]
If you use this workflow in a publication, please don’t forget to give credit to the authors by citing it using this DOI 10.5281/zenodo.15005419.
🖋️ Authors
💿 Software
This project wouldn’t be possible without the following software and their dependencies.
| Software | Reference (DOI) |
|---|---|
| iSeq | https://github.com/BioOmics/iSeq |
| pandas | https://doi.org/10.5281/zenodo.3509134 |
| Picard | https://broadinstitute.github.io/picard/ |
| Snakemake | https://doi.org/10.12688/f1000research.29032.2 |
🔬 Methods
This is a template for the Methods section of a scientific publication and is intended to serve as a starting point. Only retain paragraphs relevant to your analysis. References [ref] to the respective publications are curated in the software table above. Versions (ver) have to be read out from the respective conda environment specifications (workflow/envs/*.yaml file) or post-execution in the result directory ({module}/envs/*.yaml). Parameters that have to be adapted depending on the data or workflow configurations are denoted in squared brackets e.g., [X].
| Data Acquisition & Processing. Public sequencing data were retrieved from [GSA | SRA | ENA | DDBJ] under the accession(s) [accession_ids] using iSeq (ver) [ref]. The data were downloaded as FASTQ files (and converted to unmapped BAM (uBAM) files using Picard FastqToSam (ver) [ref], preserving sample information and read groups while supporting both single-end and paired-end sequencing data). Metadata for each dataset was collected and merged into a single Comprehensive reference file. |
The data acquisition and processing described here were performed using a publicly available Snakemake (ver) [ref] workflow 10.5281/zenodo.15005419.
🚀 Features
The workflow performs the following steps that produce the outlined results:
- Data Acquisition
- Downloads sequencing data from public repositories GSA, SRA, ENA, and DDBJ using various accession ID types
- Extracts comprehensive metadata for each dataset
- Supports parallel downloading for improved performance using threads
- Data Processing
- Automatic handling of both single-end and paired-end sequencing data
- Creation of a unified comprehensive metadata file with accession IDs and file paths
- Optional conversion from
FASTQ(as*.fastq.gz) to unmapped BAM(as*.bam) format using Picard’s FastqToSam
- Metadata-only mode for quick exploration without downloading sequence files (
metadata_only: 1) - Considerations
- Dependent on iSeq’s supported repositories and accession types
- Requires internet connectivity and sufficient storage space for downloaded data
The workflow produces the following directory structure:
{result_path}/
└── fetch_ngs/
├── metadata.csv # merged metadata for all accessions
├── .fastq_to_bam/ # processing marker files
│ └── [accession].done
└── [accession]/ # one directory per accession
├── [accession].metadata.csv # metadata for this accession
└── [sample].[bam/fastq.gz] # sequence files
🛠️ Usage
Here are some tips for the usage of this workflow:
- Run your workflow with
snakemake --resources parallel_downloads=3to restrict concurrent download jobs to three (worked well for me), thereby reducing the risk of triggering IP blacklisting from excessive parallel FTP connections. This can also be achieved by using the workflow’s profile. In case of usage as module, put the parameter into the parent workflow’s profile. - Specify accession IDs in the configuration file as a list to download multiple datasets in one run
- Use
metadata_only: 1for a quick preview of available data before committing to full downloads - Choose between
FASTQorBAMoutput formats based on your downstream analysis needs - For large datasets, consider increasing
threadsandmemparameters - For super series (e.g., GSE) or projects containing many samples, start by running in
metadata_only: 1mode to extract run accession IDs. Then use these IDs in the config to enable maximum parallelization, avoiding sequential download and conversion. - The merged metadata file can be used as a basis for sample annotation files downstream
- BAM output format (
output_format: bam) is recommended for direct integration with BAM compatible downstream analysis workflows
⚙️ Configuration
Detailed specifications can be found here ./config/README.md
📖 Examples
Explore detailed examples showcasing module usage in comprehensive end-to-end analyses (including data, configuration, annotation and results) in our MrBiomics Recipes:
🔗 Links
📚 Resources
- Recommended compatible MrBiomics Modules for downstream analyses:
- ATAC-seq Data Processing & Quantification Pipeline for processing, quantification and annotation of chromatin accessibility.
- Genome Browser Track Visualization for quality control and visual inspection/analysis of genomic regions/genes of interest or top hits.
- Split, Filter, Normalize and Integrate Sequencing Data after count quantification.
- Differential Analysis with limma to identify and visualize statistically significantly different features (e.g., genes or genomic regions) between sample groups.
- Enrichment Analysis for biomedical interpretation of (differential) analysis results using prior knowledge.
- Unsupervised Analysis to understand and visualize similarities and variations between cells/samples, including dimensionality reduction and cluster analysis. Useful for all tabular data including single-cell and bulk sequencing data.
📑 Publications
The following publications successfully used this module for their analyses.