Questions about metagnomic data at EMG compared to documentation

[renzok]

The below issue(s) questions were raised by Pascal.

Hi,

As already discussed, there is little reason to work with the OSD raw reads (other than technical or QC questions), so within the scope of the OSD analysis paper we should mostly all be using the workable reads (and assemblies, when available), but at this point I'm a little confused as to where the main workable reads may be hiding :)

Can I make a quick sanity-check with you about the location of the OSD raw and workable read data, taking for example ERR770976:
Raw reads (after demultiplexing + adapter clipping as defined on gitHub) are here:
https://www.ebi.ac.uk/ena/data/view/ERR770976
With two nice looking fastq PE files:
ftp://ftp.sra.ebi.ac.uk/vol1/ERA413/ERA413491/fastq/OSD73_R2_shotgun_raw.fastq.gz
ftp://ftp.sra.ebi.ac.uk/vol1/ERA413/ERA413491/fastq/OSD73_R1_shotgun_raw.fastq.gz
containing 1149852 * 2 reads
A certain flavor of workable reads (after quality trimming + length filtering as defined on gitHub) are here:
https://www.ebi.ac.uk/metagenomics/projects/ERP009703/samples/ERS667589/runs/ERR770976/results/versions/2.0
Which provides a link to the above raw reads, and a link to "Processed nucleotide reads (FASTA)":
https://www.ebi.ac.uk/metagenomics/projects/ERP009703/samples/ERS667589/runs/ERR770976/results/sequences/versions/2.0/export?contentType=text&exportValue=processedReads
which downloads a file which from its name look like merged reads:
ERR770976_MERGED_FASTQ_nt_reads.fasta
it contains 975769 reads, not incompatible with the raw PE read number above.

There are a number of questions:
1/ is this file ERR770976_MERGED_FASTQ_nt_reads.fasta the merged workable reads file?
2/ but then how come this file has an extra "ambiguous bas filtering" step which is not documented on gitHub in (as seen in the "Quality Control" tab of https://www.ebi.ac.uk/metagenomics/projects/ERP009703/samples/ERS667589/runs/ERR770976/results/versions/2.0)
3/ where are the workable un-merged reads which are described on gitHub as part of the "shotgun data the output files per sample".
4/ why is the "initial" (I guess "raw") number of reads (1,241,922) in the EMG "Quality Control" tab different from the raw read number on the ENA page above (1,149,852). Because the EMG analysis appears based on more reads than the supposedly official raw dataset, is it possible that EMG started with the "pre-raw" data set, ie the files directly obtained from the sequencing company?

So it seems that in fact the data located on the EMG portal are not the workable dataset described on giHub documentation, but instead an EMG specific dataset based on a different raw dataset, using a different read pre-processing?

I seem to remember we'd agreed that we should all be working with the same data sets, or else how are we to hope to compare and merge our results, plus the mat & met section is going to be one read pre-processing method for each OSD analysis partner?

Thanks ever so much for your help (and patience as this might have been said before - but trust me I did really do my best to search before writing this mail, so it might in fact just be an issue with documentation or my poor understanding).
Cheers,
Pascal

[renzok]

Hi Pascal,

To start answering your question:

• Indeed the data at EMG is not the one as described in Sequence Data Pre-Processing Documentation actually we had a bug (massive thanks for pointing us to this!), where we wrongly stated that EMG was working on our workable datasets. Actually, EMG worked on the raw datasets as already submitted to ENA. See our correction in the documentation here
• the publication of workable is still missing, but expected in the next day. Follow progress here
• I guess this leaves your 4th question still open:

4/ why is the "initial" (I guess "raw") number of reads (1,241,922) in the EMG "Quality Control" tab
different from the raw read number on the ENA page above (1,149,852). Because the EMG
analysis appears based on more reads than the supposedly official raw dataset,

Which is more up to EMG folks. @sittichpeitsche can you answer?

[almitchell]

Hi,

If we take ERR770976, the fasta PE files on the ENA site with 1,149,852 * 2 reads were the ones used by the EBI analysis pipeline - there is no other raw dataset.

The first thing the pipeline does is run SeqPrep to merge the paired end sequences. The resultant file contains 1,241,922 sequences, as illustrated in the ‘Initial reads’ section of the QC page. The reason that this number of reads is higher than the reads in the individual paired end files is that it contains merged reads plus reads that cannot be merged. So the 'initial reads file' used by the pipeline contains:
(R1+R2 merged reads) + (unmerged R1 reads) + (R2 unmerged reads).

The various QC steps (length filtering and removal of sequences with > 10% undetermined nucleotides)
are then applied to this initial reads set. As a result of these processes, we end up with a file containing 975,769 sequences, which is the processed reads file that you link to below.

I hope this makes sense. If there any anything that is unclear, please let me know and I will try to clarify.

Alex

[AVezzi]

Hi,

I had to join the conversation, as I had the same "confusion" that Pascal had.
Here in Padova we have two analysis proposals that are based on the IPR codes that are present in the "*InterPro.tsv" tables released by the EMG EBI effort.
And, as Renzo presumably remember, more than 3 weeks ago I asked to the OSD leaders if all the data (shotgun, 16S and 18S) were all "the same", coming out from an unique pre-processing step.

Today that I had the first look to the Github forum (just for the same doubt that Pascal had), I discovered that ....? What?
Just surfing a bit on the EMG data (and portal) I had clear that someone had merged the reads (that is absolutely correct), and that the number of merged reads could be more than the single raw data file as present in ENA.

The question here is:
what we have to do?
Is EBI going to release a "new version of the EMG data", based on the workable dataset that Bremen is producing? In that case, we clearly have to wait till the new release.
Instead, if the dataset will remain the actual, can we proceed with our analysis, or at the end everything will be useless, as there is no consistency among data?

Could someone answer to my last points? Now we are completely stuck
Cheers,
Ale

[renzok]

Hi @AVezzi ,

no need to be stuck or lost or frustrated. I am not aware of what you exactly want to study. However, I do think your analysis will be valid either based on EMG data or on the OSD workable data. It is just important to be aware of the differences.

Indeed EMG used the 'OSD raw dataset' and produced their own output.

However, OSD decided to to define 'workable datasets' as described in github wiki for the OSD community independently of any existing or future analysis pipeline. Main reason for that were:

• Each pipeline has there own policy of changing over time (e.g. EMG just changed shortly before OSD jamboree)
• Many other pipelines need a 'workable datasets' as input (e.g. mg-traits or SILVAnga)

In this specific case the main difference between the EMG AND osd workable are:

• EMG uses seqprep for merging, OSD uses PEAR: the differences are not that big (see attachment nicely kindly produced by Antonio)
• the main difference in results is based on the fact that OSD pre-processing uses Q20 instead of Q15 in the quality-trimming step after merging.

@genomewalker (antonio) and I also run all metagenomes workable datasets through the mg-traits pipeline. The final result will be available in two weeks. Maybe you are interested to also do your analysis on the data based on mg-traits in addition to EMG. This might give you different results or not, but definitely it will make your analysis and interpretation more robust.

ciao,renzo

[renzok]

forgot: all workable datasets are available for download now and I updated the documentatio with the aim to make thinks about workable and raw more clear!

[Zaphod-dev]

I followed the link " All workable data is available here" to https://zarafa.mpi-bremen.de/owncloud/index.php/apps/files?dir=/Shared/mgg/OSD/analysis-data/2014/datasets/workable/ and this then asks for a username and password?

[Zaphod-dev]

Also, the documentation is still a little unclear under the section "Metagenomic data" it is still written "You can browse and download the metagenomic data as well as analysis results generated by the European Bioinformatics Institute (EBI) here: https://www.ebi.ac.uk/metagenomics/projects/ERP009703/"
It should be explained that the EMG function & taxonomy results can be accessed there, but that the raw reads are stored at ENA (although linked from the EMG web page, but this to make sure everyone understand where the data is actually officially archived, not providing secondary links that participate in confusing things further) whereas the "workable reads" are available somewhere else (eg zarafa when the access is fixed). I would here explicitely state what exactly are the " Processed nucleotide reads (FASTA)" available from the EMG website, and how they differ from the "workable reads". Better repeat things than risk someone jumping to this section and missing the important points?

[Zaphod-dev]

In fact it looks like the workable data is here:
https://zarafa.mpi-bremen.de/owncloud/public.php?service=files&t=f0e297f451b9944b32f4b2366c74cde3&path=/datasets/workable/metagenomes

[AVezzi]

Thanks Antonio for your answer that allows us to go further with the analyses.
As far as I know (could you confirm this, please?) mg-traits is not producing any InterPro matches table, therefore we need to use the EMG dataset.

One possibility, and this a question to EMG folks, is to do again all the EMG analysis, using the Bremen produced workable dataset (= skipping the merging procedure of the EMG pipeline).
What the EMG folks think on it? It could be (at least for me) the only way to use the same dataset for all the OSD metagenomic analyses.
Thanks,
Ale

[renzok]

@hingamp thanks for further suggestion I will try to improve documentation