"Add" dataset window - conformity check of uploaded samples

[marcomass]

During data upload check sample consistency vs. dataset schema; complete loading only if all lines of all samples of the uploaded dataset are compliant, whereas stop upload at the first line not compliant, and report the error, the line and sample where it occurs.

Rational and details as follows:
When a new dataset is created with samples that have less region attributes than those listed in the selected dataset schema, currently uploading ends with success, but then the processing of such dataset ends (successfully) with empty result!
If one or more of the uploaded samples include more attributes than those listed in the selected schema, currently the system manages this situation by loading only the values of a number of the first/leftmost sample attributes equal to the number of attributes listed in the schema; this is the only thing that can be done, but it is right only when the additional sample attributes are the last, i.e. the rightmost, ones in the sample.
Check correctness of selected schema for the uploaded samples while creating the new dataset, and accordingly provide a message to the user (either if any of their lines contain less or more region than expected), and stop loading, so that to avoid not conformant samples in the repository.

[akaitoua]

@marcomass, since checking the regions with upload takes considerable time to be performed. I suggests to change the repository files from text to more complicated fast to read files (make our repository a columnar repository ), by using Parquet files.

This is supposed to improve the overall performance for GMQL. I will discuss this with @andreagulino and @acanakoglu

[andreagulino]

@marcomass you can test on cineca

[marcomass]

@andreagulino
I tested and it is working fine, but I think the following four checks should be added, do you agree?

• strand field: if the attribute strand does not have a valid value ("+", "-", ".", "*")
• integer fields: if a field defined as integer/long includes a float value (currently this is checked in left/right coordinates)
• greater number of fields: if the number of input fields is greater than what defined in the schema (e.g. at least an additional field, or single tab - i.e. empty string field at the end of the row)
• chromosome name: if the name used for the chromosome is compliant with what GMQL can process (see how this check has been implemented in GMQL-Importer, checkRegionData method)

[andreagulino]

@marcomass the check on "integer fields" is hard to implement, the others are ok