Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FMI Cross Check results as HTML for nightly builds #803

Closed
AnHeuermann opened this issue Sep 3, 2020 · 12 comments
Closed

FMI Cross Check results as HTML for nightly builds #803

AnHeuermann opened this issue Sep 3, 2020 · 12 comments
Assignees

Comments

@AnHeuermann
Copy link
Member

Is your feature request related to a problem? Please describe.
For participation in the FMI Cross Check (#739) we wan't to have regular tests with the cross check repo.
To get some history and to save older results we can do something similar we do for OpenModelica already: https://libraries.openmodelica.org/branches/newInst/Modelica_3.2.3/Modelica_3.2.3.html

Describe the solution you'd like
A html page could look like this:
image

Tinkering with the html we have for OpenModelica until it works would be my suggestion.

@AnHeuermann AnHeuermann self-assigned this Sep 3, 2020
@lochel
Copy link
Member

lochel commented Sep 3, 2020

Looks good! I like it 👍

@sjoelund
Copy link
Member

sjoelund commented Sep 3, 2020

Do all of the different models have exactly 42 signals to verify? Or is this just a mockup?

@AnHeuermann
Copy link
Member Author

AnHeuermann commented Sep 3, 2020

Nope, just a placeholder for now. I'm not generating any csv files for single variable diffs at the moment and don't count the number of failed variables.
But I'll add this soon

@AnHeuermann
Copy link
Member Author

AnHeuermann commented Sep 8, 2020

First HTML generated on Jenkins:
https://test.openmodelica.org/jenkins/job/Sandbox/job/Andreas/job/OMSimulator%20FMI%20Cross%20Check/lastSuccessfulBuild/artifact/OMSimulator/testsuite/fmi-cross-check/html/

This took me ages, but it's slowly getting somewhere.

[Edit] You need to login into Jenkins to be able to see the artifact.

@AnHeuermann
Copy link
Member Author

See PRs OpenModelica/OpenModelicaLibraryTesting#4 and OpenModelica/OpenModelicaBuildScripts#6 for more details on the progress.

@AnHeuermann
Copy link
Member Author

My latest generated html with #818:
screencapture-localhost-html-fmi-cross-check-html-2020-09-25-17_22_17

Please note, that on the Jenkins job this file will have less tests passed, since my local machine has more stuff installed than the Docker image used for testing.

@AnHeuermann
Copy link
Member Author

Results for Ubuntu Bionic and Focal are now online: https://libraries.openmodelica.org/fmi-crosschecking/OMSimulator/
Windows results are still missing though.

@lochel
Copy link
Member

lochel commented Sep 29, 2020

Looks good, thanks!

Why are the tests for bionic and focal sorted differently? And why do they use different OpenModelica versions?

@AnHeuermann
Copy link
Member Author

I have no clue why the are sorted differently. I'm using pythons os.listdir to walk over the created directories. Maybe python3 os.listdir behaves different on Bionic and Focal?
See https://github.com/OpenModelica/OMSimulator/blob/master/testsuite/fmi-cross-check/generateHTML.py#L138
But you can sort the result alphabetically by clicking on the header of the table, e.g. on Simulated

For the different versions of OM: I created the docker images on the same day, see https://hub.docker.com/repository/docker/anheuermann/ompython/tags?page=1 and used the Dockerfiles from OpenModelica/OpenModelicaBuildScripts#6 to create the images.
Maybe http://build.openmodelica.org/apt doesn't contain the same versions for Bionic/Focal?
@sjoelund Can you you comment on this?

We should re-run the build script and move the container to a location of OpenModelica any way.

@sjoelund
Copy link
Member

sjoelund commented Sep 29, 2020

os.listdir does not sort data. You need to sort it if you want it sorted instead of whatever internal order is stored in the filesystem.

I would never use the nightly to create stable docker images unless they require some feature in the master branch. Use a release repository: https://build.openmodelica.org/omc/builds/linux/releases/

@AnHeuermann
Copy link
Member Author

AnHeuermann commented Oct 21, 2020

We now have test results for Windows using Wine on Ubuntu Bionic (called wine64/)
https://libraries.openmodelica.org/fmi-crosschecking/OMSimulator/history/build-230%202020-10-21/

🚀

(I hope the default job will do the same as my temp job to test this and not break)

EDIT:
Of course it did 💥

@AnHeuermann
Copy link
Member Author

Remove spaces from the name of the Jenkins job. I could not figure out how to do it with spaces, so the job will fail as soon someone decides to rename the job and uses spaces.
Now everything should be up and running:
https://libraries.openmodelica.org/fmi-crosschecking/OMSimulator/history/build-v2.1.0-232%202020-10-22/wine64/fmi-cross-check.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants