Visual improvement of test reports

[MarcoMinghini]

The way test reports are generated may give the visual impression of a negative outcome even in the case that only a small number of tests have failed (i.e. the general outcome is positive), e.g. in the following example:

The test outcome can be visually changed and improved through e.g. the following measures:

• avoid to mark the status of the whole test as "failed"
• instaed, display a performance bar from 0 to 100 (see figure below) to give an immediate idea of the goodness of the outcome. The color of the colored part of the bar depends on the outcome (e.g. it is red for negative outcome, orange for average outcome, green for positive outcome).
• enrich/substitute the table (counting the number of total/count/skipped/failed/warnings/manual test suites, test cases and assertions) with a histogram (see figure below), which gives an immediate idea of how many checks are passed/failed/passed with a warning. Alternatively, for a more detailed display of results, the same histogram representation can be used but replicated for each conformance class.
• provide immediate options for accessing the test report, e.g. by downloading a PDF report or by receiving an e-mail [this may be moved to a separate improvement proposal].

NOTE: this figure is taken from the website https://seositecheckup.com, which "validates" a website against SEO performance criteria. Similar examples can be easily found elsewhere.

[thijsbrentjens]

Nice ideas.

Something to consider (I don't have a solution for it): not all checks are equally important. For example if it is required by the IR as a main operation (like downloading the entire dataset) or because other checks may depend on the result. So it could be hard or a bit misleading to express the test result in a single number or percentage.

[MarcoMinghini]

[2017.4 meeting 2018-11-15] the improvement proposal is considered to be useful and nice to have.

[AntoRot]

That would be a great improvement in order to provide the correct impression on the result of the tests.

[IGNCNIG]

It’s very useful for us this visual implementatin. We consider a very useful improvement to have this information in a visual way:

1. The porcentage of compliance of each Regulations: Metadata, Interoperability, Network services, etc.
2. The idea of histogram is a good idea but it’d be more useful if the information is classified by types of error, not only : passed , failed and warnings
3. Good idea to include the pdf report in this page

[alitka]

I agree with @thijsbrentjens. If possible, a distinction should be made between the different prioritisations of requirements.