Model Validation

It is easy to validate existing models on Evidencio. Everyting you need is a database containing more than 20 patients and the model you want to validate needs to work properly on Evidencio. There are multiple paths to create a new validation:

• Click on “Details” when looking at a list of models on Evidencio → Click on Available validations →
• Click on the tab validations when looking at a model on Evidencio → click on create validation.
• Click on the when looking at a model on Evidencio.

The first tab when opening a validation (new or edit) is always the general tab. This tab looks roughly the same as the general tab shown when editing/creating a model.

Validation details
Start your model with filling in the title. The title is the only required field necessary for saving a validation.

Model shows a link to the model being validated with the version of the referenced model. This field is important for future updates of the referenced model. Every adjustment of the model will create a new version. If your referenced model has version 1.1 and the model gets an edit where an extra line in the description was added, the model will then be saved to version 1.2 which no longer corresponds to the reference of the validation. This example, however, does not influence the outcomes of the validation. But if a newer version includes adjustments of the model coefficients, then it definitely influences the validation. Since it is never certain what adjustments were made in a newer version of the model, it is advised to always update your validation to the newest version of the referenced model.
Note: If a newer version of the model is available, Evidencio automatically suggest to update the referenced model, but the calculations in the validation have to be repeated manually.

There are three possible Statuses to choose:

• Draft
• Private
• Public

If the validation is saved on Public, an Evidencio employee gets a review request for the validation. If this was not the intention, the review request can be withdrawn by saving the validation on either Draft or Private. Make sure that at least the discrimination and calibration tab has generated results before saving the validation on Public.

It is possible to share your validation with connections on Evidencio, this way you can let connections (such as co-workers) look into your validation before saving it to public. It is necessary to save the validation at least one time before sharing is possible.

Use the Description field to give a little background information regarding the validation. No full description of the validated model is necessary, since it's referenced in the validation details.

Add the language of the validation, note that this does not have to be the same language as the selected language for the website.

Start typing in the MeSH term field and select multiple terms that belong to the topic of the validation.

If all fields are filled, make sure to save the validation and then go on to the next screen References

In this tab, all relevant sources regarding the validation can be added.

Research Authors
These fields Name, Email, and Institute are used to enter the details of the supporting authors of the validation. The authors that have a correct Email address filled in and have a tick in the notify box get an email once the validation was accepted as a public validation.

Online resources
It is recommended to add the URL of the original paper in this section, such as a link to Pubmed or a DOI. Make sure to add Tags for the provided resources. It is possible to add multiple tags to a single resource.

Related files
It is possible to add relevent files over here such as an institute logo, make sure to add tags once a file has been uploaded. Selecting some of these tags, such as the institute logo, gives the option to add a URL to the file.

Once all fields have been filled to your satisfaction, make sure to save the validation and go on to the next tab Comparison

In this tab, the patient characteristics of the model development cohort can be compared to the characteristics of the validation cohort. How well the comparison can be performed also relies on how well the original model was described and how well it was all added to Evidencio. At least fill in total population of the validation study

Characteristics can be described in three different ways:

• Categorical
• Mean continuous
• Median continuous

Add the patient characteristics of the validation cohort in the same way the characteristics were described in the referenced model. Once all the characteristics were added it is possible to let Evidencio calculate three things:

• P-Values (for categorical characteristics)
• Normal distribution plots (for mean continuous characteristics)
• Boxplots (for median continuous characteristics)

P-Values are calculated with the Fisher's exact test
Normal distribution plots are shown using the mean and the standard deviation.
Boxplots are shown using the minimum, first quartile, median, third quartile and maximum. Not all research papers describe the minimum and maximum which unables the display of a boxplot.

Once everything is filled correctly, save the validation and go on the the last tab discrimination & calibration

This field shows how well the model predicted the outcomes in the validation cohort. There are basicly two possible ways to show the outcomes of a validation study.

Compact
Using the Compact view, it is possible to add the C-index and the regression coefficient yourself.
The compact view is useful for adding validation studies that already have been published elsewhere and it is not necessary to calculate the validation anymore.

Once the model has been saved at least once, it is possible to upload a ROC plot and Calibration plot from your local files.

Full Using the Full view, it is possible to add your own data and calculate the discrimination and validation.
To calculate the discrimination and calibration on own data, make sure that the data is ordered correctly in microsoft's Excel or other similar software. Copy and paste your data from Excel to the table shown in the tab. Evidencio supports data entrees between 20 and 10000 rows. Make sure that the column of your copied data matches the columns on Evidencio. Then click on the button Match Validation Data. A pop-up window opens where it is necessary to match your input with the input possibilities of the referenced model. Missing data can be selected, and the index, positive text, and negative text are optional. If the matching was done correctly, you should be able to click the button Calculate discrimination & Calibration. Note The larger your dataset, the longer it may take to calculate the discrimination and calibration. Once the spinner stops spinning, the c-index, ROC curve, calibration plot, intercept, and regression coefficient are displayed in the Results tab.

Make sure to save the validation and if everything was filled in properly, you might select public status in the General tab to initiate a review sequence.