Tutorial #4:
Stratify the evidence

A. Ioannidis criteria

• This classification allows to stratify evidence according to guidelines described in Fusar-Poli & Radua (2018).
• This classification proposes to stratify evidence in five ordinal classes: "Class I", "Class II", "Class III", "Class IV", "Class ns".
• If you want to apply the criteria described in Fusar-Poli and Radua (2018), you simply have to select the "Ioannidis" criteria in the 4 - Evidence criteria section. All calculations will be performed automatically.
• The precise criteria used to stratify the evidence are the following:
  • Class I: number of cases > 1000, p-value of the meta-analysis < 10e-6, I² < 50%, 95% prediction interval excluding the null, p-value of the egger test > .05 and p-value of the Ioannidis test > .05
  • Class II: number of cases > 1000, p-value of the meta-analysis < 10e-6, largest study with a statistically significant effect and class I criteria not met
  • Class III: number of cases > 1000, p-value of the meta-analysis < 10e-3 and class I–II criteria not met
  • Class IV: p-value of the meta-analysis < 0.05 and class I–III criteria not met
  • Class ns: p-value of the meta-analysis >= 0.05

B. GRADE criteria

• The app also offers to rate the evidence according to criteria labelled as 'GRADE'.
• This criteria allows to stratify evidence according to four ordinal classes: "High", "Moderate", "Low", "Very low".
• If you want to apply the GRADE criteria, you simply have to select the "GRADE" criteria in the 4 - Evidence criteria section. All calculations will be performed automatically.
• Importantly, this classification should not be taken as an equivalent to the subjective approach underlying the standard GRADE criteria. However, similarly to the standard GRADE, this rating system takes into account similar criteria (such as limitations, imprecision, inconsistency and publication bias) and uses a downgrading procedure.
• Every factors start with a "High" evidence class, that could then be downgraded according to the following criteria:
  • Publication bias: a p-value of an Egger test < .10 leads to a downgrading of 1 class
  • Inconsistency: an I² value >= 50% leads to a downgrading of 1 class
  • Imprecision: a total number of participants included in the meta-analysis giving a lower power than 0.8 to detect a Cohen's = 0.20 leads to a downgrading of 1 class. A number of participants giving a lower power than 0.8 to detect a Cohen's d = 0.50, leads to a downgrading of 2 classes.
  • Limitation: a proportion of participants included in studies at low risk of bias inferior to 75% leads to a downgrading of 1 class. A proportion inferior to 50% leads to a downgrading of 2 classes.

C. Personalized criteria

• Because the "Ioannidis" and "GRADE" criteria do not necessarily provide a classification that perfectly matches the requirements of your umbrella review, the metaumbrella app offers the possibility to use a "Personalized" classification to stratify the evidence.
• This classification proposes to stratify the evidence in 5 ordinal classes "Class I", "Class II", "Class III", "Class IV" and "Class V". "Class I" is the highest class that could be achieved and "Class V" is the lowest.
• If you want to apply the Personalized criteria, start by selecting the "Personalized" criteria in the 4 - Evidence criteria section.
• When selecting the "Personalized" criteria, a list of 13 criteria appears in the left menu and can be used to stratify evidence. Only criteria that you select in the left menu will be taken into account to stratify the evidence (in the present example, only the number of studies included the meta-analysis, the p-value of the meta-analysis and the publication bias are used to stratify evidence).
• When you click on a criteria, a list of cut-off appears. If the results of the umbrella review do not fulfill the cut-off values for Class I-IV, a Class V is assigned.
• 
• The overall class achieved by a factor is equal to the lowest class achieved by all criteria used to stratify evidence.
• Once again, only criteria that you select in the left menu will be taken into account to stratify the evidence.

Below, we describe the list of 13 criteria that can be chosen to stratify the evidence:

1. n_studies: a number of studies included in the meta-analysis. If the number of studies included in the meta-analysis is strictly superior to the threshold value indicated in studies, the class for which this value is indicated can be reached.
2. total_n: a total number of participants included in the meta-analysis. If the total number of participants included in the meta-analysis is strictly superior to the threshold value indicated in total_n, the class for which this value is indicated can be reached.
3. n_cases: a number of cases included in the meta-analysis. If the number of cases included in the meta-analysis is strictly superior to the threshold value indicated in cases, the class for which this value is indicated can be reached.
4. p_value: a p-value of the pooled effect size under the random-effects model. If the p-value of the pooled effect size is strictly inferior to the threshold value indicated in p.value, the class for which this value is indicated can be reached.
5. I2: an i-squared (I2) value. If the I2 value of the meta-analysis is strictly inferior to the threshold value indicated in I2, the class for which this value is indicated can be reached.
6. imprecision: a SMD value that will be used to calculate the statistical power of the metaanalysis. If the number of participants included in the meta-analyses allows to obtain a statistical power strictly superior to 80% for the SMD value indicated in imprecision, the class for which this value is indicated can be reached.
7. rob: a percentage of participants included in studies at low risk of bias. Note that the approach to determining whether a study is at low risk of bias is left to the user. If the percentage of participants included in studies at low risk of bias is strictly superior to the threshold value indicated in rob, the class for which this value is indicated can be reached.
8. amstar: an AMSTAR rating on the methodological quality of the meta-analysis. If the AMSTAR value of the meta-analysis is strictly superior to the threshold value indicated in amstar, the class for which this value is indicated can be reached.
9. egger_p: a p-value of an Egger’s test for publication bias. If the p-value of the Egger’s test is strictly superior to the threshold value indicated in egger_p, the class for which this value is indicated can be reached.
10. esb_p: a p-value of an Ioannidis’ test for the excess of significance bias (ESB). If the p-value of the Ioannidis’ test is strictly superior to the threshold value indicated in esb_p, the class for which this value is indicated can be reached.
11. JK_p: the largest p-value obtained in the jackknife meta-analysis (JK). If the largest p-value obtained in the jackknife meta-analysis is strictly inferior to the threshold value indicated in JK_p, the class for which this value is indicated can be reached.
12. pi: a "notnull" value indicates that users request the 95% prediction interval of the metaanalysis to exclude the null value to achieve the class for which it is indicated.
13. largest_CI: a "notnull" value indicates that users request the 95% confidence interval of the largest study included in the meta-analysis to exclude the null value to achieve the class for which it is indicated.