Assessment reliability can be dramatically reduced by differing interpretations of scales and conventions as well as poor interviewing technique. This reduced reliability can have a profound effect on study power.
Maximizing inter-rater reliability
and reducing rater variability can have a profound effect on both the power and sample size requirements for a study. For example, improving the intra-class correlation coefficient (ICC)*
from 0.70 to 0.90 would increase the study power from 50% to 72%.
Traditional rater training is insufficient at best. True inter-rater reliability is rarely achieved in multi center trials. Critically, after rater training at the study outset sponsors have:
- No visibility to the quality of patients enrolled
- No visibility to quality of the assessments
Using MedAvante's central bank of expert trainers, sponsors can:
- Monitor locally administered screening and symptom-severity assessments
- Assess the subject based on the site interview
- Calibrate the site raters to the central MedAvante expert trainer cohort
- Site-based raters digitally record all their subject interviews through provided laptop.
- All the recorded interviews and source documents are uploaded to MedAvante via internet.
- Recorded Interviews & the source document both electronically tracked and automatically routed to a MedAvante trainer according to an algorithm agreed upon with the Sponsor.
- MedAvante expert trainer listens to the entire subject interview
- Provides own score of interview scale: automated workflow program will not provide site's scores until MedAvante expert trainer enters their assessment scores.
- Scores the site-based rater's clinical interviewing skills using the Rater Applied Performance Scale (RAPS).
- Provides Sponsor with the real-time Quality Scorecard & provides the site-based raters with calibrating feedback
- MedAvante expert trainers are calibrated to provide consistent feedback to all site raters to minimize rater variability.
The remediation plan is comprised of a customized algorithm to determine the frequency and number of subject interview evaluations, level of concordance required with centralized expert trainers assessments and comprehensive and consistent, personalized feedback to the site-based rater. This rigorous process, as defined in the MedAvante CQC Continuous Monitoring methodology, returns substantial rewards. Other providers may provide a remote independent review but only MedAvante measures the site assessments against a single uniform world wide standard benchmark of clinical excellence.
- To achieve this uniform benchmark, no other organization has MedAvante's on-going internal QC (20% of raters time) and rigorous quality control. If the trainers are not calibrated, they will distribute "noise" to the study.