Concordance calculator
Agreement between an AI and experts, with its confidence interval and the human ceiling next to it.
Synthetic data. The logic is the one used in production; the numbers are invented for this page.
Agreement AI vs. expert
80.8% 95% interval 72.9 to 86.9
The point estimate clears the target, the lower bound does not. That is what the report says, with no rounding in its own favour.
The ceiling is how often the experts agree exactly with each other. An AI cannot be expected to agree with them more often than they agree among themselves, so the number is published next to it.
What this calculator refuses to do
- Round the point estimate up to the target.
- Report the point estimate without its interval.
- Drop the items or the labelers that pull the number down. If a labeler is excluded, the reason has to be statistical, not convenient.