The essentials on analysing 100% of calls
- Thirty calls listened to measure almost nothing. On 30 calls, an 80% compliance rate only means that the true rate lies, with 95% confidence, between 63% and 91%. With three calls per advisor, the range runs from 21% to 94%.
- Month-on-month variations are mostly noise. Between two samples of 30 calls, a gap of at least ten points appears more than one time in two even when nothing has changed. To reliably detect a real 10-point gap, you need to evaluate close to 300 calls per period.
- Rare defects slip through the sample. A breach present in 1% of calls does not appear in any of the 30 calls listened to three months out of four. And zero breaches found on 150 calls is still compatible with 400 non-compliant calls a month.
- For an average, 400 randomly drawn calls are enough. 100% coverage serves another purpose: measuring each advisor, finding every rare defect, comparing segments, retrieving the calls that need action.
- 100% coverage does not solve everything. Sampling error disappears, the measurement error of automated analysis remains, and it does not shrink with volume. Without prioritisation, 20,000 analysed calls produce hundreds of alerts a month.
- Cost is compared per evaluated conversation. At Raisetalk's public floor price, a 5-minute call costs €0.50. A manual evaluation of the same call costs in the order of €7.50 to €10.
100% coverage is a selling point, rarely a calculation
Most pages devoted to speech analytics follow the same outline. They describe the shift from sample-based listening to automated analysis of every conversation, present it as the main benefit, then move on to features. Few of them show, with figures to back it up, why the sample is not enough.
Yet the calculation is within reach of any quality manager. It says two things the sales pitch leaves out: the usual sample measures even less than people think, and analysing every conversation does not solve every problem. We set it out here on a typical case, the same one throughout the article:
- a contact centre that handles 20,000 calls a month;
- 50 advisors split into five teams of ten, that is about 400 calls per advisor and 4,000 per team;
- a quality team that listens to three calls per advisor per month, that is 30 calls per team and 150 for the centre;
- a closely monitored compliance criterion: informing the customer that the call is being recorded.
These volumes are those of many centres. If yours differ, the tables that follow give the orders of magnitude for other sample sizes.
The calculation: what 30 calls measure
Last month, of the 30 calls listened to in one team, 24 included the information about the recording. Measured rate: 80%.
That figure is not the team's rate. It is an estimate, and its precision can be calculated. With 95% confidence, the true rate, the one you would get by listening to all 4,000 of the team's calls, lies between 63% and 91%. A 28-point range, for an indicator that is then discussed down to the last point.
| Calls evaluated | What it represents | Measured rate | 95% confidence interval | Width |
|---|---|---|---|---|
| 3 | One advisor, one month | 67% (2 out of 3) | 21% to 94% | 73 points |
| 30 | A team of ten, one month | 80% | 63% to 91% | 28 points |
| 36 | One advisor, a full year | 81% (29 out of 36) | 65% to 90% | 25 points |
| 150 | The centre, one month | 80% | 73% to 86% | 13 points |
| 400 | A dedicated random draw | 80% | 76% to 84% | 8 points |
| 20,000 | Every call in the month | 80% | 79.4% to 80.5% | 1 point |
Two observations change how this table reads.
The size of the centre hardly matters. Precision depends on the number of calls evaluated, not on the number of calls handled. Thirty calls measure a team of 4,000 calls just as poorly as a centre of 200,000.
Individual follow-up has no statistical basis. Three calls listened to a month mostly rank chance: an advisor at 2 out of 3 and a colleague at 3 out of 3 cannot be told apart. Even added up over a year, 36 calls listened to leave a 25-point range, assuming the advisor's practice has not changed in twelve months.
The calculation, to redo it on your own figures. For a rate p measured on n calls, the margin of error at 95% is roughly 1.96 × √(p × (1 − p) / n). For 80% on 30 calls: 1.96 × √(0.8 × 0.2 / 30) ≈ 0.14, that is 14 points either side. The table uses the Wilson interval, more accurate on small samples and for rates close to 0 or 100%, hence a range shifted slightly downwards: 63% to 91% rather than 66% to 94%.
How large must a gap be to justify an action plan?
The following month, the team scores 21 out of 30, or 70%. A ten-point drop: the supervisor reminds everyone of the instruction, the quality committee asks for an action plan.
The appropriate test for comparing two small samples, Fisher's exact test, gives a probability of 0.55. In other words, if nothing had changed in the team, a gap at least this large would be observed more than one time in two, through the luck of the draw alone. The action plan is responding to noise.
For the drop to be significant at the usual 5% threshold, the team would have to fall from 24 to 15 compliant calls out of 30, that is from 80% to 50%. With 30 calls per period, only massive deteriorations show up.
The question can also be asked the other way round: how many calls need to be evaluated each period to have a good chance, four in five, of detecting a drop that really exists?
| Real drop to detect, from 80% | Calls to evaluate each period |
|---|---|
| 30 points (from 80% to 50%) | about 40 |
| 20 points (from 80% to 60%) | about 80 |
| 15 points (from 80% to 65%) | about 140 |
| 10 points (from 80% to 70%) | about 290 |
| 5 points (from 80% to 75%) | about 1,100 |
Two-sided test at the 5% threshold, 80% statistical power, normal approximation.
With 30 calls per period, only disasters show up. The gaps of a few points that most quality committees discuss are, on these volumes, indistinguishable from chance. This is not a criticism of quality teams: it is the limit of the tool they were given.
The practical rule that follows comes down to one question. Before launching an action plan on a variation, ask how many calls each of the two figures being compared is based on. Even with around a hundred calls per period, a gap of less than 14 points does not allow any conclusion.
Rare defects: what the sample will not see
The average rate is not always the right question. In compliance, the question is often: does it happen, and where? A telephone canvassing call made without obtaining the consent that French law now requires, a contract taken out without the duty to advise showing in the exchange, a mandatory disclosure left out.
Suppose a breach present in 1% of calls, that is 40 calls a month for one team and 200 for the centre.
- In a team's 30 calls listened to, the probability of finding none is 74%. Three months out of four, the team concludes that all is well.
- In the 150 calls listened to for the whole centre, that probability is still 22%, more than one month in five.
And when the sample finds nothing, it proves very little. Zero breaches on 150 calls is still compatible, with 95% confidence, with a true rate of up to 2%, that is 400 non-compliant calls a month out of 20,000. Zero on a team's 30 calls is compatible with a rate of up to 9.5%, that is nearly 400 of the team's 4,000 calls. This is what statisticians call the rule of three: when no case is observed on n calls, the true rate can reach about 3 / n.
When the sample does find a case, it does not say where the others are. Spotting a breach on one call listened to means learning that there are probably dozens more, without knowing which ones. To put things right, call back the customers concerned or respond to an inspection, it is those specific calls you need, not an estimate of how many there are.
For an average, 400 calls are enough: 100% coverage serves another purpose
It would be dishonest to stop there. If the only question is the average level of a criterion across the whole centre, a well-built sample is enough: 400 randomly drawn calls give the rate to within 4 points, 1,000 calls to within 2.5 points. Listening to all 20,000 would add almost nothing to that particular question.
On two conditions, which current set-ups rarely meet:
- Volume. At 15 to 20 minutes per evaluation, 400 calls listened to a month take up roughly one full-time evaluator, for a single centre-level indicator.
- Randomness. A chosen sample is not a random sample. The calls listened to are often the ones a supervisor has flagged, those from a convenient time slot, those that led to a complaint, or those of an advisor someone wants to showcase. This selection bias is not corrected by listening to more: 400 badly chosen calls give a wrong estimate with great precision.
So 100% coverage is not primarily about estimating an average better. It serves four purposes that no reasonable sample allows.
- Measuring each advisor. With 400 calls each, an advisor's rate is known to within 4 points every month, instead of a 70-point range. The debrief then rests on established facts, with passages to back them up, and not on three calls drawn at random.
- Finding every rare defect, instead of estimating its frequency.
- Comparing segments. A call reason, a site, an outsourcer, a time slot, a campaign: each cross-tabulation requires hundreds of calls. Thirty calls spread over five call reasons leave six per cell.
- Retrieving the calls that need action. An estimate says "around 2% of sign-ups without information on the right of withdrawal". Analysing every call gives the list of the conversations concerned, with the passage at issue.
Then there is explanation. Linking an advisor behaviour to a rare outcome, a complaint or a lost sale, requires hundreds of conversations evaluated on the same scorecard. That is the job of the "mechanics" page of the management report, which compares the rate of an outcome depending on whether the behaviour is present or absent, at equal score.
What 100% coverage does not solve
Measurement error replaces sampling error
On 20,000 calls, the confidence interval shrinks to one point. But it only measures the error due to the draw. The other source of error, that of the instrument, remains untouched: an automated analysis can get a criterion wrong, and that error does not shrink with volume. If the analysis fails to recognise the recording information in 5% of the calls where it was given, the rate displayed will be 76% instead of 80%, with one-point precision. A very precise measurement of a misunderstood criterion is still wrong.
Human evaluators are not immune to this problem. Two evaluators listening to the same call do not always return the same verdict, and contact centres call the work of narrowing that gap calibration. But an evaluator's error only weighs on the few calls they listen to, whereas that of an automated analysis applies to all of them.
Hence a counter-intuitive consequence: the sample does not disappear with 100% coverage, it changes role. It no longer serves to estimate the quality of advisors, it serves to check the measurement. A double human review of a random sample of verdicts, criterion by criterion, shows where the analysis goes wrong and in which direction. The orders of magnitude of the first table apply: about 140 reviewed verdicts are enough to estimate a 90% agreement rate to within 5 points.
In Raisetalk, each verdict comes with its justification and the passages of the call it relies on, which makes reviewing quick. An evaluation modified by a manager keeps the AI's reading alongside the human decision, and the advisor can contest a criterion: disagreements remain visible, criterion by criterion. The full calibration protocol is detailed in our method for moving to 100% quality monitoring.
The volume of alerts
The second effect of 100% coverage is more immediate: everything that was invisible becomes visible, all at once. On 20,000 calls, a criterion missed in 5% of cases produces 1,000 occurrences a month, around fifty per working day. For a single criterion. If each one triggers an email, nobody reads them any more after a week, and the set-up falls back, at greater cost, into the blindness of the sample. In compliance, the situation is worse still: a breach that was detected and then left unaddressed is hard to justify during an inspection.
Sorting comes down to a single question: who can act, on what, and within what timeframe? It separates three routes.
- What calls for action on the call itself: a serious compliance breach, a threat to cancel, a formal complaint. Immediate alert, call by call, sent to a person who can act within the day.
- What calls for action on an advisor's or a team's practice: a criterion missed repeatedly. No alert per call, but a rate over the period, dealt with in the debrief.
- What calls for action on the process: a call reason on the rise, a behaviour missing everywhere, a script that does not work. An aggregated finding, ranked, presented to senior management.
A call-by-call alert is only justified if someone can act on that specific call, quickly. Everything else is read as a rate. In Raisetalk, the delivery mode is chosen rule by rule, by email or by a simple in-app notification, available on the My alerts page. The management report, for its part, ranks its findings by the gap observed, the seriousness of the criterion and the number of conversations it is based on, so that what is serious and what is widespread come before the anecdotal.
The real cost: per evaluated conversation, not per seat
That leaves the question that often settles everything. Two pricing logics coexist in this market:
- per seat: a price per user per month, independent of volume. Cost follows headcount, not activity;
- usage-based: a price per minute of analysed conversation. Cost follows volume.
To compare, only one unit works: the cost per conversation actually evaluated, as recommended in our criteria for choosing a solution.
For a per-seat licence, this cost is obtained by dividing the monthly seat price by the advisor's number of conversations. The same seat price therefore gives a cost per conversation three times higher for an advisor who handles 150 calls a month than for a colleague who handles 450. To compare with a per-minute offer, divide the seat price by the advisor's monthly minutes of conversation: you get the equivalent price per minute.
For usage-based billing, this cost is obtained by multiplying the average call duration by the price per minute. Raisetalk charges from €0.10 per minute of analysed conversation, with no licence, set-up fee or term commitment, and packages above a certain monthly volume. At the floor price, a 5-minute call costs €0.50.
Against that, manual evaluation. Evaluating a 5-minute call means listening to all of it, replaying the debatable passages, filling in the scorecard and writing a comment: allow 15 to 20 minutes. At a fully loaded cost of €30 an hour, an assumption to replace with your own, a manual evaluation costs €7.50 to €10.
| Manual evaluation | Automated analysis, Raisetalk floor price | |
|---|---|---|
| Cost of a 5-minute call | €7.50 to €10 | €0.50 |
| Calls covered with an evaluator's monthly budget (about €3,750) | 375 to 500 | about 7,500 |
| What makes the cost vary | The time spent per call | The minutes analysed |
Two clarifications, to keep the comparison honest.
- 100% coverage does not eliminate human work, it moves it. Calibrating the measurement, handling alerts, debriefs, arbitrating contestations: the quality team's time shifts from listening to action. The table compares evaluation costs, not headcounts.
- Not every minute needs to be analysed. Voicemails, calls lasting a few seconds, internal transfers, test lines: Raisetalk's telephony connectors, such as Aircall or Ringover, let you set a minimum duration, choose the call direction and exclude lines or users. The bill follows what you choose to analyse.
Moving from the sample to 100%
The calculation says why the sample is not enough. It does not say how to move beyond it without falling into the two traps described above, the poorly calibrated measurement and the avalanche of alerts. Deciding what to measure, rewriting the scorecard, calibrating, prioritising alerts, budgeting, redeploying the quality team: the six steps are detailed in our method for moving to 100% quality monitoring.
Key terms
- Confidence interval: the range within which the true value of a rate lies, with a given probability (here 95%), given the number of calls on which it was measured
- Significant difference: a gap between two rates too large to be explained, with reasonable probability, by the luck of the draw alone; the usual threshold is 5%
- Statistical power: the probability that a comparison detects a gap that really exists; the usual target is 80%
- Rule of three: when no case is observed on n calls, the true rate can reach about 3 / n
- Selection bias: an error introduced by a sample that is chosen rather than drawn at random; it does not shrink as the sample grows
- Measurement error: the gap between the verdict returned on a call and what actually happened in it, whether the verdict comes from a human or from automated analysis
- Cost per evaluated conversation: the total cost of a set-up divided by the number of conversations actually evaluated, the only unit that allows a per-seat licence to be compared with usage-based billing
Frequently asked questions
How many calls do you need to listen to in order to measure a contact centre's quality?
It all depends on the level and the precision you are aiming for. For an average rate across the centre, 250 to 400 randomly drawn calls give a precision of 4 to 5 points. To detect a 10-point drop from one month to the next, you need close to 300 calls evaluated per period. To follow each advisor, no amount of manual listening is enough: with three calls a month, the confidence interval covers more than 70 points.
Why analyse 100% of calls rather than a sample?
Not to estimate an average better, which a random sample of 400 calls already measures correctly. 100% coverage serves to measure each advisor on a solid basis, to find rare defects, to compare segments such as call reasons or sites, and above all to retrieve the calls concerned in order to act.
Is automated analysis of every call reliable?
It removes sampling error, not measurement error. An automated analysis can get a criterion wrong, and that error applies to every call. It therefore needs to be calibrated: have human evaluators review a random sample of verdicts, criterion by criterion, and rewrite the criteria on which it diverges. A verdict that comes with its justification and the relevant passages can be checked in a few seconds.
How much does analysing 100% of calls cost?
With usage-based billing, the cost is the number of minutes analysed multiplied by the price per minute. Raisetalk charges from €0.10 per minute of analysed conversation, that is €0.50 for a 5-minute call at the floor price, with no licence or commitment. A manual evaluation of the same call costs in the order of €7.50 to €10 in evaluator time.
How do you avoid being swamped by alerts?
By reserving call-by-call alerts for what requires action on the call itself, within the day: a serious breach, a threat to cancel, a complaint. Criteria missed repeatedly are read as rates by advisor and by team, and process problems in an aggregated report. Before activating a rule, count how many of the previous month's calls would have triggered it.
Getting started
Your sample gives you an estimate. To find out what is happening in all your calls, try Raisetalk on your own conversations.
