The essentials on moving to 100% quality monitoring
- Moving from sample-based listening to analysing every call changes the method, not just the tool. What was rare becomes massive, what was estimated becomes counted, and a scorecard designed for an evaluator's ear cannot be read as it stands by a machine.
- Six steps: decide what you measure and at what level, rewrite the scorecard for automated reading, calibrate the analysis against your evaluators, prioritise before activating alerts, budget by the minute, redeploy the quality team.
- Calibration is the step most often skipped. About 140 reviewed verdicts are enough to estimate, to within 5 points, the agreement between the analysis and your evaluators on a criterion. The right benchmark is not 100% agreement, it is how often your evaluators agree with each other.
- A call-by-call alert is only justified if someone can act on that call within the day. The rest is read as a rate, in a debrief or in a report.
- The quality team does not disappear. Its time shifts from listening to calibration, debriefs and action, and advisors need to know what is measured and how to contest it.
A change of method, not just of tool
Listening to three calls per advisor per month measures almost nothing. On 30 calls, an 80% compliance rate has a confidence interval running from 63% to 91%, a ten-point gap from one month to the next appears more than one time in two even when nothing has changed, and a breach present in 1% of calls goes unnoticed three months out of four. The detail of these calculations, and of what the alternative costs, is in our article Analysing 100% of calls: what the statistics say, and what it really costs.
Automated analysis of every conversation removes that limit. But plugging an analysis into a scorecard designed for listening, and alerts into all of its criteria, creates other problems: criteria the machine reads poorly, scores that advisors contest, hundreds of alerts that nobody handles. The six steps that follow are there to avoid them.
| Step | The question to settle | What comes out of it |
|---|---|---|
| 1. Decide what you measure | At what level do you need a reliable figure? | The list of indicators, with their level and the decision they serve |
| 2. Rewrite the scorecard | Can each criterion be settled on what was said? | A scorecard of observable criteria |
| 3. Calibrate the measurement | Does the analysis judge like your evaluators? | An agreement rate per criterion, and a decision per criterion |
| 4. Prioritise the alerts | Who acts, on what, within what timeframe? | Three routes: the call, the person, the process |
| 5. Budget by the minute | Which minutes really need to be analysed? | A scope, a volume, a cost per conversation |
| 6. Redeploy the quality team | What happens to the time spent listening? | A new working rhythm, known to advisors |
Step 1: decide what you measure, and at what level
Before touching the tool, list what you want to know and, for each indicator, the level at which it must be reliable: the centre, the team, the advisor, the segment or the call itself. That level is what tells you whether 100% coverage is necessary.
| What you want to know | Level | What needs to be evaluated |
|---|---|---|
| Is our average compliance rate improving? | The centre | A random sample of 400 calls may be enough |
| Which advisors should we support first? | The advisor | All of their calls |
| Which call reason concentrates complaints? | The segment | All calls for that reason |
| Which customers should we call back to remedy a breach? | The call | All calls |
For each row, note the decision the indicator must serve: train, support, correct a script, remedy, alert senior management. An indicator that serves no decision has no place in the scorecard, and it will produce alerts that nobody reads.
Step 2: rewrite the scorecard for automated reading
A scorecard designed for listening relies on the evaluator's judgement. "Showed empathy", "good command of the offer", "fluent delivery": two evaluators already do not always answer these the same way, and an automated analysis has nothing precise to look for in the transcript. Each criterion must be decidable on the basis of what was said.
| Listening criterion | Criterion rewritten for automated reading |
|---|---|
| The advisor shows empathy. | If the customer describes a concrete consequence of the problem for them, the advisor takes it into account in the solution they propose. |
| Good command of the offer. | The customer asks a question about the offer that the advisor does not answer. |
| Compliance with legal disclosures. | The advisor informs the customer that the call is being recorded, before addressing the purpose of the call. |
Four rules are enough in most cases:
- one subject per criterion, so that a "not acquired" verdict says precisely what was missing;
- a fact observable in the exchange, not an impression or an inner state attributed to the advisor or the customer;
- an explicit condition when the situation is not systematic: a criterion that starts with "If", "When" or "Whenever" is N/A when the situation does not arise, instead of penalising the call;
- compliance kept separate from rituals, and call outcomes tracked outside the score.
These rules are developed in our article on procedural or relational quality monitoring. In Raisetalk, a criterion that neither adds nor removes any points keeps its verdict without affecting the score: it remains filterable, feeds reporting and can trigger an alert.
Step 3: calibrate the measurement before publishing a score
This is the step most often skipped, and the most costly to make up for. An automated analysis can get a criterion wrong, and its error applies to every call. On 20,000 calls, a misunderstood criterion gives a wrong rate with one-point precision. And an individual score published before it has been checked gets contested, rightly, then loses its credibility for a long time.
Calibration means comparing the analysis with your evaluators, criterion by criterion, on a randomly drawn sample. The sample does not disappear with 100% coverage: it changes role. It no longer serves to estimate the quality of advisors, it serves to check the measurement.
The protocol
- Draw at random from calls already analysed. At random, not from flagged or contested calls: otherwise you measure the analysis on its hardest cases and judge it worse than it is.
- Have them evaluated by two evaluators, blind. Neither one sees the analysis's verdict or their colleague's.
- Compare, criterion by criterion: the agreement between the two evaluators, then the agreement of the analysis with each of them.
- Read the direction of the disagreements. An analysis that is stricter or more lenient than your evaluators shifts every rate in the same direction. Scattered disagreements, in both directions, distort rates little but weigh on individual scores.
- Decide criterion by criterion: go live, rewrite, or manual evaluation.
- Start again every month on a small sample, and whenever the scorecard changes.
How many verdicts to review
| Expected agreement | Target precision | Verdicts to review, per criterion |
|---|---|---|
| 90% | 5 points | about 140 |
| 90% | 3 points | about 390 |
| 95% | 3 points | about 200 |
For a conditional criterion, only the calls in which the situation arose count. A criterion that applies to one call in five therefore requires drawing about 700 calls to obtain 140 usable verdicts.
The right threshold is not 100%
The benchmark is not perfect agreement, which your evaluators do not reach among themselves either. If the analysis agrees with each of your evaluators as often as they agree with each other, it judges that criterion at least as consistently as your team, and it does so on every call.
Be wary of the raw agreement rate on criteria that are almost always acquired. If a criterion is acquired in 98% of calls, an analysis that answered "acquired" every time would agree with your evaluators 98 times out of 100 without detecting anything. Look instead at the minority cases: of the calls your evaluators judge not acquired, how many did the analysis also spot? Statisticians correct this bias with Cohen's kappa, which removes the agreement due to chance.
What you decide for each criterion
- Go live when agreement is at the level of your evaluators' own, with no marked bias in either direction.
- Rewrite when disagreements cluster around ambiguous wording: two ideas in the same criterion, a word everyone understands in their own way, an implicit condition. Before deploying the new version, run it on calls that have already been evaluated, using re-evaluation with another scorecard.
- Manual evaluation when the criterion requires information that is absent from the exchange, or a judgement you want to keep human. A criterion can be reserved for manual evaluation, alongside automated analysis: that is the principle of hybrid analysis.
In Raisetalk, each verdict comes with its justification and the passages of the call it relies on: an evaluator checks a verdict in a few seconds instead of listening to the call again. Once the scorecard is live, an evaluation modified by a manager keeps the AI's reading alongside the human decision, and every contestation of a criterion by an advisor is logged. The criteria that come up often in these corrections are the first to recalibrate.
Step 4: prioritise before switching on alerts
100% coverage makes everything that was invisible visible at once. On 20,000 calls a month, a criterion missed in 5% of cases produces around fifty occurrences per working day. If each one triggers an email, nobody reads them any more after a week.
The rule to set before activating a single alert: a call-by-call alert is only justified if someone can act on that specific call, within the day. The rest is read as a rate.
| Route | What triggers it | Form | Recipient and timeframe |
|---|---|---|---|
| The call | Serious compliance breach, threat to cancel, formal complaint | Immediate alert, by email | A named person, who acts within the day |
| The person | Criterion repeatedly missed by an advisor or a team | Rate over the period, notifications read at their own pace | The manager, at the next debrief |
| The process | Call reason on the rise, behaviour missing everywhere, script that does not work | Aggregated and ranked finding | Senior management, in the monthly report |
Three practices prevent the avalanche:
- Count before activating. Filter the previous month's conversations on the target criterion: their number is the volume of alerts the rule would have produced. If it exceeds what the recipient can handle in their day, the rule belongs to another route.
- Start with few immediate alerts. Two or three email rules at launch, reserved for the first route. The other rules notify without interrupting, and you review the set-up after a few weeks, in light of what was read and handled.
- Name each rule after what it flags. "Recording not announced on the Sales scorecard" reads at a glance, "Alert 3" much less so.
In Raisetalk, the delivery mode is chosen rule by rule: Notify by email for what needs to interrupt, Notify in Raisetalk for what only needs to be seen, on the My alerts page, which keeps every alert that fired and what has been read. That page, like an analytic report, can be the subject of a scheduled send: a periodic summary rather than a continuous stream. For the third route, the management report ranks its findings by the gap observed, the seriousness of the criterion and the number of conversations it is based on.
Step 5: budget by the minute
With usage-based billing, the budget is built in three lines.
- The gross volume: the number of calls per month multiplied by their average duration. 20,000 calls of 5 minutes make 100,000 minutes.
- What does not need analysing: voicemails, calls lasting a few seconds, internal transfers, test lines. Raisetalk's telephony connectors, such as Aircall or Ringover, let you set a minimum duration, choose the call direction and exclude lines or users.
- The price per minute: Raisetalk charges from €0.10 per minute of analysed conversation, with no licence or commitment, and packages above a certain monthly volume.
Then bring the total down to a cost per evaluated conversation, the only unit that allows usage-based billing, a per-seat licence and the cost of manual listening to be compared. For a per-seat licence, divide the monthly seat price by the advisor's minutes of conversation: you get the equivalent price per minute. The orders of magnitude, including the cost of a manual evaluation, are detailed in our calculation of the cost of analysing 100% of calls.
Start with a scope, not with the whole centre. One team or one type of call, for example cancellations or complaints, for one month. The first month is used to calibrate and tune the alerts: the pilot budget is that of the minutes in this scope, and the decision to extend is made on figures.
Step 6: redeploy the quality team, and inform advisors
Moving to 100% does not eliminate the quality team's work, it changes its nature.
| Activity | With sample-based listening | With analysis of every call |
|---|---|---|
| Listening to and scoring calls | Most of the time | Monthly calibration on a randomly drawn sample |
| Choosing which calls to listen to | Every week | No longer needed: every call is evaluated |
| Preparing a debrief | Based on three calls | Based on all of the advisor's calls, with passages to back it up |
| Handling alerts | Whenever a listening session happens to catch them | According to the routes of step 4 |
| Explaining a gap | Impossible on 30 calls | From the management report |
| Evolving the scorecard | Rarely, for lack of measurement | In light of calibration and contestations |
The time freed up goes first to the debrief, which becomes the heart of the job: an advisor improves on examples taken from their own calls, not on a score.
That leaves informing advisors, and it is not a formality. In France, no information concerning an employee personally may be collected by a system that has not been brought to their attention beforehand (article L1222-4 of the French Labour Code), and the comité social et économique (CSE), the French works council, is informed and consulted, before they are implemented, on the means used to monitor employees' activity (article L2312-38). Beyond the legal obligation, three things to say clearly:
- what is measured: behaviours observed in the exchange and verifiable facts, never an emotion or an inner state;
- how to contest: a verdict always comes with its justification and the relevant passages, and an advisor can propose the value they consider fair;
- who sees what: roles and permissions limit each reader to their own scope.
The five most common mistakes
- Plugging in the listening scorecard as it stands, without rewriting the criteria that relied on the evaluator's judgement.
- Publishing individual scores before calibrating the analysis.
- Activating an email alert for every criterion.
- Ranking advisors on a score that also depends on what they do not control, such as the call reason or a commercial decision made elsewhere.
- Cutting the quality team instead of redeploying it: the measurement deteriorates as soon as nobody checks it any more.
Key terms
- 100% quality monitoring: automated evaluation of every conversation in a given scope against a scorecard of criteria, replacing sample-based listening
- Calibration: comparing the verdicts of the automated analysis with those of human evaluators, criterion by criterion, to check that the measurement is reliable and to correct it
- Blind double review: evaluation of the same call by two people who see neither each other's verdict nor that of the analysis
- Cohen's kappa: a measure of agreement between two evaluators that removes the share of agreement due to chance
- Conditional criterion: a criterion that only applies if a specific situation arises in the exchange, and is N/A otherwise
- Alert route: the path a finding follows depending on what it calls for: action on the call, on a person or on the process
Frequently asked questions
How do you move from sample-based listening to analysing 100% of calls?
In six steps: decide what you measure and at what level, rewrite the scorecard as observable criteria, calibrate the analysis against your evaluators, prioritise alerts into three routes, budget by the minute on an initial scope, then redeploy the quality team towards calibration, debriefs and action. Inform advisors and, in France, consult the comité social et économique (CSE), the works council, if there is one, before implementation.
Should you keep human evaluators when AI analyses every call?
Yes. They no longer listen to calls at random, but check the measurement, arbitrate contestations, prepare debriefs and evolve the scorecard. Without regular calibration, an automated analysis can drift on a criterion without anyone noticing.
How many calls should you review to calibrate an automated analysis?
About 140 verdicts per criterion to estimate a 90% agreement rate to within 5 points, and about 390 for a precision of 3 points. For a conditional criterion, only the calls in which the situation arose count: you then need to draw more calls.
How do you avoid being swamped by quality alerts?
By reserving immediate alerts for what requires action on the call itself within the day, and reading the rest as rates, in a debrief or in a report. Before activating a rule, count how many of the previous month's calls would have triggered it: that is the volume it will produce.
Do advisors need to be informed that their calls are analysed automatically?
Yes. In France, an employee must be informed in advance of any system that collects information about them, and the comité social et économique (CSE), the works council, is consulted before any means of monitoring activity is implemented. Explaining what is measured, how to contest a verdict and who sees what also reduces contestations.
Getting started
The first month plays out on one scope and one scorecard. Try Raisetalk on your own conversations, and calibrate on your own calls before deciding.
