Evaluation disputes: what they tell you about your quality scorecard

A heavily disputed criterion is not necessarily a badly written one. Dispute rate and acceptance rate read criterion by criterion, the cause selected by the arbitrator, response time as a median: the method for knowing which criteria to review and which to explain, and how the Raisetalk Disputes report puts it into practice.

Blue tokens spread over a board divided into four zones, a single coral token set apart in one corner

The essentials on reading evaluation disputes

  • A dispute is information about your scorecard. Each time an advisor disputes a criterion and a manager decides, you get a human review of a result, with a reason attached.
  • Volume alone says nothing. A heavily disputed criterion may be badly written or badly understood. What separates the two is the acceptance rate, read criterion by criterion.
  • Two rates, four situations: often disputed and often accepted, the assertion is to be reviewed first; often disputed and rarely accepted, the criterion holds and needs explaining to the teams.
  • The cause decides where to make the fix. A transcription error is not solved by rewriting the criterion, an ambiguous assertion is not solved in the vocabulary.
  • A right to dispute is worth what its response time is worth. Track it as a median, along with the queue and its age.
  • The Raisetalk Disputes report brings these readings together: the matrix of criteria to review, the direction and causes of disputes, decision times, the effect on scores, and the curve that shows whether a rewrite has worked.

A dispute is information, not an incident

When every conversation is evaluated, each advisor receives far more results than with sample-based listening. The right to dispute a criterion is the natural counterpart of that coverage: it is the human review of an automated evaluation, requested by the person best placed to spot a discrepancy, the one who handled the call. We described this step in our article on disputing a criterion.

Handled one by one, disputes do justice to an advisor on a call. Read together, they answer another question, the one every quality manager asks: which criteria in my scorecard produce results that my own managers correct? No organised review supplies this signal as regularly. Advisors read their evaluations every day, across every team, and a manager confirms or rejects each of their remarks.

These requests still have to be read correctly. Three common readings lead to poor decisions.

The readings that mislead

ReadingWhat it suggestsWhat it lacks
The number of disputes"This criterion is a problem, it is the most disputed"The outcome of the requests: a criterion that is disputed and then confirmed by the arbitrators does not need rewriting
The overall acceptance rate"Our evaluations are reliable, few disputes succeed"The detail by criterion: a reassuring average can hide a criterion accepted three times out of four
The number of disputes per advisor"This advisor disputes too much"The acceptance rate of their requests: if they are often proved right, they are pointing to a criterion, not showing a behaviour

Then there is the average response time. Take an assumption: ten requests, nine of them decided in one day and one forgotten for 41 days. The average shows 5 days, whereas nine advisors out of ten had their answer the next day. The median, one day, describes what the advisor experiences; the forgotten request is read elsewhere, in the queue.

The method: five rules for reading your disputes

1. Count decisions, not cases

An advisor can dispute three criteria in the same evaluation. The manager then makes three decisions, which may differ: accept the first, reject the other two. The unit of measurement is therefore the decision to be made, criterion by criterion. And the period is counted from the date of the request, not the date of the conversation: it is the request that starts the clock.

2. Cross two rates, criterion by criterion

For each criterion, two rates are enough:

  • its dispute rate: of the evaluations where it has a result, the share where it was disputed;
  • its acceptance rate: of the requests decided, the share accepted. Pending requests are not included.

Take two criteria, each disputed forty times over a quarter. For the first, thirty requests are accepted; for the second, four. The volume is the same, the conclusion is the opposite: the first produces results that managers correct, the second holds, but advisors do not understand it.

Rarely acceptedOften accepted
Often disputedTo explain to advisors: the criterion holds, it is misunderstoodTo review first: the assertion produces results that managers correct
Rarely disputedNothing to reportTo review: few requests, but they succeed

A rate calculated on two decisions means nothing. Set a threshold, five decisions for example, below which you read numbers only.

3. Ask for the cause at the moment of acceptance

Knowing that a criterion is often corrected does not say where to correct it. The person who accepts a dispute has just listened to the passage: that is the moment to ask them why the result was wrong. A handful of causes cover most cases, and each points to a different place.

Cause selected by the arbitratorWhere to look
Transcription errorThe organisation's vocabulary, the quality of the recording
Speakers swappedThe assignment of the advisor and customer roles
Assertion to rewordThe wording of the criterion
Criterion not relevant to this callThe condition under which the criterion applies
Analysis errorThe examples of the criterion

Without this distinction, you rewrite criteria whose text was sound. A criterion accepted because of transcription errors will gain nothing from being reworded.

4. Track the response time as a median, and the queue

A right of reply handled in three weeks is no longer one. Three measures are enough to keep it: the median decision time, the share of requests decided within seven days, and the requests still pending, with the age of the oldest. A queue that grows from one week to the next shows up before advisors stop disputing.

5. Check the effect of a rewrite

The reading is only worth something if it ends with an action on the scorecard, then with a check. Note the date on which a criterion is changed, and look at its disputes before and after. If they fall, the rewrite has worked. If not, the cause lay elsewhere. The method for testing a criterion before rolling it out is described in our article on testing a quality scorecard.

The Disputes report: the method applied in Raisetalk

In Raisetalk, every score can be proved: the criterion, its justification and the passage of the call that supports it are on the same page. The dispute and its arbitration rest on that passage. The Disputes report gives the overall view. It opens from the Understand page, like the other analytical reports, and covers disputed criteria as well as requests to deactivate an evaluation.

Video: the Raisetalk Disputes report, from the indicators to a criterion's panel

Video, 2 minutes: from the indicators of the period to the criterion to rewrite.

Four indicators open the report, each compared with the previous period: decisions requested, the acceptance rate, the median decision time and the requests pending at the end of the period. No colour qualifies the volume or the acceptance rate, because a rise is neither good nor bad in itself. Only the decision time and the queue read in one direction.

The matrix of criteria to review

The Criteria to review tile places each criterion according to its two rates, with the four zones of the method: review first, to review, to explain to advisors, nothing to report. The size of the dot follows the number of accepted disputes. Only criteria with at least five decisions are placed; when some are missing, a button extends the calculation to six months.

Matrix of criteria to review in the Raisetalk Disputes report: each criterion placed according to its dispute rate and its acceptance rate, and the direction of disputes

Next to it, the direction of disputes says what advisors are asking for: to go from not acquired to acquired (a missed passage), from assessed to not assessable (the criterion did not apply to this call), or the reverse. Many requests towards "not assessable" point to an application condition to review, not an assertion.

The causes of accepted disputes

On acceptance, the arbitrator selects the cause of the discrepancy. The report gives the breakdown and indicates, for each cause, where to look: the vocabulary, the assignment of roles, the wording of the assertion, the application condition, the examples of the criterion.

Causes of accepted disputes tile in the Raisetalk Disputes report: one row per cause with the number, the share and where to look

Two sources are better than one

The criteria table adds direct corrections to disputes: the results a manager changed on their own initiative, by modifying an evaluation or by adjusting a criterion during a debrief. This is a second source, independent of the advisors. When both point to the same criteria, the signal is all the stronger.

A criterion's panel: from the matrix to the rewrite

A click on a criterion opens its panel, over six months: its assertion, its figures, the direction and causes of its disputes, the latest decisions with their reason and the arbitrator's comment, and the decisions by arbitrator, to see whether managers decide this criterion in the same way. The month-by-month curve carries a Criterion changed mark at each change: this is where the effect of a rewrite is checked.

A criterion's panel in the Raisetalk Disputes report: assertion, figures, six-month trend with the change mark, direction and latest decisions

The loop takes four steps: spot the criterion on the matrix, understand it through the reasons and the causes, rewrite it from Open the criterion, then check on the curve that disputes fall after the mark.

Decision times, the effect on scores, and who arbitrates

  • Decision times: decisions by band, from under 24 hours to over 14 days, and the age of pending requests, with a direct link to those that have been waiting for more than 7 days.
  • The effect on scores: the points returned by acceptances, the evaluations whose score changed, the difference in the average score of the scope.
  • The arbitrators: for each person who decided, their decisions, their acceptance rate and their median decision time.
  • Teams and advisors: who uses their right, with the acceptance rate always shown next to the volume.

The advisors table is not a ranking, and carries no colour of judgement. Disputing is a right: an advisor who disputes often and is often proved right is doing you a service.

Like the other reports, it can be compared with another period, shared by link, exported as a PDF and scheduled (see scheduled reports). Sent every Monday to the arbitrators, it reminds them of the queue and of overdue requests. Each person sees only the requests of their own teams, according to their roles and permissions.

Where to start

  1. Clear the queue. Open the requests that have been pending for more than 7 days and decide them. No reading of the scorecard counts while advisors are waiting for their answer.
  2. Take the "review first" criteria one by one. Read the advisors' reasons and the causes selected by the arbitrators before opening the criterion.
  3. Fix it in the right place. The assertion if it is ambiguous, the application condition if the criterion falls on calls where it has no place, the vocabulary if the transcription is at fault.
  4. Explain the criteria that hold. A criterion that is often disputed and rarely accepted calls for a team briefing, with two or three calls in support, not a rewrite.
  5. Come back a month later. The criterion's curve says whether disputes fell after the change.

Look at how your arbitrators decide, too. If a criterion is almost always accepted by one manager and almost never by another, the disagreement is about how the criterion is read, and advisors receive two answers to the same question. A calibration session between arbitrators, on a few recent decisions, often settles more than a rewrite.

What disputes do not tell you

An honest reading also states its limits.

  • An advisor disputes what works against them. A missed criterion, a criterion that had no place: disputes measure discrepancies to the advisor's disadvantage, never an evaluation that was too lenient. To check that a criterion is not granted too easily, listen to conversations where it is acquired.
  • A low acceptance rate does not prove that a criterion is reliable. It says only that the advisors' requests were not justified.
  • A team that does not dispute does not necessarily agree. It may be unaware of this right, or not make use of it. The share of advisors who have disputed at least once can be tracked in the report.
  • Small volumes do not give rates. Below five decisions, the report shows numbers, not percentages.

Key terms

  • Dispute: an advisor's request to review the result of a criterion in their evaluation, with a reason
  • Decision to be made: unit of count for disputes; a disputed criterion is one decision, a deactivation request is another
  • Dispute rate: share of evaluations where a criterion was disputed, among those where it has a result
  • Acceptance rate: accepted requests, divided by accepted and rejected requests; pending requests are not included
  • Arbitrator: manager who decides a dispute, by accepting or rejecting it
  • Cause: reason for the discrepancy, selected by the arbitrator when accepting a dispute
  • Direct correction: result changed by a manager outside a dispute, through a modification or an adjustment in a debrief
  • Median decision time: time that half of the requests do not exceed, from request to decision

Frequently asked questions

Is a high number of disputes a bad sign?

Not in itself. Advisors who dispute are reading their evaluations and using their right of reply. What matters is the outcome of the requests: many accepted disputes on the same criterion point to an assertion to review, many rejected disputes point to a criterion to explain.

What is a good acceptance rate?

There is no right value. A high rate signals criteria to review, a very low rate can signal poorly founded requests or arbitrators reluctant to go back on a result. The rate is read criterion by criterion, with the volume alongside, and compared over time.

How quickly should you answer a dispute?

Set yourself a deadline and keep to it. The Raisetalk Disputes report takes seven days as its benchmark: it gives the share of requests decided within that time and flags those that have been waiting longer. A short response time keeps the right to dispute in use.

Should you track disputes by advisor?

Yes, to check that the right is known and used, never to rank. The acceptance rate is always read next to the volume: an advisor who disputes often and is proved right is pointing to a criterion to review. In Raisetalk, this table carries no colour of judgement.

How do you know whether a criterion should be rewritten or explained?

Cross its dispute rate with its acceptance rate. Often disputed and often accepted, it is to be reviewed. Often disputed and rarely accepted, it holds and deserves an explanation to the teams. The cause selected by the arbitrators then says whether to work on the assertion, the application condition or the vocabulary.

How do you check that a rewrite has worked?

Compare the criterion's disputes before and after its change. In a criterion's panel, the month-by-month curve carries a mark at each change: disputes that fall after the mark confirm the rewrite.

Getting started

Your advisors already read their evaluations, and your managers already decide their requests. The Disputes report puts these decisions in order to say which criteria to review, which to explain, and whether the answers arrive on time. It is one more step built on Raisetalk's foundations, after the new interface: every finding tied to the passage that supports it, and now every disagreement tied to the criterion it questions. To see it in two minutes, watch the Disputes report video; to try it on your own calls, try Raisetalk for free.

A row of blue chairs facing each other in pairs, one coral pair in the centre
Quality Monitoring 12 min

Tracking coaching: checking that debriefs take place, and what they change

Evaluating is not coaching. The number of debriefs says neither whether they were completed nor which advisors were seen. Stages, times as a median, team coverage, criteria worked on before and after: the method for tracking coaching, and how the Raisetalk Debriefs report puts it into practice.

A queue of blue call bubbles, two of them coral and joined by an arc
Quality Monitoring 14 min

First call resolution rate: measuring it on your customers' real callbacks

A customer who calls back about the same thing was not served on the first call. Advisor-entered status, instant survey, same number within 7 days: the usual FCR measures mislead. The method for a reliable rate, and how Once & Done in Raisetalk applies it to 100% of calls.

A coral label is tied by four threads to four distinct objects set side by side
Quality Monitoring 16 min

Quality monitoring: what the term really covers, and why two tools carrying that label do not do the same job

Scoring advisors, checking regulatory compliance, listening to the voice of the customer, coaching a sales force: four disciplines circulate under the "quality monitoring" label. They do not have the same users, the same scorecards or the same standards of proof. The framework for telling them apart, why a tool does not carry over from one to another, and how to write a specification that does not mix them up.

A coral speech bubble with jagged edges, picked out in the middle of a stream of blue bubbles
Use Case 12 min

Customer incivility: identifying it in calls to alert and support advisors

An angry customer is not an uncivil customer. Insults, threats, discriminatory remarks: incivility targets the advisor as a person, and most of the time it goes unnoticed. A three-step method to identify it on 100% of calls, alert the right manager and support teams, without measuring emotion.

Three screens of the new Raisetalk interface: dashboard, evaluation page, contestation panel
Conversational Analysis 7 min

The new Raisetalk interface: conversational analysis moves up another gear

Raisetalk unveils its new interface, delivered the weekend of 1 November: more modern, clearer and, above all, more productive, built on the foundations that make it strong. A dashboard for every role, every score proven in one click, one way to search everywhere, and everything you rely on still in place.

Thousands of small blue tiles, of which only about thirty are lit up in coral
Quality Monitoring 14 min

Analysing 100% of calls: what the statistics say, and what it really costs

With 30 calls listened to a month, an 80% compliance rate in fact lies anywhere between 63% and 91%, and a ten-point drop cannot be told apart from chance. The calculation vendors do not show, the gap from which an action plan is justified, what analysing every call really changes, what it does not solve, and its cost per evaluated conversation.

Upload a call. Read its analysis. Five minutes.

Sign up in seconds, with no credit card and no cookies. A welcome tour walks you through the key principles, then you add your first conversation.

Early access · Your feedback shapes the product