The essentials on reading evaluation disputes
- A dispute is information about your scorecard. Each time an advisor disputes a criterion and a manager decides, you get a human review of a result, with a reason attached.
- Volume alone says nothing. A heavily disputed criterion may be badly written or badly understood. What separates the two is the acceptance rate, read criterion by criterion.
- Two rates, four situations: often disputed and often accepted, the assertion is to be reviewed first; often disputed and rarely accepted, the criterion holds and needs explaining to the teams.
- The cause decides where to make the fix. A transcription error is not solved by rewriting the criterion, an ambiguous assertion is not solved in the vocabulary.
- A right to dispute is worth what its response time is worth. Track it as a median, along with the queue and its age.
- The Raisetalk Disputes report brings these readings together: the matrix of criteria to review, the direction and causes of disputes, decision times, the effect on scores, and the curve that shows whether a rewrite has worked.
A dispute is information, not an incident
When every conversation is evaluated, each advisor receives far more results than with sample-based listening. The right to dispute a criterion is the natural counterpart of that coverage: it is the human review of an automated evaluation, requested by the person best placed to spot a discrepancy, the one who handled the call. We described this step in our article on disputing a criterion.
Handled one by one, disputes do justice to an advisor on a call. Read together, they answer another question, the one every quality manager asks: which criteria in my scorecard produce results that my own managers correct? No organised review supplies this signal as regularly. Advisors read their evaluations every day, across every team, and a manager confirms or rejects each of their remarks.
These requests still have to be read correctly. Three common readings lead to poor decisions.
The readings that mislead
| Reading | What it suggests | What it lacks |
|---|---|---|
| The number of disputes | "This criterion is a problem, it is the most disputed" | The outcome of the requests: a criterion that is disputed and then confirmed by the arbitrators does not need rewriting |
| The overall acceptance rate | "Our evaluations are reliable, few disputes succeed" | The detail by criterion: a reassuring average can hide a criterion accepted three times out of four |
| The number of disputes per advisor | "This advisor disputes too much" | The acceptance rate of their requests: if they are often proved right, they are pointing to a criterion, not showing a behaviour |
Then there is the average response time. Take an assumption: ten requests, nine of them decided in one day and one forgotten for 41 days. The average shows 5 days, whereas nine advisors out of ten had their answer the next day. The median, one day, describes what the advisor experiences; the forgotten request is read elsewhere, in the queue.
The method: five rules for reading your disputes
1. Count decisions, not cases
An advisor can dispute three criteria in the same evaluation. The manager then makes three decisions, which may differ: accept the first, reject the other two. The unit of measurement is therefore the decision to be made, criterion by criterion. And the period is counted from the date of the request, not the date of the conversation: it is the request that starts the clock.
2. Cross two rates, criterion by criterion
For each criterion, two rates are enough:
- its dispute rate: of the evaluations where it has a result, the share where it was disputed;
- its acceptance rate: of the requests decided, the share accepted. Pending requests are not included.
Take two criteria, each disputed forty times over a quarter. For the first, thirty requests are accepted; for the second, four. The volume is the same, the conclusion is the opposite: the first produces results that managers correct, the second holds, but advisors do not understand it.
| Rarely accepted | Often accepted | |
|---|---|---|
| Often disputed | To explain to advisors: the criterion holds, it is misunderstood | To review first: the assertion produces results that managers correct |
| Rarely disputed | Nothing to report | To review: few requests, but they succeed |
A rate calculated on two decisions means nothing. Set a threshold, five decisions for example, below which you read numbers only.
3. Ask for the cause at the moment of acceptance
Knowing that a criterion is often corrected does not say where to correct it. The person who accepts a dispute has just listened to the passage: that is the moment to ask them why the result was wrong. A handful of causes cover most cases, and each points to a different place.
| Cause selected by the arbitrator | Where to look |
|---|---|
| Transcription error | The organisation's vocabulary, the quality of the recording |
| Speakers swapped | The assignment of the advisor and customer roles |
| Assertion to reword | The wording of the criterion |
| Criterion not relevant to this call | The condition under which the criterion applies |
| Analysis error | The examples of the criterion |
Without this distinction, you rewrite criteria whose text was sound. A criterion accepted because of transcription errors will gain nothing from being reworded.
4. Track the response time as a median, and the queue
A right of reply handled in three weeks is no longer one. Three measures are enough to keep it: the median decision time, the share of requests decided within seven days, and the requests still pending, with the age of the oldest. A queue that grows from one week to the next shows up before advisors stop disputing.
5. Check the effect of a rewrite
The reading is only worth something if it ends with an action on the scorecard, then with a check. Note the date on which a criterion is changed, and look at its disputes before and after. If they fall, the rewrite has worked. If not, the cause lay elsewhere. The method for testing a criterion before rolling it out is described in our article on testing a quality scorecard.
The Disputes report: the method applied in Raisetalk
In Raisetalk, every score can be proved: the criterion, its justification and the passage of the call that supports it are on the same page. The dispute and its arbitration rest on that passage. The Disputes report gives the overall view. It opens from the Understand page, like the other analytical reports, and covers disputed criteria as well as requests to deactivate an evaluation.
Video, 2 minutes: from the indicators of the period to the criterion to rewrite.
Four indicators open the report, each compared with the previous period: decisions requested, the acceptance rate, the median decision time and the requests pending at the end of the period. No colour qualifies the volume or the acceptance rate, because a rise is neither good nor bad in itself. Only the decision time and the queue read in one direction.
The matrix of criteria to review
The Criteria to review tile places each criterion according to its two rates, with the four zones of the method: review first, to review, to explain to advisors, nothing to report. The size of the dot follows the number of accepted disputes. Only criteria with at least five decisions are placed; when some are missing, a button extends the calculation to six months.

Next to it, the direction of disputes says what advisors are asking for: to go from not acquired to acquired (a missed passage), from assessed to not assessable (the criterion did not apply to this call), or the reverse. Many requests towards "not assessable" point to an application condition to review, not an assertion.
The causes of accepted disputes
On acceptance, the arbitrator selects the cause of the discrepancy. The report gives the breakdown and indicates, for each cause, where to look: the vocabulary, the assignment of roles, the wording of the assertion, the application condition, the examples of the criterion.

Two sources are better than one
The criteria table adds direct corrections to disputes: the results a manager changed on their own initiative, by modifying an evaluation or by adjusting a criterion during a debrief. This is a second source, independent of the advisors. When both point to the same criteria, the signal is all the stronger.
A criterion's panel: from the matrix to the rewrite
A click on a criterion opens its panel, over six months: its assertion, its figures, the direction and causes of its disputes, the latest decisions with their reason and the arbitrator's comment, and the decisions by arbitrator, to see whether managers decide this criterion in the same way. The month-by-month curve carries a Criterion changed mark at each change: this is where the effect of a rewrite is checked.

The loop takes four steps: spot the criterion on the matrix, understand it through the reasons and the causes, rewrite it from Open the criterion, then check on the curve that disputes fall after the mark.
Decision times, the effect on scores, and who arbitrates
- Decision times: decisions by band, from under 24 hours to over 14 days, and the age of pending requests, with a direct link to those that have been waiting for more than 7 days.
- The effect on scores: the points returned by acceptances, the evaluations whose score changed, the difference in the average score of the scope.
- The arbitrators: for each person who decided, their decisions, their acceptance rate and their median decision time.
- Teams and advisors: who uses their right, with the acceptance rate always shown next to the volume.
The advisors table is not a ranking, and carries no colour of judgement. Disputing is a right: an advisor who disputes often and is often proved right is doing you a service.
Like the other reports, it can be compared with another period, shared by link, exported as a PDF and scheduled (see scheduled reports). Sent every Monday to the arbitrators, it reminds them of the queue and of overdue requests. Each person sees only the requests of their own teams, according to their roles and permissions.
Where to start
- Clear the queue. Open the requests that have been pending for more than 7 days and decide them. No reading of the scorecard counts while advisors are waiting for their answer.
- Take the "review first" criteria one by one. Read the advisors' reasons and the causes selected by the arbitrators before opening the criterion.
- Fix it in the right place. The assertion if it is ambiguous, the application condition if the criterion falls on calls where it has no place, the vocabulary if the transcription is at fault.
- Explain the criteria that hold. A criterion that is often disputed and rarely accepted calls for a team briefing, with two or three calls in support, not a rewrite.
- Come back a month later. The criterion's curve says whether disputes fell after the change.
Look at how your arbitrators decide, too. If a criterion is almost always accepted by one manager and almost never by another, the disagreement is about how the criterion is read, and advisors receive two answers to the same question. A calibration session between arbitrators, on a few recent decisions, often settles more than a rewrite.
What disputes do not tell you
An honest reading also states its limits.
- An advisor disputes what works against them. A missed criterion, a criterion that had no place: disputes measure discrepancies to the advisor's disadvantage, never an evaluation that was too lenient. To check that a criterion is not granted too easily, listen to conversations where it is acquired.
- A low acceptance rate does not prove that a criterion is reliable. It says only that the advisors' requests were not justified.
- A team that does not dispute does not necessarily agree. It may be unaware of this right, or not make use of it. The share of advisors who have disputed at least once can be tracked in the report.
- Small volumes do not give rates. Below five decisions, the report shows numbers, not percentages.
Key terms
- Dispute: an advisor's request to review the result of a criterion in their evaluation, with a reason
- Decision to be made: unit of count for disputes; a disputed criterion is one decision, a deactivation request is another
- Dispute rate: share of evaluations where a criterion was disputed, among those where it has a result
- Acceptance rate: accepted requests, divided by accepted and rejected requests; pending requests are not included
- Arbitrator: manager who decides a dispute, by accepting or rejecting it
- Cause: reason for the discrepancy, selected by the arbitrator when accepting a dispute
- Direct correction: result changed by a manager outside a dispute, through a modification or an adjustment in a debrief
- Median decision time: time that half of the requests do not exceed, from request to decision
Frequently asked questions
Is a high number of disputes a bad sign?
Not in itself. Advisors who dispute are reading their evaluations and using their right of reply. What matters is the outcome of the requests: many accepted disputes on the same criterion point to an assertion to review, many rejected disputes point to a criterion to explain.
What is a good acceptance rate?
There is no right value. A high rate signals criteria to review, a very low rate can signal poorly founded requests or arbitrators reluctant to go back on a result. The rate is read criterion by criterion, with the volume alongside, and compared over time.
How quickly should you answer a dispute?
Set yourself a deadline and keep to it. The Raisetalk Disputes report takes seven days as its benchmark: it gives the share of requests decided within that time and flags those that have been waiting longer. A short response time keeps the right to dispute in use.
Should you track disputes by advisor?
Yes, to check that the right is known and used, never to rank. The acceptance rate is always read next to the volume: an advisor who disputes often and is proved right is pointing to a criterion to review. In Raisetalk, this table carries no colour of judgement.
How do you know whether a criterion should be rewritten or explained?
Cross its dispute rate with its acceptance rate. Often disputed and often accepted, it is to be reviewed. Often disputed and rarely accepted, it holds and deserves an explanation to the teams. The cause selected by the arbitrators then says whether to work on the assertion, the application condition or the vocabulary.
How do you check that a rewrite has worked?
Compare the criterion's disputes before and after its change. In a criterion's panel, the month-by-month curve carries a mark at each change: disputes that fall after the mark confirm the rewrite.
Getting started
Your advisors already read their evaluations, and your managers already decide their requests. The Disputes report puts these decisions in order to say which criteria to review, which to explain, and whether the answers arrive on time. It is one more step built on Raisetalk's foundations, after the new interface: every finding tied to the passage that supports it, and now every disagreement tied to the criterion it questions. To see it in two minutes, watch the Disputes report video; to try it on your own calls, try Raisetalk for free.







