Customer Service Quality Assurance: What Your QA Score Is Measuring

You run a janitorial supply business, and the QA score on your support report has sat in the same green band for months. Then a facilities manager who buys from you on net-30 disputes an invoice, because one of your agents told them the restocking fee on a returned case of floor finish was waived.

You pull the history and find that agents have been waiving that fee for weeks. Every one of those tickets scored well. Each reply went out quickly and politely, with the right tag on it.

A QA score means only as much as three things behind it: what the scorecard checks, which tickets get read, and what changes when a ticket fails. You can check all three without grading a single ticket yourself.

What a Customer Service QA Score Measures

Customer service quality assurance is the regular reading of support conversations against a written standard, so someone can see whether customers got the right answer in the right way. A QA score sums up how the sampled tickets did against that standard.

Two choices are made before anyone grades a ticket, and they decide what the number means. One is what the scorecard checks, and the other is which tickets get read.

A typical scorecard spreads its points across how a reply was written. Its categories cover the voice and empathy in a reply, the writing style, and whether the agent followed procedure. Accuracy sits on the same list as one more category. That scorecard measures real things an operation needs to know, and it measures them well.

The weighting is where the restocking fee got through. A reply that’s warm and well written still collects the points for voice and style when it gets your policy wrong, so it can pass.

Which Policy the Reviewer Checks Against

Accuracy only counts if the reviewer checks each answer against your policy as it stands today. A reviewer working from last year’s policy will pass the answers your agents learned last year.

Some wrong answers should fail a ticket outright, however well the reply was written. Which ones depends on the rule and the situation, so the scorecard should name them. Wrong answers on money, on terms, or on what you promise a customer are the usual candidates.

A business selling to consumers loses money on refunds, return windows, and order changes. An agent who approves a refund past a return window you shortened in the spring has written a reply the customer will love and you will pay for.

A business selling to other businesses loses money on terms and pricing tiers, and on freight claims and purchase orders. A restocking fee waived for a commercial customer is one of those mistakes. Net-30 offered to a customer whose terms are on hold is another. Support for customers who buy on credit terms has to match those terms, and a scorecard built around tone was never set up to check them.

Which Tickets Get Read

Which tickets get read matters as much as what the scorecard checks. A random sample is mostly routine contacts, because routine contacts make up most of the queue.

Refunds and exceptions carry money and risk, and so do escalations and reopened tickets. Low ratings and large commercial orders belong on the same list. For a consumer brand, the risky set might be every refund over a set amount and every ticket a customer reopened. For a parts distributor, it might be every purchase order change and every freight claim.

Those contacts are a small share of the queue and the slowest to score. A reviewer working through a sample tends to reach the quick ones first, so the risky contacts are often the ones that get skipped.

A well-run operation reads both kinds. It grades a random sample to measure quality across the queue. It pulls the risky contacts on purpose to find problems, and it reports the two results separately so one doesn’t hide the other.

AI now lets an operation evaluate every ticket and flag the ones that look wrong or risky. That gives the targeted pull a way to reach tickets a random sample would miss, such as a polite reply that quietly waived a fee. A person still reviews what’s flagged, grades the sample, and decides what changes.

Reviewers also disagree with each other. Two people can score the same ticket differently, so a good QA team holds regular calibration sessions where reviewers grade the same tickets and settle their differences. Without them, the score moves with whoever did the grading that week.

A Score That Never Moves

QA exists to change the operation. Every failed ticket should end as one of three things: an agent coached, a macro or help article rewritten, or a policy question sent to you as a decision.

Your business keeps changing. You add products, change carriers and prices, and hire new people. Each change creates new ways for an answer to go wrong. An operation that’s reading the right tickets would keep finding those, and its list of changes would grow every month.

We’d expect the score from an operation like that to move. It would dip after a new product line or a policy change, then recover once the macro and the coaching catch up. The list of changes would explain each dip.

In your business, a targeted pull of commercial returns would have put one of those waived fees in front of a reviewer early. That month’s list would then show the restocking fee rule written into the returns macro, two agents coached on commercial returns, and one question sent to you: can the fee ever be waived for customers on contract?

That question reaches you as a decision, the same way it would in a weekly report that’s doing its job.

A score that holds steady month after month, with nothing on that list, tells you something different. In our view, it means the operation is grading itself and leaving the work unchanged, with a scorecard and a sample that let the tickets keep passing.

Your Part and Who Does the Grading

Your part in QA is small. You sign off on the scorecard once, including which errors fail a ticket outright, and then you read what changed each month. You don’t grade tickets.

Signing off means reading the list of errors that fail a ticket outright and correcting it wherever it differs from how you run the business. For the janitorial supply business, that list would include waived fees, changed payment terms, and freight credits given without approval.

The same standard applies whether your support is in-house or with a provider. A provider that runs your operation is accountable for QA as part of what you pay for. That makes the scorecard, the sample, and the list of changes yours to see. QA is one of the jobs that should sit with the provider rather than drift back to you.

An in-house lead who grades their own team while also carrying the queue is in a hard position to report a falling score, because the score reads as a verdict on the lead. If your score hasn’t moved in a year, ask who wrote the scorecard and who picks the tickets.

How HelpFlow Runs QA

Other agencies supply agents. HelpFlow runs the whole customer service operation, including QA and performance management. The scorecard, the sample, and the changes are ours to own and to show you.

We write your scorecard, and you sign off on it. If you already run QA, which is rare among the businesses we work with, we build on yours, and you sign off on that too. What we build is usually a step up from any QA a client ran before. Which wrong answers fail a ticket outright depends on the rule and the situation, and the scorecard names them.

We typically grade a random sample across all your tickets and a targeted pull of risky contacts. We’ve also built an AI-driven QA process that evaluates every ticket. Our reviewers grade the sample and the pull, and they hold regular calibration sessions.

You see the findings and what changed in our weekly report and our monthly review. When a finding needs a policy answer or a process change from you, we bring it to you sooner.

We’ve run customer service operations for 100+ ecommerce brands.

Ask for Two Things This Month

Ask whoever runs your support for two things: the scorecard they grade against, and the list of what last month’s QA changed.

Read the scorecard for what fails a ticket outright and whether your money and terms rules are on it. Read the list for coached agents, rewritten macros, and questions sent to you. A full list means QA is changing your operation. An empty list next to a steady score means nobody has been improving it.

If you’d rather receive that list every month than ask for it, that’s the operation we run.