Evaluation Forms - Kore.ai Docs

Evaluation Forms provide a flexible framework for evaluating Human Agent and AI Agent interactions using separate evaluation configurations. They support configurable assignments, evaluation criteria, scoring models, duration thresholds, Not Applicable (N/A) scoring treatment, and version-controlled updates. QA Managers can create evaluation forms based on the interaction type. Each form supports only the metrics and workflows applicable to its assigned conversation type. Evaluation Forms support two assignment types:

Human Agent: Evaluates human agent interactions routed through contact center queues. Assign forms to Queues and use Human Agent capabilities such as dispute workflows and speech-based evaluation metrics for supported voice interactions.
AI Agent: Evaluates AI Agent interactions handled by Automation AI Experience Flows. Assign forms to Experience Flows and use AI Agent-compatible evaluation metrics. AI Agent forms don’t support Human Agent capabilities such as dispute workflows or speech-based evaluation metrics.

For AI Agent-to-Human Agent handoff conversations, the system evaluates AI Agent and Human Agent segments using their respective evaluation forms and applicable capabilities. QAI analytics provide filters to analyze conversations handled by an AI Agent, Human Agent, or both.

Select the assignment type when creating the evaluation form and don’t change it later. Assign each form to either Queues (Human Agent) or Experience Flows (AI Agent), not both.

Supported Conversation Types

Evaluation Forms support the following conversation types:

Human Agent conversations: Evaluated using Human Agent evaluation forms assigned to contact center queues.
AI Agent conversations: Evaluated using AI Agent evaluation forms assigned to Automation AI Experience Flows.
AI Agent-to-Human Agent handoff conversations: The system evaluates AI Agent and Human Agent segments using the applicable evaluation workflows and capabilities.

Key Capabilities

Capability	Description
Evaluation Types	Create evaluation forms for Human Agent and AI Agent conversations.
Assignment	Assign Human Agent forms to contact center Queues and AI Agent forms to Automation AI Experience Flows. Queue and Experience Flow assignments are mutually exclusive.
Metrics	Configure metrics based on the selected assignment type. AI Agent forms support By Question (GenAI-Based), By Question (Deterministic), and By AI Agent metrics. Human Agent forms support Human Agent-specific metrics, including speech-based metrics for supported voice interactions.
Channel and Language Support	Configure evaluation forms for supported channels and languages.
Scoring	Configure percentage- or points-based scoring, pass scores, metric weights, response logic, fatal errors, and negative scoring.
Not Applicable (N/A) Scoring	Choose whether the system excludes N/A metrics from the score or awards them full weight.
Duration Thresholds	Specify the minimum interaction duration required for evaluation eligibility.
Conversation Coverage	Evaluate Human Agent, AI Agent, and AI Agent-to-Human Agent handoff conversations. In handoff conversations, the system evaluates AI Agent and Human Agent segments using their applicable evaluation forms. By Question metrics can use transcript context from both segments.
Dispute Workflow	Support dispute workflows for Human Agent evaluation forms.
Completed Audit Editing*	Authorized auditors can update manual audit responses after submitting an audit.
Versioning	Maintain version history and an audit trail for evaluation form configurations and assignments.
Default AI Agent Evaluation Form	Automatically create and assign a default AI Agent evaluation form with predefined quality metrics to all Experience Flows.

Evaluation Forms Structure

Defines the overall configuration of an evaluation form.

Section	Human Agent	AI Agent
General Settings	Configure form details, language, channel, scoring, N/A scoring, duration thresholds, pass score, and assignment type.	Configure the same general settings for AI Agent evaluation forms.
Assignments	Assign evaluation forms to contact center Queues, with Contact Direction and Conversation Source.	Assign evaluation forms to Automation AI Experience Flows for evaluating AI Agent conversations.
Evaluation Metrics	Configure supported Human Agent evaluation metrics, including speech-based metrics for supported voice interactions.	Configure AI Agent-compatible metrics, including By Question (GenAI-Based), By Question (Deterministic), and By AI Agent.
Advanced Settings	Configure dispute workflows, acknowledgement workflows, and completed audit editing permissions.	Not supported.

Access Evaluation Forms

Navigate to Quality AI > Configure > Evaluation Forms.

Evaluation Forms List

The Evaluation Forms page displays the following list:

Field	Description
Name	Displays the evaluation form name.
Description	Displays the evaluation form description.
Assigned In	Displays the assigned Queue (Human Agent) or Experience Flow (AI Agent).
Channel	Displays the assigned channel (Voice or Chat).
Pass Score	Displays the minimum passing score.
Status	Indicates whether the evaluation form is active or inactive.
Created By	Displays the user who created the evaluation form.
Search	Searches evaluation forms by name.

To create AI Agent evaluation forms, also enable the Automation AI Conversation Source.

Evaluation Processing

For conversations that transfer between an AI Agent and a Human Agent, By Question metrics can evaluate transcript context across both conversation segments. This provides additional context for AI-based evaluation while each conversation segment continues to use its own assigned Evaluation Form. This behavior applies only to supported By Question metrics.

Create a New Evaluation Form

Evaluation Form creation consists of three or four steps depending on the selected assignment type.

Configure General Settings

Select the Evaluation Forms tab.
Select + New Evaluation Forms.
Enter the following details:
- Name: Enter a name for the evaluation form.
- Description: Enter a description (optional).
- Language: Select one or more supported languages.
Select a Handled By agent to assign a queue name:
- Human Agent (Human Agent forms): Creates an evaluation form for Human Agent conversations using supported evaluation workflows.
- AI Agent (Experience Flow name): Creates an evaluation configuration for AI Agent conversations using supported evaluation capabilities.
Select a Channel type:
- Chat: Display only chat metrics (excluding speech and voice-specific Playbook metrics).
- Voice: Display all applicable voice metrics, including speech and Playbook metrics.
Select a Scoring Type:
- Percentage: Calculates scores based on configured metric weightages.
- Points: Calculates scores based on assigned metric points.
Select a Not Applicable Scoring Treatment to define how Not Applicable (N/A) metrics affect the final evaluation score for future evaluations only.
- Exclude from Score (default): Excludes the N/A metric from score calculation by removing its weightage from both the numerator and denominator. The metric doesn’t contribute to the final score.
- Award Full Weightage: Includes the N/A metric’s configured weightage in both the numerator and denominator, awarding full marks while the metric outcome remains Not Applicable.
(Optional) Set the minimum interaction duration required for evaluation by entering values in MIN and SEC. The system excludes conversations below the configured threshold from evaluation scoring.
Set a minimum Pass Score required for agents.
Select Next.

Assignment Requirements

The selected Handled By type determines the available configuration options. Human Agent forms require queue assignments, while AI Agent forms require Automation AI Experience Flow assignments. For Human Agent forms, assign at least one queue before proceeding. The selected Contact Direction determines which interaction types the system evaluates for the assigned queues.

Changes to Not Applicable Scoring Treatment apply only to future evaluations created with the updated evaluation form version. Existing evaluations retain the scoring behavior configured when they were created.
Not Applicable Scoring Treatment applies consistently to both AutoQA and manual supervisor audit scores across all metric types and versions.
Attribute and scorecard scoring are configured separately at the Agent Scorecard level.

Configure Assignments

The Assignments step defines where the evaluation form applies. Based on the selected Handled By type, assign the form to Queues for Human Agent conversations or Experience Flows for AI Agent conversations. You can assign an evaluation form to Queues or Experience Flows, but not both.

Human Agent Assignments

Search and select one or more Queues.
Configure the Contact Direction (Inbound, Outbound, or both) and Conversation Source (such as Contact Center AI (CCAI), AgentAI, or Quality AI Express) for each selected queue.
Review the selected queue assignments.
Select Next.

Evaluation metric availability depends on the conversation sources associated with the assigned queues. If an evaluation form includes both CCAI or AgentAI queues and Quality AI Express queues, the By Playbook and By Dialog evaluation metrics are unavailable because these metrics are not supported across mixed conversation sources.

AI Agent Assignments

Search and select one or more Experience Flows.
Review the selected Experience Flows.
Select Next.

Experience Flows are retrieved from the configured Automation AI environment.

Experience Flow Assignment Rules

Rule	Description
One Form Per Flow Per Channel	Each Experience Flow supports only one active evaluation form per channel. If a form exists for the same flow and channel, the system displays a validation error.
Mutually Exclusive Assignment	You can assign an evaluation form either to Queues or to Experience Flows, but not both.
Flow Source	Experience Flows available for selection come from the Automation AI configuration in the same workspace.

Queue Assignment Rules

Assignment Type	Rule
Human Agent	Assign each evaluation form to a unique Queue + Contact Direction combination. You can’t assign the same combination more than once within a form.
AI Agent	Assign each Experience Flow to only one active evaluation form per channel. You can’t combine Queue and Experience Flow assignments in the same evaluation form.

AI Agent Assignments

For Handled By = AI Agent, assign the evaluation form to Automation AI Experience Flows.

To create AI Agent evaluation forms, you must enable the Automation AI Conversation Source. If Automation AI isn’t enabled, Experience Flow assignments are unavailable, and you can’t assign AI Agent evaluation forms until the required Conversation Source is configured.

Then continue with:

Search and select one or more Experience Flows.
Review the selected Experience Flows.
Select Next.

Configure Evaluation Metrics

Evaluation metrics define the criteria used to evaluate interactions for audits and AutoQA scoring. The available metric types depend on the selected Handled By type. Manual metrics require evaluator input, while AI-based metrics evaluate interactions by using the configured AI evaluation method.

Human Agent Metrics	AI Agent Metrics
1. Search for and select one or more Evaluation Metrics.	1. Search for and select one or more Evaluation Metrics.
2. Select the Edit icon to configure the metric.	2. Select the Edit icon to configure the metric Weightage or Points, Response Outcomes, Correct Response, and Fatal Error, if applicable for compliance-critical metrics.
3. Configure the Weightage or Points, Response Outcomes, Correct Response, and Fatal Error (if applicable for compliance-critical metrics).	3. Weightage or Points, Response Outcomes, Correct Response, and Fatal Error.
4. Review the Total Positive Weightage and Total Negative Weightage.	4. Review the Total Positive Weightage and Total Negative Weightage.
5. Reorder or delete metrics, if required.	5. Reorder or delete metrics as required.
6. Select Next.	6. Select Create to save the evaluation form

Supported Metric Types

Human Agent	AI Agent
Supports By Question, By Playbook, By Speech, By Hold, By Transfer, By Value, Manual, and other Quality AI metrics.	Supports By Question (GenAI-Based), By Question (Deterministic), and By AI Agent, and Manual evaluation metrics.
Manual metrics are available only with Points scoring.	AI Agent evaluation forms don’t support By Playbook Adherence, By Hold, By Value, By Transfer, or By Speech (Dead Air, Cross-talk, Speaking Rate). These metric types don’t display in the Evaluation Metrics list.

Metric Card Configuration

Configure each metric’s Weightage, Correct Response, and Fatal Error behavior based on whether you enable Trigger Scoring.

Configuration	Description
Trigger Scoring Disabled	Displays a Weightage field to define the metric’s contribution to the overall score. You can also enable Fatal Error to fail the evaluation when the metric requirement isn’t met.
Trigger Scoring Enabled	Displays Yes and No outcome rows. Configure a Weightage, Correct Response, and Fatal Error for each outcome as needed.
Outcome Configuration	Configure Yes and No outcomes with positive, zero, or negative Weightage values. The selected Correct Response determines the score contribution, while the alternate outcome uses its configured weight. Negative scoring applies only at the outcome level.

Configure negative scoring only at the outcome level.

Configure Dispute Allocation (Human Agents Only)

Dispute Resolution Assignment

Turn on Dispute Resolution Assignment.
Select a dispute routing option:
- Same Auditor: Routes disputes to the original evaluator.
- Different Auditor: Routes disputes to another QA evaluator from the selected auditor list.
If you select Different Auditor, search for and select one or more auditors to handle disputes.
Turn on Allow Multi-Round Re-Disputes to let agents raise additional disputes if they disagree with a QA evaluator’s re-evaluation decision. Turn off this option to make the QA evaluator’s re-evaluation final.
Select Max Number of Re-Disputes to define the maximum number of additional dispute rounds allowed after re-evaluation.

If Dispute Resolution Assignment is turned off, agents can’t acknowledge or dispute completed evaluations.

Completed Audit Editing

Turn on Completed Audit Editing to let authorized users update manual audit responses after submitting an audit.
Under Select Auditor, select one or more auditors authorized to edit completed audits.
Select Create to save the configuration.

Metric Rules

The combined positive weightage must equal 100% for percentage-based scoring.
Configure the expected response and score for every metric.
Human Agent forms support all available metric types, and Fatal Error settings apply only to supported metrics.
AI Agent forms support only AI Agent metrics used during Experience Flow evaluations.

Completed Audit Editing enables authorized users to update manual audit responses after an audit is submitted. Re-audits impact the auditor score and are recorded in the trial logs. Edit permissions are based on the users added for form re-audits.

Evaluation Processing

Evaluation Behavior

During evaluation, the system selects the applicable form based on the queue, channel, and contact direction. The system skips conversations without matching assignments. Manual metrics work only with points-based scoring and remain excluded from automated scorecards.

AI Agent-to-Human Agent Handoff Evaluation

For conversations that include an AI Agent-to-Human Agent handoff, each conversation segment is evaluated using its corresponding evaluation form:

AI Agent segment: Uses the AI Agent evaluation form assigned to the corresponding Experience Flow.
Human Agent segment: Uses the Human Agent evaluation form assigned to the corresponding Queue.

Supported By Question metrics can use transcript context from both segments when evaluating the applicable conversation segment.

Form Selection Logic

During evaluation, the system selects the applicable Evaluation Form based on the configured assignment criteria.

Human Agent conversations: The system selects the Evaluation Form based on the assigned Queue, Channel, and Contact Direction. An interaction is eligible for evaluation when it’s associated with a queue that has an active evaluation form matching the configured channel and contact direction.
AI Agent conversations: The system selects the Evaluation Form based on the assigned Automation AI Experience Flow and Channel. An interaction is eligible for evaluation when it’s associated with an Experience Flow that has an active Evaluation Form for the configured channel.

For Human Agent conversations, the system determines evaluation eligibility based on the conversation’s queue assignment and associated Evaluation Form, not the handling agent’s queue membership or workforce management assignment. Conversations remain eligible when the queue, channel, and contact direction match an active Evaluation Form.

Contact Duration Evaluation

The system evaluates contact duration before scoring.

Contact Duration	Status Assigned	Result
Meets or Exceeds Threshold	—	Evaluated normally
Falls Below Threshold	Below Threshold	Excluded from scoring and quality metrics
Duration Unresolved	Duration unavailable	Excluded from evaluation

Threshold updates apply only to future interactions. A contact may qualify for one evaluation form but not another.

Scoring

Scoring Logic

Pass: Final score ≥ Pass Score threshold
Fail: Final score < Pass Score threshold

The system calculates the final score using positive and negative metric weights.

Not Applicable Scoring Treatment

A metric can have a Not Applicable outcome when a Dynamic metric’s trigger condition isn’t met or an auditor manually marks the metric as Not Applicable. Choose one of the following scoring treatments:

Treatment	Effect	Formula
Exclude from Score (default)	Excludes the Not Applicable metric’s weightage from the score calculation by removing it from both the numerator and denominator.	Score % = (Weightage of Adhered) / (Weightage of Adhered + Weightage of Not Adhered) × 100
Award Full Weightage	Includes the Not Applicable metric’s weightage as full credit in both the numerator and denominator. The metric remains marked as Not Applicable.	Score % = (Weightage of Adhered + Weightage of Not Applicable) / (Weightage of Adhered + Weightage of Not Adhered + Weightage of Not Applicable) × 100

Example: A form contains five metrics, each weighted 20, for a total weightage of 100:

3 Adhered metrics = 60 weightage
1 Not Adhered metric = 20 weightage
1 Not Applicable metric = 20 weightage

Treatment	Calculation	Score
Exclude from Score	60 / (60 + 20)	75%
Award Full Weightage	(60 + 20) / (60 + 20 + 20)	80%

The system applies this treatment to all metric types and versions for both AutoQA scores and manual supervisor audit scores. The treatment changes only the metric’s weightage contribution. The Not Applicable label, AI justification, and adherence state remain unchanged. Changes to this setting apply only to new evaluations. The system doesn’t rescore previously evaluated conversations.

Points-Based Scoring Formula

Kore Evaluation Score calculates the weighted impact of met and not-met metrics, subtracts penalties, divides by total positive weights, and multiplies the result by 100.

\left(\frac{\sum(M_{yi}\times W_{yi})-\sum(M_{ni}\times W_{ni})}{\sum(W_{yi})}\right)\times100

where,

Myi, Wyi = Adhered metrics and positive points.
Mni, Wni = Non-adhered metrics and negative points.

Percentage vs. Points Comparison

Quality AI supports two scoring methods.

Feature	Percentage-Based	Points-Based
Best For	Fewer than 20 metrics	20 or more metrics
Total Weight	Must equal 100%	No fixed maximum
Scalability	Limited	High
Weight Per Metric	Decreases as metrics increase (for example, 40 metrics ≈ 2.5% each)	Assign any point value based on importance
Weight Precision	May require fractional values (for example, 2.5%)	Uses whole-number allocation (for example, 50 points for critical metrics)
Negative Scoring	Managed within 100% structure	Allows negative points but limits them to the total positive points
Final Evaluation Score	Direct percentage (0–100)	Normalized percentage (0–100)

Weight Assignment Rules

Configuration	Percentage-Based	Points-Based
Correct Response = Yes	Positive % for Yes, zero or negative % for No	Positive points for Yes, zero or negative points for No
Correct Response = No	Positive % for No, zero or negative % for Yes	Positive points for No, zero or negative points for Yes
Validation	Total positive weight must equal 100%. Negative weight can remain within the 100% structure.	Total positive points have no upper limit. Total negative points can’t exceed total positive points.
Manual Evaluation Metrics	Not supported	Supported

Fatal Error Behavior

Fatal error configuration remains the same for both scoring types. When a fatal metric fails, the system sets the final score to 0 and marks the interaction as failed, regardless of other metric scores.

Manage Evaluation Forms

Edit a Form

Select Edit from the Evaluation Forms list. You can modify: General Settings, Queue or Experience Flow assignments, Evaluation Metrics, and Dispute Allocation (Human Agent only). Select Update to save changes.

Delete a Form

Select Delete from the Evaluation Forms list. Before deleting:

Remove all Queue or Experience Flow assignments.
Resolve dependent metric or attribute references.

If the form is in use, the system displays a warning before deletion. Select Update to save changes.

Versioning

Every update creates a new version of the Evaluation Form. Existing evaluations continue to use the version that was active when the evaluation started, while future evaluations use the latest published version. Changing the Not Applicable Scoring Treatment follows the same rule. It applies only to evaluations that start after the change, and it never re-scores evaluations already completed or in progress. The system also maintains a history of assignment and configuration changes, including changes to the Not Applicable Scoring Treatment setting.

Conversation Type Support Summary

Conversation Type	Evaluation Support
Human Agent conversation	Supports Human Agent Evaluation Forms.
AI Agent conversation	Supports AI Agent Evaluation Forms.
AI Agent → Human Agent handoff	Evaluates AI Agent and Human Agent segments using their respective assigned forms.

Switching Scoring Type

Changing the scoring type clears all existing metric weights and requires reconfiguration. Percentage totals must equal 100%, and points-based values must meet the validation rules.

Unsupported Language Issues (Form-Level)

The system blocks the update when a new language doesn’t match the supported languages for one or more associated metrics. For example, adding Hindi to a form with metrics that support only English and Dutch triggers this error. To resolve it, review each metric’s language configuration, update metrics to support the new language, and then add the language to the form once all metrics are compatible.

Metric-Level Language Limitation

The system displays this warning when you add or update a metric that doesn’t support a language configured on the form. Resolve it by configuring the required language on the metric or selecting a metric that supports all form languages.

Language Selection Behavior

Multi-language selection uses an AND condition. The system displays only By-Question metrics available in all selected languages. For example, when you select English and Dutch, the system displays only metrics available in both languages.

Channel Mode Change

Switching between Voice and Chat triggers a warning and removes speech-based metrics. Update remaining metrics and adjust weights before saving changes.

No Dispute Workflow

AI Agent evaluation forms don’t include Dispute Allocation. Agents can’t acknowledge or dispute AI Agent evaluations.

Speech Metric Limitations

Speech-based evaluation metrics aren’t available for AI Agent evaluation forms because Experience Flow evaluations analyze bot interactions rather than voice quality characteristics.

​Supported Conversation Types

​Key Capabilities

​Evaluation Forms Structure

​Access Evaluation Forms

​Evaluation Forms List

​Evaluation Processing

​Create a New Evaluation Form

​Configure General Settings

​Assignment Requirements

​Configure Assignments

​Human Agent Assignments

​AI Agent Assignments

​Experience Flow Assignment Rules

​Queue Assignment Rules

​AI Agent Assignments

​Configure Evaluation Metrics

​Supported Metric Types

​Metric Card Configuration

​Configure Dispute Allocation (Human Agents Only)

​Dispute Resolution Assignment

​Completed Audit Editing

​Metric Rules

​Evaluation Processing

​Evaluation Behavior

​AI Agent-to-Human Agent Handoff Evaluation

​Form Selection Logic

​Contact Duration Evaluation

​Scoring

​Scoring Logic

​Not Applicable Scoring Treatment

​Points-Based Scoring Formula

​Percentage vs. Points Comparison

​Weight Assignment Rules

​Fatal Error Behavior

​Manage Evaluation Forms

​Edit a Form

​Delete a Form

​Versioning

​Conversation Type Support Summary

​Switching Scoring Type

​Unsupported Language Issues (Form-Level)

​Metric-Level Language Limitation

​Language Selection Behavior

​Channel Mode Change

​No Dispute Workflow

​Speech Metric Limitations

Supported Conversation Types

Key Capabilities

Evaluation Forms Structure

Access Evaluation Forms

Evaluation Forms List

Evaluation Processing

Create a New Evaluation Form

Configure General Settings

Assignment Requirements

Configure Assignments

Human Agent Assignments

AI Agent Assignments

Experience Flow Assignment Rules

Queue Assignment Rules

AI Agent Assignments

Configure Evaluation Metrics

Supported Metric Types

Metric Card Configuration

Configure Dispute Allocation (Human Agents Only)

Dispute Resolution Assignment

Completed Audit Editing

Metric Rules

Evaluation Processing

Evaluation Behavior

AI Agent-to-Human Agent Handoff Evaluation

Form Selection Logic

Contact Duration Evaluation

Scoring

Scoring Logic

Not Applicable Scoring Treatment

Points-Based Scoring Formula

Percentage vs. Points Comparison

Weight Assignment Rules

Fatal Error Behavior

Manage Evaluation Forms

Edit a Form

Delete a Form

Versioning

Conversation Type Support Summary

Switching Scoring Type

Unsupported Language Issues (Form-Level)

Metric-Level Language Limitation

Language Selection Behavior

Channel Mode Change

No Dispute Workflow

Speech Metric Limitations