> ## Documentation Index
> Fetch the complete documentation index at: https://koreai-content-gov.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluation Forms
Evaluation Forms provide a flexible framework for evaluating Human Agent and AI Agent interactions using separate evaluation configurations. They support configurable assignments, evaluation criteria, scoring models, duration thresholds, Not Applicable (N/A) scoring treatment, and version-controlled updates.
QA Managers can create evaluation forms based on the interaction type. Each form supports only the metrics and workflows applicable to its assigned conversation type.
Evaluation Forms support two assignment types:
* **Human Agent**: Evaluates human agent interactions routed through contact center queues. Assign forms to Queues and use Human Agent capabilities such as dispute workflows and speech-based evaluation metrics for supported voice interactions.
* **AI Agent**: Evaluates AI Agent interactions handled by Automation AI Experience Flows. Assign forms to Experience Flows and use AI Agent-compatible evaluation metrics. AI Agent forms don't support Human Agent capabilities such as dispute workflows or speech-based evaluation metrics.
For AI Agent-to-Human Agent handoff conversations, the system evaluates AI Agent and Human Agent segments using their respective evaluation forms and applicable capabilities. QAI analytics provide filters to analyze conversations handled by an **AI Agent**, **Human Agent**, or both.
Select the assignment type when creating the evaluation form and don't change it later. Assign each form to either **Queues** (Human Agent) or **Experience Flows** (AI Agent), not both.
## Supported Conversation Types
Evaluation Forms support the following conversation types:
* **Human Agent conversations**: Evaluated using Human Agent evaluation forms assigned to contact center queues.
* **AI Agent conversations**: Evaluated using AI Agent evaluation forms assigned to Automation AI Experience Flows.
* **AI Agent-to-Human Agent handoff conversations**: The system evaluates AI Agent and Human Agent segments using the applicable evaluation workflows and capabilities.
## Key Capabilities
| Capability | Description |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Evaluation Types** | Create evaluation forms for **Human Agent** and **AI Agent** conversations. |
| **Assignment** | Assign **Human Agent** forms to contact center **Queues** and **AI Agent** forms to **Automation AI Experience Flows**. Queue and Experience Flow assignments are mutually exclusive. |
| **Metrics** | Configure metrics based on the selected assignment type. AI Agent forms support **By Question (GenAI-Based)**, **By Question (Deterministic)**, and **By AI Agent** metrics. Human Agent forms support Human Agent-specific metrics, including speech-based metrics for supported voice interactions. |
| **Channel and Language Support** | Configure evaluation forms for supported channels and languages. |
| **Scoring** | Configure percentage- or points-based scoring, pass scores, metric weights, response logic, fatal errors, and negative scoring. |
| **Not Applicable (N/A) Scoring** | Choose whether the system excludes N/A metrics from the score or awards them full weight. |
| **Duration Thresholds** | Specify the minimum interaction duration required for evaluation eligibility. |
| **Conversation Coverage** | Evaluate Human Agent, AI Agent, and AI Agent-to-Human Agent handoff conversations. In handoff conversations, the system evaluates AI Agent and Human Agent segments using their applicable evaluation forms. By Question metrics can use transcript context from both segments. |
| **Dispute Workflow** | Support dispute workflows for **Human Agent** evaluation forms. |
| **Completed Audit Editing**\* | Authorized auditors can update manual audit responses after submitting an audit. |
| **Versioning** | Maintain version history and an audit trail for evaluation form configurations and assignments. |
| **Default AI Agent Evaluation Form** | Automatically create and assign a default AI Agent evaluation form with predefined quality metrics to all Experience Flows. |
***
## Evaluation Forms Structure
Defines the overall configuration of an evaluation form.
| Section | Human Agent | AI Agent |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| **General Settings** | Configure form details, language, channel, scoring, N/A scoring, duration thresholds, pass score, and assignment type. | Configure the same general settings for AI Agent evaluation forms. |
| **Assignments** | Assign evaluation forms to contact center **Queues**, with Contact Direction and Conversation Source. | Assign evaluation forms to **Automation AI Experience Flows** for evaluating AI Agent conversations. |
| **Evaluation Metrics** | Configure supported Human Agent evaluation metrics, including speech-based metrics for supported voice interactions. | Configure AI Agent-compatible metrics, including **By Question (GenAI-Based)**, **By Question (Deterministic)**, and **By AI Agent**. |
| **Advanced Settings** | Configure dispute workflows, acknowledgement workflows, and completed audit editing permissions. | Not supported. |
***
## Access Evaluation Forms
Navigate to **Quality AI** > **Configure** > **Evaluation Forms**.
.
***
### Evaluation Forms List
The Evaluation Forms page displays the following list:
| Field | Description |
| --------------- | -------------------------------------------------------------------------------- |
| **Name** | Displays the evaluation form name. |
| **Description** | Displays the evaluation form description. |
| **Assigned In** | Displays the assigned **Queue** (Human Agent) or **Experience Flow** (AI Agent). |
| **Channel** | Displays the assigned channel (Voice or Chat). |
| **Pass Score** | Displays the minimum passing score. |
| **Status** | Indicates whether the evaluation form is active or inactive. |
| **Created By** | Displays the user who created the evaluation form. |
| **Search** | Searches evaluation forms by name. |
To create **AI Agent** evaluation forms, also enable the **Automation AI Conversation Source**.
***
## Evaluation Processing
For conversations that transfer between an AI Agent and a Human Agent, By Question metrics can evaluate transcript context across both conversation segments. This provides additional context for AI-based evaluation while each conversation segment continues to use its own assigned Evaluation Form.
This behavior applies only to supported By Question metrics.
***
## Create a New Evaluation Form
Evaluation Form creation consists of three or four steps depending on the selected assignment type.
### Configure General Settings
1. Select the **Evaluation Forms** tab.
2. Select **+ New Evaluation Forms**.
3. Enter the following details:
* **Name**: Enter a name for the evaluation form.
* **Description**: Enter a description (optional).
* **Language**: Select one or more supported languages.
4. Select a **Handled By** agent to assign a queue name:
* **Human Agent** (Human Agent forms): Creates an evaluation form for Human Agent conversations using supported evaluation workflows.
* **AI Agent** (Experience Flow name): Creates an evaluation configuration for AI Agent conversations using supported evaluation capabilities.
5. Select a **Channel** type:
* **Chat**: Display only chat metrics (excluding speech and voice-specific Playbook metrics).
* **Voice**: Display all applicable voice metrics, including speech and Playbook metrics.
6. Select a **Scoring Type**:
* **Percentage**: Calculates scores based on configured metric weightages.
* **Points**: Calculates scores based on assigned metric points.
7. Select a **Not Applicable Scoring Treatment** to define how **Not Applicable (N/A)** metrics affect the final evaluation score for future evaluations only.
* **Exclude from Score** (default): Excludes the N/A metric from score calculation by removing its weightage from both the numerator and denominator. The metric doesn’t contribute to the final score.
* **Award Full Weightage**: Includes the N/A metric's configured weightage in both the numerator and denominator, awarding full marks while the metric outcome remains Not Applicable.
8. (Optional) Set the minimum interaction duration required for evaluation by entering values in **MIN** and **SEC**. The system excludes conversations below the configured threshold from evaluation scoring.
9. Set a minimum **Pass Score** required for agents.
10. Select **Next**.
#### Assignment Requirements
The selected Handled By type determines the available configuration options. Human Agent forms require queue assignments, while AI Agent forms require Automation AI Experience Flow assignments.
For Human Agent forms, assign at least one queue before proceeding. The selected Contact Direction determines which interaction types the system evaluates for the assigned queues.
* Changes to **Not Applicable Scoring Treatment** apply only to future evaluations created with the updated evaluation form version. Existing evaluations retain the scoring behavior configured when they were created.
* **Not Applicable Scoring Treatment** applies consistently to both **AutoQA** and **manual supervisor audit** scores across all metric types and versions.
* Attribute and scorecard scoring are configured separately at the **Agent Scorecard** level.
### Configure Assignments
The **Assignments** step defines where the evaluation form applies. Based on the selected **Handled By** type, assign the form to **Queues** for Human Agent conversations or **Experience Flows** for AI Agent conversations. You can assign an evaluation form to **Queues** or **Experience Flows**, but not both.
#### Human Agent Assignments
1. Search and select one or more **Queues**.
2. Configure the **Contact Direction** (Inbound, Outbound, or both) and **Conversation Source** (such as Contact Center AI (CCAI), AgentAI, or Quality AI Express) for each selected queue.
3. Review the selected queue assignments.
4. Select **Next**.
Evaluation metric availability depends on the conversation sources associated with the assigned queues. If an evaluation form includes both **CCAI** or **AgentAI** queues and **Quality AI Express** queues, the **By Playbook** and **By Dialog** evaluation metrics are unavailable because these metrics are not supported across mixed conversation sources.
#### AI Agent Assignments
1. Search and select one or more **Experience Flows**.
2. Review the selected **Experience Flows**.
3. Select **Next**.
**Experience Flows** are retrieved from the configured **Automation AI** environment.
#### Experience Flow Assignment Rules
| Rule | Description |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **One Form Per Flow Per Channel** | Each Experience Flow supports only one active evaluation form per channel. If a form exists for the same flow and channel, the system displays a validation error. |
| **Mutually Exclusive Assignment** | You can assign an evaluation form either to **Queues** or to **Experience Flows**, but not both. |
| **Flow Source** | Experience Flows available for selection come from the **Automation AI** configuration in the same workspace. |
#### Queue Assignment Rules
| Assignment Type | Rule |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Human Agent** | Assign each evaluation form to a unique **Queue + Contact Direction** combination. You can't assign the same combination more than once within a form. |
| **AI Agent** | Assign each **Experience Flow** to only one active evaluation form per channel. You can't combine **Queue** and **Experience Flow** assignments in the same evaluation form. |
## AI Agent Assignments
For **Handled By = AI Agent**, assign the evaluation form to **Automation AI Experience Flows**.
To create AI Agent evaluation forms, you must enable the **Automation AI Conversation Source**. If **Automation AI** isn’t enabled, Experience Flow assignments are unavailable, and you can’t assign AI Agent evaluation forms until the required Conversation Source is configured.
Then continue with:
1. Search and select one or more **Experience Flows**.
2. Review the selected **Experience Flows**.
3. Select **Next**.
### Configure Evaluation Metrics
Evaluation metrics define the criteria used to evaluate interactions for audits and AutoQA scoring. The available metric types depend on the selected Handled By type. Manual metrics require evaluator input, while AI-based metrics evaluate interactions by using the configured AI evaluation method.
| **Human Agent Metrics** | **AI Agent Metrics** |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1. Search for and select one or more **Evaluation Metrics**. | 1. Search for and select one or more **Evaluation Metrics**. |
| 2. Select the **Edit** icon to configure the metric. | 2. Select the **Edit** icon to configure the metric **Weightage** or **Points**, **Response Outcomes**, **Correct Response**, and **Fatal Error**, if applicable for compliance-critical metrics. |
| 3. Configure the **Weightage** or **Points**, **Response Outcomes**, **Correct Response**, and **Fatal Error** (if applicable for compliance-critical metrics). | 3. **Weightage** or **Points**, **Response Outcomes**, **Correct Response**, and **Fatal Error**. |
| 4. Review the **Total Positive Weightage** and **Total Negative Weightage**. | 4. Review the **Total Positive Weightage** and **Total Negative Weightage**. |
| 5. Reorder or delete metrics, if required. | 5. Reorder or delete metrics as required. |
| 6. Select **Next**. | 6. Select **Create** to save the evaluation form |
### Supported Metric Types
| **Human Agent** | **AI Agent** |
| ----------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Supports **By Question**, **By Playbook**, **By Speech**, **By Hold**, **By Transfer**, **By Value**, **Manual**, and other Quality AI metrics. | Supports **By Question (GenAI-Based)**, **By Question (Deterministic)**, and **By AI Agent**, and **Manual** evaluation metrics. |
| Manual metrics are available only with **Points** scoring. | AI Agent evaluation forms don't support **By Playbook Adherence**, **By Hold**, **By Value**, **By Transfer**, or **By Speech** (Dead Air, Cross-talk, Speaking Rate). These metric types don't display in the Evaluation Metrics list. |
### Metric Card Configuration
Configure each metric's **Weightage**, **Correct Response**, and **Fatal Error** behavior based on whether you enable **Trigger Scoring**.
| **Configuration** | **Description** |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Trigger Scoring Disabled** | Displays a **Weightage** field to define the metric's contribution to the overall score. You can also enable **Fatal Error** to fail the evaluation when the metric requirement isn't met. |
| **Trigger Scoring Enabled** | Displays **Yes** and **No** outcome rows. Configure a **Weightage**, **Correct Response**, and **Fatal Error** for each outcome as needed. |
| **Outcome Configuration** | Configure **Yes** and **No** outcomes with positive, zero, or negative **Weightage** values. The selected **Correct Response** determines the score contribution, while the alternate outcome uses its configured weight. Negative scoring applies only at the outcome level. |
Configure negative scoring only at the outcome level.
### Configure Dispute Allocation (Human Agents Only)
#### Dispute Resolution Assignment
1. Turn on **Dispute Resolution** Assignment.
2. Select a dispute routing option:
* **Same Auditor**: Routes disputes to the original evaluator.
* **Different Auditor**: Routes disputes to another QA evaluator from the selected auditor list.
3. If you select **Different Auditor**, search for and select one or more auditors to handle disputes.
4. Turn on **Allow Multi-Round Re-Disputes** to let agents raise additional disputes if they disagree with a QA evaluator's re-evaluation decision. Turn off this option to make the QA evaluator's re-evaluation final.
5. Select **Max Number of Re-Disputes** to define the maximum number of additional dispute rounds allowed after re-evaluation.
If **Dispute Resolution Assignment** is turned off, agents can't acknowledge or dispute completed evaluations.
### Completed Audit Editing
1. Turn on **Completed Audit Editing** to let authorized users update manual audit responses after submitting an audit.
2. Under **Select Auditor**, select one or more auditors authorized to edit completed audits.
3. Select **Create** to save the configuration.
#### Metric Rules
* The combined positive weightage must equal 100% for percentage-based scoring.
* Configure the expected response and score for every metric.
* **Human Agent** forms support all available metric types, and **Fatal Error** settings apply only to supported metrics.
* **AI Agent** forms support only AI Agent metrics used during **Experience Flow** evaluations.
**Completed Audit Editing** enables authorized users to update manual audit responses after an audit is submitted. Re-audits impact the auditor score and are recorded in the trial logs. Edit permissions are based on the users added for form re-audits.
## Evaluation Processing
### Evaluation Behavior
During evaluation, the system selects the applicable form based on the queue, channel, and contact direction. The system skips conversations without matching assignments. Manual metrics work only with points-based scoring and remain excluded from automated scorecards.
#### AI Agent-to-Human Agent Handoff Evaluation
For conversations that include an AI Agent-to-Human Agent handoff, each conversation segment is evaluated using its corresponding evaluation form:
* **AI Agent segment**: Uses the AI Agent evaluation form assigned to the corresponding Experience Flow.
* **Human Agent segment**: Uses the Human Agent evaluation form assigned to the corresponding Queue.
Supported **By Question** metrics can use transcript context from both segments when evaluating the applicable conversation segment.
### Form Selection Logic
During evaluation, the system selects the applicable Evaluation Form based on the configured assignment criteria.
* **Human Agent conversations**: The system selects the **Evaluation Form** based on the assigned **Queue**, **Channel**, and **Contact Direction**. An interaction is eligible for evaluation when it's associated with a queue that has an active evaluation form matching the configured channel and contact direction.
* **AI Agent conversations**: The system selects the **Evaluation Form** based on the assigned **Automation AI Experience Flow** and **Channel**. An interaction is eligible for evaluation when it's associated with an **Experience Flow** that has an active **Evaluation Form** for the configured channel.
For **Human Agent** conversations, the system determines evaluation eligibility based on the conversation’s queue assignment and associated Evaluation Form, not the handling agent’s queue membership or workforce management assignment. Conversations remain eligible when the queue, channel, and contact direction match an active **Evaluation Form**.
#### Contact Duration Evaluation
The system evaluates contact duration before scoring.
| Contact Duration | Status Assigned | Result |
| ----------------------------------- | -------------------- | ----------------------------------------- |
| \*\* Meets or Exceeds Threshold\*\* | — | Evaluated normally |
| **Falls Below Threshold** | Below Threshold | Excluded from scoring and quality metrics |
| **Duration Unresolved** | Duration unavailable | Excluded from evaluation |
Threshold updates apply only to future interactions. A contact may qualify for one evaluation form but not another.
## Scoring
### Scoring Logic
* **Pass:** Final score ≥ Pass Score threshold
* **Fail:** Final score \< Pass Score threshold
The system calculates the final score using positive and negative metric weights.
### Not Applicable Scoring Treatment
A metric can have a Not Applicable outcome when a Dynamic metric's trigger condition isn't met or an auditor manually marks the metric as Not Applicable. Choose one of the following scoring treatments:
| **Treatment** | **Effect** | **Formula** |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Exclude from Score** (default) | Excludes the Not Applicable metric's weightage from the score calculation by removing it from both the numerator and denominator. | **Score % = (Weightage of Adhered) / (Weightage of Adhered + Weightage of Not Adhered) × 100** |
| **Award Full Weightage** | Includes the Not Applicable metric's weightage as full credit in both the numerator and denominator. The metric remains marked as Not Applicable. | **Score % = (Weightage of Adhered + Weightage of Not Applicable) / (Weightage of Adhered + Weightage of Not Adhered + Weightage of Not Applicable) × 100** |
**Example**: A form contains five metrics, each weighted 20, for a total weightage of 100:
* **3 Adhered metrics** = 60 weightage
* **1 Not Adhered metric** = 20 weightage
* **1 Not Applicable metric** = 20 weightage
| **Treatment** | **Calculation** | **Score** |
| ------------------------ | -------------------------- | --------: |
| **Exclude from Score** | 60 / (60 + 20) | **75%** |
| **Award Full Weightage** | (60 + 20) / (60 + 20 + 20) | **80%** |
The system applies this treatment to all metric types and versions for both **AutoQA scores** and **manual supervisor audit scores**. The treatment changes only the metric's weightage contribution. The **Not Applicable** label, AI justification, and adherence state remain unchanged.
Changes to this setting apply only to new evaluations. The system doesn't rescore previously evaluated conversations.
### Points-Based Scoring Formula
Kore Evaluation Score calculates the weighted impact of met and not-met metrics, subtracts penalties, divides by total positive weights, and multiplies the result by 100.
$$
\left(\frac{\sum(M_{yi}\times W_{yi})-\sum(M_{ni}\times W_{ni})}{\sum(W_{yi})}\right)\times100
$$
where,
* **Myi**, **Wyi** = Adhered metrics and positive points.
* **Mni**, **Wni** = Non-adhered metrics and negative points.
### Percentage vs. Points Comparison
Quality AI supports two scoring methods.
| Feature | Percentage-Based | Points-Based |
| -------------------------- | ------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| **Best For** | Fewer than 20 metrics | 20 or more metrics |
| **Total Weight** | Must equal 100% | No fixed maximum |
| **Scalability** | Limited | High |
| **Weight Per Metric** | Decreases as metrics increase (for example, 40 metrics ≈ 2.5% each) | Assign any point value based on importance |
| **Weight Precision** | May require fractional values (for example, 2.5%) | Uses whole-number allocation (for example, 50 points for critical metrics) |
| **Negative Scoring** | Managed within 100% structure | Allows negative points but limits them to the total positive points |
| **Final Evaluation Score** | Direct percentage (0–100) | Normalized percentage (0–100) |
### Weight Assignment Rules
| Configuration | Percentage-Based | Points-Based |
| ----------------------------- | -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| **Correct Response = Yes** | Positive % for Yes, zero or negative % for No | Positive points for Yes, zero or negative points for No |
| **Correct Response = No** | Positive % for No, zero or negative % for Yes | Positive points for No, zero or negative points for Yes |
| **Validation** | Total positive weight must equal 100%. Negative weight can remain within the 100% structure. | Total positive points have no upper limit. Total negative points can't exceed total positive points. |
| **Manual Evaluation Metrics** | Not supported | Supported |
### Fatal Error Behavior
Fatal error configuration remains the same for both scoring types. When a fatal metric fails, the system sets the final score to 0 and marks the interaction as failed, regardless of other metric scores.
## Manage Evaluation Forms
### Edit a Form
Select **Edit** from the Evaluation Forms list.
You can modify: General Settings, Queue or Experience Flow assignments, Evaluation Metrics, and Dispute Allocation (Human Agent only).
Select **Update** to save changes.
### Delete a Form
Select **Delete** from the Evaluation Forms list.
Before deleting:
* Remove all Queue or Experience Flow assignments.
* Resolve dependent metric or attribute references.
If the form is in use, the system displays a warning before deletion.
Select **Update** to save changes.
### Versioning
Every update creates a new version of the Evaluation Form. Existing evaluations continue to use the version that was active when the evaluation started, while future evaluations use the latest published version.
Changing the Not Applicable Scoring Treatment follows the same rule. It applies only to evaluations that start after the change, and it never re-scores evaluations already completed or in progress.
The system also maintains a history of assignment and configuration changes, including changes to the Not Applicable Scoring Treatment setting.
### Conversation Type Support Summary
| **Conversation Type** | **Evaluation Support** |
| ---------------------------------- | ---------------------------------------------------------------------------------- |
| **Human Agent conversation** | Supports **Human Agent Evaluation Forms**. |
| **AI Agent conversation** | Supports **AI Agent Evaluation Forms**. |
| **AI Agent → Human Agent handoff** | Evaluates AI Agent and Human Agent segments using their respective assigned forms. |
### Switching Scoring Type
Changing the scoring type clears all existing metric weights and requires reconfiguration. Percentage totals must equal 100%, and points-based values must meet the validation rules.
#### Unsupported Language Issues (Form-Level)
The system blocks the update when a new language doesn't match the supported languages for one or more associated metrics. For example, adding Hindi to a form with metrics that support only English and Dutch triggers this error.
To resolve it, review each metric’s language configuration, update metrics to support the new language, and then add the language to the form once all metrics are compatible.
#### Metric-Level Language Limitation
The system displays this warning when you add or update a metric that doesn't support a language configured on the form.
Resolve it by configuring the required language on the metric or selecting a metric that supports all form languages.
### Language Selection Behavior
Multi-language selection uses an **AND** condition. The system displays only **By-Question** metrics available in all selected languages. For example, when you select English and Dutch, the system displays only metrics available in both languages.
### Channel Mode Change
Switching between Voice and Chat triggers a warning and removes speech-based metrics. Update remaining metrics and adjust weights before saving changes.
### No Dispute Workflow
AI Agent evaluation forms don't include **Dispute Allocation**. Agents can't acknowledge or dispute AI Agent evaluations.
### Speech Metric Limitations
Speech-based evaluation metrics aren't available for AI Agent evaluation forms because Experience Flow evaluations analyze bot interactions rather than voice quality characteristics.