> ## Documentation Index
> Fetch the complete documentation index at: https://koreai-content-gov.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Crawl API

<Badge icon="arrow-left" color="gray">[Back to API List](/ai-for-service/apis/searchai/api-list)</Badge>

The Web Crawl APIs enable you to create and manage website sources, configure sitemap-based crawling, and trigger crawl operations programmatically.

## Create a Web Source

Creates a new website source for crawling.

| Field | Value |
| - | - |
| **Method** | POST |
| **Endpoint** | `<host_url>/api/public/bot/:appId/sources` |
| **Content-Type** | `application/json` |
| **Authorization** | `auth: <JWT Token>` |
| **API Scope** | Data Ingestion |

### Query Parameters

| Parameter | Description | Mandatory |
| - | - | - |
| appId | Provide your application ID here. | Yes |

### Request Parameters

| Field | Type | Required | Description |
| - | - | - | - |
| `name` | String | Yes | Name of the web source. |
| `desc` | String | No | Description of the web source. |
| `sourceType` | String | Yes | Source type. Supported values: `url`, `uploadSitemap`. |
| `url` | String | Required for `url` | Website URL to crawl. |
| `advanceOpts` | Object | No | Advanced crawl settings. |
| `advanceOpts.maxUrlLimit` | Integer | No | Maximum number of URLs to crawl. |
| `advanceOpts.crawlDepth` | Integer | No | Maximum crawl depth. |
| `authorizationEnabled` | Boolean | No | Enables authorization for the source. |

### Sample Response

```json expandable=true theme={null}
{
    "_id": "fs-52d84393-769d-5707-b413-a60726a8b385",
    "jobId": "fj-23fbd6e1-db5f-5a96-b1ff-acfdcd3791b3",
    "extractionType": "web",
    "contentSource": "web",
    "autoGen": false,
    "recentStatus": "configured",
    "name": "Example source",
    "desc": "Optional source description",
    "searchIndexId": "sidx-403e99ff-4e42-5980-b4ce-dd69db50a5d9",
    "createdBy": "u-0555dabf-d7a4-57c4-ae45-8995f25710f9",
    "lModBy": "u-0555dabf-d7a4-57c4-ae45-8995f25710f9",
    "createdOn": "2026-08-31T11:12:05.483Z",
    "lMod": "2026-08-31T11:12:05.000Z",
    "display": true,
    "rAcl": [
        "*"
    ],
    "numOfDocs": 0,
    "totalSize": 0,
    "url": "https://example.com",
    "settings": {
        "blockHttpsMsgs": false,
        "isJavaScriptRendered": false,
        "crawlBeyondSitemaps": false,
        "respectRobotTxtDirectives": true,
        "useCookies": true,
        "crawlDepth": 5,
        "maxUrlLimit": 10
    },
    "advanceSettings": {
        "scheduleOpt": false,
        "repeatInterval": "",
        "crawlEverything": true,
        "allowedOpt": false,
        "allowedURLs": [],
        "blockedOpt": false,
        "blockedURLs": [],
        "blockHttpsMsgs": false,
        "isJavaScriptRendered": false,
        "crawlBeyondSitemaps": false,
        "respectRobotTxtDirectives": true,
        "useCookies": true,
        "crawlDepth": 1,
        "maxUrlLimit": 5,
        "crawlDelay": 0,
        "processingOpts": true
    },
    "sitemaps": [],
    "validations": {
        "url": {
            "validated": true,
            "msg": "Initial URL validation successful",
            "description": ""
        },
        "networkConnectivity": {
            "validated": true,
            "msg": "Network Connectivity test successful",
            "description": ""
        }
    },
    "authorizationEnabled": false,
    "authorizationProfileId": null,
    "isURLValid": true,
    "customCrawler": {
        "enabled": false,
        "authDetails": {
            "reqHeaders": []
        },
        "useProfile": false
    },
    "filesMetaData": [],
    "sourceType": "url",
    "isDeleted": false,
    "__v": 0
}
```

### Configure an uploaded sitemap for web crawl

To configure a web source using a sitemap, complete the following steps.

**Step 1: Upload the Sitemap**

Upload the sitemap file using the public upload API. Use multipart/form-data with the following fields:

| Field | Description |
| - | - |
| `file` | Sitemap file to upload. |
| `fileContext` | Set to `findly`. |
| `fileExtension` | Extension of the uploaded sitemap file. |

Save the returned `fileId` from the response.

**Step 2: Attach the Sitemap**

Update the web source with the uploaded sitemap using the Create Web Crawl endpoint. Set the sourceType as `uploadSitemap`, as shown below.

```json expandable=true theme={null}
{
  "sourceType": "uploadSitemap",
  "fileId": "<uploaded-file-id>"
}

```

***

## Initiate a web crawl

This API triggers web crawl.

| Field | Value |
| - | - |
| **Method** | POST |
| **Endpoint** | `<host_url>/api/public/bot/:appId/sources/:sourceId/crawl` |
| **Content-Type** | `application/json` |
| **Authorization** | `auth: <JWT Token>` |
| **API Scope** | Data Ingestion |

### Query Parameters

| Parameter | Description | Mandatory |
| - | - | - |
| appId | Provide your application ID here. | Yes |
| sourceId | Provide the unique identifier for the web source. | Yes |

### Request Parameters

| Field | Type | Required | Description |
| - | - | - | - |
| `forceRecrawl` | Boolean | Yes | When forceRecrawl is true, the API triggers the existing recrawl flow. Otherwise it triggers the existing extract flow. |

```json expandable=true theme={null}
{
  "forceRecrawl": false
}
```