> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipeshub.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawling manager

# Crawling Manager API

The Crawling Manager API provides endpoints for scheduling, managing, and monitoring data crawling jobs across various connector types including Google Workspace, OneDrive, SharePoint Online, Slack, and Confluence.

## Base URL

All endpoints are prefixed with `/api/v1/crawlingManager`

## Authentication

All endpoints require:

* **Authentication** via `Authorization` header with valid JWT token
* **Admin privileges** - Only users with admin role can access these endpoints

**Headers:**

```
Authorization: Bearer <jwt_token>
```

## Supported Connectors

* `gmail` - Gmail connector
* `drive` - Google Drive connector
* `onedrive` - OneDrive connector
* `sharepointonline` - SharePoint Online connector
* `confluence` - Confluence connector
* `slack` - Slack connector
* `linear` - Linear connector
* `dropbox` - Dropbox connector
* `outlook` - Outlook connector
* `jira` - Jira connector
* `atlassian` - Atlassian connector
* `github` - GitHub connector
* `box` - Box connector
* `s3` - Amazon S3 connector
* `azure` - Azure connector
* `airtable` - Airtable connector
* `zammad` - Zammad connector
* `zendesk` - Zendesk connector

**Note:** Connector names are case-insensitive and spaces are automatically removed during processing. Job IDs are generated using the format: `crawl-{connector}-{orgId}` where connector is normalized to lowercase with hyphens.

## API Endpoints

<AccordionGroup>
  <Accordion title="POST /:connector/schedule - Schedule Job">
    Schedule a new crawling job for a specific connector type.

    **Endpoint:** `POST /api/v1/crawlingManager/:connector/schedule`

    **Parameters:**

    * `connector` (string, path) - The connector type (gmail, drive, onedrive, sharepointonline, confluence, slack, linear, dropbox, outlook, jira, atlassian, github, box, s3, azure, airtable, zammad, zendesk)

    <Tabs>
      <Tab title="Request Body">
        ```json theme={null}
        {
          "scheduleConfig": {
            "scheduleType": "daily|weekly|monthly|hourly|custom|once",
            "isEnabled": true,
            "timezone": "UTC",
            // Schedule-specific configuration based on scheduleType
          },
          "priority": 5,          // 1-10, default: 5
          "maxRetries": 3,        // 0-10, default: 3
          "timeout": 300000       // 1000-600000ms, default: 300000 (5 minutes)
        }
        ```

        **Schedule Configuration Types:**

        All schedule configurations inherit base properties:

        * `scheduleType` (required) - The type of schedule
        * `isEnabled` (boolean, default: true) - Whether the schedule is enabled
        * `timezone` (string, default: "UTC") - Timezone for schedule execution

        **Hourly Schedule:**

        ```json theme={null}
        {
          "scheduleType": "hourly",
          "minute": 0,            // 0-59, required
          "interval": 1,          // 1-24, default: 1 (every X hours)
          "isEnabled": true,
          "timezone": "UTC"
        }
        ```

        **Daily Schedule:**

        ```json theme={null}
        {
          "scheduleType": "daily", 
          "hour": 9,              // 0-23, required
          "minute": 0,            // 0-59, required
          "isEnabled": true,
          "timezone": "UTC"
        }
        ```

        **Weekly Schedule:**

        ```json theme={null}
        {
          "scheduleType": "weekly",
          "daysOfWeek": [1, 2, 3, 4, 5],  // Array of 0-6 (Sunday-Saturday), min 1 item
          "hour": 9,                      // 0-23, required
          "minute": 0,                    // 0-59, required
          "isEnabled": true,
          "timezone": "UTC"
        }
        ```

        **Monthly Schedule:**

        ```json theme={null}
        {
          "scheduleType": "monthly",
          "dayOfMonth": 1,        // 1-31, required
          "hour": 9,              // 0-23, required
          "minute": 0,            // 0-59, required
          "isEnabled": true,
          "timezone": "UTC"
        }
        ```

        **Custom Schedule (Cron):**

        ```json theme={null}
        {
          "scheduleType": "custom",
          "cronExpression": "0 9 * * 1-5",  // Required, must match pattern: ^(\S+\s+){4}\S+$
          "description": "Weekdays at 9 AM", // Optional
          "isEnabled": true,
          "timezone": "UTC"
        }
        ```

        **Once Schedule:**

        ```json theme={null}
        {
          "scheduleType": "once",
          "scheduledTime": "2024-12-31T23:59:59.000Z",  // ISO 8601 datetime string
          "isEnabled": true,
          "timezone": "UTC"
        }
        ```
      </Tab>

      <Tab title="Success Response">
        **Status:** `201 Created`

        ```json theme={null}
        {
          "success": true,
          "message": "Crawling job scheduled successfully",
          "data": {
            "jobId": "bull:crawling-scheduler:123",
            "connector": "linear",
            "scheduleConfig": {
              "scheduleType": "daily",
              "hour": 9,
              "minute": 0,
              "isEnabled": true,
              "timezone": "UTC"
            },
            "scheduledAt": "2024-01-15T10:30:00.000Z"
          }
        }
        ```
      </Tab>

      <Tab title="Error Responses">
        **Status:** `400 Bad Request`

        ```json theme={null}
        {
          "success": false,
          "message": "Invalid schedule configuration",
          "error": {
            "code": "VALIDATION_ERROR",
            "details": "Schedule type 'invalid' is not supported"
          }
        }
        ```

        **Status:** `400 Bad Request` (Disabled Schedule)

        ```json theme={null}
        {
          "success": false,
          "message": "Cannot schedule a disabled job"
        }
        ```

        **Status:** `400 Bad Request` (Past Time for Once Schedule)

        ```json theme={null}
        {
          "success": false,
          "message": "Scheduled time must be in the future"
        }
        ```

        **Status:** `401 Unauthorized`

        ```json theme={null}
        {
          "success": false,
          "message": "Unauthorized access"
        }
        ```

        **Status:** `403 Forbidden`

        ```json theme={null}
        {
          "success": false,
          "message": "Admin privileges required"
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="GET /:connector/schedule - Get Job Status">
    Retrieve the status of a scheduled crawling job for a specific connector.

    **Endpoint:** `GET /api/v1/crawlingManager/:connector/schedule`

    **Parameters:**

    * `connector` (string, path) - The connector type

    <Tabs>
      <Tab title="Success Response">
        **Status:** `200 OK`

        ```json theme={null}
        {
          "success": true,
          "message": "Job status retrieved successfully", 
          "data": {
            "id": "bull:crawling-scheduler:123",
            "name": "crawl-box",
            "data": {
              "connector": "box",
              "scheduleConfig": {
                "scheduleType": "daily",
                "hour": 9,
                "minute": 0,
                "isEnabled": true,
                "timezone": "UTC"
              },
              "orgId": "64f1a2b3c4d5e6f7a8b9c0d1",
              "userId": "64f1a2b3c4d5e6f7a8b9c0d2",
              "timestamp": "2024-01-15T09:00:00.000Z",
              "metadata": {}
            },
            "progress": 75,
            "delay": null,
            "timestamp": 1705392000000,
            "attemptsMade": 1,
            "finishedOn": null,
            "processedOn": 1705392060000,
            "failedReason": null,
            "state": "active"
          }
        }
        ```

        **Job States:**

        * `waiting` - Job is waiting to be processed
        * `active` - Job is currently being processed
        * `completed` - Job completed successfully
        * `failed` - Job failed with errors
        * `delayed` - Job is delayed for future execution
        * `paused` - Job is paused

        **Progress Tracking:** Jobs report progress at key stages: 10% (start), 20% (task service obtained), 100% (completion). Failed jobs may show partial progress.
      </Tab>

      <Tab title="Error Responses">
        **Status:** `404 Not Found`

        ```json theme={null}
        {
          "success": false,
          "message": "No scheduled job found for this connector",
          "data": null
        }
        ```

        **Status:** `401 Unauthorized`

        ```json theme={null}
        {
          "success": false,
          "message": "Unauthorized access"
        }
        ```

        **Status:** `403 Forbidden`

        ```json theme={null}
        {
          "success": false,
          "message": "Admin privileges required"
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="GET /schedule/all - Get All Job Statuses">
    Retrieve all scheduled crawling jobs for the organization.

    **Endpoint:** `GET /api/v1/crawlingManager/schedule/all`

    <Tabs>
      <Tab title="Success Response">
        **Status:** `200 OK`

        ```json theme={null}
        {
          "success": true,
          "message": "All job statuses retrieved successfully",
          "data": [
            {
              "id": "bull:crawling-scheduler:123",
              "name": "crawl-github", 
              "data": {
                "connector": "github",
                "scheduleConfig": {
                  "scheduleType": "daily",
                  "hour": 9,
                  "minute": 0,
                  "isEnabled": true,
                  "timezone": "UTC"
                },
                "orgId": "64f1a2b3c4d5e6f7a8b9c0d1",
                "userId": "64f1a2b3c4d5e6f7a8b9c0d2",
                "timestamp": "2024-01-15T09:00:00.000Z"
              },
              "progress": 100,
              "state": "completed",
              "finishedOn": 1705392120000
            },
            {
              "id": "crawl-zendesk-64f1a2b3c4d5e6f7a8b9c0d1",
              "name": "crawl-zendesk",
              "data": {
                "connector": "zendesk", 
                "scheduleConfig": {
                  "scheduleType": "weekly",
                  "daysOfWeek": [1, 3, 5],
                  "hour": 10,
                  "minute": 0,
                  "isEnabled": true,
                  "timezone": "UTC"
                },
                "orgId": "64f1a2b3c4d5e6f7a8b9c0d1",
                "userId": "64f1a2b3c4d5e6f7a8b9c0d2", 
                "timestamp": "2024-01-15T08:00:00.000Z"
              },
              "progress": 0,
              "state": "paused"
            },
            {
              "id": "crawl-linear-64f1a2b3c4d5e6f7a8b9c0d1",
              "name": "crawl-linear",
              "data": {
                "connector": "linear",
                "scheduleConfig": {
                  "scheduleType": "hourly",
                  "minute": 30,
                  "interval": 2,
                  "isEnabled": true,
                  "timezone": "UTC"
                },
                "orgId": "64f1a2b3c4d5e6f7a8b9c0d1",
                "userId": "64f1a2b3c4d5e6f7a8b9c0d2",
                "timestamp": "2024-01-15T10:30:00.000Z"
              },
              "progress": 45,
              "state": "active"
            }
          ]
        }
        ```
      </Tab>

      <Tab title="Error Responses">
        **Status:** `401 Unauthorized`

        ```json theme={null}
        {
          "success": false,
          "message": "Unauthorized access"
        }
        ```

        **Status:** `403 Forbidden`

        ```json theme={null}
        {
          "success": false,
          "message": "Admin privileges required"
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="DELETE /:connector/schedule - Remove Job">
    Remove a scheduled crawling job for a specific connector.

    **Endpoint:** `DELETE /api/v1/crawlingManager/:connector/schedule`

    **Parameters:**

    * `connector` (string, path) - The connector type

    <Tabs>
      <Tab title="Success Response">
        **Status:** `200 OK`

        ```json theme={null}
        {
          "success": true,
          "message": "Crawling job removed successfully"
        }
        ```
      </Tab>

      <Tab title="Error Responses">
        **Status:** `401 Unauthorized`

        ```json theme={null}
        {
          "success": false,
          "message": "Unauthorized access"
        }
        ```

        **Status:** `403 Forbidden`

        ```json theme={null}
        {
          "success": false,
          "message": "Admin privileges required"
        }
        ```

        **Status:** `500 Internal Server Error`

        ```json theme={null}
        {
          "success": false,
          "message": "Internal server error",
          "error": {
            "code": "INTERNAL_ERROR",
            "details": "Failed to remove job from queue"
          }
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="DELETE /schedule/all - Remove All Jobs">
    Remove all scheduled crawling jobs for the organization.

    **Endpoint:** `DELETE /api/v1/crawlingManager/schedule/all`

    <Tabs>
      <Tab title="Success Response">
        **Status:** `200 OK`

        ```json theme={null}
        {
          "success": true,
          "message": "All crawling jobs removed successfully"
        }
        ```
      </Tab>

      <Tab title="Error Responses">
        **Status:** `401 Unauthorized`

        ```json theme={null}
        {
          "success": false,
          "message": "Unauthorized access"
        }
        ```

        **Status:** `403 Forbidden`

        ```json theme={null}
        {
          "success": false,
          "message": "Admin privileges required"
        }
        ```

        **Status:** `500 Internal Server Error`

        ```json theme={null}
        {
          "success": false,
          "message": "Internal server error"
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="POST /:connector/pause - Pause Job">
    Pause a scheduled crawling job for a specific connector.

    **Endpoint:** `POST /api/v1/crawlingManager/:connector/pause`

    **Parameters:**

    * `connector` (string, path) - The connector type

    <Tabs>
      <Tab title="Success Response">
        **Status:** `200 OK`

        ```json theme={null}
        {
          "success": true,
          "message": "Crawling job paused successfully",
          "data": {
            "connector": "airtable",
            "orgId": "64f1a2b3c4d5e6f7a8b9c0d1",
            "pausedAt": "2024-01-15T10:30:00.000Z"
          }
        }
        ```
      </Tab>

      <Tab title="Error Responses">
        **Status:** `400 Bad Request` (No Job Found)

        ```json theme={null}
        {
          "success": false,
          "message": "No active job found to pause"
        }
        ```

        **Status:** `400 Bad Request` (Already Paused)

        ```json theme={null}
        {
          "success": false,
          "message": "Job is already paused"
        }
        ```

        **Status:** `401 Unauthorized`

        ```json theme={null}
        {
          "success": false,
          "message": "Unauthorized access"
        }
        ```

        **Status:** `403 Forbidden`

        ```json theme={null}
        {
          "success": false,
          "message": "Admin privileges required"
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="POST /:connector/resume - Resume Job">
    Resume a paused crawling job for a specific connector.

    **Endpoint:** `POST /api/v1/crawlingManager/:connector/resume`

    **Parameters:**

    * `connector` (string, path) - The connector type

    <Tabs>
      <Tab title="Success Response">
        **Status:** `200 OK`

        ```json theme={null}
        {
          "success": true,
          "message": "Crawling job resumed successfully",
          "data": {
            "connector": "dropbox", 
            "orgId": "64f1a2b3c4d5e6f7a8b9c0d1",
            "resumedAt": "2024-01-15T10:35:00.000Z"
          }
        }
        ```
      </Tab>

      <Tab title="Error Responses">
        **Status:** `400 Bad Request`

        ```json theme={null}
        {
          "success": false,
          "message": "No paused job found to resume"
        }
        ```

        **Status:** `401 Unauthorized`

        ```json theme={null}
        {
          "success": false,
          "message": "Unauthorized access"
        }
        ```

        **Status:** `403 Forbidden`

        ```json theme={null}
        {
          "success": false,
          "message": "Admin privileges required"
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="GET /stats - Get Queue Statistics">
    Retrieve statistics about the crawling job queue.

    **Endpoint:** `GET /api/v1/crawlingManager/stats`

    <Tabs>
      <Tab title="Success Response">
        **Status:** `200 OK`

        ```json theme={null}
        {
          "success": true,
          "message": "Queue statistics retrieved successfully",
          "data": {
            "waiting": 5,
            "active": 2,
            "completed": 120,
            "failed": 3,
            "delayed": 1,
            "paused": 2,
            "repeatable": 8,
            "total": 141
          }
        }
        ```

        **Statistics Fields:**

        * `waiting` - Number of jobs waiting to be processed
        * `active` - Number of jobs currently being processed
        * `completed` - Number of completed jobs
        * `failed` - Number of failed jobs
        * `delayed` - Number of delayed jobs
        * `paused` - Number of paused jobs
        * `repeatable` - Number of repeatable/scheduled jobs
        * `total` - Total number of jobs across all states
      </Tab>

      <Tab title="Error Responses">
        **Status:** `401 Unauthorized`

        ```json theme={null}
        {
          "success": false,
          "message": "Unauthorized access"
        }
        ```

        **Status:** `403 Forbidden`

        ```json theme={null}
        {
          "success": false,
          "message": "Admin privileges required"
        }
        ```

        **Status:** `500 Internal Server Error`

        ```json theme={null}
        {
          "success": false,
          "message": "Failed to retrieve queue statistics"
        }
        ```
      </Tab>
    </Tabs>
  </Accordion>
</AccordionGroup>

## Data Types

<Tabs>
  <Tab title="Enums">
    ```typescript theme={null}
    enum CrawlingScheduleType {
      HOURLY = 'hourly',
      DAILY = 'daily', 
      WEEKLY = 'weekly',
      MONTHLY = 'monthly',
      CUSTOM = 'custom',
      ONCE = 'once'
    }
    ```
  </Tab>

  <Tab title="Interfaces">
    ```typescript theme={null}
    interface CrawlingJobData {
      connector: string;
      scheduleConfig: ICrawlingSchedule;
      orgId: string;
      userId: string;
      timestamp: Date;
      metadata?: Record<string, any>;
    }

    interface ScheduleJobOptions {
      priority?: number;        // 1-10, default: 5
      maxRetries?: number;      // 0-10, default: 3  
      timeout?: number;         // 1000-600000ms, default: 300000 (5 minutes)
      metadata?: Record<string, any>;
    }

    interface JobStatus {
      id: string | undefined;
      name: string;
      data: CrawlingJobData;
      progress: JobProgress;   // Number (0-100) or object from BullMQ
      delay: number | undefined;
      timestamp: number;
      attemptsMade: number;
      finishedOn: number | undefined;
      processedOn: number | undefined; 
      failedReason: string | undefined;
      state: string;            // waiting, active, completed, failed, delayed, paused
    }
    ```
  </Tab>
</Tabs>

## System Configuration

<Tabs>
  <Tab title="Rate Limiting">
    The API includes built-in rate limiting and concurrency controls:

    * Maximum 5 concurrent jobs per queue
    * Jobs are automatically retried with exponential backoff (5000ms initial delay) on failure
    * Stalled jobs are detected after 30 seconds
    * Maximum 3 retry attempts for failed jobs
    * Job history retention: Last 10 completed and 10 failed jobs per connector type
    * Jobs are removed and recreated when updating schedules (no job modification)
  </Tab>

  <Tab title="Job Lifecycle">
    1. **Scheduled** - Job is created and scheduled based on configuration
    2. **Waiting** - Job is queued and waiting for worker availability
    3. **Active** - Job is being processed by a worker
    4. **Completed** - Job finished successfully
    5. **Failed** - Job failed after all retry attempts
    6. **Paused** - Job execution is paused (manual action)

    <Note>
      **Paused Jobs:** When a job is paused, its configuration is stored in memory and the active job is removed from the queue. Resuming recreates the job with the stored configuration. Paused jobs persist only during the service runtime and are lost on restart.
    </Note>
  </Tab>

  <Tab title="Sync Events">
    Upon successful job completion, the system publishes sync events:

    * `drive` → `SyncDriveEvent`
    * `gmail` → `SyncGmailEvent`
    * `onedrive` → `SyncOneDriveEvent`
    * `sharepointonline` → `SyncSharePointOnlineEvent`
    * `linear` → `SyncLinearEvent`
    * `dropbox` → `SyncDropboxEvent`
    * `outlook` → `SyncOutlookEvent`
    * `jira` → `SyncJiraEvent`
    * `atlassian` → `SyncAtlassianEvent`
    * `github` → `SyncGithubEvent`
    * `box` → `SyncBoxEvent`
    * `s3` → `SyncS3Event`
    * `azure` → `SyncAzureEvent`
    * `airtable` → `SyncAirtableEvent`
    * `zammad` → `SyncZammadEvent`
    * `zendesk` → `SyncZendeskEvent`
    * `confluence` and `slack` → `ReindexAllRecordEvent` (fallback)

    Events include metadata such as `orgId`, `connector`, `origin: 'CONNECTOR'`, and timestamps for tracking and audit purposes.
  </Tab>
</Tabs>

## Error Handling

All endpoints follow a consistent error response format:

```json theme={null}
{
  "success": false,
  "message": "Error description",
  "error": {
    "code": "ERROR_CODE",
    "details": "Additional error details"
  }
}
```

**Common HTTP Status Codes:**

* `200` - Success
* `201` - Created (for scheduling jobs)
* `400` - Bad Request (validation errors, invalid configuration)
* `401` - Unauthorized (missing or invalid authentication)
* `403` - Forbidden (insufficient privileges, admin required)
* `404` - Not Found (job not found)
* `500` - Internal Server Error