Connector Manager API

The Connector Manager Service provides centralized management of authentication tokens and connector configurations for third-party integrations. This service handles OAuth flows, token lifecycle management, and connector configurations for various external services like Google Workspace, OneDrive, SharePoint, Confluence, and more.

Base URL

All endpoints are prefixed with /api/v1/connectors

Authentication

All endpoints require authentication via Bearer token:
Authorization: Bearer YOUR_TOKEN
Internal endpoints use scoped token authentication for service-to-service communication with specific scopes:
  • FETCH_CONFIG - For configuration updates and token refresh operations

Architecture Overview

The Connector Manager Service is built on a Node.js backend with MongoDB for data persistence, Redis for caching, and ETCD for secure configuration storage. The service consists of several key components:
  • Token Management - Handles OAuth token lifecycle including creation, refresh, and revocation
  • Connector Configuration - Manages connector settings and credentials
  • Event Broadcasting - Kafka-based events for system integration
  • Health Monitoring - Service health checks and dependency monitoring
The service integrates with these components:
  • Configuration Manager - Manages encrypted configuration storage
  • IAM Service - Handles user authentication and authorization
  • Kafka - Event streaming for token and connector state changes
  • ETCD - Encrypted key-value storage for sensitive configurations
  • Backend Connector Service - Python service that handles actual connector implementations

Data Models

Connectors

Connector configurations that define enabled/disabled state and metadata for third-party integrations.

Token Events

Events broadcast when tokens are created, refreshed, revoked, or expired.

Entity Events

Events broadcast when connectors are enabled or disabled, triggering synchronization workflows.

Configuration

Encrypted storage of OAuth credentials, API keys, and service endpoints.

API Endpoints

Manage connector configurations including retrieval, updates, OAuth flows, and filtering options.
Retrieve all connectors for the organization (admin access required).
Endpoint: GET /api/v1/connectors/Access Control: Admin privileges requiredAuthentication: Bearer token required
Retrieve all currently active/enabled connectors.
Endpoint: GET /api/v1/connectors/activeAuthentication: Bearer token required
Retrieve all currently inactive/disabled connectors.
Endpoint: GET /api/v1/connectors/inactiveAuthentication: Bearer token required
Retrieve configuration for a specific connector.
Endpoint: GET /api/v1/connectors/config/:connectorNamePath Parameters:
  • connectorName: Connector name (string, required)
Access Control: Admin privileges required
GET /api/v1/connectors/config/OneDrive
Update configuration for a specific connector.
Endpoint: PUT /api/v1/connectors/config/:connectorNamePath Parameters:
  • connectorName: Connector name (string, required)
Access Control: Admin privileges requiredRequest Body Parameters:
ParameterTypeRequiredDescription
authobjectNoAuthentication configuration
syncobjectNoSynchronization settings
filtersobjectNoFilter configurations
baseUrlstringYesBase URL for the connector
{
    "auth": {
        "clientId": "xxxxxxx",
        "clientSecret": "xxxxxxxx",
        "tenantId": "xxxxxxx",
        "hasAdminConsent": true,
        "redirectUri": ""
    },
    "sync": {
        "selectedStrategy": "SCHEDULED",
        "scheduledConfig": {}
    },
    "filters": {}
}
Retrieve the configuration schema for a specific connector type.
Endpoint: GET /api/v1/connectors/schema/:connectorNamePath Parameters:
  • connectorName: Connector name (string, required)
Access Control: Admin privileges required
Enable or disable a specific connector.
Endpoint: POST /api/v1/connectors/toggle/:connectorNamePath Parameters:
  • connectorName: Connector name (string, required)
Access Control: Admin privileges required
Handle OAuth authorization flows for connector authentication.
Generate OAuth authorization URL for a connector.
Endpoint: GET /api/v1/connectors/:connectorName/oauth/authorizePath Parameters:
  • connectorName: Connector name (string, required)
Query Parameters:
ParameterTypeRequiredDescription
baseUrlstringYesBase URL for the connector service
Access Control: Admin privileges required
GET /api/v1/connectors/google_workspace/oauth/authorize?baseUrl=https://api.example.com
Handle OAuth callback after user authorization.
Endpoint: GET /api/v1/connectors/:connectorName/oauth/callbackPath Parameters:
  • connectorName: Connector name (string, required)
Query Parameters:
ParameterTypeRequiredDescription
codestringNoAuthorization code from OAuth provider
statestringNoState parameter for CSRF protection
errorstringNoError code if authorization failed
baseUrlstringYesBase URL for the connector service
Access Control: Admin privileges required
GET /api/v1/connectors/google_workspace/oauth/callback?code=AUTH_CODE&state=STATE&baseUrl=https://api.example.com
Exchange OAuth authorization code for access and refresh tokens (Google Workspace specific).
Endpoint: POST /api/v1/connectors/getTokenFromCodeAccess Control: Admin privileges requiredRequest Body Parameters:
ParameterTypeRequiredDescription
tempCodestringYesOAuth authorization code from Google
{
  "tempCode": "4/0AX4XfWjYX3Z9X8Y7W6V5U4T3S2R1Q0P"
}
Process:
  1. Retrieves Google Workspace configuration from Configuration Manager
  2. Exchanges authorization code for access and refresh tokens via Google’s token endpoint
  3. Verifies the ID token matches the authenticated user’s email
  4. Stores credentials securely via Configuration Manager
  5. Creates or updates connector in MongoDB
  6. Publishes connector enabled event to Kafka
  7. Determines enabled apps based on OAuth scopes received
Manage connector-specific filtering options and configurations.
Retrieve filter options for a specific connector.
Endpoint: GET /api/v1/connectors/filters/:connectorNamePath Parameters:
  • connectorName: Connector name (string, required)
Access Control: Admin privileges required
Save or update filter options for a specific connector.
Endpoint: POST /api/v1/connectors/filters/:connectorNamePath Parameters:
  • connectorName: Connector name (string, required)
Access Control: Admin privileges requiredRequest Body Parameters:
ParameterTypeRequiredDescription
filterOptionsobjectYesFilter configuration object
{
  "filterOptions": {
    "fileTypes": ["pdf", "docx"],
    "folders": {
      "includeSharedDrives": false,
      "excludedFolders": ["/Archive", "/Personal"]
    },
    "dateRange": {
      "enabled": true,
      "from": "2024-01-01",
      "to": "2024-12-31"
    }
  }
}
Internal endpoints for service-to-service communication and system maintenance.
Refresh an expired access token using a refresh token (Google Workspace specific).
Endpoint: POST /api/v1/connectors/internal/refreshIndividualConnectorTokenAuthentication: Requires scoped token with FETCH_CONFIG scopeProcess:
  1. Retrieves refresh token from Configuration Manager
  2. Exchanges refresh token for new access token via Google’s token endpoint
  3. Implements retry logic with exponential backoff (3 attempts)
  4. Updates stored credentials via Configuration Manager
  5. Maintains existing refresh token and expiry times
Note: This endpoint automatically retrieves the refresh token from secure storage and exchanges it for a new access token.
Update the application configuration for the connector manager service.
Endpoint: POST /api/v1/connectors/updateAppConfigAuthentication: Requires scoped token with FETCH_CONFIG scopeNote: This endpoint reloads the application configuration from the configuration manager and updates the dependency injection container.
Monitor the health and status of the Connector Manager Service and its dependencies.
Retrieve the health status of the service and all its dependencies.
Endpoint: GET /api/v1/healthAuthentication: No authentication required for health checks

Event System

The Connector Manager Service broadcasts events through Kafka to notify other services about token and connector state changes. These events trigger actions like connector synchronization, indexing updates, and audit logging.

Event Topics

TopicDescription
token-eventsToken lifecycle events (creation, refresh, revocation, expiration)
entity-eventsConnector state change events (enabled, disabled)

Event Types

Event TypeDescription
TOKEN_CREATEDTriggered when a new token is acquired
TOKEN_REFRESHEDTriggered when a token is refreshed
TOKEN_REVOKEDTriggered when a token is revoked
TOKEN_EXPIREDTriggered when a token expires
appEnabledTriggered when a connector is enabled
appDisabledTriggered when a connector is disabled

Event Payload Structures

{
  "eventId": "event_123456",
  "eventType": "TOKEN_REFRESHED",
  "tokenReferenceId": "token_ref_789",
  "serviceType": "GOOGLE_DRIVE",
  "accountId": "user@example.com",
  "timestamp": 1714208400000
}
Service Types:
  • ONEDRIVE
  • GOOGLE_DRIVE
  • CONFLUENCE
  • JIRA
{
  "eventType": "appEnabled",
  "timestamp": 1714208400000,
  "payload": {
    "orgId": "org123456",
    "appGroup": "Google Workspace",
    "appGroupId": "connector_123",
    "credentialsRoute": "https://cm.example.com/api/v1/configurationManager/internal/connectors/googleWorkspaceCredentials",
    "refreshTokenRoute": "https://cm.example.com/api/v1/connectors/internal/refreshIndividualConnectorToken",
    "apps": ["DRIVE", "GMAIL", "CALENDAR"],
    "syncAction": "immediate"
  }
}
Sync Actions:
  • none - No synchronization required
  • immediate - Start synchronization immediately
  • scheduled - Schedule synchronization for later
{
  "eventType": "appDisabled",
  "timestamp": 1714208400000,
  "payload": {
    "orgId": "org123456",
    "appGroup": "Google Workspace",
    "appGroupId": "connector_123",
    "apps": ["DRIVE", "GMAIL", "CALENDAR"]
  }
}

Schema Definitions

interface AppConfig {
  // Authentication secrets
  jwtSecret: string;
  scopedJwtSecret: string;
  cookieSecret: string;
  rsAvailable: string;

  // Service endpoints
  communicationBackend: string;
  frontendUrl: string;
  iamBackend: string;
  authBackend: string;
  cmBackend: string;
  kbBackend: string;
  esBackend: string;
  storageBackend: string;
  tokenBackend: string;
  aiBackend: string;
  connectorBackend: string;
  connectorPublicUrl: string;
  indexingBackend: string;

  // Infrastructure configuration
  kafka: {
    brokers: string[];
    sasl?: {
      mechanism: 'plain' | 'scram-sha-256' | 'scram-sha-512';
      username: string;
      password: string;
    };
  };
  redis: {
    host: string;
    port: number;
    password?: string;
    db?: number;
  };
  mongo: {
    uri: string;
    db: string;
  };
  qdrant: {
    port: number;
    apiKey: string;
    host: string;
    grpcPort: number;
  };
  arango: {
    url: string;
    db: string;
    username: string;
    password: string;
  };
  etcd: {
    host: string;
    port: number;
    dialTimeout: number;
  };
  smtp: {
    host: string;
    port: number;
    username?: string;
    password?: string;
    fromEmail: string;
  } | null;
  storage: {
    storageType: string;
    endpoint: string;
  };
}
enum ServiceType {
  ONEDRIVE = 'ONEDRIVE',
  GOOGLE_DRIVE = 'GOOGLE_DRIVE',
  CONFLUENCE = 'CONFLUENCE',
  JIRA = 'JIRA',
}

enum TokenEventType {
  TOKEN_CREATED = 'TOKEN_CREATED',
  TOKEN_REFRESHED = 'TOKEN_REFRESHED',
  TOKEN_REVOKED = 'TOKEN_REVOKED',
  TOKEN_EXPIRED = 'TOKEN_EXPIRED',
}

interface ITokenEvent {
  eventId: string;
  eventType: TokenEventType;
  tokenReferenceId: string;
  serviceType: ServiceType;
  accountId: string;
  timestamp: Number;
}
enum SyncAction {
  None = 'none',
  Immediate = 'immediate',
  Scheduled = 'scheduled',
}

enum EventType {
  AppEnabledEvent = 'appEnabled',
  AppDisabledEvent = 'appDisabled',
}

interface Event {
  eventType: EventType;
  timestamp: number;
  payload: AppEnabledEvent | AppDisabledEvent;
}

interface AppEnabledEvent {
  orgId: string;
  appGroup: string;
  appGroupId: string;
  credentialsRoute?: string;
  refreshTokenRoute?: string;
  apps: string[];
  syncAction: SyncAction;
}

interface AppDisabledEvent {
  orgId: string;
  appGroup: string;
  appGroupId: string;
  apps: string[];
}
enum GoogleWorkspaceApp {
  Drive = 'DRIVE',
  Gmail = 'GMAIL',
  Calendar = 'CALENDAR',
}

interface GoogleWorkspaceApps {
  apps: GoogleWorkspaceApp[];
}

const scopeToAppMap: { [key: string]: GoogleWorkspaceApp } = {
  'https://www.googleapis.com/auth/gmail.readonly': GoogleWorkspaceApp.Gmail,
  'https://www.googleapis.com/auth/calendar.readonly': GoogleWorkspaceApp.Calendar,
  'https://www.googleapis.com/auth/drive.readonly': GoogleWorkspaceApp.Drive,
};
interface HealthStatus {
  status: 'healthy' | 'unhealthy';
  timestamp: string;
  services: {
    redis: string;
    kafka: string;
    mongodb: string;
    arangodb: string;
    KVStoreservice: string;
  };
}
// Update connector config validation
const updateConnectorConfigSchema = z.object({
  body: z.object({
    auth: z.any(),
    sync: z.any(),
    filters: z.any(),
    baseUrl: z.string(),
  }),
  params: z.object({
    connectorName: z.string(),
  }),
});

// OAuth authorization URL validation
const getOAuthAuthorizationUrlSchema = z.object({
  params: z.object({
    connectorName: z.string(),
  }),
  query: z.object({
    baseUrl: z.string(),
  }),
});

// OAuth callback validation
const handleOAuthCallbackSchema = z.object({
  params: z.object({
    connectorName: z.string(),
  }),
  query: z.object({
    baseUrl: z.string(),
    code: z.string().optional(),
    state: z.string().optional(),
    error: z.string().optional(),
  }),
});

Configuration Management

The Connector Manager Service uses ETCD as a key-value store for managing configurations securely. All sensitive information, such as OAuth client IDs/secrets and access tokens, are encrypted using AES-256 before storage.

Configuration Types

The service manages several types of configurations:
  1. Service Connection Information
    • Database connection strings (MongoDB, ArangoDB)
    • Message broker details (Kafka brokers, authentication)
    • Cache settings (Redis host, port, credentials)
  2. Service Discovery
    • Backend service endpoints (IAM, Auth, Configuration Manager, etc.)
    • Frontend URL for OAuth redirects
  3. Authentication Secrets
    • JWT signing keys
    • Scoped JWT secrets for service-to-service communication
    • Cookie encryption secrets
  4. Connector Credentials
    • OAuth client IDs and secrets
    • Access tokens and refresh tokens
    • Token expiration timestamps

Security Measures

  • All sensitive configuration data is encrypted using AES-256 before storage in ETCD
  • Encryption keys are securely managed and not stored in plaintext
  • Scoped JWT tokens are used for internal service communication
  • Token refresh operations include retry logic with exponential backoff

Error Handling

All endpoints return structured error responses with specific HTTP status codes:
{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid request parameters",
    "details": "Connector name is required"
  },
  "meta": {
    "requestId": "req-error-123",
    "timestamp": "2025-04-27T13:20:00.000Z"
  }
}
Common HTTP Status Codes:
  • 200 - Success
  • 201 - Created (for new resources)
  • 400 - Bad Request (validation errors, missing parameters)
  • 401 - Unauthorized (missing or invalid authentication)
  • 403 - Forbidden (insufficient privileges, admin required)
  • 404 - Not Found (connector or resource not found)
  • 500 - Internal Server Error (backend service failures)
Backend Service Error Handling:
  • Connection refused → “Connector Service is currently unavailable. Please check your network connection or try again later.”
  • Service-specific errors → Mapped to appropriate HTTP status codes with detailed error messages
  • Automatic retry logic with exponential backoff for failed requests (3 retries)

Integration Architecture

Backend Connector Service Integration

The Connector Manager Service acts as a proxy to a Python-based Backend Connector Service that handles the actual connector implementations. Most endpoints forward requests to this backend service and return the responses. Request Flow:
  1. Client sends request to Connector Manager Service
  2. Connector Manager validates authentication and authorization
  3. Request is forwarded to Backend Connector Service
  4. Backend Connector Service processes the request
  5. Response is returned through Connector Manager to client

Configuration Manager Integration

The Connector Manager Service integrates closely with the Configuration Manager for:
  1. Secure Credential Storage - OAuth client IDs, secrets, and tokens
  2. Service Discovery - Endpoint URLs for backend services
  3. Configuration Management - Connector settings and preferences
  4. Encrypted Data Handling - All sensitive data is encrypted before storage

Google Workspace OAuth Flow

For Google Workspace connectors, the service implements a complete OAuth 2.0 flow:
  1. Authorization Request - Generate authorization URL with appropriate scopes
  2. Code Exchange - Exchange authorization code for access and refresh tokens
  3. Token Verification - Verify ID token matches authenticated user
  4. Credential Storage - Securely store tokens via Configuration Manager
  5. Connector Activation - Enable connector and publish events
  6. Scope Mapping - Map OAuth scopes to enabled applications (Drive, Gmail, Calendar)

Token Lifecycle Management

The service provides automatic token management with the following features:
  1. Token Acquisition - OAuth flow handling and token exchange
  2. Token Refresh - Automatic refresh before expiration with retry logic
  3. Token Revocation - Secure token deletion and cleanup
  4. Event Broadcasting - Kafka events for token state changes
  5. Error Handling - Retry logic and fallback mechanisms