Token Manager Service

The Token Manager Service manages authentication tokens for third-party integrations and connectors within your organization’s ecosystem. It handles token lifecycle management, including creation, refreshing, and revocation of tokens for various services like Google Workspace, ensuring secure and seamless integration with external systems.

Architecture Overview

The Token Manager Service is built on a Node.js backend with MongoDB for data persistence and Redis for caching. It leverages several key components:

ETCD - For storing sensitive configuration and token data in an encrypted format
Kafka - For event-driven communication between services
IAM Service - Handles user authentication and authorization
Configuration Manager - Manages application settings and connector configurations

The service securely stores OAuth credentials and automatically refreshes tokens when they expire, maintaining continuous connectivity with integrated systems.

Connectors API

The Connectors API enables integration with third-party services such as Google Workspace.

Get Connector Credentials

Retrieves connector credentials for the authenticated user’s organization.

GET /api/v1/connectors/credentials

Set Connector Credentials

Sets or updates connector credentials for the authenticated user’s organization.

POST /api/v1/connectors/credentials

Delete Connector Credentials

Deletes connector credentials for the authenticated user’s organization.

DELETE /api/v1/connectors/credentials

Download Connector Credentials

Downloads the connector credentials as a JSON file.

GET /api/v1/connectors/credentials/download

Get Connectors Status

Retrieves the status of all connectors for the authenticated user’s organization.

GET /api/v1/connectors/status

Get Connector Configuration

Retrieves the configuration for a specific connector.

GET /api/v1/connectors/config?service=googleWorkspace

Set Connector Configuration

Sets or updates the configuration for a specific connector.

POST /api/v1/connectors/config?service=googleWorkspace

Disable Connector

Disables a specific connector for the authenticated user’s organization.

POST /api/v1/connectors/disable?service=googleWorkspace

Enable Connector

Enables a specific connector for the authenticated user’s organization.

POST /api/v1/connectors/enable?service=googleWorkspace

Exchange OAuth Code for Tokens

Exchanges an OAuth authorization code for access and refresh tokens.

POST /api/v1/connectors/getTokenFromCode

Refresh Individual Connector Token

Refreshes an expired access token using a refresh token. This endpoint is for internal service communication.

POST /api/v1/connectors/internal/refreshIndividualConnectorToken

Update App Configuration

Updates the application configuration for the token manager service. This endpoint is for internal service communication.

POST /api/v1/connectors/updateAppConfig

Health API

The Health API allows for monitoring the health status of the Token Manager Service and its dependencies.

Get Health Status

Retrieves the health status of the service and its dependencies.

GET /api/v1/health

Schema Definitions

Application Config Schema

interface AppConfig {
  // JWT and authentication secrets
  jwtSecret: string;
  scopedJwtSecret: string;
  cookieSecret: string;
  rsAvailable: string;

  // Service endpoints
  communicationBackend: string;
  frontendUrl: string;
  iamBackend: string;
  authBackend: string;
  cmBackend: string;
  kbBackend: string;
  esBackend: string;
  storageBackend: string;
  tokenBackend: string;
  aiBackend: string;
  connectorBackend: string;
  indexingBackend: string;
  
  // Kafka configuration
  kafka: {
    brokers: string[];
    sasl?: {
      mechanism: 'plain' | 'scram-sha-256' | 'scram-sha-512';
      username: string;
      password: string;
    };
  };
  
  // Redis configuration
  redis: {
    host: string;
    port: number;
    password?: string;
    db?: number;
  };
  
  // MongoDB configuration
  mongo: {
    uri: string;
    db: string;
  };

  // Qdrant configuration
  qdrant: {
    port: number;
    apiKey: string;
    host: string;
    grpcPort: number;
  };

  // ArangoDB configuration
  arango: {
    url: string;
    db: string;
    username: string;
    password: string;
  };
  
  // ETCD configuration
  etcd: {
    host: string;
    port: number;
    dialTimeout: number;
  };

  // SMTP configuration
  smtp: {
    host: string;
    port: number;
    username?: string;
    password?: string;
    fromEmail: string;
  } | null;
  
  // Storage configuration
  storage: {
    storageType: string;
    endpoint: string;
  };
}

Connectors Config Schema

interface ConnectorsConfig {
  // Organization and user identifiers
  orgId: string;
  name: string;
  lastUpdatedBy: string;
  
  // Status
  isEnabled: boolean;
  
  // Timestamps
  createdAt: Date;
  updatedAt: Date;
}

Token Event Schema

enum ServiceType {
  ONEDRIVE = 'ONEDRIVE',
  GOOGLE_DRIVE = 'GOOGLE_DRIVE',
  CONFLUENCE = 'CONFLUENCE',
}

enum TokenEventType {
  TOKEN_CREATED = 'TOKEN_CREATED',
  TOKEN_REFRESHED = 'TOKEN_REFRESHED',
  TOKEN_REVOKED = 'TOKEN_REVOKED',
  TOKEN_EXPIRED = 'TOKEN_EXPIRED',
}

interface ITokenEvent {
  eventId: string;
  eventType: TokenEventType;
  tokenReferenceId: string;
  serviceType: ServiceType;
  accountId: string;
  timestamp: Number;
}

Entity Event Schema

enum SyncAction {
  None = 'none',
  Immediate = 'immediate',
  Scheduled = 'scheduled',
}

enum EventType {
  AppEnabledEvent = 'appEnabled',
  AppDisabledEvent = 'appDisabled',
}

interface Event {
  eventType: EventType;
  timestamp: number;
  payload: AppEnabledEvent | AppDisabledEvent;
}

interface AppEnabledEvent {
  orgId: string;
  appGroup: string;
  appGroupId: string;
  credentialsRoute?: string;
  refreshTokenRoute?: string;
  apps: string[];
  syncAction: SyncAction;
}

interface AppDisabledEvent {
  orgId: string;
  appGroup: string;
  appGroupId: string;
  apps: string[];
}

Google Workspace Types

enum GoogleWorkspaceApp {
  Drive = 'DRIVE',
  Gmail = 'GMAIL',
  Calendar = 'CALENDAR',
}

interface GoogleWorkspaceApps {
  apps: GoogleWorkspaceApp[];
}

const scopeToAppMap: { [key: string]: GoogleWorkspaceApp } = {
  'https://www.googleapis.com/auth/gmail.readonly': GoogleWorkspaceApp.Gmail,
  'https://www.googleapis.com/auth/calendar': GoogleWorkspaceApp.Calendar,
  'https://www.googleapis.com/auth/drive': GoogleWorkspaceApp.Drive,
};

Health Status Schema

interface HealthStatus {
  status: 'healthy' | 'unhealthy';
  timestamp: string;
  services: {
    redis: string;
    kafka: string;
    mongodb: string;
    arangodb: string;
    KVStoreservice: string;
  };
}

Configuration Management

The Token Manager Service uses ETCD as a key-value store for managing configurations securely. All sensitive information, such as OAuth client IDs/secrets and access tokens, are encrypted before storage.

Configuration Types

The service manages several types of configurations:

Service Connection Information
- Database connection strings (MongoDB, ArangoDB)
- Message broker details (Kafka brokers, authentication)
- Cache settings (Redis host, port, credentials)
Service Discovery
- Backend service endpoints (IAM, Auth, Configuration Manager, etc.)
- Frontend URL for OAuth redirects
Authentication Secrets
- JWT signing keys
- Scoped JWT secrets for service-to-service communication
- Cookie encryption secrets
Connector Credentials
- OAuth client IDs and secrets
- Access tokens and refresh tokens
- Token expiration timestamps

Security Measures

All sensitive configuration data is encrypted using AES-256 before storage in ETCD. The encryption keys themselves are securely managed and not stored in plaintext in the codebase or configuration files.

Event-Driven Architecture

The Token Manager Service uses Kafka as an event bus to communicate token-related events to other services. This enables other components to react to token lifecycle events appropriately.

Token Events

Token events are published to the token-events Kafka topic and include:

TOKEN_CREATED: Emitted when a new token is acquired
TOKEN_REFRESHED: Emitted when a token is refreshed
TOKEN_REVOKED: Emitted when a token is explicitly revoked
TOKEN_EXPIRED: Emitted when a token has expired

Entity Events

Entity events are published to the entity-events Kafka topic and include:

AppEnabledEvent: Emitted when a connector is enabled
AppDisabledEvent: Emitted when a connector is disabled

These events trigger synchronization jobs in other services, such as initiating content indexing when a new data source is connected.

Welcome To PipesHub

System Overview

Authentication

Mail Configuration

AI Providers

Connectors

User Management

Deployment

Developer

Additional Resources

Token Manager

Token Manager Service

Architecture Overview

Connectors API

Get Connector Credentials

Set Connector Credentials

Delete Connector Credentials

Download Connector Credentials

Get Connectors Status

Get Connector Configuration

Set Connector Configuration

Disable Connector

Enable Connector

Exchange OAuth Code for Tokens

Refresh Individual Connector Token

Update App Configuration

Health API

Get Health Status

Schema Definitions

Configuration Management

Configuration Types

Security Measures

Event-Driven Architecture

Token Events

Entity Events

Welcome To PipesHub

System Overview

Authentication

Mail Configuration

AI Providers

Connectors

User Management

Deployment

Developer

Additional Resources

​Token Manager Service

​Architecture Overview

​Connectors API

​Get Connector Credentials

​Set Connector Credentials

​Delete Connector Credentials

​Download Connector Credentials

​Get Connectors Status

​Get Connector Configuration

​Set Connector Configuration

​Disable Connector

​Enable Connector

​Exchange OAuth Code for Tokens

​Refresh Individual Connector Token

​Update App Configuration

​Health API

​Get Health Status

​Schema Definitions

​Configuration Management

​Configuration Types

​Security Measures

​Event-Driven Architecture

​Token Events

​Entity Events

Token Manager Service

Architecture Overview

Connectors API

Get Connector Credentials

Set Connector Credentials

Delete Connector Credentials

Download Connector Credentials

Get Connectors Status

Get Connector Configuration

Set Connector Configuration

Disable Connector

Enable Connector

Exchange OAuth Code for Tokens

Refresh Individual Connector Token

Update App Configuration

Health API

Get Health Status

Schema Definitions

Configuration Management

Configuration Types

Security Measures

Event-Driven Architecture

Token Events

Entity Events