​

Storage Service API

​

Base URL

​

Authentication

Authorization: Bearer YOUR_TOKEN

• STORAGE_TOKEN - For storage operations
• FETCH_CONFIG - For configuration updates

​

Architecture Overview

1. Amazon S3 - For cloud-based storage on AWS
2. Azure Blob Storage - For cloud-based storage on Microsoft Azure
3. Local Storage - For on-premises file storage

• Configuration Manager - Manages storage configuration settings stored in etcd
• IAM Service - Handles user authentication and authorization
• Key-Value Store - Provides access to configuration data from etcd

​

Storage Configuration

​

Authentication Modes

1. User Authentication - Uses standard user JWT tokens for operations performed by end users
2. Service Authentication - Uses scoped tokens with specific permissions for service-to-service communication

​

Data Models

​

Document

• Basic information (name, path, size, MIME type)
• Versioning metadata and history
• Storage location details
• Permissions and ownership
• Custom metadata

​

Version History

• Previous versions of the document
• Metadata for each version (size, creation time, creator)
• Storage locations for each version
• Version notes and labels

​

API Endpoints

{
  "documentName": "large-dataset",
  "documentPath": "analytics/data",
  "isVersionedFile": false,
  "extension": "csv",
  "permissions": "owner"
}

​

Schema Definitions

interface Document {
  // Basic document information
  documentName: string;
  alternateDocumentName?: string;
  documentPath?: string;
  isVersionedFile: boolean;
  orgId: mongoose.Types.ObjectId;
  
  // Document metrics
  mutationCount?: number;
  sizeInBytes?: number;
  mimeType?: string;
  extension: string;
  
  // Access control
  permissions?: 'owner' | 'editor' | 'commentator' | 'readonly';
  initiatorUserId: mongoose.Types.ObjectId | null;
  
  // Versioning
  versionHistory?: DocumentVersion[];
  currentVersion: number;
  
  // Metadata
  customMetadata?: CustomMetadata[];
  tags?: string[];
  
  // Timestamps
  createdAt?: number;
  updatedAt?: number;
  
  // Deletion status
  deletedByUserId?: mongoose.Types.ObjectId;
  isDeleted: boolean;
  
  // Storage information
  s3?: StorageInfo;
  azureBlob?: StorageInfo;
  local?: StorageInfo;
  storageVendor: StorageVendor;
}

​

Supported MIME Types

​

Storage Provider Implementation

interface S3StorageConfig {
  accessKeyId: string;
  secretAccessKey: string;
  region: string;
  bucketName: string;
}

https://{bucket}.s3.{region}.amazonaws.com/{orgId}/PipesHub/{path}/{documentId}/current/{name}.{ext}

​

File Size Limits & Direct Upload

​

Error Handling

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid request parameters",
    "details": {
      "field": "documentName",
      "issue": "Document name cannot contain extensions"
    }
  },
  "timestamp": "2025-04-27T13:20:00.000Z"
}

• VALIDATION_ERROR - Invalid request parameters
• NOT_FOUND - Document not found
• BAD_REQUEST - Invalid request format
• FORBIDDEN - File format mismatch or access denied
• INTERNAL_SERVER_ERROR - Storage service errors

• Extension validation failures
• File size limit exceeded
• Storage vendor configuration errors
• Version control constraint violations
• MIME type validation failures

• 200 - Success
• 308 - Permanent Redirect (for direct uploads)
• 400 - Bad Request
• 401 - Unauthorized
• 403 - Forbidden
• 404 - Not Found
• 500 - Internal Server Error

​

Configuration Management

{
  "storageType": "s3|azureBlob|local",
  "credentials": {
    // Provider-specific configuration
  }
}

​

Important Notes

1. Schema Discrepancy: There’s a mismatch between the rollback validator schema (expects version in query) and controller implementation (reads version from body). Documentation follows schema specification.
2. File Extensions: Automatically detected from original filename and validated against comprehensive MIME type mappings.
3. Version Control: Optional per document via isVersionedFile flag, maintains complete history with metadata and notes.
4. Storage Abstraction: Seamless switching between storage vendors via configuration without code changes.
5. Organization Isolation: All documents are organization-scoped (orgId) for secure multi-tenancy.
6. Soft Deletion: Documents are marked as deleted (isDeleted: true) rather than physically removed.
7. Direct Upload: Large files bypass API server for optimal performance, using vendor-specific presigned URLs.
8. Internal APIs: Service-to-service communication with scoped authentication using specific token scopes.
9. Mutation Tracking: Documents track change count (mutationCount) for auditing and optimization.
10. MIME Type Validation: Extensive validation against supported file types with automatic extension detection.
11. Path Structure: Organized as {orgId}/PipesHub/{documentPath}/{documentId}/current|versions/ for clear file organization.
12. Cross-Platform Support: Local storage handles Windows, macOS, and Linux filesystem differences transparently.