Storage Service API

The Storage Service provides robust file storage, versioning, and retrieval capabilities across your organization’s infrastructure. This service abstracts away the underlying storage vendor implementation, allowing seamless interaction with files regardless of where they’re physically stored.

Base URL

All endpoints are prefixed with /api/v1/document

Authentication

All endpoints require authentication via Bearer token:

Authorization: Bearer YOUR_TOKEN

Internal endpoints use scoped token authentication for service-to-service communication with specific scopes:

STORAGE_TOKEN - For storage operations
FETCH_CONFIG - For configuration updates

Architecture Overview

The Storage Service is built on a Node.js backend with MongoDB for metadata persistence. It provides a flexible storage interface that supports three storage vendor backends:

Amazon S3 - For cloud-based storage on AWS
Azure Blob Storage - For cloud-based storage on Microsoft Azure
Local Storage - For on-premises file storage

The service integrates with other components such as:

Configuration Manager - Manages storage configuration settings stored in etcd
IAM Service - Handles user authentication and authorization
Key-Value Store - Provides access to configuration data from etcd

Storage Configuration

Storage configuration is maintained in etcd under the /services/storage path. The configuration determines which storage vendor is active and the credentials needed to access it. The service dynamically loads this configuration at runtime and can switch between storage vendors without restarting.

Authentication Modes

The Storage Service offers two authentication modes:

User Authentication - Uses standard user JWT tokens for operations performed by end users
Service Authentication - Uses scoped tokens with specific permissions for service-to-service communication

Data Models

Document

The Document model represents metadata about files stored in the system, including:

Basic information (name, path, size, MIME type)
Versioning metadata and history
Storage location details
Permissions and ownership
Custom metadata

Version History

Each versioned document maintains a complete history of changes, including:

Previous versions of the document
Metadata for each version (size, creation time, creator)
Storage locations for each version
Version notes and labels

API Endpoints

Document Management

Core document operations including upload, retrieval, and deletion.

POST /upload - Upload Document

Upload a new document to the storage service.

Request
Response

Endpoint: POST /api/v1/document/uploadRequest Body Parameters:

Parameter	Type	Required	Description
file	File	Yes	The file to upload (multipart/form-data)
documentName	string	Yes	Name of the document (no extensions allowed)
documentPath	string	No	Optional path for document storage
alternateDocumentName	string	No	Alternative name for the document
permissions	string	No	Access permissions (owner, editor, commentator, readonly)
customMetadata	object	No	Custom metadata to associate with the document
isVersionedFile	string	Yes	Whether the document supports versioning (“true”/“false”)

File Size Limits:

User endpoints: 1GB (1024 * 1024 * 1000 bytes)
Internal endpoints: 100MB (1024 * 1024 * 100 bytes)

Direct Upload Threshold: 0MB (all files trigger direct upload for S3/Azure for large files)Validation Rules:

Document name cannot contain extensions or forward slashes
File must have a valid extension with supported MIME type
File buffer is required in multipart form data

Supported MIME Types: Extensive list including documents (PDF, Office), images (JPEG, PNG, SVG), archives (ZIP, RAR), and many others.

Status: 200 OK (normal upload) or 308 Permanent Redirect (large files requiring direct upload)Normal Upload Response:

{
  "_id": "60f8a5b3e6d8f32a94c12345",
  "documentName": "presentation",
  "documentPath": "org123/PipesHub/marketing/campaigns/60f8a5b3e6d8f32a94c12345",
  "alternateDocumentName": null,
  "isVersionedFile": true,
  "sizeInBytes": 2048576,
  "extension": ".pptx",
  "storageVendor": "s3",
  "s3": {
    "url": "https://bucket.s3.region.amazonaws.com/org123/PipesHub/marketing/campaigns/60f8a5b3e6d8f32a94c12345/current/presentation.pptx"
  },
  "versionHistory": [
    {
      "version": 0,
      "s3": {
        "url": "https://bucket.s3.region.amazonaws.com/org123/PipesHub/marketing/campaigns/60f8a5b3e6d8f32a94c12345/versions/v0.pptx"
      },
      "createdAt": 1626955195000,
      "size": 2048576,
      "extension": ".pptx"
    }
  ],
  "createdAt": 1626955195000,
  "updatedAt": 1626955195000,
  "isDeleted": false,
  "currentVersion": 0,
  "mutationCount": 1
}

Large File Response (308 Redirect):

{
  "_id": "60f8a5b3e6d8f32a94c67890",
  "documentName": "large-video",
  "documentPath": "media/videos",
  "isVersionedFile": false,
  "extension": ".mp4",
  "storageVendor": "s3"
}

Headers for large files:

Location: Presigned URL for direct upload
x-document-id: Document ID for reference
x-document-name: Document name

POST /internal/upload - Internal Upload

Internal endpoint for service-to-service document uploads.

Request
Response

Endpoint: POST /api/v1/document/internal/uploadAuthentication: Requires scoped token with STORAGE_TOKEN scope.Request Body: Same structure as public upload endpoint.File Size Limit: 100MB for internal uploads.

POST /placeholder - Create Document Placeholder

Create a document placeholder for future direct uploads.

Request
Response

Endpoint: POST /api/v1/document/placeholderRequest Body Parameters:

Parameter	Type	Required	Description
documentName	string	Yes	Name of the document (no extensions allowed)
documentPath	string	Yes	Path for document storage
alternateDocumentName	string	No	Alternative name for the document
permissions	string	No	Access permissions
metaData	object	No	Custom metadata
isVersionedFile	boolean	No	Whether the document supports versioning
extension	string	Yes	File extension (e.g., “pdf”)

{
  "documentName": "large-dataset",
  "documentPath": "analytics/data",
  "isVersionedFile": false,
  "extension": "csv",
  "permissions": "owner"
}

Validation:

Document name cannot contain extensions or forward slashes
Extension must be provided separately

Status: 200 OK

{
  "_id": "60f8a5b3e6d8f32a94c67890",
  "documentName": "large-dataset",
  "documentPath": "analytics/data",
  "isVersionedFile": false,
  "extension": ".csv",
  "storageVendor": "s3",
  "createdAt": 1626955245000,
  "updatedAt": 1626955245000,
  "isDeleted": false,
  "currentVersion": 0
}

POST /internal/placeholder - Internal Create Placeholder

Internal endpoint for creating document placeholders.

Request
Response

Endpoint: POST /api/v1/document/internal/placeholderAuthentication: Requires scoped token with STORAGE_TOKEN scope.Request Body: Same structure as public placeholder endpoint.

GET /:documentId - Get Document By ID

Retrieve document metadata by ID.

Request
Response

Endpoint: GET /api/v1/document/:documentIdPath Parameters:

documentId: MongoDB ObjectId (24-character hex string)

Request Headers:

Authorization: Bearer token (required)

Request Body: Contains fileBuffer field for validation (per DocumentIdParams schema)

Status: 200 OK

{
  "_id": "60f8a5b3e6d8f32a94c12345",
  "documentName": "presentation",
  "documentPath": "org123/PipesHub/marketing/campaigns/60f8a5b3e6d8f32a94c12345",
  "alternateDocumentName": null,
  "isVersionedFile": true,
  "sizeInBytes": 2048576,
  "extension": ".pptx",
  "mimeType": "application/vnd.openxmlformats-officedocument.presentationml.presentation",
  "storageVendor": "s3",
  "s3": {
    "url": "https://bucket.s3.region.amazonaws.com/org123/PipesHub/marketing/campaigns/60f8a5b3e6d8f32a94c12345/current/presentation.pptx"
  },
  "versionHistory": [
    {
      "version": 0,
      "s3": {
        "url": "https://bucket.s3.region.amazonaws.com/org123/PipesHub/marketing/campaigns/60f8a5b3e6d8f32a94c12345/versions/v0.pptx"
      },
      "createdAt": 1626955195000,
      "size": 2048576,
      "extension": ".pptx"
    }
  ],
  "permissions": "owner",
  "initiatorUserId": "507f1f77bcf86cd799439011",
  "createdAt": 1626955195000,
  "updatedAt": 1626955195000,
  "isDeleted": false,
  "currentVersion": 0,
  "mutationCount": 1
}

GET /internal/:documentId - Internal Get Document

Internal endpoint for retrieving document metadata.

Request
Response

Endpoint: GET /api/v1/document/internal/:documentIdAuthentication: Requires scoped token with STORAGE_TOKEN scope.

DELETE /:documentId/ - Delete Document

Mark a document as deleted (soft delete).

Request
Response

Endpoint: DELETE /api/v1/document/:documentId/Path Parameters:

documentId: MongoDB ObjectId (24-character hex string)

Note: The endpoint includes a trailing slash in the route definition.

Status: 200 OK

{
  "_id": "60f8a5b3e6d8f32a94c12345",
  "documentName": "presentation",
  "isDeleted": true,
  "deletedByUserId": "5fb3b7c8e1d8f32a94c98765",
  "updatedAt": 1626958795000
}

DELETE /internal/:documentId/ - Internal Delete

Internal endpoint for deleting documents.

Request
Response

Endpoint: DELETE /api/v1/document/internal/:documentId/Authentication: Requires scoped token with STORAGE_TOKEN scope.Query Parameters:

hard (optional): true removes the document instead of marking it deleted. Its stored files are removed first; if that fails, the failure is logged and the document is removed anyway.

Same response structure as public delete endpoint. With hard=true the response is { "deleted": true }, also when the document was already gone.

DELETE /internal/:documentId/purge - Internal Purge

Removes a document for good: every stored copy of its file (the current file and each version), then its metadata. PipesHub calls this when the record that owned the document is deleted, such as a file uploaded to a collection.

Request
Response

Endpoint: DELETE /api/v1/document/internal/:documentId/purgeAuthentication: Requires scoped token with STORAGE_TOKEN scope.Path Parameters:

documentId: MongoDB ObjectId (24-character hex string)

{ "purged": 1 }

purged is 0 when the document was already gone, so repeating the call is safe. If a stored file can’t be removed, the request fails with 503 and the document is kept, so the call can be retried.

DELETE /internal/records/:virtualRecordId/purge - Internal Purge Processed Record

Removes the processed version of a record that indexing stored: the record_{virtualRecordId} and metadata_{virtualRecordId} documents, wherever they are filed (under the record’s folder path, or under records/{virtualRecordId} for older records), with their files. Records with identical content share one virtual record id, so PipesHub calls this only once no record uses it.

Request
Response

Endpoint: DELETE /api/v1/document/internal/records/:virtualRecordId/purgeAuthentication: Requires scoped token with STORAGE_TOKEN scope.Path Parameters:

virtualRecordId: 1 to 128 letters, digits, - and _. It is matched against exact document names and the exact records/{virtualRecordId} path, never as a prefix.

{ "purged": 2 }

As with the document purge, purged is 0 when the documents were already gone, and a file storage can’t remove fails the request with 503, keeping the documents for a retry.

Document Download & Access

Operations for downloading and accessing document content.

GET /:documentId/download - Download Document

Get a signed URL to download a document or stream it for local storage.

Request
Response

Endpoint: GET /api/v1/document/:documentId/downloadPath Parameters:

documentId: MongoDB ObjectId (24-character hex string)

Query Parameters:

Parameter	Type	Required	Description
version	string	No	Version number to download (transformed to number, must be > 0)
expirationTimeInSeconds	string	No	Expiration time for the signed URL (transformed to number, must be > 0, default: 3600)

Validation:

Version is transformed from string to number and must be greater than 0 if provided
expirationTimeInSeconds is transformed from string to number and must be greater than 0 if provided
Version must exist in document’s version history
Only versioned documents can specify version parameter
Storage vendor must match current configuration

Status: 200 OKFor S3/Azure (JSON Response):

{
  "signedUrl": "https://bucket.s3.region.amazonaws.com/org123/PipesHub/marketing/campaigns/60f8a5b3e6d8f32a94c12345/current/presentation.pptx?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=..."
}

For Local Storage: File is streamed directly with appropriate headers:

Content-Type: Based on file extension via getMimeType()
Content-Disposition: attachment; filename="documentName.extension"

GET /internal/:documentId/download - Internal Download

Internal endpoint for downloading documents.

Request
Response

Endpoint: GET /api/v1/document/internal/:documentId/downloadAuthentication: Requires scoped token with STORAGE_TOKEN scope.Query Parameters: Same as public download endpoint.

GET /:documentId/buffer - Get Document Buffer

Get the document’s content as a buffer.

Request
Response

Endpoint: GET /api/v1/document/:documentId/bufferPath Parameters:

documentId: MongoDB ObjectId (24-character hex string)

Query Parameters:

Parameter	Type	Required	Description
version	string	No	Version number to retrieve (transformed to number, min: 0)

Validation:

Version is transformed from string to number via .pipe(z.number().min(0).optional())
Version must exist if specified
Version cannot exceed available version history length

Status: 200 OKResponse: Binary buffer data of the document content.Error Response (when buffer retrieval fails):

"Failed to get buffer"

GET /internal/:documentId/buffer - Internal Get Buffer

Internal endpoint for retrieving document buffers.

Request
Response

Endpoint: GET /api/v1/document/internal/:documentId/bufferAuthentication: Requires scoped token with STORAGE_TOKEN scope.Query Parameters: Same as public buffer endpoint.

PUT /:documentId/buffer - Update Document Buffer

Update a document’s content.

Request
Response

Endpoint: PUT /api/v1/document/:documentId/bufferPath Parameters:

documentId: MongoDB ObjectId (24-character hex string)

Request Body Parameters:

file: File to upload (multipart/form-data with field name ‘file’)

File Size Limit: 100MB (1024 * 1024 * 100 bytes)Validation: Uses DocumentIdParams schema which includes fileBuffer in bodyBehavior:

Updates the document content in storage
Increments mutation count
Updates document size metadata

Status: 200 OK

"https://bucket.s3.region.amazonaws.com/org123/PipesHub/marketing/campaigns/60f8a5b3e6d8f32a94c12345/current/presentation.pptx"

Response: Storage URL of the updated document.

PUT /internal/:documentId/buffer - Internal Update Buffer

Internal endpoint for updating document buffers.

Request
Response

Endpoint: PUT /api/v1/document/internal/:documentId/bufferAuthentication: Requires scoped token with STORAGE_TOKEN scope.Request Body: Same as public buffer update endpoint.

Document Versioning

Version control operations for document lifecycle management.

POST /:documentId/uploadNextVersion - Upload Next Version

Upload a new version of an existing document.

Request
Response

Endpoint: POST /api/v1/document/:documentId/uploadNextVersionPath Parameters:

documentId: MongoDB ObjectId (24-character hex string)

Request Body Parameters:

Parameter	Type	Required	Description
file	File	Yes	The new file content (multipart/form-data with field name ‘file’)
currentVersionNote	string	No	Note for the current version (if modified)
nextVersionNote	string	No	Note for the next version

File Size Limit: 100MB (1024 * 1024 * 100 bytes)Validation:

Document must support versioning (isVersionedFile: true)
File extension must match existing document extension
Document existence and access verification

Behavior:

Checks if current document differs from last version
If changed, creates version entry for current state with currentVersionNote
Uploads new version with nextVersionNote
Updates current document content
Increments mutation count

Status: 200 OK

{
  "_id": "60f8a5b3e6d8f32a94c12345",
  "documentName": "presentation",
  "isVersionedFile": true,
  "mutationCount": 2,
  "sizeInBytes": 2359296,
  "versionHistory": [
    {
      "version": 0,
      "s3": {
        "url": "https://bucket.s3.region.amazonaws.com/org123/PipesHub/marketing/campaigns/60f8a5b3e6d8f32a94c12345/versions/v0.pptx"
      },
      "createdAt": 1626955195000,
      "size": 2048576,
      "extension": ".pptx"
    },
    {
      "version": 1,
      "s3": {
        "url": "https://bucket.s3.region.amazonaws.com/org123/PipesHub/marketing/campaigns/60f8a5b3e6d8f32a94c12345/versions/v1.pptx"
      },
      "note": "Updated version",
      "createdAt": 1626958895000,
      "size": 2359296,
      "extension": ".pptx",
      "initiatedByUserId": "5fb3b7c8e1d8f32a94c98765",
      "mutationCount": 2
    }
  ],
  "updatedAt": 1626958895000
}

POST /internal/:documentId/uploadNextVersion - Internal Upload Next Version

Internal endpoint for uploading next version.

Request
Response

Endpoint: POST /api/v1/document/internal/:documentId/uploadNextVersionAuthentication: Requires scoped token with STORAGE_TOKEN scope.Request Body: Same structure as public upload next version endpoint.

POST /:documentId/rollBack - Roll Back to Previous Version

Restore a document to a previous version.

Request
Response

Endpoint: POST /api/v1/document/:documentId/rollBackPath Parameters:

documentId: MongoDB ObjectId (24-character hex string)

Query Parameters:

Parameter	Type	Required	Description
version	string	Yes	Version number to roll back to (transformed to number, min: 0)

Request Body Parameters:

Parameter	Type	Required	Description
note	string	Yes	Note about the rollback operation

Validation:

Uses RollBackToPreviousVersionSchema which extends GetBufferSchema
Version comes from query parameter (inherited from GetBufferSchema)
Note comes from request body
Document must support versioning
Target version must exist and be less than current version count

Note: There appears to be a discrepancy between the validator schema (which expects version in query) and the controller implementation (which reads version from body). The schema specification is documented here.

Status: 200 OK

{
  "_id": "60f8a5b3e6d8f32a94c12345",
  "documentName": "presentation",
  "mutationCount": 3,
  "versionHistory": [
    {
      "version": 0,
      "s3": {
        "url": "https://bucket.s3.region.amazonaws.com/org123/PipesHub/marketing/campaigns/60f8a5b3e6d8f32a94c12345/versions/v0.pptx"
      },
      "createdAt": 1626955195000,
      "size": 2048576,
      "extension": ".pptx"
    },
    {
      "version": 1,
      "s3": {
        "url": "https://bucket.s3.region.amazonaws.com/org123/PipesHub/marketing/campaigns/60f8a5b3e6d8f32a94c12345/versions/v1.pptx"
      },
      "note": "Updated version",
      "createdAt": 1626958895000,
      "size": 2359296,
      "extension": ".pptx",
      "initiatedByUserId": "5fb3b7c8e1d8f32a94c98765"
    },
    {
      "version": 2,
      "s3": {
        "url": "https://bucket.s3.region.amazonaws.com/org123/PipesHub/marketing/campaigns/60f8a5b3e6d8f32a94c12345/versions/v2.pptx"
      },
      "note": "Rolled back due to errors",
      "createdAt": 1626959195000,
      "size": 2048576,
      "extension": ".pptx",
      "initiatedByUserId": "5fb3b7c8e1d8f32a94c98765",
      "mutationCount": 3
    }
  ],
  "sizeInBytes": 2048576,
  "updatedAt": 1626959195000
}

POST /internal/:documentId/rollBack - Internal Roll Back

Internal endpoint for rolling back document versions.

Request
Response

Endpoint: POST /api/v1/document/internal/:documentId/rollBackAuthentication: Requires scoped token with STORAGE_TOKEN scope.Request Body: Same structure as public rollback endpoint.

Direct Upload & Utilities

Direct upload capabilities and utility functions.

POST /:documentId/directUpload - Get Direct Upload URL

Generate a presigned URL for directly uploading to storage.

Request
Response

Endpoint: POST /api/v1/document/:documentId/directUploadPath Parameters:

documentId: MongoDB ObjectId (24-character hex string)

Request Body: Empty object (per DirectUploadSchema)Use Case: For large files that need to be uploaded directly to storage vendor without going through the API server.Validation:

Document and document path must exist
Storage vendor configuration must be valid
Uses DirectUploadSchema validation

Status: 200 OK

{
  "signedUrl": "https://bucket.s3.region.amazonaws.com/org123/PipesHub/analytics/data/60f8a5b3e6d8f32a94c67890?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...",
  "documentId": "60f8a5b3e6d8f32a94c67890"
}

Behavior:

Updates document storage metadata with base URL
Presigned URL expires in 1 hour (3600 seconds)
Different implementations for S3 vs Azure Blob storage
For S3: Updates document.s3 field
For Azure: Updates document.azureBlob field

POST /internal/:documentId/directUpload - Internal Direct Upload

Internal endpoint for generating direct upload URLs.

Request
Response

Endpoint: POST /api/v1/document/internal/:documentId/directUploadAuthentication: Requires scoped token with STORAGE_TOKEN scope.

GET /:documentId/isModified - Check If Document Is Modified

Check if a document has been modified since its last version.

Request
Response

Endpoint: GET /api/v1/document/:documentId/isModifiedPath Parameters:

documentId: MongoDB ObjectId (24-character hex string)

Validation: Uses DocumentIdParams schemaBehavior:

Compares current document buffer with latest version in history (or version 0)
Returns boolean indicating modification status
Uses buffer comparison for accurate change detection

Status: 200 OK

true

Response: Boolean value indicating if document has been modified.

GET /internal/:documentId/isModified - Internal Check Modified

Internal endpoint for checking document modification status.

Request
Response

Endpoint: GET /api/v1/document/internal/:documentId/isModifiedAuthentication: Requires scoped token with STORAGE_TOKEN scope.

Configuration

Internal configuration management endpoints for service administration.

POST /updateAppConfig - Update App Configuration

Internal endpoint for updating application configuration.

Request
Response

Endpoint: POST /api/v1/document/updateAppConfigAuthentication: Requires scoped token with FETCH_CONFIG scope.Behavior:

Reloads application configuration from configuration manager
Updates storage configuration in dependency injection container
Recreates storage controller with new configuration
Allows dynamic reconfiguration without service restart

Status: 200 OK

{
  "message": "Storage configuration updated successfully",
  "config": {
    "storage": {
      "endpoint": "https://storage.example.com",
      "maxFileSize": 1073741824
    }
  }
}

Schema Definitions

interface Document {
  // Basic document information
  documentName: string;
  alternateDocumentName?: string;
  documentPath?: string;
  isVersionedFile: boolean;
  orgId: mongoose.Types.ObjectId;
  
  // Document metrics
  mutationCount?: number;
  sizeInBytes?: number;
  mimeType?: string;
  extension: string;
  
  // Access control
  permissions?: 'owner' | 'editor' | 'commentator' | 'readonly';
  initiatorUserId: mongoose.Types.ObjectId | null;
  
  // Versioning
  versionHistory?: DocumentVersion[];
  currentVersion: number;
  
  // Metadata
  customMetadata?: CustomMetadata[];
  tags?: string[];
  
  // Timestamps
  createdAt?: number;
  updatedAt?: number;
  
  // Deletion status
  deletedByUserId?: mongoose.Types.ObjectId;
  isDeleted: boolean;
  
  // Storage information
  s3?: StorageInfo;
  azureBlob?: StorageInfo;
  local?: StorageInfo;
  storageVendor: StorageVendor;
}

interface DocumentVersion {
  version?: number;
  userAssignedVersionLabel?: string;
  s3?: StorageInfo;
  azureBlob?: StorageInfo;
  local?: StorageInfo;
  note?: string;
  extension?: string;
  currentVersion?: number;
  createdAt?: number;
  mutationCount?: number;
  initiatedByUserId?: mongoose.Schema.Types.ObjectId;
  size?: number;
}

interface StorageInfo {
  url: string;
  localPath?: string;  // Only used for local storage
}

interface CustomMetadata {
  key: string;
  value: mongoose.Schema.Types.Mixed;
}

enum StorageVendor {
  S3 = 's3',
  AzureBlob = 'azureBlob',
  Local = 'local'
}

interface FilePayload {
  documentPath: string;
  buffer: Buffer;
  mimeType: string;
  isVersioned: boolean;
}

interface StorageServiceResponse<T> {
  statusCode: number;
  data?: T;
  msg?: string;
}

// Headers schema used across endpoints
export const Headers = z.object({
  authorization: z.string(),
});

// Query parameter transformations
const versionTransform = z.string()
  .optional()
  .transform((val) => (val ? Number(val) : undefined))
  .refine((num) => num === undefined || num > 0, {
    message: "version must be greater than zero",
  });

const expirationTransform = z.string()
  .optional()
  .transform((val) => (val ? Number(val) : undefined))
  .refine((num) => num === undefined || num > 0, {
    message: "expirationTimeInSeconds must be greater than zero",
  });

// Upload validation with fileBuffer constraints
export const UploadNewSchema = z.object({
  body: z.object({
    documentName: z.string(),
    documentPath: z.string().optional(),
    alternateDocumentName: z.string().optional(),
    permissions: z.string().optional(),
    customMetadata: z.any().optional(),
    isVersionedFile: z.string(),
    fileBuffer: z.any().optional(),
    fileBuffers: z.array(z.any()).optional(),
  }).refine(
    (data) => data.fileBuffer || data.fileBuffers,
    { message: 'fileBuffer or fileBuffers must be present' }
  ),
  query: z.object({}),
  params: z.object({}),
  headers: Headers,
});

Supported MIME Types

The storage service supports an extensive list of MIME types including:

Document Formats
Image Formats
Archive Formats
Media & Other Formats

PDF: application/pdf
Microsoft Office: Word (.docx, .doc), Excel (.xlsx, .xls), PowerPoint (.pptx, .ppt)
OpenDocument: .odt, .ods, .odp
Text: .txt, .rtf, .csv, .md, .mdx
Web: .html, .css, .js, .json, .xml
eBooks: .epub
Google Workspace: .gdoc, .gsheet, .gslides, .gdraw

Common: .jpg, .jpeg, .png, .gif, .svg, .webp
Raw/Professional: .tiff, .bmp, .ico, .psd, .icns
Modern: .heic, .heif, .avif, .jxl, .jp2, .jpx

Common: .zip, .rar, .7z, .tar, .gz
System: .cab, .rpm, .deb, .cpio
Compression: .bz2, .xz, .zst, .lz

Audio: .mp3, .wav, .flac, .aac, .ogg, .ape, .amr
Video: .mp4, .avi, .mkv, .mov, .webm, .3gp
Fonts: .ttf, .otf, .woff, .woff2, .eot
CAD/3D: .dwg, .dxf, .stl, .glb, .gltf
Scientific: .fits, .dcm, .nii

Storage Provider Implementation

The Storage Service uses an adapter pattern to abstract the underlying storage provider implementation.

Amazon S3
Azure Blob Storage
Local Storage

Features:

Direct file uploads/downloads via AWS SDK
Signed URLs with configurable expiration (default: 1 hour)
Multipart uploads for large files
Object versioning support
Comprehensive error handling with specific S3 error types

Configuration:

interface S3StorageConfig {
  accessKeyId: string;
  secretAccessKey: string;
  region: string;
  bucketName: string;
}

URL Structure:

https://{bucket}.s3.{region}.amazonaws.com/{orgId}/PipesHub/{path}/{documentId}/current/{name}.{ext}

Error Handling:

StorageConfigurationError - Missing credentials
StorageUploadError - Upload failures
StorageDownloadError - Download failures
PresignedUrlError - URL generation failures
MultipartUploadError - Multipart operation failures

Features:

Blob storage operations via Azure SDK
SAS token generation for secure access
Container-based organization with automatic creation
Blob versioning support
Stream handling for large files

Configuration:

interface AzureBlobStorageConfig {
  azureBlobConnectionString?: string;  // Alternative 1
  accountName?: string;                // Alternative 2
  accountKey?: string;                 // Alternative 2
  containerName: string;               // Required
  endpointProtocol?: string;           // Default: https
  endpointSuffix?: string;             // Default: core.windows.net
}

Limitations:

Multipart uploads not implemented (throws MultipartUploadError)
Use direct upload instead for large files

SAS Token Features:

Read permissions for downloads
Write permissions for uploads
Configurable expiration (default: 1 hour)
Content disposition support for downloads

Features:

Filesystem-based storage with OS-specific mount paths
File streaming for efficient delivery
Directory-based organization
Cross-platform path handling (Windows, macOS, Linux)
Direct file serving without intermediate storage

Configuration:

interface LocalStorageConfig {
  mountName: string;  // e.g., "PipesHub"
  baseUrl: string;    // API endpoint base URL
}

Mount Paths:

macOS: ~/Library/{mountName}
Windows: ~/AppData/{mountName}
Linux: ~/.local/{mountName}

File URLs:

file:///path/to/mount/{orgId}/PipesHub/{path}/{documentId}/current/{name}.{ext}

URL Normalization for Downloads:

Converts file:// URLs to API download endpoints
Pattern: {baseUrl}/api/v1/document/{documentId}/download
Enables web access to local files through API proxy

File Size Limits & Direct Upload

Size Limits
Direct Upload Flow

Public Endpoints:

Upload: 1GB (1024 * 1024 * 1000 bytes)
Buffer operations: 100MB (1024 * 1024 * 100 bytes)

Internal Endpoints:

All operations: 100MB (1024 * 1024 * 100 bytes)

Direct Upload Threshold:

Set via maxFileSizeForPipesHubService constant
Currently configured to 0MB (all files trigger direct upload evaluation)
Applied only to S3 and Azure Blob storage
Local storage always uses direct API upload

Client uploads file that exceeds threshold
API creates document placeholder
API returns 308 Permanent Redirect with presigned URL
Client uploads directly to storage vendor using presigned URL
Storage vendor handles the upload
Document metadata is updated with final storage location

Headers in redirect response:

Location: Presigned upload URL (expires in 1 hour)
x-document-id: Document identifier for tracking
x-document-name: Document name for reference

Advantages:

Reduces API server load for large files
Improves upload performance
Leverages storage vendor’s optimized upload handling

Error Handling

All endpoints return structured error responses:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid request parameters",
    "details": {
      "field": "documentName",
      "issue": "Document name cannot contain extensions"
    }
  },
  "timestamp": "2025-04-27T13:20:00.000Z"
}

Common Error Codes:

VALIDATION_ERROR - Invalid request parameters
NOT_FOUND - Document not found
BAD_REQUEST - Invalid request format
FORBIDDEN - File format mismatch or access denied
INTERNAL_SERVER_ERROR - Storage service errors

Storage-Specific Errors:

Malformed ids: a documentId in the path that is not a 24-character hex id answers 400 with VALIDATION_ERROR, and a storage token whose orgId or userId is not one answers 400 with HTTP_BAD_REQUEST. The message names the id to fix. A well-formed id that matches no document is still 404.
Extension validation failures
File size limit exceeded
Storage vendor configuration errors
Version control constraint violations
MIME type validation failures

HTTP Status Codes:

200 - Success
308 - Permanent Redirect (for direct uploads)
400 - Bad Request
401 - Unauthorized
403 - Forbidden
404 - Not Found
500 - Internal Server Error

Configuration Management

Storage Configuration
Service Integration

Storage configuration is maintained in etcd under /services/storage path:

{
  "storageType": "s3|azureBlob|local",
  "credentials": {
    // Provider-specific configuration
  }
}

Dynamic Configuration:

Service loads configuration at startup and via watch mechanism
Can switch storage vendors without service restart
Configuration changes trigger adapter reinitialization
Graceful handling of configuration errors

Key-Value Store (etcd):

Storage configuration management under /services/storage
Endpoint configuration under /services/endpoints
Dynamic configuration updates via watch mechanism
Service discovery for configuration manager endpoints

Configuration Manager:

Centralized configuration storage and validation
Configuration update API for dynamic reconfiguration
Service-specific configuration routing
Configuration versioning and rollback capabilities

IAM Service:

User authentication and authorization
Organization and user context extraction for multi-tenancy
Scoped token validation for internal APIs
User existence validation for shared operations

Important Notes

Schema Discrepancy: There’s a mismatch between the rollback validator schema (expects version in query) and controller implementation (reads version from body). Documentation follows schema specification.
File Extensions: Automatically detected from original filename and validated against comprehensive MIME type mappings.
Version Control: Optional per document via isVersionedFile flag, maintains complete history with metadata and notes.
Storage Abstraction: Seamless switching between storage vendors via configuration without code changes.
Organization Isolation: All documents are organization-scoped (orgId) for secure multi-tenancy.
Deletion: The delete endpoints mark a document as deleted (isDeleted: true). The exception is the internal delete with ?hard=true, which removes the document and makes one attempt to remove its stored files. Deleting a record, folder or knowledge base in PipesHub removes its stored files and their documents through the internal purge endpoints, asynchronously. The delete request finishes once the records are gone from the graph, and the indexing service removes the stored content afterwards:
- An uploaded file’s removal is scheduled before its record is deleted. If it can’t be scheduled, nothing is deleted and the request returns 503.
- While a record still lists the file, the indexing service re-schedules the removal, for up to a day after the delete. A file a record still lists after that belongs to a record that was never deleted, so it is kept.
- If the record is gone but storage can’t remove the file, or the record can’t be checked, the removal is re-scheduled with a delay of up to 5 minutes until it succeeds, with no time limit.
- A record’s processed content is removed by its delete event or, if that event is lost, by the indexing service’s periodic orphan sweep.
Direct Upload: Large files bypass API server for optimal performance, using vendor-specific presigned URLs.
Internal APIs: Service-to-service communication with scoped authentication using specific token scopes.
Mutation Tracking: Documents track change count (mutationCount) for auditing and optimization.
MIME Type Validation: Extensive validation against supported file types with automatic extension detection.
Path Structure: Organized as {orgId}/PipesHub/{documentPath}/{documentId}/current|versions/ for clear file organization.
Cross-Platform Support: Local storage handles Windows, macOS, and Linux filesystem differences transparently.

Welcome To PipesHub

System Overview

Authentication

Mail Configuration

Web Search

AI Providers

Connectors

Integrations

Agents

Toolsets

MCP Server

User Management

Deployment

Developer

Additional Resources

Storage Service API

Storage Service API

Base URL

Authentication

Architecture Overview

Storage Configuration

Authentication Modes

Data Models

Document

Version History

API Endpoints

Schema Definitions

Supported MIME Types

Storage Provider Implementation

File Size Limits & Direct Upload

Error Handling

Configuration Management

Important Notes

​Storage Service API

​Base URL

​Authentication

​Architecture Overview

​Storage Configuration

​Authentication Modes

​Data Models

​Document

​Version History

​API Endpoints

​Schema Definitions

​Supported MIME Types

​Storage Provider Implementation

​File Size Limits & Direct Upload

​Error Handling

​Configuration Management

​Important Notes

Storage Service API

Base URL

Authentication

Architecture Overview

Storage Configuration

Authentication Modes

Data Models

Document

Version History

API Endpoints

Schema Definitions

Supported MIME Types

Storage Provider Implementation

File Size Limits & Direct Upload

Error Handling

Configuration Management

Important Notes