Overview
This guide will walk you through deploying PipesHub on Google Cloud Platform (GCP) using a Virtual Machine instance. You’ll set up a VM with the required specifications, install Docker, and deploy the PipesHub application.Database Architecture
PipesHub uses a modern, distributed database architecture:- MongoDB: Primary document store for user data, configurations, and metadata
- ArangoDB: Multi-model database for graph relationships and complex queries
- Qdrant: High-performance vector database for semantic search and AI embeddings
- Redis: In-memory cache for session management and real-time data
- etcd: Distributed key-value store for service discovery and configuration management
Minimum Requirements
Before you begin, ensure your VM meets these specifications:- CPU: 4 cores (minimum)
- RAM: 16 GB (minimum)
- Storage: 100 GB SSD or higher (recommended)
- PipesHub uses multiple databases (MongoDB, ArangoDB, Qdrant, Redis, etcd)
- Storage requirements grow with indexed documents and vector embeddings
- OS: Ubuntu 22.04 LTS or 24.04 LTS (recommended)
For production workloads with large document collections, consider 200 GB or more storage to accommodate database growth and vector embeddings.
Recommended GCP VM Instance Types
Choose an instance type based on your workload requirements:Standard Workloads
- n2-standard-4: 4 vCPUs, 16 GB memory
- Balanced performance for most use cases
- Latest generation compute-optimized
- n2d-standard-4: 4 vCPUs, 16 GB memory (AMD EPYC)
- Cost-effective alternative with AMD processors
Cost-Optimized
- e2-standard-4: 4 vCPUs, 16 GB memory
- Most cost-effective option
- Suitable for steady-state workloads
Deployment Steps
1
Create a GCP VM Instance
- Go to the GCP Console
- Navigate to Compute Engine > VM Instances
- Click Create Instance
-
Configure your instance:
- Name: Choose a descriptive name (e.g.,
pipeshub-prod) - Region/Zone: Select a region close to your users
- Machine configuration: Select one of the recommended instance types
- Boot disk:
- Operating system: Ubuntu
- Version: Ubuntu 22.04 LTS or 24.04 LTS
- Boot disk type: Balanced persistent disk or SSD persistent disk
- Size: 100 GB (200 GB recommended for production)
- Firewall:
- ✅ Allow HTTP traffic
- ✅ Allow HTTPS traffic
- Name: Choose a descriptive name (e.g.,
- Click Create to launch your instance
2
Configure Firewall Rules
After creating your VM, configure firewall rules to allow traffic:
- Go to VPC Network > Firewall
- Click Create Firewall Rule
-
Configure the rule:
- Name:
allow-pipeshub - Target tags: Add a network tag (e.g.,
pipeshub-server) - Source IP ranges:
0.0.0.0/0(or restrict to your organization’s IP range) - Protocols and ports:
- ✅ tcp:80
- ✅ tcp:443
- ✅ tcp:3000
- Name:
- Go back to your VM instance and add the network tag under Edit > Network tags
3
Connect to Your VM
Connect to your VM using SSH:
4
Update System Packages
Once connected, update your system:
5
Install Docker
Install Docker using the official Docker installation script:For detailed instructions, see the official Docker installation guide.
To run Docker commands without
sudo, add your user to the docker group:6
Install Additional Dependencies
Install required network utilities:
7
Clone PipesHub Repository
Clone the PipesHub repository:
8
Configure Environment Variables
Copy the environment template and configure your settings:Edit the Important settings to update:
.env file to set your configuration:SECRET_KEY: Generate a secure random keyFRONTEND_PUBLIC_URL: Set to your domain or VM’s external IP- Any other service-specific passwords or keys
9
Start PipesHub
Start the PipesHub application using Docker Compose:This command will:View logs:
- Download all required Docker images
- Create and start all containers
- Run the application in detached mode
10
Stop PipesHub (When Needed)
To stop the services:To stop and remove all data (⚠️ use with caution):
Configure HTTPS Access
You have several options to set up HTTPS:Option 1: Nginx Reverse Proxy
Configure Nginx as a reverse proxy to terminate HTTPS traffic and forward to the PipesHub frontend:Option 2: Cloudflare Tunnel
Use Cloudflare Tunnel for zero-configuration HTTPS:Option 3: GCP Load Balancer
Use GCP’s built-in Load Balancer with managed SSL certificates:- Go to Network Services > Load Balancing
- Create an HTTPS Load Balancer
- Configure backend to point to your VM instance on port 3000
- Set up a managed SSL certificate for your domain
Access PipesHub
Once deployed, access PipesHub at:- HTTPS (production):
https://your-domain.com
The first startup may take a few minutes as Docker pulls images and initializes the databases (MongoDB, ArangoDB, Qdrant, Redis, and etcd).
Post-Deployment Configuration
After accessing PipesHub for the first time:- Complete the onboarding setup
- Choose your account type (Individual or Enterprise)
- Configure your AI models and connectors
- Set up user management and permissions
Troubleshooting
White Screen After Deployment
Cause: You’re accessing PipesHub over HTTP instead of HTTPS. Solution: Set up HTTPS using one of the methods described above.Cannot Access on Port 3000
Cause: Firewall rules not configured or service not running. Solution:Docker Permission Denied
Cause: User doesn’t have Docker permissions. Solution:Out of Memory or CPU Issues
Cause: Instance type doesn’t meet minimum requirements. Solution: Upgrade to a larger instance type with at least 4 cores and 16 GB RAM.Monitoring and Maintenance
View Logs
Check Service Health
Update PipesHub
Backup Data
PipesHub uses multiple databases and storage systems. Choose one of the following backup strategies:Option 1: Native Database Tools (Recommended for Production)
Option 1: Native Database Tools (Recommended for Production)
Use database-specific backup tools that create consistent snapshots without downtime:Advantages:
- No downtime required
- Consistent database snapshots
- Safe for production environments
- Uses native tools designed for each database
Option 2: Stop Application First (Simple and Safe)
Option 2: Stop Application First (Simple and Safe)
If you prefer a simpler approach and can tolerate downtime:Advantages:
- Simple and straightforward
- Single backup file for all data
- Guaranteed data consistency
- Easy to automate
Automated Backup Script
Automated Backup Script
Create a backup script for regular automated backups:Features:
- Automated daily backups at 2 AM
- 30-day retention policy
- Automatic cleanup of old backups
- Optional GCS upload for off-site storage
- Logging for monitoring
For production environments, upload backups to Google Cloud Storage (GCS) for long-term retention and disaster recovery. Use
gsutil to automate uploads.Restore Data
Choose the restore method that matches your backup strategy:Restore from Native Backup Tools
Restore from Native Backup Tools
Use this method if you created backups using Option 1 (native database tools):
Restore from Volume Backup
Restore from Volume Backup
Use this method if you used Option 2 (stopped application backup):
Restore from Automated Backup Script
Restore from Automated Backup Script
If you used the automated backup script:
Verify Restore
Verify Restore
After restoring, verify that all services are running correctly:Verification Checklist:
- ✅ All containers are running
- ✅ No error messages in logs
- ✅ All databases respond to health checks
- ✅ Application UI is accessible
- ✅ User data is visible
- ✅ Connectors are functioning













