Skip to main content

Overview

This guide will walk you through deploying PipesHub on Google Cloud Platform (GCP) using a Virtual Machine instance. You’ll set up a VM with the required specifications, install Docker, and deploy the PipesHub application.

Database Architecture

PipesHub uses a modern, distributed database architecture:
  • MongoDB: Primary document store for user data, configurations, and metadata
  • ArangoDB: Multi-model database for graph relationships and complex queries
  • Qdrant: High-performance vector database for semantic search and AI embeddings
  • Redis: In-memory cache for session management and real-time data
  • etcd: Distributed key-value store for service discovery and configuration management

Minimum Requirements

Before you begin, ensure your VM meets these specifications:
  • CPU: 4 cores (minimum)
  • RAM: 16 GB (minimum)
  • Storage: 100 GB SSD or higher (recommended)
    • PipesHub uses multiple databases (MongoDB, ArangoDB, Qdrant, Redis, etcd)
    • Storage requirements grow with indexed documents and vector embeddings
  • OS: Ubuntu 22.04 LTS or 24.04 LTS (recommended)
For production workloads with large document collections, consider 200 GB or more storage to accommodate database growth and vector embeddings.
Choose an instance type based on your workload requirements:

Standard Workloads

  • n2-standard-4: 4 vCPUs, 16 GB memory
    • Balanced performance for most use cases
    • Latest generation compute-optimized
  • n2d-standard-4: 4 vCPUs, 16 GB memory (AMD EPYC)
    • Cost-effective alternative with AMD processors

Cost-Optimized

  • e2-standard-4: 4 vCPUs, 16 GB memory
    • Most cost-effective option
    • Suitable for steady-state workloads

Deployment Steps

1

Create a GCP VM Instance

  1. Go to the GCP Console
  2. Navigate to Compute Engine > VM Instances
  3. Click Create Instance
  4. Configure your instance:
    • Name: Choose a descriptive name (e.g., pipeshub-prod)
    • Region/Zone: Select a region close to your users
    • Machine configuration: Select one of the recommended instance types
    • Boot disk:
      • Operating system: Ubuntu
      • Version: Ubuntu 22.04 LTS or 24.04 LTS
      • Boot disk type: Balanced persistent disk or SSD persistent disk
      • Size: 100 GB (200 GB recommended for production)
    • Firewall:
      • ✅ Allow HTTP traffic
      • ✅ Allow HTTPS traffic
  5. Click Create to launch your instance
2

Configure Firewall Rules

After creating your VM, configure firewall rules to allow traffic:
  1. Go to VPC Network > Firewall
  2. Click Create Firewall Rule
  3. Configure the rule:
    • Name: allow-pipeshub
    • Target tags: Add a network tag (e.g., pipeshub-server)
    • Source IP ranges: 0.0.0.0/0 (or restrict to your organization’s IP range)
    • Protocols and ports:
      • ✅ tcp:80
      • ✅ tcp:443
      • ✅ tcp:3000
  4. Go back to your VM instance and add the network tag under Edit > Network tags
3

Connect to Your VM

Connect to your VM using SSH:
4

Update System Packages

Once connected, update your system:
5

Install Docker

Install Docker using the official Docker installation script:
For detailed instructions, see the official Docker installation guide.
To run Docker commands without sudo, add your user to the docker group:
6

Install Additional Dependencies

Install required network utilities:
7

Clone PipesHub Repository

Clone the PipesHub repository:
8

Configure Environment Variables

Copy the environment template and configure your settings:
Edit the .env file to set your configuration:
Important settings to update:
  • SECRET_KEY: Generate a secure random key
  • FRONTEND_PUBLIC_URL: Set to your domain or VM’s external IP
  • Any other service-specific passwords or keys
Never commit the .env file to version control. Keep your secrets secure!
9

Start PipesHub

Start the PipesHub application using Docker Compose:
This command will:
  • Download all required Docker images
  • Create and start all containers
  • Run the application in detached mode
Check the status of your containers:
View logs:
10

Stop PipesHub (When Needed)

To stop the services:
To stop and remove all data (⚠️ use with caution):

Configure HTTPS Access

HTTPS is required for production deployments. PipesHub enforces stricter security checks, and browsers will block certain requests when the application is served over HTTP. If you see a white screen after deployment, this is likely the cause.
You have several options to set up HTTPS:

Option 1: Nginx Reverse Proxy

Configure Nginx as a reverse proxy to terminate HTTPS traffic and forward to the PipesHub frontend:
Get a free SSL certificate using Let’s Encrypt:

Option 2: Cloudflare Tunnel

Use Cloudflare Tunnel for zero-configuration HTTPS:

Option 3: GCP Load Balancer

Use GCP’s built-in Load Balancer with managed SSL certificates:
  1. Go to Network Services > Load Balancing
  2. Create an HTTPS Load Balancer
  3. Configure backend to point to your VM instance on port 3000
  4. Set up a managed SSL certificate for your domain
For detailed HTTPS setup instructions, refer to the Quickstart Guide.

Access PipesHub

Once deployed, access PipesHub at:
  • HTTPS (production): https://your-domain.com
The first startup may take a few minutes as Docker pulls images and initializes the databases (MongoDB, ArangoDB, Qdrant, Redis, and etcd).

Post-Deployment Configuration

After accessing PipesHub for the first time:
  1. Complete the onboarding setup
  2. Choose your account type (Individual or Enterprise)
  3. Configure your AI models and connectors
  4. Set up user management and permissions
For detailed onboarding instructions, see the Onboarding Guide.

Troubleshooting

White Screen After Deployment

Cause: You’re accessing PipesHub over HTTP instead of HTTPS. Solution: Set up HTTPS using one of the methods described above.

Cannot Access on Port 3000

Cause: Firewall rules not configured or service not running. Solution:

Docker Permission Denied

Cause: User doesn’t have Docker permissions. Solution:

Out of Memory or CPU Issues

Cause: Instance type doesn’t meet minimum requirements. Solution: Upgrade to a larger instance type with at least 4 cores and 16 GB RAM.

Monitoring and Maintenance

View Logs

Check Service Health

Update PipesHub

Backup Data

Never use simple file copy methods (like tar) on live database volumes. This can result in corrupted backups and data loss. Always use native database backup tools or stop the application before backing up.
PipesHub uses multiple databases and storage systems. Choose one of the following backup strategies:
If you prefer a simpler approach and can tolerate downtime:
Advantages:
  • Simple and straightforward
  • Single backup file for all data
  • Guaranteed data consistency
  • Easy to automate
Create a backup script for regular automated backups:
Features:
  • Automated daily backups at 2 AM
  • 30-day retention policy
  • Automatic cleanup of old backups
  • Optional GCS upload for off-site storage
  • Logging for monitoring
For production environments, upload backups to Google Cloud Storage (GCS) for long-term retention and disaster recovery. Use gsutil to automate uploads.

Restore Data

Always test your backup and restore procedures in a non-production environment before relying on them for disaster recovery.
Choose the restore method that matches your backup strategy:
Use this method if you created backups using Option 1 (native database tools):
Use this method if you used Option 2 (stopped application backup):
If you used the automated backup script:
After restoring, verify that all services are running correctly:
Verification Checklist:
  • ✅ All containers are running
  • ✅ No error messages in logs
  • ✅ All databases respond to health checks
  • ✅ Application UI is accessible
  • ✅ User data is visible
  • ✅ Connectors are functioning

Next Steps

Support

Need help?
  • 📚 Check our FAQ
  • 💬 Join our community discussions
  • 🐛 Report issues on GitHub