​

External Services

Our architecture leverages several industry-standard external services to provide a robust, scalable, and efficient system. Below is an overview of each service and its role in our ecosystem.

​

Apache Kafka

Apache Kafka is an open-source distributed event streaming platform used for building real-time data pipelines and streaming applications. In our architecture, Kafka enables event-driven communication between services, making the system loosely coupled and highly scalable. Learn more about Apache Kafka

​

MongoDB

MongoDB is a document-oriented NoSQL database used for high volume data storage. We utilize MongoDB as our primary persistence layer for storing and retrieving application data in JSON-like documents with dynamic schemas. Learn more about MongoDB

​

etcd

etcd is a distributed, reliable key-value store for the most critical data of a distributed system. We use etcd for managing configuration across our services, ensuring consistency and availability of configuration parameters. Learn more about etcd

​

Graph database (Neo4j or ArangoDB)

PipesHub stores the knowledge graph — the relationships between records, users, and metadata — in a graph database. Which one an instance uses is set by the DATA_STORE environment variable. Services do not open a Neo4j or ArangoDB driver directly, so the same features work on either backend. Neo4j is the default. It is what install.sh selects for both the full and slim deployment types, what backend/env.template ships with, and what the Helm chart enables out of the box. Set DATA_STORE=neo4j. ArangoDB is the alternative, a multi-model database supporting graph, document, and key-value models. Existing installations that were set up on ArangoDB continue to run on it, and the installer keeps that choice when it finds an ArangoDB data volume. Set DATA_STORE=arangodb. Learn more about Neo4j · Learn more about ArangoDB

​

Qdrant

Qdrant is a vector similarity search engine designed for production-ready AI applications. We use Qdrant as our vector database for efficient storage, management, and search of high-dimensional vectors for machine learning models and embeddings. Learn more about Qdrant

​

Redis

Redis is an open-source, in-memory data structure store used as a database, cache, message broker, and streaming engine. In our architecture, Redis serves as a high-performance key-value store for caching and real-time data operations. Learn more about Redis