PipesHub External Service
Documentation for the PipesHub External Service
External Services
Our architecture leverages several industry-standard external services to provide a robust, scalable, and efficient system. Below is an overview of each service and its role in our ecosystem.
Apache Kafka
Apache Kafka is an open-source distributed event streaming platform used for building real-time data pipelines and streaming applications. In our architecture, Kafka enables event-driven communication between services, making the system loosely coupled and highly scalable.
MongoDB
MongoDB is a document-oriented NoSQL database used for high volume data storage. We utilize MongoDB as our primary persistence layer for storing and retrieving application data in JSON-like documents with dynamic schemas.
etcd
etcd is a distributed, reliable key-value store for the most critical data of a distributed system. We use etcd for managing configuration across our services, ensuring consistency and availability of configuration parameters.
ArangoDB
ArangoDB is a multi-model database supporting graph, document, and key-value data models. We leverage ArangoDB specifically for its graph database capabilities, enabling efficient storage and querying of connected data.
Qdrant
Qdrant is a vector similarity search engine designed for production-ready AI applications. We use Qdrant as our vector database for efficient storage, management, and search of high-dimensional vectors for machine learning models and embeddings.
Redis
Redis is an open-source, in-memory data structure store used as a database, cache, message broker, and streaming engine. In our architecture, Redis serves as a high-performance key-value store for caching and real-time data operations.