Skip to main content
This page describes the databases and brokers along the bottom of the system overview diagram. A default install starts these services for you. Use this page to learn what each service is for, and when you would switch to the other option. The sections follow the same order as the diagram: search data, application data, then how services pass work to each other and where configuration is stored.

​
Knowledge graph (Neo4j or ArangoDB)

The knowledge graph is how PipesHub stores that a Slack thread, a Drive file, and a person belong together. Permission checks and citations follow those links. You choose one graph database with the DATA_STORE environment variable. Application services talk to a graph abstraction, so the product behaves the same on either backend.
  • Neo4j is the default (DATA_STORE=neo4j). The default install, backend/env.template, and the Helm chart all use Neo4j.
  • ArangoDB is the other option (DATA_STORE=arangodb). If an existing install already has an ArangoDB data volume, the installer keeps ArangoDB.

​
Vector search (Qdrant)

After Indexing parses a document, an embedding model turns each chunk of text into a vector. Qdrant stores those vectors and finds similar text when Query answers a question. A default install uses Qdrant. The application also supports OpenSearch and Redis (VECTOR_DB_TYPE=opensearch or VECTOR_DB_TYPE=redis). Those values apply only when the app container receives VECTOR_DB_TYPE. The default Docker Compose file does not pass that variable, so Qdrant stays in use unless you add it to the service environment.

​
Application data (MongoDB)

MongoDB stores application data that is not part of the knowledge graph: sessions, file metadata, and similar state.

​
Files

The contents of files are not stored in MongoDB. They live on local disk, S3, or Azure Blob, through the storage module in the API.

​
How services hand work to each other

Connectors publish records, and Indexing consumes them. That path is the event bus in the architecture diagram.
  • On a local machine, the event bus is Redis Streams (MESSAGE_BROKER=redis). Kafka is not started.
  • In a larger deployment, the event bus is Apache Kafka (MESSAGE_BROKER=kafka).
The Python services stay the same. Only the transport between them changes.

​
Cache, configuration, and Redis

Redis always runs. It can do three jobs, depending on how you deploy:
  1. Cache — frequently used data and locks.
  2. Encrypted configuration — the default key-value store for settings (KV_STORE_TYPE=redis).
  3. Event bus — Redis Streams, when MESSAGE_BROKER=redis.
In a larger deployment, Kafka takes over the event bus. Redis continues to provide the cache, and it remains the configuration store unless you switch that store to etcd.

​
Optional: etcd

etcd is the other place to keep encrypted configuration. A default install uses Redis for this. To use etcd, set KV_STORE_TYPE=etcd and enable the Compose profile kv-etcd.

​
etcd settings

These environment variables tell PipesHub how to reach etcd. They belong on the pipeshub-ai service, which runs the Python services and the Node.js API; the etcd service doesn’t read them. The Compose files already set ETCD_URL, ETCD_HOST and ETCD_TIMEOUT there, pointing at the bundled etcd service, and leave ETCD_PORT at its default. When etcd authentication is in use, the Python services renew their login by themselves when etcd’s login token expires (after 5 minutes by default), so they keep working without a restart. The password is never written to the logs.
The Compose files don’t set ETCD_USERNAME or ETCD_PASSWORD, so add them to the pipeshub-ai service’s environment if you use them. They only give PipesHub a login; turning authentication on in etcd itself is a separate step, done with etcdctl. Wrong values, or values set against an etcd that doesn’t have authentication turned on, stop the Python services from connecting at start-up; before this release, they were silently ignored.The Node.js API does not log in to etcd. Don’t turn on etcd authentication while the Node.js API uses etcd as its configuration store (KV_STORE_TYPE=etcd), or it won’t be able to reach etcd.
If the configuration store is down when a service starts and saves its vector collection setting, the service stops with “Could not persist the initial collection strategy to the config store”. The message ends with the next step: check that the config store (Redis or etcd, whichever KV_STORE_TYPE selects) is running and reachable, then restart the service.