Skip to main content
With web search on, chat and agents can search the web and read the pages they find. Answers cite each page by its title, and the chat shows every search and every page read as a step you can open.
Chat reply steps: Searched the web with a table of linked results, and Read a web page naming Quickstart on docs.pipeshub.com

A reply's steps: the search lists every result with a link, and the page read names the page.

It works out of the box: with no provider configured, PipesHub uses free search engines and needs no API key. Add a provider when you want more reliable or higher-volume search, or to keep searches on your own servers.

​
Providers

PipesHub has its own page reader, so you only need Firecrawl or Crawl4AI for sites the built-in reader can’t get through, such as sites that block automated clients.

​
Add a provider

1

Open the settings

As an admin, go to Workspace → Web Search.
Web Search Configuration page listing providers with their status and Set as default buttons

Workspace → Web Search, with SearXNG as the default and Serper and Crawl4AI configured.

2

Configure the provider

Click the provider’s configure button and fill in its fields:
  • API Key for Serper, Tavily, Exa and Brave Search.
  • Server URL for SearXNG and Crawl4AI, for example http://searxng:8080.
  • Firecrawl: an API Key for Firecrawl Cloud. For a self-hosted server, also set its Server URL.
  • Crawl4AI: an API Token if the server was started with one.
SearXNG configuration panel with a Server URL field

SearXNG needs only its server address.

3

Save

Saving runs one real search (or, for Crawl4AI, one real page read). If the check fails, the provider is not saved and the error says why.
4

Make it the default

Click Set as default to have chat use the provider. Crawl4AI only reads pages, so it can’t be the default.
An agent can use a different provider from the default: choose it in the agent builder’s Web Search section. Only admins see API keys and server addresses. Other members see them masked.

​
When a provider fails

A search tries several sources in this order:
  1. The chosen provider: the agent’s own, or else the default.
  2. The other configured search providers, in the order you added them.
  3. The free engines.
So a wrong key or a provider outage doesn’t stop the search. A source that rejected its key or blocked the server is skipped for a few minutes. Pages are read by the built-in reader first. If it fails, for example on a bot check or a timeout, the configured Firecrawl and Crawl4AI servers are tried. A failure about the address itself (not a public website, too many redirects, not a valid address, or not allowed) ends there: that address is never sent to Firecrawl or Crawl4AI. A search that fails is reported as a failure, with the reason. It is never reported as finding nothing.

​
Self-hosted SearXNG and Crawl4AI

The Docker Compose setup includes both servers as optional profiles.
1

Turn on the profiles

In .env, add search-searxng and/or fetch-crawl4ai to COMPOSE_PROFILES:
2

Set their secrets

3

Start the stack and add the providers

Run docker compose up -d. Then add SearXNG at http://searxng:8080 and Crawl4AI at http://crawl4ai:11235 under Workspace → Web Search. For Crawl4AI, enter the same token as CRAWL4AI_API_TOKEN.
Crawl4AI configuration panel with Server URL and API Token fields

Crawl4AI takes its server address and the token from .env.

Both servers run on their own network. They can reach the internet, but not PipesHub or its databases. PipesHub reaches them through a small gateway container, which only forwards to the two servers.
If you run your own SearXNG server, its settings.yml must list json under search.formats, or saving it fails. Crawl4AI servers older than 0.9.4 are refused, because they let anyone who can reach them run code or reach internal addresses.

​
Pages built by JavaScript

Some sites send an almost empty page and build their content with scripts. When PipesHub gets such a page, it opens it in its headless browser to read the finished content. If the browser fails, it keeps the page as it was served. To turn this off, set WEB_FETCH_RENDER_JS=off. It is also off whenever WEB_CONNECTOR_ALLOWED_HOSTS is set. PipesHub reads HTML, PDF (pages with a text layer), plain text, Markdown, CSV and JSON.

​
Deep research

In agent mode, the chat’s + menu has a Deep research switch. It needs Web Search turned on. With it on, the agent:
  • splits the question into parts and searches them in parallel;
  • reads the best pages in full and checks key facts against two sources;
  • writes a report with citations, with the direct answer first.
The + menu with Internal Search, Web Search and Deep research switches turned on

Deep research in the + menu. It turns off when Web Search is off.

Deep research takes longer and uses more searches. The switch stays on for your next chats until you turn it off.

​
Limits

When an answer reaches its limit, the agent answers with what it has found. Each search has 12 seconds and each page read 20 seconds. The built-in reader stops at 10 MB; Firecrawl and Crawl4AI return the page’s text without that limit.

​
Safety

  • PipesHub only reads public websites. Addresses that point inside your network, such as localhost, private IP ranges or cloud metadata addresses, are refused. The built-in reader also checks every redirect.
  • Firecrawl and Crawl4AI follow redirects on their own servers, so PipesHub checks only the address it sends them and the one they land on. Run them where they can’t reach your internal services. The Docker Compose profiles above do this; a server you run elsewhere and add by its Server URL needs the same isolation.
  • Text from the web is marked as untrusted, so the model treats it as information, not instructions. A page can’t use it to change what the agent does.
  • A citation link the model invents is removed from the answer.

​
Troubleshooting

When a step fails, open it in the chat to see why. Common reasons: