LobeHub
LobeHub

Full-Text Search

LobeHub uses full-text search for product data such as agents, topics, messages, files, knowledge bases, chat groups, and memories. This is separate from the web-search tools that an agent can call.

LobeHub supports three product-search backends:

BackendBest fitAdditional operations
pg_searchA deployment that already runs PostgreSQL with the ParadeDB pg_search extension and wants the simplest topologyPostgreSQL maintains the BM25 indexes; no separate search-sync worker is required
elasticsearchA larger deployment that needs BM25 search quality and can operate a dedicated search service and persistent sync jobRequires an Elasticsearch service (external, or the optional single-node service in the official Docker Compose file), an initial backfill, PostgreSQL change capture, and a continuously scheduled incremental consumer
pg_likeA personal or small deployment on plain PostgreSQL that does not want to run ElasticsearchNone; substring matching runs directly against the existing tables without extensions, indexes, or sync workers

FTS_SEARCH_PROVIDER selects one backend for the whole deployment. The accepted values are pg_search, elasticsearch, and pg_like; the default is pg_search. LobeHub does not silently fall back to another backend when the selected provider is unavailable.

Database migrations install pg_search and its BM25 indexes only when the database provides the extension, and skip that step otherwise. On PostgreSQL without pg_search, set FTS_SEARCH_PROVIDER=pg_like or elasticsearch before starting LobeHub. While the provider is pg_search, including when it is unset, the database migration step stops with an error naming the fix if the extension or any LobeHub BM25 index is missing, so the Docker container does not start and a Vercel build fails.

Choose a backend

Keep the default provider when your PostgreSQL service supports pg_search, its BM25 indexes fit comfortably with the rest of the database workload, and you prefer not to operate a separate search service.

For a self-managed database, the LobeHub Docker examples use the paradedb/paradedb:latest-pg17 image and preload pg_search. Apply the normal LobeHub database migrations to install the extension and the LobeHub-managed BM25 indexes, then keep:

FTS_SEARCH_PROVIDER=pg_search

If you install pg_search on a database that ran the LobeHub migrations without it, create the extension and the BM25 indexes from a LobeHub source checkout before selecting this provider:

*.sh
Shell
psql "$DATABASE_URL" -c 'CREATE EXTENSION IF NOT EXISTS pg_search'
psql "$DATABASE_URL" -f packages/database/migrations/0093_add_bm25_indexes_with_icu.sql

Use pg_like on plain PostgreSQL

Choose pg_like when your PostgreSQL service cannot install pg_search (many managed providers) and the deployment is small enough that a dedicated search service is not worth operating. It needs no extension, no additional index, and no sync worker: every search runs as case-insensitive substring matching (ILIKE) over the same tables and permission scopes as pg_search.

FTS_SEARCH_PROVIDER=pg_like

What to expect compared with the other backends:

  • Matching. A row matches when every word of the query appears in one searchable field. There is no stemming, no typo tolerance, and no phrase proximity. Chinese, Japanese, and Korean text works without segmentation because matching is substring based.
  • Ranking. Results are ordered by a simple relevance score: a whole-field match ranks above a prefix match, which ranks above a phrase match, which ranks above rows that merely contain every word; title-like fields weigh more than body text.
  • Performance. Queries scan the scoped rows instead of using a search index. This is fine for personal and small-team instances; for large message or memory tables, prefer pg_search or Elasticsearch.

New databases and upgrades both work on PostgreSQL without pg_search: the migrations skip the extension and BM25 indexes there. If you previously replaced the 0090_enable_pg_search.sql or 0093_add_bm25_indexes_with_icu.sql migration files to get past a failed upgrade, remove that override.

Use Elasticsearch

Choose Elasticsearch when search needs independent capacity, when you want to separate search storage from transactional PostgreSQL, and when you can keep a sync job running continuously. Personal and small deployments whose database provider is ending pg_search should consider the service-free pg_like backend first.

Neon stopped offering pg_search to new projects on March 19, 2026. Retirement timelines for existing projects follow the notice Neon sends for that project. LobeHub users who receive such a notice should use Migrate from pg_search on Neon to choose pg_like or Elasticsearch. Check Neon's current pg_search notice before acting.

There are two supported ways to run Elasticsearch. Both use the same backfill and incremental sync commands and the same ES_INDEX_NAMESPACE; they differ only in how LobeHub reaches the cluster.

External Elasticsearch (Elastic Cloud or a managed cluster)

  • an Elasticsearch project reachable from the LobeHub server over HTTPS;
  • the official ICU analysis plugin, which is bundled in Elastic Cloud Serverless and must be installed on every node of a self-managed cluster;
  • ES_URL, ES_API_KEY, and a stable ES_INDEX_NAMESPACE;
  • a full backfill followed by continuous Outbox synchronization;
  • FTS_SEARCH_PROVIDER=elasticsearch only after the backfill and incremental queue are complete.

Plain HTTP is accepted only for loopback addresses during local development. LobeHub refuses to send an API key to any other host over plaintext HTTP.

Docker Compose single-node Elasticsearch

The official docker-compose/deploy/docker-compose.yml ships an optional, disabled-by-default elasticsearch service plus the backfill and sync commands built from the official LobeHub image. It is intended for a single-host deployment that wants Elasticsearch without an external account. Elasticsearch runs with security disabled and is reachable only inside the Compose network, so the plaintext, no-API-key connection must be enabled explicitly with ES_ALLOW_INSECURE_HTTP=true. See Run Elasticsearch with Docker Compose below.

Database support for both options

Elasticsearch works with PostgreSQL with or without pg_search: LobeHub migrations skip the BM25 setup on a database that does not provide the extension. When the database already has pg_search objects, Elasticsearch takes over search traffic after the backfill, and the objects can be removed later with the supported cleanup command.

See Migrate from pg_search on Neon for guidance on choosing a backend and starting the migration.

How Elasticsearch synchronization works

PostgreSQL remains the source of truth. Enabling the Elasticsearch path installs change-capture triggers that coalesce source changes into a durable Outbox. The recurring consumer writes those changes to Elasticsearch. Search requests retrieve candidate identifiers from Elasticsearch, then LobeHub hydrates and authorizes the results against PostgreSQL before returning them.

The operating sequence is therefore:

*.txt
Plaintext
PostgreSQL write
  -> capture trigger
  -> FTS Outbox
  -> recurring fts-search:sync consumer
  -> Elasticsearch index
  -> candidate search
  -> PostgreSQL permission check and hydration

The full reindex is only the initial snapshot. It does not replace the recurring consumer. Keep fts-search:sync scheduled for as long as Elasticsearch serves search traffic.

Run Elasticsearch with Docker Compose

The optional services in docker-compose/deploy/docker-compose.yml are gated by Compose profiles, so a default docker compose up neither downloads nor starts them:

ServiceProfileRole
elasticsearchelasticsearchSingle node built locally from the pinned official image plus analysis-icu, with a named data volume, a health check, and no published port
fts-search-reindexelasticsearch-reindexOne-off backfill / status command from the lobehub/lobehub image; its checkpoint lives in the fts-search-reindex-state volume
fts-search-syncelasticsearch-syncLong-running incremental consumer from the lobehub/lobehub image; restarts and logs when it hits failed or dead-letter work

Resource notes: the node uses a 1 GB JVM heap by default (ES_JAVA_OPTS); keep the heap at or below half of the memory available to the container and plan for at least 2 GB of RAM for Elasticsearch alone. Linux hosts must set vm.max_map_count=262144. The first docker compose up with the profile enabled builds the image once from the pinned official image and installs analysis-icu, so that build needs outbound access to docker.elastic.co and artifacts.elastic.co; later container recreations work offline. The build context is the elasticsearch/Dockerfile next to the Compose file, which setup.sh downloads as well. To upgrade Elasticsearch, change the version in both image and build.args of the elasticsearch service and run docker compose up -d --build.

  1. Enable the node. In .env, uncomment the Elasticsearch block and keep pg_search serving requests:

    *.sh
    Shell
    COMPOSE_PROFILES=elasticsearch
    ES_URL=http://elasticsearch:9200
    ES_ALLOW_INSECURE_HTTP=true
    ES_INDEX_NAMESPACE=lobehub
    # FTS_SEARCH_PROVIDER stays pg_search (the default) until the last step

    Then start the stack and wait for every service, including the new node, to become healthy; the database migrations of the same release create the Outbox schema:

    docker compose up -d --wait
  2. Backfill. Inspect the state, then run the one-off backfill. --apply installs the PostgreSQL change capture, creates the 14 indexes with the ICU mapping, copies the data, and creates the aliases:

    *.sh
    Shell
    docker compose run --rm fts-search-reindex --status
    docker compose run --rm fts-search-reindex --apply --fresh-run --yes

    Only the first run uses --fresh-run. If it is interrupted, run the same command without --fresh-run to resume from the checkpoint volume. Repeat --status until the run is ready_for_incremental_sync, every entity is completed, and every failedCount is 0.

  3. Start continuous sync. Add the sync profile and recreate the stack:

    *.sh
    Shell
    COMPOSE_PROFILES=elasticsearch,elasticsearch-sync
    *.sh
    Shell
    docker compose up -d
    docker compose logs -f fts-search-sync

    The worker runs fts-search-elasticsearch-sync.cjs --max-steps=8 --interval-seconds=15 --yes in a loop (FTS_SEARCH_SYNC_INTERVAL_SECONDS changes the pause between empty drains). It exits non-zero on failed or dead-letter work; Compose restarts it, so a container that keeps restarting means the queue needs attention. Keep it running for as long as Elasticsearch serves search.

  4. Switch explicitly. When docker compose run --rm fts-search-reindex --status shows pending, ready, retrying, inFlight, dead, and revisionLag all at 0, set FTS_SEARCH_PROVIDER=elasticsearch in .env and recreate the application container:

    docker compose up -d lobe

    Nothing switches the provider automatically. Rolling back means restoring FTS_SEARCH_PROVIDER=pg_search and recreating lobe; the sync worker may keep running.

External target: the fts-search-reindex and fts-search-sync services depend only on PostgreSQL, not on the bundled node. To run them against Elastic Cloud instead, leave the elasticsearch profile out of COMPOSE_PROFILES, set ES_URL and ES_API_KEY in .env, and do not set ES_ALLOW_INSECURE_HTTP; steps 2 to 4 are otherwise identical.

Rebuild the current Elasticsearch schema generation

When a projection or capture behavior change requires every historical document to be regenerated, first upgrade every application and continuous-sync worker. Then build a new physical generation from PostgreSQL without changing the mapping version:

*.sh
Shell
docker compose run --rm fts-search-reindex \
  --apply --rebuild-current --entity=messages --yes

The command prints a run ID and an index such as <namespace>-messages-v1-r<run-id>. If interrupted, run the same command with --run-id=<run-id>. Continuous sync writes both the existing and rebuilding generations; do not pause application writes or the sync worker. Creating the checkpoint briefly fences only the PostgreSQL tables that can emit the selected entities; resuming the same checkpoint does not repeat that fence. After the backfill reports ready_for_incremental_sync and the messages Outbox is idle, promote the exact physical index:

*.sh
Shell
docker compose run --rm fts-search-reindex \
  --promote --entity=messages --generation=<exact-index> --yes

The alias and logical schema version remain unchanged. Keep the previous generation during an observation window so the alias can be promoted back to its exact name. Once rollback is no longer needed, run --purge --entity=messages --yes. Purge installs an exact-name auto-create guard before deleting each detached old index. Self-managed clusters may run --retire --entity=messages --yes first to close old indexes; Elastic Cloud Serverless does not support index close, so skip retire and purge directly.

Security boundary of this mode: ES_ALLOW_INSECURE_HTTP=true permits plaintext HTTP to a non-loopback host and lets ES_API_KEY be omitted. It never permits sending an API key over plaintext HTTP, so do not combine it with ES_API_KEY and an http:// URL. The Elasticsearch service publishes no port; never add one, because the node accepts unauthenticated requests. The Elastic Cloud path is unchanged: without this variable LobeHub still requires HTTPS and an API key.

Configuration reference

VariablePurpose
FTS_SEARCH_PROVIDERDeployment-wide provider: pg_search, elasticsearch, or pg_like
ES_URLElasticsearch endpoint. HTTPS, or plaintext HTTP only for loopback or, with ES_ALLOW_INSECURE_HTTP=true, a private Compose hostname such as elasticsearch
ES_API_KEYElasticsearch API key with index creation, bulk write, refresh, count, alias, and search permissions. Required unless ES_ALLOW_INSECURE_HTTP=true
ES_ALLOW_INSECURE_HTTPtrue opts in to plaintext HTTP without an API key for an Elasticsearch node reachable only inside a private container network. Never sends a key over HTTP
ES_INDEX_NAMESPACEStable deployment-owned prefix for physical indexes and aliases
FTS_SEARCH_SYNC_ENABLEDEnables the incremental consumer; set to true only after the initial backfill is ready. The Compose sync service sets it itself

Compose-only variables read by docker-compose/deploy/docker-compose.yml:

VariablePurpose
COMPOSE_PROFILESelasticsearch starts the node; elasticsearch,elasticsearch-sync also starts the sync worker
ES_JAVA_OPTSJVM heap of the Elasticsearch container, default -Xms1g -Xmx1g
FTS_SEARCH_SYNC_INTERVAL_SECONDSSeconds the sync worker sleeps after a drain that found no more work, default 15

Do not reuse one namespace for unrelated LobeHub deployments. Do not change the namespace while a migration or incremental consumer is active.

LobeHub exports bounded OpenTelemetry metrics and traces without recording raw queries, user IDs, document IDs, or indexed text. The main metric families are:

MetricWhat it answers
fts_search_backend_operations_totalRequest volume and failures by provider, entity, operation, and result
fts_search_backend_operation_durationEnd-to-end backend latency
fts_search_backend_result_countRequested, candidate, and PostgreSQL-hydrated result counts
fts_search_elasticsearch_requests_totalActual Elasticsearch request volume and result
fts_search_elasticsearch_request_durationElasticsearch request duration including response parsing
fts_search_elasticsearch_request_sizeSerialized request-body size
fts_search_elasticsearch_response_decoded_sizeDecoded response-body size
fts_search_elasticsearch_response_hitsHits returned per Elasticsearch request
fts_search_elasticsearch_server_tookProcessing time reported by Elasticsearch

Traces use spans named fts.search.backend.<operation>. Use request counts, bytes, hits, server time, and indexed storage together when investigating Elasticsearch cost; any single signal alone is insufficient. See Grafana Observability for the self-hosted OpenTelemetry stack.

Operating rules

  • Back up PostgreSQL and rehearse on an isolated database copy and Elasticsearch target before a production migration.
  • Keep the same durable checkpoint directory for every resume of one backfill. Never run two workers against the same checkpoint and physical indexes.
  • Keep pg_search serving requests until every entity is complete, failures are zero, and the incremental queue has caught up.
  • Use the supported cleanup command after cutover; users do not need to copy database-object SQL.
  • Keep the Elasticsearch Outbox, capture triggers, and recurring consumer. They are required by the active Elasticsearch backend and are unrelated to Neon's removal of pg_search.