Pipelines overview
==================

Pipelines is an EDB Postgres AI component that automates the
transformation of raw data into vector-searchable insights. It
integrates :ref:`pgvector iterative index scans <pgvector iterative index scans>`  search capabilities with built-in data
preparation tools, keeping knowledge bases synchronized with source data
without manual intervention.

Integrated data lifecycle
-------------------------

Pipelines automates the journey from raw data to searchable insights. By
managing data ingestion, embedding generation, and indexing, it ensures
your vector stores stay synchronized with your source data.

- **Flexible model integration**: Choose from supported local models,
  such as HuggingFace, or connect to OpenAI-compatible external APIs.

- **Preprocessing automation**: Built-in data preparation steps handle
  data cleaning and transformation directly within AI pipelines.

- **Eliminate stale data**: Auto-processing keeps embeddings current,
  significantly reducing the hallucinations often caused by outdated
  vector data.

- **Unified semantic search**: The intelligent knowledge base feature
  allows you to query text and images using a single function call,
  regardless of whether the data lives in Postgres tables or
  S3-compatible object storage.

Pipeline architecture
---------------------

A pipeline is configured using the ``aidb.create_pipeline()`` function
and consists of three main aspects:

- \*\* :ref:`Creating pipelines <Creating pipelines>`  \*\* — pipelines read from Postgres tables or from
  external cloud storage (S3, GCS, Azure) via :ref:`Postgres File System (PGFS) </pg_extensions/pgfs/>`  .

- \*\* :ref:`Pipeline steps <Pipeline steps>`  \*\* — up to 10 sequential operations transform
  your data. Steps include chunking, parsing, OCR, summarization, and
  embedding into a knowledge base. Each step is configured using a
  dedicated helper function.

- \*\* :ref:`Orchestration <Orchestration>`  \*\* — auto-processing modes (``Live`` ,
  ``Background`` , or ``Disabled`` ) keep your knowledge base in sync
  with source data automatically, without manual intervention.

Supported models
----------------

The AIDB extension supports a diverse range of Open Encoder LLMs from
HuggingFace (running locally on your Postgres node) and OpenAI Encoders
(via cloud API). You can view all available options in the
``aidb.model_providers`` table.