AI pipelines
============

Pipelines can ingest, prepare and vectorize data for semantic retrieval
and RAG applications. They integrate :ref:`pgvector iterative index scans <pgvector iterative index scans>`  search capabilities
and can keep the knowledge base synchronized with your source data
automatically.

Example: a knowledge base for RAG
---------------------------------

This walkthrough builds a knowledge base over internal support articles
and shows each retrieval stage independently. For the complete version,
see :ref:`End-to-end example: Knowledge base for RAG <End-to-end example: Knowledge base for RAG>`  , which adds model registration, a reranking demo
where the cross-encoder corrects a wrong vector-search result, and a
single statement that retrieves, reranks, and generates an answer with
verified output.

Set up the pipeline
^^^^^^^^^^^^^^^^^^^

The source is a ``support_articles`` table holding 16 short articles
about database operations — take the ``CREATE TABLE`` and ``INSERT``
statements from the full example’s :ref:`source data step <Examples>`  . The pipeline embeds
each article body with ``bge-small-en-v1.5-f16`` , one of the built-in
llama.cpp embedding models — pre-registered, runs locally, and its model
file downloads automatically on first use:

.. code:: sql

   SELECT aidb.create_pipeline(
       name               => support_kb,
       source             => support_articles,
       source_key_column  => id,
       source_data_column => body,
       step_1             => KnowledgeBase,
       step_1_options     => aidb.knowledge_base_config(
           model       => bge-small-en-v1.5-f16,
           data_format => Text
       ),
       auto_processing    => Disabled
   );

Run the pipeline
^^^^^^^^^^^^^^^^

With ``auto_processing => 'Disabled'`` , nothing runs until you say so.
Process all existing rows once:

.. code:: sql

   SELECT aidb.run_pipeline(support_kb);

Retrieve: semantic search
^^^^^^^^^^^^^^^^^^^^^^^^^

Every article body is now embedded and indexed. Paste raw error text
into a semantic search — the top match is the resolution, found by
meaning rather than keywords:

\`\`\ ``sql SELECT key, value, distance::numeric(5,3) AS distance FROM aidb.retrieve_text('support_kb', 'application logs show: FATAL: sorry, too many clients already', 3); __OUTPUT__ key |                                                                                                          value                                                                                                          | distance -----+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------``
# AI pipelines\`

.. csv-table::
  :header: ------------------------------------,---------------------------------------------------------------------------------------------------------
  :widths: 10,30
  :align: left
  :class: longtable

  :ref:`Pipeline steps<Pipeline steps>` ,"Available step types: ChunkText, ParseHtml, ParsePdf, PerformOcr, SummarizeText, KnowledgeBase."
  :ref:`Orchestration<Orchestration>` ,"Auto-processing modes, background workers, observability, and error handling."
  ":ref:`AI pipelines <Pipelines API reference>` ","Full API reference for pipeline types, views, CRUD functions, and config helpers."
  :ref:`Examples<Examples>` ,End-to-end worked examples.

.. toctree::
  :maxdepth: 3

  data-pipelines--data-processing
  data-pipelines--pipeline-steps
  data-pipelines--orchestration--index
  data-pipelines--reference
  data-pipelines--example