AI pipelines

AI pipelines#

Pipelines can ingest, prepare and vectorize data for semantic retrieval and RAG applications. They integrate pgvector iterative index scans search capabilities and can keep the knowledge base synchronized with your source data automatically.

Example: a knowledge base for RAG#

This walkthrough builds a knowledge base over internal support articles and shows each retrieval stage independently. For the complete version, see End-to-end example: Knowledge base for RAG , which adds model registration, a reranking demo where the cross-encoder corrects a wrong vector-search result, and a single statement that retrieves, reranks, and generates an answer with verified output.

Set up the pipeline#

The source is a support_articles table holding 16 short articles about database operations — take the CREATE TABLE and INSERT statements from the full example’s source data step . The pipeline embeds each article body with bge-small-en-v1.5-f16 , one of the built-in llama.cpp embedding models — pre-registered, runs locally, and its model file downloads automatically on first use:

SELECT aidb.create_pipeline(
    name               => support_kb,
    source             => support_articles,
    source_key_column  => id,
    source_data_column => body,
    step_1             => KnowledgeBase,
    step_1_options     => aidb.knowledge_base_config(
        model       => bge-small-en-v1.5-f16,
        data_format => Text
    ),
    auto_processing    => Disabled
);

Run the pipeline#

With auto_processing => 'Disabled' , nothing runs until you say so. Process all existing rows once:

SELECT aidb.run_pipeline(support_kb);
Contents