AI Factory Pipelines ==================== **AI Factory Pipelines** is a core capability of **EDB Postgres® AI** (EDB PG AI), enabling intelligent, automated AI data workflows directly in your Postgres clusters — including **Retrieval-Augmented Generation (RAG)**, **Knowledge Bases**, and vector-powered applications. Pipelines abstracts away complexity in preparing, managing, and serving AI data for Postgres — integrating with **Vector Engine** and enabling **Sovereign AI** patterns where your data stays inside your database or trusted object storage. Why Pipelines? -------------- Modern AI workloads — especially RAG and Gen AI — require: - Clean, preprocessed data - Consistent embeddings - Efficient vector indexes - Real-time updates when source data changes Manually managing these steps is complex and error-prone. Pipelines automates this lifecycle: - Prepares and cleans data - Runs embedding generation - Maintains vector indexes - Supports auto-processing and Knowledge Bases Pipelines helps eliminate “hallucinations” from stale embeddings and makes building reliable Gen AI easier. What You Can Do --------------- With Pipelines, you can: - Automate **vector embedding pipelines** for text and image data - Maintain **Knowledge Bases** with real-time or batch updates - Perform **semantic and similarity search** using :ref:`Vector Engine (PGvector) ` - Integrate with **Gen AI Builder** and other AI Factory components - Build fully auditable, controlled **Sovereign AI** systems on your data How It Works ------------ Preparers ^^^^^^^^^ Preparers define how source data is cleaned, chunked, and prepared for embedding. You can create preparers for: - Postgres tables (``aidb.create_table_preparer()`` ) - Object storage (S3-compatible) (``aidb.create_volume_preparer()`` ) Once created, you can run: - **Bulk processing** (``aidb.bulk_data_preparation()`` ) - **Auto-processing** (``aidb.set_auto_preparer()`` ) to keep embeddings up to date automatically. See :ref:`Pipelines reference - Preparers ` to learn more. Knowledge Bases ^^^^^^^^^^^^^^^ Knowledge Bases provide a simple abstraction for RAG: - Define a Knowledge Base (``aidb.create_table_knowledge_base()`` , ``aidb.create_volume_knowledge_base()`` ) - Automatically manage embeddings and vector indexes - Use ``aidb.retrieve_key()`` and ``aidb.retrieve_text()`` for semantic search Supports: - Text and image data - Multi-modal models (e.g., CLIP) - Integration with Gen AI Builder’s retrieval flows See :ref:`Knowledge Base Concepts ` . Auto-Processing ^^^^^^^^^^^^^^^ Pipelines supports flexible auto-processing modes: - Sync or async - Table or volume sources - Configurable batch sizes and triggers See :ref:`Pipeline auto-processing ` for details. Who Should Use Pipelines? ------------------------- Pipelines is ideal for: - Gen AI Builders creating **enterprise Knowledge Bases** - Data Engineers building **semantic search pipelines** - AI teams needing **automated RAG pipelines** - Architects implementing **Sovereign AI** in regulated environments When to Use Pipelines --------------------- Use Pipelines when you need to: - Keep embeddings current with changing data - Reduce RAG errors caused by stale or incomplete embeddings - Power multi-modal Gen AI search across text and images - Maintain full auditability and governance of embedding processes Configuration values explained ------------------------------ - Chunk size and overlap: Smaller chunks improve grounding; overlap helps preserve context across chunks. Tune based on document structure. - Embedding model selection: Use a consistent model for ingestion and query. Choose for your domain and language. - Auto‑processing cadence: Align with data change frequency and SLA. Consider sync vs. async modes. - Batch sizes and parallelism: Increase for throughput but monitor storage, network, and memory. - Similarity threshold and top‑K: Higher thresholds reduce noise; tune top‑K for recall vs. latency. Learn More ---------- - :ref:`Knowledge Base Concepts ` - :ref:`Pipelines reference - Preparers ` - :ref:`Pipeline auto-processing ` - :ref:`Supported Embedding Models ` - :ref:`Vector Engine (PGvector) ` **AI Accelerator Pipelines** helps you build trusted, scalable **Sovereign AI** solutions on top of your Postgres data — keeping AI pipelines clean, current, and production-ready. .. toctree:: :maxdepth: 3 Usage--pipeline--Start_here--index Usage--pipeline--Get_started--index Usage--pipeline--Core_concepts--index Usage--pipeline--Storage_and_access--index Usage--pipeline--Reference--index Usage--pipeline--Rel_notes--index Usage--pipeline--Legal--index