Pipelines reference - Preparers =============================== This reference documentation for Pipelines preparers includes information on the functions and views available in the aidb extension related to :ref:`Pipelines reference - Preparers ` . See :ref:`Pipelines knowledge bases usage ` and :ref:`Where to find SDK references and examples? ` for more details. Views ----- aidb.preparers ^^^^^^^^^^^^^^ Also referenceable as ``aidb.preps`` , the ``aidb.preparers`` view contains information about the configured data preparers. It includes the following columns: .. csv-table:: :header: Column,Type,Description :widths: 12,15,25 :align: left :class: longtable id,INTEGER,"" name,TEXT,Name of the preparer. operation,:ref:`aidb.DataPreparationOperation` ,The kind of processing step to perform. destination_schema,TEXT,Schema of the destination table to store the output data. destination_table,TEXT,Name of the destination table to store the output data. destination_key_column,TEXT,Column of the destination table that references the key in source data. destination_data_column,TEXT,Column of the destination table to store the output data. options,JSONB,"Configuration options for the data preparation operation. Uses the same API as the :ref:`**vector-native capabilities** ` ." source_type,TEXT,Type of source data the preparer is working with. Can be either 'Table' or 'Volume'. source_schema,TEXT,Schema of the table with the source data the preparer will process. Applies only to preparers of `Table` source type. source_table,TEXT,Name of the table with the source data the preparer will process. Applies only to preparers of `Table` source type. source_data_column,TEXT,Column in the source table with the source data the preparer will process. Applies only to preparers of `Table` source type. source_key_column,TEXT,Name of the key column in the source table for reference with the output processed data. Applies only to preparers of `Table` source type. source_volume_name,TEXT,Name of the volume to use as a data source. Applies only to preparers of `Volume` source type. Types ----- aidb.DataPreparationOperation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ``aidb.DataPreparationOperation`` type is an enum that represents the different types of preprocessing steps that can be performed: - ChunkText - SummarizeText - ParseHtml - ParsePdf - PerformOcr Functions --------- aidb.create_table_preparer ^^^^^^^^^^^^^^^^^^^^^^^^^^ Creates a preparer with a source data table. Parameters ^^^^^^^^^^ .. csv-table:: :header: Parameter,Type,Default,Description :widths: 15,12,12,20 :align: left :class: longtable name,TEXT,Required,Name of the preparer operation,:ref:`aidb.DataPreparationOperation` ,Required,Type of data preparation operation source_table,TEXT,Required,Name of the source data table source_data_column,TEXT,Required,Column in the source table containing the raw data destination_table,TEXT,Required,Name of the destination table destination_data_column,TEXT,Required,Column in the destination table for processed data source_key_column,TEXT,'id',Unique column in the source table to use as key to reference the rows. destination_key_column,TEXT,'id',Key column in the destination table that references the `source_key_column` options,JSONB,'{}'::JSONB,"Configuration options for the data preparation operation. Uses the same API as the :ref:`**vector-native capabilities** ` ." ``aidb.create_preparer_for_table`` (deprecated) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ **Replaced by** :ref:`aidb.create_table_preparer ` with same arguments. aidb.create_volume_preparer ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Creates a preparer for a given PGFS volume. .. _parameters-1: Parameters ^^^^^^^^^^ .. csv-table:: :header: Parameter,Type,Default,Description :widths: 15,12,12,20 :align: left :class: longtable name,TEXT,Required,Name of the preparer operation,:ref:`aidb.DataPreparationOperation` ,Required,Type of data preparation operation source_volume_name,TEXT,Required,Name of the source volume containing the raw data destination_table,TEXT,Required,Name of the destination table destination_data_column,TEXT,Required,Column in the destination table for processed data destination_key_column,TEXT,'id',Key column in the destination table that uniquely identifies the processed data options,JSONB,'{}'::JSONB,"Configuration options for the data preparation operation. Uses the same API as the :ref:`**vector-native capabilities** ` ." invoker_role,TEXT,NULL,"Role owning the tables, pipelines, and background job execution." ``aidb.create_preparer_for_volume`` (deprecated) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ **Replaced by** :ref:`aidb.create_volume_preparer ` with same arguments. aidb.bulk_data_preparation ^^^^^^^^^^^^^^^^^^^^^^^^^^ Executes the configured data preparation operation on all data from the specified preparer’s source. .. _parameters-2: Parameters ^^^^^^^^^^ .. csv-table:: :header: Parameter,Type,Description :widths: 12,15,25 :align: left :class: longtable preparer_name,TEXT,Name of the preparer aidb.set_auto_preparer ^^^^^^^^^^^^^^^^^^^^^^ Sets the automatic processing mode for this preparer. This function is used to enable and disable automatic data preparation: ``Live`` mode enables the Postgres trigger-based automation and ``Disabled`` disables all automation. .. _parameters-3: Parameters ^^^^^^^^^^ .. csv-table:: :header: Parameter,Type,Default,Description :widths: 15,12,12,20 :align: left :class: longtable preparer_name,TEXT,"",Name of preparer mode,:ref:`aidb.PipelineAutoProcessingMode` ,"",Desired processing mode Example ^^^^^^^ .. code:: sql SELECT aidb.set_auto_preparer(test_preparer, Live); SELECT aidb.set_auto_preparer(test_preparer, Disabled); ``aidb.set_preparer_auto_processing`` (deprecated) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ **Replaced by** :ref:`aidb.set_auto_preparer ` with same arguments. aidb.delete_preparer ^^^^^^^^^^^^^^^^^^^^ Deletes the preparer’s configuration. .. _parameters-4: Parameters ^^^^^^^^^^ .. csv-table:: :header: Parameter,Type,Description :widths: 12,15,25 :align: left :class: longtable preparer_name,TEXT,Name of preparer to delete .. Note:: This function doesn't delete the destination table or any data in it.