Knowledge bases reference
=========================

Reference for knowledge base functions and views. For guide-style
documentation, see :ref:`Knowledge bases reference <Knowledge bases reference>`  .

Views
-----

aidb.knowledge_bases
^^^^^^^^^^^^^^^^^^^^

Also accessible as ``aidb.kbs`` . Lists all knowledge bases in the
database with their full configuration.

.. csv-table::
  :header: Column,Type,Description
  :widths: 12,15,25
  :align: left
  :class: longtable

  `id`,integer,Internal identifier
  `name`,text,Knowledge base name
  `vector_schema`,text,Schema of the embeddings table
  `vector_table`,text,Embeddings storage table
  `model_name`,text,Embedding model used
  `distance_operator`,:ref:`aidb.DistanceOperator<aidb.DistanceOperator>` ,Vector distance function used for retrieval
  `distance_operator_sql`,text,"The distance operator in SQL-operator syntax (e.g. `<->`, `<=>`)"
  `vector_data_column`,text,Embeddings column
  `vector_key_column`,text,Key column in the embeddings table
  `vector_index`,jsonb,Vector index configuration
  `pipeline_ids`,integer[],IDs of all pipelines attached to this knowledge base
  `pipeline_names`,text[],Names of all pipelines attached to this knowledge base

--------------

aidb.knowledge_base_stats
^^^^^^^^^^^^^^^^^^^^^^^^^

Also accessible as ``aidb.kbstat`` . Provides current processing
statistics for all knowledge base pipelines.

:raw-latex:`\d a`idb.knowledge_base_stats View
“aidb.knowledge_base_stats” Column \| Type \| Collation \| Nullable \|
Default ————+———+———–+———-+——— name \| text \| \| \| pipelines \|
integer \| \| \| <- number of pipelines attached embeddings \| bigint \|
\| \| <– this is the old “count(embeddings)” status \| text \| \| \| <–
the same status field we have in “aidb.pipeline_metrics” but we collect
the “worst” status of all the connected pipelines

Example
^^^^^^^

.. code:: sql

   SELECT * from aidb.kbm;

   __OUTPUT__
                    name                 | pipelines | embeddings |  status
   - -------------------------------------+-----------+------------+----------
    public.pipeline_pipeline__7471a      |         1 |          5 | UpToDate
    public.pipeline_pipeline__7471b      |         1 |          5 | UpToDate
    public.pipeline_animal_facts_kb      |         1 |        372 | UpToDate
    public.pipeline_animal_facts_kb_bert |         1 |        372 | UpToDate
    public.mpkb_shared_vectors           |         2 |          4 | UpToDate
   (5 rows)

--------------

Types
-----

aidb.DistanceOperator
^^^^^^^^^^^^^^^^^^^^^

Vector distance function used during retrieval.

.. csv-table::
  :header: Value,Description
  :widths: 10,30
  :align: left
  :class: longtable

  `L2`,Euclidean distance
  `InnerProduct`,Inner product
  `Cosine`,Cosine similarity
  `L1`,L1 distance
  `Hamming`,Hamming distance
  `Jaccard`,Jaccard distance

--------------

aidb.PipelineDataFormat
^^^^^^^^^^^^^^^^^^^^^^^

Format of data in a pipeline source or volume.

.. csv-table::
  :header: Value,Description
  :widths: 10,30
  :align: left
  :class: longtable

  `Text`,Plain text
  `Image`,Binary image data
  `Pdf`,PDF documents

--------------

aidb.PipelineAutoProcessingMode
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Auto-processing mode for a knowledge base pipeline.

.. csv-table::
  :header: Value,Description
  :widths: 10,30
  :align: left
  :class: longtable

  `Live`,Immediate processing via Postgres triggers on each data change
  `Background`,Periodic processing via a Postgres background worker
  `Disabled`,No automatic processing; run manually with `aidb.bulk_embedding()`

--------------

Knowledge base functions
------------------------

aidb.set_auto_knowledge_base
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Updates the auto-processing mode for an existing knowledge base.

Parameters
^^^^^^^^^^

.. csv-table::
  :header: Parameter,Type,Default,Description
  :widths: 15,12,12,20
  :align: left
  :class: longtable

  `knowledge_base_name`,TEXT,Required,Name of the knowledge base.
  `mode`,:ref:`aidb.PipelineAutoProcessingMode<aidb.PipelineAutoProcessingMode>` ,Required,New auto-processing mode.
  `batch_size`,INTEGER,NULL,Records processed per batch (Background and Disabled modes).
  `background_sync_interval`,INTERVAL,NULL,Polling interval (Background mode).

Examples
^^^^^^^^

.. code:: sql

   SELECT aidb.set_auto_knowledge_base(my_kb, Live);
   SELECT aidb.set_auto_knowledge_base(my_kb, Background, background_sync_interval => 1 minute);
   SELECT aidb.set_auto_knowledge_base(my_kb, Disabled, batch_size => 200);

aidb.retrieve_key
^^^^^^^^^^^^^^^^^

Returns the source record keys and distances for the top matching
embeddings, without fetching source data. In ``part_ids`` , the call
also returns, for each step in a multi-step pipeline, the IDs of the
parts that match the query. For example if a single source record is
split into 10 parts by a “ChunkText” step, then ``part_ids`` will
indicate which chunk matched.

.. _parameters-1:

Parameters
^^^^^^^^^^

.. csv-table::
  :header: Parameter,Type,Default,Description
  :widths: 15,12,12,20
  :align: left
  :class: longtable

  `knowledge_base_name`,TEXT,Required,Name of the knowledge base.
  `query`,TEXT,Required,Query text to search with.
  `topk`,INTEGER,`1`,Number of results to return.
  `deduplicate`,BOOLEAN,`true`,"Return each source result only once even if multiple ""parts"" (e.g. chunks/pages) match"

Returns
^^^^^^^

.. csv-table::
  :header: Column,Type,Description
  :widths: 12,15,25
  :align: left
  :class: longtable

  `key`,text,Source record key
  `distance`,double precision,Vector distance from the query
  `part_ids`,bigint[],Which parts i.e. pipeline step results match (chunk or page)
  `pipeline_name`,text,The name of the pipeline processing this source record

.. _example-1:

Example
^^^^^^^

.. code:: sql

   SELECT * FROM aidb.retrieve_key(public.pipeline_animal_facts_kb, birds, topk=>3);

   __OUTPUT__
    key |      distance      | part_ids |  pipeline_name
   - ----+--------------------+----------+-----------------
    5   | 1.1931772046185758 | {0,0}    | animal_facts_kb
    93  | 1.1980633963685476 | {2,0}    | animal_facts_kb
    98  | 1.2150919866080878 | {2,0}    | animal_facts_kb
   (3 rows)

--------------

aidb.retrieve_text
^^^^^^^^^^^^^^^^^^

Returns source text and distances for the top matching embeddings by
joining the embeddings table with the source table. In ``part_ids`` ,
the call also returns, for each step in a multi-step pipeline, the IDs
of the parts that match the query. For example if a single source record
is split into 10 parts by a “ChunkText” step, then ``part_ids`` will
indicate which chunk matched. The return column ``intermediate_steps``
will contain the actual result text of the intermediate step results.

.. _parameters-2:

Parameters
^^^^^^^^^^

.. csv-table::
  :header: Parameter,Type,Default,Description
  :widths: 15,12,12,20
  :align: left
  :class: longtable

  `knowledge_base_name`,TEXT,Required,Name of the knowledge base.
  `query_string`,TEXT,Required,Query text to search with.
  `number_of_results`,INTEGER,`0`,Number of results to return. Uses `topk` default if `0`.
  `knowledge_base_name`,TEXT,Required,Name of the knowledge base.
  `query`,TEXT,Required,Query text to search with.
  `topk`,INTEGER,`1`,Number of results to return.
  `deduplicate`,BOOLEAN,`true`,"Return each source result only once even if multiple ""parts"" (e.g. chunks/pages) match"

.. _returns-1:

Returns
^^^^^^^

.. csv-table::
  :header: Column,Type,Description
  :widths: 12,15,25
  :align: left
  :class: longtable

  `key`,text,Source record key
  `value`,text,Source text content
  `distance`,double precision,Vector distance from the query
  `part_ids`,bigint[],Which parts i.e. pipeline step results match (chunk or page)
  `pipeline_name`,text,The name of the pipeline processing this source record
  `intermediate_steps`,jsonb,"The matching results from the intermediate steps; i.e. the values belonging to ""part_ids"""

.. _example-2:

Example
^^^^^^^

.. code:: sql

   SELECT * FROM aidb.retrieve_text(my_kb, waterproof jacket, 3);

   __OUTPUT__
     key  |                       value                        |      distance
   - ------+----------------------------------------------------+--------------------
    19337 | Men Stripes Waterproof Shell Jacket                | 0.2994317672742334
    55018 | Women All-Weather Hiking Anorak                    | 0.3804609668507203
   (2 rows)

--------------

aidb.delete_knowledge_base
^^^^^^^^^^^^^^^^^^^^^^^^^^

Removes a knowledge base; i.e. its destination vector table and its
configuration. Also removes all attached Pipelines. Does not delete the
underlying source table.

.. _parameters-3:

Parameters
^^^^^^^^^^

.. csv-table::
  :header: Parameter,Type,Description
  :widths: 12,15,25
  :align: left
  :class: longtable

  `knowledge_base_name`,TEXT,Name of the knowledge base to delete.

.. _example-3:

Example
^^^^^^^

--------------

Volume functions
----------------

aidb.create_volume
^^^^^^^^^^^^^^^^^^

Creates an AIDB volume from a PGFS storage location for use as a
knowledge base data source. See :ref:`Configuring external storage <Configuring external storage>`  for setting up the PGFS
storage location first.

.. _parameters-4:

Parameters
^^^^^^^^^^

.. csv-table::
  :header: Parameter,Type,Description
  :widths: 12,15,25
  :align: left
  :class: longtable

  `name`,TEXT,Name for the volume.
  `server_name`,TEXT,Name of the PGFS storage location.
  `path`,TEXT,Sub-path within the storage location.
  `mime_type`,TEXT,"Data type: `Text`, `Image`, or `Pdf`."

.. _example-4:

Example
^^^^^^^

.. code:: sql

   SELECT aidb.create_volume(pdf_volume, my_s3_location, /, Pdf);

--------------

aidb.list_volumes
^^^^^^^^^^^^^^^^^

Lists all AIDB volumes in the database.

.. code:: sql

   SELECT * FROM aidb.list_volumes();

--------------

aidb.delete_volume
^^^^^^^^^^^^^^^^^^

Deletes an AIDB volume. Note: deleting the underlying PGFS storage
location also deletes all volumes built on top of it.

.. _parameters-5:

Parameters
^^^^^^^^^^

.. csv-table::
  :header: Parameter,Type,Description
  :widths: 12,15,25
  :align: left
  :class: longtable

  `volume_name`,TEXT,Name of the volume to delete.

.. code:: sql

   SELECT aidb.delete_volume(pdf_volume);