Data parsing
============

EDB Postgres AI provides built-in functions to parse and extract text
from various data formats, including HTML and PDF. These functions are
designed to handle the complexities of different formats while providing
clean, structured text output that can be easily processed in subsequent
pipeline steps.

Parse HTML
----------

The ``aidb.parse_html()`` function extracts readable text from HTML
strings, stripping away tags while preserving the logical structure of
the content.

.. code:: sql

   SELECT * FROM aidb.parse_html(
       html =>
           <h1>Hello, world!</h1>
           <p>This is my first web page.</p>
           <p>
               It contains some <strong>bold text</strong>, some <em>italic test</em>, and a <a href="https://google.com" target="_blank">link</a>.
           </p>

           <img src="postgres_logo.png" alt="Postgres Logo Image">

           <ol>
               <li>List item</li>
               <li>List item</li>
               <li>List item</li>
           </ol>,
       options => {"method": "StructuredPlaintext"} -- Default
   );

   __OUTPUT__
                           parse_html
   - ----------------------------------------------------------
    Hello, world!                                            +
                                                             +
    This is my first web page.                               +
                                                             +
    It contains some bold text, some italic test, and a link.+
                                                             +
    Postgres Logo Image                                      +
    List item                                                +
    List item                                                +
    List item                                                +

   (1 row)

- The ``method`` determines how the HTML is parsed:

  - ``StructuredPlaintext`` (default) — Uses an algorithmic approach to
    extract raw text from HTML, ideal for simple data ingestion.

  - ``StructuredMarkdown`` — Converts HTML into Markdown-like text. This
    retains important syntactical context (like headers and lists),
    which can help LLMs better understand the documents’s original
    hierarchy.

Parse PDF
---------

The ``aidb.parse_pdf()`` function extracts text from binary PDF data.
Because PDFs are often structurally complex, this function offers robust
error handling.

.. code:: sql

   SELECT * FROM aidb.parse_pdf(
       bytes => decode(255044462d312e340a25b89a929d0a312030206f626a3c3c2f547970652f4361
                       74616c6f672f50616765732033203020523e3e0a656e646f626a0a322030206f
                       626a3c3c2f50726f64756365722847656d426f782047656d426f782e50646620
                       312e37202831372e302e33352e313034323b202e4e4554204672616d65776f72
                       6b29292f4372656174696f6e4461746528443a32303231313032383135313732
                       312b303227303027293e3e0a656e646f626a0a332030206f626a3c3c2f547970
                       652f50616765732f4b6964735b34203020525d2f436f756e7420312f4d656469
                       61426f785b302030203539352e3332203834312e39325d3e3e0a656e646f626a
                       0a342030206f626a3c3c2f547970652f506167652f506172656e742033203020
                       522f5265736f75726365733c3c2f466f6e743c3c2f46302036203020523e3e3e
                       3e2f436f6e74656e74732035203020523e3e0a656e646f626a0a352030206f62
                       6a3c3c2f4c656e6774682035393e3e73747265616d0a42540a2f463020313220
                       54660a3120302030203120313030203730322e3733363636363720546d0a2848
                       656c6c6f20576f726c642129546a0a45540a656e6473747265616d0a656e646f
                       626a0a362030206f626a3c3c2f547970652f466f6e742f537562747970652f54
                       797065312f42617365466f6e742f48656c7665746963612f4669727374436861
                       722033322f4c61737443686172203131342f5769647468732037203020522f46
                       6f6e7444657363726970746f722038203020523e3e0a656e646f626a0a372030
                       206f626a5b323738203237382030203020302030203020302030203020302030
                       2030203020302030203020302030203020302030203020302030203020302030
                       2030203020302030203020302030203020302030203020302037323220302030
                       2030203020302030203020302030203020302030203020302039343420302030
                       2030203020302030203020302030203020302030203535362035353620302030
                       203020302030203020323232203020302035353620302030203333335d0a656e
                       646f626a0a382030206f626a3c3c2f547970652f466f6e744465736372697074
                       6f722f466c6167732033322f466f6e744e616d652f48656c7665746963612f46
                       6f6e7446616d696c792848656c766574696361292f466f6e7457656967687420
                       3530302f4974616c6963416e676c6520302f466f6e7442426f785b2d31363620
                       2d3232352031303030203933315d2f436170486569676874203731382f584865
                       69676874203532332f417363656e74203731382f44657363656e74202d323037
                       2f5374656d482037362f5374656d562038383e3e0a656e646f626a0a78726566
                       0a3020390a303030303030303030302036353533352066200a30303030303030
                       303135203030303030206e200a30303030303030303539203030303030206e20
                       0a30303030303030313739203030303030206e200a3030303030303032353720
                       3030303030206e200a30303030303030333436203030303030206e200a303030
                       30303030343531203030303030206e200a303030303030303537332030303030
                       30206e200a30303030303030373733203030303030206e200a747261696c6572
                       0a3c3c2f526f6f742031203020522f49445b3c39333932413539463342453742
                       383430383035443632373436453841344632393e3c3933393241353946334245
                       3742383430383035443632373436453841344632393e5d2f496e666f20322030
                       20522f53697a6520393e3e0a7374617274787265660a3938380a2525454f460a, hex),
       options => {"method": "Structured", "allow_partial_parsing": true} -- Default
   );

   __OUTPUT__
    part_id |     text
   - --------+--------------
          0 | Hello World!+
            |
   (1 row)

- The ``method`` determines how the PDF is parsed:

  - ``Structured`` method (default): Relies on the standard PDF
    specification to identify text blocks. Structural PDF parsing that
    relies on the PDF specification can struggle with PDFs that are not
    spec-compliant, especially if they include complex or proprietary
    encodings and different fonts.

  - ``allow_partial_parsing`` : A safety flag that tells the system
    whether to continue to parse PDFs when the parser encounters errors
    on one or more pages. Setting this to ``true`` (default) ensures you
    get as much data as possible from non-compliant or complex files.

- Page mapping: The resulting ``part_id`` column maps directly to the
  specific page index from which the text was extracted, allowing you to
  maintain the original document reference in your AI pipeline.

..  Tip::
   Both parsing operations perform unnesting. This means they transform a single input (like a multi-page PDF) into multiple rows of data, automatically generating a `part_id`  to keep the segments in the correct order.