Model Serving Explained ======================= **Model Serving** in **AI Factory** allows you to deploy AI models as scalable, production-grade inference services — running on your Kubernetes infrastructure. It provides a **Kubernetes-native architecture** based on **KServe**, giving your models the ability to serve predictions and embeddings over network-accessible APIs. AI Factory Model Serving is optimized to support enterprise-class AI workloads with: - **GPU-accelerated infrastructure** - **Flexible scaling** - **Integrated observability** - **Sovereign AI alignment** — models run under your governance - **Seamless integration** with Gen AI Builder, Knowledge Bases, and other AI Factory pipelines ``---`` Before you start ---------------- Prerequisites for understanding Model Serving: - Familiarity with **Kubernetes** basics - Understanding of **KServe** and **InferenceService** - Awareness of **Model Library** → Model Serving workflow in AI Factory - Understanding of **Sovereign AI** principles — models running under your governance Suggested reading: - :ref:`Model Serving Concepts ` - :ref:`Model Library Explained In Depth ` - :ref:`Fundamental AI Concepts ` ``---`` How Model Serving works ----------------------- Core stack ^^^^^^^^^^ .. csv-table:: :header: Layer,Purpose :widths: 10,30 :align: left :class: longtable **AI Factory** ,Provides infrastructure and Model Serving APIs **Hybrid Manager Kubernetes Cluster** ,Hosts model-serving workloads **KServe** ,Manages model serving lifecycle and APIs **InferenceService** ,Deployed model resource **Model Library** ,Manages model image versions **GPU Nodes** ,Run high-performance model serving pods **User Applications** ,Call model endpoints via REST/gRPC Key components ^^^^^^^^^^^^^^ - **InferenceService** — Kubernetes CRD representing a deployed model. - **ServingRuntime / ClusterServingRuntime** — Define reusable runtime configurations. - **Model containers** — Currently focused on **NVIDIA NIM** containers in AI Factory 1.2. - **Observability** — Integrated Prometheus-compatible metrics, Kubernetes logging. ``---`` Supported models ---------------- AI Factory Model Serving currently supports **NVIDIA NIM** containers for: .. csv-table:: :header: Model Type,Example Usage :widths: 10,30 :align: left :class: longtable **Text Completion** ,"LLM agents, Assistants" **Text Embeddings** ,"Knowledge Bases, RAG" **Text Reranking** ,RAG pipelines **Image Embeddings** ,Multi-modal search **Image OCR** ,Document extraction See: :ref:`Supported models ` ``---`` Deployment architecture ----------------------- ``Applications → Model Endpoints (REST/gRPC) → KServe → GPU-enabled Kubernetes → Model Containers`` - Each model is isolated in its own **InferenceService**. - KServe manages: - Model lifecycle (start, stop, update) - Scaling (including scale-to-zero) - Endpoint routing (REST/gRPC) - GPU resources are provisioned and scheduled via Hybrid Manager integration. ``---`` Patterns of use --------------- Gen AI Builder ^^^^^^^^^^^^^^ - LLM endpoints power **Assistants** and **Agents**. - Embedding models support hybrid RAG pipelines. Knowledge Bases ^^^^^^^^^^^^^^^ - Embedding models serve **vectorization** needs. - Retrieval and reranking models power **semantic search** pipelines. Custom applications ^^^^^^^^^^^^^^^^^^^ - Business applications can consume **InferenceService** endpoints for: - Real-time predictions - Image analysis - Text processing ``---`` Best practices -------------- - Deploy models via the **Model Library → Model Serving** flow to ensure governance. - Use **ClusterServingRuntime** for reusable runtime configs. - Monitor **GPU utilization** and model latency closely. - Test **scale-to-zero** configurations for readiness in production. - Ensure **Model Library tags** are versioned and documented. - Regularly audit deployed **InferenceServices** as part of Sovereign AI governance. ``---`` Summary ------- Model Serving in AI Factory provides a robust, scalable architecture for serving **production AI models**: - Kubernetes-native serving with KServe - GPU acceleration and optimized serving runtimes - Integrated observability and governance - Tight integration with AI Factory components: Gen AI Builder, Knowledge Bases, custom AI pipelines Model Serving helps you implement **Sovereign AI** — with your models, on your infrastructure, under your control. ``---`` Next steps ---------- - :ref:`How-To Create an InferenceService for NVIDIA NIM Container ` - :ref:`How-To Verify InferenceServices and GPU usage ` - :ref:`How-To Deploy an NVIDIA NIM Microservices available on build.nvidia.comcontainer with KServe ` - :ref:`Model Library Explained In Depth ` ``---`` Model Serving gives you a powerful foundation for building intelligent applications and data products — securely, scalably, and under your governance — as part of **EDB Postgres® AI**. ``---``