Observability for Model Serving
===============================

Observability helps you ensure that your deployed AI models are running
efficiently and reliably within AI Factory.

Model Serving in AI Factory uses KServe on Kubernetes to serve models.
This provides built-in options to monitor:

- Model serving status and availability

- Resource usage (CPU, Memory, GPU)

- Inference performance and throughput

``---``

Key monitoring capabilities
---------------------------

KServe InferenceService status
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You can inspect model serving status directly via Kubernetes:

.. code:: sql

   kubectl get inferenceservice -n <namespace>

Common status fields include:

- Ready / NotReady

- URL endpoint

- Current replicas

- Allocated resources (GPU, CPU, Memory)

For detailed inspection:

.. code:: shell

   kubectl describe inferenceservice <name> -n <namespace>

``---``

GPU utilization monitoring
^^^^^^^^^^^^^^^^^^^^^^^^^^

If your models are deployed on GPU nodes, monitor GPU usage to optimize
resource allocation.

Example:

.. code:: shell

   kubectl top node

For deeper GPU-specific metrics (if supported):

.. code:: shell

   nvidia-smi

``---``

Prometheus and Grafana integration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If Prometheus is configured, AI Factory model serving exposes metrics
through KServe:

Prometheus annotations to enable scraping:

.. code:: yaml

   serving.kserve.io/enable-prometheus-scraping: "true"
   prometheus.kserve.io/port: "8000"
   prometheus.kserve.io/path: "/v1/metrics"

You can build Grafana dashboards to monitor:

- Inference requests per second

- Latency and error rates

- GPU utilization trends

- Pod restarts and health

``---``

Logs and debugging
^^^^^^^^^^^^^^^^^^

You can access detailed logs from model serving pods:

.. code:: shell

   kubectl logs -f <pod-name> -n <namespace>

Use logs to:

- Debug model loading or initialization issues

- Review inference request behavior

- Monitor response handling and error conditions

``---``

Hybrid Manager observability integration
----------------------------------------

When running AI Factory in conjunction with Hybrid Manager (HM), you can
integrate model serving observability into the broader HM observability
stack.

See :ref:`How-To Verify InferenceServices and GPU usage <How-To Verify InferenceServices and GPU usage>`  for additional guidance on using HM observability
tools with Model Serving.

``---``

Best practices
--------------

- Enable Prometheus scraping on InferenceServices for production.

- Regularly monitor GPU usage to optimize capacity.

- Set proper resource requests/limits in InferenceService specs.

- Leverage Grafana dashboards for real-time observability.

- Use log inspection to support debugging and tuning.

``---``

Related links
-------------

- :ref:`How-To Deploy an NVIDIA NIM Microservices available on build.nvidia.comcontainer with KServe <How-To Deploy an NVIDIA NIM Microservices available on build.nvidia.comcontainer with KServe>`  

- :ref:`Model Serving Concepts <Model Serving Concepts>`  

- :ref:`How-To Verify InferenceServices and GPU usage <How-To Verify InferenceServices and GPU usage>`  

``---``

By applying these observability practices, you can operate your AI
Factory Model Serving workloads with greater confidence, proactively
identifying and addressing issues before they impact users.