Integrate DeepEval with SAP AI Core

November 1, 2025 ☼ Python ☼ SAP ☼ LLM

DeepEval is an open-source framework designed to evaluate Large Language Models (LLMs) across various dimensions. By integrating DeepEval with SAP AI Core, you can seamlessly assess the performance of your deployed LLM models within the SAP ecosystem.

In this post I will show you how to set up DeepEval if you access your LLMs via SAP AI Core.

Prerequisites

• An SAP AI Core instance with access to your LLM models.
• Python environment with DeepEval installed. You can install it via pip:

pip install deepeval

Set Up DeepEval with SAP AI Core

By default, DeepEval uses gpt-4.1 to power all of its evaluation metrics. Usually you would authenticate with OpenAI using an API key.

OPENAI_API_KEY=<your-openai-api-key>

However, since we want to use SAP AI Core, we need to create a custom LLM wrapper. The key here is to extend the DeepEvalBaseLLM class and implement the required methods to interact with SAP AI Core’s Generative AI Hub.

import json
from typing import Any
from deepeval.models import DeepEvalBaseLLM
from gen_ai_hub.proxy.core.proxy_clients import get_proxy_client
from gen_ai_hub.proxy.langchain.openai import ChatOpenAI
proxy_client = get_proxy_client("gen-ai-hub")
class SapAiCoreGptJudge(DeepEvalBaseLLM):
    """
    Custom LLM class for DeepEval that uses SAP Cloud SDK for AI.
    This class integrates SAP AI Core's Generative AI Hub with DeepEval's
    evaluation framework, supporting both synchronous and asynchronous operations.
    Args:
        model_name: Name of the model deployed in SAP AI Core (e.g., 'gpt-4o', 'gpt-4o-mini')
        temperature: Controls randomness in generation (0.0 to 1.0)
        max_tokens: Maximum number of tokens to generate
        **model_kwargs: Additional keyword arguments to pass to the model
    """
    def __init__(self, model_name: str = "gpt-4o-mini", temperature: float = 0.0, max_tokens: int = 2000, **model_kwargs: Any):
        """Initialize the SAP AI Core LLM parameters"""
        self.model_name = model_name
        self.temperature = temperature
        self.max_tokens = max_tokens
        self.model_kwargs = model_kwargs
        self._model = None
    def load_model(self) -> ChatOpenAI:
        """
        Load and return the SAP AI Core chat model.
        This method initializes the ChatOpenAI client from gen_ai_hub SDK
        that connects to SAP AI Core's Generative AI Hub.
        Returns:
            ChatOpenAI: Configured chat model instance
        """
        if self._model is None:
            self._model = ChatOpenAI(proxy_model_name=self.model_name, temperature=self.temperature, max_tokens=self.max_tokens, **self.model_kwargs)
        return self._model
    def generate(self, prompt: str) -> str:
        """
        Generate a response synchronously.
        Args:
            prompt: The input prompt string
        Returns:
            str: The generated response content
        """
        chat_model = self.load_model()
        response = chat_model.invoke(prompt)
        # Normalize response.content to guaranteed str (avoid returning Any)
        content_raw = getattr(response, "content", "")
        if isinstance(content_raw, str):
            return content_raw
        return json.dumps(content_raw)
    async def a_generate(self, prompt: str) -> str:
        """
        Generate a response asynchronously.
        This method uses the async interface provided by the SAP SDK
        to enable non-blocking evaluation execution.
        Args:
            prompt: The input prompt string
        Returns:
            str: The generated response content
        """
        chat_model = self.load_model()
        response = await chat_model.ainvoke(prompt)
        content_raw = getattr(response, "content", "")
        if isinstance(content_raw, str):
            return content_raw
        return json.dumps(content_raw)
    def get_model_name(self) -> str:
        """
        Return the model name for identification.
        Returns:
            str: name of the model
        """
        return self.model_name

You can read more about creating custom LLMs in DeepEval’s documentation.

Let’s see an example of how to use this custom LLM with DeepEval to evaluate a model.

from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
gpt_judge = SapAiCoreGptJudge(model_name="gpt-4o-mini", temperature=0.0, max_tokens=1000)
metric = AnswerRelevancyMetric(model=gpt_judge, threshold=0.7)
test_case = LLMTestCase(
    input="What is photosynthesis?",
    actual_output="Photosynthesis is the process by which plants convert light into energy.",
    retrieval_context=["Photosynthesis occurs in chloroplasts..."],
)
metric.measure(test_case)
print(f"Score: {metric.score}")
print(f"Reason: {metric.reason}")

---

If you have any suggestions, questions, corrections or if you want to add anything please DM or tweet me: @zanonnicola