Diving Deeper with Structured Outputs

Helping enhance your understanding and optimal usage of structured outputs and LLMs

In the previous article, we were introduced to structured outputs using OpenAI. Since the general availability release in ChatCompletions API (v1.40.0), structured outputs have been applied across dozens of use cases, and spawned numerous threads on OpenAI forums.

In this article, our aim is to provide you with an even deeper understanding, dispel some misconceptions, and give you some tips on how to apply them in the most optimal manner possible, across different scenarios.

## Structured outputs overview

Structured outputs are a way of enforcing the output of an LLM to follow a pre-defined schema — usually a JSON schema. This works by transforming the schema into a context free grammar (CFG), which during the token sampling step, is used together with the previously generated tokens, to inform which subsequent tokens are valid. It’s helpful to think of it as creating a regex for token generation.

OpenAI API implementation actually tracks a limited subset of JSON schema features. With more general structured output solutions, such as Outlines, it is possible to use a somewhat larger subset of the JSON schema, and even define completely custom non-JSON schemas — as long as one has access to an open weight model. For the purpose of this article, we will assume the OpenAI API implementation.

### JSON Schema and Pydantic

According to JSON Schema Core Specification, “JSON Schema asserts what a JSON document must look like, ways to extract information from it, and how to interact with it”. JSON schema defines six primitive types — null, boolean, object, array, number and string. It also defines certain keywords, annotations, and specific behaviours. For example, we can specify in our schema that we expect an array and add an annotation that minItems shall be 5 .

Pydantic is a Python library that implements the JSON schema specification. We use Pydantic to build robust and maintainable software in Python. Since Python is a dynamically typed language, data scientists do not necessarily think in terms of variable types — these are often implied in their code. For example, a fruit would be specified as:

```python

fruit = "apple"

```

…while a function declaration that returns “fruit” from some piece of data would often be specified as:

```python

def get_fruit(data):

return data

```

Pydantic on the other hand allows us to generate a JSON-schema compliant class, with properly annotated variables and type hints, making our code more readable/maintainable and in general more robust, i.e.

```python

class Fruit(BaseModel):

fruit: str

```

OpenAI actually strongly recommends the use of Pydantic for specifying schemas, as opposed to specifying the “raw” JSON schema directly. There are several reasons for this. Firstly, Pydantic is guaranteed to adhere to the JSON schema specification, so it saves you extra pre-validation steps. Secondly, for larger schemas, it is less verbose, allowing you to write cleaner code, faster. Finally, the openai Python package actually does some housekeeping, like setting additionalProperties to False for you, whereas when defining your schema “by-hand” using JSON, you would need to set these manually, for every object in your schema (failing to do so results in a rather annoying API error).

## Limitations

As we alluded to previously, the ChatCompletions API provides a limited subset of the full JSON schema specification. There are numerous keywords that are not yet supported, such as minimum and maximum for numbers, and minItems and maxItems for arrays — annotations that would be otherwise very useful in reducing hallucinations, or constraining the output size.

Certain formatting features are also unavailable. For example, the following Pydantic schema would result in API error when passed to response_format in ChatCompletions:

```python

class Post(BaseModel):

date_published: datetime

```

It would fail because openai package has no format handling for datetime , so instead you would need to set date_published as a str and perform format validation (e.g. ISO 8601 compliance) post-hoc.

Other key limitations include the following:

- Schemas must be able to be represented using Pydantic models.

- Schemas must be able to be serialized to JSON.

- Schemas must be able to be validated using Pydantic models.

- Schemas must be able to be deserialized from JSON.

## Tips and tricks

With all this in mind, we can now go through a couple of use cases with tips and tricks on how to enhance the performance when using structured outputs.

### Creating flexibility using optional parameters

Let’s say we are building a web scraping application where our goal is to collect specific components from the web pages. For each web page, we supply the raw HTML in the user prompt, give specific scraping instructions in the system prompt, and define the following Pydantic model:

```python