Polymorphic responses

[papegaaij]

The API we develop uses quite some inheritance in its models. The approach Kiota takes with allOf suits us quite well, but we are having problems defining paths with polymorphic responses. Our API is a bit too large to share here, but I've created a similar situation using the inheritance example from the documentation. This example API has one path that returns a directoryObject, which can be either a user or a group.

If the response references the directoryObject, the resulting code only contains this directoryObject and no code is generated for the user and group. In this situation the entire typing is lost. If I change the type of the response to a oneOf, the resulting code does contain model classes for directoryObject, user and group. However, the result type of the request method will be a union type or wrapper. Although this is technically correct, it is not very convenient. The OpenAPI specification needs to enumerate all possible types and the user of the API has to check the returned type on every call.

I'm looking for a way to have the request method use directoryObject as return type, but use the actual subtype (user or group) as value.

The example OpenAPI specification is:

{
  "openapi" : "3.0.1",
  "info" : {
    "title" : "OpenAPI polymorphism",
    "version" : "1"
  },
  "servers" : [ {
    "url" : "http://localhost:8080"
  } ],
  "paths" : {
    "/example" : {
      "get" : {
        "summary" : "Read a polymorphic type",
        "operationId" : "example",
        "responses" : {
          "200" : {
            "description": "Example",
            "content" : {
              "application/json" : {
                "schema" : {
                  "$ref" : "#/components/schemas/microsoft.graph.directoryObject"
                }
              }
            }
          }
        }
      }
    }
  },
  "components" : {
    "schemas" : {
      "microsoft.graph.directoryObject": {
        "allOf": [
          {
            "title": "directoryObject",
            "required": ["@odata.type"],
            "type": "object",
            "properties": {
              "@odata.type": {
                "type": "string",
                "default": "#microsoft.graph.directoryObject"
              }
            }
          }
        ],
        "discriminator": {
          "propertyName": "@odata.type",
          "mapping": {
            "#microsoft.graph.user": "#/components/schemas/microsoft.graph.user",
            "#microsoft.graph.group": "#/components/schemas/microsoft.graph.group"
          }
        }
      },
      "microsoft.graph.user": {
        "allOf": [
          { "$ref": "#/components/schemas/microsoft.graph.directoryObject" },
          {
            "title": "user",
            "type": "object"
          }
        ]
      },
      "microsoft.graph.group": {
        "allOf": [
          { "$ref": "#/components/schemas/microsoft.graph.directoryObject" },
          {
            "title": "group",
            "type": "object"
          }
        ]
      }
    }
  }
}

[baywet]

Hi @papegaaij
Thanks for using Kiota and for reaching out.
Which language are you generating for? What you're describing should already be the case but some languages are more mature than others?
Which version of kiota are you generating with? Have you tried with 1.3.0?

[papegaaij]

Hi @baywet
I was using 1.2.1, but just tried with 1.3.0 as well. I've generated code for CSharp, Go and Java from the example OpenAPI specification I posted above, and all 3 generate very similar output. There is no models package/directory and the response of the single operation is placed in a DirectoryObjectResponse. This does not contain the user or group types, only the directoryObject. When I put additional properties in user, these will not be generated anywhere in the code.

It could be that the example I provided is wrong or not in the format expected by Kiota. OpenAPI is rather free and there are many ways to write the same spec. If so, could you please explain how to change to spec? The documentation only contains an example about the inheritance, not with a polymorphic response.

[baywet]

It might be because your user and group schemas don't have any properties of themselves and are getting trimmed (unless you simplified them for this discussion).
Could you compare your definition with our official ones please? https://github.com/microsoftgraph/msgraph-metadata/blob/master/openapi/v1.0/openapi.yaml

Case and point /groups/id/members returns a collection of directory objects (wrap in a response class in that case, but similar to yours) https://github.com/microsoftgraph/msgraph-sdk-dotnet/blob/10eaf4b4bc7f0dc535b41938402bedb60d95cc7f/src/Microsoft.Graph/Generated/Models/DirectoryObjectCollectionResponse.cs#L17

[papegaaij]

@baywet It seems only those types are generated that are explicitly referenced from responses. If I change the example OpenAPI by adding a second path and have that path return the user, I get both directoryObject and user in the model, but still no group. I suspect the large msgraph spec references most of the types directly or indirectly.

[baywet]

The trimming logic is supposed to include directly derived types of types that are referenced specifically for this downcast/discriminator scenario.
For sanity, do you get a different result with 1.1?

[papegaaij]

No, the outcome is the same with 1.1.3. I've also added a property to both user and group, to make sure these were not trimmed because they are empty, but that doesn't help either.

I decided to take a different approach and trim down the MSGraph openapi.yaml and I think I may have found the cause. In the example on https://learn.microsoft.com/nl-nl/openapi/kiota/models the base class is directoryObject, but it contains a reference to its superclass entity. I removed that reference to keep the inheritance 1 level deep, however, I left the remainder of the type the same. It turns out that the allOf in the base class confuses Kiota and breaks the polymorphism. If I change the definition of directoryObject to the following, it works fine again:

      "microsoft.graph.directoryObject": {
        "title": "directoryObject",
        "required": ["@odata.type"],
        "type": "object",
        "properties": {
          "@odata.type": {
            "type": "string",
            "default": "#microsoft.graph.directoryObject"
          }
        },
        "discriminator": {
          "propertyName": "@odata.type",
          "mapping": {
            "#microsoft.graph.user": "#/components/schemas/microsoft.graph.user",
            "#microsoft.graph.group": "#/components/schemas/microsoft.graph.group"
          }
        }
      },

It turns out that our own OpenAPI has the exact same issue, with the base type consisting of a single allOf.

[baywet]

Thanks for the additional information. I'm guessing this is somewhat related to #2720 and #2434.
It really looks like a duplicate of #2434 in facts... Would you mind trying the earlier form for directoryObject, but removing any other property besides the allOf please?

[papegaaij]

@baywet I don't really understand what you mean. The original form only had a single property, the discriminator. I don't see any part besides the allOf I could remove workout breaking the example. This is the original definition:

      "microsoft.graph.directoryObject": {
        "allOf": [
          {
            "title": "directoryObject",
            "required": ["@odata.type"],
            "type": "object",
            "properties": {
              "@odata.type": {
                "type": "string",
                "default": "#microsoft.graph.directoryObject"
              }
            }
          }
        ],
        "discriminator": {
          "propertyName": "@odata.type",
          "mapping": {
            "#microsoft.graph.user": "#/components/schemas/microsoft.graph.user",
            "#microsoft.graph.group": "#/components/schemas/microsoft.graph.group"
          }
        }
      },

[baywet]

I mean this

 "microsoft.graph.directoryObject": {
        "allOf": [
          {
            "title": "directoryObject",
            "required": ["@odata.type"],
            "type": "object",
            "properties": {
              "@odata.type": {
                "type": "string",
                "default": "#microsoft.graph.directoryObject"
              }
            }
          }
        ]
      },

[papegaaij]

I've tried several different setups, but as soon as the base type has an allOf, the inheritance is broken. I've also tried adding the microsoft.graph.entity with an allOf, and it gives the same issue. The base type cannot have an allOf.

In the same category, I've also spotted an issue with the subtypes: these must have an allOf and it must have exactly 2 parts. If, for some reason, I split the allOf of the subtype in 3 parts, the reference to its supertype, the first half of the properties and the second half of the properties, code generation is also broken: only the second half of the properties get generated.

It seems Kiota is quite strict in the format of the OpenAPI file, while the OpenAPI specification allows for a lot of variation. Maybe Kiota could benefit from a normalization step in the generation process, where these variations are all converted into a single canonical format?

[baywet]

base type being an all of and derailing inheritance: if you could come up with a unit test as found in KiotaBuilderTests it'd be super helpful to investigate this issue.

x > 2 entries: this should get flattened into a single class (with all properties) after implementing #2438 (version 1.2)

We did a lot of thinking and adjustments on the types projection. The gist being JSON schema (what OpenAPI uses to describe the payloads) is a validation schema and not an IDL. People use AllOf to materialize inheritance by convention, there are a couple of different conventions going on, but it's not what JSON schema is designed for. While there are ongoing efforts to define an IDL vocab (extension) for JSON schema, we're still a long way out from broad adoption. For this reason we've had to make judgement calls.

[papegaaij]

@baywet as you can see, I've created a PR with 2 new tests. Please note that these were my first lines of C# ever, so it could be that I did not do things entirely correct.