Schema for issues.fields.customfield_* is not generalized

[akoebbe-uu]

The schema defined in the IssueStream of this tap will not work for everyone because Jira uses something akin to EAV for custom fields. One project's issues.fields.customfield_10004 may be a string (how it's currently defined), but mine is a number. Therefore, the validation will always fail. Either the schema needs to allow for any datatype in issues.fields.customfield_*, or the tap needs to inspect Jira's field schema API to dynamically set the schema correctly for the instance it's extracting.

Field information API: /rest/api/3/field/ (Docs)

[prakharcode]

Hi @akoebbe-uu , I think we can create some sort of schema generator for this tap but don't know if that's super handy. @edgarrmondragon I know that tap-salesforce does something like this.
But is it included in general plan of meltano, to have such schema generator when source provide a schema itself.

[akoebbe-uu]

@prakharcode I think you're right. I'm pretty sure every instance of Jira will have different field types for custom fields. Meaning customfield_10006 may be the "Sprint" field for one Jira instance and "Story Points" for another. So I'm pretty sure this is mandatory.

The /rest/api/3/field endpoint will get the information needed to generate the schema. See: https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-issue-fields/#api-rest-api-3-field-get

[akoebbe-uu]

Here's a specific case where the extract fails because of the custom field mismatch.

In the tap code, customfield_10005 is defined as the following. This looks like the "Sprint" field schema.

tap-jira/tap_jira/streams.py

Lines 1680 to 1693 in 80638d8

	Property(
	"customfield_10005",
	ArrayType(
	ObjectType(
	Property("id", IntegerType),
	Property("boardId", IntegerType),
	Property("name", StringType),
	Property("state", StringType),
	Property("goal", DateTimeType),
	Property("startDate", DateTimeType),
	Property("completeDate", DateTimeType),
	)
	),
	),

But in an instance I'm trying to pull from the Jira API defines the schema as:

  {
    "id": "customfield_10005",
    "key": "customfield_10005",
    "name": "Business Value",
    "untranslatedName": "Business Value",
    "custom": true,
    "orderable": true,
    "navigable": true,
    "searchable": true,
    "clauseNames": [
      "Business Value",
      "Business Value[Number]",
      "cf[10005]"
    ],
    "schema": {
      "type": "number",
      "custom": "com.atlassian.jira.plugin.system.customfieldtypes:float",
      "customId": 10005
    }
  }