Null value terminology

[only1chunts]

Previously we have been encouraging the use of the INSDC set of null terms when required:

term	definition
not applicable	information is inappropriate to report, can indicate that the standard itself fails to model or represent the information appropriately
missing-not collected	information of an expected format was not given because it has not been collected
missing- not provided	information of an expected format was not given, a value may be given at the later stage
missing - restricted access	information exists but can not be released openly because of privacy concerns

These have worked OK when used, so there may not be any need to change things, but I wanted to raise this option for discussion. Could /should we be using the HL7 set of "Null Flavors" instead: https://terminology.hl7.org/3.0.0/ValueSet-v3-NullFlavor.html

As these are more comprehensive than the INSDC set.

[cmungall]

Do we want the same value set for each field?

Do we always allow missing values? E.g. if a field is required, so we allow missing value enums?

[lschriml]

Hello @only1chunts - the null standards were devised with the GSC in the CIG working group and we agreed to adopt and promote their use with the INSDC. If the HL7 values are in wide use/useful to our community, I would suggest, we look to see how they map to the INSDC terms, and add them as synonyms where possible.

Cheers,
Lynn

[only1chunts]

@lschriml , I'm good with just using the set of 4 simple terms that we already encourage, I'm just raising the possibility that there are other options out there that are more comprehensive in their coverage.
@cmungall , Where null values are permissible I think we should be expecting the same set of 4 options as listed above. As for which terms/slots we allow null values.... thats probably a bigger discussion! Does a valid null value fulfill the mandatory requirement? in some cases maybe it does. Where we specify a CV/enum then we should probably start checking which of the null values are permissible and including those in the list.

[turbomam]

What are the consequences of making a term mandatory, specifying a non-string Value syntax like {float}, and then allowing these stringy INSDC null terms?

[only1chunts]

INSDC are now moving to a more extensive set of missing value terms to help provide more details on the reason the value is missing https://www.insdc.org/submitting-standards/missing-value-reporting/

[pbuttigieg]

This is a bit off. You can't validly add these values to fields that have types like number or boolean. It's more normal to leave missing values missing or add NA, then include a separate property where explanations of the missingness can be included.

Otherwise we're creating the need for users to write tools to understand a bespoke vocabulary (under unclear governance), and thats not very good practice for a standards organisatiin.

[Woolly-at-EBI]

This is the latest list of missing values is below:
INSDC page was not clear, e.g. “missing:control sample” or “not applicable:control sample”
AI: TWG: add the terms to the YAML
AI: Chris: add details to the GenSC.org website

not collected|not provided|restricted access|missing: control sample|missing: sample group|missing: synthetic construct|missing: lab stock|missing: third party data|missing: data agreement established pre-2023|missing: endangered species|missing: human-identifiable

[only1chunts]

I have updated the gensc.org page to include the INSDC missing value terms.
Note - I did not include the option "missing: data agreement established pre-2023" as I feel that is very INSDC specific as the agreement it refers to is the one agreed by the INSDC partners.

[Woolly-at-EBI]

Agree Chris, that "missing: data agreement established pre-2023" is very INSDC specific.

[mslarae13]

See https://www.gensc.org/pages/standards/checklists.html#:~:text=Missing%20value%20reporting,remove%20mandatory%20terms!).

Terms to add need reviewed as some don't exactly match INSDC - https://www.insdc.org/submitting-standards/missing-value-reporting/
Need to check with @Woolly-at-EBI about the terms as it's inconsistent in insdc documentation.
not applicable - control sample vs missing: control sample shown in examples below.

• missing: control sample - Information is not applicable as the sample represents a negative control sample collected in a lab.
• missing: sample group - Information is not applicable as the sample represents a group of samples that do not have a single origin. E.g. for co-assembly or transcriptome assembly.
• Not collected - information of an expected format was not given because it has not been collected
• Not provided - information of an expected format was not given, a value may be given at the later stage
• Restricted access - information exists but can not be released openly because of privacy concerns
• missing: synthetic construct - Information does not exist as the sample represents an ab-initio synthetic construct.
• missing: lab stock - Information was not collected as the sample represents a cultured cell line or model organism under long-term lab control.
• missing: third party data - Information does not exist as the metadata was not collected or reported in records predating the 2023 agreement. For use in Third Party data submissions.
• missing: endangered species - Information can not be reported as the target organism is endangered e.g. on the IUCN red-list.
• missing: human-identifiable - Information can not be reported as the metadata would make the sample human-identifiable.

Adjustments, remove capitalization & change : to something else as : is not friendly to all software.

[Woolly-at-EBI]

The INSDC web page URL is the correct one. https://www.insdc.org/submitting-standards/missing-value-reporting/

These are the exact expected values, if missing values are used in INSDC where mandatory values can not be provided:
not applicable
not collected
not provided
restricted access
missing: control sample
missing: sample group
missing: synthetic construct
missing: lab stock
missing: third party data
missing: data agreement established pre-2023
missing: endangered species
missing: human-identifiable

In ENA this is the relevant regex we are using where a mandatory field allows missing values.
|(^not applicable$)|(^not collected$)|(^not provided$)|(^restricted access$)|(^missing: control sample$)|(^missing: sample group$)|(^missing: synthetic construct$)|(^missing: lab stock$)|(^missing: third party data$)|(^missing: data agreement established pre-2023$)|(^missing: endangered species$)|(^missing: human-identifiable$)

There is no capitalisation for any of these missing values.
I doubt that removing a ":" will be amenable to INSDC in the short term as this got pushed out as a standard last year, so it would need to be mapped before submission to INSDC repos. ":" is not as difficult for sw to handle as things like back-slashes, but yes there may still be some problems

[Woolly-at-EBI]

Yes, it is a bit inconsistent with "missing:" being prefixed sometimes. There is this sentence of guidance on the web page

When reporting a missing mandatory field, the eight granular ‘reporting level’ terms need to be preceded with the term ‘missing: ’ to declare both the absence of a true value as well as the reason.

[turbomam]

goals:

• traceability
• consistency, or explanations for inconsistency

eg

• why is bare 'not applicable' acceptable but bare 'missing' isn't?
• why is 'missing: control sample' acceptable, when 'control sample' has the "top level" parent 'not applicable'

[turbomam]

@pbuttigieg, @mslarae13 and I are curious about having clear GSC guidance on missing data annotations. If INSDC can't provide crystal clear guidance, we can still provide mappings to their namespace

[turbomam]

Also including textual missing data indicators and numeric fields leads to poor computability

We should have a pattern for indicating missing data

[turbomam]

I can report the relative abundance of the verbatim INSDC missing value indicators, but it would be harder to anticipate all of the ways submitters have modified the INSDC codes

[mslarae13]

not applicable is allowed along, no additional information
Is 'missing' allowed on its own?

From what Peter provided, would the MissingValueEnum really be

• not applicable
• not collected
• not provided
• restricted access
• missing: control sample
• missing: sample group
• missing: synthetic construct
• missing: lab stock
• missing: third party data
• missing: data agreement established pre-2023
• missing: endangered species
• missing: human-identifiable

Making "missing" valid means numeric fields are not text, and that's inconsistent.
@Woolly-at-EBI to still check in & confirm.

[pbuttigieg]

I propose we start a new issue on how GSC standards are going to handle the reporting of MAR, MNAR, MCAR, and other forms of missing data generically.

We absolutely should not allow strings like "missing" into numeric fields.

We could map to or allow INSDC values (among others) in a broader specification on how to add explanations to empty values, without messing up data types

[Woolly-at-EBI]

Here are is the rst file (I had to give it a .txt extension) and I rendered it into PDF via HTML(from the rst) so that one can see it more clearly. Had to learn the wonders of pandoc, so a useful side effect.

Reporting Missing Values.pdf
missing-values.rst.txt

• these are what I want to make live on the INSDC page, as these minor changes increase the clarity, when my colleague Maira is back from her holidays. (she controls acceptance of PRs for the user documentation)

Colman(the ENA product owner) agreed that it was inconsistent with not all terms needing the "missing: " prefix, and will propose that as a change at the next INSDC meeting(May 2025!).

and to answer Montana's question, following the logic of the documentation, all the highest level terms which include "missing" ought to be accepted as a standalone term. Currently, the regex at ENA does not support this. I am now checking this with the NCBI and DDJB, if a yes, I will dink the regex's at least at ENA to accept such. If not I will push strongly for any early inclusion.

[mslarae13]

Converting to discussion until we resolve GSC methods

[mslarae13]

Assuming @Woolly-at-EBI 's table is approved, the missing values that are allowed by INSDC are

• not applicable
• not collected
• not provided
• restricted access
• missing: control sample
• missing: sample group
• missing: synthetic construct
• missing: lab stock
• missing: third party data
• missing: endangered species
• missing: human-identifiable

@pbuttigieg has made the point that when a slot requires a value like ph we should NOT allow text for "missing"

name: ph
description: Ph measurement of the sample, or liquid portion of sample, or aqueous
  phase of the fluid
title: pH
examples:
- value: '7.2'
from_schema: https://w3id.org/mixs
keywords:
- ph
slot_uri: MIXS:0001001
alias: ph
domain_of:
- FoodFarmEnvironment
- HydrocarbonResourcesCores
- HydrocarbonResourcesFluidsSwabs
- MicrobialMatBiofilm
- MiscellaneousNaturalOrArtificialEnvironment
- Sediment
- Soil
- WastewaterSludge
- Water
range: float

Decisions to make BEFORE making issues

• How will GSC capture missing values?
• For non-string or non-text slots, how do we validate missing values
• What values are allowed
• Confirm missing value information should only be permissible in REQUIRED fields. Optional fields do NOT need missing value qualifiers
• How will LinkML accomodate this?

[pbuttigieg]

The cleanest solution is to add two sub-properties to each MIxS property (term). This is easy in JSON or YAML type data, less so in tabular formats.

The first property should express the more globally accepted and useful type of missingness (MAR, MCAR, MNAR, Structured missingness)

The second property could allow levels like those below.

not applicable
not collected
not provided
restricted access

Unfortunately, the other levels (below) are poorly constructed and should not be base values. They can be added to a description sub-property, as some could be used to explain the levels above (e.g. endangered species explains "not provided" or perhaps "restricted"). That suggests a third, more verbose property.

missing: control sample
missing: sample group
missing: synthetic construct
missing: lab stock
missing: third party data
missing: endangered species
missing: human-identifiable

Confirm missing value information should only be permissible in REQUIRED fields. Optional fields do NOT need missing value qualifiers

Missing values apply everywhere - it doesn't make sense to limit them to required fields. That's poor data management.

[lschriml]

A while ago, We worked with the INSDC to define the values. If we want to change the options, we should reconvene that working group.Cheers,LynnSent from my iPhoneOn Jul 17, 2024, at 8:25 PM, Pier Luigi Buttigieg ***@***.***> wrote: The cleanest solution is to add two sub-properties to each MIxS property (term). This is easy in JSON or YAML type data, less so in tabular formats. The first property should express the more globally accepted and useful type of missingness (MAR, MCAR, MNAR, Structured missingness) The second property could allow levels like those below. not applicable not collected not provided restricted access Unfortunately, the other levels (below) are poorly constructed and should not be base values. They can be added to a description sub-property, as some could be used to explain the levels above (e.g. endangered species explains "not provided" or perhaps "restricted"). That suggests a third, more verbose property. missing: control sample missing: sample group missing: synthetic construct missing: lab stock missing: third party data missing: endangered species missing: human-identifiable Confirm missing value information should only be permissible in REQUIRED fields. Optional fields do NOT need missing value qualifiers Missing values apply everywhere - it doesn't make sense to limit them to required fields. That's poor data management. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[Woolly-at-EBI]

This is an important and helpful discussion. Always much to learn and improve, so thank you.

Agree with Lynn, to improve the controlled vocabulary a joint INSDC GSC working group is a good way forwards.

I heard from my colleagues at DDJB and NCBI over the last few days that:

1. /^missing$/ is allowable(exception at the DDJB for collection_date and geo_loc_name). ( So I am now confirming with my fellow ENA content team to allow /^missing$/ too.)
2. besides the above, all INSDC members representative had the same understanding of the allowable "missing values”

To confirm: in the INSDC checklists, optional fields do not need missing value qualifiers. A complication is that the same field is mandatory in certain checklists and then optional in other checklists.

The expanded INSDC missing value reporting I believe was part of the effort around making the spatiotemporal fields(collection_date and geo_loc_name) mandatory and recording some granularity of why it was missing.

As Pier indicates there are different data management practices. In INSDC, we are coping with the real world with many including historical and other situations, so it is a balancing act. For example with endangered species, quite often the location will deliberately not been revealed.

[pbuttigieg]

I heard from my colleagues at DDJB and NCBI over the last few days that:

1. /^missing$/ is allowable(exception at the DDJB for collection_date and geo_loc_name). ( So I am now confirming with my fellow ENA content team to allow /^missing$/ too.)

Sure, but this doesn't mean it's good practice for a global standard - this is more how one implementation handles messy data.

2. besides the above, all INSDC members representative had the same understanding of the allowable "missing values”

To confirm: in the INSDC checklists, optional fields do not need missing value qualifiers. A complication is that the same field is mandatory in certain checklists and then optional in other checklists.

Not so clear on what is meant here.

The expanded INSDC missing value reporting I believe was part of the effort around making the spatiotemporal fields(collection_date and geo_loc_name) mandatory and recording some granularity of why it was missing.

This is also better accomplished with a generic missingness module.

As Pier indicates there are different data management practices. In INSDC, we are coping with the real world with many including historical and other situations, so it is a balancing act. For example with endangered species, quite often the location will deliberately not been revealed.

It's never a good idea to mix data type in a standard, however. Legacy data is indeed going to be a headache, but we can't let that hold back improvements for the future.

[pbuttigieg]

A while ago, We worked with the INSDC to define the values. If we want to change the options, we should reconvene that working group.

@lschriml I don't think is needed, it's more that we parse the outputs of that group and place them in more logically consistent slots