index.qmd

---
title: Python package for metadata schemas
title-slide-attributes:
  data-background-image: assets/kul-title-bg.png
  data-background-size: contain
author: Mariana Montes and Ronny Moreas
date: 2024-05-29
format:
  revealjs:
    theme: [default, assets/kul.scss]
    footer: iRODS UGM 2024, Amsterdam
    logo: assets/kul-logo.png
    echo: true
code-annotations: hover
---

```{python}
# | label: setup
# | include: false
import os, os.path
import json
from datetime import date
from irods.session import iRODSSession
from mango_mdschema import Schema

home_dir = "/icts/home/datateam_icts_icts_test/irodsugm_demo"

env_file = ".irods_environment.json"

session = iRODSSession(irods_env_file=env_file)
```

## Outline

- The ManGO Metadata Schema Manager
- From JSON to validation
- From a Python dictionary to AVUs
- Conclusion

# The ManGO Metadata Schema Manager

## Form to add metadata

::: {style="overflow-y: scroll; height: 600px;"}
![](img/empty-mdschema-form.png)
:::

::: footer
[ManGO Schema Manager](github.com/kuleuven/mango-metadata-schemas)
:::

## View of the schema metadata

::: {style="overflow-y: scroll; height: 600px;"}
![](img/view-metadata.png)
:::

::: footer
[ManGO Schema Manager](github.com/kuleuven/mango-metadata-schemas)
:::

## The Schema Manager

::: {style="overflow-y: scroll; height: 600px;"}
![](img/schema-manager.png)
:::

::: footer
[ManGO Schema Manager](github.com/kuleuven/mango-metadata-schemas)
:::

## Metadata Schemas as JSON
book-v2.0.0-published.json
```{python}
#| echo: false
with open("book-v2.0.0-published.json", "r") as f:
    schema_json = json.load(f)
schema_json
```

## Minimal example

```sh
pip install mango-mdschema
```

<br>

```{.python filename="add_schema_metadata.py" code-line-numbers="|2,9,10|1,5,6|3,8,11|"}
import json
from irods.session import iRODSSession
from mango_mdschema import Schema

with open("metadata_file.json", "r") as f:
  my_metadata = json.load(f) # a dictionary

my_schema = Schema("book-v2.0.0-published.json")
with iRODSSession(irods_env_file=env_file) as session:
  irods_object = session.collections.get(home_dir).data_objects[0]
  my_schema.apply(irods_object, my_metadata) # includes validation
```

# From JSON to validation

## Metadata schemas as JSON

book-v2.0.0-published.json
```{python}
#| echo: false
schema_json
```

## Interpretation via the Python package

```{python}
book_schema = Schema("book-v2.0.0-published.json")  # <1>
print(book_schema)
```

1. The optional `prefix` argument lets you tailor it to your implementation.

## Field requirements

```{python}
book_schema.print_requirements("publishing_date")
```

```{python}
book_schema.print_requirements("publisher")
```

## Field requirements

```{python}
book_schema.print_requirements("cover_colors")
```

# From a Python dictionary to AVUs

## Required fields and defaults

```{python}
#| output-location: fragment
my_metadata = {
    "title": "An exemplary book",                                        # <1>
    "author": [                                                          # <2>
        {"name": "Fulano De Tal", "email": "fulano.detal@kuleuven.be"},  # <2>
        {"name": "Jane Doe", "email": "jane.doe@kuleuven.be"},           # <2>
    ],                                                                   # <2>
    "ebook": "Available",
    "publishing_date": "2024-02-01",                                     # <3>
    "cover_colors": ["red", "magenta", "yellow", "turquoise"],           # <4>
}
book_schema.validate(my_metadata)                                        # <5>
```

1. A required field without a default
2. A repeatable composite field
3. A repeatable date
4. A field with multiple possible values, not all of them valid
5. Required fields with default are filled in.

## Error messages

```{python}
#| error: true
#| output-location: fragment
book_schema.validate(
    {
        "title": "Some title",
        "author": {"name": "Jane Doe", "email": "sweetdoe@email.eu"},
        "publishing_date": date.today(),
    }
)
```

```{python}
# | error: true
#| output-location: fragment
book_schema.validate(
    {
        "title": "Some title",
        "author": {"name": "Jane Doe", "email": "jane.doe@kuleuven.be"},
        "publishing_date": "01/01/1990",
    }
)
```

## Warnings

```{python}
# | message: true
# | warning: true
#| output-location: fragment
import logging

logger = logging.getLogger("mango_mdschema")
logger.setLevel(logging.INFO)

book_schema.validate(my_metadata)
```

## Write: from dictionaries to namespacing

```{python}
irods_object = session.collections.get(home_dir).data_objects[0]
irods_object.metadata.items()
```

```{python}
#| output-location: fragment
avus = book_schema.to_avus(my_metadata)
avus
```

## Write: from dictionaries to namespacing

```{python}
book_schema.apply(irods_object, my_metadata)
irods_object.metadata.items()
```

## Read: from AVUs back to dictionaries

```{python}
#book_schema.from_avus(avus)
book_schema.extract(irods_object)
```

# Conclusion

## Metadata schemas with Python

:::: {.columns}

::: {.column width="50%"}
### Metadata schemas

- Format validation
- Required fields and default values
- Hierarchical structure

:::

::: {.column width="50%"}
### Python

- Processing data in badges
- Reading metadata from files
- E.g. metadata with data ingestion

:::

::::

::: callout-tip
You don't need ManGO, these are also standalone applications!
:::


## `mango-mdschema`

- Offers validation, writing and reading of structured metadata

- Schemas are described in JSON, can be designed in the manager

- Metadata can be hierarchical, rendered with namespacing

- Input can be automatized, output can be parsed and rendered in the portal

# Thank you! {background-color="#DCE7F0"}

[github.com/kuleuven/mango-mdschema](https://github.com/kuleuven/mango-mdschema)
[github.com/kuleuven/mango-metadata-schemas](https://github.com/kuleuven/mango-metadata-schemas)

::: footer
:::


```{python}
#| include: false
irods_object.metadata.remove_all()
session.cleanup()
```