DigiPres Publications Index v2.0

[anjackson]

Leading on from #2

Proposed features

• Zenodo/Zotero/Google Sheet as faceted sources.

Ideas

• Shift to a widely-used data model for collection and aggregation #6
• Keywords analysis and options #7
• Permalinks for the iPRES Index? publications#8 and for search!
• iPRES 2023 example presentation without a paper publications#21
• Can't search for exact matches publications#22
• Match authors across papers with different forms of name publications#24
• Fix use of license instead of licence in publication model (see FIXME)
• iPRES:
  • Would be nice to have more detail on conferences, e.g. keynote speakers and awards.
  • Link publications DB to digipres.org as well as direct to source repos? Because of metadata support?
  • Need to work with iPRES Steering Group to work out best way forward.
  • Could become the 'master' records, allowing augmentation (as proposed by Micky) e.g. adding Author's and distinct entities with ORCIDs etc. Clean up data, especially keywords, etc. But only if iPRES and the source repository hosts are okay with it.
  • Check the data, e.g. Kieran points out that Preservica have presented on SharePoint at bake-off and lightning talks, so it seems the search may be missing something.
  • Keywords are an interesting case. Align with CHARM? Follow the example of the Taxonomy of Digital Research Activities in the Humanities? Literary Warrant, etc.
• We have https://github.com/digipres/policies - this could also be integrated into the Practices index as a new table.
• Tighten up the SQL schema, using this.
• Integrate the policies documents as a separate table in the database?
• Also download full-text and index that. See Support grabbing web resources e.g. blogs #8
• Process PDFs with PyMuPDF (AGPL) for simple text or, for more fancy data...
• Process PDFs with GROBID, and use that to get more metadata. Maybe match up citations. Possibly compare PDF link extraction tools? See also https://github.com/unt-libraries/etd-to-urls
• The Datasette version can make links to search results. The current website version cannot. It would be good to have a search results page in the HTML version.
• Improve Datasette-Lite for this use case:
  • e.g. set truncate_cells_html suggested by https://digipres.club/@raffaele/112218790913192333
  • Go through Tim Sherratt's work to look for good ideas.
  • If possible, contribute any modifications back to the core Datasette Lite project.
  • Use this and this to re-order columns so the default Datasette view is more useful.
• Other sources:
  • Add Zenodo support and pull in the Digital Preservation community #3
  • Add Zotero support and pull in Ethan's emulation bibliography - https://www.zotero.org/ethan-gates/collections/ERZIYJ3T
  • https://pasig.figshare.com/pasigcdmx
  • From our history: PADI Digital Preservation Resources
  • Add specialist blogs as a facet of the DPPI #9

From Micky:

• add country based on institution cf https://zenodo.org/records/1462500
• institution types?
• generate author networks?

Are there some parts where we do need community editing workflows to manage some aggregation data? Like the iPRES conference metadata? Are there tools for supporting analysis and visualisation? See digipres/registries-of-practice-project#16

[anjackson]

That paper at iPRES, applying https://maartengr.github.io/BERTopic/ to a different corpus of digital preservation papers, seemed to mirror what I'd found with spacy. You don't get much that makes sense when you've only got metadata to work with. I suspect this is generally true that domains with terms of art and difficult to integrate with generic language tools, at least without a decently large corpus. Perhaps this needs the full-text to be in place?