Merge branch 'main' into movetutorials

.github/workflows/build.yml

@@ -53,3 +53,12 @@ jobs:
           accountId: 472bdad691b4483dea759eadb37110bd
           command: pages deploy "_build/html" --project-name=lamin-docs
           gitHubToken: ${{ secrets.GITHUB_TOKEN }}
+      - uses: edumserrano/find-create-or-update-comment@v2
+        if: github.event_name == 'pull_request'
+        with:
+          issue-number: ${{ github.event.pull_request.number }}
+          body-includes: "Deployment URL"
+          comment-author: "github-actions[bot]"
+          body: |
+            Deployment URL: ${{ steps.cloudflare.outputs.deployment-url }}
+          edit-mode: replace

docs/changelog/soon/bionty.md

@@ -1,12 +1,2 @@
-- 🎨 Squash migrations [PR](https://github.com/laminlabs/bionty/pull/162) [@sunnyosun](https://github.com/sunnyosun)
-- 🎨 Use field defaults [PR](https://github.com/laminlabs/bionty/pull/158) [@sunnyosun](https://github.com/sunnyosun)
-- ✅ Clean up test properly [PR](https://github.com/laminlabs/bionty/pull/161) [@Zethson](https://github.com/Zethson)
-- Fix jobs run on fork CI [PR](https://github.com/laminlabs/bionty/pull/159) [@Zethson](https://github.com/Zethson)
-- ✨ Add fork protected CI [PR](https://github.com/laminlabs/bionty/pull/157) [@Zethson](https://github.com/Zethson)
-- 📝 Add symbol mapping FAQ reference to Gene  [PR](https://github.com/laminlabs/bionty/pull/149) [@Zethson](https://github.com/Zethson)
-- 🎨 Rename import_from_source [PR](https://github.com/laminlabs/bionty/pull/150) [@Zethson](https://github.com/Zethson)
-- 📝 Fix keep docstring standardize gene [PR](https://github.com/laminlabs/bionty/pull/148) [@Zethson](https://github.com/Zethson)
-- ✨ Add field support for bionty base [PR](https://github.com/laminlabs/bionty/pull/146) [@Zethson](https://github.com/Zethson)
-- 🎨 Raise `OrganismNotSet` instead of `AssertionError` when `organism` is not set [PR](https://github.com/laminlabs/bionty/pull/144) [@Zethson](https://github.com/Zethson)
-- 🎨 Proper deprecation warning [PR](https://github.com/laminlabs/bionty/pull/138) [@Zethson](https://github.com/Zethson)
-- 🐛 Hide from_public from TOC [PR](https://github.com/laminlabs/bionty/pull/135) [@Zethson](https://github.com/Zethson)
+- 🎨 Simplify coupling to `lamindb_setup` [PR](https://github.com/laminlabs/bionty/pull/171) [@falexwolf](https://github.com/falexwolf)
+- 🎨 Make search consistent with the lamindb implementation [PR](https://github.com/laminlabs/bionty/pull/170) [@Koncopd](https://github.com/Koncopd)

docs/changelog/soon/lamindb.md

@@ -1,2 +1,7 @@
-- ✨ Enable Django field validation and Literal validation [PR](https://github.com/laminlabs/lamindb/pull/2177) [@sunnyosun](https://github.com/sunnyosun)
-- ⚡️ Additional search rule for truncated searches [PR](https://github.com/laminlabs/lamindb/pull/2183) [@Koncopd](https://github.com/Koncopd)
+- 📝 Move `tutorial` and `tutorial2` to `lamin-docs` [PR](https://github.com/laminlabs/lamindb/pull/2184) [@falexwolf](https://github.com/falexwolf)
+- 💚 Fix tiledbsoma installation in CI [PR](https://github.com/laminlabs/lamindb/pull/2208) [@Koncopd](https://github.com/Koncopd)
+- ♻️ Simplify coupling of `bionty` and `lamindb-setup` [PR](https://github.com/laminlabs/lamindb/pull/2207) [@falexwolf](https://github.com/falexwolf)
+- ♻️ Refactor setup bionty sources [PR](https://github.com/laminlabs/lamindb-setup/pull/906) [@falexwolf](https://github.com/falexwolf)
+- 🍱 Add `.rds` as a valid simple path suffix [PR](https://github.com/laminlabs/lamindb-setup/pull/905) [@falexwolf](https://github.com/falexwolf)
+- 🍱 Add `.qs` file as a valid simple suffix [PR](https://github.com/laminlabs/lamindb-setup/pull/904) [@falexwolf](https://github.com/falexwolf)
+- 🎨 Make search in bionty consistent with the lamindb implementation [PR](https://github.com/laminlabs/lamindb/pull/2205) [@Koncopd](https://github.com/Koncopd)

docs/changelog/soon/laminr.md

@@ -1,34 +1,2 @@
-- add badge [PR](https://github.com/laminlabs/laminr/pull/82) [@rcannood](https://github.com/rcannood)
-- Add more file handlers [PR](https://github.com/laminlabs/laminr/pull/81) [@rcannood](https://github.com/rcannood)
-- update desc [PR](https://github.com/laminlabs/laminr/pull/77) [@rcannood](https://github.com/rcannood)
-- update installation instructions [PR](https://github.com/laminlabs/laminr/pull/74) [@rcannood](https://github.com/rcannood)
-- Modify Registry$print() to show fields by module [PR](https://github.com/laminlabs/laminr/pull/71) [@lazappi](https://github.com/lazappi)
-- add lifecycle badge [PR](https://github.com/laminlabs/laminr/pull/73) [@rcannood](https://github.com/rcannood)
-- Adjust print colours [PR](https://github.com/laminlabs/laminr/pull/69) [@lazappi](https://github.com/lazappi)
-- 🎨 Reword laminr docs [PR](https://github.com/laminlabs/laminr/pull/72) [@Zethson](https://github.com/Zethson)
-- Improve API error messages [PR](https://github.com/laminlabs/laminr/pull/70) [@lazappi](https://github.com/lazappi)
-- Add Records container class [PR](https://github.com/laminlabs/laminr/pull/59) [@lazappi](https://github.com/lazappi)
-- Refactor documentation [PR](https://github.com/laminlabs/laminr/pull/67) [@rcannood](https://github.com/rcannood)
-- Fix macOS runner [PR](https://github.com/laminlabs/laminr/pull/66) [@rcannood](https://github.com/rcannood)
-- Add check_requires() function and update installation docs [PR](https://github.com/laminlabs/laminr/pull/56) [@lazappi](https://github.com/lazappi)
-- Remove link tables from print output [PR](https://github.com/laminlabs/laminr/pull/55) [@lazappi](https://github.com/lazappi)
-- Add progress bar [PR](https://github.com/laminlabs/laminr/pull/58) [@lazappi](https://github.com/lazappi)
-- Add get_records() to API and df() to Registry [PR](https://github.com/laminlabs/laminr/pull/54) [@lazappi](https://github.com/lazappi)
-- update readme [PR](https://github.com/laminlabs/laminr/pull/51) [@rcannood](https://github.com/rcannood)
-- simplify lamin_cli installation instructions [PR](https://github.com/laminlabs/laminr/pull/33) [@rcannood](https://github.com/rcannood)
-- update version, add luke as an author [PR](https://github.com/laminlabs/laminr/pull/39) [@rcannood](https://github.com/rcannood)
-- Document classes [PR](https://github.com/laminlabs/laminr/pull/36) [@rcannood](https://github.com/rcannood)
-- Make InstanceAPI print methods public [PR](https://github.com/laminlabs/laminr/pull/37) [@lazappi](https://github.com/lazappi)
-- Add printing to remaining classes [PR](https://github.com/laminlabs/laminr/pull/31) [@lazappi](https://github.com/lazappi)
-- Improve error messages [PR](https://github.com/laminlabs/laminr/pull/30) [@rcannood](https://github.com/rcannood)
-- Update documentation [PR](https://github.com/laminlabs/laminr/pull/29) [@rcannood](https://github.com/rcannood)
-- Return `NULL` when a record's related field is empty [PR](https://github.com/laminlabs/laminr/pull/28) [@rcannood](https://github.com/rcannood)
-- minor fix in usage vignette [PR](https://github.com/laminlabs/laminr/pull/32) [@rcannood](https://github.com/rcannood)
-- Add usage vignette [PR](https://github.com/laminlabs/laminr/pull/18) [@rcannood](https://github.com/rcannood)
-- Add a simple unit test which queries laminlabs/lamindata [PR](https://github.com/laminlabs/laminr/pull/27) [@rcannood](https://github.com/rcannood)
-- bump action from v4.5.0 to v4 [PR](https://github.com/laminlabs/laminr/pull/26) [@rcannood](https://github.com/rcannood)
-- Add `Record` print methods [PR](https://github.com/laminlabs/laminr/pull/22) [@lazappi](https://github.com/lazappi)
-- Log into Lamin in CI [PR](https://github.com/laminlabs/laminr/pull/23) [@rcannood](https://github.com/rcannood)
-- Allow unauthenticated users to connect to an instance [PR](https://github.com/laminlabs/laminr/pull/19) [@rcannood](https://github.com/rcannood)
-- Create pkgdown docs [PR](https://github.com/laminlabs/laminr/pull/13) [@rcannood](https://github.com/rcannood)
-- Auto-generate changelog at `docs.lamin.ai/changelog` [PR](https://github.com/laminlabs/laminr/pull/15) [@falexwolf](https://github.com/falexwolf)
+- Cosmetic fixes to `get-started` [PR](https://github.com/laminlabs/laminr/pull/102) [@falexwolf](https://github.com/falexwolf)
+- Polish the get-started guide [PR](https://github.com/laminlabs/laminr/pull/97) [@falexwolf](https://github.com/falexwolf)

docs/changelog/soon/usecases.md

@@ -1,19 +1,9 @@
-- 🎨 Use Field defaults [PR](https://github.com/laminlabs/findrefs/pull/2) [@sunnyosun](https://github.com/sunnyosun)
-- 🎨 Set `blank=True` [PR](https://github.com/laminlabs/ourprojects/pull/2) [@sunnyosun](https://github.com/sunnyosun)
-- ✨ Add `blank=True` [PR](https://github.com/laminlabs/clinicore/pull/14) [@sunnyosun](https://github.com/sunnyosun)
-- 🎨 Add blank=True [PR](https://github.com/laminlabs/cellregistry/pull/2) [@sunnyosun](https://github.com/sunnyosun)
-- 🎨 Squash migrations [PR](https://github.com/laminlabs/wetlab/pull/75) [@sunnyosun](https://github.com/sunnyosun)
-- ✨ Add `blank=True` [PR](https://github.com/laminlabs/wetlab/pull/74) [@sunnyosun](https://github.com/sunnyosun)
-- 💚 Lower bound on tiledbsoma in by_datatype [PR](https://github.com/laminlabs/lamin-usecases/pull/169) [@Koncopd](https://github.com/Koncopd)
-- 🎨 Add missing validate call [PR](https://github.com/laminlabs/lamin-usecases/pull/168) [@Zethson](https://github.com/Zethson)
-- 🐛Fix docstring [PR](https://github.com/laminlabs/ourprojects/pull/1) [@Zethson](https://github.com/Zethson)
-- ✨ More flexible Curator & 🐛 support for paths [PR](https://github.com/laminlabs/cellxgene-lamin/pull/104) [@Zethson](https://github.com/Zethson)
-- 📝Polish docstrings [PR](https://github.com/laminlabs/findrefs/pull/1) [@Zethson](https://github.com/Zethson)
-- 🎨 Use from_source instead of from_public [PR](https://github.com/laminlabs/lamin-usecases/pull/167) [@Zethson](https://github.com/Zethson)
-- 💚 Fix CI [PR](https://github.com/laminlabs/lamin-mlops/pull/20) [@Koncopd](https://github.com/Koncopd)
-- 📝 Polish scrna guide, add another cell type feature: cell_type_untrusted [PR](https://github.com/laminlabs/lamin-usecases/pull/166) [@falexwolf](https://github.com/falexwolf)
-- ✨ First draft of `Cell` registry [PR](https://github.com/laminlabs/cellregistry/pull/1) [@falexwolf](https://github.com/falexwolf)
-- 🔥 Remove `Curator.add_validated_from()` [PR](https://github.com/laminlabs/cellxgene-lamin/pull/105) [@falexwolf](https://github.com/falexwolf)
-- 📝 Polish scrna guide, add another cell type feature: `cell_type_untrusted` [PR](https://github.com/laminlabs/lamin-usecases/pull/162) [@falexwolf](https://github.com/falexwolf)
-- 🔥 Remove `Curator.add_validated_from()` [PR](https://github.com/laminlabs/lamin-usecases/pull/164) [@falexwolf](https://github.com/falexwolf)
-- 🎨 Fix Curator.from_mudata [PR](https://github.com/laminlabs/lamin-usecases/pull/165) [@sunnyosun](https://github.com/sunnyosun)
+- ✨ Add Person and Reference models [PR](https://github.com/laminlabs/ourprojects/pull/5) [@Zethson](https://github.com/Zethson)
+- fix typo [PR](https://github.com/laminlabs/cellregistry/pull/4) [@Zethson](https://github.com/Zethson)
+- 🎨 Re-create 1st migration [PR](https://github.com/laminlabs/ourprojects/pull/4) [@sunnyosun](https://github.com/sunnyosun)
+- 🎨 Clean up [PR](https://github.com/laminlabs/ourprojects/pull/3) [@sunnyosun](https://github.com/sunnyosun)
+- ⬆️ Clean up [PR](https://github.com/laminlabs/cellregistry/pull/3) [@sunnyosun](https://github.com/sunnyosun)
+- 🎨 Clean up [PR](https://github.com/laminlabs/clinicore/pull/15) [@sunnyosun](https://github.com/sunnyosun)
+- ⬆️ Update lamindb [PR](https://github.com/laminlabs/findrefs/pull/4) [@sunnyosun](https://github.com/sunnyosun)
+- ⬆️ Update lamindb deps [PR](https://github.com/laminlabs/wetlab/pull/76) [@sunnyosun](https://github.com/sunnyosun)
+- 💚 Fix literal [PR](https://github.com/laminlabs/lamin-usecases/pull/170) [@sunnyosun](https://github.com/sunnyosun)

docs/glossary.md

@@ -2,26 +2,35 @@
 
 ```{glossary}
 
+FAIR
+    FAIR data is data which meets the principles of findability, accessibility, interoperability, and reusability [wikipedia](https://en.wikipedia.org/wiki/FAIR_data).
+
+GUI
+    Graphical user interface, for instance, a browser-based data catalog.
+
 feature
-    A feature is an individual measurable property of a phenomenon [[Wikipedia](https://en.wikipedia.org/wiki/Feature_(machine_learning))], a measured event like a microscopy image or transcriptomic readout of a biological system.
+    A feature is a property of a measurement [[Wikipedia](https://en.wikipedia.org/wiki/Feature_(machine_learning))]. It's equivalent to a {term}`variable` in statistics and is typically equated with a dimension of a dataset.
 
-    It's equivalent to the term "independent {term}`variable`" in statistics, but is the preferred term to denote dimensions of "feature spaces" in machine learning.
+    LaminDB comes with a {class}`~lamindb.Feature` registry to organize dataset dimensions and equates them with statistical variables.
 
 label
     A label refers to a descriptor or tag that is assigned to something to describe, identify, or categorize it.
 
+lakehouse
+    A data lakehouse combines the flexibility and cost-effectiveness of a data lake with the data management and ACID transaction support of a data warehouse, enabling both structured and unstructured data analytics in a single framework. Some of the early lakehouse frameworks were Databrick's [Delta Lake](https://delta.io/), Google's [BigLake](https://cloud.google.com/biglake), Amazon's [Lake Formation](). Later examples include [Dremio](https://www.dremio.com/), [Starburst](https://www.starburst.io/) and others. Here is a [blog post](https://cloud.google.com/blog/products/data-analytics/unify-data-lakes-and-warehouses-with-biglake-now-generally-available) from Google, a [blog post](https://aws.amazon.com/blogs/big-data/build-a-lake-house-architecture-on-aws/) from AWS, a [glossary entry](https://www.databricks.com/glossary/data-lakehouse) and a [paper](https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf) from Databricks.
+
+ORM
+    Object-relational mapper. In LaminDB every sub-class of `Record` (every instance of `Registry`) is an ORM that corresponds to a SQL table in the underlying metadata database [wikipedia](https://en.wikipedia.org/wiki/Object%E2%80%93relational_mapping).
+
 observation
     In statistics (machine learning), an observation refers to a particular measured instance of a set of random variable.
 
     In biology, an observation typically corresponds to measuring (reading out) a set of properties from a biological sample.
 
 record
-    A record is a data structure that consists in [fields](https://en.wikipedia.org/wiki/Field_(computer_science)), typically of different types but in a fixed sequence [[Wikipedia](https://en.wikipedia.org/wiki/Record_(computer_science))].
-
-    Importantly, we refer to instances of [Registry](https://lamin.ai/docs/lamindb.core.registry) as records. Once a record is inserted into a database table, it becomes a row in that table.
-    Every `Registry` class (in LaminDB) has a 1:1 correspondence with a database table and a django [model](https://docs.djangoproject.com/en/4.2/topics/db/models/), every row in a database table has a 1:1 correspondence with a record.
+    A record is a data structure that consists in a sequence of typed [fields](https://en.wikipedia.org/wiki/Field_(computer_science)) that store values [[Wikipedia](https://en.wikipedia.org/wiki/Record_(computer_science))].
 
-    A record often stores jointly measured {term}`variables <variable>` in its fields, but in general allows updating fields when more information becomes available or changes.
+    In LaminDB, a metadata record is modeled as a {class}`~lamindb.Record`.
 
 sample
     In biology, a sample is an instance or part of a biological system.

docs/includes/features-lamindb.md

@@ -1,8 +1,14 @@
-**Manage data & metadata with a unified Python API (“lakehouse”).**
+Any LaminDB instance comes with an underlying SQL metadata database to organize files, folders, and arrays across any number of storage locations.
 
-- Query & search across artifacts ({class}`~lamindb.Artifact`) & metadata records ({class}`~lamindb.core.Record`): {class}`~lamindb.core.Record.filter`, {class}`~lamindb.core.Record.search`
-- Query large array stores: {class}`~lamindb.Artifact.open` → [guide](cellxgene)
-- Cache artifacts on disk & load them into memory: {class}`~lamindb.Artifact.cache`, {class}`~lamindb.Artifact.load`
+The following detailed specs are for the Python package `lamindb`. For the analogous R package `laminr`, see the [R docs](https://laminr.lamin.ai).
+
+**Manage data & metadata with a unified API (“lakehouse”).**
+
+- Query & search across data & metadata: {class}`~lamindb.core.Record.filter`, {class}`~lamindb.core.Record.search`
+- Model important entities as an {term}`ORM` which their own metadata registry: {class}`~lamindb.core.Record`
+- Model files and folders as datasets & models via one class: {class}`~lamindb.Artifact`
+- Slice large array stores: {class}`~lamindb.Artifact.open` → [guide](cellxgene)
+- Cache & load artifacts: {class}`~lamindb.Artifact.cache`, {class}`~lamindb.Artifact.load`
 - Manage features & labels: {class}`~lamindb.Feature`, {class}`~lamindb.FeatureSet`, {class}`~lamindb.ULabel`
 - Plug-in custom [schemas](/schemas) & manage schema migrations
 - Use array formats in memory & storage: [DataFrame](/tutorial), [AnnData](/arrays), [MuData](multimodal), [SOMA](cellxgene), ... backed by [parquet](/tutorial), [zarr](/arrays), [TileDB](cellxgene), [HDF5](/arrays), [h5ad](/arrays), [DuckDB](rxrx), ...

docs/integrations.md

@@ -1,5 +1,18 @@
 # Integrations
 
+## Schema modules
+
+Additional registries can be found here:
+
+- [bionty](bionty): Basic biological entities, coupled to public ontologies
+- [wetlab](wetlab): Basic wetlab entities
+- [clinicore](clinicore): Basic clinical entities
+- [cellregistry](cellregistry): A registry for single cells
+- [ourprojects](ourprojects): Manage projects, teams & references
+- [omop](omop): OMOP Common Data Model
+
+Please [reach out](https://lamin.ai/contact) to have your schema module featured.
+
 ## Git
 
 Sync tracked scripts in lamindb with git commits: [guide](track.ipynb#sync-scripts-with-github)