Skip to content

Commit

Permalink
{Documentation} Update ADT docs for object model release
Browse files Browse the repository at this point in the history
Summary:
This diff applies most of the changes we want to support the newly released ADT object models. Here is a list of all changes included in this diff:

* Update main ADT landing page to include information on object models
* Update sequences count in two locations: main ADT landing page, and dataset download page. This is because when we updated to v2.0, we broke subsequences into their own sequence which then increased the sequence count.
* Add object models page with instructions for downloading and viewing models
* Note that DinoToy model was not released
* Update link to paper to CVPR link, not ArXiv

Reviewed By: xiaqingp

Differential Revision: D63294727

fbshipit-source-id: ae4d4e428153e55cf827a895dfc40e79a3014528
  • Loading branch information
nickcharron authored and facebook-github-bot committed Sep 25, 2024
1 parent 6beb665 commit 719e38a
Show file tree
Hide file tree
Showing 5 changed files with 145 additions and 20 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -6,24 +6,41 @@ title: Overview

## Overview

Project Aria Tools provides Python and C++ APIs to access the [Aria Digital Twin (ADT) dataset](https://www.projectaria.com/datasets/adt/) ([paper](https://arxiv.org/abs/2306.06362)).
Project Aria Tools provides Python and C++ APIs to access the [Aria Digital Twin (ADT) dataset](https://www.projectaria.com/datasets/adt/)
([paper](https://openaccess.thecvf.com/content/ICCV2023/html/Pan_Aria_Digital_Twin_A_New_Benchmark_Dataset_for_Egocentric_3D_ICCV_2023_paper.html)).

## About the data
ADT provides raw and synthesized sensor data from Project Aria glasses, combined with groundtruth data generated using a motion capture system including depth images, device trajectories, object trajectories and bounding boxes, and human tracking. We also provide processed sensor data from our [Machine Perception Services](/ARK/mps/mps.mdx). Go to [ADT Data Format](data_format.mdx) to see a full list of the data we provide.
ADT provides raw and synthesized sensor data from Project Aria glasses, combined with groundtruth data generated using a motion capture
system including depth images, device trajectories, object trajectories and bounding boxes, and human tracking.
We also provide processed sensor data from our [Machine Perception Services](/ARK/mps/mps.mdx).
Go to [ADT Data Format](data_format.mdx) to see a full list of the data we provide.

The ADT dataset contains 222 sequences recording single and dual-person activities. The data was recorded in two spaces: an apartment and a single room office. There are 74 single-instance dynamic objects shared between the two spaces.
The ADT dataset contains 236 sequences recording single and dual-person activities.
The data was recorded in two spaces: an apartment and a single room office.
There are 74 single-instance dynamic objects shared between the two spaces.

Go to the [Getting Started Tutorial](getting_started.mdx) to explore the sample dataset (available on Google colab, no download necessary) or the [Dataset Download page](dataset_download.mdx) to get started.
Go to the [Getting Started Tutorial](getting_started.mdx) to explore the sample dataset (available on Google colab, no download necessary)
or the [Dataset Download page](dataset_download.mdx) to get started.

The sample dataset is a single-user dataset with body pose in the Apartment. It is a pretty representative example that should give you an idea of the dataset.
The sample dataset is a single-user dataset with body pose in the Apartment.
It is a pretty representative example that should give you an idea of the dataset.

In addition to the sensor data and annotated ground truth data, we also provide high quality 3D object models (in .glb) for each of the objects in ADT.
For more info on downloading and viewing these models, see [Object Model](object_models.mdx) page.

:::info NEW UPDATE!
3D object model release is new as of September 2024
:::

### Apartment scene
170 sequences were recorded in the apartment scene. The apartment comprised of a living room, kitchen, dining room and bedroom and contained 281 unique stationary objects.
284 sequences were recorded in the apartment scene. The apartment comprised of a living room, kitchen, dining room and bedroom and contained
281 unique stationary objects.

Given some objects have multiple instances that may differ slightly, the apartment has 324 stationary object instances in total.

### Office scene
52 sequences were recorded in the office scene, a single room with minimal office furniture. The office room contained 15 unique stationary objects and 20 stationary object instances.
52 sequences were recorded in the office scene, a single room with minimal office furniture.
The office room contained 15 unique stationary objects and 20 stationary object instances.


### Activities
Expand All @@ -46,13 +63,6 @@ The dual-person activities included:
Every activity has 10 to 50 sequences and the activity names are embedded into the sequence_names.


### Other statistics:

* Number of multi-person sequences: 77
* Number of sequences with no skeletons: 110
* Number of sequences with 1 skeleton: 92
* Number of sequences with 2 skeleton: 20

:::info
We provide a mix of datasets where users may or may not be wearing an Aria and/or a bodysuit.
Please refer to the `skeleton_aria_association.json` to see the case for each specific sub-sequence.
Expand All @@ -64,7 +74,9 @@ The ADT section of the wiki covers:
* [Getting Started](getting_started.mdx)
* A quickstart tutorial available as a Google colab project or install Project Aria Tools Python package to run locally, run the ADT notebook to access and visualize ground-truth data.
* [Dataset Download](dataset_download.mdx)
* A walkthrough of using `adt_benchmark_dataset_downloader` to download the published ADT dataset.
* A walkthrough of using `aria_dataset_downloader` to download the ADT dataset.
* [Object Models](object_models.mdx)
* Everything you need to know about the released ADT object models including: a walkthrough of using `dtc_object_downloader` to download the ADT object models, and brief description of data content and how to visualize the models
* [Data Format](data_format.mdx)
* How ADT data is organized and stored
* [Data Loader](data_loader.mdx)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ title: Dataset Download
## Overview

This page covers how to download the sample Aria Digital Twin (ADT) sequence as well as how to download specific sequences and types of data. Follow the instructions to download the sample datasets and from there you'll be able to use the CLI to download more data.
For information on how to download ADT object models, see [Object Models](object_models.mdx) page

The sample dataset is a single-user dataset with body pose in the Apartment. This is a pretty representative example that should give you an idea of the dataset.

Expand Down Expand Up @@ -39,7 +40,7 @@ Scroll down to the bottom of the page. Enter your email and select **Access the

Once you've selected **Access the Datasets** you'll be taken back to the top of the ADT page.

Scroll down the page to select **Aria Digital Twin Download Links** and download the file to the folder $HOME/Downloads.
Scroll down the page to select **ADT Sequences** link and download the file to the folder $HOME/Downloads.

<div style={{textAlign: 'center'}}>

Expand All @@ -56,8 +57,6 @@ You can redownload the download links whenever they expire

```bash
mkdir -p $HOME/Documents/projectaria_tools_adt_data

mv $HOME/Downloads/aria_digital_twin_dataset_download_urls.json $HOME/Documents/projectaria_tools_adt_data/
```

### Step 3 : Download the sample sequence (~500MB) via CLI:
Expand All @@ -77,7 +76,7 @@ For more information on the content in the other sequences, see the Data Content
## Download the Aria Digital Twin (ADT) benchmark dataset
### Data size

The Aria Digital Twin dataset consists of 217 sequences. It is split into 4 data types that can be downloaded individually, plus MPS data. Go to [Project Aria Machine Perception Services](/ARK/mps/mps.mdx) for more information about MPS data. The MPS data is also broken into chunks that can be included or excluded at download time. The size of each data type is shown below. ADT data without MPS is approximately 3.5TB.
The Aria Digital Twin dataset consists of 236 sequences. It is split into 4 data types that can be downloaded individually, plus MPS data. Go to [Project Aria Machine Perception Services](/ARK/mps/mps.mdx) for more information about MPS data. The MPS data is also broken into chunks that can be included or excluded at download time. The size of each data type is shown below. ADT data without MPS is approximately 3.5TB.

| **Data type** | What's included | Per sequence size | Total size for all sequences |
| --- | --- | --- | --- |
Expand All @@ -91,7 +90,6 @@ The Aria Digital Twin dataset consists of 217 sequences. It is split into 4 data
| MPS SLAM online calibration | Online calibrations | < 20 MB | 1.2 GB |



### Download via CLI

ADT uses the general Aria dataset downloader, which is available in the [`projectaria_tools` PyPI (pip install) package](/data_utilities/installation/installation_python.mdx).
Expand Down
115 changes: 115 additions & 0 deletions website/docs/open_datasets/aria_digital_twin_dataset/object_models.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
---
sidebar_position: 35
title: Object Models
---
# ADT Object Models

## Overview

This page covers how to download the 3D object models associated with the Aria Digital Twin (ADT) dataset,
and briefly describe the data content and how to visualize the models.

By downloading the object models as described in this page, you agree that you have read and accepted the
terms of the [Aria Digital Twin Object Models License Agreement](https://www.projectaria.com/datasets/adt/license/).

## Download the ADT Object Models

### Step 0: install project_aria_tools package and create venv if not done before

Follow Step 0 to Step 3 in [Getting Started](/docs/open_datasets/aria_digital_twin_dataset/getting_started).

### Step 1 : Signup and get the download links file

To use the downloader CLI, you need to download a file which contains all data URLs plus some metadata for the downloader script.
We currently offer two ways of getting this file:

Visit [ADT website](https://www.projectaria.com/datasets/adt/) and sign up.

Scroll down to the bottom of the page. Enter your email and select **Access the Datasets**.

![Screenshot from website showing download dataset](/img/open_datasets/aria_digital_twin_dataset/download_page.png)

Once you've selected **Access the Datasets** you'll be taken back to the top of the ADT page.

Scroll down the page to select **ADT Object 3D Models** link and download the file to the folder $HOME/Downloads.

<div style={{textAlign: 'center'}}>

![ADT Website Signup Image](/img/open_datasets/aria_digital_twin_dataset/download_button.png)

</div>

:::info The download-links file will expire in 14 days
You can redownload the download links whenever they expire
:::


### Step 2 : Set up a folder for object models

```bash
mkdir -p $HOME/Documents/adt_object_library
```

### Step 3 : Download the object models via CLI:

To download all ADT models (~5.5 GB), run the following command from your Python virtual environment:

```bash
dtc_object_downloader -r ADT -c {PATH_TO_YOUR_CDN_FILE} -o $HOME/Documents/adt_object_library
```

To download only a select subset of ADT models, run the following command from your Python virtual environment:

```bash
dtc_object_downloader -r ADT -c {PATH_TO_YOUR_CDN_FILE} -o $HOME/Documents/adt_object_library -l {SELECTED_OBJECT_NAMES}
```

You can retrieve object names from the ADT data providers using your downloaded sequence data, or have a look at the instances.json file in your sequence data.

:::warning
For legal reasons, we have **removed** the object **DinoToy** from the available models, however, this object is still present in the ADT sequence data.
:::

### Troubleshooting

Check [Project Aria Tools troubleshooting](/docs/data_utilities/installation/troubleshooting) if you are having issues using this guide.

## Object Model Data Format

When downloading the object models using dtc_object_downloader, each object will download into its own folder, named using the instance name of the object.
Under each object folder, there will be three files as shown below.

```
|InstanceName1|
├──3d-asset.glb # model file
├──CC_BY-SA.txt # license file
├──metadata.json
|InstanceName2|
{...}
```

All poses associated with the objects are stored in each sequence ground truth data, not in the downloaded object data.
For example code to learn how to load object models and their associated poses (including static and dynamic objects),
refer to the [ADT Rerun Viewer](https://github.com/facebookresearch/projectaria_tools/blob/main/projectaria_tools/utils/viewer_projects_adt.py)

## Visualizing Object Models

The GLB files for each object model can be viewed using any off-the-shelf GLB viewer.
For example, the open web browser based [Babylon viewer](https://sandbox.babylonjs.com/) can be used by dragging and dropping the GLB file.

To visualize multiple object models, along with their ground truth poses, the ADT rerun viewer can be used.
To run the viewer, enter the following command from your virtual environment with projectaria_tools installed:


```
viewer_projects_adt --sequence_path {PATH_TO_YOUR_SEQUENCE} --object_library_path $HOME/Documents/adt_object_library --display_objects {INSTANCE_NAMES}
```
![ADT Model Viewer Screenshot](/img/open_datasets/aria_digital_twin_dataset/rerun-adt-models.png)

Example command to produce the above visualization:

```
viewer_projects_adt --sequence_path $HOME/Documents/adt_sequence_data/Apartment_release_golden_skeleton_seq100_M1292 \
--object_library_path $HOME/Documents/adt_object_library \
--display_objects BlackCoffeeTable BirdHouseToy YellowArmChair GreySofa RedArmChair
```
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 719e38a

Please sign in to comment.