Skip to content

Latest commit

 

History

History
55 lines (42 loc) · 2.9 KB

README.md

File metadata and controls

55 lines (42 loc) · 2.9 KB
Raw

Data Management Toolkit

This module manages data collection and storage in this project, it has two submodules:

  • IO Watcher and Down-sampler (watcher): this program watch local disk for new experiment generated data, which we call them experiment log. For every new log entry, the program upload the point cloud to a pre-defined object storage service. It also send metadata of experiment to backend
  • Service Backend: the backend receives metadata from watchers. It stores the metadata in a MongoDB database. The metadata is also fetched from backend module in training.

Note: This toolkit is adapted from UniFolding.

Get Started

The datamanagement service is managed by Docker Compose. It is recommended to prepare a dedicated host for Serivece Backend Module and Database and high performance object storage service. You should configure the IO Watcher module on the machine where the Unifolding Controller is deployed. The IO Wathcer module should be able to reach out to Service Backend via network faster than 100Mbps (1Gbps connection is recommended).

We have prepared an example for you. This example uses MinIO to provide object storage. Start the backend by:

cd manifests/docker/backend
docker compose up -d

This will start the MinIO object storage, the MongoDB database and the Service Backend module.

Then, you should configure the watchers to monitor your log directory. Modify the manifests/docker/watcher/docker-compose.yaml so that the path of logs generated by Unifolding Controller is mapped to container

# generate a docker-compose file that runs rabbitmq, mongodb, minio, with recommended settings. username should be "unifolding" and password should be "unifolding"
version: '3.7'
services:
  watcher: # pick a name
    image: davidliyutong/unifolding-watcher:latest
    restart: always
    environment:
      - UNIFOLDING_API_URL=http://backend:8080, # shoudl point to the backend
      - UNIFOLDING_APP_WATCHPATH=/container_data # modify this path, match the volume mapping configuration
      - UNIFOLDING_OSS_HOST=minio # modify this oss hostname
      - UNIFOLDING_OSS_PORT=9000 # modify this oss port
      - UNIFOLDING_OSS_BUCKET=unifolding-test # modify this bucket name
      - UNIFOLDING_OSS_ACCESSKEY=unifolding-admin # modify this access_key
      - UNIFOLDING_OSS_SECRETKEY=unifolidng-secret # modify this secret_key
    volumes:
      - ./host_data:/container_data # map the correct data to $UNIFOLDING_APP_WATCHPATH

networks:
  default:
    external: true
    name: unifolding

You can configure an arbitrary number of watcher on arbitrary number of machines. This toolkit is suitable for large scale robot farms. Once configured, use this command to start watchers.

cd manifests/docker/backend
docker compose up -d

The watcher will scan the directory and upload new logs. The uploaded logs are moved to archives sub-directory.