Multi Target container discussion

[VoVAllen]

Ref issue: #403

One of envd's targets is to support most container-related requirements related to data science, from development to production, from data processing stage to final serving. Therefore, users may want to build multiple images from a single repo.

Different requirements between stages:

• data processing
  • Scenario: From SQL to Tensor
• Develop model
  • Requirement: VSCode, ZSH, SSH, CUDA-DEVEL
  • Quickly changed the image
• Training job
  • Requirement: Experiment tracking/monitoring, observability, CUDA-RUNTIME
  • Rarely changed the image
• Evaluation
  • In real-world data science, evaluation stage can be fully separate from training, due to complex downstream systems
• Serving
  • Requirement: Serving observability, CUDA-RUNTIME

Possible Scenarios:

• Train on small set of data
  • Researcher usually tries to train on a small subset of data to verify code correctness
  • python train.py --small
• Train on full dataset
  • Researcher will train on the full dataset to get the result
  • python train.py
  • Upload artifact
  • Model/Data versioning
• Hyperparameter tuning
  • launch multiple python trian.py --lr xxx
• Evaluation
  • Get Artifact from training stage
  • May need to setup a cluster
  • python train.py --eval
• Unit test for models
  • Simple unit test to test correctness
  • Not so common in data science, got idea from Earthly
• Compress Model
  • Compress model by quantization
• Serving
  • python serve.py

Possible syntax (from #403):

version = "2.9.1"

def dev_env(version):
    # Use nvidia/cuda:devel image
    base(runtime="devel")
    install.python_packages(name=[
        "tensorflow=="+version,
        "tensorboard"
    ])
    config.jupyter(password="", port=8888)
    shell("zsh")
    config.vscode(extension=[
        "ms-python.python"
    ])

def serving_env(version):
    base(runtime="runtime")
    install.python_packages(name=[
        "tensorflow=="+version,
    ])

def build():
    cuda(version="11.6", cudnn="8")
    dev_env(version)

def serve():
    serving_env(version)
    cuda(version="11.6", cudnn="8")
    entrypoint("python server.py")