README.md

Lambda Diffusers

Additional models and pipelines for 🤗 Diffusers created by Lambda Labs

• Stable Diffusion Image Variations
• Pokemon text to image

Installation

git clone https://github.com/LambdaLabsML/lambda-diffusers.git
cd lambda-diffusers
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Stable Diffusion Image Variations

A fine-tuned version of Stable Diffusion conditioned on CLIP image embeddings to enabel Image Variations.

• Download the weights ported to 🤗 Diffusers here.
• See the original training repo here.

Usage

from pathlib import Path
from lambda_diffusers import StableDiffusionImageEmbedPipeline
from PIL import Image
import torch
device = "cuda" if torch.cuda.is_available() else "cpu"
pipe = StableDiffusionImageEmbedPipeline.from_pretrained("lambdalabs/sd-image-variations-diffusers")
pipe = pipe.to(device)
im = Image.open("your/input/image/here.jpg")
num_samples = 4
image = pipe(num_samples*[im], guidance_scale=3.0)
image = image["sample"]
base_path = Path("outputs/im2im")
base_path.mkdir(exist_ok=True, parents=True)
for idx, im in enumerate(image):
    im.save(base_path/f"{idx:06}.jpg")

Pokemon text to image

Stable Diffusion fine tuned on Pokémon by Lambda Labs.

Put in a text prompt and generate your own Pokémon character, no "prompt engineering" required!

If you want to find out how to train your own Stable Diffusion variants, see this example from Lambda Labs.

Girl with a pearl earring, Cute Obama creature, Donald Trump, Boris Johnson, Totoro, Hello Kitty

Model description

Trained on BLIP captioned Pokémon images using 2xA6000 GPUs on Lambda GPU Cloud for around 15,000 step (about 6 hours, at a cost of about $10).

Usage

import torch
from diffusers import StableDiffusionPipeline
from torch import autocast

pipe = StableDiffusionPipeline.from_pretrained("lambdalabs/sd-pokemon-diffusers", torch_dtype=torch.float16)  
pipe = pipe.to("cuda")

prompt = "Yoda"
scale = 10
n_samples = 4

# Sometimes the nsfw checker is confused by the Pokémon images, you can disable
# it at your own risk here
disable_safety = False

if disable_safety:
  def null_safety(images, **kwargs):
      return images, False
  pipe.safety_checker = null_safety

with autocast("cuda"):
  images = pipe(n_samples*[prompt], guidance_scale=scale).images

for idx, im in enumerate(images):
  im.save(f"{idx:06}.png")

Benchmarking inference

Detailed benchmark documentation can be found here.

Setup

Before running the benchmark, make sure you have completed the repository installation steps.

You will then need to set the huggingface access token:

1. Create a user account on HuggingFace and generate an access token.
2. Set your huggingface access token as the ACCESS_TOKEN environment variable:

export ACCESS_TOKEN=<hf_...>

Usage

Launch the benchmark script to append benchmark results to the existing benchmark.csv results file:

python ./scripts/benchmark.py

Results

Links

• Captioned Pokémon dataset
• Model weights in Diffusers format
• Original model weights
• Training code

Trained by Justin Pinkney (@Buntworthy) at Lambda Labs.