[feat: Benchmarking Workflow] add stuff for a benchmarking workflow #5839

sayakpaul · 2023-11-17T06:22:45Z

What does this PR do?

Adds utilities running benchmarks as an automated workflow. I think this could lift a lot of manual burden off of our shoulders.

As was discussing with @williamberman.

Curious to know what others think about it. @patrickvonplaten @DN6

Some comments in line.

The final artifact of the workflow would be a dataset like this: https://huggingface.co/datasets/diffusers/benchmarks.

sayakpaul · 2023-11-17T06:23:16Z

.github/workflows/benchmark.yml

+name: Benchmarking tests
+
+on:
+  pull_request:
+    branches:
+      - main
+  push:
+    branches:
+      - ci-*


Should ideally be done weekly or bi-weekly and only on main. For testing, I am running it on PRs.

I think bi-weekly (twice a month?) Works here. Or even monthly is a good cadence.

Yup. Will keep it to bi-weekly. Will incorporate that change once we're one with how we want to report the benchmarks and the pipelines we want to benchmark.

Should change this to a cron job

Yes this will be changed.

Change to cron job.

.github/workflows/benchmark.yml

HuggingFaceDocBuilderDev · 2023-11-17T07:23:27Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

patrickvonplaten · 2023-11-21T10:54:42Z

I like the idea - @DN6 wdyt?

sayakpaul · 2023-11-27T02:07:36Z

@DN6 a gentle ping.

.github/workflows/benchmark.yml

benchmarks/benchmark_sd.py

DN6 · 2023-11-27T14:08:22Z

Left a couple of comments, but overall LGTM 👍🏽 @sayakpaul

* EulerDiscreteScheduler add `rescale_betas_zero_snr`

This reverts commit 821726d.

This reverts commit 3dc2362.

sayakpaul · 2023-12-07T11:09:52Z

@DN6 could you go through PR rigorously? https://huggingface.co/datasets/diffusers/benchmarks is the reported dataset. Have a look at it too :)

Once the workflow successfully completes, I will make the changes necessary for the crown job. Are we sticking to bi-weekly for now?

sayakpaul · 2023-12-09T04:10:46Z

@DN6 one final review please :)

DN6

LGTM 👍🏽

…uggingface#5839) * add poc for benchmarking workflow. * import * fix argument * fix: argument * fix: path * fix * fix * path * output csv files. * workflow cleanup * append token * add utility to push to hf dataset * fix: kw arg * better reporting * fix: headers * better formatting of the numbers. * better type annotation * fix: formatting * moentarily disable check * push results. * remove disable check * introduce base classes. * img2img class * add inpainting pipeline * intoduce base benchmark class. * add img2img and inpainting * feat: utility to compare changes * fix * fix import * add args * basepath * better exception handling * better path handling * fix * fix * remove * ifx * fix * add: support for controlnet. * image_url -> url * move images to huggingface hub * correct urls. * root_ckpt * flush before benchmarking * don't install accelerate from source * add runner * simplify Diffusers Benchmarking step * change runner * fix: subprocess call. * filter percentage values * fix controlnet benchmark * add t2i adapters. * fix filter columns * fix t2i adapter benchmark * fix init. * fix * remove safetensors flag * fix args print * fix * feat: run_command * add adapter resolution mapping * benchmark t2i adapter fix. * fix adapter input * fix * convert to L. * add flush() add appropriate places * better filtering * okay * get env for torch * convert to float * fix * filter out nans. * better coment * sdxl * sdxl for other benchmarks. * fix: condition * fix: condition for inpainting * fix: mapping for resolution * fix * include kandinsky and wuerstchen * fix: Wuerstchen * Empty-Commit * [Community] AnimateDiff + Controlnet Pipeline (huggingface#5928) * begin work on animatediff + controlnet pipeline * complete todos, uncomment multicontrolnet, input checks Co-Authored-By: EdoardoBotta <[email protected]> * update Co-Authored-By: EdoardoBotta <[email protected]> * add example * update community README * Update examples/community/README.md --------- Co-authored-by: EdoardoBotta <[email protected]> Co-authored-by: Patrick von Platen <[email protected]> * EulerDiscreteScheduler add `rescale_betas_zero_snr` (huggingface#6024) * EulerDiscreteScheduler add `rescale_betas_zero_snr` * Revert "[Community] AnimateDiff + Controlnet Pipeline (huggingface#5928)" This reverts commit 821726d. * Revert "EulerDiscreteScheduler add `rescale_betas_zero_snr` (huggingface#6024)" This reverts commit 3dc2362. * add SDXL turbo * add lcm lora to the mix as well. * fix * increase steps to 2 when running turbo i2i * debug * debug * debug * fix for good * fix and isolate better * fuse lora so that torch compile works with peft * fix: LCMLoRA * better identification for LCM * change to cron job --------- Co-authored-by: Patrick von Platen <[email protected]> Co-authored-by: Dhruv Nair <[email protected]> Co-authored-by: Aryan V S <[email protected]> Co-authored-by: EdoardoBotta <[email protected]> Co-authored-by: Beinsezii <[email protected]>

add poc for benchmarking workflow.

4c185f0

sayakpaul commented Nov 17, 2023

View reviewed changes

.github/workflows/benchmark.yml Outdated Show resolved Hide resolved

sayakpaul commented Nov 17, 2023

View reviewed changes

.github/workflows/benchmark.yml Outdated Show resolved Hide resolved

sayakpaul added 4 commits November 17, 2023 12:02

import

945ab17

fix argument

b4debda

fix: argument

22966a1

fix: path

12424a3

sayakpaul added 3 commits November 17, 2023 13:00

fix

122d5d9

fix

c20d254

path

43544ed

Merge branch 'main' into feat/ci-benchmarking

ada65b3

DN6 reviewed Nov 27, 2023

View reviewed changes

.github/workflows/benchmark.yml Show resolved Hide resolved

DN6 reviewed Nov 27, 2023

View reviewed changes

benchmarks/benchmark_sd.py Outdated Show resolved Hide resolved

sayakpaul added 12 commits November 28, 2023 08:32

Merge branch 'main' into feat/ci-benchmarking

f59e315

output csv files.

3c05e41

workflow cleanup

24b68fd

append token

8e2088e

add utility to push to hf dataset

01584c7

fix: kw arg

853035b

better reporting

46aaf96

fix: headers

d626eef

better formatting of the numbers.

ab12fe6

better type annotation

1bb531e

fix: formatting

2df4aba

moentarily disable check

939fe5c

Beinsezii and others added 5 commits December 7, 2023 13:56

EulerDiscreteScheduler add rescale_betas_zero_snr (#6024)

3dc2362

* EulerDiscreteScheduler add `rescale_betas_zero_snr`

Revert "[Community] AnimateDiff + Controlnet Pipeline (#5928)"

26a8c00

This reverts commit 821726d.

Revert "EulerDiscreteScheduler add rescale_betas_zero_snr (#6024)"

8db59d7

This reverts commit 3dc2362.

Merge branch 'main' into feat/ci-benchmarking

f76ba5b

add SDXL turbo

4e7fb4d

sayakpaul marked this pull request as ready for review December 7, 2023 10:52

add lcm lora to the mix as well.

e2df761

sayakpaul added 15 commits December 8, 2023 00:07

fix

2588853

Merge branch 'main' into feat/ci-benchmarking

81d56de

increase steps to 2 when running turbo i2i

a7fd2c3

Merge branch 'main' into feat/ci-benchmarking

191ebf6

debug

b878a29

debug

1389d0e

debug

b2d35be

fix for good

d78609d

fix and isolate better

b3897f8

fuse lora so that torch compile works with peft

8289baa

fix: LCMLoRA

dd54366

Merge branch 'main' into feat/ci-benchmarking

d6966b4

better identification for LCM

51acace

Merge branch 'main' into feat/ci-benchmarking

65b97e8

change to cron job

80e8311

sayakpaul requested a review from DN6 December 9, 2023 04:11

DN6 approved these changes Dec 12, 2023

View reviewed changes

sayakpaul merged commit 1d686ba into main Dec 12, 2023
16 checks passed

sayakpaul deleted the feat/ci-benchmarking branch December 12, 2023 05:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat: Benchmarking Workflow] add stuff for a benchmarking workflow #5839

[feat: Benchmarking Workflow] add stuff for a benchmarking workflow #5839

sayakpaul commented Nov 17, 2023 •

edited

Loading

sayakpaul Nov 17, 2023

DN6 Nov 27, 2023

sayakpaul Nov 28, 2023 •

edited

Loading

DN6 Nov 29, 2023

sayakpaul Nov 29, 2023

DN6 Dec 7, 2023

HuggingFaceDocBuilderDev commented Nov 17, 2023

patrickvonplaten commented Nov 21, 2023

sayakpaul commented Nov 27, 2023

DN6 commented Nov 27, 2023

sayakpaul commented Dec 7, 2023

sayakpaul commented Dec 9, 2023

DN6 left a comment

[feat: Benchmarking Workflow] add stuff for a benchmarking workflow #5839

[feat: Benchmarking Workflow] add stuff for a benchmarking workflow #5839

Conversation

sayakpaul commented Nov 17, 2023 • edited Loading

What does this PR do?

sayakpaul Nov 17, 2023

Choose a reason for hiding this comment

DN6 Nov 27, 2023

Choose a reason for hiding this comment

sayakpaul Nov 28, 2023 • edited Loading

Choose a reason for hiding this comment

DN6 Nov 29, 2023

Choose a reason for hiding this comment

sayakpaul Nov 29, 2023

Choose a reason for hiding this comment

DN6 Dec 7, 2023

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Nov 17, 2023

patrickvonplaten commented Nov 21, 2023

sayakpaul commented Nov 27, 2023

DN6 commented Nov 27, 2023

sayakpaul commented Dec 7, 2023

sayakpaul commented Dec 9, 2023

DN6 left a comment

Choose a reason for hiding this comment

sayakpaul commented Nov 17, 2023 •

edited

Loading

sayakpaul Nov 28, 2023 •

edited

Loading