add test-invoke-pip.yml #1521

mauwii · 2022-11-20T11:15:43Z

this workflow is similar to the conda test workflow, but instead is using pip to install requirements. It is testing

requirements-lin-cuda.txt
requirements-lin-amd.txt
requirements-mac-mps-cpu.txt

with Python 3.9 and 3.10.

IMPORTANT

this removes -e .from those 3 requirement files (imho should be removed from the others as well). So the workflow now is to install requirements, and afterwards pip install -e .

since it requires to have torch already installed also restore origin requirements-base.txt after suc. test in my fork

mauwii · 2022-11-20T12:28:02Z

This comment is obsolete now 🙈

# IMPORTANT

This removes the `requirements-base.txt` from some requirements files (only the ones I can test here and local, but I suppose it should be removed from the others as well).

For me the installation was also not working locally with the `requirements-base.txt` included in the `requirements.txt` (same error as in workflow, `torch` needs to be installed first), which makes me think that this only worked on system with preinstalled / shared `torch` package.

keturn · 2022-11-21T00:45:52Z

I don't know all the ins and outs of pip dependency resolution, but the concern I would have about splitting it into multiple commands is whether or not it resolves things the same way when it's not given the dependency tree all at once. Especially when modifiers like --pre and --extra-index-url are involved.

The error looks like the one this was intended to fix: XPixelGroup/BasicSR#514

which is tagged as if it were released in basicsr 1.4.2. And yet, the logs here show this using 1.4.2 and encountering that error on line 8 of setup.py. The line removed by that PR???

mauwii · 2022-11-21T00:55:12Z

I am not sure if I can follow you. The error appears for example here, which was when installing basicsr-1.4.1.tar.gz. This was exactly the same error I got when doing a pip install --prefer-binary -r requirements.txt (where requirements.txt was a symlink to environments-and-requirements/requirements-mac-mps-cpu.txt.

But yes, I am with you, if possible it would be better to have the requirements-base.txt called from within the environments requirements*.txt

keturn · 2022-11-21T01:11:46Z

Wait, how did it end up with 1.4.1? We have a requirement for 1.4.2, that's what it downloaded... oh.

INFO: pip is looking at multiple versions of realesrgan to determine which version is compatible with other requirements. This could take a while.
Collecting realesrgan
  Downloading realesrgan-0.2.9-py3-none-any.whl (26 kB)
  Downloading realesrgan-0.2.8-py3-none-any.whl (25 kB)
  Downloading realesrgan-0.2.7-py3-none-any.whl (26 kB)
  Downloading realesrgan-0.2.6-py3-none-any.whl (26 kB)
  Downloading realesrgan-0.2.5.0-py3-none-any.whl (25 kB)
Collecting basicsr>=1.3.3.11
  Downloading basicsr-1.4.1.tar.gz (171 kB)

[See the pip manual on Backtracking for why it's downloading so many versions.]

So initially pip did get us the right version of basicsr, but then something has it going back a bunch of versions of realesrgan, until it finally goes back so far that it finds a realesrgan requiring the broken basicsr. What a mess!

requirements-base currently doesn't specify a version dependency for realesrgan at all. Can we add one to prevent it from going back that far?

It'll likely break, because there was some reason it thought it wouldn't be able to use the current version of realesrgan in the first place, but at least it will tell us what that reason is and maybe that's something we can fix.

add `basicsr>=1.4.2` to requirements-base.txt remove second installation step

mauwii · 2022-11-21T01:29:23Z

Very nice finding @keturn, now we get the next error:

error: [Errno 2] No such file or directory: '/home/runner/work/InvokeAI/InvokeAI/scripts/load_models.py'

…nvokeAI into add-test-invoke-pip-workflow

mauwii · 2022-11-21T02:05:08Z

When I replace realesrgan with git+https://github.com/invoke-ai/Real-ESRGAN.git#egg=realesrgan, I get this error (which at least points out where the broken basicsr dependencie comes from):

The conflict is caused by:
    The user requested basicsr>=1.4.2
    gfpgan 1.3.8 depends on basicsr==1.4.1

mauwii · 2022-11-21T03:11:00Z

informed @lstein that I need information about how to run preload_models.py non interactive

lstein · 2022-11-21T13:26:38Z

Is preload_models.py --no-interactive not working?

or rather, I think you may be asking for all the options to be set on the command line, like preload_models.py --output_dir=/foo/bar/outputs --models_dir=/foo/bar/models --configs_dir=/etc/configs ?? That's a good idea.

lstein · 2022-11-21T13:29:20Z

When I replace realesrgan with git+https://github.com/invoke-ai/Real-ESRGAN.git#egg=realesrgan, I get this error (which at least points out where the broken basicsr dependencie comes from):
The conflict is caused by:

    The user requested basicsr>=1.4.2

    gfpgan 1.3.8 depends on basicsr==1.4.1

basicsr-1.4.2 fails to load on Windows computers. Starting out with a new venv or conda environment, pip install basicsr==1.4.2 seems to work ok, but then python -mbasicsr gets you a module not found error. 1.4.1 works fine.

This is why the requirements in realesrgan and gfpgan were downgraded to use 1.4.1.

lstein

This looks good to me, but I think @tildebyte should check.

lstein · 2022-11-23T23:07:43Z

@mauwii is this ready for a merge, and will it fix the current CI errors?

lstein

This is ugly, but there are now three scripts that call the same codebase: preload_models.py, load_models.py, and configure_invokeai.py. I really only want to have the last one, but I don't want to alter the CI files without @mauwii 's blessing.

use same 4 git packages as for pip

lstein · 2022-11-24T22:56:24Z

I'm getting the following dependency error after running pip install -r requirements.txt; pip install -e .:

Traceback (most recent call last):������������������������������������������������������������������������������������������������� 
  File "/home/lstein/.local/bin/invoke.py", line 4, in <module>�������������������������������������������������������������������� 
    __import__('pkg_resources').require('InvokeAI==2.1.4')������������������������������������������������������������������������� 
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3254, in <module>������������������������������������������ 
    def _initialize_master_working_set():������������������������������������������������������������������������������������������ 
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3237, in _call_aside��������������������������������������� 
    f(*args, **kwargs)������������������������������������������������������������������������������������������������������������� 
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3266, in _initialize_master_working_set�������������������� 
    working_set = WorkingSet._build_master()��������������������������������������������������������������������������������������� 
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 586, in _build_master�������������������������������������� 
    return cls._build_from_requirements(__requires__)������������������������������������������������������������������������������ 
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 599, in _build_from_requirements��������������������������� 
    dists = ws.resolve(reqs, Environment())���������������������������������������������������������������������������������������� 
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 792, in resolve�������������������������������������������� 
    raise VersionConflict(dist, req).with_context(dependent_req)������������������������������������������������������������������� 
pkg_resources.ContextualVersionConflict: (numpy 1.17.4 (/usr/lib/python3/dist-packages), Requirement.parse('numpy<1.26.0,>=1.18.5'), {'scipy'})

mauwii · 2022-11-24T22:57:10Z

Are you using a clean environment? I mean the runners work to do the installation, so I doubt that there is any requirements error at all ....

mauwii · 2022-11-24T22:57:53Z

And the runners succeed even with python 3.9 and python 3.10 xD

mauwii · 2022-11-24T23:17:28Z

mostly suspiccious to me is this /usr/lib/python3/dist-packages/pkg_resources/__init__.py

when you are using a clean and new .venv, this should be the python3 path to your venv and i doubt that you created this in /usr/lib/ 🤔

Also strange is from where the numpy 1.17.4 should be 🙈

Vargol · 2022-11-26T17:09:57Z

IMPORTANT

this removes -e .from those 3 requirement files (imho should be removed from the others as well). So the workflow now is to install requirements, and afterwards pip install -e .

I go away for a few days...

Why was this done, pip install -r requirements....txt was successfully installing a working InvokeAi before this change.

mauwii · 2022-11-26T18:24:53Z

For me on mac with M1 it didn't, neither for lstein, neither for the Runners.... So it was implemented to fix the Installation and it was discussed here and in discord. But nobody would stop you from implementing -e . back in the requirements and make it work 😉

Vargol · 2022-11-26T18:26:56Z

For me on mac with M1 it didn't, neither for lstein, neither for the Runners.... So it was implemented to fix the Installation and it was discussed here and in discord. But nobody would stop you from implementing -e . back in the requirements and make it work 😉

So basically your saying you broke the pip install cos the conda install is broken ???

I'd love to fix it but conda is not getting anywhere near any computer I have to use python on.

mauwii · 2022-11-26T18:30:36Z

The Conda fix was implemented after the pip fix and has nothing to do with the pip fix.

Maybe you should re-read what I wrote, which was speaking about the pip install which you mentioned...

keturn · 2022-11-26T18:31:44Z

because the pip install was broken. You can see this PR went through a lot of runs trying to find something that worked: https://github.com/invoke-ai/InvokeAI/actions/workflows/test-invoke-pip.yml?query=branch%3Aadd-test-invoke-pip-workflow+event%3Apull_request

Vargol · 2022-11-26T18:32:04Z

I don't know when it doesn't work for the guys I've been using it right from the beginning.

Vargol · 2022-11-26T18:39:45Z

I wonder if its because you took out the prefer binary option

mauwii · 2022-11-26T18:40:48Z

So I guess you did not try a Installation from a clean venv, which indeed did not work. I did not spend hours of my life to fix something which is not broken, just for fun or to break it for everybody....

I wonder if its because you took out the prefer binary option

just take a look at the history, there are more than enough failing runs where you could look at ....

Vargol · 2022-11-26T18:43:06Z

Sorry, I know nothing about Github actions would know where to start

Vargol · 2022-11-26T18:45:24Z

Just edited the -e . back in on my own machine and i the pip went without issue.

python -m venv InvokeAIPip
cd InvokeAIPip
. ./bin/activate
git clone https://github.com/invoke-ai/InvokeAI.git
cd InvokeAI
git switch development
ln -sf environments-and-requirements/requirements-mac-mps-cpu.txt .

edit equirements-mac-mps-cpu.txt

pip install --prefer-binary -r requirements-mac-mps-cpu.txt
pip install -e .

mauwii · 2022-11-26T18:48:35Z

Why do you do a pip install -e . when you have it edited back into the requirements?

I would suggest you open a PR if it is working to install now with -e . in the requirements. as I say I was not the only person who was affected with problems, also the runners didn't manage to install, that is all I have to say here ....

Vargol · 2022-11-26T18:51:29Z

vBecause I clumsily copied an extra line from the script I used, the had a exit where I did the edit.
I didn't do the -e . install

% cat create_sd_stop.sh
set -x
python -m venv InvokeAIPip
cd InvokeAIPip
. ./bin/activate
git clone https://github.com/invoke-ai/InvokeAI.git
cd InvokeAI
git switch development
ln -sf environments-and-requirements/requirements-mac-mps-cpu.txt .
exit <-------
pip install --prefer-binary -r requirements-mac-mps-cpu.txt
pip install -e .
export PYTORCH_ENABLE_MPS_FALLBACK=1
#python scripts/preload_models.py --no-interactive
cp configs/models.yaml.example configs/models.yaml
cp ../../preflight_prompts.txt .
cp ../../curly.png .
cp ../../Lincoln-and-Parrot-512* .

1185 ./create_sd_stop.sh
1186 vi InvokeAIPip/InvokeAI/requirements-mac-mps-cpu.txt
1187 deactivate
1188 cd InvokeAIPip
1189 . bin/activate
1190 cd InvokeAI
1191 cat ../../create_sd_stop.sh
1192 pip install --prefer-binary -r requirements-mac-mps-cpu.txt
1193 cat ../../create_sd_stop.sh

Vargol · 2022-11-26T19:14:26Z

#1570 raised

mauwii added 2 commits November 20, 2022 12:12

add test-invoke-pip.yml

3f450c6

update requirements-base.txt to fix tests

96a3982

mauwii marked this pull request as draft November 20, 2022 12:07

install requirements-base.txt separate

cf61858

since it requires to have torch already installed also restore origin requirements-base.txt after suc. test in my fork

mauwii marked this pull request as ready for review November 20, 2022 12:24

mauwii requested a review from lstein November 20, 2022 12:24

restore origin requirements

1da9f66

add `basicsr>=1.4.2` to requirements-base.txt remove second installation step

mauwii added the CI-CD Continuous integration / Continuous delivery label Nov 21, 2022

mauwii self-assigned this Nov 21, 2022

mauwii and others added 4 commits November 21, 2022 02:29

Merge branch 'development' into add-test-invoke-pip-workflow

39cb688

re-add previously overseen req in lin-cuda

7e7c63b

Merge branch 'add-test-invoke-pip-workflow' of github.com:invoke-ai/I…

85f3418

…nvokeAI into add-test-invoke-pip-workflow

fix typo in setup.py - scripts/preload_models.py

a0393e5

mauwii added 3 commits November 21, 2022 03:22

use GFBGAN from branch basicsr-1.4.2

e215b44

remove basicsr>=1.4.2 from base reqs

4bdee4d

add INVOKEAI_ROOT to env

66eb6aa

lstein reviewed Nov 22, 2022

View reviewed changes

lstein requested a review from tildebyte November 22, 2022 13:32

Merge branch 'development' into add-test-invoke-pip-workflow

fb616f2

lstein approved these changes Nov 23, 2022

View reviewed changes

mauwii added 6 commits November 24, 2022 22:44

fix job name

6ffd4d6

fix matrix.stable-diffusion-model

d6fec55

restructure matrix

be8027a

fix activate conda env step

e9d6585

update the environment yamls

601526f

use same 4 git packages as for pip

Merge branch 'fix-test-invoke-conda' into add-test-invoke-pip-workflow

600ac8d

mauwii mentioned this pull request Nov 24, 2022

fix test invoke conda #1548

Closed

mauwii added 3 commits November 25, 2022 00:27

rename job in test-invoke-conda

a421352

add tqdm to environment-lin-amd.yml

e0b98a6

fix python commands in test-invoke-conda.yml

317ccb8

mauwii merged commit 2433cc3 into development Nov 25, 2022

mauwii deleted the add-test-invoke-pip-workflow branch November 25, 2022 00:24

Vargol mentioned this pull request Nov 26, 2022

Fix for extra pip install -e . for mac pip install #1570

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add test-invoke-pip.yml #1521

add test-invoke-pip.yml #1521

mauwii commented Nov 20, 2022 •

edited

Loading

mauwii commented Nov 20, 2022 •

edited

Loading

keturn commented Nov 21, 2022

mauwii commented Nov 21, 2022 •

edited

Loading

keturn commented Nov 21, 2022 •

edited

Loading

mauwii commented Nov 21, 2022

mauwii commented Nov 21, 2022

mauwii commented Nov 21, 2022

lstein commented Nov 21, 2022 •

edited

Loading

lstein commented Nov 21, 2022

lstein left a comment

lstein commented Nov 23, 2022

lstein left a comment

lstein commented Nov 24, 2022

mauwii commented Nov 24, 2022

mauwii commented Nov 24, 2022

mauwii commented Nov 24, 2022 •

edited

Loading

Vargol commented Nov 26, 2022 •

edited

Loading

IMPORTANT

mauwii commented Nov 26, 2022

Vargol commented Nov 26, 2022

mauwii commented Nov 26, 2022

keturn commented Nov 26, 2022

Vargol commented Nov 26, 2022

Vargol commented Nov 26, 2022

mauwii commented Nov 26, 2022

Vargol commented Nov 26, 2022

Vargol commented Nov 26, 2022

mauwii commented Nov 26, 2022

Vargol commented Nov 26, 2022

Vargol commented Nov 26, 2022

add test-invoke-pip.yml #1521

add test-invoke-pip.yml #1521

Conversation

mauwii commented Nov 20, 2022 • edited Loading

IMPORTANT

mauwii commented Nov 20, 2022 • edited Loading

keturn commented Nov 21, 2022

mauwii commented Nov 21, 2022 • edited Loading

keturn commented Nov 21, 2022 • edited Loading

mauwii commented Nov 21, 2022

mauwii commented Nov 21, 2022

mauwii commented Nov 21, 2022

lstein commented Nov 21, 2022 • edited Loading

lstein commented Nov 21, 2022

lstein left a comment

Choose a reason for hiding this comment

lstein commented Nov 23, 2022

lstein left a comment

Choose a reason for hiding this comment

lstein commented Nov 24, 2022

mauwii commented Nov 24, 2022

mauwii commented Nov 24, 2022

mauwii commented Nov 24, 2022 • edited Loading

Vargol commented Nov 26, 2022 • edited Loading

IMPORTANT

mauwii commented Nov 26, 2022

Vargol commented Nov 26, 2022

mauwii commented Nov 26, 2022

keturn commented Nov 26, 2022

Vargol commented Nov 26, 2022

Vargol commented Nov 26, 2022

mauwii commented Nov 26, 2022

Vargol commented Nov 26, 2022

Vargol commented Nov 26, 2022

mauwii commented Nov 26, 2022

Vargol commented Nov 26, 2022

Vargol commented Nov 26, 2022

mauwii commented Nov 20, 2022 •

edited

Loading

mauwii commented Nov 20, 2022 •

edited

Loading

mauwii commented Nov 21, 2022 •

edited

Loading

keturn commented Nov 21, 2022 •

edited

Loading

lstein commented Nov 21, 2022 •

edited

Loading

mauwii commented Nov 24, 2022 •

edited

Loading

Vargol commented Nov 26, 2022 •

edited

Loading