Conda recipe #404

apriltuesday · 2023-11-28T14:31:53Z

The conda directory will be submitted (with updated checksum etc.) to bioconda-recipes, at which point CMAT would be available as usual from the bioconda channel.

If you want to test locally before it's released on bioconda, you should be able to do the following:

Pull the branch and replace the source and build sections in conda/meta.yaml with:

source:
  path: ../
 
build:
  number: 0
  noarch: generic
  preserve_egg_dir: True

Install conda if you haven't already
Run the following:

# Install conda-build in base environment - see https://docs.conda.io/projects/conda-build/en/stable/install-conda-build.html
conda activate base
conda install conda-build
 
# Create new environment called "cmat-test" for testing locally
conda create -n cmat-test python=3.8
conda activate cmat-test
 
# Build and install local recipe - see https://docs.conda.io/projects/conda-build/en/stable/user-guide/tutorials/build-pkgs.html#building-and-installing
cd /path/to/cmat/conda  # directory containing the recipe
conda build --python 3.8 .
conda install --use-local cmat

The build will run only minimal tests as per Bioconda guidelines, for an end-to-end test you can do this:

mkdir testdir && cd testdir
cmat annotate --output_dir . --clinvar /path/to/cmat/tests/output_generation/resources/end2end/input.xml.gz --resume

apriltuesday · 2023-11-28T15:26:26Z

conda/meta.yaml

+    - pytest-cov==2.10.0
+    - requests==2.31.0
+    - requests-mock==1.8.0
+    - retry==0.9.2


Not sure why these requirements have to be present in both host and run sections, but it's the only way I could get it to work...

tcezard

Looks good.

tcezard · 2023-11-29T14:59:35Z

bin/cmat/cmat

+# ones in Andries Feder's Cladebreaker (https://github.com/andriesfeder/cladebreaker)
+# and Robert A. Petit III's Bactopia (https://bactopia.github.io).
+
+VERSION=3.0.6.dev3


Is it worth moving the version to a text file and read that from both the setup.py and this wrapper script?

I think there's a chicken and egg problem, because this wrapper script uses the version to figure out where the shared files are, which presumably would include the separate text file...

Perhaps the conda build script could modify this wrapper to include the correct version number, but I'd have to think it through.

Is there not a better way to store the version, especially when there's also the {% set version = "3.0.6.dev3" %} at the meta.yaml file?
We end up having the same information in different places, no?

Yes, right now the version is in 3 places, which is not great...

I think the version has to be in at least 2 places though, i.e. somewhere in this repo and in bioconda-recipes (which is what everything in the conda directory is reflecting).

I will try to get the setup.py and the wrapper script to use a single source though.

I managed to get this working with 2 sources of version without too much difficulty. I also created #406 for automating the conda release which I believe should let us have just 1 source of version, as the automated release would generate the meta.yaml file using the same VERSION file.

M-casado

I'm currently trying to test it locally. The installation is taking a long while, so I'll be back after it finishes. For now, a few comments in-line, plus:

Documentation. Should we include a remark on how to install it through Conda in the README? Especially if we are expecting all to use a virtual environment.

M-casado · 2023-12-04T16:38:17Z

bin/cmat/cmat

+# ones in Andries Feder's Cladebreaker (https://github.com/andriesfeder/cladebreaker)
+# and Robert A. Petit III's Bactopia (https://bactopia.github.io).
+
+VERSION=3.0.6.dev3


Is there not a better way to store the version, especially when there's also the {% set version = "3.0.6.dev3" %} at the meta.yaml file?
We end up having the same information in different places, no?

conda/build.sh

M-casado · 2023-12-04T16:59:42Z

Update on the local testing

Failed tests but successful build (?)

$ conda build --python 3.8 .
WARNING: No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.22
Adding in variants from internal_defaults
Adding in variants from config.variant
Attempting to finalize metadata for cmat
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
BUILD START: ['cmat-3.0.6.dev3-py38_0.tar.bz2']
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /home/mcasado/miniconda3/conda-bld/cmat_1701707488374/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac


The following NEW packages will be INSTALLED:
...
...

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Copying /mnt/c/Users/mcasado/Documents/GitHub/FORKED-CMAT/eva-opentargets to /home/mcasado/miniconda3/conda-bld/cmat_1701707488374/work/
source tree in: /home/mcasado/miniconda3/conda-bld/cmat_1701707488374/work
export PREFIX=/home/mcasado/miniconda3/conda-bld/cmat_1701707488374/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac
export BUILD_PREFIX=/home/mcasado/miniconda3/conda-bld/cmat_1701707488374/_build_env
export SRC_DIR=/home/mcasado/miniconda3/conda-bld/cmat_1701707488374/work
Processing $SRC_DIR
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: cmat
  Building wheel for cmat (setup.py): started
  Building wheel for cmat (setup.py): finished with status 'done'
  Created wheel for cmat: filename=cmat-3.0.6.dev3-py3-none-any.whl size=74876 sha256=0e75ececb87ade93d76de21441af4f6febdf16a8e3fb75ba6b99442938ee2a1c
  Stored in directory: /tmp/pip-ephem-wheel-cache-p98354hu/wheels/85/5d/9e/287fe51c87339d3783955109b3d9d030cf7b1504d4c9556ce4
Successfully built cmat
Installing collected packages: cmat
Successfully installed cmat-3.0.6.dev3

Packaging cmat
Packaging cmat-3.0.6.dev3-py38_0
number of files: 77
Fixing permissions
Packaged license file/s.
INFO :: Time taken to mark (prefix)
        0 replacements in 0 files was 0.01 seconds
WARNING: Importing conda-verify failed.  Please be sure to test your packages.  conda install conda-verify to make this message go away.
TEST START: /home/mcasado/miniconda3/conda-bld/noarch/cmat-3.0.6.dev3-py38_0.tar.bz2
Adding in variants from /tmp/tmptvph2k83/info/recipe/conda_build_config.yaml
Renaming /home/mcasado/miniconda3/conda-bld/cmat_1701707488374/_build_env prefix directory '/home/mcasado/miniconda3/conda-bld/cmat_1701707488374/_build_env' to '/home/mcasado/miniconda3/conda-bld/cmat_1701707488374/build_prefix_moved_cmat-3.0.6.dev3-py38_0_linux-64'
shutil.move(/home/mcasado/miniconda3/conda-bld/cmat_1701707488374/_build_env prefix)=/home/mcasado/miniconda3/conda-bld/cmat_1701707488374/_build_env, dest=/home/mcasado/miniconda3/conda-bld/cmat_1701707488374/build_prefix_moved_cmat-3.0.6.dev3-py38_0_linux-64)
Renaming work directory '/home/mcasado/miniconda3/conda-bld/cmat_1701707488374/work' to '/home/mcasado/miniconda3/conda-bld/cmat_1701707488374/work_moved_cmat-3.0.6.dev3-py38_0_noarch'
shutil.move(work)=/home/mcasado/miniconda3/conda-bld/cmat_1701707488374/work, dest=/home/mcasado/miniconda3/conda-bld/cmat_1701707488374/work_moved_cmat-3.0.6.dev3-py38_0_noarch)
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /home/mcasado/miniconda3/conda-bld/cmat_1701707488374/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold


The following NEW packages will be INSTALLED:
...
...

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
export PREFIX=/home/mcasado/miniconda3/conda-bld/cmat_1701707488374/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold
export SRC_DIR=/home/mcasado/miniconda3/conda-bld/cmat_1701707488374/test_tmp
import: 'cmat'
import: 'cmat'
+ cmat
/usr/bin/env: ‘bash\r’: No such file or directory
WARNING: Tests failed for cmat-3.0.6.dev3-py38_0.tar.bz2 - moving package to /home/mcasado/miniconda3/conda-bld/broken
TESTS FAILED: cmat-3.0.6.dev3-py38_0.tar.bz2

Failed installation

Even when it said that the installation was successful in the previous prompt, I guess since the tests were not passed, it moved everything to the broken directory of conda, and it wasn't found when to be installed:

$ conda install --use-local cmat
Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - cmat

Current channels:

  - file:///home/mcasado/miniconda3/conda-bld/linux-64
  - file:///home/mcasado/miniconda3/conda-bld/noarch
  - https://conda.anaconda.org/conda-forge/linux-64
  - https://conda.anaconda.org/conda-forge/noarch
  - https://conda.anaconda.org/bioconda/linux-64
  - https://conda.anaconda.org/bioconda/noarch
  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

apriltuesday · 2023-12-05T12:16:41Z

For your installation error @M-casado, are you using WSL by chance? /usr/bin/env: ‘bash\r’ seems to indicate a problem with line endings, possibly due to how git is handling them (see first answer here). This should only be an issue with this local installation method since conda install won't use git, but you can try the suggestions in that Stack Overflow answer if you like.

@tcezard Btw, did you get it working on Mac?

Documentation. Should we include a remark on how to install it through Conda in the README? Especially if we are expecting all to use a virtual environment.

Yes definitely, I'll add it once it's been actually added to the bioconda channel, since that requires another PR and I'm not sure how long that will take...

M-casado · 2023-12-07T09:49:36Z

conda/meta.yaml

@@ -1,4 +1,4 @@
-{% set version = "3.0.6.dev3" %}
+{% set version = "3.0.6.dev4" %}


Just in case it's a hassle for you to change this manually. Now that we have the version within the file VERSION, would it be handy to have a very simple python script that replaces this bit of version in the meta.yaml, triggered by a GitHub action?

For example, something like the following, summarised by our friend GPT:

import re version_file_path = 'VERSION' yaml_file_path = 'meta.yaml' with open(version_file_path, 'r') as file: version = file.read().strip() with open(yaml_file_path, 'r') as file: yaml_content = file.read() new_yaml_content = re.sub(r'{% set version = ".*" %}', f'{{% set version = "{version}" %}}', yaml_content) with open(yaml_file_path, 'w') as file: file.write(new_yaml_content)

Again, only if it's less of a hassle to keep in mind that the VERSION is also in the YAML file

Or actually, rather than modify this YAML file, why not have the other files read the file meta.yaml instead of VERSION? That way, the version exists only in one place. We could use a regex to find the version within this yaml file and extract it instead

I think it's okay for now, but it's something to keep in mind for a future PR (e.g. in #406)... we can decide the direction of information flow (i.e. from VERSION to YAML or vice versa) and how it's triggered there.

M-casado · 2023-12-07T09:53:54Z

@apriltuesday

are you using WSL by chance? /usr/bin/env: ‘bash\r’ seems to indicate a problem with line endings

You are very right. I encounter that problem from time to time. but normally the errors are more informative. It is probably what you mentioned. I'll wait then till you publish it to test the installation through conda.

M-casado · 2023-12-07T09:50:07Z

conda/meta.yaml

@@ -1,4 +1,4 @@
-{% set version = "3.0.6.dev3" %}
+{% set version = "3.0.6.dev4" %}


Again, only if it's less of a hassle to keep in mind that the VERSION is also in the YAML file

M-casado · 2023-12-07T09:51:07Z

conda/meta.yaml

@@ -1,4 +1,4 @@
-{% set version = "3.0.6.dev3" %}
+{% set version = "3.0.6.dev4" %}


Or actually, rather than modify this YAML file, why not have the other files read the file meta.yaml instead of VERSION? That way, the version exists only in one place. We could use a regex to find the version within this yaml file and extract it instead

apriltuesday added 3 commits November 28, 2023 14:10

add conda recipe and wrapper script

c83ad7a

fix biomart query and conda dependencies

c20af12

clean up build.sh

5066e4a

apriltuesday commented Nov 28, 2023

View reviewed changes

apriltuesday self-assigned this Nov 28, 2023

apriltuesday requested review from tcezard and M-casado November 28, 2023 15:29

tcezard approved these changes Dec 1, 2023

View reviewed changes

M-casado reviewed Dec 4, 2023

View reviewed changes

single source of version for wrapper and setup

1f3a405

M-casado reviewed Dec 7, 2023

View reviewed changes

M-casado approved these changes Dec 7, 2023

View reviewed changes

apriltuesday merged commit 4d367cd into EBIvariation:master Dec 7, 2023
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conda recipe #404

Conda recipe #404

apriltuesday commented Nov 28, 2023 •

edited

Loading

apriltuesday Nov 28, 2023

tcezard left a comment

tcezard Nov 29, 2023

apriltuesday Dec 4, 2023

M-casado Dec 4, 2023

apriltuesday Dec 5, 2023

apriltuesday Dec 6, 2023

M-casado left a comment

M-casado Dec 4, 2023

M-casado commented Dec 4, 2023

apriltuesday commented Dec 5, 2023

M-casado Dec 7, 2023

M-casado Dec 7, 2023

M-casado Dec 7, 2023

apriltuesday Dec 7, 2023 •

edited

Loading

M-casado commented Dec 7, 2023

M-casado Dec 7, 2023

M-casado Dec 7, 2023

		@@ -1,4 +1,4 @@
		{% set version = "3.0.6.dev3" %}
		{% set version = "3.0.6.dev4" %}

Conda recipe #404

Conda recipe #404

Conversation

apriltuesday commented Nov 28, 2023 • edited Loading

Choose a reason for hiding this comment

tcezard left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

M-casado left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

M-casado commented Dec 4, 2023

Update on the local testing

Failed tests but successful build (?)

Failed installation

apriltuesday commented Dec 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apriltuesday Dec 7, 2023 • edited Loading

Choose a reason for hiding this comment

M-casado commented Dec 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apriltuesday commented Nov 28, 2023 •

edited

Loading

apriltuesday Dec 7, 2023 •

edited

Loading