Pycurv fails to parallel process

[Kathir-66]

I was trying to run pycurv, for the meshes.
It is very slow and kept running forever for the meshes of IMM and relatively slow for OMM than before. I tested on 2 systems one with 10 cores and 30 cores.

So once I executed the code with 30 cores, I monitored using the htop command. To see the usage of CPUs it looks like it is just using one CPU at a time. I am attaching the pictures below for reference

The config file

data_dir: "./inputs/"
work_dir: "./morphometrics/"
segmentation_values: # The value used to represent each feature, and the name of the feature.
  OMM: 1
  IMM: 2
  ER: 
surface_generation:
  angstroms: false # Default scaling adjusts so surfaces are nm-scaled, if set to true surfaces are not rescaled and remain angstrom-scaled.
  ultrafine: true # If true, the surface is generated with a new high quality protocol. This makes calculation much slower but the surface is much better for sampling density.
  mesh_sampling: 1 # Approximate mesh sampling rate, 
  simplify: False # Non-simplified surfaces can be VERY slow, but are better for sampling the coulomb potential.
  max_triangles: 150000 # If simplifying and more than this number of triangles are generated, the surface is decimated to this number
  extrapolation_distance: 1.5 # Distance in nm (or angstroms, if angstrom setting is True) to extrapolate the surface
  octree_depth: 9 # Increase if the surface is too smooth, decrease if surface is too jagged - but has a HUGE effect.
  point_weight: 0.7 # Smaller numbers have stronger weight towards fitting segmented points, larger numbers have stronger weight towards generating a smooth surface
  neighbor_count: 400 # Number of neighbors used for normal estimation. Should not need to adjust this, but I am exposing it as a tuning variable. Values over 50 seem to all be equivalently good...
  smoothing_iterations: 1 # Number of smoothing iterations to perform for estimating normals. You should usually leave this at 1.
curvature_measurements:
  radius_hit: 9 # This corresponds to the radius of the smallest feature of interest, roughly, for neighborhood determination. 8-15 seems to be a good range for mitochondria
  min_component: 30 # The minimum number of triangles for a component to be considered for curvature measurement.
  exclude_borders: 1 # Values greater than zero exclude the border from the curvature calculation by n nm/angstroms.
distance_and_orientation_measurements:
  mindist: 3 # Minimum distance between two points for them to be considered for distance measurement.
  maxdist: 400 # Maximum distance between two points for them to be considered for distance measurement.
  tolerance: 0.1 # Tolerance for determining if a triangle is contacted by a line.
  verticality: true # Measure verticality (angle relative to growth plane). 
  intra: # Intra-surface distance/orientation (verticality) measurements
    - IMM
    - OMM
    - ER
  relative_orientation: true # Measure relative orientation (angle relative to nearest neighbor surface).
  inter: # Intersurface distance/orientation measurements. Writes out symmetrically so you do not need to give symmetric options.
    OMM:
      - IMM
      - ER
cores: 30 # Number of cores to use for parallel processing.

htop command before running pycurv

htop command after running pycurv

I tested this both for the conda environment and docker. The results are the same.

[bbarad]

Pycurv only uses certain steps for parallel processing as far as I understand it. I think the bigger issue is that the new ultrafine meshing protocol is making triangles too small (and too numerous) by default. It worked fine for my small test mesh but I have heard often that the ultrafine settings are too aggressive. Can you let me know if turning ultrafine off yields better/faster results? You can also turn simplify back on!

[bbarad]

Recent commit should revert the default - I'll leave this open till we get a more complete resolution!