Skip to content

Meeting notes

Jump to bottom
William F. Broderick edited this page Oct 22, 2018 · 28 revisions

Meeting 16: Oct 15

We discussed how to make the software effort more inclusive to everyone in the lab.
Our takeaway is: we need to reduce the barrier to entry and to show more clearly the benefits of a functional and shared code-base.
Three practical ideas to work on now:
i) invite a guest to help bring momentum to this effort. For example, it'd be awesome to get Jeremy Freeman or Deep Ganguli to come give us a tutorial.
ii) organize code review sessions to learn good practices and feel comfortable sharing code.
iii) start other repositories to host projects that relate to other people's current work. For example:

  • diverse selectivity and gain control (filters, ..)
  • modulated variability (simulating, fitting, ..)
  • efficient coding (encoding, decoding, ..)
  • perceptual metrics & Co. (msssim, NLPD, eigendist, ..)

Meeting 15: Oct 12

  • Discussed changes to pyrshow
  • TODOS:
    • make consistent list of attributes for all pyramids (transparently named)
      • num_orientation, num_scales, complex (True/False), image_size,
    • combine SFpyr and SCFpyr, just differing based on complex flag
    • rename files to the name of the class (Gpyr.py -> GaussianPyramid.py)
    • rename filters (filter1 -> filter up, filter down based on whether they're for up- or down-sampling)
    • change how pyramid data (pyr.pyr) is stored:
      • a big 1d vector with indices corresponding to the band and level (using pyr_utils.LB2idx / idx2LB) -- probably only want this in the torch version, so take whatever the method is and have the python version raise an exception
      • instead, have a dictionary whose keys are (level, orientation), "residual_high" and "residual_low", who contains the arrays of pyramid coefficients
    • make pyramid import hierarchical, but keep tools flat
    • for pyrshow:
      • handle complex pyramids by interleaving them
      • make showing residuals optional
    • for imshow, when vrange is auto, have the titles how data range, not vrange -- actually have printed range always be data range
    • throw down notes about how to make more pythonic version of current showPyr (with the different levels shown at actual size), instead of putting that together
    • make steerable pyramids parameterized by number of orientations, rather than order (and explain that relationship in the docstring)

Meeting 14: Sep 28

  • Generalizing imshow further and merging branches where necessary
  • TODO: Built with Sphinx using a theme provided by Read the Docs.

Meeting 13: Sep 21

Talked through some of the TODOs for pytorch code and listed some in tutorial 04

Meeting 12: Sep 12

new meeting schedule: Fridays 1-3pm

  • Billy works on pyrshow
  • Zahra works on the pytorch version of Spyr
  • Nikhil works on the pytorch version of SFpyr
  • Pierre works on the original Spyr, SFpyr and SFCpyr

Meeting 11: August 30

initializing the pytorch branch

Meeting 10: August 9

Discussed in which data format the pyramids should be stored (important when accessing sub levels/bands, eg. for reconstruction with partial info, or for plotting). Options include: list, dictionary, or large vector with proper index handling. Playing with pointers can be the source of nasty bugs (cf. hacking C code). So is switching between list index and lev/band pair. Maybe this is a place where dictionaries would be helpful?

Discussed the existence of multiple inverses from overcomplete representations, and talked through the Gaussian and Laplacian Pyramid examples. Options include: upConv to reverse corrDn, the Moore-Penrose pseudo inverse (least l2 norm), sparse inverse (least l1 norm).

Discussed filter choice: filter size / speed, reconstruction error / quantization of digital signals (https://www.dxomark.com/)

TODO: getting rid of the pyramid util file and implement those functions using @staticmethod within the correct class. https://www.geeksforgeeks.org/class-method-vs-static-method-python/ This way the code will be better organized and you can still call the static functions without constructing the class
NOTE: but don't overuse objects, the tree of dependencies can quickly become hard to parse (which piece of code actually runs when I use a given child class?)

Meeting 9: August 2

Struggled with implementation details of Wpyr. Finally fixed the reshaping in wrapper.py and got rid of the res argument.

Note on staggering for the Wavelets code:
The high pass filter is constructed from the low pass filter by modulating and flipping it. Odd-length filters will also need third step: spatial shift by one sample. This guaranties the filters will be orthogonal to each other and to shift by two versions of each other.
In practice though, odd-lenght filters are not spatially shifted by one sample in the modulateFilp function. Instead, this is taken care of by the start argument of CorrDn and upConv. Convolution with the high pass filter always starts on the second index, while convolution with the low pass filter starts on the first index for odd-length filters (staggered) or second index for even-length filters (not staggered).

Combinations of boundary handling and filter type that work (im)properly:

  • can't use reflected boundaries with asymmetric filters (daub)
  • can use 'reflect1' and symmetric odd-length QMFs
  • can use 'reflect2' and symmetric even-length QMFs

TODO:

  • fix height argument of Wpyr, so that it is consistent with Spyr
  • make arguments names consistent across init and methods and pyramid types
  • raise warning for odd sized signals and even sized filters
  • write pyrshow in display_tools
  • handle saving with desired resolution
  • handle extra vranges (share mid gray zero, user specified vrange)
  • handle indep vranges in animshow
  • then move on to Spyr, SFpyr, and SFCpyr

Meeting 8: July 26

Discussed filter normalization conventions and the sqrt2 in 'binom'.

Discussed the modulateFlip function, orthogonality and how it underlies the Wavelet constructions.

Discussed the Wavelet bugs, which looks like a an indexing problem: ie. where does corrDn start. To figure this out, look at single stage first and explicitly build corresponding linear transform matrix to visualize the problem. TODO print a warning if corrDn is called with different filter shape and sub-sampling shape.

One extra function to be implemented: grab generalized neighborhoods of pyramid coefficients (in space, ori and scale). Matlab has im2col and col2im but only allows for full density or sampled at the size of the block. We would need a more general function allowing for user specified subsampling and boundary handling (ala convolution code). Maybe take inspiration from scikit-image view as blocks / as images.

Worked on the display code.

Meeting 7: July 19

Discussed the organization of:

  • pyramid base class, pyramid child classes and their relationship
  • display_tools (imshow, pyrshow, animshow)

Question:

  • why is scaling of binomial filters dimension independent and un normalized? (neither l1 nor l2)

Meeting 6: July 12

Wrestling with installation on different platforms

  • giving up on windows support

TODO
Let the specifications in the config file resolve most of these issues automatically at installation time.

Meeting 5: June 28

Eero gave an introduction to pyramids: Gaussian, Laplacian, Wavelets, Steerable

Meeting 4: June 20

Pytorch FFT / conv:

  • Is mixed radix implemented in pytorch.fft ?
  • Boundary handling - important especially in a multi scale context
  • Extrapolation - double refection in 1D, but does not generalize (?!)
  • Reflexion boundaries, add three reflected copies - math equivalent to DCT and corresponding convolution theorem
  • NOTE: pytorch padding layers offers great flexibility (https://pytorch.org/docs/stable/nn.html?#padding-layers)

Made repository private and pushed first commits

  1. ran automated script from 2to3 - https://docs.python.org/3/library/2to3.html#to3-reference
  2. Jimmy/Billy/pe started to organize:
  • C code (and functions that interact with the C code, for handling convolutions mostly)
  • get_filters
  • synthetic_images
  • image_tools
  1. TODO:
  • display - using matplotlib
  • pyramids - (use full name, eg. LaplacianPyramid - not Lpyr)

Meeting 3: June 14

Created GitHub organization
Waiting for academic approval

Discussed existing implementations out there:
Scikit-image
Pytorch SSIM
https://github.com/Po-Hsun-Su/pytorch-ssim
https://github.com/jorge-pessoa/pytorch-msssim

Meeting 2: June 7

big picture [Eero]

{x, θ, y} : inputs, outputs, parameters

fix adjust
simulate {x, θ} {y}
fit {x, y} {θ}
synthesize {y, θ} {x}

Some action items:

  1. Eero will transition our lab account (https://github.com/LabForComputationalVision) to an “organization” account, that will be more flexible/better for lab use.

  2. Eero will (gradually) with Zahra’s help, move our old matlab coder sources to GitHub, and eventually replace the lab software page (http://www.cns.nyu.edu/~lcv/software.php) with a direct redirect the gitHub page.

  3. Overall, preference for use of pyTorch (but TensorFlow and matlab also acceptable).

matlabPyrTools:

  • Image display / stats [plain python]
  • Pyramid display / stats [plain python]
  • Multi-scale pyramid implementations: Gaussian/Laplacian/Wavelet/Steerable pyramids [pyTorch]
  • Convolution, with subsampling, and boundary handling [add new features/capabilities to existing pyToorch convolution]

Other current LCV code (from software page):

  • MS-SSIM - is there a (good) pyTorch version?
  • NLPD
  • CBP -
  • End-to-end-comression (Johannes) - should package up the matlab code that’s on the software page
  • MT model
  • GSM denoising code [python version?]
  • Texture synthesis code [pyton version?]

Jimmy - Retina Efficient Coding - tf [private for now]
Jimmy/Yifei - LogDet - tf [Jimmy will submit pull request to tensorFlow folks]
Olivier - steerablePyramid - pt [Nikhil has a forked version of Olivier’s code]
Billy - MAD competition - tf [Billy will re-write in pyTorch]

Meeting 1: May 30 (Eero was out of town)

The stability flexibility trade-off

LCV GitHub organization
In the spirit of healthy and reproducible research we aim for a clean Lab GitHub page containing repositories with well documented and functional code.
Billy - Set GitHub organization, partition into public / private repositories, give access to lab members

Choice of Software
We generally lean in the direction of pyTorch (pt) over TensorFlow (tf) because of its flexibility (note that version 1.0 of pt will be released in the coming weeks)
We think it should be possible to make up for the missing elements from this library (imaginary numbers, FFT, Lop Rop, etc)

INVENTORY

Ready
Content on the LCV website
NLP distance- theano
GDN coder - m
Content of share/lcv/toolbox

Jimmy - Retina Efficient Coding - tf
Jimmy Yifei - LogDet - tf
Olivier - steerablePyramid - pt
Billy - MAD competition - tf (maybe pt)

In progress
Jimmy Kede - Gaussianization - tf (maybe pt)
Manu - MT model and motion cloud stimuli - python
Billy - pypyrtools in python 3
PE - speed comparison C vs pt
Nikhil - optimization for textures - pt
Paul - V1 normalization model, based off of Goris - python and tf

In question
Alex - eigendistortion compatible with differentiable representation models
Olivier - geodesics optimization
Caro - Modulated poisson
Manu - convolutional LNP - m and pt
Manu - auditory textures (already on McDermott web?)
Manu - CPB - m

Next meetings Thursdays at 2pm.
From now on planning and notes from meeting will be on the Github wiki.
After the first wave (pushing code that is ready), these meetings will be an occasion to get coding help from one another

Clone this wiki locally