Add orthogonal initialization

[noahfarr]

Hi,

I would like to use orthogonal initialization as I would like to use it for a project of mine. Is it planned to add it in the near future?

[awni]

There is no plan that I know of to add it. If you are up for it you can send a PR. Otherwise we can add it soon.. I don't think it should be too difficult.

[noahfarr]

Its not trivial as you need e.g. qr decomposition and idk if thats implemented in mlx, so you either have to use some third party package or implement that as well

[awni]

We have a QR decomposition. The main limitation is it runs on the CPU.. so that part of the initialization needs to run on the CPU.

[noahfarr]

Ah very nice. Then I maybe will put together a pull request in a few days