python pass registration

[makslevental]

This PR demonstrates linking C++ directly to the python bindings (without going through the C API).

In fact it demonstrates something more exciting: tunneling python callbacks all the way into MLIR passes:

def print_ops(op):
    print(op.name)

aie.dialects.aie.register_python_pass_demo_pass(print_ops)

module = """
module {
  %12 = AIE.tile(1, 2)
  %buf = AIE.buffer(%12) : memref<256xi32>
  %4 = AIE.core(%12)  {
    %0 = arith.constant 0 : i32
    %1 = arith.constant 0 : index
    memref.store %0, %buf[%1] : memref<256xi32>
    AIE.end
  }
}
"""

mlir_module = Module.parse(module)
PassManager.parse("builtin.module(python-pass-demo)").run(
        mlir_module.operation
 )

This will print

AIE.tile
AIE.buffer
arith.constant
arith.constant
memref.store
AIE.end
AIE.core
builtin.module

by calling print_ops from inside the pass (where there's a this->getOperation()->walk([this](Operation *op) { func(wrap(op)); });).

The only thing that needs to be changed for this to work is to build MLIR with RTTI. This sounds scary (maybe?) but it doesn't seem to matter at all (my wheels are built and tested with RTTI now for months...).

In the future (very soon) this pass will do more than just demonstrate the capability (at which point it will be renamed appropriately).

Note, this is stacked on top of #711 so be sure to exclude the first commit from the diff view.

[stephenneuendorffer]

The only thing that needs to be changed for this to work is to build MLIR with RTTI. This sounds scary (maybe?) but it doesn't seem to matter at all (my wheels are built and tested with RTTI now for months...).

I don't think there's a real problem compiling with RTTI: It just adds overhead that LLVM is mostly designed to avoid. IIRC there might also be some other c++ python interop problems, I seem to remember discussing with Stella at one point.

[makslevental]

I don't think there's a real problem compiling with RTTI: It just adds overhead that LLVM is mostly designed to avoid. IIRC there might also be some other c++ python interop problems, I seem to remember discussing with Stella at one point.

I was under the same impression but after chatting with @stellaraccident offline about this, I think the consensus is it's best to avoid it if not needed. Well good news! Turns out you can indeed avoid it - the current version puts the pass behind a "blood-brain" barrier (just a separate TU with a pure Python C API interface) and compiles the object file with -fno-rtti. Locally it works without compiling upstream with RTTI enabled but let's see how the test here shakes out.

[stellaraccident]

Just don't cross the streams. Commingling c++ code compiled with and without it for the same types is a recipe for bad things.

I doubt you'll get support for running rtti builds upstream to test. Ditto in iree.

[makslevental]

@AndraBisca @fifield I would like to merge this.

Can you please take a look because you both are working on the bindings and tell me if there's anything you don't understand. Additionally, I have tested this locally (on Mac and Linux, the latter with with chess) but not Windows nor with Peano. If either of you could test locally to double check that would be helpful.

[AndraBisca]

This looks really good! Thank you for all the helpful comments you added in the files as well!

[fifield]

I guess this makes some assumptions about directory layout that I don't follow since I get a failure here. Is it possible to get cmake or something else to fill in the path?

[makslevental]

that directory is CMAKE_BIN/lib but but you need to build LLVM/MLIR with the execution engine enabled for them to be copied to that location

mlir-aie/python/CMakeLists.txt

Line 236 in 8ed3f7b

${CMAKE_BINARY_DIR}/lib

[makslevental]

I can gate this test by availability of the execution engine.

[makslevental]

Also it's possible if you built without DCMAKE_PLATFORM_NO_VERSIONED_SONAME=ON then the .so got version and the script doesn't pick it up (fixed in #771).