How to customize connections between layers

[ghanimmustafa]

I am exploring an algorithim that selects only the most critical connections between layers. For example we are considering this baseline network:
What would be the best approach to still keep supporting HLS4ML flow and adjust the connections (by removing or keeping some of them) between each layer?

# Define the architecture based on the provided specifications
def build_custom_model(input_shape=(28, 28, 1)):
    model = models.Sequential()

    # First 2D Convolutional Layer
    model.add(layers.Conv2D(filters=6, kernel_size=7, strides=2, padding='valid', activation='relu', input_shape=input_shape))
    # Output shape: (11, 11, 6) -> 726 total outputs (11*11*6)

    # Second Convolutional Layer (Simulating a 3D convolution)
    model.add(layers.Conv2D(filters=50, kernel_size=7, strides=4, padding='valid', activation='relu'))
    # Output shape: (2, 2, 50) -> 200 total outputs (2*2*50)
    
    # Flatten layer to convert 3D outputs to 1D
    model.add(layers.Flatten())  # Output shape: (200)

    # Fully Connected Layer with 20 neurons
    model.add(layers.Dense(20, activation='relu'))  # Output shape: (20)

    # Classifier Layer with 10 output neurons
    model.add(layers.Dense(10, activation='softmax'))  # Output shape: (10)

    return model

[jmitrevs]

Zero weights can be removed by the compiler when synthesizing,

[ghanimmustafa]

How to achieve this? My NN with zero weights in a dense layer is estimated by HLS to have exactly the same resources of the same network without zero weights.

[jmitrevs]

This is done automatically, but it's not always possible. If you use a reuse factor of 1, then it's straightforward to remove the multiplier. If you use a different reuse factor, then all the weights that map to that multiplier need to be 0 for the multiplier to be removed.

[ghanimmustafa]

I think you are right. I'm now stuck in the dilemma of using reuse factor of 1 and getting this error:
INFO: [HLS 200-489] Unrolling loop 'Accum2' (firmware/nnet_utils/nnet_dense_latency.h:56) in function 'nnet::dense_wrapper<ap_fixed<32, 16, (ap_q_mode)4, (ap_o_mode)0, 0>, ap_fixed<32, 16, (ap_q_mode)4, (ap_o_mode)0, 0>, config9>' completely with a factor of 10. INFO: [HLS 200-489] Unrolling loop 'Result' (firmware/nnet_utils/nnet_dense_latency.h:64) in function 'nnet::dense_wrapper<ap_fixed<32, 16, (ap_q_mode)4, (ap_o_mode)0, 0>, ap_fixed<32, 16, (ap_q_mode)4, (ap_o_mode)0, 0>, config9>' completely with a factor of 10. INFO: [HLS 200-489] Unrolling loop 'Product1' (firmware/nnet_utils/nnet_dense_latency.h:37) in function 'nnet::dense_wrapper<ap_fixed<32, 16, (ap_q_mode)4, (ap_o_mode)0, 0>, ap_fixed<32, 16, (ap_q_mode)4, (ap_o_mode)0, 0>, config7>' completely with a factor of 450. ERROR: [XFORM 203-504] Stop unrolling loop 'Product1' (firmware/nnet_utils/nnet_dense_latency.h:37) in function 'nnet::dense_wrapper<ap_fixed<32, 16, (ap_q_mode)4, (ap_o_mode)0, 0>, ap_fixed<32, 16, (ap_q_mode)4, (ap_o_mode)0, 0>, config7>' because it may cause large runtime and excessive memory usage due to increase in code size. Please avoid unrolling the loop or form sub-functions for code in the loop body.

Or using a very large reuse factor (e.g, > 4000) to get HLS estimiations working but without showing any benefit or pruning the neural network. I may try next to implement each layer or some layers separately to see if it will pass the HLS scheduling.

[bo3z]

Several additional comments, in addition to @jmitrevs comments: hls4ml will always optimise away zero weights (connections) when Reuse Factor (RF) = 1.

When RF > 1 and you use Latency strategy; there will be some optimisations but the number of DSPs used isn't quite predictable until after HLS synthesis because HLS re-orders multiplications when all of the inputs (in this case weights and inputs) are in registers. This is done automatically by the HLS compiler to balance the expression complexity. So in short, with Latency strategy and RF > 1, there is no guarantee of the multiplication order within a layer, but high-sparsity unstructured pruning should still help.

For Resource strategy and RF > 1, weights are stored in memory and the order of multiplication is well-known before compilation. In this case, you can do some structured pruning, such as hardware-aware pruning: https://arxiv.org/abs/2308.05170 which is also part of hls4ml (see the documentation under docs/advanced/) (key here is to use the DenseUnrolled strategy, as indicated by the docs). Note, however, this approach doesn't scale very well with many parameters.

A couple of comments on the code snipped above:

• I find that Vitis HLS is slightly better at unrolling complex loops; I managed to unroll around 16k multiplications with Vitis HLS (but with a low precision), so maybe consider using Vitis HLS instead of Vivado HLS.
• I don't expect that synthesising individual layers will help a lot. You will get results quicker, indeed, but the layers are synthesised sequentially by the HLS compiler. So if a layer gets stuck as part of the full network synthesis it is also likely to get stuck if synthesised individually. Also, when synthesising individual layers its important to keep in mind that once the layers are stitched together they don't exceed the resources of the FPGA. But, yes, synthesising individual layers for some design space exploration can be useful.

Finally, quantisation-aware training can really help reduce the resource footprint of a model on an FPGA, and it doesn't depend on the RF. For more information, see tutorial: https://github.com/fastmachinelearning/hls4ml-tutorial/blob/main/part4_quantization.ipynb

[bo3z]

What you are referring to is pruning and as @jmitrevs points out, hls4ml will optimise away zero weights (connections) in the Latency strategy. There are many different ways of pruning a neural network; the most basic one is using TensorFlow Model Optimization, which is linked with hls4ml in this tutorial: https://github.com/fastmachinelearning/hls4ml-tutorial/blob/main/part3_compression.ipynb. You could also look into more advanced techniques, such as hardware-aware pruning: https://arxiv.org/abs/2308.05170 which is also part of hls4ml (see the documentation under docs/)