Unsupported SFP+ Module

[JaredLThompson]

I am starting this thread to troubleshoot an unidentified SFP Module.

I have a Finisar fcbn510qe2c03, which is a QSFP that fans out to 4 10gbps SFP+ Transceivers. The QSW-2104-2S does not link the transceiver. I connected a FTDI to the serial header to troubleshoot this issue.

When I connect the non-supported module, I do indeed get an error message:

BCMCLI>  not found, use the first 16 bytes! 
fport=12, data[0]=ff, data[1]=04, data[3]=00, data[8]=08, data[12]=67 

Another SFP+ module in the same cable also brings up the same message:

not found, use the first 16 bytes! 
fport=12, data[0]=ff, data[1]=04, data[3]=00, data[8]=08, data[12]=67 

I then plugged in two other supported SFP/SFP+ Modules to get the corresponding message:

10gbps SFP+ MM Transceiver:

BCMCLI> found! i=0 
fport=12, data[0]=03, data[1]=04, data[3]=10, data[8]=00, data[12]=67 

1gbps SFP MM Transceiver:

BCMCLI> found! i=0 
fport=12, data[0]=03, data[1]=04, data[3]=00, data[8]=00, data[12]=0d 

It appears that this is reading the SFP management data that identifies the transceiver capabilities, documented https://members.snia.org/document/dl/25916

The first byte, data[0], corresponds to the Identifier.

• 0x03 is SFP or SFP+
• 0xff is "vendor specific"

I believe that the switch is discarding this because it does not recognize the FF identifier. The other bytes correspond to link speed, transmission medium, etc, and seem like they are all supported.

I will be looking at the source code mentioned in other issues attached to this repo to see if there is a way to get the switch to accept the FF transceiver identification.

[danieltwagner]

Have you tried connecting another 10g DAC and seeing if that works?
Based on my spelunking back when I set up this repo it might be possible to directly modify the binary to accept your DAC similar to how I modified the timeout but obviously building firmware from source would be dramatically better.

[danieltwagner]

It would appear that it's specifically looking for 0x0304 as a byte sequence. Here's how you could obtain a modified binary to flash onto your device that will look only for 0xFF04 in the metadata, and hence will not work with regular SFP or SFP+ modules, might brick your QSFP breakout or the switch itself, eat you alive or spontaneously combust. Or maybe it'll work for your particular cable. You've been warned.

$ dd skip=393216 iflag=skip_bytes bs=3092020 count=1 if=flash-qsw-2104-2s.bin of=flash1.bin
$ cp flash1.bin qsfp.bin
$ printf '\xff' | dd of=qsfp.bin bs=1 seek=$((0xc0fda)) conv=notrunc
$ diff <(xxd flash1.bin) <(xxd qsfp.bin)
49406c49406
< 000c0fd0: 8342 4ed0 5c1a 13f8 012b 032a f8d1 a3eb  .BN.\....+.*....
---
> 000c0fd0: 8342 4ed0 5c1a 13f8 012b ff2a f8d1 a3eb  .BN.\....+.*....
$ python3 img_to_fw_update.py qsfp.bin SW-UT2206QG_1.00-0.07-qsfp

[JaredLThompson]

I agree that it is looking for 0x0304 (0x03=sfp/sfp+; 0x04=2 wire interface).

I am sure it will rip a hole in the very fabric of space and time, but I will probably take the chance to see if it does indeed recognize the transceiver.

I would like to build the firmware from source, but I am not exactly sure what source it was build from. I am reading through #1 which has a lot of good info.

I will let you know the results of the testing the modified binary. :-)

[JaredLThompson]

So the good news is that this experiment didn't result in a black hole being formed around the QNAP.

As expected, the QSFP is now 'found'

BCMCLI> found! i=0 
fport=13, data[0]=ff, data[1]=04, data[3]=00, data[8]=08, data[12]=67 
found! i=0 
fport=12, data[0]=ff, data[1]=04, data[3]=00, data[8]=08, data[12]=67 

And the sfp+ is not:

BCMCLI> not found, use the first 16 bytes! 
fport=12, data[0]=03, data[1]=04, data[3]=10, data[8]=00, data[12]=67 
not found, use the first 16 bytes! 
fport=13, data[0]=03, data[1]=04, data[3]=10, data[8]=00, data[12]=67 

However, everything still behaves the same way; the SFP+ links and passes traffic, the QSFP does not link. So there is more to it than identifying the module. :-) It was a fun experiement. I will have to look at building firmware from source if I want to use this QSFP. (Or maybe it is likely that the Broadcom hardware just doesn't recognize it)

I think that this 'issue' can be closed. Thanks for providing the instructions on how to replace that lookup, you definitely found the right part of the firmware.

[JaredLThompson]

Closing this, I thought it was a simple swap of values, there is more to it than that. Will need to compile firmware from source to get the QSFP module to recognize.

[danieltwagner]

I take it there weren't any other helpful logs that appeared and would hint at what went wrong? Agreed that building from source is likely the way forward... trying to go deep in the disassembled binary is going to be quite painful.

[JaredLThompson]

I am still looking through the logs, figuring out how the sfp_get command works, etc. It was surprising that the SFP+'s still worked, even though it stated "Not found". I am working through Gidra, and am looking at the "cbx_link_scan_init" routine, label at 1015684c.