U280 build static failed when using DDR memory

[qianyich]

Hi Coyote Maintainers:

I am a PhD student from Northeastern University in Boston, US. I am considering using Coyote RDMA part in our project to communicate with Mellanox RDMA NIC. I am currently trying the static shell, but I am encountering the following errors.

Vivado: 2022.1
Board: Alveo U280

Step to reproduce:

1. comment out "set(HBM_SPLIT 1)" in CMakeLists.txt in examples_hw
2. remove u280 from the HBM_DEV and add it into DDR_DEV in FIndCoyoteHW.cmake
3. add set(N_DDR_CHAN 1) in elseif(FDEV_NAME STREQUAL "u280") block in FIndCoyoteHW.cmake based on Error Building Static Application using CMake (U250) #74 (comment)

Error message when flow_comp.tcl is executed for shell compilation:

** Starting shell compilation ...
**
Command: open_checkpoint /home/qianyich/Coyote/examples_hw/build_hw_ddr/checkpoints/shell_linked.dcp

Starting open_checkpoint Task

Time (s): cpu = 00:00:00.05 ; elapsed = 00:00:00.03 . Memory (MB): peak = 2724.719 ; gain = 5.938 ; free physical = 76417 ; free virtual = 90001
INFO: [Device 21-403] Loading part xcu280-fsvh2892-2L-e
Netlist sorting complete. Time (s): cpu = 00:00:07 ; elapsed = 00:00:08 . Memory (MB): peak = 4748.578 ; gain = 77.000 ; free physical = 74409 ; free virtual = 87993
INFO: [Netlist 29-17] Analyzing 14880 Unisim elements for replacement
INFO: [Netlist 29-28] Unisim Transformation completed in 3 CPU seconds
WARNING: [Netlist 29-1115] Found multi-term driver net: inst_static/inst_int_static/util_ds_buf/U0/<const0>.
INFO: [Project 1-479] Netlist was created with Vivado 2022.1
INFO: [Project 1-570] Preparing netlist for logic optimization
WARNING: [Constraints 18-550] Could not create 'IBUF_LOW_PWR' constraint because net 'inst_shell/xclk' is not directly connected to top level port. Synthesis is ignored for IBUF_LOW_PWR but preserved for implementation.
WARNING: [Constraints 18-550] Could not create 'IBUF_LOW_PWR' constraint because net 'inst_static/inst_int_static/clk_wiz_0/clk_in1' is not directly connected to top level port. Synthesis is ignored for IBUF_LOW_PWR but preserved for implementation.
INFO: [Timing 38-478] Restoring timing data from binary archive.
INFO: [Timing 38-479] Binary timing data restore complete.
INFO: [Project 1-856] Restoring constraints from binary archive.
INFO: [Power 33-23] Power model is not available for xiphy_riu_or
INFO: [Power 33-23] Power model is not available for genVref.u_hpio_vref
INFO: [Timing 38-35] Done setting XDC timing constraints.
INFO: [Constraints 18-4422] SNAPPING_MODE is set so implementation will use the DERIVED_RANGES which may be different then the GRID_RANGES.
INFO: [Constraints 18-4421] A LAGUNA range, without usable sites, was added to DERIVED_RANGES for purposes of keeping the Programmable Unit together within the reconfigurable pblock.
INFO: [Project 1-853] Binary constraint restore complete.
Reading XDEF placement.
Reading placer database...
Reading XDEF routing.
Read XDEF Files: Time (s): cpu = 00:00:00.73 ; elapsed = 00:00:00.75 . Memory (MB): peak = 9436.719 ; gain = 0.000 ; free physical = 70266 ; free virtual = 83850
Restored from archive | CPU: 0.750000 secs | Memory: 4.954178 MB |
Finished XDEF File Restore: Time (s): cpu = 00:00:00.73 ; elapsed = 00:00:00.75 . Memory (MB): peak = 9436.719 ; gain = 0.000 ; free physical = 70266 ; free virtual = 83850
INFO: [IP_Flow 19-234] Refreshing IP repositories
INFO: [IP_Flow 19-1704] No user IP repositories specified
INFO: [IP_Flow 19-2313] Loaded Vivado IP repository '/tools/Xilinx/Vivado/2022.1/data/ip'.
Netlist sorting complete. Time (s): cpu = 00:00:00.11 ; elapsed = 00:00:00.11 . Memory (MB): peak = 9436.719 ; gain = 0.000 ; free physical = 70374 ; free virtual = 83958
INFO: [Project 1-111] Unisim Transformation Summary:
  A total of 3321 instances were transformed.
  CFGLUT5 => CFGLUT5 (SRL16E, SRLC32E): 352 instances
  DSP48E1 => DSP48E2 (DSP_ALU, DSP_A_B_DATA, DSP_C_DATA, DSP_MULTIPLIER, DSP_M_DATA, DSP_OUTPUT, DSP_PREADD, DSP_PREADD_DATA): 3 instances
  IBUFDS => IBUFDS (DIFFINBUF, IBUFCTRL): 1 instance
  IOBUFDS => IOBUFDS (DIFFINBUF, IBUFCTRL, INV, OBUFT(x2)): 18 instances
  IOBUFE3 => IOBUFE3 (IBUFCTRL, INBUF, OBUFT_DCIEN): 72 instances
  LUT6_2 => LUT6_2 (LUT5, LUT6): 31 instances
  OBUFDS => OBUFDS_DUAL_BUF (INV, OBUF(x2)): 1 instance
  RAM16X1D => RAM32X1D (RAMD32(x2)): 7 instances
  RAM256X1S => RAM256X1S (MUXF7(x2), MUXF8, RAMS64E(x4)): 9 instances
  RAM32M => RAM32M (RAMD32(x6), RAMS32(x2)): 336 instances
  RAM32M16 => RAM32M16 (RAMD32(x14), RAMS32(x2)): 2089 instances
  RAM32X1D => RAM32X1D (RAMD32(x2)): 169 instances
  RAM64M => RAM64M (RAMD64E(x4)): 9 instances
  RAM64M8 => RAM64M8 (RAMD64E(x8)): 224 instances

INFO: [Project 1-604] Checkpoint was created with Vivado v2022.1 (64-bit) build 3526262
open_checkpoint: Time (s): cpu = 00:06:15 ; elapsed = 00:03:21 . Memory (MB): peak = 9436.719 ; gain = 6720.906 ; free physical = 70372 ; free virtual = 83957
Command: opt_design -directive Explore
INFO: [Vivado_Tcl 4-136] Directive used for opt_design is: Explore
Attempting to get a license for feature 'Implementation' and/or device 'xcu280'
INFO: [Common 17-349] Got license for feature 'Implementation' and/or device 'xcu280'
Running DRC as a precondition to command opt_design

Starting DRC Task
INFO: [DRC 23-27] Running DRC with 8 threads
INFO: [Project 1-461] DRC finished with 0 Errors, 108 Warnings
INFO: [Project 1-462] Please refer to the DRC report (report_drc) for more information.

Time (s): cpu = 00:00:40 ; elapsed = 00:00:34 . Memory (MB): peak = 9514.406 ; gain = 77.688 ; free physical = 70170 ; free virtual = 83765

Starting Logic Optimization Task

Phase 1 Generate And Synthesize MIG Cores
ERROR: [Mig 66-99] Memory Core Error - [inst_shell/inst_int_ddr/ddr4_0] Either port(s) c0_sys_clk_p, c0_sys_clk_n is/are not placed or un-supported clocking structure/circuit for memory ip instance. Please refer to clocking section of PG150 for supported clocking structures.
ERROR: [Mig 66-99] Memory Core Error - [inst_shell/inst_int_ddr/ddr4_0] Port(s) c0_sys_clk_n,c0_sys_clk_p is/are not placed. Assign all ports to valid sites.
Phase 1 Generate And Synthesize MIG Cores | Checksum: 1ec633aea

Time (s): cpu = 00:00:00.35 ; elapsed = 00:00:00.38 . Memory (MB): peak = 9514.406 ; gain = 0.000 ; free physical = 69960 ; free virtual = 83555
INFO: [Common 17-83] Releasing license: Implementation
25 Infos, 3 Warnings, 0 Critical Warnings and 3 Errors encountered.
opt_design failed
** CERR: ERROR: [Opt 31-306] MIG/Advanced IO Wizard Cores generation Failed.

**
INFO: [Common 17-206] Exiting Vivado at Fri Sep 20 12:11:50 2024...

I double-checked the ddr_0 constraints for u280 and I think it is all correct, but I have no idea why the tool throws this error.

[qianyich]

The problem seems to be solved by removing hbm signals in cyt_top template and shell_top template. I will update if that works.

[qianyich]

Everything was built successfully. Then I loaded the .bit file through JTAG. I compiled and inserted the driver module successfully. I also compiled the sw program successfully, but running it gave me the following error. I ran the program with sudo privilege.

Why is the number of regions equal to 4? I just set N_REGIONS to 1. Also, I have several bitstreams under bitstream folder, which one should I load?

qianyich:[~/Coyote/examples_hw/build_hw_ddr]$ ls bitstreams/
cyt_top.bin  cyt_top.bit  cyt_top.ltx  cyt_top_pblock_inst_shell_partial.bin  cyt_top_pblock_inst_shell_partial.bit  cyt_top_pblock_inst_shell_partial.ltx

root@rcl-server1:/home/qianyich/Coyote/examples_sw/build_sw/bin# ./test


-- PARAMS
-----------------------------------------------
Number of regions: 4
Hugepages: 1
Mapped pages: 1
Streaming: HOST
Number of repetitions (thr): 10000
Number of repetitions (lat): 100
Starting transfer size: 1024
Ending transfer size: 1048576

terminate called after throwing an instance of 'std::runtime_error'
  what():  bThread could not be obtained, vfid: 1

dmesg after running program:

[  997.271890] fpga device 0 opened, spid 7125, ref cnt 0
[  997.271894] registration succeeded, cpid 2, hpid 7125, spid 7125
[  997.271895] fpga_ioctl():reading config 0x100020000000f
[  997.271898] fpga_mmap():fpga dev. 0, memory mapping config region at 382fe0130000 of size 10000
[  997.271903] fpga_mmap():fpga dev. 0, memory mapping user ctrl region at 382fe0120000 of size 10000
[  997.271905] x86/PAT: reserve_ram_pages_type failed [mem 0xa6b7a7000-0xa6b7a7fff], track 0x2, req 0x2
[  997.271906] fpga_mmap():fpga dev. 0, memory mapping writeback regions at fffff000 of size 400
[  997.271923] mmu_handler_gup():hpid found = 7125
[  997.271925] mmu_handler_gup():map not present
[  997.271933] tlb_get_user_pages():allocated 4096 bytes for page pointer array for 512 pages @0x00000000e689afe9
[  997.271934] tlb_get_user_pages():pages=0x00000000e689afe9
[  997.450225] tlb_get_user_pages():get_user_pages_remote(7f1dc0000, n_pages = 512, page start = 17c0000, hugepages = 1)
[  997.450236] card_alloc():user card buffer allocated @ 40400000, n_pages 512, huge 1, device 0
[  997.450237] tlb_create_map():creating new TLB entry, vaddr_fa 7f1dc0000, paddr 17c0000000, strm 1, cpid 2, hpid 7125, hugepage 1
[  997.450239] fpga_ioctl():user mapping vFPGA 0 handled
[  997.535862] tlb_create_unmap():unmapping TLB entry, vaddr_fa 7f1dc0000, hpid 7125, hugepage 1

dmesg after insert the driver module:

[  604.406531] loading: Coyote PCIe driver ...
[  604.406598] probe (pdev = 0x00000000f4c5b747, pci_id = 0x00000000a03fef92)
[  604.406710] pci device node 000000006467c74f enabled
[  604.406710] fpga device 0, pci bus 3b, slot 00
[  604.406720] pci bus master capability enabled
[  604.406724] enabling MSI-X
[  604.406912] obtained 22 MSI-X irqs
[  604.406912] pci request regions
[  604.406914] pci regions obtained
[  604.406915] mapping BAR 0, 1048576 bytes to be mapped
[  604.406921] BAR0 at 0x382ff0000000 mapped at 0x000000002bfc2af3, length=1048576, (1048576)
[  604.406922] BAR #1 is not present
[  604.406922] mapping BAR 2, 65536 bytes to be mapped
[  604.406926] BAR2 at 0xab000000 mapped at 0x00000000d5ad0dd1, length=65536, (65536)
[  604.406926] BAR #3 is not present
[  604.406927] mapping BAR 4, 268435456 bytes to be mapped
[  604.406935] BAR4 at 0x382fe0000000 mapped at 0x00000000ea4921a7, length=268435456, (268435456)
[  604.406936] sizeof(dma_addr_t) == 8
[  604.406936] probing for h2c engine 0 at 00000000d5ad0dd1
[  604.406938] engine ID = 0x1fc0, channel ID = 0
[  604.406938] found h2c 0 engine at 00000000d5ad0dd1
[  604.406940] engine 000000007f18f5ee name h2c alignments=0x00010140
[  604.406941] engine_create():ioread32(0x00000000d5b572f8) = 0x00000001 (dummy read flushes writes).
[  604.406941] probing for h2c engine 1 at 00000000e66c0f5c
[  604.406943] engine ID = 0x1fc0, channel ID = 1
[  604.406943] found h2c 1 engine at 00000000e66c0f5c
[  604.406944] engine 000000007f54713e name h2c alignments=0x00010140
[  604.406945] engine_create():ioread32(0x00000000b07d0cf3) = 0x00000001 (dummy read flushes writes).
[  604.406946] probing for h2c engine 2 at 0000000059876676
[  604.406947] engine ID = 0x1fc0, channel ID = 2
[  604.406947] found h2c 2 engine at 0000000059876676
[  604.406948] engine 00000000301d06cb name h2c alignments=0x00010140
[  604.406949] engine_create():ioread32(0x00000000d450d92e) = 0x00000001 (dummy read flushes writes).
[  604.406950] probing for c2h engine 0 at 0000000096afe810
[  604.406951] engine ID = 0x1fc1, channel ID = 0
[  604.406952] found c2h 0 engine at 0000000096afe810
[  604.406953] engine 00000000599ca134 name c2h alignments=0x00010140
[  604.406954] engine_create():ioread32(0x0000000055051458) = 0x00000001 (dummy read flushes writes).
[  604.406954] probing for c2h engine 1 at 000000008b61b1e3
[  604.406955] engine ID = 0x1fc1, channel ID = 1
[  604.406956] found c2h 1 engine at 000000008b61b1e3
[  604.406957] engine 0000000037d007a0 name c2h alignments=0x00010140
[  604.406958] engine_create():ioread32(0x000000003e96b39e) = 0x00000001 (dummy read flushes writes).
[  604.406958] probing for c2h engine 2 at 000000009d36d05d
[  604.406959] engine ID = 0x1fc1, channel ID = 2
[  604.406960] found c2h 2 engine at 000000009d36d05d
[  604.406961] engine 000000002305b530 name c2h alignments=0x00010140
[  604.406962] engine_create():ioread32(0x0000000058316cb5) = 0x00000001 (dummy read flushes writes).
[  604.406962] found 6 engines
[  604.406969] deployment static probe 000ff1ce
[  604.406970] deployment shell probe 000ff1ce
[  604.406972] detected 1 virtual FPGA regions, 2 FPGA channels
[  604.406974] enabled AVX 1, enabled writeback 1
[  604.406976] sTLB order 10, sTLB assoc 4, sTLB page size 4096
[  604.406979] lTLB order 9, lTLB assoc 2, lTLB page size 2097152
[  604.406981] enabled host streams 1, enabled card streams (mem) 1
[  604.406982] enabled dynamic reconfiguration 0
[  604.406983] enabled RDMA 0, port 0
[  604.406984] enabled TCP/IP 0, port 0
[  604.406985] creating sysfs entry ...
[  604.412920] reconfig device regions allocated, major number 502
[  604.412954] allocated memory for reconfig device
[  604.413001] reconfiguration device created
[  604.413002] reconfiguration device added
[  604.413003] vFPGA device regions allocated, major number 501
[  604.413011] allocated memory for fpga devices
[  604.413133] allocated memory for descriptor writeback, vaddr ffff8b2f6b7a7000, paddr fffff000
[  604.413165] virtual FPGA device 0 created
[  604.413185] all virtual FPGA devices added
[  604.413205] using IRQ#97 with vFPGA 0
[  604.413214] using IRQ#120 with reconfiguration device
[  604.413216] ioread32(0x000000000327547f) returned 0x00000000 (user_int_request).
[  604.413217] ioread32(0x00000000c40215b5) returned 0x00000000 (channel_int_request)
[  604.413218] probe returning 0

[JonasDann]

Hi, glad you want to use Coyote for your work. Number of regions is set to 4 as a default for this example software application. The runtime error sounds like it can't open all the regions it expects which makes sense if you set the number of regions to 1 for your synthesis. Could you try passing the correct number of regions as a parameter?

[qianyich]

When I try to install the driver, it gave me the following:

[ 2678.408767] loading: Coyote PCIe driver ...
[ 2678.408838] probe (pdev = 0x00000000211ac513, pci_id = 0x000000004920a4d0)
[ 2678.408946] pci device node 00000000d071311d enabled
[ 2678.408947] fpga device 0, pci bus 3b, slot 00
[ 2678.408957] pci bus master capability enabled
[ 2678.408960] enabling MSI-X
[ 2678.409158] obtained 22 MSI-X irqs
[ 2678.409159] pci request regions
[ 2678.409161] pci regions obtained
[ 2678.409162] mapping BAR 0, 1048576 bytes to be mapped
[ 2678.409168] BAR0 at 0x382ff0000000 mapped at 0x000000004a4ddfc4, length=1048576, (1048576)
[ 2678.409168] BAR #1 is not present
[ 2678.409168] mapping BAR 2, 65536 bytes to be mapped
[ 2678.409172] BAR2 at 0xab000000 mapped at 0x000000009c11737d, length=65536, (65536)
[ 2678.409173] BAR #3 is not present
[ 2678.409173] mapping BAR 4, 268435456 bytes to be mapped
[ 2678.409312] BAR4 at 0x382fe0000000 mapped at 0x00000000e5a0870a, length=268435456, (268435456)
[ 2678.409313] sizeof(dma_addr_t) == 8
[ 2678.409313] probing for h2c engine 0 at 000000009c11737d
[ 2678.409315] engine ID = 0x1fc0, channel ID = 0
[ 2678.409315] found h2c 0 engine at 000000009c11737d
[ 2678.409316] engine 0000000072c6ed49 name h2c alignments=0x00010140
[ 2678.409318] engine_create():ioread32(0x00000000d291ae46) = 0x00000001 (dummy read flushes writes).
[ 2678.409319] probing for h2c engine 1 at 0000000084f2982f
[ 2678.409320] engine ID = 0x1fc0, channel ID = 1
[ 2678.409321] found h2c 1 engine at 0000000084f2982f
[ 2678.409322] engine 0000000081133421 name h2c alignments=0x00010140
[ 2678.409323] engine_create():ioread32(0x00000000844b6161) = 0x00000001 (dummy read flushes writes).
[ 2678.409323] probing for h2c engine 2 at 00000000867c618d
[ 2678.409324] engine ID = 0x1fc0, channel ID = 2
[ 2678.409325] found h2c 2 engine at 00000000867c618d
[ 2678.409326] engine 00000000908ea376 name h2c alignments=0x00010140
[ 2678.409327] engine_create():ioread32(0x00000000c203b279) = 0x00000001 (dummy read flushes writes).
[ 2678.409327] probing for c2h engine 0 at 00000000a74e27b6
[ 2678.409329] engine ID = 0x1fc1, channel ID = 0
[ 2678.409329] found c2h 0 engine at 00000000a74e27b6
[ 2678.409330] engine 00000000237f0c4a name c2h alignments=0x00010140
[ 2678.409331] engine_create():ioread32(0x000000008fc18df9) = 0x00000001 (dummy read flushes writes).
[ 2678.409332] probing for c2h engine 1 at 00000000d8796399
[ 2678.409333] engine ID = 0x1fc1, channel ID = 1
[ 2678.409334] found c2h 1 engine at 00000000d8796399
[ 2678.409335] engine 00000000c686cd5d name c2h alignments=0x00010140
[ 2678.409336] engine_create():ioread32(0x00000000d57f2577) = 0x00000001 (dummy read flushes writes).
[ 2678.409336] probing for c2h engine 2 at 0000000032afad0e
[ 2678.409337] engine ID = 0x1fc1, channel ID = 2
[ 2678.409338] found c2h 2 engine at 0000000032afad0e
[ 2678.409339] engine 000000007495fec0 name c2h alignments=0x00010140
[ 2678.409340] engine_create():ioread32(0x00000000883b34e5) = 0x00000001 (dummy read flushes writes).
[ 2678.409340] found 6 engines
[ 2678.409347] deployment static probe 000ff1ce
[ 2678.569048] deployment shell probe ffffffff
[ 2678.919859] detected -1 virtual FPGA regions, -1 FPGA channels
[ 2679.270685] enabled AVX 1, enabled writeback 1
[ 2679.796936] ------------[ cut here ]------------
[ 2679.796937] kernel BUG at /home/qianyich/Coyote/driver/fpga_dev.c:119!
[ 2679.796957] invalid opcode: 0000 [#1] SMP NOPTI
[ 2679.796969] CPU: 0 PID: 4051 Comm: kworker/0:0 Tainted: G           OE     5.4.0-150-generic #167~18.04.1-Ubuntu
[ 2679.796988] Hardware name: Dell Inc. PowerEdge R740xd/04FC42, BIOS 2.4.8 11/26/2019
[ 2679.797007] Workqueue: events work_for_cpu_fn
[ 2679.797021] RIP: 0010:read_shell_config+0x113/0x42b [coyote_drv]
[ 2679.797035] Code: 83 e6 0f 48 89 70 30 48 8b 51 18 48 c1 ea 08 83 e2 0f 89 50 20 48 8b 49 18 48 c1 e9 14 83 e1 3f 48 89 48 08 48 83 f9 0c 74 02 <0f> 0b 41 bc 01 00 00 00 89 f1 48 c7 40 10 00 10 00 00 4c 89 e7 48
[ 2679.797069] RSP: 0018:ffffab13c8047dd8 EFLAGS: 00010206
[ 2679.797081] RAX: ffff940f40f822a0 RBX: ffff94023cc79418 RCX: 000000000000003f
[ 2679.797096] RDX: 000000000000000f RSI: 000000000000000f RDI: ffff940350c07480
[ 2679.797110] RBP: ffffab13c8047df8 R08: ffff9403510350c0 R09: ffff940350c07480
[ 2679.797124] R10: ffffab13c8047cd8 R11: 0000000000000001 R12: 0000000000000000
[ 2679.797138] R13: ffff94023cc79418 R14: ffff940f4b351000 R15: ffff940f4b3510b0
[ 2679.797153] FS:  0000000000000000(0000) GS:ffff940351000000(0000) knlGS:0000000000000000
[ 2679.797169] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2679.797181] CR2: 000055b2c9201138 CR3: 0000000adc85a004 CR4: 00000000007606f0
[ 2679.797195] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2679.797209] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 2679.797223] PKRU: 55555554
[ 2679.797229] Call Trace:
[ 2679.797240]  pci_probe+0x225/0x404 [coyote_drv]
[ 2679.797252]  local_pci_probe+0x47/0xa0
[ 2679.797262]  work_for_cpu_fn+0x1a/0x30
[ 2679.797271]  process_one_work+0x20f/0x400
[ 2679.797281]  worker_thread+0x22a/0x410
[ 2679.797291]  kthread+0x121/0x140
[ 2679.797299]  ? process_one_work+0x400/0x400
[ 2679.797309]  ? kthread_park+0x90/0x90
[ 2679.797318]  ret_from_fork+0x1f/0x40
[ 2679.797327] Modules linked in: coyote_drv(OE+) ebtable_filter ebtables xt_conntrack vboxnetadp(OE) vboxnetflt(OE) nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo br_netfilter bridge stp llc vboxdrv(OE) aufs overlay ip6table_filter ip6_tables iptable_filter xt_MASQUERADE xt_nat xt_tcpudp xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bpfilter nls_iso8859_1 binfmt_misc intel_rapl_msr intel_rapl_common isst_if_common ipmi_ssif skx_edac nfit cdc_ether usbnet mii ftdi_sio x86_pkg_temp_thermal usbserial intel_powerclamp coretemp kvm_intel kvm rapl input_leds dell_smbios xclmgmt(OE) dcdbas wmi_bmof dell_wmi_descriptor intel_cstate bnxt_re ib_uverbs lpc_ich mei_me mei ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter mac_hid sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp nfsd libiscsi scsi_transport_iscsi auth_rpcgss nfs_acl lockd grace parport_pc sunrpc ppdev lp parport ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456
[ 2679.797353]  async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mgag200 ses hid_generic enclosure drm_vram_helper i2c_algo_bit ttm crct10dif_pclmul drm_kms_helper crc32_pclmul usbhid syscopyarea ghash_clmulni_intel sysfillrect hid aesni_intel sysimgblt crypto_simd nvme fb_sys_fops mpt3sas cryptd raid_class glue_helper nvme_core drm ahci scsi_transport_sas bnxt_en tg3 libahci wmi
[ 2679.797597] ---[ end trace f453f4917d62fbe3 ]---
[ 2679.800690] RIP: 0010:read_shell_config+0x113/0x42b [coyote_drv]
[ 2679.800705] Code: 83 e6 0f 48 89 70 30 48 8b 51 18 48 c1 ea 08 83 e2 0f 89 50 20 48 8b 49 18 48 c1 e9 14 83 e1 3f 48 89 48 08 48 83 f9 0c 74 02 <0f> 0b 41 bc 01 00 00 00 89 f1 48 c7 40 10 00 10 00 00 4c 89 e7 48
[ 2679.800740] RSP: 0018:ffffab13c8047dd8 EFLAGS: 00010206
[ 2679.800752] RAX: ffff940f40f822a0 RBX: ffff94023cc79418 RCX: 000000000000003f
[ 2679.800766] RDX: 000000000000000f RSI: 000000000000000f RDI: ffff940350c07480
[ 2679.800780] RBP: ffffab13c8047df8 R08: ffff9403510350c0 R09: ffff940350c07480
[ 2679.800794] R10: ffffab13c8047cd8 R11: 0000000000000001 R12: 0000000000000000
[ 2679.800809] R13: ffff94023cc79418 R14: ffff940f4b351000 R15: ffff940f4b3510b0
[ 2679.800824] FS:  0000000000000000(0000) GS:ffff940351000000(0000) knlGS:0000000000000000
[ 2679.800840] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2679.800852] CR2: 000055b2c9201138 CR3: 0000000adc85a004 CR4: 00000000007606f0
[ 2679.800867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2679.800881] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 2679.800896] PKRU: 55555554

[maximilianheer]

Hi @qianyich, thanks for your ongoing reports. Unfortunately, we are no longer actively supporting U280 as a platform. To begin with: Did you use coyote v1 or the current main-branch? Following the error print-out in your driver insertion, there seems to be something seriously wrong with the compilation (check [ 2679.796937] kernel BUG at /home/qianyich/Coyote/driver/fpga_dev.c:119!
[ 2679.796957] invalid opcode: 0000 [#1] SMP NOPTI). We haven't encountered this yet. What we can offer to you is to apply for a user account on our HACC-cluster (https://github.com/fpgasystems/hacc), where we also have U280s. You could then try to build and deploy the design there.