Skip to content

Latest commit

 

History

History
2307 lines (1536 loc) · 77.9 KB

README.md

File metadata and controls

2307 lines (1536 loc) · 77.9 KB
Raw

Custom Firmware Development Guide for Full Device Emulation

Table of Contents

Part 1: Foundational Concepts

  1. Introduction
  2. Key Definitions
  3. Device Compatibility
  4. Requirements
  5. Gathering Donor Device Information
  6. Initial Firmware Customization
  7. Vivado Project Setup and Customization

Part 2: Intermediate Concepts and Implementation

  1. Advanced Firmware Customization
  2. Emulating Device-Specific Capabilities
  3. Transaction Layer Packet (TLP) Emulation

Part 3: Advanced Techniques and Optimization

  1. Building, Flashing, and Testing
  2. Advanced Debugging Techniques
  3. Troubleshooting
  4. Emulation Accuracy and Optimizations
  5. Best Practices for Firmware Development
  6. Additional Resources
  7. Contact Information
  8. Support and Contributions

Part 1: Foundational Concepts

1. Introduction

1.1 Purpose of the Guide

The primary purpose of this guide is to provide a step-by-step approach to developing custom Direct Memory Access (DMA) firmware for FPGA-based devices to emulate PCIe hardware accurately. This enables applications such as hardware testing, system debugging, security research, and hardware emulation.

By following this guide, you will learn how to:

  • Gather necessary information from a donor device.
  • Customize firmware to emulate a specific hardware device.
  • Set up the development environment using tools like Vivado and Visual Studio Code.
  • Understand key concepts related to PCIe and DMA operations.

1.2 Target Audience

This guide is intended for:

  • Firmware Developers: Engineers interested in creating custom firmware for hardware emulation, testing, or bypassing hardware restrictions.
  • Hardware Engineers: Professionals working on hardware testing and development who need to emulate specific devices.
  • Security Researchers: Individuals conducting vulnerability assessments, malware analysis, or security testing requiring hardware emulation.
  • FPGA Enthusiasts: Hobbyists and learners interested in FPGA customization and low-level hardware emulation.

1.3 How to Use This Guide

The guide is divided into three parts:

  • Part 1: Foundational Concepts: Covers the basic concepts, setup, and initial steps required to start firmware development for device emulation.
  • Part 2: Intermediate Concepts and Implementation: Delves into more complex topics such as advanced firmware customization, TLP emulation, and initial debugging techniques.
  • Part 3: Advanced Techniques and Optimization: Explores advanced debugging, troubleshooting, optimization strategies, and best practices.

It is recommended to follow the guide sequentially to build a solid understanding before tackling advanced topics.

2. Key Definitions

Understanding the terminology is crucial for effectively following this guide. Below are key definitions related to PCIe, DMA, and device emulation:

  • DMA (Direct Memory Access): A capability that allows hardware devices to read from or write to system memory directly, without CPU intervention, enabling high-speed data transfers.
  • TLP (Transaction Layer Packet): The fundamental unit of communication in the PCIe architecture, encapsulating control and data information.
  • BAR (Base Address Register): Registers in PCIe devices that define memory and I/O address regions, mapping device memory into the system memory space.
  • FPGA (Field-Programmable Gate Array): A reconfigurable integrated circuit that can be programmed to perform specific hardware functions.
  • MSI/MSI-X (Message Signaled Interrupts): Mechanisms used by PCIe devices to send interrupts to the CPU without using traditional interrupt lines.
  • Device Serial Number (DSN): A unique identifier associated with a specific device, often used for advanced device identification.
  • PCIe Configuration Space: A standardized memory area where PCIe devices provide information about themselves and configure operational parameters.
  • Donor Device: A PCIe hardware device used to extract configuration and identification details for the purpose of emulating its behavior on an FPGA.

3. Device Compatibility

3.1 Supported FPGA-Based Hardware

While this guide focuses on the Squirrel DMA (35T) card due to its accessibility, the methodologies are adaptable to other FPGA-based DMA hardware:

  • Squirrel (35T)
    • Description: Affordable FPGA-based DMA device suitable for standard memory acquisition and device emulation.
  • Enigma-X1 (75T)
    • Description: Mid-tier FPGA offering enhanced resources, ideal for more demanding memory operations.
  • ZDMA (100T)
    • Description: High-performance FPGA optimized for rapid memory interactions, suitable for extensive memory reads/writes.
  • Kintex-7
    • Description: Advanced FPGA with robust capabilities for complex projects and large-scale DMA solutions.

3.2 PCIe Hardware Considerations

To ensure smooth emulation, several PCIe-specific features must be addressed:

  • IOMMU/VT-d Settings
    • Recommendation: Disable IOMMU (Intel's VT-d) or AMD's equivalent to allow unrestricted DMA access.
    • Rationale: IOMMU can restrict DMA operations, potentially interfering with memory acquisition and emulation.
  • Kernel DMA Protection
    • Recommendation: Disable Kernel DMA Protection features in modern systems.
    • Steps:
      • Windows: Disable features like Secure Boot or Virtualization-Based Security (VBS) in BIOS/UEFI settings.
      • Caution: Disabling these features can expose the system to security risks; ensure you're operating in a secure environment.
  • PCIe Slot Requirements
    • Recommendation: Use a compatible PCIe slot that matches the FPGA device's requirements (e.g., x1, x4).
    • Rationale: Ensures optimal performance and compatibility with the host system.

3.3 System Requirements

  • Host System
    • Processor: Multi-core CPU (Intel i5/i7 or AMD equivalent)
    • Memory: Minimum 16 GB RAM
    • Storage: SSD with at least 100 GB free space
    • Operating System: Windows 10/11 (64-bit) or compatible Linux distribution
  • Peripheral Devices
    • JTAG Programmer: For flashing firmware onto the FPGA
    • PCIe Slot: Ensure the host system has an available PCIe slot compatible with the DMA card

4. Requirements

4.1 Hardware

  • Donor PCIe Device
    • Purpose: Source of device IDs and configuration data for emulation.
    • Examples: Network adapters, storage controllers, or any generic PCIe card not in use.
  • DMA FPGA Card
    • Description: FPGA-based device capable of performing DMA operations.
    • Examples: Squirrel (35T), Enigma-X1 (75T), ZDMA (100T), Kintex-7
  • JTAG Programmer
    • Purpose: For flashing firmware onto the FPGA.
    • Examples: Xilinx Platform Cable USB II, Digilent JTAG-HS3

4.2 Software

  • Xilinx Vivado Design Suite
    • Description: FPGA development software for synthesizing and building firmware projects.
    • Download: Xilinx Vivado
  • Visual Studio Code
  • PCILeech-FPGA
  • Arbor
    • Description: PCIe device scanning tool for gathering device information.
    • Download: Arbor by MindShare
    • Note: Requires account creation; offers a 14-day trial.
  • Alternative Tools
    • Telescan PE
      • Description: PCIe traffic analysis tool as an alternative to Arbor.
      • Download: Teledyne LeCroy Telescan PE
      • Note: Free but requires manual registration approval.

4.3 Environment Setup

4.3.1 Install Xilinx Vivado Design Suite

  • Steps:
    1. Visit the Xilinx Vivado Download Page.
    2. Download the appropriate version compatible with your FPGA device.
    3. Run the installer and follow the on-screen instructions.
    4. Select necessary components during installation.
    5. Launch Vivado to ensure proper installation.

4.3.2 Install Visual Studio Code

  • Steps:
    1. Visit the Visual Studio Code Download Page.
    2. Download and install for your operating system.
    3. Install extensions for Verilog or VHDL support (e.g., Verilog-HDL/SystemVerilog).

4.3.3 Clone the PCILeech-FPGA Repository

  • Steps:
    1. Open a terminal or command prompt.
    2. Navigate to your desired directory: 
      cd ~/Projects/
    3. Clone the repository: 
      git clone https://github.com/ufrisk/pcileech-fpga.git
    4. Navigate to the cloned directory: 
      cd pcileech-fpga

4.3.4 Set Up a Clean Development Environment

  • Recommendation: Work in an isolated environment to prevent unintended interactions.
  • Steps:
    1. Use a dedicated development machine or virtual machine.
    2. Ensure no other applications interfere with PCIe operations or FPGA programming.

5. Gathering Donor Device Information

Accurate device emulation hinges on meticulously extracting and replicating critical information from the donor device. This comprehensive data collection enables your FPGA to faithfully mimic the target hardware's PCIe configuration and behavior, ensuring compatibility and functionality when interfacing with the host system.

5.1 Using Arbor for PCIe Device Scanning

Arbor is a robust and user-friendly tool designed for in-depth scanning of PCIe devices. It provides detailed insights into the configuration space of connected hardware, making it an invaluable resource for extracting the necessary information for device emulation.

5.1.1 Install Arbor

To begin utilizing Arbor for device scanning, you must first install the software on your system.

Steps:

  1. Visit the Arbor Download Page:

    • Navigate to the official Arbor Download Page using your preferred web browser.
    • Ensure you are accessing the site directly to avoid any malicious redirects.

  2. Create an Account (if required):

    • Arbor may require you to create a user account to access the download links.
    • Provide the necessary information, such as your name, email address, and organization.
    • Verify your email if prompted, to activate your account.

  3. Download Arbor:

    • Once logged in, locate the download section for Arbor.
    • Select the version compatible with your operating system (e.g., Windows 10/11 64-bit).
    • Click the Download button and save the installer to a known location on your computer.

  4. Install Arbor:

    • Locate the downloaded installer file (e.g., ArborSetup.exe).
    • Right-click the installer and select Run as administrator to ensure it has the necessary permissions.
    • Follow the on-screen instructions to complete the installation process.
      • Accept the license agreement.
      • Choose the installation directory.
      • Opt to create desktop shortcuts if desired.

  5. Verify Installation:

    • Upon completion, ensure that Arbor is listed in your Start Menu or on your desktop.
    • Launch Arbor to confirm it opens without errors.

5.1.2 Scan PCIe Devices

With Arbor installed, you can proceed to scan your system for connected PCIe devices.

Steps:

  1. Launch Arbor:

    • Double-click the Arbor icon on your desktop or find it via the Start Menu.
    • If prompted by User Account Control (UAC), allow the application to make changes to your device.

  2. Navigate to the Local System Tab:

    • In the Arbor interface, locate the navigation pane or tabs.
    • Click on Local System to access tools for scanning the local machine.

  3. Scan for PCIe Devices:

    • Look for a Scan or Rescan button, typically located at the top or bottom of the interface.
    • Click Scan/Rescan to initiate the detection process.
    • Wait for the scanning process to complete; this may take a few moments depending on the number of devices connected.

  4. Review Detected Devices:

    • Once the scan is complete, Arbor will display a list of all detected PCIe devices.
    • The devices are usually listed with their names, device IDs, and other identifying information.

5.1.3 Identify the Donor Device

Identifying the correct donor device is crucial for accurate emulation.

Steps:

  1. Locate Your Donor Device in the List:

    • Scroll through the list of devices detected by Arbor.
    • Look for the device matching the make and model of your donor hardware.
    • Devices may be listed by their vendor names, device types, or function.

  2. Verify Device Details:

    • Click on the device to select it.
    • Confirm that the Device ID and Vendor ID match those of your donor device.
      • Tip: These IDs are typically found in the device's documentation or on the manufacturer's website.

  3. View Detailed Configuration:

    • With the device selected, find and click on an option like View Details or Properties.
    • This will open a detailed view showing the device's configuration space and capabilities.

  4. Cross-Reference with Physical Hardware:

    • If multiple similar devices are listed, cross-reference the Slot Number or Bus Address with the physical slot where the donor device is installed.

5.1.4 Capture Device Data

Extracting detailed information from the donor device is essential for accurate emulation.

Information to Extract:

  • Device ID (0xXXXX):
    • A 16-bit identifier unique to the device model.
  • Vendor ID (0xYYYY):
    • A 16-bit identifier assigned to the manufacturer.
  • Subsystem ID (0xZZZZ):
    • Identifies the specific subsystem or variant.
  • Subsystem Vendor ID (0xWWWW):
    • Identifies the vendor of the subsystem.
  • Revision ID (0xRR):
    • Indicates the revision level of the device.
  • Class Code (0xCCCCCC):
    • A 24-bit code that defines the type of device (e.g., network controller, storage device).
  • Base Address Registers (BARs):
    • Registers defining the memory or I/O space the device uses.
    • Includes BAR0 through BAR5, each potentially 32 or 64 bits.
  • Capabilities:
    • Lists supported features such as MSI/MSI-X, power management, PCIe link speed, and width.
  • Device Serial Number (DSN):
    • A 64-bit unique identifier, if the device supports it.

Steps:

  1. Navigate to the PCI Config Tab:

    • Within the device's detailed view, find and select the PCI Config or Configuration Space tab.

  2. Record Relevant Details:

    • Carefully document each of the required fields.
    • Use screenshots or copy the values into a text file or spreadsheet for accuracy.
    • Ensure hexadecimal values are noted correctly, including the 0x prefix if used.

  3. Expand Capability Lists:

    • Look for sections labeled Capabilities or Advanced Features.
    • Document each capability and its parameters (e.g., MSI count, power states supported).

  4. Examine BARs in Detail:

    • For each BAR, note:
      • BAR Number (e.g., BAR0):
      • Type (Memory or I/O):
      • Bit Width (32-bit or 64-bit):
      • Size (e.g., 256 MB):
      • Prefetchable Status (Yes/No):

  5. Save the Data for Reference:

    • Compile all the information into a well-organized document.
    • Label each section clearly for easy reference during firmware customization.

  6. Double-Check Entries:

    • Review all recorded data to ensure accuracy.
    • Correct any discrepancies by revisiting the Arbor interface.

5.2 Extracting and Recording Device Attributes

After capturing the data, it's crucial to understand the significance of each attribute and ensure they've been accurately documented.

Ensure You Have Accurately Recorded the Following:

  1. Device ID:

    • Purpose: Uniquely identifies the device model.
    • Usage: Essential for the host OS to load the correct driver.

  2. Vendor ID:

    • Purpose: Identifies the manufacturer.
    • Usage: Used in conjunction with Device ID to match device drivers.

  3. Subsystem ID and Subsystem Vendor ID:

    • Purpose: Specifies the subsystem's device and vendor IDs, allowing for differentiation between variants.
    • Usage: Important for devices with multiple configurations or OEM-specific versions.

  4. Revision ID:

    • Purpose: Indicates the hardware revision.
    • Usage: Helps in identifying specific hardware versions that may require different drivers or firmware.

  5. Class Code:

    • Purpose: Categorizes the device type (e.g., mass storage, network controller).
    • Usage: Allows the OS to understand the device's general function.

  6. Base Address Registers (BARs):

    • Purpose: Define the memory or I/O address regions that the device will use.
    • Usage: Critical for mapping device memory into the system address space.

  7. Capabilities:

    • Purpose: Lists the advanced features the device supports.
    • Examples:
      • MSI/MSI-X: Message Signaled Interrupts for efficient interrupt handling.
      • Power Management: States like D0, D1, D2, D3hot, D3cold.
      • PCIe Link Speed/Width: Determines the data transfer capabilities.

  8. Device Serial Number (DSN):

    • Purpose: A unique 64-bit identifier for the device.
    • Usage: Used for advanced identification and may be required by some drivers.

Best Practices:

  • Organize the Data:

    • Create a structured document or spreadsheet.
    • Use clear headings and subheadings for each attribute.

  • Include Units and Formats:

    • Indicate units for sizes (e.g., MB, KB).
    • Use consistent formatting for hexadecimal values (e.g., 0x1234).

  • Cross-Reference with Specifications:

    • If available, consult the device's datasheet to verify values.
    • This can help identify any discrepancies or unusual configurations.

  • Secure the Data:

    • Store the collected information securely.
    • Be mindful of any proprietary or confidential information.

6. Initial Firmware Customization

With the donor device's information meticulously documented, the next phase involves customizing your FPGA's firmware to emulate the donor device accurately. This involves modifying the PCIe configuration space and ensuring that memory mappings align correctly.

6.1 Modifying Configuration Space

The PCIe configuration space is a critical component that defines how the device is recognized and interacts with the host system. Customizing this space to match the donor device is essential for successful emulation.

6.1.1 Navigate to the Configuration File

The configuration space is defined within a specific SystemVerilog (.sv) file in your project.

Path:

  • Standard Path:

    pcileech-fpga/pcileech-wifi-main/src/pcie_7x_0_core_top.v

  • Alternative Path (Depending on Directory Structure):

    src\pcie_7x\pcie_7x_0_core_top.v

6.1.2 Open the File in Visual Studio Code

Editing the configuration file requires a suitable code editor that supports syntax highlighting for SystemVerilog.

Steps:

  1. Launch Visual Studio Code:

    • Click on the VS Code icon or find it via the Start Menu.

  2. Open the File:

    • Use File > Open File or press Ctrl + O.
    • Navigate to the configuration file path mentioned above.
    • Select pcileech_pcie_cfg_a7.sv and click Open.

  3. Verify Syntax Highlighting:

    • Ensure that the editor recognizes the .sv file extension.
    • If necessary, install extensions for SystemVerilog support.

  4. Familiarize Yourself with the File Structure:

    • Scroll through the file to understand the existing assignments and comments.
    • Look for sections where configuration registers are defined.

6.1.3 Modify Device ID and Vendor ID

Updating these identifiers is crucial for the host system to recognize the emulated device as the donor.

Steps:

  1. Search for cfg_deviceid:

    • Use the search functionality (Ctrl + F).
    • Locate the line defining cfg_deviceid.

  2. Update Device ID:

    cfg_deviceid <= 16'hXXXX;  // Replace XXXX with the donor's Device ID
    • Example:
      • If the donor's Device ID is 0x1234, update as: 
        cfg_deviceid <= 16'h1234;

  3. Search for cfg_vendorid:

    • Locate the line defining cfg_vendorid.

  4. Update Vendor ID:

    cfg_vendorid <= 16'hYYYY;  // Replace YYYY with the donor's Vendor ID
    • Example:
      • If the donor's Vendor ID is 0xABCD, update as: 
        cfg_vendorid <= 16'hABCD;

  5. Ensure Correct Formatting:

    • Verify that hexadecimal values are prefixed with 16'h.
    • Maintain consistent indentation and commenting style.

6.1.4 Modify Subsystem ID and Revision ID

These identifiers provide additional details about the device variant and hardware revision.

Steps:

  1. Search for cfg_subsysid:

    • Locate the line defining cfg_subsysid.

  2. Update Subsystem ID:

    cfg_subsysid <= 16'hZZZZ;  // Replace ZZZZ with the donor's Subsystem ID
    • Example:
      • If the donor's Subsystem ID is 0x5678, update as: 
        cfg_subsysid <= 16'h5678;

  3. Search for cfg_subsysvendorid:

    • Locate the line defining cfg_subsysvendorid.

  4. Update Subsystem Vendor ID (if applicable):

    cfg_subsysvendorid <= 16'hWWWW;  // Replace WWWW with the donor's Subsystem Vendor ID
    • Example:
      • If the donor's Subsystem Vendor ID is 0x9ABC, update as: 
        cfg_subsysvendorid <= 16'h9ABC;

  5. Search for cfg_revisionid:

    • Locate the line defining cfg_revisionid.

  6. Update Revision ID:

    cfg_revisionid <= 8'hRR;   // Replace RR with the donor's Revision ID
    • Example:
      • If the donor's Revision ID is 0x01, update as: 
        cfg_revisionid <= 8'h01;

6.1.5 Update Class Code

The Class Code informs the host of the device type and function.

Steps:

  1. Search for cfg_classcode:

    • Locate the line defining cfg_classcode.

  2. Update Class Code:

    cfg_classcode <= 24'hCCCCCC;  // Replace CCCCCC with the donor's Class Code
    • Example:
      • If the donor's Class Code is 0x020000 (Ethernet Controller), update as: 
        cfg_classcode <= 24'h020000;

  3. Verify Correct Bit Width:

    • Ensure that the Class Code is a 24-bit value.
    • Hexadecimal values should be prefixed with 24'h.

6.1.6 Save Changes

After making all modifications, it's important to save and review the changes.

Steps:

  1. Save the File:

    • Click File > Save or press Ctrl + S.

  2. Review Changes:

    • Re-read the modified lines to confirm accuracy.
    • Check for any syntax errors or typos.

  3. Optional - Use Version Control:

    • If using Git or another version control system, commit your changes with a meaningful message.
      • Example: 
        git add pcileech_pcie_cfg_a7.sv
git commit -m "Updated PCIe configuration with donor device identifiers"

6.2 Inserting the Device Serial Number (DSN)

The Device Serial Number (DSN) is a unique identifier that some devices utilize for advanced features. Including it enhances the authenticity of the emulation.

6.2.1 Locate the DSN Field

The DSN is typically defined in the same configuration file.

Steps:

  1. Search for cfg_dsn:

    • In pcileech_pcie_cfg_a7.sv, use the search function (Ctrl + F) to find cfg_dsn.

  2. Understand the Existing Assignment:

    • The DSN may be set to a default value or zeroed out. 
      cfg_dsn <= 64'h0000000000000000;  // Default DSN

6.2.2 Insert the DSN

Updating the DSN involves setting it to the exact value from the donor device.

Steps:

  1. Update cfg_dsn:

    cfg_dsn <= 64'hXXXXXXXX_YYYYYYYY;  // Replace with donor DSN
    • Example:
      • If the donor's DSN is 0x0011223344556677, update as: 
        cfg_dsn <= 64'h0011223344556677;

  2. Handle DSN Unavailability:

    • If the donor device does not have a DSN or it is not required, set it to zero: 
      cfg_dsn <= 64'h0000000000000000;  // No DSN

  3. Ensure Correct Formatting:

    • The DSN is a 64-bit value; ensure it's properly formatted.
    • Use the 64'h prefix for hexadecimal values.

  4. Add Comments for Clarity:

    • Include a comment indicating the DSN source. 
      cfg_dsn <= 64'h0011223344556677;  // Donor DSN

6.2.3 Save Changes

Finalize the modifications by saving and reviewing.

Steps:

  1. Save the File:

    • Click File > Save or press Ctrl + S.

  2. Verify the Syntax:

    • Look for any red underlines or error indications in the editor.
    • Correct any issues before proceeding.

  3. Document the Changes:

    • If using version control, commit the updates with an appropriate message.
      • Example: 
        git commit -am "Inserted donor Device Serial Number (DSN) into configuration"

7. Vivado Project Setup and Customization

With the firmware files updated to reflect the donor device's configuration, the next step is to integrate these changes into the Vivado project. This involves generating the project files, customizing IP cores, and preparing the design for synthesis and implementation.

7.1 Generating Vivado Project Files

Vivado uses Tcl scripts to automate project creation and configuration. By running these scripts, you ensure that all settings are correctly applied based on your FPGA device.

7.1.1 Open Vivado

Starting with a fresh session of Vivado ensures that previous settings or projects do not interfere with your current work.

Steps:

  1. Launch Vivado:

    • Find the Vivado application in your Start Menu or desktop.
    • Click to open it.

  2. Select the Correct Version:

    • If multiple versions are installed, ensure you are using the one compatible with your FPGA (e.g., Vivado 2020.1).

  3. Wait for the Startup Screen:

    • Allow Vivado to fully initialize before proceeding.

7.1.2 Access the Tcl Console

The Tcl Console allows you to execute scripts and commands directly.

Steps:

  1. Open the Tcl Console:

    • In the Vivado interface, go to the menu bar.
    • Click on Window > Tcl Console.
    • The Tcl Console will appear at the bottom of the window.

  2. Adjust Console Size (Optional):

    • Drag the console's top border to resize it for better visibility.

  3. Clear Previous Commands:

    • If any commands are present, you can clear them for a clean start.

7.1.3 Navigate to the Project Directory

Ensure that the Tcl Console is pointing to the correct directory where your project scripts are located.

For Squirrel DMA (35T):

Path:

  • Your project directory, typically: 
    C:/Users/YourUsername/Documents/pcileech-fpga/pcileech-wifi-main/

Steps:

  1. Set the Working Directory:

    • In the Tcl Console, enter: 
      cd C:/Users/YourUsername/Documents/pcileech-fpga/pcileech-wifi-main/
      • Replace the path with the actual location on your system.

  2. Verify the Directory Change:

    • Enter pwd in the Tcl Console.
    • The console should display the current directory, confirming the change.

7.1.4 Generate the Vivado Project

Running the appropriate Tcl script will set up the project with all necessary configurations.

Steps:

  1. Run the Tcl Script:

    • For Squirrel (35T): 
      source vivado_generate_project_squirrel.tcl -notrace
    • For Enigma-X1 (75T): 
      source vivado_generate_project_enigma_x1.tcl -notrace
    • For ZDMA (100T): 
      source vivado_generate_project_100t.tcl -notrace

  2. Wait for Script Completion:

    • The script will execute several commands:
      • Create the project.
      • Add source files.
      • Configure project settings.
    • Monitor the Tcl Console for progress messages.
    • Address any errors that may occur, such as missing files or incorrect paths.

  3. Confirm Project Generation:

    • Upon completion, the console will indicate that the project has been created.
    • The project files (.xpr and associated directories) will be present in the project directory.

7.1.5 Open the Generated Project

Now that the project is generated, you can open it within Vivado for further customization.

Steps:

  1. Open the Project:

    • In Vivado, click File > Open Project.
    • Navigate to your project directory.

  2. Select the Project File:

    • For Squirrel: 
      pcileech_squirrel_top.xpr
    • Click on the .xpr file to select it.

  3. Click Open:

    • Vivado will load the project, displaying the design hierarchy and sources.

  4. Verify Project Contents:

    • In the Project Manager window, ensure that all source files are listed.
    • Check for any warnings or errors upon opening.

7.2 Modifying IP Blocks

The PCIe IP core is a critical component that must be configured to match the donor device's specifications. Customizing the IP core ensures that the FPGA behaves identically to the donor hardware at the PCIe protocol level.

7.2.1 Access the PCIe IP Core

The PCIe IP core is an instantiated IP block within your Vivado project.

Steps:

  1. Locate the PCIe IP Core:

    • In the Sources pane, ensure the Hierarchy tab is selected.
    • Expand the design hierarchy to find the PCIe IP core.
      • It is typically named pcie_7x_0.xci or similar.

  2. Open the IP Customization Window:

    • Right-click on pcie_7x_0.xci.
    • Select Customize IP from the context menu.
    • The IP Configuration window will open.

  3. Wait for IP Settings to Load:

    • The IP customization interface may take a few moments to initialize.
    • Ensure that all options and tabs are fully loaded before proceeding.

7.2.2 Customize Device IDs and BARs

Configuring the device identifiers within the IP core is crucial for correct enumeration by the host system.

Steps:

  1. Navigate to Device and Vendor Identifiers:

    • In the IP customization window, select the Device and Vendor Identifiers tab or section.

  2. Enter the Device ID:

    • Find the field labeled Device ID.
    • Enter the donor's Device ID (e.g., 0x1234).

  3. Enter the Vendor ID:

    • Locate the Vendor ID field.
    • Input the donor's Vendor ID (e.g., 0xABCD).

  4. Enter the Subsystem ID and Subsystem Vendor ID:

    • Input the Subsystem ID (e.g., 0x5678).
    • Input the Subsystem Vendor ID (e.g., 0x9ABC).

  5. Set the Revision ID:

    • Enter the Revision ID (e.g., 0x01).

  6. Set the Class Code:

    • Enter the Class Code (e.g., 0x020000 for Ethernet Controller).

  7. Configure Other Identifiers (if available):

    • Some IP cores allow setting Programming Interface, Device Capabilities, etc.
    • Match these to the donor device as needed.

7.2.3 Configure BAR Sizes

The BARs define how the device maps its internal memory and registers to the host system.

Steps:

  1. Navigate to Base Address Registers (BARs):

    • Select the BARs tab or section in the IP customization window.

  2. Configure Each BAR:

    • For BAR0 to BAR5, set the following parameters based on the donor device:
      • Enable BAR: Check or uncheck to match donor device.
      • BAR Size: Select the size from the dropdown (e.g., 256 MB, 64 KB).
      • BAR Type:
        • Memory (32-bit Addressing)
        • Memory (64-bit Addressing)
        • I/O
      • Prefetchable: Check if the donor's BAR is prefetchable.

  3. Example Configuration:

    • BAR0:
      • Enabled
      • Size: 256 MB
      • Type: Memory (64-bit)
      • Prefetchable: Yes
    • BAR1:
      • Disabled (if the donor device does not use BAR1)

  4. Ensure Alignment and Non-Overlapping Spaces:

    • Verify that the total memory mapped does not exceed the FPGA's capabilities.
    • Ensure that BAR sizes align with PCIe specification requirements.

  5. Advanced Settings (if applicable):

    • Some devices may have special requirements, such as expansion ROM BAR.
    • Configure these settings if necessary.

7.2.4 Finalize IP Customization

After configuring all necessary settings, you need to apply the changes.

Steps:

  1. Review All Settings:

    • Go through each tab in the IP customization window.
    • Confirm that all entries match the donor device's specifications.

  2. Apply Changes:

    • Click OK or Generate to apply the settings.
    • If prompted, confirm that you wish to proceed with the changes.

  3. Regenerate IP Core:

    • Vivado will regenerate the IP core to reflect the new configurations.
    • Monitor the Messages pane for any errors or warnings.

  4. Update IP in Project:

    • Ensure that the updated IP core is correctly integrated into your project.
    • Vivado may prompt to update IP dependencies; allow it to do so.

7.2.5 Lock the IP Core

Locking the IP core prevents unintended changes during synthesis and implementation.

Purpose:

  • Prevent Overwrites: Ensures that your manual configurations are preserved.
  • Maintain Consistency: Keeps the IP core in a known state throughout the build process.

Steps:

  1. Open the Tcl Console:

    • In Vivado, if not already open, go to Window > Tcl Console.

  2. Execute the Lock Command:

    • Enter the following command: 
      set_property -name {IP_LOCKED} -value true -objects [get_ips pcie_7x_0]
    • Press Enter to execute.

  3. Verify the Lock:

    • Check the Messages pane for confirmation.
    • The IP core should now be marked as locked.

  4. Unlocking (if necessary):

    • To make further changes in the future, you can unlock the IP core: 
      set_property -name {IP_LOCKED} -value false -objects [get_ips pcie_7x_0]
    • Remember to re-lock it after making changes.

  5. Document the Action:

    • Note in your project documentation that the IP core has been locked.
    • This helps team members understand the project's configuration state.

Part 2: Intermediate Concepts and Implementation

8. Advanced Firmware Customization

To achieve a precise emulation of the donor device, further in-depth customization of the firmware is necessary. This involves aligning the PCIe parameters, adjusting Base Address Registers (BARs), and emulating power management and interrupt mechanisms to match the donor device's specifications. These steps ensure that the emulated device interacts seamlessly with the host system and behaves identically to the original hardware.

8.1 Configuring PCIe Parameters for Emulation

Accurate emulation requires that the PCIe parameters of your FPGA device are meticulously configured to match those of the donor device. This includes settings such as the PCIe link speed, link width, capability pointers, and maximum payload sizes. Proper configuration ensures compatibility with the host system and the correct operation of drivers and applications that interact with the device.

8.1.1 Matching PCIe Link Speed and Width

The PCIe link speed and width are critical parameters that determine the data throughput and performance of the device. Matching these settings with the donor device is essential for accurate emulation.

Steps:

  1. Access PCIe IP Core Settings:

    • Open Your Vivado Project:

      • Launch Vivado and open the project you previously created or modified.
      • Ensure that all source files are correctly added to the project.

    • Locate the PCIe IP Core:

      • In the Sources pane, expand the hierarchy to find the PCIe IP core instance, typically named pcie_7x_0.
      • The file associated with the IP core is usually pcie_7x_0.xci.

    • Customize the IP Core:

      • Right-click on pcie_7x_0.xci and select Customize IP.
      • The IP customization window will open, displaying various configuration options.

  2. Set Maximum Link Speed:

    • Navigate to Link Parameters:

      • In the IP customization window, click on the Link Parameters tab or section.
      • This section contains settings related to the PCIe link's characteristics.

    • Configure Maximum Link Speed:

      • Find the Maximum Link Speed option.
      • Set it to match the donor device's link speed.
        • Example:
          • If the donor device operates at Gen2 (5.0 GT/s), select 5.0 GT/s.
          • If it operates at Gen1 (2.5 GT/s) or Gen3 (8.0 GT/s), select the corresponding option.
      • Note: Ensure that your FPGA and the physical hardware support the selected link speed.

  3. Set Link Width:

    • Configure Link Width:
      • In the same Link Parameters section, locate the Link Width setting.
      • Set it to match the donor device's link width.
        • Example:
          • If the donor device uses a x4 link, set the Link Width to 4.
          • Options typically include 1, 2, 4, 8, 16 lanes.
      • Note: The physical connectors and the FPGA must support the selected link width.

  4. Save and Regenerate:

    • Apply Changes:

      • After configuring the link speed and width, click OK to apply the changes.
      • Vivado may prompt you to regenerate the IP core due to the changes made.
      • Confirm and allow the regeneration process to complete.

    • Verify Settings:

      • Once regeneration is complete, revisit the IP core settings to ensure the configurations are correctly applied.
      • Check for any warnings or errors in the Messages window.

8.1.2 Setting Capability Pointers

Capability pointers in the PCIe configuration space point to various capability structures, such as MSI, power management, and others. Correctly setting these pointers ensures that the host system can locate and utilize the device's capabilities.

Steps:

  1. Locate Capability Pointer in Firmware:

    • Open Configuration File:

      • In Visual Studio Code, open the pcileech_pcie_cfg_a7.sv file located at: 
        pcileech-fpga/pcileech-wifi-main/src/pcileech_pcie_cfg_a7.sv

    • Understand the Capability Pointer:

      • The capability pointer is an 8-bit register that points to the first capability structure in the PCIe configuration space, usually starting after the standard configuration header.

  2. Set Capability Pointer Value:

    • Find the Assignment for cfg_cap_pointer:

      • Search for the line in the code where cfg_cap_pointer is assigned. 
        cfg_cap_pointer <= 8'hXX; // Current value

    • Update the Capability Pointer:

      • Replace XX with the donor device's capability pointer value.
        • Example:
          • If the donor device's capability pointer is 0x60, update the line to: 
            cfg_cap_pointer <= 8'h60; // Updated to match donor device

    • Ensure Correct Alignment:

      • Capability structures must be aligned on a 4-byte boundary.
      • The capability pointer should point to a valid offset within the configuration space.

  3. Save Changes:

    • Save the Configuration File:

      • After making the changes, save the file by clicking File > Save or pressing Ctrl + S.

    • Verify Syntax:

      • Ensure there are no syntax errors introduced by the changes.

    • Comment for Clarity:

      • Add a comment explaining the change for future reference. 
        cfg_cap_pointer <= 8'h60; // Set to donor's capability pointer at offset 0x60

8.1.3 Adjusting Maximum Payload and Read Request Sizes

These parameters define the maximum amount of data that can be transferred in a single PCIe transaction. Matching these settings with the donor device ensures compatibility and optimal performance.

Steps:

  1. Set Maximum Payload Size:

    • Access Device Capabilities:

      • In the PCIe IP core customization window, navigate to the Device Capabilities or Capabilities tab.

    • Configure Max Payload Size Supported:

      • Find the Max Payload Size Supported setting.
      • Set it to the value supported by the donor device.
        • Options:
          • 128 bytes, 256 bytes, 512 bytes, 1024 bytes, 2048 bytes, 4096 bytes.
        • Example:
          • If the donor device supports a maximum payload size of 256 bytes, select 256 bytes.

  2. Set Maximum Read Request Size:

    • Configure Max Read Request Size Supported:
      • In the same tab, find the Max Read Request Size Supported setting.
      • Set it to match the donor device's capability.
        • Example:
          • If the donor supports a maximum read request size of 512 bytes, select 512 bytes.

  3. Adjust Firmware Parameters:

    • Open pcileech_pcie_cfg_a7.sv:

      • Ensure that the configuration file is open in Visual Studio Code.

    • Update Firmware Constants:

      • Locate the lines where max_payload_size_supported and max_read_request_size_supported are defined. 
        max_payload_size_supported <= 3'bZZZ; // Current value
max_read_request_size_supported <= 3'bWWW; // Current value

    • Set the Appropriate Values:

      • Replace ZZZ and WWW with the binary representations of the sizes.
        • Mapping:
          • 128 bytes: 3'b000
          • 256 bytes: 3'b001
          • 512 bytes: 3'b010
          • 1024 bytes: 3'b011
          • 2048 bytes: 3'b100
          • 4096 bytes: 3'b101
        • Example:
          • For 256 bytes payload size: 
            max_payload_size_supported <= 3'b001; // Supports up to 256 bytes
          • For 512 bytes read request size: 
            max_read_request_size_supported <= 3'b010; // Supports up to 512 bytes

  4. Save Changes:

    • Save the File:

      • After updating the values, save the file.

    • Verify Consistency:

      • Ensure that the values in the firmware match those configured in the PCIe IP core.

    • Add Comments:

      • Document the changes for future reference. 
        max_payload_size_supported <= 3'b001; // 256 bytes as per donor device
max_read_request_size_supported <= 3'b010; // 512 bytes as per donor device

8.2 Adjusting BARs and Memory Mapping

Base Address Registers (BARs) define the memory regions that the device exposes to the host. Correctly configuring the BARs and memory mapping is crucial for accurate emulation and proper operation of device drivers.

8.2.1 Setting BAR Sizes

Configuring the BAR sizes ensures that the device requests the correct amount of address space during enumeration and that the host maps these regions appropriately.

Steps:

  1. Access BAR Configuration:

    • Customize PCIe IP Core:

      • In Vivado, right-click on pcie_7x_0.xci and select Customize IP.

    • Navigate to BARs Tab:

      • In the IP customization window, click on the Base Address Registers (BARs) tab.

  2. Configure BAR Sizes and Types:

    • Match Donor Device's BARs:

      • For each BAR (BAR0 to BAR5), set the size and type to match the donor device.

    • Set BAR Sizes:

      • Select the appropriate size from the dropdown for each BAR.
        • Example:
          • If BAR0 is 64 KB, set BAR0 Size to 64 KB.
          • If BAR1 is 128 MB, set BAR1 Size to 128 MB.

    • Set BAR Types:

      • Choose between 32-bit or 64-bit addressing for each BAR.
      • Specify if the BAR is of type Memory or I/O.
      • Set Prefetchable status based on the donor device.

    • Enable or Disable BARs:

      • Ensure that only the BARs used by the donor device are enabled.

  3. Update BRAM Configurations:

    • Adjust BRAM IP Cores:

      • In the ip directory, locate the BRAM configurations corresponding to the BARs.
        • Files: 
          pcileech-fpga/pcileech-wifi-main/ip/bram_bar_zero4k.xci
pcileech-fpga/pcileech-wifi-main/ip/bram_pcie_cfgspace.xci

    • Modify BRAM Sizes:

      • Open each BRAM IP core and adjust the memory size to match the corresponding BAR size.
      • Ensure that the total memory does not exceed the FPGA's capacity.

  4. Save and Regenerate:

    • Apply Changes:

      • After configuring the BARs and updating BRAM sizes, click OK in the IP customization window.

    • Regenerate IP Cores:

      • Vivado may prompt you to regenerate the IP cores due to the changes.
      • Allow the regeneration to complete.

    • Check for Errors:

      • Review the Messages window for any warnings or errors related to BAR configurations.

8.2.2 Defining BAR Address Spaces in Firmware

With the BAR sizes and types set, you need to define how the firmware handles accesses to these BARs.

Steps:

  1. Open the BAR Controller File:

    • Locate the Source File:
      • In Visual Studio Code, open: 
        pcileech-fpga/pcileech-wifi-main/src/pcileech_tlps128_bar_controller.sv

  2. Map Address Ranges:

    • Define Address Decoding Logic:

      • Implement logic to detect when a BAR is accessed based on the address. 
        always_comb begin
  if (bar_hit[0]) begin
    // Handle accesses to BAR0
  end else if (bar_hit[1]) begin
    // Handle accesses to BAR1
  end
  // Continue for additional BARs
end

    • Implement BAR Access Handling:

      • For each BAR, define how reads and writes are managed.
        • Example: 
          if (bar_hit[0]) begin
  case (addr_offset)
    16'h0000: data_out <= reg0;
    16'h0004: data_out <= reg1;
    // Additional registers
    default: data_out <= 32'h0;
  endcase
end

  3. Implement Address Decoding Logic:

    • Calculate Address Offsets:

      • Use the incoming address to calculate offsets within the BAR. 
        addr_offset = incoming_address - bar_base_address[0];

    • Handle Data Transfers:

      • Implement logic for read and write operations. 
        if (cfg_write) begin
  // Write data to the appropriate register
end else if (cfg_read) begin
  // Read data from the appropriate register
end

  4. Save Changes:

    • Save the File:

      • After implementing the logic, save the pcileech_tlps128_bar_controller.sv file.

    • Verify Functionality:

      • Ensure that the logic correctly handles all possible accesses.

8.2.3 Handling Multiple BARs

Properly managing multiple BARs is essential for devices that expose multiple memory or I/O regions.

Steps:

  1. Implement Logic for Each BAR:

    • Separate Logic Blocks:

      • For clarity, create separate code blocks for each BAR within the controller. 
        // BAR0 Handling
if (bar_hit[0]) begin
  // BAR0 specific logic
end
// BAR1 Handling
if (bar_hit[1]) begin
  // BAR1 specific logic
end

    • Define Registers and Memories:

      • Allocate registers or memory blocks for each BAR as needed.

  2. Ensure Non-Overlapping Address Spaces:

    • Validate Address Ranges:

      • Confirm that the address spaces for each BAR do not overlap.
      • Align BAR sizes to power-of-two boundaries as per PCIe specifications.

    • Update Address Decoding:

      • Adjust the address decoding logic to account for the sizes and bases of each BAR.

  3. Test BAR Accesses:

    • Simulation Testing:

      • Use simulation tools to test read and write operations to each BAR.
      • Verify that the correct data is read or written.

    • Hardware Testing:

      • After programming the FPGA, use software tools on the host to access each BAR.
      • Example:
        • Use lspci on Linux to inspect BAR mappings.
        • Write test programs that perform memory-mapped I/O to the BARs.

8.3 Emulating Device Power Management and Interrupts

Emulating power management features and implementing interrupts are critical for devices that need to interact closely with the host operating system's power and interrupt handling mechanisms.

8.3.1 Power Management Configuration

Implementing power management allows the device to support various power states, contributing to system-wide power efficiency and compliance with operating system expectations.

Steps:

  1. Enable Power Management in PCIe IP Core:

    • Access Capabilities:

      • In the PCIe IP core customization window, select the Capabilities tab.

    • Enable Power Management:

      • Check the option for Power Management to include the capability in the device's configuration space.

  2. Set Power States Supported:

    • Configure Supported States:
      • Specify which power states the device supports, such as:
        • D0 (Fully On)
        • D1, D2 (Intermediate States)
        • D3hot, D3cold (Low Power States)
      • Match these settings to the donor device's capabilities.

  3. Implement Power State Logic in Firmware:

    • Open pcileech_pcie_cfg_a7.sv:

      • Modify the firmware to handle power state transitions.

    • Handle Power Management Registers:

      • Implement read and write access to the Power Management Control and Status Register (PMCSR). 
        // PMCSR Address
localparam PMCSR_ADDRESS = 12'h44; // Example address

// PMCSR Register
reg [15:0] pmcsr_reg;

// Handle PMCSR Writes
always @(posedge clk) begin
  if (cfg_write && cfg_address == PMCSR_ADDRESS) begin
    pmcsr_reg <= cfg_writedata[15:0];
    // Update power state based on pmcsr_reg[1:0]
  end
end

    • Manage Power State Effects:

      • Implement the logic to alter device behavior based on the current power state.

  4. Save Changes:

    • Save the Firmware File:

      • Ensure all modifications are saved.

    • Verify Functionality:

      • Test the power management features through simulation or hardware testing.

8.3.2 MSI/MSI-X Configuration

Implementing MSI/MSI-X allows the device to use message-based interrupts, which are more efficient and scalable than traditional pin-based interrupts.

Steps:

  1. Enable MSI/MSI-X in PCIe IP Core:

    • Access Interrupts Configuration:

      • In the PCIe IP core customization window, navigate to the Interrupts or MSI/MSI-X tab.

    • Select Interrupt Type:

      • Choose MSI or MSI-X based on the donor device.

    • Configure Number of Supported Vectors:

      • Set the number of interrupt vectors to match the donor device.
        • MSI supports up to 32 vectors.
        • MSI-X supports up to 2048 vectors.

    • Enable Capabilities:

      • Ensure that the MSI or MSI-X capabilities are included in the device's configuration space.

  2. Implement Interrupt Logic in Firmware:

    • Open pcileech_pcie_tlp_a7.sv:

      • Modify the firmware to handle interrupt generation.

    • Define Interrupt Signals:

      • Declare signals for MSI/MSI-X requests. 
        reg msi_req;

    • Implement Interrupt Generation Logic:

      • Define conditions under which an interrupt is triggered. 
        // Example Interrupt Condition
wire interrupt_condition = /* condition logic */;

// Generate MSI Interrupt
always @(posedge clk) begin
  if (interrupt_condition) begin
    msi_req <= 1'b1;
  end else begin
    msi_req <= 1'b0;
  end
end

    • Connect to PCIe Core:

      • Ensure that the msi_req signal is properly connected to the PCIe IP core's interrupt interface.

  3. Save Changes:

    • Save the Firmware File:

      • After implementing the interrupt logic, save the file.

    • Check for Timing Constraints:

      • Verify that the new logic does not introduce timing violations.

8.3.3 Implementing Interrupt Handling Logic

Defining when and how interrupts are generated is essential for the device's interaction with the host's interrupt handling mechanisms.

Steps:

  1. Define Interrupt Conditions:

    • Identify Trigger Events:

      • Determine specific events that should cause an interrupt.
        • Examples:
          • Data ready for processing.
          • Error conditions.
          • Completion of a task.

    • Implement Condition Logic:

      • Use combinational or sequential logic to detect these events.

  2. Create Interrupt Generation Module:

    • Modular Design:

      • Implement the interrupt logic as a separate module for clarity and reuse. 
        module interrupt_controller(
  input wire clk,
  input wire reset,
  input wire event_trigger,
  output reg msi_req
);
  always @(posedge clk or posedge reset) begin
    if (reset) begin
      msi_req <= 1'b0;
    end else if (event_trigger) begin
      msi_req <= 1'b1;
    end else begin
      msi_req <= 1'b0;
    end
  end
endmodule

    • Integrate with Main Firmware:

      • Instantiate the module and connect it to the main firmware logic.

  3. Ensure Proper Timing and Sequencing:

    • Adhere to PCIe Specifications:

      • Ensure interrupts are generated and cleared according to the protocol.

    • Manage Interrupt Latency:

      • Optimize logic to minimize delay between event occurrence and interrupt generation.

  4. Test Interrupt Delivery:

    • Simulation:

      • Use simulation tools to verify that interrupts are generated correctly.

    • Hardware Testing:

      • Program the FPGA and use host-side software to confirm that interrupts are received and handled.

    • Debugging Tools:

      • Utilize Integrated Logic Analyzer (ILA) cores to monitor signals in real time.

  5. Save Changes:

    • Finalize Code:

      • Ensure all changes are saved and documented.

    • Review and Refine:

      • Iterate on the design as needed based on testing results.

10. Transaction Layer Packet (TLP) Emulation

Transaction Layer Packets (TLPs) are the fundamental units of communication in PCIe. Accurate TLP emulation is crucial for the device to interact properly with the host system.

10.1 Understanding and Capturing TLPs

10.1.1 Learning the TLP Structure

  • Components:

    • Header: Contains fields such as Transaction Layer Packet Type (Type), Length, Requester ID, Tag, Address, etc.
    • Data Payload: Present in Memory Write and some other TLPs.
    • CRC: Ensures data integrity.

  • Understanding TLP Types:

    • Memory Read Request
    • Memory Read Completion
    • Memory Write
    • Configuration Read/Write
    • Vendor-Defined Messages

10.1.2 Capturing TLPs from the Donor Device

  • Steps:

    1. Set Up a PCIe Protocol Analyzer:

      • Use hardware tools like Teledyne LeCroy PCIe Analyzers.

    2. Capture Transactions:

      • Monitor the donor device during normal operation and record the TLPs.

    3. Analyze Captured TLPs:

      • Use the analyzer's software to dissect the TLPs and understand their structure and sequence.

10.1.3 Documenting Key TLP Transactions

  • Steps:

    1. Identify Critical Transactions:

      • Focus on TLPs that are essential for device initialization, configuration, data transfer, and error handling.

    2. Create Detailed Documentation:

      • For each key TLP, note the field values, sequence, and conditions under which it is sent.

    3. Understand Timing and Sequencing:

      • Pay attention to the timing between TLPs and the required response times.

10.2 Crafting Custom TLPs for Specific Operations

10.2.1 Implementing TLP Handling in Firmware

  • Files to Modify:

    • pcileech_pcie_tlp_a7.sv 
      pcileech-wifi-main/src/pcileech_pcie_tlp_a7.sv

  • Steps:

    1. Create TLP Generation Functions:

      • In pcileech_pcie_tlp_a7.sv, write functions to assemble TLPs with the required headers and payloads.
      • Example: 
        function automatic [127:0] generate_tlp;
  input [15:0] requester_id;
  input [7:0] tag;
  input [7:0] length;
  input [31:0] address;
  input [31:0] data;
  begin
    generate_tlp = { /* TLP Header and Payload */ };
  end
endfunction

    2. Handle TLP Reception:

      • Implement logic to parse incoming TLPs and extract necessary information.
      • Use state machines to manage different TLP types.

    3. Ensure Compliance:

      • Verify that the TLPs conform to the PCIe specification regarding format and timing.

    4. Implement Completion Handling:

      • For Memory Read Requests, generate appropriate Completion TLPs.

    5. Save Changes:

      • Save the file after implementing the changes.

10.2.2 Handling Different TLP Types

  • Memory Read Requests:

    • Implementation:

      • Parse the request header.
      • Fetch data from the appropriate memory location.
      • Assemble and send a Completion TLP with the data.

  • Memory Write Requests:

    • Implementation:

      • Receive the TLP and extract the data payload.
      • Write the data to the specified memory location.

  • Configuration Read/Write Requests:

    • Implementation:

      • Access the configuration space registers.
      • For reads, return the requested data.
      • For writes, update the register values.

  • Vendor-Defined Messages:

    • Implementation:

      • Implement parsing and response logic for any vendor-specific messages as per the donor device's protocol.

10.2.3 Validating TLP Timing and Sequence

  • Steps:

    1. Use Simulation Tools:

      • Simulate the firmware using test benches to validate TLP handling.

    2. Monitor with ILA:

      • Insert an ILA core to capture TLP-related signals during hardware testing.

    3. Check Timing Constraints:

      • Ensure that TLPs are processed and responded to within the allowed timing windows specified by the PCIe standard.

    4. Compliance Testing:

      • Use PCIe compliance tools to verify adherence to the standard.

    5. Save Changes:

      • Save all modified files after testing and validation.

Part 3: Advanced Techniques and Optimization

11. Building, Flashing, and Testing

After all customizations, it's time to build the firmware, program it onto the FPGA, and thoroughly test it to ensure proper functionality.

11.1 Synthesis and Implementation

11.1.1 Running Synthesis

Synthesis converts your high-level code into a gate-level representation.

  • Steps:

    1. Start Synthesis:

      • In Vivado, click Run Synthesis in the Flow Navigator.

    2. Monitor Progress:

      • Watch for any warnings or errors.
      • Common Warnings:
        • Unconnected Ports: Ensure all necessary signals are connected.
        • Timing Constraints Not Met: May need to adjust constraints.

    3. Review Synthesis Report:

      • Check the Utilization Summary to ensure the design fits on the FPGA.

11.1.2 Running Implementation

Implementation maps the synthesized design onto the FPGA's resources.

  • Steps:

    1. Start Implementation:

      • After successful synthesis, click Run Implementation.

    2. Analyze Timing Reports:

      • Ensure that all timing constraints are met.
      • Address Violations:
        • Adjust logic or constraints to fix setup or hold time violations.

    3. Verify Placement:

      • Check that critical components are placed optimally.

11.1.3 Generating Bitstream

The bitstream is the binary file used to program the FPGA.

  • Steps:

    1. Generate Bitstream:

      • Click Generate Bitstream.

    2. Wait for Completion:

      • This may take some time depending on design complexity.

    3. Review Bitstream Generation Log:

      • Ensure no errors occurred during generation.

11.2 Flashing the Bitstream

11.2.1 Connecting the FPGA Device

  • Steps:

    1. Prepare Hardware:

      • Ensure the FPGA board is powered and connected via JTAG.
      • Refer to your FPGA board's manual for specific connection instructions.

    2. Open Hardware Manager:

      • In Vivado, navigate to Flow Navigator > Program and Debug > Open Hardware Manager.

11.2.2 Programming the FPGA

  • Steps:

    1. Connect to the Target:

      • In the Hardware Manager, click Open Target and select Auto Connect.
      • Vivado should detect your FPGA device.

    2. Program Device:

      • In the Hardware window, right-click on your FPGA device and select Program Device.
      • Select the generated bitstream file (with .bit extension).
      • Click Program to flash the firmware onto the FPGA.
      • Wait for the programming process to complete.

11.2.3 Verifying Programming

  • Steps:

    1. Check Status:

      • Ensure the programming completes without errors.
      • Vivado will display a success message upon completion.

    2. Observe LEDs or Indicators:

      • Some FPGA boards have LEDs indicating successful programming or active status.

11.3 Testing and Validation

11.3.1 Verifying Device Enumeration

  • Windows:

    • Steps:
      1. Open Device Manager:
        • Press Win + X and select Device Manager.
      2. Check Device Properties:
        • Look under the appropriate device category (e.g., Network Adapters, Storage Controllers).
        • Confirm that the Device ID, Vendor ID, and other identifiers match those of the donor device.

  • Linux:

    • Steps:
      1. Use lspci: 
        lspci -nn
      2. Verify Device Listing:
        • Check that the emulated device appears with correct IDs.
        • Example Output: 
          03:00.0 Network controller [0280]: VendorID DeviceID

11.3.2 Testing Device Functionality

  • Steps:

    1. Install Necessary Drivers:

      • Use the donor device's drivers if required.
      • Install them as per the manufacturer's instructions.

    2. Perform Functional Tests:

      • Run applications that interact with the device.
      • Test data transfers, configurations, and any special functions.
      • Examples:
        • For a network card, perform ping tests or data streaming.
        • For a storage controller, perform read/write operations.

    3. Monitor System Behavior:

      • Check for system stability and absence of errors.
      • Ensure that the device behaves as expected under various workloads.

11.3.3 Monitoring for Errors

  • Windows:

    • Steps:
      1. Check Event Viewer:
        • Press Win + X and select Event Viewer.
        • Navigate to Windows Logs > System.
      2. Look for PCIe-Related Errors:
        • Search for warnings or errors related to PCIe or the specific device.

  • Linux:

    • Steps:
      1. Check dmesg Logs: 
        dmesg | grep pci
      2. Identify Issues:
        • Look for messages indicating problems with PCIe communication or device initialization.

12. Advanced Debugging Techniques

When issues arise, advanced debugging tools and techniques can help identify and resolve problems efficiently.

12.1 Using Vivado's Integrated Logic Analyzer

The Integrated Logic Analyzer (ILA) allows real-time monitoring of internal FPGA signals.

12.1.1 Inserting ILA Cores

  • Steps:

    1. Add ILA IP Core:

      • In Vivado, open the IP Catalog.
      • Search for ILA.
      • Instantiate the ILA core in your design.

    2. Connect Signals:

      • Attach the signals you wish to monitor to the ILA probes.
      • Example: 
        ila_0 your_ila_instance (
  .clk(clk),
  .probe0(signal_to_monitor)
);
      • File Path: 
        pcileech-wifi-main/src/pcileech_squirrel_top.sv

12.1.2 Configuring Trigger Conditions

  • Steps:

    1. Set Probe Properties:

      • Define the width of each probe to match the signal widths.

    2. Define Triggers:

      • In the ILA dashboard, set conditions that will trigger data capture.
      • Example:
        • Trigger when a specific TLP type is detected or when an error condition occurs.

12.1.3 Capturing and Analyzing Data

  • Steps:

    1. Run the Design:

      • Program the FPGA with the ILA-enabled bitstream.

    2. Open Hardware Manager:

      • Access the ILA interface within Vivado.

    3. Capture Data:

      • Arm the ILA and wait for the trigger condition.
      • Once triggered, the ILA will capture the waveform data.

    4. Analyze Waveforms:

      • Use the waveform viewer to inspect signal behavior.
      • Identify anomalies or verify correct operation.

12.2 PCIe Traffic Analysis Tools

Using external tools can provide deeper insights into PCIe communications.

12.2.1 PCIe Protocol Analyzers

  • Examples:

    • Teledyne LeCroy PCIe Analyzers
    • Keysight PCIe Analyzers

  • Steps:

    1. Set Up Analyzer:

      • Connect the analyzer between the host system and FPGA device.

    2. Configure Capture Settings:

      • Define the scope of data to capture (e.g., specific TLP types, error conditions).

    3. Capture Traffic:

      • Record PCIe transactions during device operation.

    4. Analyze Results:

      • Examine TLPs for compliance and correctness.
      • Identify any protocol violations or unexpected behaviors.

12.2.2 Software-Based Tools

  • Examples:

    • Wireshark with PCIe Plugins
    • ChipScope Pro (for Xilinx devices)

  • Steps:

    1. Install Necessary Plugins:

      • Ensure PCIe support is enabled in the tool.

    2. Monitor PCIe Bus:

      • Capture and display PCIe packets.

    3. Analyze Communications:

      • Look for anomalies or errors in the data.
      • Verify that TLPs are correctly formed and sequenced.

13. Troubleshooting

This section provides solutions to common problems you may encounter during firmware development and testing.

13.1 Device Detection Issues

Problem: The FPGA device is not recognized by the host system.

Possible Causes and Solutions:

  1. Incorrect Device IDs:

    • Cause: Mismatch between the IDs in the firmware and what the host expects.
    • Solution: Verify and correct the Device ID, Vendor ID, and Subsystem ID in your firmware.

  2. PCIe Link Training Failure:

    • Cause: The PCIe link is not established.
    • Solution:
      • Check physical connections.
      • Ensure the Link Width and Link Speed are correctly configured.

  3. Power Issues:

    • Cause: Insufficient power to the FPGA device.
    • Solution: Verify power supply connections and voltage levels.

  4. Firmware Errors:

    • Cause: Errors in the firmware prevent proper operation.
    • Solution: Review code for syntax errors or misconfigurations.

13.2 Memory Mapping and BAR Configuration Errors

Problem: The device's memory regions are not accessible, or accessing them causes system errors.

Possible Causes and Solutions:

  1. Incorrect BAR Sizes or Types:

    • Cause: BAR configurations do not match the donor device.
    • Solution: Adjust BAR sizes and types in both the PCIe IP Core and firmware.

  2. Address Decoding Errors:

    • Cause: Firmware fails to correctly interpret addresses.
    • Solution: Debug the address decoding logic in your firmware.

  3. Overlapping Address Spaces:

    • Cause: BARs overlap or conflict with other devices.
    • Solution: Ensure BAR addresses are correctly aligned and do not overlap.

13.3 DMA Performance and TLP Errors

Problem: Slow data transfer rates or errors occur during DMA operations, or TLPs are malformed.

Possible Causes and Solutions:

  1. Inefficient DMA Logic:

    • Cause: DMA engine not optimized.
    • Solution: Implement buffering and pipelining to enhance throughput.

  2. Malformed TLPs:

    • Cause: Incorrect TLP formatting.
    • Solution: Review TLP assembly code to ensure compliance with the PCIe specification.

  3. Flow Control Issues:

    • Cause: Improper handling of flow control credits.
    • Solution: Implement proper flow control mechanisms in the firmware.

14. Emulation Accuracy and Optimizations

Enhancing emulation accuracy ensures compatibility and performance, making the emulated device indistinguishable from the donor.

14.1 Techniques for Accurate Timing Emulation

  • Implement Timing Constraints:

    • Use Vivado's timing constraints to match the timing characteristics of the donor device.
    • Apply constraints to critical paths to ensure they meet the required setup and hold times.

  • Use Clock Domain Crossing (CDC) Techniques:

    • Properly handle signals crossing between different clock domains to prevent metastability.
    • Use synchronizers or FIFOs as appropriate.

  • Simulate Device Behavior:

    • Use simulation tools to model and verify the device's behavior under different conditions.
    • Validate that the timing and sequencing of operations match the donor device.

14.2 Dynamic Response to System Calls

  • Implement State Machines:

    • Design state machines that allow the device to respond dynamically to various commands and states.
    • Ensure that the device can handle unexpected or out-of-order requests gracefully.

  • Monitor and Respond to Host Commands:

    • Implement logic to decode and respond to configuration writes, vendor-specific commands, and other interactions.
    • Update internal registers and states accordingly.

  • Optimize Firmware Logic:

    • Refine the firmware code to reduce latency and improve responsiveness.
    • Remove unnecessary delays or bottlenecks in the data paths.

15. Best Practices for Firmware Development

Following best practices helps maintain code quality, facilitates collaboration, and ensures the longevity of your project.

15.1 Continuous Testing and Documentation

  • Regular Testing:

    • Test the firmware after each significant change to catch issues early.
    • Use test benches and simulation to validate logic before implementation.

  • Automated Testing:

    • Implement automated test scripts to verify functionality and performance.
    • Use continuous integration tools if working in a team environment.

  • Maintain Documentation:

    • Document the design, including block diagrams, state machines, and interfaces.
    • Keep track of changes with clear commit messages and update design documents accordingly.

15.2 Managing Firmware Versioning

  • Use Version Control Systems:

    • Employ systems like Git to manage code versions and collaborate with others.
    • Organize the repository with clear directory structures and naming conventions.

  • Tag Releases and Milestones:

    • Tag stable versions of the firmware for future reference.
    • Use branches for experimental features or major changes.

  • Backup and Recovery:

    • Regularly backup your work to prevent data loss.
    • Use cloud-based repositories or local backups as appropriate.

15.3 Security Considerations

  • Secure Coding Practices:

    • Follow guidelines to prevent common vulnerabilities such as buffer overflows or race conditions.
    • Validate all inputs and handle errors gracefully.

  • Data Protection:

    • Ensure that any sensitive data handled by the device is protected.
    • Implement encryption or access controls if necessary.

  • Compliance and Ethics:

    • Be aware of legal and ethical considerations related to device emulation.
    • Ensure compliance with relevant laws, regulations, and licensing agreements.

16. Additional Resources

Enhance your understanding and stay updated with the following resources:

17. Contact Information

If you need assistance, have questions, or wish to collaborate, feel free to reach out. I'm available to provide guidance, troubleshoot complex problems, or discuss ideas in detail.

Discord: VCPU | Server Invite Link

18. Support and Contributions

Your support helps maintain and improve this guide and related projects.

Donations

  • Crypto Donations (LTC):
    • Address: MPMyQD5zgy2b2CpDn1C1KZ31KmHpT7AwRi

If you found this guide helpful and want to support ongoing work, consider contributing. Every donation helps in continuing to create, share, and support the community.

Special Bonus: If you donate, reach out on Discord (VCPU) to receive a personal thank you and possibly additional resources or assistance.

Note: If you need me to review your implementation or troubleshoot issues, please mark the relevant sections with //VCPU-REVIEW// and provide detailed explanations of the problems you're encountering.

End of Guide