Introduction:
PCI Express is a serial expansion bus standard operating at multi-gigabit data rates. It is the third generation, high performance I/O bus which is used for interconnecting peripheral devices. PCI Express provides a higher rate of data transfer and lower latency compared to the older PCI and PCI-X technologies which implemented parallel I/O buses. Each device that is connected to the motherboard via PCI Express link has a dedicated point-to-point connection and as it is not sharing the same bus, it doesn’t have to compete for bandwidth. PCI Express is based on the point-to-point topology where dedicated serial links are connecting every device to the root complex. This article implements a simple design to demonstrate how to write and read data to Aller Artix-7 FPGA Board with M.2 Interface which acts as a PCI Express endpoint device. Let us get started!
Hardware required:
- Host PC with Linux or Windows (Linux preferred)
- Aller Artix-7 FPGA Board with M.2 Interface
- Xilinx Platform Cable USB II (JTAG cable)
Software required:
- Xilinx Vivado Design Suite 2023.2.1
- RW-Everything (for Windows host)
Step 1:
Download and install Vivado Board Support Package files for Aller_200t from here. Follow the README.md file on how to install Vivado Board Support Package files for Numato Lab’s boards.
Step 2:
Open Xilinx Vivado Design Suite. Go to File -> Project -> New. The “New Project” wizard will open. Click “Next”.
Step 3:
In the Project Name tab, give the appropriate project name and location and click “Next”.
Step 4:
In the “Project Type” window, select “RTL Project” and select the option “Do not specify sources at this time”. Click “Next”.
Step 5:
In the “Default Part” wizard, select Boards and choose “numato.com” as Vendor. Select “Aller_200T” and click “Next”. If Aller_200T is not listed, make sure board support files are installed correctly. Click “Finish” in the next wizard. A new Vivado project will open with the selected settings.
Step 6:
Under Flow Navigator, select “Create Block Design” in IP Integrator. Give a name to the block design. This article used “system” for the block design name. You can use the name as per your preference.
Step 7:
Go to the Diagram window and click “Add IP” from the toolbar as shown in the image below. Type “PCIe” in the search box and double click “7 Series Integrated Block for PCI Express” IP to customize it.
Step 8:
Double click on pcie_7x_0 IP, this will opens the “Re-customize IP” window. In the Basic tab, Select “Lane Width” as “X4”, “Maximum Link Speed” as “5.0 GT/s”. Leave the other tabs in their default state and click “OK”.
Step 9:
Save the design and then right-click on the pcie_7x_0 IP block in the block diagram and select “Open Example IP design”. Enter a location as to where the example project has to be created and then click “OK”.
It will open an example project for the PCI Express Endpoint Device as per the customized IP settings. This tutorial uses this generated example project by Xilinx. It already has RTL logic enabling users to write data to FPGA and read back from it via PCI Express.
Step 10:
In the pcie_7x_0 IP example design, there is a user_lnk_up logic to indicate that the PCIe link between the host PC and the FPGA is ready to exchange the data when we connect the FPGA board to the PCIe slot of the motherboard. Aller board features an RGB led. Hence, connect the user_lnk_up logic output to the blue led and connect the complement output of user_lnk_up to the green led. The blue led will glow when the PCIe link is ready to exchange the data and the green led will glow when the PCIe link is not ready or when the host PC and FPGA are attempting to establish communication or when communication with the FPGA is lost due to errors on the transmission channel. So by connecting the user_lnk_up to these LEDs, we can observe that the PCIe link between the host PC and the FPGA is ready to exchange the data or not. Also, add the counter to check whether the PCIe clock is working or not, and assign the counter output to red led. So that we can observe the RGB led blinking when the board is detected by the host PC.
To add this logic in the example design, open the Verilog file from the Design Sources category in the “Sources”.
Declare the RGB led ports in the Ports declaration of the Top module(xlinx_pcie_2_1_ep_7x.v).
And under Wire Declarations, declare the ‘counter’ variable as register and assign the ‘user_lnk_up’, ‘compliment of user_lnk_up’ and ‘counter’ output to RGB led.
And then add this counter implementation after the I/O BUFFERS instantiation.
Step 11:
Open the .xdc file from the “Constraints” category under the Sources “Hierarchy” tab.
Step 12:
Select everything and delete it. Simply copy the following constraints in that file.
set_property PACKAGE_PIN AB20 [get_ports sys_rst_n]
set_property IOSTANDARD LVCMOS33 [get_ports sys_rst_n]
set_property PULLUP true [get_ports sys_rst_n]
set_property -dict {PACKAGE_PIN B8} [get_ports {pci_exp_rxp[0]}]
set_property -dict {PACKAGE_PIN A8} [get_ports {pci_exp_rxn[0]}]
set_property -dict {PACKAGE_PIN B4} [get_ports {pci_exp_txp[0]}]
set_property -dict {PACKAGE_PIN A4} [get_ports {pci_exp_txn[0]}]
set_property -dict {PACKAGE_PIN D11} [get_ports {pci_exp_rxp[1]}]
set_property -dict {PACKAGE_PIN C11} [get_ports {pci_exp_rxn[1]}]
set_property -dict {PACKAGE_PIN D5} [get_ports {pci_exp_txp[1]}]
set_property -dict {PACKAGE_PIN C5} [get_ports {pci_exp_txn[1]}]
set_property -dict {PACKAGE_PIN B10} [get_ports {pci_exp_rxp[2]}]
set_property -dict {PACKAGE_PIN A10} [get_ports {pci_exp_rxn[2]}]
set_property -dict {PACKAGE_PIN B6} [get_ports {pci_exp_txp[2]}]
set_property -dict {PACKAGE_PIN A6} [get_ports {pci_exp_txn[2]}]
set_property -dict {PACKAGE_PIN D9} [get_ports {pci_exp_rxp[3]}]
set_property -dict {PACKAGE_PIN C9} [get_ports {pci_exp_rxn[3]}]
set_property -dict {PACKAGE_PIN D7} [get_ports {pci_exp_txp[3]}]
set_property -dict {PACKAGE_PIN C7} [get_ports {pci_exp_txn[3]}]
set_property -dict { PACKAGE_PIN "AB21" IOSTANDARD LVCMOS33 SLEW FAST} [get_ports { led_red }]
set_property -dict { PACKAGE_PIN "AB22" IOSTANDARD LVCMOS33 SLEW FAST} [get_ports { led_green }]
set_property -dict { PACKAGE_PIN "U20" IOSTANDARD LVCMOS33 SLEW FAST} [get_ports { led_blue }]
create_clock -name sys_clk -period 10 [get_ports sys_clk_p]
set_property PACKAGE_PIN F6 [get_ports sys_clk_p]
set_property PACKAGE_PIN E6 [get_ports sys_clk_n]
set_false_path -to [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]
set_false_path -to [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S1}]
set_case_analysis 1 [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]
set_case_analysis 0 [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S1}]
set_property DONT_TOUCH true [get_cells -of [get_nets -of [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]]]
set_false_path -from [get_ports sys_rst_n]
set_property BITSTREAM.CONFIG.CONFIGRATE 16 [current_design]
set_property BITSTREAM.GENERAL.COMPRESS TRUE [current_design]
set_property BITSTREAM.CONFIG.SPI_BUSWIDTH 4 [current_design]
Step 13:
Right-Click on “Generate Bitstream” under the PROGRAM AND DEBUG section of Flow Navigator. and select “Bitstream Settings”.
Step 14:
In “Bitstream Settings”, select “-bin_file” and click “OK”.
Step 15:
Now, in the Project Manager Tab, click on “Generate Bitstream” under the “PROGRAM AND DEBUG” section to synthesis, implement and to generate the bitstream.
Note:
Before generating the bitstream, go to the project folder, and browse to this location: pcie_7x_0_ex.srcs\sources_1\ip\pcie_7x_0\source and change the site location to as shown in the image below in the pcie_7x_0-PCIE_X0Y0.xdc file.
Step 16:
After Successful generation of the bitstream, Open Hardware Manager.
Step 17:
Make sure Aller Board is connected to JTAG and power supply. Click on ‘Open target’ and ‘Auto Connect’. Vivado Hardware Manager will connect to Aller.
Step 18:
Right-click on the device and select ‘Add Configuration Memory Device…’. It will open a new window. Select Manufacturer as ‘Micron’, Density(Mb) as ‘1024’ and Type as ‘spi’. Select device ‘mt25ql 01g-spi-x1_x2_x4’ and click on OK.
Step 19:
A Dialogue window will appear asking if you want to program the configuration device now. Click Yes and program the .bin bitstream file which is located in “<project location>/ProjectName.runs/impl_1” directory.
Step 20:
Once the device is programmed, test it on a Windows or Linux machine.
Communicating with Aller via PCI Express on Linux Machines:
Step 1:
Download the complete pcimem application code zip file and unzip it in a specific location. Go to the command line terminal and check the PCIe base address by using the command
lspci -vv the output of the command is shown below. Make sure the Aller Board is inserted correctly into the PCIe Slot of the host system’s motherboard. If the host is unable to detect Aller (which should show up as “Memory controller: Xilinx Corporation Device 7024” as in the image below), make sure the board is inserted correctly into PCIe Slot and do a soft reset after the host is powered up. A soft-reset after the host is powered up helps the host detect FPGA-based PCIe devices.
Step 2:
In the command line terminal, open the path where you saved the ‘pciemem’ code. First, compile the C program by using the command “make”. Once it is compiled successfully, use the following command:
sudo ./pcimem /dev/mem f7000000 w 0xffffff14
Here,
f7000000: indicates the base address + offset, it is the address to which write is performed.
w: indicates whether it is word, byte or half-word.
0xffffff14: 32-bit data value for write purpose.
You will observe the following output indicating that the 32-bit data has been written to the specified address and read back from it. If the written data matches the data read, it means data was successfully written to Aller.
Communicating with Aller via PCI Express on Windows Machines:
Step 1:
For Windows machines, use RW-Everything software to write data.
Step 2:
Insert Aller in the PCI Express slot of the host system’s motherboard. Power up the host and then do a soft restart again. Boot into Windows. Open RW-Everything and click on “PCI Devices”. It will open the PCI Devices window.
Step 3:
Select Xilinx PCIe Device from the dropdown list.
Step 4:
Locate the BAR Address from the addresses on the left side.
Step 5:
Double-click to open the BAR Address and select any one of the address memory locations and write some data to it. The data is persistent on Aller as long as it is powered up. We can verify it by closing and re-opening the application, which should show the data written previously, indicating that the write operation was successful.
So, this was a basic introduction to getting started with PCI Express using Aller Artix-7 FPGA Board with M.2 Interface. PCI Express offers a lot more capability such as DMA transfers and bus mastering. High-performance PCI Express projects will most necessarily need custom drivers for either Windows or Linux, depending on the Operating System which is to be used. This article is just a start of the big journey into PCI Express. We encourage you to keep moving forward!