Introduction:
PCI Express is a serial expansion bus standard operating at multi-gigabit data rates. It is the third generation, high performance I/O bus which is used for interconnecting peripheral devices. PCI Express provides higher rate of data transfer and lower latency compared to the older PCI and PCI-X technologies which implemented parallel I/O bus. Each device which is connected to the motherboard via PCI Express link actually has a dedicated point-to-point connection and as it is not sharing the same bus, it doesn’t have to compete for bandwidth. PCI Express is based on the point-to-point topology where there are dedicated serial links connecting every device to the root complex. This article implements a simple design to demonstrate how to write and read data to Nereid Kintex 7 PCI Express Development Board which acts as a PCI Express endpoint device. Lets get started!
Hardware required:
- Host PC with Linux or Windows (Linux preferred)
- Nereid Kintex 7 PCI Express FPGA Development Board
- Xilinx Platform Cable USB II (JTAG cable)
Software required:
- Vivado 2023.2.1
- RW-Everything (for Windows host)
Step 1:
Download and install Vivado Board Support Package files for Nereid from here. Follow the README.md file on how to install Vivado Board Support Package files for Numato Lab’s boards.
Step 2:
Open Vivado and select “Create Project” from the “Quick Start” section.
Step 3:
Click Next and then in Project Name tab, give appropriate project name and location and click “Next” till you see the ‘Default Part’ tab.
Step 4:
Select “Boards” from “Default Part” tab, choose “numato.com” as Vendor. Select “Nereid” and click “Next”. If Nereid is not listed, make sure board support files are installed correctly. Click “Finish”. A new Vivado project will open with the selected settings.
Step 5:
Click “Next” and then “Finish” to create the project. Click “Create Block Diagram” from IP integrator tab on left, give a name for block design and click OK.
Step 6:
Click “Add IP” from the toolbar as shown in the image below. Type “pcie” in the search box and double click “7 Series Integrated Block for PCI Express” IP to customize it.
Step 7:
The “Re-customize IP” window should open. In Basic tab, Select “Lane Width” as “X4”, “Maximum Link Speed” as “5.0 GT/s”. In “BARs” Tab, select Size Value as “512”. Leave the other tabs in their default state and click OK.
Step 8:
Right-click on the pcie_7x_0 IP block in the block diagram and click “Open Example IP design…”.
It will open an example project for the PCI Express Endpoint Device as per the customised IP settings. This tutorial uses this generated example project by Xilinx. It already has RTL logic enabling users to write data to FPGA and read back from it via PCI Express.
Step 9:
Open .xdc file from “Constraints” category under Sources “Hierarchy” tab.
Step 10:
Select everything and delete it. Simply copy the following constraints in that file.
set_property PACKAGE_PIN E21 [get_ports sys_rst_n]
set_property IOSTANDARD LVCMOS33 [get_ports sys_rst_n]
set_property -dict {PACKAGE_PIN H1} [get_ports {pci_exp_txn[0]}]
set_property -dict {PACKAGE_PIN H2} [get_ports {pci_exp_txp[0]}]
set_property -dict {PACKAGE_PIN K1} [get_ports {pci_exp_txn[1]}]
set_property -dict {PACKAGE_PIN K2} [get_ports {pci_exp_txp[1]}]
set_property -dict {PACKAGE_PIN M1} [get_ports {pci_exp_txn[2]}]
set_property -dict {PACKAGE_PIN M2} [get_ports {pci_exp_txp[2]}]
set_property -dict {PACKAGE_PIN P1} [get_ports {pci_exp_txn[3]}]
set_property -dict {PACKAGE_PIN P2} [get_ports {pci_exp_txp[3]}]
set_property -dict {PACKAGE_PIN J3} [get_ports {pci_exp_rxn[0]}]
set_property -dict {PACKAGE_PIN J4} [get_ports {pci_exp_rxp[0]}]
set_property -dict {PACKAGE_PIN L3} [get_ports {pci_exp_rxn[1]}]
set_property -dict {PACKAGE_PIN L4} [get_ports {pci_exp_rxp[1]}]
set_property -dict {PACKAGE_PIN N3} [get_ports {pci_exp_rxn[2]}]
set_property -dict {PACKAGE_PIN N4} [get_ports {pci_exp_rxp[2]}]
set_property -dict {PACKAGE_PIN R3} [get_ports {pci_exp_rxn[3]}]
set_property -dict {PACKAGE_PIN R4} [get_ports {pci_exp_rxp[3]}]
set_property LOC IBUFDS_GTE2_X0Y3 [get_cells refclk_ibuf]
create_clock -name sys_clk -period 10 [get_ports sys_clk_p]
set_property PACKAGE_PIN K6 [get_ports sys_clk_p]
set_property PACKAGE_PIN K5 [get_ports sys_clk_n]
set_false_path -to [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]
set_false_path -to [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S1}]
create_generated_clock -name clk_125mhz_x0y0 [get_pins pcie_7x_0_support_i/pipe_clock_i/mmcm_i/CLKOUT0]
create_generated_clock -name clk_250mhz_x0y0 [get_pins pcie_7x_0_support_i/pipe_clock_i/mmcm_i/CLKOUT1]
create_generated_clock -name clk_125mhz_mux_x0y0 \
-source [get_pins pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/I0] \
-divide_by 1 \
[get_pins pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/O]
create_generated_clock -name clk_250mhz_mux_x0y0 \
-source [get_pins pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/I1] \
-divide_by 1 -add -master_clock [get_clocks -of [get_pins pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/I1]] \
[get_pins pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/O]
set_clock_groups -name pcieclkmux -physically_exclusive -group clk_125mhz_mux_x0y0 -group clk_250mhz_mux_x0y0
set_false_path -from [get_ports sys_rst_n]
set_property BITSTREAM.CONFIG.CONFIGRATE 16 [current_design]
set_property BITSTREAM.GENERAL.COMPRESS TRUE [current_design]
set_property BITSTREAM.CONFIG.SPI_BUSWIDTH 1 [current_design]
Step 11:
Right-Click “Generate Bitstream” and select “Bitstream Settings”
Step 12:
In “Bitstream Settings”, select “-bin_file” and click OK.
Step 13:
In Project Manager Tab, click “Run Synthesis”. If synthesis is successful, click “Run Implementation”. On successful completion of implementation step, select “Generate Bitstream”.
Step 14:
After Successful generation of bitstream, Open Hardware Manager.
Step 15:
Make sure Nereid Board is connected to JTAG and power supply. Click on ‘Open target’ and ‘Auto Connect’. Vivado Hardware Manager will connect to Nereid.
Step 16:
Left-click on the device and select ‘Add Configuration Memory Device…’. It will open a new window. Select Manufacturer as ‘ISSI’, Density(Mb) as ‘128’ and Type as ‘spi’. Select device ‘is251p128f-spi-x1_x2_x4’ and click on OK.
Step 17:
A Dialogue window will appear asking if you want to program the configuration device now. Click Yes and program the .bin bitstream file which is located in “<project location>/ProjectName.runs/impl_1” directory.
Step 18:
Once the device is programmed, test it on a Windows or Linux machine.
Communicating with Nereid via PCI Express on Linux Machines:
Step 1:
Download the complete pcimem application code zip file and unzip it in a specific location. Go to the command line terminal and check the PCIe base address by using the command
lspci -vv the output of the command is shown below. Make sure the Nereid Board is inserted correctly into the PCIe Slot of host system’s motherboard. If the host is unable to detect Nereid (which should show up as “RAM memory: Xilinx Corporation” as in the image below), make sure the board is inserted correctly into PCIe Slot and do a soft-reset after the host is powered up. A soft-reset after the host is powered up helps the host detect FPGA based PCIe devices.
Step 2:
In command line terminal, open the path where you saved the ‘pciemem’ code. First compile the C program by using the command “make”. Once it is compiled successfully, use the following command:
sudo ./pcimem /dev/mem 4273994752 w 0xffffff12
Here,
4273994752: indicates the base address + offset, it is the address to which write is performed and it is the decimal value of hex: 0xfebffc00 (in the above image)
w: indicates whether it is word, byte or half word.
0xffffff12: 32-bit data value for write purpose.
You will observe the following output indicating that the 32-bit data has been written to the specified address and read back from it. If the written data matches the data read, it means data was successfully written to Nereid.
Communicating with Nereid via PCI Express on Windows Machines:
Step 1:
For Windows machine we will use RW-Everything software to write data.
Step 2:
Insert Nereid in the PCI Express slot of the host system’s motherboard. Power up the host and then do a soft-restart again. Boot into Windows. Open RW-Everything and click on “PCI Devices”. It will open PCI Devices window.
Step 3:
Select Xilinx PCIe Device from dropdown list.
Step 4:
Locate the BAR Address from the addresses on the left-side.
Step 5:
Double-click to open the BAR Address and select any one of the address memory location and write some data to it. The data is persistent on Nereid as long as it is powered up. We can verify it by closing and re-opening the application, which should show the data written previously, indicating that the write operation was successful.
So, this was a basic introduction into getting started with PCI Express using Nereid Kintex 7 PCI Express FPGA Board. PCI Express offers lot more capability such as DMA transfers and bus mastering. High-performance PCI Express projects will most necessarily need custom drivers for either Windows or Linux, depending on the Operating System which is to be used. This article is just a start of the big journey into PCI Express. We encourage you to keep moving forward!