Introduction:
PCI Express is a serial expansion bus standard operating at multi-gigabit data rates. It is the third generation, high-performance I/O bus which is used for interconnecting peripheral devices. PCI Express provides a higher rate of data transfer and lower latency compared to the older PCI and PCI-X technologies which implemented parallel I/O buses. Each device that is connected to the motherboard via PCI Express link actually has a dedicated point-to-point connection and as it is not sharing the same bus, it doesn’t have to compete for bandwidth. PCI Express is based on the point-to-point topology where dedicated serial links are connecting every device to the root complex. This article implements a simple design to demonstrate how to write and read data to the Tagus – Artix 7 PCI Express Development Board which acts as a PCI Express endpoint device. Let’s get started!
Hardware required:
- Host PC with Linux or Windows (Linux preferred)
- Tagus – Artix 7 PCI Express Development Board
- Xilinx Platform Cable USB II (JTAG cable)
Software required:
- Xilinx Vivado Design Suite 2023.2.1
- RW-Everything (for Windows host)
Step 1:
Open Xilinx Vivado Design Suite. Go to File -> Project -> New. The “New Project” wizard will open. Click “Next”.
Step 2:
Enter a name and location for the project. This article uses the project name “PCIe” but feel free to name it as per your preference. Select the checkbox “Create project subdirectory” as shown in the image below and proceed with the “Next” button.
Step 3:
Select “RTL Project” as project type, and select the checkbox as shown below:
Step 4:
At the “Default Part” step, select the “Boards” tab. Select “Tagus” from the list. If the Tagus is not listed, make sure board support files are installed correctly. After selecting the board, click on Next then Finish. The project will be created.
Step 5:
Under Flow Navigator, select “Create Block Design” in IP Integrator. Give a name to the block design. This article used “PCIe” for the block design name. You can use the name as per your preference.
Step 6:
Go to the Diagram window, right-click and select “Add IP” from the popup menu. Search for “7 Series Integrated Block for PCI Express”. Add it to the block design by double-clicking.
Step 7:
Double click on pcie_7x_0 IP to customize the IP. In the Re-customize IP window, go to the Basic tab and select Lane Width as X1, Maximum Link Speed as 5.0 GT/s. Leave the other tabs in their default state and click OK.
Step 8:
Save the design and then right-click on the pcie_7x_0 IP block in the block diagram and select “Open Example IP design”. Enter a location as to where the example project has to be created and then click “OK”.
It will open an example project for the PCI Express Endpoint Device as per the customized IP settings. This tutorial uses this generated example project by Xilinx. It already has RTL logic enabling users to write data to FPGA and read back from it via PCI Express.
Step 9:
In the pcie_7x_0 IP example design, there is a user_lnk_up logic to indicate that the PCIe link between the host PC and the FPGA is ready to exchange the data when we connect the FPGA board to the PCIe slot of the motherboard. Aller board features an RGB led. Hence, connect the user_lnk_up logic output to the blue led and connect the complement output of user_lnk_up to the green led. The blue led will glow when the PCIe link is ready to exchange the data and the green led will glow when the PCIe link is not ready or when the host PC and FPGA are attempting to establish communication or when communication with the FPGA is lost due to errors on the transmission channel. So by connecting the user_lnk_up to these LEDs, we can observe that the PCIe link between the host PC and the FPGA is ready to exchange the data or not. Also, add the counter to check whether the PCIe clock is working or not, and assign the counter output to red led. So that we can observe the RGB led blinking when the board is detected by the host PC.
To add this logic, open the Verilog file from the Design Sources category under the Sources tab
In the Verilog code, in the ports declarations, add the following:
And under Wire Declarations, add the following:
And then under the I/O Buffers section, add the counter implementation:
Step 10:
Open the .xdc file from the “Constraints” category under the Sources tab.
Select everything and delete it. Simply copy the following constraints in that file.
set_property PACKAGE_PIN W20 [get_ports sys_rst_n]
set_property IOSTANDARD LVCMOS33 [get_ports sys_rst_n]
# set_property PULLUP true [get_ports sys_rst_n]
set_property PACKAGE_PIN W21 [get_ports led_red]
set_property IOSTANDARD LVCMOS33 [get_ports led_red]
set_property PACKAGE_PIN W22 [get_ports led_green]
set_property IOSTANDARD LVCMOS33 [get_ports led_green]
set_property PACKAGE_PIN AA20 [get_ports led_blue]
set_property IOSTANDARD LVCMOS33 [get_ports led_blue]
set_property LOC IBUFDS_GTE2_X0Y3 [get_cells refclk_ibuf]
set_property -dict {PACKAGE_PIN F6} [get_ports sys_clk_p]
set_property -dict {PACKAGE_PIN E6} [get_ports sys_clk_n]
set_property -dict {PACKAGE_PIN B8} [get_ports {pci_exp_rxp[0]}]
set_property -dict {PACKAGE_PIN A8} [get_ports {pci_exp_rxn[0]}]
set_property -dict {PACKAGE_PIN B4} [get_ports {pci_exp_txp[0]}]
set_property -dict {PACKAGE_PIN A4} [get_ports {pci_exp_txn[0]}]
#
create_clock -name sys_clk -period 10 [get_ports sys_clk_p]
#
#
set_false_path -to [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]
set_false_path -to [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S1}]
#
#
set_case_analysis 1 [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]
set_case_analysis 0 [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S1}]
set_property DONT_TOUCH true [get_cells -of [get_nets -of [get_pins {pcie_7x_0_support_i/pipe_clock_i/pclk_i1_bufgctrl.pclk_i1/S0}]]]
set_false_path -from [get_ports sys_rst_n]
set_property BITSTREAM.GENERAL.COMPRESS TRUE [current_design]
set_property BITSTREAM.CONFIG.CONFIGRATE 66 [current_design]
set_property CONFIG_VOLTAGE 3.3 [current_design]
set_property CFGBVS VCCO [current_design]
set_property BITSTREAM.CONFIG.SPI_BUSWIDTH 4 [current_design]
set_property BITSTREAM.CONFIG.SPI_FALL_EDGE YES [current_design]
Step 11:
Now, click on “Generate Bitstream” under the “PROGRAM AND DEBUG” section to synthesis, implement and to generate the bitstream.
Step 12:
To program the board, open the target by clicking on the “Open Target” in “Open Hardware Manager” in the “Program and Debug” section of the Flow Navigator window. Select “Auto Connect”.
If the device is detected successfully, then select “Add Configuration Memory Device” by right click on the target device “xc7a100t_0” as shown below.
Select the memory device “mt25ql128-spi-x1_x2_x4 (which is equivalent to n25q128-3.3v-spi-x1_x2_x4)”, then click OK.
A dialogue window will appear asking if you want to program the configuration device now. Click OK and program the .bin bitstream file which is located in “<project location>/ProjectName.runs/impl_1” directory.
Step 13:
Once the device is programmed, test it on a Windows or Linux machine.
Communicating with the Tagus via PCI Express on Linux Machines:
Step 1:
Download the complete pcimem application code zip file and unzip it in a specific location. Go to the command line terminal and check the PCIe base address by using the command
lspci -vv the output of the command is shown below. Make sure the Tagus Board is inserted correctly into the PCIe Slot of the host system’s motherboard. If the host is unable to detect Tagus (which should show up as “Memory controller: Xilinx Corporation Device 7021” as in the image below), make sure the board is inserted correctly into PCIe Slot and do a soft reset after the host is powered up. A soft-reset after the host is powered up helps the host detect FPGA-based PCIe devices.
Step 2:
In the command-line terminal, open the path where you saved the ‘pciemem’ code. First, compile the C program by using the command “make”. Once it is compiled successfully, use the following command:
sudo ./pcimem /dev/mem 4156555264 w 0xffffff12
Here,
4156555264: indicates the base address + offset, it is the address to which write is performed and it is the decimal value of hex: 0xf7c00000 (in the above image)
w: indicates whether it is a word, a byte, or a half-word.
0xffffff12: 32-bit data value for write purpose.
You will observe the following output indicating that the 32-bit data has been written to the specified address and read back from it. If the written data matches the data read, it means data was successfully written to the Tagus.
Communicating with the Tagus via PCI Express on Windows Machines:
Step 1:
For Windows machines, use RW-Everything software to write data.
Step 2:
Insert Tagus board in PCIe slot of host system’s motherboard. Open RW-Everything, Click PCI Devices. It will open PCI Devices.
Step 3:
Select Xilinx PCIe Device from the drop-down list.
Step 4:
Locate the BAR Address from the addresses on the left side.
Step 5:
Double-click to open the BAR Address and select any one of the address bits and write some data to it. The data is retained even if you close and open the application indicating that the write operation is successful.
