Skip to content

Fix pciedriver and portalmem to support newest kernel versions #198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Fix pciedriver and portalmem to support newest kernel versions #198

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0afe2bb
Fix typos in aws f1 scripts
tchomphoochan Mar 2, 2025
aacc176
Fixes for Ubuntu 20.04 and 24.04
ljz-1109 Mar 2, 2025
2ab6a30
Uncomment unnecessarily commented code
tchomphoochan Mar 4, 2025
3551006
Fix janky hex string
tchomphoochan Mar 4, 2025
4d6b4a7
Typo extra space
tchomphoochan Mar 5, 2025
d481d2f
This condition could never occur
tchomphoochan Apr 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,3 +62,7 @@ kc705g2
ac701g2
*.ltx
zedboard_ubuntu
driverversion.h
*.o
*.d
*.mod
3 changes: 1 addition & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,6 @@ uninstall:
rm -vf $(MODULES_LOAD_D_DIR)/`basename $$fname` ; \
done;
(cd drivers/pcieportal; make uninstall)
make -C pcie/connectalutil uninstall
for fname in $(UDEV_RULES) ; do \
rm -f $(UDEV_RULES_DIR)/$$fname ; \
done
Expand Down Expand Up @@ -117,7 +116,7 @@ ifeq ($(shell uname), Darwin)
port install asciidoc
easy_install ply
else
if [ -f /usr/bin/yum ] ; then yum install gmp strace python-argparse python-ply python-gevent; else apt-get install libgmp10 strace python-ply python-gevent; fi
if [ -f /usr/bin/yum ] ; then yum install gmp strace python-argparse python-ply python3-gevent; else apt-get install libgmp10 strace python-ply python3-gevent; fi
if [ -f /usr/lib/x86_64-linux-gnu/libgmp.so ] ; then ln -sf /usr/lib/x86_64-linux-gnu/libgmp.so /usr/lib/x86_64-linux-gnu/libgmp.so.3 ; fi
if [ ! -f /usr/lib64/libgmp.so.3 ] && [ -f /usr/lib64/libgmp.so.10 ] ; then ln -s /usr/lib64/libgmp.so.10 /usr/lib64/libgmp.so.3; fi
endif
Expand Down
2 changes: 1 addition & 1 deletion boardinfo/awsf1.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"bsvdefines" : ["XILINX=1", "VirtexUltrascale", "PhysAddrWidth=40", "DataBusWidth=512", "XsimHostInterface", "AWSF1=1",
"MemTagSize=16", "MemServerTags=8",
"DEFAULT_NOPROGRAM=1",
"CONNECTAL_BITS_DEPENDENCES=build/checkpoints/to_aws/mkTop.SH_CL_routed.dcp", "CONNECTAL_RUN_SCRIPT=$(CONNECTALDIR)/scripts/run.aws"],
"CONNECTAL_BITS_DEPENDENCES=build/checkpoints/to_aws/mkTop.SH_CL_routed.dcp", "CONNECTAL_RUN_SCRIPT=$(CONNECTALDIR)/scripts/aws/run.awsf1"],
Copy link

@jameyhicks-cmt jameyhicks-cmt Mar 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this was ever really debugged, but it's good to fix the typo

"os" : "ubuntu",
"partname" : "xcvu9p-flgb2104-2-i",
"TOP" : "AwsF1Top",
Expand Down
4 changes: 2 additions & 2 deletions cpp/poller.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,8 +187,8 @@ void* PortalPoller::event(void)
if (rc < 0)
fprintf(stderr, "[%s:%d] read error %d\n", __FUNCTION__, __LINE__, errno);
for (int i = 0; i < numWrappers; i++) {
if (!portal_wrappers)
fprintf(stderr, "Poller: No portal_instances revents=%d\n", portal_fds[i].revents);
// if (!portal_wrappers)
Copy link

@jameyhicks-cmt jameyhicks-cmt Mar 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want to comment out this code

Copy link
Author

@tchomphoochan tchomphoochan Mar 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Not sure why it was commented.

// fprintf(stderr, "Poller: No portal_instances revents=%d\n", portal_fds[i].revents);
Portal *instance = portal_wrappers[i];
if (trace_poller)
fprintf(stderr, "Poller: event tile %d fpga%d fd %d handler %p parent %p\n",
Expand Down
27 changes: 24 additions & 3 deletions drivers/pcieportal/pcieportal.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@
#include <linux/poll.h> /* poll_table, etc. */
#include <asm/uaccess.h> /* copy_to_user, copy_from_user */
#include <linux/dma-buf.h>
// #include <linux/dma-mapping.h>
Copy link

@jameyhicks-cmt jameyhicks-cmt Mar 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have these files gone away? If so we need an appropriate ifdef around these lines

Copy link
Author

@tchomphoochan tchomphoochan Mar 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I'm not sure yet. I was integrating parts of the changes implemented by @ljz-1109 and must have missed that.

I can look into this and introduce necessary #ifdefs.

Copy link
Author

@tchomphoochan tchomphoochan Mar 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Not sure why it was commented. Those files still remain to these days.

Those files are needed for the recent kernel API for setting up DMA. While they seem to be transitively included through other files (hence commenting out works), it's probably best to make those includes explicit.

// #include <linux/dma-direction.h>
#include <linux/pci_regs.h>
#include "driverversion.h"

#include "pcieportal.h"
Expand Down Expand Up @@ -356,7 +359,11 @@ static int portal_mmap(struct file *filp, struct vm_area_struct *vma)
//vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
off = this_portal->extra->dma_handle;
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(6,3,0)
vma->vm_flags |= VM_IO;
#else
vm_flags_set(vma, VM_IO);
#endif
if (io_remap_pfn_range(vma, vma->vm_start, off >> PAGE_SHIFT,
vma->vm_end - vma->vm_start, vma->vm_page_prot))
return -EAGAIN;
Expand Down Expand Up @@ -424,7 +431,11 @@ static int portal_dma_pcis_mmap(struct file *filp, struct vm_area_struct *vma)
vma->vm_pgoff = off >> PAGE_SHIFT;
//vma->vm_flags |= VM_IO | VM_RESERVED;

#if LINUX_VERSION_CODE < KERNEL_VERSION(6,3,0)
vma->vm_flags |= VM_IO;
#else
vm_flags_set(vma, VM_IO);
#endif
if (io_remap_pfn_range(vma, vma->vm_start, off >> PAGE_SHIFT,
vma->vm_end - vma->vm_start, vma->vm_page_prot))
return -EAGAIN;
Expand Down Expand Up @@ -590,8 +601,13 @@ printk("[%s:%d]\n", __FUNCTION__, __LINE__);
printk("config block ID %x\n", ioread32(this_board->bar2io + 0x3000));
}
/* set DMA mask */
#if LINUX_VERSION_CODE < KERNEL_VERSION(5,18,0)
if (pci_set_dma_mask(dev, DMA_BIT_MASK(48))) {
printk(KERN_ERR "%s: pci_set_dma_mask failed for 48-bit DMA\n", DEV_NAME);
printk(KERN_ERR "%s: pci_dma_set_mask failed for 48-bit DMA\n", DEV_NAME);
#else
if (dma_set_mask(&dev->dev, DMA_BIT_MASK(48))) {
printk(KERN_ERR "%s: dma_set_mask failed for 48-bit DMA\n", DEV_NAME);
#endif
err = -EIO;
goto BARS_MAPPED_label;
}
Expand Down Expand Up @@ -839,7 +855,7 @@ static

MODULE_DEVICE_TABLE(pci, pcieportal_id_table);

static pci_ers_result_t pcieportal_error_detected(struct pci_dev *pdev, enum pci_channel_state error)
static pci_ers_result_t pcieportal_error_detected(struct pci_dev *pdev, pci_channel_state_t error)
{
printk(KERN_ERR "%s:%s: pcie error %d\n", DEV_NAME, __FUNCTION__, error);
return PCI_ERS_RESULT_CAN_RECOVER;
Expand Down Expand Up @@ -904,7 +920,12 @@ static int pcieportal_init(void)
int status;

printk("[%s:%d]\n", __FUNCTION__, __LINE__);
pcieportal_class = class_create(THIS_MODULE, "Connectal");
#if LINUX_VERSION_CODE < KERNEL_VERSION(6,4,0)
pcieportal_class = class_create(THIS_MODULE, "Connectal");
#else
pcieportal_class = class_create("Connectal");
#endif

if (IS_ERR(pcieportal_class)) {
printk(KERN_ERR "%s: failed to create class Connectal\n", DEV_NAME);
return PTR_ERR(pcieportal_class);
Expand Down
51 changes: 45 additions & 6 deletions drivers/portalmem/portalmem.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@
#define DRIVER_NAME "portalmem"
#define DRIVER_DESCRIPTION "Memory management between HW and SW processes"

MODULE_IMPORT_NS(DMA_BUF);

static struct miscdevice miscdev;

static void free_buffer_page(struct page *page, unsigned int order)
Expand Down Expand Up @@ -320,21 +322,58 @@ pa_dma_buf_end_cpu_access(struct dma_buf *dmabuf,
#endif
}

#ifdef DMAARGS
#error "DMAARGS is expected not to be defined. Modify this code to use other names."
#endif
#ifdef DMABUFMAP_T
#error "DMABUFMAP_T is expected not to be defined. Modify this code to use other names."
#endif

#if (LINUX_VERSION_CODE < KERNEL_VERSION(4,6,0))
#define DMAARGS 0, 0, 0
#else
#define DMAARGS 0
#endif

#if (LINUX_VERSION_CODE < KERNEL_VERSION(5,11,0))
#elif (LINUX_VERSION_CODE < KERNEL_VERSION(5,18,0))
#define DMABUFMAP_T struct dma_buf_map
#else
#define DMABUFMAP_T struct iosys_map
#endif

#if (LINUX_VERSION_CODE < KERNEL_VERSION(5,11,0))
static void *pa_dma_buf_vmap(struct dma_buf *dmabuf)
{
struct pa_buffer *buffer = dmabuf->priv;
pa_dma_buf_begin_cpu_access(dmabuf,
#if (LINUX_VERSION_CODE < KERNEL_VERSION(4,6,0))
0, 0,
#endif
0);
pa_dma_buf_begin_cpu_access(dmabuf, DMAARGS);
return buffer->vaddr;
}
#else
static int pa_dma_buf_vmap(struct dma_buf *dmabuf, DMABUFMAP_T *map)
{
struct pa_buffer *buffer = dmabuf->priv;
pa_dma_buf_begin_cpu_access(dmabuf, DMAARGS);
map->is_iomem = 0;
map->vaddr = buffer->vaddr;
return 0;
}
#endif

#if (LINUX_VERSION_CODE < KERNEL_VERSION(5,11,0))
static void pa_dma_buf_vunmap(struct dma_buf *dmabuf, void *vaddr)
{
printk("%s: dmabuf %p vaddr %p\n", __FUNCTION__, dmabuf, vaddr);
}
#else
static void pa_dma_buf_vunmap(struct dma_buf *dmabuf, DMABUFMAP_T *map)
{
printk("%s: dmabuf %p vaddr %p\n", __FUNCTION__, dmabuf, map->vaddr);
}
#endif

#undef DMABUFMAP_T
#undef DMAARGS


static struct dma_buf_ops dma_buf_ops = {
Expand All @@ -354,7 +393,7 @@ static struct dma_buf_ops dma_buf_ops = {
.map_atomic = pa_dma_buf_kmap,
.unmap_atomic = pa_dma_buf_kunmap,
#endif
#if !(defined(RHEL_MAJOR) && RHEL_MAJOR >= 8)
#if (LINUX_VERSION_CODE < KERNEL_VERSION(4,19,0)) && !(defined(RHEL_MAJOR) && RHEL_MAJOR >= 8)
.map = pa_dma_buf_kmap,
.unmap = pa_dma_buf_kunmap,
#endif
Expand Down
6 changes: 3 additions & 3 deletions pcie/pcieflat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
from __future__ import print_function

import fcntl, glob, json, struct, subprocess, sys
from gmpy import mpz
from gmpy2 import mpz
BNOC_TRACE = 0xc008b508
Trace_len = 144
struct_traceData = '@ L I I 16I 16I'
Expand Down Expand Up @@ -415,7 +415,7 @@ if __name__ == '__main__':
base = int(tlpfirst[0:16],16)
base = int(tlpfirst[0:8],16)
else:
fd = open(devfilenames[0], 'rw')
fd = open(devfilenames[0], 'w')
try:
while 1:
buf = fcntl.ioctl(fd, PCIE_CHANGE_ENTRY, ' ' * ChangeEntry_len)
Expand All @@ -441,7 +441,7 @@ if __name__ == '__main__':
buf = fcntl.ioctl(fd, BNOC_GET_TLP, ' ' * Tlp_len)
foo = ''
for x in buf:
Copy link

@jameyhicks-cmt jameyhicks-cmt Mar 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow, I'm sorry this code is so ugly. Seems like it should be buf.hex() or reverse(buf).hex() or something like that but if your changes work I am OK with them.

Copy link
Author

@tchomphoochan tchomphoochan Mar 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto previous comment. I'm having trouble understanding how this code works so I'm just taking the working change for granted now.

Copy link
Author

@tchomphoochan tchomphoochan Mar 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

foo = "%02x" % ord( x ) + foo
foo = "%02x" % x + foo
tlplog.append(foo)
fcntl.ioctl(fd, BNOC_ENABLE_TRACE, 1)
#for foo in tlplog:
Expand Down
2 changes: 1 addition & 1 deletion scripts/Makefile.connectal.build
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,7 +315,7 @@ bin/cvcsim: $(DTOP)/bin/libconnectal-sim.so verilog bin/cvcsim

vlsim: $(DTOP)/obj_dir/vlsim

VERILATOR_ARGS?= -O3 -CFLAGS "-I$(CONNECTALDIR)/cpp -I$(DTOP)/jni -O $(PROF_FLAGS)" -LDFLAGS "-O $(PROF_FLAGS)" --profile-cfuncs --output-split 20000
VERILATOR_ARGS?= -O3 -CFLAGS "-I$(CONNECTALDIR)/cpp -I$(DTOP)/jni -O $(PROF_FLAGS)" -LDFLAGS "-O $(PROF_FLAGS)" --profile-cfuncs --output-split 20000 --no-timing
VERILATOR_ARGS += $(VERILATOR_PROJECT_ARGS)

$(DTOP)/obj_dir/vlsim.mk: $(DTOP)/bin/libconnectal-sim.so verilog
Expand Down
2 changes: 1 addition & 1 deletion scripts/aws/notify_via_sns.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/usr/bin/env python33
#!/usr/bin/env python3

# Amazon FPGA Hardware Development Kit
#
Expand Down