Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Python interperter distclean isn't working #2895

Open
1 task done
PetervdPerk-NXP opened this issue Dec 17, 2024 · 7 comments
Open
1 task done

[BUG] Python interperter distclean isn't working #2895

PetervdPerk-NXP opened this issue Dec 17, 2024 · 7 comments
Assignees
@tmedicci
Labels
Type: Bug Something isn't working

Comments

@PetervdPerk-NXP
Copy link
Contributor

Description / Steps to reproduce the issue

Try to compile python interpreter using

$ cd nuttx
$ make distclean
$ ./tools/configure.sh rv-virt:python
$ make -j$(nproc)
$ ls -l nuttx

Run make distclean or make apps_distclean`
Python compile objects are still there.

Dry run doesn't show removal of the files

make apps_distclean -n
make -C /home/peter/nuttx/apps/interpreters/ distclean APPDIR="/home/peter/nuttx/apps"
rm -f  Kconfig
rm -f  .kconfig

On which OS does this issue occur?

[OS: Linux]

What is the version of your OS?

Ubuntu 22.04

NuttX Version

master

Issue Architecture

[Arch: risc-v]

Issue Area

[Area: Interpreters]

Verification

  • I have verified before submitting the report.
@PetervdPerk-NXP PetervdPerk-NXP added the Type: Bug Something isn't working label Dec 17, 2024
@tmedicci
Copy link
Contributor

Hi @PetervdPerk-NXP , were you able to build it properly in the last run?

I remember seeing something similar when I was developing and the sources were not being properly tracked. Can you try cleaning the nuttx and the nuttx-apps repository with git clean -ffdx, trying to build it and then using make distclean, please?

@tmedicci
Copy link
Contributor

tmedicci commented Dec 18, 2024 •
edited
Loading

I tried and it worked as expected on our docker image:

docker run -it ghcr.io/apache/nuttx/apache-nuttx-ci-linux:latest /bin/bash -c "                                                               
  cd /tmp ; ls -la
  pwd ;
  git clone https://github.com/apache/incubator-nuttx nuttx;
  git clone https://github.com/apache/incubator-nuttx-apps apps ;
  pushd nuttx ;
  git checkout master;         
  echo NuttX Source: https://github.com/apache/nuttx/tree/\$(git rev-parse HEAD) ; popd ;
  pushd apps  ;
  git checkout master;
  echo NuttX Apps: https://github.com/apache/nuttx-apps/tree/\$(git rev-parse HEAD) ; popd ;
  cd nuttx;
  make distclean; ./tools/configure.sh rv-virt:python; make -j$(nproc);
  echo \"Python files before make distclean\"
  ls -la ../apps/interpreters/python;
  echo \"Python files after make distclean\"
  make distclean;
  ls -la ../apps/interpreters/python;
"

The build logs show the folder ../apps/interpreters/python after building the firmware and, then, after a make distclean:

Python files before make distclean
total 99564
drwxr-xr-x  6 root root     4096 Dec 18 13:06 .
drwxr-xr-x 14 root root     4096 Dec 18 13:03 ..
-rw-r--r--  1 root root        0 Dec 18 13:06 .built
-rw-r--r--  1 root root        0 Dec 18 13:05 .depend
-rw-r--r--  1 root root       86 Dec 18 13:03 .gitignore
-rw-r--r--  1 root root     1562 Dec 18 13:03 Kconfig
-rw-r--r--  1 root root     1322 Dec 18 13:03 Make.defs
-rw-r--r--  1 root root    12038 Dec 18 13:05 Make.dep
-rw-r--r--  1 root root     7259 Dec 18 13:03 Makefile
drwxr-xr-x 22 root root     4096 Dec 18 13:05 Python
-rw-r--r--  1 root root      601 Dec 18 13:03 Setup.local
drwxr-xr-x  4 root root     4096 Dec 18 13:05 build
-rw-r--r--  1 root root      723 Dec 18 13:03 config.site
drwxr-xr-x  4 root root     4096 Dec 18 13:05 install
-rw-r--r--  1 root root     4471 Dec 18 13:03 mount_modules.c
-rw-r--r--  1 root root  7633300 Dec 18 13:06 mount_modules.c.tmp.apps.interpreters.python.o
-rw-r--r--  1 root root  7633300 Dec 18 13:06 mount_modules.c.tmp.apps.interpreters.python_1.o
drwxr-xr-x  2 root root     4096 Dec 18 13:03 patch
-rw-r--r--  1 root root     5972 Dec 18 13:06 python.c.tmp.apps.interpreters.python.o
-rw-r--r--  1 root root     5972 Dec 18 13:06 python.c.tmp.apps.interpreters.python_1.o
-rw-r--r--  1 root root 47000166 Dec 18 13:06 romfs_cpython_modules.h
-rw-r--r--  1 root root  7621632 Dec 18 13:06 romfs_cpython_modules.img
-rw-r--r--  1 root root 31968446 Dec 18 13:04 v3.13.0.zip
Python files after make distclean
total 48
drwxr-xr-x  3 root root 4096 Dec 18 13:06 .
drwxr-xr-x 14 root root 4096 Dec 18 13:06 ..
-rw-r--r--  1 root root   86 Dec 18 13:03 .gitignore
-rw-r--r--  1 root root 1562 Dec 18 13:03 Kconfig
-rw-r--r--  1 root root 1322 Dec 18 13:03 Make.defs
-rw-r--r--  1 root root 7259 Dec 18 13:03 Makefile
-rw-r--r--  1 root root  601 Dec 18 13:03 Setup.local
-rw-r--r--  1 root root  723 Dec 18 13:03 config.site
-rw-r--r--  1 root root 4471 Dec 18 13:03 mount_modules.c
drwxr-xr-x  2 root root 4096 Dec 18 13:03 patch

Can you provide more details about your environment?

@PetervdPerk-NXP
Copy link
Contributor Author 

peter@SMW015329:~/nuttx/apps/interpreters/python$ git clean -ffdx
Removing Python/
Removing build/
Removing install/
Removing v3.13.0.zip
peter@SMW015329:~/nuttx/apps/interpreters/python$ ls
Kconfig  Make.defs  Makefile  Setup.local  config.site  mount_modules.c  patch

$ ./tools/configure.sh rv-virt:python
$ make -j$(nproc)

# Lots of output

checking size of fpos_t... 0
checking size of size_t... configure: error: in `/home/peter/nuttx/apps/interpreters/python/build/target':
configure: error: cannot compute sizeof (size_t)
See `config.log' for more details
make[2]: *** [Makefile:107: /home/peter/nuttx/apps/interpreters/python/build/target/Makefile] Error 77
make[1]: *** [Makefile:57: /home/peter/nuttx/apps/interpreters/python_depend] Error 2
make: *** [tools/Unix.mk:630: pass2dep] Error 2
peter@SMW015329:~/nuttx/nuttx$
peter@SMW015329:~/nuttx/nuttx$ make distclean

Compile failure I'll try make distclean

peter@SMW015329:~/nuttx/apps/interpreters/python$ ls
Kconfig  Make.defs  Makefile  Python  Setup.local  build  config.site  install  mount_modules.c  patch  v3.13.0.zip

So yeah most likely make distclean would work on a working build, but a build fail then you can't clean through make distclean

Also which compiler are you using? Because I think my failure has to do with my gcc version.

@tmedicci
Copy link
Contributor 

peter@SMW015329:~/nuttx/apps/interpreters/python$ git clean -ffdx
Removing Python/
Removing build/
Removing install/
Removing v3.13.0.zip
peter@SMW015329:~/nuttx/apps/interpreters/python$ ls
Kconfig  Make.defs  Makefile  Setup.local  config.site  mount_modules.c  patch

$ ./tools/configure.sh rv-virt:python
$ make -j$(nproc)

# Lots of output

checking size of fpos_t... 0
checking size of size_t... configure: error: in `/home/peter/nuttx/apps/interpreters/python/build/target':
configure: error: cannot compute sizeof (size_t)
See `config.log' for more details
make[2]: *** [Makefile:107: /home/peter/nuttx/apps/interpreters/python/build/target/Makefile] Error 77
make[1]: *** [Makefile:57: /home/peter/nuttx/apps/interpreters/python_depend] Error 2
make: *** [tools/Unix.mk:630: pass2dep] Error 2
peter@SMW015329:~/nuttx/nuttx$
peter@SMW015329:~/nuttx/nuttx$ make distclean

Compile failure I'll try make distclean

peter@SMW015329:~/nuttx/apps/interpreters/python$ ls
Kconfig  Make.defs  Makefile  Python  Setup.local  build  config.site  install  mount_modules.c  patch  v3.13.0.zip

So yeah most likely make distclean would work on a working build, but a build fail then you can't clean through make distclean

Also which compiler are you using? Because I think my failure has to do with my gcc version.

I see... Yeah, would need to investigate why it fails to clean the files if the build fails too...

About the toolchain, we always recommend using the same CI uses: https://nuttx.apache.org/docs/latest/platforms/risc-v/qemu-rv/boards/rv-virt/index.html#risc-v-toolchain

@PetervdPerk-NXP
Copy link
Contributor Author

Any idea where the CI downloads the 64 bit of the riscv toolchain I only see this?


###############################################################################
# Build image for tool required by RISCV builds
###############################################################################
FROM nuttx-toolchain-base AS nuttx-toolchain-riscv
# Download the latest RISCV GCC toolchain prebuilt by xPack
RUN mkdir -p riscv-none-elf-gcc && \
  curl -s -L "https://github.com/xpack-dev-tools/riscv-none-elf-gcc-xpack/releases/download/v13.2.0-2/xpack-riscv-none-elf-gcc-13.2.0-2-linux-x64.tar.gz" \
  | tar -C riscv-none-elf-gcc --strip-components 1 -xz

But when building it wants the 64 bit version

peter@SMW015329:~/nuttx/nuttx$ ./tools/configure.sh rv-virt:python
  Copy files
  Select CONFIG_HOST_LINUX=y
  Refreshing...
CP: arch/dummy/Kconfig to /home/peter/nuttx/nuttx/arch/dummy/dummy_kconfig
CP: boards/dummy/Kconfig to /home/peter/nuttx/nuttx/boards/dummy/dummy_kconfig
LN: platform/board to /home/peter/nuttx/apps/platform/dummy
LN: include/arch to arch/risc-v/include
LN: include/arch/board to /home/peter/nuttx/nuttx/boards/risc-v/qemu-rv/rv-virt/include
LN: drivers/platform to /home/peter/nuttx/nuttx/drivers/dummy
LN: include/arch/chip to /home/peter/nuttx/nuttx/arch/risc-v/include/qemu-rv
LN: arch/risc-v/src/chip to /home/peter/nuttx/nuttx/arch/risc-v/src/qemu-rv
LN: arch/risc-v/src/board to /home/peter/nuttx/nuttx/boards/risc-v/qemu-rv/rv-virt/src
Loaded configuration '.config'
Configuration saved to '.config'
peter@SMW015329:~/nuttx/nuttx$ make
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found
Create version.h
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found

@tmedicci
Copy link
Contributor

Any idea where the CI downloads the 64 bit of the riscv toolchain I only see this?


###############################################################################
# Build image for tool required by RISCV builds
###############################################################################
FROM nuttx-toolchain-base AS nuttx-toolchain-riscv
# Download the latest RISCV GCC toolchain prebuilt by xPack
RUN mkdir -p riscv-none-elf-gcc && \
  curl -s -L "https://github.com/xpack-dev-tools/riscv-none-elf-gcc-xpack/releases/download/v13.2.0-2/xpack-riscv-none-elf-gcc-13.2.0-2-linux-x64.tar.gz" \
  | tar -C riscv-none-elf-gcc --strip-components 1 -xz

But when building it wants the 64 bit version

peter@SMW015329:~/nuttx/nuttx$ ./tools/configure.sh rv-virt:python
  Copy files
  Select CONFIG_HOST_LINUX=y
  Refreshing...
CP: arch/dummy/Kconfig to /home/peter/nuttx/nuttx/arch/dummy/dummy_kconfig
CP: boards/dummy/Kconfig to /home/peter/nuttx/nuttx/boards/dummy/dummy_kconfig
LN: platform/board to /home/peter/nuttx/apps/platform/dummy
LN: include/arch to arch/risc-v/include
LN: include/arch/board to /home/peter/nuttx/nuttx/boards/risc-v/qemu-rv/rv-virt/include
LN: drivers/platform to /home/peter/nuttx/nuttx/drivers/dummy
LN: include/arch/chip to /home/peter/nuttx/nuttx/arch/risc-v/include/qemu-rv
LN: arch/risc-v/src/chip to /home/peter/nuttx/nuttx/arch/risc-v/src/qemu-rv
LN: arch/risc-v/src/board to /home/peter/nuttx/nuttx/boards/risc-v/qemu-rv/rv-virt/src
Loaded configuration '.config'
Configuration saved to '.config'
peter@SMW015329:~/nuttx/nuttx$ make
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found
Create version.h
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found
/usr/bin/bash: line 1: riscv64-unknown-elf-gcc: command not found

You need to export it to your PATH env var, just like it's done on CI:
https://github.com/apache/nuttx/blob/c1b665c5d5ca5eac130300395597e59d3add0803/tools/ci/docker/linux/Dockerfile#L465

So, basically, it doesn't matter where you download it, as long as you export it to your PATH. I created an alias on my ~/.zshrc (~/.bashrc for Ubuntu, if I'm not mistaken) to automate inserting it to my PATH when needed:

alias get_nuttx='export PATH="/home/tiago/Downloads/xpack-riscv-none-elf-gcc-13.2.0-2/bin:$PATH"'

@PetervdPerk-NXP
Copy link
Contributor Author

Okay got it now turns out it fails on GCC 10.2, but GCC 13.2 works fine.
On a succesfull build distclean works.

Created /home/peter/nuttx/apps/interpreters/python/build/target/lib/python313.zip (3.11 MiB)
CPP:  /home/peter/nuttx/nuttx/boards/risc-v/qemu-rv/rv-virt/scripts/ld.script-> /home/peter/nuttx/nuttx/boards/risc-v/qeLD: nuttx
riscv-none-elf-ld: warning: /home/peter/nuttx/nuttx/nuttx has a LOAD segment with RWX permissions
Memory region         Used Size  Region Size  %age Used
peter@SMW015329:~/nuttx/nuttx$ make distclean
peter@SMW015329:~/nuttx/nuttx$ ls ../apps/interpreters/python/
Kconfig  Make.defs  Makefile  Setup.local  config.site  mount_modules.c  patch
peter@SMW015329:~/nuttx/nuttx$

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tmedicci tmedicci

Labels
Type: Bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tmedicci @PetervdPerk-NXP