_generated/ffi.py missing from binary install

[raffienficiaud]

Hi,

I just noticed this:

$ echo $VIRTUAL_ENV
/tmp/toto
$ pip install -U pangocairocffi
Processing /root/.cache/pip/wheels/5e/b4/90/3317813d0940443f4e9302caf89a8b7fe79eaf174823abadea/pangocairocffi-0.3.0-py3-none-any.whl
Requirement already satisfied, skipping upgrade: cairocffi>=1.0.2 in /tmp/toto/lib/python3.8/site-packages (from pangocairocffi) (1.1.0)
Requirement already satisfied, skipping upgrade: cffi>=1.1.0 in /tmp/toto/lib/python3.8/site-packages (from pangocairocffi) (1.14.3)
Requirement already satisfied, skipping upgrade: pangocffi>=0.4.0 in /tmp/toto/lib/python3.8/site-packages (from pangocairocffi) (0.6.0)
Requirement already satisfied, skipping upgrade: setuptools>=39.2.0 in /tmp/toto/lib/python3.8/site-packages (from cairocffi>=1.0.2->pangocairocffi) (50.3.0)
Requirement already satisfied, skipping upgrade: pycparser in /tmp/toto/lib/python3.8/site-packages (from cffi>=1.1.0->pangocairocffi) (2.20)
Installing collected packages: pangocairocffi
Successfully installed pangocairocffi-0.3.0

but then

$ ls -al /tmp/toto/lib/python3.8/site-packages/pangocairocffi/_generated/
drwxr-xr-x 3 root root 4096 Sep 23 10:13 .
drwxr-xr-x 4 root root 4096 Sep 23 10:13 ..
drwxr-xr-x 2 root root 4096 Sep 23 10:13 __pycache__
-rw-r--r-- 1 root root   47 Sep 23 10:13 ffi_pango.py

While if I do

$ pip install --no-binary :all: -U pangocairocffi
Collecting pangocairocffi
  Using cached pangocairocffi-0.3.0.tar.gz (20 kB)
Requirement already satisfied, skipping upgrade: cffi>=1.1.0 in /tmp/toto/lib/python3.8/site-packages (from pangocairocffi) (1.14.3)
Requirement already satisfied, skipping upgrade: cairocffi>=1.0.2 in /tmp/toto/lib/python3.8/site-packages (from pangocairocffi) (1.1.0)
Requirement already satisfied, skipping upgrade: pangocffi>=0.4.0 in /tmp/toto/lib/python3.8/site-packages (from pangocairocffi) (0.6.0)
Requirement already satisfied, skipping upgrade: pycparser in /tmp/toto/lib/python3.8/site-packages (from cffi>=1.1.0->pangocairocffi) (2.20)
Requirement already satisfied, skipping upgrade: setuptools>=39.2.0 in /tmp/toto/lib/python3.8/site-packages (from cairocffi>=1.0.2->pangocairocffi) (50.3.0)
Skipping wheel build for pangocairocffi, due to binaries being disabled for it.
Installing collected packages: pangocairocffi
    Running setup.py install for pangocairocffi ... done
Successfully installed pangocairocffi-0.3.0

$ ls -al /tmp/toto/lib/python3.8/site-packages/pangocairocffi/_generated/
drwxr-xr-x 3 root root  4096 Sep 23 10:16 .
drwxr-xr-x 4 root root  4096 Sep 23 10:16 ..
drwxr-xr-x 2 root root  4096 Sep 23 10:16 __pycache__
-rw-r--r-- 1 root root 29120 Sep 23 10:16 ffi.py
-rw-r--r-- 1 root root    47 Mar 23  2019 ffi_pango.py

So the file ffi.py is properly set up, while it does not seem to be part of the binary archive. Taking a deeper look, when wheel is installed and upon first installation, the generated whl file in the cache does not contain the ffi.py file neither. Any subsequent installation in a virtual environment will have the same issue.

[leifgehrmann]

Can you go into more detail on your setup?

I am not surprised it is failing to install the wheel, mainly because there shouldn't be a wheel for pangocairocffi, nor for pangocffi and cairocffi.

Both pangocffi and pangocairocffi require setup.py install to run when being installed by pip because ffi.py is tightly linked to the platform it is being installed on (i.e. Windows, MacOS, Linux). Because of this, wheels cannot be created for these packages (As far as I know). Perhaps there is a way around this, but I believe this is what cairocffi does as well.

I'm actually quite curious where pangocairocffi-0.3.0-py3-none-any.whl that appears in your cache comes from. Unless I'm mistaken, that artifact doesn't come from pypi. Maybe clearing your cache would make pip default back to installing from source?

I've booted up a fresh Ubuntu instance in VirtualBox to try and replicate your commands for myself.

user@user-VirtualBox:~/Documents/pangocairocffi-test$ python3 -m venv venv
user@user-VirtualBox:~/Documents/pangocairocffi-test$ source venv/bin/activate
(venv) user@user-VirtualBox:~/Documents/pangocairocffi-test$ pip install -U pip
Collecting pip
  Downloading pip-20.2.3-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 1.7 MB/s
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Uninstalling pip-20.0.2:
      Successfully uninstalled pip-20.0.2
Successfully installed pip-20.2.3
(venv) user@user-VirtualBox:~/Documents/pangocairocffi-test$ pip install -U pangocairocffi
Collecting pangocairocffi
  Downloading pangocairocffi-0.3.0.tar.gz (20 kB)
Collecting cairocffi>=1.0.2
  Downloading cairocffi-1.1.0.tar.gz (68 kB)
     |████████████████████████████████| 68 kB 2.6 MB/s
Collecting cffi>=1.1.0
  Using cached cffi-1.14.3-cp38-cp38-manylinux1_x86_64.whl (410 kB)
Collecting pangocffi>=0.4.0
  Downloading pangocffi-0.6.0.tar.gz (37 kB)
Requirement already satisfied, skipping upgrade: setuptools>=39.2.0 in ./venv/lib/python3.8/site-packages (from cairocffi>=1.0.2->pangocairocffi) (44.0.0)
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Using legacy 'setup.py install' for pangocairocffi, since package 'wheel' is not installed.
Using legacy 'setup.py install' for cairocffi, since package 'wheel' is not installed.
Using legacy 'setup.py install' for pangocffi, since package 'wheel' is not installed.
Installing collected packages: pycparser, cffi, cairocffi, pangocffi, pangocairocffi
    Running setup.py install for cairocffi ... done
    Running setup.py install for pangocffi ... done
    Running setup.py install for pangocairocffi ... done
Successfully installed cairocffi-1.1.0 cffi-1.14.3 pangocairocffi-0.3.0 pangocffi-0.6.0 pycparser-2.20
(venv) user@user-VirtualBox:~/Documents/pangocairocffi-test$ ls -la venv/lib/python3.8/site-packages/pangocairocffi/_generated/
total 48
drwxrwxr-x 3 user user  4096 Sep 25 22:06 .
drwxrwxr-x 4 user user  4096 Sep 25 22:06 ..
-rw-rw-r-- 1 user user    47 Mar 23  2019 ffi_pango.py
-rw-rw-r-- 1 user user 29120 Sep 25 22:06 ffi.py
drwxrwxr-x 2 user user  4096 Sep 25 22:06 __pycache__
(venv) user@user-VirtualBox:~/Documents/pangocairocffi-test$

(This is running python 3.8 and pip 20.2.3 on Ubuntu 20.04.1 LTS)

This is the installation process I would except, where it downloads pangocairocffi, cairocffi, and pangocffi as tar.gz, and any other dependencies as a wheel. Then it runs setup.py for those non-wheel dependencies.

[raffienficiaud]

Hi,

Yes, so the whl is directly created by pip itself: from a setup.py this is created in order I believe to accelerate subsequent installations. If wheel is installed, I believe you will have the issue as well: as I understand it, pip will first generate the whl and cache it, then it will install from the whl (and not from the folder where setup.py is). If the whl is not fully installable, the issue pops up.

I emitted all the commands in a naked docker container.

[leifgehrmann]

Thanks for clarifying! I somehow missed in your original comment that you installed wheel. I can now replicate the issue.

This also explains why a lot of things are failing in CI pipelines and readthedocs.org (see #9)! I would've never figured it out so thank you!

I'm unsure what the correct procedure is. Ultimately, I'm fairly certain I need to find a way to stop pip from running bdist_wheel, but pip's documentation doesn't hint of a way to configure this at the package level.

I've found this stack overflow thread: https://stackoverflow.com/questions/58289062/prevent-pip-from-caching-a-package which suggests forcing failures when pip attempts to create a bdist_wheel. Seems evil, but I guess it might work.

When I have time this week I'll play around with that approach and see if it's any good.

In the short term, I hope the --no-binary :all: is a good enough work around!

[naveen521kk]

Both pangocffi and pangocairocffi require setup.py install to run when being installed by pip because ffi.py is tightly linked to the platform it is being installed on (i.e. Windows, MacOS, Linux). Because of this, wheels cannot be created for these packages (As far as I know). Perhaps there is a way around this, but I believe this is what cairocffi does as well.

Platform specific wheel can be created and is widely used for a variety of other projects.

Now, on closely looking into this, there is a _generated/ffi.py folder in the wheel generated also, I still don't understand why it is not extracted though. Stopping pip from caching wheel is impossible and is not recommended.

I don't know much about cffi, but I think if the generated ffi.py are platform specific those can be build as platform specific wheels and then published to PyPi.

[raffienficiaud]

I may do not be aware of all the details, but as soon as you have a dependency to libraries outside the pip environment (in this case pango which may be installed in various manner, brew, apt etc) then making a precompiled wheel is not ideal and likely to break. It is not only about the target operating system, but also how those libraries are installed.

[naveen521kk]

But I don't think this depends on the way pango is installed though because looking at the ffi generator code I could say that.

What if those wheels include pango binaries? I made an issue on that. #14

[leifgehrmann]

Hi @raffienficiaud, sorry for the long wait! I've published a new release for pangocffi (v0.8.0) and pangocairocffi (v0.4.0). Can you try installing again?

[leifgehrmann]

Closing this issue since I believe it has been resolved in pangocffi (v0.8.0) and pangocairocffi (v0.4.0).

Discussion about platform specific wheels can be continued in #14