[Trac-dev] Installing Trac and Trac plugins in wheel format

[RjOllos]

I've been experimenting how to install Trac and plugins in wheel format
using pip, and encountering various nuances. It appears that the egg cache
is not needed when installing wheels, which would simplify the
configuration (1).

I've found no issues if a wheel is available on PyPI. However, it gets more
complex when installing from a tarball/zip or installing from a repository.
I haven't found a single pip command that will force install of a wheel
when installing from a tarball/zip or repository. The workaround I
described in (2) is to first create the wheels, then install the wheels.

It gets more complex if the installation requirements are more complex. For
example, I want to install Genshi 0.6 with Trac 1.2 to workaround #11184
(3). There are two complications here:
* There is no wheel for Genshi 0.6 on PyPI
* If steps other than those described below are followed, it's easy to end
up with Genshi 0.7.
* It's easy to end up with Genshi compiled without speedups (I still
haven't solved this for the case of using requirements.txt)

Create virtualenv:

~/tmp$virtualenv pve
New python executable in /Users/rjollos/tmp/pve/bin/python2.7
Also creating executable in /Users/rjollos/tmp/pve/bin/python
Installing setuptools, pip, wheel...pidone.
(pve) ~/tmp$pip install -U pip setuptools wheel
Collecting pip
Using cached pip-9.0.1-py2.py3-none-any.whl
Collecting setuptools
Using cached setuptools-34.2.0-py2.py3-none-any.whl
Collecting wheel
Using cached wheel-0.29.0-py2.py3-none-any.whl
Collecting packaging>=16.8 (from setuptools)
Using cached packaging-16.8-py2.py3-none-any.whl
Collecting six>=1.6.0 (from setuptools)
Using cached six-1.10.0-py2.py3-none-any.whl
Collecting appdirs>=1.4.0 (from setuptools)
Using cached appdirs-1.4.0-py2.py3-none-any.whl
Collecting pyparsing (from packaging>=16.8->setuptools)
Using cached pyparsing-2.1.10-py2.py3-none-any.whl
Installing collected packages: pip, six, pyparsing, packaging, appdirs,
setuptools, wheel
Found existing installation: pip 8.0.2
Uninstalling pip-8.0.2:
Successfully uninstalled pip-8.0.2
Found existing installation: setuptools 19.6.2
Uninstalling setuptools-19.6.2:
Successfully uninstalled setuptools-19.6.2
Found existing installation: wheel 0.26.0
Uninstalling wheel-0.26.0:
Successfully uninstalled wheel-0.26.0
Successfully installed appdirs-1.4.0 packaging-16.8 pip-9.0.1
pyparsing-2.1.10 setuptools-34.2.0 six-1.10.0 wheel-0.29.0
(pve) ~/tmp$python --version
Python 2.7.13

Create wheels from packages specified in requirements.txt:

(pve) ~/tmp$cat requirements.txt
genshi==0.6
trac
(pve) ~/tmp$pip wheel --no-cache-dir --wheel-dir wheels -r requirements.txt
Collecting genshi==0.6 (from -r requirements.txt (line 1))
Downloading Genshi-0.6.tar.gz (433kB)
100% |████████████████████████████████| 440kB 1.7MB/s
Collecting trac (from -r requirements.txt (line 2))
Downloading Trac-1.2-py2-none-any.whl (3.8MB)
100% |████████████████████████████████| 3.8MB 1.5MB/s
Saved ./wheels/Trac-1.2-py2-none-any.whl
Collecting setuptools>=0.6 (from trac->-r requirements.txt (line 2))
Downloading setuptools-34.2.0-py2.py3-none-any.whl (389kB)
100% |████████████████████████████████| 399kB 1.8MB/s
Saved ./wheels/setuptools-34.2.0-py2.py3-none-any.whl
Collecting six>=1.6.0 (from setuptools>=0.6->trac->-r requirements.txt (line
2))
Downloading six-1.10.0-py2.py3-none-any.whl
Saved ./wheels/six-1.10.0-py2.py3-none-any.whl
Collecting packaging>=16.8 (from setuptools>=0.6->trac->-r requirements.txt
(line 2))
Downloading packaging-16.8-py2.py3-none-any.whl
Saved ./wheels/packaging-16.8-py2.py3-none-any.whl
Collecting appdirs>=1.4.0 (from setuptools>=0.6->trac->-r requirements.txt (line
2))
Downloading appdirs-1.4.0-py2.py3-none-any.whl
Saved ./wheels/appdirs-1.4.0-py2.py3-none-any.whl
Collecting pyparsing (from packaging>=16.8->setuptools>=0.6->trac->-r
requirements.txt (line 2))
Downloading pyparsing-2.1.10-py2.py3-none-any.whl (56kB)
100% |████████████████████████████████| 61kB 2.3MB/s
Saved ./wheels/pyparsing-2.1.10-py2.py3-none-any.whl
Skipping trac, due to already being wheel.
Skipping setuptools, due to already being wheel.
Skipping six, due to already being wheel.
Skipping packaging, due to already being wheel.
Skipping appdirs, due to already being wheel.
Skipping pyparsing, due to already being wheel.
Building wheels for collected packages: genshi
Running setup.py bdist_wheel for genshi ... done
Stored in directory: /Users/rjollos/tmp/wheels
Successfully built genshi
(pve) ~/tmp$ls wheels/
Genshi-0.6-py2-none-any.whl pyparsing-2.1.10-py2.py3-none-any.whl
Trac-1.2-py2-none-any.whl setuptools-34.2.0-py2.py3-none-any.whl
appdirs-1.4.0-py2.py3-none-any.whl six-1.10.0-py2.py3-none-any.whl
packaging-16.8-py2.py3-none-any.whl

I tried to install with "wheel install", but tracd and trac-admin don't get
installed. Oddly, there is no output from "wheel install".

(pve) ~/tmp$pip freeze
appdirs==1.4.0
packaging==16.8
pyparsing==2.1.10
six==1.10.0
(pve) ~/tmp$wheel install --wheel-dir wheels -r requirements.txt
(pve) ~/tmp$pip freeze
appdirs==1.4.0
Genshi==0.6
packaging==16.8
pyparsing==2.1.10
six==1.10.0
Trac==1.2(pve)
(pve) ~/tmp$ls -d pve/lib/python2.7/site-packages/*.egg-info
ls: pve/lib/python2.7/site-packages/*.egg-info: No such file or directory
(pve) ~/tmp$ls -d pve/lib/python2.7/site-packages/*.dist-info
pve/lib/python2.7/site-packages/Genshi-0.6.dist-info
pve/lib/python2.7/site-packages/Trac-1.2.dist-info
pve/lib/python2.7/site-packages/appdirs-1.4.0.dist-info
pve/lib/python2.7/site-packages/packaging-16.8.dist-info
pve/lib/python2.7/site-packages/pip-9.0.1.dist-info
pve/lib/python2.7/site-packages/pyparsing-2.1.10.dist-info
pve/lib/python2.7/site-packages/setuptools-34.2.0.dist-info
pve/lib/python2.7/site-packages/six-1.10.0.dist-info
pve/lib/python2.7/site-packages/wheel-0.30.0a0.dist-info
(pve) ~/tmp$trac-admin trac initenv
-bash: trac-admin: command not found
(pve) ~/tmp$ls pve/bin/
activate easy_install-2.7 python-config
activate.csh pip python2
activate.fish pip2 python2.7
activate_this.py pip2.7 wheel
easy_install python

"pip install" does work, by pointing to the wheels dir:

(pve) ~/tmp$pip install wheels/*.whl
Processing ./wheels/Genshi-0.6-py2-none-any.whl
Processing ./wheels/Trac-1.2-py2-none-any.whl
Requirement already satisfied: appdirs==1.4.0 from file:///Users/rjollos/tmp/wheels/appdirs-1.4.0-py2.py3-none-any.whl
in ./pve/lib/python2.7/site-packages
Requirement already satisfied: packaging==16.8 from file:///Users/rjollos/tmp/wheels/packaging-16.8-py2.py3-none-any.whl
in ./pve/lib/python2.7/site-packages
Requirement already satisfied: pyparsing==2.1.10 from file:///Users/rjollos/tmp/wheels/pyparsing-2.1.10-py2.py3-none-any.whl
in ./pve/lib/python2.7/site-packages
Requirement already satisfied: setuptools==34.2.0 from file:///Users/rjollos/tmp/wheels/setuptools-34.2.0-py2.py3-none-any.whl
in ./pve/lib/python2.7/site-packages
Requirement already satisfied: six==1.10.0 from file:///Users/rjollos/tmp/wheels/six-1.10.0-py2.py3-none-any.whl
in ./pve/lib/python2.7/site-packages
Installing collected packages: Genshi, Trac
Successfully installed Genshi-0.6 Trac-1.2


The key points seem to be:
* Use a requirements.txt if you need to pin installation versions. It's
easy to end up with Genshi 0.7 by running "pip wheel .. genshi==0.6"
followed by "pip wheel ... trac". The latter command will also download
Genshi 0.7 and build a wheel, unless "--no-deps" is used.
* Use "pip wheel" followed by "pip install" to avoid getting eggs when the
wheel isn't available on PyPI, or when installing from a repository.

With these steps I was able to get all the packages installed to
trac-hacks.org in the wheel format, and the somewhat complex
requirements.txt:

pytz
babel
docutils
psycopg2
pygments
httplib2
oauth2
pillow
mollom
spambayes
dnspython
genshi==0.6
trac
svn+https://trac-hacks.org/svn/changelogmacro/trunk/
svn+https://trac-hacks.org/svn/footnotemacro/trunk/
svn+https://trac-hacks.org/svn/svnauthzadminplugin/1.0/
svn+https://trac-hacks.org/svn/accountmanagerplugin/trunk/
svn+https://trac-hacks.org/svn/acronymsplugin/0.11/
svn+https://trac-hacks.org/svn/autowikifyplugin/trunk/
svn+https://trac-hacks.org/svn/extracturlplugin/0.11/
svn+https://trac-hacks.org/svn/fullblogplugin/0.11/
svn+https://trac-hacks.org/svn/includemacro/trunk/
svn+https://trac-hacks.org/svn/traciniadminpanelplugin/trunk/
svn+https://trac-hacks.org/svn/newsflashmacro/0.11/
svn+https://trac-hacks.org/svn/pollmacro/trunk/
svn+https://trac-hacks.org/svn/serversideredirectplugin/1.0/
svn+https://svn.edgewall.org/repos/trac/plugins/1.2/spam-filter/
svn+https://trac-hacks.org/svn/tagsplugin/trunk/
svn+https://trac-hacks.org/svn/tocmacro/0.11/
svn+https://trac-hacks.org/svn/voteplugin/trunk/
svn+https://trac-hacks.org/svn/wikiextrasplugin/trunk/
svn+https://trac-hacks.org/svn/xmlrpcplugin/trunk/
svn+https://trac-hacks.org/svn/wikiautocompleteplugin/trunk/
svn+https://trac-hacks.org/svn/trachacksplugin/trunk/

Please let me know if you have any other tips, or advice on accomplishing
(2).

I'm thinking it would be good to create a TracInstallVirtualEnv recipe that
describes how to install to a virtual environment using pip and wheels.

- Ryan

(1) https://trac.edgewall.org/wiki/TracModWSGI#Averybasicscript
(2) http://stackoverflow.com/questions/42269290/single-pip-command-to-install-a-tarball-or-zip-as-a-wheel
(3) https://trac.edgewall.org/wiki/TracFaq#Q:WhydoIseeIndexError:popfromemptylistwhentryingtoaddanattachment
http://blog.dscpl.com.au/2015/04/introducing-modwsgi-express.html

[Thomas Moschny]

Using your requirements.txt, things worked well here:

# virtualenv t
# source t/bin/activate
# pip install -U pip setuptools wheel
# pip wheel -w wheels -r requirements.txt
# pip wheel -w wheels --global-option=--with-speedups genshi==0.6
# rm -vf wheels/Genshi-0.6-py2-none-any.whl
# pip install wheels/*.whl

In theory, one should be able to add global options to the
requirements.txt file:

...
genshi==0.6 --global-option=--with-speedups
...

but in my tests that didn't work - it is honored by "pip install", but
ignored by "pip wheel". Also, it has unwanted global effects, see
https://github.com/pypa/pip/issues/4118.


The more general question is, whether it is really an issue if
packages are installed from a wheel or as egg. Using this recipe (and
your unmodified requirements.txt) to directly installing packages

# virtualenv t
# source t/bin/activate
# pip install -U pip setuptools wheel
# pip install --global-option=--with-speedups genshi==0.6
# pip install -r requirements.txt

will not put any zipped eggs in t/lib/python2.7/site-packages, so I
guess in that case also no egg cache will be used?

- Thomas

[RjOllos]

Post by Thomas Moschny
# virtualenv t
# source t/bin/activate
# pip install -U pip setuptools wheel
# pip wheel -w wheels -r requirements.txt
# pip wheel -w wheels --global-option=--with-speedups genshi==0.6
# rm -vf wheels/Genshi-0.6-py2-none-any.whl
# pip install wheels/*.whl
In theory, one should be able to add global options to the
...
genshi==0.6 --global-option=--with-speedups
...
but in my tests that didn't work - it is honored by "pip install", but
ignored by "pip wheel". Also, it has unwanted global effects, see
https://github.com/pypa/pip/issues/4118.

Thanks, I was seeing the same behavior, but was unsure why. If we could
make a whl available on PyPI that would avoid the need for using the
--global-option flag.
https://genshi.edgewall.org/ticket/606#comment:2

Post by Thomas Moschny
The more general question is, whether it is really an issue if
packages are installed from a wheel or as egg. Using this recipe (and
your unmodified requirements.txt) to directly installing packages
# virtualenv t
# source t/bin/activate
# pip install -U pip setuptools wheel
# pip install --global-option=--with-speedups genshi==0.6
# pip install -r requirements.txt
will not put any zipped eggs in t/lib/python2.7/site-packages, so I
guess in that case also no egg cache will be used?

I'm also assuming no egg cache is necessary.
https://trac.edgewall.org/ticket/12288#comment:10

- Ryan

[RjOllos]

Post by RjOllos
I tried to install with "wheel install", but tracd and trac-admin don't
get installed. Oddly, there is no output from "wheel install".

Console scripts are installed with "wheel install-scripts" (1), or using
"pip install" rather "wheel install".

(pve) ~/tmp$pip wheel --no-cache-dir --wheel-dir wheels -r requirements.txt
(pve) ~/tmp$wheel install --wheel-dir wheels -r requirements.txt
(pve) ~/tmp$ls pve/bin/trac*
ls: pve/bin/trac*: No such file or directory
(pve) ~/tmp$wheel install-scripts trac
(pve) ~/tmp$ls pve/bin/trac*
pve/bin/trac-admin pve/bin/tracd

Anyway, I just wanted to understand that the behavior is by design. I
suspect we'd always use "pip install" to install wheels.

- Ryan

(1) http://wheel.readthedocs.io/en/latest/#setuptools-scripts-handling