hip pip hooray

Published on July 14, 2017

I've been working with Python at work for a while (circa 8 years). Work is at a Local Authrority and is a MSFT Windows environment, sat behind proxy servers and a firewall. Home is a Linux environment and certainly working with the standard Python package manager (pip) and Python virtual environments in a pythonic way is easier - it's how this blog was developed.

At work I had pretty much accepted that pip was unavailable and that if we needed additional packages (think lxml, pypyodbc, pdb) then I would need to find someone who'd ported to Windows and could provide a Windows installer or download and install source code from github manually. I could then maintain these on a file-share and reference them in any code that needed them. However, a few nudges here and there and some helpful network people permitted access to pypi.org (and therefore pip) at work immediately making the install procedure a lot easier.

This was a good day:

A milestone for me as I now have the entire Python repository avaiable to use and other options for intergrate their use and maintenance with other Python based platforms and environments at work.

There is still work to do in terms of productionising the use of pip, including offereing an alternative Python installation which currently piggy backs off an ESRI ArcMap Desktop install. Despite the use of Python by ArcMap the application, given the nature of most corproate software development cycles, the Python packages supplied with the software are out of date when compared against the current versions in the Python repositories (see above).

In terms of PSN compliance and the general feeling about Python at work, being a Microsoft based Local Authority where a support contract provides a tangible confirmation of security that typcially satisfies the management tier. In an environment like this Open Source software is not always seen as a serious viable alternative, regardleess of the fact that many large government sites use Open Source and Python extensively (including data.gov.uk which is build on CKAN - referenced here) .

The inclusion of pip with the Python installation used by ArcMap at work makes it an option to upgrade and add additional Python packages should a user wish to do so however this carries risks in that it alters the Python environment produced by the software installation. If certain packages require specifiic versions an upgrade may impact on any tools using them. In particular the use of Numpy and Pandas as well as xlrd and xlrw. Our organisation also include a managed roll out of the ArcMap Desktop software so making underlying changes to dependencies is not encouraged for support reasons.

One approach I've taken is to install the Python Virtualenv module which would permit the creation of standalone Python environments where modules are installed separately of the ArcMap Python installation, meaning I can upgrade and install and Python packages to my hearts content without running the risk of it affecting the Python environment used by ArcMap. Indeed the use of Virtualenv is well documented online as an approach for best practise when it comes to Python development generally, - just Google "Python, Development, Virtualenv, Best Pracise"

Installing and using vitualenv is covered in the next blog post but does required a fully functioning pip setup which the rest of this post describes.

The main problem is to ensure that the environment being used to access the Python package resources are avaiable through any firewall / proxy configurations that the organisation may have in place. You may need to check this with your network administrators if unsure. Once access is permitted the --proxy parameter in pip can be passed to a pip command that needs online access such as checking for outdated commands. If this suceeds - congratulations pip is working. An example is in the image above.

Here is an example of the proxy setting not being used and the subsequent error message returned:

Pip does not seem to pass the --proxy parameter automatically in the cases where a package install has dependecies which must also be included. The permenant fix for including the proxy setting during every pip call is to include a couple of Environment Variables. I mention previously we're a Windows shop and while I am fortunate enough to have enough rights on my laptop to do this I am aware that not everyone has this privilege.

Fortunately there are two ways to do this - one being permanent, the other being just for the session only only. The former is done using the Environment Variables GUI the other is setting them using the command prompt: Two are needed:

The can be set in the command prompt in the following way:

Having done this pip should be accessible without needed to pass the --proxy parameter:

To get started with pip it's worth spending some time on the pypi site and checking out the help files. It's also worth noting that this is the central Python repository where all the Python modules are maintained. One of the great things about using Virtualbox is that, used correctly it takes the anxiety of experimenting and learning to use Python modules since these are in a self-contained environment that will not affect the rest of the PC.

The next post is this blog describes making use of Python virtual environments to leverage additional Python functionality while ensuring it remains neatly separated from the rest of the system, minimising the impact of making changes to a users' system.


Additional comments have been disabled for this entry

An entry posted on July 14, 2017.

Category: Python

Tags: pip , python , windows and work