What did you Expect - a look at Python module Pexpect

On Tuesday I presented on Pexpect at the Grenoble Python User Group, a short presentation as Rene Ribaud was presenting DocOpt and staticjinja.

Pexpect is a pure-Python module implementing Expect functionality.  Expect is functionality to enable to automate interactions with some interactive process where the user is obliged to interact with the program.  Expect was originally written in tcl but versions have since appeared in many other languages.

When using expect you spawn a new process and then interact with that process through a series of interactions where the expect() is called and blocks until a suitable match occurs, e.g. a password prompt, and then send() is used to send text to the process as if sent by the user typing at the keyboard.

I started using pexpect because I had a special use case.  When installing a particular N-node OpenStack platform it was necessary to pass by several machines/accounts to reach the controller or compute nodes.  As machines and user accounts were automatically created with unknown passwords and installed ssh keys, it was necessary to perform a sequence of 3 ssh hops and 3 sudo calls to reach a compute or controller node.

Using pexpect it was possible to auto-login across those 6 accounts, by sending the password when necessary.  A script was written mhop_ssh.py to perform such multi-hop logins and even to install ssh keys along the way thus allowing traditional direct ssh connection to the controller or compute nodes.

I've placed my presentation notebook and the mhop_ssh.py script on github here.

For testing purposes I created a group of virtual servers using Docker, described here.

Using Python with MongoDB

On Thursday evening I hosted the Grenoble Python User Group - short on speakers I proposed to present on "Using Python with MongoDB" - not easy at the moment with lots of NFV work and preparing to move house.

Anyway, I made it and so I've posted both my slides on SlideShare

and the Jupyter notebook ("what else?") on github

I hope to use this as the basis for some future presentations/labs around Python and NoSQL databases.

At last a FutureLearn downloader ...

I try to be less of a freeloader these days but there's still a temptation when you need a tool to just wait for someone else to provide it.

As coursera-dl and edx-dl have existed for a while I just assumed that a futurelearn-dl would be created soon ... real soon ... real soon now ...

But it didn't happen.
I signed up for FutureLearn courses, as you do, and I didn't follow them all as I couldn't carry the videos around with me on my tablet (I have a 45 min commute morning and evening, will also be mainly "offline" in Chile soon).

I would search occasionally for the fabled futurelearn-dl ... with no luck.

Then a year ago there was hope when someone posted a gist of a futurelearn_dl.sh, see here.
So I tried that, I modified it a bit and I managed to semi-automatically download some videos from an FL course, but it was semi-manual ... no good.

Anyway, yesterday I tried again with my modified bash script but I couldn't get it to work, couldn't get it to login anymore.  But as some people had requested that I post my modified version I decided to swallow my pride and create a repo in the hope that someone else would get it working.

But the shame of that badly coded, unworking bash script was too much so I thought well we'll see how far I get with a Python3 version ...

Surprise, less than 24 hours later I have a less than 600-line Python3 script which does the job ... mostly ...

Basically I do
    futurelearn-dl.py myaccount mypassword data-to-insight 2

to download all available material from run 1 of  the currently running 'data-to-insight' course,
or

    futurelearn-dl.py myaccount mypassword data-to-insight 2 1

to download just week 1.

You can obtain the course id 'data-to-insight' and the course run number '2'  in this case by logging
into the site and navigating to the appropriate course page:
    https://www.futurelearn.com/courses/data-to-insight/2/todo/3004

From the URL we can see the course_id and the run number.

The repo I created is here:

    https://github.com/mjbright/futurelearn-dl

Let me know if it's any use, and file issues for any problems (I won't have much time until December I'm afraid).

Time to do my homework ... well, maybe I'll just head outdoors instead ... it's healthier !

"Ipython vers Jupyter", my Jupyter talk at Pyconfr, Pau, France.

I've posted the slides of my talk about Jupyter which I gave at Pyconfr, in Pau, France last Saturday, here on Slideshare, or here with the IPython notebook source on github.

I'll put a link to the talk video when that is published, though I'm a little ashamed that my slides were seriously cropped (lesson learnt: never believe them when they tell you you'll have 1440x960 resolution ... you'll get 1024x768 like anyone else !!).  Erreur de debutant as they say around here ;-)

It wasn't squeezing all that into 20 mins including a demo, so I hope to go much farther in other conferences later.

I was pleased to be able to use Damian Avila's RISE (Live reveal) extension allowing to edit and present the notebook in slide format.  For me it'a a killer app for Jupyter ... your mileage may vary.

I'll also post comments about the questions I got during and after the presentation.

One question in particular was whether or not there was a tutorial describing the use of RISE for producing presentations.  In fact apart from the github page itself, here, with it's README, I'm not aware of any tutorial but I do think it's needed.  I'll try soon to write a short tutorial based on my experiences.

Jupyter - an amazing tool

So after months - well years already - of me saying Docker-Docker-Docker, and my colleagues thinking I'm Docker sponsored given the preponderance of Docker t-shirts I wear, there's a new kid on the block.

From now on it's going to be Docker-Jupyter-Docker-Jupyter ...

Jupyter, the multi-language notebook software derived from the IPython project is now in it's 1.0 release since August.

On Thursday I presented Jupyter at a Meetup of our local Grenoble Python User Group.

This was a useful dry-run for my presentation at Pyconfr in Pau on October 17th (French).
Though that will be a 20 mins presentation, quite a bit shorter.

The presentation mixed slides and demo integrated thanks to the excellent Jupyter RISE extension from Damian Avila.  This allowed to execute code within the presentation slides themselves in the browser.  I find this very powerful and it's something I will continue to develop for my personal and professional presentations and demos.

You can see an example below of code being executed within the presentation slide itself in a browser. In this case demonstrating the use of Sympy to solve a quadratic equation.

Note that the use of Jupyter means that this is live code.  I could modify the markdown (add more descriptive text other than the title), or modify the Input cell to express a different formula and then solve that equation within the presentation.
Live coding !!

I've posted the Jupyter notebook I used on github here:
                2015-Sep-24_Pyugre_HP_DataAnalysisLiftOffForJupyter.ipynb

Note: I've also submitted a proposal for Confoo in Montreal next February ... you can vote for that talk here (but you'll need to register)

Note: In a followup post here I describe how I used Jupyter to perform some basic Docker demos the next day during a training session.

Setting up Anaconda/Py3k/Selenium/Django under Windows, for TDD with Python

This is a description of the initial setup under Windows allowing me to follow along with the O’Reilly book ‘Test-Driven Development with Python’ which I'm currently reviewing.

I’d have done all my TDD with Python work under Ubunutu/Docker if my trusty little Atom-based machine hadn’t gone west recently (just a fan died, but I still haven’t gotten around to replacing that special sized fan).

So I decided to test natively on a Windows8 machine - this would be a good opportunity to test that the book doesn’t depend on Linux.

Linux
In the meantime, due to some strange Windows connection problems I also tried the code on a Linux Mint system just to be sure. No problems there. I also managed to workaround the connection problems seen on Windows, described later.

PythonAnywhere
Note: Appendix A of the book describes how to use the PythonAnywhere, a PaaS (Platform as a Service), allowing you to run the code in the cloud in a Python-hosting environment. I did try this but unfortunately couldn't figure out how to combine the site-generated web app and my github repository. It's unfortunate that the appendix isn't more explicit about how to do this.

Upgrading Anaconda to Python3

On Windows I work from a Cygwin shell, but decided I wanted to use my Anaconda installation.

Well first of all I realized my Anaconda was running Python 2.7.6, not Python3 which the book, quite rightly, uses.  A bit of Googling allowed me to install Python3 under Anaconda as a separate environment, as described here.

So I installed a Python3 environment into Anaconda using (bare in mind I’m using Cygwin so I have Unix-style PATHs here):

cd /c/Progs/Anaconda/

conda create -n py3k python=3 anaconda

The Anaconda activate script wasn’t doing what I needed - in my Cygwin environment - so I decided to explicitly add appropriate paths to my PATH environment variable by adding the following lines at the end of my .profile:

echo "Adding Anaconda python3 to PATH" >&2

export PATH=/c/Progs/Anaconda/envs/py3k:$PATH

export PATH=/c/Progs/Anaconda/envs/py3k/Scripts:$PATH

Now source your .profile and verify that python -V produces something like:

Python 3.3.5 :: Anaconda 2.0.1 (64-bit)

Note: The Anaconda installation installs Python3 as python.exe and Pip as pip.exe, which could be

confusing as the book refers to python3 and pip3 as executeables.

First functional test run

At this point I ran the first version of the functional_test.py script described on page 5 of the book.

This failed because I hadn’t installed selenium yet.

So a quick

   pip install selenium

instantly fixed this.

I reran the test which failed because it wasn’t seeing my ‘Portable Firefox’.

I Googled to see how to point Selenium to my Portable Firefox but decided it would be much easier to just install a standard Firefox, so I did that.

I reran the test which now opened a new Firefox window, and then subsequently failed because I didn’t have Django installed yet, this had been described in the book pre-requisites.

Installing Django 1.7

As described in the book, if django 1.7 isn’t yet available, the following command will fail

   pip install django==1.7

and so we need to install from another source.  Unfortunately the books’ link to django1.7 under github was no longer valid (hey, this is the ‘wibbly wobbly web’ and so my link below probably won’t work for you either … but with a bit of Googling you’ll get there!)

I performed a pip install to install a pre-release Django 1.7 as described on the download page:

pip install https://www.djangoproject.com/download/1.7c3/tarball/

which installed Django 1.7 release candidate3 without problem.

Update: Django 1.7 is now released.
So

   pip install django==1.7

is now sufficient.

Second functional test run

I then reran the test and got the expected error as described on page 5, because Django wasn’t yet up and running.

$ python functional_tests.py

Traceback (most recent call last):

 File "functional_tests.py", line 7, in <module>

   assert 'Django' in browser.title

AssertionError

I tried the

django-admin.py startproject superlists

command from page 6 of the book, but this complained that the file couldn’t be found (maybe due to me using Cygwin?), noticing a file django-admin.exe in the Anaconda Scripts directory of my Python3 environment I decided to try that:

django-admin.exe startproject superlists

returned instantly and I saw that the project had been created locally in sub directory superlists.

I then started my new Django project:

          cd superlists

          python manage.py runserver

and then relaunched the functional tests.  This time the browser opened displaying the Django welcome page “It worked!”.  The functional test now succeeded because it correctly detected the word Django in the browser page.

Up and Running

This got me operational for running the initial tests in the book.

I’ll update this blog post later with any further steps such as MongoDB installation.

Workaround for Windows connection problems

During chapter 6 the functional tests start to create a second firefox session.
At this point I started getting some connection errors under Windows.

 File "C:\Progs\Anaconda\envs\py3k\lib\socket.py", line 426, in create_connection

   sock.connect(sa)

ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it

For a while I just put up with this, but I decided to insert some print statements into my functional_tests/test.py file to determine at what point the connection error was occuring.
I found that inserting a print statement at the end of each of the two browser sessions prevented this error from occuring !

However, later in the book (you see the book really covers everything you need !!) it's mentioned that it's sufficient to insert a self.browser.refresh() before .quit(). I tried that before the .quit() in my functional_tests and at the end of the test script and the problem went away - without the need for any print statements !