The Art of the Command Line - ls (seen from Jupyter ... seen from MyBinder)

In the spirit of the github repo "the art of the command-line", or Chris Allen's cheatsheets I wanted to put together some cheat sheets and notes about GNU, unix, linux utilities.

But I wanted to produce them using the bash_kernel of the Jupyter notebook thereby allowing to have a notebook representation of the cheat sheet in an executable form (providing Jupyter+bash_kernel are available of course).  I'll put these cheatsheets on github, in a fork of the art-of-the-command-line, here.

In an earlier draft of this blog post I wrote

Ideally, I'd like to point you to an online executeable version of these cheatsheets in the same way we can reference tmpnb, tryjupyter.org or binder.org notebooks but unfortunately none of them provide a bash_kernel.  At least on github you can view them, and also clone them to try out locally.

Thankfully I realized today that this is not the case.
Thanks to Docker, thanks to the Binder project (https://mybinder.org) you can run such notebooks online.

By clicking on the link at the bottom of this page you will launch a container on the mybinder site prepopulated with my ls tutorial notebook in a form that you can modify.

So here's a very first shot at that looking at that venerable performer 'ls'.  OK, not the most exciting example for the moment but more will come, especially combinations of tools.
But why ls?

Well many moons ... many many many moons ago actually, like in the 80s/90s I would look at some HP-UX pages and be surprised - I didn't know about that option!  So I vowed to periodically return to man pages and have another look.  Of course 20-30 years on I don't do that as there are so many commands ... but I should ... sometimes at least.

So I figured that probably even ls has a few surprises - like it's only very recent that I discovered the '-h' human readable option on ls, du and even sort (unfortunately OS-X seems to miss this option on sort).  So as an example on Linux, or Windows/Cygwin, instead of looking at disk usage with
    du -s * | sort -n
I now do
    du -sh * | sort -h
which will show disk space in human readable form and sorted, e.g.
    10k tmp/
    2.1M src/
    1.5G Downloads/

Anyway, you can run the actual notebook tutorial here

In later posts I produce some more exciting tutorial notebooks, and I'll also talk more about Binder.
I have a lot of plans for Jupyter, Binder, Docker, ...

Creating a Docker Build Lab with Jupyter

In keeping with my "Jupyter with everything" moniker (not to mention Docker, Docker, Docker), I recently had the opportunity to create and run a Docker Build Lab with about 20 participants.

To actually create the Docker Lab I used Jupyter with the bash_kernel to create and document the examples.

Using Jupyter was extremely useful - especially when ill, sat in a hotel room finishing the lab (work was crazy in the weeks before the conference).  It becomes extremely easy and fast to rerun the entire notebook to be sure that everything works and recreate a printable document for use.

Although the lab execution was performed in RHEL7.1 virtual machines, I prepared the lab on a Windows PC running a Ubuntu-LTS via Vagrant.  This worked perfectly - though some students were confused to see mention of 'vagrant' user within the lab document (I'll have to be a bit more rigourous on that next time).

I had some small problems creating a lab document which I wanted to be able to distribute in class as a PDF..
I had trouble producing a PDF (well I could just print the notebook in Jupyter but the stylesheet was causing this to print without colour which made it less readable - hacked around this to print to PDF).
My hack worked and so I could distribute a colour PDF.

I decided to archive to github the Jupyter notebook (.ipynb) also exported as HTML and markdown(.md) aswell as the PDF.

The lab is here https://github.com/mjbright/DOCKER_LABS.
This is actually a clone of the repo I used (whence some links may go to the original version) as I want to work this lab as a series of labs in future conferences.

This was definitely a beta run.

I'd like to break this lab down into several labs
- Docker build essentials: focussing more on the low-level details
- Building an application: going step by step to building an orchestrated solution

Hopefully I'll get the chance to rerun this at MixIT here in France in April, or at HPE TSS in June.

Suggestions for improvement are more than welcome.

"Ipython vers Jupyter", my Jupyter talk at Pyconfr, Pau, France.

I've posted the slides of my talk about Jupyter which I gave at Pyconfr, in Pau, France last Saturday, here on Slideshare, or here with the IPython notebook source on github.

I'll put a link to the talk video when that is published, though I'm a little ashamed that my slides were seriously cropped (lesson learnt: never believe them when they tell you you'll have 1440x960 resolution ... you'll get 1024x768 like anyone else !!).  Erreur de debutant as they say around here ;-)

It wasn't squeezing all that into 20 mins including a demo, so I hope to go much farther in other conferences later.

I was pleased to be able to use Damian Avila's RISE (Live reveal) extension allowing to edit and present the notebook in slide format.  For me it'a a killer app for Jupyter ... your mileage may vary.

I'll also post comments about the questions I got during and after the presentation.

One question in particular was whether or not there was a tutorial describing the use of RISE for producing presentations.  In fact apart from the github page itself, here, with it's README, I'm not aware of any tutorial but I do think it's needed.  I'll try soon to write a short tutorial based on my experiences.

Jupyter - an amazing tool

So after months - well years already - of me saying Docker-Docker-Docker, and my colleagues thinking I'm Docker sponsored given the preponderance of Docker t-shirts I wear, there's a new kid on the block.

From now on it's going to be Docker-Jupyter-Docker-Jupyter ...

Jupyter, the multi-language notebook software derived from the IPython project is now in it's 1.0 release since August.

On Thursday I presented Jupyter at a Meetup of our local Grenoble Python User Group.

This was a useful dry-run for my presentation at Pyconfr in Pau on October 17th (French).
Though that will be a 20 mins presentation, quite a bit shorter.

The presentation mixed slides and demo integrated thanks to the excellent Jupyter RISE extension from Damian Avila.  This allowed to execute code within the presentation slides themselves in the browser.  I find this very powerful and it's something I will continue to develop for my personal and professional presentations and demos.

You can see an example below of code being executed within the presentation slide itself in a browser. In this case demonstrating the use of Sympy to solve a quadratic equation.

Note that the use of Jupyter means that this is live code.  I could modify the markdown (add more descriptive text other than the title), or modify the Input cell to express a different formula and then solve that equation within the presentation.
Live coding !!

I've posted the Jupyter notebook I used on github here:
                2015-Sep-24_Pyugre_HP_DataAnalysisLiftOffForJupyter.ipynb

Note: I've also submitted a proposal for Confoo in Montreal next February ... you can vote for that talk here (but you'll need to register)

Note: In a followup post here I describe how I used Jupyter to perform some basic Docker demos the next day during a training session.

Jupyter - an educational tool - for demonstrating or teaching about Docker

On Friday I did an N-th presentation and demo of Docker at work as part of an OpenStack training.

Whilst the slides were in Powerpoint-* the demo was conducted principally using Jupyter, the multi-language notebook tool derived from IPython.

* -  this time, but I plan to use Jupyter/RISE for an integrated presentation/demo next time.

As I'd presented Jupyter at a Meetup of our local Grenoble Python User Group on Thursday, and I'm preparing to present at Pyconfr in Pau on October 17th (French) I was eager to really apply Jupyter for some educational use.

Note: I've also submitted a proposal for Confoo in Montreal next February ... you can vote for that talk here (but you'll need to register)

I was also keen to test new formats for presenting.
On Thursday I'd successfully mixed presentation with demo - using the excellent Jupyter/RISE extension from Damian Avila - allowing me to execute code within the presentation slides themselves in the browser.
I find this very powerful and it's something I felt I really needed for my Docker demos.

So this time I didn't have time to redo my slideset, but I did have time - a morning - to put together a demo environment.

When I have more time I'll extend that work to be a much more comprehensive demo.  I've quite a lot of ideas for that around basic Docker capabilities but also other tools such as Machine, Compose, Swarm ... and more ... but more on that in a later post ...

Here's a screenshot of just one part of the demo.
We can see text describing what's happening (this is entered as Markdown), followed by the input cell where we enter real commands, followed by the output from the command.

So this allows to demonstrate and document live coding and later publish the result -as the above Jupyter notebook posted on github.

Here's the notebook showing the demo steps, presented as a narrative thanks to the notebook format.  You can view directly the notebook rendered on github here:

2015-Sep-25_Docker_Demo/2015-Sep-25_Docker_Demo.ipynb

or if you want to run the notebook in a suitable environment you can download the raw notebook here:

https://raw.githubusercontent.com/mjbright/jupyter_notebooks/master/2015-Sep-25_Docker_Demo/2015-Sep-25_Docker_Demo.ipynb

To use that notebook you'll need to have Docker installed and Jupyter.
I don't intend to describe the intricacies of installing Docker and Jupyter on various systems, there are many articles available on this.  I do just provide an overview below.

This was the first post on using Jupyter to demo Docker, there will be many more to follow as I evangelize both tools at work and in conferences.

---

Setup Notes

For Docker you'll need to install directly on a Linux machine or if on Windows or Mac OSX you'll need to use a virtualized Linux.  You can use the Docker Toolbox (basically a combination of Kitematic GUI, VirtualBox and a linux image).

My demo was performed on a Windows machine.
I personally prefer to have more control over my virtual machines, and so I use Vagrant to install my virtual machine running under VirtualBox.
The Vagrantfile I used is available here.

I also used Jupyter to run/record the demo.
The best way to run Jupyter on either Windows or Linux is to install the Anaconda distribution, available from Continuum Analytics here.

You can find out more about the Jupyter project here, or try it out online here

Using different kernels with Jupyter notebooks under Windows8/Cygwin ... the story so far

Not a post of very general interest but this might be of help to those using Jupyter on Windows.

I've been experimenting quite a bit with IPython and Jupyter notebooks recently.
I've been using them for various MOOCs such as EdX's courses on Apache Spark and Machine Learning or Courseras' Data Science Specialization.

This is all good preparation for my

• Python User Group Meetup "Data Analysis ... lift-off for Jupyter" on 24th September and
• Pyconfr "IPython vers Jupyter" presentation on 17th October.

For the moment my best luggables are my corporate Windows laptop or my wifes' MacBook Air (great but I struggle with the Azerty keyboard ...) so I intend to present with the Windows laptop but this does present some problems (other than just it's weight!).

So I'm planning on using my day to day Cygwin environment with a "native" Jupyter, via the Anaconda suite.  I'll also have a couple of backup solutions just in case

• on the same Wvagrant/Ubuntu/Docker/Jupyter stack
• and a remote Fedora/vagrant/Ubuntu/Docker/Jupyter stack ...

can't be too careful.

I'd be satisfied with the Windows environment if I could get "everything" working without problems.

out of the box I had Jupyter with Python3 OK ("no brainer" thanks to Anaconda).

The R Kernel

The first kernel I added was the 'irkernel' for R so I could do my MOOCs.  That went surprisingly well, once I'd installed R, installed the irkernel package I just had to manually set the path to R in my irkernel/kernel.json file.  A surprisingly easy start.

RISE

Next up was Damian Avilas' LIVE Reveal.js module RISE.  This also installed like a charm.

Great, I'll be able to do my reveal slides directly in the Jupyter notebook ... and I can even embed live code !!

Bash_kernel

I really wanted bash_kernel as I spend most of my day in the Cygwin bash in ConEmu.

This failed to install because of a dependency on pexpect ... as I'd already tried in the past with a Windows version of pexpect I gave up a little easily for now.

That would have been so nice ... looks like I'll need yet another vagrant box to be able to do my Docker demos from Jupyter then!

Update: 2016-May-16 - Metakernel_bash
I've recently been looking at Calystos' Metakernel repository on github.
I've created a fork which allows me to use Metakernel on Cygwin/Windows and also integrates some graphical capabilities.  Unfortunately the graphical capabilities only apply to the Metakernel_bash for the moment, not to other Metakernel magics, I hope to do some Pull Requests to Calysto later.

In the meantime you can try out my fork at
https://github.com/mjbright/metakernel

The original Metakernel from Calysto is available here
https://github.com/Calysto/metakernel

I'll be using this for my EuroPython 2016 talk in July.

IJavaScript

Next up was IJavaScript but alas this has a dependency on the node module gyp which was failing to install.  JSDOM3 was suggested as an old version without the gyp dependency but this failed too.

So I'll just have to make do with %%javascript IPython magic.

IJulia

Finally to the point of this post.

It seemed rather fitting to have the Python/Julia/R triumvirate as a minimum language set.

So I installed the latest Julia v0.3.1, I ran Pkg.add("IJulia") from the Julia console as suggested.

However when I launched the Julia kernel from the Jupyter notebook it was failing to find the Compat library.

To cut a long story short. I found that launching Julia from a DOS command prompt or Windows file explorer then Compat was visible ("using Compat" works).  I've never used Julia at all before but I could see that Compat and other libraries were being stored under my Windows USERPROFILE at

C:\Users\mjbright where as under cygwin I have my HOME set to D:\.

If I launched the DOS command console from Cygwin then I would get Compat not found from "using Compat" in the Julia console.

Setting HOME or JULIA_PKGDIR to appropriate values allowed Julia to find the Compat library.

Now I didn't want to modify my HOME directory generally so I tried setting on the command line.

    JULIA_PKGDIR="C:/Users/mjbright/.julia/v0.3"    c/Progs/Dev/Julia-0.3.11/bin/julia.exe 

or

    HOME=c:/Users/mjbright/    c/Progs/Dev/Julia-0.3.11/bin/julia.exe

both did the job.

but prepending these to "jupyter notebook" produced the assertion error reproduced below.

NOTE: I did try launching jupyter notebook from a DOS command prompt and that worked fine .. but please no, I just don't do DOS command prompts unless I *really* have to !  I wanted something I can call from my Cygwin bash command-line.

The solution was to create a minimal bash script to launch the notebook which sets one of these variables and then launch the notebook via cmd.exe (!).

So now I have:

JULIA_PKGDIR="C:/Users/mjbright/.julia/v0.3" cmd /c c:/Progs/Anaconda3/Scripts/jupyter.exe notebook

not pretty but it does the job ...

OK, now to get some real stuff done ...


Error log when launching jupyter directly from cygwin:

[I 17:47:36.652 NotebookApp] Kernel started: 5d07f88a-0e81-4e9b-8844-fd9d88cf6d5f^M
[W 17:47:36.663 NotebookApp] 404 GET /kernelspecs/julia-0.3/logo-64x64.png (::1) 3.91ms referer=http://localhost:8887/notebooks/Untitled5.ipynb?kernel_name=julia-0.3^M
Starting kernel event loops.A^@s^@s^@e^@r^@t^@i^@o^@n^@ ^@f^@a^@i^@l^@e^@d^@!^@
^@
^@P^@r^@o^@g^@r^@a^@m^@:^@ ^@C^@:^@\^@P^@r^@o^@g^@s^@\^@D^@e^@v^@\^@J^@u^@l^@i^@a^@-^@0^@.^@3^@.^@1^@1^@\^@b^@i^@n^@\^@j^@u^@l^@i^@a^@.^@e^@x^@e^@
^@F^@i^@l^@e^@:^@ ^@s^@r^@c^@/^@w^@i^@n^@/^@r^@e^@q^@-^@i^@n^@l^@.^@h^@,^@ ^@L^@i^@n^@e^@ ^@2^@2^@3^@
^@
^@E^@x^@p^@r^@e^@s^@s^@i^@o^@n^@:^@ ^@0^@
^@[W 17:47:46.685 NotebookApp] Timeout waiting for kernel_info reply from 5d07f88a-0e81-4e9b-8844-fd9d88cf6d5f^M
[I 17:47:48.649 NotebookApp] KernelRestarter: restarting kernel (1/5)^M
WARNING:root:kernel 5d07f88a-0e81-4e9b-8844-fd9d88cf6d5f restarted^M
Starting kernel event loops.
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6bfdd386 -- uv_poll_stop at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
uv_poll_stop at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
stop_watching at poll.jl:263
close at C:\Users\mjbright\.julia\v0.3\ZMQ\src\ZMQ.jl:121
jlcall_close_1370 at (unknown line)
jl_apply_generic at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
jl_profile_is_running at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
jl_gc_run_all_finalizers at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
uv_atexit_hook at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
jl_exit at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
_start at client.jl:404
jlcall__start_364 at (unknown line)
jl_apply_generic at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
unknown function (ip: 4200686)
julia_trampoline at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
unknown function (ip: 4202914)
unknown function (ip: 4199436)
unknown function (ip: 4199739)
BaseThreadInitThunk at C:\windows\system32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\windows\SYSTEM32\ntdll.dll (unknown line)
RtlUserThreadStart at C:\windows\SYSTEM32\ntdll.dll (unknown line)

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6bfdd386 -- uv_poll_stop at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
uv_poll_stop at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
stop_watching at poll.jl:263
close at C:\Users\mjbright\.julia\v0.3\ZMQ\src\ZMQ.jl:121