The Art of the Command Line - ls (seen from Jupyter ... seen from MyBinder)

In the spirit of the github repo "the art of the command-line", or Chris Allen's cheatsheets I wanted to put together some cheat sheets and notes about GNU, unix, linux utilities.

But I wanted to produce them using the bash_kernel of the Jupyter notebook thereby allowing to have a notebook representation of the cheat sheet in an executable form (providing Jupyter+bash_kernel are available of course).  I'll put these cheatsheets on github, in a fork of the art-of-the-command-line, here.

In an earlier draft of this blog post I wrote

Ideally, I'd like to point you to an online executeable version of these cheatsheets in the same way we can reference tmpnb, tryjupyter.org or binder.org notebooks but unfortunately none of them provide a bash_kernel.  At least on github you can view them, and also clone them to try out locally.

Thankfully I realized today that this is not the case.
Thanks to Docker, thanks to the Binder project (https://mybinder.org) you can run such notebooks online.

By clicking on the link at the bottom of this page you will launch a container on the mybinder site prepopulated with my ls tutorial notebook in a form that you can modify.

So here's a very first shot at that looking at that venerable performer 'ls'.  OK, not the most exciting example for the moment but more will come, especially combinations of tools.
But why ls?

Well many moons ... many many many moons ago actually, like in the 80s/90s I would look at some HP-UX pages and be surprised - I didn't know about that option!  So I vowed to periodically return to man pages and have another look.  Of course 20-30 years on I don't do that as there are so many commands ... but I should ... sometimes at least.

So I figured that probably even ls has a few surprises - like it's only very recent that I discovered the '-h' human readable option on ls, du and even sort (unfortunately OS-X seems to miss this option on sort).  So as an example on Linux, or Windows/Cygwin, instead of looking at disk usage with
    du -s * | sort -n
I now do
    du -sh * | sort -h
which will show disk space in human readable form and sorted, e.g.
    10k tmp/
    2.1M src/
    1.5G Downloads/

Anyway, you can run the actual notebook tutorial here

In later posts I produce some more exciting tutorial notebooks, and I'll also talk more about Binder.
I have a lot of plans for Jupyter, Binder, Docker, ...

Using different kernels with Jupyter notebooks under Windows8/Cygwin ... the story so far

Not a post of very general interest but this might be of help to those using Jupyter on Windows.

I've been experimenting quite a bit with IPython and Jupyter notebooks recently.
I've been using them for various MOOCs such as EdX's courses on Apache Spark and Machine Learning or Courseras' Data Science Specialization.

This is all good preparation for my

• Python User Group Meetup "Data Analysis ... lift-off for Jupyter" on 24th September and
• Pyconfr "IPython vers Jupyter" presentation on 17th October.

For the moment my best luggables are my corporate Windows laptop or my wifes' MacBook Air (great but I struggle with the Azerty keyboard ...) so I intend to present with the Windows laptop but this does present some problems (other than just it's weight!).

So I'm planning on using my day to day Cygwin environment with a "native" Jupyter, via the Anaconda suite.  I'll also have a couple of backup solutions just in case

• on the same Wvagrant/Ubuntu/Docker/Jupyter stack
• and a remote Fedora/vagrant/Ubuntu/Docker/Jupyter stack ...

can't be too careful.

I'd be satisfied with the Windows environment if I could get "everything" working without problems.

out of the box I had Jupyter with Python3 OK ("no brainer" thanks to Anaconda).

The R Kernel

The first kernel I added was the 'irkernel' for R so I could do my MOOCs.  That went surprisingly well, once I'd installed R, installed the irkernel package I just had to manually set the path to R in my irkernel/kernel.json file.  A surprisingly easy start.

RISE

Next up was Damian Avilas' LIVE Reveal.js module RISE.  This also installed like a charm.

Great, I'll be able to do my reveal slides directly in the Jupyter notebook ... and I can even embed live code !!

Bash_kernel

I really wanted bash_kernel as I spend most of my day in the Cygwin bash in ConEmu.

This failed to install because of a dependency on pexpect ... as I'd already tried in the past with a Windows version of pexpect I gave up a little easily for now.

That would have been so nice ... looks like I'll need yet another vagrant box to be able to do my Docker demos from Jupyter then!

Update: 2016-May-16 - Metakernel_bash
I've recently been looking at Calystos' Metakernel repository on github.
I've created a fork which allows me to use Metakernel on Cygwin/Windows and also integrates some graphical capabilities.  Unfortunately the graphical capabilities only apply to the Metakernel_bash for the moment, not to other Metakernel magics, I hope to do some Pull Requests to Calysto later.

In the meantime you can try out my fork at
https://github.com/mjbright/metakernel

The original Metakernel from Calysto is available here
https://github.com/Calysto/metakernel

I'll be using this for my EuroPython 2016 talk in July.

IJavaScript

Next up was IJavaScript but alas this has a dependency on the node module gyp which was failing to install.  JSDOM3 was suggested as an old version without the gyp dependency but this failed too.

So I'll just have to make do with %%javascript IPython magic.

IJulia

Finally to the point of this post.

It seemed rather fitting to have the Python/Julia/R triumvirate as a minimum language set.

So I installed the latest Julia v0.3.1, I ran Pkg.add("IJulia") from the Julia console as suggested.

However when I launched the Julia kernel from the Jupyter notebook it was failing to find the Compat library.

To cut a long story short. I found that launching Julia from a DOS command prompt or Windows file explorer then Compat was visible ("using Compat" works).  I've never used Julia at all before but I could see that Compat and other libraries were being stored under my Windows USERPROFILE at

C:\Users\mjbright where as under cygwin I have my HOME set to D:\.

If I launched the DOS command console from Cygwin then I would get Compat not found from "using Compat" in the Julia console.

Setting HOME or JULIA_PKGDIR to appropriate values allowed Julia to find the Compat library.

Now I didn't want to modify my HOME directory generally so I tried setting on the command line.

    JULIA_PKGDIR="C:/Users/mjbright/.julia/v0.3"    c/Progs/Dev/Julia-0.3.11/bin/julia.exe 

or

    HOME=c:/Users/mjbright/    c/Progs/Dev/Julia-0.3.11/bin/julia.exe

both did the job.

but prepending these to "jupyter notebook" produced the assertion error reproduced below.

NOTE: I did try launching jupyter notebook from a DOS command prompt and that worked fine .. but please no, I just don't do DOS command prompts unless I *really* have to !  I wanted something I can call from my Cygwin bash command-line.

The solution was to create a minimal bash script to launch the notebook which sets one of these variables and then launch the notebook via cmd.exe (!).

So now I have:

JULIA_PKGDIR="C:/Users/mjbright/.julia/v0.3" cmd /c c:/Progs/Anaconda3/Scripts/jupyter.exe notebook

not pretty but it does the job ...

OK, now to get some real stuff done ...


Error log when launching jupyter directly from cygwin:

[I 17:47:36.652 NotebookApp] Kernel started: 5d07f88a-0e81-4e9b-8844-fd9d88cf6d5f^M
[W 17:47:36.663 NotebookApp] 404 GET /kernelspecs/julia-0.3/logo-64x64.png (::1) 3.91ms referer=http://localhost:8887/notebooks/Untitled5.ipynb?kernel_name=julia-0.3^M
Starting kernel event loops.A^@s^@s^@e^@r^@t^@i^@o^@n^@ ^@f^@a^@i^@l^@e^@d^@!^@
^@
^@P^@r^@o^@g^@r^@a^@m^@:^@ ^@C^@:^@\^@P^@r^@o^@g^@s^@\^@D^@e^@v^@\^@J^@u^@l^@i^@a^@-^@0^@.^@3^@.^@1^@1^@\^@b^@i^@n^@\^@j^@u^@l^@i^@a^@.^@e^@x^@e^@
^@F^@i^@l^@e^@:^@ ^@s^@r^@c^@/^@w^@i^@n^@/^@r^@e^@q^@-^@i^@n^@l^@.^@h^@,^@ ^@L^@i^@n^@e^@ ^@2^@2^@3^@
^@
^@E^@x^@p^@r^@e^@s^@s^@i^@o^@n^@:^@ ^@0^@
^@[W 17:47:46.685 NotebookApp] Timeout waiting for kernel_info reply from 5d07f88a-0e81-4e9b-8844-fd9d88cf6d5f^M
[I 17:47:48.649 NotebookApp] KernelRestarter: restarting kernel (1/5)^M
WARNING:root:kernel 5d07f88a-0e81-4e9b-8844-fd9d88cf6d5f restarted^M
Starting kernel event loops.
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6bfdd386 -- uv_poll_stop at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
uv_poll_stop at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
stop_watching at poll.jl:263
close at C:\Users\mjbright\.julia\v0.3\ZMQ\src\ZMQ.jl:121
jlcall_close_1370 at (unknown line)
jl_apply_generic at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
jl_profile_is_running at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
jl_gc_run_all_finalizers at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
uv_atexit_hook at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
jl_exit at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
_start at client.jl:404
jlcall__start_364 at (unknown line)
jl_apply_generic at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
unknown function (ip: 4200686)
julia_trampoline at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
unknown function (ip: 4202914)
unknown function (ip: 4199436)
unknown function (ip: 4199739)
BaseThreadInitThunk at C:\windows\system32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\windows\SYSTEM32\ntdll.dll (unknown line)
RtlUserThreadStart at C:\windows\SYSTEM32\ntdll.dll (unknown line)

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6bfdd386 -- uv_poll_stop at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
uv_poll_stop at C:\Progs\Dev\Julia-0.3.11\bin\libjulia.dll (unknown line)
stop_watching at poll.jl:263
close at C:\Users\mjbright\.julia\v0.3\ZMQ\src\ZMQ.jl:121