[Conference - CodeEurope.pl] Developing Micro-services on Kubernetes

In April I had the chance to present at CodeEurope.pl, first in Warsaw on Apr 24th, and then in Wroclaw ("wroslof" was my best attempt at pronouncing Wroclaw) on Apr 26th.

It was a fun event, very well organized by Tomasz Kreczmar who headed up the team at Absolvent.pl, and it was a great experience to exchange with the other speakers on similar career tracks to myself.

So, why am I writing about this now?

Well, I'm preparing for some talks at Pyconfr, in Lille in October and needed to finally clean up the demo I'd prepared for CodeEurope.pl.  Although the slides were ready at the time I spent an intense but creative afternoon in a hotel  preparing my demo.

Recently it took me a while to figure out how I'd run the demo -  between jobs in April I'd created the demo on a borrowed MacBook.

So I wanted to take the time to properly document the setting up and running of the demo as I was pretty please with the result.

I've done that now, and that version is available at https://github.com/mjbright/ConferenceDemo.2018-04_codeeurope-microservices.

Slides from the presentation are available at https://mjbright.github.io/Talks/index.html#201804_codeeu.

I will be improving this demo and will post the new code and slides when that's done.  I intend to improve the visualization in particular with a clean slate implementation.

I'll also post a video of the demo, but right now I have to prepare my 3 Pyconfr [English version] sessions ... hope to see you in Lille!

Kubernetes Micro-service Demo with Visualization

The demo was showing some typical operations you would perform when deploying an application to Kubernetes.  In this case the application was a very simple Flask application communicating with a Redis backend acting as a counter.

Whilst that's not rocket science it allowed to show

- Deployment of the application components

- Exposing the components as services

- Using Nginx as an Ingress Controller

- Scaling of the Flask pods

- Rolling upgrade of the Flask pods

- ... and *piece de maitre* a cobbled together demo dashboard

Demo dashboard

I decided to create a "demo dashboard" where I could run commands, invoke the service and visualize the results all in the same browser window.

To be honest I don't want to reinvent the wheel, and I'd much rather use something like JupyterLab - I will certainly try that in the future - but I needed a quick and dirty solution.  I'm quite pleased with what I cobbled together in a (long) afternoon and I will continue to develop that before Pyconfr next month.

The demo dashboard, see below, integrates:

- In the top line a button to invoke my Flask app service, alongside the output "counter value=..."

- An integrated terminal using ttyd

- A kubernetes visualizer derived from Brendans Burns' live-k8s-visualizer on github

Below is a screenshot showing the application button, and the output showing the current counter value.  Below that we see the apply of flask-deployment.yaml to change the version of the flask-app image, thus provoking a rolling upgrade.

In the visualizer we can see the upgrade happening ... the blue "flask-app:v2" Pods are being started whilst the green "flask-app:v1" Pods are still running.  Once all v2 Pods are deployed the v1 Pods will disappear.

Thanks to the CodeEurope team, and Tomasz, I really hope the event will be on again next year !

PyCon US in Portland - notes from our Jupyter/IPython-in-depth tutorial

Last week I was in Portland for PyConUS.

Thursday I contributed to the Jupyter/IPython-in-depth tutorial led by Matthias Buissionier and MinRK core Jupyter project members.

Tutorials

IPython and Jupyter in Depth: High productivity, interactive Python

2017-05-18 Tutorials, 9 a.m. – 12:20 p.m.

I'd previously run part of this tutorial at EuroPython 2016 in Bilbao, so I was familiar with some of the github ipython-in-depth content - though there is an immense amount of materials from which it is necessary to choose/adapt.

It was a great opportunity to be able to co-present with Matthias and Min and meet some core Jupyter team members.

Matthias had done quite a bit of work updating the repo which has been used for various workshops since 2012, pre-dating even the announcement of the Jupyter/IPython bigsplit (at SciPy in 2014?).

It was proposed that I present how to deploy JupyterHub in the Cloud as I'm familiar with Docker/Kubernetes/Google Cloud Platform.

I was really pleased with that for a couple of reasons

- one of the main questions I had at the end of my 3hr Bilbao tutorial was how do we install JupyterHub
  At the time it was harder to install and I'd hoped to address this at some point.

  In the meantime the Jupyter project, supported by the Berkeley Institute for Data Science Education Program has further developed JupyterHub and in particular installation procedures have been developed and published.

- that procedure makes use of the Kubernetes Helm project "The Kubernetes Package Manager", something which has been on my todo-list for a while, so this was an opportunity to see how easy that is to use

Matthias pointed me to the "Zero to JupyterHub" website which describes in a very clear way how to deploy JupyterHub on Google Cloud Platform or Microsoft Azure.  In fact those instructions could be used on any  Kubernetes platform.

I used those instructions to
- deploy a basic jupyter hub image
- extend this by building a docker-stacks base-notebook image, with ipython-in-depth repo integrated
- extend this to a more complete image dervied from the docker-stacks minimal-notebook image to which I added Thomas Kluyvers' bash kernel

The deployed example is absolutely not for production being http based without any realy login authentication.  But the "Zero to JupyterHub" explains how to add proper authentication for example using OAuth2 with Google.

All this is well explained in the "Zero to JupyterHub" docs here
http://zero-to-jupyterhub.readthedocs.io/en/latest/

PyCon US in Portland from today ... notes from our Container Orchestration tutorial

So here I am jet-lagged in Portland, hours before the opening keynotes of PyConUS.

I've been here a couple of days already as on Wednesday Haikel Guemar and I ran our Container Orchestration workshop which we're developing and running in conferences along with Mario Loriedo.

Thursday I contributed to the Jupyter/IPython-in-depth tutorial led by Matthias Buissionier and MinRK core Jupyter project members, I'll make another post about that session.


Container Orchestration Tutorial

Tutorials

ContainerOrchestration.py: The tutorial session

2017-05-17 Tutorials, 1:20 p.m. – 4:40 p.m.

The intention was to
- cover each of Docker Swarm, Kubernetes and Apache Mesos
    - using both command-line and Python API bindings (Marathon in the case of Mesos)
- allow attendees to work through the materials-*
    - with their preferred choice of running on their machine (Linux, MacOS or Windows) or in the cloud.

* - currently on a branch at ContainerOrchestration/Labs/tree/PyConUS2017

To facilitate their setup I'd sent setup instructions in advance, but due to a conference website glitch it seemed most people did not receive.  Anticipating that many attendees wouldn't be able to install in advance I loaded up USB keys with executables (Docker for XX, Toolbox, Minikube, Anaconda Python, some modules, and docker-images) in advance.  5G in all.

The tutorial started out badly with barely working WiFi (I also had Chrome crash on me minutes before, and my PC very slow) which was a frustrating experience for all.

So in the end I demonstrated the tutorial contents (Docker Swarm and Kubernetes) on my own machine.
I ran both command-line and Python examples from within Jupyter.  I like to create and possibly run tutorials this way as it can be possible to build fully runnable - whence easily testable tutorial materials as well as mixture with markdown and rich media.

Container Orchestration Tutorial - contributions welcome
I welcome participants to look at those materials on github, run through the tutorials themselves and provide feedback by whatever means (comments here, e-mail, or preferably github issues or PRs).

Suggestions/contributions are welcome for
- bugs, suggestions for the code itself
- suggestions for clearer text description of steps
- suggested new features, or organization of the lab (we intend to make more modular)

New features could include
- more features of existing container orchestrators
- use of different language bindings such as Go, Rust (Mesos has native Rust api bindings)
- new orchestrators such as Nomad, Kontena or Morpheus

Haikel, Mario and I intend to continually develop this tutorial and run the lab or present on Container Orchestration in different conferences.  These materials will be finalized at the above url before being merged in to the main branch.

DockerCon 2017

So much has already been said about DockerCon 2017 and the many announcements so I won't go there, I'm way too late ... but here's a short (oops ! long !!) post on my trip to DockerCon in Austin, TX at the end of April.
So here is a more personal account of DockerCon which for me was a fantastic event full of opportunities.

As mentioned in this "Unikernels, Unikernels, Unikernels ... DockerCon !! post, I really appreciate the support I received from many people starting with my wife Paulina, as it was not in easy circumstances.

As I tweeted I really appreciate the community that Docker Inc. have built around Docker (Moby), Meetups, Docker Captains and they go out of their way to be inclusive - everyone has a role to play and it is valued.

I appreciated the hand-written card I received as a thank-you as a speaker, I was equally impressed to see Docker employees also receive hand-written thank-you cards for their efforts - e.g. Amir for his Moby Mingle contributions.

DockerCon 2015
This was my second DockerCon, having been lucky enough to attend DockerCon 2015 in San Francisco.
That was a fantastic event, that I was so lucky to attend especially as expenses were covered by Docker after winning the "Docker Paris Hackathon" in Dec 2014 with Mario Loriedo - and I should point out that Mario was the brains behind that Hackathon win with "our Sublime Docker" entry.

I really enjoyed DockerCon 2015, able to attend the pre-conference Hackathon (my proposed entry, "The Phedds" from the @PhenomalDocker team, and the resulting hack).  That was a great opportunity to make contacts before the conference and the conference itself was really interesting - my best conference so far.


I mention all this just to emphasize that DockerCon 2017 was even better in my experience crammed full of events and possibilities to meet people and make interesting contacts.

DockerCon 2017 - VIP reception
It all started with a missed connection at Houston which I thought would make me miss the "VIP reception" for speakers on the Monday evening, but thankfully as I dropped my suitcase in my Austin hotel room at 20h35 with only 25 mins left to find the venue and attend I figured it was worth it ... in my quest for contacts in the Docker eco-system ..

Not easy joining a party in full swing but thankfully Victor Coisne the Docker Community Manager saw me and said hi ... just as I was leaving ... that made all the difference, so I stayed a while meeting a few people such as Docker Captain Dan Finneran of HPE - we work in neighbouring teams, Mano Marks and several others.  A great start to DockerCon.

Before DockerCon 2017 - I can't sleep so I have to run ...

I love to run or cycle when I travel ... usually cycling is impractical (though I remember a crazy ride downtown HongKong and then around the Island one time ... but it had taken a while to find somewhere to rent a bike).

I love discovering a place like that and now I use runtastic to capture the experience, it's nice to browse through my runs seeing where I've been ... the last 4 runs have been Austin, Montreal, Madrid and Grenoble ...

Jet-lagged, I went out for a run at 6am.  It was pretty dark even on the way back before 7am so it was a little dangerous in the park - almost fell going down a step I didn't see.  I was amazed at the number of runners I saw, some of them sensibly running with headlamps strapped to their forehead.

DockerCon 2017 - Moby Mingles
Docker introduced the concept of Moby Mingles for this conference allowing anybody to propose a subject to discuss in 1-1 meetings.  I would have proposed something in Unikernels but of course Amir of the MirageOS team had already done that.

There were many interesting subjects proposed but the three I was really interested in were the three proposed by Amir: Unikernels, FaaS (functions as a service) and IoT.  In the end I settled for Uniikernels as this was a golden opportunity to exchange with Amir prior to my presentaton the next day.  He invited Johnny Mkhael along also, this was great as Johnny and I had some interesting talks and as he is Paris based we hope to meet in the future - especially as he lives in the same town as my Chilean sister in law, Marcela.

Johnny impressed me by his blogging on http://jmkhael.io/ straight away at DockerCon, I could learn something from him ;-)

DockerCon 2017 - the conference ...
Then of course the conference itself started with the keynotes.

I didn't intend to attend many of the presentations as I figured it would be better to spend time either meeting people, looking at the booths and preparing my own presentation !

I did attend both days keynotes which I really enjoyed - for me the announcement of LinuxKit was really important.

I also attended Abby Fullers (Amazon Developer Evangelist) "Creating Effective Images" and Brendan Greggs (Netflix) excellent "Container Performance Analysis" talks.

DockerCon 2017 - My Unikernels presentation
I was surprised to discover that the 20 minute "community" presentations would be in part of the exhibition area rather than in a separate room.  Learning this I immediately decided to not attend any of those sessions feeling that they would be difficult to follow because of the background noise of the exhibition area.  Nevertheless as a speaker I didn't have any problem and felt at ease presenting to a sea of faces and I didn't even think to estimate how many people were attending ... 100 maybe?

I'd looked at the agenda and decided I could overrun my 20 min slot if needed which I did, though when I stopped at 29 mins and opened for questions they cut the mike ;-)  Nevertheless, I was pleased that several people remained and we discussed for 15 to 20 mins.

I was very pleased over the following days that several people came up to me with positive comments on my presentation - I think this is more that there is a thirst for information on Unikernels today - and I had the chance to meet more MirageOS team members such as Mindy Preston and Heidi Howard (a string believer in Unikernels - a Univ. of Cambridge post doc. rather than MirageOS team member i think).

I was also contacted by Per Uber the CEO of IncludeOS who wanted to discuss about their product.  We had some very interesting discussion.  IncludeOS is not a legacy OS as shown in my slides - not a very wise move calling any active project "legacy" of course.

DockerCon 2017 - the exhibition area

After my talk I was pretty pleased to be able to unwind a little and spent some time browsing the different booths, collecting a few t-shirts along the way.

Actually, my favourite t-shirt was the HPE/Docker t-shirt for it's very quiet tone ...  black t-shirt with small HPE and Docker logos makes this quite a wearable t-shirt - not just another pyjama top !

DockerCon 2017 - The internals summit

The internals summit was held on the Thursday and was a great chance to meet Docker employees, Docker captains and organizers and others.  The goal was to learn about specific Docker, or should I say Moby projects ...

In the afternoon we had different breakout sessions and so I chose to join the Unikernels group initially where we had very interesting disucssions lead by Amir Chaudhary.
I really appreciate how Amir is making efforts to federate the Unikernels community across diverse technology implementations.

I was also very pleased to get the chance to meet Matt Bajor, who's presentation "Look Ma, no OS!" I'd watched and briefly Idit Levine creator of the Unik project.

I later attended the LinuxKit/Infrakit session but joined the session a little late.

DockerCon 2017 - The organizer summit (in parallel with the captain's summit)

On the Friday morning there was an organizer summit, a great opportunity to meet other organizers and discuss what works, what doesn't in organizing meetups and to here from Karen Bazja, Lisa McNicol and Victor Coisne about plans for future organizer community events.

It was appreciated that the meeting ended with a lunch which we shared with the Docker Captains ... a great way to meet more people, in particular I finally got to chat with Alex Ellis.

The journey home
Unfortunately a delayed flight from Newark to Montreal meant an unplanned stopover in Montreal.
The good news was that I managed to change the flight AirCanada were proposing (from a 3 flight periple Montreal-Toronto-Munich-Lyon taking 15 hr to a direct 7hr flight to Geneva) allowing me to have a nearly full day in Montreal before leaving.

Not wanting to take too many risks I headed to Mont Royal and had a nice run before wisely heading back to the hotel to get my bags and off to the airport.

A panoramic view from Mont Royal

I wouldn't have chosen the delay, but this was a nice few hours recompensation ...

Unikernels, Unikernels, Unikernels ... DockerCon !!!

Cutting to rhe chase ... my DockerCon Unikernels talk video and slides are now available

It's the end of a series for me, a fairly tough 6 months of working holidays and weekends to advance on Container Orchestration and Unikernels by presenting in conferences, so finally a blog post or two.

After our Container Orchestration and labs (with Haikel Guemar, Mario Loriedo) at LinuxCon/ContainerCon in Berlin last October I decided to investigate Unikernels.

I was intrigued by reports of "7 Unikernel Projects to Take On Docker in 2015",
and then Dockers' "purchase of Unikernel Systems" in Jan 2016,
closely followed by Brian Cantrills' post "Unikernels are unfit for production"
and very interesting discussions on "Hacker News".

Bryan can be deliciously outspoken by the way ...

The culmination of this initial investigation was being able to present at DockerCon 2017 in Austin in April.
That was such an amazing opportunity for me and I have to say a big thanks to those who supported me on this (My long suffering wife, Paulina, of course !! Christian Schutz, Bruno Cornec, Patrick Masse at HPE, Jenny Burcio and Victor Coisne at Docker Inc) and thanks also to Amir Chaudhry, Mindy Preston and Heidi Howard of the MirageOS team for their encouragement at the event.

The talk video and slides are now available

There were a few steps along the road to DockerCon ...

From that initial interest in Unikernels I decided to respond to a few call for papers and was initially accepted at DevConf.cz, so to warm up I did a  "Docker Grenoble" Meetup to prepare.  Thanks to those who participated to that "alpha release" presentation ... and for the great comments and questions.

Slides at https://www.slideshare.net/MichaelBright3/2017-jan19-meetupunikernels.


Shortly after came DevConf.czhttps://devconf.cz/, at Brno in the Czech Republic close to RedHat's European center of engineering.  This was my first, but hopefully not last time attending or presenting at DevConf.cz.  The first day started with Mario and I running our Container Orchestration lab.  On the morning of my Unikernels presentation I was preparing in my hotel room when I started to see tweets about a Unikernels debate between Steve Pousty "OpenShift Developer Advocate" and Dan Walsh "Mr. SELinux".  The tweets had me thinking there was quite an anti-Unikernel sentiment at the conference which was a little worrying - in fact I've since watched the video and seen that Steve brilliantly argued that Unikernels are worth investigating ...

Slides at https://www.slideshare.net/MichaelBright3/2017-jan29-devconfczunikernels

I had the chance to present on NFV, and Docker tools presentations and Docker 101 ad Orchestration labs with Bruno Cornec at HPE TSS in Cannes in February ... but no Unikernels.  I was amazed one afternoon to check my e-mail to see a mail from Jenny Burcio and Victor Coisne telling me I'd been accepted to give a talk at DockerCon ... very important to me !

I scurried back to Grenoble on the Thursday evening just in time to be able to present on Unikernels at Snowcamp.iohttps://snowcamp.io/2017/en/welcome in Grenoble.  Thanks to the Snowcamp team for that opportunity.

Slides at here https://www.slideshare.net/MichaelBright3/2017-feb10-snowcampiounikernels


In March HPE TES (aka "HPE High Performance Computing and Open Source Linux Technical Excellence Symposium 2017") I had the chance to present on Unikernels,

The official slides are not available, but were based on this slide set on github

So after a series of quite different presentations, some with no demo, some with one demo and the Snowcamp.io presentation where I feverishly created some demos whilst watching the previous speaker session .... I was ready to present at DockerCon.

Despite the initial intention to show several Unikernel technologies, I decided that this wouldn't work in my 20 min slot (well no one was after me so I stole 29 mins in the end ...).

I decided to give an overview of the technologies and follow this with a demo of an example MirageOS unikernel for a DNS server (a modified version of the example https://github.com/mirage/mirage-skeleton/tree/master/applications/dns) showing
- how "mirage configure -t unix" could be used to build a standard Linux executable of the application
- how "mirage configure -t ukvm" could be used to build a bootable unikernel image (no OS folks !) of the application
- running the unikernel locally under ukvm
- running the unikernel under Google Compute Engine

Note: There is no currently available IaaS which is ideally suited to running Unikernels, with GCE it was necessary to create an image with 1Gby disk for example, but the point was to show that the Unikernel technology is there even if the IaaS aren't ready yet.

That's it folks, thanks to all those who contributed in different ways, it's really appreciated !

Next up is a Container Orchestration lab with Haikel, and a Jupyter-in-depth lab with Matthias Buissonier of the Jupyter core team at PyConUS next week in Portland, Oregon.

After that a bit of rest before reflecting on how to get to DockerCon Europe ...

FLOSS Grenoble Meetup: tmux + tmuxinator

On Wednesday 16th November we had the first FLOSS (Free Libre Open Source Software) Meetup Group in Grenoble.  The group was created recently by Rene Ribaud and myselff and Bruno Cornec are co-organizers.

To set the ball rolling I proposed a presentation of "tmux + tmuxinator".  Tmux is a screen multiplexer, similar to GNU Screen - in fact if you're happy with GNU Screen already, well you might as well stick with that.  That said tmux is a much more active project achieving similar functionality since it's debut in 2011.

The following diagram illustrates the main concepts of tmux.

We deal with a tmux session, which is composed of one or more windows - where a window fills a terminal console, such as a Gnome terminal window.  Windows can be split up into panes, allowing to have several applications or shells running in different panes of the same window view as shown below.

Thus image shows a GNOME terminal, where the currently displayed window is split into 3 panes.
Note in the status bar we see win1* (indicating that we are displaying window 'win1') and win2 another window which we do not see here.

Within this one window we can see 3 shells each in it's own pane

So in a window we could mix several applications, for example a vim text editing session, a web server and a command-line shell for example.

The concept of a session is very useful.
If you disconnect from your tmux session it continues to run in the background.
This can be particularly useful when connecing to a remote server for example, whether you choose to disconnect or lose the network connection you session continues to run and you can reconnect to it at will.

Another useful aspect of the session is that it allows you to group related tasks within the same session.
You might have several sessions running dedicated to tasks such as
- performance monitoring
- (command-line) editing a presentation, and web serving it
- etc ...

That's a very quick overview of tmux, you can find more information in the presentation here, or on the tmux web site, or awesome-tmux list of useful tmux related resources.

Enjoy tmux !

Docker Grenoble: Docker Global Mentor Week meetup

Last week we rose to the challenge of Docker Global Mentor Week by holding a #learndocker Meetup in Grenoble, France hosted by HPE Grenoble - one of 100's of meetups worldwide.  Global Mentor Week was announced in October, on the Docker Blog here.  Just enough time to see the post, signup as a Mentor, become a Docker Meetup co-organizer and persuade the boss to buy us Pizzas.

Docker Grenoble Meetup, 17th November - on Meetup
So we had everything we needed, and we had 36 participants, not bad for a start and I was glad to get quite a few new members of the Docker Grenoble Meetup Group.

Unfortunate that Manuel Vacelet, who created the Meetup Group wasn't available, but the Docker Global Mentor Week dates were fixed and I was at Docker Paris on the Monday, driving down to Grenoble on the Tuesday, presenting at Docker FLOSS Meetup on the Wednesday ... well, you get the idea.

So, we had a good turnout and it seems people were very pleased with the event, the most common comment being "when's the next one ...".

It was essential to have some mentors though, and we had 4, a big thanks to
Rene Ribaud ( @Uggla_ ), Vincent Misson, Christiophe Larsonneur, and Nicolas Thomas ( @thomnico ) for their help making the evening a success.

We also had Stephane Bureau of HPE pass by to take photos.  Much appeciated, thanks Stephane!
tweet

Great participation at #dockermeetup at #HPE Grenoble France 17Nov #Learndocker @Docker @HPE_FR pic.twitter.com/SOypx8VQiU

— Stephane Bureau (@SBUCloud) November 18, 2016

Another great aid was the Docker provided trainings (beginner to advanced) which students could follow.
Also one or 2 people tried the new Play With Docker site allowing to play with Docker online.

So when is the next Docker Global Mentor Week?

When is the next Docker Grenoble Meetup - well that depends on you if you want to present - otherwise we'll probably present on subjects such as Unikernels and also Kubernetes early in 2017.

Dirty COW vulnerability - is your Linux patched?

A bit late to the game I realized I should be patching my linux systems, especially if I continue to use Docker for various "appliances" (simply using Docker to invoke tools via Docker images without having to worry about installation/compilation/dependencies ...).

Apart from some currently powered down kit I need to remember to fix later (a couple of NUCs running Ubuntu 16.04 LTS, some Raspberry Pis) I've a desktop and a laptop both running Fedora 24.

The goal of this post is to gather the essential information so that you can quickly check if you have the vulnerability and how to patch it.

What is the Dirty COW vulnerability

It's a serious linux kernel flaw, ably described in this excellent blog post "Dirty COW - (CVE-2016-5195) - Docker Container Escape".  

Is my system vulnerable?

This blog post details how to fix the vulnerability on several Operating Systems, this may be all you need but I propose we check before/after if the vulnerability is present.

This page lists several POCs (Proof of Concept) of the exploit.

Trying the dirtyc0w.c POC, raw C code is here.

Create a temporary directory, e.g. /tmp/dirtycow

$ mkdir /tmp/dirtycow
$ cd /tmp/dirtycow

Copy/paste that code into a file named dirtyc0w.c
Then follow the instructions" in that file, reproduced here:

Change to the root use and create a test file foo, then exit this root shell.

$ sudo -s
# echo this is not a test > foo
# chmod 0404 foo
# exit

Now as a normal user, list the file
Note that this file is readable but no one can write to the file, normally ...

$ ls -lah foo
-r-----r-- 1 root root 19 Oct 20 15:23 foo
$ cat foo
this is not a test

Now we will build and run the dirtyc0w proof of concept as follows:

$ gcc -pthread dirtyc0w.c -o dirtyc0w
$ ./dirtyc0w foo m00000000000000000
mmap 56123000
madvise 0
procselfmem 1800000000
$ cat foo
m00000000000000000

Note that running our executable as a normal user allowed us to exploit this bug and overwrite a file for which we don't have access rights.   We're vulnerable ...


Fixing the vulnerability

Well hopefully just performing an OS update, e.g. as described in the same blog post referred to earlier wll fix the problem and this post will not go further except to cover my case, Fedora 24.

In my case I was running an "old" Linux kernel 4.7.6 where this bug report, Bug 1384344, at comment #43 informs us that the fix is available in kernel 4.8.4.

Of course all that's needed on Fedora 24 now is to perform a standard OS update using

sudo dnf update

But dont' forget to reboot !!

Then retry the above vulnerability check - no need to recompile, but you need to regenerate an initial file foo with known contents before running the check.

Back to work ...

OK, time to get back to work, it's one of those Saturdays of following the rabbit down the hole ...

Container Con Europe 2016: Container Orchestration

Early October I had the chance to go to ContainerCon Europe in Berlin, annexe to LinuxCon Europe where I was co-presenting with Haikel Guemar and Mario Loriedo of RedHat on our respective sessions.

We shared 3 sessions:
- Mario ran his "5 containers for 5 languages" tutorial
- Haikel ran the "Container Orchestration" Lab
- I ran the "Container Orchestration: Which Conductor?" talk

But first things first, after an early start to get to the airport, on arrival at my hotel I had to go for a run, bascally around the "Tiergarten" and to the Brandenburg gate.

5 Container Patterns for 5 Languages

Based upon an earlier lab session Mario conducted at BreizhCamp, this session looks at various patterns of usage for Docker containers such as "copying items into the build", "mounting items in a volume", "mounting the docker control socket", etc ...

The slides for Mario's workshop are here

Container Orchestration Lab

All materials for this lab were provided on github here and here with the specific lab steps described at these links:

DockerSwarm.md

Kubernetes.md

Mesos.md

We provided access to a remote environment which some people used, or they could reproduce the labs on their own laptop - they needed just Vagrant and VirtualBox.

Though we experienced some connectivity problems the lab went well.

A big thanks to Mario and Haikel for their help in preparing and running the lab !

Container Orchestration: Which Conductor?

In this session we provided an overview of the need for container orchestration, of the currently available orchestration engines with a particular focus on Docker Swarm, Kubernetes and Apache Mesos.

Mario provided us with a demonstration of Docker "swarm mode".

The slides for this session are online here

The Conference LinuxCon Europe / ContainerCon Europe

This was my first "LinuxCon" since 2012 in Barcelona.  That time I was on holiday, so it was great to have HPE support to come to Berlin and actually present and run labs this time.

I liked the conference venue (I remember there being less space to move around at the Barcelona event) and enjoyed spending time visiting the various booths ... of which the first I saw when entering the booth area was the HPE booth.  The HPE booth was generally pretty lively with 2 or 3 people manning the stand, and various freebies, or beer or tombolas animating the stand.

There were many things about the conference that I enjoyed, such as Solomon Hykes' keynote, where he announced the InfraKit project and then bravely did the first commit of this to github.com during his talk.  Nice.

I also appreciated Mark Atwoods' closing keynote speech, dressed in Star Trek attire, comparing the 30, 40, 50s (which decade?) radio and collaborative culture in the US with the collaborative culture of Open Source software in particular as it pertains to the HPE "The Machine" project.

There were conference outings of course, in the evenings, or cycling or jogging in the morning.  This photo is from our jogging through the Tiergarten, on to the Brandenburg gate.  We had an excellent tour guide who led the jogging, providing many interesting facts about Berlin.

What's next?

Well LinuxCon, ContainerCon and CloudOpen will be merged into one conference entitled "The Linux Foundation Open Source Summit" in 2017, as announced here.

We hope to present again on these and other subjects (such as Unikernels) at various conferences.

Mario presented the latest version of his "Container Patterns" tutorial at the Nantes Docker Meetup on 28th November and has proposed the talk in other conferences. Mario's latest slides are here. 

For my part I've run some internal sessions at HPE and will present more specifically on "Docker Swarm" at HPE TSS in Cannes in February.

I hope we will rerun the Container Orchestration talks and labs and for that I've reproduced them in a common repository here.

I've proposed these for several conferences in 2017 ... still waiting for CfP results ...

EuroPython 2016: Jupyter for Everything Else

In July I had the chance to run a 3 hour Ipython-in-depth tutorial and to present "Jupyter for Everything Else" at EuroPython in Bilbao.  This is a poor attempt to write up about that work.  I had great plans to write up that work after the conference but was then busy preparing for ContainerCon Europe ... and so on ...

I ran the 3 hours IPython tutorial on the Monday afternoon.
I'd prepared well asking people to install Anaconda Python and Jupyter in advance, but I assumed few people would actually do that - so I came armed with 5 USB stcks with Linux, Windows and OSX versions of the Anaconda distribution to be free from any possible "conference wifi" deficiencies.

I was very pleased to have help from Gil (@LuRsT, whose name I still don't know ... it would have been impossible to start the tutorial and run around distributing USB keys, so Gil saved the day - thanks Gil.

My slides for the IPython-in-depth tutorial are here, and the github repo here (almost an exact clone from the offcial ipython repo of the same name).

The tutorial went well.  I was a little concerned that we were about 45 at the beginning about about 15-20 people at the end ... but that was after 3 hours with a conference break in the middle.  I was assured this was normal and anyway I could see that those who invested the full 3 hours were very happy to have done so.

On the Friday I presented "Jupyter for Everything Else".  This was intended to look at some alternative uses for Jupyter notebooks, in particular I like being able to use bash kernels to create notebooks of command-line experiments.  I have for  example run Docker demos, or labs using Jupyter+bash kernel.

My slides for that session are available here, the github repo here and the YouTube video below


I also demonstrated some extensions to the metakernel_bash  from Calysto, which allow to display images in line in the notebook, or create graphics using GraphViz dot language, or even invoke graphics such as Lightning or Bokeh servers from the bash command-line.  Unfortunately, I still haven't cleaned up that code in a form which I could create a Pull Request - shame on me ... that's why this blog post is soooo late.

Another important subject was on the use of the mybinder project for publishing live notebooks online.
I've already   blogged about publishing live command-line tutorials using mybinder bash metakernel here, in particular I published a live (you can run and modify the code) tcpdump tutorial, in "tcpdump (online runnable) tutorial under MyBinder/Jupyter".

Another subject I covered was OpenStack monitoring from a Jupyter Notebook (using Python3 kernel) using nbconvert under cron to execute the notebook and create graphical (html) results which could be sent by e-mail.   This allowed me to provide daily status reports for 4 OpenStack platforms that our NFV team manages.

tcpdump (online runnable) tutorial under MyBinder/Jupyter

Preparing for my "Jupyter-for-Everything-Else" presentation at EuroPython in Bilbao I wanted to go a step further with the idea of doing tutorials (live command-line tutorials !) on http://mybinder.org.

You can cut to the chase and run the current tutorials by going to the INDEX PAGE at this link.
There are some instructions on how to use the notebooks there.

NOTE: This will launch a new jupyter server on mybinder.org any time you press on this link.  If left unattended the notebook server will be culled and your work lost.  You can save your work using the "File -> Download as -> Notebook (.ipynb)" menu item.

Following on from my previous post "The Art of the Command Line - ls (seen from Jupyter ... seen from MyBinder)" I've finally added another tutorial.  Shame on me, that repo was sitting there with a wondrous "ls tutorial" example ... not exactly the most technical tutorial.
But it's been tough finding time with 2 house warmings, family visiting, Tango Argentin 4 nights in a row and other excuses (thankfully watching "the foot" Euro 2016 is not one of my vices).


So today I investigated adding tcpdump into the same MyBinder image.  Quite easy once I realized I needed to add an "apt-get update" at the beginning of my Dockerfile so that tcpdump was visible in the repos.  I then added a few "apt-get install -y" into the Dockerfile so that the generated image has tcpdump and other networking tools already installed.


I also installed rakudo so that I can demonstrate Perl6 1-liners at EuroPython ... prepare your tomatoes folks!


I've now added an INDEX page into the image (a notebook of course) with links into the currently running server - clicking on the link below to the INDEX page will start a new server.

Follow the link at the start of this article to launch your own server using my 'mjbright/binder_jupyter_notebook' image.

NOTE: The sources for this image are available on github are available here https://github.com/mjbright/binder_jupyter_notebook


Pull requests for the Dockerfile or with new example notebooks are welcome !
At the moment Bash, Python2 and Python3 kernels are included in the image.

You can run the actual notebook tutorial here