GSOC GNS3 Docker support: The road so far

So midterm evaluations are ending soon and I'd like to write about my progress before that. If you remember my last update it was about how to write a new GNS3 module. Probably the biggest issue you'll run into is implementing links between between various nodes. This is because GNS3 is a jungle of different technologies, all with their own networking technologies. Implementing Docker links is no different.

Docker is different kind of virtualization than what GNS3 has been using until now -> OS-level virtualization. VMware, for instance uses full virtualization. You can read more about the difference on one of the million articles on the Internet. An important thing to note is that Docker uses namespaces to manage its network interfaces. More on this here: https://docs.docker.com/articles/networking/#container-networking. It's great, go read it!

GNS3 uses UDP tunnels for connecting its various VM technologies. This means that it after creating a network interface on the virtual machine, it allocates a UDP port on that interface. But this is REALLY not that easy to do in Docker because a lot of the virtualization technologies have UDP tunnels built in - Docker doesn't. Assuming you've read the article above, this is how it will work (still having trouble with it):

1.  Create a veth pair
2. Allocate UDP port on one end of veth pair
3. Wait for container to start and then push the other interface into container namespace
4. Connect interface to ubridge

If you're wondering what ubridge is -> it's a great little piece of technology that allows you to connect udp tunnels and interfaces. Hardly anyone's heard of it but GNS3 has been using it for their VMware machines for quite some time: https://github.com/GNS3/ubridge

The biggest problem with this is that this is all hidden deep inside GNS3 code which makes you constantly aske the question: "Where the hell should I override this??" Also, you have to take into consideration unforseen problems like the one I've mention earlier: You have to actually start the container in order to create the namespace and push the veth interface into it.

Another major problem that was solved is that Docker container require a running process without which they'll just terminate. I've decided to make an official Docker image to be used for Docker containers:  https://github.com/gcetusic/vroot-linux. It's not yet merged as part of GNS3. Basically, it uses a sleep command to act as a dummy init process and also installs packages like ip, tcpdump, netstat etc. It's a great piece of  code and you can use it independently of GNS3. In the future I expect there'll be a setting, something like "Startup command" so users will be able to use their own Docker images with their own init process.

It's been bumpy road so far, solving problems I haven't really thought about when I was writing the proposal but Docker support is slowly getting there.  

GNS3 architecture and writing a new VM implementation

Last time I wrote a post I talked about what GNS3 does and how Docker fits into this. What I failed to mention and some you already familiar with GNS3 may know, the GNS3 software suite actually comes in two relatively separate parts:

1. https://github.com/GNS3/gns3-server
2. https://github.com/GNS3/gns3-gui

The GUI is a Qt-based management interface that sends HTTP requests to specific endpoints to the server defined in one of its files. These endpoints normally handle the basic stuff you would expect for VM instance to do: start/stop/suspend/restart/delete. For example, sending a POST request to /projects/{project_id}/virtualbox/vms creates a new Virtualbox instance handled in virtualbox_handler.py. You might run into some trouble getting the GUI to run, especially if you're using the latest development code like me because with the latest development version Qt4 was replaced with Qt5 and a lot of Linux distributions out there don't yet have Qt5 in their repositories. The installation instructions only deal with Qt4 and Ubuntu so it's up to you to trudge through numerous compile and requirement errors.

Generally, every virtualization technology (Dynamips, VirtualBox, QEMU) has the request handler, the VM manager responsible for managing available VM instances and the VM handler that knows how to start/stop/suspend/restart/delete. Going back to the request handler, if we wanted to start a previously created VM instance, sending a POST request to /projects/{project_id}/docker/images/{id}/start would do it. Once the request gets routed to a specific method, it usually fetches the singleton manager object responsible for that particular VM technology like VirtualBox or Docker that can fetch the Python object representing the VM instance based on the ID in the request. This VM instance object has the methods that can do various things with the instance, but are specific for VirtualBox or Docker or Qemu. 

Here are some important files that the current Docker implementation uses but there are equivalent files for other kinds of virtual devices:

SERVER

• handlers/docker_handler.py - HTTP request handlers calling the Docker manager
• modules/docker/__init__.py - Manager class that knows how to fetch and list Docker containers
• modules/docker/docker_vm.py - Docker VM class whose methods manipulate specific containers
• modules/docker/docker_error.py - Error class that mostly just overrides a base error class
• schemas/docker.py - request schemas determining allowed and required arguments in requests

GUI

• modules/docker/dialogs - folder containing code for GUI Qt dialogs
• docker_vm_wizard.py - Wizard to create a new VM type, configure it and save it as a template from which the VM instances will be instantiated. Concretely, for Docker you choose from a list of available images from which a container will be created but this really depends on what you're using for virtualization.
• modules/docker/ui - folder with Qt specifications that generate Python files that are then used to define how the GUI reacts to user interactions
• modules/docker/pages - GUI pages and interactions are defined here by manipulating the previously generated Python Qt class objects
• modules/docker/__init__.py - classes that handles the Docker specific functionality of the GUI  like loading and saving of settings
• modules/docker/docker_vm.py - this part does the actual server requests and handles the responses
• modules/docker/settings.py - general Docker and also container specific settings

This seems like quite a complicated setup but the important thing to remember is that if you want to add your own virtualization technology you have to make equivalent files to those above in a new folder. My advice is to copy the files of an already existing similar VM technology and go from there. All of the classes inherit from base classes that require some methods to exist, otherwise it will fail spectacularly. As is true with every object oriented language, you should try to leave most of the work to the base classes by overriding the required methods but if the methods they use make no sense, write custom code that circumvents their usage completely. A lot of the code used in one technology may seem useless and redundant in another. For example, VirtualBox has a lot of boilerplate code that manages the location of its vboxmanage command that's completely useless in Docker which uses docker-py to handle all container related actions. The core of GNS3 is written with modularity in mind but with all the (very) different virtualization technologies it supports you're bound to do a hack here or there if you don't want to completely retumble the rest of the code a couple of times.

Cheers until next time when I'll be talking about how to connect vastly different VMs via links.

GNS Docker support

GNS3 Docker support

So the coding session for GSOC finally began this week. I got accepted with the GNS Docker support project and here is the project introduction and my plan of attack.

GNS3 is a network simulator that uses faithfully simulates network nodes. Docker is a highly flexible VM platform that uses Linux namespacing and cgroups to isolate processes inside what are effectively virtual machines. This would enable GNS3 users to create their custom virtual machines and move beyond the limitations of nodes that are network oriented and because of its lightweight implementation, would make it possible to run thousands of standalone servers on GNS3.
Right now GNS3 supports QEMU, VirtualBox and Dynamips (a Cisco IOS emulator). The nodes in GNS3 and the links between them can be thought of as virtual machines that have their own network stacks and communicate amongst themselves like separate machine on any other "real" network. While this is nice by itself, QEMU and VirtualBox are "slow" virtualization technologies because they provide full virtualization -> they can run any OS but this comes at a price. So while QEMU and VirtualBox can run various network services, it's not very efficient. Docker, on the other hand, uses kernel-level virtualization which means it's the same OS but processes are grouped together and different groups isolated between themselves, effectively creating a virtual machine. That's why Docker and other such virtualization solutions are extremely fast and can run thousands of GNS3 nodes -> no translation layer between host and guest systems because they run the same kernel. Docker is quite versatile when it comes to managing custom made kernel-based VMs. It takes the load of the programmer so he/she doesn't have to think about disk space, node startup, process isolation etc. Links between Docker containers pose an additional problem. In its current form, GNS3 uses UDP networking (tunnels) for all communication between all nodes. The advantage is that this is done in userland. It is very simple and works on all OSes without requiring admin privileges. However, using UDP tunnels has proven to be more difficult to integrate new emulators to GNS3 because they usually do not support UDP networking out of the box. OpenvSwitch is a production quality, multilayer virtual switch and interconnecting Docker containers and alleviating the problem of new emulators requires at least basic support for OpenvSwitch ports in GNS3. Additionally, this would enable Docker links in GNS3 to be manipulated through Linux utilities like netem and tc that are specialized for such tasks, something not possible with UDP tunnels.

Let's start coding!

Google Summer of Code - prepost

This year (my last year as a student!) I decided to apply for GSOC (Google Summer of Code). It's been on my academical TODO list for a few years but haven't had the time or the motivation to apply. Since this is my last chance, I decided to have a go.

Now, a (not so) short intro...Google summer of code is a stipend programme for open source software that students (bsc, ms, phd) can apply to. Basically, different organizations like GNOME, Debian, CERN and MIT register and submit their proposals and open source project ideas to Google. Google then goes through the proposals and selects somewhere between 100 and 200 organizations. Once the organizations have been selected, students can contact them regarding different project ideas. These ideas are mostly already defined by the organizations but students can propose their own ideas. Once the students find a couple of organizations of interest, they usually contact the organizations before sending the official proposal to Google and before student registrations are open. It's important to start as early as possible to see about project ideas or even come to an agreement that the organization will pick them as one of the students for the project. Once a student feels confident enough and once Google opens student registrations, he/she can register, provide whatever info Google requires them and submit their proposals. Some organizations will require additional info in the proposal so make sure to check that with your prospective orgs. Organizations assign mentors to specific project ideas, pick student proposals and then Google evaluates the proposals and finally picks the students who will be given a stipend during their GSOC enrollment.

IMPORTANT: Being picked by the organization the student proposal is meant for doesn't mean you'll get accepted by Google -> after the organizations have been selected and after student proposal deadline, Google assigns student slots to organizations. So if the organization picked 4 students but Google assigns 2 slots, only 2 students will be accepted for that organization. But they may get selected for a different organization. That's why most students submit several proposals to different organizations. Basically, Google has a limited number of slots and distributes them between organizations.

Most organizations get 1-2 slots but organizations that have a vast number of student applications and are longtime contributors to GSOC get more slots. For example, Python Software Foundation (PSF) is a longtime member of GSOC and an umbrella organization. What this means is that this organization applied for GSOC, got accepted and then accepts other Python projects under its fold. Even some projects that haven't been accepted by Google (maybe because of limited number of slots) later get into GSOC through these umbrella organizations. Google is often extremely generous with slots given to umbrella organizations like PSF and Apache Software Foundation but if a large number of projects get under an umbrella organization, organizations might still get only 1-2 slots and some might theoretically even miss out. It depends.

Enough of the short intro, let's get back to what I originally wanted to write about. Since Python is the only programming language I trust myself to use to get the job done without the language being the obstacle, I've decided to work exclusively on Python projects. Now, I don't want to get stuck with a project that I'd do only for the money (and the amount Google pays isn't negligible from where I'm from) so I picked up three really cool projects to apply for:

• GNS3 - higly veracious network simulator of real, physical networks.
• NetworkX - a software package for the creation, manipulation, and study of complex networks.
• SunPy - Python for Solar Physics

Cool, right? All three projects I've mentioned are part of the PSF umbrella organization. PSF requires students to keep a blog of their project progress and that's the reason this blog exists. It did take some of my time from writing the actual proposals but now that I'm writing it I think it's actually a nice idea.
They're ordered based on preference. I'm working on my masters thesis, porting a network simulator developed at my university from FreeBSD to Linux. That's why GNS3 is first on my list. I can elaborate on the idea there and use it in GSOC. Concretely, the project idea is "Docker support for GNS3".

Background: right now GNS3 supports QEMU, VirtualBox and Dynamips (a Cisco IOS emulator). We can think of the nodes in GNS3 and the links between them as virtual machines that have their own network stacks and communicate amongst themselves like separate machine on any other "real" network. While this is nice by itself, QEMU and VirtualBox are "slow" virtualization technologies because they provide full virtualization -> you can run any OS on them. So while QEMU and VirtualBox can run various network services, it's not very efficient. Docker, on the other hand, uses kernel-level virtualization which means it's the same OS but processes are grouped and different groups isolated between themselves, effectively creating a VM. That's why it's extremely fast and can run thousands of GNS3 nodes -> no calls between host and guest systems, it's the same kernel! Docker is quite versatile when it comes to managing custom made kernel-based VMs. It takes the load of the programmer so he/she doesn't have to think about disk space, node startup and isolation etc.

The second project, NetworkX is basically a graph analysis software written in Python. You define your graph with nodes and edges and run various graph algorithms on it. Before Google announced the selected organizations, NetworkX has been on the PSF GSOC wiki page, one of the first. They're the first organization I've contacted. While for GNS3 I just chose one of the already available project ideas, for NetworkX I've proposed to make a Tkinter GUI since it doesn't have one. It would enable users draw edges and graphs without actually writing programs. This wasn't exactly rejected but one of the core developers explained to me in a lengthy email that while they appreciate the effort, NetworkX is moving away from any kind of GUI development and that I should probably pick one of the existing ideas. So I chose to write a backend API for NetworkX.

Background: Until now, graph data has been represented as volatile Python dictionaries. It would make sense to provide a flexible backend interface to easily create modules for graph storage to efficiently access graphs at any time. This would usually include graph databases since their data representation is close what NetworkX does and such databases have efficient algorithms to access graph data but it shouldn't be restricted to such storages. Case in point would be document-store databases that can more or less directly save Python dictionaries as JSON data and load them. SQL databases are somewhat trickier because their data representation isn't directly compatible with graph.

The last project, SunPy, is a software package for solar physics computations. Now, while the previous two projects are more along the line of what I usually do and study, this is more of a whim! I mean, solar physics! Cool! The project ideas is to refactor on of its modules called Lightcurve. I have to admit I don't know a lot (close to nothing) about solar physics but this refactor project has more to do with actual Python refactoring. I'll probably have to learn something about solar physics, which I'd like to but although GNS3 was added to PSF organization list after SunPy, I've put most of my effort into writing the proposal for GNS3  because of my thesis with a touch of regret that I won't work on software that researches the Sun!

Whichever project gets selected, I'm sure it's going to be a fun and educational experience for me. Keeping my fingers crossed for the next post where I'll write in more detail about the project I'll (hopefully) work on.

Cheers