At some point, I firmly decided not to be a developer as I found the repetitive chunks of code that kept popping up pretty tedious, even for me. That’s when it dawned on me that there is this whole other aspect of IT in the background, i.e. the support for all the code that gets churned out daily. So, I went into the search for a job where I would work with stuff code runs on instead of working on the code. Luckily the folks at CROZ were just starting up a summer internship for, among others, systems engineers.
A couple of weeks later I got the call, went in for what was probably the most relaxed interview of my life and I was in. I always liked the idea of knowing what makes stuff tick and of course how to make things tick how I want them to. Coming in I expected to get a server rack, some cable and a pair of crimping pliers but boy was I in for a treat!
The project was something way more modern than expected and something both me and my teammates knew squat about – OpenShift! Having been greeted with a pep talk, we set out on a two-week course of educational workshops. The number of technologies covered was hefty and afterward, we set out to put together a functioning Nextcloud instance in the container space of OpenShift. From my perspective – focusing more on the security part of the entire thing – the first thing to set up was access to the OpenShift platform for everyone. It took a couple of days to get my bearings and set up my first in a long line of LDAP connections. Having seen how it’s done, I’ve gathered more than enough skills to do it over and over.
Next, I had to set up a bit of Ansible magic to automatically assign new projects for a list of specific users, mainly myself and the other interns. This was quite a fun trip down the YAML rabbit hole, and one of the first real encounters with Ansible. The simplicity of the configuration files was just a mirage for the real power hid behind legions of Ansible modules. These modules and their documentation spent quite some time on my screen as I went through them looking for suitable ones to use and how to properly use them. As time went on and I came back to the modules, I saw how much space there was for further improvement in writing them.
Now that everyone had their playgrounds all set up, we could go out and each, on our own, set up a personal instance of Nextcloud to see what it is and how it acts. This in turn led to a long frustration of not knowing that OpenShift has some quirks that aren’t present on Kubernetes/Docker for which the original container images were built. The most profound quirk was the fact that OpenShift didn’t allow for default ports such as 80, 443 and anything else below 1024 to be used by containers. So on I went to explore ways of changing the immutable container.
What I learned doing this I will probably never forget. The first thing was how Docker builds images, or, to be precise, how Dockerfiles work. It’s actually quite simple in itself, but it still takes a bit of playing around to see what you can and cannot do. The end result was a Docker image with a bit of customization to the Apache server setting its default ports to a higher value that could be used by unprivileged users. This would later grow into an image hosted on Quay.io with its own GitHub connected build trigger to ease my updating process which I’m really proud of.
I’m not going to lie – this wasn’t a quick Google search of a problem. It took me about a week or maybe more to figure the port thing out. Of course, there was always another possibility of changing the OpenShift security policy to allow for privileged container execution, but I felt this was the better option. I think this was the most complicated part of the entire deployment process.
Having finally made a functional version of the Nextcloud container I shared it with the others as they were doing their own things. Next up was getting a database up and running and thanks to OpenShift it was a breeze – basically just a few clicks on the web console. Afterward, I got an introduction to Keycloak and general identity management and was instructed to get an instance up and running so the users can use an SSO functionality. This involved playing around with LDAP, with the settings on Nextcloud and with Keycloak itself. It’s a powerful but – as with everything else – ultimately simple tool for managing user authentication. 2FA was added to users logging in and there was a bit of a hiccup getting the LDAP connection up and running, but nothing a quick message to the guys mentoring us didn’t solve.
After that, I played around with getting a bash script to auto-deploy everything I did over the past few weeks. This was before setting up the Quay repository so it involved calling a Docker build and push and deploying everything to OpenShift with Ansible. What followed soon was polishing the entire project. This meant getting proper mock users, stress testing for multiple requests at once and large file uploads, finding the best plugins to make Nextcloud our own, changing the database from the humble MySQL to the bigger and stronger PostgreSQL and making some tweaks to the Keycloak user-base.
Finally, after 3 months of learning and churning through tutorials and manuals, we had pooled all our instances together, got a nice auto-deployment via Helm charts and Operators, logging and monitoring with the ELK stack and of course a shiny functional Nextcloud instance ready for the Student Show Off. As far as I could tell, everyone was and still is excited for this project to go from our summer project into full production, so we can replace our current system with this powerhouse, allowing us a safe and smart alternative for sharing internal documents. It should be said that for all the great stuff OpenShift has to offer, it is lacking in some areas and it still isn’t a universal solution for running code. But more on that in another article. As for my experience in building a cluster-based system for the first time, I must say it was incredibly exciting and fun. Looking back, it was an experience from which I will take more than my fair share of knowledge and skill.
Petar Ivanković Milošević
