OpenShift CLI oneliners to get your containers started quickly

Configuring assets in your OpenShift namespace usually involves a lot of clicking and typing in the GUI. For some, it involves juggling massive yaml files containing the same information, but machine readable.
Both involve quite a bit of work and a good understanding of OpenShift and its inner workings to do the right things in the right order.

Luckily, there is a CLI tool to aid in this: the OpenShift Command Line Interface, a.k.a. OpenShift CLI, or oc for short.
It can be found here, and is freely available with support from Red Hat.
But, as with any tool, it also has its own complexity and quirks.
Below, I’ll discuss two commons situations and how oc can help you get started quickly!

I’m assuming a successfully installed oc tool and a valid (e.g. logged in) session to your OpenShift instance for the next part of this blog.

Create a runnable image from source (dockerfile)

Let’s assume you have created the perfect Dockerfile to build an image and ultimately a container of your application.
Perhaps it is a base image for future Java-based application development? Or even an extended openshift-jenkins-slave image? Use your imagination here!
Fact is: everything that’s needed to build the Docker Image on your machine is also in a git repository and is reachable by OpenShift. Great!

OpenShift is very much capable to build your Image, and store it in its internal Docker Repository in an ImageStream, but some steps have to be taken.

Source secrets

OpenShift will have to check out the code, and for that it requires a username / password combination. This needs to be set up at your VCS (here: GitLab), and the combination has to be registered in OpenShift. This is done via a Secret which can later be referred to as a Source Secret.

To create a secret in OpenShift, the first of our oneliners is introduced:

oc create secret generic {name-of-your-secret} \
--type=kubernetes.io/basic-auth \
--from-literal={username}={password} \
--namespace {namespace}

Before you contact me about this: yes. I know it’s supposed to be a oneliner, and it is split into multiple lines here; technically making it not a oneliner anymore. It’s a oneliner at heart! Just spread out to make it more legible. 😀

Actual content

Having OpenShift being able to read stuff from a git repository is nice and all but worthless until you actually read something from a git repository!

Luckily, OpenShift has a nice on-size-fits-all command to read from an repository and create *all* necessary OpenShift / Kubernetes objects required to actually run the artifact on the platform. This is the command oc new-app and it can use several strategies (--strategy) to accomplish the goal.
Here, we use the docker strategy to take a Docker file and build it into an Image which is ultimately run as Deployment on OpenShift and can be exposed through a Route on the platform to external entities.

The trick is to just point to your repository, provide the source secret and give everything an appropriate name.

oc new-app https://gitlab.local/{group}/{repository}.git \
--strategy=docker \
--source-secret={name-of-your-secret} \
--name={name-of-your-build} \
--namespace {namespace}

This will just create a build and start the build, but not deploy it into your namespace and make it accessible:

oc new-build https://gitlab.local/{group}/{repository}.git \
--strategy=docker \
--source-secret={name-of-your-secret} \
--name={name-of-your-build} \
--namespace {namespace}

Troubleshooting

Every once in a while you probably want to know why your application is not behaving as it should in a namespace. The oc binary can help you here as well!

Reading logs

Logs for pods and/or containers can be veiwed from the OpenShift GUI, but who needs a GUI when there’s a perfectly fine CLI?
Logs from pods can be obtained by using the pod-name:

oc logs {name-of-your-pod} \
--follow=true \
--namespace {namespace}

Or just one container in a pod:

oc logs {name-of-your-pod} \
--container {name-of-the-container-in-the-pod} \
--namespace {namespace}

You can also select pods based on a label:

oc logs --selector {label}={value} \
--namespace {namespace}

Debugging pods

When a container or pods gets stuck in a CrashLoopBackoff state, this will not fix itself. Usually, there’s something wrong with the configuration of the pod. But when the thing won’t start, it is hard to check what’s wrong, right?

A debug instance of the pod will help!
This instance is run without the usual docker ENTRYPOINT but instead /bin/sh is swapped in as a starting command and any liveness or readiness checks are disabled. This gives you a running pod without the actual program started but with all other things equal to a regular start.
The CLI will drop you right into a shell on the pod.

oc debug {name-of-your-pod} \
--namespace {namespace}

More?

There are probably loads more oneliners for your OpenShift Client needs. Let me know if you want me to find some more on a particular subject!

Jenkins + OpenShift: running custom Jenkins agents to perform your work

Working with a managed Kubernetes solution like OpenShift is really a mixed bag of emotions. It’s a great blessing as a lot of the regular infrastructure like hosts, scaling, load balancing, network stuff, and user management is managed by the platform and in extension by you, but it also imposes some constraints and an added complexity in managing your applications.
Suddenly, you’re not logging into hosts anymore but are administering them through a web-based console. Everything should be stateless, and where are my logs going?!

The use of Jenkins is quite nicely integrated into the ‘legacy’ version of OpenShift, 3.x. We are running 3.11(.146).
Setting up a Jenkins instance in your OpenShift Namespace is quite literally as easy as pointing to a Jenkins Pipeline in your Version Control System, and ‘everything’ happens by itself!
A fresh instance of Jenkins is provisioned, the referenced Pipeline is loaded, and you can even click a shiny button ‘Start Pipeline’ from OpenShift. Logging in to Jenkins is done through SSO via OpenShift and all should be well.
But is it?

Often it is.
But sometimes, you need more.

For instance: when you want to run a Jenkins agent node with some very specific tooling.
As worker nodes (agents) for Jenkins are usually separate hosts with a specific toolchain, this will not work out of the box with the automatically provisioned Jenkins.
There’s always the option of manually spinning up instances of your container of choice and configure the container to be a Jenkins Agent. You’ll also need to edit the Jenkins config to accept this container as an Agent node and hope you touched on all the important configuration settings.
I’m not saying it cannot be done, it is just a lot of work. Also: manually maintaining runtime config is laborious and must be redone when migrating the config to other namespaces or when something is accidentally deleted (e.g. your namespace is emptied by an overzealous coworker).

Everything-as-code

Configuration is part of ‘everything’ and thus should be code.
Conveniently, there is a Jenkins Plugin to aid us here: Jenkins Kubernetes.
This plugin enables us to run Agent pods directly on our Kubernetes cluster (i.e. OpenShift), and more importantly, define this Agent based on an image without configuring something in Jenkins itself. Everything is automagically loaded in Jenkins if one is to add some configuration files to the OpenShift namespace. Yay!
Jenkins is notoriously heavy on XML, and this is no exception. Time to load some XML into yaml on OpenShift!

As everything should be as code, it is time to create OpenShift configuration files containing the Jenkins configuration, pointing to the image we are trying to create containers from, to run our software the way we want it.
Luckily, the process is pretty straightforward. Let’s start!

Identify the custom Jenkins Agent image

Red Hat conveniently provides the Container section in the Red Hat Ecosystem Catalog. The images suited to be used as agents can be found by searching for ‘jenkins’ and are usually named jenkins-agent-*.
Here, we want to run a Java11 based Maven agent on OpenShift 3, so openshift3/jenkins-agent-maven-36-rhel7 is the image of our choice.

Jenkins plugin configuration

At its heart, the configuration looks something like this:

<org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
  <inheritFrom></inheritFrom>
  <name>java11-maven</name>
  <instanceCap>2147483647</instanceCap>
  <idleMinutes>0</idleMinutes>
  <label>java11-maven</label>
  <serviceAccount>jenkins</serviceAccount>
  <nodeSelector>beta.kubernetes.io/os=linux</nodeSelector>
  <volumes/>
  <containers>
    <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
      <name>jnlp</name>
      <image>registry.redhat.io/openshift3/jenkins-agent-maven-36-rhel7</image>
      <privileged>false</privileged>
      <alwaysPullImage>true</alwaysPullImage>
      <workingDir>/tmp</workingDir>
      <command></command>
      <args>${computer.jnlpmac} ${computer.name}</args>
      <ttyEnabled>false</ttyEnabled>
      <resourceRequestCpu></resourceRequestCpu>
      <resourceRequestMemory></resourceRequestMemory>
      <resourceLimitCpu></resourceLimitCpu>
      <resourceLimitMemory></resourceLimitMemory>
      <envVars/>
    </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
  </containers>
  <envVars/>
  <annotations/>
  <imagePullSecrets/>
  <nodeProperties/>
</org.csanchez.jenkins.plugins.kubernetes.PodTemplate>

Please note the important stuff here: (and I know it’s a lot of XML for just a small config, but bear with me)
name, label, nodeSelector and image.

With name, you can set the name of the ContainerTemplate in the Jenkins GUI. A little bit more important is label. This is how you can ‘select’ a ContainerTemplate from within a Jenkins pipeline. The nodeSelector is to help OpenShift figure out on which nodes the container can be run on (here, we want to use Linux-based nodes) and with image you pinpoint the specific RedHat Jenkins Agent image you previously selected from the repository.

Making Jenkins load the config

The Jenkins Plugin responsible for running workers on OpenShift automatically loads extra PodTemplates from *any* ConfigMap with certain labels. This has to be the label role=jenkins-slave.

As OpenShift only deals with yamls, one would create an OpenShift yaml like below:

apiVersion: v1
data:
  template1: "<org.csanchez.jenkins.plugins.kubernetes.PodTemplate>\r\n      <inheritFrom></inheritFrom>\r\n      <name>java11-maven</name>\r\n      <instanceCap>2147483647</instanceCap>\r\n      <idleMinutes>0</idleMinutes>\r\n      <label>java11-maven</label>\r\n      <serviceAccount>jenkins</serviceAccount>\r\n      <nodeSelector>beta.kubernetes.io/os=linux</nodeSelector>\r\n      <volumes/>\r\n      <containers>\r\n        <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>\r\n          <name>jnlp</name>\r\n          <image>registry.redhat.io/openshift3/jenkins-agent-maven-36-rhel7</image>\r\n          <privileged>false</privileged>\r\n          <alwaysPullImage>true</alwaysPullImage>\r\n          <workingDir>/tmp</workingDir>\r\n          <command></command>\r\n          <args>${computer.jnlpmac} ${computer.name}</args>\r\n          <ttyEnabled>false</ttyEnabled>\r\n          <resourceRequestCpu></resourceRequestCpu>\r\n          <resourceRequestMemory></resourceRequestMemory>\r\n          <resourceLimitCpu></resourceLimitCpu>\r\n          <resourceLimitMemory></resourceLimitMemory>\r\n          <envVars/>\r\n        </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>\r\n      </containers>\r\n      <envVars/>\r\n      <annotations/>\r\n      <imagePullSecrets/>\r\n      <nodeProperties/>\r\n    </org.csanchez.jenkins.plugins.kubernetes.PodTemplate>"
kind: ConfigMap
metadata:
  labels:
    role: jenkins-slave
  name: jenkins-slave

The next time Jenkins loads (e.g. when you delete the current pod!) this ConfigMap is picked up automatically, and Jenkins presents you with this glorious result:

Manage -> Nodes -> Configure Clouds -> Kubernetes -> Pod Templates:

Using the Jenkins Agent image in your Pipeline

As mentioned before, once the PodTemplate is loaded into Jenkins, it can be referenced just like any other ‘agent’ in your Pipeline.

stage('Build & Unit test') {
    agent {
        label 'java11-maven'
    }
    environment {
        VERSION = sh script: 'mvn help:evaluate -Dexpression=project.version -q -DforceStdout', returnStdout: true
    }
    steps {
        echo "Building & testing version ${VERSION}"
        sh "export JAVA_HOME=/etc/alternatives/java_sdk_11 && mvn verify"
    }
}

Conclusion

Extending the number of use cases for an embedded instance of Jenkins on your OpenShift namespace, is very valuable for teams to operate independently and create software the way the team wants to. With strictly adhering to the everything-as-code adage, by creating config for a fully hands-off provisioned Jenkins instance, it is just another tool for successful teams to be even more successful with the scarce resources they have.

The accidental monolith

It is one of those current buzzwords, next to agile, cloud and containers: microservices.
Hyped by Gartner and “mister Service Oriented Architecture” Thomas Erl, this new way of creating business value by building your application as efficiently as possible is to decimate all issues we have had with creating applications in the past decades.
As you would probably have guessed from my slightly sarcastic tone, I’m not quite sure it is the solution to all the wrongs as it is trivially easy to wind up with a very common application type: the monolith.

While working various jobs in the integration realm in both the commercial and public sectors, I’ve seen the tempting promise of the ability to create loosely coupled but (functionally) strongly cohesive enterprise applications being made by a variety of architectural principles, with the previous one being Service Oriented Architecture.

We are taking a little detour here by veering off track and roping in an enterprise architecture during a discussion of a software architecture. Please bear with me on this one, I’ll try to be brief here.
This section is skippable for those who whish to get to the point quickly. 😉

Going off track

History

Unfortunately, I’ve also seen these principles being trampled under the hooves of the day-to-day business or even the occasional vendor of Enterprise Software (â„¢) as they completely hijacked the principle and twisted it to fit their own goals.
(I’m looking at you here, Oracle)

SOA is an enterprise architecture instead of an application architecture like microservices, and is promising a lot of good things in the SOA Manifesto by prioritizing business value, strategic goals, extensibility and flexibility over short term gains, custom implementations and (local or early) optimization. To be fair: this is awesome! Getting the most out of your resources in terms of business value is never not a great thing to strive for and these principles in the SOA Manifesto are, in my opinion, excellent tools to actually do so.

However, these principles also prove to be really hard to adhere to as not focussing on short term gains also means that, for a very long time, you have nothing to show for all investments being made in this new architectural principle and new application. Often, this ‘long time’ is too long for an organisation due to either financial or political pressure (or both!) and they start cutting corners by focussing on getting things done sooner instead of getting them done correctly.

Also, the SOA Manifesto was written in 2009 and last updated somewhere in 2013 which means it is *really old* and generally considered to be outdated. Personally, I’m convinced the form (the manifesto itself) might be outdated, but the premise of the manifesto is still highly valuable and applies to many fields in architecture and software development. Especially the principles focussing on flexibility, business value and evolutionary refinement.

Current

As Service Oriented Architecture (among the tech visionaries) goes the same route as our fax machines have (the use of them has become increasingly scarce and is generally frowned upon), a new paradigm has popped up: microservices.
This time another grand master of software development has stepped up to do the marketing for this method: Martin Fowler.

The Microservices Architecture uses the concept of Bounded Context: divide an application into functionally complete and separable subsections which can have their own lifecycles.
Generally, microservices are associated with other new-fangled techniques like containerization, continuous deployment and the cloud. I feel this is not correct, as microservices should be regarded as technology independent as the concepts would apply not only to new tech but also to old tech. I could create a microservices based application using proven technology like j2ee applications on an application container (java on JBoss), C++ applications running on a variety of Windows machines or even server side Javascript (NodeJS) based applications hosted in the cloud. What matters in microservices is the Bounded Context.

Martin Fowler has the following to say about microservices:

The term “Microservice Architecture” has sprung up over the last few years to describe a particular way of designing software applications as suites of independently deployable services. While there is no precise definition of this architectural style, there are certain common characteristics around organization, business capability, automated deployment, intelligence in the endpoints, and decentralized control of languages and data.

In short:

  • There is no formal definition,
  • The application consists of independently deployable ‘services’,
  • There is decentralised control of languages and data.

This last part catches the essence of the Bounded Context, albeit rather cryptically. In a microservices architecture, the application is ‘split up’ into chunks, where each of the chunks has a distinctive and restricted set of responsibilities and data.
A simple example is an HR-system in a large enterprise: it owns and handles all data and processes of your employees. Nothing more, nothing less.

A Bounded Context is just that: a chunk of your application, which owns the data of a specific subject (personnel, documents, cases, invoices, inventory) and also provides all the services the organisation needs regarding this data (CRUD and probably a lot more). Please note these chunks are not created along technical boundaries but through functional ones. These Bounded Contexts routinely cross into all three tiers of a multitier architecture, and that is completely fine!
Furthermore such a chunk is completely independent from the rest of the application, has a clear and stable interface and even has the added benefit of being the right size, i.e. to not have need shared ownership across multiple teams.

By splitting up your application into independent pieces and connecting them through well documented and stable interfaces, they are easier to maintain as any changes to a single piece have no impact on other pieces of your application apart from the interactions through the interfaces. This should mean less regression issues and also less complexity.

Getting back on track: the problem

Splitting up your application into independent microservices is hard.
Let me rephrase and blockquote that:

Splitting up your application into independent microservices is really hard.

Microservices tend to either be too big (and become cumbersome, over-complicated and bloated) or too small, leading to a heap of problems nobody anticipated in advance.
There even are names for the latter of the issues: nanoservices or picoservices.

An example of an application in an ideal world:

This application consists of 8 well connected components and should not be hard to comprehend as the number of connections is still low.
By having your application split into (high) tens or even hundreds of pieces (instead of a more manageable amount like 5 to 8) it becomes neigh impossible to keep track of all the relations between the nanoservices. When you consider a nanoservice not only has a version of its API but also its very own version, you might end up with a spaghetti-like mesh of wires between the nanoservices, of which nobody has an overview. It needs no explanation that keeping track of this spaghetti will become increasingly difficult.


A mess mesh.

This also translates into difficulties if you want to release a new business function of the application. This functional change might impact up to 30% of your nanoservices, and suddenly you need to not only modify, but also test and release a large amount of “independent” artefacts at once. All the while making sure your change does not cause regressions somewhere else in the application. Due to the opaque nature of the spaghetti your application this might even be harder than it sounds.

A possible solution for this is to stop treating individual nanoservies as independent artefacts and to start bundling them. Either into functional shards (which might have been a larger microservice from the beginning) or even into a single artefact which gets promoted from DEV to your PROD eventually.

This sounds like a familiar thing, right? Well, that’s because it is!
By tying together the ‘independent’ microservices and releasing them all together instead of releasing the services separately, a monolith has been created.
I’m sure that was not the intended outcome, but here we are. You have just achieved the ultimate anti pattern in microservices: the accidental monolith.

The solution

If only it were that easy, right?
Alas, quickly scrolling down to the header of a blog post which states ‘solution’ will not save you from this pitfall or get you out of your predicament as there simply is no easy fix.

The true solution is getting the bounds of your Bounded Contexts right. Nothing more, and nothing less.

You should have a manageable amount of microservices, where each one is functionally ‘complete’ in its own subject, is independent enough, has a clearly defined interface and is responsible for its own data.
Furthermore, your organization should focus on achieving business value in the long run and not get distracted by merely achieving short term goals and taking shortcuts, or just harvesting low hanging fruit. Having mature software development teams, respecting (and actually having) standards and guidelines, adhering to a versioning scheme (e.g. Semantic Versioning), routinely eliminating and preventing technical debt and trying to continuously improve all that is possible will minimize the chances of an accidental monolith forming. Thereby maximizing the success of the development of the application.

Jenkins: Working with Credentials in your pipeline

Security is mandatory, and should always have been, so eventually, you’ll run into the requirement of having to pass credentials to a system in your pipelines.

It could be you trying to authenticate with your Binary Repository Manager (e.g. Nexus, Artifactory), your Source Code Management system (e.g. SVN, git, TFS) or an internal system like JIRA or a ChatOps-like application (Mattermost, Slack), but credentials are most likely necessary and mandatory.
Luckily, Jenkins knows the concept of a secret and calls it a credential. For more information on this topic, please see the documentation.

These credentials can also be used in your pipeline, and they are implemented as the helper function credentials() in the environment block.

This post is part of a Series in which I elaborate on my best practices for running Jenkins at scale which might benefit Agile teams and CI/CD efforts.

In this post, I’ll try to show the usage of the helper function in your Jenkins Pipeline, and how to write tests to validate that.
Read More »

Jenkins: Testing conditional logic for stages in your pipeline

Some steps in your development or release process can only be executed when the conditions are right. An example of this is that releases to Production can only be done from the production branch, or that deployments to Acceptance can only occur when they are approved by a specific user.
This is where conditions shine, and they are implemented in Jenkins Pipelines as a when block.

This post is part of a Series in which I elaborate on my best practices for running Jenkins at scale which might benefit Agile teams and CI/CD efforts.

Whenever there’s logic involved in your code, one has to be extra careful that the behaviour of this code is both correct and consistent over time. Tests help with both.
Read More »

Jenkins: Testing with post conditions in your pipeline

The behaviour of your pipeline can be much more complex than the simple success/failure flow shown in the previous blog post.
The Jenkins declarative pipeline syntax has a lot more ways to decide whether to execute a step or not, and this also requires a proper test scenario.

This post is part of a Series in which I elaborate on my best practices for running Jenkins at scale which might benefit Agile teams and CI/CD efforts.

This post will focus on the post section, which is very much like a regular stage but is only executed when specific requirements are met.

Read More »

Jenkins: Validating the behaviour of your pipeline

The recent posts on this blog about Jenkins have been preparing us for this: validation of the behaviour of your Jenkins Pipeline in a test, in order to be able to check what the impact is of any changes you make to this pipeline, before it lands on your live instance and possibly influences the teams that are working with it.

This post is part of a Series in which I elaborate on my best practices for running Jenkins at scale which might benefit Agile teams and CI/CD efforts.

So far, we’ve seen how to set up your project in your IDE, how to run a pipeline from your Shared Library, and how to write your first (simple) test for this pipeline.

Read More »

Jenkins: Testing a full declarative pipeline in your Shared Library

In the previous posts, I’ve shown how to set up your Jenkins Shared Library, create Custom Pipeline Steps in it, set up the test frameworks, run complete pipelines from your Library and write tests for your Custom Pipeline Steps. Now the time has come to test the full declarative pipelines!

This post is part of a Series in which I elaborate on my best practices for running Jenkins at scale which might benefit Agile teams and CI/CD efforts.

Remember why we are going through all this trouble? We want to have as little issues as possible during the development of functionality in pipelines and steps, and want to achieve this through:

  • Documenting the behaviour of pipelines/steps in tests
  • Validating the current implementation of pipelines/steps
  • Removing the need for a ‘live’ environment through mocking
  • Enabling the possibility for debugging the code in an IDE

Read More »

Jenkins: Preparing your Shared Library for tests

After explaining how to set up your Shared Library, how to build it and how to run complete pipelines from this Library, it is time to create tests for the behaviour of the Library itself.

This post is part of a Series in which I elaborate on my best practices for running Jenkins at scale which might benefit Agile teams and CI/CD efforts.

As I’ve detailed before, being able to test the behaviour of your Library is instrumental in running Jenkins at scale. You just do not want to have to ‘test’ any changes in your pipelines or underlying code on a live production system. Ever.
The code which is being developed by the team(s) is usually held to very high standards by implementing a strict development process with multiple checks and quality gates: why should the code for your development process be treated differently?

Read More »

Jenkins: Running a declarative pipeline from your Shared Library

As shown in the previous blog post, Jenkins enables you to write your own pipeline code, which can be shared among all pipelines in your Jenkins instance.

This post is part of a Series in which I elaborate on my best practices for running Jenkins at scale which might benefit Agile teams and CI/CD efforts.

Where we’ve focussed on custom steps previously, I’ll now demonstrate how to create and use full declarative pipelines from your Shared Library.
This way you can minimize the amount of duplicated code in your projects by getting nearly all pipeline configuration from a central location.

Read More »