I’ve been distracted for over a year now, writing a (~500 page) end-to-end tutorial on constructing data-centric platforms with Kubernetes. The book is titled “Advanced Platform Development with Kubernetes: Enabling Data Management, the Internet of Things, Blockchain, and Machine Learning”

A little more than a year ago, Apress reached out and asked if I would write a book on Kubernetes for them, mirroring the wide range of projects I develop (and write about) for my clients. I have been building data-centric platforms for almost twenty years, spanning everything from my early days on the aggregation of massive volumes of international log files for Disney to fan-driven location data for Nine Inch Nails. And in the last decade, retailers with point-of-sale, logistics, and inventory systems, marketers leveraging social media metrics, fleet operators with demanding telematics platforms, and manufacturers with advanced IIoT (industrial internet of things) networks.

Kafka is a fast, horizontally scalable, fault-tolerant, message queue service. Kafka is used for building real-time data pipelines and streaming apps.

There are a few Helm based installers out there including the official Kubernetes incubator/kafka. However, in this article, I walk through applying a surprisingly small set of Kubernetes configuration files needed to stand up high performance, highly available Kafka. Manually applying Kubernetes configurations gives you a step-by-step understanding of the system you are deploying and limitless opportunities to customize.

Blockchain technologies have been made famous by Cryptocurrencies such as Bitcoin and Ethereum. However, the concepts behind Blockchain are far more reaching than their support for cryptocurrency. Blockchain technologies now support any digital asset, from signal data to complex messaging, to the execution of business logic through code. Blockchain technologies are rapidly forming a new decentralized internet of transactions.

Kubernetes is an efficient and productive platform for the configuration, deployment, and management of private blockchains. Blockchain technology is intended to provide a decentralized transaction ledger, making it a perfect fit for the distributed nature of Kubernetes Pod deployments. The Kubernetes network infrastructure provides the necessary elements for security, scalability and fault tolerance needed for private or protected Blockchains.

kubefwd helps to enable a seamless and efficient way to develop applications and services on a local workstation. Locally develop applications that intend to interact with other services in a Kubernetes cluster. kubefwd allows applications with connection strings like http://elasticsearch:9200/ or tcp://db:3306 to communicate into the remote cluster. kubefwd can be used to reduce or eliminate the need for local environment specific connection configurations.

Developing services in a Microservices architecture presents local development challenges, especially when the service you are developing needs to interact with a mixture of other services. Microservices, like any other applications are rarely ever self-contained and often need access to databases, authentication services, and other public or private APIs. Loosely-coupled applications still have couplings, they happen on a higher layer of the application stack and often through TCP networking.

FaaS or Function as a Service also known as Serverless computing implementations are gaining popularity. Discussed often are the cost savings and each implementations relationship to the physical and network architecture of a specific platform or vendor. While many of the cost and infrastructure advantages of FaaS are compelling, its only one of many advantages. Below, I hope to demonstrate how easy it is to develop and deploy FaaS components into a custom Kubernetes cluster. The functions I develop are nearly all business logic, and I believe therein lies the advantage, high-density business logic. Functions can have a higher degree of focus directly on business logic and communication with other services. Functions can communicate with other functions, microservices or monoliths. In this article, I demonstrate this with Elasticsearch.

In days before Kubernetes, writing code in IDEs on our local workstation meant we had only a few options for developing server-side-API-style services that needed to connect to a database. We could set up a database server on our local workstation manually or use packages like MAMP/WAMP, or run big virtual servers managed with Vagrant. Even after we got the database running, we needed a good set of data to work with, and that often meant asking a DBA or Sysadmin for SQL dumps from an environment in which we have no access.

Background

High availability and high performance often mean burdensome complexity. Data replication, application, network and infrastructure redundancy, anything we can do to avoid a single point of failure. However, what happens when one of those points do fail? Alternatively, a cascade of problems causing slowdowns and back pressure builds to constant far too high for indexing to catch up?

This guide walks through a process for setting up Kibana within a namespace on a Kubernetes cluster. If you followed along with Production Grade Elasticsearch on Kubernetes then aside from personal or corporate preferences, little modifications are necessary for the configurations below.

Project Namespace

I use the-project as a namespace for all my examples and testing. Kubernetes Namespaces are the main delimiter I use for security and organization. Configuration files are organized by project, and in Kubernetes, these projects are separated by namespace; therefore I always include a namespace configuration. There is no harm in asking Kubernetes to create a namespace that already exists; an error is returned confirming its existence.

Installing production ready, Elasticsearch 6.2 on Kubernetes requires a hand full of simple configurations. The following guide is a high-level overview of an installation process using Elastic’s recommendations for best practices. The Github project kubernetes-elasticsearch-cluster is used for the Elastic Docker container and built to operate Elasticsearch with nodes dedicated as Master, Data, and Client/Ingest.

The Docker container docker-elasticsearch, a “Ready to use, lean and highly configurable Elasticsearch container image.” by pires is sufficient for use in this guide. However, the txn2/k8s-es wraps it with a few minor preset environment variables to simplify configuration. I use the docker image txn2/k8s-es:v6.2.3 in the examples below.

RBAC (Role Based Access Control) allows our Kubernetes clusters to provide the development team better visibility and access into the development, staging and production environments than it has have ever had in the past. Developers using the command line tool kubectl, can explore the network topology of running microservices, tail live server logs, proxy local ports directly to services or even execute shells into running pods.

Kubernetes and GitlabCI are the central components of our DevOps toolchain and have increased our productivity by many multiples over the traditional approaches of the past.

Using ingress-nginx on Kubernetes makes adding CORS headers painless. Kubernetes ingress-nginx uses annotations as a quick way to allow you to specify the automatic generation of an extensive list of common nginx configuration options.

Example ingress configuration enabling CORS:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: api
  namespace: fuse
  labels:
    app: api
  annotations:
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://admin.example.com"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
spec:
  rules:
  - host: api.example.com
    http:
      paths:
      - backend:
          serviceName: api-example
          servicePort: 80
        path: /api
  tls:
  - hosts:
    - api.example.com
    secretName: example-tls

You can check the nginx configuration file generated by Kubernetes ingress-nginx on any of the ingress controller pods.

Basic Auth is one of the oldest and easiest ways to secure a web page or API endpoint. Basic Auth does not have many features and lacks the sophistication of more modern access controls (see Ingress Nginx Auth Examples). However, Basic Auth is supported by nearly every major web client, library, and utility. Basic Auth is secure, stable and perfect for quick security on Kubernetes projects. Basic Auth can easily we swapped out later as requirements demand or provide a foundation for implementations such as OAuth 2 and JWT.

txToken is a small high performance microservice utility container. txToken is used for adding JSON Web Token based security to existing or new API development. txToken is specifically for systems that communicate in JSON over HTTP. txToken is called from a client with a JSON post body and passes received JSON to a remote endpoint. JSON retrieved from a remote endpoint is used to create a JWT token with an HS256 symmetrically encrypted signature.

Overview

Use cert-manager to get port 443/https running with signed x509 certificates for Ingress on your Kubernetes Production Hobby Cluster. cert-manager is the successor to kube-lego and the preferred way to “automatically obtain browser-trusted certificates, without any human intervention.” using Let’s Encrypt.

You need to install Helm first if you do not already have it. Otherwise, check out my article Helm on Custom Kubernetes, especially if you are following along with my Production Hobby Cluster guides.

Install cert-manager

helm install --name cert-manager --namespace kube-system stable/cert-manager

After Helm installs cert-manager you end up with a ServiceAccount ClusterRole, ClusterRoleBinding, Deployment and a couple of Pods named cert-manager-cert-manager in the kube-system namespace. Helm additionaly installs three CustomResourceDefinitions for cert-manager (custom resources are not namespaced):

Helm is the de facto package manager for Kubernetes. If you are looking to start using Helm or want to test its capabilities, I suggest you set up a Production Hobby Cluster. This article is a continuation of the Production Hobby Cluster configuration but should be entirely useful on its own.

From https://github.com/kubernetes/helm

• Helm has two parts: a client (helm) and a server (tiller)
• Tiller runs inside of your Kubernetes cluster and manages releases (installations) of your charts.
• Helm runs on your laptop, CI/CD, or wherever you want it to run.
• Charts are Helm packages that contain at least two things:
• A description of the package (Chart.yaml)
• One or more templates, which contain Kubernetes manifest files
• Charts can be stored on disk, or fetched from remote chart repositories (like Debian or RedHat packages)

Start with installing Helm on your local workstation. Read the Helm installation instructions or merely follow along below.

Customize the Upstream Nameservers used by kube-dns by Pods when looking up external hostnames from within a Kubernetes cluster. I found that adding custom Upstream Nameservers to my kube-dns solved many issues encountered in in the past with external hostname resolution on individual Pods.

If you want to experiment on a production-like cluster, I suggest reading my article “Production Hobby Cluster” for a guide on setting up a fun, cheap-yet-robust experimental cluster.

The following configuration sets the Upstream Nameservers to use Google’s DNS servers 8.8.8.8 and 8.8.4.4.

There are more than a handful of ways to set up port 80 and 443 web ingress on a custom Kubernetes cluster. Specifically a bare metal cluster. If you are looking to experiment or learn on a non-production cluster, but something more true to production than minikube, I suggest you check out my previous article Production Hobby Cluster, a step-by-step guide for setting up a custom production capable Kubernetes cluster.

This article builds on the Production Hobby Cluster guide. The following closely the official deploy ingress Installation Guide with a few adjustments suitable for the Production Hobby Cluster, specifically the use of a DaemonSet rather than a Deployment and leveraging hostNetwork and hostPort for the Pods on our DaemonSet. There are quite a few ingress nginx examples in the official repository if you are looking for a more specific implementation.

I use a few Kubernetes clusters on a daily basis, and I use kubectl to access and configure them from my workstation. There are dozens of ways to configure kubectl however I find the following method the easiest for me to manage and not make a mess.

I also set up test clusters from time-to-time, and so keeping my configs organized is, so I don’t confuse myself or make a mess.

For this post, I’ll talk about four clusters and how I have them organized. The following is a list of the clusters I am managing:

When setting up nginx ingress on Kubernetes for a private Docker Registry, I ran into an error when trying to push an image to it.

Error parsing HTTP response: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.9.14</center>\r\n</body>\r\n</html>\r\n"

The “413 Request Entity Too Large” error is something many accustomed to running nginx as a standard web server/proxy. nginx is configured to restrict the size of files it will allow over a post. This restriction helps avoid some DoS attacks. The default values are easy to adjust in the Nginx configuration. However, in the Kubernetes world things are a bit different. I prefer to do things the Kubernetes way; however, there is still a lack of established configuration idioms, since part of its appeal is flexibility. One thousand different ways of doing something means ten thousand variations of remedies to every potential problem. Googling errors in the Kubernetes world leads to a mess of solutions not always explicitly tied to an implementation. I am hoping that as Kubernetes continues to gain popularity more care will be taken to provide context for solutions to common problems.

I use Minikube to run a local Kubernetes single node cluster (cluster?). However, I also work with a custom production cluster for work. This cluster consists of development and production nodes. I often need to switch between working on my local Minikube and the online Kubernetes cluster.

TIP: Visit the kubectl Cheat Sheet often.

The default configuration kubectl is stored in ~/.kube/config and if you have Minikube installed, it added the context minikube to your config.

The following is a collection of articles, videos, and notes on Microservices. The Microservices architecture is a variant of the service-oriented architecture (SOA), a collection of loosely coupled services.

Articles

• Background concept - “Open Data: Small Pieces Loosely Joined”, Tim O’Reilly
• Modern software design problems and solutions - “12-Fractured Apps”, Kelsey Hightower (SysAdmin @ Google)
• 12-Factor Defined - “The 12-Factor App”, Adam Wiggins
• Pros and Cons of Microservices - “Microservices” and Microservice Trade-Offs, Martin Fowler
• “What are containers and why do you need them?” - CIO
• “Containers bring a skinny new world of virtualization to Linux” - ITWorld

Videos

• “Microservices”, Martin Fowler
• “The Evolution of Microservices”, Martin Fowler
• “The State of the Art in Microservices” with Docker, Adrian Cockroft

Notes

Key Goals of Microservices

• Rapid development
• Continuous deployment

Best Practices

• Version Control - All code and configuration should be versioned.
• Log to Standard Out - This unifies the log collection process.
• Package Dependencies - Ensure the stability of the build process.

Twelve-Factor Principles

• Portable - Service (container) should be able to be run anywhere.
• Continually Deployable - Able to deploy any time without disruption.
• Scalable - Multiple copies should be able to run concurrently (stateless)

JSON Web Tokens (JWT) - Client -> Server trust.

• Compact, self-contained method for transferring secure data as a JSON object.
• Use for Authentication and Information Exchange
• See https://jwt.io/
• A server creates a token, and the client uses token to make requests.

Containers - Docker

• Docker is simply an API on top of existing process isolation technology.
• Independent packages
• Namespace Isolation

Alpine Linux

Alpine Linux Small. Simple. Secure. Alpine Linux is a security-oriented, lightweight Linux distribution based on musl libc and busybox.

• Deeper Reading and Resources
  • Free Courses
  • Paid Courses
• Prerequisites
• Test Installation
• Get some status
  • Architecture
• Create a Deployment
• Useful Commands

Deeper Reading and Resources

Free Courses

• Official Kubernetes Tutorials
• Udacity: Scalable Microservices with Kubernetes
• Kubernetes The Hard Way For anyone planning to support a production Kubernetes cluster and wants to understand how everything fits together.

Paid Courses

• Lynda: Learning Kubernetes
• Lynda: Kubernetes: Native Tools
• Udemy: Learn DevOps: The Complete Kubernetes Course
• Udemy: Learn DevOps: Advanced Kubernetes Usage

Prerequisites

• Install Docker
• Install Virtualbox
• Install Kubectl
• Install Minikube

Test Installation

$ minikube version
minikube version: v0.25.0

$ minikube start
Starting local Kubernetes v1.9.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.

$ minikube addons list
- addon-manager: enabled
- coredns: disabled
- dashboard: enabled
- default-storageclass: enabled
- efk: disabled
- freshpod: disabled
- heapster: disabled
- ingress: disabled
- kube-dns: enabled
- registry: disabled
- registry-creds: disabled
- storage-provisioner: enabled


# enable heapster for CPU and mem
$ minikube addons enable heapster
heapster was successfully enabled

# open the dashboard (in a browser)
$ minikube dashboard

Get some status

# are we running a cluster?
$ kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443

# we should have a minikube node
$ kubectl get nodes
NAME       STATUS    ROLES     AGE       VERSION
minikube   Ready     <none>    2d        v1.9.0

Architecture

Read Kubernetes Basics for a better understanding.