I’ve been distracted for over a year now, writing a (~500 page) end-to-end tutorial on constructing data-centric platforms with Kubernetes. The book is titled “Advanced Platform Development with Kubernetes: Enabling Data Management, the Internet of Things, Blockchain, and Machine Learning”
A little more than a year ago, Apress reached out and asked if I would write a book on Kubernetes for them, mirroring the wide range of projects I develop (and write about) for my clients. I have been building data-centric platforms for almost twenty years, spanning everything from my early days on the aggregation of massive volumes of international log files for Disney to fan-driven location data for Nine Inch Nails. And in the last decade, retailers with point-of-sale, logistics, and inventory systems, marketers leveraging social media metrics, fleet operators with demanding telematics platforms, and manufacturers with advanced IIoT (industrial internet of things) networks.
Advanced Platform Development with Kubernetes
What You'll Learn
- Build data pipelines with MQTT, NiFi, Logstash, MinIO, Hive, Presto, Kafka and Elasticsearch
- Leverage Serverless ETL with OpenFaaS
- Explore Blockchain networking with Ethereum
- Support a multi-tenant Data Science platform with JupyterHub, MLflow and Seldon Core
- Build a Multi-cloud, Hybrid cluster, securely bridging on-premise and cloud-based Kubernetes nodes
Furthermore, the clients behind these verticals have begun looking for an edge, often found beyond standard PaaS offerings available today. Yet these clients from established organizations and age-old industries have little tolerance for the risk associated with a multi-year development on technologies whose value is speculative. Will blockchain revolutionize logistics and finance? Will machine learning produce artificial intelligence capable of replacing legacy decisions making systems? The safe bet is to wait. If there is gold in any of these mountains the hyper clouds will find it, and they will happily sell it to everyone, by the hour, megabyte, or IOPS.
Not every organization should be developing custom software, let alone advanced data platforms. However, the barrier to entry is lowering every day. I have been building platforms for twenty years, and never has a single technology increased productivity in this practice more than Kubernetes has in the last five. Kubernetes is truly a platform for building platforms, capable of harnessing the wide breadth of new technologies released into the open-source landscape almost daily.
On October 28, 2018, IBM announced a $34 billion deal to buy Red Hat, the company behind Red Hat Enterprise Linux (RHEL) and, more recently, Red Hat OpenShift, an enterprise Kubernetes-based application platform. What we see is $34 billion of evidence that Cloud-native and open source technologies centered on the Linux ecosystem and empowered by Kubernetes is leading disruption in enterprise software application development. IBM sells platforms, and yet it looks to capitalize on the value of a system tailored to platform development. IBM is speculating that the next trend in PaaS offerings is itself a platform for developing platforms. An organization does not spend $34 billion to on-board brand and technology without a market. Who is demanding the capabilities of OpenShift/Kubernetes? Systems, solutions, and software architects, full-stack developers, programmers, integrators, and DevOps engineers. These are the people and roles responsible for the technology that powers the business logic.
I did not want to write a book on administering Kubernetes or how it works (although you’ll likely learn this along the way). There are a ton of excellent books and tutorials on the matter. I wanted to write about what gets the lion’s share of hits on my blog: “How to run X on Kubernetes.” I don’t know why you want to run a Blockchain network on Kubernetes; I have my reasons for doing so. I don’t know why you need to interconnect Kafka, NiFi, MinIO, Hive, Keycloak, Cassandra, MySQL, Zookeeper, Mosquitto, Elasticsearch, Logstash, Kibana, Presto, OpenFaaS, Ethereum, Jupyter, MLflow, and Seldon Core. You don’t need Kubernetes to run these excellent applications. You don’t need Kubernetes to run containers networked across the globe on various cloud providers communicating over secure VPNs, on-premises, on a mix of bare metal servers and virtual machines. I know that I, and the many roles mentioned above want to, (evidenced by Kubernetes enormous success) and it’s likely because they are building sophisticated and modern platforms and see the value Kubernetes brings to this endeavor.
“Advanced Platform Development with Kubernetes: Enabling Data Management, the Internet of Things, Blockchain, and Machine Learning” is a 500 page tutorial on the work I do daily. The examples are scaled-down yet real and fully functional. If you are an entrepreneur with dreams of building your own AWS or Azure or if constructing enterprise-capable data-centric platforms is what you do for work or hobby, my book is looking to inspire you and give you traction where I have found it myself.
Weekend Projects
All source code and configuration manifests are open source and available online at: https://github.com/apk8s/book-source
“Advanced Platform Development with Kubernetes: Enabling Data Management, the Internet of Things, Blockchain, and Machine Learning” will give you an equivalent of ten weekend projects, covering all the technology mentioned above and more, broken down as follows:
- Week 1: DevOps Infrastructure
- Week 2: Development Environment
- Week 3: In-Platform CI/CD
- Week 4: Pipeline
- Week 5: Indexing and Analytics
- Week 6: Data Lakes
- Week 7: Data Warehouses
- Week 8: Routing and Transformation
- Week 9: Platforming Blockchain
- Week 10: Platforming AIML
Custom Kubernetes
Quite a few books and how-tos specialize in AKS, EKS, and GKE. “Advanced Platform Development with Kubernetes: Enabling Data Management, the Internet of Things, Blockchain, and Machine Learning” focuses on building Custom Kubernetes clusters and illustrates this by using generic (and cheap) compute instances (VMs) offered by Vultr, Digital Ocean, Linode, Hetzner and Scaleway.
Technology
“Advanced Platform Development with Kubernetes: Enabling Data Management, the Internet of Things, Blockchain, and Machine Learning” covers the following technology:
- Chapter 2: DevOps Infrastructure (source)
- Chapter 3: Development Environment (source)
- Chapter 4: In-Platform CI/CD (source)
- Chapter 5: Pipeline (source)
- Chapter 6: Indexing and Analytics (source)
- Chapter 7: Data Lakes (source)
- Chapter 8: Data Warehouses (source)
- Chapter 9: Routing and Transformation (source)
- Chapter 10: Platforming Blockchain (source)
- Chapter 11: Platforming AIML (source)
- Kilo (Automated WireGuard)
- Nvidia / CUDA
- Bash / Shell Scripting
- Raspberry Pi
- CronJob with Python
- MLflow
- Seldon Core
Advanced Platform Development with Kubernetes
What You'll Learn
- Build data pipelines with MQTT, NiFi, Logstash, MinIO, Hive, Presto, Kafka and Elasticsearch
- Leverage Serverless ETL with OpenFaaS
- Explore Blockchain networking with Ethereum
- Support a multi-tenant Data Science platform with JupyterHub, MLflow and Seldon Core
- Build a Multi-cloud, Hybrid cluster, securely bridging on-premise and cloud-based Kubernetes nodes
This blog post, titled: "Advanced Platform Development with Kubernetes: Enabling Data Management, the Internet of Things, Blockchain, and Machine Learning" by Craig Johnston, is licensed under a Creative Commons Attribution 4.0 International License.
SUPPORT
Order my new Kubernetes book: Advanced Platform Development with Kubernetes: Enabling Data Management, the Internet of Things, Blockchain, and Machine Learning