Blockchain technologies have been made famous by Cryptocurrencies such as Bitcoin and Ethereum. However, the concepts behind Blockchain are far more reaching than their support for cryptocurrency. Blockchain technologies now support any digital asset, from signal data to complex messaging, to the execution of business logic through code. Blockchain technologies are rapidly forming a new decentralized internet of transactions.
Advanced Platform Development with Kubernetes
What You'll Learn
- Build data pipelines with MQTT, NiFi, Logstash, MinIO, Hive, Presto, Kafka and Elasticsearch
- Leverage Serverless ETL with OpenFaaS
- Explore Blockchain networking with Ethereum
- Support a multi-tenant Data Science platform with JupyterHub, MLflow and Seldon Core
- Build a Multi-cloud, Hybrid cluster, securely bridging on-premise and cloud-based Kubernetes nodes
Kubernetes is an efficient and productive platform for the configuration, deployment, and management of private blockchains. Blockchain technology is intended to provide a decentralized transaction ledger, making it a perfect fit for the distributed nature of Kubernetes Pod deployments. The Kubernetes network infrastructure provides the necessary elements for security, scalability and fault tolerance needed for private or protected Blockchains.
Ethereum is a Cryptocurrency as well as a platform. Ethereum “is a decentralized platform that runs smart contracts: applications that run exactly as programmed without any possibility of downtime, censorship, fraud or third-party interference.”
Setting up Ethereum on Kubernetes
In my opinion, the best way to learn and understand the capabilities of this technology is by implementing your own private production ready Blockchain. Beginner tutorials often get you up and running on a local workstation leaving out important production implementation details; other quick start guides use package managers like Helm that do all the work for you. However, both these approaches leave gaps, either in understanding how to migrate and scale into a production system in the case of local development or the case of package managers hiding implementation details and limiting configuration options.
The following is an idiomatic Kubernetes setup. Providing and describing each configuration step in leveraging the Kubernetes core concept of declarative state configuration.
The transaction and miner nodes run the official ethereum/client-go:release-1.8 Docker containers.
Advanced Platform Development with Kubernetes
What You'll Learn
- Build data pipelines with MQTT, NiFi, Logstash, MinIO, Hive, Presto, Kafka and Elasticsearch
- Leverage Serverless ETL with OpenFaaS
- Explore Blockchain networking with Ethereum
- Support a multi-tenant Data Science platform with JupyterHub, MLflow and Seldon Core
- Build a Multi-cloud, Hybrid cluster, securely bridging on-premise and cloud-based Kubernetes nodes
Bare Metal / Custom Kubernetes
This guide builds a private Ethereum Blockchain on a custom Kubernetes cluster and avoids vendor lock-in or vendor specific instructions. If you don’t already have a cluster, I recommend setting up a production ready Kubernetes hobby cluster.
Resource Organization / Namespaces
Namespaces in Kubernetes allow you to separate services or projects into logical groups. In this guide, I use the fictional namespace the-project. Applications within the namespace can easily find each other by their Service name. Networking within a Kubernetes Pod uses internal DNS to locate a service by its name (or name.namespace).
RBAC or Role Based Access Control for a Namespaces offers a suitable solution to securing the configuration and operation of private blockchain to a group of external users or systems. See Kubernetes Team Access - RBAC for developers and QA for details on Namespace-based security.
Private Blockchain Network Topology
The articles Using Helm to Deploy Blockchain to Kubernetes and Building a Private Ethereum Consortium on Microsoft’s Developer Blog offer some great illustrations of this private Blockchain setup, following implementation is a deconstruction of the referenced Helm Chart.
Bootnode
A bootnode is a “Stripped down version of our Ethereum client implementation that only takes part in the network node discovery protocol, but does not run any of the higher level application protocols. It can be used as a lightweight bootstrap node to aid in finding peers in private networks.”
Bootnode Service
The Bootnode Service provides the endpoints eth-bootnode:30301 and eth-bootnode:8080 to Pod with the selector eth-bootnode defined further down in the Bootnode Deployment.
Create the file 110-bootnode-service.yml:
apiVersion: v1
kind: Service
metadata:
name: eth-bootnode
namespace: the-project
labels:
app: eth-bootnode
spec:
selector:
app: eth-bootnode
clusterIP: None
ports:
- name: discovery
port: 30301
protocol: UDP
- name: http
port: 8080
Apply the configuration:
kubectl create -f ./110-bootnode-service.yml
Bootnode Deployment
Bootnode Pods use the official ethereum/client-go Docker container, in this case, version alltools-release-1.8. Bootnode Pods start with an initialization container (initContainers section) named genkey. The genkey init container run the command bootnode --genkey=/etc/bootnode/node.key.
As can be seen the bootnode asks for a key. Each ethereum node, including a bootnode, is identified by an enode identifier. These identifiers are derived from a key. Therefore you will need to give the bootnode such key. Since we currently don’t have one, we can instruct the bootnode to generate a key (and store it in a file) before it starts.
Once the genkey init container has completed the Bootnode Pod run two containers, bootnode on port 30301/UDP is the running bootnode and bootnode-server is a simple command-line web server that responds on port 8080/TCP with the Ethereum node address.
The following is an example response from the bootnode-server exposed by the service added above (ellipses added for brevity):
curl eth-bootnode:8080
enode://[email protected]:30301
Create the file 120-bootnode-deployment.yml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: eth-bootnode
namespace: the-project
labels:
app: eth-bootnode
spec:
replicas: 2
revisionHistoryLimit: 1
selector:
matchLabels:
app: eth-bootnode
template:
metadata:
labels:
app: eth-bootnode
spec:
containers:
- name: bootnode
image: ethereum/client-go:alltools-release-1.8
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: ".5"
requests:
cpu: "0.25"
command: ["/bin/sh"]
args:
- "-c"
- "bootnode --nodekey=/etc/bootnode/node.key --verbosity=4"
volumeMounts:
- name: data
mountPath: /etc/bootnode
ports:
- name: discovery
containerPort: 30301
protocol: UDP
- name: bootnode-server
image: ethereum/client-go:alltools-release-1.8
imagePullPolicy: IfNotPresent
command: ["/bin/sh"]
args:
- "-c"
- "while [ 1 ]; do echo -e \"HTTP/1.1 200 OK\n\nenode://$(bootnode -writeaddress --nodekey=/etc/bootnode/node.key)@$(POD_IP):30301\" | nc -l -v -p 8080 || break; done;"
volumeMounts:
- name: data
mountPath: /etc/bootnode
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
ports:
- containerPort: 8080
initContainers:
- name: genkey
image: ethereum/client-go:alltools-release-1.8
imagePullPolicy: IfNotPresent
command: ["/bin/sh"]
args:
- "-c"
- "bootnode --genkey=/etc/bootnode/node.key"
volumeMounts:
- name: data
mountPath: /etc/bootnode
volumes:
- name: data
emptyDir: {}
Apply the configuration:
kubectl create -f ./120-bootnode-deployment.yml
Bootnode Registrar
bootnode-registrar is a “Registrar for Geth Bootnodes. bootnode-registrar resolves a DNS address record to a enode addresses that can then be consumed by geth –bootnodes=
Later on in this configuration the miner and tx Pods initialize by calling the eth-bootnode-registrar:80 service to receive a comma-separated list of bootnodes as Ethereum node address.
An example response from this services (ellipses added for brevity):
enode://[email protected]:30301,enode://[email protected]:30301
Bootnode Registrar Service
Create the file 130-bootnode-registrar-service.yml:
apiVersion: v1
kind: Service
metadata:
name: eth-bootnode-registrar
namespace: the-project
labels:
app: eth-bootnode-registrar
spec:
selector:
app: eth-bootnode-registrar
type: ClusterIP
ports:
- port: 80
targetPort: 9898
Apply the configuration:
kubectl create -f ./130-bootnode-registrar-service.yml
Bootnode Registrar Deployment
The Bootnode Registrar Deployment uses the bootnode-registrar container. bootnode-registrar is open source and you can quickly build a custom container with the provided Dockerfile. In this configuration uses the pre-built container provided.
If you are using a custom namespace be sure to change the BOOTNODE_SERVICE environment variable to the namespace you are using: “eth-bootnode.the-project.svc.cluster.local”.
The bootnode-registrar Pod runs a single Docker container named bootnode-registrar, running the jpoon/bootnode-registrar:v1.0.0 image from Docker hub.
Create the file 140-bootnode-registrar-deployment.yml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: eth-bootnode-registrar
namespace: the-project
labels:
app: eth-bootnode-registrar
spec:
replicas: 1
revisionHistoryLimit: 1
selector:
matchLabels:
app: eth-bootnode-registrar
template:
metadata:
labels:
app: eth-bootnode-registrar
spec:
containers:
- name: bootnode-registrar
image: jpoon/bootnode-registrar:v1.0.0
imagePullPolicy: IfNotPresent
env:
- name: BOOTNODE_SERVICE
value: "eth-bootnode.the-project.svc.cluster.local"
ports:
- containerPort: 9898
Apply the configuration:
kubectl create -f ./140-bootnode-registrar-deployment.yml
Ethstats Dashboard
The eth-netstats project provides an incredible dashboard interface for monitoring Ethereum nodes. eth-netstats consumes stats provided by the Ethereum geth nodes. Geth is the command line interface for running a full Ethereum node implemented in Go and deployed as miner and tx
Read the article Ethereum Ethstats - Learning the Ethereum Blockchain through its metrics if you are new to Blockchain or Ethereum terminology. In the article, I review essential metrics and their meaning.
Ethstats Dashboard Service
Create the file 210-ethstats-service.yml:
apiVersion: v1
kind: Service
metadata:
name: eth-ethstats
namespace: the-project
labels:
app: eth-ethstats
spec:
selector:
app: eth-ethstats
type: ClusterIP
ports:
- port: 80
targetPort: http
Apply the configuration:
kubectl create -f ./210-ethstats-service.yml
Ethstats Dashboard Secret
The geth nodes pass their Hostname and a shared password to Ethstats. A Kubernetes Secret stores the password as base64 encoded string.
You can create a password on MacOs or Linux terminals by piping a string to the command base64, or you can use the site base64decode.org to create a base64 encoded string.
Create Ethstats API Password:
echo -n 'changeme' | base64
Y2hhbmdlbWU=
The following Secret eth-ethstats is mounted by Ethstats and the geth node Pods further down this guide.
Create the file 220-ethstats-secret.yml:
apiVersion: v1
kind: Secret
metadata:
name: eth-ethstats
namespace: the-project
labels:
app: eth-ethstats
type: Opaque
data:
# "changeme"
WS_SECRET: "Y2hhbmdlbWU="
Apply the configuration:
kubectl create -f ./220-ethstats-secret.yml
Ethstats Dashboard Deployment
The Ethstats dashboard Deployment uses the Docker ethereumex/eth-stats-dashboard container from Ethereum Expertise. The container uses the environment variable WS_SECRET to set the password the Ethereum nodes use in calling to report their stats. The eth-ethstats Secret created above defines the password.
Create the file 230-ethstats-deployment.yml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: eth-ethstats
namespace: the-project
labels:
app: eth-ethstats
spec:
replicas: 1
revisionHistoryLimit: 1
selector:
matchLabels:
app: eth-ethstats
template:
metadata:
labels:
app: eth-ethstats
spec:
containers:
- name: ethstats
image: ethereumex/eth-stats-dashboard:latest
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 3000
env:
- name: WS_SECRET
valueFrom:
secretKeyRef:
name: eth-ethstats
key: WS_SECRET
Apply the configuration:
kubectl create -f ./230-ethstats-deployment.yml
ethereumex/eth-stats-dashboard container
Ethstats Dashboard Ingress
To access the Ethstats Dashboard from outside of the cluster, you need to have Ingress configured on your cluster. If you do not have an Ingress controller installed, I suggest reading Ingress on Custom Kubernetes for a quick guide on setting up ingress-nginx.
The example below uses the secret imti-dev-production-tls to provide an SSL cert. Let’s Encrypt is a free, automated, and open Certificate Authority and a simple way to get up and running on a custom Kubernetes cluster. I suggest reading Let’s Encrypt, Kubernetes to get up and running quickly with SSL.
Lastly, the following Ingress configuration uses annotations to specify a Basic Auth password from the secret imti-basic-auth, if you are new to Basic Auth on Kubernetes Ingress I suggest reading Basic Auth on Kubernetes Ingress, which gives clear instructions for generating a secret that can be used with ingress-nginx.
Create the file 240-ethstats-ingress.yml:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: eth-ethstats
namespace: the-project
labels:
app: eth-ethstats
annotations:
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: imti-basic-auth
nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
spec:
rules:
- host: ethstats.dcp.dev.d4l.cloud
http:
paths:
- backend:
serviceName: eth-ethstats
servicePort: 80
path: /
tls:
- hosts:
- ethstats.eth.imti.co
secretName: imti-dev-production-tls
Apply the configuration:
kubectl create -f ./240-ethstats-ingress.yml
Geth Configuration
Geth is the command line interface for running a full ethereum node implemented in Go. Geth is used to mine Eth, transfer funds between addresses, create contracts and send transactions, explore block history and many other operations.
The following configurations use Geth to initialize the new private blockchain, configure miners and operate transaction only nodes.
Ethereum Genesis Block
Read Explaining the Genesis Block in Ethereum for a better understanding of the Ethereum Genesis Block.
Geth nodes initialize themselves and start the new private blockchain with the first block (Ethereum Genesis Block) defined in the key genesis.json.
You should customize the value networkid in the ConfigMap and the “chainId” in the genesis.json. Ethereum nodes must have the same genesis block and networkid to join a network. Large networkids typically indicate private networks.
Geth ConfigMap
The Geth miner and transaction pods mount the Geth ConfigMap created here.
Along with customizing the networkid/chainId, you may also add Ethereum accounts to fund when the Genesis Block gets created. These can be any addresses you like. Use MetaMask or MyEtherWallet for an easy way to manage accounts or install Geth on your local workstation and type geth account new to create some new accounts.
This private network has a limited number of miners running with restricted CPU access, so we want to keep the difficulty low.
Read the official documentation on private networks configuration for more details on choosing a network id and creating the Genesis Block.
Create the file 310-geth-configmap.yml:
apiVersion: v1
kind: ConfigMap
metadata:
name: eth-geth
namespace: the-project
labels:
app: eth-geth
data:
networkid: "8189450821"
genesis.json: |-
{
"config": {
"chainId": 8189450821,
"homesteadBlock": 0,
"eip150Block": 0,
"eip155Block": 0,
"eip158Block": 0
},
"difficulty": "0x400",
"gasLimit": "2000000",
"nonce" : "0x0000000000000000",
"alloc": {
"0x58917D55dA991da576F148FD7E3E05a34666988b": { "balance": "100000000000000000000" },
"0x29bb385cF8ae4Cc49dBd10CcdA5e3d591D831527": { "balance": "200000000000000000000" },
"0xf1c9C9a1Ba591588147c2A729c470D8AFA91a04d": { "balance": "300000000000000000000" }
}
}
Apply the configuration:
kubectl create -f ./310-geth-configmap.yml
Geth Miner Nodes
The following configuration deploys three Geth miner nodes that share a Secret used for creating their coinbase accounts.
Geth Miner Secret
A Kubernetes Secret is used to store a common password each miner uses for creating an Ethereum account funded by block creation rewards; this is called the coinbase. This password is used to unlock the associated Ethereum account to transfer Eth gained from the minder nodes to other accounts. However, the coinbase may be configured for each miner can at any time.
You can create a password on MacOs or Linux terminals by piping a string to the command base64, or you can use the site base64decode.org to create a base64 encoded string.
Create Ethstats API Password:
echo -n 'password' | base64
cGFzc3dvcmQ=
Create the file 320-geth-miner-secret.yml:
apiVersion: v1
kind: Secret
metadata:
name: eth-geth-miner
namespace: the-project
labels:
app: eth-geth-miner
type: Opaque
data:
# echo -n 'password' | base64
accountsecret: "cGFzc3dvcmQ="
Apply the configuration:
kubectl create -f ./320-geth-miner-secret.yml
Geth Miner Deployment
This configuration starts with three miner Pods but can easily be scaled. Pods consist of one main container named geth-miner and two initialization containers named init-genesis and eth-geth-miner.
The pods request a minimum CPU allotment of 0.25 cores and maximum of 1.
Create the file 330-geth-miner-deployment.yml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: eth-geth-miner
namespace: the-project
labels:
app: eth-geth-miner
spec:
replicas: 3
revisionHistoryLimit: 1
selector:
matchLabels:
app: eth-geth-miner
template:
metadata:
labels:
app: eth-geth-miner
spec:
containers:
- name: geth-miner
image: ethereum/client-go:release-1.8
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: ".5"
requests:
cpu: "0.25"
command: ["/bin/sh"]
args:
- "-c"
- "geth --bootnodes=`cat /root/.ethereum/bootnodes` --mine --etherbase=0 --networkid=${NETWORK_ID} --ethstats=${HOSTNAME}:${ETHSTATS_SECRET}@${ETHSTATS_SVC} --verbosity=5"
env:
- name: ETHSTATS_SVC
value: eth-ethstats.dcp
- name: ETHSTATS_SECRET
valueFrom:
secretKeyRef:
name: eth-ethstats
key: WS_SECRET
- name: NETWORK_ID
valueFrom:
configMapKeyRef:
name: eth-geth
key: networkid
ports:
- name: discovery-udp
containerPort: 30303
protocol: UDP
- name: discovery-tcp
containerPort: 30303
volumeMounts:
- name: data
mountPath: /root/.ethereum
initContainers:
- name: init-genesis
image: ethereum/client-go:release-1.8
imagePullPolicy: IfNotPresent
args:
- "init"
- "/var/geth/genesis.json"
volumeMounts:
- name: data
mountPath: /root/.ethereum
- name: config
mountPath: /var/geth
- name: create-account
image: ethereum/client-go:release-1.8
imagePullPolicy: IfNotPresent
command: ["/bin/sh"]
args:
- "-c"
- "printf '$(ACCOUNT_SECRET)\n$(ACCOUNT_SECRET)\n' | geth account new"
env:
- name: ACCOUNT_SECRET
valueFrom:
secretKeyRef:
name: eth-geth-miner
key: accountsecret
volumeMounts:
- name: data
mountPath: /root/.ethereum
- name: get-bootnodes
image: ethereum/client-go:release-1.8
imagePullPolicy: IfNotPresent
command: ["/bin/sh"]
args:
- "-c"
- |-
apk add --no-cache curl;
CNT=0;
echo "retreiving bootnodes from $BOOTNODE_REGISTRAR_SVC"
while [ $CNT -le 90 ]
do
curl -m 5 -s $BOOTNODE_REGISTRAR_SVC | xargs echo -n >> /geth/bootnodes;
if [ -s /geth/bootnodes ]
then
cat /geth/bootnodes;
exit 0;
fi;
echo "no bootnodes found. retrying $CNT...";
sleep 2 || break;
CNT=$((CNT+1));
done;
echo "WARNING. unable to find bootnodes. continuing but geth may not be able to find any peers.";
exit 0;
env:
- name: BOOTNODE_REGISTRAR_SVC
value: eth-bootnode-registrar
volumeMounts:
- name: data
mountPath: /geth
volumes:
- name: data
emptyDir: {}
- name: config
configMap:
name: eth-geth
Apply the configuration:
kubectl create -f ./330-geth-miner-deployment.yml
Geth Transaction Nodes
Geth Transaction Nodes eth-geth-tx are only used to create transactions on the private blockchain.
Geth Transaction Nodes Service
Access to Geth Transaction Nodes is through a Service named eth-geth-tx on ports 8545 and 8546.
Create the file 410-geth-tx-service.yml:
apiVersion: v1
kind: Service
metadata:
name: eth-geth-tx
namespace: the-project
labels:
app: eth-geth-tx
spec:
selector:
app: eth-geth-tx
type: ClusterIP
ports:
- name: rpc
port: 8545
- name: ws
port: 8546
Apply the configuration:
kubectl create -f ./410-geth-tx-service.yml
Geth Transaction Nodes Deployment
Issuing transactions on the private Blockchain occur through a set of two Geth Transaction Nodes; these Pods are setup similar to miners running the official ethereum/client-go:release-1.8. Geth Transaction Node consist of one main container named geth-tx and two initialization containers init-genesis and get-bootnodes.
Create the file 420-geth-tx-deployment.yml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: eth-geth-tx
namespace: the-project
labels:
app: eth-geth-tx
spec:
replicas: 2
selector:
matchLabels:
app: eth-geth-tx
template:
metadata:
labels:
app: eth-geth-tx
spec:
containers:
- name: geth-tx
image: ethereum/client-go:release-1.8
imagePullPolicy: IfNotPresent
command: ["/bin/sh"]
args:
- "-c"
- "geth --bootnodes=`cat /root/.ethereum/bootnodes` --rpc --rpcapi=eth,net,web3 --rpccorsdomain='*' --ws --networkid=${NETWORK_ID} --ethstats=${HOSTNAME}:${ETHSTATS_SECRET}@${ETHSTATS_SVC} --verbosity=5"
env:
- name: ETHSTATS_SVC
value: eth-ethstats.dcp
- name: ETHSTATS_SECRET
valueFrom:
secretKeyRef:
name: eth-ethstats
key: WS_SECRET
- name: NETWORK_ID
valueFrom:
configMapKeyRef:
name: eth-geth
key: networkid
ports:
- name: rpc
containerPort: 8545
- name: ws
containerPort: 8546
- name: discovery-udp
containerPort: 30303
protocol: UDP
- name: discovery-tcp
containerPort: 30303
volumeMounts:
- name: data
mountPath: /root/.ethereum
initContainers:
- name: init-genesis
image: ethereum/client-go:release-1.8
imagePullPolicy: IfNotPresent
args:
- "init"
- "/var/geth/genesis.json"
volumeMounts:
- name: data
mountPath: /root/.ethereum
- name: config
mountPath: /var/geth
- name: get-bootnodes
image: ethereum/client-go:release-1.8
imagePullPolicy: IfNotPresent
command: ["/bin/sh"]
args:
- "-c"
- |-
apk add --no-cache curl;
CNT=0;
echo "retreiving bootnodes from $BOOTNODE_REGISTRAR_SVC"
while [ $CNT -le 90 ]
do
curl -m 5 -s $BOOTNODE_REGISTRAR_SVC | xargs echo -n >> /geth/bootnodes;
if [ -s /geth/bootnodes ]
then
cat /geth/bootnodes;
exit 0;
fi;
echo "no bootnodes found. retrying $CNT...";
sleep 2 || break;
CNT=$((CNT+1));
done;
echo "WARNING. unable to find bootnodes. continuing but geth may not be able to find any peers.";
exit 0;
env:
- name: BOOTNODE_REGISTRAR_SVC
value: eth-bootnode-registrar.dcp
volumeMounts:
- name: data
mountPath: /geth
volumes:
- name: data
emptyDir: {}
- name: config
configMap:
name: eth-geth
Apply the configuration:
kubectl create -f ./420-geth-tx-deployment.yml
Conclusion
Depending on your difficulty and resources it may take 10-20 minutes to mine your first block in this private chain. However, the difficulty is automatically adjusted, and you may begin to mine blocks at around 20-30 seconds on average.
Port Forwarding / Local Development
Check out kubefwd for a simple command line utility that bulk forwards services of one or more namespaces to your local workstation.
Additional Resources
- go-ethereum Private-network Github Wiki
- MetaMask Ethereum account management
This blog post, titled: "Ethereum Blockchain on Kubernetes: Deploy a Private Ethereum Blockchain on a Custom Kubernetes Cluster." by Craig Johnston, is licensed under a Creative Commons Attribution 4.0 International License.
SUPPORT
Order my new Kubernetes book: Advanced Platform Development with Kubernetes: Enabling Data Management, the Internet of Things, Blockchain, and Machine Learning