Professional notes by Craig Johnston
long-form, short-form, working drafts · since 2008
long-form, short-form, working drafts · since 2008
VOL. XIX · MMXXVI
82 NOTES IN PRINT
82 NOTES IN PRINT
Plates
Every note's diagram, all on one page. Atlas-style contact sheet of the imti.co plate library. Click any plate to open the note.
82 plates
flowchart TB
A([Agent edits code]) --> B{{git commit}}
B --> H[/PreToolUse hook/]
H --> C{Trivial diff?<br/>Doc-only?<br/>Plan mode?}
C -->|yes| OK([allow commit])
C -->|no| D{Artifact valid?<br/>hash matches?<br/>verdict CLEAN?}
D -->|yes| OK
D -->|no| DENY[/deny + return<br/>review prompt/]
DENY --> SUB([general-purpose<br/>sub-agent review])
SUB --> F{Findings?}
F -->|N findings| FIX([fix in working tree])
FIX --> SUB
F -->|CLEAN| ART[(write artifact:<br/>diff hash + verdict)]
ART --> B
The Pre-Commit Review Gate
flowchart LR
A1[Familiar tool bias] -.- M1
A2[SQL is the default hammer] -.- M1
A3[Routing hints feel optional] -.- M1
M1[Mode A: Path of Least Resistance] --> P
B1[Reads the tool description] -.- M2
B2[Repeats it back accurately] -.- M2
B3[Acts on the training-data prior anyway] -.- M2
M2[Mode B: Comprehension is not Compliance] --> P
P((BAD AGENT<br/>OUTPUT))
The Two Failure Modes That Break Your AI Data Agent
flowchart TB
CRON{{"CronJob · every 15 min"}}
CRON --> WIN
PG[("PostgreSQL 16<br/>240 GB · 300M rows")]
PG -.->|"JDBC date window"| WIN
subgraph PIPE[" PySpark · local mode "]
direction TB
WIN["Date-windowed read<br/><i>WHERE event_at BETWEEN ?</i>"]
KEY["SHA-2 deterministic _id<br/><i>idempotent on retry</i>"]
BULK["OpenSearch bulk action<br/><i>refresh=False</i>"]
WIN --> KEY --> BULK
end
BULK -->|"_id-keyed upsert"| OS[("OpenSearch 2.13<br/>indexed docs")]
PostgreSQL to OpenSearch with PySpark on Kubernetes
gitGraph
commit id: "init"
commit id: "core code"
branch release/sign
commit id: "goreleaser.yml"
commit id: "cosign keyless"
commit id: "syft sbom"
checkout main
merge release/sign
commit id: "v1.0.0" tag: "v1.0.0"
commit id: "fix darwin arm64"
commit id: "v1.0.1" tag: "v1.0.1"
GoReleaser with Cosign Signing and Syft SBOM
mindmap
root((AI on a Leash<br/>for Go))
Linting
golangci-lint
43 enabled linters
revive overrides
Testing
table-driven
race detector
mutation: gremlins
Coverage
codecov.yml
per-package thresholds
Security
gosec
govulncheck
Release
GoReleaser
cosign + SBOM
Convention
GNUmakefile
CLAUDE.md
go-leash skill
AI on a Leash: Complete Go Project Configuration
quadrantChart
title Where AI Coding Approaches Sit
x-axis Low AI Autonomy --> High AI Autonomy
y-axis Low Verification Rigor --> High Verification Rigor
quadrant-1 AI on a Leash
quadrant-2 Hand-Crafted Rigor
quadrant-3 Cowboy Codes
quadrant-4 Vibe Coding
Karpathy Vibe Coding: [0.85, 0.10]
Ralph Bash Loop: [0.78, 0.22]
Senior Solo Dev: [0.12, 0.55]
Pair Programming + AI: [0.40, 0.65]
AI on a Leash: [0.82, 0.88]
Ralph's Uncle
flowchart TB
AI(["AI proposes Go code"])
AI --> COMP{{"go build"}}
COMP -->|type error| F1[/"specific error message<br/><i>cannot use x (type T) as Y</i>"/]
F1 -.->|feedback| AI
COMP -->|pass| LINT{{"golangci-lint · 43 linters"}}
LINT -->|reject| F2[/"lint rule violated<br/><i>ineffassign, errcheck, gocritic...</i>"/]
F2 -.->|feedback| AI
LINT -->|pass| TEST{{"go test -race"}}
TEST -->|fail| F3[/"failure trace<br/><i>plus race conditions</i>"/]
F3 -.->|feedback| AI
TEST -->|pass| OUT(["idiomatic, type-safe, race-free code"])
Go's Constraints and Idioms Make AI Coding Better
stateDiagram-v2
[*] --> Skeptical: hears about CVEs
Skeptical --> Curious: context IS the hard problem
Curious --> Building: let me try one server
Building --> Stuck: hits a known flaw
Stuck --> Building: workaround / pin / fork
Building --> Production: metadata investment pays off
Production --> Hardened: real-world feedback loop
Hardened --> [*]: metadata survives the protocol
note right of Stuck
OAuth complexity
Scaling limits
Spec ambiguity
end note
note right of Hardened
Even if MCP dies,
your documented
data remains.
end note
MCP Is Flawed. Build With It Anyway.
erDiagram
BUCKET ||--o{ OBJECT : contains
OBJECT ||--o{ VERSION : "has versions"
OBJECT ||--o{ TAG : "tagged with"
OBJECT ||--o| ACL : "guarded by"
OBJECT ||--|| METADATA : describes
MCP_TOOL }o--|| OBJECT : reads
MCP_TOOL ||--o{ PRESIGNED_URL : issues
AI_AGENT ||--o{ MCP_TOOL : invokes
BUCKET {
string name PK
string region
timestamp created_at
}
OBJECT {
string key PK
bigint size_bytes
string content_type
string etag
}
VERSION {
string version_id PK
timestamp created_at
boolean is_latest
}
TAG {
string key PK
string value
}
ACL {
string canned_policy
string owner_id
}
METADATA {
string user_meta
string content_disposition
}
PRESIGNED_URL {
string url
timestamp expires_at
string scope
}
MCP_TOOL {
string name
string permission
}
AI_AGENT {
string client_id
string transport
}
AI Data Lake Access with MCP and S3
sequenceDiagram
autonumber
actor U as Analyst
participant A as Claude / AI
participant M as mcp-trino
participant T as Trino
participant PG as PostgreSQL
participant S3 as S3 / Iceberg
participant K as Kafka
U->>A: Last quarter orders by region?
A->>M: list_schemas()
M->>T: SHOW SCHEMAS
T-->>M: catalogs and tables
M-->>A: schema graph
A->>M: execute_query(SQL)
M->>T: parse + federated plan
T->>PG: SELECT customers
T->>S3: scan parquet partitions
T->>K: scan recent stream
PG-->>T: rows
S3-->>T: rows
K-->>T: rows
T-->>M: federated result set
M-->>A: rows + metadata
A-->>U: answer with chart
AI Data Warehouse Access with MCP and Trino
flowchart TB
APP(["Your App"])
APP -->|"postgres:5432"| HOSTS
HOSTS["/etc/hosts entry"]
HOSTS -->|"127.1.27.1:5432"| KFWD
KFWD{{"kubefwd listener"}}
KFWD -->|"port-forward channel"| KAPI
KAPI["kubectl port-forward"]
KAPI -->|"TCP to pod IP"| POD
POD[("postgres pod in cluster")]
kubefwd: Forward Kubernetes Services to Localhost by Name
stateDiagram-v2
[*] --> Idle: kubefwd --tui
Idle --> Forwarding: add namespace<br/>or API call
Forwarding --> Connected: hosts updated<br/>listener bound
Connected --> Dropped: pod restart<br/>or VPN drop
Dropped --> Reconnecting: auto-reconnect
Reconnecting --> Connected: success
Reconnecting --> Failed: max retries
Failed --> Reconnecting: manual retry
Connected --> Idle: remove forward
Idle --> [*]: exit
note right of Reconnecting
exponential backoff
with jitter
end note
kubefwd in 2026: Interactive TUI and Auto-Reconnect
flowchart TB
GOAL(["Developer needs cluster services locally"])
GOAL --> OLD
GOAL --> NEW
subgraph OLD[" The old way "]
direction TB
O1["open the right cluster context"]
O2["kubectl port-forward x N services"]
O3["track local ports manually"]
O4["restart everything when pods cycle"]
O1 --> O2 --> O3 --> O4
end
subgraph NEW[" With the kubefwd MCP "]
direction TB
N1["ask AI: wire up namespace foo"]
N2["AI lists pods and services"]
N3["AI forwards everything by name"]
N4["AI auto-reconnects on pod cycle"]
N1 --> N2 --> N3 --> N4
end
OLD -.->|"painful"| FAIL[("time wasted")]
NEW -.->|"seamless"| WIN[("code against<br/>redis:6379 directly")]
AI-Assisted Kubernetes Development with kubefwd
sequenceDiagram
autonumber
actor C as External Service
participant K as Ingress / LB
participant P as portpxy
participant L as HandleHttpRequest<br/>(NiFi :10001)
participant F as NiFi Flow
C->>K: POST /webhook/orders
K->>P: forwarded
P->>P: route lookup<br/>by path or host
P->>L: proxy to :10001
L->>F: FlowFile created
F->>F: route, transform, publish
F->>L: HandleHttpResponse
L-->>P: 200 OK
P-->>K: 200 OK
K-->>C: 200 OK
Note over P: dynamic listeners,<br/>no manifest changes
Apache NiFi: Dynamic HTTP Listeners with portpxy
mindmap
root((Advanced JOLT))
Cardinality
ONE - force single
MANY - force array
MANY for nested values
Modify
=concat
=toLower / =toUpper
=sum / =divide
=literalString
Wildcards
"*" any field
"&" reference back
"$" copy a key as value
Chained Specs
shift then default
cardinality then modify
multi-pass cleanup
Apache NiFi: JOLT Transformations Part 2
flowchart TB
IN[/"Input JSON"/]
IN -->|"1"| S["shift · move and rename"]
S -->|"2"| D["default · fill missing"]
D -->|"3"| R["remove · drop fields"]
R -->|"4"| SO["sort · alphabetize"]
SO -->|"5"| C["cardinality · array vs scalar"]
C -->|"6"| M["modify · transform values"]
M --> OUT[/"Output JSON"/]
Apache NiFi: JOLT Transformations Part 1
erDiagram
AUTH_PROVIDER ||--o{ USER : authenticates
USER ||--o{ MEMBERSHIP : "belongs to"
GROUP ||--o{ MEMBERSHIP : contains
GROUP ||--o{ POLICY : granted
USER ||--o{ POLICY : "directly granted"
POLICY }o--|| RESOURCE : "applies to"
POLICY ||--|| ACTION : permits
CERTIFICATE ||--|| USER : "identifies via DN"
AUTH_PROVIDER {
string type
string config
}
USER {
string identity PK
string distinguished_name
timestamp last_login
}
GROUP {
string name PK
string identity_provider
}
MEMBERSHIP {
string user_id FK
string group_id FK
}
POLICY {
string id PK
string action
string resource_path
}
RESOURCE {
string path PK
string type
}
ACTION {
string verb PK
}
CERTIFICATE {
string subject_dn PK
string issuer
timestamp not_after
}
Apache NiFi: Securing Your Data Flows
flowchart TB
USER([Internet User])
USER --> ING
subgraph EDGE[ Edge ]
ING["Ingress Controller<br/>+ TLS"]
end
ING --> SVC
subgraph NS[ NiFi Namespace ]
SVC["nifi-service<br/>headless"]
subgraph SS[ NiFi StatefulSet ]
direction LR
N0["nifi-0"]
N1["nifi-1"]
N2["nifi-2"]
end
subgraph ZK[ ZooKeeper Ensemble ]
direction LR
Z0["zk-0"]
Z1["zk-1"]
Z2["zk-2"]
end
SVC --> N0
SVC --> N1
SVC --> N2
N0 -.->|coordination| Z0
N1 -.->|coordination| Z1
N2 -.->|coordination| Z2
end
subgraph STORAGE[ Persistent Volumes ]
direction LR
PV1[("flowfile_repo")]
PV2[("content_repo")]
PV3[("provenance_repo")]
end
N0 --> PV1
N0 --> PV2
N0 --> PV3
Apache NiFi: Production Kubernetes Deployment
quadrantChart
title Linear Algebra in Go HPC Approaches
x-axis Pure Go --> Native acceleration
y-axis Low throughput --> High throughput
quadrant-1 Optimized and parallel
quadrant-2 Convenient and slow
quadrant-3 Naive loops
quadrant-4 Single-threaded native
Plain nested for-loop: [0.10, 0.10]
goroutines over slices: [0.22, 0.42]
gonum default pure Go: [0.32, 0.48]
gonum cgo OpenBLAS: [0.78, 0.82]
SIMD via x sys cpu: [0.62, 0.68]
LAPACK direct: [0.86, 0.88]
GPU offload via cgo: [0.95, 0.95]
Linear Algebra in Go: High-Performance Computing
flowchart TB
X[/"input x"/] --> M1
W1[("W1")] --> M1
M1["matmul x · W1"] --> A1{{"ReLU"}}
A1 --> H[/"hidden h"/]
H --> M2
W2[("W2")] --> M2
M2["matmul h · W2"] --> A2{{"softmax"}}
A2 --> O[/"output y_hat"/]
O --> L["loss L"]
L -.->|"∂L/∂W2"| W2
L -.->|"∂L/∂W1 chain rule"| W1
Linear Algebra in Go: Neural Network Foundations
sankey-beta
Feature 1,Total Variance,12
Feature 2,Total Variance,18
Feature 3,Total Variance,9
Feature 4,Total Variance,15
Feature 5,Total Variance,11
Feature 6,Total Variance,8
Feature 7,Total Variance,14
Feature 8,Total Variance,13
Total Variance,PC1 - 52%,52
Total Variance,PC2 - 24%,24
Total Variance,PC3 - 12%,12
Total Variance,PC4 to PC8 - 12%,12
Linear Algebra in Go: PCA Implementation
flowchart TB
D[("Training data X, y")]
D --> AUG["augment X with bias column"]
AUG --> N["normal equations: X^T X β = X^T y"]
N --> R["ridge: add λ I to X^T X"]
R --> SOLVE["solve via mat.Solve / Cholesky"]
SOLVE --> CV["k-fold cross-validation"]
CV --> COEF[/"trained coefficients β"/]
Linear Algebra in Go: Building a Regression Library
mindmap
root((Statistics in Go))
Descriptive
mean
median
variance
stddev
Distributions
Normal
Uniform
Exponential
Poisson
Multivariate
covariance matrix
correlation matrix
Mahalanobis distance
Hypothesis tests
t-test
chi-squared
Kolmogorov-Smirnov
Linear Algebra in Go: Statistics and Data Analysis
flowchart TB
A[("Matrix A (m by n)")]
A --> ATA["form A^T A"]
ATA --> EIG["eigendecompose"]
EIG --> V["V = right singular vectors"]
EIG --> SIG["Σ = sqrt(eigenvalues)"]
V --> COMBINE["U_i = (A · v_i) / σ_i"]
SIG --> COMBINE
COMBINE --> RESULT[/"A = U · Σ · V^T"/]
RESULT --> APPS["applications"]
APPS --> P["pseudoinverse"]
APPS --> L["low-rank approximation"]
APPS --> R["dimensionality reduction"]
Linear Algebra in Go: SVD and Decompositions
stateDiagram-v2
[*] --> Init: random unit vector v_0
Init --> Multiply: w = A · v_k
Multiply --> Normalize: v_{k+1} = w / ||w||
Normalize --> CheckConverge: compare to v_k
CheckConverge --> Multiply: not converged
CheckConverge --> Compute: converged
Compute --> Output: λ = v^T A v
Output --> [*]
note right of Multiply
power iteration
converges to the
dominant eigenvalue
end note
Linear Algebra in Go: Eigenvalue Problems
quadrantChart
title Linear System Solvers
x-axis Iterative --> Direct
y-axis Sparse matrices --> Dense matrices
quadrant-1 Direct dense
quadrant-2 Iterative dense
quadrant-3 Iterative sparse
quadrant-4 Direct sparse
Gaussian elimination: [0.85, 0.85]
LU decomposition: [0.80, 0.80]
Cholesky pos-def: [0.75, 0.88]
Conjugate Gradient: [0.20, 0.30]
GMRES: [0.15, 0.25]
Jacobi: [0.10, 0.50]
Gauss-Seidel: [0.18, 0.55]
Sparse LU: [0.85, 0.20]
Linear Algebra in Go: Solving Linear Systems
erDiagram
MAT_INTERFACE ||--o{ DENSE : implements
MAT_INTERFACE ||--o{ VECDENSE : implements
MAT_INTERFACE ||--o{ SYMDENSE : implements
MAT_INTERFACE ||--o{ TRIDENSE : implements
MAT_INTERFACE ||--o{ DIAGDENSE : implements
DENSE ||--|{ MATRIX_OP : supports
VECDENSE ||--|{ VECTOR_OP : supports
DENSE ||--|| TRANSPOSE : has
DENSE ||--o| INVERSE : has
MAT_INTERFACE {
method At
method Dims
method T
}
DENSE {
int rows
int cols
slice data
}
VECDENSE {
int n
slice data
}
SYMDENSE {
int n
bool symmetric
}
TRIDENSE {
bool upper_or_lower
}
DIAGDENSE {
slice diagonal
}
MATRIX_OP {
method Mul
method Add
method T
}
VECTOR_OP {
method Dot
method Norm
}
TRANSPOSE {
Dense matrix
}
INVERSE {
bool exists
}
Linear Algebra in Go: Matrix Fundamentals
mindmap
root((Vectors in Go))
Construction
mat.NewVecDense
from float64 slice
zero vector
unit vector
Arithmetic
add
subtract
scalar multiply
Inner products
dot product
magnitude
angle between
Spatial
cross product
projection
orthogonal basis
Linear Algebra
span
linear combination
basis vectors
Linear Algebra in Go: Vectors and Basic Operations
gitGraph
commit id: "Why Kubernetes"
branch concepts
commit id: "containers"
commit id: "k8s primitives"
commit id: "operators"
checkout main
merge concepts
branch data
commit id: "Cassandra"
commit id: "Elasticsearch"
commit id: "Kafka"
checkout main
merge data
branch iot
commit id: "MQTT"
commit id: "edge nodes"
checkout main
merge iot
branch ml
commit id: "TensorFlow"
commit id: "model serving"
checkout main
merge ml
branch chain
commit id: "Ethereum"
commit id: "smart contracts"
checkout main
merge chain
commit id: "Platform" tag: "v1.0"
Advanced Platform Development with Kubernetes
sankey-beta
Raw Data,Cleaning,100
Cleaning,Feature Engineering,90
Cleaning,Discarded outliers,10
Feature Engineering,Linear Models,28
Feature Engineering,Tree Models,30
Feature Engineering,Neural Nets,22
Feature Engineering,Clustering,10
Linear Models,Predictions,28
Tree Models,Predictions,30
Neural Nets,Predictions,22
Clustering,Insights,10
Linear Algebra: Practical Applications in ML
flowchart TB
D[("Centered data X")]
D --> PATH{"choose path"}
PATH -->|"covariance"| COV["C = X^T X / (n-1)"]
COV --> EIG["eigendecompose C"]
EIG --> SORT
PATH -->|"direct"| SVD["SVD of X"]
SVD --> SORT
SORT["sort by σ^2 / λ descending"]
SORT --> SELECT["select top-k components"]
SELECT --> PROJECT["project Z = X V_k"]
PROJECT --> OUT[("Reduced data Z")]
Linear Algebra: Principal Component Analysis
mindmap
root((SVD applications))
Decomposition
A = U Sigma V^T
orthonormal U and V
diagonal Sigma
Pseudoinverse
A^+ = V Sigma^+ U^T
Moore-Penrose
least squares solver
Low rank
truncate to top k
data compression
noise reduction
Dimensionality reduction
PCA via SVD
latent factors
Image processing
lossy compression
denoising
Recommender systems
matrix factorization
collaborative filtering
Linear Algebra: Singular Value Decomposition
flowchart TB
P[/"Overdetermined: Ax = b, m > n"/]
P --> CHOICE{"path"}
CHOICE -->|"normal equations"| N["A^T A x = A^T b"]
CHOICE -->|"QR factorization"| QR["A = Q R"]
N --> SOLVE
QR --> Q["form Q^T b"]
Q --> R["R x = Q^T b<br>back-substitute"]
R --> SOLVE
SOLVE["solve for x"]
SOLVE --> X[/"x_LS minimizes ||Ax - b||"/]
X --> RES[/"residual r = b - A x_LS"/]
Linear Algebra: Least Squares and Regression
stateDiagram-v2
[*] --> Start: vectors v_1 ... v_k
Start --> First: u_1 = v_1 / norm
First --> Next: i = 2
Next --> Subtract: u_i = v_i - sum proj of v_i onto u_1...u_{i-1}
Subtract --> Normalize: u_i = u_i / norm
Normalize --> Check: i < k?
Check --> Next: yes, i = i+1
Check --> Done: i == k
Done --> [*]: orthonormal basis u_1 ... u_k
note right of Subtract
remove components
already spanned
by u_1 ... u_{i-1}
end note
Linear Algebra: Orthogonality and Projections
mindmap
root((Eigenvalues part 2))
Diagonalization
A factored as P D P inverse
D holds the eigenvalues
P holds the eigenvectors
Matrix powers
compute A k via diagonalization
far cheaper than direct multiplication
Complex eigenvalues
rotation matrices
conjugate pairs
Symmetric matrices
real eigenvalues
orthogonal eigenvectors
Applications
Markov chains
stability analysis
vibration modes
Google PageRank
Linear Algebra: Eigenvalues and Eigenvectors Part 2
flowchart TB
A[("Square matrix A (n by n)")]
A --> CHAR["characteristic polynomial<br>det(A - lambda I) = 0"]
CHAR --> ROOTS["solve for roots"]
ROOTS --> EVALS["eigenvalues lambda_1 ... lambda_n"]
EVALS --> EACH{"for each lambda_i"}
EACH --> NULL["solve (A - lambda_i I) v = 0"]
NULL --> NSPACE["null space basis"]
NSPACE --> EVECS[/"eigenvectors v_i"/]
EVALS --> PAIRS[/"eigen pairs (lambda_i, v_i)"/]
EVECS --> PAIRS
Linear Algebra: Eigenvalues and Eigenvectors Part 1
erDiagram
VECTOR_SPACE ||--o{ SUBSPACE : contains
VECTOR_SPACE ||--|| BASIS : "spanned by"
BASIS ||--|{ VECTOR : "list of"
BASIS ||--|| DIMENSION : determines
SUBSPACE ||--|| SPAN : "equals span of vectors"
SUBSPACE ||--o| ORTHOGONAL_COMPLEMENT : has
LINEAR_COMBINATION }o--o{ VECTOR : "weights"
SUBSPACE ||--|| BASIS : "has its own"
VECTOR_SPACE {
field scalars
op addition
op scalar_mult
}
SUBSPACE {
bool closed_under_add
bool closed_under_scalar
bool contains_zero
}
BASIS {
bool linearly_independent
bool spans
}
VECTOR {
slice components
}
DIMENSION {
int n
}
SPAN {
bool generates_subspace
}
ORTHOGONAL_COMPLEMENT {
int dim
}
LINEAR_COMBINATION {
slice coefficients
}
Linear Algebra: Vector Spaces and Subspaces
flowchart TB
A[("Matrix A (n by n)")]
A --> DET["compute det(A)"]
DET --> CHECK{"det = 0?"}
CHECK -->|"yes"| SING[/"Singular: no inverse"/]
CHECK -->|"no"| METHOD{"choose method"}
METHOD -->|"adjugate"| ADJ["A^-1 = adj(A) / det(A)"]
METHOD -->|"Gauss-Jordan"| GJ["row reduce [A | I] to [I | A^-1]"]
METHOD -->|"LU"| LU["solve LU x_i = e_i for each column"]
ADJ --> INV[/"inverse A^-1"/]
GJ --> INV
LU --> INV
Linear Algebra: Matrix Inverses and Determinants
sequenceDiagram
autonumber
participant Aug as Augmented [A | b]
participant Up as Upper triangular form
participant X as Solution x
Note over Aug: forward elimination
Aug->>Aug: pivot at (1,1) clear column 1 below
Aug->>Aug: pivot at (2,2) clear column 2 below
Aug->>Up: continue to (n,n)
Note over Up: now upper triangular
Up->>X: back-substitute from bottom row
X-->>Aug: solution vector x
Linear Algebra: Systems of Linear Equations
erDiagram
NDARRAY ||--o{ ELEMENT : contains
NDARRAY ||--|| SHAPE : has
NDARRAY ||--|| DTYPE : has
NDARRAY ||--|{ MATRIX_OP : supports
NDARRAY ||--|| TRANSPOSE : has_T
NDARRAY ||--o| INVERSE : has
MATRIX_OP }o--o{ NDARRAY : produces
NDARRAY {
slice data
int ndim
int size
}
ELEMENT {
float64 value
int row
int col
}
SHAPE {
tuple dims
}
DTYPE {
string name
int bytes
}
MATRIX_OP {
method matmul
method add
method elementwise
}
TRANSPOSE {
view of_array
}
INVERSE {
bool exists
}
Linear Algebra: Matrices
flowchart TB
V[/"Vector v as numpy array"/]
V --> ARITH{"operation"}
ARITH -->|"add"| ADD["v + w"]
ARITH -->|"scale"| SCALE["c * v"]
ARITH -->|"dot"| DOT["v · w = sum v_i w_i"]
ARITH -->|"cross"| CROSS["v x w in R^3"]
ARITH -->|"magnitude"| MAG["||v|| = sqrt(v · v)"]
ARITH -->|"projection"| PROJ["proj_w v = (v · w / w · w) w"]
DOT --> ANGLE["cos(theta) = v · w / (||v|| ||w||)"]
MAG --> UNIT["unit vector v / ||v||"]
Linear Algebra: Vectors
flowchart TB
EXT([Producers and Consumers])
EXT --> SVC
subgraph CLUSTER[ Kafka Namespace ]
SVC[broker Service - headless]
subgraph SS[ Kafka StatefulSet - 3 brokers ]
direction LR
B0[broker-0]
B1[broker-1]
B2[broker-2]
end
subgraph ZK[ ZooKeeper Ensemble ]
direction LR
Z0[zk-0]
Z1[zk-1]
Z2[zk-2]
end
SVC --> B0
SVC --> B1
SVC --> B2
B0 -.->|coordination| Z0
B1 -.->|coordination| Z1
B2 -.->|coordination| Z2
end
PV0[(broker-0 PVC)]
PV1[(broker-1 PVC)]
PV2[(broker-2 PVC)]
B0 --> PV0
B1 --> PV1
B2 --> PV2
Kafka on Kubernetes
sequenceDiagram
autonumber
participant Geth as Geth node (private chain)
participant Stats as ethstats backend
participant DB as in-memory store
participant UI as ethstats dashboard
actor Op as Operator
Geth->>Stats: register identity (signed)
Stats->>DB: add node entry
loop every block or interval
Geth->>Stats: push metrics (hashrate, peers, gas, latency)
Stats->>DB: update metrics
end
Op->>UI: open dashboard
UI->>Stats: subscribe (websocket)
Stats-->>UI: stream node states
UI-->>Op: hashrate, peers, gas, latency
Ethereum Ethstats
erDiagram
NODE ||--o{ PEER : connects_to
NODE ||--|| KEYSTORE : has
NODE ||--o{ ACCOUNT : owns
ACCOUNT ||--o{ TRANSACTION : signs
BLOCK ||--o{ TRANSACTION : includes
GENESIS ||--|| CHAIN_CONFIG : defines
NODE ||--|| GENESIS : initialized_with
NODE {
string nodekey
string enode_url
string role
}
PEER {
string remote_enode
}
KEYSTORE {
string path
}
ACCOUNT {
string address PK
bigint balance
}
BLOCK {
bytes hash PK
bytes parent_hash
int number
timestamp time
}
TRANSACTION {
bytes hash PK
string from_addr
string to_addr
bigint value
}
GENESIS {
json config
int chain_id
}
CHAIN_CONFIG {
string consensus
bigint difficulty
}
Ethereum Blockchain on Kubernetes
gitGraph
commit id: "Genesis"
commit id: "Block 1"
commit id: "Block 2"
branch fork
commit id: "Block 3a"
checkout main
commit id: "Block 3"
commit id: "Block 4"
commit id: "Block 5" tag: "longest chain wins"
commit id: "Block 6"
commit id: "Block 7"
commit id: "Block 8" tag: "finalized"
Blockchain
sequenceDiagram
autonumber
actor Dev
participant K as kubefwd
participant API as Kubernetes API
participant Hosts as /etc/hosts
Dev->>K: sudo kubefwd svc -n myns
K->>API: list services in namespace
API-->>K: services es, db, redis, ...
loop for each service
K->>Hosts: add 127.1.27.x entry for service name
K->>API: open port-forward to pod
end
K-->>Dev: ready
Note over Dev: app uses elasticsearch:9200 normally
Dev->>Hosts: resolve elasticsearch
Hosts-->>Dev: 127.1.27.5
Dev->>K: TCP to 127.1.27.5:9200
K->>API: tunnel to ES pod
Kubernetes Port Forwarding for Local Development
flowchart TB
C([HTTP client])
C --> ING[Ingress Controller]
ING --> KL[Kubeless function HTTP trigger]
KL --> POD[Python runtime pod auto-created]
POD --> CODE[your function code]
CODE --> ES[(Elasticsearch)]
ES -.->|results| CODE
CODE -.->|JSON response| C
FaaS on Kubernetes
mindmap
root((Elasticsearch Queries))
Term level
term
terms
range
exists
wildcard
Full text
match
multi match
query string
simple query
Compound
bool
must
should
must not
filter
Aggregations
metrics
buckets
pipelines
Specialized
geo distance
nested
parent child
Elasticsearch Essential Queries
flowchart LR
DEV([Local IDE or curl])
DEV --> KFW{port-forward channel}
KFW --> SVC[Elasticsearch Service]
SVC --> POD[(ES pod)]
POD -.->|JSON results| DEV
Remote Query Elasticsearch on Kubernetes
sequenceDiagram
autonumber
actor C as IoT / POS client
participant RX as rxtx
participant DB as bbolt (local WAL)
participant RT as rtBeat
participant ES as Elasticsearch
C->>RX: POST JSON
RX->>DB: append with seq + timestamp
RX-->>C: 200 OK (immediate)
Note over C,RX: client never waits for ES
loop every batch interval
RX->>DB: read next batch
RX->>RT: POST batch
RT->>ES: bulk index
alt ES healthy
ES-->>RT: 200
RT-->>RX: ok
RX->>DB: drop batch
else ES down or slow
RT-->>RX: error / timeout
Note over RX,DB: keep batch, retry next interval -<br/>backpressure absorbed at the edge
end
end
High Traffic JSON Data into Elasticsearch on Kubernetes
quadrantChart
title Dashboard tools for Elasticsearch
x-axis Hard to set up --> Easy to set up
y-axis Limited features --> Powerful
quadrant-1 Easy and powerful
quadrant-2 Easy and limited
quadrant-3 Hard and limited
quadrant-4 Hard and powerful
Kibana: [0.60, 0.82]
Grafana ES plugin: [0.65, 0.62]
Custom React app: [0.12, 0.92]
Discover only: [0.80, 0.38]
Excel pivot: [0.85, 0.18]
Kibana on Kubernetes
erDiagram
CLUSTER ||--o{ MASTER_NODE : coordinates_via
CLUSTER ||--o{ DATA_NODE : stores_via
CLUSTER ||--o{ CLIENT_NODE : ingests_via
CLUSTER ||--o{ INDEX : contains
INDEX ||--o{ SHARD : split_into
SHARD ||--o{ REPLICA : replicated_as
DATA_NODE ||--o{ SHARD : hosts
DATA_NODE ||--|| PVC : backed_by
CLUSTER {
string name
int version
}
MASTER_NODE {
bool dedicated
string role
}
DATA_NODE {
bool dedicated
bigint disk_gb
}
CLIENT_NODE {
bool dedicated
string role
}
INDEX {
string name PK
int primary_shards
int replicas
}
SHARD {
int shard_id PK
string node
}
REPLICA {
int replica_id
string node
}
PVC {
string storage_class
bigint size_gb
}
Production Grade Elasticsearch on Kubernetes
erDiagram
USER ||--o{ ROLEBINDING : subject_of
SERVICEACCOUNT ||--o{ ROLEBINDING : subject_of
GROUP ||--o{ ROLEBINDING : subject_of
ROLEBINDING ||--|| ROLE : grants
ROLEBINDING ||--|| NAMESPACE : scoped_to
CLUSTERROLEBINDING ||--|| CLUSTERROLE : grants
ROLE ||--o{ RULE : contains
CLUSTERROLE ||--o{ RULE : contains
RULE ||--o{ VERB : permits
RULE ||--o{ RESOURCE : applies_to
USER {
string identity PK
string cert_subject
}
SERVICEACCOUNT {
string name PK
string namespace
}
GROUP {
string name PK
}
ROLE {
string name PK
string namespace
}
CLUSTERROLE {
string name PK
}
ROLEBINDING {
string name PK
string namespace
}
CLUSTERROLEBINDING {
string name PK
}
RULE {
slice api_groups
}
VERB {
string action
}
RESOURCE {
string kind
string name
}
NAMESPACE {
string name PK
}
Kubernetes Team Access - RBAC for developers and QA
mindmap
root((Python Plotting))
Matplotlib core
figure
axes
subplots
pyplot
Plot types
line
scatter
bar
histogram
pie
Seaborn additions
regplot
pairplot
jointplot
heatmap
violinplot
Style
set_style
color palettes
themes
Integration
pandas DataFrames
numpy arrays
scipy stats
Python Data Essentials - Matplotlib and Seaborn
sequenceDiagram
autonumber
actor C as Client
participant API as txpdf API
participant Q as Job Queue
participant W as Worker
participant CHR as Headless Chrome
participant S as Object Storage
C->>API: POST /generate {url, opts}
API->>Q: enqueue job
API-->>C: 202 Accepted + job_id
Q->>W: dispatch job
W->>CHR: navigate(url)
CHR-->>W: page rendered
W->>CHR: print to PDF
CHR-->>W: PDF bytes
W->>S: upload pdf
C->>API: GET /status/{job_id}
API-->>C: ready + download url
C->>S: GET pdf
Webpage to PDF Microservice
gitGraph
commit id: "init"
branch feature/login
commit id: "handler"
commit id: "tests"
checkout main
merge feature/login
commit id: "build image"
commit id: "deploy staging"
commit id: "smoke tests"
commit id: "v0.1.0" tag: "v0.1.0"
commit id: "deploy prod"
commit id: "fix typo"
commit id: "v0.1.1" tag: "v0.1.1"
A Microservices Workflow with Golang and Gitlab CI
erDiagram
DATAFRAME ||--|| INDEX : has
DATAFRAME ||--o{ COLUMN : contains
DATAFRAME ||--o{ ROW : contains
COLUMN ||--|| SERIES : is_a
COLUMN ||--|| DTYPE : has
SERIES ||--|| INDEX : aligned_with
SERIES ||--o{ VALUE : contains
GROUPBY }o--|| DATAFRAME : groups
GROUPBY ||--o{ AGG : produces
DATAFRAME {
int rows
int cols
tuple shape
}
INDEX {
string name
type dtype
bool unique
}
COLUMN {
string name
}
ROW {
any index_label
}
SERIES {
bigint length
}
DTYPE {
string kind
int itemsize
}
VALUE {
any data
}
GROUPBY {
list keys
}
AGG {
method func
}
Python Data Essentials - Pandas
mindmap
root((NumPy))
ndarray
shape
dtype
strides
Construction
array
zeros
ones
arange
linspace
Math
add
multiply
dot
matmul
Reduction
sum
mean
std
max
argmax
Reshaping
reshape
transpose
flatten
stack
Broadcasting
scalar to array
shape compatibility
Linear algebra
solve
eig
svd
Python Data Essentials - Numpy
sequenceDiagram
autonumber
actor C as Client
participant P as n2proxy (Go)
participant S as Backend service (PHP, etc.)
C->>P: HTTP request
P->>P: parse + validate headers
P->>P: strip XSS / injection patterns
alt request safe
P->>S: forwarded request
S-->>P: response
P->>P: optional response scrub
P-->>C: 200 + body
else request blocked
P-->>C: 403 Forbidden
end
Reverse Proxy in Golang
flowchart TB
USER([Developer])
USER -->|"go install"| GN[gophernotes]
GN --> REG[register kernel.json with Jupyter]
REG --> JUP[Jupyter Lab or Notebook]
JUP --> NB[New notebook]
NB --> CHOOSE{select kernel}
CHOOSE -->|Go| KERN[gophernotes kernel]
KERN --> EXEC[run Go cells inline]
EXEC --> OUT[output rendered as text or HTML]
Golang to Jupyter
mindmap
root((Python 3 essentials))
Types
str
int
float
list
tuple
dict
set
Control flow
if elif else
for
while
try except
Functions
def
lambda
args and kwargs
decorators
Classes
class
init
inheritance
dunder methods
Modules
import
from x import y
pip install
Stdlib
os
sys
json
datetime
pathlib
Essential Python 3
sequenceDiagram
autonumber
actor B as Browser
participant ING as Ingress-nginx
participant API as API service
Note over B: cross-origin XHR
B->>ING: OPTIONS /api/x (preflight)
Note over ING: CORS annotations on ingress
ING-->>B: 204 + Access-Control-Allow-Origin headers
alt preflight allowed
B->>ING: GET /api/x (actual request)
ING->>API: forward
API-->>ING: response data
ING-->>B: 200 + CORS headers
else preflight rejected
Note over B: browser blocks the request
end
CORS on Kubernetes Ingress Nginx
sequenceDiagram
autonumber
actor C as Client
participant ING as Ingress-nginx
participant SEC as basic-auth secret (htpasswd)
participant API as backend service
C->>ING: GET /api (no Authorization header)
ING-->>C: 401 + WWW-Authenticate Basic
Note over C: client prompts for username and password
C->>ING: GET /api (Authorization Basic ...)
ING->>SEC: bcrypt verify against htpasswd
alt valid credentials
ING->>API: forward
API-->>ING: response
ING-->>C: 200 + body
else invalid
ING-->>C: 401
end
Basic Auth on Kubernetes Ingress
flowchart TB
C([Client app])
C -->|POST JSON credentials| TX[txToken proxy]
TX -->|forward credentials| AUTH[Authentication endpoint]
AUTH -->|user object as JSON| TX
TX --> SIGN[sign with HS256]
SIGN --> JWT[/JWT token/]
JWT -.->|return| C
C -->|subsequent requests with Bearer JWT| API[Protected API]
API --> VERIFY[verify HS256 signature]
VERIFY --> CLAIM[extract claims]
CLAIM --> ALLOW[/access granted/]
JWT Microservice, Tokenize Remote APIs
stateDiagram-v2
[*] --> Requested: ingress annotation
Requested --> Challenged: ACME http-01 or dns-01
Challenged --> Verified: challenge served on .well-known
Verified --> Issued: signed by Let's Encrypt
Issued --> Active: secret stored in namespace
Active --> Renewing: 30 days before expiry
Renewing --> Issued: success
Renewing --> Failed: rate limit or DNS issue
Failed --> Renewing: backoff retry
note right of Verified
cert-manager handles
the challenge automatically
via a solver pod
end note
Let's Encrypt, Kubernetes
mindmap
root((Helm))
Concepts
Chart
Release
Repository
Values
Lifecycle
install
upgrade
rollback
uninstall
Chart structure
Chart yaml
values yaml
templates folder
charts subfolder
Common operations
helm repo add
helm install
helm upgrade
helm list
helm uninstall
Helm on Custom Kubernetes
flowchart TB
POD([Pod resolves api.example.com])
POD --> KD[kube-dns / CoreDNS]
KD --> CHECK{cluster-local name?}
CHECK -->|yes| SVC[(K8s Service ClusterIP)]
CHECK -->|no, external| UP[upstream nameservers]
UP --> G1[8.8.8.8 Google DNS]
UP --> G2[8.8.4.4 Google DNS]
G1 --> ANS[/A record or CNAME/]
G2 --> ANS
ANS -.->|return| POD
Kubernetes Custom Upstream DNS
flowchart TB
EXT([External user])
EXT --> LB[LoadBalancer or NodePort]
LB --> ING[Ingress-nginx Controller pod]
ING --> RULES{host / path rules}
RULES -->|app1.example.com| S1[Service: app1]
RULES -->|app2.example.com| S2[Service: app2]
RULES -->|/api| S3[Service: api]
S1 --> P1[(app1 pods)]
S2 --> P2[(app2 pods)]
S3 --> P3[(api pods)]
Ingress on Custom Kubernetes
erDiagram
KUBECONFIG ||--o{ CLUSTER : registers
KUBECONFIG ||--o{ USER : registers
KUBECONFIG ||--o{ CONTEXT : defines
CONTEXT ||--|| CLUSTER : binds_to
CONTEXT ||--|| USER : binds_to
CONTEXT ||--o| NAMESPACE : default_namespace
KUBECONFIG ||--|| CURRENT_CONTEXT : tracks
KUBECONFIG {
string path
}
CLUSTER {
string name PK
string server
string ca_cert
}
USER {
string name PK
string token
string client_cert
}
CONTEXT {
string name PK
string cluster_ref
string user_ref
}
NAMESPACE {
string name PK
}
CURRENT_CONTEXT {
string name
}
kubectl Context Multiple Clusters
sequenceDiagram
autonumber
actor U as User
participant T as Terminal
participant DU as diskutil
participant K as Kernel
U->>T: diskutil list
T->>DU: query
DU->>K: enumerate devices
K-->>DU: disk0, disk2, ...
DU-->>T: identifiers + sizes
T-->>U: device list
U->>T: diskutil eraseDisk JHFS+ Backup disk2
T->>DU: eraseDisk
DU->>K: unmount + format + remount
K-->>DU: ok
DU-->>T: Volume Backup created
T-->>U: done
Formatting Drives on MacOS
flowchart TB
ERR[/"x509: certificate signed by unknown authority"/]
ERR --> Q1{kubeconfig has the cluster CA?}
Q1 -->|no| F1[copy CA bundle from cluster]
Q1 -->|yes| Q2{server URL matches CA SAN?}
Q2 -->|no| F2[regenerate cert with correct SAN]
Q2 -->|yes| Q3{cert expired?}
Q3 -->|yes| F3[rotate cluster certs]
Q3 -->|no| F4[check client clock / NTP]
F1 --> OK[/connection works/]
F2 --> OK
F3 --> OK
F4 --> OK
Kubectl x509 Unable to Connect
flowchart TB
USER([External traffic])
USER --> LB[LoadBalancer / NodePort]
subgraph CP[Control plane node]
API[kube-apiserver]
SCHED[scheduler]
CTRL[controller-manager]
ETCD[(etcd)]
API <--> ETCD
end
subgraph W1[Worker node 1]
KL1[kubelet]
POD1[(pods)]
end
subgraph W2[Worker node 2]
KL2[kubelet]
POD2[(pods)]
end
LB --> KL1
LB --> KL2
KL1 -.->|register| API
KL2 -.->|register| API
CNI[CNI overlay network]
POD1 -.- CNI
POD2 -.- CNI
Production Hobby Cluster
stateDiagram-v2
[*] --> Waiting: cron schedule
Waiting --> Running: scheduled tick
Running --> Syncing: rsync over SSH
Syncing --> Verifying: checksum compare
Verifying --> Done: match
Verifying --> Retrying: mismatch
Retrying --> Syncing: backoff retry
Done --> Waiting: next interval
Running --> Failed: SSH or network error
Failed --> Waiting: log + skip cycle
rSync Files on Interval
erDiagram
CUSTOMER ||--o{ ORDER : places
ORDER ||--|{ ORDER_ITEM : contains
PRODUCT ||--o{ ORDER_ITEM : ordered_as
CUSTOMER ||--o{ ADDRESS : has
CUSTOMER {
int id PK
string name
string email
}
ADDRESS {
int id PK
int customer_id FK
string street
string city
}
ORDER {
int id PK
int customer_id FK
date placed_at
decimal total
}
ORDER_ITEM {
int order_id FK
int product_id FK
int quantity
decimal price
}
PRODUCT {
int id PK
string sku
string name
decimal price
}
SQL Foundations
sequenceDiagram
autonumber
actor U as User
participant T as Terminal
participant DU as diskutil
participant DD as dd
participant SD as SD card
U->>T: diskutil list
T-->>U: device list
U->>T: diskutil unmountDisk disk2
T->>DU: unmount
DU->>SD: eject volumes
DU-->>T: ok
U->>T: sudo dd if=image.iso of=rdisk2 bs=4m
T->>DD: dd raw write
DD->>SD: write blocks
SD-->>DD: bytes written
DD-->>T: records out
U->>T: diskutil eject disk2
T->>DU: eject
DU-->>T: ok
Burn SD Images on MacOs
sequenceDiagram
autonumber
actor C as Client
participant ING as Ingress-nginx
participant API as API service
Note over C: client uploads 50MB JSON
C->>ING: POST /api/data (50MB)
Note over ING: default proxy-body-size 1m
ING-->>C: 413 Request Entity Too Large
Note over ING: ops adds annotation -<br/>proxy-body-size 100m
C->>ING: POST /api/data (50MB) again
ING->>API: forward (within new limit)
API-->>ING: 200
ING-->>C: 200 OK
Kubernetes - 413 Request Entity Too Large
stateDiagram-v2
[*] --> Local: minikube start
Local --> Production: kubectl config use-context prod
Production --> Local: kubectl config use-context minikube
Local --> Staging: use-context staging
Staging --> Local: use-context minikube
Production --> Staging: use-context staging
Staging --> Production: use-context prod
note right of Local
developer machine
single-node cluster
end note
note right of Production
remote multi-node
requires CA cert + token
end note
Kubernetes Remote Control
mindmap
root((Microservices))
Principles
single responsibility
loosely coupled
independently deployable
polyglot persistence
Patterns
API gateway
service discovery
circuit breaker
saga
sidecar
Operations
containerization
orchestration
observability
tracing
Communication
sync HTTP / gRPC
async pub sub
events
Data
database per service
eventual consistency
CQRS
Microservices & Kubernetes
sequenceDiagram
autonumber
actor U as Developer
participant T as Terminal
participant DK as Docker engine
participant CQL as cqlsh container
participant CASS as Cassandra cluster
U->>T: docker run --rm cassandra:3 cqlsh
T->>DK: pull image (cached)
DK->>CQL: run cqlsh entrypoint
CQL->>CASS: connect on 9042
CASS-->>CQL: protocol handshake
CQL-->>U: cqlsh prompt
U->>CQL: SELECT * FROM ks.users LIMIT 10
CQL->>CASS: CQL query
CASS-->>CQL: result set
CQL-->>U: rows
U->>CQL: exit
CQL-->>T: container removed
Don't Install cqlsh
flowchart LR
USR([Developer])
USR -->|"alias emacs"| AL[shell alias]
AL --> DR["docker run -it --rm<br/>-v PWD:/work emacs-image"]
DR --> CON[(Alpine container with emacs)]
CON --> EDIT[edit files in /work]
EDIT --> EXIT[container exits]
EXIT -.->|files persisted to host| USR
Don't Install Emacs
mindmap
root((Kubernetes basics))
Workloads
Pod
Deployment
StatefulSet
DaemonSet
Job
CronJob
Networking
Service
Ingress
NetworkPolicy
DNS
Storage
Volume
PersistentVolume
PersistentVolumeClaim
StorageClass
Config
ConfigMap
Secret
Access
Namespace
RBAC
ServiceAccount
Tools
kubectl
kubeadm
minikube
helm
Kubernetes Overview
flowchart LR
PROC[(/proc/cpuinfo)]
PROC -->|cat| TXT[full CPU info text]
TXT -->|grep Serial| LINE["Serial line"]
LINE -->|cut -d colon -f 2| OUT[/serial number/]
Raspberry Pi - Serial Number