IMTI

(I Make Things Interesting) Architecting, Developing, nixCraft, DevOps, AI/ML, Blockchain

High Traffic JSON Data into Elasticsearch on Kubernetes

Instant, reliable, send and forget.

IOT devices, Point-of-Sale systems, application events or any client that sends data destined for indexing in Elasticsearch often need to send and forget, however, unless that data is of low value there needs to be assurance that arrives at its final destination. Back-pressure and database outages can pose a considerable threat to data integrity. Contents Background Overview Development Environment the-project Namespace The Project: Weather (wx) Data rxtx for Store-and-Forward wx-rxtx Service wx-rxtx StatefulSet rtBeat to Collect, Buffer and Publish wx-rtbeat Service wx-rtbeat ConfigMap wx-rtbeat Deployment Client Simulation / Kubernetes Cron Performance Conclusion Port Forwarding / Local Development Reference Support this blog!

Posted by Craig Johnston Wednesday, July 18, 2018

FEATURED TAGS
blockchain data data-science development devops docker elasticsearch golang ingress iot kubectl kubernetes microservices python raspberry-pi security utils
ABOUT ME

I lead a talented team of software development and creative engineers, covering many industries looking to collect, analyze, move, buffer, queue, process and present data in significant ways. My expertise and that of my team revolve around microservices, artificial intelligence, algorithms, machine learning and blockchain technologies.

BOOKMARKS
REDDIT
AWESOME
DOCKER CONTAINERS