It’s A Match!

&It’s A Match!

Brief Introduction032

Wenting GongSoftware Engineer, Observability, TinderGithub: https://github.com/christine-gongLinkedin: .gong@gotinder.com3

Agenda034

Agenda Our Monitoring Journey with Grafana- Applications running as VMs and Containers- Infrastructure resources- Datasource and Dashboard Automation Demo5

2 Years Ago036

Let’s meet some of our Tinder Engineering team members:AliceBobObservability TeamBackend TeamCharlieCloud Infrastructure Team037

Let’s build our observability infrastructure to monitor the healthof all the services!CharlieCloud Infrastructure TeamHmmm, we also need a central place for engineers to checkand view the real-time metrics!AliceObservability Team038

Initial ScenarioAround fifty microservices runningwith AWS EC2 instances and ELBs9

AWS Cloudwatch and Grafana should help with that!AliceObservability Team 0310

Solution11

Solution12

Ok! Now we can view and check the services real-time healthmetrics from Grafana!AliceObservability TeamGreat!BobBackendTeamCharlieCloud InfraTeam0313

BobBackend TeamWe backend engineers also want to know the details about ourservices!E.g. the P50, P95, P99 latency, the requests status for differentroutes, etcThat sounds fair enough! We need to investigate some opensource monitoring solutions!AliceObservability Team0314

AliceObservability Team0315

Prometheus is the best option for us!AliceObservability Team0316

Solution17

A few months later.0318

Our business grows so fast that one prometheus server couldnot handle them all.BobBackend TeamYeah, we should have a better solution to make our monitoringinfra more scalable.AliceObservability Team0319

Let’s get each service an assigned prometheus server!AliceObservability Team0320

Solution21

SolutionPrometheus Datasource22

Good loud InfraTeam0323

Picture from ide-what-marketing-metrics-to-track-and-report0324

The metrics are great! Can we keep them longer so that we cansee the trends and compare them when necessary?BobBackend TeamThat’s a good point!AliceObservability Team0325

Compare two approachesApproachIncrease the retentionperiod for all prometheusserversHave a separateprometheus archiveserver for long-termmetricsProsCons Super EASYUses the resources and more efficiently Not all metrics are neededfor longer retentionIncreased costs and resources wastedfor unnecessary metricsExtra setupModule owners need tounderstand and updateexisting configuration26

Compare two approachesApproachIncrease the retentionperiod for all prometheusserversHave a separateprometheus archiveserver for long-termmetricsProsCons Super EASYUses the resources and more efficiently Not all metrics are neededfor longer retentionIncreased costs and resources wastedfor unnecessary metricsExtra setupModule owners need tounderstand and updateexisting configuration27

SolutionLet’s only archive those key metrics for longer time!AliceObservability Team28

Well oud InfraTeam0329

Picture from Adam Tow, AllThingsD.com0330

Picture from Adam Tow, AllThingsD.com0331

Our services expand so aggressively, it is time to move toKubernetes for deploying and managing containerizedapplications at scale.CharlieCloud Infrastructure Team0332

Our services expand so aggressively, it is time to move toKubernetes for deploying and managing containerizedapplications at scale.Kubernetes was designed to give developers more velocity,efficiency and agility.CharlieCloud Infrastructure Team0333

Our services expand so aggressively, it is time to move toKubernetes for deploying and managing containerizedapplications at scale.CharlieCloud InfrastructureTeamKubernetes was designed to give developers more velocity,efficiency and agility.Then we should definitely support the monitoring for K8Senvironment!AliceObservability Team0334

CharlieCloud InfrastructureTeam35

Developers should not redo the metrics instrumenting, we needto stick with Prometheus in Kubernetes as well!AliceObservability Team ?Prometheus Operator0336

Solution - metrics for all servicesprom-operatorPrometheus datasources37

Solution: archived metrics for longer retentionprom-operator38

lieCloud InfraTeam0339

Speaking of, sometimes issue happens inside k8scluster. We need the capability to monitor the k8s health itself.CharlieCloud Infrastructure TeamSure thing! Kubernetes has a large, rapidly growing ecosystem.Its services, support, and tools are widely available [1]AliceObservability Team0340

Solution41

Cloud InfraTeam0342

Are infrastructure resources being taken care of? Do we havemetrics for them?BobBackend TeamLet me look into that!AliceObservability Team0343

Infrastructure MonitoringComponentMethodElasticsearch ClustersPrometheus elasticsearch exporterKafka ClustersPrometheus kafka exporter,Prometheus kafka consumer group exporterAWS RDS InstancesPrometheus cloudwatch exporterAWS DynamodbPrometheus cloudwatch exporter44

Now everything gets settled!With the help of k8s, we can scale the testing environmentswith monitoring! Could we reduce manual dashboardcreation work?CharlieCloud Infrastructure TeamWe should definitely do that! Also Grafana has many usefulAPIs, I can develop that automation utils!AliceObservability Team0345

Wrote my Grafana dashboard automation utils with the help ofGrafanalib in Python with a Http client wrapper. Would help alot when duplicating similar dashboards for all clusters and allenvironmentsAliceObservability Team46

Summary-Have Grafana as a central monitoring place- Detailed real-time metrics for each service- Longer term historical key metrics available-Monitor the health of all the microservices running in containers and VMsMonitor the health of Kubernetes clustersMonitor the health of all of our infrastructure resources, e.g.Elasticsearch, Dynamodb, Redis, etcDatasource auto-discovery and Dashboard auto-creation-47

Demo Time0348

Reference[1] is-kubernetes/[2] Emoji icons supplied by EmojiOne[3] Kubernetes related icons s49

QUESTIONS?

THANKS

Kafka Clusters Prometheus kafka exporter, Prometheus kafka consumer group exporter AWS RDS Instances Prometheus cloudwatch exporter AWS Dynamodb Prometheus cloudwatch exporter. 4503 Alice . - Have Grafana as a central monitoring