A Complete Introduction To Monitoring Kubernetes With New .

A Complete Introduction to Monitoring Kubernetes with New RelicKubernetes ObservabilityWe recommend that Kubernetes observability begins withthese five practices:1. Visualizing services2. Monitoring health and capacity3. Correlating Kubernetes events4. Understanding APM correlation5. Investigating end-user experienceVisualizing servicesSuggested alertingWhen deploying the New Relic Kubernetes integrationfor the first time in an account, a default set of alertconditions is deployed to the account. The alert policyis configured without a notification channel to avoidunwanted alerts.You can customize the alert conditions’ thresholds to yourenvironment and update the alert policy to send notifications. For more, see the New Relic Infrastructure alertsdocumentation.When working in a Kubernetes environment, it can be difficult to untangle the dependencies between applicationsand infrastructure, and navigate all of the entities—containers, pods, nodes, deployments, namespaces, and soon—that may be involved in a troubleshooting effort. Youneed to observe performance and dependencies acrossthe entire Kubernetes environment.You should be able to visualize key parts of your services,including: The structure of your application and its dependencies The interactions between various microservices, eventhose that are intermingled across your machine clusterMonitoring cluster health and capacityKubernetes environments vary from deployment todeployment, but they have all a handful of key components, resources, and potential errors in common. The following sections introduce best practices, including tips forhow to use New Relic and alerts, for monitoring the healthand capacity of any Kubernetes environment: Track cluster resource usage Monitor node resource consumption Monitor for missing pods Find pods that aren’t runningHow New Relic helps Troubleshoot container restartsThe cluster explorer provides a multi-dimensional rep- Track container resource usageresentation of a Kubernetes cluster that allows teams to Monitor storage volumesdrill down into Kubernetes data and metadata in a highfidelity, curated UI that simplifies complex environments.Teams can use cluster explorer to more quickly troubleshoot failures, bottlenecks, and other abnormal behavioracross their Kubernetes environments. Monitor the control plane: etcd, the API server, thescheduler, and the controller managerTrack cluster resource usageWhen you administer clusters, you need enough usableresources in your cluster to avoid running into issues whenscheduling pods or deploying containers. If you don’t have07

A Complete Introduction to Monitoring Kubernetes with New Relicenough capacity to meet the minimum resource requirements of all your containers, scale up your nodes’ capacityor add more nodes to distribute the workload.You should know: What percentage of cluster resources you’re using atany given time If your clusters are over- or under-provisionedHow New Relic helpsNew Relic tracks resource consumption (used cores andmemory) for each Kubernetes node. That lets you trackthe number of network requests sent across containerson different nodes within a distributed service. You canalso track resource metrics for all containers on a specificnode—regardless of which service they belong to. How much demand have you’ve placed on your systemsHow New Relic helpsOur Kubernetes integration monitors and tracks aggregated core and memory usage across all nodes in yourAlways ensure your current deployment has sufficientcluster. This allows you to meet resource requirements forresources to scale. You don’t want new node deploymentsoptimal application performance.blocked by a lack of resources.Suggested alertingSet alerts so you’ll be notified if hosts stop reporting orif a node’s CPU or memory usage drops below a desiredthreshold.Monitor for missing podsFrom time to time, you may find your cluster is missing aSuggested alertingpod. A pod can go missing if the engineers did not providesufficient resources when they scheduled it. The pod maySet alerts on the cores and memory usage of the hosts inhave never started; it’s in a restart loop; or it went missingyour cluster.because of an error in its configuration.Monitor node resource consumptionTo make sure Kubernetes does its job properly, you needto confirm the health and availability of pod deployments.Beyond simply keeping track of nodes in your cluster, youA pod deployment defines the number of instancesneed to monitor the CPU, memory, and disk usage forthat need to be present for each pod, including backupKubernetes nodes (workers and masters) to ensure allinstances. (In Kubernetes, this is referred to as a Replica-nodes in your cluster are healthy.Use this data to ensure: You have enough nodes in your cluster The resource allocations to existing nodes is sufficientfor deployed applications You’re not hitting any resource limits08Set). Sometimes the number of active pods is not specified in the Replicas field on each deployment. Even if theyare specified, Kubernetes may determine if it can runanother instance based on resources the administratorhas defined.forbidden: exceeded quota: compute-resources,requested: pods 1, used: pods 1, limited: pods 1

A Complete Introduction to Monitoring Kubernetes with New RelicHow New Relic helpsNew Relic makes it easier to avoid this issue by knowingthe resource limitations of the cluster.If you don’t have enough resources to schedule a pod,add more container instances to the cluster or exchangea container instance for one with the appropriate amountof resources. In general, you can use the New Relic Kubernetes integration to monitor for missing pods and immediately identify deployments that require attention. Thisoften creates an opportunity to resolve resource or configuration issues before they affect application availabilityor performance.When you see that pods aren’t running, you’ll wantto know: If there are any pods in a restart loop How often are requests failing If there are resource issues or configuration errorsHow New Relic helpsAs noted, if you have resource issues or configurationerrors, Kubernetes may not be able to schedule the pods.In such cases, you want to check the health of your deployments, and identify configuration errors or resource issues.With the New Relic Infrastructure Kubernetes integration,you can use default deployment data to discover and trackpods that may not be running and sort them by clusterand namespace.Additionally, you can analyze further root causes of terminated pods, with the terminated pods metric. For example,if a pod is terminated because its application memory hasSuggested alertingSet an alert for when a deployment’s missing pods valuerises above a certain threshold for a certain period. If thereached the memory limit set on the containers, it will bekilled by the out of memory (OOM) killer. In such cases,New Relic will expose the reason for pod termination.number of available pods for a deployment falls belowthe number of pods you specified when you created thedeployment, the alert will trigger. The alert will be appliedto each deployment that matches the filters you set.Find pods that aren’t runningKubernetes dynamically schedules pods into the cluster,but if you have resource issues or configuration errors,scheduling will likely fail. If a pod isn’t running or evenscheduled, that means there’s an issue with either the podor the cluster, or with your entire Kubernetes deployment.Suggested alertingSet alerts on the status of your pods. Alerts should trigger when a pod has a status of “Failed,” ”Pending,” or“Unknown” for the period of time you specify.09

A Complete Introduction to Monitoring Kubernetes with New RelicTroubleshoot container restartsWhen monitoring container resource usage, you’ll wantIn normal conditions, containers should not restart. Con-to know:tainer restarts are a sign that you’re likely hitting a mem- If your containers are hitting resource limits and af fectingory limit in your containers. Restarts can also indicate anissue with either the container itself or its host. Additionally, because of the way Kubernetes schedules containers, it can be difficult to troubleshoot container resourceissues since Kubernetes will restart—or kill—containerswhen they hit their limits.Monitoring container restarts helps you understand: If any containers are in a restart loop How many container restarts occurred in X amountof time Why containers are restartingHow New Relic helpsA running count of container restarts is part of the defaultcontainer data New Relic gathers with the Kubernetesintegration.Suggested alertingThis an optional alerting scenario. Kubernetes automatically restarts containers, and setting up an alert will giveyou an immediate, useful notification, but don’t let container restarts interrupt your sleep.Track container resource usageMonitoring container resource usage helps you ensurethat their containers and applications remain healthy. Forexample, if a container hits its limit for memory usage, thekubelet agent might kill it.10the performance of their applications If there are spikes in resource consumption If there is a pattern to the distribution of errors percontainerHow New Relic helpsFirst, identify the minimum amount of CPU and memorya container requires to run—which needs to be guaranteed by the cluster—and monitor those resources withNew Relic.Second, monitor container resource limits. These are themaximum amounts of resources that the container will beallowed to consume. In Kubernetes, resource limits areunbounded by default.

A Complete Introduction to Monitoring Kubernetes with New RelicThis type of monitoring can help proactively resolveWith Kubernetes volume monitoring in New Relic, youresource usage issues before they affect your application.can monitor your volumes, and set alerts on them so thatSuggested alertingSet alerts on container CPU and memory usage and onyou get informed as soon as a volume reaches a certainthreshold—a proactive approach to limiting issues withapplication performance or availability.limits for those metrics.Monitor storage volumesOne thing you definitely want to avoid when running a

Kubernetes makes it easy to deploy and operate applica-tions in a microservice architecture. It does so by creating an abstraction layer on top of a group of hosts, so that development teams can deploy their applications and let Kubernetes manage: Controlling resource consumption by application