Monitoring Guide - Couchbase

Monitoring: Data Serviceep clock cas drift threshold exceededep data read failedNumber of disk read failuresep data write failedNumber of disk write failuresep dcp 2i backoffNumber of backoffs for index DCP connectionsep dcp 2i countNumber of internal second index DCP connections inthis bucketep dcp 2i items remainingNumber of secondary index items remaining to be sentto consumer in this bucketep dcp 2i items sentNumber of secondary index items per second beingsent for a producer for this bucketep dcp 2i producer countNumber of secondary index senders for this bucketep dcp 2i total backlog sizeTotal size in bytes of the DCP backlog for secondaryindexesep dcp 2i total bytesNumber of bytes per second being sent for secondaryindexes DCP connectionsep dcp cbas backoffNumber of backoffs for Analytics DCP connectionsep dcp cbas countNumber of internal Analytics DCP connections in thisbucketep dcp cbas items remainingNumber of Analytics items remaining to be sent toconsumer in this bucketep dcp cbas items sentNumber of Analytics items per second being sent for aproducer for this bucketep dcp cbas producer countNumber of Analytics senders for this bucketep dcp cbas total backlog sizeTotal size in bytes of the DCP backlog for Analyticsep dcp cbas total bytesNumber of bytes per second being sent for AnalyticsDCP connectionsep dcp eventing backoffNumber of backoffs for Eventing DCP connectionsep dcp eventing countNumber of internal Eventing DCP connections in thisbucketep dcp eventing items remainingNumber of Eventing items remaining to be sent toconsumer in this bucketep dcp eventing items sentNumber of Eventing items per second being sent for aproducer for this bucketep dcp eventing producer countNumber of Eventing senders for this bucketep dcp eventing total backlog sizeTotal size in bytes of the DCP backlog for Eventingep dcp eventing total bytesNumber of bytes per second being sent for EventingDCP connectionsep dcp fts backoffNumber of backoffs for FTS DCP connectionsep dcp fts countNumber of internal FTS DCP connections in this bucketep dcp fts items remainingNumber of FTS items remaining to be sent to consumerin this bucketNumber of FTS items per second being sent for aCouchbase Professional Services9

Monitoring: Data Serviceep dcp fts items sentNumber of FTS items per second being sent for aproducer for this bucketep dcp fts producer countNumber of FTS senders for this bucketep dcp fts total backlog sizeTotal size in bytes of the DCP backlog for FTSep dcp fts total bytesNumber of bytes per second being sent for FTS DCPconnectionsep dcp other backoffNumber of backoffs for other DCP connectionsep dcp other countNumber of other DCP connections in this bucketep dcp other items remainingNumber of items remaining to be sent to consumer inthis bucketep dcp other items sentNumber of items per second being sent for a producerfor this bucketep dcp other producer countNumber of other senders for this bucketep dcp other total backlog sizeTotal size in bytes of the DCP backlog for analyticsotherep dcp other total bytesNumber of bytes per second being sent for other DCPconnections for this bucketep dcp replica backoffNumber of backoffs for replication DCP connectionsep dcp replica countNumber of internal replication DCP connections in thisbucketep dcp replica items remainingNumber of replication items remaining to be sent toconsumer in this bucketep dcp replica items sentNumber of replication items per second being sent for aproducer for this bucketep dcp replica producer countNumber of replication senders for this bucketep dcp replica total backlog sizeTotal size in bytes of the DCP backlog for replicationep dcp replica total bytesNumber of bytes per second being sent for replicationDCP connectionsep dcp views indexes backoffNumber of backoffs for view/index DCP connectionsep dcp views indexes countNumber of internal view/index DCP connections in thisbucketep dcp views indexes items remainingNumber of view/index items remaining to be sent toconsumer in this bucketep dcp views indexes items sentNumber of view/index items per second being sent for aproducer for this bucketep dcp views indexes producer countNumber of views/index senders for this bucketep dcp views indexes total backlog sizeTotal size in bytes of the DCP backlog for views/indexesep dcp views indexes total bytesNumber of bytes per second being sent forviews/indexes DCP connectionsep dcp views backoffNumber of backoffs for view DCP connectionsep dcp views countNumber of internal view DCP connections in this bucketNumber of view items remaining to be sent to consumerCouchbase Professional Services10

Monitoring: Data Serviceep dcp views items remainingNumber of view items remaining to be sent to consumerin this bucketep dcp views items sentNumber of view items per second being sent for aproducer for this bucketep dcp views producer countNumber of view senders for this bucketep dcp views total backlog sizeTotal size in bytes of the DCP backlog for viewsep dcp views total bytesNumber of bytes per second being sent for view DCPconnectionsep dcp xdcr backoffNumber of backoffs for XDCR DCP connectionsep dcp xdcr countNumber of internal XDCR DCP connections in thisbucketep dcp xdcr items remainingNumber of XDCR items remaining to be sent toconsumer in this bucketep dcp xdcr items sentNumber of XDCR items per second being sent for aproducer for this bucketep dcp xdcr producer countNumber of XDCR senders for this bucketep dcp xdcr total backlog sizeTotal size in bytes of the DCP backlog for XDCRep dcp xdcr total bytesNumber of bytes per second being sent for XDCR DCPconnectionsep diskqueue drainTotal number of items per second being written to diskin this bucketep diskqueue fillTotal number of items per second being put on the diskqueue in thisep diskqueue itemsTotal number of items waiting to be written to disk in thisbucketep flusher todoNumber of items currently being written.ep item commit failedNumber of times a transaction failed to commit due tostorage errors.ep kv sizeTotal amount of user data cached in RAM in this bucketep max sizeThe maximum amount of memory this bucket can use.ep mem high watHigh water mark for auto-evictionsep mem low watLow water mark for auto-evictionsep meta data memoryTotal amount of item metadata consuming RAM in thisbucketep num non residentThe number of non-resident items.ep num ops del metaNumber of delete operations per second for this bucketas the target for XDCRep num ops del ret metaNumber of delRetMeta operations.ep num ops get metaNumber of metadata read operations per second for thisbucket as the target for XDCRep num ops set metaNumber of set operations per second for this bucket asthe target for XDCRCouchbase Professional Services11

Monitoring: Data Serviceep num ops set ret metaep num value ejectsTotal number of items per second being ejected to diskin this bucketep oom errorsNumber of times unrecoverable OOMs happened whileprocessing operations.ep ops createTotal number of new items being inserted into thisbucketep ops updateNumber of items updated on disk per second for thisbucketep overheadExtra memory used by transient data like persistencequeues, replication queues, checkpoints, etc.ep queue sizeNumber of items queued for storage.ep replica ahead exceptionsTotal number of ahead exceptions for all replicavBucketsep replica hlc driftThe sum of total abs drift for the node's activevBucketsep replica hlc drift countThe sum of total abs drift count for the node's activevBucketsep resident items ratePercentage of all items cached in RAM in this bucketep tmp oom errorsNumber of back-offs sent per second to client SDKs dueto "out of memory" situations from this bucketep vb totalTotal number of vBuckets for this bucketevictionsNumber of items per second evicted from this bucketget hitsNumber of get operations per second for data that thisbucket containsget missesNumber of get operations per second for data that thisbucket does not containhibernated requestsNumber of hibernated requestshibernated wakedNumber of times hibernated wakedhit ratioPercentage of get requests served with data from thisbucketincr hitsNumber of increment operations per second for datathat this bucket containsincr missesNumber of increment operations per second for datathat this bucket does not containmem usedAmount of Memory usedmissesTotal amount of operations per second for that that thebucket does not containopsTotal amount of operations per second (includingXDCR) to this bucketrest requestsswap totalswap usedCouchbase Professional Services12

Monitoring: Data Serviceswap usedvb active ejectNumber of items per second being ejected to disk from"active"vb active itm memoryAmount of active user data cached in RAM in thisbucketvb active meta data memoryAmount of active item metadata consuming RAM in thisbucketvb active numNumber of vBuckets in the "active" state for this bucketvb active num non residentNumber of non-resident items.vb active ops createNew items per second being inserted into "active"vBuckets in this bucketvb active ops updateNumber of items updated on disk per second for thisbucketvb active queue ageSum of disk queue item age in milliseconds for "active"vBucketsvb active queue drainNumber of active items per second being written to diskin this bucketvb active queue fillNumber of active items per second being put on theactive item disk queue in this bucketvb active queue sizeNumber of active items waiting to be written to disk inthis bucketvb active resident items ratioPercentage of active items cached in RAM in thisbucketvb active sync write aborted countNumber of vbucket writes abortedvb active sync write accepted countNumber of vbucket writes acceptedvb active sync write committed countNumber of vbucket writes committedvb avg active queue ageAverage age in seconds of active items in the activeitem queue for this bucketvb avg pending queue ageAverage age in seconds of pending items in the pendingitem queue for this bucket and should be transientduring rebalancingvb avg replica queue ageAverage age in seconds of replica items in the replicaitem queue for this bucketvb avg total queue ageAverage age in seconds of all items in the disk writequeue for this bucketvb pending curr itemsNumber of items in "pending" vBuckets in this bucketand should be transient during rebalancingvb pending ejectNumber of items per second being ejected to disk from"pending" vBuckets in this bucket and should betransient during rebalancingvb pending itm memoryAmount of pending user data cached in RAM in thisbucket and should be transient during rebalancingvb pending meta data memoryAmount of pending item metadata consuming RAM inthis bucket and should be transient during rebalancingCouchbase Professional Services13

Monitoring: Data ServiceNumber of vBuckets in the "pending" state for thisvb pending numbucket and should be transient during rebalancingvb pending num non residentNumber of non-resident items.vb pending ops createNew items per second being instead into "pending"vBuckets in this bucket and should be transient duringrebalancingvb pending ops updateNumber of items updated on disk per second for thisbucketvb pending queue ageSum of disk queue item age in milliseconds.vb pending queue drainNumber of pending items per second being written todisk in this bucket and should be transient duringrebalancingvb pending queue fillNumber of pending items per second being put on thepending item disk queue in this bucket and should betransient during rebalancingvb pending queue sizeNumber of pending items waiting to be written to disk inthis bucket and should be transient during rebalancingvb pending resident items ratioPercentage of items in pending state vbuckets cachedin RAM in this bucketvb replica curr itemsNumber of items in "replica" vBuckets in this bucketvb replica ejectNumber of items per second being ejected to disk from"replica" vBuckets in this bucketvb replica itm memoryAmount of replica user data cached in RAM in thisbucketvb replica meta data memoryAmount of replica item metadata consuming in RAM inthis bucketvb replica numNumber of vBuckets in the "replica" state for this bucketvb replica num non residentNumber of non-resident items.vb replica ops createNew items per second being inserted into "replica"vBuckets in this bucketvb replica ops updateNumber of items updated on disk per second for thisbucketvb replica queue ageSum of disk queue item age in milliseconds for "replica"vBucketsvb replica queue drainNumber of replica items per second being written to diskin this bucketvb replica queue fillNumber of replica items per second being put on thereplica item disk queue in this bucketvb replica queue sizeNumber of replica items waiting to be written to disk inthis bucketvb replica resident items ratioPercentage of replica items cached in RAM in thisbucketvb total queue ageSum of disk queue item age in milliseconds.Couchbase Professional Services14

Monitoring: Data Servicexdc opsIncoming XDCR operations per second for this bucketGET Cluster-Wide Individual Bucket StatsThese endpoints are informational and should not be used for monitoring as they are an aggregate for theentire and cluster and the best practice is to monitor each node individually.Insecure: T}/statsSecure: KET}/statsExample: With an average for all samplesBUCKET "travel-sample"curl \--user Administrator:password \--silent \--request GET \--data zoom minute \http://localhost:8091/pools/default/buckets/ BUCKET/stats \jq -r '.op.samples to entries[] select(.key ! "timestamp") .key ": " (.value add / length tostring)'GET Node-Level Individual Bucket StatsEach node in the cluster running the data service should be monitoring individually using the endpoint listedbelow.Insecure: T}/nodes/{NODE}/statsSecure: KET}/nodes/{NODE}/statsExample: Stats for Individual NodeThe following example demonstrates how to retrieve the bucket stats for a specific node.BUCKET "travel-sample"NODE "172.17.0.2:8091"curl \--user Administrator:password \--silent \--request GET \--data zoom minute \http://localhost:8091/pools/default/buckets/ BUCKET/nodes/ NODE/stats \jq -r -c '.op.samples Couchbase Professional Services15

Monitoring: Data Servicejq -r -c '.op.samples "cmd get: " (.cmd get add / length tostring) "\ncmd set: " (.cmd set add / length tostring) "\ncurr connections: " (.curr connections add / length tostring) "\ncurr items: " (.curr items add / length tostring) "\ncurr items tot: " (.curr items tot add / length tostring) "\ndecr hits: " (.decr hits add / length tostring) "\ndecr misses: " (.decr misses add / length tostring) "\ndelete hits: " (.delete hits add / length tostring) "\ndelete misses: " (.delete misses add / length tostring) "\nep bg fetched: " (.ep bg fetched add / length tostring) "\nevictions: " (.evictions add / length tostring) "\nget hits: " (.get hits add / length tostring) "\nget misses: " (.get misses add / length tostring) "\nhit ratio: " (.hit ratio add / length tostring) "\nincr hits: " (.incr hits add / length tostring) "\nincr misses: " (.incr misses add / length tostring) "\nmisses: " (.misses add / length tostring) "\nops: " (.ops add / length tostring)"\nxdc ops: " (.xdc ops add / length tostring)'Key Metrics to MonitorCouchbase MetricDescriptionResponseThese four metricstogether giveinsight into howmemory is used bythe data service.mem used /ep kv sizerepresentsfragmentationwithin the KVengine.mem usedep kv sizeep mem high watout mem used isthe actual memoryutilization whereasep kv size is thesum of themetadata andvalues expected tobe inRAM.mem used /memoryTotalshould be lessthan 90%.ep kv size /ep mem high watrepresents yourCouchbase Professional ServicesThe amount of fragmentation(mem used / ep kv size) you shouldexpect will depend on the workload,but in general, alert if this valueexceeds 115%.If mem used /memoryTotal are consistently near90%, that is a trigger to add additionalmemory or nodes to the cluster. If thisvalue approaches 100%, then youcould face an Out of Memory error andthe Couchbase process could be killedor crash.Once ep kv size ep mem high wat, Couchbase willstart ejecting data to disk. This may beexpected depending on your use case,but caching use cases will always wantep kv size to be lower thanep mem high wat.16

Monitoring: Data Servicequota utilization.ep mem high watis the maximumRAM the bucket isexpected to use.ep meta data memoryThe amount ofmemory usedspecifically fordocumentmetadata. In ValueEjection mode, it'spossible fordocumentmetadata todisplace documentvalues in cache,reducing cache hitrates andincreasinglatencies.Create a baseline forep meta data memory /ep mem high wat. If this valueexceeds 30% andvb active resident items ratio is not100%, consider configuring FullEjection on the bucket.ep queue sizeThe amount ofdata waiting to bewritten to disk. Alarge valuetypically indicatesthe server is diskIO bound. If thisvalue exceeds1,000,000 items,the server will startsending tmp oom(backoff)messages to theapplication.Create a baseline for this value as"normal" will be dependent upon yourworkload and available disk IO. Alert at2x of baseline. You may need to addnodes or increase the per-node diskIO.ep flusher todoThe number ofitems currentlybeing written todisk. Combinedwithep queue size,this represents thetotal disk writequeue on theserver.Create a baseline for this value as"normal" will be dependent upon yourworkload and available disk IO. Alert at2x of baseline.vb avg total queue ageThe average timein seconds that awrite is in queuebefore persistingto disk. Thisrepresents thelocal node'sexposure topotential data loss.Create a baseline for this value as"normal" will be dependent upon yourworkload and available disk IO. Alert at2x of baseline.ep dcp replica items remainingCouchbase Professional ServicesThe number ofitems in the internode replicationqueue. Thisrepresents thecluster's exposureto potential dataCreate a baseline for this value as"normal" will be dependent upon yourworkload and available network IO.Alert at 2x of baseline.17

Monitoring: Data Servicecluster's exposureto potential dataAlert at 2x of baseline.loss.The total numberof KV operationsoccurring againstthe node.Create a baseline for this value as"normal" will be dependent on yourworkload. Alert at 2x of baseline.Abnormally high operations couldmean an unexpected change to theapplication or unusual applicationtraffic patterns.The number of KVGET operationsoccurring againstthe node.Create a baseline for this value as"normal" will be dependent on yourworkload. Alert at 3x of baseline.Abnormally high operations couldmean an unexpected chang

This document describes how to poll the Couchbase REST API to obtain metrics for an external monitoring system, describes which metrics are most important to monitor, and provides guidance on how to interpret those metrics. Obtaining Couchbase Metrics Couchbase exposes monitoring metrics via REST APIs with responses returned in JSON format .