Troubleshooting CPS VDRA

Troubleshooting CPS vDRADiameter Error Codes and ScenariosResult-CodeResult-Code ValueDescriptionDIAMETER INVALID AVP VALUE 5004-DIAMETER MISSING AVP5005Mandatory AVP in request message ismissing.DIAMETER RESOURCES5006A request was received that cannot beauthorized because the user has alreadyexpended allowed resources. An example ofthis error condition is a user that is restrictedto one dial-up PPP port, attempts to establisha second PPP connection.EXCEEDEDDIAMETER CONTRADICTING 5007AVPSServer has identified that AVPs are presentthat are contradictory to each other.DIAMETER AVP NOT ALLOWED 5008Message is received by node (Server) thatcontain AVP must not be present.DIAMETER AVP OCCURS5009If message contains the a AVP number oftimes that exceeds permitted occurrence ofAVP in message definition.5010In response of CER if no common applicationsupported between the peers.5011Self explanatory.5012Message rejected because of unspecifiedreasons.5013When an unrecognized bit in the Diameterheader is set to one.5014Self explanatory.5015Self explanatory.5016For example, marking AVP to Mandatorywhile message definition doesn't say so.5017In response of CER if no common securitymechanism supported between the peers.TOO MANY TIMESDIAMETER NO COMMONAPPLICATIONDIAMETER UNSUPPORTEDVERSIONDIAMETER UNABLETO COMPLYDIAMETER INVALID BITIN HEADERDIAMETER INVALIDAVP LENGTHDIAMETER INVALIDMESSAGE LENGTHDIAMETER INVALID AVPBIT COMBODIAMETER NO COMMONSECURITYTroubleshooting CPS vDRA8

Troubleshooting CPS vDRAPolicy DRA Error CodesPolicy DRA Error CodesNon-compliant Diameter requests are checked for errors in routing AVP and P-bits. The following tabledescribes the error codes and the reasons for errors in Diameter requests:Table 2: Policy DRA Error CodesPolicy DRA Error StringError CodeSub-codeDescriptionNo application routefound3002001Route List AvailabilityStatus is “Unavailable”Timeout triggered3002002Timeout triggeredNo peer group3002003No peer groupSession DB Error3002004Session DB ErrorBinding DB Error3002005Binding DB ErrorNo key for binding lookup 3002006No key for binding lookupBinding not found3002007Binding not foundMessage loop detected3005008Message loop detectedParsing exception withmessage3009009Parsing exception withmessageCRD DB Error3002010CRD DB ErrorRetries exceeded3002011Retries exceededNo peer route3002012No peer routing rulefound for a Realm-only ornon-peer Destination-HostP-bit not set3002013P-bit in the Requestmessage is set to “0”Missing Origin-Host AVP 5005014Mandatory Origin-HostAVP missingMissing Origin-RealmAVP5005015Mandatory Origin-RealmAVP missingMissingDestination-Realm AVP5005016MandatoryDestination-Realm AVPmissingNo avp found in requestfor SLF lookup type3002101No avp found in requestfor SLF lookup typeSLF DB Error3002102SLF DB ErrorTroubleshooting CPS vDRA9

Troubleshooting CPS vDRADefault HTTP Error CodesPolicy DRA Error StringError CodeSub-codeDescriptionSLF credential not found 3002in DB103SLF credential not foundin DBSLF Destination type not 3002found in DB104SLF Destination type notfound in DBDestination not found inSLF Mapping Table3002105Destination not found inSLF Mapping TableBinding DB Overload3002022Binding record limitexceededDefault HTTP Error CodesYou can configure the HTTP response error code (such as 4xx, 5xx) corresponding to each vDRA Rest APIJSON error response code for the GET binding (for example imsi, imsiApn, msisdn, msisdnApn, ipv4, ipv6).For more information about the CRD, see the CPS vDRA Configuration Guide.If you do not configure the Rest API HTTP Error Code in the CRD, vDRA uses the default HTTP error codesfor GET binding Rest API.The following table lists the default HTTP error codes:Table 3: Default HTTP Error CodesvDRA Rest API Error CodeHTTP Error CodeHTTP Reason-Phrase1001 (INTERNAL ERROR)500Internal Server Error2014 (DATA NOT FOUND)404Not Found2019 (INVALID API FORMAT) 400Bad RequestDebug ping / ping6Run the following commands to check ping connectivity from the VM to other nodes using IPv4 and IPv6:# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# debug ping wtc2b1fdrd02v –n IPv4 address admin@orchestrator# debug ping6 wtc2b1fdrd02v -n IPv6 address Where: -n:Debug tracerouteRun the following commands to check traceroute connectivity from the VM to other nodes:Troubleshooting CPS vDRA10

Troubleshooting CPS vDRADebug tcpdumpIPv4:# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# debug traceroute VMHOST IPv4address IPv6:# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# debug traceroute VMHOST -6 IPv6address Debug tcpdumpUse the following command to get packet capture from the VM. Specify interface and port details to avoidbig packet capture files.If you use the -i any option, you may see the same packet twice: once as it traverses the VMs interface, andagain when it traverses the Docker container's virtual interface.# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# debug tcpdump wtc2b1fdrd01v test.pcap60s -s 0 -i ens162 port 3868admin@orchestrator# debug packet-capture gather directory test debugadmin@orchestrator# debug packet-capture purgeYou can download the packer capture file from : https:// master ip /orchestrator/downloads/ after loggingin to https:// master ip /After you download the file, delete the packet capture files to clean up the disk space.Monitoring Application LogsUse the following commands to monitor application logs :# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# monitor log applicationDebug Tech to Capture LogsRun the following command to capture SVN, CRD, logs, and save it at http:// masterip /orchestrator/downloads/debug/tech/:# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# debug techMonitoring Container LogsUse the following command to monitor specific container logs:Troubleshooting CPS vDRA11

Troubleshooting CPS vDRAMonitoring Orchestrator Logs# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# monitor log container container-name Monitoring Orchestrator LogsUse the following command to monitor orchestrator logs during an upgrade/downgrade:# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# monitor log container orchestrator include AUDITIf the CLI is not accessible or is giving errors when executing commands, use the following command fromthe master VM for more information:cps@ {DRM-hostname}: docker logs orchestratorChange CLI User PasswordIf you know the existing password, use the following steps to change the user password in CLI:# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# aaa authentication users user fpasapi change-passwordValue for 'old-password' ( string ): *****Value for 'new-password' ( string ): ********Value for 'confirm-password' ( string ): ********If you do not know the password, use the following commands to reset the password:# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# configadmin@orchestrator(config)# aaa authentication users user fpasapi gid 100uid 9000 homedir "" ssh keydir "" password password admin@orchestrator(config-user-apiuser)# commitCommit complete.admin@orchestrator(config-user-apiuser)# endRestart Docker ContainerIf the commands show docker service or system diagnostics show errors, check the docker service forany unhealthy processes. If there are unhealthy processes, use the command monitor container logs toview logs and then restart the docker container.Action# node: DRA Master / DB Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# show docker service tab exclude HEALTHYadmin@orchestrator# show system diagnostics tab exclude passing# container-name is unhealthy process container id.admin@orchestrator# docker restart container-id container-name Troubleshooting CPS vDRA12

Troubleshooting CPS vDRACheck DNS ConfigCheck DNS ConfigCheck the VMs dnsmasq file to verify whether the DNS entries are present; if not, perform the followingsteps:# node: DRA Master# user: cpscps@ {DRM-hostname}: cat /data/dnsmasq/etc/dnsmasq.conf# If DNS entries are missing, perform the following steps:cps@ {DRM-hostname}: cliadmin@orchestrator# show running-config network dns save /data/config/dns cli backupadmin@orchestrator# configadmin@orchestrator(config)# no network dnsadmin@orchestrator(config)# commitadmin@orchestrator(config)# endadmin@orchestrator# configadmin@orchestrator(config)# load merge /data/config/dns cli backupadmin@orchestrator(config)# commitadmin@orchestrator(config)# endadmin@orchestrator# exitcps@ {DRM-hostname}: cat /data/dnsmasq/etc/dnsmasq.confRedeploy Master VMWhen the master VM is deleted or redeployed for some reason, you must make it part of the existing cluster.workaround to make it part of the cluster as described in the following steps:# node: DRA Master# user: cps# After the master VM is redeployed, log into the master VM, and wait til# cpsinstall is completecps@ {DRM-hostname}: journalctl -u cpsinstall.service –f# Verify that the following log apperas: log date time stamp master-0bootstrap.sh[1521]: Install script completed.# Once cpsinstall is finished; execute the following# commands on the master VM in the order specified.cps@ {DRM-hostname}: docker stop (docker ps -a -q)cps@ {DRM-hostname}: docker rm (docker ps -a -q)cps@ {DRM-hostname}: weave launch-router --ipalloc-init consensus 3cps@ {DRM-hostname}: sudo rm -rf /data/orchestratorcps@ {DRM-hostname}: sudo rm /var/cps/bootstrap-statuscps@ {DRM-hostname}: sudo /root/bootstrap.shcps@ {DRM-hostname}: ssh-keygen -f "/home/cps/.ssh/known hosts" -R[localhost]:2024Remove MongoDB Replica Set MemberPerform the following steps to remove a replica set member from MongoDB.CautionThe command nocorrect.database clusterdeletes the configuration completely, so ensure the information is# node: DRA Master# user: cpscps@ {DBM-hostname}: cliTroubleshooting CPS vDRA13

Troubleshooting CPS vDRAMonitoring MongoDB Logscps@ {DBM-hostname}: configadmin@orchestrator(config)# no database cluster binding shardbinding-shard-1 shard-server fn6-1a1bs1kadmin@orchestrator(config)# commitadmin@orchestrator(config)# end#connect to the replica set primary member container to remove the node, take a note#of port of the replica setcps@ {DBM-hostname}: docker connect mongo-s104root@mongo-s104:/# mongo --port 27033rs-binding-shard-1:PRIMARY rs.status()#Note the name of the member from rs status output and then input it to#rs.remove to remove the memberrs-binding-shard-1:PRIMARY nitoring MongoDB LogsThe MongoDB logs are stored under /data/mongod-node/db on every VM that has mongod instancerunning.Clean the DatabasePerform the following steps if you want to clean the database and recreate a fresh database.WarningAll the data will be lost.# node: DRA Master# user: cpscps@ {DBM-hostname}: cli# Stop all the application process:cps@ {DBM-hostname}: system stop# Wait for some time till all the application proceses stop.# You can check the process using the commands:# show scheduling status and show system status# Repeat the following steps in all the database nodescps@ {DBM-hostname}: rm -rf /data/configdb/*cps@ {DBM-hostname}: rm -rf /data/mongod-node/db/*cps@ {DBM-hostname}: rm -rf /mmapv1-tmpfs- port /*cps@ {DBM-hostname}: cli# Restart the system:cps@ {DBM-hostname}: system startReset the CLI ConfigurationPerform the following steps to reset the CLI configuration:CautionThe complete configuration will be reset.# node: DRA Master# user: cpscps@ {DRM-hostname}: docker exec -it orchestrator bashcps@ {DRM-hostname}: /var/confd/bin/confd load -D -m -l/data/cdb//*.xmlTroubleshooting CPS vDRA14

Troubleshooting CPS vDRAPolicy DRA Logger LevelsPolicy DRA Logger LevelsPolicy DRA Application logs are available for debugging purposes.Note that turning on logs in a production system can have a substantial impact on the system performanceand is not recommended.Enabling and Disabling LogsUse the orchestrator CLI to enable and disable application logs.admin@orchestrator# logger set ?Possible completions: logger name admin@orchestrator# logger set com.broadhop.dra.service ?Possible completions:debug error info off trace warnadmin@orchestrator# logger clear com.broadhop.dra.service ?Possible completions: cr View Log LevelsThe different log levels in the order of increasing details in the log are: Error (error logs) Warn (error and warning logs) Info Debug Trace (all logs)The default log level is warn.Use the following orchestrator CLI command to view the current log levels set on a per application modulebasis.admin@orchestrator# show logger levelLoggerCurrent jdiameterwarnView LogsTo view application logs continuously similar to the tail–fcommand, use the following command:“monitor log application”Troubleshooting CPS vDRA15

Troubleshooting CPS vDRACommon LoggersTo view application logs that were previously collected in a consolidated log file (similar to the more command),use the following command:show log applicationCommon LoggersThe following table describes the different loggers and their default log level:Table 4: Common LoggersLogger NameDescriptionDefault Log Levelcom.broadhop.dra.servicePolicy DRA application logs. This warndisplays logs from various modulesof the Policy DRA system.dra.tracePolicy DRA audit logs. Thiswarndisplays a summary of the Diametermessage request and responseprocessing.org.jdiameterjDiameter module logs. Thiswarndisplays logs from various modulesof the jDiameter module.com.broadhop.dra.session.Checks and deletes stale sessions. ckcom.broadhop.dra.service.mongo.sharding.implThis logger provides logs about the warnbinding and API handlingoperations managed by the Worker.com.mongodbLogging related to MongoDBlibrary as the Worker invokesMongoDB API for databaseoperations.com.broadhop.dra.servicevDRA routing-related messages to warndebug issues in ubleshooting CPS vDRA16vDRA stack-related logs to enable warndebugging at stack level. To beused with org.jdiameter log level.vDRA logs related to controlmessaging.warnwarn

Troubleshooting CPS vDRACommon Troubleshooting StepsCommon Troubleshooting StepsCPS vDRA LogsStep 1Use the following command in CLI to view the consolidated application logs.admin@orchestrator[master-0]# show log applicationStep 2Use the following command in CLI to view the consolidated engine logs.admin@orchestrator[master-0]# show log engineCounters and StatisticsCheck for statistics generated at pcrfclient01/02 in /var/broadhop/stats and counters in beans at jmxterminal.System Health ChecksView System StatusUse the following command to view the system status and verify whether the system is running, or if anyupgrade or downgrade is in progress, and whether it is 100% deployed.APP VNF# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# show system statusDB VNF# node: DRA DB Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# show system statusIf system is not 100% deployed, use the following command to view the current scheduling status: systemscheduling statusView System DiagnosticsUse the following command to view the system disgnostics and debug failed processes.APP VNF# node: DRA Master# user: cpsTroubleshooting CPS vDRA17

Troubleshooting CPS vDRACheck System Scheduling Statuscps@ {DRM-hostname}: cliadmin@orchestrator# show system diagnostics tab exclude passingDB VNF# node: DRA DB Master# user: cpscps@ {DBM-hostname}: cliadmin@orchestrator# show system software tabadmin@orchestrator# show system diagnostics tab exclude passingYou can monitor the log of the container using the command: monitorcontainer logsCheck System Scheduling StatusUse the following command to verify the installer scheduler status. The scheduler must reach haproxy-int-api1 500 and all states indicate running.APP VNF# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# show scheduling statusDB VNF# node: DRA DB Master# user: cpscps@ {DBM-hostname}: cliadmin@orchestrator# show scheduling statusCheck Docker EngineUse the following commands to check the docker engine: show docker engineare connected. tab:Check docker engine connected status to verify whether all VM engines show running-config docker tab: Check the running configuration of the docker to verify whetherall VMs are registered to the Master VM correctly and whether all VMs are shown with internal IP andscheduling slots.APP VNFCommand:# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# show docker engine tabCommand:# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# show running-config docker tabTroubleshooting CPS vDRA18

Troubleshooting CPS vDRACheck Docker ServiceDB VNFCommand:# node: DRA DB Master# user: cpscps@ {DBM-hostname}: cliadmin@orchestrator# show docker engine tabCommand:# node: DRA DB Master# user: cpscps@ {DBM-hostname}: cliadmin@orchestrator# show running-config docker tabCheck Docker ServiceUse the following commands to check the docker service: showdocker service tab:to verify whether all the docker services are running. showdocker service tab exclude HEALTHY: to view unhealthy docker services.APP VNFCommand:# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# show docker service tabbCommand:# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# show docker service tab exclude HEALTHYDB VNFCommand:# node: DRA DB Master# user: cpscps@ {DBM-hostname}: cliadmin@orchestrator# show docker service tabCommand:# node: DRA DB Master# user: cpscps@ {DBM-hostname}: cliadmin@orchestrator# show docker service tab exclude HEALTHYView Alert StatusCheck the alert status in both VNFs and verify that there are no issues.Troubleshooting CPS vDRA19

Troubleshooting CPS vDRATroubleshooting ApplicationAPP VNF# node: DRA Master# user: cpscps@ {DRM-hostname}: cliadmin@orchestrator# show alert status tab include firingDB VNF# node: DB Master# user: cpscps@ {DBM-hostname}: cliadmin@orchestrator# show alert status tab include firingTroubleshooting ApplicationCall FailuresIn case of call failures, check the Peer Connection, Binding Monitoring, Peer Errors, Error Result Code inCentral GUI as described:1. Log into the Central GUI as admin.In the Peer Monitoring, filter by the host where call failures are observed.If there is any problem with con

cps@ {DRM-hostname}: journalctl -u cpsinstall.service –f # Verify that the following log apperas: log date time stamp master-0 bootstrap.sh[1521]: Install script completed. # Once cpsinstall is finished; execute the following # commands on the master VM in the order specified. cps@ {DRM-hostname}: docker stop (docker ps -a -q)