WebSphere MQ Monitoring - HelpSystems

This rule, which runs 24/7, monitors for message AMQ9202 in message file AMQMSG within library QMQM. If any of these messages are found to be present, an alert is raised and a message sent to the Message Console. Possible solutions to this issue include: An FFDC report is generated that may help you diagnose the cause of failure. Use the probe id in the FFDC to search for known problems. The queue manager error logs and @System error log may have other messages related to the failure that may help you with resolution. 80 Error on receive from host &3 An error occurred when receiving data from a host. This may be due to a communications failure. Return codes 10054 (Windows), 73 (AIX), 131 (Solaris), 232 (HP-UX), 104 (Linux) or 3246 (iSeries) means that the connection is reset by peer (ECONNRESET). This usually indicates a problem in the TCP/ IP network. This rule, which runs 24/7, monitors for message AMQ9208 in message file AMQMSG within library QMQM. If any of these messages are found to be present, an alert is raised and a message sent to the Message Console. Possible solutions to this issue include: Use TCP/IP packet and sniffer traces to determine why the reset occurred. 90 Connection to host &3 closed An error occurred receiving data from a host connection. The connection to the remote host has unexpectedly terminated. This is most likely caused by incorrect settings in the /etc/services and INETD.CONF files. This rule, which runs 24/7, monitors for message AMQ9209 in message file AMQMSG within library QMQM. If any of these messages are found to be present, an alert is raised and a message sent to the Message Console. Possible solutions to this issue include: Verify that INETD.CONF and /etc/services files match the name of the queue manager and listener port Try using the runmqlsr command on the remote queue manager (receiver side). 100 A communications error for &4 occurred An unexpected error occurred in communications. This is most likely caused by the inetd configuration file not having the correct information or syntax. This rule, which runs 24/7, monitors for message AMQ9213 in message file AMQMSG within library QMQM. If any of these messages are found to be present, an alert is raised and a message sent to the Message Console. Possible solutions to this issue include: Try using the runmqlsr command on the remote queue manager (receiver side) WebSphere MQ Monitoring template 2-7

110 The &4 responder program could not be started An attempt was made to start an instance of the responder program, but the program was rejected. The most common cause of this issue is that the inetd configuration file did not have the correct information or syntax. This rule, which runs 24/7, monitors for message AMQ9228 in message file AMQMSG within library QMQM. If any of these messages are found to be present, an alert is raised and a message sent to the Message Console. Possible solutions to this issue include: Try using the runmqlsr command on the remote queue manager (receiver side) 120 Channel negotiation failed Channel between this machine and the remote machine could not be established due to a negotiation failure. This error can be raised for many reasons. This rule, which runs 24/7, monitors for message AMQ9503 in message file AMQMSG within library QMQM. If any of these messages are found to be present, an alert is raised and a message sent to the Message Console. Possible solutions to this issue include: An FFDC report is generated that may help you diagnose the cause of failure. Use the probe id in the FFDC to search for known problems. The queue manager error logs and @System error log may have other messages related to the failure that may help you with resolution. 130 Message sequence number error for channel &3 The local and remote queue managers do not agree on the next message sequence number. A message with sequence number &1 has been sent when sequence number &2 was expected. The most common causes of this issue are: Manually resetting a receiver channel message sequence number Deleting and redefining a channel Two instances of the same receiver channel This rule, which runs 24/7, monitors for message AMQ9526 in message file AMQMSG within library QMQM. If any of these messages are found to be present, an alert is raised and a message sent to the Message Console. Possible solutions to this issue include: Never reset a receiver channel’s message sequence number. 2- 8 Halcyon Specialist Monitoring

C HAPTER WEBSPHERE MQ MONITORING CHAPTER3 MQ Monitoring Record & Playback Scripts Halcyon’s Record & Playback is used to run 4 specially created scripts to monitor WebSphere’s MQ Channels. Providing that you have Record & Playback installed on your system, at the command line type: GO HRP400/HRP Figure 3.1 Starting Halcyon Record & Playback WebSphere MQ Monitoring 3-9

The Record & PlayBack main menu is displayed. Figure 3.2 Record & Playback main menu Select Option 3 Edit Scripts to open the Edit Scripts display. Figure 3.3 Edit Scripts display

Within this display are the four scripts which monitor WebSphere MQ. They are: CHKMQM CHKMQMCHL CHKMQMCHLG CHKMQMQ CHKMQM (WebSphere MQ Manager Checking) Script The CHKMQM script checks the queue managers to ensure that they are active. In its default format, this script checks for two queue managers. If you have more (or less) than two queue managers you must edit the script to include the additional managers. Figure 3.4 CHKMQM script If a queue manager is found to be in an inactive status, the following program is called: HRPDATA/SNDMGRSTS This sends the Message ID: MQM0003 to Message Queue: QSYSOPR WebSphere MQ Monitoring 3-11

The message text of this message ID reads: MQ Queue Manager &1 is &2 where: &1 represents the Queue Manager name &2 represents the current status CHKMQMCHL (Active MQ Channels Running) Script The CHKMQMCHL script checks that MQ Channels that are meant to be active, are running. As this script checks for specific channels, it requires editing with the correct channel configuration for your operating environment. You may need to add further channels as required Figure 3.5 CHKMQMCHL script The MQ Manager Name is identified in line 7 of this script: 0007.00 K 20/7 'WRKMQMCHL MQMNAME('VLEARTHQM')' Lines 15, 28 and 35 of this script identify the MQ Channels that exist within the MQ Manager. 0015.00 P IF &SCR(&Y/8:20) 'FIPVP01.TO.NEPTUNE ' 0028.00 P IF &SCR(&Y/8:20) 'VLMERCURY.TO.VENUS ' 0035.00 P IF &SCR(&Y/8:20) 'VLSATURN.TO.VLCPAD '

Note: Remember to edit the MQ Manager Name and Active MQ Channels within this script to replicate your own environment. If a channel is found to be inactive, the following program is called: HRPDATA/SNDCHLSTS This sends the Message ID: MQM0002 to Message Queue: QSYSOPR The message text of this message ID reads: MQ Channel &1 has status &2 where: &1 represents the Channel name &2 represents the current status CHKMQMCHLG (ALL MQ Channels Running) Script This is basically the same as the CHKMQMCHL script but it checks ALL MQ Channels rather than just those that specified within the CHKMQMCHL script. Figure 3.6 CHGMQMCHLG Script WebSphere MQ Monitoring 3-13

Note: Remember to edit this script to reflect your own operating environment configuration. CHKMQMQ (Message Queue Depth) Script This script checks the Message Queue Depth on the System Dead Letter Queue and any other queue beginning with Q*. Figure 3.7 CHKMQMQ Script There are two separate checks made against two variables: &LOWCOUNT 500 &HIGHCOUNT 5000 These limits can be amended so that they are more representative of the queue depths experienced in your environment. In both cases, an alert is raised if the queue depth exceeds the specified value. The first criteria (&LOWCOUNT) is designed as a type of ‘early-warning’ safeguard to the second (&HIGHCOUNT) criteria. Using the &LOWCOUNT criteria as an example: If the depth of System Dead Letter Queue exceeds 500, the following program is called: HRPDATA/SNDMQDEPTH 3- 1 4 Halcyon Specialist Monitoring

This sends the Message ID: MQM0001 to Message Queue: QSYSOPR The message text of this message ID reads: MQ Message Queue &1 at depth &2 where: &1 represents the Message Queue name &2 represents the current queue depth Editing the script for remaining message queues: Page down the script so that line 48 is visible: 0048.00 K 20/7 'WRKMQMQ QNAME(Q*)' Edit this line to determine the additional message queues to be monitored. WebSphere MQ Monitoring 3-15

C HAPTER SETTING MQ MONITORING RULES CHAPTER4 Once the Record and Playback Scripts have been amended for your own environment, you need to set a Message Queue rule in order that the generated messages cause actions to be invoked. Providing that you are using Halcyon’s Level 2 - Systems Operations Suite or above, you can additionally set Performance Rules to ensure that subsystems are active and that MQ jobs are running. Creating the MQ Message Rule 1 Log onto your Halcyon Suite and operating environment as usual: GO HALxxxx/HALCYON (where xxxx represents the name of your environment). 2 From the main menu select option 5 Work with Rules. 3 With the cursor positioned next to the Message Queue Rules, press F11 to expand the display. 4 Position the cursor next to the QSYSOPR Message Queue and press F11 again to display the rules that already exist in this group. 5 Page Down and position the cursor against the last entry in this Rule Group. 6 Type 1 and press ENTER to insert a new rule into this Rule Group. You are now ready to enter the details of the rule. 1 Type the Rule Description as All HRP Messages 2 Press Page Down. 3 Type 2 against the existing entry to change the rule detail. Press ENTER 4- 1 6 Halcyon Specialist Monitoring

4 Tab down to the Library parameter and type HRPDATA over the existing entry. 5 Move the cursor back up to the Message File parameter and press F4. Note: 6 From the pop-up display, select the HRPALTMSGF entry. Press ENTER . At this point a warning message, advising that the entered criteria may cause unpredictable results is displayed. This can be ignored. Press Enter again. Figure 4.1 Amending Message Queue Rule detail 7 Against the newly amended rule, select option 8 Count/Method to change the current count method. Press ENTER . 8 Change the Message Count method from *JOBNBR to *MSGDTA. Press ENTER . You are now ready to start adding actions to this rule. Adding Actions This rule will have three actions applied in order to: Notify you of the alert Reset the rule Delete any repeats of the alert occurring within the following 60 minute period Setting MQ Monitoring Rules 4-17

Adding a Console action This action sends a notification of the rule alert to a Console view of your choice. 1 Position the cursor so that it is in the lower half of the Add Message Queue Action display (under the Actions line). Press F6. This opens the Add Message Queue Action display. 2 In the Action Type field, type CONSOLE and press ENTER . 3 In the System parameter, leave the setting as *LOCAL to send the alert to the iSeries Message Console or specify the name of the Remote Location on which another Message Console or Enterprise Console version is running. 4 Press ENTER to confirm. Adding a Reset Rule action This action resets the rule following an alert being sent to the console. 1 Position the cursor so that it is in the lower half of the Add Message Queue Action display (under the Actions line). Press F6. This opens the Add Message Queue Action display. 2 In the Action Type field, type RSTRULE and press ENTER . 3 In the Delay parameter, type 3600 to specify the delay before the reset rule action runs, as being 60 minutes. 4 Press ENTER to confirm. Adding a Delete Alert action This action deletes the alert providing all actions have been completed without error. 1 Position the cursor so that it is in the lower half of the Add Message Queue Action display (under the Actions line). Press F6. This opens the Add Message Queue Action display. 2 Change the Action Sequence Number to 20. 3 In the Action Type field, type DELETE and press ENTER . 4 Press ENTER to confirm. Changing the Console Count To prevent multiple alerts being sent to the Console, change the message count to 1, so that the rule only triggers on the first occurrence of the alert being raised. To do this: 1 Position the cursor next to the Console action in the list of actions. Type 8 and press ENTER . 2 Change the ‘Perform action for’ parameter to 1. Press ENTER . 3 The Console action (and its associated Reset Rule action) now have a count of 1 applied. 4- 1 8 Halcyon Specialist Monitoring

Press ENTER to complete the creation of the Message Queue rule. Figure 4.2 Message Queue Rule with associated actions Creating Performance Rules Note: Performance Rules are used to ensure that critical subsystems are active and that required MQ Jobs are running. You must have Halcyon Level 2 - Systems Operations Suite or higher to be able to set Performance Rules. 1 Log onto your Halcyon Suite and operating environment as usual: GO HALxxxx/HALCYON (where xxxx represents the name of your environment). 2 From the main menu select option 5 Work with Rules. 3 With the cursor positioned next to Performance Rules, type 1 to create a new Performance Rule Group. 4 Enter the Performance Group Name as HRP and the Description as HRP Performance Rules. Press ENTER to confirm. Setting MQ Monitoring Rules 4-19

You are now ready to create the first rule. Checking the WebSphere MQ Subsystem is active 1 Type 1 against the HRP Performance Rules Group to insert a new rule. Press ENTER . 2 Enter the Rule description as WebSphere MQ Subsystem Monitoring. Press PAGE DOWN . 3 Type 2 against the existing entry to change the rule detail. Press ENTER . 4 Change the Rule Type to *SUBSYSTEM. Press ENTER . 5 Enter the Subsystem Name as QMQM and the Value as *INACTIVE. Press ENTER 6 Position the cursor so that it is in the lower half of the Add Message Queue Action display (under the Actions line). Press F6. This opens the Add Performance Action display. 7 Add an action that is suitable for your notification requirements. Press ENTER . 8 Add further actions as required. Checking MQ Listener Jobs This rule checks that the WebSphere MQ Listener Jobs are running as expected. 1 Type 1 against the HRP Performance Rules Group to insert a new rule. Press ENTER . 2 Enter the Rule description as Check MQ Listener Jobs. Press PAGE DOWN . 3 Type 2 against the existing entry to change the rule detail. Press ENTER . 4 Change the Rule Type to *JOB. Press ENTER . 5 Change the Job Name from *ALL to RUNMQLSR. Tab down to the Subsystem parameter. 6 Enter the Subsystem as QMQM. Tab down to the Statistic parameter. 7 Enter the Statistic as JOBCOUNT. Tab down to the Comparison parameter. 8 Enter the Comparison as *LT (Less than). Tab down to the Value parameter. 9 Enter the values as 1 (or the number of MQ listeners that currently operate in your WebSphere environment. Press ENTER . 10Position the cursor so that it is in the lower half of the Add Message Queue Action display (under the Actions line). Press F6. This opens the Add Performance Action display. 11Add an action that is suitable for your notification requirements. Press ENTER . 12Add further actions as required. 4- 2 0 Halcyon Specialist Monitoring

Checking WebSphere MQ Jobs This rule checks that there are Jobs running within the MQ Subsystem. 1 Type 1 against the HRP Performance Rules Group to insert a new rule. Press ENTER . 2 Enter the Rule description as Check WebSphere Jobs. Press PAGE DOWN . 3 Type 2 against the existing entry to change the rule detail. Press ENTER . 4 Change the Rule Type to *JOB. Press ENTER . 5 Leave the Job Name as *ALL. Tab down to the Subsystem parameter. 6 Enter the Subsystem as QMQM. Tab down to the Statistic parameter. 7 Enter the Statistic as JOBCOUNT. Tab down to the Comparison parameter. 8 Enter the Comparison as *LT (Less than). Tab down to the Value parameter. 9 Enter the values as 1 (or the number of MQ listeners that currently operate in your WebSphere environment. Press ENTER . 10Type 2 against the existing entry to change the rule detail again. Press ENTER . 11Tab down to the Allow alerts when subsystem not active parameter and change the entry from *YES to *NO. Press ENTER . 12Add an action that is suitable for your notification requirements. Press ENTER . 13Add further actions as required. Performance Rules are now complete. Use the Chapter 5 - “Checking/Editing CL Source and Message File” to amend any CL Source and Message File configurations Setting MQ Monitoring Rules 4-21

C HAPTER SCHEDULING THE RUNNING OF THE RECORD & PLAYBACK SCRIPTS CHAPTER5 In order that the Record & Playback scripts monitor WebSphere MQ on a regular and ongoing basis, a schedule must be created and applied to ensure consistent monitoring. In best practice and for ease-of use, Halcyon recommends that the schedule is set up using Halcyon Advanced Job Scheduler (available in Halcyon’s Level 4 Operations Center Suite or as a standalone product). Otherwise, a TCP/IP rule can be created (using a false IP Address) that then causes a command action to be run that invokes the Record & Playback scripts. Creating a TCP/IP rule to schedule the Record & Playback scripts To create a TCP/IP schedule rule: 1 From the Halcyon Suite main menu select option 5 Work with Rules. 2 Position the cursor next to the TCP/IP rule. Type 1 and press ENTER . 3 Type HRPSched as the Group Name and HRP Scheduler as the Description. Press ENTER . 4 Against this newly created group, type 1 (to insert a new rule) and press ENTER . 5 Type HRP Script Scheduling as the Rule Description. Tab down to the Check Interval parameter and ensure the entry is set to 300. Press PAGE DOWN . 6 Type 2 against the existing entry to change the rule detail. Press ENTER . 7 Tab down to the Remote Internet Address field and type 1.2.3.4. (or any other IP address that does not exist). Enter the Description as HRP Scheduler. 5- 2 2 Halcyon Specialist Monitoring

8 Tab down to the Wait Time parameter and change the current entry to 1 (must be right-aligned). Press ENTER . 9 Posi

WebSphere MQ, a member of the WebSphere family from IBM, is the most popular system for cross- platform messaging, providing assured delivery of messages across 35 plus IBM and non-IBM platforms, including IBM mainframe and midrange, Windows, Linux, and UNIX. WebSphere MQ is often referred to as "MQ" or "MQ Series".