IBM Tivoli Web Response Monitor And IBM Tivoli Web

v Leveraging of the ITM KBB trace set feature that enables customizable totalnumber of files, file size, and file reuse. This allows capturing ofhard-to-reproduce and long-running problems with predetermined disk spacerequirements.CPU usage managementThe fixpack provides the following improvements to CPU management:v Allows you to specify maximum Analyzer CPU utilization limitsv The Analyzer monitors its own CPU usage and adjusts internal processingcontrol by applying heuristic rules and algorithms, maintaining CPUconsumption within an 85% to 100% range.v The Analyzer produces the best possible monitoring transaction throughput perresource availability and CPU constraints.v The default Analyzer behavior is no limit.New CPU usage management parametersThe following are the new CPU usage management parameters:v KFC CPU ENFORCE MAX LIMITv KFC CPU TARGET THRESHOLDv KFC CPU MANAGE PERIODv KFC CPU ACTION INTERVALSee a description of these parameters at “Parameters affecting CPU usagemanagement” on page 16.Storage utilization improvementsThe fixpack provides the following improvements to storage utilization:v Readjustment of headers, data, sessions, and work buffer sizes to fit multiplebuffers into a 4K pagev Initial allocation of multiple pages of buffers per poolv Allocation of frequent recycled header and data buffers by page instead of byindividual bufferv Because buffers are used and returned in random order, the Analyzer neverreleases buffers from initial allocation sets while performing buffer contractionsince they are in contiguous page storage.v Under high network loads, tens of thousands of TCP connections come and goper minute. Changing the background task session clean up frequency resolveslarge session pool allocation problems.v Network activities always take place in bursts, even under very high networkloads. Frequent buffer pools contractions as a method to reduce overall storageusage actually causes high storage fragmentation. Thus, the Analyzer's methodof reducing the buffer pools contraction frequency and contraction percentageavoids thrashing and helps to preserve steady state buffer pool allocation sets.v Most systems handle similar network loads every day. The Analyzercontinuously maintains calculated system steady state buffer pool allocationcharacteristics and saves them across product restarts. This enables the Analyzer,based on actual runtime history, to initially allocate contiguous buffers in storagethat closely match steady-state needs, thereby requiring little buffer poolexpansion or contraction and avoiding the possibility of storage fragmentation.Chapter 3. Product Enhancements7

v On UNIX systems, captured frames are stored in capture buffers allocated pernetwork MTU size. However, the majority of packets are smaller than themaximum negotiated size. Allocation of capture buffers based on actual framesize helps to reduce Analyzer storage requirements.Throughput efficiencyThe fixpack provides the following improvements in throughput efficiency:v Implementation of dynamic and customizable packet captures on UNIX systemsv Increase of network protocol processing parallelism by implementation ofconfigurable protocol task thread poolsv Reduction of response time data filter processing and formatting of KFC1 APItransmission data delays by implementing a configurable output task threadpoolv Improvement of KFC1 communication I/O efficiencyNew WRM Collector environment variable:SM3 LOG SAMPLE PERCENTAGEThis new environment variable, specified in the kflmenv file, allows the specifiedpercentage of transactions to be logged and inserted into the WRM database. It canbe used to control the volume of information recorded. The default is 100%. Theunit is an integer. For example:SM3 LOG SAMPLE PERCENTAGE 75New WSA Engine features and enhancementsThe following are new WSA Engine features and enhancements provided by thefixpack:Shared filter enhancementThis performance enhancement reduces system resource utilization for both theAnalyzer Agents and WSA Engine and significantly decreases network activitybetween all Analyzer Agents and the WSA Engine. This is accomplished by sharingapplication segment data which is common to multiple WSA Engine transactiondefinitions. At WSA Engine startup, these common application segments aredetermined and then a single transaction filter is defined for each commonapplication segment to the appropriate Analyzer Agent. Each time the transactionfilter data is sent to the WSA Engine, it is shared by all interested transactiondefinitions.Filter data retention algorithmThis performance enhancement reduces the number of filter data answer areas thatmust be retained during Engine transaction profiling activity. This lowers storageutilization and decreases internal processing queue lengths.New default value forTRANSACTION AGE INTERVAL SECONDS parameterDue to improved Analyzer Agent transaction filter data delivery, this existingglobal WSA Engine configuration file parameter should be set to 15. This decreasesthe number of filter data answer areas that are retained during Engine transactionprofiling activity. The default value is now 15 seconds. For example:8IBM Tivoli Web Response Monitor and IBM Tivoli Web Segment Analyzer: Notes for Fixpack 1.2.0-TIV-W3 Analyzer-IF0003

TRANSACTION AGE INTERVAL SECONDS 15New parameter: HTTP TRANSACTIONSThis new WSA Engine configuration file parameter allows URLs for HTTPtransactions to not be recorded if so desired. It is specified at the individualtransaction definition level. The default is Yes to record the URL for HTTPtransactions. Indicate either Yes or No. For example:HTTP TRANSACTIONS NoNew parameter: EXCLUDE RESPONSE TIME RANGEThis new Engine configuration file parameter enables unwanted transactions suchas a maintenance transaction like a long running heartbeat to be excluded fromresponse time averages so that these averages are not skewed. It is specified at theindividual TIER or OUTBOUND level within a transaction definition. There are nodefaults for this parameter. A comma separated minimum and maximum value inmilliseconds must be specified. The following example will ignore any transactionfrom 59 to 61 seconds, inclusive:EXCLUDE RESPONSE TIME RANGE 59000, 61000Chapter 3. Product Enhancements9

10IBM Tivoli Web Response Monitor and IBM Tivoli Web Segment Analyzer: Notes for Fixpack 1.2.0-TIV-W3 Analyzer-IF0003

Chapter 4. Best Practices in High Load EnvironmentsThe Analyzer component captures and processes network data for monitoringapplication transactions. Depending upon your environment and applications'characteristics, the Analyzer might process millions of network packets per minute.The Analyzer would require considerable amounts of system storage and CPUcycles for its operation under high network and transaction workload conditions.The work demand is beyond the Analyzer's control, and frequently, it is alsobeyond your control or predication. Furthermore, network activities always occurin bursts. The Analyzer must be able to handle sudden bursts of transactionactivities that are far above average steady-state expected volumes andthroughputs.The Analyzer accomplishes high load demand through a balanced approach of thefollowing five strategies:v Increasing process parallelismSee “Increasing process parallelism.”v Optimizing storage utilizationSee “Optimizing storage utilization” on page 12.v Minimizing output delaySee “Minimizing output delay” on page 13.v Improving I/O efficiencySee “Improving I/O efficiency” on page 14.v CPU usage managementSee “CPU usage management” on page 15.In addition, this chapter describes best practices for configuring the WRM andWSA products in high load environments, which are made possible by the fixpack.See “Best Practices in a High Load Environment for WRM” on page 16 and “BestPractices in a High Load Environment for WSA” on page 17.Increasing process parallelismNetwork communication takes place between two end point parties. The Analyzercaptures network transmission data and groups them by logical communicationconnections, or sessions, per the communication parties’ IP addresses and portnumbers. A session key is easily generated using addresses and ports. It enablesthe Analyzer to queue any captured network packet quickly to its session anchorthrough a hashing lookup algorithm. There are thousands of active sessions at anyone time, and the Analyzer must process all of their network and applicationprotocols in order to calculate the application transaction response time.The Analyzer sets up a protocol task thread pool so that it can process sessionsconcurrently. The sessions are assigned to protocol tasks of the thread pool basedon a least-busy formula. This is important because sessions correspond toapplication transactions and they vary in duration and complexity. By using theleast-busy formula instead of a simple round-robin approach, it achieves balanced11

workloads among all protocol tasks in the thread pool, thereby avoidingabnormalities in session processing delays.The following Analyzer parameter controls protocol task thread pool:KFC MAX PROTOCOL POOL SIZE: The total number of protocol tasks in thethread pool. The default is 20. You may need to increase pool size per workload.Check the Analyzer active and maximum session buffer used operation matrix andkeep the average session per task below 1000 and the total pool size less than 100.For very high volumes, consult IBM Software Support.Optimizing storage utilizationStorage is critical to the Analyzer's operation and performance. Network volume isbeyond the Analyzer's control, yet it must capture and process them regardless ofcircumstance. All jobs in a system are given a set of virtual space for its programand data storage. This usually corresponds to operating system architectureaddressability limits.In AIX, however, the process’s data segment size is controlled by the LDR CNTRLenvironment variable and the default is only 256 MB. So for high volumeenvironments, higher process storage limits must be set for the Analyzer to operatesatisfactorily. This parameter must be set as an environment variable for therunning process and cannot be preset in system configuration files. The tablebelow describes the variable's settings and the corresponding process data segmentlimits. Because this is simply virtual storage size imposed on the Analyzer, themaximum setting is recommended.Table 1. LDR CNTRL SettingsLDR CNTRL SettingAdditional Data SegmentsProcess Memory LimitUnset0 (Default)256 MBLDR CNTRL-MAXDATA 0x10000001512 MBLDR CNTRL-MAXDATA 0x20000002768 MBLDR CNTRL-MAXDATA 0x300000031 GBLDR CNTRL-MAXDATA 0x400000041.25 GBLDR CNTRL-MAXDATA 0x500000051.5 GBLDR CNTRL-MAXDATA 0x600000061.75 GBLDR CNTRL-MAXDATA 0x700000072.0 GBLDR CNTRL-MAXDATA 0x800000082.25 GBAnalyzer processing in degraded modeRegardless of design precautions and care in implementation, the Analyzer can stillrun out of storage because of network volumes and limits on machine capabilities.In such a situation, the Analyzer will continue processing in degraded modethrough the following methods:v Inhibiting new session monitoring while continuing to process existing sessionsvvvv12Reducing network packet capture ratesReducing transaction data content copy sizeSpeeding up HTTP object merge processes and increasing object queue sizesDropping of response time data if an I/O buffer cannot be obtainedIBM Tivoli Web Response Monitor and IBM Tivoli Web Segment Analyzer: Notes for Fixpack 1.2.0-TIV-W3 Analyzer-IF0003

The Analyzer resumes normal operation as soon as storage constraints are relieved.Parameters affecting Analyzer storage usageThe following parameters affect Analyzer storage usage:Table 2. Parameters Affecting Analyzer Storage UsageParameterDescriptionKFC BUFFER POOL STATSpecifies whether to periodically output buffer poolutilization statistics. The default is Yes.KFC BUFFER STAT INTERVALThe buffer pool utilization statistics output interval. Thedefault is 300 seconds.KFC MAX APPLICATION DATA SIZECAUTION:Do not modify this parameter unless instructed to doso by IBM Software Support.The transaction reply content size copied for applicationprotocol interpretation. The default size is 32768 bytes.KFC BUFFER POOL CONTRACTION PERCENTCAUTION:Do not modify this parameter unless instructed to doso by IBM Software Support.The target buffer pool contraction percentage of the freebuffer. For example, 80 means releasing 20% of unusedbuffer. The default is 90.Minimizing output delayThe WRM Collector and WSA Engine are Analyzer applications. They definemonitoring criteria to the Analyzer through the KFC1 API. The Analyzer creates afilter corresponding to the monitoring requests. The application can define as manyfilters as needed, and different application filters may be either identical or similar.The Analyzer processes captured data per network and application protocols andcreates transaction response time data records. Every transaction response timedata record must be examined by every filter for its applicability. Once atransaction response time data record passes a filter, the response time dataattributes must be formatted into a transmission buffer to be delivered to theowning application. It is evident that it is a time-consuming procedure to runresponse time data records through all filter criteria and format all transactionattributes and timestamps into transmission buffers.WRM monitors individual Web page response times. From the monitoringperspective, it does not matter that the Web page response time data is deliveredto the WRM Collector a second after or a minute after the Analyzer creates theresponse time data record, as long as it is delivered and response time is calculatedcorrectly. However, WSA imposes additional time constraints on the Analyzer,because WSA correlates transaction response time data from several Analyzers andall data must be available in a short time frame in order for them to be validcorrelation candidates. Consequently, the Analyzer must deliver response time dataresults to the WSA Engine in near-real time.Parameters affecting Analyzer outputThe following parameters affect Analyzer output:Chapter 4. Best Practices in High Load Environments13

Table 3. Parameters Affecting Analyzer OutputParameterDescriptionKFC MAX OUTPUT TASK POOL SIZEThe output worker thread pool size. The default is 10.Check the Analyzer operation metrics Max RTDB OutputQueue Delay and Max RTDB Output Transaction Delayfirst to see if the delay is greater than the WSA timeframes. In general, no changes are necessary for arrivalrates less than 40,000 per minute.KFC TCP REPORT HTTP OBJECTSSpecifies whether to output or discard HTTP embeddedobject response time data for a TCP filter type (WSA).The default is No.KFC HTTP REPORT OBJECTSSpecifies whether to output or discard HTTP embeddedobject response time data for an HTTP filter type (WRM).The default is Yes.KFC HTTP OUTPUT UNMERGED OBJECTSSpecifies whether to output or discard unmerged HTTPpage embedded objects. The default is No.Improving I/O efficiencyNetwork I/O frequently contributes to program throughput bottleneck. Under highload conditions, the report rate for Analyzer-to-WRM/WSA-collector applicationsexceeds tens of thousands of transaction response time data records per minute.Any slight network issue might cause immediate output queue build up that leadsto significant increases in storage requirements. Furthermore, unless input networkvolume decreases, the output throughput might not recover, resulting in seriousperformance problems.Parameters affecting KFC1 API I/OThe following parameters affect KFC1 API I/O:Table 4. Parameters Affecting KFC1 API I/OParameterDescriptionKFC ENV API SPECIFY NICSpecifies an alternate network interface for connectingthe KFC1 API client to the Analyzer. If not specified, thefirst NIC is used. Specify the NIC using its assigned IPaddress in dot decimal format.KFC TIME SYNC REQUIREDSpecifies whether to enable or disable the timestampclock synchronization procedure between the KFC1 APIclient and the Analyzer. WSA requests timestampssynchronization. The default is No.KFC TIMESYNC THRESHOLD PERCENTSets timestamp synchronization delta accuracy. Thedefault is 25%. Higher accuracy requires more iterations,thus longer KFC1 client application start up times.KFC REPORT TRANS ARRIVAL RATESpecifies whether to enable or disable outputtingtransaction response time data arrival counts per minute.The default is No.KFC API MEDIASERVER LISTEN PORTSpecifies the Analyzer application server listening port.The default is 12121.14IBM Tivoli Web Response Monitor and IBM Tivoli Web Segment Analyzer: Notes for Fixpack 1.2.0-TIV-W3 Analyzer-IF0003

Table 4. Parameters Affecting KFC1 API I/O (continued)ParameterDescriptionKFC API CALLBACK TABLE LIMITCAUTION:Do not modify this parameter unless instructed to doso by IBM Software Support.Specifies the maximum KFC1 API callback result tablesize. The default is 12,000. The result table is reusedwhen this size is exceeded in order to support a highresponse time data arrival rate that is otherwise unableto be handled by the client application.CPU usage managementThe Analyzer may require high CPU and storage system resources in order toprocess high network transaction volumes. When the Analyzer requires moresystem resources due to workload, your applications, such as Web servers anddatabases, will also require more system resources because they also incur highvolume stress. Consequently, the Analyzer could impact business processing for allprograms competing for the same limited resources.The Analyzer implements an internal CPU management facility. This capabilityallows you to specify the maximum CPU limit that the Analyzer is allowed to usefor response time monitoring processing regardless of workload. The Analyzercontinuously monitors its CPU usage. The CPU usage is derived from thecumulative Analyzer CPU time, both Kernel and User time, from the beginning ofa management period to the end of that management period. The total CPU time isthen normalized against total processor time online in order to calculate the CPUusage percentage of the management period. Upon confirmation of consecutiveCPU usage, the Analyzer adjusts its internal work management and dispatchingheuristic rules, thus maintaining its CPU consumption below allowed limits.Nevertheless, the Analyzer still attempts to process and deliver the best possibleresponse time monitoring throughput within such resource constraints.In the chart below, the Analyzer’s CPU limit is set at 45% and the Analyzersuccessfully tracks its CPU usage at a 85% to 100% of 45% (38.25% to 24%) targetzone for a given workload volume (50,000 per minute) for this particularmachine.Chapter 4. Best Practices in High Load Environments15

Parameters affecting CPU usage managementThe following parameters affect CPU usage management:Table 5. Parameters Affecting CPU usage managementParameterDescriptionKFC CPU ENFORCE MAX LIMITSpecifies whether to turn on or off the CPU managementfeature. The default is No (turned off) so that theAnalyzer uses as much CPU as needed per workload.KFC CPU TARGET THRESHOLDSets the maximum allowed Analyzer CPU usage limit asa percentage. The minimum is 10% and the maximum is100%. The default is 40%.KFC CPU MANAGE PERIODThe CPU time calculation period. The default is 60seconds. Setting it to less than 30 seconds is ignored.KFC CPU ACTION INTERVALThe number of consecutive CPU usage managementperiods needed of either higher or lower than the targetthreshold in order for the Analyzer to initiate adjustmentactions. The default is 2.KFC CPU STATSpecifies whether to periodically output CPU usage data.The default is No. This parameter is effective regardlessof the KFC CPU ENFORCE MAX LIMIT setting.Best Practices in a High Load Environment for WRMIn a high load or high transaction arrival rate environment for WRM, the followingconsiderations and suggestions should be noted:v Consider reducing the number of recorded transactions by lowering thetransaction sampling percentage: For example:SM3 LOG SAMPLE PERCENTAGE 75v

1.2.0-TIV-W3_Analyzer-IF0003 for the Analyzer component of the IBM Tivoli Web Response Monitor (WRM) 2.0.0 and IBM Tivoli Web Segment Analyzer (WSA) 2.0.0 products. This document provides details about the following: v Limitations and new prerequisites for the products that you nee