ObjectStore Client Profiling

ObjectStore^® Client Profiling

• Configuration
• Use
  • Command line parameters
  • Activation of TerminalServer support
  • Server display
  • Client display
• remote profiling

The program cxDBProf is used for the remote monitoring of ClassiX® applications and to detect bottlenecks and unexpected locks on the database ObjectStore.

cxDBProf is based on two sources of information:

• Status information is evaluated, which the server makes available via the clients. This information can also be obtained via "ossvrstat" to be displayed.
• The clients themselves send information about the processed InstantView® messages and procedures via the ClassiX® remote profiling protocol.

See also CXOSLOG.EXE - a version without window interface, running as a Windows Service.

Configuration

The monitoring of the applications is done via the ClassiX® profile protocol (cxpp). This requires a common UDP port on the client and on the computer running the profiler.
For this purpose, an entry is made in the file " services " is required. Under Windows NT this is located in the directory " %SYSTEMROOT%\system32\drivers\etc "(mostly "C:\Windows\System32\drivers\etc").
The entry has the format:

Cxpp /udp

Whereby it is to be replaced by a number greater than 1024. The same entry must exist on the client and the server. If no corresponding value is entered on one of the systems, the default value 11111 is used.

Use

Command line parameters

cxDBprof.exe knows 5 command line parameters. All are optional, so that the program can be started without parameters. The parameters can be specified in any order. There are the following parameters:

cxDBprof.exe -HServer host name -TerminalServer host name -Database -LLogging.ini -SInterval

Notes:
The parameter values must be written directly after the parameter name. For example "-LC:\Temp\Logging.ini".

If the parameter "-S" has been specified, the (accumulated) server statistics will be started immediately, i.e. it does not have to be started via the menu first. The interval (in milliseconds) must be entered as a number and is optional, e.g. "-S60000" for one minute. The default is 200.

The parameter "-D (database) must contain the complete path to the database. If the database is on another computer this can be a UNC path (e.g. "\\\Server\Sharing\Classix\Projects\classix.cxd").

Activation of TerminalServer support

In order for the assignment between client name and a terminal server on which a ClassiX® application is executed, each terminal server must first be registered. This - and thus the name assignment - is only possible if a server version of TerminalServer is installed on the computer running the ProfServ.

You can register a TerminalServer via the menu item "View/TerminalServer options". Here you can add terminal servers (button "Add") or remove button "Del".

When adding, further options are queried: type of Terminal Server (Windows Terminal Server, Citrix Metaframe or Automatic Detection) and host name. If the specified computer is recognised as a Terminal Server, it will appear in the list.

Note: Citrix is not yet supported due to lack of documentation!

Server display

Under "File" a connection to DB servers and to clients accessing the database can be established. The server display provides a dynamic display of the status of all clients currently accessing the server. To open a server display, select "File/Connect to DB Server". Then, enter the server name in the "DB Server" field. The server name can be entered as a pure host name (e.g. "CL"), as a complete FQDN (Fully Qualified Domain Name) (e.g. "CL.CX.DE") or as an IP address (e.g. 192.168.1.34). The resolve button attempts to resolve a name.

Additionally, query speed and database can be specified. From the database specification the segment names are read out, which are required in the display of locking conflicts in the client display (see below). These details can be changed later with the DB server options. The database specification can either be a locally accessible file name (both from the server and the DBProf computer) or the path to the server database (:, i.e. the path locally from the server's point of view).

If the SystemOut path is specified, it is used to assign the ClassiX® user names to the currently logged on clients in the client overview.
The database server and the SystemOut directory can be set by environment variables.

Environment variable	Description
CX_DATABASE_SERVER	The database server
CX_SYSTEM_OUT	The output directory of the logs (SystemOut)

In addition to the window with all access details (next screenshot), the overview checkbox also opens an overview display in which all active clients are shown together with their status and locking conflicts, especially locking clusters, are highlighted in colour.

Overview display

The title of the overview window shows the respective database server. If possible, the clients are listed in the server window with their names, otherwise with IP addresses or process ID. The following statuses are possible:

Without status	The client is not currently causing any database activity, nor is there any server transaction open. However, a client transaction (i.e. one that is only known in the cache manager) can still be open.
Idle (Txn Open)	The client is not currently causing database activity, but has a transaction open on the server.
Working	The client is currently causing database activity, i.e. it has locks on at least one object in the database. In addition, the load that the client is causing (Work) and the time that this work has already taken is displayed.
Dead	The client is involved in a deadlock
Wait...	The client is waiting for the allocation of a lock that is currently held by another application. Further information relates to whether the client wants the lock to be read (R) or written (W), and whether a lock should be fetched for a specific range within a segment (Range, the normal case), for a whole segment or for the whole database.

Additionally, colours are used to indicate locking clusters. For example, if Client A is waiting for Client B and Client B is waiting for Client C, A, B and C are displayed in one colour. If client D is also waiting for client E, D and E are displayed in an additional colour.

As the data can change very quickly, it is possible to freeze the display. This is done via the stop button in the toolbar or the menu "Edit/Stop". The display is brought back to life by the "Run" button or the menu "Edit/Start". This setting affects the collection process and the display of the client display.

Detailed view

Server window with 2 active and one dead client

The title of the server window shows the respective database server and the number of connected clients. If possible, the clients are listed in the server window with their names, otherwise with IP addresses. A number of details are displayed for each client (see figure). The values displayed are the cumulative values since the client logged on to the database server. So not the cumulated values since the start of cxDBProf. By double-clicking on the name of the client you can access the client display.

Column labeling	Description
Client	Name of the client and process ID of the ClassiX process.
Locking conflicts	Frequency with which this client had to wait to get a lock on a page because a lock has already been issued to another client.
Waiting times	Total time this client has waited for locks to be released.
Deadlocks	Frequency with which the server has decided that this client has fallen victim to a deadlock. The client is then informed that he/she must cancel a certain transaction so that other clients can finish their transactions.
Transaction Aborts	Number of aborted transactions known to the server. If the transaction has run out of the cache without interaction with the server by then, the abort is not known to the server either.
Pages read	Number of pages and the corresponding amount of data sent by the server to the client. This can help to determine the cache size by keeping as much data as possible in the cache.
Written Pages	Number of pages and the corresponding amount of data sent by the client to the server. Data transfer only takes place if commits are successful. The data transferred per commit should not exceed the propagation buffer size of the server.
Commits	Number of completed transactions known to the server. If no data is changed in a transaction, the server is not necessarily informed about the completion. This should roughly correspond to the number of all transactions.
Readonly commits	Number of completed transactions for which the server has determined that they have not changed any data.
Locking conflict with	Which client currently holds a lock that this client is waiting for.
Conflict in segment	In which segment is a lock held that this client is waiting for.

Initially, the list contains only active clients. When a client logs off, its entry in the list is highlighted in gray. By clicking on the "Delete dead clients" button (red "X") the inactive (dead) clients can be deleted from the list.

If a client has to wait for one (or more) clients (locking), its entry in the list is coloured red and in the column "Locking conflict with" the client(s) are listed for which the client has to wait.

Client display

The client display is opened with "File/Connect to Client". In the "Client" box, enter the name of the client computer to be monitored. The resolve button corresponds to the "DB server" field in the server display (see above).

The "maximum depth" is currently not taken into account, but will later be used to display InstantView® commands only up to a certain nesting depth.

Under "Server Infos" you enter which DB server the client accesses. The field "DB server" is only activated if status information is to be obtained from the DB server.

The client display is divided into two parts. In the middle of the window is a slider that can be dragged to the left and right. When remote profiling is enabled on the client, the InstantView® profile statements are displayed on the left side.

On the right side, the advanced status information from the server is displayed if it is selected in the client option. The most interesting display is probably the lock entries. For better understanding: "Read-lock" does not mean that an application holds a read-lock, but that it is waiting for a read-lock. Read- and writelocks holds an application in the working sections.

In the example (figure below), an application on the CL.CX.DE computer is waiting for a read-lock to be issued on an area in segment 1080 ("salesOrderSlaveS"). The area starts at sector 104 and has a length of 8 sectors. The Read-Lock cannot be granted because an application on MOBIL2.CX.DE has already locked one of the objects in this area with a higher priority.

There is an InstantView® application that can be used to display all objects located in the specified area based on this information (see "Displaying the objects causing a locking conflict").

If "unknown" is displayed instead of a segment name, the DB server display must be made known to the database via the menu item "Display/DB server options" so that the segments belonging to the database can be loaded (see server display).

Logging

The following information can also be collected and logged using the cxosdblog service.

Statistical data on the server and all clients can be logged in a log file. The same applies to locking conflicts. First a configuration file has to be created:

log4cplus.rootLogger=INFO, FILE_LOGGER

log4cplus.appender.FILE_LOGGER=log4cplus::RollingFileAppender
log4cplus.appender.FILE_LOGGER.MaxFileSize=5MB
log4cplus.appender.FILE_LOGGER.MaxBackupIndex=10
log4cplus.appender.FILE_LOGGER.File=cxdbprof.log
log4cplus.appender.FILE_LOGGER.ImmediateFlush=true
log4cplus.appender.FILE_LOGGER.Append=true
log4cplus.appender.FILE_LOGGER.layout=log4cplus::PatternLayout
log4cplus.appender.FILE_LOGGER.layout.ConversionPattern=%D{%Y-%m-%d %H:%M:%S,%%q} %5p %c - %m%n

log4cplus.logger.cxdbprof=INFO

If necessary, enter the file name here (relative to the current directory). Environment variables can be accessed via ${VARIABLE}, absolute paths are also possible. If necessary, enter the file name here (relative to the current directory). Environment variables can be accessed via ${VARIABLE}, absolute paths are also possible.

If necessary, enter the file name here (relative to the current directory). Environment variables can be accessed via ${VARIABLE}, absolute paths are also possible. If necessary, enter the file name here (relative to the current directory). Environment variables can be accessed via ${VARIABLE}, absolute paths are also possible.

cxDBprof logs events of three different categories:

Category	Meaning	Example
cxdbprof.main	Start and end	ClassiX Profile Server 1.35 started ClassiX Profile Server terminated
cxdbprof.stat	Statistics	see below
cxdbprof.lock	Locking Conflict	see below

Statistical data about the server or a client is always written to the log if something has changed:

2006-05-11 15:37:59,762 INFO cxdbprof.stat - ; client, clientID, time, receive_msgs, callback_msgs, KB_read, KB_written, commit, readonly_commit, abort,lock_timeouts, lock_waits, deadlocks, total_lock_wait_time [,propagations, KB_propagated, KB_direct]
2006-05-11 15:38:08,762 INFO cxdbprof.stat - . 9 84 0 276 0 0 1 0 0 0 0 0 0 0 0
2006-05-11 15:38:09,762 INFO cxdbprof.stat - . 10 2 0 0 0 0 0 0 0 0 0 0 0 0 0
2006-05-11 15:38:12,980 INFO cxdbprof.stat - . 13 792 0 1267 0 0 5 0 0 0 0 0 0 0 0
2006-05-11 15:38:13,980 INFO cxdbprof.stat - md 231 14 792 0 1267 0 0 5 0 0 0 0 0
2006-05-11 15:38:13,980 INFO cxdbprof.stat - . 14 65 0 247 0 0 1 0 0 0 0 0 0 0 0
2006-05-11 15:38:14,980 INFO cxdbprof.stat - md 231 15 64 0 247 0 0 1 0 0 0 0 0

The first line describes the individual data fields. In the logfile these are separated by the tabulator character.

Data field	Description
client	Name of the client, where the dot stands for the server
clientID	ID that the ObjectStore server gives to each client. It should not be confused with the process ID!
time	Time elapsed since program start (by cxDBprof) in seconds
receive_msgs	Number of messages that the server has received from clients. A message can be a request for an action such as opening the DB, getting a page, updating the DB, executing a transaction, aborting a transaction, closing the DB, and other things. The number tells you how often clients communicate with the server. Accordingly, if the number is small, the request to the server is small.
callback_msgs	Number of callback messages that the server sends to the clients. A callback message is a message that the server sends to client A when client B requests data from a page that is cached by client A. If this number is high, this means that an application often wants to write to data that other clients want to read or write to.
KB_read	Amount of data read, in KBytes Number of kilobytes of data that the server sent to clients to read. Monitor this statistic to help determine whether you need to enlarge client cache files. Compare kilobytes read for a given client with the number of commits and the size of the client cache file.
KB_written	Written data volume, in KBytes Number of kilobytes of modified data that the clients sent to the server. Written data is data involved in a commit. It must be buffered, and it is logged if it cannot go directly to a database because it is not being written past the current end of a cluster. When analysis is concerned with the number of transactions per second, the number of kilobytes written is an important factor.
commit	Number of commits (completed R/W transactions) that the server knows about. If a client does not modify data during a transaction, the client may not inform the server that a transaction is completed. With this number the transactions per second can be calculated.
readonly_commit	Number of readonly commits: Executed transactions which, according to the server, were not involved in data changes. Usually the client does not inform the server about such commits, so the number should be small. Read-only commits are like simple aborts and do not cost a lot of time?
abort	Number of aborted transactions logged by the server. If the client has not sent a message to the server, it can cancel a transaction without notifying the server. Does ClassiX® only abort transactions due to lock conflicts? (Timeouts?)
lock_timeouts	Number of lock timeouts: Number of all attempts by the clients to lock a page that is already locked because the set waiting time had already expired.
lock_waits	Number of blockades that did not lead to a timeout Number of attempts by all clients to block a page that is already blocked by another client. Next to the number of lock waits you can see the average time you wait for a lock in round brackets. (ObjectStore divides the total time by the number of lock waits.
deadlocks	Number of deadlocks Number of times the server will declare a client a deadlock "victim" and notice that it needs to complete a single transaction so that other clients can complete their transactions.
total_lock_wait_time	Waiting time for locks, cumulative (in msec)

Events such as deadlock or blocking (read operation blocks write operation, and vice versa) are also logged:

2006-05-11 14:37:35,740 INFO cxdbprof.lock - st 219 mr(564) 205 Segment 32
2006-05-11 14:37:35,740 INFO cxdbprof.lock - ct 207 mr(564) 205 Segment 32

2006-05-11 15:39:11,245 INFO cxdbprof.lock - cl 234 st(2148) 233 mr(1868) 231 Segment 32

The individual columns are separated by the tabulator character (represented by _ in the example).

Client st is blocked by mr
Client ct is also blocked by mr

Client cl is blocked by the clients st and mr

The reading from left to right: Client xxx is blocked by client(s) yyy (and zzz and ...) in segment abc. The number in brackets is the process number, the number right of the client name is the client ID of ObjectStore.

remote profiling

To display the InstantView® statements of a client with the cxDBProf, remote profiling must be activated on the client side. This is done with the InstantView® command

profiles()

which must be the name of the server where cxDBProf is running. If you specify a computer name that is reachable but not running cxDBProf, this will not result in an error, but the information sent will be lost (in a sink, as theorists or workflow experts would say).

To switch off, the usual command for deactivating profiling is used:

Profiles(OFF)

Remote profiling can also be activated remotely via a remote message. For this purpose, the "ClassiX® Remote Message Protocol" (cxrmp) must first be set. For this purpose, both the sender and the receiver must be configured in the file " Services " (see "Configuration") the entry

cxrmp /udp

must be available. <Port# is a value to be selected identically on the client and server side from 1024.

If remote messaging has been set up, the sender can enable or disable remote profiling on the receiver's side by means of an InstantView® instruction in the following manner:

"Profile()" "" SendMsg(, REMOTE)

"Profile(OFF)" "" SendMsg(, REMOTE)

To receive the message, the recipient must run an InstantView® application that responds to the message as follows:

: Execute

cxDBProf server name and receiver name must be selected according to the configuration. must be a message that is understood uniformly on the sender and receiver side.