Logging and Troubleshooting
Overview
RadiantOne uses the log4J v2 API for logging.
Although a variety of log files are generated by RadiantOne, the ones most useful for troubleshooting are associated with RadiantOne Universal Directory and FID, Control Panel, Zookeeper, the real-time persistent cache refresh and global sync components: Connectors, Sync Engine and Agents.
Root Logging Location/Drive
The log4J configurations for the RadiantOne components use paths that include ${rli:logging.root). You can see an example of this for the log4j2-vds.json configuration in the screen below.
Figure 1: Log4J Configuration Settings in Zookeeper
The default root location/drive is identified by the environment variable RLI_HOME.
This is configured in <RLI_HOME>\config\logging\logging.properties.
logging.root=${env:RLI_HOME}
You can change the value of logging.root to indicate another location/drive. Restart the RadiantOne services if you change the default logging root.
This is a node-specific configuration as each node may need to log to its own drive/location.
Log Levels
There are different levels of logging available for the components described in this guide. The options are: Off, Fatal, Error, Warn, Info, Debug, and Trace.
Off
Logging is turned off completely for the server.
Fatal
Logs events that may cause the RadiantOne service to not be able to respond. An example would be if a single client was using all threads and the RadiantOne service could not respond to anyone else but this client.
Error
Logs any error message encountered by RadiantOne (database connection problems, fatal errors…etc.).
Warn Logs any warning messages encountered by RadiantOne (client disconnected before response could be sent, and errors, fatal messages…etc.).
Info
Logs all access to RadiantOne. All actions taken by RadiantOne and the results (including warnings, errors, and fatal messages).
Debug Logs very detailed information about the actions taken by RadiantOne.
Trace
This log level is relevant only for the Radiant Logic development team to understand problems and is used for intensive troubleshooting.
Running System Checks
During the RadiantOne installation, a series of system checks are performed to verify that important system requirements are met. These include number of CPUs, machine memory, disk throughput, disk space, maximum number of file descriptors (applicable to Linux), maximum resident set size (applicable to Linux), maximum amount of virtual memory available to RadiantOne (applicable to Linux), maximum number of memory map areas for processes (applicable to Linux), and swappiness (applicable to Linux).
To execute these system checks after installation, from a command line, run cluster.bat system-check (cluster.sh is the script on Linux). Below is an example of launching the command on Windows.
C:\radiantone\vds\bin\advanced>cluster.bat system-check
Another series of checking related to DNS resolution (for all cluster nodes), Zookeeper network latency (between all cluster nodes), Zookeeper read and write throughput, can be triggered with cluster.bat remote-check (cluster.sh is the script on Linux). These checks are mostly applicable to communication between cluster nodes. Below is an example of launching the command on Windows.
C:\radiantone\vds\bin\advanced>cluster.bat remote-check
If you experience any problems, it is best to run these system checks prior to restarting any RadiantOne services. The result of these checks can provide a glimpse at the current system environment and help determine if something in the environment is contributing to your problem.
The items checked are outlined below.
- CPU Cores - minimum 2 cores.
- Machine Memory - minimum 15GB.
- Disk Throughput - minimum 100MB/sec.
- Disk Space - minimum 20GB.
- DNS resolution - all hostnames must be resolvable.
- Zookeeper network latency - maximum 15ms.
- Zookeeper read throughput - minimum 25MB/sec.
- Zookeeper write throughput - minimum 5MB/sec.
- Linux Open File Descriptors - minimum 48k.
- Linux Maximum resident set size - requires unlimited.
- Linux Memory Map Areas for processes - minimum 256KB.
- Linux Swappiness - swap to be disabled or swappiness <=20.
- Linux Maximum amount of virtual memory available to a process launched by the user associated with the RadiantOne installation - requires unlimited.
Data Statistics
In general, knowing statistics about your data can be helpful for troubleshooting.
To get statistics about the entries in your view, you can use the LDIFStatistics function of the <RLI_HOME>/bin/advanced/ldif-utils utility. Once you have an LDIF file containing your entries, pass the file name and path to the utility with the -f property.
If you have nested groups, and want them included in the results, include -n in the command.
The syntax of the command is:
ldif-utils LDIFStatistics -f <ldif_file_path> [-n (to get nested group stats)]
The results include the following statistics about entries (non-group), groups and objectclasses:
###### Entries statistics ######
Entry count – number of entries
Max attributes per entry
###### Non-group entry statistics ######
AVG attributes per entry
Max entry size in bytes
AVG entry size in bytes
Max attribute size
AVG attributes size (non-objectclass)
###### Groups statistics ######
Group count – number of group entries
Groups Statistics: [
### Groups SIZE_RANGE_NAME statistics ###
Group entry count
Max members
AVG members
Max entry size in bytes
AVG entry size in bytes
###### ObjectClass Statistics ######
### objectclass_name statistics ###
Entry count – number of entries
Max attributes per entry
AVG attributes per entry
Max entry size in bytes
AVG entry size in bytes
Max attribute size
AVG attributes size
RDN Types: [rdn_name]
Entry count per branch: {branch_dn=entrycount_x},
The following would be an example of the command and statistics returned.
C:\radiantone\vds\bin\advanced>ldif-utils LDIFStatistics -f "C:\radiantone\vds\vds_server\ldif\export\mydirectory.ldif"
###### Entries statistics ######
Entry count: 10014
Max attributes per entry: 19
###### Non-group entry statistics ######
AVG attributes per entry: 8
Max entry size: 956 bytes
AVG entry size: 813 bytes
Max attribute size: 1
AVG attributes size (non-objectclass): 1
###### Groups statistics ######
Group count: 2
Groups Statistics: [
### Groups LESS_THAN_10 statistics ###
-Group entry count: 1
-Max members: 2
-AVG members: 2
-Max entry size: 578 bytes
-AVG entry size: 578 bytes,
### Groups BETWEEN_1K_AND_10K statistics ###
-Group entry count: 1
-Max members: 10000
-AVG members: 10000
-Max entry size: 571 KB
-AVG entry size: 571 KB]
###### ObjectClass Statistics ######
### organization statistics ###
-Entry count: 1
-Max attributes per entry: 8
-AVG attributes per entry: 8
-Max entry size: 403 bytes
-AVG entry size: 403 bytes
-Max attribute size: 1
-AVG attributes size: 1
-RDN Types: [o]
-Entry count per branch: {root=1},
### groupofuniquenames statistics ###
-Entry count: 1
-Max attributes per entry: 9
-AVG attributes per entry: 9
-Max entry size: 578 bytes
-AVG entry size: 578 bytes
-Max attribute size: 2
-AVG attributes size: 2
-RDN Types: [cn]
-Entry count per branch: {ou=groups,o=companydirectory=1},
### organizationalunit statistics ###
-Entry count: 11
-Max attributes per entry: 8
-AVG attributes per entry: 8
-Max entry size: 468 bytes
-AVG entry size: 439 bytes
-Max attribute size: 1
-AVG attributes size: 1
-RDN Types: [ou]
-Entry count per branch: {o=companydirectory=11},
### inetorgperson statistics ###
-Entry count: 10000
-Max attributes per entry: 19
-AVG attributes per entry: 19
-Max entry size: 956 bytes
-AVG entry size: 814 bytes
-Max attribute size: 1
-AVG attributes size: 1
-RDN Types: [uid]
-Entry count per branch: {ou=inventory,o=companydirectory=1000, ou=management,o=companydirectory=1000, ou=human resources,o=companydirectory=1000, ou=product development,o=companydirectory=1000, ou=accounting,o=companydirectory=1000, ou=information technology,o=companydirectory=1000, ou=customer service,o=companydirectory=1000, ou=sales,o=companydirectory=1000, ou=quality assurance,o=companydirectory=1000, ou=administration,o=companydirectory=1000},
### groupofurls statistics ###
-Entry count: 1
-Max attributes per entry: 9
-AVG attributes per entry: 9
-Max entry size: 571 KB
-AVG entry size: 571 KB
-Max attribute size: 10000
-AVG attributes size: 10000
-RDN Types: [cn]
-Entry count per branch: {ou=groups,o=companydirectory=1}]
Done in 1169ms
Sharing Sample Data with Radiant Logic Support
It is often difficult for the Radiant Logic support team to reproduce problems seen by a customer without having some sample data to test with. Many companies are uncomfortable with sharing their data, so RadiantOne includes a utility that anonymizes attribute values (except for object classes) while maintaining original data distribution, case, special characters, relationships (e.g. group membership), range of values and length of values.
The utility is in <RLI_HOME>/bin/advanced/LdifAnonymizer.bat (.sh on Linux) and has the following usage:
LdifAnonymizer.bat -f <ldif_file_path> [-i <ignored attributes>] [-d <dn attributes>][-s <secret attributes>]
-f is the source LDIF file containing the entries. You must have an LDIF file containing the data prior to using the LdifAnonymizer utility.
-i is optional and can contain a comma-separated list of attributes that should be preserved in the source LDIF file and not modified with random values.
-d is optional and can contain a comma-separated list of DN-type attributes (those that have a DN for the value).
-s is optional and can contain a comma-separated list of attributes to keep secret, meaning that the process randomizes these attributes within the different ranges instead of using the shuffling sets.
The utility generates a modified LDIF file (
In addition to sending an anonymized LDIF file containing sample data, it is also helpful to share data statistics with the Radiant Logic support team.
Sharing Data Statistics with Radiant Logic Support
RadiantOne logs statistics related to operations it receives. This includes average execution time, peak execution time, and whether the operation was successful. No actual data (entries/attributes) is logged, only metadata. This log is primarily for Radiant Logic support to have key information to assist with troubleshooting.
This logging is enabled by default and can be managed from the Main Control Panel > Settings Tab > Logs section > Statistics > Statistics Analyzer Settings sub-section. The log location is <RLI_HOME>/vds_server/logs/stats.log. This logging is enabled by default and calculates statistics during 1-minute intervals prior to saving to the stats.log.
For each RadiantOne Universal Directory (HDAP) store or persistent cache initialization, statistics are calculated for the total number of entries and sub-categorized by branches and object classes. The average and peak number of attributes per entry, and the average and peak size (in KB) per entry are also calculated. This information is logged into the <RLI_HOME>/vds_server/logs/stats.log. This logging is enabled by default and can be managed from the Main Control Panel > Settings Tab > Logs section > Statistics > Init Statistics Settings sub-section.
Expert Mode
Some settings in the Main Control Panel are accessible only in Expert Mode. To switch to Expert Mode, click the Logged in as, (username) drop-down menu and select Expert Mode.
Figure 2: Switching Control Panel to Expert Mode
The Main Control Panel saves the last mode (Expert or Standard) it was in when you log out and returns to this mode automatically when you log back in. The mode is saved on a per-role basis.