The capture connector configuration dictates the process for detecting changes on the source objects. The type of data source determines which capture methods are available.

This section focuses on configuring the connector type. For details on the behavior of and properties for database connectors (Timestamp, Counter, Changelog), LDAP connectors (changelog or persistent search), and Active Directory connectors (usnChanged or DirSync), please see the Connector Properties.

For database backends (JDBC-accessible), the connector change detection options are:

• Changelog – This connector type relies on a database table that contains all changes that have occurred on the base tables (that the RadiantOne identity view is built from). This typically involves having triggers on the base tables that write into the log/changelog table. However, an external process may be used instead of triggers. The connector picks up changes from the changelog table.

• Timestamp – This connector type has been validated against Oracle, SQL Server, MySQL, MariaDB, PostgreSQL, Snowflake, and Apache Derby. The database table must have a primary key defined for it and an indexed column that contains a timestamp/date value. This value must be maintained and modified accordingly for each record that is updated.

  For Oracle databases, the timestamp column type must be one of the following: TIMESTAMP, DATE, TIMESTAMP WITH TIME ZONE, `TIMESTAMP WITH LOCAL TIME ZONE1.

  For SQL Server database, the timestamp column type must be one of the following: SMALLDATETIME, DATETIME, DATETIME2

  For MYSQL or MariaDB databases, the timestamp column type must be one of the following: TIMESTAMP, DATETIME

  For PostgreSQL databases, the timestamp column type must be one of the following: TIMESTAMP, timestamp without time zone (equivalent to timestamp), TIMESTAMPTZ, timestamp with time zone (equivalent to timestamptz)

  For Snowflake, the timestamp column type must be: TIMESTAMPNTZ

  For Derby databases, the timestamp column type must be: TIMESTAMP

  The DB Timestamp connector leverages the timestamp column to determine which records have changed since the last polling interval. This connector type does not detect delete operations. If you have a need to detect and propagate delete operations from the database, you should choose a different connector type like DB Changelog or DB Counter.

• Counter - This connector type is supported for any database table that has an indexed column that contains a sequence-based value that is automatically maintained and modified for each record that is added/updated. This column must be one of the following types: BIGINT, DECIMAL, INTEGER, or NUMERIC. If DECIMAL or NUMERIC are used, they should be declared without numbers after dot: DECIMAL(6,0) not as DECIMAL(6,2). The DB Counter connector leverages this column to determine which records have changed since the last polling interval. This connector type can detect delete operations as long as the table has a dedicated "Change Type" column that indicates one of the following values: insert, update, delete. If the value is empty or something other than insert, update, or delete, an update operation is assumed.

Database Changelog Connector

RadiantOne generates the SQL scripts which create the configuration needed to support the DB Changelog Connector. These scripts can be downloaded and applied by your database administrator or the connector configuration process can apply them directly.

• create_user.sql - Creates the log table user and the log table schema.
• create_capture.sql - Creates the log table and the triggers on the base table.
• drop_capture.sql - Drops the triggers and the log tabl1. Note: for some databases the file is empty.
• drop_user.sql - Drops the log table user and schem1. Note: for some databases the file is empty.

This section describes generating and executing the scripts to configure the database changelog connector for real-time persistent cache refresh. The following steps assume the database backend has a changelog table that contains changed records that need to be propagated to destinations. The changelog table must have two key columns named RLICHANGETYPE and RLICHANGEID. RLICHANGETYPE must indicate insert, update or delete, dictating what type of change was made to the record. RLICHANGEID must be a sequence-based, auto-incremented INTEGER that contains a unique value for each record. The DB Changelog connector uses RLICHANGEID to maintain a cursor to keep track of processed changes.

To configure the database changelog connector for real-time persistent cache refresh:

1. From the Control Panel, go to Setup > Directory Namespace > Namespace Design.

2. Select the root naming context that contains the identity view that is cached.

3. On the right side, click the CACHE tab.

4. If you haven't already created the cache, click +CREATE NEW CACHE and browse to the branch in the RadiantOne namespace that you would like to store in persistent cache and click CREATE. If you already have created the cache, click ... > Edit inline with the cached branch.

5. Select Real Time for the refresh type. A table displays indicating which connectors require configuration.

6. Click the inline with the connector to configure.

1. Select DB Changelog from the Connector Type drop-down list.
2. Indicate the user name and password for the connector's dedicated credentials for connecting to the log table. If you do not have the user name and password, contact your DBA for the credentials.
3. Enter the log table name using the proper syntax for your database (e.g. {USER}.{TABLE}_LOG) or accept the default. Change the value for this property only if you are creating the log table manually and the capture connector does not calculate the log table name correctly. Be sure to use the correct syntax if you change the value.

1. Define any other needed Connector Properties.
2. Select SAVE.
3. In the connector table, click </> to apply the scripts.

1. A message is displayed that asks if you want to apply the scripts to configure the log table immediately or not. You can also download the scripts to the local machine.

1. To apply now, select Apply Now and click CLOSE. Otherwise, select Apply Later and click DOWNLOAD CONFIGURE SQL SCRIPTS.

1. Click NEXT to initialize the cache and once initialized, you can manage the cache store properties.

Log table name syntax

Proper syntax for the Log Table Name must include both the schema name and the table name separated with a period. Values for this property may contain quote marks as required by the database. In most cases, the double quote mark (") is used, but some databases use a single quote (') or back quote (`). The following examples explain the property's syntax and usage.

Example 1:

For Postgres, if the schema is rli_con, and log table name is test_log, the property should be one of the following.

rli_con.test_log

Or with optional quoting:

"rli_con"."test_log"

Example 2:

For SQL Server, if the schema is RLI_CON, and log table name is TEST_LOG, the property should be one of the following.

RLI_CON.TEST_LOG

Or with optional quoting:

"RLI_CON"."TEST_LOG"

Example 3:

If schema and/or table name contain mixed-case characters, they must be quoted. For example, if the schema is Rli_Con, and log table name is Test_Log, the property should be as follows.

"Rli_con"."Test_log"

Database Timestamp Connector

The following steps assume your backend database table has a primary key defined and contains a timestamp column. The timestamp column name is required for configuring the connector. The timestamp column database types supported are described in the Database connectors section.

To configure the database timestamp connector for real-time persistent cache refresh:

1. From the Control Panel, go to Setup > Directory Namespace > Namespace Design.

2. Select the root naming context that contains the identity view that is cached.

3. On the right side, click the CACHE tab.

4. If you haven't already created the cache, click +CREATE NEW CACHE and browse to the branch in the RadiantOne namespace that you would like to store in persistent cache and click CREATE. If you already have created the cache, click ... > Edit inline with the cached branch.

5. Select Real Time for the refresh type. A table displays indicating which connectors require configuration.

6. Click the inline with the connector to configure.

1. Select DB Timestamp from the Connector Type drop-down list.

2. Indicate the column name in the database table that contains the timestamp. An example is shown below.

1. Define any other needed Connector Properties.
2. Select SAVE.
3. Click NEXT to initialize the cache and once initialized, you can manage the cache store properties.

Database Counter Connector

The following steps assume your database backend table contains an indexed column that contains a sequence-based value that is automatically maintained and modified for each record that is added, updated or deleted. The DB Counter connector uses this column to maintain a cursor to keep track of processed changes. The counter column database types supported are described in the Database connectors section.

1. From the Control Panel, go to Setup > Directory Namespace > Namespace Design.

2. Select the root naming context that contains the identity view that is cached.

3. On the right side, click the CACHE tab.

4. If you haven't already created the cache, click +CREATE NEW CACHE and browse to the branch in the RadiantOne namespace that you would like to store in persistent cache and click CREATE. If you already have created the cache, click ... > Edit inline with the cached branch.

5. Select Real Time for the refresh type. A table displays indicating which connectors require configuration.

6. Click the inline with the connector to configure.

1. Select DB Counter from the Connector Type drop-down list.

2. Indicate the change type column and the counter column names in the database table that contains the counter. An example is shown below.

1. Define any other needed Connector Properties.
2. Select SAVE.
3. Click NEXT to initialize the cache and once initialized, you can manage the cache store properties.

Database Connector Failover

This section describes the failover mechanism for the database connectors.

The database connectors leverage the failover server that has been configured for the data source. When you configure a data source for your backend database, select a failover database server from the drop-down list. The failover server must be configured as a RadiantOne data source. See the screen shot below for how to indicate a failover server for the Data Sources from the Control Panel > Setup > Data Catalog > Data Sources > [selected data source] > Details tab > Failover section.

If a connection cannot be made to the primary server, the connector tries to connect to the failover server configured in the data source. If a connection to both the primary and failover servers fails, the retry count goes up. The connector repeats this process until the value configured in "Max Retries on Connection Error" is reached. There is no automatic failback, meaning once the primary server is back online, the connector does not automatically go back to it.

For directory backends (LDAP-accessible), the connector change detecion options are:

• Changelog - leverages the cn=changelog naming context in the LDAP directory to detect changes.
• Persistent Search - leverages the LDAP persistent search control to detect changes.

Changelog

The connector leverages a changelog that has been enabled on the backend directory. The connector picks up changes from the cn=changelog naming context based on a polling interval. The changenumber attribute in the changelog entries is used to keep track of the processed changes. The changelog must be enabled in the backend directory. Please check with your directory vendor for instructions on how to enable the changelog.

Persistent search

Any LDAP directory that offers a persistent search mechanism can use the Persistent Search connector type. Novell eDirectory is an example of an LDAP source that supports persistent search. Others include Red Hat Directory, IBM TDS, CA Directory and RadiantOne Directory. The connector issues a persistent search and gets notified by the directory server when information changes. If the connector is shut down (either deliberately or due to failure), the delete operations that occurred in the directory are lost. Once the connector is back online there is no way to detect the delete operations that occurred while it was down. The only exception to this is for IBM TDS directories. It stores deleted entries and the capture connector is able to read them, and based on timestamp, determine if the change occurred while the connector was offline.

There are three change detection mechanisms: USNChanged, DirSync and Hybrid. If you are virtualizing and detecting changes from a Global Catalog, then you must use the USNChanged changed connector because the DirSync and Hybrid connectors cannot detect change events on sub-domains.

• DirSync - leverages Active Directory DirSync to detect changes.
• USNChanged - leverages the USNChanged attribute managed by Active Directory to detect changes.
• Hybrid - leverages capabilities of both Active Directory DirSync and USNChanged values to detect changes.

The flowchart below helps to decide which change detection mechanism to use.

DirSync

This connector type retrieves changes that occur to entries by passing a cookie that identifies the directory state at the time of the previous DirSync search. The first time the DirSync capture connector is started, it stores a cookie in a cursor file. At the next polling interval, the connector performs a DirSync search to detect changes by sending the current cookie. To use the DirSync control, the Bind DN connecting to the directory must have the DS-Replication-Get-Changes extended right, which can be enabled with the "Replicating Directory Changes" permission, on the root of the partition being monitored. By default, this right is assigned to the Administrator and LocalSystem accounts on domain controllers.

If you are virtualizing and detecting changes from a Global Catalog, then you must use the Active Directory USNChanged changed connector because the DirSync connector cannot detect change events on sub-domains.

USNChanged

This connector type keeps track of changes based on the uSNChanged attribute for the entry. Based on a configured polling interval, the connector connects with the user and password configured in the connection string/data source and checks the list of changes stored by Active Directory. The connector internally maintains the last processed change number (uSNChanged value) and this allows for the recovery of all changes that occur even if the connector is down (deliberately or due to failure).

If capturing the sequence of events is critical, use the DirSync connector instead of USNChanged because it processes events in the order in which they occur instead of prioritizing and processing inserts and updates before deletes.

Hybrid

This connector type uses a combination of the uSNChanged and DirSync change detection mechanisms. The first time the connector starts, it gets a new cookie and the highest uSNchanged number. When the connector gets a new change (modify or delete), it makes an additional search using the DN of the entry and fetches the entry from AD. The fetched entry contains the uSNChanged attribute, so the connector updates the cursor values for both for the cookie and the last processed uSNchanged number.

When the connector restarts, uSNChanged detection catches the entries that have been modified or deleted while the connector was stopped. The LDAP search uses the last processed uSNChanged number to catch up. After the connector processes all entries, it requests a new cookie from AD (not from the cursor) and switches to DirSync change detection.

If you are using the RadiantOne Directory, a persistent cache on a proxy view of a RadiantOne Directory, or a nested persistent cache view (a cached view used in another cached view), the connector type is noted as HDAP trigger automatically and cannot be changed. This is a special trigger mechanism that publishes changes directly into the queue to invoke the synchronization as opposed to requiring a connector component to detect the change.

This section describes the failover mechanism for the LDAP (changelog), Persistent Search and Active Directory connectors.

The directory connectors leverage the failover servers that have been configured for the data source. When you configure a data source for your backend directory, you need to indicate the list of failover servers in order of priority. When the connector fails over, it uses the failover servers in the order they are listed. See the screen shot below for how to indicate a failover server for the Data Sources from the Main Control Panel.

If a connection cannot be made to the primary server and the maximum number of retry attempts has been exhausted, the connector connects to the failover servers in the order they are listed. There is no automatic failback, meaning once the primary server is back online, the connector does not automatically go back to it.

This failover mechanism is supported for Active Directory, OpenDJ, Oracle Directory Server Enterprise Edition (Sun Directory v7), Oracle Unified Directory (OUD). In addition, any LDAP directory implementing cn=changelog and replicationCSN attribute or the persistent search control is also supported.

For the Active Directory USNChanged connector, the failover logic leverages the Active Directory replication vectors [replUpToDateVector], and the failover servers configured at the level of the RadiantOne data source associated with Active Directory, in order to determinate which server(s) the connector switches to in case of failure. Since the replication vector contains all domains, in addition to some possibly retired domains, the connector narrows down the list of possible failover candidates to only the ones listed as failover servers in the RadiantOne data source associated with the Active Directory backend. If there are no failover server defined for the data source, all domains in the replication vector are possible candidates for failover.

[replUpToDateVector] definition: The non-replicated attribute replUpToDateVector is an optional attribute on the naming context root of every naming context replica. If this vector is unavailable, the connector is suspended.

The ReplUpToDateVector type is a tuple with the following fields:

• uuidDsa: The invocation ID of the DC that assigned usnHighPropUpdate.
• usnHighPropUpdate: A USN at which an update was applied on the DC identified by uuidDsa.
• timeLastSyncSuccess: The time at which the last successful replication occurred from the DC identified by uuidDsa; for replication latency reporting only.

[replUpToDateVector] example:

01ca6e90-7d20-4f9c-ba7b-823a72fc459e @ USN 2210490 @ Time 2005-08-21 15:54:21

1d9bb4b6-054a-440c-aedf-7a3f28837e7f @ USN 26245013 @ Time 2007-02-27 10:17:33

24980c9d-39fa-44d7-a153-c0c5c27f0577 @ USN 4606302 @ Time 2006-08-20 23:33:09

At run-time, the connector retrieves the entire list of servers defined in the replication vector and reduces the number of possible failover candidates based on failover servers defined in the RadiantOne data source. The list of potential failover servers is stored at each polling interval. When the current server fails, the connector decides to switch to the closest candidate by selecting the server with the maximum timestamp from the up-to-dateness vector. The capture connector's cursor will be assigned the value from the up-to-dateness vector for the failover server. If the closest candidate fails as well, the connector tries with a second closest candidate and so on.

For the AD Hybrid connector, the failover process starts when the number of exceptions exceeds either the Maximum Retries on Error or Maximum Retries on Connection Error value. The failover servers are specified in the data source associated with Active Directory. The Active Directory up-to-date vector is used to determine the failover server and the value of the new cursor. If the up-to-date vector is unavailable from the current server, failover is not possible. If this happens, verify that AD replication is correctly configured. After the failover server is found, uSNChanged detection catches the entries that have been modified or deleted since the connector's failure. The LDAP search uses the last processed uSNChanged number to catch up.

After the connector processes all entries, it requests a new cookie from Active Directory and switches to DirSync change detection.