Kafka Producer Connector

The RadiantOne Kafka apply connector leverages the Apache Kafka Producer API to publish a stream of records to one or more Kafka topics.

The Kafka capture and apply connectors are independent, meaning that they do not need to be deployed together. The source(s) and target(s) could be any combination of Kafka topics, LDAP-accessible data stores, JDBC-accessible data stores, or web-services (SCIM or REST accessible).

This section assumes that you have access to an existing Kafka instance that you can connect to. Once you gain access, follow the instructions in this section.

To sync between RadiantOne and Kafka topics, you need to configure the following.

• Properties file
• Data source
• Schemas
• Views
• Sync topologies

You can configure the properties file either for unencrypted, anonymous connections, or for connections where encryption and authentication are required. This section describes how to configure both versions of the properties file.

Configuring the Properties File for Unencrypted Anonymous Connections

In a text editor, enter the path on your RadiantOne server containing the bootstrap.servers value with the name and port of your Kafka server. i.e. kafka.mycompany.com:9092.

Figure 1: Producer.properties text file for unencrypted anonymous connections

Configuring the Properties File for Encryption and Authentication

In cases where encryption and authentication are required, additional properties must be included in the producer.properties file.

In this example, PLAIN SASL is used to specify a username and password for authentication. Because the connection is encrypted, you must create a Java truststore containing the trusted CAs which issued the certificate used by the Kafka broker. The ssl.truststore.location needs to be the full path to the file on your RadiantOne server. It is recommended that you store both the producer.properties and truststore.jks file in your $RLI_HOME/vds_server/custom directory so that it replicates to follower nodes in your cluster.

Figure 2: Producer.properties text file for a Kafka broker requiring encryption and authentication

In this section, the producer data source is created.

To configure the producer data source:

1. In the Main Control Panel, navigate to Settings > Server Backend > Custom Data Sources.

2. From the Add Custom menu, select Kafka.

3. Name the data source. In this example, the data source is named kafkaproducer.

4. Enter values for the following properties.

   Property Name	Value
   Topic Name	(The name of the topic to which you wish to publish)
   Message Format	(The name of the changeEventConvertor, such as GoldenGateJSONConvertor)
   Producer Properties File	(The location of the properties file, i.e. $RLI_HOME/vds_server/custom/producer.properties)

   

   Figure 3: Kafka Producer

5. Click Save.

   Note

   If a note displays stating that the connection to the data source failed, click Yes to save anyway.

Based on the JSON messages you expect to process with your Kafka connector, you need to define appropriate schema.

1. In the Main Control Panel, navigate to the Context Builder tab.

2. On the Schema Manager tab, click .

3. Select Custom and click OK.

4. Select the Kafka data source you created in Configuring the Producer Data Source and click Next.

5. Enter a name for your Kafka schema. In this example, the schema is named kafkaexample.

6. Click Create Schema. The new schema is displayed.

7. In the pane on the left, expand the Objects node in the tree view.

8. Right click on Tables and click Add New Object.

   

   Figure 4: Adding a new table

9. Name the table and enter an LDAP Object Class name. In this example, the table is named worker. Click OK.

10. In the pane on the left, expand the table you created in the previous step.

11. Right click on Fields and select Add New Attribute.

12. Complete the Add New Attribute dialog box.

13. Repeat the above step for all fields which will be processed through your Kafka connector.

    

    Figure 5: Kafka Schema

14. Click Save.

In this section, a new naming context representing the outgoing Kafka Producer is added. If you plan to consume identity data to a Kafka topic, a separate view is required.

1. In the Main Control Panel, navigate to the Context Builder tab.

2. On the View Designer tab, click .

3. Name the view. In this example, the schema is named kafkaexample.

4. Select the schema you created in Configuring the Producer Schema. Click OK.

5. In the pane on the left, select the view’s folder icon and click New Content.

6. Select your table name and click OK.

   

   Figure 6: New View Definition

7. Make any updates you need for the RDN and Attribute settings.

   

   Figure 7: New View with Attributes

8. Click Save.

9. Navigate to the Directory Namespace tab.

10. Click .

11. Enter a context name, i.e. o=kafkaexample.

12. Choose Virtual Tree and click Next.

13. Select Use an existing view.

14. Click Browse. Navigate to the saved .dvx file for the view you created. Click OK.

15. Click OK.

16. Click OK to acknowledge the alert that your virtual tree has been created.

If you are configuring Radiant Logic Synchronization to act as both a consumer and producer with Kafka, you need a separate mounted view with the appropriate schema.

Synchronization uses the settings you configured for your kafkaproducer data source to call the correct changeConvertor, connect to your Kafka endpoint, and publish to the specified topic.

1. In the Main Control Panel, navigate to the Synchronization tab.

2. Click .

3. Select your Source and Destination naming contexts (i.e. the destination should be the kafkaproducer data source configured in Configuring the Producer Data Source) and click OK.

4. Click Configure.

   Note

   Assuming you are using an HDAP source, nothing is required to be configured on the Capture tile.

5. Click the Transformation tile and configure your mappings.

6. Click the Apply tile and start the pipeline.