Kafka Consumer Connector

The RadiantOne Kafka capture connector leverages the Apache Kafka Consumer API to subscribe to topics and process streams of records produced by them.

The Kafka capture and apply connectors are independent, meaning that they do not need to be deployed together. The source(s) and target(s) could be any combination of Kafka topics, LDAP-accessible data stores, JDBC-accessible data stores, or web-services (SCIM or REST accessible).

This section assumes that you have access to an existing Kafka instance that you can connect to. Once you gain access, follow the instructions in this section.

To sync between RadiantOne and Kafka topics, you need to configure the following.

• Data source
• Schemas
• Views
• Sync topologies

A data source is required for the Consumer connector.

To configure the Consumer data source:

1. In the Main Control Panel, navigate to Settings > Server Backend > Custom Data Sources.

2. From the Add Custom menu, select Kafka.

3. Name the data source. In this example, the data source is named kafkaconsumer.

4. Enter any value for each of the following properties; a value for these properties is required but ignored.

   • Topic Name
   • Message Format
   • Producer Properties File

   

   Figure 1: Kafka Consumer Data Source

   Note

   Additional data source settings are configured in Configuring synchronization as a Kafka Consumer.

5. Click Save.

   Note

   If a note displays stating that the connection to the data source failed, click Yes to save anyway.

If you use the same schema for both your consumer and producer, no further schema configuration is required. If you plan to use a different message format for publishing to Kafka, that schema must also be created. See Configuring the Producer Schema for more information.

In this section, a new naming context representing the incoming Kafka Consumer is added. If you plan to publish identity data to a Kafka topic, a separate view is required. If you use both a consumer and producer, and have already setup your producer schema, no further schema configuration is required.

1. In the Main Control Panel, navigate to the Context Builder tab.

2. On the View Designer tab, click .

3. Name the view. In this example, the schema is named kafkaExample.

4. Select the schema you created in Configuring the Schema. Click OK.

5. In the pane on the left, select the view’s folder icon and click New Content.

6. Select your table name and click OK.

   

   Figure 2: New View Definition

7. Make any updates you need for the RDN and Attribute settings.

   

   Figure 3: New View with Attributes

8. Click Save.

9. Navigate to the Directory Namespace tab.

10. Click .

11. Enter a context name, i.e. o=kafkaexample.

12. Choose Virtual Tree and click Next.

13. Select Use an existing view.

14. Click Browse. Navigate to the saved .dvx file for the view you created. Click OK.

15. Click OK.

16. Click OK to acknowledge the alert that your virtual tree has been created.

If you’re configuring synchronization to act as both a consumer and producer with Kafka, you need a separate mounted view with the appropriate schema.

1. In the Main Control Panel, navigate to the Synchronization tab.

2. Click .

3. Select your source and destination naming contexts and click OK.

4. Click Configure.

5. Click the Capture tile and fill in the following fields.

   Field Name	Description
   Topic Name	The name of your topic (i.e. Workday).
   Kafka Consumer Properties	Contains bootstrap.servers= followed by the name and port number for your Kafka server. See note below for more information.
   Message Format	The name of your changeConvertor, such as GoldenGateJSONConvertor or KafkaExample (without .java on the end)
   Table Name	Required only if you are using KafkaGoldenGate formatting

   

   Figure 4: Capture connector properties using KafkaExample changeConvertor

bootstrap.servers=kafka.mycompany.com:9094,sasl.mechanism=PLAIN,security.protocol=SASL_SSL,ssl.truststore.location=/radiantone/vds/vds_server/custom/truststore.jks,sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="accountname" password="password";

1. Click Save.

2. Click the Transformation tile. Select a Transformation Type from the drop-down menu.

3. Expand the Mappings section and map attributes as required.

   

   Figure 5: Sample Mappings

4. Click the Apply tile and start your pipeline.