Skip to main content

Using Kafka Connect Connectors with Conduit

The Conduit Kafka Connect Wrapper connector is a special connector that allows you to use Kafka Connect connectors with Conduit. Conduit doesn't come bundled with Kafka Connect connectors, but you can use it to bring any Kafka Connect connector with Conduit.

This connector gives you the ability to:

  • Easily migrate from Kafka Connect to Conduit.
  • Remove Kafka as a dependency to move data between data infrastructure.
  • Leverage a datastore if Conduit doesn't have a native connector.

Since the Conduit Kafka Connect Wrapper itself is written in Java, but most of Conduit's connectors are written in Go, it also serves as a good example of the flexibility of the Conduit Plugin SDK.

Let's begin.

How it works

To use the Kafka Connect wrapper connector, you'll need to:

  1. Download the conduit-kafka-connect-wrapper latest release, at the time of writing this, we'll go with v0.4.3 and download the file
  2. Download Kafka Connect JARs and any dependencies you would like to add.
  3. Create a pipeline configuration file.
  4. Run Conduit.


To begin, download the conduit-kafka-connect-wrapper latest release, at the time of writing this, we'll go with v0.4.3 and download the file

Downloading the release is our preferred option to get the connector JAR. However, another option to get the JAR file is to build it from source, this could be useful in case you added your own changes to the connector and wanted to test them out, to do that we will clone the connector:

git clone [email protected]:ConduitIO/conduit-kafka-connect-wrapper.git

Then, we need to build the connector. The Kafka Connect wrapper connector is written in Java, so it needs to be compiled.

cd conduit-kafka-connect-wrapper


Running scripts/ will create a directory called dist with following contents:

  1. A script (which runs the connector). This script starts a connector instance.
  2. Directory libs. This is where you add the connector JAR itself, and put the Kafka connector JARs and their dependencies (if any).

Now that we have everything setup, we can add the Kafka Connect connectors.

Download a connector and its dependencies

The libs directory is where you put the Kafka Connect connector JARs and their dependencies (if any). The wrapper plugin will automatically load connectors and all the other dependencies from a libs directory.

To download a connector from a Maven repository and all of its dependencies, you can use scripts/ For example:

./scripts/ io.example jdbc-connector 2.1.3

For usage, run ./scripts/ --help.

You can also download them manually if needed, we will use the PostgreSQL Kafka Connect JDBC Connector. To install, add the following:

This connector allows you to connect to any JDBC database.

Create Pipeline Configuration File

Now that the Kafka Connect connectors included in lib, we can use it in a pipeline configuration file.

  1. Install Conduit.
  2. Create a pipeline configuration file: Create a folder called pipelines at the same level as your Conduit binary. Inside of that folder create a file named jdbc-to-file.yml, check Specifications for more details about Pipeline Configuration Files.
version: 2.0
- id: kafka-connect-pipeline
status: running
description: This pipeline is for testing
- id: jdbc-kafka-connect
type: source
plugin: standalone:conduit-kafka-connect-wrapper
wrapper.connector.class: "io.aiven.connect.jdbc.JdbcSourceConnector",
connection.url: "jdbc:postgresql://localhost/conduit-test-db",
connection.user: "username",
connection.password: "password", "id",
mode: "incrementing",
tables: "customers",
topic.prefix: "my_topic_prefix"
- id: file-dest
type: destination
plugin: builtin:file
path: "path/to/the/file.txt"
  1. Run Conduit! And see how simple it is to migrate to Conduit.

Note that the wrapper.connector.class should be a class which is present on the classpath, i.e. in one of the JARs in the libs directory. For more information, check the Wrapper Configuration section.