Adding built-in Connectors to Conduit

Built-in connectors offer better performance when compared to standalone ones, which is why in some cases it's desirable to have a custom build of Conduit that includes additional built-in connectors.

The simplest way to achieve so is to write a small application that embeds Conduit (i.e. uses Conduit as a library) and adds one or more connectors to its default configuration.

note

For the purpose of this example, we're going to add the Conduit Snowflake connector to Conduit as a built-in connector, and we're going to name our application custom-conduit-app pushed to github.com/conduitio-labs.

Initialize your own Conduit Application

First we'll create the directory structure for our custom Conduit application and initialize a Go module:

mkdir -p custom-conduit-app && touch custom-conduit-app/main.go
cd custom-conduit-app
go mod init github.com/conduitio-labs/custom-conduit-app

Add the connector as a built-in plugin

Once that is done, we need to write a main function that adds the Snowflake connector to the default Conduit configuration. Later we'll run Conduit with this custom configuration.

Write the main function

custom-conduit-app/main.go

package main

import (
	snowflake "github.com/conduitio-labs/conduit-connector-snowflake"
	"github.com/conduitio/conduit/cmd/conduit/cli"
	"github.com/conduitio/conduit/pkg/conduit"
)

func main() {
	// Get the default configuration, including all built-in connectors
	cfg := conduit.DefaultConfig()

	// Add the Snowflake connector to list of built-in connectors
	cfg.ConnectorPlugins["snowflake"] = snowflake.Connector

	cli.Run(cfg)
}

Add your dependencies

go get github.com/conduitio/conduit
go get github.com/conduitio-labs/conduit-connector-snowflake
go mod tidy

Build your application binary

This custom version of Conduit can be built with:

custom-conduit-app

go build -o custom-conduit main.go

Check the Snowflake connector is included

In order to interact with Conduit, you need to make sure you run the service first.

important

By running Conduit, you will also process any pipelines configured in the default configuration (./custom-conduit-app/pipelines).

Assuming you're only going through this guide, no pipelines have been configured yet, so no data would be processed.

custom-conduit-app

./custom-conduit run

In a different terminal session, you can check that the Snowflake connector has been included in the build by listing all the connector plugins as built-in:

Using the Conduit CLI

custom-conduit-app

./custom-conduit connector-plugins list
+---------------------------+----------------------------------------------------------------------+
|           NAME            |                               SUMMARY                                |
+---------------------------+----------------------------------------------------------------------+
| builtin:[email protected]      | A file source and destination plugin for Conduit.                    |
| builtin:[email protected] | A plugin capable of generating dummy records (in different formats). |
| builtin:[email protected]     | A Kafka source and destination plugin for Conduit, written in Go.    |
| builtin:[email protected]        | A destination connector that logs all incoming records.              |
| builtin:[email protected]  | Conduit connector for PostgreSQL                                     |
| builtin:[email protected]         | An S3 source and destination plugin for Conduit, written in Go.      |
| builtin:[email protected]  | An Snowflake source plugin for Conduit, written in Go.               |
+---------------------------+----------------------------------------------------------------------+

Using the API

curl localhost:8080/v1/connectors/plugins | jq '.[].name'
"builtin:[email protected]"
"builtin:[email protected]"
"builtin:[email protected]"
"builtin:[email protected]"
"builtin:[email protected]"
"builtin:[email protected]"
"builtin:[email protected]"

Next steps

Once you have confirmed that your Conduit Connector is available in your application, you are ready to transfer any pipeline you had previously tested to the custom Conduit application:

custom-conduit-app

mkdir pipelines
mv $PREVIOUSLY_TESTED_CONDUIT/pipelines/* pipelines/
./custom-conduit run

Or configure it to read from the desired pipelines directory:

custom-conduit-app

./custom-conduit run --pipelines.path $YOUR_PIPELINES_DIRECTORY

note

If you, on the other hand, didn't have any pipeline previously configured and you'd like to get started, check out this page on how to build a pipeline. Just remember that any conduit command from the terminal would need to be replaced by your custom conduit binary ./custom-conduit.