Installing Standalone Connectors

Beside the built-in connectors shipped with Conduit there is a list of connectors that can be added to Conduit as plugins. These are called standalone connectors.

To install a standalone connector you first need the compiled connector binary. A binary can normally be downloaded from the latest release in the connector's GitHub repository (this may vary in 3rd party connectors not developed by the Conduit team). Make sure to download the binary that matches your operating system and architecture.

tip

Alternatively you can build the binary yourself as if you were installing your own custom standalone connector.

Installing a Connector in Conduit

Conduit loads standalone connectors at startup. The connector binaries need to be placed in the connectors directory relative to the Conduit binary so Conduit can find them. Alternatively, the path to the standalone connectors can be adjusted using the CLI flag -connectors.path, for example:

./conduit run --connectors.path=/path/to/connectors/

Names of the connector binaries are not important, since Conduit is getting the information about connectors from connectors themselves (using their gRPC API).

Find out how to reference your connector.

Record Size Limitations

When using standalone connectors, the maximum size of a record is limited by the gRPC go library grpc/grpc-go. This means that the maximum size of a record is limited to 4 MB by default. If you try to send a record larger than this limit, you will see an error like this in the logs:

2022-07-28T14:00:01+00:00 ERR node stopped 
  error="node stopped with error: failed to receive ack: 
         error receiving ack: grpc: received message larger than max (5000117 vs. 4194304)"

This limitation can be changed by setting the --connectors.max-receive-record-size flag when starting Conduit. For example, to set the maximum record size to 10 MB, you can run:

conduit run --connectors.max-receive-record-size 10485760

This option could be also be set in the configuration file or with the environment variable CONDUIT_CONNECTORS_MAX_RECEIVE_RECORD_SIZE.

warning

For this configuration to take effect, your standalone connector must be using conduit-connector-sdk >= v0.14.0