In this article, we are going to walk through, step by step, how to build a Conduit connector.
Conduit connectors communicate with Conduit by either writing records into the pipeline (source connector) and/or the other way around (destination connector).
For this example, we are going to build an Algolia destination connector. The goal of this connector is to give the user the ability to send data to Algolia. In the context of search engines, this is called indexing. Since Conduit is a generic tool to move data between data infrastructure, with this new connector we can index data from any Conduit Source (PostgreSQL, Kafka, etc.).
By default, Conduit ships with a REST API that allows you to automate the creation of data pipelines and connectors. To make it easy to get started with the API, we have provided a Swagger UI to visualize and interact with the Condiut without having to write any code...yet 😉.
After you start Conduit, if you navigate to http://localhost:8080/openapi/, you will see a page that looks like this:
Then, after you test the API, you can write code to make the equilivent request. For example, here is how you would make a request using the axios Node.js library.
In this guide, we will build a data pipline that moves data between files. This example is a great to get started with Conduit on a local machine, but it's also the foundation of use cases such as log aggregation.
Everytime that data is appended to the src.log, data will be move in real-time to dest.log.