Skip to main content

Getting Started with Pipeline Configuration Files

Pipeline configuration files give you the ability to define pipelines that are provisioned by Conduit at startup. It's as simple as creating a YAML file that defines pipelines, connectors, processors, and their corresponding configurations.

Example pipeline (file to file)

Create a folder called pipelines at the same level as your Conduit binary. Inside of that folder create a file named file-to-file.yml.

# Conduit binary
├── conduit
# Folder with pipeline configurations
└── pipelines
# Pipeline configuration file
└── file-to-file.yml

Copy following content into file-to-file.yml:

version: 2.0
- id: file-to-file
# run pipeline on startup
status: running
description: >
Example pipeline reading from file "" and writing into file "example.out".

- id:
type: source
# use the built-in file plugin as the source
plugin: builtin:file
# read data from
path: ./

- id: example.out
type: destination
# use the built-in file plugin as the destination
plugin: builtin:file
# write data to example.out
path: ./example.out
# output the raw payload as a string
sdk.record.format: template
sdk.record.format.options: '{{ printf "%s" .Payload.After }}'

Now start Conduit. You should see a log line saying that the pipeline file-to-file was created:

$ ./conduit
2023-04-01T12:34:56+00:00 INF pipeline configs provisioned component=provisioning.Service created=["file-to-file"] deleted=[] pipelines_path=./pipelines

Conduit is now running the pipeline file-to-file which continuously reads lines added to file ./ and copies them to file ./example.out. Try writing a line to ./ and checking the content of ./example.out.

$ echo "hello conduit" >>
$ cat example.out
hello conduit