Getting Started with Pipeline Configuration Files

Pipeline configuration files give you the ability to define pipelines that are provisioned by Conduit at startup. It's as simple as creating a YAML file that defines pipelines, connectors, processors, and their corresponding configurations.

Example pipeline

tip

In our Conduit repository, you can find more examples, but to ilustrate a simple use case we'll show a pipeline using a file as a source, and another file as a destination. Check out the different specifications to see the different configuration options.

Create a folder called pipelines at the same level as your Conduit binary. Inside of that folder create a file named file-to-file.yml.

│ # Conduit binary
├── conduit
│ # Folder with pipeline configurations
└── pipelines
    │ # Pipeline configuration file
    └── file-to-file.yml

Copy following content into file-to-file.yml:

pipelines/file-to-file.yml

version: 2.2
pipelines:
  - id: file-to-file
    # run pipeline on startup
    status: running
    description: >
      Example pipeline reading from file "example.in" and writing into file "example.out".

    connectors:
      - id: example.in
        type: source
        # use the built-in file plugin as the source
        plugin: builtin:file
        settings:
          # read data from example.in
          path: ./example.in

      - id: example.out
        type: destination
        # use the built-in file plugin as the destination
        plugin: builtin:file
        settings:
          # write data to example.out
          path: ./example.out
          # output the raw payload as a string
          sdk.record.format: template
          sdk.record.format.options: '{{ printf "%s" .Payload.After }}'

Now start Conduit. You should see a log line saying that the pipeline file-to-file was created:

$ ./conduit
/.../
2023-04-01T12:34:56+00:00 INF pipeline configs provisioned component=provisioning.Service created=["file-to-file"] deleted=[] pipelines_path=./pipelines
/.../

Conduit is now running the pipeline file-to-file which continuously reads lines added to file ./example.in and copies them to file ./example.out. Try writing a line to ./example.in and checking the content of ./example.out.

$ echo "hello conduit" >> example.in
$ cat example.out
hello conduit