Skip to main content

How to configure Conduit

Conduit accepts CLI flags, environment variables and a configuration file to configure its behavior. Each CLI flag has a corresponding environment variable and a corresponding field in the configuration file. Conduit uses the value for each configuration option based on the following priorities:

CLI flags​

CLI flags (highest priority) - if a CLI flag is provided it will always be respected, regardless of the environment variable or configuration file. To see a full list of available flags run conduit --help:

Conduit CLI is a command-line that helps you interact with and manage Conduit.

Usage:
  conduit [flags]
  conduit [command]

Pipelines
  pipelines   Initialize and manage pipelines

Additional Commands:
  help        Help about any command
  init        Initialize Conduit with a configuration file and directories.

Flags:
      --api.enabled                                            enable HTTP and gRPC API (default true)
      --connectors.path string                                 path to standalone connectors' directory (default "/home/haris/projects/conduitio/conduit/connectors")
      --db.badger.path string                                  path to badger DB (default "/home/haris/projects/conduitio/conduit/conduit.db")
      --db.postgres.connection-string string                   postgres connection string, may be a database URL or in PostgreSQL keyword/value format
      --db.postgres.table string                               postgres table in which to store data (will be created if it does not exist) (default "conduit_kv_store")
      --db.sqlite.path string                                  path to sqlite3 DB (default "/home/haris/projects/conduitio/conduit/conduit.db")
      --db.sqlite.table string                                 sqlite3 table in which to store data (will be created if it does not exist) (default "conduit_kv_store")
      --db.type string                                         database type; accepts badger,postgres,inmemory,sqlite (default "badger")
      --grpc.address string                                    address for serving the gRPC API (default ":8084")
  -h, --help                                                   help for conduit
      --http.address string                                    address for serving the HTTP API (default ":8080")
      --log.format string                                      sets the format of the logging; accepts json, cli (default "cli")
      --log.level string                                       sets logging level; accepts debug, info, warn, error, trace (default "info")
      --pipelines.error-recovery.backoff-factor int            backoff factor applied to the last delay (default 2)
      --pipelines.error-recovery.max-delay duration            maximum delay before restart (default 10m0s)
      --pipelines.error-recovery.max-retries int               maximum number of retries (default -1)
      --pipelines.error-recovery.max-retries-window duration   amount of time running without any errors after which a pipeline is considered healthy (default 5m0s)
      --pipelines.error-recovery.min-delay duration            minimum delay before restart (default 1s)
      --pipelines.exit-on-degraded                             exit Conduit if a pipeline enters a degraded state
      --pipelines.path string                                  path to the directory that has the yaml pipeline configuration files, or a single pipeline configuration file (default "/home/haris/projects/conduitio/conduit/pipelines")
      --preview.pipeline-arch-v2                               enables experimental pipeline architecture v2 (note that the new architecture currently supports only 1 source and 1 destination per pipeline)
      --processors.path string                                 path to standalone processors' directory (default "/home/haris/projects/conduitio/conduit/processors")
      --schema-registry.confluent.connection-string string     confluent schema registry connection string
      --schema-registry.type string                            schema registry type; accepts builtin,confluent (default "builtin")
  -v, --version                                                version for conduit

Use "conduit [command] --help" for more information about a command.

Environment variables​

Environment variables (lower priority) - an environment variable is only used if no CLI flag is provided for the same option. Environment variables have the prefix CONDUIT and contain underscores instead of dots and hyphens (e.g. the flag -db.postgres.connection-string corresponds to CONDUIT_DB_POSTGRES_CONNECTION_STRING).

Configuration file​

Configuration file (lowest priority) - Conduit by default loads the file conduit.yaml placed in the same folder as Conduit. The path to the file can be customized using the CLI flag -config. It is not required to provide a configuration file and any value in the configuration file can be overridden by an environment variable or a flag. The file content should be a YAML document where keys can be hierarchically split on .. For example:

db:
  type: postgres # corresponds to flag -db.type and env variable CONDUIT_DB_TYPE
  postgres:
    connection-string: postgres://localhost:5432/conduitdb # -db.postgres.connection-string or CONDUIT_DB_POSTGRES_CONNECTION_STRING

scarf pixel conduit-site-docs-using