Skip to main content

avro.decode

Decodes a field's raw data in the Avro format.

Description​

The processor takes raw data (bytes or a string) in the specified field and decodes it from the Avro format into structured data. It extracts the schema ID from the data, downloads the associated schema from the schema registry and decodes the payload. The schema is cached locally after it's first downloaded.

If the processor encounters structured data or the data can't be decoded it returns an error.

This processor is the counterpart to avro.encode.

Configuration parameters​

version: 2.2
pipelines:
  - id: example
    status: running
    connectors:
      # define source and destination ...
    processors:
      - id: example
        plugin: "avro.decode"
        settings:
          # The password to use with basic authentication. This option is
          # required if auth.basic.username contains a value. If both
          # auth.basic.username and auth.basic.password are empty basic
          # authentication is disabled.
          # Type: string
          auth.basic.password: ""
          # The username to use with basic authentication. This option is
          # required if auth.basic.password contains a value. If both
          # auth.basic.username and auth.basic.password are empty basic
          # authentication is disabled.
          # Type: string
          auth.basic.username: ""
          # The field that will be decoded.
          # For more information about the format, see [Referencing
          # fields](https://conduit.io/docs/processors/referencing-fields).
          # Type: string
          field: ".Payload.After"
          # The path to a file containing PEM encoded CA certificates. If this
          # option is empty, Conduit falls back to using the host's root CA set.
          # Type: string
          tls.ca.cert: ""
          # The path to a file containing a PEM encoded certificate. This option
          # is required if tls.client.key contains a value. If both
          # tls.client.cert and tls.client.key are empty TLS is disabled.
          # Type: string
          tls.client.cert: ""
          # The path to a file containing a PEM encoded private key. This option
          # is required if tls.client.cert contains a value. If both
          # tls.client.cert and tls.client.key are empty TLS is disabled.
          # Type: string
          tls.client.key: ""
          # URL of the schema registry (e.g. http://localhost:8085)
          # Type: string
          url: ""

Examples​

Decode a record field in Avro format​

This example shows the usage of the avro.decode processor. The processor decodes the record's.Key field using the schema that is downloaded from the schema registry and needs to exist under the subjectexample-decode. In this example we use the following schema:

{
  "type":"record",
  "name":"record",
  "fields":[
    {"name":"myString","type":"string"},
    {"name":"myInt","type":"int"}
  ]
}

Configuration parameters​

version: 2.2
pipelines:
  - id: example
    status: running
    connectors:
      # define source and destination ...
    processors:
      - id: example
        plugin: "avro.decode"
        settings:
          field: ".Key"
          url: "http://127.0.0.1:54322"

Record difference​

Before
After
1
{
1
{
2
  "position": "dGVzdC1wb3NpdGlvbg==",
2
  "position": "dGVzdC1wb3NpdGlvbg==",
3
  "operation": "create",
3
  "operation": "create",
4
  "metadata": {
4
  "metadata": {
5
    "key1": "val1"
5
    "key1": "val1"
6
  },
6
  },
7
-
  "key": "\u0000\u0000\u0000\u0000\u0001\u0006bar\u0002",
7
+
  "key": {
8
+
    "myInt": 1,
9
+
    "myString": "bar"
10
+
  },
8
  "payload": {
11
  "payload": {
9
    "before": null,
12
    "before": null,
10
    "after": null
13
    "after": null
11
  }
14
  }
12
}
15
}


scarf pixel conduit-site-docs-processors