Kafka Connect - Connector Plugin

About

Connector is a component of the connect framework that coordinates data streaming by managing tasks

A connector instance is a logical job. Each connector instance coordinates a set of tasks that actually copy the data.

A connector is run by a worker.

Articles Related

Management

See:

• Distributed
• Standalone

The easiest way to create, configure, and manage connectors is with Confluent Control Center.

State

The following states are possible for a connector or one of its tasks:

• UNASSIGNED: The connector/task has not yet been assigned to a worker.
• RUNNING: The connector/task is running.
• PAUSED: The connector/task has been administratively paused.
• FAILED: The connector/task has failed (usually by raising an exception, which is reported in the status output).

Concept

Type

• Sink
• Source

Built-in consumer, producer

Kafka Connect uses the standard Java producer and consumers to communicate with Kafka.

• Consumer
• Producer

All producer configs and consumer configs can be overridden by prefixing them with producer. or consumer., respectively.

List

Actual Connector

See list connector available for a worker (in distributed mode).

Externally Available

What connectors can I install ?

• https://www.confluent.io/product/connectors/
• Connector hub

Dev Guide / Build

• https://docs.confluent.io/current/connect/devguide.html

• Archetype

mvn archetype:generate -DarchetypeGroupId=com.github.jcustenborder.kafka.connect -DarchetypeArtifactId=kafka-connect-quickstart -DarchetypeVersion=0.11.0.0-cp1

• https://medium.com/@criccomini/so-you-want-to-build-a-kafka-connector-source-edition-357dce7d55f4

Configuration

Connector configuration file are given:

• when a worker is started in standalone mode
• via Rest Api for the distributed mode. See Distributed. The Connector configuration is persisted into Kafka. Every time you start a worker, it will start all the connectors that were running when it was stopped. The configuration data will be persisted in a the internal topic connect-configs.

Common properties:

• name - Unique name for the connector. Attempting to register again with the same name will fail.
• connector.class - The Java class for the connector
• tasks.max - The maximum number of tasks that should be created for this connector. The connector may create fewer tasks if it cannot achieve this level of parallelism.
• key.converter and value.converter - (optional) to override the default converter configuration

Sink connectors:

• topics - A list of topics to use as input for this connector

Offset

Even if you delete and recreate the connector, if the connector has the same name it will retain the same offsets previously stored.

Documentation / Reference

• Kafka Connector
• http://docs.confluent.io/current/connect/index.html