Skipping bad records with the Kafka Connect JDBC sink connector
How to manually skip bad records when using the Kafka Connect JDBC sink connector to prevent pipeline halts.
Robin Moffatt is a Principal DevEx Engineer and seasoned conference speaker with 15+ years of experience presenting at top events like QCon, Devoxx, Kafka Summit, and Strata. He shares insights on developer experience, distributed systems, and cloud technologies through his blog, YouTube, and public talks.
491 articles from this blog
How to manually skip bad records when using the Kafka Connect JDBC sink connector to prevent pipeline halts.
A guide to using Kafka Connect to stream data from Kafka topics to Elasticsearch 7, covering configuration, common errors, and schema options.
A tutorial on using kafkacat to copy data between Kafka clusters via Unix pipelines for quick, simple data replication.
A Developer Advocate shares practical advice on managing time, avoiding burnout, and maintaining well-being while traveling extensively for tech conferences and meetups.
A developer's travel itinerary for tech conferences and meetups focused on Apache Kafka and software architecture for late 2019.
A guide to resetting offsets for Kafka Connect source connectors by modifying data in the Kafka offset topic, using tools like kafkacat.
How to configure a Kafka Connect sink connector to start reading from the end of a topic instead of the beginning.
A guide on troubleshooting and resetting Kafka consumer group offsets when using Replicator to copy data between clusters.
A technical guide on manually migrating Alfred's clipboard history database between Macs using symbolic links and Dropbox.
An updated guide on creating technical diagrams using Concepts app on iPad, detailing a workflow for presentations and talks.
How to manually delete a Kafka Connect connector by writing a tombstone message to its internal config topic using kafkacat.
A technical guide on automating the restart of failed Kafka Connect tasks using bash scripts and the REST API.
A guide to securely externalizing secrets like passwords in Kafka Connect configuration using KIP-297 and the FileConfigProvider.
Explains how to delete a Kafka Connect connector by directly modifying its configuration topic when the REST API is unavailable.
A workaround technique for exploding arrays in ksqlDB using manual indexing, simulating EXPLODE/UNNEST functionality.
Explains the role of Kafka Connect converters, demonstrating that JSON source data doesn't automatically become structured messages.
A technical guide on how to read Kafka Connect offset data using the Confluent REST Proxy over HTTP, including example commands.
A technical tutorial on using CASE statements in ksqlDB to pivot and aggregate event stream data, similar to SQL PIVOT operations.
Guide on configuring KSQL to connect to a secured Confluent Cloud Schema Registry, including Docker setup and error troubleshooting.
A technical guide exploring stream-stream joins in KSQL using orders and shipments data, covering setup, aggregations, and join implementation.