In a previous article, I explained how to evolve JSON Schemas, based on whether backward, forward, or full (both backward and forward) compatibility is desired. In JSON Schema, the value of additionalProperties determines whether the JSON content model is open (if the value is true), closed (if the value is false), or partially open (if […]
The Rust Schema Registry Client is a fully asynchronous Rust client library for interacting with the Confluent Schema Registry. It allows you to enforce a Data Contract between a producer and a consumer. For example, you can Serialize and deserialize Kafka records using Avro, Protobuf, and JSON Schema Specify data quality rules using Google Common […]
JSONata is an expressive, declarative functional language that is as powerful as older transformation technologies such as XSLT, but with a concise syntax. When working with Confluent Schema Registry, JSONata can be used to define complex schema migration rules that can ensure that schema evolution does not break existing consumers. Another area where JSONata can […]
Today’s data scientist has a plethora of options when processing data in Apache Kafka. In terms of stream processing, popular options are Kafka Streams, ksqlDB, and Apache Flink. In terms of real-time analytics, one can choose from Apache Druid, Clickhouse, Apache Pinot, and Apache Doris. One aspect of these systems is that they all support […]
The following post originally appeared in the Confluent blog on October 18, 2023. The Confluent Schema Registry plays a pivotal role in ensuring that producers and consumers in a streaming platform are able to communicate effectively. Ensuring the consistent use of schemas and their versions allows producers and consumers to easily interoperate, even when schemas […]
The Confluent Schema Registry recently added support for tags, metadata, and rules, which together support the concept of a data contract. Data quality rules in a data contract can be expressed using the Google Common Expression Language (CEL), while migration rules can be expressed using JSONata. In this article, I’ll provide some tips on understanding […]
JSONata is a declarative functional language for querying and transforming JSON data. It was inspired by the path semantics of XPath. Whereas other languages that are inspired by XPath also incorporate SQL-like elements, JSONata stays true to its XPath roots and augments the path semantics of XPath with additional features to increase its power. JSONata […]
While working with Apache Kafka and Confluent Schema Registry, I often spend time switching between tools like kcat (formerly kafkacat) the Kafka Avro console consumer/producer the Kafka JSON Schema console consumer/producer the Kafka Protobuf console consumer/producer the Schema Registry Maven plugin Recently I decided to combine much of the functionality of the above tools into […]
In a recent article, I presented rules for evolving JSON Schemas in a backward, forward, and fully compatible manner. The backward compatibility rules for Protocol Buffers (Protobuf) are much simpler, and most of them are outlined in the section “Updating A Message Type” in the Protobuf Language Guide. The Confluent Schema Registry uses these rules […]
“The world is made of events, not things.” — Carlo Rovelli “Every company is becoming software.” — Jay Kreps In The Order of Time, the physicist Carlo Rovelli argues that the theory of relativity compels us to view the world not as made of things or entities, but as events or processes. In the world […]