Vector v0.22.0 release notes
- The
journaldsource deadlocks almost immediately (#12966). Fixed in v0.22.1. - The
kubernetes_logssource does not work with k3s/k3d (#12989). Fixed in v0.22.1. - Vector would panic when reloading configuration using the
compressionorconcurrencyoptions due to a deserialization failure (#12919). Fixed in v0.22.1. - When using a component that creates a unix socket,
vector validateno longer creates the socket (#13018). This causes the default SystemD unit file to fail to start Vector since it runsvector validatebefore starting Vector. Fixed in v0.22.1. - VRL sometimes miscalculates type definitions when conditionals are used causing later usages of values assigned in conditionals to not require type coercion as they should (#12948). Fixed in v0.22.1.
- Metrics from AWS components were tagged with an
endpointincluding the full path of the request. For theaws_s3sink this caused cardinality issues since the AWS S3 key is included in the URL. Fixed in v0.22.3. - The
gcp_pubsubsource would log errors due to attempting to fetch too quickly when it has no acknowledgements to pass along. Fixed in v0.22.3. - Vector shuts down when a configured source codec (
decoding.codec) receives invalid data. Fixed in v0.23.1.
The Vector team is pleased to announce version 0.22.0!
Be sure to check out the upgrade guide for breaking changes in this release.
Important: as part of this release, we have promoted the new implementation of disk buffers (buffer.type = "disk_v2") to the default implementation (buffer.type = "disk"). Any existing disk buffers (disk_v1
or disk) will be automatically migrated. We have rigorously tested this migration, but recommend making
a back up of the disk buffers (in the configured data_dir, typically in /var/lib/vector) to roll back if
necessary. Please see the release highlight for additional
updates about this migration.
In addition to the new features, enhancements, and fixes listed below, this release adds:
- Support for iteration has landed in VRL. Now you can
dynamically map unknown key/value pairs in objects and items in arrays. This replaces some common use cases for
the
luatransform with the much more performantremaptransform. - New native event codecs for Vector. We are still rolling out the
new codec support to all sinks, but this will allow sending events (logs, metrics, and traces) between Vector
instances via transports like
kafkarather than being limited to the gRPCvectorsource and sink. - A new GCP PubSub (
gcp_pubsub) source to consume events from GCP PubSub. - A new
websocketsink was added to send events to a remote websocket listener. - New VRL functions for encrypting and decrypting data.
We also made additional performance improvements this release increasing the average throughput by up to 50% for common topologies (see our soak test framework).
| experiment | Δ mean | Δ mean % | confidence |
|---|---|---|---|
| splunk_transforms_splunk3 | 5.98MiB | 58.22 | 100.00% |
| datadog_agent_remap_blackhole | 20.1MiB | 43.44 | 100.00% |
| splunk_hec_route_s3 | 5.28MiB | 35.34 | 100.00% |
| syslog_regex_logs2metric_ddmetrics | 1.84MiB | 15.62 | 100.00% |
| syslog_log2metric_splunk_hec_metrics | 2.52MiB | 15.59 | 100.00% |
| datadog_agent_remap_datadog_logs | 9.6MiB | 15.02 | 100.00% |
| http_to_http_json | 2.78MiB | 13.19 | 100.00% |
| syslog_humio_logs | 1.91MiB | 12.23 | 100.00% |
| syslog_splunk_hec_logs | 1.85MiB | 12.11 | 100.00% |
| syslog_loki | 1.42MiB | 9.53 | 100.00% |
Vector Changelog
7 new features
VRL now includes two new functions for encrypting and decrypting field values: encrypt and decrypt. A random_bytes function was added to make it easy to generate initialization vectors for the
encryptfunction.See the highlight for more details about this new functionality.
- The
socketandsyslogsources now allow configuration of the permissions to use when creating a unix socket viasocket_file_modewhenmode = "unix"is used.
Thanks to @Sh4d1 for contributing this change! - VRL now allows for a simple form of string templating via
{{ some_variable }}syntax. We will be expanding support for templating over time. This does mean that any strings that had{{ }}in them already now need to be escaped. See the upgrade guide for details. - Vector has two new codecs that can be used on sources and sinks to encode as Vector’s native
representation:
nativeandnative_json. This makes it easier to send events between Vector instances on transports likekafka. It also makes it possible to send metrics to Vector from an external process (such as when using theexecsource) without needing to use theluatransform to convert logs to metrics. Previously, these generic sources (likeexecorhttp) could only receive logs. See the release highlight for more about this new feature and how to use it. - A new
gcp_pubsubsource was added for consuming events from GCP PubSub. - A new
websocketsink was added for sending events to a remote websocket listener.
Thanks to @zshell31 for contributing this change! - A new
is_jsonfunction was added to VRL. This allows more efficient checking of whether the incoming value is JSON vs. trying to parse it usingparse_jsonand checking if there was an error.
Thanks to @nabokihms for contributing this change!
13 enhancements
- The
journaldsource now processes data more efficiently by continuing to read new data while waiting for read data to be processed by Vector. Vector’s configuration interpolation of environment variables has been enhanced to both allow setting of default values and returning an error message if an expected environment variable is unset or empty. The syntax matches bash interpolation syntax:
${VARIABLE:-default}evaluates to default if VARIABLE is unset or empty in the environment.${VARIABLE-default}evaluates to default only if VARIABLE is unset in the environment.${VARIABLE:?err}exits with an error message containing err if VARIABLE is unset or empty in the environment.${VARIABLE?err}exits with an error message containing err if VARIABLE is unset in the environment.
Thanks to @hhromic for contributing this change!- The Datadog sinks now retry requests that failed due to an invalid API key. This avoids data loss in the case that an API key is revoked.
- The
kubernetes_logssource now tags emitted internal metrics withpod_namespace. - The
datadog_metricssink now supports sending aggregated summary metrics (typically scraped from a Prometheus exporter) to Datadog. Previously these metrics were dropped at the sink. - The
datadog_metricssink now supports sending aggregated summary metrics (typically scraped from a Prometheus exporter) to Datadog. Previously these metrics were dropped at the sink. - The RPM package now adds the created
vectoruser to thesystemd-journal-remotegroup to be able to consume journald events from a remote system. This matches the Debian package. - The
kubernetes_logssource now allows configuration ofextra_namespace_label_selectorwhich Vector will to use select the pods to capture the logs of, if set, based on labels attached to the pod namespace. This is similar to theextra_label_selectoroption which applies to pod labels.
Thanks to @anapsix for contributing this change! - The
kubernetes_logssource now reads events in order whenever a pod log file rotates. Previously Vector could start reading the new file before it finished processing the previous one, resulting in the logs being out-of-order.
Thanks to @sillent for contributing this change! The
parse_jsonfunction now takes an optionalmax_depthparameter to control how far it will recurse when deserializing the event. Once the depth limit is hit, the remainder of the fields is left as raw JSON in the deserialized event.For example:
parse_json!("{"1": {"2": {"3": {"4": {"5": {"6": "finish"}}}}}}", max_depth: 5)Yields:
{ "1": { "2": { "3": { "4": { "5": "{"6": "finish"}" } } } } }The default remains no max depth limit.
Thanks to @nabokihms for contributing this change!- A new
component_received_events_counthistogram metric was added to record the sizes of event batches passed around in Vector’s internal topology. Note that this is different than sink-level batching. It is mostly useful for debugging performance issues in Vector due to low internal batching. - The
httpsource now allows configuration of the HTTP method to expect requests with via the newmethodoption. Previously it only allowed POST requests.
Thanks to @r3b-fish for contributing this change! - All components now emit consistent metrics in accordance with Vector’s component specification.
16 bug fixes
- The
splunk_hecsource now correctly handles negative acknowledgements from sinks. Previously it would mark the request including the rejected events as delivered. In Splunk’s acknowledgement protocol, this means returningtruefor theackIDfor the request, but now it correctly returnsfalse, indicating the request is not acknowledged. The
gcp_stackdriver_metricssink now requires configuration of labels at the top-level to match thegcp_stackdriver_logssink. Previously these were nested under.labels.See the upgrade guide for more details.
- The
new_relicsink health check now considers any 200-level response a success. It used to require a 200 which did not match what New Relic actually returns: 202. - When using Vector’s ability to load configuration from a directory (
--config-dir), Vector now ignores subdirectories starting with a.. - The
aws_s3sink now only sets thex-amz-taggingheader if tags are being applied. Specifying an empty value was incompatible with Ceph. - The VRL type definition for
.andparse_xmlwas corrected to be a map of any field/value rather than specifically an empty map. This could cause later false positives with type issues during VRL compilation. VRL now correctly updates the type definition of variables defined in one scope, that are mutated in another.
For example:
foo = 1 { foo = "bar" } upcase(foo)Would previously fail to compile because VRL thinks
foois an integer when, in fact, it has been reassigned to a string.- The
internal_metricssource now correctly tags emitted metrics withhostandpidwhenhost_keyandpid_keyare configured, respectively, on theinternal_metricssource. - The
socketsource now discards UDP frames greater than the configuredmax_length(whenmode = "udp"). Previously these were truncated rather than discarded, which did not match the behavior whenmode = "tcp". Allsocketsource modes are now consistent with dropping messages greater thanmax_length. - The
internal_logssource occasionally missed some events generated early in Vector’s start-up, before the component was initialized. This was remedied so that theinternal_logssource more reliably captures start-up events. - The
parse_ruby_hashVRL function can now parse hashes that contain a symbol as the value, such as{ "key" => :foo }. - The
logfunction in VRL no longer wraps logged string values in quotes. This was causing double quoting for sink encodings likejson.
Thanks to @nabokihms for contributing this change! - The
aws_s3source now handles S3 object keys that contain spaces. Previously Vector would encounter a 404 when querying for objects due to not decoding spaces correctly from the SQS object notification. - GCP sinks now correctly handle authentication token refreshing from the metadata service when the health check fails.
- The
httpconfig provider now correctly repolls when an error is encountered.
Thanks to @jorgebay for contributing this change! - The
logfmtsink codec and as theencode_logfmtfunction now correctly wrap values that contain quotes (") in quotes and escape the inner quote.
Thanks to @jalaziz for contributing this change!
1 chore
- Vector now automatically migrates disk buffers from
disk_v1todisk_v2as described in Switching the default implementation of disk buffers todisk_v2. In a future release, support forv1disk buffers will be dropped.
What’s next
v2
implementation. This means if you set type = "disk" you will get the new buffer implementation. In a future release, we will remove the legacy
disk buffers. To continue using the v1 disk buffers, for now, set type = "disk_v1".