Vector v0.19.0 release notes
- A regression was introduced that changed the name of the data directory for sinks using a disk buffer. This means, when upgrading from
v0.18.0, if there was any data in existing disk buffers, it will not be sent. Vector’s starting with clean or empty disk buffers are unaffected. Fixed in v0.19.1. See #10430 for more details. - Presence of
framing.character_delimited.delimiterin configs causes Vector to fail to start withinvalid type: string. Fixed in v0.19.1. - When using
decoding.codecon sources, invalid data will cause the source to cease processing. Fixed in v0.19.2. encoding.only_fieldsfailed to deserialize correctly for sinks that used fixed encodings (i.e. those that don’t haveencoding.codec). Fixed in v0.19.2.- Buffers using
when_fullofblockwere incorrectly countingbuffer_events_totalby including discarded events. Fixed in v0.19.2. - Transforms neglect to tag logs and metrics with their component span tags (like
component_id). Fixed in v0.19.2.
The Vector team is pleased to announce version 0.19.0!
In addition to the below features, enhancements, and fixes, we’ve been
hard at work improving Vector’s performance and were able to move the
needle 10-100% for most configurations in our soak test
framework
from the last release, v0.18.
Be sure to check out the upgrade guide for breaking changes in this release.
Vector Changelog
6 new features
- The
splunk_hecsource and sink have added support for the acknowledgement part of Splunk’s HTTP Event Collector protocol. This improves delivery guarantees for data from Splunk clients and when sending events to Splunk. See the highlight article for more details. - The
datadog_agentsource is now able to accept metrics from the Datadog Agent; however, some changes are pending in the Datadog Agent to be able to send metrics to Vector. We expect this to be released in version 6.33 / 7.33 of the Datadog Agent. - The
humio_logssink now transmits sub-millisecond timestamps to Humio. - The
splunk_hecsource and sink have added support for passing the channel token events were sent with from the source to the sink. This makes it easier to use Vector in-between a Splunk sender and receiver to transform the data. See the highlight article for more details. - A new config option has been added to the
elasticsearchsink to allow suppressing thetypefield from being sent by Vector. This field is deprecated in Elasticsearch v7 and will be removed in v8. - A new
connection_limitoption has been added to TCP-based sources likesocketandsyslogto limit the number of allowed TCP connections. This can be useful to limit resource utilization by Vector.
10 enhancements
- Added end-to-end acknowledgement for the
aws_s3source. VRL now allows for writing multi-line string literals by ending the line with a backslash (\).
Example:
.thing = "foo \ bar" assert(.thing == "foo bar")A couple of enhancements have been made to the
influxdb_logssink to modify how Vector encodes events.A
measurementconfig field was added to allow overriding the measurement name (previously hardcoded tovector). Along with this thenamespaceoption was made optional asmeasurementcan be used to set the full name directly.For example:
[sinks.log_to_influxdb] type = "influxdb_logs" measurement = "vector-logs" endpoint = "http://localhost:9999"Now outputs events like:
vector-logs,metric_type=logs,host=example.com message="hello world",size=10 {timestamp}A
metric_typeconfig field was added to allow customizing the metric type (previously hardcoded tolog).For example:
[sinks.log_to_influxdb] type = "influxdb_logs" measurement_type = "foo" endpoint = "http://localhost:9999"Now outputs events like:
ns.vector,metric_type=foo,host=example.com message="hello world",size=10 {timestamp}
Thanks to juvenn for contributing this change!- The
kubernetes_logssource has been updated to read older pod logs first. This should result in better behavior with Vector releasing file handles for rotated pod files more quickly. - Vector’s support for environment variable expansion in
configuration files now allows
.s in the variable names as these commonly appear in environment variables set by Java properties files. VRL has added new functions for interacting with event metadata.:
get_metadata_field("key")set_metadata_field("key", "value")remove_metadata_field("key")
Right now, the only event metadata that is accessible is Datadog API keys (
datadog_api_key) or Splunk HEC channel tokens (splunk_hec_token) that are associated by the source, but we expect metadata use-cases to grow.This can be used with, for example, CSV enrichment tables to lookup the
datadog_api_keyto use with events based on other metadata.- The
statsdsink now compresses histograms to result in smaller payloads without data loss. - Vector’s CPU utilization has improved by running eligible transforms on multiple cores when possible. Previously, a transform could be a significant bottleneck since only one copy of it was ran which would result Vector under-utilizing available CPU resources. See the highlight article for more details.
- We have improved Vector’s performance for most use-cases by re-introducing the jemalloc memory allocator as the allocator for *nix platforms. We continue to evaluate other allocators to see if they are a better fit for Vector’s allocation patterns.
- Fix metric emission for the
blackholesink when a rate limit lower than1024was used.
7 bug fixes
- The
headers_keyconfig option for thekafkasource was restored. This was accidentally renamed toheaders_fieldin v0.18.0. For compatibility with v0.18.0,headers_fieldwill also be accepted. - The
lokisink now accepts any 200-level HTTP response from servers as success. This was added for compatibility with other Loki-compatible APIs like cLoki which didn’t respond with the expected 204 response code. - The
hostandpidfields are correctly added to all internal logs now. Previously they were only added to start-up logs, but not logs while Vector was running. - Fix a panic that could occur during when reloading Vector config that requires shutting down and recreating a sink.
- Vector, when using
--config-dir, no longer tries to load unknown file extensions. This was a regression inv0.18. - The
max_lengthoption available on somedecodingframers for sources previously caused Vector to stop decoding a given input stream (like a TCP connection) when a frame that was too big was encountered. It now correctly just discards that frame and continues. - Previously the
framing.character_delimited.delimiteroption available on some sources allowed for characters greater than a byte, but the implementation assumed the delimiter was only one byte. Vector now correctly errors if a delimiter that is greater than a byte is used. Only byte delimiters are allowed for efficiency in scanning.