Get the most out of Logstash

A good pipeline structure in Logstash gets you the most out of it!

Here are three tips for best practices in Logstash and thinks I would have known when I started to work with it.

1. No long-running operations

When you deploy Logstash you start it with either a default configuration or with your own settings.

However you will probably be limited in workers and the batch size.

This means that when you have some longer-running pipelines you will be limited in the size of the input as your Logstash instance gets stuck in job processing. Or in the worst case your instance will break!

Which actions you can take here are:

1. Try to not run long-running operations and keep them as minimal as possible. This will assure that workers are not too long busy.
2. If you can, pre-convert your data before sending it to your Logstash instance, so that you don’t need to keep the limited computing power of your instance. Logstash should not have a lot of logic in it’s pipelines and don’t implement any business logic. In the best case it should just take some data, perform as little manipulations as possible and output it.
3. Run only the pipelines which you really need, so that you can save some resources in any cases.

2. Check the architectural pipeline patterns

Make use of the multiple pipelines feature if you need to separate logic from each other.

If you need one logical block of filter operations for many different input types, you should not implement this x times. Check the pipeline-to-pipeline communcation for how to address multiple pipelines and sticking them together:

- pipeline.id: upstream
  config.string: input { stdin {} } output { pipeline { send_to => [myVirtualAddress] } }
- pipeline.id: downstream
  config.string: input { pipeline { address => myVirtualAddress } }

What you should also keep in mind, is that Logstash itself allows to implement architectural patterns which should help you get a better usage of it.

3. Keep the plugins in your mind

Always keep in your mind that for most use cases there is an existing plugin. Check the documentation and the GitHub page (and issues!) to get more information of the plugin.

But sometimes you don’t find a plugin. In this case you should make use of Ruby code execution.

Further reading

• Logstash performance tuning
• Logstash performance troubleshooting
• Concepts of Logstash