Articles on AWS, Golang, Containers, Linux, Infrastructure and more.
Hey, I’ve recently bought a Raspberry Pi 3B+ , and that seemed like a great target for a Concourse worker run at. It turns out that just the process of building and adapting the Concourse binary itself was already an interesting thing to do, so here I share the lessons learned and how that process looked like. To provide ARM -compiled binaries for node_extra_exporter (a Rust-based Prometheus…
Hey, If you’ve been working with web servers for a little while, you certainly have already hit the classic “ address already in use ” (EADDRINUSE). Here in this article, we go through not only how to see whether such condition as conditioned to happen (by looking at the list of open sockets), but also verify in the actual Kernel code paths where that check happens. In case…
Hey, The file covered in this article, /proc/<pid>/stack , is the one that motivated me to learn more about /proc and get The Month of Proc . It’s such a useful thing when you’re unaware of what is the state of a given process. Meanwhile, I’ve noticed that it’s not very well known by people getting started with Linux. Here in this post, you’ll get to know more about…
Hey, When dealing with web servers, it’s not uncommon to face the problem of “too many files open”. Usually, people solve that by modifying a number using ulimit -n , and then, sometimes that doesn’t work as such setting is per-process, and not system-wide. In this article, I go through how that ulimit setting works under the hood, and how you can make use of /proc to…
Hey, Something that is very common to get wrong when starting with Linux containers is to think that free and other tools like top should report the memory limits. Here you’ll not only go through why that happens and how to get it right, but also take a look at where is the Kernel looking for information when you ask it for memory statistics. Also, if you’re curious about how the code…
Hey, Continuing with the Month of /proc , today’s blog post is about how reading /proc works (yep, the directory) . Not only is this article’s content about /proc , but also about reading directories in general ( expect syscalls and Kernel inspection ). If you’ve been curious about how listing directory entries works under the hood, this is for you! This is the second article in…
Hey, Although many people that are accustomed to Linux are aware of the existence of /proc and what some files over there can do, many lack the understanding of what goes behind the scenes to power such filesystem (myself included before writing this article). If you’ve been wondering about how /proc works under the hood, stay tuned! This is the first article in a series of 30 articles…
Hey! I’ve been wanting to better understand what are some of the capabilities of procfs ( /proc ) , so I thought of a little challenge for me: for each day, for 30 days , I’ll be posting a new article about a single file under /proc that should be useful for someone either introspecting a system (or wanting to mutate some kernel param). Given that so many tools touch the /proc…
Hey, Today I spotted something very interesting in my Twitter (I’m @cirowrc there, by the way): To learn bpftrace, there is a reference guide and one-liners tutorial: https://t.co/EztuxTK9iJ https://t.co/foCH2TUCpv — Brendan Gregg (@brendangregg) October 8, 2018 That’s right, dtrace functionality for Linux via eBPF “superpowers”. To be very honest, I know almost…
Hey, This month I’ve been interested in increasing the overall time that a user spends navigating here in the blog. Being article recommendation something that the blog was missing, I went for it. Check out how you can do it for your static site too! The idea Implementing a Recommended Articles list in Hugo Displaying the recommendation list Closing thoughts The idea Given that my content…
Hey, While it’s prevalent to see implementations of TCP servers in Golang, it’s not very common to see the same when it comes to UDP. Besides the many differences between UDP and TCP, using Go it feels like these are pretty much alike, except for little details that arise from each protocol specifics. If you feel like some Golang UDP knowledge would be valuable, make sure you stick to…
Hey, Another day I was trying to make sure that a given process that I was running was using a specific binary that I had built, but I couldn’t figure out: ps would only show me the non-absolute path. # How could I know what is the absolute path of the # `hugo` binary, assuming that I could have multiple # `hugo` binaries in `$PATH`? ps PID TTY TIME CMD 4153 ttys000 5:14.98 hugo serve <<<…
Hey, This week I wanted to discover a bit more about how dmesg works under the hood. In the past, I wanted to have alerting based on error messages popping at dmesg , so, maybe by trying to create something that uses the same thing that dmesg uses under the hood, I could better understand it. Also, I knew that in some systems, it was possible to gather the same information from kern.log and that…
Hey, I’ve been doing some work in a Linux VM recently, and having to use secure copy ( scp ) every time just to copy a tcpdump capture to my MacOS machine to inspect it with Wireshark is not very fun. VM MACOS tcpdump --> capture scp linux --> /tmp/capture wireshark /tmp/capture Given that Wireshark can inspect packets flowing in real time, what if Wireshark perform the equivalent of a tail…
Hey, Being accustomed to tying together multiple bash programs with the pipe operator, sometimes I’ve seen myself not being able to easily do so when a command expected the input to come from a file instead of stdin . Given that `dirname` doesn't expect input from `stdin`, piping to it doesn't work as expected Using process substitution we can make sure that pretty much every command can…
Hey, Oftentimes I see myself needing to check response times while making a bunch of requests quickly. Most of the tools out there either do not continuously make new requests (via an entirely new TCP connection) or don’t expose connection times. Snapshot of the `httpstat` tool httpstat is a great tool that does the job really well. It’s a single binary that you can put under your…
Hey, Hugo v0.43 just got released today (see the GitHub Releases page for Download links and notes about all that’s included in v0.43) with great news! From my point of view, the most significant feature in this release (as noted in the gohugo.io website) is the introduction of the asset pipeline ( Hugo Pipes ), a way of allowing the user to specify transformations to be applied to assets (…
Differently from the blog posts , these are smaller posts which are usually more direct; not a lot of plain-text explanation - more code.
Hey, I’ve been trying to improve the Hugo template that I created for this blog, and one of the things I wanted to do was inject the contents of an index.css file into a index.amp.html layout file - Accelerated Mobile Pages (AMP) requires us to have our custom styles inlined in the HTML (see Custom Amp Styles ). Here I go through a way of achieving that - using awk - and a way of not…
Hey, Some days ago I wrote about how you can configure YouCompleteMe to navigate the Linux source code . One great benefit of going through such setup is that it showed me how I could use the very same concepts to develop eBPF code better by leveraging autocompletion and jumps to definitions and declarations. While it’s common for people to just embed their eBPF code directly into Python…
Hey, When Docker Swarm mode got announced, one of the big features included was the routing mesh . Image from docs.docker.com Although the feature indeed works as expected, there’s the possibility that you might not want to have all of your nodes accepting connections and performing the job of a load-balancer. In this blog post, I go through what are the fundamental blocks that the routing…
Hey, Some time ago I needed to check whether AWS EC2 instance profiles would work fine with Docker Registry, and guess what? It does! What interests me the most in such setup is how it facilitates a scenario where you can have a highly available internal registry, having a load-balancer in front of a set of registries and granting privileges to only those who really need it. Given that the pushes…
Hey, Today I was looking at the internal struct that ends up being filled as the result of parsing the Docker Registry configuration, and doing that I found that in the master branch of the repository there’s already support for metrics scraping by Prometheus (see configuration.go ), something that used to be only available in OpenShift (see openshift/origin issue ). It surprised me that…
Hey, These days I’ve been doing some research that involves looking at the Linux kernel to figure things out. Since I’ve been using YouCompleteMe for a reasonable amount of time, and that I love how simple it is to configure it, why not give a try to using it for inspecting the Linux kernel? Example of YouCompleteMe searching available functions. ps.: for a browser experience, bootlin…
Hey, This week I needed to check whether in a given situation a given error would occur when connecting to a TCP server, so why not go back to the man pages and review the steps? Here’s what I come up with! I hope it’s going to be useful for you who’s reading. /** * Start by specifying our dependencies. * * As we're using only standard dependencies you'd find in a * Linux…
Hey, I’ve been working on an tc + ebpf -based load-balancer that I’ll soon talk about here in this blog, and one of the things I wanted to do was test such load-balancing feature. In my line of thought, I wondered: if I’m going to redirect the traffic by changing the destination address of the packets, that means that I’ll need somehow to have a different internet set up in…
Hey, There’s been some time since an issue has been open under SwarmKit to address the lack of support to having privileged containers (see https://github.com/docker/swarmkit/issues/1030) . Although pull requests have been made (see https://github.com/docker/swarmkit/pull/1129, for instance), it seems like there’s no intention by the Docker team to have the feature added before…
Hey, Some days ago I needed to tweak Docker a little bit to check if changing some parameters of SwarmKit would make it work more reliably. Given that the value I needed changing was hardcoded, no /etc/docker/daemon.json configuration would do the job. That meant that a fork would be needed. It turns out that building and running a forked version of Docker is not complicated once you get to know…
Hey, This week I wanted to make the logs of a Docker build easier to read, and remembering how some tools prefix their outputs with a name, I thought that doing so would be a good way to go. In such scenario, I had the following structure: . ├── Dockerfile # The Dockerfile that instruct the │ # image building process │ └── scripts # directory full of scripts to be run ├── first.sh # Ideally, each…
Hey, If you’ve ever wondered whether you can tie Access Control Lists (ACLs) with maps in HAProxy, the answer is: yes. Let’s tailor a scenario here: based on a map, decide whether a request to a given domain should be answered by the current frontend or not - in the negative case, forward the request to a different frontend. One application that can be thought from this is the case…
Hey, This is a quick post for those who might need to perform HTTP health checks against a running HAProxy instance. From my perspective, three ways of doing it: using the monitor-uri directive (you should use this one); using a custom HTTP file with errorfile directive; using a lua script. While the first and seconds should work in any HAProxy installation, the third requires lua , which your…
Hey, Those who already have deployed Docker to an EC2 instance might have noticed (or not) that, within the containers, you’re able to perform requests to the EC2 metadata service, discovering information about the host from within these containers. While in some cases that’s a desired thing (i.e., not needing to explicitly pass credentials to containers and make use of instance…
Hey, While Docker takes some time to expose more options to service creation (for instance, limiting the maximum number of PIDs of a service), it’s important that they are still enforced (at least with a default number). Many docker options that are not exposed to docker swarm mode yet - see add more options to service create / `service update . It turns out though that there’s a way…
Hey, I’ve wanted to cover some Linux networking basics, and I felt that going through the exercise of setting up a straightforward TCP server would do good, and writing a guide during that process would certainly force me to go over details that I’d usually skip. If you’re curious (or just wanted a tutorial) on how to set up a TCP server using Linux and C, this is for you. The…
Hey, I came through a great article this week ( Write yourself a strace in 70 lines of code ) and decided to give a try, going through every step with a bunch of care such that I’d understand each piece. While writing the code, I came up with the following schematics of how everything works together: Let’s dive into it. Executing a new process with tracing enabled By controlling the…
Hey, this is a quick follow up for the article I wrote about the locks quota of AWS EFS ( The first limit you’ll hit on AWS EFS: Locks ) and an article about settings up AWS networking with Terraform ( A practical look at basic AWS Networking with Terraform ). Here I take the concept of creating multiple subnets in a VPC explored in the second article and then tie with the AWS EFS…
Hey, I noticed that recently a bunch of people has been getting to the website through queries about HAProxy in the context of AWS - mainly how to receive traffic through a set of instances and forward them to another set of machines. As I aim at making this blog as much practical as I can, I feel that if I taught the basics around AWS networking first, then I could have the ground set for further…
Hey, While people generally know (and agree) that cAdvisor is the guy in the room to keep track of container metrics, there’s a sort of hidden feature of the docker daemon that people don’t take into account: the daemon by itself can be monitored too - see Collect Docker Metrics with Prometheus . You indeed can determine whether the daemon is running by checking the systemd metrics via…
Hey, some days ago HTTP2 server push has been added to Nginx (at least the open source version). That’s great news since this was one of the most interesting features from HTTP2 that Nginx was lacking. Given that I didn’t use server push myself so far, I decided to learn a bit more about it and give a try to Nginx' implementation. A minimal HTTP2 Server push example in Go Verifying…
Hey, I’ve been exploring the new Go support for AWS Lambda that has been announced by AWS early this year (see Announcing Go Support for AWS Lambda ), and there’s a little detail of it that I found very interesting: at its core, it uses Go’s net/rpc package as a mean of communication between the AWS infrastructure and your code. Having never used net/rpc myself, I decided to…
Hey, a month ago I was looking at some issues with a container running a MySQL instance against an NFS-mounted directory on the host (should you do this? maybe not). The issues seemed pretty weird as I’m not a MySQL guy and there were all sorts of errors popping up related to disk quotas. Sure, we did have a bunch of space: df -h fs-<...>.amazonaws.com:/ 8.0E 1.6G 8.0E 1% /mnt/nfs The…
Hey, this week the team behind Grafana released a beta release of Grafana v5 (which is almost there - check the milestone ) and I got very happy to know that now it’s easier to get an instance from ground up without having to use the UI to configure the dashboards. The niceness of that feature is that now you can redeploy a Grafana container that has a bunch of dashboards without needing to…
Hey, some time ago I had the challenge of setting a monitoring system for some machines in AWS. I end up taking Prometheus as the system to retrieve the metrics from the sources and Grafana as the way of graphing them. Nothing fancy, pretty standard. After that, I had a problem though. How could I test this setup locally? Everything was very much tied to how service discovery worked within an AWS…
Hey, this blog post is the continuation of the previous post ( How to build and run Concourse CI locally ). In this one, I try to understand better how one of its main components works internally to keep track of pipelines and have builds triggered. Let’s dig into what goes on when a pipeline is submitted to Concourse. The API As a consumer of the “concourse service”, the API…
Hey, close to a year ago I got in touch with concourse.ci when evaluating a tool to perform builds whenever changes happened in a git repository and even though the team I worked with end up creating a custom tailored system (we needed much less than what concourse - or something like Jenkins - provided) I think Concourse does a lot right. This post is not meant to teach you the concepts behind…
Hey, since some months ago I’ve been struggling a bit with the amount of information that I consume in a single day. It’s just outrageous that sometimes I’d put myself in a zombie mode and just keep following news (even if something like HackerNews which, in theory, should be a good source of interesting information) for so long and not really produce much. I decided to try…
Hey, This is a quick tip on how you can switch from using ACLs (access control lists) to use maps for selecting backends based on a request parameter. The usefulness of using maps instead of acls is that sometimes it might be easier to update the mappings rather than the ACLs (as well as allowing us to perform mutations on the maps without needing to spawn a new process due to the possibility of…
Hey, I just got an idea about a tool, and given that I think it’s a type of thing that pairs nicely with a blog to drive people to it, I decided to create one. Given that I already have everything set up for this one ( ops.tips , I mean), I thought about sharing my setup with you so that you can also adopt it - if you think it’s cool. ps.: this is not meant to be advertising for AWS.…
Hey, Recently, HAProxy 1.8 got announced , and it came with some pretty good news: HTTP/2 is automatically detected and processed in HTTP frontends negotiating the “h2” protocol name based on the ALPN or NPN TLS extensions. At the moment the HTTP/2 frames are converted to HTTP/1.1 requests before processing , so they will always appear as 1.1 in the logs (and in server logs). No HTTP/2…
![](https://ops.tips/blog/raspberry-pi-concourse-ci-worker/"https://ops.tips/blog/-/images/concourse-raspberry-pi-diagram.svg")
![](https://ops.tips/blog/how-linux-creates-sockets/"https://ops.tips/blog/-/images/socket-how-does-it-work.svg")
![](https://ops.tips/blog/using-procfs-to-get-process-stack-trace/"https://ops.tips/blog/-/images/procfs-capturing-stack.svg")
![](https://ops.tips/blog/proc-pid-limits-under-the-hood/"https://ops.tips/blog/-/images/ulimits-example.svg")
![](https://ops.tips/blog/why-top-inside-container-wrong-memory/"https://ops.tips/blog/-/images/meminfo-uncontained.svg")
![](https://ops.tips/blog/how-is-proc-able-to-list-pids/"https://ops.tips/blog/-/images/ls-proc.svg")
![](https://ops.tips/blog/what-is-slash-proc/"https://ops.tips/blog/-/images/vfs-abstraction.svg")
![](https://ops.tips/blog/article-recommendation-using-hugo/"https://ops.tips/blog/-/images/recommended-articles.svg")
![](https://ops.tips/blog/tcpdump-to-wireshark-in-realtime/"https://ops.tips/blog/-/images/wireshark-cant-read-pipes.png")
![](https://ops.tips/blog/bash-process-substitution/"https://ops.tips/blog/-/images/pipe-to-dirname-fails.svg")
![](https://ops.tips/gists/measuring-http-response-times-curl/"https://ops.tips/gists/-/images/httpstat.png")
![](https://ops.tips/blog/minifying-and-bundling-assets-using-hugo/"https://ops.tips/blog/-/images/hugo-asset-pipeline.svg")
![](https://ops.tips/gists/shell-replacing-variable-by-file-content/"https://ops.tips/gists/-/images/awk-templater-idea.svg")
![](https://ops.tips/blog/developing-ebpf-with-autocompletion-support/"https://ops.tips/blog/-/images/map-lookup-autocomplete.png")
![](https://ops.tips/blog/blocking-ingress-traffic-to-docker-swarm-worker-machines/"https://ops.tips/blog/-/images/ingress-routing-mesh.png")
![](https://ops.tips/gists/aws-s3-private-docker-registry/"https://ops.tips/gists/-/images/s3-registry-interaction.svg")
![](https://ops.tips/gists/retrieving-docker-registry-metrics-using-prometheus/"https://ops.tips/gists/-/images/blob-upload-latency.png")
![](https://ops.tips/gists/navigating-the-linux-kernel-source-with-youcompleteme/"https://ops.tips/gists/-/images/function-search.png")
![](https://ops.tips/blog/using-network-namespaces-and-bridge-to-isolate-servers/"https://ops.tips/blog/-/images/lb-overview.svg")
![](https://ops.tips/blog/adding-privileged-to-docker-swarm-mode/"https://ops.tips/blog/-/images/fork-docker-privileged-journey.svg")
![](https://ops.tips/blog/compiling-your-own-forked-docker-release/"https://ops.tips/blog/-/images/docker-ce-overview.svg")
![](https://ops.tips/gists/how-to-haproxy-acl-map/"https://ops.tips/gists/-/images/haproxy-acl-map-config.svg")
![](https://ops.tips/blog/blocking-docker-containers-from-ec2-metadata/"https://ops.tips/blog/-/images/ec2-metadata-block-overview.svg")
![](https://ops.tips/blog/run-docker-with-forked-runc/"https://ops.tips/blog/-/images/docker-containerd-runc-interaction.svg")
![](https://ops.tips/blog/a-tcp-server-in-c/"https://ops.tips/blog/-/images/tcp-connection-overview.svg")
![](https://ops.tips/gists/using-c-to-inspect-linux-syscalls/"https://ops.tips/gists/-/images/ptrace-syscall-flow.svg")
![](https://ops.tips/gists/how-aws-efs-multiple-availability-zones-terraform/"https://ops.tips/gists/-/images/aws-efs-architecture.svg")
![](https://ops.tips/blog/a-pratical-look-at-basic-aws-networking/"https://ops.tips/blog/-/images/aws-net-intro-highlevel.svg")
![](https://ops.tips/gists/how-to-collect-docker-daemon-metrics/"https://ops.tips/gists/-/images/docker-daemon-grafana-metrics.png")
![](https://ops.tips/blog/nginx-http2-server-push/"https://ops.tips/blog/-/images/h2-header.svg")
![](https://ops.tips/gists/example-go-rpc-client-and-server/"https://ops.tips/gists/-/images/go-rpc-architecture.svg")
![](https://ops.tips/blog/limits-aws-efs-nfs-locks/"https://ops.tips/blog/-/images/efs-mysql-err.svg")
![](https://ops.tips/blog/initialize-grafana-with-preconfigured-dashboards/"https://ops.tips/blog/-/images/grafana-flow.svg")
![](https://ops.tips/blog/simulating-aws-tags-in-local-prometheus/"https://ops.tips/blog/-/images/prom-aws-overview.svg")
![](https://ops.tips/blog/what-happens-when-pipeline-is-created-in-concourse-ci/"https://ops.tips/blog/-/images/cc2-atc.svg")
![](https://ops.tips/blog/how-to-run-concourse-ci-locally/"https://ops.tips/blog/-/images/cc-head.png")
![](https://ops.tips/blog/extension-to-block-websites/"https://ops.tips/blog/-/images/webrequest-flow.png")
![](https://ops.tips/gists/replacing-haproxy-acls-by-haproxy-maps/"https://ops.tips/gists/-/images/haproxy-acl-overview.svg")
![](https://ops.tips/blog/publishing-a-blog-with-aws/"https://ops.tips/blog/-/images/build-process.svg")
![](https://ops.tips/blog/haproxy-http2/"https://ops.tips/blog/-/images/haproxy-http2-overview.svg")