vi/vim - disable visual mode and enable syntax highlighting
In vi/vim, to disable visual mode and keep the syntax highlighting: Add to your ~/.vimrc file: set mouse= syntax on
21zoo Labs - Assorted Stuff
Recent content on 21zoo Labs - Assorted Stuff
Latest posts
Check from the shell if Github is up and accessible
$ nc -vz github.com 22 Connection to github.com port 22 [tcp/ssh] succeeded! $
Limit Log Disk Space Used by Journald on Linux
What to do when journal is taking up a lot of space in your /var/log/ directory? You can limit the disk space that is used by journald (the logging system) by modifying the following file: /etc/systemd/journald.conf Look for the line #SystemMaxUse= Uncomment it and set the max limit to e.g. 100M SystemMaxUse=100M This will cap the max disk usage for journald at 100MB. Afer modifying the file you…
How to exclude metrics from ingestion into your Prometheus server
If you’re scraping a target and there are certain metrics you don’t want to ingest (for instance because they are too high cardinality and take up too much space) then you can add the following to your target config: - job_name: something_exporter static_configs: - targets: - localhost.com:9119 metric_relabel_configs: # dropping all go garbage collection metrics - not needed -…
How to delete a timeseries / metrics from Prometheus
First, you need to enable the admin API via the command line: --web.enable-admin-api Now you can delete a metric by calling the respective endpoint, for example by using curl. curl -v -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=node_cpu_seconds_total' This will delete all timeseries called node_cpu_seconds_total. The actual data still exists on disk and is cleaned up…
Use drone.io to publish a Docker image to ECR and copy files to S3
Here’s an example on how to use the drone.io CI/CD system to build a new Docker image, push it to ECR and copy some files to S3 as part of the build process. kind: pipeline type: docker name: default steps: # Your build steps go here. For example: - name: build image: alpine commands: - make build # Upload some files to S3 - name: upload files to s3 image: plugins/s3 settings: bucket:…
Migrate AWS Lambda Golang Functions to the "al2.provided" Runtime
AWS is deprecating the go1.x runtime on Lambda and it’s time to udpate your Golang lambda functions. Functions need to migrate to the al2.provided runtime and it’s pretty straight-forward. In the AWS console select al2.provided as the new runtime and when you compile your Golang handler you need to create an executable named bootstrap. Here’s a sample Golang lambda program:…
Run CloudSQL proxy with docker-compose
Using CloudSQL proxy allows you to access your CloudSQL isntance without exposing its port to the big bad internet. First, create a service account in your project and place the service-account-key-file.json key file in the current directory. docker-compose.yml services: cloudsql-proxy: image: gcr.io/cloudsql-docker/gce-proxy:1.30.1 volumes: - ./service-account-key-file.json:/config ports: -…
Use Gitea as a Auth Provider for oauth2_proxy
To use Gitea as a provider with the oauth2_proxy use this config: proxy.cfg authenticated_emails_file = "./emails.txt" redirect_url = "https://<< host running opauth2_proxy >>/oauth2/callback" cookie_secret = "<< cookie secret" provider = "github" provider_display_name = "Gitea" client_id = "<< client_id as generated by Gitea >>" client_secret = "<< client_secret as generated by Gitea >>"…
Make "go get" use ssh instead of https
To make go get github.com/user/repo use ssh instead of https run this line: git config --global --add url."git@github.com:".insteadOf "https://github.com/" It will add a section to your ~/.gitconfig file use ssh instead of https when you run go get ..., allowing you to use your ssh key to authenticate to e.g. Github and access private repos.
Go Modules - How to Update All Dependencies
To update all dependencies in a project that uses go modules enabled run: $ go get -u -m
How to Make a DNS Lookup in Golang
Save this snippet in a file and run with go run . or go run . google.com package main import ( "fmt" "net" "os" ) func main() { addr := "stuff.21zoo.com" if len(os.Args) > 1 { addr = os.Args[1] } ips, err := net.LookupIP(addr) if err != nil { fmt.Printf("net.LookupIP( %s ) err: %s", addr, err) return } for _, ip := range ips { fmt.Printf("%s has address %s\n", addr, ip.
Prometheus & Kubernetes - Configure to Scrape Pods
In your prometheus.yml file, make sur you use pod for the role when setting up the kubernetes_sd_configs. - job_name: 'kubernetes-pods' kubernetes_sd_configs: - role: pod relabel_configs: # only scrape when annotation prometheus.io/scrape: 'true' is set - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels:…
Prometheus Mongodb Exporter - Correct DB User Permissions
When you see the error Failed to get local.oplog_rs collection stats. in your mongodb exporter logs, make sure the DB user account you use has the right permissions. db.getSiblingDB("admin").createUser({ user: "mongodb_exporter", pwd: "password", roles: [ { role: "clusterMonitor", db: "admin" }, { role: "read", db: "local" } ] }) And then configure the exporter to use the mongodb_exporter user…
Postgres - Get the Current Number of Open Connections
Get the Current Number of Open Connections for a Postgres DB: SELECT * FROM pg_stat_database;
Using AppEngine go112 with CloudSQL Postgres - How to set the DB URI?
With the recent upgrade of AppEngine Golang from go111 to go112 you might have to change your database DSNs if you’re using CloudSQL and want to connect using the /cloudsql/ socket. DB DSN format for CloudSQL Postgres and Golang go112: \ "user=<<USER>> password=<<PWD> host=/cloudsql/<<CONNECTION NAME>>/ dbname=<<NAME>>" The instance connection name can be found on Instance Detail page of…
How to Calculate the Distance Between Two Lat,Long Points in Golang
This will return the distance between two points in miles (based on the Haversine formula). To get the distance in kilometers, multiply with 1.60934. func distance(lat1 float64, lng1 float64, lat2 float64, lng2 float64) float64 { radlat1 := float64(math.Pi * lat1 / 180) radlat2 := float64(math.Pi * lat2 / 180) theta := float64(lng1 - lng2) radtheta := float64(math.Pi * theta / 180) dist :=…
How to Run the Google Firestore Emulator
You can run the Firestore emulator by running: gcloud beta emulators firestore start and then set the FIRESTORE_EMULATOR_HOST environment variable as per the console output (e.g. run export FIRESTORE_EMULATOR_HOST=::1:8505). This requires the Google Cloud SDK and a Java 8+ JRE installed and on your system PATH.
How to prevent Nginx from Caching DNS for Proxy Upstreams
Normal use of proxy_pass server { proxy_pass http://upstream-host.com:8080; } Instead, use a variable in your nginx config to disable nginx caching the DNS for upstream-host.com like this: server { ... resolver 127.0.0.1; set $backend "http://upstream-host.com:8080"; proxy_pass $backend; ... } When nginx is running inside a Docker container then you need to use: resolver 127.0.0.11 ipv6=off;
Reload Nginx config inside a Docker container
Check the new configuration: $ docker container exec <<< container name >> nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful $ Apply the new configuration: $ docker container exec <<< container name >> nginx -s reload 2019/05/20 19:14:29 [notice] 7#7: signal process started $
Golang Sub Tests
t.Run() is a great way to improve table-driven tests and give them names: func TestFunc(t *testing.T) { tests := map[string]struct { input string input2 string want string }{ "simple": {input: "abc", input2: "#", want: "result"}, "wrong": {input: "def", input2: "$", want: "ok"}, "no": {input: "abc", input2: "^", want: "not-ok"}, "yes": {input: "xyz", input2: "&", want: "please"}, } for name, tst…
git push --force-with-lease
Instead of using git push --force use git push --force-with-lease. It will update remote references only if it has the same value as the remote-tracking branch we have locally and reduce the risk of accidentally overwriting someone else’s work. You can use alias gpf='git push --force-with-lease' to make it more convenient. Add it to your ~/.profile so it’s loaded automatically.
Interactive Bash Shell in Kubernetes
When you need a quick shell in your Kubernetes cluster: kubectl run my-shell --rm -i --tty --image ubuntu -- bash Warning: this will create the shell in the default namespace normally has resource constraints that will lead to termiantion of processes that take up too much CPU or memory. If this is a problem, add --namespace <<ns>> where <<ns>> is an existing namespace and the shell will be…
How to Forward a Port to Different IP using "nc"
For example, this will redirect all TCP connections to the local port 8001 to port 80 on IP 1.1.1.1 nc -l -p 8001 -c "nc 8.8.8.8 80"
Delete a Tag from Docker Hub
First, authenticate with Docker Hub: export USERNAME="<< your Docker Hub username >>" export PASSWORD="<< your Docker Hub password >>" TOKEN=`curl -s -H "Content-Type: application/json" -X POST \ -d '{"username": "'$USERNAME'", "password": "'$PASSWORD'"}' \ https://hub.docker.com/v2/users/login/ | jq -r .token` Now delete the image tag: \ export ORG="oliver006" export IMAGE="drone-gcf" export…
drone-gcf - a Google Cloud Function plugin for drone.io
Google Cloud Funtions allow you to execute small pieces of code, written in NodeJS, Python or Golang and whenever a certain event is triggered. The events can be e.g. a HTTP request, a message published to PubSub, a file uploaded to GCS and so on. The drone.io CI/CD server is a simple yet powerful and extendable Continuous Delivery platform that supports a plugin architecture where plugins a re…
How to query Prometheus from Python
query_prometheus.py from __future__ import print_function import requests import sys if len(sys.argv) != 3: print('query_prometheus.py << prometheus server URL >> "<< query >>" ') print() print("""Example: query_prometheus.py http://localhost:9090 'irate(http_requests_total{code="200"}[1m])' """) print() sys.exit(1) response = requests.get('{0}/api/v1/query'.format(sys.argv[1]), params={'query':…
Docker - Clean Up Unused Volumes
From here - a shell script to easily remove unused containers and volumes: #!/bin/bash # remove exited containers: docker ps --filter status=dead --filter status=exited -aq | xargs -r docker rm -v # remove unused images: docker images --no-trunc | grep '<none>' | awk '{ print $3 }' | xargs -r docker rmi # remove unused volumes: find '/var/lib/docker/volumes/' -mindepth 1 -maxdepth 1 -type d | grep…
Golang HTTP Client - How to not follow redirects
client: &http.Client{ CheckRedirect: func(req *http.Request, via []*http.Request) error { return http.ErrUseLastResponse }, } resp, err := client.Do(req) ...
Golang - Parallelize Tests
You can run Golang tests in parallel by calling t.Parallel() in the very beginning of a test function. For example: func TestRunThisInParallel(t *testing.T) { t.Parallel() // the actual test } Tests for a specific package by default are executed sequentially, but then using t.Parallel() a test is marked as safe for parallel execution within the test package. Only those tests marked as parallel…
Crontab - run a command every 5 minutes
To run a command via cron every 5 minutes: * any value , value list separator - range of values / step values
Install Deluge Torrent Client on Ubuntu
First, install the Deluge PPA to get the latest releases: sudo add-apt-repository ppa:deluge-team/ppa sudo apt-get update sudo apt-get install deluge To install the headless version run: sudo apt-get install deluged deluge-web deluge-console then run deluged to start the daemon and now you can use e.g. deluge-console to us the command line client. To install the GUI version run: sudo apt-get…
How to set a default namespace for "kubectl"
Kubernetes uses namespaces. If you don’t specify any, it will use the default namespace. You can use a “Context” if you want all your kubectl commands to use the same namespace. $ kubectl config set-context kube-cluster-ctx --namespace=my-namespace Context "kube-cluster-ctx" created. You have to also start using the context once it’s created like so: $ kubectl config use-context…
Golang Cross Compile for Linux Macos and Windows
For Linux desktop: $ GOOS=linux GOARCH=amd64 go build For MacOS: $ GOOS=darwin GOARCH=arm64 go build For old Windows desktop: $ GOOS=windows GOARCH=386 go build
How to get the SSL cert expiration date from PEM file
How to get the the SSL cert expiration date from PEM file openssl x509 -enddate -noout -in << pem file name >>
Prometheus Instrumentation & Metrics Best Practices
Good read: Prometheus Instrumentation Best Practices
Notes on Programming in C
Good read: Notes on programming in C by Rob Pike. On complexity: Rule 1. You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is. Rule 2. Measure. Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the…
What was the best CS paper you read in 2017?
Condensed from here: https://news.ycombinator.com/item?id=16035402 The Case for Learned Index Structures https://arxiv.org/pdf/1712.01208v1.pdf State the Problem Before Describing the Solution (1978) https://lamport.azurewebsites.net/pubs/state-the-problem.pdf Formal Verification of an OS Kernel (2009) https://www.sigops.org/sosp/sosp09/papers/klein-sosp09.pdf Chord…
How to Design a Scalable Rate Limiting Algorithm
Link: How to Design a Scalable Rate Limiting Algorithm
Django SMTP Settings for Gmail
Django SMTP Settings for Gmail EMAIL_USE_TLS = True EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend' EMAIL_HOST = 'smtp.gmail.com' EMAIL_HOST_USER = '<<username>>@gmail.com' # your Gmail username EMAIL_HOST_PASSWORD = 'xxxx' # your Gmail password EMAIL_PORT = 587 DEFAULT_FROM_EMAIL = EMAIL_HOST_USER And enable low security apps here.
Remove or Clear Last Login Information on Linux
In particular, the information returned by lastlog and last/lastb # for lastlog rm -f /var/log/lastlog && touch /var/log/lastlog # for last/lastb rm /var/log/wtmp && touch /var/log/wtmp rm /var/log/btmp && touch /var/log/btmp
Enable Swap on Ubuntu
Enable Swap on Ubuntu sudo fallocate -l 2G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
Install Docker and Docker Compose on Ubuntu
Install Docker and Docker Compose on Ubuntu # docker: curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" sudo apt-get update apt-cache policy docker-ce sudo apt-get install -y docker-ce # docker-compose: sudo curl -L…
Configure SSH to use a Jump Host
Configure SSH to use a Jump Host for ssh connections: [ Laptop ] --ssh--> [ Jump Host ] --ssh--> [ Host ] ~/.ssh/config Host jump-host User jump-host-username Hostname << JUMP HOST IP >> Host host User host-username Hostname << HOST IP >> ProxyCommand ssh -q -W %h:%p jump-host
How to define multi-line strings in yaml/yml
key: > a very very long string –> "a very very long string" And with literal newlines: key: | another very very long string –> "another very very\nlong string"
Bash Shell - Append Date to Filename
Append the current date to a filename in Bash shell: $ fname="/var/log/output-$(date +"%Y-%m-%d").log" $ echo $fname /var/log/output-2017-11-14.log $
Automatically Delete Docker Container After Running
Automatically delete Docker container after running: docker run --rm my-image
Disable password access in sshd
Disable password access in sshd /etc/ssh/sshd_config ChallengeResponseAuthentication no PasswordAuthentication no UsePAM no
How to create custom Nginx error pages for 404s
How to create custom Nginx error pages for 404s nginx.conf: error_page 404 /404.html;
Update Nginx to latest version on Ubuntu
Update Nginx to the latest version on Ubuntu: sudo add-apt-repository ppa:nginx/stable sudo apt-get update sudo apt-get install nginx