Recent content on Brian Cunnie's Technical Blog
8.8.8.8 is Google’s “ free, global DNS resolution service ”, so popular that it’s been lampooned by XKCD . It has a wonderfully straightforward front-end: the IP address 8.8.8.8 . But that begs the question: what’s on the back-end? Do the authoritative nameservers receive queries from 8.8.8.8 or some other IP address? Spoiler: the queries don’t originate from…
This Blog Post Is Not For You This blog post is directed towards people who are working with the BOSH vSphere CPI (Cloud Provider Interface), which is not you. There are more interesting things to read. If you want suggestions, try Ulysses by Sir Alfred Lord Tennyson, a poem about an aged hero seeking to recapture his adventures of youth. Challenge: Extending the Size of the Root Disk I’d…
If you don’t like seeing the “Your connection is not private” or “Warning: Potential Security Risk Ahead” when you browse to your NSX Manager, then you may want to install a TLS certificate from a commercial CA (Certificate Authority). This post tells you how. This NSX manager has a certificate from issued from Sectigo. Note that the padlock in the address bar shows…
* If you don’t count the amount of time spent maintaining the on-premise equipment. Abstract My 48-VM (virtual machine) homelab configuration costs me approximately $430/month in hardware, electricity, virtualization software, and internet, but an equivalent configuration on AWS (Amazon Web Services) would cost $1,660/month (almost four times as expensive)! Disclosures: I work for VMware,…
We’re going to set up automated backups for a vCenter which we were forced to rebuild over the winter break because the unexpected reboot of the file server hosting the iSCSI datastore backing the vCenter’s disk drive caused unrecoverable database corruption, and we had no backups. Log into your TrueNAS server via its web interface, e.g. https://nas.nono.io Browse to…
Concourse CI/CD (continuous integration/continuous delivery) can create multi-platform Docker images. This blog post describes how. A multi-platform docker image is one that contains “ variants for different architectures ”. Docker images are often created for a single architecture (“instruction set architecture” or “ ISA ”), typically Intel’s/AMD’s…
Quickstart First, create your key and your CSR (Certificate Signing Request). In the following example, we are creating a CSR for our vCenter host, “vcenter-80.nono.io”: CN = vcenter-80.nono.io # 'CN' is the abbreviation for 'Common Name' openssl genrsa -out $CN.key 3072 openssl req \ -new \ -key $CN.key \ -out $CN.csr \ -sha256 \ -subj '/C=US/ST=California/L=San…
Network Diagram. We want to maximize the throughput from the blue box (the client) to the green box (HAProxy) Summary We were able to push through almost 450 MB/sec through HAProxy (which terminated our SSL) by carefully matching our 4-core HAProxy with 2 x 4-core Gorouters (which were on a much slower ESXi host). Results Bandwidth MB/second Configuration 201.27MB 1 HAProxy: 1 vCPU 136.47MB 1…
The Cloud Foundry Acceptance Tests are the gold standard to test the proper functioning of your Cloud Foundry deployment. This guide tells you how to run them. When in doubt, refer to the README . Quick Start cd ~/workspace/ git clone git@github.com:cloudfoundry/cf-acceptance-tests.git cd cf-acceptance-tests . ./.envrc cp example-cats-config.json cats-config.json export CONFIG = cats-config.json…
Recreating the Cluster We want to recreate our cluster while preserving our Vault and Concourse data (we want to recreate our GKE regional cluster as a zonal cluster to take advantage of the GKE free tier which saves us $74.40 per month). Note: when we say, “recreate the cluster”, we really mean, “recreate the cluster”. We destroy the old cluster, including our worker nodes…
Is it worth switching your VMware vSphere VM’s SCSI (small computer system interface) from the LSI Logic Parallel controller to the VMware Paravirtual SCSI controller? Except for ultra-high-end database servers (> 1M IOPS ( input/output operations per second)), the answer is “no”; the difference is negligible. Our benchmarks show that VMware’s Paravirtual SCSI (small…
In our previous post, we configured our GKE Concourse CI server, which was the capstone of the series. But we were wrong: this post is the capstone in the series. In this post, we install Vault and configure our Concourse CI server to use Vault to retrieve secrets. Installation Most of these instructions are derived from the Hashicorp tutorial, Vault on Kubernetes Deployment Guide . Create a DNS A…
In our previous post, we configured our GKE (Google Kubernetes Engine) to use Let’s Encrypt TLS certificates. In this post, the capstone of our series, we install Concourse CI. Installation These instructions are a more-opinionated version of the canonical instructions for the Concourse CI Helm chart found here: https://github.com/concourse/concourse-chart . First Install: with Helm We use…
In our previous blog post, we configured ingress to our Kubernetes cluster but were disappointed to discover that the TLS certificates were self-signed. In this post we’ll remedy that by installing cert-manager, the Cloud native certificate management tool. Disclaimer: most of this blog post was lifted whole cloth from the most-excellent cert-manager documentation . We merely condensed it &…
In our previous blog post, we set up our Kubernetes cluster and deployed a pod running nginx, but the experience was disappointing—we couldn’t browse to our pod. Let’s fix that by deploying the nginx Ingress controller. Acquire the External IP Address (Elastic IP) We’ll use the Google Cloud console to acquire the external address [ external address ] for our load balancer.…
Let’s deploy Concourse , a continuous-integration, continuous delivery (CI/CD) application (similar to Jenkins and CircleCI ). We’ll deploy it to Google Cloud , to our Google Kubernetes Engine (GKE). In this post, we’ll use HashiCorp ’s Terraform to create our cluster. We assume you’ve already installed the terraform command-line interface (CLI) and created a Google…
Why am I creating a new blog? What was wrong with the old blog? Why don’t I use Medium ? The short version: The old blog is frozen in time, like a prince caught in amber 1 or a dandy in aspic 2 . I can no longer post to it. The old blog, the Pivotal Engineering Journal , which many of Pivotal’s engineers contributed to, was archived a year after VMware acquired Pivotal. Every…
My co-worker Belinda Liu turned to me and said, “I don’t like these tests at all; they’re hard to follow, and I’m not sure what they’re testing.” I looked at the tests that I had spent much of yesterday afternoon working on. She was right: they were hard to follow (even for me, who had written some of them!). How had we gotten here? Our code was straightforward,…
0. Abstract HAProxy is an optional load balancer included in the canonical open source Cloud Foundry deployment . Its intended use is on IaaSes (Infrastructures as a Service) that do not offer built-in load balancers [0]. On vSphere, this means without the optional network virtualization solutions, NSX-T and NSX-V. This blog post describes how to assign an IPv6 address to an HAProxy load balancer…
Abstract “How much faster will my VM’s disks be if I upgrade my ZFS -based (Z File System) NAS to 10 GbE?” The disks will be faster, in some cases, much faster. Our experience is that sequential read throughput will be 1.4✕ faster, write throughput, 10✕ faster, and IOPS, 1.6✕ faster. We ran a three-hour benchmark on our NAS server before and after upgrading to 10 GbE. We ran the…
Abstract Smartphone authenticator apps such as Google Authenticator and Authy implement software tokens that are “two-step verification services using the Time-based One-time Password Algorithm (TOTP) and HMAC-based One-time Password algorithm (HOTP)” Smartphone TOTP, a form of Two-factor authentication (2FA), displays a 6-digit code derived from a shared secret, updating every thirty…
Abstract By using tcpdump to troubleshoot an elusive error, we uncovered a man-in-the-middle (MITM) ssh proxy installed by our information security (InfoSec) team to harden/protect a set of machines which were accessible from the internet. The ssh proxy in question was Palo Alto Network’s (PAN) Layer 7 (i.e. it worked on any port, not solely ssh’s port 22) proxy, and was discovered when we…
When customers with vSphere+NSX-T-based foundations apply a stemcell update, update a tile, or upgrade PAS (Pivotal Application Service) from 2.2 to 2.3, their Cloud Foundry may become unreachable as their NSX-T static load balancer server pools have been emptied. This blog post describes a method to ensure availability during upgrades. We use a combination of customized Operations Manager…
The following section is the new Quickstart for installing a TLS certificate on vCenter 7 vCenter 7 Quickstart On your vCenter, navigate to Menu → Administration → Certificates → Certificate Management On the __MACHINE_CERT tile, click Actions , select Generate Certificate Signing Request (CSR) . Enter the appropriate info; for inspiration, this is what we entered:
0. Overview [Disclaimer: the author works for Pivotal Software, of which Dell is an investor. Dell is also an owner of VMware] It’s helpful to know the performance characteristics of disks when selecting a disk type. For example, the performance of a database server will be greatly affected by the IOPS of the underlying storage. Similarly, a video-streaming server will be affected by the…
0. Abstract BOSH is a VM orchestrator; a BOSH Director creates, configures, monitors, and deletes VMs. The BOSH Director interoperates with a number of IaaSes (Infrastructure as a Service), one of which is VMware vSphere, a virtualization platform. BOSH traditionally operates exclusively within the IPv4 networking space (i.e. the BOSH Director has an IPv4 address (e.g. 10.0.0.6), and the VMs which…
“BOSH deploys Concourse, and Concourse deploys BOSH” —Cloud Foundry koan A BOSH Director is a VM (virtual machine) orchestrator which is itself a VM. BOSH solves the problem of keeping its VMs’ applications (operating systems (stemcells) and releases) up-to-date with the command, bosh deploy ; however, this begs the question, “what keeps the BOSH Director itself…
0. Abstract A BOSH director is a virtual machine (VM) orchestrator which deploys VMs to various Infrastructures as a Service (IaaS) such as Amazon Web Services (AWS) and Google Cloud Platform (GCP). The BOSH Command Line (CLI) communicates with the director over Secure Sockets Layer (SSL). While most BOSH directors are deployed with self-signed certificates, it is possible to configure a BOSH…
VMware’s vSphere is an Infrastructure as a Service (IaaS) which runs Virtual Machines (VMs). BOSH is a VM orchestrator which automates the creation of VMs. NSX-T is a pluggable Network backend for vSphere (and other hypervisors). NSX-T allows the creation of opaque networks in vSphere, networks whose detail and configuration of the network is unknown to vSphere and which is managed outside…
When Hacker News picked up Part 1 of our series of blog posts on running public NTP servers, a contributor said, “I wish he’d explained … what they ultimately did (since there’s no part 3 that I can find).” We had dropped the ball — we had never concluded the series, had never written part 3, had never described the strategies to mitigate the data transfer costs.…
This blog post describes the procedure we followed to use the beta BOSH command line interface (CLI) to deploy an nginx webserver with a native IPv6 address (i.e. 2600:1f16:0a62:5c00::4) to AWS in addition to its IPv4 Elastic IP address (i.e. 52.15.73.90). We were then able to browse the webserver via the IPv6 protocol. BOSH does not support IPv6. This is a proof-of-concept. Do not apply IPv6 to…
VMWare NSX is a network virtualization platform (frequently paired with the vSphere IaaS (Infrastructure as a Service)). It includes features such as Load Balancers (LBs) and firewall rules, features often found in public-facing IaaSes (e.g. AWS (Amazon Web Services), GCE (Google Compute Engine), and Microsoft Azure) but not native to vSphere. BOSH , a VM orchestrator, includes hooks to…
In this blog post, we describe the procedure we followed in order to create a custom Google Compute Engine (GCE) stemcell with a user cunnie whose ~/.ssh/authorized_keys is pre-populated with a specific public key. Customizing stemcells is highly discouraged — it voids your warranty, and opens a host of problems which will only cause pain. This post is intended as an educational demonstration of…
When PowerDNS released version 4.0.1 of their authoritative nameserver, we rushed to update our BOSH Release (which was at version 4.0.0). We thought it would be a walk in the park, but instead it was an epic fail (a final release which couldn’t be deployed because the blobs were broken). In this blog post we describe the procedure we ultimately followed to successfully create an updated…
The Concourse Continuous Integration (CI) server has an API endpoint that displays a badge which shows health of your project: http(s):// concourse-server /api/v1/pipelines/ pipeline-name /jobs/ job-name /badge 0. Abstract Open Source projects that have CI (e.g. Bootstrap , Node.js ) often feature status badges (also known as images or icons) to display the health of their projects. CI servers…
Abstract Concourse is a continuous integration (CI) server. It can be deployed manually or via BOSH . In this blog post, we describe the BOSH deployment of a Concourse CI server to natively accept Secure Sockets Layer (SSL) connections without using a load balancer. This may reduce the complexity and cost [ELB-pricing] of a Concourse deployment. 2016-09-12: This blog post is obsolete. Newer…
vSphere users ask, “How do I configure my BOSH director so that it can communicate with my vCenter but the director’s deployed VMs can’t?” One technique is to use a multi-homed BOSH director combined with the BOSH Networking Release (a BOSH release which enables customization of the VM’s routing table, allowing more routes than the default gateway). Network Diagram…
[2016-04-06: This Blog Post is out-of-date; Please refer to the official Concourse documentation for instructions how to install a Concourse server] Continuous Integration (CI) is often used in conjunction with test-driven development (TDD); however, CI servers often bring their own set of challenges: they are usually “snowflakes”, uniquely configured machines that are difficult to…