Recent content on Zespre's Blog
In the previous post , we introduced KubeVirtBMC and showed how it provides virtual BMC endpoints for KubeVirt VMs. We tested it with raw IPMI and Redfish commands. That was fun, but the real power of KubeVirtBMC shines when you pair it with actual bare-metal provisioning tools. In this post, we’ll walk through a complete end-to-end demo: using Metal3 to manage and provision KubeVirt VMs…
When you provision bare-metal servers using software, such as Tinkerbell , Metal3 , or even Foreman , you need a BMC (Baseboard Management Controller) to power them on, set the boot order, and control them remotely. But what if you want to test your provisioning software without real hardware? Moreover, what if your test environment runs on Kubernetes with KubeVirt ? KubeVirtBMC solves this…
Why Sometimes, you need a quick and temporary way to serve Helm charts without setting up a full-fledged chart repository like ChartMuseum or relying on an external service. Whether you’re testing a chart, sharing it within your local network, or running an ad-hoc deployment, a simple and lightweight solution can save time. This guide shows how to instantly spin up a Helm chart server using…
Overview It’s possible to provision machines from scratch to act like running a Harvester cluster but in a poor man’s fashion. Though the functionality is not as rich as what a legitimate Harvester cluster is capable of, it helps us understand the core technology of an HCI solution in terms of: Compute: Kubernetes (the container orchestration platform) and KubeVirt (enabling…
The Harvester team often uses the ipxe-examples to spin up live and VM-based clusters for development and testing. This write-up introduces an alternative way of doing it. By leveraging the VirtualBMC project , it’s possible to use Seeder to provision Harvester clusters with virtual machines. Background Harvester HCI Harvester is a modern, open, interoperable, HCI solution built on…
The dockershim has been announced as deprecated since Kubernetes v1.20. And Kubernetes v1.23 is about to step into its EOL after the end of this month. So it’s time to move forward! Let’s upgrade the cluster to v1.24. This article is solely for the personal record about changing container runtimes to upgrade Kubernetes from v1.23 to v1.24. Prerequisites Firstly, make sure what container runtime…
There’s an old saying: One must first polish his/her tools to do a good job. I love to polish things but hesitate to do real jobs. That’s my problem, and I have realized that for a long time. Anyway, in this article, I’d like to briefly share how I set up the on-prem infrastructure for blog hosting featuring: static blog with version control containerized environment managed by…
These are important but not so commonly used. Therefore, I noted down some critical parts of it in case I need them in the future. Create Keys The interactive way: gpg --expert --full-gen-key Delete Keys Delete Subkeys $ gpg --edit-key $KEYID gpg> list gpg> key 1 gpg> delkey gpg> save Delete Specific Secret Keys $ gpg-connect-agent 'HELP DELETE_KEY' /bye # DELETE_KEY [--force|--stub-only]…
Imagine a scenario: There are three nodes and each of them runs a daemon. The daemon will generate a token file in the name of its hostname under a specific location, say /tmp/node-01 . To form a fully functional cluster, the daemon on one node needs to know the tokens of the other two nodes. The only way is to sync those token files across these three nodes so that each node has all three nodes’…
As you may have already known, I have a 4-node Kubernetes cluster, which was installed using kubeadm . When I was trying to deploy metrics-server on my cluster using the official Helm chart , I got the following situation: $ kubectl -n metrics-server get deploy NAME READY UP-TO-DATE AVAILABLE AGE metrics-server 0/3 3 0 26h $ kubectl -n metrics-server get po NAME READY STATUS RESTARTS AGE…
Foreword I use Jekyll , a static site generator, to build my blog. And put the workflow of posting new articles under a CI/CD pipeline with Drone CI , integrated with Gitea via webhook. All of these went very well until someday I had my home cluster restarted (yeah, the aforementioned components are all running on my home cluster). The Crime Scene The docker build step of Drone CI was failed…
Inlets Inlets is a lightweight tunneling tool. It’s not a SaaS product, it’s a self-hosted tunneling solution. That means you have total control over it. According to the documentation of Inlets PRO , there are various use cases: Exposing services from a private network Self-hosting HTTP endpoints with Let’s Encrypt integration (so you have HTTPS) Connecting local Kubernetes with…
CentOS 7 + Zabbix 4.0 # yum install epel-release # yum install yum-utils # yum-config-manager --enable rhel-7-server-optional-rpms Zabbix Installation Adding Zabbix Repository # rpm -Uvh https://repo.zabbix.com/zabbix/4.0/rhel/7/x86_64/zabbix-release-4.0-1.el7.noarch.rpm MySQL # rpm -Uvh https://repo.mysql.com/mysql80-community-release-el7-3.noarch.rpm # sed -i 's/enabled=1/enabled=0/'…
It’s always recommended to view web pages through HTTPS connections, even it’s just a static HTML page. So, as a content provider, it’s my duty to host websites with HTTPS. To enable HTTPS on the web server like Apache or Nginx, valid certificates are required. In my case, I have bought and configured a domain name on Gandi.net for my home cluster. It’s better to have…
cloud-init’s behavior can be configured via user-data. User-data can be given by the user at instance launch time. Boot Stages Generator Local Network Config Final Generator When booting under systemd, a generator will run that determines if cloud-init.target should be included in the boot goals. By default, this generator will enable cloud-init. Local The purpose of the local stage is:…
Firstly, expand RAID 5 by adding a new hard drive into it. It will take about 1 day to reconstruct the RAID, so please be patient. root@pve2:~# pvresize -v /dev/mapper/36d0946603cb2db0024e931c9a882ced7 Wiping internal VG cache Wiping cache of LVM-capable devices Archiving volume group 'pve' metadata ( seqno 446 ) . Resizing volume '/dev/mapper/36d0946603cb2db0024e931c9a882ced7' to 10545266688…
Overview of Hadoop High Availability Before we can go about configuring YARN HA it’s important that you have: Setup a Hadoop and YARN cluster Setup a ZooKeeper cluster High Availability in Hadoop should be considered in two parts: HDFS & YARN. Both need a ZooKeeper cluster. HDFS High Availability NameNode Architecture In a typical HA cluster, two or more separate machines are configured as…
Introduction ZooKeeper is commonly used in distributed systems to manage configuration information, naming services, distributed synchronization, quorum, and state. In addition, distributed systems rely on ZooKeeper to implement consensus, leader election, and group management. In a production environment, each ZooKeeper node should be run on separate host. This prevents service disruption due to…
Redfish Data Model All resources are linked from a service entry point (root), always located at URL /redfish/v1 . Major resource types are structured in “collections” to allow for standalone, multi-node, or aggregated rack-level systems. And additional related resources fan out from members within these collections. There are three main collections: ComputerSystem: properties expected…
Poudriere is used by FreeBSD project to build the official pkg repositories. Prerequisites sudo portsnap fetch sudo portsnap extract sudo portsnap fetch sudo portsnap update ZFS sudo zpool create -m /poudriere ztank /dev/vtbd1 Installation sudo pkg install poudriere Configuration Poudriere’s main config file is /usr/local/etc/poudriere.conf : ZPOOL=ztank ZROOTFS=/poudriere…
Placement API is a separate REST API stack and data model used to track resource provider inventories and usages , along with different classes of resources. Concepts Resource Providers Resource Classes The types of resources consumed are tracked as classes. VCPU MEMORY_MB DISK_GB … The nova-compute service code will begin reporting resource provider inventory and usage information as soon…
GRUB In file /etc/dhcp/dhcpd.conf : ddns-update-style interim; allow booting; allow bootp; ignore client-updates; set vendorclass = option vendor-class-identifier; option pxe-system-type code 93 = unsigned integer 16; subnet 10.190.21.0 netmask 255.255.255.0 {} subnet 10.190.21.0 netmask 255.255.255.0 { default-lease-time 2160000; max-lease-time 432000; option routers 10.190.21.254; next-server…
Create New LUN with existing iSCSI Target on Synology We can see there is already one LUN called pve-vmimages mapped to pve-vmimages iSCSI target. Now create pve-vmimages-ext LUN and map it to the same iSCSI target. The result should look like the following: Rescan for New LUN on Proxmox VE Hosts root@pve1:~# iscsiadm --mode session --rescan root@pve1:~# lvmdiskscan…
root@pve1:~# lvmdiskscan /dev/mapper/36d094660267abf0024ff75e573972158 [ 4.91 TiB ] /dev/mapper/35000c500a0d6c0ef [ 1.64 TiB ] /dev/mapper/35000c500a0d688eb [ 1.64 TiB ] /dev/mapper/35000c500a0d68dbf [ 1.64 TiB ] /dev/mapper/35000c500a0d6a79b [ 1.64 TiB ] /dev/mapper/36001405a660c610d0d43d4ad7d80f6dd [ 1.50 TiB ] LVM physical volume /dev/sdi2 [ 256.00 MiB ] /dev/sdi3 [ 445.75 GiB ] LVM physical…
Lifecycle Controller in Recovery Mode Because disk space is not enough on pve1 , I’m planning to use 4 unused HDDs in the chassis using iDRAC’s build RAID on-the-fly feature. However, it told me the job cannot be done due to lifecycle controller issue. The lifecycle controller is in recovery mode, not enabled. So I had to reboot the server. How to Solve This racadm>>get…
記憶體挑選準則 時脈越高越好 時序越低越好 記憶體種類 揮發性記憶體 Volatile Memory Static Random Access Memory (SRAM) Dynamic Random Access Memory (DRAM) 非揮發性記憶體 Non-Volatile Memory Read-Only Memory (ROM) Flash Memory DRAM Level Channel 每組記憶體控制器個別對應一個 channel。 DIMM 早前消費者必須購買記憶體顆粒直接插在主機板上，而後才漸漸發展出將多組記憶體顆粒焊在一片電路板上的做法，隨著時代演進依序出現以下兩種配置： Single In-line Memory Module (SIMM) 頻寬 32-bit Dual In-line Memory Module (DIMM) 頻寬 64-bit Rank…
Main branches master 👍 Production-ready develop ⚔️ Branch-off from the master Main branch for the development ✔️ Suppose to be always stable Supporting branches feature 🚀 branch name: feature/JIRA-298 Branch-off from the develop Where new feature to work on Need to be tested hotfix 🚑 branch name: hotfix/JIRA-352 Merge back to master and develop…
In addition to import self-signed certificate into system, you need to import it to another place. That’s called Java truststore . Here’s one little Java class file called SSLPoke.class to let you check if you can connect the target with SSL (whatever HTTPS, LDAPS, POP3S, etc.). C:\Users\nobody\Downloads>java SSLPoke repo.maven.apache.org 443 sun.security.validator.ValidatorException :…
Check path environment variable Check Java home environment variable Replacing the following three files under C:\Program Files (x86)\Common Files\Oracle\Java\javapath with the right ones. java.exe javaw.exe javaws.exe Configuring registry settings using regedit.exe : Key: HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Runtime Environment Name: CurrentVersion Value: 1.8
Availability Zone This article only talks about OpenStack data plane availability, control plane availability is out of the scope. Think availability zone as a logical subdivision of resources into failure domain, allowing cloud applications to intelligently deploy in ways to maximize their availability. One of the things that complicates use of availability zones is that each OpenStack project…
OpenStack version: Newton Ironic API In this installation guide, Ironic API will be installed on controller node. Prerequisites Create essential user, service, and endpoint information for Ironic with OpenStack admin credential. openstack user create --password password --email ironic@example.com ironic openstack role add --project service --user ironic admin openstack service create --name ironic…
Introduction Corosync is the messaging layer inside your cluster. It is responsable for several things like: Cluster membership and messaging thanks to the Totem Single Ring Ordering and Membership protocol Quorum calculation Availability manager Setup RRP supports various mode of operation: Active: both rings will be active and in use Passive: only one of the N ring is in use, the second one will…
Standard asynchronous replication is not a synchronous cluster. Keep in mind that stand and semi synchronous replication do not guarantee that the environments replication data coherence data integrity Statement-Based Row-Based Mixed replication First fact you absolutely need to remember is MySQL Replication is single threaded, which means if you have any long running write query it clogs…
Identity: Keystone Default-Scoped (May be unscoped) $ curl -i -H 'Content-Type: application/json' -d ' { 'auth': { 'identity': { 'methods': ['password'], 'password': { 'user': { 'name': 'admin', 'domain': { 'id': 'default' }, 'password': 'password' } } } } }' http://controller:5000/v3/auth/tokens; echo HTTP/1.1 201 Created Date: Tue, 18 Dec 2018 08:47:51 GMT Server: Apache/2.4.18 ( Ubuntu )…
Server-Side Installation sudo apt install build-essential automake libssl-dev liblzo2-dev libbz2-dev zlib1g-dev wget https://tinc-vpn.org/packages/tinc-1.0.33.tar.gz tar -xf tinc-1.0.33.tar.gz cd tinc-1.0.33 ./configure --prefix = make sudo make install Configuration /etc/stunnel pid = /var/run/stunnel.pid [tincd] cert = /etc/stunnel/vpnserver.crt key = /etc/stunnel/vpnserver.key accept =…
After installing whole new Windows 10 on my new computer, I installed Oracle Java version 8 update 162, which made me unable to use BMC Java console. Due to security policy, new version Java disable MD5withRSA by default. Comment out the following line in C:\\Program Files\Java\jre1.8.0_162\lib\security\java.security : #jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024, DSA keySize < 1024…
$ echo 'This is an input test' > test.txt $ exec 3< test.txt $ cat <& 3 This is an input test $ exec 4> test.txt $ echo 'This is a truncate write test' >& 4 $ cat test.txt This is a truncate write test $ exec 5>> test.txt $ echo 'This is an append write test' >& 5 $ cat test.txt This is a truncate write test This is an append write test exec 3<&- exec 4<&- exec 5<&- References Playing with file…
Heartbeat is not the best tool to get a redundant haproxy setup, it was designed to build clusters, which is a lot different from having two redundant stateless network equipments. Network oriented tools such as keepalived or ucarp are the best suited for that task. The difference between those two families is simple: A cluster-oriented product such as heartbeat will ensure that a shared resource…
$ mkdir certs $ openssl req \ -newkey rsa:4096 -nodes \ -sha256 \ -keyout certs/domain.key \ -x509 \ -days 365 \ -out certs/domain.crt $ docker secret create domain.crt certs/domain.crt $ docker secret create domain.key certs/domain.key $ docker secret ls ID NAME DRIVER CREATED UPDATED 108vdrp0wpa7sl2b99gvqp2gd domain.crt 4 days ago 4 days ago rjx8c7h8j0k2yb19dxae25nji domain.key 4 days ago 4 days…
This is an relatively simple guide about how to install DRBD (Distributed Replicated Block Device) on two CentOS 6.9 hosts in active/passive mode as block-level data failover. DRBD Introduction The DRBD is a software-based, shared-nothing, replicated storage solution mirroring the content of block devices (hard disks, partitions, logical volumes etc.) between hosts. DRBD mirrors data in real time…
Elasticsearch curl -L -O <https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.2.0-linux-x86_64.tar.gz> tar -xzvf elasticsearch-7.2.0-linux-x86_64.tar.gz cd elasticsearch-7.2.0 config/elasticsearch.yml network.host : 0.0.0.0 cluster.initial_master_nodes : [ '172.16.169.17' ] sudo sysctl -w vm.max_map_count = 262144 ./bin/elasticsearch Kibana curl -L -O…
Direct server return is usually shortened to DSR. In DSR mode, the load-balancer routes packets to the backends without changing anything in it but the destination MAC address. The backends process the requests and answer directly to the clients, without passing through the load-balancer. The backends must have the service IP configured on a loopback to be able to accept the requests. TCP…
NAT stands for Network Address Translation. In the NAT mode, the load-balancer will route traffic between user and server by changing destination IP address of the packets. TCP Connection Overview TCP connection is established between the client and the server. The load-balancer just ensures a client is always forwarded to the same server. Data Flow As shown below, the clients get connected to the…
As KDDI/NOS’ requests, we’re constructing a new network architecture of our development site which consists of three parts: KCPS Bare-Metal: for CKKB (HaaS-like project) KCPS: for original KCPS Dev: internal use of BAMPI team To make the new network architecture work, the main challenges are: VLAN Routing DHCP relay ACL Currently, Cisco N3K is our core router (actually it is a layer 3…
Note: This article assumes you are using MySQL 5.7.6 and later. Default Root Password of Newly Installed MySQL 5.7 After you installed MySQL community server 5.7 on fresh Linux, you will need to find out the temporary password from /var/log/mysqld.log to login as root. grep 'temporary password' /var/log/mysqld.log Run mysql_secure_installation to change new password The above method is for normal…
HaaS is a emerging type of cloud service model which aims physical machines rather than virtual machines. Due to this key differences, in a typical HaaS deployment, network configuration should be VLAN-oriented. HaaS utilizes Peregrine and OpenStack Neutron to achieve tenant network isolation, DHCP and layer 3 routing functionalities. To configure Neutron to support HaaS, we need to modify Neutron…
A split-brain condition is the result of a Cluster Partition, where each side believes the other is dead, and then proceeds to take over resources as though the other side no longer owned any resources. After this, a variety of Bad Things Will Happen - including destroying shared disk data. This is the result of acting on incomplete information - neglecting Dunns Law. That is, when a node is…
This tricky network issue consists of: Subnet overlapping ARP request not ignored Proxy ARP race condition Asymmetric routing arp_ignore arp_ignore - INTEGER Define different modes for sending replies in response to received ARP requests that resolve local target IP addresses: 0 - (default): reply for any local target IP address, configured on any interface 1 - reply only if the target IP address…
Currently BAMPI uses remote executor to do various types of tasks. Apart from Ironic’s agent pattern, remote executor is based on SSH. For high availability design, we need to consider a lot of situations, one is “while during task execution, BAMPI-1 fails”. In this situation, remote executor is dead, but the task being run still remains on the target bare-metal. So how does…
You probably familiar with manipulating screen operations in your screen sessions. How about “outside” the screen session? Here are some tips about it. If you’re interested, please have a look. Manipulate Screen from Outer Space Create screen session with a window running specified command: screen -dmS testing -t shell bash Turn on log: screen -S testing -pshell -X logfile…