Bargain 5 Node Cluster of PINE A64+
Building a 5-node cluster for under £75 (£190 including five quadcore PINE A64+s)After the interest in my cluster of Raspberry Pi 3s in April, I was keen to try building a cluster with some of the other excellent SBCs now on the market. PINE 64 in the States generously sent me 5 of their latest PINE A64+ ARM boards, each with a 64-bit quad core ARM A53 SoC running at 1.2GHz, gigabit Ethernet and 1GB of DDR3 memory. Launched as a Kickstarter-funded project it was hugely successful, raising over $1.7 million dollars from tens of thousands of enthusiasts.
This article includes an Updated design 2017: Universal cluster case (May 2017)
The completed cluster measures 146.4 (w) × 151 (h) × 216mm (d) and weighs 1.5kg. (5.6 × 5.9 × 8.3", 52oz)
PINE A64+ vs Raspberry Pi 3
The PINE A64+ board has similar specifications to the Raspberry Pi 3, but has faster memory, much faster networking and yet is quite a bit cheaper even when you include international shipping to the UK:
| Model | PINE A64+ | Raspberry Pi 3 |
|---|---|---|
| SoC | 4-core ARM A53 1.2GHz | 4-core ARM A53 1.2GHz |
| Memory | 1GB DDR3 (2GB option) | 1GB DDR2 |
| Network | 1000Mbps | 100Mbps |
| WiFi | option | 802.11bgn |
| Bluetooth | option | 4.1 + BLE |
| Infrared | yes (header) | no |
| Storage | microSD card | microSD card |
| USB spare | 2 fitted | 4 fitted |
| Video | HDMI 1.4 CEC, DSI-LCD | HDMI, DSI |
| Camera ports | CSI | CSI |
| Audio | 3.5mm, SPDIF (header) | 3.5mm |
| Microphone | yes (headphone+mic jack) | no |
| Size | 127 × 79mm | 85 × 56mm |
| Power | 1.0 → 2.9W 2A max, microUSB | 1.2 → 2.1W 2.5A max, microUSB |
| Launched | Q2 2016 | Q1 2016 |
| Price (UK) | £25.661 | £33.59 |
1US$19 A64+ + $12 shipping + 20% VAT + 0% import duty = £25.66 | ||
Which are MOST important if you bought/made a cluster?
3D design in SketchUp
I modified my original Raspberry Pi cluster design using the free version of SketchUp and built rough 3D templates of the PINE A64+s, network switch, USB hub, sockets, etc. I didn’t bother to include ventilation slots/grills in the 3D model.
- Download SKP file for SketchUp 2013 (old 2016 design)
2D design in Inkscape
I used the free Inkscape application for 2D design, ready for exporting to the laser cutter. Each colour is a different pass of the laser, at different power/speed levels, so the green lines are cut first to make holes for ports/screws/ventilation, pink are extra cuts to help extract delicate parts, orange is text/lines that are etched and finally blue cuts the outside of each panel.
Download the (old 2016 design) files for laser cutting on one 600×400×3mm sheet:
Design Changes from the Raspberry Pi 3 cluster
Although similar to my cluster of Raspberry Pi 3s, I made a number of changes and improvements:- I kept the horizontal mounting rail design but the A64+ has M3 holes which are easier to find parts for, rather than M2.5 on the Pi. And the holes are further apart because the overall board size is much bigger than the Pi. Screwing the plastic nuts onto horizontal rails is a bit tedious, and I’d like to 3D print a C-shaped ‘clip’ that holds the boards in place along each rail, or perhaps use tight elastic washers.
- Internal Case Fan vs External – I was sure that the A64+ boards would benefit from active cooling, and so I stuck with an ultraquiet 92mm fan, but moved it inside the case, and to the back rather than on top.
- Case USB ports – while perfectly functional, I never really liked the look of the twin USB ports on my Pi cluster, so for the A64+ cluster I sourced a more compact 2-port USB mount.
- Internal USB hub on case side vs Internal shelf – attaching the USB hub to the case side panel meant I could simplify the design (it can now be cut from a single sheet of 600×400mm acrylic), and so reduce the costs. Removing the shelf would have reduced the rigidity of the case, but screwing the horizontal mounting rails to the side panels keep it secure. I also moved the USB hub right up to the rear case panel, so it could be plugged in without needing a case-mounted socket & lead, which again saves costs, and avoids the need for soldering a custom cable.
- Right-angle USB extenders – I didn’t realise until everything else was finished that the USB cables would stick out too far at the back and block the fan so I had a last-minute purchase of two right-angle extenders so I could connect up the USB. A neater solution would be making/buying a single cable.
- Anear 60W 6 Port USB vs SAVFY 50W 6 Port USB – I was hoping to find a 6-port USB hub that could be screwed directly onto a case panel (without needing any glue), but the cheap SAVFY 50W proved unsuitable, as it was almost impossible to remove the outer casing and had no mounting holes in the PCB. So I stuck with the Anear 60W hub.
- Green microUSB cables vs Boring black – bit of a stretch to call them pine-coloured... ? Could do with being slightly longer to more easily reach the furthest two boards.
- Flat 25cm LAN cables vs Round 15cm – I loved the rainbow LAN cables in my RPi3 build, but it was a tight fit bending them in the case. These flat cables bend far more easily, but 25cm is too long really and 15cm would be neater.
- White LAN cables vs Boring grey – goes nicely with Gelid fan?
- Gelid Solutions Silent 9 fan vs Nanoxia Deep Silence fan – I was very happy with the performance of the Nanoxia fan (and their excellent customer service) but wanted to try out a cheaper option. The Gelid rubber grommets are thicker than the Nanoxia ones, so I increased the case mounting hole diameters by 0.5mm.
- Gigabit vs 10Gigabit-switch – the A64+ network ports are each 1000Mbps (10× faster than the Pi), so using at least a 1000Mbps switch is a no-brainer really. A 10Gbps switch would eliminate any bottleneck (e.g., if the A64+s were simultaneously saturating their links talking to the outside network), however these are still very expensive at £600+, and too large to fit inside this case. The new Asus XG-U2008 switch looks promising.
- Ceramic heatsinks vs Aluminium – rather than the small 14x14x5mm aluminium heatsinks I’ve used for most of these SBCs, this time I tried some innovative 25x25x5mm ceramic heatsinks that extend over the edge of the SoC to give a greater cooling area, and feature a “micro porous ceramic finned” structure. They seem to work well, and I’m going to write a separate article that compares some of the different heatsinks available.
- Universal cluster case – the A64+ boards are much bigger than the Pi, which made squeezing them into the cluster case a challenge, especially with the internal USB hub. The obvious answer would be to have made the case larger, but I’m trying to design a cluster case that will fit any of the ARM SBCs, simply by swapping the 2 side panels. Work in progress!
Power, temperature & cooling
At idle, the entire system of five A64+s, network switch & 5V fan sips a mere 9W, and at 100% load it still only uses 19W in total. There is the possibility of further reducing the power requirements by disabling HDMI?Is a fan needed? The PINE A64+ boards dissipate their heat better than any of the other SBCs I’ve tried (perhaps helped by the much larger surface area of the boards?), including the Raspberry Pi 3. You could certainly use the cluster without any fan, but I would fit a 5V fan if you want maximum performance or are going to run the cluster at high load for extended periods.
The current release of Debian Linux (3.10.102-2-pine64-longsleep kernel) is unusual in that it ships with the CPU performance governer locked to 1152MHz. I think this will be changed in a future release, and I’ve configured my boards to use the ondemand governer that dynamically adjusts the CPU speed between 480MHz and 1152MHz as required. This reduces electricity used as well as keeping the boards cooler:echo ondemand | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >/dev/null
The USB hub can supply up to 60W (2.4A per A64+), which is more than enough for a couple of power-hungry external devices to be plugged into the USB ports. Using:
cat /sys/devices/virtual/thermal/thermal_zone0/tempto measure the SoC core temperature, the cluster idles at 33.4°C (92°F) with cooling from the 5V fan.
At 100% load and the 5V fan, using:
sysbench --test=cpu --cpu-max-prime=200000 --num-threads=4 run &the SoC core temperatures reached a stable 63.2°C (146°F). If you don’t provide at least some active cooling, the SoCs will get as high as 80°C and automatically throttle down their clock speed, to avoid overheating. They can safely run long-term at that temperature, but you don’t get maximum performance.
Exactly the same case design should work with the A64 model.
Silent cooling
To cool down the cluster I fitted a single 92mm fan inside the rear of the case. I used a (effectively) silent fan recommended by Quietpc.com, the Gelid Silent 9 (£5.53).At 5V I have to get my ear within 50-75mm (2-3") to hear even the slightest whisper from the fan, and the supplied rubber gromets definitely do a good job of isolating the case from any small vibrations. However at 12V the fan is quite audible in an otherwise silent room, so I was looking for the voltage that would provide enough cooling, while keeping the fan silent.
For a single A64+ you might not need a fan at all, or perhaps a small 40-60mm one.
| description | heatsinks? | idle | 100% load | performance |
|---|---|---|---|---|
| Case, 12V fan, 1500 rpm | yes | 30.4°C | 53.8°C | OK |
| Case, 7V fan, ? rpm | yes | 31.4°C | 57.6°C | OK |
| Case, 5V fan, ? rpm | yes | 33.4°C | 63.2°C | OK |
| Case, no fan | yes | 40.8°C | 75.6°C | throttles |
Temperatures between the different A64+s was spread over 13 degrees, so an average of 75.6°C (168°F) actually had three of the boards throttling.
Powering the fan
Unlike PC motherboards, a A64+ doesn’t have a standard 3 or 4-pin fan header, but it is quite easy to power the fan using the GPIO 5V power rail instead. WARNING you can’t run anything that needs a lot of current this way – however a low-power fan should be fine. You do need to make sure you have a quality power supply, and not too many power-hungry USB devices plugged into your A64+, otherwise you’ll get crashes & rebooting. The main GPIO pins are for data, but using the VDD_5V power rail on pin 2 or 4 should work in theory.This should be enough to run the fan with 5V, or if you need more cooling, splice a step-up/boost converter in the middle to adjust the speed of the fan, by controlling its voltage between 5V and 12V.
Building the A64+ Cluster
- Remove network switch case (2 small screws)
Attach the network switch to case base, using 4× 6mm brass spacers + 4× 14mm bolts + nuts. This only fits one way around. (The external LAN cables route neatly underneath the switch PCB) - Remove USB hub case (tricky – needs to be carefully prised open, there are no screws). Fit 4× 6mm brass standoffs to the case side, using screws
Superglue the metal top of the USB hub to the brass standoffs. This needs to be correctly positioned so that the USB hub power socket sticks out of the back of the case when it is fitted together![Building A64+ Cluster case 2]()
- Screw 2 external LAN ports to inside of the case back (has “AC100-240V” etched on the outside)
Fit the 92mm case fan to inside of the case back using the supplied rubber grommets. The fan logo should face inwards if you want the fan to blow cold air into the case. The Gelid rubber grommets are a VERY tight fit and might need to be pushed into place with a small screwdriver/pen as well as pulling quite hard!
Clip case back to case base (marked C+D)
Plug external LAN into network switch ports 1+3 (no room to use 1+2)![Building A64+ Cluster case 3]()
- Place a heatsink on the SoC chip of each A64+, and hold in place for a few seconds to let the adhesive stick
Slide 5 A64+s onto 4 threaded rods using 48 nuts to secure. This might be faster with the rod in an electric screwdriver? Leave 9mm space at left end, space each A64+ 22mm apart. The LAN+HDMI side of the A64+s point towards the front of the case
Attach case sides to A64+ rods (EF+AB), secure with 8 metal nuts.![Building A64+ Cluster case 4]()
- Fit the 3-pin fan plug into the GPIO pins of one of the A64+s, so that black (ground) connects to pin 6, red (power 5-12V) to pin 2, and yellow (fan speed) is unconnected. To remove the individual wires from the plug, depress the tiny metal tabs (e.g., with a small screwdriver), and the wires will slide out. I cut off the yellow wire which I wasn’t interested in, and slid the wires back into the plug so that red was on the left, the middle was empty, and black was on the right:
![Building A64+ Cluster case 5]()
- Plug A64+ LAN cables into network switch ports 4-8 (be very careful if removing these later – it is easy to break tiny plastic clips on the switch ports)
Clip case sides to case bottom (E+F, A+B)
Plug USB hub to network switch DC power
Plug green USB cables into USB hub![Building A64+ Cluster case 6]()
- Route HDMI cable through case
Screw external twin USB to front case
Bolt external HDMI to front case, 2 nuts
Plug 2 external USBs + right-angle adaptors into any A64+ (or 2 separate A64+s)
Plug A64+ LAN cables into A64+s
Plug A64+ micro USB cables into A64+s
Plug HDMI cable + right-angle adaptor into any A64+![Building A64+ Cluster case 7]()
-
Clip case front to case bottom (G+H)
Clip case lid to case sides, front & back (M+N+O+P+Q+R+S+T)
Attach self-adhesive rubber feet to underside of base
Bill of materials
Most of these parts were sourced from individual sellers on Amazon or eBay, which of course racks up the postage charges. If there were enough demand, it would be cheaper to bulk buy the parts and have a kit with everything you need to build the cluster.
| Edimax ES-5800G V3 Gigabit Ethernet Switch | £13.49 |
| White flat 25cm Cat6 LAN cables (5 pack) | £4.45 |
| Green micro USB cables (5 pack) | £9.45 |
| USB Charger Anear 60W 6 Port USB | £14.99 |
| M3 steel screws 8mm (10 from a 20 pack) | £1.70 |
| RJ45 male to female screw mount (2 pack) | £1.98 |
| M3 steel hex nuts (4 from a 5 pack) | £1.10 |
| M3 steel screws 14mm (4 from a 5 pack) | £1.35 |
| M3 brass female standoff 6mm (8 from a 10 pack) | £0.99 |
| M3 steel threaded bar 150mm + nuts (4 from a 5 pack) | £5.50 |
| 0.5m HDMI male to female panel mount (inc. bolts) | £2.39 |
| Twin USB female socket to male cable | £2.19 |
| USB right-angle male to female socket extender (2 pack) | £1.98 |
| Polyurethane rubber feet (4 from a 12 pack) | £2.99 |
| M3 nylon hex nuts (48 from a 100 pack) | £1.29 |
| 3mm extruded clear perspex 600×400mm | £5.32 |
| Laser cutting charge | n/a |
| HDMI 270 degree adaptor | £0.99 |
| Gelid Silent 9 92mm case fan | £5.53 |
| Subtotal inc P&P | £77.68 |
|---|---|
| PINE A64+ at US$19/each (5 pack)1 | £93.56 |
| Kingston class 10 16Gb microSDHC card (5 pack) | £16.95 |
| Ceramic heat sink 25mm × 5mm (5 pack) | £4.74 |
| Total inc P&P | £192.93 |
1The PINE A64+ is duty free to import into the UK, costs US$18.00 to ship 5 boards from the USA, and there is UK VAT at 20% bringing the total to £93.56.
The cluster of A64+s (excluding the boards themselves) is cheaper to build than the Pi3s: this is mainly because I’d found better deals on some of the components, as well as simplifying the design to remove the case shelf + AC socket.
Show me the Benchmarks!!
I’m writing a separate article benchmarking this PINE A64+ cluster against my original Raspberry Pi 3 cluster, as well as detailed benchmarks comparing the individual boards. Watch this space...Clusters of other Single Board Computers
So far I’ve built clusters using the following ARM boards:- DIY 5 Node Cluster of Raspberry Pi 3s
- 40-core ARM cluster using the NanoPC-T3
- 5 Node Cluster of Orange Pi Plus 2e
- Bargain 5 Node Cluster of PINE A64+
- ClusterHAT with 4× Raspberry Pi Zero
- 96-core ARM supercomputer using the NanoPi-Fire3
The NanoPC-T3, Orange Pi Plus 2e and PINE A64+ clusters mostly share the same components, including the acrylic case panels – only the 2 side panels are unique because the boards are different sizes. I’m planning to revise my original Raspberry Pi cluster design so it will share most of these same case/components.
I’d like to build a small cluster of all the current crop of sub-$100 ARM SBCs, comparing the different features, and with detailed benchmarks. e.g., Odroid C2/XU4 and the Banana Pi M3. Please email me if you’d like to send boards for review.
Software to run on a cluster?
or... What is it for??Education, training, blah, blah... well personally I’m just running a Debian Linux provided by Pine64.Pro on each A64+ for now, and I’m going to experiment with things like load-balanced web/database servers.
Running Docker on ARM on each node looks like an excellent way of controlling the cluster.
Updated design 2017: Universal cluster case
I’ve made a handful of minor improvements to the design described above, but the essential details of the cluster remain the same. The total cost has gone up by £1, but the case is now more rigid, and the cluster cabling slightly neater. This case should work for any of the ARM SBCs, simply by swapping the 2 side panels.
The USB hub is now securely attached to the case by carefully drilling two 3mm holes through the plastic end-piece of the hub, allowing it to be bolted to the rear case panel. The hub is still attached to the side case panel, but now with two M3 4mm brass standoffs glued to the heatsink of the hub, instead of four 6mm standoffs. This design is much stronger for repeated (dis)connections of the power cable.
The front-panel twin USB port has been replaced by two separate USB ports with right-angled male plugs, so there is no need for the two right-angle extender cables.
| 0.5m HDMI male to female panel mount (inc. bolts) | £2.39 |
| 30cm HDMI male to female panel mount (no screws) | £1.95 |
| M2.5 steel screws 8mm (2 from a 5 pack) | £1.25 |
| Twin USB female socket to male cable | £2.19 |
| USB right-angle male to female socket extender (2 pack) | £1.98 |
| 25cm USB female panel mount to right-angle male plug (2 pack) | £2.38 |
| M3 brass female standoff 6mm (8 from a 10 pack) | £0.99 |
| M3 brass female standoff 6mm (4 from a 50 pack) | £0.99 |
| M3 brass female standoff 4mm (2 from a 50 pack) | £0.99 |
| M3 steel hex nuts (4 from a 5 pack) | £1.10 |
| M3 steel hex nuts (6 from a 10 pack) | £0.99 |
| M3 steel screws 8mm (10 from a 20 pack) | £1.70 |
| M3 steel screws 8mm (8 from a 10 pack) | £1.50 |
| M3 steel screws 6mm (2 from a 5 pack) | £1.20 |
| Subtotal inc P&P | £78.58 |
|---|
Download the 2017 design files for laser cutting on one 600×400×3mm sheet:
Nick Smith, August 2016 & May 2017.
