FOURGANG Twitter thread

Back to the FOURGANG project

@mrled: Been working on a home cluster project for a while and I finally hit a milestone today - a kernel node can finally load a kernel over tftp!!!! ✨✨✨ https://t.co/MZypYJD0I6
@mrled: The cluster nodes are Odroid HC2 ARM machines. I want to play with LizardFS and determine its performance at small scale (just a few nodes). Going to use Docker Swarm and other high availability systems so my home services are fault tolerant
@mrled: The USB UART for these boards has some VERY FUCKING BRIGHT LEDS on it lol https://t.co/ObmJ0f0j3A
@mrled: Booting a little bit further... https://t.co/4DbtLoloz4
@mrled: The Odroid HC2 on the right is booting from the Raspberry Pi 2 on the left https://t.co/zs4rFmQaOX
@mrled: The Pi 2 is very slow 😵 at least compared to the Odroid HC2, so I have an Odroid XU4 in the mail to replace it. The XU4 and the HC2 are based off the same platform... kernels, device trees, etc can be shared.
@mrled: I was trying to hold off because I already had the Pi. The Pi is the DNS server for the network, and gives hostnames to cluster nodes, so its resolv.conf needs to point to itself for DNS. And if DNSSEC is turned on in BIND, it needs to connect to other DNS servers securely.
@mrled: BUT THE PI DOESNT HAVE A HARDWARE CLOCK. When it boots, the time is off by days/weeks, so DNS lookups to external hosts fail. Normally you’d use an NTP client to set the clock on boot, but you need to reach the NTP pool by DNS.
@mrled: (Technically you could hard code someone’s NTP server’s IP address, but this is considered bad manners, and is not robust besides. The pools are made by round robin DNS, so if you want to use Internet NTP servers correctly, you must have working DNS.)
@mrled: And I could have gotten a hardware clock module for the Pi, or tried to solve it some other way... but the XU4 has a clock (and the backup battery is cheap), and it’s faster, so here we are
@mrled: New Odroid XU4Q to replace my raspberry pi 2 as the bootstrapper for the cluster. Shit this thing is so much faster. You can see the backup battery for the rtc right above the Ethernet cable, so it shouldn’t lose its clock on a reboot, breaking DNSSEC and DNS resolution. https://t.co/0rwts4298e
@mrled: The XU4 really sped this project up until RSI and the holidays really slowed it down. Made some more progress recently though.
@mrled: Some first draft diagrams when I thought I’d use as 12v to 5v converter of some kind, before I realized I’d be better off with a separate 5v PSU for the XU4 and USB hub (ignore the “12v->2v” typo, it should be “12v->5v”). https://t.co/BgZq6Dy5ry
@mrled: More recent layout, with the longer screwdown terminal present for 12v, and new 5v PSU present next to it’s shorter screwdown terminal. https://t.co/q0SKzT09pU
@mrled: And then this evening I started to lay out the board in pencil. (I went, well, back to the drawing board.) https://t.co/MKgyEOlMzW
@mrled: That last photo was taken with the board rotated 180 degrees from previous photos. Shows the layout of where the HC2 nodes will go. The switch is on the left, and there will be a groove over the lines with X through them - to fit the groove on the side of the HC2 enclosure
@mrled: That way the HC2 nodes will lie flat on their sides. I’ll use little bracket things like this to hold each node in place but allow it to be removed for troubleshooting, hard drive replacement, etc. (There’s no groove in this pic, so the nodes are a leaning a bit.) https://t.co/DDmoSFLwcy
@mrled: Some Dremel work today. Worked on a project box to house my wall plug receptical. Clearly I’m no Dremel artist. https://t.co/9s7VYSwNtE
@mrled: But it does look halfway decent with the power inlet module installed!! https://t.co/ON6DscdsL4
@mrled: And here it is put together with a port for the wires. The wires will carry AC power from the inlet to my two PSUs, which will convert them to 12V DC for the HC2s, switch, and router; and to 5V DC for the XU4 and USB hub. https://t.co/64kOLEZeep
@mrled: I did all that work just holding the Dremel in my hand, with the plastic pieces in a vise. I didn’t have any guide for the Dremel to make straight edges - not sure if there’s a way I could have, this is my first time ever using one.
@mrled: Time for more amateurish Dremeling. I finished all the grooves for the 6 possible HC2 nodes (I only have 4 currently), plus an extra one in the middle because I fucked up my reference point :( https://t.co/lFotAQXcEf
@mrled: I used the straight edge guide with the drill bit for this, which seemed like the right thing to do? based on the Dremel kit I have. However, maybe I could’ve made a jig and used a circular cutter for more precision.
@mrled: (Not that it *really* matters, since the ugly grooves will be hidden by the HC2s on top of them. Except for the extra one 😔 and the two grooves for nodes I haven’t decided to buy yet 😔.)
@mrled: And now I’m super tired. Wanted to also install some of the threaded inserts I got which will let me attach/remove components to my back board repeatedly using standard 6-32 computer case screws... but I’m not doin that shit tonight. https://t.co/a47P6OeKgV
@mrled: Never used these before either but it was really important that I could remove a component to fix or replace it without needing to screw a new hole in the plywood every time. And it’s convenient that they come in 6-32, considering I have zillions of those screws lying around.
@mrled: Turns out I don’t have the drill bit large enough to drill the pilot for my threaded inserts :(. I did lay out where all the inserts will go though.
@mrled: This SHOULD be the “final” layout... not that I didn’t immediately start thinking of how I’d expand this in the future. https://t.co/Lya25akbHs
@mrled: I finally Committed today and installed all my threaded inserts. A few are slightly off - they’re a little hard to install straight and it’s hard to drill by hand as precisely as some of my components have been manufactured :( ... still, was able to confirm everything will fit! https://t.co/KumYA1EWup
@mrled: This board will attach to the side of the main back board, and the switch will attach to it. Had to do it this way because the switch needs to lay flat against what it’s attaching to, but ran out of room on the main board. https://t.co/g5Gb0E71j7
@mrled: Drilling into the boards was causing significant splintering so I sanded around the inserts, and on the edges and corners too. The boards I got are nice and flat, but the plywood layers seem a little brittle. The cut against the grain for the small board caused some damage.
@mrled: Finished attaching the back board for the switch to the main back board on Friday ✨ https://t.co/HoZwBRYWxc
@mrled: Ended up using 3.5” framing screws to go from the face of the switch back board into the edge of the main back board (first photo in last tweet). Tested first on a piece of scrap, which is good because I learned that my impact driver would split the board 😬
@mrled: That corner bracket isn’t my most precise work... I marked the holes for screws (which are smaller 1.5” wood screws) before attaching with the framing screws, and that attachment is 1/16” from flush, so it’s a bit wonky
@mrled: But it’s sturdy and it works!! :)
@mrled: Then yesterday I finally got to start ATTACHING COMPONENTS OMG. Here’s the 12V power supply - those corner brackets weren’t the MOST nice thing I could have used, but they did have the advantage of already being in my house https://t.co/fyTPcz9AHO
@mrled: Attaching it to the other side started to show a minor flaw in my screw plan - I had wanted to use 6-32 threading for the brass inserts, thinking I could use the hundreds of computer case screws I have lying around. That’s the silver screw going into the backboard. https://t.co/phOsVE7ogA
@mrled: But you can see the washer underneath it - that’s because the screw head was small enough to fit in the hole on the black corner bracket I got. Sigh.
@mrled: To make matters worse, only the very longest of the screws I had on hand would go through a washer and bracket and into the brass insert, so I really had to hunt for the right ones.
@mrled: In other cases, like the screw terminals, most of the screw heads I have are too BIG to fit, so I had to go hunting for 6-32 screws with smaller heads. (A normal screw is leaning against the right one for comparison in this pic.) https://t.co/WRmlE3qdy1
@mrled: (Also, look how uneven the two terminals are - they came in the same pack on amazon lol.) https://t.co/uRdTMpbOQp
@mrled: For the master node SSD bracket, I ended up having to cut it so screws with normal sized heads would fit, which looked very cool. Used a Dremel with a fiberglass cutting wheel, and some eye protection of course. https://t.co/Ki3vVE2TDD
@mrled: I can’t believe this thing can just cut through metal like this. Dremel skills might be the most exciting thing I’ve learned over the whole project. https://t.co/Ge0aYIsaPq
@mrled: Also attached some black angle brackets to the USB hub so it fits on its side. Again, dremel, this time with the drill bit. You might notice the silver brackets - this was the screw head problem again, and the #6 washers I have were even too small, so I improvised. https://t.co/xAnTBt9xkO
@mrled: I’ve ordered some #8 washers that should solve this problem, but for now the only issue is that it looks ugly. Also ordered a bunch of 1/2” 6-32 screws so I’ll have ones that are long enough.
@mrled: A shot to show that there is plenty of room inside the plastic housing for nuts to keep the 6-32 screws in place on the USB hub https://t.co/FPZl7srKPP
@mrled: Making so much progress!! These two were taken about a half hour apart yesterday. Installing the components is so satisfying, I can finally see it coming together ✨✨✨✨ https://t.co/u5QxUaNkqK
@mrled: That brings us to the end of the day yesterday. Well, actually, just to the afternoon - had an overnight trip planned and we got back about an hour and a half ago.
@mrled: Today I worked on installing the actual cluster nodes, and it worked out as well as I hoped! When they’re installed this way it’s easy to screw them down and they are quite solid, not gonna move. https://t.co/EJjzraOWmt
@mrled: (You might notice the damned screws are FLATHEAD. These were 1/2” 6-32 screws I already had, but being flathead they’re obviously garbage. The screws I ordered today are 1/2” 6-32 Phillips and as I un/re-mount units for testing, I’ll replace their screws.)
@mrled: With the 1/2” screws I can easily loosen the cluster nodes without having to completely unscrew them from the back board https://t.co/w2sYVoB2yn
@mrled: Also attached the switch and router today, which means...
@mrled: I’ve attached all the components!! Currently that means only 4 cluster nodes with empty slots for 2 more. This is SO exciting, it looks how I planned it and everything is solid... even with some mistakes, the plan worked!! 💎 https://t.co/RG1uaEK44J
@mrled: Next step is wiring power, which is going to involve a lot of soldering and testing one by one. After that, Ethernet/power/serial cable installation and routing, and then finally software. (Oh god, right, there is still a lot of software work to do.)
@mrled: Did some initial playing around with power, discovered I failed to order ground wire and only have red/black +/- wire for AC in. Also, I don’t know how to wire this switch! Need to do more research. https://t.co/AXbY5ziJIg
@mrled: Also wired power from screw terminal to the router. Not sure I made the right choices here either - the lamp wire I got has very thick shielding and is kinda bulky, and there isn’t much room to maneuver when attaching to the screw terms. https://t.co/F9roo3VHNW
@mrled: Also, the damned lamp cord doesn’t mark one side or the other, so I have to keep double checking to make sure I’m wiring negative to negative and positive to positive.

Back to the FOURGANG project