Scaling “E-Waste”: LLMs on Multiple Decommissioned GPUs

The critical finding last time was that: in this world of decommissioned enterprise hardware, three and four GPU setups are counter-intuitively durable against diminishing returns. Within a homelab GPU appliance, you can just keep adding GPUs to proportionally scale performance. Given the low cost of older ~16GB GPUs, this puts LLMs larger than the ~10B parameter single GPU models within reach for a local daily driver.

To better understand how multiple GPUs handle diverse workloads, I’ve expanded our benchmarking tool with over 130 new tests tailored for LLMs. Following requests from commenters, I also integrated power consumption monitoring to compare efficiency across different builds. Detailed results and analysis to follow.

To reiterate: these cards are cheap because they run hot and ship without cooling, a non-starter for the typical secondary market customer. The project underwriting all this research has been my standalone active cooler, which is almost ready for release. If you’re interested in helping push it over the line by stress testing a sample, I’ll be recruiting beta testers using the mailing list.

Continue reading →

Benchmarking 15 “E-Waste” GPUs with Modern Workloads

Decommissioned NVIDIA enterprise GPUs are one of the last remaining sources of idle VRAM. K80 with 24GB of GDDR5 sells for $60, P100-16GB for around $75 and V100-16GB for under $200. Shortcomings and all, it’s important to understand if we can utilize these widely available cards in the modern era. This project to benchmark tesla GPUs has been in the works for almost a year and over the winter I was able to spend many kilowatt hours heating up my studio with GPUs.

 

Continue reading →

Saving a Secondhand Tripod

There are several semi-professional thrifters in my life that regularly rotate through the different Goodwill stores in our city prowling for deals. Their habit exposes me to a bit of second hand second hand shopping, and I always request camera equipment. Above is the current state of the tripod pile.

Often times these donated tripods don’t come fitted with their adapter plates for attaching to cameras. This nominally terminal condition is easily recoverable in the age of 3D printing.

Continue reading →

Introducing a GPU Server Benchmark for Tesla GPUs

Previously a plan was set in motion to benchmark my collection of mighty Tesla GPUs. The set of cooler manifolds are designed, a GPU server benchmark suite has been created, and the time has come to start working through the spreadsheet. I have long suspected that the older multi node cards could be fantastic for image processing. Finally we can quantify how much life is left in these older cards.

Continue reading →

First look at the New GPU Cooler Prototype

For the past while I’ve been working on a major redesign of my high performance gpu cooler project.

The rapid ascent of the LLM into the collective consciousness has sent the big players into a frenzy over datacenter GPUs. This is putting accelerating downward pressure on the price of all used compute GPUs, even the historically pricey stuff. P100s can be had for ~$100, V100 16GB are selling for ~$500, any day now the lower VRAM Ampere cards are going to drop below $1000…

We need a new cooler that can support any of these 300W GPUs, can be installed as densely as the cards, and is easy on the ears for homelab use. Pictured below are four prototype coolers installed on Tesla P100s:

Continue reading →

1kW of GPUs on the OpenBenchTable

In the conclusion of my previous homelab post, I pled to the eBay gods begging for a 4xP100 system. My prayers were heard, possibly by a malevolent spirit as a V100 16GB for $400 surfaced. More money than I’d be willing to spend on a P100 but the cheapest I’d ever seen a V100, I fell to temptation. To use all four cards, I needed something bigger than the Rosewill RSV-R4100U. Enter the OpenBenchTable, and some 3d printed parts I designed to be able to securely mount four compute GPUs:

Continue reading →

Hardware for Engineering Streams

My beloved blog post still sits on the throne as the most effective format for engineering projects. To me, Inlining code, photographs, CAD models and schematics in an interactive way trumps other mediums. This level of interactivity closes the gap between reader and material, allowing an independent relationship with the subject outside of the story being told by the author.

Working on stream to an online audience has a similar effect, the unedited and interactive format yielding a real understanding of the creators process and technique.

For a while there, I’d settled into a nice habit of broadcasting project development live on twitch. Two moves later, things have settled down enough in my personal life that I feel it’s time to try to get back into this habit.

Before we get started again, I took some time to improve the ergonomics (both hardware and software) of my stream setup. The following documents a few smaller projects, all in service of these upcoming broadcasts.

Continue reading →

Stage Pieces for Bensbeendead.

There’s an unspoken tenant of the maker movement that demands: calls for bespoke engineering work from friends should always be answered. Maker projects for friends pay dividends in net-happiness injected into the world.

My ‘ol pal Ben (A.K.A Bensbeendead.) knows this kind of work is a favorite of mine. So when he asked, of course I jumped at the opportunity to design and manufacture some “elbows” that mount his laptop and controller atop RGB stage lighting.

Continue reading →

Perfecting the Sliger CX4200A: Rear Exhaust Fans + Drive Shelf

In the years since the GANce days, Rosewill’s massive 4U chassis, RSV-L4000U has housed my daily-driver virtualization host. The guts of the build are almost identical to the original spec, but it has come time to move into a smaller case. CX4200A from Sliger won out because of a few factors.

The main one is that, following some flooding in my city, someone discarded a Hoffman EWMW482425 26U short-depth rack with a flawless glass door and an MSRP over over $1000:

Server rack sitting near dumpster.
Before…
Sliger CX4200A in server rack.
After! CX4200A is the big red one.

Under the cover of darkness, a co-conspirator and I were able to heist the rack from its resting place back to mine.

Judging by the bits left in inside, it looks to have served as a housing for telecom gear. Much of the rackable gear I’ve come into over the years is similarly short-depth with one exception, the Rosewill.

The 25″ of depth is way too much for the rack. It was great for being able to work on tesla cooler, plenty of room for weird coolers and hard drives. The weight, the physical mass of the thing is also just too much. I’ve moved apartments twice since acquiring the Rosewill and have dreaded moving it both times. These aren’t RSV-L4000U’s fault, these are features for the majority of users. Just not for me right now.

Enter the Sliger CX4200A and a few modifications to make it perfect for my needs:

Continue reading →

Recycling an Old Friend

There’s a triple-point energy to working on something out at the edge of your abilities. Enumerated alongside the possibilities for failure are visions of the finished piece, installed and gloriously humming along, that make the long nights ahead less intimidating.

In late October of 2018, I began the CAD for such a project. Telapush had just closed our largest deal to date. We were to contracted to build a massive illuminated sign that would map social media interactions to custom animations to be displayed in real time. The piece was to run, un-attended for the entire month of December in the Center Court of the Prudential Center mall, inside of Boston’s most distinctive skyscraper.

Erin and I had been pouring effort into Telapush for some time, and this installation was to be the largest tangible result from that effort. Serious people spending real money and expecting actual results. Boston was my new home and I felt that this project was my chance to make a good first impression to this new and intimidating place. This heightened importance, paired with the reverence for the challenge, coaxed out some good engineering; the project was a rousing success. More details about this installation can be found in my portfolio entry on the project.

Luckily, one element of the installation that I was able to retain possession of after the installation concluded was the LED matrix. An overwhelming array of 1215 addressable LEDs, and a 100W power supply to drive them.

Still brimming with usefulness, and valuable as a totem commemorating the successful project, the light bar was filed into storage. Unfortunately, I lacked the bandwidth and inspiration to pick it back up. In the past few months however, on quiet nights, I could hear it calling out to me from the crawlspace, pleading to be reanimated. And after a few years of rest, this post describes how this favorite project is given a new lease on life to enhance my 3D printing workflow.

Continue reading →