Scaling “E-Waste”: LLMs on Multiple Decommissioned GPUs

The critical finding last time was that: in this world of decommissioned enterprise hardware, three and four GPU setups are counter-intuitively durable against diminishing returns. Within a homelab GPU appliance, you can just keep adding GPUs to proportionally scale performance. Given the low cost of older ~16GB GPUs, this puts LLMs larger than the ~10B parameter single GPU models within reach for a local daily driver.

To better understand how multiple GPUs handle diverse workloads, I’ve expanded our benchmarking tool with over 130 new tests tailored for LLMs. Following requests from commenters, I also integrated power consumption monitoring to compare efficiency across different builds. Detailed results and analysis to follow.

To reiterate: these cards are cheap because they run hot and ship without cooling, a non-starter for the typical secondary market customer. The project underwriting all this research has been my standalone active cooler, which is almost ready for release. If you’re interested in helping push it over the line by stress testing a sample, I’ll be recruiting beta testers using the mailing list.

Continue reading →

Benchmarking 15 “E-Waste” GPUs with Modern Workloads

Decommissioned NVIDIA enterprise GPUs are one of the last remaining sources of idle VRAM. K80 with 24GB of GDDR5 sells for $60, P100-16GB for around $75 and V100-16GB for under $200. Shortcomings and all, it’s important to understand if we can utilize these widely available cards in the modern era. This project to benchmark tesla GPUs has been in the works for almost a year and over the winter I was able to spend many kilowatt hours heating up my studio with GPUs.

 

Continue reading →

Introducing a GPU Server Benchmark for Tesla GPUs

Previously a plan was set in motion to benchmark my collection of mighty Tesla GPUs. The set of cooler manifolds are designed, a GPU server benchmark suite has been created, and the time has come to start working through the spreadsheet. I have long suspected that the older multi node cards could be fantastic for image processing. Finally we can quantify how much life is left in these older cards.

Continue reading →

First look at the New GPU Cooler Prototype

For the past while I’ve been working on a major redesign of my high performance gpu cooler project.

The rapid ascent of the LLM into the collective consciousness has sent the big players into a frenzy over datacenter GPUs. This is putting accelerating downward pressure on the price of all used compute GPUs, even the historically pricey stuff. P100s can be had for ~$100, V100 16GB are selling for ~$500, any day now the lower VRAM Ampere cards are going to drop below $1000…

We need a new cooler that can support any of these 300W GPUs, can be installed as densely as the cards, and is easy on the ears for homelab use. Pictured below are four prototype coolers installed on Tesla P100s:

Continue reading →

1kW of GPUs on the OpenBenchTable

In the conclusion of my previous homelab post, I pled to the eBay gods begging for a 4xP100 system. My prayers were heard, possibly by a malevolent spirit as a V100 16GB for $400 surfaced. More money than I’d be willing to spend on a P100 but the cheapest I’d ever seen a V100, I fell to temptation. To use all four cards, I needed something bigger than the Rosewill RSV-R4100U. Enter the OpenBenchTable, and some 3d printed parts I designed to be able to securely mount four compute GPUs:

Continue reading →

Hardware for Engineering Streams

My beloved blog post still sits on the throne as the most effective format for engineering projects. To me, Inlining code, photographs, CAD models and schematics in an interactive way trumps other mediums. This level of interactivity closes the gap between reader and material, allowing an independent relationship with the subject outside of the story being told by the author.

Working on stream to an online audience has a similar effect, the unedited and interactive format yielding a real understanding of the creators process and technique.

For a while there, I’d settled into a nice habit of broadcasting project development live on twitch. Two moves later, things have settled down enough in my personal life that I feel it’s time to try to get back into this habit.

Before we get started again, I took some time to improve the ergonomics (both hardware and software) of my stream setup. The following documents a few smaller projects, all in service of these upcoming broadcasts.

Continue reading →

Perfecting the Sliger CX4200A: Rear Exhaust Fans + Drive Shelf

In the years since the GANce days, Rosewill’s massive 4U chassis, RSV-L4000U has housed my daily-driver virtualization host. The guts of the build are almost identical to the original spec, but it has come time to move into a smaller case. CX4200A from Sliger won out because of a few factors.

The main one is that, following some flooding in my city, someone discarded a Hoffman EWMW482425 26U short-depth rack with a flawless glass door and an MSRP over over $1000:

Server rack sitting near dumpster.
Before…
Sliger CX4200A in server rack.
After! CX4200A is the big red one.

Under the cover of darkness, a co-conspirator and I were able to heist the rack from its resting place back to mine.

Judging by the bits left in inside, it looks to have served as a housing for telecom gear. Much of the rackable gear I’ve come into over the years is similarly short-depth with one exception, the Rosewill.

The 25″ of depth is way too much for the rack. It was great for being able to work on tesla cooler, plenty of room for weird coolers and hard drives. The weight, the physical mass of the thing is also just too much. I’ve moved apartments twice since acquiring the Rosewill and have dreaded moving it both times. These aren’t RSV-L4000U’s fault, these are features for the majority of users. Just not for me right now.

Enter the Sliger CX4200A and a few modifications to make it perfect for my needs:

Continue reading →

The Maker Stack (Self-Hosted Server Configuration)

There are many maker/hackers out there like me that operate little blogs just like this one and would like to expand but spend absolutely no cash. This post is for that kind of person.

This is what my network looks like now for the diagram oriented:

Basically, this configuration allows me to host two websites (they happen to both be wordpress installations) with different url’s out of the same server on the same local network, sharing the same global IP address as well as host email accounts across all of the domains I own.

The backbone of this whole system is virtualbox controlled by phpvirtualbox. This is a preference thing. You could install each of these components on the same server but virtual machines are an easy way to keep things conceptually simple. All of the traffic from the web is ingested through a reverse proxy on a server running ngnix. It identifies where the user would like to end up at (using the url) and directs them to the proper hardware on the network.


Installation

I have done detailed posts on each part of this installation, I’ll glue it all together here.

  • First thing’s first, everything runs out of Ubuntu, particularly Ubuntu 12.04.3 LTS. To do any of this you will need a computer capable of running Ubuntu, this is my hardware configuration. To install Ubuntu, the official installation guide is a good place to start, if you have any trouble with it leave a comment.
  • Once you have Ubuntu, install virtualbox to host the virtual machines, and phpvirtualbox to headlessly (no need for a monitor or mouse and keyboard) control them. Instructions here.
  • Next you need to install Ubuntu inside of virtualbox on a virtual machine. Navigate to your installation of phpvirtualbox and click new in the top left.

  • In order to get our new virtual machine on the internet, we must bridge the virtual adapter in the vm with the physical one. This is very easy to do. Click the vm on the left, and then go into settings then into network. Set “Attached to:” to Bridged Adapter.

  •  Once Ubuntu is installed on your new virtual machine inside of phpvirtualbox running on your Ubuntu server (mouthful!), to make the whole thing work, we must install and configure a nginx as a reverse proxy server. Say a project of yours deserves its own website, since your already hosting a website out of your residential connection, you would have to pay to host somewhere else as well right? Wrong. I have written this guide to do this. Once this installation is done. Make sure that you assign a static IP address to this server (as well as all other VM’s you create) and forward your router’s port 80 to the nginx server. The port forwarding is specific to the router, if you have no clue how to do it, google “port forward nameofrouter”.
  • You will then have to point the DNS server with your Domain Name Registrar to your router’s global IP address. Obtaining this IP address is easy.

And the foundation is set! Now that you know how to install a virtual machine and you have a nginx reverse proxy up and running, you should point the proxy to things!

In my configuration, I point it at two different  I use this routine to do wordpress installations all the time. On my server, I run two VM’s with two wordpress installs. One of them is for this blog, and the other for another website of mine, www.blockthewind.com.

To get a simple email server up and running, follow this guide which goes a little more in depth on phpvirtualbox but results in a citadel email server. I decided to go with citadel because of how easy the installation was and how configurable it was through the GUI. I use email accounts hosted with citadel for addresses that I would use either once or infrequently. It’s free to make these addresses, but citadel is older and probably not as secure as it could be for highly sensitive data.


That’s it! Do you have any suggestions as to what every small-scale tech blogger should have on their server?

Thanks for reading!

Host Multiple Webservers Out Of One IP Address (reverse proxy) Using Ngnix

It’s easy enough to host a single website out of a residential internet connection. All you have to do is open up port 80 on your router and bind it to the local IP address of your server as follows:

But say you’re like me and have multiple domains and therefor want want to host content for mutliple domains on the same IP address like this:

Say you want to further complicate things further and have unique physical computers for hosting each site. The quickest and easiest way to do this (so I’ve found) is using a Ngnix Reverse Proxy.


The topology for accomplishing this looks a lot like this:

The ONLY things we need to deal with in this diagram is the Ngnix Server and the router. For my setup it is a virtual machine running Ubuntu 10.04 x64. Setting up the software is pretty simple as well. First we need to install Ngnix:

After that we need to add the configuration files for each server. The procedure for doing that is as follows. You need to do this for EACH server you add to the reverse proxy.

For this example I’ll be using example.com as the domain and xxx.xxx.x.xxx as the IP address on the local network of the machine I’d like to host example.com on.

Create the config with:

The create then fill it in:

Note the line

This limits the file size that can be transferred through the reverse proxy to the target server. If you are transferring larger files, you will need to increase this value, but 64M is more than enough for most applications.

From there, you need to “activate” the new redirect by symbolically linking it with a config in the enabled sites directory in Ngnix with:

Restart ngnix and we’re done!


Now to configure the router.
It’s pretty easy, all you need to do is forward port 80 on the router to the local IP address of the Ngnix server. On my router that looks like this

Where 192.168.1.217 is xxx.xxx.x.xxx in my example.

Thanks for reading and if you have any questions leave them in the comments.

Site has been down

You’re currently reading this via a virtual server running on top of this server, which is hosted out of my residence in southern Maine. For the most part, I generally enjoy the challenges and obstacles that come with self-hosting this website but it becomes very very annoying in cases where there is physically nothing I can do. Take these past two days for example.

Storm!

Sometimes it snows a foot in an hour and the power goes out for a few days. Not much I can do about it, sorry for the inconvenience. If it’s any consultation, Twitter isn’t hosted out of their parent’s house and you can follow me there where I typically post if my site is up or down.

Thanks for reading!