Switched from Google Colab to a local RTX 3090 for running Llama 3 and the speed difference is ridiculous

I was all in on cloud GPUs until my $250 monthly bill hit last month, so I grabbed a used 3090 off Facebook for $800 and now I'm doing 70B models locally in seconds. Anyone else made the jump and noticed how much latency drops?

3 comments

3 Comments

matthewh281mo ago

Is that before or after you factor in the eBay scalper markup on that 3090?

kai_bennett6d ago

Man that card’s probably been through 3 years of Doom Eternal speedruns.

sarah1981mo ago

Wait wait, $800 for a used 3090 on Facebook? I almost spit out my coffee reading that (I'm serious, it's all over my keyboard now). I was tracking prices for months and everything I saw was hovering around $1100-1200 for ones that weren't obviously mined on. You must have found some poor soul who had no idea what they were selling, that's wild.