Switched from a cloud-based LLM to a local one on my laptop

I chose between running a 7B model locally versus paying $20/month for GPT-4, and the local one actually handled my code reviews faster after I tweaked the quantization settings - anyone else ditched the cloud for a local setup?

2 comments

2 Comments

abby_henderson6d ago

A local model can't keep up with live API updates though.

hugo2366d ago

My neighbor's cat could train a better model on his laptop than half these API services running live updates right now. @abby_henderson I swear I asked Claude 3.5 a question about today's weather three hours ago and it gave me yesterday's forecast with a straight face. Maybe it's just me but at least my local model can't embarrass me in front of my boss during a meeting. Idk I'll take the occasional outdated info over paying $20 a month for weather facts from last week.