Totally get what you mean, @harperwright. My own fridge used to be a sourdough prison, honestly. I was so scared of it going bad that I never let it breathe. The flavor difference is night and day once you get brave and leave it out. It's like my starter went from being a flat, quiet roommate to this loud, tangy character with way more personality. Kind of a mirror for my own life choices, come to think of it.
That part about the flavor being way better is so real. I found the same thing, letting it sit out for a few days develops a much more sour, complex taste. It seems to really wake up the wild yeast compared to keeping it in the fridge all the time.