How did you not notice the grain was fighting you until the end? Sanding against it must have felt like rubbing a cat the wrong way. At least now you know for next time, and your table is probably extra smooth!
See that's where I disagree with @the_angela. Sometimes you gotta go against the grain a little on the final passes, especially with tricky woods. Fighting it can actually smooth out those weird fuzzy spots the first direction leaves behind. Did it on my last oak piece because the grain swirled like crazy, and it came out perfect. Feels like glass now, no scratches.