***The consensus is usually that the
expanding universe is like a polka dotted baloon being blown up. Then
they ignore the fact that when the baloon starts out,
it is coming outward from a central point.

The problem of the "central point" or 'primal radiant' and absence
thereof was discussed in depth in another thread just a couple of days
ago.
The sphere of our visible cosmos or 'known universe'
is what decoupled from the 'Bang' point and moved some distance away
from it. Our sphere of visibility (SoV) is limited in size due to the
finite speed of light since the BB (the 'Horizon Problem'). But the
expansion speed *of space itself* knows no limitation, thus our SoV got
carried a considerable distance during the initial expansion (Guth's
"inflation").
Analogies like the "balloon" or "raisin bread" are
flawed in that they depict the polka dots or "raisins" as galaxies (or
galaxy clusters) moving away from a central point of origin ('primal
radiant'). In order for these analogies to work correctly, the "raisins"
or polka dots would have to represent entire SoVs having moved away from
the 'Bang' point, since the 'Bang' point IS the primal radiant of the
universe. So naturally there is no primal radiant *inside* our SoV, and
the illusion is that the BB occured "everywhere at once".
And there is an infinite number of SoVs, because
anywhere you're at in the universe, you're automatically at the center
of your own SoV.. with no central point of origin.