Why Wasn't ISS Built Sooner?

Since a major point of the shuttle was to construct a
space station why didn't construction start with like the
third or fourth shuttle launch in the 1980s? Why didn't
more people question the absence of any serious station
plan then? I recall seeing lots of newspaper articles
in the late 1970s depicting how the fancy new space
station might look and there was a lot of talk about
that sort of thing but it all faded away sometime BEFORE
the first shuttle launch replaced with an emphasis on
flying habit modules and labs in the shuttle cargo bay.
Or am I remembering wrong?

-McDaniel