Nobody really cares, and temple attendance is pretty bad anyway.
If you don't go, you don't get your magic underwear, which is a form of spiritual protection, or your endowment, which is your personal fiefdom in Mormon heaven.
If you don't believe in magic underwear and aren't a megalomaniac, there's no point in going to the temple. You can still do work on behalf of the dead. For example,
Forum:
Recovery Board