Some of the moments we consider highly important really do not last that long when you think about it. When going to the gym, if you are doing 3 sets of 5 for 3 different lifts, each set probably lasts about 1 minute each. That makes 9 minutes of time spent going hard. 9 minutes is not a lot of time, in the grand scheme of the day, but those are 9 very important minutes.
Yes, you need to devote time for rest between sets too, but I hope my point makes sense.