Some of the moments we consider highly important really do not last that long when you think about it. When going to the gym, if you are doing 3 sets of 5 for 3 different lifts, each set probably lasts about 1 minute each. That makes 9 minutes of time spent going hard. 9 minutes is not a lot of time, in the grand scheme of the day, but those are 9 very important minutes.
Yes, you need to devote time for rest between sets too, but I hope my point makes sense.
And contrary to the impression many people want to give about sex (namely their own performance and how long they can go at it), sex probably does not last all that long for most people. Even the longest outliers, the act itself does not last that long when you consider the grand total of your life. Yet again, many people make a huge deal out of it.
Often, it is not the quantity of time to consider, but the quality.
Alnost Everyone has time to exercise 10 seconds per day
Someone could run at top sprinting speed for 10 seconds every day and get very strong fast twitch leg muscles
It is not worth it to devote your life to the point of becoming a slave in order to have relations with women for a few minutes
Or think about most competitions, be it sports, gaming, music, or any other activity you can think of. The competition itself is just minutes of time out of the day (when factoring in only the moments where some activity is happening). Yet people prepare a crazy amount of time just for the few minutes of competition.
Once more, we see significance in the quality, not quantity, in that time spent.