I think you could do it with an early enough start and if the weather cooperated.
I did it as a 3 day hike last July. Up to Trail Camp. Summit and Trail Camp and then out the third day.
I started hiking around 7am. Was at Trail Camp around noon. I spent the rest of that day reading and talking to other folks. I guess if I had been inclined I might have done it as you suggest. Altho, I also got a bit of queasiness between Mirror Lake and Trail Camp - might have been "lunch" at 10am of cheese and dried fruit at Outpost Camp.
But then again, summitting the second day, I was able to spend a good two hours up there - just taking it all in (my third time).
And I again spent the rest of that day, lounging around, hiking out the 3rd day. I found that 3rd day tedious, particularly those last 2 or 3 miles.