Hard to give a one size fits all answer to your question. You need to think in terms of the risks and what you would be prepared and capable to do in order to deal with them. Some of the risks that come to mind are: climbing down in the dark (headlamps are always a good thing to carry), being too tired or not knowing the trail well enough to make it back down to camp (carry enough gear/food/water to survive the night out), running out of water or food (carry extra), late afternoon thiunderstorms (may need to turn around or get off the mountian fast). I generally figure half the time going down as it takes going up, so at 4 pm turnaround would mean about 3 1/2 hours of good light. If it took you about 7 hours to climb (at the the most difficult parts of) the trail going up, then 4 pm would be OK. I would tend to make it more like 3 pm if I were climbing with someone who's abilities were uncertain.
