There really is no bad time to visit San Clemente. It's more so a preference of what type of activities you are looking to do and how busy you want it to be.