That's right - you only use hyphens when the phrase is used as an adjective. For example: Marcus is a down-to-earth guy. He likes to get straight to the point.