Its designer says the hat telegraphed conservatism, while fashion critics thought it had a more imperialistic vibe.