I've been playing Assassins Creed Brotherhood and my notoriety won't go down.?

I unlocked the Auditore cape and I put it on bht I decided to take it off because I didn't want to have to pay to fix my armor. When I took I off I realized that my notoriety meter wont go down and I'm always notorious. And it makes the game a **** load harder can someone help me bring it down!