In order to get clean in the shower, many people use loofah. It helps the skin become soft. The puff hangs in the shower on a hook or faucet, waiting to be used. Moreover, they boost circulation when the skin is scrubbed and the dead skin cells are sloughed off.
If you don’t want to part with it, then you should better rinse it off after you use it on your skin and take it away from the shower so that it can dry. Replace the puff with a new one every couple of months in order to make sure nothing hides inside. Smell it and if it is unpleasant, then you should get rid of it. It probably comes from the bacteria. Just buy a new one.