I agree, it's frustrating that even when the totem isn't the target, its natural timer will reset mobs. It can't be intended and for sure it feels awful.