Still receiving slack metabot pulses after disabling

I have disabled the slack credentials in our Metabase instance, but still receiving pulses. Just wondering if this is expected? I’ve now deleted the token also, hoping to stop them.