My workflow's been to temporary disable that, do the stuff I need, then re-enable it. It's a bit clunky, but I don't install stuff outside a virtualenv frequently enough for it to be a major pain in the neck.
Yeah I use pyenv + virtualenvwrapper and there are a few packages I am fine with having in the top-level pyenv version, rather than in any particular virtual environment: black, requests, click, etc.