Bar charts and user toggling certain bars

I’m designing a bar chart for a dashboard. The bar chart will display various categories of items and display their inventory counts. In a separate UI (one that only high-level managers have access to), the manager can enable or disable some of these categories. So a bar chart with a total of 12 bars, may only be configured to show, say 9 (in the case where the manager disabled 3 of the categories).

My assumption is that I should display the 9 bars in exactly the same spot they would be if there were all 12, hide the 3 bars that the manager turned off, but do not condense the bar chart because that would mean some bars are moving around and could be confusing to the user if they are used to seeing a particular category in a particular spot. I am also thinking of graying out the categories that are disabled so they can still see the text, and above it would be blank space.

Am I wrong here? Should the entire table be condensed instead of leaving blank space for phantom bars? Would moving around a position of a bar be confusing for the user?