Milwaukee residents are saying they weren't kept in the loop about plans to move an ICE facility to the Northwest Side.