You can manually remove files in the Package IP window, but this is not recommended as Package IP window becomes out-of-sync with the associated Vivado project.
To manually remove a file from a file group, in the File Groups page, right-click the respective file and click Remove File.