I do x-ray cameras, never visible light optics but here I am, trying to change a collimated detector array into a lensed array. I am such an idiot about this stuff I barely know where to look to find the answer I am questing after, so I came here. I bet someone knows the answer off-hand...
Can you heat up the part which the plug is pressing into? We perform shrink fits all the time with aluminum, and your interference range of 0.001 ~ 0.003 is right in the reasonable value for a shrink fit. If you could heat up the 'hole' to be roughly 240 deg. F hotter than the plug, you will...