The optically active Nitrogen-Vacancy (NV) defect in diamond holds substantial promise as a potential single photon source for quantum cryptography. Here we report enhanced spectral properties of an NV centre through a high-quality single-moded cavity coupled to an optical fibre. We use the secure bit rate of the BB84 quantum key distribution protocol as our figure of merit to demonstrate the significant improvements from employing state of the art cavities