The distance-dependent interaction of an emitter with a plasmonic nanoparticle or surface forms the basis for the application of such systems within optoelectronics. Semiconductor quantum dots (QDs) are robust emitters due to their photostability. A key challenge is the formation of well-defined assemblies containing QDs and plasmonic nanoparticles in high purity. Here, we present the translation of DNA-based self-assembly to assemble metal and semiconductor nanocrystals into hybrid structures. The high purity of the assemblies, including dimers and higher-order core-satellite structures, allows fundamental investigation of the plasmon-exciton interaction. In contrast to the increase in the QD emission rate and enhancement in steady-state photoluminescence observed for overlap between the QD emission and localized surface plasmon resonance, significant detuning of these energies leads to lengthening of the QD emission lifetime (reduction of the emission rate) up to 1.7-fold and enhancement in steady-state photoluminescence of 15-75%. These results are understood in terms of the Purcell effect, where the gold nanoparticle acts as a damped, nanoscale cavity. We show that the response is driven by the interference experienced by the emitter for parallel and perpendicular field orientations. This provides a mechanism for control of the emission rate of a QD by a metal nanoparticle and a much wider range of lifetimes than previously understood.