Habitat fragmentation can increase the chance of population bottlenecks and inbreeding, and may ultimately lead to reduced fitness and local extinction. Notelaea lloydii is a native olive species endemic to Australia and listed as vulnerable due to its restricted distribution. A recent molecular systematics study has revealed there might be some geographic structuring among the N. lloydii populations. Therefore, we undertook a genome wide Single Nucleotide Polymorphism (SNP) analysis to determine levels and patterns of genetic diversity, inbreeding and gene flow within and among N. lloydii populations in south-eastern Queensland (SE-QLD). Furthermore, as the reproductive phase of a plant’s life history has a profound influence on genetic diversity, life history reproductive traits were also studied. Our SNP analysis revealed low genetic diversity, inbreeding and significant genetic structuring even among proximate populations. Results of a flower and fruit bagging experiment in two consecutive seasons revealed that N. lloydii produced many flowers but only a few fruits survived to maturity. There were no differences in bagged and un-bagged flowering and fruiting rates and therefore, we conclude that the high fruit abortion rate was probably due to inbreeding depression and/or suboptimal conditions, rather than pollinator availability and insect attack. Overall, results of this study indicate that the populations of N. lloydii are small, inbred and genetically isolated and represent unique management units that require local conservation management due to ongoing threats associated with urbanisation.