An obvious increase is witnessed in the incidence of type 2 diabetes mellitus (T2DM) in the elderly with changes in people′s lifestyles and the aging of the population. Compared with non-diabetic elderly patients, the risk of cognitive decline increased by 1.5 times in the elderly T2DM patients. Studies have shown that T2DM can affect cognitive function in the elderly, and increasing attention has also been drawn to the relationship between persistent hyperglycemia and cognitive impairment in the elderly with T2DM. However, few studies have focused on the relationship between blood glucose fluctuation and cognitive impairment and its mechanism in those patients. This article reviews the potential mechanism of glucose fluctuation and cognitive impairment in elderly T2DM patients.