People with cancer or those who have beaten the disease are 40% more likely to suffer memory problems than those who have never had the disease, researchers said.
It is thought memory problems could be caused by the effects of treatment, such as chemotherapy, radiotherapy and hormone therapies.
An alternative theory connects the brain functioning issues to the way the cancer tumours grow and behave, which could change brain chemistry.
Research presented at the American Association for Cancer Research meeting in Denver focused on almost 10,000 people from a wide range of backgrounds taking part in the National Health and Nutrition Examination Survey.
Memory problems were reported more often by people who currently had cancer (14%) than by those who did not (8%).
People who had suffered cancer in the past also had around a 50% higher chance of suffering memory loss.
The researchers, led by a team at the University of Miami Miller School of Medicine, concluded: "Overall, participants with cancer had a 40% greater likelihood of having memory problems that interfered with daily functioning."