- Research Article
- Open Access
- Open Peer Review
Did online publishers “get it right”? Using a naturalistic search strategy to review cognitive health promotion content on internet webpages
© The Author(s). 2017
- Received: 30 October 2016
- Accepted: 6 June 2017
- Published: 15 June 2017
One of the most common uses of the Internet is to search for health-related information. Although scientific evidence pertaining to cognitive health promotion has expanded rapidly in recent years, it is unclear how much of this information has been made available to Internet users. Thus, the purpose of our study was to assess the reliability and quality of information about cognitive health promotion encountered by typical Internet users.
To generate a list of relevant search terms employed by Internet users, we entered seed search terms in Google Trends and recorded any terms consistently used in the prior 2 years. To further approximate the behaviour of typical Internet users, we entered each term in Google and sampled the first two relevant results. This search, completed in October 2014, resulted in a sample of 86 webpages, 48 of which had content related to cognitive health promotion. An interdisciplinary team rated the information reliability and quality of these webpages using a standardized measure.
We found that information reliability and quality were moderate, on average. Just one retrieved page mentioned best practice, national recommendations, or consensus guidelines by name. Commercial content (i.e., product promotion, advertising content, or non-commercial) was associated with differences in reliability and quality, with product promoter webpages having the lowest mean reliability and quality ratings.
As efforts to communicate the association between lifestyle and cognitive health continue to expand, we offer these results as a baseline assessment of the reliability and quality of cognitive health promotion on the Internet.
Evidence is converging to suggest that individual behavioural choices may influence age-related changes in cognition. Epidemiological research suggests that among the most promising behavioural choices to support cognitive health include: managing cardiovascular risk factors; engaging in educational or other mentally stimulating activities; being physically active; developing social supports; using alcohol in moderation; avoiding use of tobacco; and eating a nourishing diet such as a Mediterranean diet [1, 2]. Since interactions between genetic factors and behavioural choices may convey different degrees of advantage to different individuals, [3, 4] fostering attention to multiple risk management and health promotion strategies is increasingly advocated [5–8]. Although there remains some hesitation about promoting such strategies on the basis of observational studies rather than randomized controlled trials, [9, 10] it is nevertheless estimated that reducing lifestyle-related risk factors by 10 to 25% in the human population has potential to prevent approximately 1 to 3 million cases of dementia worldwide .
Despite a more significant media focus on cognitive health promotion in recent years, surveys of the North American public suggest that misperceptions about the role of lifestyle in cognitive health continue. For instance, nearly half the participants in a large American survey believed that there is “nothing a person can do to help maintain a healthy memory and reduce the chance of Alzheimer's disease” . Americans are also more likely to believe that doing crossword puzzles reduces risk of Alzheimer’s disease than to believe that engagement in regular physically activity reduces risk . As the increasing prevalence of dementia continues to place pressure on health care and social systems,  gaps in public knowledge about strategies to promote cognitive health are concerning.
Social marketing campaigns have successfully promoted healthy behavioural choices to the general public for decades. For instance, the VERB™ campaign saw a 34% increase in the free-time physical activity of American pre-teens,  and the messages promoted in Canada’s long-lived ParticipACTION campaign were recognized by 85% of Canadians in 2002 . Given considerable potential for well-constructed social marketing campaigns to influence lifestyle, North American governments and special interest groups are aligning themselves with a global movement [17, 18] to disseminate available evidence about promoting cognitive health and managing risk factors for cognitive decline to the wider public. For example, the USA Centers for Disease Control and Prevention are collaborating with the American Alzheimer’s Association and other organizations on a Healthy Brain Initiative to improve population cognitive health and support those with cognitive impairment . Similarly, Alzheimer Society of Canada  has recommended government sponsorship of public health initiatives to promote lifestyle choices (e.g., engagement in physical activity and exercise, healthy diet) to reduce incident cases of Alzheimer’s disease.
Internet-based communication may be a useful component of public health campaigns and social marketing initiatives to improve brain health, since the Internet is widely used by North Americans to search for information related to health. For example, approximately 50% of Americans and nearly two thirds of Canadians use the Internet to search for this purpose, and this is one of the top uses for the Internet in Canada [20, 21]. There is also evidence that the public actively uses the Internet for health research across the adult lifespan [20, 21]. Despite these facts, reviews of the quality of Internet-based information about cognitive health are largely absent. One study reviewed the type of information provided by health care systems, public health departments, and senior centres in the United States to determine how much and what kind of information was provided about lifestyle-related strategies to promote cognitive health . Of 70 health websites identified for review, only 40% provided information about strategies to promote cognitive health. Content was variable, and only half of these websites included any information about physical activity – arguably the best-evidenced strategy .
With a focus on health care systems, public health departments, and senior centres, this review, completed by Laditka and colleagues in 2012,  illustrated the range of information available across diverse regions or communities in the USA. Laditka’s approach yielded an excellent foundation upon which to develop public health strategies that can be regionally adapted, but did not provide information about the information Internet users are most likely to be exposed to. Therefore, the objective of this study was to survey the nature of information about cognitive health that is currently available to the North American public using typical Internet search strategies and results.
Identification of Webpages
We then continued to model our webpage search after typical Internet search behaviour. For instance, since the majority of Internet users use Google as a search engine [23–26], we selected Google as the search engine for this study. Google search results are ranked based on a number of factors, including the number of times a page has been accessed, geographical location,  the number of other websites linking to a particular website, [28, 29] and other indicators of website relevance and popularity . Given that our objective was to employ a naturalistic search strategy, we accepted these features of the search engine as true to life. Nevertheless, we made one exception to avoid the possibility of confounding search results. Since personal browsing histories are also factored into Google search algorithms, we cleared the web browser’s history and conducted all searches with the browser set to “private”.
Research on Internet search behavior demonstrates that the average searcher views a single page of search results . Generally, the first hyperlink on the page of search results is likely to be clicked , and the likelihood of clicking additional results decreases to 50% for the second item, about 30% for the third, and so forth . In addition, about 50% of the time, Internet users who are searching for health information visit just one to three webpages after scanning this initial page of search results . Therefore, to generate a set of search results approximating those an average user is most likely to encounter and explore, we entered each of our search terms in Google and recorded the first two relevant hits from the first page of search results. When multiple webpages from the same website surfaced in the search, we included each distinct webpage in the search results. This Google search was completed in October 2014. Search results were initially archived as webpage hyperlinks, and were subsequently archived as PDF files at the outset of the content analysis, in January 2015.
Content analysis of Webpages
Webpage analysis was executed in three stages. First, we documented general characteristics of the webpage and corresponding website. Next, we described any content associated with cognitive health promotion or prevention. Finally, we rated the quality of the information provided on the webpage.
In order to maintain a focus on webpages rather than websites, we did not visit any hyperlinks contained within the webpages we reviewed. However, we did review the full content of web documents that were clearly intended to be read across multiple pages, as indicated by expressions such as “next”, or “page 1 of 2”.
Webpage and website characteristics
To document the general characteristics of the webpages, F.F. recorded the URL (Internet address) of each webpage, the date the webpage was accessed, its type (e.g., informational, shopping, discussion forum), the primary focus (i.e., cognitive health, dementia, or other), and the extent of advertising content (e.g., promotion of a specific product or company, advertising content only; or no advertising content). F.F. also documented the apparent involvement of health experts in creating the information provided on the page, and reference to national guidelines or recommendations about lifestyle strategies such as physical activity and nutrition. Finally, the website corresponding to the webpage was also reviewed in order to document the country of origin and ownership structure (e.g., commercial, not-for-profit government). P.H. confirmed these results.
Type of content
Next, we set out to describe whether or not each webpage contained information about promoting cognitive health, preventing risk of cognitive decline, or preventing further decline (i.e., for those with existing impairment). In order to replicate key aspects of a previous review of health system and seniors centre websites,  we also documented the presence or absence of information about particular strategies for cognitive health, such as diet or nutrition, physical activity, mental activity, use of technology, social involvement, and alternative medicine. During this stage of the analysis, P.H. and M.O. independently reviewed the content of 20% of the webpages and then evaluated their agreement level. The agreement level was 98% across all categories. Inter-rater reliability was then assessed within each relevant domain using Krippendorf’s alpha . Resulting values of .83 to 1.0 confirmed a high level of agreement within each content domain. For the few disagreements that arose, the two raters discussed the webpage content until consensus was achieved. P.H. then completed the remainder of the content review.
Quality of content
The final step in our analysis was to rate the quality of information on each webpage using a 16-item standardized rating tool designed for a mixed user group of information producers, health professionals, and patients. DISCERN  is a fifteen-item scale used to rate the quality of published health information. The scale has previously been used in studies examining the quality of health information provided on webpages [36–39]. Scale scores are correlated with other indicators of website quality and information quality [38, 40] and show good internal consistency .
Eight DISCERN items are used to rate the trustworthiness (i.e., reliability) of the information presented (e.g., relevance, clarity of sources, bias) and a further seven items are used to rate information quality. The original scale focuses on quality of information for treatment choices (e.g., how the treatment works, benefits, risks), and we modified the scale to focus on prevention strategies. For example, the item, “Does it describe what would happen if no treatment is used?” was rephrased as, “Does it describe what would happen if no prevention strategy is used?” A 16th DISCERN item asked for a global rating of publication quality based on previous responses, with responses ranging from low (implying serious shortcomings) to high (implying minimal shortcomings). All items are rated using a Likert-type scale of 1 (“No”) to 5 (“Yes”) with “partially” as the neutral option.
Four of the authors (P.H., M.D., M.O, & A.C.) formed a rating team to provide DISCERN item ratings for each webpage containing content about promoting cognitive health. To ensure that all raters were well-acquainted with DISCERN, each rater independently analyzed five webpages. The rating team then met to discuss observed discrepancies and agree on consensus ratings for these webpages. The remaining 43 webpages were divided approximately equally among the rating team such that two members of the team were assigned to review each webpage independently.
We calculated inter-rater reliability using Gwet’s AC2 [41, 42]. This statistic was recently proposed as an alternative to more commonly used agreement coefficients, such as Cohen’s Kappa and Krippendorf’s alpha [43, 44]. Designed for ordinal data, AC2 can be applied when there are more than two raters. Another helpful feature of AC2 is that disagreements are weighted based on their seriousness . The statistic is also robust in cases of high agreement, rare occurrences, or skewed data . AC2 was calculated for every pair of raters on each of the 16 DISCERN items. A quadratic weighting was chosen for the analysis of disagreements. AC2 was greater than 0.7 for 12 of the 16 DISCERN items. For the four items that did not meet this threshold, AC2 ranged from 0.58–0.64. For each of these four items, the weighted percent agreement was 88% or higher. This indicates that we achieved acceptable levels of inter-rater reliability for most items, with marginally acceptable reliability on five items.
Nevertheless, consistent with a best practice approach to rater agreement,  the rating team made an a priori decision to review items with more serious disagreement closely and use a consensus-finding approach in these cases. Therefore, once all pairs of raters had completed their work, results were reviewed item-by-item. For each pair of raters, when one member’s item rating was within a single point of the other’s, the team chose to take the average of the two scores as the consensus rating. When the item rating differed more than this, the full team reviewed the webpage together and decided on a consensus rating.
We then proceeded to calculate DISCERN’s reliability and quality subscale scores. A previously reported characteristic of DISCERN is that certain questions do not apply well to some publications . This issue affected just 1 in 1290 items in our dataset. In this instance, we calculated DISCERN subscale scores dividing the sum of ratings by the number of items completed to arrive at a prorated score .
Although content expertise is not required for DISCERN ratings,  raters were asked to respond to an additional item that did rely on content expertise (P.H., M.D., M.O, & A.C. are trained health and social scientists with respective expertise in clinical psychology; geriatric clinical neuropsychology; health product marketing; and clinical dietetics and nutrition science). The additional item, once again rated on a 1 to 5 (poor to excellent) Likert-type scale, was: What is your global rating of the quality of evidence on this page? Gwet’s AC2 was .70, suggesting acceptable agreement overall. Once again, the team reviewed discrepant scores to improve agreement. When discrepancies were within one point, the average of the two raters’ scores as the final score; otherwise, a consensus-finding approach was used.
Web page and site characteristics
Characteristics of the websites associated with a sample of 86 webpages
Number (%) of websites
Country of origin:
United States of America
Website ownership structure:
Primary focus of webpage:
Advertising content on webpage:
Promotion of a specific product or company
Mixed advertising content on page
No advertising content
Webpage indicates expert involvement:
Name, credentials, and affiliation noted
Name and credentials noted
No reference to experts
Webpage refers to consensus guidelines:
Of the 86 webpages analyzed, 4 (4.7%) were from universities, 5 (5.8%) were from government organizations, 38 (44.2%) were from not-for-profit organizations, and the remaining 39 (45.3%) were from commercial organizations. The majority of the commercial webpages (33.7%) were content providers; that is, webpages that provide information on health-related topics alongside paid advertising and sponsored content. The website webMD.com is a prime example of such a webpage. The remaining commercial webpages (11.6%) were product promoters; that is, webpages that promote products or services for a particular brand or company. For instance, included among these were webpages promoting Dr. Perlmutter’s books on the alleged benefits of grain-free diets.
Pages were most frequently formatted as information pages, and less frequently as wikis, scholarly works, news articles, or discussion forums. For additional details, see Table 1.
Type of content
Level of information about preventing cognitive decline or promoting cognitive health within a sample of 86 webpages
Web page characteristic
Any information on promoting cognitive health
Any information about preventing cognitive decline
Any information on either prevention or promotion
Any information about preventing further decline
Specific information about promoting cognitive health and/or preventing decline
Identification of specific strategies to prevent cognitive decline or promote cognitive health in a sample of 86 webpages
Diet and nutrition
Alternative medicines, herbs, medical foods, vitamins
Brain games or specific electronic interfaces
Other (e.g., smoking cessation, head protection)
Just one site (1.2%) mentioned best practice and/or consensus guidelines by name. In this case, the American Heart Association’s exercise guidelines were mentioned. Two additional sites (2.3%) made generic reference to dietary allowances or recommended servings of fruits and vegetables (see Table 1).
Quality of content
Average DISCERN score for 48 webpages
Subscale or item
Information reliability subscale
Information quality subscale
Global rating of the overall quality of the publication as a source of information about prevention strategies
Global rating of the quality of evidence provided in the publication
Top 10 webpages ranked by information reliability, information quality, and subject matter expert ratings
By information reliability
By information quality
By subject matter expert rating
Relationship between quality of content and webpage characteristics
Based on our subjective impressions of webpage content, we hypothesized statistically significant differences in the quality, reliability, and SME ratings across content provider, product promoter, and non-commercial webpages, and that these differences would favour content provider and non-commercial webpages over product promoter webpages. We used a series of one-way ANOVA to test this hypothesis.
In the first ANOVA, reliability of information was the dependent variable, and owner type (content provider, product promoter, or non-commercial) was the independent variable (factor). Reliability differed by owner type (F = 12.01, p < 0.05), and post-hoc comparisons revealed that non-commercial webpages rated significantly more reliable than content provider webpages (M non-comm = 3.6 vs. M content = 3.0, p < .05) as well as product promoter webpages (M non-comm = 3.6 vs. M product = 1.9, p < .05). There was also a significant difference in reliability between content providers and product promoters; content providers were rated as significantly more reliable than product promoters (M content = 3.0 vs. M product = 1.9, p < .05).
Similarly, a one-way ANOVA with subject-matter expert rating of evidence quality as the dependent variable and owner type as the factor showed that evidence quality differed by owner type (F = 10.8, p < .05). Non-commercial webpages were rated significantly higher than both content provider (M non-comm = 3.2 vs. M content = 2.1, p < .05) and product promoter webpages (M non-comm = 3.2 vs. M product = 1.2, p < .05). Content provider webpages, however, were not rated significantly higher than product promoter webpages (M content = 2.1 vs. M product = 1.2, p > .05).
A different pattern emerged when comparing the three types of webpages in terms of quality ratings. A one-way ANOVA with information quality as the dependent variable and owner type as the factor showed that information quality differed by owner type (F = 5.28, p < .05), but post-hoc comparisons revealed that the only significant difference in quality ratings was between non-commercial and product promoter webpages (M non-comm = 2.79 vs. M product = 1.8, p < .05). There were no significant differences between the non-commercial and content-provider webpages (M non-comm = 2.79 vs. M content = 2.27, p = .05) or between the content-provider and product promoter webpages (M content = 2.2 vs. M product = 1.8, p > .05).
This review of webpages, compiled using a naturalistic search strategy for information about strategies to promote brain health, has demonstrated that, for the most part, Internet publishers did not “get it right”. Based on scores on a standardized measure, the vast majority of cognitive health promotion webpages considered in this review did not present reliable, high quality information to readers. These findings are of critical importance since they represent the information available to most individuals who are seeking to teach themselves about cognitive health.
The range of cognitive health promotion strategies observed
A prior review of cognitive health information on the Internet focused on the content of health care system, public health department, and senior centre websites . This review found that health care systems were more likely to provide cognitive health promotion and prevention content than public health departments and seniors centres. Nutritious eating, physical activity, and mental activity were each promoted on approximately one fifth of health care system websites, and social activity was promoted on approximately 16% of websites. In the current review, the pattern of results differed, with nutritional strategies surfacing in nearly 50% of websites, mental and physical activity in about 30%, and social activity in 11%. ‘Brain games’ and dietary supplements also surfaced regularly in our review, though not in the prior one. The divergent content of health system and general Internet websites raises some questions about the preponderance of information about ‘brain games’, dietary supplements, and nutrition-related information in the general Internet search. It is plausible that health system websites provide a more cautious and balanced application of the evidence base than websites retrieved using a general Internet search.
Physical activity and exercise is the most robustly supported strategy to mitigate the impacts of cognitive aging,  reduce risk of mild cognitive impairment, and delay onset of dementia, [48, 49] particularly after mild cognitive impairment [50, 51]. In fact, it is the primary non-pharmacologic intervention for persons with dementia [52, 53]. Although many webpages we reviewed presented information on multiple strategies to support cognition, physical activity was only referenced in approximately 30% of these.
Still, we were encouraged to see that nearly half of the webpages we reviewed presented multiple strategies for promoting cognitive health and delaying decline. This trend is consistent with emerging evidence that a number of lifestyle strategies may help to slow cognitive decline [1, 2] and that genetic predispositions may confer differential advantages for some of these over others, depending on the individual [3, 4]. Given the present state of the evidence, we expect that the next several years will see Internet publishers with interests in cognitive health redouble their efforts to present multiple options for promoting cognitive health and delaying decline.
Information reliability and quality
Exercise and nutrition were the most frequently mentioned strategies for maintaining cognitive health or delaying decline. Surprisingly, although consensus and/or best practice recommendations for exercise and nutrition are in place in each of the countries from which our pool of webpages originated, [54–58] only one webpage specifically referred to such guidelines. Among the eight DISCERN items we used to assess the reliability of information presented on the webpages we reviewed, one inquired about the sources used to generate the information, another assessed whether other sources of information are offered, and a third evaluated the level of bias in the information presented. Citing available consensus guidelines would be a very useful way for health information publishers to address each of these indicators of information reliability.
Recommendations concerning diet and food surfaced in nearly half of the sampled websites, far more often than any other health promotion strategy. Although the Mediterranean diet and other high-nutrition diets show potential as a cognitive health strategy, [59–66] we found that the vast majority of the nutrition-related webpages that surfaced in our search contained inaccurate recommendations or other misinformation. This is worrisome, given that 51% of people use the Internet as a source of nutrition information, and many perceive Internet sources to be credible . Of all webpages generated in our search, pages containing a focus on diet and nutrition were most likely to contain advertising content, even when the pages were not explicitly designed to market products.
Product promoter sites and marketing ethics
Webpages designed to sell a product or promote a company (i.e., product promoters) represented approximately 11.6% of our webpage sample. Most of these product promoter sites that surfaced in our search marketed nutrition products and “brain games”. This very specific focus is in contrast with a more complex message derived from research on the efficacy and effectiveness of such products. For example, vitamin-rich diets are recommended over single vitamin supplements or nutrition products for mitigating effects of cognitive aging [58, 68]. Furthermore, although several studies show improvement in the cognitive skills directly exercised by ‘brain games’ or computerized cognitive training,  there is significant controversy as to whether these skills are generalizable to everyday life . or lasting . Put succinctly, “while data is promising and negative side effects from computer-based cognitive enhancement strategies are likely minimal, the current base of scientific evidence is insufficient to support many commercial claims of the efficacy of their computerized cognitive enhancement systems” .
As more and more people turn to the Internet for health-related information, the number of commercial webpages is not likely to diminish. One troubling aspect of our results has to do with webpages promoting a specific product or company. These pages scored consistently lower than non-commercial webpages in terms of both reliability and quality. Whereas content providers fund their operations by selling advertising space and sponsored content but provide diverse information, product promoter webpages are entirely devoted to selling one particular solution and viewpoint of a health condition. Regulators might want to be especially vigilant in monitoring product promoters’ claims. For example, the ‘brain training’ program Lumosity was marketed with claims of benefits such as improved performance in school, work, and sport, and protection against cognitive decline, all said to be evidence-based . The US Federal Trade Commission (FTC) raised charges against the parent company, Lumos Labs, alleging that these claims deceived consumers. In January 2016, as a result of the complaint, Lumos Labs received a $50 million judgment and was reminded to rely on rigorous scientific evidence to make claims about any everyday or health benefits associated with their product. Internet-based claims about Lumosity now appear to be much more objective. For instance, their homepage presently states, “we need to do more research to determine the connection between improved assessment scores and everyday tasks in participants’ lives” .
The first principle, non-malfeasance, encourages marketers to avoid knowingly doing harm. Although many of the products that surfaced in our search may not have important benefits, most are unlikely to cause harm. Nevertheless, some products that surfaced in our search, including restricted diets and dietary supplements, do have risks attached .
According to the second principle, non-deception, marketers should emphasize trust and avoid both blatant deception and misleading consumers. For instance, Lumos Labs was found to have misled consumers to increase adoption of Lumosity. Careful attention to this ethical principle could have promoted consumer interests and prevented the legal action against the company.
The third principle emphasizes protecting vulnerable segments of the market against exploitation. Generally, consumers have difficulty assessing the quality of claims associated with neuroscience . Having a health or mental health condition may contribute to additional vulnerability . For instance, older adults who have compromised mental and cognitive health also tend to have lower health literacy, which may increase risk of exploitation in health-oriented marketing .
An additional ethical principle is distributive justice, which emphasizes fair distribution of goods. Although Internet may not be regularly accessible to all North American consumers, it is accessible to the grand majority. Internet-based marketing thus enhances fairness (albeit imperfectly) by providing access to information and products designed to promote cognitive health across geographic regions and a number of social strata.
A final ethical principle for marketers is that of stewardship, which recognizes that profit motivations should not result in long-term costs to society. The principle of stewardship can be extended to include the potential social costs of propagating misinformation. For instance, it is likely that professional time is being expended to correct misinformation at public expense.
These principles overlap and intersect. For instance, as a result of the misleading use of science to promote Lumosity, some consumers may not have searched for other, more beneficial resources. Since Lumos Labs marketed their product to vulnerable clients (e.g., people who were at greater risk for cognitive impairment; people with specific health conditions),  the misleading information may have been more consequential, perhaps even violating the principle of non-malfeasance. As clinicians’ efforts to provide balanced information to potentially vulnerable patients were thwarted by overstated claims that Lumosity was backed by science, the company wasted and undermined social resources, violating the principle of social stewardship.
Are all commercial sites suspect?
Our analysis showed that product promoter webpages, one type of commercial webpage, scored lower on DISCERN’s reliability and quality subscales than content provider webpages and non-commercial webpages. Nevertheless, content provider webpages, another type of commercial webpage, did not differ from non-commercial webpages on DISCERN’s information quality subscale. Considering the high percentage of commercial webpages returned in our search, this is an important finding. While the reliability of the content provider webpages could be improved (e.g., by crediting sources), at least the quality of the information people are being exposed to while searching for health-related topics on commercial content provider webpages (e.g., WebMD.com) seems to be on par with that provided by not-for-profit organizations.
Could commercial sites be superior in any way?
A recent study examined user perceptions of website quality when searching for information on a particular health topic . The study compared hospital websites and commercial content provider websites such as WebMD, and found that users had significantly more favourable attitudes toward the commercial websites. This suggests that some characteristics of content provider websites are compelling to users.
Although we did not formally assess the readability and usability of the webpages that surfaced in our study, we noted a tendency for more comprehensive articles (e.g., Wikipedia and academic articles; see Table 5) to receive higher reliability scores, and wondered whether Internet users would expend the time and effort required to mine these lengthy articles for useful information. In keeping with this, health communications researchers have observed that higher quality content correlates with lower readability and reduced usability . Besides improving access to high-quality, reliable information about brain health, website readability and usability also deserve consistent attention if this information is to attract the attention of the average Internet user [81–85]. Moreover, it may also be important to assess whether people searching for information about cognitive health are indeed average users, and to design websites to suit the intended audience. For instance, one recent study demonstrated that older adults find websites organized around keyword tags easier to navigate than hierarchically organized websites . Since cognitive capacity is also known to interact with website usability,  website designers should also consider the needs of special populations .
Research Strengths & Limitations
This study has some important strengths and limitations. Methodological strengths of the study include employment of a naturalistic search strategy to approximate user results and behaviour, use of a standardized measure of information quality, use of a consensus rating process to generate webpage ratings, and resolution of rater discrepancies through a thorough review and discussion.
Our methods to enhance generalizability were imperfect. For instance, given that the Google search algorithm includes geography as one of hundreds of variables that influences search results, our locale may reduce the replicability of our sample of webpages. Furthermore, Internet websites are constantly being updated for currency, and webpage presence and content may change quickly over time. Similarly, search results are likely to change based on social trends. Thus, the study is best conceptualized as a snapshot in time from a particular vantage point.
Finally, to ensure a focus on cognitive health, we evaluated webpages rather than websites. Thus, we did not evaluate the overall quality of health information available on websites such as Wikipedia, WebMD.com, Mayo Clinic, or Alzheimer Society. We did, however, evaluate the quality of information about cognitive health provided on webpages associated with these sites. Since a review of the full websites is likely to lead to somewhat different conclusions about information quality or reliability, it is important that our results not be over-generalized.
This study relied on a unimodal search engine based strategy to represents health information searches on the Internet. However, it is also possible some health-conscious searchers access information from the Internet without using search engines. For instance, people may search for Smartphone applications instead of or in addition to reviewing websites, [89, 90] and these searches typically take place within an application store. In addition, researchers have begun to explore health information dissemination on social media platforms such as Facebook and Twitter . Although the most recent available research on this suggests that search engines do remain the starting point for health information searches,  given the pace of change on the Internet, this is something that should receive sustained research attention.
In ongoing research on health information searching, the interaction of time and search behaviour should also be further explored. For instance, it is possible that some health-conscious searchers come to favour sites that surface frequently in their searches, such as those listed in Table 5, and eventually begin their searches at these sites, rather than in a search engine, or refine their searches to include the name of the health website. Further, some recent research suggests that health information searches are systematic, with consistencies in general themes but variance in specific keywords or strategies seen over the span of several days . There is also some evidence that health information searching is multi-modal and includes in vivo interactions with health care providers, [93, 94] which subsequently help to determine the content of Internet searches .
Recommendations for the busy clinician
Empathy. Work to understand why patients are mentioning Internet sources, and respond in a manner that aligns with their questions and motivations.
Education. Patients may not realize that the polarized and distilled way that scientific results are communicated on the Internet differs greatly from the balanced and qualified approach adopted in reputable scientific journals . They can be taught to look for positive indicators of trustworthiness, such as reference to scientific evidence or national guidelines. They can also be taught to be more suspicious of claims made on webpages used to market products, or of scientific claims made by people with financial interest in a company . Clinicians can also identify websites they are comfortable recommending (e.g., see Table 5).
Hope. Many patients believe unsubstantiated claims because they give hope. Rather than leaning toward the extremes of confrontation or avoidance, we urge clinicians to state the facts, and foster hope by offering evidence-based alternatives such as physical activity and other healthy lifestyle choices. Clinicians can also remind a patient that since each of us has a distinct genetic heritage, if there is any “one size fits all” strategy to improve cognitive health, it is to achieve balance across a number of simple, healthy lifestyle choices, including regular exercise, a nutritious diet, and personally engaging activities [5–8]. Again, some currently available sites do this well (see Table 5).
Respect for autonomy. Even when a claim is not substantiated by research, there may be some personal value in pursuing it, such as having fun, being trendy, or expressing autonomy. Clinicians need not dissuade patients from these pursuits so long as there is no likelihood of harm. However, it may be useful to revisit these ideas over time, introducing evidence-based alternatives as enthusiasm wanes.
These results are offered as a baseline assessment of cognitive health promotion efforts on the Internet, in anticipation of national and global efforts to increase availability of information about the effects of lifestyle on cognitive health. This review of webpages, compiled using a naturalistic search strategy for information about strategies to promote brain health, has demonstrated that the vast majority of Internet publishers did not “get it right”. As a compelling example of this, only one webpage that surfaced in our search made direct reference to consensus guidelines, despite that a number of relevant guidelines are available. Additional work must be done to promote high-quality, reliable information about cognitive health on the Internet. For busy clinicians negotiating these issues with their patients, we recommend warning patients that webpages designed to promote products are more likely than other webpages to provide misleading information, and showing patients how to search for evidence-based guidelines.
Funding for research assistance was provided through a $5000 Research Grant from St. Thomas More College, University of Saskatchewan.
Availability of data and materials
The data used in this study is publicly available on the Internet, but there is a chance that these data will change over time. The compiled data analysed during the current study are available from the corresponding author on reasonable request.
PH, MO, MD, and AC have respective expertise in clinical psychology; geriatric clinical neuropsychology; health product marketing; and clinical dietetics and nutrition science; AC is also a graduate student in pharmacy and nutrition. JS, TF, and FF are graduate students in interdisciplinary studies, clinical psychology, and applied psychology. PH conceptualized the article and drafted the initial manuscript. PH, MO, MD, AC, and TF contributed to literature review. PH, MO, MD, and AC rated webpages using DISCERN and provided subjective global ratings. JS determined the search strategy and completed the webpage search as a paid research assistant under the supervision of PH. MD selected and completed all statistical analyses. PH and MD co-reviewed data and analyses for accuracy. FF reviewed the characteristics of webpages and was supervised by PH and MD. All authors edited the manuscript. PH formatted the article for publication. PH is guarantor. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Hughes TF, Ganguli M. Modifiable midlife risk factors for late-life cognitive impairment and dementia. Curr Psychiatr Rev. 2009; doi:10.2174/157340009788167347.
- Imtiaz B, Tolppanen AM, Kivipelto M, Soininen H. Future directions in Alzheimer's disease from risk factors to prevention. Biochem Pharmacol. 2014; doi:10.1016/j.bcp.2014.01.003.
- Kivipelto M, Rovio S, Ngandu T, Kåreholt I, Eskelinen M, Winblad B, et al. Apolipoprotein E ɛ4 magnifies lifestyle risks for dementia: a population-based study. J Cell Mol Med. 2008; doi:10.1111/j.1582-4934.2008.00296.x.
- Mangialasche F, Kivipelto M, Solomon A, Fratiglioni L. Dementia prevention: current epidemiological evidence and future perspective. Alzheimers Res Ther. 2012; doi:10.1186/alzrt104.
- Ngandu T, Lehtisalo J, Solomon A, Levälahti E, Ahtiluoto S, Antikainen R, et al. A 2 year multidomain intervention of diet, exercise, cognitive training, and vascular risk monitoring versus control to prevent cognitive decline in at-risk elderly people (FINGER): a randomised controlled trial. Lancet. 2015; doi:10.1016/S0140-6736(15)60461-5.
- Norton S, Matthews FE, Barnes DE, Yaffe K, Brayne C. Potential for primary prevention of Alzheimer's disease: an analysis of population-based data. Lancet Neurol. 2014; doi:10.1016/S1474-4422(14)70136-X.
- Sindi S, Mangialasche F, Kivipelto M. Advances in the prevention of Alzheimer's Disease. F1000prime Rep. 2015; doi:10.12703/P7-50.
- Vauzour D, Camprubi-Robles M, Miquel-Kergoat S, Andres-Lacueva C, Bánáti D, Barberger-Gateau P, et al. Nutrition for the ageing brain: towards evidence for an optimal diet. Ageing Res Rev. 2016; doi:10.1016/j.arr.2016.09.010.
- Patterson C, Feightner JW, Garcia A, Hsiung GYR, MacKnight C, Sadovnick AD. Diagnosis and treatment of dementia: 1. CMAJ: Risk assessment and primary prevention of Alzheimer disease; 2008. doi:10.1503/cmaj.070796.Google Scholar
- Daviglus ML, Plassman BL, Pirzada A, Bell CC, Bowen PE, Burke JR, et al. Risk factors and preventive interventions for Alzheimer disease: state of the science. Arch Neurol. 2011; doi:10.1001/archneurol.2011.100.
- Barnes DE, Yaffe K. The projected effect of risk factor reduction on Alzheimer's disease prevalence. Lancet Neurol. 2011; doi:10.1016/S1474-4422(11)70072-2.
- Connell CM, Roberts JS, McLaughlin SJ. Public opinion about Alzheimer disease among blacks, Hispanics, and whites: results from a national survey. Alzheimer Dis & Assoc Disord. 2007; doi:10.1097/WAD.0b013e3181461740.
- Anderson LN, McCaul KD, Langley LK. Common-sense beliefs about the prevention of Alzheimer's disease. Aging Ment Health. 2011; doi:10.1080/13607863.2011.569478.
- Dudgeon S. Rising tide: the impact of dementia on Canadian society: a study Alzheimer Society of Canada. 2010. http://www.alzheimer.ca/~/media/Files/national/Advocacy/ASC_Rising_Tide_Full_Report_e.pdf. Accessed 10 Jan 2015.Google Scholar
- Huhman M, Potter LD, Wong FL, Banspach SW, Duke JC, Heitzler CD. Effects of a mass media campaign to increase physical activity among children: year-1 results of the VERB campaign. Pediatr. 2005; doi:10.1542/peds.2005-0043.
- Edwards P. No country mouse: thirty years of effective marketing and health communications. Can J Public Health. 2004;95(Suppl):6–13.Google Scholar
- Shah H, Albanese E, Duggan C, Rudan I, Langa KM, Carrillo MC, et al. Research priorities to reduce the global burden of dementia by 2025. Lancet Neurol. 2016; doi:10.1016/S1474-4422(16)30235-6.
- Winblad B, Amouyel P, Andrieu S, Ballard C, Brayne C, Brodaty H, et al. Defeating Alzheimer's disease and other dementias: a priority for European science and society. Lancet Neurol. 2016; doi:10.1016/S1474-4422(16)00062-4.
- Baker S, et al. Healthy brain initiative: a national public health road map to maintaining cognitive health. Centers for Disease Control and Prevention & Alzheimer's Association. 2007; https://www.cdc.gov/aging/pdf/2013-healthy-brain-initiative.pdf. Accessed 31 July 2015.
- Amante DJ, Hogan TP, Pagoto SL, English TM, Lapane KL. Access to care and use of the internet to search for health information: results from the US national health interview survey. J Med Internet Res. 2015; doi:10.2196/jmir.4126.
- CANSIM, Table 358–0130. Statistics Canada. 2010. http://www.statcan.gc.ca/tables-tableaux/sum-som/l01/cst01/comm29a-eng.htm. Accessed 10 Jan 2015.
- Laditka JN, Laditka SB, Lowe KB. Promoting cognitive health: a web site review of health systems, public health departments, and senior centers. Am J Alz Dis Other Dementias. 2012; doi:10.1177/1533317512460564.
- Fox S. Health websites: some are more equal than others. Washington: Pew Research Centre; 2010. http://www.pewinternet.org/2010/01/21/health-websites-some-are-more-equal-than-others/. Accessed 10 Jan 2015.Google Scholar
- Halavais A. Search engine society. Cambridge: John Wiley & Sons; 2013.Google Scholar
- Sullivan D. Google still world’s most popular search engine by far, but share of unique searchers dips slightly. Redding, CT: Third Door Weekly; 2013. http://searchengineland.com/google-worlds-most-popular-search-engine-148089. Accessed 25 Mar 2017.Google Scholar
- Yu J. The psychology of search intent: converting moments that matter. London: Search Engine Watch; 2017. https://searchenginewatch.com/2017/02/15/the-psychology-of-search-intent-converting-moments-that-matter/. Accessed 25 Mar 2017.Google Scholar
- Algorithms. Google. 2015. https://www.google.com/intl/en_us/insidesearch/howsearchworks/algorithms.html. Accessed 10 Jan 2015.
- Brin S, Page L. Reprint of: the anatomy of a large-scale hypertextual web search engine. Comput Netw. 2012; doi:10.1016/j.comnet.2012.10.007.
- Split second search. Google. 2015. https://static.googleusercontent.com/media/www.google.com/en//intl/en_us/insidesearch/howsearchworks/assets/searchInfographic.pdf. Accessed 10 Jan 2015.
- Jansen BJ, Spink A. How are we searching the world wide web? A comparison of nine search engine transaction logs. Information Processing & Management. 2006; doi:10.1016/j.ipm.2004.10.007.
- Jerath K, Ma L, Park YH. Consumer click behavior at a search engine: the role of keyword popularity. J Mark Res. 2014; doi:10.1509/jmr.13.0099.
- Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ. 2002; doi:10.1136/bmj.324.7337.573.
- Morahan-Martin JM. How internet users find, evaluate, and use online health information: a cross-cultural review. CyberPsychol Beh. 2004; doi:10.1089/cpb.2004.7.497.
- Hayes AF, Krippendorff K. Answering the call for a standard reliability measure for coding data. Commun Methods Meas. 2007; doi:10.1080/19312450709336664.
- Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health. 1999; doi:10.1136/jech.53.2.105.
- Barnes C, Harvey R, Wilde A, Hadzi-Pavlovic D, Wilhelm K, Mitchell PB. Review of the quality of information on bipolar disorder on the internet. Aust N Z J Psychiatry. 2009; doi:10.1080/00048670903179137.
- Dillon WA, Prorok JC, Seitz DP. Content and quality of information provided on Canadian dementia websites. Can Geriatr J. 2013; doi:10.5770/cgj.16.40.
- Griffiths KM, Christensen H. Quality of web based information on treatment of depression: cross sectional survey. BMJ. 2000; doi:10.1136/bmj.321.7275.1511.
- Morel V, Chatton A, Cochand S, Zullino D, Khazaal Y. Quality of web-based information on bipolar disorder. J Affect Dis. 2008; doi:10.1016/j.jad.2008.01.007.
- Ademiluyi G, Rees CE, Sheard CE. Evaluating the reliability and validity of three tools to assess the quality of health information on the internet. Patient Educ Couns. 2003; doi:10.1016/S0738-3991(02)00124-6.
- Gwet KL. Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol. 2008; doi:10.1348/000711006X126600.
- Gwet KL. Handbook of inter-rater reliability: the definitive guide to measuring the extent of agreement among raters. 4th ed. Advanced Analytics: Gaithersburg; 2014.Google Scholar
- Blood E, Spratt KF. Disagreement on agreement: two alternative agreement coefficients. SAS Glob Forum. 2007;186:1–12.Google Scholar
- Lacy S, Watson BR, Riffe D, Lovejoy J. Issues and best practices in content analysis. Journal Mass Commun Q. 2015; doi:10.1177/1077699015607338.
- Krippendorff K. Reliability in content analysis: some common misconceptions and recommendations. Human Commun Res. 2004; doi:10.1093/hcr/30.3.411.
- Charnock D. The DISCERN handbook. Oxford: Radcliffe Medical Press; 1998. http://www.discern.org.uk/discern.pdf. Accessed 10 Jan 2015Google Scholar
- Angevaren M, Aufdemkampe G, Verhaar HJJ, Aleman A, Vanhees L. Physical activity and enhanced fitness to improve cognitive function in older people without known cognitive impairment. Cochrane Database Syst Rev. 2008; doi:10.1002/14651858.CD005381.pub3.
- Abbott RD, White LR, Ross GW, Masaki KH, Curb JD, Petrovitch H. Walking and dementia in physically capable elderly men. JAMA. 2004; doi:10.1001/jama.292.12.1447.
- Hamer M, Chida Y. Physical activity and risk of neurodegenerative disease: a systematic review of prospective evidence. Psychol Med. 2009; doi:10.1017/S0033291708003681.
- Lautenschlager NT, Cox K, Kurz AF. Physical activity and mild cognitive impairment and Alzheimer’s disease. Curr Neurol Neurosci Rep. 2010;10:352–8.View ArticlePubMedGoogle Scholar
- Heyn P, Abreu BC, Ottenbacher KJ. The effects of exercise training on elderly persons with cognitive impairment and dementia: a meta-analysis. Arch Phys Med Rehabil. 2004;85:1694–704.View ArticlePubMedGoogle Scholar
- Hogan DB, Bailey P, Black S, Carswell A, Chertkow H, Clarke B, et al. Diagnosis and treatment of dementia: 5. Nonpharmacologic and pharmacologic therapy for mild to moderate dementia. CMAJ. 2008;179:1019–26.View ArticlePubMedPubMed CentralGoogle Scholar
- Forbes D, Forbes SC, Blake CM, Thiessen EJ, Forbes S. Physical activity programs for persons with dementia. Cochrane Database Syst Rev. 2015; doi:10.1002/14651858.CD006489.pub4.
- Fact sheet 5: Older adults (65+ years). In UK physical activity guidelines. Chief Medical Office, London; 2011. https://www.gov.uk/government/publications/uk-physical-activity-guidelines. Accessed 10 Jan 2015.
- Australia’s Physical Activity and Sedentary Behaviour Guidelines for Adults (18–64 years). Department of Health, Sydney; 2012. http://www.health.gov.au/internet/main/publishing.nsf/content/health-pubhlth-strateg-phys-act-guidelines. Accessed 10 Jan 2015.
- Haskell WL, Lee IM, Pate RR, Powell KE, Blair SN, Franklin BA, et al. Physical activity and public health: updated recommendation for adults from the American College of Sports Medicine and the American Heart Association. Circulation. 2007; doi:10.1249/mss.0b013e3180616b27.
- Tremblay MS, Warburton DE, Janssen I, Paterson DH, Latimer AE, Rhodes RE, et al. New Canadian physical activity guidelines. Appl Physiol Nutr Metab. 2011; doi:10.1139/H11-009.
- Tucker JM, Welk GJ, Beyler NK. Physical activity in US adults: compliance with the physical activity guidelines for Americans. Am J Prev Med. 2011;40:454–61.View ArticlePubMedGoogle Scholar
- Feart C, Samieri C, Barberger-Gateau P. Mediterranean diet and cognitive health: an update of available knowledge. Curr Opin Clin Nutr Metab Care. 2015; doi:10.1097/MCO.0000000000000131.
- Morley J. Cognition and nutrition. Curr Opin Clin Nutr Metab Care. 2014; doi:10.1097/MCO.0000000000000005.
- Lourida I, Soni M, Thompson-Coon J, Purandare N, Lang IA, Ukoumunne OC, et al. Mediterranean diet, cognitive function, and dementia: a systematic review. Epidemiology. 2013; doi:10.1097/EDE.0b013e3182944410.
- Wu L, Sun D. Adherence to Mediterranean diet and risk of developing cognitive disorders: An updated systematic review and meta-analysis of prospective cohort studies. Sci Rep. 2017; http://0-dx.doi.org.brum.beds.ac.uk/10.1038/srep41317.
- Dominguez LJ, Barbagallo M. The relevance of nutrition for the concept of cognitive frailty. Curr Opin Clin Nutr Metab Care. 2017; doi:10.1097/MCO.0000000000000337.
- Hardman RJ, Kennedy G, Macpherson H, Scholey AB, Pipingas A. Adherence to a Mediterranean-style diet and effects on cognition in adults: a qualitative evaluation and systematic review of longitudinal and prospective trials. Front Nutr. 2016; doi:10.3389/fnut.2016.00022.
- Tucker KL. Nutrient intake, nutritional status, and cognitive function with aging. Ann N Y Acad Sci. 2016; doi:10.1111/nyas.13062.
- Shlisky J, Bloom DE, Beaudreault AR, Tucker KL, Keller HH, Freund-Levi Y, et al. Nutritional considerations for healthy aging and reduction in age-related chronic disease. Adv Nutr. 2017; doi:10.3945/an.116.013474.
- Goodman S, Hammond D, Pillo-Blocka F, Glanville T, Jenkins R. Use of nutritional information in Canada: national trends between 2004 and 2008. J Nutr Educ Behav. 2011; doi:10.1016/j.jneb.2011.02.008.
- Shah R. The role of nutrition and diet in Alzheimer disease: a systematic review. JAMDA. 2013; doi:10.1016/j.jamda.2013;01.014.
- Kueider AM, Parisi JM, Gross AL, Rebok GW. Computerized cognitive training with older adults: a systematic review. PLoS One. 2012; doi:10.1371/journal.pone.0040588.
- Green CS, Bavelier D. Exercising your brain: a review of human brain plasticity and training-induced learning. Psychol Aging. 2008; doi:10.1037/a0014345.
- Reijnders J, van Heugten C, van Boxtel M. Cognitive interventions in healthy older adults and people with mild cognitive impairment: a systematic review. Ageing Res Rev. 2013; doi:10.1016/j.arr.2012.07.003.
- Jak AJ, Seelye AM, Jurick SM. Crosswords to computers: a critical review of popular approaches to cognitive enhancement. Neuropsychol Rev. 2013; doi:10.1007/s11065-013-9226-5.
- Lumosity to pay $2 million to settle FTC deceptive advertising charges for its “Brain Training” program. Federal Trade Commission; 2016. https://www.ftc.gov/news-events/press-releases/2016/01/lumosity-pay-2-million-settle-ftc-deceptive-advertising-charges. Accessed 27 Mar 2017.
- Our scientists research the efficacy of Lumosity. Lumosity, 2017. http://www.lumosity.com. Accessed 27 Mar 2017.
- Laczniak GR, Murphy PE. Normative perspectives for ethical and socially responsible marketing. J Macromarketing. 2006; doi:10.1177/0276146706290924.
- Statement of Ethics. American Marketing Association; 2017. https://www.ama.org/AboutAMA/Pages/Statement-of-Ethics.aspx. Accessed 27 Mar 2017.
- Chancellor B, Chatterjee A. Brain branding: when neuroscience and commerce collide. AJOB Neurosci. 2011; doi:10.1080/21507740.2011.611123.
- Chesser AK, Keene Woods N, Smothers K, Rogers N. Health literacy and older adults: a systematic review. Gerontol Geriatr Med. 2016; doi:10.1177/2333721416630492.
- Cudmore BA, Bobrowski PE, Kiguradze T. Encouraging consumer searching behavior on healthcare web sites. J Consum Mark. 2011; doi:10.1108/07363761111143187.
- Sutherland LA, Wildemuth B, Campbell MK, Haines PS. Unraveling the web: an evaluation of the content quality, usability, and readability of nutrition web sites. J Nutr Educ Behav. 2005; doi:10.1016/S1499-4046(06)60160-7.
- Cline RJ, Haynes KM. Consumer health information seeking on the internet: the state of the art. Health Educ Res. 2001; doi:10.1093/her/16.6.671.
- Nielsen J. Top ten mistakes in web design. Neilson Norman Group. 1996. http://nngroup.com Accessed 28 Dec 2015.
- Nielsen J. Ten good deeds in web design. Neilson Norman Group 1999. http://nngroup.com. Accessed 28 Dec 2015.
- Zhang P, Von Dran GM. Satisfiers and dissatisfiers: a two-factor model for website design and evaluation. J Am Soc Inf Sci. 2000; doi:10.1002/1097-4571(2000)9999:9999%3C::AID-ASI1039%3E3.0.CO;2-O.
- Zhang Y, Sun Y, Xie B. Quality of health information for consumers on the web: a systematic review of indicators, criteria, tools, and evaluation results. J Assoc Inf Sci Technol. 2015; doi:10.1002/asi.23311.
- Pak R, Price M, Thatcher J. Age-sensitive design of online health information: comparative usability study. J Med Internet Res. 2009; doi:10.2196/jmir.1220.
- Czaja SJ, Sharit J, Lee CC, Nair SN, Hernández MA, Arana N, et al. Factors influencing use of an e-health website in a community sample of older adults. J Am Med Inform Assoc. 2013; doi:10.1136/amiajnl-2012-000876.
- Rotondi AJ, Sinkule J, Haas GL, Spring MB, Litschge CM, Newhill CE, et al. Designing websites for persons with cognitive deficits: design and usability of a psychoeducational intervention for persons with severe mental illness. Psychol Serv. 2007; doi:10.1037/1541-15126.96.36.199.
- Firth J, Torous J. Smartphone apps for schizophrenia: a systematic review. JMIR Mhealth Uhealth. 2015; doi:10.2196/mhealth.4930.
- Pandey A, Hasan S, Dubey D, Sarangi S. Smartphone apps as a source of cancer information: changing trends in health information-seeking behavior. J Cancer Educ. 2013; doi:10.1007/s13187-012-0446-9.
- Robillard JM, Johnson TW, Hennessey C, Beattie BL, Illes J. Aging 2.0: health information about dementia on twitter. PLoS One. 2013; doi:10.1371/journal.pone.0069861.
- White RW, Horvitz E. From health search to healthcare: explorations of intention and utilization via query logs and user surveys. J Am Med Inform Assoc. 2014; doi:10.1136/amiajnl-2012-001473.
- Diviani N, Van den Putte B, Meppelink CS, van Weert JC. Exploring the role of health literacy in the evaluation of online health information: insights from a mixed-methods study. Patient Educ Couns. 2016; doi:10.1016/j.pec.2016.01.007.
- Cotten SR, Gupta SS. Characteristics of online and offline health information seekers and factors that discriminate between them. Soc Sci Med. 2004; doi:10.1016/j.socscimed.2004.02.020.
- Kutner M, Greenberg E, Jin Y, Paulsen C. The health literacy of America's adults: results from the 2003 National Assessment of adult literacy [NCES 2006–483]. US Department of Education: Washington; 2006. http://files.eric.ed.gov/fulltext/ED493284.pdf. Accessed 18 July 2016.Google Scholar