what is theoretical empirical literature review

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Module 2 Chapter 3: What is Empirical Literature & Where can it be Found?

In Module 1, you read about the problem of pseudoscience. Here, we revisit the issue in addressing how to locate and assess scientific or empirical literature . In this chapter you will read about:

• distinguishing between what IS and IS NOT empirical literature
• how and where to locate empirical literature for understanding diverse populations, social work problems, and social phenomena.

Probably the most important take-home lesson from this chapter is that one source is not sufficient to being well-informed on a topic. It is important to locate multiple sources of information and to critically appraise the points of convergence and divergence in the information acquired from different sources. This is especially true in emerging and poorly understood topics, as well as in answering complex questions.

What Is Empirical Literature

Social workers often need to locate valid, reliable information concerning the dimensions of a population group or subgroup, a social work problem, or social phenomenon. They might also seek information about the way specific problems or resources are distributed among the populations encountered in professional practice. Or, social workers might be interested in finding out about the way that certain people experience an event or phenomenon. Empirical literature resources may provide answers to many of these types of social work questions. In addition, resources containing data regarding social indicators may also prove helpful. Social indicators are the “facts and figures” statistics that describe the social, economic, and psychological factors that have an impact on the well-being of a community or other population group.The United Nations (UN) and the World Health Organization (WHO) are examples of organizations that monitor social indicators at a global level: dimensions of population trends (size, composition, growth/loss), health status (physical, mental, behavioral, life expectancy, maternal and infant mortality, fertility/child-bearing, and diseases like HIV/AIDS), housing and quality of sanitation (water supply, waste disposal), education and literacy, and work/income/unemployment/economics, for example.

Three characteristics stand out in empirical literature compared to other types of information available on a topic of interest: systematic observation and methodology, objectivity, and transparency/replicability/reproducibility. Let’s look a little more closely at these three features.

Systematic Observation and Methodology. The hallmark of empiricism is “repeated or reinforced observation of the facts or phenomena” (Holosko, 2006, p. 6). In empirical literature, established research methodologies and procedures are systematically applied to answer the questions of interest.

Objectivity. Gathering “facts,” whatever they may be, drives the search for empirical evidence (Holosko, 2006). Authors of empirical literature are expected to report the facts as observed, whether or not these facts support the investigators’ original hypotheses. Research integrity demands that the information be provided in an objective manner, reducing sources of investigator bias to the greatest possible extent.

Transparency and Replicability/Reproducibility.   Empirical literature is reported in such a manner that other investigators understand precisely what was done and what was found in a particular research study—to the extent that they could replicate the study to determine whether the findings are reproduced when repeated. The outcomes of an original and replication study may differ, but a reader could easily interpret the methods and procedures leading to each study’s findings.

What is NOT Empirical Literature

By now, it is probably obvious to you that literature based on “evidence” that is not developed in a systematic, objective, transparent manner is not empirical literature. On one hand, non-empirical types of professional literature may have great significance to social workers. For example, social work scholars may produce articles that are clearly identified as describing a new intervention or program without evaluative evidence, critiquing a policy or practice, or offering a tentative, untested theory about a phenomenon. These resources are useful in educating ourselves about possible issues or concerns. But, even if they are informed by evidence, they are not empirical literature. Here is a list of several sources of information that do not meet the standard of being called empirical literature:

• your course instructor’s lectures
• political statements
• advertisements
• newspapers & magazines (journalism)
• television news reports & analyses (journalism)
• many websites, Facebook postings, Twitter tweets, and blog postings
• the introductory literature review in an empirical article

You may be surprised to see the last two included in this list. Like the other sources of information listed, these sources also might lead you to look for evidence. But, they are not themselves sources of evidence. They may summarize existing evidence, but in the process of summarizing (like your instructor’s lectures), information is transformed, modified, reduced, condensed, and otherwise manipulated in such a manner that you may not see the entire, objective story. These are called secondary sources, as opposed to the original, primary source of evidence. In relying solely on secondary sources, you sacrifice your own critical appraisal and thinking about the original work—you are “buying” someone else’s interpretation and opinion about the original work, rather than developing your own interpretation and opinion. What if they got it wrong? How would you know if you did not examine the primary source for yourself? Consider the following as an example of “getting it wrong” being perpetuated.

Example: Bullying and School Shootings . One result of the heavily publicized April 1999 school shooting incident at Columbine High School (Colorado), was a heavy emphasis placed on bullying as a causal factor in these incidents (Mears, Moon, & Thielo, 2017), “creating a powerful master narrative about school shootings” (Raitanen, Sandberg, & Oksanen, 2017, p. 3). Naturally, with an identified cause, a great deal of effort was devoted to anti-bullying campaigns and interventions for enhancing resilience among youth who experience bullying.  However important these strategies might be for promoting positive mental health, preventing poor mental health, and possibly preventing suicide among school-aged children and youth, it is a mistaken belief that this can prevent school shootings (Mears, Moon, & Thielo, 2017). Many times the accounts of the perpetrators having been bullied come from potentially inaccurate third-party accounts, rather than the perpetrators themselves; bullying was not involved in all instances of school shooting; a perpetrator’s perception of being bullied/persecuted are not necessarily accurate; many who experience severe bullying do not perpetrate these incidents; bullies are the least targeted shooting victims; perpetrators of the shooting incidents were often bullying others; and, bullying is only one of many important factors associated with perpetrating such an incident (Ioannou, Hammond, & Simpson, 2015; Mears, Moon, & Thielo, 2017; Newman &Fox, 2009; Raitanen, Sandberg, & Oksanen, 2017). While mass media reports deliver bullying as a means of explaining the inexplicable, the reality is not so simple: “The connection between bullying and school shootings is elusive” (Langman, 2014), and “the relationship between bullying and school shooting is, at best, tenuous” (Mears, Moon, & Thielo, 2017, p. 940). The point is, when a narrative becomes this publicly accepted, it is difficult to sort out truth and reality without going back to original sources of information and evidence.

What May or May Not Be Empirical Literature: Literature Reviews

Investigators typically engage in a review of existing literature as they develop their own research studies. The review informs them about where knowledge gaps exist, methods previously employed by other scholars, limitations of prior work, and previous scholars’ recommendations for directing future research. These reviews may appear as a published article, without new study data being reported (see Fields, Anderson, & Dabelko-Schoeny, 2014 for example). Or, the literature review may appear in the introduction to their own empirical study report. These literature reviews are not considered to be empirical evidence sources themselves, although they may be based on empirical evidence sources. One reason is that the authors of a literature review may or may not have engaged in a systematic search process, identifying a full, rich, multi-sided pool of evidence reports.

There is, however, a type of review that applies systematic methods and is, therefore, considered to be more strongly rooted in evidence: the systematic review .

Systematic review of literature. A systematic reviewis a type of literature report where established methods have been systematically applied, objectively, in locating and synthesizing a body of literature. The systematic review report is characterized by a great deal of transparency about the methods used and the decisions made in the review process, and are replicable. Thus, it meets the criteria for empirical literature: systematic observation and methodology, objectivity, and transparency/reproducibility. We will work a great deal more with systematic reviews in the second course, SWK 3402, since they are important tools for understanding interventions. They are somewhat less common, but not unheard of, in helping us understand diverse populations, social work problems, and social phenomena.

Locating Empirical Evidence

Social workers have available a wide array of tools and resources for locating empirical evidence in the literature. These can be organized into four general categories.

Journal Articles. A number of professional journals publish articles where investigators report on the results of their empirical studies. However, it is important to know how to distinguish between empirical and non-empirical manuscripts in these journals. A key indicator, though not the only one, involves a peer review process . Many professional journals require that manuscripts undergo a process of peer review before they are accepted for publication. This means that the authors’ work is shared with scholars who provide feedback to the journal editor as to the quality of the submitted manuscript. The editor then makes a decision based on the reviewers’ feedback:

• Accept as is
• Accept with minor revisions
• Request that a revision be resubmitted (no assurance of acceptance)

When a “revise and resubmit” decision is made, the piece will go back through the review process to determine if it is now acceptable for publication and that all of the reviewers’ concerns have been adequately addressed. Editors may also reject a manuscript because it is a poor fit for the journal, based on its mission and audience, rather than sending it for review consideration.

Indicators of journal relevance. Various journals are not equally relevant to every type of question being asked of the literature. Journals may overlap to a great extent in terms of the topics they might cover; in other words, a topic might appear in multiple different journals, depending on how the topic was being addressed. For example, articles that might help answer a question about the relationship between community poverty and violence exposure might appear in several different journals, some with a focus on poverty, others with a focus on violence, and still others on community development or public health. Journal titles are sometimes a good starting point but may not give a broad enough picture of what they cover in their contents.

In focusing a literature search, it also helps to review a journal’s mission and target audience. For example, at least four different journals focus specifically on poverty:

• Journal of Children & Poverty
• Journal of Poverty
• Journal of Poverty and Social Justice
• Poverty & Public Policy

Let’s look at an example using the Journal of Poverty and Social Justice . Information about this journal is located on the journal’s webpage: http://policy.bristoluniversitypress.co.uk/journals/journal-of-poverty-and-social-justice . In the section headed “About the Journal” you can see that it is an internationally focused research journal, and that it addresses social justice issues in addition to poverty alone. The research articles are peer-reviewed (there appear to be non-empirical discussions published, as well). These descriptions about a journal are almost always available, sometimes listed as “scope” or “mission.” These descriptions also indicate the sponsorship of the journal—sponsorship may be institutional (a particular university or agency, such as Smith College Studies in Social Work ), a professional organization, such as the Council on Social Work Education (CSWE) or the National Association of Social Work (NASW), or a publishing company (e.g., Taylor & Frances, Wiley, or Sage).

Indicators of journal caliber.  Despite engaging in a peer review process, not all journals are equally rigorous. Some journals have very high rejection rates, meaning that many submitted manuscripts are rejected; others have fairly high acceptance rates, meaning that relatively few manuscripts are rejected. This is not necessarily the best indicator of quality, however, since newer journals may not be sufficiently familiar to authors with high quality manuscripts and some journals are very specific in terms of what they publish. Another index that is sometimes used is the journal’s impact factor . Impact factor is a quantitative number indicative of how often articles published in the journal are cited in the reference list of other journal articles—the statistic is calculated as the number of times on average each article published in a particular year were cited divided by the number of articles published (the number that could be cited). For example, the impact factor for the Journal of Poverty and Social Justice in our list above was 0.70 in 2017, and for the Journal of Poverty was 0.30. These are relatively low figures compared to a journal like the New England Journal of Medicine with an impact factor of 59.56! This means that articles published in that journal were, on average, cited more than 59 times in the next year or two.

Impact factors are not necessarily the best indicator of caliber, however, since many strong journals are geared toward practitioners rather than scholars, so they are less likely to be cited by other scholars but may have a large impact on a large readership. This may be the case for a journal like the one titled Social Work, the official journal of the National Association of Social Workers. It is distributed free to all members: over 120,000 practitioners, educators, and students of social work world-wide. The journal has a recent impact factor of.790. The journals with social work relevant content have impact factors in the range of 1.0 to 3.0 according to Scimago Journal & Country Rank (SJR), particularly when they are interdisciplinary journals (for example, Child Development , Journal of Marriage and Family , Child Abuse and Neglect , Child Maltreatmen t, Social Service Review , and British Journal of Social Work ). Once upon a time, a reader could locate different indexes comparing the “quality” of social work-related journals. However, the concept of “quality” is difficult to systematically define. These indexes have mostly been replaced by impact ratings, which are not necessarily the best, most robust indicators on which to rely in assessing journal quality. For example, new journals addressing cutting edge topics have not been around long enough to have been evaluated using this particular tool, and it takes a few years for articles to begin to be cited in other, later publications.

Beware of pseudo-, illegitimate, misleading, deceptive, and suspicious journals . Another side effect of living in the Age of Information is that almost anyone can circulate almost anything and call it whatever they wish. This goes for “journal” publications, as well. With the advent of open-access publishing in recent years (electronic resources available without subscription), we have seen an explosion of what are called predatory or junk journals . These are publications calling themselves journals, often with titles very similar to legitimate publications and often with fake editorial boards. These “publications” lack the integrity of legitimate journals. This caution is reminiscent of the discussions earlier in the course about pseudoscience and “snake oil” sales. The predatory nature of many apparent information dissemination outlets has to do with how scientists and scholars may be fooled into submitting their work, often paying to have their work peer-reviewed and published. There exists a “thriving black-market economy of publishing scams,” and at least two “journal blacklists” exist to help identify and avoid these scam journals (Anderson, 2017).

This issue is important to information consumers, because it creates a challenge in terms of identifying legitimate sources and publications. The challenge is particularly important to address when information from on-line, open-access journals is being considered. Open-access is not necessarily a poor choice—legitimate scientists may pay sizeable fees to legitimate publishers to make their work freely available and accessible as open-access resources. On-line access is also not necessarily a poor choice—legitimate publishers often make articles available on-line to provide timely access to the content, especially when publishing the article in hard copy will be delayed by months or even a year or more. On the other hand, stating that a journal engages in a peer-review process is no guarantee of quality—this claim may or may not be truthful. Pseudo- and junk journals may engage in some quality control practices, but may lack attention to important quality control processes, such as managing conflict of interest, reviewing content for objectivity or quality of the research conducted, or otherwise failing to adhere to industry standards (Laine & Winker, 2017).

One resource designed to assist with the process of deciphering legitimacy is the Directory of Open Access Journals (DOAJ). The DOAJ is not a comprehensive listing of all possible legitimate open-access journals, and does not guarantee quality, but it does help identify legitimate sources of information that are openly accessible and meet basic legitimacy criteria. It also is about open-access journals, not the many journals published in hard copy.

An additional caution: Search for article corrections. Despite all of the careful manuscript review and editing, sometimes an error appears in a published article. Most journals have a practice of publishing corrections in future issues. When you locate an article, it is helpful to also search for updates. Here is an example where data presented in an article’s original tables were erroneous, and a correction appeared in a later issue.

• Marchant, A., Hawton, K., Stewart A., Montgomery, P., Singaravelu, V., Lloyd, K., Purdy, N., Daine, K., & John, A. (2017). A systematic review of the relationship between internet use, self-harm and suicidal behaviour in young people: The good, the bad and the unknown. PLoS One, 12(8): e0181722. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5558917/
• Marchant, A., Hawton, K., Stewart A., Montgomery, P., Singaravelu, V., Lloyd, K., Purdy, N., Daine, K., & John, A. (2018).Correction—A systematic review of the relationship between internet use, self-harm and suicidal behaviour in young people: The good, the bad and the unknown. PLoS One, 13(3): e0193937.  http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0193937

Search Tools. In this age of information, it is all too easy to find items—the problem lies in sifting, sorting, and managing the vast numbers of items that can be found. For example, a simple Google® search for the topic “community poverty and violence” resulted in about 15,600,000 results! As a means of simplifying the process of searching for journal articles on a specific topic, a variety of helpful tools have emerged. One type of search tool has previously applied a filtering process for you: abstracting and indexing databases . These resources provide the user with the results of a search to which records have already passed through one or more filters. For example, PsycINFO is managed by the American Psychological Association and is devoted to peer-reviewed literature in behavioral science. It contains almost 4.5 million records and is growing every month. However, it may not be available to users who are not affiliated with a university library. Conducting a basic search for our topic of “community poverty and violence” in PsychINFO returned 1,119 articles. Still a large number, but far more manageable. Additional filters can be applied, such as limiting the range in publication dates, selecting only peer reviewed items, limiting the language of the published piece (English only, for example), and specified types of documents (either chapters, dissertations, or journal articles only, for example). Adding the filters for English, peer-reviewed journal articles published between 2010 and 2017 resulted in 346 documents being identified.

Just as was the case with journals, not all abstracting and indexing databases are equivalent. There may be overlap between them, but none is guaranteed to identify all relevant pieces of literature. Here are some examples to consider, depending on the nature of the questions asked of the literature:

• Academic Search Complete—multidisciplinary index of 9,300 peer-reviewed journals
• AgeLine—multidisciplinary index of aging-related content for over 600 journals
• Campbell Collaboration—systematic reviews in education, crime and justice, social welfare, international development
• Google Scholar—broad search tool for scholarly literature across many disciplines
• MEDLINE/ PubMed—National Library of medicine, access to over 15 million citations
• Oxford Bibliographies—annotated bibliographies, each is discipline specific (e.g., psychology, childhood studies, criminology, social work, sociology)
• PsycINFO/PsycLIT—international literature on material relevant to psychology and related disciplines
• SocINDEX—publications in sociology
• Social Sciences Abstracts—multiple disciplines
• Social Work Abstracts—many areas of social work are covered
• Web of Science—a “meta” search tool that searches other search tools, multiple disciplines

Placing our search for information about “community violence and poverty” into the Social Work Abstracts tool with no additional filters resulted in a manageable 54-item list. Finally, abstracting and indexing databases are another way to determine journal legitimacy: if a journal is indexed in a one of these systems, it is likely a legitimate journal. However, the converse is not necessarily true: if a journal is not indexed does not mean it is an illegitimate or pseudo-journal.

Government Sources. A great deal of information is gathered, analyzed, and disseminated by various governmental branches at the international, national, state, regional, county, and city level. Searching websites that end in.gov is one way to identify this type of information, often presented in articles, news briefs, and statistical reports. These government sources gather information in two ways: they fund external investigations through grants and contracts and they conduct research internally, through their own investigators. Here are some examples to consider, depending on the nature of the topic for which information is sought:

• Agency for Healthcare Research and Quality (AHRQ) at https://www.ahrq.gov/
• Bureau of Justice Statistics (BJS) at https://www.bjs.gov/
• Census Bureau at https://www.census.gov
• Morbidity and Mortality Weekly Report of the CDC (MMWR-CDC) at https://www.cdc.gov/mmwr/index.html
• Child Welfare Information Gateway at https://www.childwelfare.gov
• Children’s Bureau/Administration for Children & Families at https://www.acf.hhs.gov
• Forum on Child and Family Statistics at https://www.childstats.gov
• National Institutes of Health (NIH) at https://www.nih.gov , including (not limited to):
• National Institute on Aging (NIA at https://www.nia.nih.gov
• National Institute on Alcohol Abuse and Alcoholism (NIAAA) at https://www.niaaa.nih.gov
• National Institute of Child Health and Human Development (NICHD) at https://www.nichd.nih.gov
• National Institute on Drug Abuse (NIDA) at https://www.nida.nih.gov
• National Institute of Environmental Health Sciences at https://www.niehs.nih.gov
• National Institute of Mental Health (NIMH) at https://www.nimh.nih.gov
• National Institute on Minority Health and Health Disparities at https://www.nimhd.nih.gov
• National Institute of Justice (NIJ) at https://www.nij.gov
• Substance Abuse and Mental Health Services Administration (SAMHSA) at https://www.samhsa.gov/
• United States Agency for International Development at https://usaid.gov

Each state and many counties or cities have similar data sources and analysis reports available, such as Ohio Department of Health at https://www.odh.ohio.gov/healthstats/dataandstats.aspx and Franklin County at https://statisticalatlas.com/county/Ohio/Franklin-County/Overview . Data are available from international/global resources (e.g., United Nations and World Health Organization), as well.

Other Sources. The Health and Medicine Division (HMD) of the National Academies—previously the Institute of Medicine (IOM)—is a nonprofit institution that aims to provide government and private sector policy and other decision makers with objective analysis and advice for making informed health decisions. For example, in 2018 they produced reports on topics in substance use and mental health concerning the intersection of opioid use disorder and infectious disease,  the legal implications of emerging neurotechnologies, and a global agenda concerning the identification and prevention of violence (see http://www.nationalacademies.org/hmd/Global/Topics/Substance-Abuse-Mental-Health.aspx ). The exciting aspect of this resource is that it addresses many topics that are current concerns because they are hoping to help inform emerging policy. The caution to consider with this resource is the evidence is often still emerging, as well.

Numerous “think tank” organizations exist, each with a specific mission. For example, the Rand Corporation is a nonprofit organization offering research and analysis to address global issues since 1948. The institution’s mission is to help improve policy and decision making “to help individuals, families, and communities throughout the world be safer and more secure, healthier and more prosperous,” addressing issues of energy, education, health care, justice, the environment, international affairs, and national security (https://www.rand.org/about/history.html). And, for example, the Robert Woods Johnson Foundation is a philanthropic organization supporting research and research dissemination concerning health issues facing the United States. The foundation works to build a culture of health across systems of care (not only medical care) and communities (https://www.rwjf.org).

While many of these have a great deal of helpful evidence to share, they also may have a strong political bias. Objectivity is often lacking in what information these organizations provide: they provide evidence to support certain points of view. That is their purpose—to provide ideas on specific problems, many of which have a political component. Think tanks “are constantly researching solutions to a variety of the world’s problems, and arguing, advocating, and lobbying for policy changes at local, state, and federal levels” (quoted from https://thebestschools.org/features/most-influential-think-tanks/ ). Helpful information about what this one source identified as the 50 most influential U.S. think tanks includes identifying each think tank’s political orientation. For example, The Heritage Foundation is identified as conservative, whereas Human Rights Watch is identified as liberal.

While not the same as think tanks, many mission-driven organizations also sponsor or report on research, as well. For example, the National Association for Children of Alcoholics (NACOA) in the United States is a registered nonprofit organization. Its mission, along with other partnering organizations, private-sector groups, and federal agencies, is to promote policy and program development in research, prevention and treatment to provide information to, for, and about children of alcoholics (of all ages). Based on this mission, the organization supports knowledge development and information gathering on the topic and disseminates information that serves the needs of this population. While this is a worthwhile mission, there is no guarantee that the information meets the criteria for evidence with which we have been working. Evidence reported by think tank and mission-driven sources must be utilized with a great deal of caution and critical analysis!

In many instances an empirical report has not appeared in the published literature, but in the form of a technical or final report to the agency or program providing the funding for the research that was conducted. One such example is presented by a team of investigators funded by the National Institute of Justice to evaluate a program for training professionals to collect strong forensic evidence in instances of sexual assault (Patterson, Resko, Pierce-Weeks, & Campbell, 2014): https://www.ncjrs.gov/pdffiles1/nij/grants/247081.pdf . Investigators may serve in the capacity of consultant to agencies, programs, or institutions, and provide empirical evidence to inform activities and planning. One such example is presented by Maguire-Jack (2014) as a report to a state’s child maltreatment prevention board: https://preventionboard.wi.gov/Documents/InvestmentInPreventionPrograming_Final.pdf .

When Direct Answers to Questions Cannot Be Found. Sometimes social workers are interested in finding answers to complex questions or questions related to an emerging, not-yet-understood topic. This does not mean giving up on empirical literature. Instead, it requires a bit of creativity in approaching the literature. A Venn diagram might help explain this process. Consider a scenario where a social worker wishes to locate literature to answer a question concerning issues of intersectionality. Intersectionality is a social justice term applied to situations where multiple categorizations or classifications come together to create overlapping, interconnected, or multiplied disadvantage. For example, women with a substance use disorder and who have been incarcerated face a triple threat in terms of successful treatment for a substance use disorder: intersectionality exists between being a woman, having a substance use disorder, and having been in jail or prison. After searching the literature, little or no empirical evidence might have been located on this specific triple-threat topic. Instead, the social worker will need to seek literature on each of the threats individually, and possibly will find literature on pairs of topics (see Figure 3-1). There exists some literature about women’s outcomes for treatment of a substance use disorder (a), some literature about women during and following incarceration (b), and some literature about substance use disorders and incarceration (c). Despite not having a direct line on the center of the intersecting spheres of literature (d), the social worker can develop at least a partial picture based on the overlapping literatures.

Figure 3-1. Venn diagram of intersecting literature sets.

Take a moment to complete the following activity. For each statement about empirical literature, decide if it is true or false.

Social Work 3401 Coursebook Copyright © by Dr. Audrey Begun is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Literature Review: 3 Essential Ingredients

The theoretical framework, empirical research and research gap

By: Derek Jansen (MBA) | Reviewer: Eunice Rautenbach (DTech) | July 2023

Writing a comprehensive but concise literature review is no simple task. There’s a lot of ground to cover and it can be challenging to figure out what’s important and what’s not. In this post, we’ll unpack three essential ingredients that need to be woven into your literature review to lay a rock-solid foundation for your study.

This post is based on our popular online course, Literature Review Bootcamp . In the course, we walk you through the full process of developing a literature review, step by step. If it’s your first time writing a literature review, you definitely want to use this link to get 50% off the course (limited-time offer).

Overview: Essential Ingredients

• Ingredients vs structure
• The theoretical framework (foundation of theory)
• The empirical research
• The research gap
• Summary & key takeaways

Ingredients vs Structure

As a starting point, it’s important to clarify that the three ingredients we’ll cover in this video are things that need to feature within your literature review, as opposed to a set structure for your chapter . In other words, there are different ways you can weave these three ingredients into your literature review. Regardless of which structure you opt for, each of the three components will make an appearance in some shape or form. If you’re keen to learn more about structural options, we’ve got a dedicated post about that here .

1. The Theoretical Framework

Let’s kick off with the first essential ingredient – that is the theoretical framework , also called the foundation of theory . 

The foundation of theory, as the name suggests, is where you’ll lay down the foundational building blocks for your literature review so that your reader can get a clear idea of the core concepts, theories and assumptions (in relation to your research aims and questions) that will guide your study. Note that this is not the same as a conceptual framework .

Typically you’ll cover a few things within the theoretical framework:

Firstly, you’ll need to clearly define the key constructs and variables that will feature within your study. In many cases, any given term can have multiple different definitions or interpretations – for example, different people will define the concept of “integrity” in different ways. This variation in interpretation can, of course, wreak havoc on how your study is understood. So, this section is where you’ll pin down what exactly you mean when you refer to X, Y or Z in your study, as well as why you chose that specific definition. It’s also a good idea to state any assumptions that are inherent in these definitions and why these are acceptable, given the purpose of your study.

Related to this, the second thing you’ll need to cover in your theoretical framework is the relationships between these variables and/or constructs . For example, how does one variable potentially affect another variable – does A have an impact on B, B on A, and so on? In other words, you want to connect the dots between the different “things” of interest that you’ll be exploring in your study. Note that you only need to focus on the key items of interest here (i.e. those most central to your research aims and questions) – not every possible construct or variable.

Lastly, and very importantly, you need to discuss the existing theories that are relevant to your research aims and research questions . For example, if you’re investigating the uptake/adoption of a certain application or software, you might discuss Davis’ Technology Acceptance Model and unpack what it has to say about the factors that influence technology adoption. More importantly, though, you need to explain how this impacts your expectations about what you will find in your own study . In other words, your theoretical framework should reveal some insights about what answers you might expect to find to your research questions .

If this sounds a bit fluffy, don’t worry. We deep dive into the theoretical framework (as well as the conceptual framework) and look at practical examples in Literature Review Bootcamp . If you’d like to learn more, take advantage of the limited-time offer (60% off the standard price).

Need a helping hand?

2. The Empirical Research

Onto the second essential ingredient, which is  empirical research . This section is where you’ll present a critical discussion of the existing empirical research that is relevant to your research aims and questions.

But what exactly is empirical research?

Simply put, empirical research includes any study that involves actual data collection and analysis , whether that’s qualitative data, quantitative data, or a mix of both . This contrasts against purely theoretical literature (the previous ingredient), which draws its conclusions based exclusively on logic and reason , as opposed to an analysis of real-world data.

In other words, theoretical literature provides a prediction or expectation of what one might find based on reason and logic, whereas empirical research tests the accuracy of those predictions using actual real-world data . This reflects the broader process of knowledge creation – in other words, first developing a theory and then testing it out in the field.

Long story short, the second essential ingredient of a high-quality literature review is a critical discussion of the existing empirical research . Here, it’s important to go beyond description . You’ll need to present a critical analysis that addresses some (if not all) of the following questions:

• What have different studies found in relation to your research questions ?
• What contexts have (and haven’t been covered)? For example, certain countries, cities, cultures, etc.
• Are the findings across the studies similar or is there a lot of variation ? If so, why might this be the case?
• What sorts of research methodologies have been used and how could these help me develop my own methodology?
• What were the noteworthy limitations of these studies?

Simply put, your task here is to present a synthesis of what’s been done (and found) within the empirical research, so that you can clearly assess the current state of knowledge and identify potential research gaps , which leads us to our third essential ingredient.

The Research Gap

The third essential ingredient of a high-quality literature review is a discussion of the research gap (or gaps).

But what exactly is a research gap?

Simply put, a research gap is any unaddressed or inadequately explored area within the existing body of academic knowledge. In other words, a research gap emerges whenever there’s still some uncertainty regarding a certain topic or question.

For example, it might be the case that there are mixed findings regarding the relationship between two variables (e.g., job performance and work-from-home policies). Similarly, there might be a lack of research regarding the impact of a specific new technology on people’s mental health. On the other end of the spectrum, there might be a wealth of research regarding a certain topic within one country (say the US), but very little research on that same topic in a different social context (say, China).

These are just random examples, but as you can see, research gaps can emerge from many different places. What’s important to understand is that the research gap (or gaps) needs to emerge from your previous discussion of the theoretical and empirical literature . In other words, your discussion in those sections needs to start laying the foundation for the research gap.

For example, when discussing empirical research, you might mention that most studies have focused on a certain context , yet very few (or none) have focused on another context, and there’s reason to believe that findings may differ. Or you might highlight how there’s a fair deal of mixed findings and disagreement regarding a certain matter. In other words, you want to start laying a little breadcrumb trail in those sections so that your discussion of the research gap is firmly rooted in the rest of the literature review.

But why does all of this matter?

Well, the research gap should serve as the core justification for your study . Through your literature review, you’ll show what gaps exist in the current body of knowledge, and then your study will then attempt to fill (or contribute towards filling) one of those gaps. In other words, you’re first explaining what the problem is (some sort of gap) and then proposing how you’ll solve it.

Key Takeaways

To recap, the three ingredients that need to be mixed into your literature review are:

• The foundation of theory or theoretical framework
• The empirical or evidence-based research

As we mentioned earlier, these are components of a literature review and not (necessarily) a structure for your literature review chapter. Of course, you can structure your chapter in a way that reflects these three components (in fact, in some cases that works very well), but it’s certainly not the only option. The right structure will vary from study to study , depending on various factors.

If you’d like to get hands-on help developing your literature review, be sure to check out our private coaching service , where we hold your hand through the entire research journey, step by step. 

Psst… there’s more!

This post is an extract from our bestselling short course, Literature Review Bootcamp . If you want to work smart, you don't want to miss this .

very good , as the first writer of the thesis i will need ur advise . please give me a piece of idea on topic -impact of national standardized exam on students learning engagement . Thank you .

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 5. The Literature Review
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• Give a new interpretation of old material or combine new with old interpretations,
• Trace the intellectual progression of the field, including major debates,
• Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
• Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

• Place each work in the context of its contribution to understanding the research problem being studied.
• Describe the relationship of each work to the others under consideration.
• Identify new ways to interpret prior research.
• Reveal any gaps that exist in the literature.
• Resolve conflicts amongst seemingly contradictory previous studies.
• Identify areas of prior scholarship to prevent duplication of effort.
• Point the way in fulfilling a need for additional research.
• Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE: Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most convincing or least convincing?
• Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

• Current Situation : Information necessary to understand the current topic or focus of the literature review.
• Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
• History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
• Standards : Description of the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

• Sources in your literature review do not clearly relate to the research problem;
• You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
• Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
• Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• Does not describe the search procedures that were used in identifying the literature to review;
• Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
• Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

• Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
• Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
• Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

• << Previous: Theoretical Framework
• Next: Citation Tracking >>
• Last Updated: Sep 4, 2024 9:40 AM
• URL: https://libguides.usc.edu/writingguide

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation
• What is a Literature Review? | Guide, Template, & Examples

What is a Literature Review? | Guide, Template, & Examples

Published on 22 February 2022 by Shona McCombes . Revised on 7 June 2022.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research.

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarise sources – it analyses, synthesises, and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Why write a literature review, examples of literature reviews, step 1: search for relevant literature, step 2: evaluate and select sources, step 3: identify themes, debates and gaps, step 4: outline your literature review’s structure, step 5: write your literature review, frequently asked questions about literature reviews, introduction.

• Quick Run-through
• Step 1 & 2

When you write a dissertation or thesis, you will have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and scholarly context
• Develop a theoretical framework and methodology for your research
• Position yourself in relation to other researchers and theorists
• Show how your dissertation addresses a gap or contributes to a debate

You might also have to write a literature review as a stand-alone assignment. In this case, the purpose is to evaluate the current state of research and demonstrate your knowledge of scholarly debates around a topic.

The content will look slightly different in each case, but the process of conducting a literature review follows the same steps. We’ve written a step-by-step guide that you can follow below.

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research objectives and questions .

If you are writing a literature review as a stand-alone assignment, you will have to choose a focus and develop a central question to direct your search. Unlike a dissertation research question, this question has to be answerable without collecting original data. You should be able to answer it based only on a review of existing publications.

Make a list of keywords

Start by creating a list of keywords related to your research topic. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list if you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can use boolean operators to help narrow down your search:

Read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

To identify the most important publications on your topic, take note of recurring citations. If the same authors, books or articles keep appearing in your reading, make sure to seek them out.

You probably won’t be able to read absolutely everything that has been written on the topic – you’ll have to evaluate which sources are most relevant to your questions.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models and methods? Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• How does the publication contribute to your understanding of the topic? What are its key insights and arguments?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible, and make sure you read any landmark studies and major theories in your field of research.

You can find out how many times an article has been cited on Google Scholar – a high citation count means the article has been influential in the field, and should certainly be included in your literature review.

The scope of your review will depend on your topic and discipline: in the sciences you usually only review recent literature, but in the humanities you might take a long historical perspective (for example, to trace how a concept has changed in meaning over time).

Remember that you can use our template to summarise and evaluate sources you’re thinking about using!

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It’s important to keep track of your sources with references to avoid plagiarism . It can be helpful to make an annotated bibliography, where you compile full reference information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

You can use our free APA Reference Generator for quick, correct, consistent citations.

Prevent plagiarism, run a free check.

To begin organising your literature review’s argument and structure, you need to understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly-visual platforms like Instagram and Snapchat – this is a gap that you could address in your own research.

There are various approaches to organising the body of a literature review. You should have a rough idea of your strategy before you start writing.

Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarising sources in order.

Try to analyse patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organise your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text, your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

If you are writing the literature review as part of your dissertation or thesis, reiterate your central problem or research question and give a brief summary of the scholarly context. You can emphasise the timeliness of the topic (“many recent studies have focused on the problem of x”) or highlight a gap in the literature (“while there has been much research on x, few researchers have taken y into consideration”).

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, make sure to follow these tips:

• Summarise and synthesise: give an overview of the main points of each source and combine them into a coherent whole.
• Analyse and interpret: don’t just paraphrase other researchers – add your own interpretations, discussing the significance of findings in relation to the literature as a whole.
• Critically evaluate: mention the strengths and weaknesses of your sources.
• Write in well-structured paragraphs: use transitions and topic sentences to draw connections, comparisons and contrasts.

In the conclusion, you should summarise the key findings you have taken from the literature and emphasise their significance.

If the literature review is part of your dissertation or thesis, reiterate how your research addresses gaps and contributes new knowledge, or discuss how you have drawn on existing theories and methods to build a framework for your research. This can lead directly into your methodology section.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your  dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, June 07). What is a Literature Review? | Guide, Template, & Examples. Scribbr. Retrieved 3 September 2024, from https://www.scribbr.co.uk/thesis-dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a theoretical framework | a step-by-step guide, what is a research methodology | steps & tips.

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health (m-health) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jun 20, 2024 9:08 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

Summer Undergraduate Research Experience (SURE)

• Start Your Research Project

What is a Literature Review?

Scholarly article types.

• Find Background Information
• Find Articles
• Evaluate Sources
• Manage Citations and Sources
• SURE Research Scenarios

Scholarly Workflow

What is a literature review?

A literature review provides an overview of the scholarly literature (e.g. books, articles, dissertations, proceedings) relevant to an area of research or theory. The review typically will include a summary of the major questions in a area and critical evaluations of work that has already been done. Literature reviews are also helpful for their comprehensive bibliographies. This  webpage by the UC Santa Cruz Library  does a good job of explaining lit reviews.

Literature reviews typically include these components:

• An overview of the subject
• Organization of relevant publications into subtopics, theoretical areas, or key debates
• An analysis and discussion of how various works relate to one another the the relevant questions
• A discussion of unresolved questions or future directions
• Some will also include discussions of key data collection and analysis methodologies

The following resources are great places to start when compiling a comprehensive bibliography.

• Oxford Bibliographies
• Browse by Political Science, Economics or just run search. Note that you can limit by access that Emory has. In cases where we do not, check discoverE, and request needed chapters.
• Great resource for building initial literature reviews, identifying important figures in the literature, and engaging with major theories. Emory has a subscription to all the Handbooks.
• In-depth reviews of the literature in a discipline, published annually, with an emphasis on overviews and more recent approaches and theories.  Note that the HTML version is good for tracking citations, but does not have needed page numbers.

Scholarly journals may contain various types of scholarly literature. Knowing the different types of articles will help you find the best resources for your research needs.

• Research articles (aka empirical articles)  - primary sources of authors reporting their own studies.
• Theoretical articles  - concepts, frameworks, models, and perspectives.
• Literature reviews  - summary of literature on a given topic.
• Meta-analyses  - use statistical methods to summarize studies.
• Systematic reviews  - high-level summary of literature to answer focused--often clinical--question.
• Case studies and clinical trials  - describe real patient cases/treatments.
• Conference materials  - proceedings, notes, etc.
• Commentaries, book reviews, opinions, letters, etc.  - opinion and thought pieces.
• << Previous: Start Your Research Project
• Next: Find Background Information >>
• Last Updated: Jun 4, 2024 4:14 PM
• URL: https://guides.libraries.emory.edu/sure

Henry Whittemore Library

Research skills hub.

• Click on a section, choose a topic. On computers, page opens on the right. On mobile, scroll past menu to read.
• Who are Our 'Experts'?
• The Information Timeline
• Peer Review
• The Internet vs. Academic Databases
• What kind of Info is in Databases?
• What's in OUR Library's Collection? This link opens in a new window
• Data vs Information
• Popular vs. Scholarly
• Grey Literature
• Primary, Secondary, Tertiary Sources
• What KIND of Info do You Need?
• Find & Narrow Topics
• Keywords from your Topic
• Concept / Mind Mapping
• Advanced Database Features
• Use Boolean Connectors
• Use Phrases
• Use Database Fields
• Use Truncation
• Use Database Limiters
• Use Synonyms
• Use Wildcards
• Use Database Subjects
• Find Better Keywords DURING Your Search
• Google Advanced Search Tricks
• Search Google Scholar
• Reference Mining
• Citation Tracking
• About Call Numbers
• Periodical's Content Not the Same as the Periodical's Website!
• From a scholarly journal - But what IS it??
• Anatomy of a Scientific Research Article
• Quantitative vs Qualitative Research Articles
• Theoretical vs Empirical Research Articles

Theoretical vs Empirical Articles

• Types of Review Articles
• Identify the Research Study Type
• Spotting Bad Science
• Types of Scientific Evidence
• Found it in a newspaper - What is it??
• More About News Evaluation
• Webste Evaluation
• Fear & Loathing on Social Media
• Pseudoscience on Social Media -the Slippery Slope
• Getting Full Text During Library Site Searches
• Find Free Full Text on the Internet
• Got a Citation? Check the Library for Full Text
• Make an Interlibrary Loan (ILL) Request
• Better Organize What You Found
• Organize with Zotero
• Organize with Google Docs
• Organize using personal accounts in databases
• Just store stuff on your laptop?
• Print / Save as PDF Website Info Without the Ads
• Avoid Plagiarism
• Why SO MANY citation systems?
• Why Citations in the Text?
• Citing vs Attribution
• Paraphrase Correctly
• Create a Bibliography
• Create an Annotated Bibliography
• Our Complete Citation Guide This link opens in a new window

Theoretical Research  is a logical exploration of a system of beliefs and assumptions, working with abstract principles related to a field of knowledge.

• Essentially...theorizing

Empirical Research is    based on real-life direct or indirect observation and measurement of phenomena by a researcher.

• Basically... Collecting data by Observing or Experimenting

• << Previous: Quantitative vs Qualitative Research Articles
• Next: Types of Review Articles >>
• Last Updated: Sep 6, 2024 3:23 PM
• URL: https://library.framingham.edu/research-skills-hub

Library Socials

Contact us:.

[email protected] Phone: (508) 626-4650 [email protected] Phone: (508) 626-4654

Search this site

Frequently asked questions

What is the difference between a literature review and a theoretical framework.

A literature review and a theoretical framework are not the same thing and cannot be used interchangeably. While a theoretical framework describes the theoretical underpinnings of your work, a literature review critically evaluates existing research relating to your topic. You’ll likely need both in your dissertation .

Frequently asked questions: Dissertation

Dissertation word counts vary widely across different fields, institutions, and levels of education:

• An undergraduate dissertation is typically 8,000–15,000 words
• A master’s dissertation is typically 12,000–50,000 words
• A PhD thesis is typically book-length: 70,000–100,000 words

However, none of these are strict guidelines – your word count may be lower or higher than the numbers stated here. Always check the guidelines provided by your university to determine how long your own dissertation should be.

A dissertation prospectus or proposal describes what or who you plan to research for your dissertation. It delves into why, when, where, and how you will do your research, as well as helps you choose a type of research to pursue. You should also determine whether you plan to pursue qualitative or quantitative methods and what your research design will look like.

It should outline all of the decisions you have taken about your project, from your dissertation topic to your hypotheses and research objectives , ready to be approved by your supervisor or committee.

Note that some departments require a defense component, where you present your prospectus to your committee orally.

A thesis is typically written by students finishing up a bachelor’s or Master’s degree. Some educational institutions, particularly in the liberal arts, have mandatory theses, but they are often not mandatory to graduate from bachelor’s degrees. It is more common for a thesis to be a graduation requirement from a Master’s degree.

Even if not mandatory, you may want to consider writing a thesis if you:

• Plan to attend graduate school soon
• Have a particular topic you’d like to study more in-depth
• Are considering a career in research
• Would like a capstone experience to tie up your academic experience

The conclusion of your thesis or dissertation should include the following:

• A restatement of your research question
• A summary of your key arguments and/or results
• A short discussion of the implications of your research

The conclusion of your thesis or dissertation shouldn’t take up more than 5–7% of your overall word count.

For a stronger dissertation conclusion , avoid including:

• Important evidence or analysis that wasn’t mentioned in the discussion section and results section
• Generic concluding phrases (e.g. “In conclusion …”)
• Weak statements that undermine your argument (e.g., “There are good points on both sides of this issue.”)

Your conclusion should leave the reader with a strong, decisive impression of your work.

While it may be tempting to present new arguments or evidence in your thesis or disseration conclusion , especially if you have a particularly striking argument you’d like to finish your analysis with, you shouldn’t. Theses and dissertations follow a more formal structure than this.

All your findings and arguments should be presented in the body of the text (more specifically in the discussion section and results section .) The conclusion is meant to summarize and reflect on the evidence and arguments you have already presented, not introduce new ones.

A theoretical framework can sometimes be integrated into a  literature review chapter , but it can also be included as its own chapter or section in your dissertation . As a rule of thumb, if your research involves dealing with a lot of complex theories, it’s a good idea to include a separate theoretical framework chapter.

While a theoretical framework describes the theoretical underpinnings of your work based on existing research, a conceptual framework allows you to draw your own conclusions, mapping out the variables you may use in your study and the interplay between them.

A thesis or dissertation outline is one of the most critical first steps in your writing process. It helps you to lay out and organize your ideas and can provide you with a roadmap for deciding what kind of research you’d like to undertake.

Generally, an outline contains information on the different sections included in your thesis or dissertation , such as:

• Your anticipated title
• Your abstract
• Your chapters (sometimes subdivided into further topics like literature review , research methods , avenues for future research, etc.)

When you mention different chapters within your text, it’s considered best to use Roman numerals for most citation styles. However, the most important thing here is to remain consistent whenever using numbers in your dissertation .

In most styles, the title page is used purely to provide information and doesn’t include any images. Ask your supervisor if you are allowed to include an image on the title page before doing so. If you do decide to include one, make sure to check whether you need permission from the creator of the image.

Include a note directly beneath the image acknowledging where it comes from, beginning with the word “ Note .” (italicized and followed by a period). Include a citation and copyright attribution . Don’t title, number, or label the image as a figure , since it doesn’t appear in your main text.

Definitional terms often fall into the category of common knowledge , meaning that they don’t necessarily have to be cited. This guidance can apply to your thesis or dissertation glossary as well.

However, if you’d prefer to cite your sources , you can follow guidance for citing dictionary entries in MLA or APA style for your glossary.

A glossary is a collection of words pertaining to a specific topic. In your thesis or dissertation, it’s a list of all terms you used that may not immediately be obvious to your reader. In contrast, an index is a list of the contents of your work organized by page number.

The title page of your thesis or dissertation goes first, before all other content or lists that you may choose to include.

The title page of your thesis or dissertation should include your name, department, institution, degree program, and submission date.

Glossaries are not mandatory, but if you use a lot of technical or field-specific terms, it may improve readability to add one to your thesis or dissertation. Your educational institution may also require them, so be sure to check their specific guidelines.

A glossary or “glossary of terms” is a collection of words pertaining to a specific topic. In your thesis or dissertation, it’s a list of all terms you used that may not immediately be obvious to your reader. Your glossary only needs to include terms that your reader may not be familiar with, and is intended to enhance their understanding of your work.

A glossary is a collection of words pertaining to a specific topic. In your thesis or dissertation, it’s a list of all terms you used that may not immediately be obvious to your reader. In contrast, dictionaries are more general collections of words.

An abbreviation is a shortened version of an existing word, such as Dr. for Doctor. In contrast, an acronym uses the first letter of each word to create a wholly new word, such as UNESCO (an acronym for the United Nations Educational, Scientific and Cultural Organization).

As a rule of thumb, write the explanation in full the first time you use an acronym or abbreviation. You can then proceed with the shortened version. However, if the abbreviation is very common (like PC, USA, or DNA), then you can use the abbreviated version from the get-go.

Be sure to add each abbreviation in your list of abbreviations !

If you only used a few abbreviations in your thesis or dissertation , you don’t necessarily need to include a list of abbreviations .

If your abbreviations are numerous, or if you think they won’t be known to your audience, it’s never a bad idea to add one. They can also improve readability, minimizing confusion about abbreviations unfamiliar to your reader.

A list of abbreviations is a list of all the abbreviations that you used in your thesis or dissertation. It should appear at the beginning of your document, with items in alphabetical order, just after your table of contents .

Your list of tables and figures should go directly after your table of contents in your thesis or dissertation.

Lists of figures and tables are often not required, and aren’t particularly common. They specifically aren’t required for APA-Style, though you should be careful to follow their other guidelines for figures and tables .

If you have many figures and tables in your thesis or dissertation, include one may help you stay organized. Your educational institution may require them, so be sure to check their guidelines.

A list of figures and tables compiles all of the figures and tables that you used in your thesis or dissertation and displays them with the page number where they can be found.

The table of contents in a thesis or dissertation always goes between your abstract and your introduction .

You may acknowledge God in your dissertation acknowledgements , but be sure to follow academic convention by also thanking the members of academia, as well as family, colleagues, and friends who helped you.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

In a thesis or dissertation, the discussion is an in-depth exploration of the results, going into detail about the meaning of your findings and citing relevant sources to put them in context.

The conclusion is more shorter and more general: it concisely answers your main research question and makes recommendations based on your overall findings.

In the discussion , you explore the meaning and relevance of your research results , explaining how they fit with existing research and theory. Discuss:

• Your  interpretations : what do the results tell us?
• The  implications : why do the results matter?
• The  limitation s : what can’t the results tell us?

The results chapter or section simply and objectively reports what you found, without speculating on why you found these results. The discussion interprets the meaning of the results, puts them in context, and explains why they matter.

In qualitative research , results and discussion are sometimes combined. But in quantitative research , it’s considered important to separate the objective results from your interpretation of them.

Results are usually written in the past tense , because they are describing the outcome of completed actions.

The results chapter of a thesis or dissertation presents your research results concisely and objectively.

In quantitative research , for each question or hypothesis , state:

• The type of analysis used
• Relevant results in the form of descriptive and inferential statistics
• Whether or not the alternative hypothesis was supported

In qualitative research , for each question or theme, describe:

• Recurring patterns
• Significant or representative individual responses
• Relevant quotations from the data

Don’t interpret or speculate in the results chapter.

To automatically insert a table of contents in Microsoft Word, follow these steps:

• Apply heading styles throughout the document.
• In the references section in the ribbon, locate the Table of Contents group.
• Click the arrow next to the Table of Contents icon and select Custom Table of Contents.
• Select which levels of headings you would like to include in the table of contents.

Make sure to update your table of contents if you move text or change headings. To update, simply right click and select Update Field.

All level 1 and 2 headings should be included in your table of contents . That means the titles of your chapters and the main sections within them.

The contents should also include all appendices and the lists of tables and figures, if applicable, as well as your reference list .

Do not include the acknowledgements or abstract in the table of contents.

The abstract appears on its own page in the thesis or dissertation , after the title page and acknowledgements but before the table of contents .

An abstract for a thesis or dissertation is usually around 200–300 words. There’s often a strict word limit, so make sure to check your university’s requirements.

In a thesis or dissertation, the acknowledgements should usually be no longer than one page. There is no minimum length.

The acknowledgements are generally included at the very beginning of your thesis , directly after the title page and before the abstract .

Yes, it’s important to thank your supervisor(s) in the acknowledgements section of your thesis or dissertation .

Even if you feel your supervisor did not contribute greatly to the final product, you must acknowledge them, if only for a very brief thank you. If you do not include your supervisor, it may be seen as a snub.

In the acknowledgements of your thesis or dissertation, you should first thank those who helped you academically or professionally, such as your supervisor, funders, and other academics.

Then you can include personal thanks to friends, family members, or anyone else who supported you during the process.

Ask our team

Want to contact us directly? No problem.  We  are always here for you.

• Email [email protected]
• Start live chat
• Call +1 (510) 822-8066
• WhatsApp +31 20 261 6040

Our team helps students graduate by offering:

• A world-class citation generator
• Plagiarism Checker software powered by Turnitin
• Innovative Citation Checker software
• Professional proofreading services
• Over 300 helpful articles about academic writing, citing sources, plagiarism, and more

Scribbr specializes in editing study-related documents . We proofread:

• PhD dissertations
• Research proposals
• Personal statements
• Admission essays
• Motivation letters
• Reflection papers
• Journal articles
• Capstone projects

Scribbr’s Plagiarism Checker is powered by elements of Turnitin’s Similarity Checker , namely the plagiarism detection software and the Internet Archive and Premium Scholarly Publications content databases .

The add-on AI detector is powered by Scribbr’s proprietary software.

The Scribbr Citation Generator is developed using the open-source Citation Style Language (CSL) project and Frank Bennett’s citeproc-js . It’s the same technology used by dozens of other popular citation tools, including Mendeley and Zotero.

You can find all the citation styles and locales used in the Scribbr Citation Generator in our publicly accessible repository on Github .

• Advertise with us
• Monday, September 09, 2024

Most Widely Read Newspaper

PunchNG Menu:

• Special Features
• Sex & Relationship

ID) . '?utm_source=news-flash&utm_medium=web"> Download Punch Lite App

Project Chapter Two: Literature Review and Steps to Writing Empirical Review

Kindly share this story:

• Conceptual review
• Theoretical review,
• Empirical review or review of empirical works of literature/studies, and lastly
• Conclusion or Summary of the literature reviewed.
• Decide on a topic
• Highlight the studies/literature that you will review in the empirical review
• Analyze the works of literature separately.
• Summarize the literature in table or concept map format.
• Synthesize the literature and then proceed to write your empirical review.

All rights reserved. This material, and other digital content on this website, may not be reproduced, published, broadcast, rewritten or redistributed in whole or in part without prior express written permission from PUNCH.

Contact: [email protected]

Stay informed and ahead of the curve! Follow The Punch Newspaper on WhatsApp for real-time updates, breaking news, and exclusive content. Don't miss a headline – join now!

VERIFIED NEWS: As a Nigerian, you can earn US Dollars with REGULAR domains, buy for as low as $24, resell for up to $1000. Earn $15,000 monthly. Click here to start.

Latest News

Hos urges perm secs to accelerate civil service reforms, why osimhen, chelsea deal collapsed - mikel obi, fake nnpc director in efcc custody over n100m fraud, inec opens portal for ondo gov poll media accreditation, edo poll: inec accredits 114 media organisations.

[ICYMI] Why some Nigerian men are not biological fathers of their children – Psychologists

Lorem ipsum dolor sit amet, conse adipiscing elit.

• Tools and Resources
• Customer Services
• Contentious Politics and Political Violence
• Governance/Political Change
• Groups and Identities
• History and Politics
• International Political Economy
• Policy, Administration, and Bureaucracy
• Political Anthropology
• Political Behavior
• Political Communication
• Political Economy
• Political Institutions
• Political Philosophy
• Political Psychology
• Political Sociology
• Political Values, Beliefs, and Ideologies
• Politics, Law, Judiciary
• Post Modern/Critical Politics
• Public Opinion
• Qualitative Political Methodology
• Quantitative Political Methodology
• World Politics
• Share This Facebook LinkedIn Twitter

Article contents

Rediscovering reputation through theory and evidence.

• Mark J. C. Crescenzi Mark J. C. Crescenzi Department of Political Science, University of North Carolina at Chapel Hill
•  and  Bailee Donahue Bailee Donahue Department of Political Science, University of North Carolina at Chapel Hill
• https://doi.org/10.1093/acrefore/9780190228637.013.282
• Published online: 28 June 2017

Reputation as it applies to the arena of international relations is information adhering to a state or its leaders about behavioral or intentional characteristics relating to cooperation or conflict. The study of reputation in world politics has waxed and waned in recent decades, but is enjoying a renaissance both in terms of theoretical and empirical analysis. We review the origins of the study of reputation in world politics, as well as the post-Cold War context that contributed to reputation’s apparent demise. We then focus on the recent rediscovery of reputation through the development of new theoretical and empirical analyses. These works have overcome earlier challenges to the conceptualization and measurement of reputation to improve our understanding of how this phenomenon affects coordination, cooperation, and conflict among and between states in the international arena.

• world politics
• cooperation
• coordination
• empirical international relations theory

Introduction

Reputation as it has been applied in the study of world politics is a term used to describe information adhering to a state or its leaders about behavioral or intentional characteristics relating to cooperation or conflict. States often obtain reputations for some defining quality that others use to predict the state’s actions. For example, a state can develop a reputation for being trustworthy, or reliable, and this information may improve its ability to enter into contracts such as alliances with other states. States can also develop reputations for being aggressive or untrustworthy, which can exacerbate international crises and increase the chance of war. In its purest sense, reputation can also be an end to its own, and states can cultivate reputations for being honorable or develop high status. As a source of information, reputation is gleaned from past interactions, contracts, or statements, often but not always involving other states in the global arena.

Reputation in and of itself can be considered a desirable attribute (especially when dealing with concepts such as status or honor), but it is frequently tied to a desire to signal an attributional quality. This information is then used by others to predict intentions and behavior. The empirical veracity of this assumption, however, has been contested in previous research. In the immediate aftermath of the Cold War, new scholarship called the conventional wisdom surrounding reputation into question. As a result, the concept of reputation as it relates to world politics developed a reputation of its own as fool’s gold. Only recently have scholars returned to the idea that reputations not only exist but can systematically influence political outcomes in international relations. Through innovations in theoretical modeling, research design, and data collection, this new scholarship has overcome many of the issues that reputation’s critics identified in earlier work. In this article we trace the evolution of the study of reputation’s role in world politics, focusing on the importance of the theoretical and empirical developments that have enabled new scholarship in recent years to improve our understanding of this elusive but important phenomenon.

Reputation has historically enjoyed a prominent position in the study of international relations. From Thucydides’ historiography of the Peloponnesian War, reputation was cast as a central component of the motivation for war. Much later, when Schelling crafted his works on strategic bargaining in world politics, reputation played a central part in establishing credible threats in pursuit of peace through deterrence. Throughout the Cold War, reputation was considered to be an essential component of credible deterrence.

Readers will undoubtedly recognize elements of the following discussion in the articles of this volume concerning bargaining theory and deterrence. This is because, like power, reputation is a means to an end, and as such it is often conflated with those ends. Also like power, reputation is observationally elusive and difficult to measure. Reputation is also linked to similar concepts, such as resolve, face, honor, status, and even identity. This conceptual ambiguity makes the systematic study of reputation challenging.

As the end of the Cold War shook the rafters of academic research, scholars began to notice that the concept of reputation in world politics bore a resemblance to Hans Christian Anderson’s classic tale, The Emperor’s New Clothes . As Hopf ( 1998 ) pointed out, many of the theoretical arguments espousing reputation’s importance had reached those conclusions based on assumptions rather than evidence. The disjuncture between reputation’s role in theory and our inability to empirically observe reputation at work led to a crisis in the study of reputation. Scholars such as Hopf ( 1998 ), Mercer ( 1996 ), and Press ( 2005 ) emphasized the importance of perceptions and socially constructed contexts surrounding reputations, concluding reputation’s influence on deterrence to be elusive and mercurial. Suddenly the role of reputation in world politics was in doubt.

In recent years, however, reputation has made a comeback. This resurgence appears to be a result of a confluence of three theoretical and empirical developments. First, the role of reputation in the phenomena of cooperation and coordination did not suffer the same existential crisis as in conflict and deterrence. As the political economy literature continued to rely on reputation as a mechanism for coordination, particularly in repeated interaction contexts, this research began to diffuse back into studies of conflict processes. Second, new theories of reputation emerged from scholars such as Crescenzi ( 2007 , 2017 ), Kydd ( 2005 ), Sartori ( 2005 ), Sechser ( 2010 ), Tomz ( 2007b ), Walter ( 2009 ), and Yarhi-Milo ( 2014 ). These new theoretical treatments shed the constraints of focusing only on deterrence and credible threats, and this broader treatment of reputation produced important new empirical implications for study. Third, the resurgence of behavioral economics and its application to the study of world politics had opened up opportunities to use experimental research designs to study reputation. These innovations improve our ability to match up empirical evidence with theoretical arguments.

Classic Theories and Their Critics

Regarding the question of reputation and international war, classic theories and their critiques are characterized by their focus on a state’s use of reputation as a mechanism for conveying information about resolve. This resolve is either with respect to the willingness to fight for one’s honor, as in the case of Thucydides, or the willingness to punish, as in the case of Schelling ( 1960 , 1966 ) and Huth ( 1988 ). This notion of resolve is a key component of deterrence theory, a vast body of work that overlaps with theories of reputation. Critiques of the notion that resolve could be conveyed through reputation focused on the intersubjective context of resolve that causes resolve to be too specific to transform into reputational information.

In two landmark works, Schelling ( 1960 , 1966 ) presented what would quickly become the cornerstone for the study of reputation in the modern emergence of the field of international relations. In The Strategy of Conflict ( 1960 ), Schelling explored how actors can construct credible threats in bargaining. According to Schelling, threats are meant to deter an adversary from a course of behavior that the issuer of the threat has deemed incompatible with her incentives. One dynamic method of credibly threatening an adversary is to stake one’s reputation on fulfilling the threat and ensuring that future bargains will be negatively impacted by defection in the present. Six years later, in Arms and Influence , Schelling noted that one of the few things in the world worth fighting for is “face.” Face, he argued, is a combination of the state’s reputation for taking action and the expectations other countries have of its behavior. This observation became prevalent in the literature as a fundamental component of the study of reputation in international relations and a key component of U.S. foreign policy throughout the Cold War.

Much of the empirical support for reputation in this classical framework is linked to the study of deterrence. Two of the most important studies, Huth and Russet ( 1984 ) and Huth ( 1988 ), provided a robust set of tests concerning the question of whether deterrence works, as well as reputation’s role in that success. Huth and Russet developed an expected utility model of immediate deterrence to determine under what circumstances deterrence is successful. They find that immediate deterrence is most likely to be successful when economic and political-military ties are strongest between the defender state and the state that is being threatened. In contrast to classical deterrence theory, they find that the defender state’s previous behavior has little bearing on the probability of successful deterrence, which suggests that a state’s reputation may not be as important in successful deterrence.

Four years later, however, Huth ( 1988 ) followed up this research with a major study that established a clear link between a state’s reputation for following through with deterrence commitments and the success and failure of future deterrence events. In this analysis, Huth studied 58 cases of extended-immediate deterrence to determine under what political and military conditions this form of deterrence will succeed or fail. Huth found that a past record of backing down or intransigence in confrontations with attackers increases the likelihood of deterrence failure. In these instances, a reputation for a lack of resolve or aggression increases the likelihood for deterrence to fail.

As the Cold War ended and sent reverberations of change throughout the world as well as those who study it, reputation did not escape introspection. Just as Schelling anchored the classic models that took reputation as a given component of the bargaining and deterrence processes, Mercer ( 1996 ) anchored the challenge against reputation as a systematic and predictable influence in world politics. Mercer theorized on the role that cognitive biases play in the construction of reputation. Mercer argued that altering a reputation in the eyes of a state’s allies and enemies is very difficult. Contrary to decades of foreign policy prescription, Mercer argued that reputations may not be worth fighting for.

Mercer’s arguments echo earlier cracks in the conventional wisdom surrounding reputation. In an important critique of classical deterrence theory, Hopf ( 1994 ) analyzed the lessons learned by the Soviets in various episodes of engagement between the United States and the Soviets from 1965 to 1990 by analyzing statements made by Soviet foreign policy-makers and academics. He was intrinsically skeptical of reputation as a meaningful construct, pointing out that all too often reputation seems to matter in theoretical analyses because reputation is assumed to do so in the first place. Hopf suggested that the fundamental principle of deterrence theory remains valid, however. If a state seeks to prevent an adversary from obtaining a piece of territory, it is important to clearly communicate the state’s resolve and capabilities. Hopf argued that the scope of expressing resolve and capability needs to be expanded to include the plethora of deterrent instruments available to a state rather than simply focusing on military tools. Perhaps most vexing is Press’ ( 2005 ) analysis that set out to find the role of past actions in decision-making and found none. Press argued that, rather than assessing their adversaries’ credibility by looking at past behaviors of commitment and credibility, policymakers are focused on the present crisis and do not consider the past when selecting a policy. According to Press, reputation does not play a crucial role in crisis behavior. Instead, policymakers are swamped by information that is uniquely important in the time and space of the crisis at hand. In essence, in times of crisis, considering the reputation of an adversary becomes a luxury that leaders and states cannot afford.

These criticisms of reputation are not without critiques of their own. In his critique of Mercer’s book, Copeland ( 1997 ) identified inconsistencies in Mercer’s logic. Chief among these inconsistencies is that reputations for resolve may be hard to change not because these reputations are hard to form but rather these reputations are imbedded in the labels that states have already given each other. Whether a state is seen as an ally or as a foe matters for that state’s reputation. Copeland moved from his critique to create a potential causal framework to understand how reputations become established between actors and to suggest that, in future research, scholars should specify under what conditions certain kinds of reputations form.

Huth ( 1997 ) also pushed back on the demise of reputation’s role in deterrence. He noted that there is a significant gap between the intuition that reputations are an important cause of interstate conflict and the paucity of data suggesting that reputations are consequential. He argued that the theory of reputation in deterrence has not been adequately proven with rigorous empirical tests. He then laid out an agenda for further research.

Overall, however, critics of reputation as an influential process in the phenomena of war revealed an important set of problems that hindered our ability to isolate the effects of reputation. Perhaps because of the parallels between reputation and identity, scholars keyed in on the situational social construction of reputation. Whereas empirical analyses were treating reputation as a common attribution, critics emphasized the situational complexity that clung to reputational information like baggage. Ultimately, the question of whether reputations matter remained unresolved, and the study of reputation in conflict stagnated.

Classic Theory of Reputation and Cooperation

Reputation also serves as a mechanism for cooperation and coordination, particularly in situations where there exists an expectation of repeated interaction. It is interesting to note that in the arena of cooperation and coordination, reputation as a contributing factor did not suffer the same existential crisis that it did in the study of conflict. Instead of being conflated with resolve and status, in the study of cooperation the notion of reputation is more akin to trust, reciprocity, and precedent. As a result, reputation has enjoyed a sustained presence in the study of coordination in repeated interaction contexts. Eventually this research diffused over to the study of conflict in repeated interaction contexts as well, aiding the resurgence of studies in the conflict processes field.

In the study of coordination, the focus is less specific than a mechanism for resolve. For example, Axelrod ( 1984 ) showed how reputation organically emerges from the combination of being able to observe past behavior and expectations of future interaction. In this groundbreaking study of cooperation under anarchy, Axelrod found that future interactions between actors help establish cooperation among egoists. The prospect of having to interact with an actor again can loom on the present interaction and change the outcome of the current interaction such that cooperation becomes a more likely strategy. Similarly, the practice of reciprocity is enabled by reputations for cooperation. Axelrod and Keohane ( 1985 ) provided a formative analysis of cooperation under the condition of anarchy, citing reputation as a key mechanism for ensuring continued cooperation. International regimes, they argued, make it easier to build a reputation for practicing reciprocity. Reputations become important assets to states because states are more willing to make agreements with states that respond to cooperation in kind, creating an efficient escape from the prisoner’s dilemma.

In the arena of sovereign debt, past repayments may convey a state’s reputation for its ability or willingness to repay new debts in the future. Eaton and Gersovitz ( 1981 ) developed a theory of borrowing by poor countries in private markets. Based on this theory, they developed an econometric model of borrowing by poor countries in the international financial market where default results in a future loss of access to international debt markets. They determined that states are incentivized to repay debts in the present due to concerns for future access to borrowing to smooth consumption in the future. Thus, maintaining a good reputation for repayment is essential in maintaining access to lending. Bulow and Rogoff ( 1989 ) took issue with the reputational approach to understanding sovereign debt repayment practices, however, because this approach requires that no actors will sell financial assets to the debtor in default. This assumption is not supported empirically, as past repayment record has little bearing on a country’s ability to borrow. Instead, Bulow and Rogoff argued that the primary motivation for repayment is direct sanctions imposed by lenders on debtor countries. They developed a dynamic model of international lending that is less dependent on reputation and repayment risk. Rogoff ( 1987 ) provided an excellent overview of reputational models of monetary policy. He argued that reputational considerations can reasonably constrain the government’s incentives to inflate the currency. Rogoff noted that the models of reputation in monetary policy are sensitive to changes in information structure and have a substantial number of equilibria. He concluded that that it is plausible that reputational constraints will prevent governments from partaking in episodes of surprise inflation but that the sensitivity of the models under review supplants issues of cooperation with issues of coordination.

Tomz ( 2007a ) supplied an important resolution to the debate over reputation’s role in sovereign debt, and identified reputation as a key mechanism for lending and repayment in sovereign debt. In his novel study, Tomz developed a reputational theory of sovereign debt. Using data spanning three centuries, Tomz found that reputation has been a key component to lending and repayment. Reputation, in turn, is an important enforcement mechanism in continued cooperation between states in the international debt market.

New Theories of Reputation

One of the ways in which the study of reputation became revitalized was through the development of new theories linking reputation to political decisions and behavior. For example, Simmons ( 2000 ) provided a cornerstone for the study of reputation in compliance and commitments in international law, while Downs and Jones ( 2002 ) thought theoretically about the role of reputation in treaty compliance. From a political economy perspective, Mailath and Samuelson ( 2006 ) provided an expanded analysis of reputation’s role in repeated games and long-run relationships. Others honed in on mid-level theorizing in more specific political contexts. As a result, scholars are now able to identify reputational effects outside of the contexts of deterrence or repeated play prisoner’s dilemma models. Here we review some of the new theoretical contributions in the study of conflict processes, international institutions, and international political economy.

Reputation, Status, and International Conflict

One of the most interesting theoretical innovations deals with the reputations of leaders. Wolford ( 2007 ) developed a new theoretical analysis of leader reputations within the framework of leader tenure. His dynamic analysis of the changing information needs of leaders revealed an intrinsic motivation for leaders to build their reputations early in their tenure. Wolford built a game theoretic model of leadership turnover and international conflict. He assumed that since successive leaders of a state can vary in their willingness to use force, incumbents have an incentive to build a reputation for resolve early on to improve their bargaining outcomes over time. Antagonists, in turn, have an incentive to test a new leader’s resolve through crisis bargaining. Gibler ( 2008 ) also investigated the role of reputation at the leader level of analysis. He examined whether a state’s past behavior in alliances, upholding or defecting from alliance commitments, has an impact on future alliance commitments and conflicts. Gibler demonstrated that alliance reputations do affect the formation of alliances and the behavior of the alliance partners in disputes.

At the same time, new theories emerged at the state level. Crescenzi ( 2007 , 2017 ) theorized that the situational contexts that surround reputation are conditioned by temporal and relational dynamics. He presented a model of reputational learning among states and examined how this learning affects international conflict. He then tested whether states are more likely to experience conflict if at least one of the states has a reputation for hostility. Moreover, he used an exponential decay function to model the diminishing effects of history on reputation over time. Crescenzi found that information outside of the dyadic interaction between the two states has a significant impact on the likelihood for conflict, but that this reputational information degrades over time and is less important for dyads once they have established a direct history of interaction. In subsequent analyses, Crescenzi, Kathman, and Long ( 2007 ) and Crescenzi ( 2017 ) expanded this study to show that a reputation for hostility also increases the likelihood for escalation to war.

Similarly, Weisiger and Yarhi-Milo ( 2015 ) theorize that reputation has a dynamic impact over time. The authors critiqued recent work on reputations dealing with resolve that suggest that reputations do not form and that past actions do not influence behavior in future interactions. The authors developed a theory that suggests that reputation should decline over time, they tested their hypothesis by developing a decay function and found that past actions do in fact inform reputations. Wiegand ( 2011 ) focused on the use of reputation to transfer resolve from one situation to another. She developed a theory for why states involved in territorial disputes are more likely to initiate militarized interstate disputes (MIDs) when compared to states that are involved in other types of interstate disputes. States sometimes use MIDs not only to signal resolve to the opposing state but also to signal resolve to other states in conflict with the initiator state.

Walter ( 2009 ) anchored a new focus on the role of reputation in internal, civil conflicts. Walter developed an information argument based on the logic of the chain store paradox that highlights the need for states to respond to internal challenges with future potential challenges in mind. Thus, even when it appears that an individual challenge should be accommodated by the state, the need to preserve a reputation for intransigence may result in a lack of accommodation by the state. Walter provided evidence that reputational concerns strongly influence state behavior in separatist conflicts when the state thinks that it may face additional separatist challengers in the future.

New theories also emerged with a focus on the role of reputation in threats and crisis situations.

In contrast to previous research that links credibility to reputations for resolve, Guisinger and Smith ( 2002 ) argued that credibility is linked to the gains made by having a reputation for honesty in diplomacy. The authors suggested that when a reputation for honesty resides within a leader rather than the country, citizens will have an incentive to remove leaders caught bluffing. Guisinger and Smith developed a game theoretic model to look at the interaction. Similarly, Sartori ( 2005 ) expanded the scope of classical deterrence theory to look at the value placed on the importance of reputation and of honesty in establishing effective diplomatic relations. Being honest most of the time allows states to maintain reputations for honesty, which in turn improves the state’s ability to demonstrate resolve in future disputes. Sartori note that there is a drawback: sometimes a state will acquiesce in a crisis when they may have been able to successfully bluff.

Sechser ( 2010 ) argued that reputation concerns can cause compellent threats to fail when they would otherwise be expected to succeed. In his study, Sechser attempted to address why compellent threats issued against weaker adversaries often fail. Sechser argued that reputation can be viewed as a strategic problem and that challengers should anticipate the reputation costs associated with appeasing an aggressor. A challenger that recognizes the costs associated with appeasement may be able to offer side payments to mitigate these costs. The argument is then illustrated by a case study of the Russo-Finnish crisis in 1939 . In a subsequent analysis, Sechser ( 2016 ) analyzed when states choose to pay high costs to protect their reputations and when they are willing to tarnish that reputation. He argued that states value their reputations less and are more willing to concede to coercive threats when the state does not expect a subsequent challenge. Using the Militarized Compellent Threats data set, Sechser found support for his theory.

Important progress has also occurred in the study of honor, status, and face and conflict. Clare and Danilovic ( 2012 ) conducted an empirical test of the logic of Schelling’s famous argument that “face” is worth fighting for. Clare and Danilovic examined the interaction between reputation and interests when states initiate challenges in crisis bargaining. They find that the role of reputation is mediated by a state’s interests when initiating challenges. Dafoe and Caughey ( 2016 ) focused even more explicitly on status, reputation, and resolve. They analyzed the role that cultural origins of leaders play in heightening concern for reputations for resolve. They analyzed the role that the culture of honor in the American South plays in Southern U.S. presidents’ conflict behavior. They found that international conflicts under Southern presidents are more likely to result in the use of force, last longer, and are more likely to end in victory than those conflicts under non-Southern presidents.

Reputation and International Cooperation

The study of reputation has expanded in the fields of international institutions and international political economy as well. Progress in the field of international institutions has centered on reputation as a signal for compliance. Simmons ( 2000 ) investigated why governments make international legal commitments. Focusing her inquiry on international monetary law, she argued that reputational concerns explain patterns of compliance with international law. She found that governments are more likely to commit and comply with legal obligations if other countries in the region do. Guzman ( 2008 ) argued that reputation serves as an enforcement mechanism for compliance with international law. In his book, Guzman created a rational choice theory of how international law can be a sustained institution without an enforcement mechanism. A state’s concern over a good reputation for compliance with international law is a key mechanism for sustaining cooperation. Guzman further argued that a state may have several competing reputations for compliance in different issue areas. The importance of maintaining a reputation for compliance in the realm of international law may also vary based on the dyad and on the regime.

Carnegie and Dolan ( 2016 ) argued that international assistance can undermine a government’s reputation by making it appear weak. Governments will at times reject aid when they are able to send a credible signal, when they care very little for assisting the affected citizens, and when rejecting aid is important for maintaining status. Carnegie and Dolan tested their hypotheses empirically using an original data set on political response to natural disasters.

Reputation can also impact sanctions behavior. Peterson ( 2013 ) argued that a state that is the target of sanctions will look to the previous response of the state imposing sanctions to states that were resistant. When a sender state has recently backed down, the sender state is more likely to face resistance. Peterson then examined U.S. sanction threats spanning from 1971 to 2000 and found support for his theory. Miller ( 2014 ) noted that there is a selection effect at work in the imposition of nonproliferation sanctions by the United States: only states that are not effectively deterred by the threat of sanctions pursue nuclear proliferation. This selection effect results in mixed evidence regarding the effectiveness of nonproliferation sanctions. The United States continues to sanction those states that are not deterred to maintain a reputation for sanctioning to ensure that other states are deterred. Miller tested his hypotheses using both qualitative case studies and quantitative methods.

Finally, new theories help explain the dynamics of reputation and coordination in international institutions, nongovernmental organizations, and international law. Gent, Crescenzi, Menninga, and Reid ( 2015 ), for example, strategically modeled the importance and consequences of reputation in the interactions between nongovernmental organizations (NGOs) and their donors. They analyzed how concerns over reputation can lead to NGOs' altering their behavior such that it becomes detrimental to achieving policy goals. This focus allowed Gent et al. to capture the strategic interaction between donors and NGOs where donors select those NGOs that have a reputation for success, and NGOs are then incentivized to select projects that are more easily accomplished but not necessarily the most efficient for achieving the NGOs' goals.

In the arena of international institutions and contracts, scholars have demonstrated that reputations can affect not just the existence, but the quality, of international contracts. Jensen and Johnston ( 2011 ) argued that states suffer reputational costs when they fail to honor contracts but governments that have economies that are natural-resource-dependent are not as sensitive to these reputational costs. As a result, they argued, natural-resource-dependent states are more likely to expropriate risk and to have contract disputes. They used a measure of expropriation risk to test their theory. Similarly, Crescenzi and colleagues ( 2012 ) demonstrated that states with a reputation for upholding their alliance commitments over time are more likely to form alliances in future years with other states. A good reputation as an ally not only affects the likelihood of future alliances, but also affects the quality of the contracts as well. Mattes ( 2012 , 2015 ) used a tightly focused mix of theory and empirical analysis of alliance contracts to show that states with poor alliance behavior in the past are likely to face more stringent and constraining alliance contracts in the future.

Empirical Innovations

In this final section, some of the research is highlighted that has helped innovate the study of reputation in international relations through new developments in research design and data collection. The analyses provide creative approaches to solving one of the most difficult challenges to the study of reputation. That is, observing and measuring reputation outside of the theoretical realm and incorporating measures of reputation into empirical models of conflict and cooperation processes.

New Approaches to Data

One understudied approach is to use a network analysis framework to study reputation. Craik ( 2008 ) offered a network model of reputation at the individual level, albeit with a social psychological focus rather than focusing on political outcomes. Craik argued that the individual can have multiple ongoing reputations, and that reputation dynamics are best understood in a network framework. Scholars may be able to improve our understanding of how reputations spread or change across network ties, as well as when and how reputations reverberate within or across network structures.

New data sets also enable improved focus on the study of reputation. Sechser ( 2011 ) broke new ground with an ambitious data set on Militarized Compellent Threats (MCTs). By collecting data on threats as well as actions, Sechser unlocked the ability to examine the impact of threats and preconflict crisis behavior on state reputation. His work allows us to understand when large states may use compellent threats differently due to concerns about their reputations. Similarly, McManus ( 2014 ) created an original data set that measures the level of resolve of statements issued by U.S. presidents during MIDs. McManus found that a higher level of resolved statements is associated with favorable resolution of the disputes. Tokdemir and Akcinaroglu ( 2016 ) presented an innovative data set that identified the reputation of terror groups, which may help explain discrete actions within the context of terror campaigns.

New data have also been useful in the study of international institutions and economic behavior. Two major new data sets have led to innovative research on the study of reputation in alliances and economic sanctions. Leeds, Ritter, Mitchell, and Long ( 2002 ) collected detailed and important information on alliances in world politics. Scholars such as Crescenzi, Kathman, Kleinberg, and Wood ( 2012 ), Crescenzi ( 2017 ), and Mattes ( 2012 ) used the data to test arguments concerning the impact of reputation on alliance formation and alliance contract qualities. Similarly, Morgan, Bapat, and Krustev ( 2009 ) introduced an innovative new data set that identified the threats as well as impositions of economic sanctions. By adding the essential first step of the threat stage, the data allowed Peterson ( 2013 ) to examine when states follow through with their threats and how a reputation for following through impacts the success of future sanctions threats.

Experiments

With the resurgence of behavioral economics in the social sciences, and the experimental research design that often comes with this approach, scholars have recently leveraged experiments to test hypotheses involving reputation as a causal factor in decision-making. Two lines of research established a platform for this research as it has been applied to questions of cooperation and crisis. First, Tomz ( 2007a , 2007b , 2008 ) established an innovative experimental approach to examining reputation in international political economy. Recent studies of the credibility of international threats have suggested that domestic audience costs are a mechanism that ensures the credibility of a threat. In his article, Tomz developed an experiment embedded in public opinion surveys to study the dynamics of domestic audience costs. He found that audience costs may arise because people are concerned with the international reputation of the country or leader. He then complemented this research with interviews with U.S. voters and British policymakers regarding preferences and expectations over international law. He found that individuals are more likely to oppose otherwise identical policies if they violate international law. The concerns with violating international law that individuals have are partially a result of worry over a loss in reputation.

Tingley and Walter ( 2011 ) provided the other anchor to the use of experiments in empirical analyses of reputation. Tingley and Walter focused on the notion of reputation-building as a purposive strategy. Rather than assuming that reputations simply emerge and asking the question of how they matter, they compellingly argued that states seek out reputations and will invest in obtaining reputations that serve their interests. The incentives for reputation-building were analyzed through a series of laboratory experiments. The paper uses comparative statistics from a repeated entry-deterrence game to isolate how incentives for reputation-building may change as the number of entrants changes. This revival of Schelling’s intuition had been difficult to assess empirically until this research unlocked a new avenue for investigation.

New scholarship has taken advantage of this platform and launched a series of important experimental studies linking reputation to both compliance and conflict. Gray and Hicks ( 2014 ) conducted a survey experiment to evaluate the effect of international agreements on a country’s reputation when few prior beliefs are held regarding the country. They found that joining in international agreements with countries that have good reputations will result in a reputation for less risk, whereas entering into agreements with countries with bad reputations will result in a higher risk assessment. Similarly, Fjelstul, Weeks, Tomz, and Reiter ( 2015 ) analyzed the domestic audience costs faced by leaders violating military alliance agreements. The authors argued that audience costs are higher for those leaders who violate general alliances rather than defensive alliances but that the difference in audience costs should vanish if the ally is a victim of aggression. Their hypotheses were then tested using an original survey experiment.

Beyond coordination problems, experimental research designs are informing our understanding of reputation’s role in international crisis behavior. Kertzer ( 2016 ) creatively unpacked the state to reveal how domestic audiences focus on reputation and resolve. In his book, Kertzer addressed why some leaders display resolve in confrontations while others do not. This study combined laboratory and survey experiments with studies of great power military intervention to develop a theory to explain the ways leaders and members of the public define situations and the trade-offs between the cost of fighting and the costs of backing down based on risk preferences, honor orientation, and self-control.

Last, new studies focus on the notion and importance of status. Renshon ( 2015 ) examined the interaction between status and the judgment of political and military leaders. In his study of the microfoundations of status, Renshon conducted an experiment where status concerns are randomly assigned prior to a task where individuals are tasked with increasing their commitment to winning or cutting their losses. Renshon conducted this test on a sample of political and military leaders as well as a series of control subjects. He found that power is an important mitigating factor in concerns for status. Those who have a high power mindset are less likely to be concerned with status. In his work with Dafoe and Huth, Renshon presents a theory of influence-specific reputations. Renshon and colleagues ( 2018 ) argue that reputations will adhere more to actors who have greater influence in the relevant decision-making prices. They find evidence of country-specific reputations being developed in this manner as well as leader-specific reputations. The authors deploy two survey experiments to test their theory.

Conclusions

As we reflect on this revitalization, it is apparent that the progress is due to three factors. First, scholars are able to build off of more sophisticated theoretical platforms to establish clear and focused arguments concerning reputation in world politics. Gent and colleagues ( 2015 ), for example, used the theoretical progress established in Mailath and Samuelson ( 2006 ) to investigate reputation’s influence on NGO behavior. Experimental designs such as Kertzer ( 2016 ) or Renshon ( 2015 ) are encouraged by the resurgence of behavioral economics both substantively and in research design. As such, this new knowledge concerning reputation is a product of a cumulative and maturing science.

Second, new data generation has enabled scholars to investigate the impact of reputation in mid-range phenomena, such as sanctions, alliances, or state–rebel group negotiations. Important data sets such as Leeds and colleagues’ ( 2002 ) ATOP dataset on alliance behavior, Morgan and colleagues’ ( 2009 ) data set on economic sanctions and threats of sanctions, and Sechser’s ( 2011 ) data on militarized compellent threats all provide resources that others have utilized to test new and focused arguments concerning the role of reputation.

Third, this progress was made possible in part by the ability of scholars to set aside the unique and complicated context of deterrence theory and the role of reputation in threat credibility in deterrence. The highly specific contexts of deterrence crises lend themselves to the critique that Mercer ( 1996 ) made when he argued that reputations are too specific to be useful in different crisis contexts. By stepping back and putting less weight on the role of reputation to be the critical determinant of peace versus war, scholars have improved our understanding of this elusive phenomenon.

New applications are quickly emerging, such as Mitton’s ( 2015 ) analysis of Schelling’s reputation arguments applied to the role of the United States in the current conflict in Syria, or Joshi and Quinn’s ( 2016 ) research analyzing the reputations states earn by keeping (or breaking) promises with rebel groups. As we continue to hone our understanding of when and how reputation matters in world politics, we can only expect this revitalization of research to continue.

• Axelrod, R. (1984). The evolution of cooperation . New York: Basic Books.
• Axelrod, R. , & Keohane, R. O. (1985). Achieving cooperation under anarchy: Strategies and institutions. World Politics , 38 (1), 226–254.
• Bulow, J. , & Rogoff, K. (1989). A constant recontracting model of sovereign debt. Journal of Political Economy , 97 (1), 155–178.
• Carnegie, A. , & Dolan, L. (2016). The effects of aid on recipients’ reputations: Evidence from natural disaster responses . Unpublished manuscript.
• Clare, J. , & Danilovic, V. (2012). Reputation for resolve, interests, and conflict. Conflict Management and Peace Science , 29 (1), 3–27.
• Copeland, D. C. (1997). Do reputations matter? Security Studies , 7 , 33–71.
• Craik, K. H. (2008). Reputation: A network interpretation : Oxford: Oxford University Press.
• Crescenzi, M. J. C. (2007). Reputation and interstate conflict. American Journal of Political Science , 51 (2), 382–396.
• Crescenzi, M. J. C. (2017). Of friends and foes: Reputation and learning in world politics . Oxford: Oxford University Press.
• Crescenzi, M. J. C. , Kathman, J. D. , & Long, S. B. (2007). Reputation, history, and war. Journal of Peace Research , 44 (6), 651–667.
• Crescenzi, M. J. C. , Kathman, J. D. , Kleinberg, K. B. , & Wood, R. M. (2012). Reliability, reputation and alliance formation. International Studies Quarterly , 56 (2), 259–274.
• Dafoe, A. , & Caughey, D. (2016). Honor and war. World Politics , 68 (2), 341–381.
• Downs, G. W. , & Jones, M. J. (2002). Reputation, compliance and international law. Journal of Legal Studies , 31 (1), 95–114.
• Eaton, J. , & Gersovitz, M. (1981). Debt with potential repudiation: Theoretical and empirical analysis. The Review of Economic Studies , 48 (2), 289–309.
• Fjelstul, J. , Weeks, J. , Tomz, M. , & Reiter, D. (2015). Alliance terms and audience costs: An experimental study of the microfoundations of alliance compliance . Unpublished manuscript.
• Gent, S. E. , Crescenzi, M. J. , Menninga, E. J. , & Reid, L. (2015). The reputation trap of NGO accountability. International Theory , 7 (3), 426–463.
• Gibler, D. M. (2008). The costs of reneging reputation and alliance formation. Journal of Conflict Resolution , 52 (3), 426–454.
• Gray, J. , & Hicks, R. P. (2014). Reputations, perceptions, and international economic agreements. International Interactions , 40 (3), 325–349.
• Guisinger, A. , & Smith, A. (2002). Honest threats: The interaction of reputation and political institutions in international crises. Journal of Conflict Resolution , 46 (2), 175–200.
• Guzman, A. (2008). How international law works: A rational choice theory . Oxford: Oxford University Press.
• Hopf, T. (1994). Peripheral visions: Deterrence theory and American foreign policy in the third world, 1965–1990 . Ann Arbor: University of Michigan Press.
• Hopf, T. (1998). The promise of constructivism in international relations theory. International Security , 23 (1), 171–200.
• Huth, P. (1988). Extended deterrence and the outbreak of war. The American Political Science Review , 82 (2), 423–443.
• Huth, P. (1997). Reputations and deterrence: A theoretical and empirical assessment. Security Studies , 7 (1), 72–99.
• Huth, P. , & Russet, B. (1984). What makes deterrence work? Cases from 1900 to 1980. World Politics , 36 (4), 496–526.
• Jensen, N. M. , & Johnston, N. P. (2011). Political risk, reputation and the resource curse. Comparative Political Studies , 44 (6), 662–688.
• Joshi, M. , & Quinn, J. M. (2016). Watch and learn: Spillover effects of peace accord implementation on non-signatory armed groups. Research and Politics , 3 (1), 1–7.
• Kertzer, J. D. (2016). Resolve in international politics . Princeton, NJ: Princeton University Press.
• Kydd, A. H. (2005). Trust and mistrust in international relations . Princeton, NJ: Princeton University Press.
• Leeds, B. A. , Ritter, J. , Mitchell, S. , & Long, A. (2002). Alliance treaty obligations and provisions, 1815–1944. International Interactions , 28 (3), 237–260.
• Mailath, G. J. , & Samuelson, L. (2006). Repeated games and reputations: Long-run relationships . Oxford: Oxford University Press.
• Mattes, M. (2012). Reputation and alliance design. International Organizations , 66 (4), 679–707.
• Mattes, M. (2015). Democratic reliability, precommitment of successor governments, and the choice of alliance commitment. International Organization , 66 (1), 153–172.
• McManus, R. W. (2014). Fighting words: The effectiveness of statements for resolve in international conflict. Journal of Peace Research , 51 (6), 726–740.
• Mercer, J. (1996). Reputation and international politics . Ithaca, NY: Cornell University Press.
• Miller, N. L. (2014). The secret success of nonproliferation sanctions. International Organization , 64 (4), 913–944.
• Mitton, J. (2015). Selling Schelling short: Reputations and American coercive diplomacy after Syria. Contemporary Security Policy , 36 (3), 408–431.
• Morgan, T. C. , Bapat, N. , & Krustev, V. (2009). The threat and imposition of economic sanctions, 1971–2000. Conflict Management and Peace Science , 26 (1), 92–110.
• Peterson, T. M. (2013). Sending a message: The reputation effect of U.S. sanction behavior. International Studies Quarterly , 57 , 672–682.
• Press, D. G. (2005). Calculating credibility: How leaders assess military threats . Ithaca, NY: Cornell University Press.
• Renshon, J. (2015). Losing face and sinking costs: Experimental evidence on the judgment of political and military leaders. International Organization , 69 (3), 659–695.
• Renshon, J. , Dafoe, A. , & Huth, P. (2018). Leader influence and reputation formation in world politics.. American Journal of Political Science , 62 (2), 325–339.
• Rogoff, K. (1987). Reputational constraints on monetary policy . Paper presented at the Carnegie-Rochester Conference Series on Public Policy, North-Holland.
• Sartori, A. E. (2005). Deterrence by diplomacy . Princeton, NJ: Princeton University Press.
• Schelling, T. (1960). The strategy of conflict . Cambridge, MA: Harvard University Press.
• Schelling, T. (1966). Arms and influence . New Haven, CT: Yale University Press.
• Sechser, T. S. (2010). Goliath’s curse: Coercive threats and asymmetric power. International Organization , 64 (4), 627–660.
• Sechser, T. S. (2011). Militarized compellent threats, 1918–2001. Conflict Management and Peace Science , 28 (4), 377–401.
• Sechser, T. S. (2016). Reputation and signaling in coercive bargaining Journal of Conflict Resolution . Advance online publication.
• Simmons, B. A. (2000). International law and state behavior: Commitment and compliance in international monetary affairs. American Political Science Review , 94 (4), 819–835.
• Tingley, D. H. , & Walter, B. F. (2011). The effect of repeated play on reputation building: An experimental approach. International Organization , 65 (2), 343–365.
• Tokdemir, E. , & Akcinaroglu, S. (2016). Reputation of terror groups dataset measuring popularity of terror groups. Journal of Peace Research , 53 (2), 268–277.
• Tomz, M. (2007a). Domestic audience cost in international relations: An experimental approach. International Organization , 61 (4), 821–840.
• Tomz, M. (2007b). Reputation and international cooperation . Princeton, NJ: Princeton University Press.
• Tomz, M. (2008). Reputation and the effect of international law on preferences and beliefs . Unpublished manuscript.
• Walter, B. F. (2009). Reputation and civil war: Why separatist conflicts are so violent . Cambridge, U.K.: Cambridge University Press.
• Weisiger, A. , & Yarhi-Milo, K. (2015). Revisiting reputation: How past actions matter in international politics. International Organizations , 69 (2), 473–495.
• Wiegand, K. E. (2011). Militarized territorial disputes: States’ attempts to transfer reputation for resolve. Journal of Peace Research , 48 (1), 101–113.
• Wolford, S. (2007). The turnover trap: New leaders, reputation, and international conflict. American Journal of Political Science , 51 (4), 772–788.
• Yarhi-Milo, K. (2014). Knowing the adversary: Leaders, intelligence, and assessment of intentions in international relations . Princeton, NJ: Princeton University Press.

Related Articles

• Negotiation in Foreign Policy
• Compliance in International Relations

Printed from Oxford Research Encyclopedias, Politics. Under the terms of the licence agreement, an individual user may print out a single article for personal use (for details see Privacy Policy and Legal Notice).

date: 09 September 2024

• Cookie Policy
• Privacy Policy
• Legal Notice
• Accessibility
• [185.66.14.236]
• 185.66.14.236

Character limit 500 /500

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 04 September 2024

How to avoid sinking in swamp: exploring the intentions of digitally disadvantaged groups to use a new public infrastructure that combines physical and virtual spaces

• Chengxiang Chu 1   na1 ,
• Zhenyang Shen 1   na1 ,
• Hanyi Xu 2   na1 ,
• Qizhi Wei 1 &
• Cong Cao   ORCID: orcid.org/0000-0003-4163-2218 1  

Humanities and Social Sciences Communications volume  11 , Article number:  1135 ( 2024 ) Cite this article

Metrics details

• Science, technology and society

With advances in digital technology, physical and virtual spaces have gradually merged. For digitally disadvantaged groups, this transformation is both convenient and potentially supportive. Previous research on public infrastructure has been limited to improvements in physical facilities, and few researchers have investigated the use of mixed physical and virtual spaces. In this study, we focused on integrated virtual and physical spaces and investigated the factors affecting digitally disadvantaged groups’ intentions to use this new infrastructure. Building on a unified theory of the acceptance and use of technology, we focused on social interaction anxiety, identified the characteristics of digitally disadvantaged groups, and constructed a research model to examine intentions to use the new infrastructure. We obtained 337 valid data from the questionnaire and analysed them using partial least squares structural equation modelling. The results showed positive relationships between performance expectancy, perceived institutional support, perceived marketplace influence, effort expectancy, and facilitating conditions. The influence of psychological reactance was significantly negative. Finally, social interaction anxiety had a regulatory effect on performance expectancy, psychological reactance, perceived marketplace influence, and effort expectancy. Its effects on perceived institutional support and facilitating conditions were not significant. The results support the creation of inclusive smart cities by ensuring that the new public infrastructure is suitable for digitally disadvantaged groups. Meanwhile, this study presents new theoretical concepts of new public infrastructures, mixed physical and virtual spaces, which provides a forward-looking approach to studying digitally disadvantaged groups in this field and paves the way for subsequent scholars to explore the field in theory and literature.

Similar content being viewed by others

The impact of small-scale green infrastructure on the affective wellbeing associated with urban sites

Economic inequalities and discontent in European cities

The appeal of cities may not wane due to the COVID-19 pandemic and remote working

Introduction.

Intelligent systems and modernisation have influenced the direction of people’s lives. With the help of continuously updated and iteratively advancing technology, modern urban construction has taken a ‘big step’ in its development. As China continues to construct smart cities, national investment in public infrastructure has steadily increased. Convenient and efficient public infrastructure has spread throughout the country, covering almost all aspects of residents’ lives and work (Guo et al. 2016 ). Previously, public infrastructure was primarily physical and located in physical spaces, but today, much of it is virtual. To achieve the goal of inclusive urban construction, the government has issued numerous relevant laws and regulations regarding public infrastructure. For example, the Chinese legislature solicited opinions from the community on the ‘Barrier-free environmental construction law of the People’s Republic of China (Draft)’.

Virtual space, based on internet technology, is a major factor in the construction of smart cities. Virtual space can be described as an interactive world built primarily on the internet (Shibusawa, 2000 ), and it has underpinned the development of national public infrastructure. In 2015, China announced its first national pilot list of smart cities, and the government began the process of building smart cities (Liu et al. 2017 ). With the continuous updating and popularisation of technologies such as the internet of things and artificial intelligence (AI) (Gu and Iop, 2020 ), virtual space is becoming widely accessible to the public. For example, in the field of government affairs, public infrastructure is now regularly developed in virtual spaces, such as on e-government platforms.

The construction of smart cities is heavily influenced by technological infrastructure (Nicolas et al. 2020 ). Currently, smart cities are being developed, and the integration of physical and virtual spaces has entered a significant stage. For example, when customers go to an offline bank to transact business, they are often asked by bank employees to use online banking software on their mobile phones, join a queue, or prove their identities. Situations such as these are neither purely virtual nor entirely physical, but in fields like banking, both options need to be considered. Therefore, we propose a new concept of mixed physical and virtual spaces in which individuals can interact, share, collaborate, coordinate with each other, and act.

Currently, new public infrastructure has emerged in mixed physical and virtual spaces, such as ‘Zheli Office’ and Alipay, in Zhejiang Province, China (as shown in Fig. 1 ). ‘Zheli Office’ is a comprehensive government application that integrates government services through digital technology, transferring some processes from offline to online and greatly improving the convenience, efficiency, and personalisation of government services. Due to its convenient payment facilities, Alipay is continuously supporting the integration of various local services, such as live payments and convenient services, and has gradually become Zhejiang’s largest living service platform. Zhejiang residents can handle almost all government and life affairs using these two applications. ‘Zheli Office’ and Alipay are key to the new public infrastructure in China, which is already leading the world in terms of a new public infrastructure that combines physical and virtual spaces; thus, China provided a valuable research context for this study.

This figure shows the new public infrastructure has emerged in mixed physical and virtual spaces.

There is no doubt that the mixing of physical and virtual spaces is a helpful trend that makes life easier for most people. However, mixed physical and virtual spaces still have a threshold for their use, which makes it difficult for some groups to use the new public infrastructure effectively. Within society, there are people whose living conditions are restricted for physiological reasons. They may be elderly people, people with disabilities, or people who lack certain abilities. According to the results of China’s seventh (2021) national population census, there are 264.02 million elderly people aged 60 years and over in China, accounting for 18.7 per cent of the total population. China is expected to have a predominantly ageing population by around 2035. In addition, according to data released by the China Disabled Persons’ Federation, the total number of people with disabilities in China is more than 85 million, which is equivalent to one person with a disability for every 16 Chinese people. In this study, we downplay the differences between these groups, focusing only on common characteristics that hinder their use of the new public infrastructure. We collectively refer to these groups as digitally disadvantaged groups who may have difficulty adapting to the new public infrastructure integrating mixed physical and virtual spaces. This gap not only makes the new public infrastructure inconvenient for these digitally disadvantaged groups, but also leads to their exclusion and isolation from the advancing digital trend.

In the current context, in which the virtual and the real mix, digitally disadvantaged groups resemble stones in a turbulent flowing river. Although they can move forward, they do so with difficulty and will eventually be left behind. Besides facing the inherent inconveniences of new public infrastructure that integrates mixed physical and virtual spaces, digitally disadvantaged groups encounter additional obstacles. Unlike the traditional public infrastructure, the new public infrastructure requires users to log on to terminals, such as mobile phones, to engage with mixed physical and virtual spaces. However, a significant proportion of digitally disadvantaged groups cannot use the new public infrastructure effectively due to economic costs or a lack of familiarity with the technology. In addition, the use of facilities in physical and virtual mixed spaces requires engagement with numerous interactive elements, which further hinders digitally disadvantaged groups with weak social or technical skills.

The United Nations (UN) has stated the creation of ‘sustainable cities and communities’ as one of its sustainable development goals, and the construction of smart cities can help achieve this goal (Blasi et al. 2022 ). Recent studies have pointed out that the spread of COVID-19 exacerbated the marginalisation of vulnerable groups, while the lack of universal service processes and virtual facilities has created significant obstacles for digitally disadvantaged groups (Narzt et al. 2016 ; C. H. J. Wang et al. 2021 ). It should be noted that smart cities result from coordinated progress between technology and society (Al-Masri et al. 2019 ). The development of society should not be at the expense of certain people, and improving inclusiveness is key to the construction of smart cities, which should rest on people-oriented development (Ji et al. 2021 ). This paper focuses on the new public infrastructure that integrates mixed physical and virtual spaces. In it, we aim to explore how improved inclusiveness can be achieved for digitally disadvantaged groups during the construction of smart cities, and we propose the following research questions:

RQ1 . In a situation where there is a mix of physical and virtual spaces, what factors affect digitally disadvantaged groups’ use of the new public infrastructure?

RQ2 . What requirements will enable digitally disadvantaged groups to participate fully in the new public infrastructure integrating mixed physical and virtual spaces?

To answer these questions, we built a research model based on the unified theory of acceptance and use of technology (UTAUT) to explore the construction of a new public infrastructure that integrates mixed physical and virtual spaces (Venkatesh et al. 2003 ). During the research process, we focused on the attitudes, willingness, and other behavioural characteristics of digitally disadvantaged groups in relation to mixed physical and virtual spaces, aiming to ultimately provide research support for the construction of highly inclusive smart cities. Compared to existing research, this study goes further in exploring the integration and interconnection of urban public infrastructure in the process of smart city construction. We conducted empirical research to delve more deeply into the factors that influence digitally disadvantaged groups’ use of the new public infrastructure integrating mixed physical and virtual spaces. The results of this study can provide valuable guidelines and a theoretical framework for the construction of new public infrastructure and the improvement of relevant systems in mixed physical and virtual spaces. We also considered the psychological characteristics of digitally disadvantaged groups, introduced psychological reactance into the model, and used social interaction anxiety as a moderator for the model, thereby further enriching the research results regarding mixed physical and virtual spaces. This study directs social and government attention towards the issues affecting digitally disadvantaged groups in the construction of inclusive smart cities, and it has practical implications for the future digitally inclusive development of cities in China and across the world.

Theoretical background and literature review

Theoretical background of utaut.

Currently, the theories used to explore user acceptance behaviour are mainly applied separately in the online and offline fields. Theories relating to people’s offline use behaviour include the theory of planned behaviour (TPB) and the theory of reasoned action (TRA). Theories used to explore users’ online use behaviour include the technology acceptance model (TAM). Unlike previous researchers, who focused on either physical or virtual space, we focused on both. This required us to consider the characteristics of both physical and virtual spaces based on a combination of user acceptance theories (TPB, TRA, and TAM) and UTAUT, which was proposed by Venkatesh et al. ( 2003 ) in 2003. These theories have mainly been used to study the factors affecting user acceptance and the application of information technology. UTAUT integrates user acceptance theories to examine eight online and offline scenarios, thereby meeting our need for a theoretical model for this study that could include both physical and virtual spaces. UTAUT includes four key factors that directly affect users’ acceptance and usage behaviours: performance expectancy, facilitating conditions, social influence, and effort expectancy. Compared to other models, UTAUT has better interpretation and prediction capabilities for user acceptance behaviour (Venkatesh et al. 2003 ). A review of previous research showed that UTAUT has mainly been used to explore usage behaviours in online environments (Hoque and Sorwar, 2017 ) and regarding technology acceptance (Heerink et al. 2010 ). Thus, UTAUT is effective for exploring acceptance and usage behaviours. We therefore based this study on the belief that UTAUT could be applied to people’s intentions to use the new public infrastructure that integrates mixed physical and virtual spaces.

In this paper, we refine and extend UTAUT based on the characteristics of digitally disadvantaged groups, and we propose a model to explore the willingness of digitally disadvantaged groups to use the new public infrastructure integrating mixed physical and virtual spaces. We categorised possible influences on digitally disadvantaged groups’ use of the new public infrastructure into three areas: user factors, social factors, and technical factors. Among the user factors, we explored the willingness of digitally disadvantaged groups to use the new public infrastructure based on their performance expectancy and psychological reactance, as performance expectations are one of the UTAUT variables. To consider situations in which some users resist using new technologies due to cognitive bias, we combined (Hoque and Sorwar, 2017 ) showing that resistance among elderly people is a key factor affecting their adoption of mobile medical services with the theory of psychological reactance and introduced psychological reactance as an independent variable (Miron and Brehm, 2006 ). Among the social factors, we expanded the UTAUT social influence variable to include perceived institutional support and perceived marketplace influence. The new public infrastructure cannot be separated from the relevant government policies and the economic development status of the society in which it is constructed. Therefore, we aimed to explore the willingness of digitally disadvantaged people to use the new public infrastructure in terms of perceived institutional support and perceived marketplace influence. Among the technical factors, we explored the intentions of digitally disadvantaged groups to use new public infrastructure based on effort expectancy and facilitating conditions—both variables taken from UTAUT. In addition, considering that users with different levels of social interaction anxiety may have different levels of intention to use the new public infrastructure, we drew on research regarding the moderating role of consumer technological anxiety in adopting mobile shopping and introduced social interaction anxiety as a moderating variable (Yang and Forney, 2013 ). Believing that these modifications would further improve the interpretive ability of UTAUT, we considered it helpful to study the intentions of digitally disadvantaged groups to use the new public infrastructure.

Intentions to use mixed physical and virtual spaces

Many scholars have researched the factors that affect users’ willingness to use intelligent facilities, which can be broadly divided into two categories: for-profit and public welfare facilities. In the traditional business field, modern information technologies, such as the internet of things and AI, have become important means by which businesses can reduce costs and expand production. Even in traditional industries, such as agriculture (Kadylak and Cotten, 2020 ) and aquaculture (Cai et al. 2023 ), virtual technology now plays a significant role. Operators hope to use advanced technology to change traditional production and marketing models and to keep pace with new developments. However, mixed physical and virtual spaces should be inclusive for all people. Already, technological development is making it clear that no one will be able to entirely avoid mixed physical and virtual spaces. The virtualisation of public welfare facilities has gradually emerged in many areas of daily life, such as electronic health (D. D. Lee et al. 2019 ) and telemedicine (Werner and Karnieli, 2003 ). Government affairs are increasingly managed jointly in both physical and virtual spaces, resulting in an increase in e-government research (Ahn and Chen, 2022 ).

A review of the literature over the past decade showed that users’ willingness to use both for-profit and public welfare facilities is influenced by three sets of factors: user factors, social factors, and technical factors. First, regarding user factors, Bélanger and Carter ( 2008 ) pointed out that consumer trust in the government and technology are key factors affecting people’s intentions to use technology. Research on older people has shown that self-perceived ageing can have a significant impact on emotional attachment and willingness to use technology (B. A. Wang et al. 2021 ). Second, social factors include consumers’ intentions to use, which may vary significantly in different market contexts (Chiu and Hofer, 2015 ). For example, research has shown that people’s willingness to use digital healthcare tools is influenced by the attitudes of the healthcare professionals they encounter (Thapa et al. 2021 ). Third, technical factors include appropriate technical designs that help consumers use facilities more easily. Yadav et al. ( 2019 ) considered technical factors, such as ease of use, quality of service provided, and efficiency parameters, in their experiments.

The rapid development of virtual technology has inevitably drawn attention away from the physical world. Most previous researchers have focused on either virtual or physical spaces. However, scholars have noted the increasing mixing of these two spaces and have begun to study the relationships between them (Aslesen et al. 2019 ; Cocciolo, 2010 ). Wang ( 2007 ) proposed enhancing virtual environments by inserting real entities. Existing research has shown that physical and virtual spaces have begun to permeate each other in both economic and public spheres, blurring the boundaries between them (K. F. Chen et al. 2024 ; Paköz et al. 2022 ). Jakonen ( 2024 ) pointed out that, currently, with the integration of digital technologies into city building, the role of urban space in various stakeholders’ lives needs to be fully considered. The intermingling of physical and virtual spaces began to occur in people’s daily work (J. Chen et al. 2024 ) during the COVID-19 pandemic, which enhanced the integration trend (Yeung and Hao, 2024 ). The intermingling of virtual and physical spaces is a sign of social progress, but it is a considerable challenge for digitally disadvantaged people. For example, people with disabilities experience infrastructure, access, regulatory, communication, and legislative barriers when using telehealth services (Annaswamy et al. 2020 ). However, from an overall perspective, few relevant studies have considered the mixing of virtual and physical spaces.

People who are familiar with information technology, especially Generation Z, generally consider the integration of physical and virtual spaces convenient. However, for digitally disadvantaged groups, such ‘science fiction’-type changes can be disorientating and may undermine their quality of life. The elderly are an important group among the digitally disadvantaged groups referred to in this paper, and they have been the primary target of previous research on issues of inclusivity. Many researchers have considered the factors influencing older people’s willingness to use emerging technologies. For example, for the elderly, ease of use is often a prerequisite for enjoyment (Dogruel et al. 2015 ). Iancu and Iancu ( 2020 ) explored the interaction of elderly with technology, with a particular focus on mobile device design. The study emphasised that elderly people’s difficulties with technology stem from usability issues that can be addressed through improved design and appropriate training (Iancu and Iancu, 2020 ). Moreover, people with disabilities are an important group among digitally disadvantaged groups and an essential concern for the inclusive construction of cities. The rapid development of emerging technologies offers convenience to people with disabilities and has spawned many physical accessibility facilities and electronic accessibility systems (Botelho, 2021 ; Perez et al. 2023 ). Ease of use, convenience, and affordability are also key elements for enabling disadvantaged groups to use these facilities (Mogaji et al. 2023 ; Mogaji and Nguyen, 2021 ). Zander et al. ( 2023 ) explored the facilitators of and barriers to the implementation of welfare technologies for elderly people and people with disabilities. Factors such as abilities, attitudes, values, and lifestyles must be considered when planning the implementation of welfare technology for older people and people with disabilities (Zander et al. 2023 ).

In summary, scholars have conducted extensive research on the factors influencing intentions to use virtual facilities. These studies have revealed the underlying logic behind people’s adoption of virtual technology and have laid the foundations for the construction of inclusive new public infrastructure. Moreover, scholars have proposed solutions to the problems experienced by digitally disadvantaged groups in adapting to virtual facilities, but most of these scholars have focused on the elderly. Furthermore, scholars have recently conducted preliminary explorations of the mixing of physical and virtual spaces. These studies provided insights for this study, enabling us to identify both relevant background factors and current developments in the integration of virtual spaces with reality. However, most researchers have viewed the development of technology from the perspective of either virtual space or physical space, and they have rarely explored technology from the perspective of mixed physical and virtual spaces. In addition, when focusing on designs for the inclusion of digitally disadvantaged groups, scholars have mainly provided suggestions for specific practices, such as improvements in technology, hardware facilities, or device interaction interfaces, while little consideration has been given to the psychological characteristics of digitally disadvantaged groups or to the overall impact of society on these groups. Finally, in studying inclusive modernisation, researchers have generally focused on the elderly or people with disabilities, with less exploration of behavioural differences caused by factors such as social anxiety. Therefore, based on UTAUT, we explored the willingness of digitally disadvantaged groups to use the new public infrastructure integrating mixed physical and virtual spaces in a Chinese context (as shown in Fig. 2 ).

This figure explores the willingness of digitally disadvantaged groups to use the new public infrastructure integrating mixed physical and virtual spaces in a Chinese context.

Research hypotheses

User factors.

Performance expectancy is defined as the degree to which an individual believes that using a system will help him or her achieve gains in job performance (Chao, 2019 ; Venkatesh et al. 2003 ). In this paper, performance expectancy refers to the extent to which digitally disadvantaged groups obtain tangible results from the use of the new public infrastructure. Since individuals have a strong desire to improve their work performance, they have strong intentions to use systems that can improve that performance. Previous studies in various fields have confirmed the view that high performance expectancy can effectively promote individuals’ sustained intentions to use technology (Abbad, 2021 ; Chou et al. 2010 ; S. W. Lee et al. 2019 ). For example, the role of performance expectancy was verified in a study on intentions to use e-government (Zeebaree et al. 2022 ). We believe that if digitally disadvantaged groups have confidence that the new public infrastructure will help them improve their lives or work performance, even in complex environments, such as mixed physical and virtual spaces, they will have a greater willingness to use it. Therefore, we developed the following hypothesis:

H1: Performance expectancy has a positive impact on digitally disadvantaged groups’ intentions to use the new public infrastructure integrating mixed physical and virtual spaces.

Brehm ( 1966 ) proposed the psychological reactance theory in 1966. According to this theory, when individuals perceive that their freedom to make their own choices is under threat, a motivational state to restore that freedom is awakened (Miron and Brehm, 2006 ). Psychological reactance manifests in an individual’s intentional or unintentional resistance to external factors. Previous studies have shown that when individuals are in the process of using systems or receiving information, they may have cognitive biases that lead to erroneous interpretations of the external environment, resulting in psychological reactance (Roubroeks et al. 2010 ). Surprisingly, cognitive biases may prompt individuals to experience psychological reactance, even when offered support with helpful intentions (Tian et al. 2020 ). In this paper, we define psychological resistance as the cognitive-level or psychological-level obstacles or resistance of digitally disadvantaged groups to the new public infrastructure. This resistance may be due to digitally disadvantaged groups misunderstanding the purpose or use of the new public infrastructure. For example, they may think that the new public infrastructure will harm their self-respect or personal interests. When digitally disadvantaged groups view the new public infrastructure as a threat to their status or freedom to make their own decisions, they may develop resistance to its use. Therefore, psychological reactance cannot be ignored as an important factor potentially affecting digitally disadvantaged groups’ intentions to use the new public infrastructure. Hence, we developed the following hypothesis:

H2: Psychological reactance has a negative impact on digitally disadvantaged groups’ intentions to use the new public infrastructure integrating mixed physical and virtual spaces.

Social factors

In many countries, the main providers of public infrastructure are government and public institutions (Susilawati et al. 2010 ). Government decision-making is generally based on laws or government regulations (Acharya et al. 2022 ). Government decision-making procedures affect not only the builders of infrastructure, but also the intentions of users. In life, individuals and social organisations tend to abide by and maintain social norms to ensure that their behaviours are socially attractive and acceptable (Bygrave and Minniti, 2000 ; Martins et al. 2019 ). For example, national financial policies influence the marketing effectiveness of enterprises (Chen et al. 2021 ). Therefore, we believe that perceived institutional support is a key element influencing the intentions of digitally disadvantaged groups to use the new public infrastructure. In this paper, perceived institutional support refers to digitally disadvantaged groups’ perceived policy state or government support for using the new public infrastructure, including institutional norms, laws, and regulations. Existing institutions have mainly been designed around public infrastructure that exists in physical space. We hope to explore whether perceived institutional support for digitally disadvantaged groups affects their intentions to use the new public infrastructure that integrates mixed physical and virtual spaces. Thus, we formulated the following hypothesis:

H3: Perceived institutional support has a positive impact on digitally disadvantaged groups’ intentions to use the new public infrastructure integrating mixed physical and virtual spaces.

Perceived marketplace influence is defined as actions or decisions that affect the market behaviour of consumers and organisations (Joshi et al. 2021 ; Leary et al. 2014 ). In this paper, perceived marketplace influence is defined as the behaviour of others using the new public infrastructure that affects the intentions of digitally disadvantaged groups to use it. Perceived marketplace influence increases consumers’ perceptions of market dynamics and their sense of control through the influence of other participants in the marketplace (Leary et al. 2019 ). Scholars have explored the impact of perceived marketplace influence on consumers’ purchase and use intentions in relation to fair trade and charity (Leary et al. 2019 ; Schneider and Leonard, 2022 ). Schneider and Leonard ( 2022 ) claimed that if consumers believe that their mask-wearing behaviour will motivate others around them to follow suit, then this belief will in turn motivate them to wear masks. Similarly, when digitally disadvantaged people see the people around them using the new public infrastructure, this creates an invisible market that influences their ability and motivation to try using the infrastructure themselves. Therefore, we developed the following hypotheses:

H4: Perceived marketplace influence has a positive impact on digitally disadvantaged groups’ intentions to use the new public infrastructure integrating mixed physical and virtual spaces.

Technical factors

Venkatesh et al. ( 2003 ) defined effort expectancy as the ease with which individuals can use a system. According to Tam et al. ( 2020 ), effort expectancy positively affects individuals’ performance expectancy and their sustained intentions to use mobile applications. In this paper, effort expectancy refers to the ease of use of the new public infrastructure for digitally disadvantaged groups: the higher the level of innovation and the more steps involved in using a facility, the poorer the user experience and the lower the utilisation rate (Venkatesh and Brown, 2001 ). A study on the use of AI devices for service delivery noted that the higher the level of anthropomorphism, the higher the cost of effort required by the customer to use a humanoid AI device (Gursoy et al. 2019 ). In mixed physical and virtual spaces, the design and use of new public infrastructure may become increasingly complex, negatively affecting the lives of digitally disadvantaged groups. We believe that the simpler the new public infrastructure, the more it will attract digitally disadvantaged groups to use it, while also enhancing their intentions to use it. Therefore, we formulated the following hypothesis:

H5: Effort expectancy has a positive impact on digitally disadvantaged groups’ intentions to use the new public infrastructure integrating mixed physical and virtual spaces.

Venkatesh et al. ( 2003 ) defined facilitating conditions as the degree to which an individual believes that an organisation and its technical infrastructure exist to support the use of a system. In this paper, facilitating conditions refer to the external conditions that support digitally disadvantaged groups in using the new public infrastructure, including resources, knowledge bases, skills, etc. According to Zhong et al. ( 2021 ), facilitating conditions can affect users’ attitudes towards the use of face recognition payment systems and, further, affect their intentions to use them. Moreover, scholars have shown that facilitating conditions significantly promote people’s intentions to use e-learning systems and e-government (Abbad, 2021 ; Purohit et al. 2022 ). Currently, the new public infrastructure involves mixed physical and virtual spaces, and external facilitating conditions, such as a ‘knowledge salon’ or a training session, can significantly promote digitally disadvantaged groups’ intentions and willingness to the infrastructure. Therefore, we developed the following hypothesis:

H6: Facilitating conditions have a positive impact on digitally disadvantaged groups’ intentions to use the new public infrastructure integrating a mixed physical and virtual spaces.

Moderator variable

Magee et al. ( 1996 ) claimed that social interaction anxiety is an uncomfortable emotion that some people experience in social situations, leading to avoidance, a desire for solitude, and a fear of criticism. In this paper, social interaction anxiety refers to the worries and fears of digitally disadvantaged groups about the social interactions they will be exposed to when using the new public infrastructure. Research has confirmed that people with high levels of dissatisfaction with their own bodies are more anxious in social situations (Li Mo and Bai, 2023 ). Moreover, people with high degrees of social interaction anxiety may feel uncomfortable in front of strangers or when observed by others (Zhu and Deng, 2021 ). Digitally disadvantaged groups usually have some physiological inadequacies and may be rejected by ‘normal’ groups. Previous studies have shown that the pain caused by social exclusion is positively correlated with anxiety (Davidson et al. 2019 ). Digitally disadvantaged groups may have higher degrees of dissatisfaction with their own physical abilities, which may exacerbate any social interaction anxiety they already have. We believe that high social interaction anxiety is a common characteristic of digitally disadvantaged groups, defining them as ‘different’ from other groups.

In mixed physical and virtual spaces, if the design of the new public infrastructure is not friendly and does not help digitally disadvantaged groups use it easily, their perceived social exclusion is likely to increase, resulting in a heightened sense of anxiety. However, compared with face-to-face and offline social communication, online platforms offer convenience in terms of both communication method and duration (Ali et al. 2020 ). Therefore, people with a high degree of social interaction anxiety frequently prefer and are likely to choose online social communication (Hutchins et al. 2021 ). However, digitally disadvantaged groups may be unable to avoid social interaction by using the facilities offered in virtual spaces. Therefore, we believe that influencing factors may have different effects on intentions to use the new public infrastructure, according to the different levels of social interaction anxiety experienced. Therefore, we predicted the following:

H7: Social interaction anxiety has a moderating effect on each path.

Research methodology

Research background and cases.

To better demonstrate the phenomenon of the new public infrastructure integrating mixed physical and virtual spaces, we considered the cases of ‘Zheli Office’ (as shown in Fig. 3 ) and Alipay (as shown in Fig. 4 ) to explain the two areas of government affairs and daily life affairs, which greatly affect the daily lives of residents. Examining the functions of ‘Zheli Office’ and Alipay in mixed physical and virtual spaces allowed us to provide examples of the new public infrastructure integrating mixed physical and virtual spaces.

This figure shows the ‘Zheli Office’, it is a comprehensive government application that integrates government services through digital technology, transferring some processes from offline to online and greatly improving the convenience, efficiency, and personalisation of government services.

This figure shows Alipay, it supports the integration of various local services, such as live payments and convenient services, and has gradually become Zhejiang’s largest living service platform.

‘Zheli Office’ provides Zhejiang residents with a channel to handle their tax affairs. Residents who need to manage their tax affairs can choose the corresponding tax department through ‘Zheli Office’ and schedule the date and time for offline processing. Residents can also upload tax-related materials directly to ‘Zheli Office’ to submit them to the tax department for preapproval. Residents only need to present the vouchers generated by ‘Zheli Office’ to the tax department at the scheduled time to manage tax affairs and undergo final review. By mitigating long waiting times and tedious tax material review steps through the transfer of processes from physical spaces to virtual spaces, ‘Zheli Office’ greatly optimises the tax declaration process and saves residents time and effort in tax declaration.

Alipay provides residents with a channel to rent shared bicycles. Residents who want to rent bicycles can enter their personal information on Alipay in advance and provide a guarantee (an Alipay credit score or deposit payment). When renting a shared bicycle offline, residents only need to scan the QR code on the bike through Alipay to unlock and use it. When returning the bike, residents can also click the return button to automatically lock the bike and pay the fee anytime and anywhere. By automating leasing procedures and fee settlement in virtual spaces, Alipay avoids the tedious operations that residents experience when renting bicycles in physical stores.

Through the preceding two examples, we demonstrate the specific performance of the integration of virtual spaces and physical spaces. The government/life affairs of residents, such as tax declarations, certificate processing, transportation, shopping, and various other affairs, all require public infrastructure support. With the emergence of new digital trends in residents’ daily lives, mixed physical and virtual spaces have produced a public infrastructure that can support residents’ daily activities in mixed physical and virtual spaces. Due to the essential differences between public infrastructure involving mixed physical and virtual spaces and traditional physical and virtual public infrastructures, we propose a new concept—new public infrastructure. This is defined as ‘a public infrastructure that supports residents in conducting daily activities in mixed physical and virtual spaces’. It is worth noting that the new public infrastructure may encompass not only the virtual spaces provided by digital applications but also the physical spaces provided by machines capable of receiving digital messages, such as smart screens, scanners, and so forth.

The UN Sustainable Development Goal Report highlights that human society needs to build sustainable cities and communities that do not sacrifice the equality of some people. Digitally disadvantaged groups should not be excluded from the sustainable development of cities due to the increasing digitalisation trend because everyone should enjoy the convenience of the new public infrastructure provided by cities. Hence, ensuring that digitally disadvantaged groups can easily and comfortably use the new public infrastructure will help promote the construction of smart cities, making them more inclusive and universal. It will also promote the development of smart cities in a more equal and sustainable direction, ensuring that everyone can enjoy the benefits of urban development. Therefore, in this article, we emphasise the importance of digitally disadvantaged groups in the construction of sustainable smart cities. Through their participation and feedback, we can build more inclusive and sustainable smart cities in the future.

Research design

The aim of this paper was to explore the specific factors that influence the intentions of digitally disadvantaged groups to use the new public infrastructure integrating mixed physical and virtual spaces, and to provide a rational explanation for the role of each factor. To achieve this goal, we first reviewed numerous relevant academic papers. This formed the basis of our research assumptions and helped determine the measurement items we included. Second, we collected data through a questionnaire survey and then analysed the data using partial least squares structural equation modelling (PLS-SEM) to explore the influence of the different factors on digitally disadvantaged groups’ intentions to use the new public infrastructure. Finally, we considered in depth the mechanisms by which the various factors influenced digitally disadvantaged groups’ intentions to use mixed physical and virtual spaces.

We distributed a structured questionnaire to collect data for the study. To ensure the reliability and validity of the questionnaire, we based the item development on the scales used in previous studies (as shown in Appendix A). The first part of the questionnaire concerned the participants’ intentions to use the new public infrastructure. Responses to this part of the questionnaire were given on a seven-point Likert scale to measure the participants’ agreement or disagreement with various statements, with 1 indicating ‘strong disagreement’ and 7 indicating ‘strong agreement’. In addition, we designed cumulative scoring questions to measure the participants’ social interaction anxiety according to Fergus’s Social Interaction Anxiety Scale (Fergus et al. 2012 ). The second part of the questionnaire concerned the demographic characteristics of the participants, including but not limited to gender, age, and education level. Participants were informed that completing the survey was voluntary and that they had the right to refuse or withdraw at any time. They were informed that the researchers would not collect any personal information that would make it possible to identify them. Only after we had obtained the participants’ consent did we commence the questionnaire survey and data collection. Since the new public infrastructure referred to in this study was quite abstract, it was not conducive to the understanding and perceptions of digitally disadvantaged groups. Therefore, to better enable the respondents to understand our concept of the new public infrastructure, we simplified it to ‘an accessible infrastructure’ and informed them about typical cases and the relevant context of this study before they began to complete the questionnaire.

Once the questionnaire design was finalised, we conducted a pretest to ensure that the questions met the basic requirements of reliability and validity and that the participants could accurately understand the questions. In the formal questionnaire survey stage, we distributed the online questionnaire to digitally disadvantaged groups based on the principle of simple random sampling and collected data through the Questionnaire Star platform. Our sampling principle was based on the following points: first, the respondents had to belong to digitally disadvantaged groups and have experienced digital divide problems; second, they had to own at least one smart device and have access to the new public infrastructure, such as via ‘Zheli Office’ or Alipay, and third, they must have used government or daily life services on ‘Zheli Office’ or Alipay at least once in the past three months. After eliminating any invalid questionnaires, 337 valid completed questionnaires remained. The demographic characteristics of the participants are shown in Table 1 . In terms of gender, 54.30% of the participants were male, and 45.70% were female. In terms of age, 64.09% of the participants were aged 18–45 years. In terms of social interaction anxiety, the data showed that 46.59% of the participants had low social interaction anxiety, and 53.41% had high social interaction anxiety.

Data analysis

PLS-SEM imposes few restrictions on the measurement scale, sample size, and residual distribution (Ringle et al. 2012 ). However, the environment in which the research object was located was relatively new, so we added two special variables—psychological reactance and perceived institutional support—to the model. The PLS-SEM model was considered suitable for conducting exploratory research on the newly constructed theory and research framework. Building on previous experience, the data analysis was divided into two stages: 1) the measurement model was used to evaluate the reliability and validity of the experiment, and 2) the structural model was used to test the study hypotheses by examining the relationships between the variables.

Measurement model

First, we tested the reliability of the model by evaluating the reliability of the constructs. As shown in Table 2 , the Cronbach’s alpha (CA) range for this study was 0.858–0.901, so both extremes were higher than the acceptable threshold (Jöreskog, 1971 ). The composite reliability (CR) scores ranged from 0.904 to 0.931; therefore, both extremes were above the threshold of 0.7 (Bagozzi and Phillips, 1982 ) (see Table 2 ).

We then assessed the validity. The test for structural validity included convergent validity and discriminant validity. Convergent validity was mainly verified by the average variance extracted (AVE) value. The recommended value for AVE is 0.5 (Kim and Park, 2013 ). In this study, the AVE values for all structures far exceeded this value (the minimum AVE value was 0.702; see Table 2 ). This result showed that the structure of this model was reliable. The Fornell–Larcker criterion is commonly used to evaluate discriminant validity; that is, the square root of the AVE should be far larger than the correlations for other constructs, meaning that each construct best explains the variance of its own construct (Hair et al. 2014 ), as shown in Table 3 . The validity of the measurement model was further evaluated by calculating the cross-loading values of the reflection construct. It can clearly be seen from Table 4 that compared with other constructs included in the structural model, the indicators of the reflection metric model had the highest loading on their potential constructs (Hair et al. 2022 ), indicating that all inspection results met the evaluation criterion for cross-loading.

In addition, we used the heterotrait-monotrait (HTMT) ratio of correlations to analyse discriminant validity (Henseler et al. 2015 ). Generally, an HTMT value greater than 0.85 indicates that there are potential discriminant validity risks (Hair et al. 2022 ), but Table 5 shows that the HTMT ratios of the correlations in this study were all lower than this value (the maximum value was 0.844).

Structural model

Figure 5 presents the evaluation results for the structural model for the whole sample. The R 2 value for the structural model in this study was 0.740; that is, the explanatory power of the model regarding intention to use was 74.00%. The first step was to ensure that there was no significant collinearity between the predicted value structures, otherwise there would be redundancy in the analysis (Hair et al. 2019 ). All VIF values in this study were between 1.743 and 2.869 and were therefore lower than the 3.3 threshold value for the collinearity test (Hair et al. 2022 ), which proved that the path coefficient had not deviated. This also proves that the model had a low probability of common method bias.

This figure shows the evaluation results for the structural model.

As shown in Fig. 5 , performance expectation ( β  = 0.505, p  < 0.001), perceived institutional support ( β  = 0.338, p  < 0.001), perceived marketplace influence ( β  = 0.190, p  < 0.001), effort expectation ( β  = 0.176, p  < 0.001) and facilitating conditions ( β  = 0.108, p  < 0.001) all had significant and positive effects on intention to use. Moreover, the results showed that the relationship between psychological reaction ( β  = −0.271, p  < 0.001) and intention to use was negative and significant. Therefore, all the paths in this paper, except for the moderator variables, have been verified.

Multi-group analysis

To study the moderating effect between the independent variables and the dependent variables, Henseler et al. ( 2009 ) recommended using a multigroup analysis (MGA). In this study, we used MGA to analyse the moderating effect of different levels of social interaction anxiety. We designed six items for social interaction anxiety (as shown in Appendix A). According to the subjects’ responses to these six items and based on the principle of accumulation, questionnaires with scores of 6–20 indicated low social interaction anxiety, while questionnaires with scores of 28–42 indicated high social interaction anxiety. Questionnaires with scores of 21–27 were considered neutral and eliminated from the analysis involving social interaction anxiety. Based on multigroup validation factor analysis, we determined the component invariance, the configurable invariance, and the equality between compound variance and mean (Hair et al. 2019 ). As shown in Formula 1 , we used an independent sample t -test as a significance test, and a p -value below 0.05 indicated the significance of the parameters.

As shown in Table 6 , under social factors, the p -value for perceived institutional support in relation to intention to use was 0.335, which failed the significance test. This showed that there were no differences between the different degrees of social interaction anxiety. For technical factors, the p -value for facilitating conditions in relation to intention to use was 0.054, which again failed the test. This showed that there were no differences between the different levels of social interaction anxiety. However, the p -values for performance expectancy, psychological reaction, perceived marketplace influence, and effort expectancy in relation to intention to use were all less than 0.05; therefore, they passed the test for significance. This revealed that different degrees of social interaction anxiety had significant effects on these factors and that social interaction anxiety moderated some of the independent variables.

Next, we considered the path coefficients and p- values for the high and low social anxiety groups, as shown in Table 6 . First, with different levels of social anxiety, performance expectation had significantly different effects on intention to use, with low social anxiety ( β  = −0.129, p  = 0.394) failing the test and high social anxiety ( β  = 0.202, p  = 0.004) passing the test. This shows that high social anxiety levels had a greater influence of performance expectations on intention to use than low social anxiety levels. Second, psychological reactance showed significant differences in its effect on intention to use under different degrees of social anxiety, with low social anxiety ( β  = 0.184, p  = 0.065) failing the test and high social anxiety ( β  = −0.466, p  = 0.000) passing the test. Third, with different levels of social anxiety, perceived marketplace influence had significantly different effects on intention to use. Of these, perceived marketplace influence had a significant effect with low social anxiety levels ( β  = 0.312, p  = 0.001) but not with high social anxiety levels ( β  = 0.085, p  = 0.189). Finally, with differing degrees of social anxiety, expected effort had significantly different effects on intention to use. Of these, expected effort was insignificant at a low social anxiety level ( β  = −0.058, p  = 0.488), but it was significant at a high social anxiety level ( β  = 0.326, p  = 0.000). Therefore, different degrees of social interaction anxiety had significantly different effects on performance expectation, psychological reactance, perceived marketplace influence, and effort expectation.

Compared with previous studies, this study constituted a preliminary but groundbreaking exploration of mixed physical and virtual spaces. Moreover, we focused on the inclusivity problems encountered by digitally disadvantaged groups in these mixed physical and virtual spaces. We focused on performance expectancy, psychological reactance, perceived institutional support, perceived marketplace influence, effort expectancy, and facilitating conditions as the six factors, with intention to use being the measure of the perceived value of the new public infrastructure. However, digitally disadvantaged groups, depending on their own characteristics or social influences, can provoke different responses from the general population in their social interactions. Therefore, we added social interaction anxiety to the model as a moderating variable, in line with the assumed psychological characteristics of digitally disadvantaged groups. The empirical results revealed a strong correlation between influencing factors and intention to use. This shows that this model has good applicability for mixed physical and virtual spaces.

According to the empirical results, performance expectancy has a significant and positive impact on intention to use, suggesting that the mixing of the virtual and the real will create usage issues and cognitive difficulties for digitally disadvantaged groups. However, if the new public infrastructure can capitalise on the advantages of blended virtual and physical spaces, it could help users build confidence in its use, which would improve their intentions to use it. Furthermore, users’ intentions to use and high social interaction anxiety are likely to be promoted by performance expectancy. In most cases, social interaction anxiety stems from self-generated avoidance, isolation, and fear of criticism (Schultz and Heimberg, 2008 ). This may result in highly anxious digitally disadvantaged groups being reluctant to engage with others when using public facilities (Mulvale et al. 2019 ; Schou and Pors, 2019 ). However, the new public infrastructure is often unattended, which could be an advantage for users with high social anxiety. Therefore, the effect of performance expectancy in promoting intentions to use would be more significant in this group.

We also found that the psychological reactance of digitally disadvantaged groups had a reverse impact on their intentions to use technology in mixed physical and virtual spaces. However, social interaction anxiety had a moderating effect on this, such that the negative effect of psychological reactance on intention to use the new public infrastructure was more pronounced in the group with high social interaction anxiety. Facilities involving social or interactive factors may make users with high social interaction anxiety think that their autonomy is, to some extent, being violated, thus triggering subconscious resistance. The communication anxiety of digitally disadvantaged groups stems not only from the new public infrastructure itself but also from the environment in which it is used (Fang et al. 2019 ). Complex, mixed physical and virtual spaces can disrupt the habits that digitally disadvantaged groups have developed in purely physical spaces, resulting in greater anxiety (Hu et al. 2022 ), while groups with high levels of social anxiety tend to remain independent because they prefer to maintain their independence. Therefore, a high degree of social interaction anxiety will induce psychological reactance towards using the new public infrastructure.

The results of this paper shed further light on the role of social factors. In particular, the relationship between perceived institutional support and intention to use reflects the fact that perceived institutional support plays a role in promoting digitally disadvantaged groups’ intentions to use the new public infrastructure. This indicates that promotion measures need to be introduced by the government and public institutions if digitally disadvantaged groups are to accept the new public infrastructure. The development of a new public infrastructure integrating mixed physical and virtual spaces requires a high level of involvement from government institutions to facilitate the inclusive development of sustainable smart cities (Khan et al. 2020 ). An interesting finding of this study was that there were no significant differences between the effects of either high or low levels of social interaction anxiety on perceived institutional support and intention to use. This may be because social interaction anxiety mainly occurs in individuals within their close microenvironments. The policies and institutional norms of perceived institutional support tend to act at the macro level (Chen and Zhang, 2021 ; Mora et al. 2023 ), so levels of social interaction anxiety do not differ insignificantly between perceived institutional support and intentions to use the new public infrastructure.

We also found that digitally disadvantaged groups with low social interaction anxiety were more influenced by perceived marketplace influence. Consequently, they were more willing to use the new public infrastructure. When the market trend is to aggressively build a new public infrastructure, companies will accelerate their infrastructure upgrades to keep up with the trend (Hu et al. 2023 ; Liu and Zhao, 2022 ). Companies are increasingly incorporating virtual objects into familiar areas, forcing users to embrace mixed physical and virtual spaces. In addition, it is inevitable that digitally disadvantaged groups will have to use the new public infrastructure due to the market influence of people around them using this infrastructure to manage their government or life issues. When digitally disadvantaged groups with low levels of social interaction anxiety use the new public infrastructure, they are less likely to feel fearful and excluded (Kaihlanen et al. 2022 ) and will tend to be positively influenced by the use behaviours of others to use the new public infrastructure themselves (Troisi et al. 2022 ). The opposite is true for groups with high social interaction anxiety, which leads to significant differences in perceived marketplace influence and intentions to use among digitally disadvantaged groups with different levels of social interaction anxiety.

Existing mixed physical and virtual spaces exhibit exceptional technical complexity, and the results of this study affirm the importance of technical factors in affecting intentions to use. In this paper, we emphasised effort expectancy as the ease of use of the new public infrastructure (Venkatesh et al. 2003 ), which had a significant effect on digitally disadvantaged groups with high levels of social interaction anxiety but no significant effect on those with low levels of social interaction anxiety. Digitally disadvantaged groups with high levels of social interaction anxiety are likely to have a stronger sense of rejection due to environmental pressures if the new public infrastructure is too cumbersome to run or operate; they may therefore prefer using simple facilities and services. Numerous scholars have proven in educational (Hu et al. 2022 ), medical (Bai and Guo, 2022 ), business (Susanto et al. 2018 ), and other fields that good product design promotes users’ intentions to use technology (Chen et al. 2023 ). For digitally disadvantaged groups, accessible and inclusive product designs can more effectively incentivise their intentions to use the new public infrastructure (Hsu and Peng, 2022 ).

Facilitating conditions are technical factors that represent facility-related support services. The study results showed a significant positive effect of facilitating conditions on intention to use. This result is consistent with the results of previous studies regarding physical space. Professional consultation (Vinnikova et al. 2020 ) and training (Yang et al. 2023 ) on products in conventional fields can enhance users’ confidence, which can then be translated into intentions to use (Saparudin et al. 2020 ). Although the form of the new public infrastructure has changed in the direction of integration, its target object is still the user in physical space. Therefore, better facilitating conditions can enhance users’ sense of trust and promote their intentions to use (Alalwan et al. 2017 ; Mogaji et al. 2021 ). Concerning integration, because the new public infrastructure can assume multiple forms, it is difficult for digitally disadvantaged groups to know whether a particular infrastructure has good facilitating conditions. It is precisely such uncertainties that cause users with high social interaction anxiety to worry that they will be unable to use the facilities effectively. They may then worry that they will be burdened by scrutiny from strangers, causing resistance. Even when good facilitating conditions exist, groups with high social interaction anxiety do not necessarily intend to use them. Therefore, there were no significant differences between the different levels of social interaction anxiety in terms of facilitating conditions and intention to use them.

Theoretical value

In this study, we mainly examined the factors influencing digitally disadvantaged groups’ intentions to use the new public infrastructure consisting of mixed physical and virtual spaces. The empirical results of this paper make theoretical contributions to the inclusive construction of mixed spaces in several areas.

First, based on an understanding of urban development involving a deep integration of physical space with virtual space, we contextualise virtual space within the parameters of public infrastructure to shape the concept of a new public infrastructure. At the same time, by including the service system, the virtual community, and other non-physical factors in the realm where the virtual and the real are integrated, we form a concept of mixed physical and virtual spaces, which expands the scope of research related to virtual and physical spaces and provides new ideas for relevant future research.

Second, this paper makes a preliminary investigation of inclusion in the construction of the new public infrastructure and innovatively examines the factors that affect digitally disadvantaged groups’ willingness to use the mixed infrastructure, considering them in terms of individual, social, and technical factors. Moreover, holding that social interaction anxiety is consistent with the psychological characteristics of digitally disadvantaged groups, we introduce social interaction anxiety into the research field and distinguish between the performance of subjects with high social interaction anxiety and the performance of those with low social interaction anxiety. From the perspective of digitally disadvantaged groups, this shows the regulatory effect of social interaction anxiety on users’ psychology and behaviours. These preliminary findings may lead to greater attention being paid to digitally disadvantaged groups and prompt more studies on inclusion.

In addition, while conducting background research, we visited public welfare organisations and viewed government service lists to obtain first-hand information about digitally disadvantaged groups. Through our paper, we encourage the academic community to pay greater attention to theoretical research on digitally disadvantaged groups in the hope that deepening and broadening such research will promote the inclusion of digitally disadvantaged groups in the design of public infrastructure.

Practical value

Based on a large quantity of empirical research data, we explored the digital integration factors that affect users’ intentions to use the new public infrastructure. To some extent, this provides new ideas and development directions for inclusive smart city construction. Inclusion in existing cities mainly concerns the improvement of specific technologies, but the results of this study show that technological factors are only part of the picture. The government should introduce relevant policies to promptly adapt the new public infrastructure to digitally disadvantaged groups, and the legislature should enact appropriate laws. In addition, the study results can guide the design of mixed physical and virtual spaces for the new public infrastructure. Enterprises can refer to the results of this study to identify inconveniences in their existing facilities, optimise their service processes, and improve the inclusiveness of urban institutions. Furthermore, attention should be paid to the moderating role of social interaction anxiety in the process. Inclusive urban construction should not only be physical but should closely consider the inner workings of digitally disadvantaged groups. The government and enterprises should consider the specific requirements of people with high social interaction anxiety, such as by simplifying the enquiry processes in their facilities or inserting psychological comfort measures into the processes.

Limitations and future research

Due to resource and time limitations, this paper has some shortcomings. First, we considered a broad range of digitally disadvantaged groups and conducted a forward-looking exploratory study. Since we collected data through an online questionnaire, there were restrictions on the range of volunteers who responded. Only if participants met at least one of the conditions could they be identified as members of digitally disadvantaged groups and participate in a follow-up survey. To reduce the participants’ introspection and painful recollections of their disabilities or related conditions, and to avoid expected deviations from the data obtained through the survey, we made no detailed distinction between the participants’ degrees of impairment or the reasons for impairment. We adopted a twofold experimental approach.: first, a questionnaire that was too detailed might have infringed on the participants’ privacy rights, and second, since little research has been conducted on inclusiveness in relation to mixed physical and virtual spaces, this work was pioneering. Therefore, we paid greater attention to digitally disadvantaged groups’ intentions to use the new public infrastructure. In future research, we could focus on digitally disadvantaged individuals who exhibit the same deficiencies, or further increase the sample size to investigate the participants’ intentions to use the new public infrastructure in more detail.

Second, different countries have different economic development statuses and numbers of digitally disadvantaged groups. Our study mainly concerned the willingness of digitally disadvantaged groups to use the new public infrastructure in China. Therefore, in the future, the intentions of digitally disadvantaged groups to use new public infrastructures involving mixed physical and virtual spaces can be further explored in different national contexts. Furthermore, in addition to the effects of social interaction anxiety examined in this paper, future researchers could consider other moderators associated with individual differences, such as age, familiarity with technology, and disability status. We also call for more scholars to explore digitally disadvantaged groups’ use of the new public infrastructure to promote inclusive smart city construction and sustainable social development.

Previous researchers have explored users’ intentions to use virtual technology services and have analysed the factors that influence those intentions (Akdim et al. 2022 ; Liébana-Cabanillas et al. 2020 ; Nguyen and Dao, 2024 ). However, researchers have mainly focused on single virtual or physical spaces (Scavarelli et al. 2021 ; Zhang et al. 2020 ), and the topic has rarely been discussed in relation to mixed physical and virtual spaces. In addition, previous studies have mainly considered the technology perspective (Buckingham et al. 2022 ; Carney and Kandt, 2022 ), and the influence of digitally disadvantaged groups’ psychological characteristics and the effect of the overall social environment on their intentions to use have largely been ignored. To fill this gap, we constructed a UTAUT-based model for intentions to use the new public infrastructure that involved a mixing of physical and virtual spaces. We considered the mechanisms influencing digitally disadvantaged groups’ use of the new public infrastructure, considering them from the perspectives of individual, social, and technical factors. We processed and analysed 337 valid samples using PLS-SEM. The results showed that there were significant correlations between the six user factor variables and intention to use the new public infrastructure. In addition, for digitally disadvantaged groups, different degrees of social interaction anxiety had significantly different effects on the impacts of performance expectancy, psychological reactance, perceived marketplace influence, and effort expectancy on intention to use, while there were no differences in the impacts of perceived institutional support and facilitating conditions on intention to use.

In the theoretical value, we build on previous scholarly research on the conceptualisation of new public infrastructures, mixed physical and virtual spaces (Aslesen et al. 2019 ; Cocciolo, 2010 ), arguing for user, social and technological dimensions influencing the use of new public infrastructures by digitally disadvantaged groups in mixed physical and virtual spaces, and for the moderating role of social interaction anxiety. Meanwhile, this study prospectively explores the new phenomenon of digitally disadvantaged groups using new public infrastructures in mixed physical and virtual spaces, which paves the way for future scholars to explore the field both in theory and literature. In the practical value, the research findings will be helpful in promoting effective government policies and corporate designs and in prompting the development of a new public infrastructure that better meets the needs of digitally disadvantaged groups. Moreover, this study will help to direct social and government attention to the problems that exist in the use of new public infrastructures by digitally disadvantaged groups. It will have a significant implication for the future development of smart cities and urban digital inclusiveness in China and worldwide.

Data availability

The datasets generated during and/or analysed during the current study are not publicly available due to the confidentiality of the respondents’ information but are available from the corresponding author upon reasonable request for academic purposes only.

Abbad MMM (2021) Using the UTAUT model to understand students’ usage of e-learning systems in developing countries. Educ. Inf. Technol. 26(6):7205–7224. https://doi.org/10.1007/s10639-021-10573-5

Article   Google Scholar  

Acharya B, Lee J, Moon H (2022) Preference heterogeneity of local government for implementing ICT infrastructure and services through public-private partnership mechanism. Socio-Economic Plan. Sci. 79(9):101103. https://doi.org/10.1016/j.seps.2021.101103

Ahn MJ, Chen YC (2022) Digital transformation toward AI-augmented public administration: the perception of government employees and the willingness to use AI in government. Gov. Inf. Q. 39(2):101664. https://doi.org/10.1016/j.giq.2021.101664

Akdim K, Casalo LV, Flavián C (2022) The role of utilitarian and hedonic aspects in the continuance intention to use social mobile apps. J. Retail. Consum. Serv. 66:102888. https://doi.org/10.1016/j.jretconser.2021.102888

Al-Masri AN, Ijeh A, Nasir M (2019) Smart city framework development: challenges and solutions. Smart Technologies and Innovation for a Sustainable Future, Cham

Google Scholar  

Alalwan AA, Dwivedi YK, Rana NP (2017) Factors influencing adoption of mobile banking by Jordanian bank customers: extending UTAUT2 with trust. Int. J. Inf. Manag. 37(3):99–110. https://doi.org/10.1016/j.ijinfomgt.2017.01.002

Ali A, Li C, Hussain A, Bakhtawar (2020) Hedonic shopping motivations and obsessive–compulsive buying on the internet. Glob. Bus. Rev. 25(1):198–215. https://doi.org/10.1177/0972150920937535

Ali U, Mehmood A, Majeed MF, Muhammad S, Khan MK, Song HB, Malik KM (2019) Innovative citizen’s services through public cloud in Pakistan: user’s privacy concerns and impacts on adoption. Mob. Netw. Appl. 24(1):47–68. https://doi.org/10.1007/s11036-018-1132-x

Almaiah MA, Alamri MM, Al-Rahmi W (2019) Applying the UTAUT model to explain the students’ acceptance of mobile learning system in higher education. IEEE Access 7:174673–174686. https://doi.org/10.1109/access.2019.2957206

Annaswamy TM, Verduzco-Gutierrez M, Frieden L (2020) Telemedicine barriers and challenges for persons with disabilities: COVID-19 and beyond. Disabil Health J 13(4):100973. https://doi.org/10.1016/j.dhjo.2020.100973.3

Article   PubMed   PubMed Central   Google Scholar  

Aslesen HW, Martin R, Sardo S (2019) The virtual is reality! On physical and virtual space in software firms’ knowledge formation. Entrepreneurship Regional Dev. 31(9-10):669–682. https://doi.org/10.1080/08985626.2018.1552314

Bagozzi RP, Phillips LW (1982) Representing and testing organizational theories: a holistic construal. Adm. Sci. Q. 27(3):459–489. https://doi.org/10.2307/2392322

Bai B, Guo ZQ (2022) Understanding users’ continuance usage behavior towards digital health information system driven by the digital revolution under COVID-19 context: an extended UTAUT model. Psychol. Res. Behav. Manag. 15:2831–2842. https://doi.org/10.2147/prbm.S364275

Bélanger F, Carter L (2008) Trust and risk in e-government adoption. J. Strategic Inf. Syst. 17(2):165–176. https://doi.org/10.1016/j.jsis.2007.12.002

Blasi S, Ganzaroli A, De Noni I (2022) Smartening sustainable development in cities: strengthening the theoretical linkage between smart cities and SDGs. Sustain. Cities Soc. 80:103793. https://doi.org/10.1016/j.scs.2022.103793

Botelho FHF (2021) Accessibility to digital technology: virtual barriers, real opportunities. Assistive Technol. 33:27–34. https://doi.org/10.1080/10400435.2021.1945705

Brehm, JW (1966). A theory of psychological reactance . Academic Press

Buckingham SA, Walker T, Morrissey K, Smartline Project T (2022) The feasibility and acceptability of digital technology for health and wellbeing in social housing residents in Cornwall: a qualitative scoping study. Digital Health 8:20552076221074124. https://doi.org/10.1177/20552076221074124

Bygrave W, Minniti M (2000) The social dynamics of entrepreneurship. Entrepreneurship Theory Pract. 24(3):25–36. https://doi.org/10.1177/104225870002400302

Cai Y, Qi W, Yi FM (2023) Smartphone use and willingness to adopt digital pest and disease management: evidence from litchi growers in rural China. Agribusiness 39(1):131–147. https://doi.org/10.1002/agr.21766

Carney F, Kandt J (2022) Health, out-of-home activities and digital inclusion in later life: implications for emerging mobility services. Journal of Transport & Health 24:101311. https://doi.org/10.1016/j.jth.2021.101311

Chao CM (2019) Factors determining the behavioral intention to use mobile learning: an application and extension of the UTAUT model. Front. Psychol. 10:1652. https://doi.org/10.3389/fpsyg.2019.01652

Chen HY, Chen HY, Zhang W, Yang CD, Cui HX (2021) Research on marketing prediction model based on Markov Prediction. Wirel. Commun. Mob. Comput. 2021(9):4535181. https://doi.org/10.1155/2021/4535181

Chen J, Cui MY, Levinson D (2024) The cost of working: measuring physical and virtual access to jobs. Int. J. Urban Sci. 28(2):318–334. https://doi.org/10.1080/12265934.2023.2253208

Chen JX, Wang T, Fang ZY, Wang HT (2023) Research on elderly users’ intentions to accept wearable devices based on the improved UTAUT model. Front. Public Health 10(12):1035398. https://doi.org/10.3389/fpubh.2022.1035398

Chen KF, Guaralda M, Kerr J, Turkay S (2024) Digital intervention in the city: a conceptual framework for digital placemaking. Urban Des. Int. 29(1):26–38. https://doi.org/10.1057/s41289-022-00203-y

Chen L, Zhang H (2021) Strategic authoritarianism: the political cycles and selectivity of China’s tax-break policy. Am. J. Political Sci. 65(4):845–861. https://doi.org/10.1111/ajps.12648

Chiu YTH, Hofer KM (2015) Service innovation and usage intention: a cross-market analysis. J. Serv. Manag. 26(3):516–538. https://doi.org/10.1108/josm-10-2014-0274

Chou SW, Min HT, Chang YC, Lin CT (2010) Understanding continuance intention of knowledge creation using extended expectation-confirmation theory: an empirical study of Taiwan and China online communities. Behav. Inf. Technol. 29(6):557–570. https://doi.org/10.1080/01449290903401986

Cocciolo A (2010) Alleviating physical space constraints using virtual space? A study from an urban academic library. Libr. Hi Tech. 28(4):523–535. https://doi.org/10.1108/07378831011096204

Davidson CA, Willner CJ, van Noordt SJR, Banz BC, Wu J, Kenney JG, Johannesen JK, Crowley MJ (2019) One-month stability of cyberball post-exclusion ostracism distress in Adolescents. J. Psychopathol. Behav. Assess. 41(3):400–408. https://doi.org/10.1007/s10862-019-09723-4

Dogruel L, Joeckel S, Bowman ND (2015) The use and acceptance of new media entertainment technology by elderly users: development of an expanded technology acceptance model. Behav. Inf. Technol. 34(11):1052–1063. https://doi.org/10.1080/0144929x.2015.1077890

Fang ML, Canham SL, Battersby L, Sixsmith J, Wada M, Sixsmith A (2019) Exploring privilege in the digital divide: implications for theory, policy, and practice. Gerontologist 59(1):E1–E15. https://doi.org/10.1093/geront/gny037

Article   PubMed   Google Scholar  

Fergus TA, Valentiner DP, McGrath PB, Gier-Lonsway SL, Kim HS (2012) Short forms of the social interaction anxiety scale and the social phobia scale. J. Personal. Assess. 94(3):310–320. https://doi.org/10.1080/00223891.2012.660291

Garone A, Pynoo B, Tondeur J, Cocquyt C, Vanslambrouck S, Bruggeman B, Struyven K (2019) Clustering university teaching staff through UTAUT: implications for the acceptance of a new learning management system. Br. J. Educ. Technol. 50(5):2466–2483. https://doi.org/10.1111/bjet.12867

Gu QH, Iop (2020) Frame-based conceptual model of smart city’s applications in China. International Conference on Green Development and Environmental Science and Technology (ICGDE), Changsha, CHINA

Book   Google Scholar  

Guo MJ, Liu YH, Yu HB, Hu BY, Sang ZQ (2016) An overview of smart city in China. China Commun. 13(5):203–211. https://doi.org/10.1109/cc.2016.7489987

Gursoy D, Chi OHX, Lu L, Nunkoo R (2019) Consumers acceptance of artificially intelligent (AI) device use in service delivery. Int. J. Inf. Manag. 49:157–169. https://doi.org/10.1016/j.ijinfomgt.2019.03.008

Hair, JF, Hult, GTM, Ringle, CM, & Sarstedt, M (2022). A primer on partial least squares structural equation modeling (PLS-SEM) (Third edition. ed.). SAGE Publications, Inc

Hair Jr JF, Sarstedt M, Hopkins L, Kuppelwieser VG (2014) Partial least squares structural equation modeling (PLS-SEM): an emerging tool in business research. Eur. Bus. Rev. 26(2):106–121. https://doi.org/10.1108/ebr-10-2013-0128

Hair JF, Risher JJ, Sarstedt M, Ringle CM (2019) When to use and how to report the results of PLS-SEM. Eur. Bus. Rev. 31(1):2–24. https://doi.org/10.1108/ebr-11-2018-0203

Heerink M, Kröse B, Evers V, Wielinga B (2010) Assessing acceptance of assistive social agent technology by older adults: the almere model. Int. J. Soc. Robot. 2(4):361–375. https://doi.org/10.1007/s12369-010-0068-5

Henseler J, Ringle CM, Sarstedt M (2015) A new criterion for assessing discriminant validity in variance-based structural equation modeling. J. Acad. Mark. Sci. 43(1):115–135. https://doi.org/10.1007/s11747-014-0403-8

Henseler, J, Ringle, CM, & Sinkovics, RR (2009). The use of partial least squares path modeling in international marketing. In RR Sinkovics & PN Ghauri (Eds.), New Challenges to International Marketing (Vol. 20, pp. 277-319). Emerald Group Publishing Limited. https://doi.org/10.1108/S1474-7979 (2009)0000020014

Hoque R, Sorwar G (2017) Understanding factors influencing the adoption of mHealth by the elderly: an extension of the UTAUT model. Int. J. Med. Inform. 101:75–84. https://doi.org/10.1016/j.ijmedinf.2017.02.002

Hsu CW, Peng CC (2022) What drives older adults’ use of mobile registration apps in Taiwan? An investigation using the extended UTAUT model. Inform. Health Soc. Care 47(3):258–273. https://doi.org/10.1080/17538157.2021.1990299

Hu J, Zhang H, Irfan M (2023) How does digital infrastructure construction affect low-carbon development? A multidimensional interpretation of evidence from China. J. Clean. Prod. 396(9):136467. https://doi.org/10.1016/j.jclepro.2023.136467

Hu TF, Guo RS, Chen C (2022) Understanding mobile payment adaption with the integrated model of UTAUT and MOA model. 2022 Portland International Conference on Management of Engineering and Technology (PICMET), Portland, OR, USA

Hutchins N, Allen A, Curran M, Kannis-Dymand L (2021) Social anxiety and online social interaction. Aust. Psychologist 56(2):142–153. https://doi.org/10.1080/00050067.2021.1890977

Iancu I, Iancu B (2020) Designing mobile technology for elderly. A theoretical overview. Technol. Forecast. Soc. Change 155(9):119977. https://doi.org/10.1016/j.techfore.2020.119977

Jakonen, OI (2024). Smart cities, virtual futures? - Interests of urban actors in mediating digital technology and urban space in Tallinn, Estonia. Urban Studies , 17. https://doi.org/10.1177/00420980241245871

Ji TT, Chen JH, Wei HH, Su YC (2021) Towards people-centric smart city development: investigating the citizens’ preferences and perceptions about smart-city services in Taiwan. Sustain. Cities Soc. 67(14):102691. https://doi.org/10.1016/j.scs.2020.102691

Jöreskog KG (1971) Simultaneous factor analysis in several populations. Psychometrika 36(4):409–426. https://doi.org/10.1007/BF02291366

Joshi Y, Uniyal DP, Sangroya D (2021) Investigating consumers’ green purchase intention: examining the role of economic value, emotional value and perceived marketplace influence. J. Clean. Prod. 328(8):129638. https://doi.org/10.1016/j.jclepro.2021.129638

Kadylak T, Cotten SR (2020) United States older adults’ willingness to use emerging technologies. Inf. Commun. Soc. 23(5):736–750. https://doi.org/10.1080/1369118x.2020.1713848

Kaihlanen AM, Virtanen L, Buchert U, Safarov N, Valkonen P, Hietapakka L, Hörhammer I, Kujala S, Kouvonen A, Heponiemi T (2022) Towards digital health equity-a qualitative study of the challenges experienced by vulnerable groups in using digital health services in the COVID-19 era. BMC Health Services Research 22(1):188. https://doi.org/10.1186/s12913-022-07584-4

Khan HH, Malik MN, Zafar R, Goni FA, Chofreh AG, Klemes JJ, Alotaibi Y (2020) Challenges for sustainable smart city development: a conceptual framework. Sustain. Dev. 28(5):1507–1518. https://doi.org/10.1002/sd.2090

Kim S, Park H (2013) Effects of various characteristics of social commerce (s-commerce) on consumers’ trust and trust performance. Int. J. Inf. Manag. 33(2):318–332. https://doi.org/10.1016/j.ijinfomgt.2012.11.006

Leary RB, Vann RJ, Mittelstaedt JD (2019) Perceived marketplace influence and consumer ethical action. J. Consum. Aff. 53(3):1117–1145. https://doi.org/10.1111/joca.12220

Leary RB, Vann RJ, Mittelstaedt JD, Murphy PE, Sherry JF (2014) Changing the marketplace one behavior at a time: perceived marketplace influence and sustainable consumption. J. Bus. Res. 67(9):1953–1958. https://doi.org/10.1016/j.jbusres.2013.11.004

Lee DD, Arya LA, Andy UU, Sammel MD, Harvie HS (2019) Willingness of women with pelvic floor disorders to use mobile technology to communicate with their health care providers. Female Pelvic Med. Reconstructive Surg. 25(2):134–138. https://doi.org/10.1097/spv.0000000000000668

Lee SW, Sung HJ, Jeon HM (2019) Determinants of continuous intention on food delivery apps: extending UTAUT2 with information quality. Sustainability 11(11):3141. https://doi.org/10.3390/su11113141 . 15

Li Mo QZ, Bai BY (2023) Height dissatisfaction and loneliness among adolescents: the chain mediating role of social anxiety and social support. Curr. Psychol. 42(31):27296–27304. https://doi.org/10.1007/s12144-022-03855-9

Liébana-Cabanillas F, Japutra A, Molinillo S, Singh N, Sinha N (2020) Assessment of mobile technology use in the emerging market: analyzing intention to use m-payment services in India. Telecommun. Policy 44(9):102009. https://doi.org/10.1016/j.telpol.2020.102009 . 17

Liu HD, Zhao HF (2022) Upgrading models, evolutionary mechanisms and vertical cases of service-oriented manufacturing in SVC leading enterprises: product-development and service-innovation for industry 4.0. Humanities Soc. Sci. Commun. 9(1):387. https://doi.org/10.1057/s41599-022-01409-9 . 24

Liu ZL, Wang Y, Xu Q, Yan T, Iop (2017) Study on smart city construction of Jiujiang based on IOT technology. 3rd International Conference on Advances in Energy, Environment and Chemical Engineering (AEECE), Chengdu, CHINA

Magee WJ, Eaton WW, Wittchen H-U, McGonagle KA, Kessler RC (1996) Agoraphobia, simple phobia, and social phobia in the National Comorbidity Survey. Arch. Gen. Psychiatry 53(2):159–168

Article   CAS   PubMed   Google Scholar  

Martins R, Oliveira T, Thomas M, Tomás S (2019) Firms’ continuance intention on SaaS use - an empirical study. Inf. Technol. People 32(1):189–216. https://doi.org/10.1108/itp-01-2018-0027

Miron AM, Brehm JW (2006) Reactance theory - 40 Years later. Z. Fur Sozialpsychologie 37(1):9–18. https://doi.org/10.1024/0044-3514.37.1.9

Mogaji E, Balakrishnan J, Nwoba AC, Nguyen NP (2021) Emerging-market consumers’ interactions with banking chatbots. Telematics and Informatics 65:101711. https://doi.org/10.1016/j.tele.2021.101711

Mogaji E, Bosah G, Nguyen NP (2023) Transport and mobility decisions of consumers with disabilities. J. Consum. Behav. 22(2):422–438. https://doi.org/10.1002/cb.2089

Mogaji E, Nguyen NP (2021) Transportation satisfaction of disabled passengers: evidence from a developing country. Transportation Res. Part D.-Transp. Environ. 98:102982. https://doi.org/10.1016/j.trd.2021.102982

Mora L, Gerli P, Ardito L, Petruzzelli AM (2023) Smart city governance from an innovation management perspective: theoretical framing, review of current practices, and future research agenda. Technovation 123:102717. https://doi.org/10.1016/j.technovation.2023.102717

Mulvale G, Moll S, Miatello A, Robert G, Larkin M, Palmer VJ, Powell A, Gable C, Girling M (2019) Codesigning health and other public services with vulnerable and disadvantaged populations: insights from an international collaboration. Health Expectations 22(3):284–297. https://doi.org/10.1111/hex.12864

Narzt W, Mayerhofer S, Weichselbaum O, Pomberger G, Tarkus A, Schumann M (2016) Designing and evaluating barrier-free travel assistance services. 3rd International Conference on HCI in Business, Government, and Organizations - Information Systems (HCIBGO) Held as Part of 18th International Conference on Human-Computer Interaction (HCI International), Toronto, CANADA

Nguyen GD, Dao THT (2024) Factors influencing continuance intention to use mobile banking: an extended expectation-confirmation model with moderating role of trust. Humanities Soc. Sci. Commun. 11(1):276. https://doi.org/10.1057/s41599-024-02778-z

Nicolas C, Kim J, Chi S (2020) Quantifying the dynamic effects of smart city development enablers using structural equation modeling. Sustain. Cities Soc. 53:101916. https://doi.org/10.1016/j.scs.2019.101916

Paköz MZ, Sözer C, Dogan A (2022) Changing perceptions and usage of public and pseudo-public spaces in the post-pandemic city: the case of Istanbul. Urban Des. Int. 27(1):64–79. https://doi.org/10.1057/s41289-020-00147-1

Perez AJ, Siddiqui F, Zeadally S, Lane D (2023) A review of IoT systems to enable independence for the elderly and disabled individuals. Internet Things 21:100653. https://doi.org/10.1016/j.iot.2022.100653

Purohit S, Arora R, Paul J (2022) The bright side of online consumer behavior: continuance intention for mobile payments. J. Consum. Behav. 21(3):523–542. https://doi.org/10.1002/cb.2017

Ringle CM, Sarstedt M, Straub DW (2012) Editor’s Comments: A Critical Look at the Use of PLS-SEM in “MIS Quarterly”. MIS Q. 36(1):III–XIV

Roubroeks MAJ, Ham JRC, Midden CJH (2010) The dominant robot: threatening robots cause psychological reactance, especially when they have incongruent goals. 5th International Conference on Persuasive Technology, Copenhagen, DENMARK

Saparudin M, Rahayu A, Hurriyati R, Sultan MA, Ramdan AM, Ieee (2020) Consumers’ continuance intention use of mobile banking in Jakarta: extending UTAUT models with trust. 5th International Conference on Information Management and Technology (ICIMTech), Bandung, Indonesia

Scavarelli A, Arya A, Teather RJ (2021) Virtual reality and augmented reality in social learning spaces: a literature review. Virtual Real. 25(1):257–277. https://doi.org/10.1007/s10055-020-00444-8

Schneider AB, Leonard B (2022) From anxiety to control: mask-wearing, perceived marketplace influence, and emotional well-being during the COVID-19 pandemic. J. Consum. Aff. 56(1):97–119. https://doi.org/10.1111/joca.12412

Schou J, Pors AS (2019) Digital by default? A qualitative study of exclusion in digitalised welfare. Soc. Policy Adm. 53(3):464–477. https://doi.org/10.1111/spol.12470

Schultz LT, Heimberg RG (2008) Attentional focus in social anxiety disorder: potential for interactive processes. Clin. Psychol. Rev. 28(7):1206–1221. https://doi.org/10.1016/j.cpr.2008.04.003

Shibusawa H (2000) Cyberspace and physical space in an urban economy. Pap. Regional Sci. 79(3):253–270. https://doi.org/10.1007/pl00013610

Susanto A, Mahadika PR, Subiyakto A, Nuryasin, Ieee (2018) Analysis of electronic ticketing system acceptance using an extended unified theory of acceptance and use of technology (UTAUT). 6th International Conference on Cyber and IT Service Management (CITSM), Parapat, Indonesia

Susilawati C, Wong J, Chikolwa B (2010) Public participation, values and interests in the procurement of infrastructure projects in Australia: a review and future research direction. 2010 International Conference on Construction and Real Estate Management, Brisbane, Australia

Tam C, Santos D, Oliveira T (2020) Exploring the influential factors of continuance intention to use mobile Apps: extending the expectation confirmation model. Inf. Syst. Front. 22(1):243–257. https://doi.org/10.1007/s10796-018-9864-5

Teo T, Zhou MM, Fan ACW, Huang F (2019) Factors that influence university students’ intention to use Moodle: a study in Macau. EtrD-Educ. Technol. Res. Dev. 67(3):749–766. https://doi.org/10.1007/s11423-019-09650-x

Thapa S, Nielsen JB, Aldahmash AM, Qadri FR, Leppin A (2021) Willingness to use digital health tools in patient care among health care professionals and students at a university hospital in Saudi Arabia: quantitative cross-sectional survey. JMIR Med. Educ. 7(1):e18590. https://doi.org/10.2196/18590

Tian X, Solomon DH, Brisini KS (2020) How the comforting process fails: psychological reactance to support messages. J. Commun. 70(1):13–34. https://doi.org/10.1093/joc/jqz040

Troisi O, Fenza G, Grimaldi M, Loia F (2022) Covid-19 sentiments in smart cities: the role of technology anxiety before and during the pandemic. Computers in Human Behavior 126:106986. https://doi.org/10.1016/j.chb.2021.106986

Venkatesh V, Brown SA (2001) A longitudinal investigation of personal computers in homes: adoption determinants and emerging challenges. MIS Q. 25(1):71–102. https://doi.org/10.2307/3250959

Venkatesh V, Morris MG, Davis GB, Davis FD (2003) User acceptance of information technology: toward a unified view. MIS Q. 27(3):425–478. https://doi.org/10.2307/30036540

Vinnikova A, Lu LD, Wei JC, Fang GB, Yan J (2020) The Use of smartphone fitness applications: the role of self-efficacy and self-regulation. International Journal of Environmental Research and Public Health 17(20):7639. https://doi.org/10.3390/ijerph17207639

Wang BA, Zhang R, Wang Y (2021) Mechanism influencing older people’s willingness to use intelligent aged-care products. Healthcare 9(7):864. https://doi.org/10.3390/healthcare9070864

Wang CHJ, Steinfeld E, Maisel JL, Kang B (2021) Is your smart city inclusive? Evaluating proposals from the US department of transportation’s smart city challenge. Sustainable Cities and Society 74:103148. https://doi.org/10.1016/j.scs.2021.103148

Wang XY (2007) Mutually augmented virtual environments for architecural design and collaboration. 12th Computer-Aided Architectural Design Futures Conference, Sydney, Australia

Werner P, Karnieli E (2003) A model of the willingness to use telemedicine for routine and specialized care. J. Telemed. Telecare 9(5):264–272. https://doi.org/10.1258/135763303769211274

Yadav J, Saini AK, Yadav AK (2019) Measuring citizens engagement in e-Government projects - Indian perspective. J. Stat. Manag. Syst. 22(2):327–346. https://doi.org/10.1080/09720510.2019.1580908

Yang CC, Liu C, Wang YS (2023) The acceptance and use of smartphones among older adults: differences in UTAUT determinants before and after training. Libr. Hi Tech. 41(5):1357–1375. https://doi.org/10.1108/lht-12-2021-0432

Yang K, Forney JC (2013) The moderating role of consumer technology anxiety in mobile shopping adoption: differential effects of facilitating conditions and social influences. J. Electron. Commer. Res. 14(4):334–347

Yeung HL, Hao P (2024) Telecommuting amid Covid-19: the Governmobility of work-from-home employees in Hong Kong. Cities 148:104873. https://doi.org/10.1016/j.cities.2024.104873

Zander V, Gustafsson C, Stridsberg SL, Borg J (2023) Implementation of welfare technology: a systematic review of barriers and facilitators. Disabil. Rehabilitation-Assistive Technol. 18(6):913–928. https://doi.org/10.1080/17483107.2021.1938707

Zeebaree M, Agoyi M, Agel M (2022) Sustainable adoption of e-government from the UTAUT perspective. Sustainability 14(9):5370. https://doi.org/10.3390/su14095370

Zhang YX, Liu HX, Kang SC, Al-Hussein M (2020) Virtual reality applications for the built environment: Research trends and opportunities. Autom. Constr. 118:103311. https://doi.org/10.1016/j.autcon.2020.103311

Zhong YP, Oh S, Moon HC (2021) Service transformation under industry 4.0: investigating acceptance of facial recognition payment through an extended technology acceptance model. Technology in Society 64:101515. https://doi.org/10.1016/j.techsoc.2020.101515

Zhu DH, Deng ZZ (2021) Effect of social anxiety on the adoption of robotic training partner. Cyberpsychology Behav. Soc. Netw. 24(5):343–348. https://doi.org/10.1089/cyber.2020.0179

Download references

Acknowledgements

This research was supported by the National Social Science Foundation of China, grant number 22BGJ037; the Fundamental Research Funds for the Provincial Universities of Zhejiang, grant number GB202301004; and the Zhejiang Province University Students Science and Technology Innovation Activity Program, grant numbers 2023R403013, 2023R403010 & 2023R403086.

Author information

These authors contributed equally: Chengxiang Chu, Zhenyang Shen, Hanyi Xu.

Authors and Affiliations

School of Management, Zhejiang University of Technology, Hangzhou, China

Chengxiang Chu, Zhenyang Shen, Qizhi Wei & Cong Cao

Law School, Zhejiang University of Technology, Hangzhou, China

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualisation: C.C., CX.C. and ZY.S.; Methodology: CX.C. and HY.X.; Validation: ZY.S. and QZ.W.; Formal analysis: HY.X.; Investigation: CX.C., ZY.S. and HY.X.; Resources: C.C.; Data curation: CX.C. and HY.X.; Writing–original draft preparation: CX.C, ZY.S., HY.X. and QZ.W.; Writing–review & editing: CX.C and C.C.; Visualisation: ZY.S. and HY.X.; Supervision: C.C.; Funding acquisition: C.C., CX.C. and ZY.S.; all authors approved the final manuscript to be submitted.

Corresponding author

Correspondence to Cong Cao .

Ethics declarations

Ethical approval.

Ethical approval for the involvement of human subjects in this study was granted by Institutional Review Board of School of Management, Zhejiang University of Technology, China, Reference number CC-2023-1-0008-0005-SOM-ZJUT.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A. Measurement items

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Chu, C., Shen, Z., Xu, H. et al. How to avoid sinking in swamp: exploring the intentions of digitally disadvantaged groups to use a new public infrastructure that combines physical and virtual spaces. Humanit Soc Sci Commun 11 , 1135 (2024). https://doi.org/10.1057/s41599-024-03684-0

Download citation

Received : 28 October 2023

Accepted : 29 August 2024

Published : 04 September 2024

DOI : https://doi.org/10.1057/s41599-024-03684-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Title: Social Media Marketing - A Systematic Review of Quantitative Empirical Studies

Authors : Bernd W. Wirtz; Isabell Balzer

Addresses : N/A ' N/A

Abstract : Purpose - The main purpose of this paper is to review the available research in the area of social media marketing and provide an overview of the research potential to identify research gaps and define specific areas for further research. Method - Through various reading and structuring loops, we examined 355 peer-reviewed quantitative empirical articles according to key themes, research objectives, research perspective, statistical method, and data collection method. Descriptive statistics were used to analyze research foci and identify research gaps. Findings - We were able to identify four research periods within social media marketing research: (1) emergence of SMM research, (2) growing interest, (3) underpinning of the theories, and (4) paradigm shift in research. We also identified the most interesting (e.g., \'success with SMM\') and underrepresented social media marketing research topics (e.g., \'B2B influence factors\'). In addition, the lack of a two-way research orientation (customer and provider) is evident and should be addressed in the future. Finally, we uncovered the need for further exploratory research in SMM. Limitations - Because we focused on quantitative empirical research, we cannot provide insights into the entire SMM research body. In addition, we cannot guarantee the completeness of the data because relevant publications may not have been retrieved due to the limitation of the search terms. Moreover, there is a potential loss of information, as we have summarized information to present it better. Implications - Future research should focus on two-sided research dealing with manager responses in SNS and customer loyalty and comparing manager responses and consumer reviews on social media performance. In addition, future research could include exploratory research questions. Research on the effects of social media marketing in a business-to-business context is also of great importance in the future. Originality - There has been no systematic review of the quantitative literature on social media marketing. Therefore, we attribute originality to our approach and results.

Keywords : Social media marketing; literature review; empirical research; quantitative research.

DOI : 10.1504/JBM.2023.141304

Journal of Business and Management, 2023 Vol.29 No.1, pp.80 - 114

Published online: 05 Sep 2024 *

Keep up-to-date

• Our Newsletter ( subscribe for free )
• New issue alerts
• Inderscience is a member of publishing organisations including:

Advertisement

Reliability and validity assessment of instrument to measure sustainability practices at shipping ports in India

• Open access
• Published: 03 September 2024
• Volume 5 , article number  236 , ( 2024 )

Cite this article

You have full access to this open access article

• L. Kishore 1 ,
• Yogesh P. Pai 2 &
• Parthesh Shanbhag 3  

Sustainability has emerged as one of the most critical factors influencing the competitiveness of maritime shipping ports. This emergence has led to a surge in research publications on port sustainability-related topics. However, despite the increasing awareness and adoption of sustainability practices, documented literature on empirical studies with survey and interview data is very limited. Moreover, the existence of validated instruments to objectively assess sustainability through sustainability practices for shipping ports in India needs to be traced. This study contributes by validating an instrument to evaluate objectively sustainability practices in shipping ports by adopting a four-stage process, starting with item identification based on an extensive literature review, instrument evaluation by subject matter experts, assessing of the instrument with suitable content validation indices, and finally evaluating the validity and reliability of the hypothesized theoretical model. For content validation, Content Validity Index, Cohens Kappa coefficient, and Lawshe’s Content Validity Ratio were computed with the assessment by a subject matter expert panel comprising six members from the port industry as well as academicians cum researchers in the field of shipping ports. The content-validated instrument was administered to 200 samples comprising officer category port employees. The measurement model was evaluated and validated using the Confirmatory Factor Analysis to assess the extent to which the measured variables represent the theoretical construct of the study and ascertain the factor structure. The empirically validated instrument met the required guidelines of model fit, reliability, and construct validity measures and was found to be a confirmed model for measuring sustainability practices in shipping ports. Structural Equation Modeling methodology was adopted to explain the variance and the path relationship between the higher-order and lower-order constructs of sustainability. The results indicate that the economic dimensions are the major contributors to the overall sustainability of the port as they drive investments in environmental and social dimensions, leading to overall sustainable development. The study’s findings will be helpful for researchers, academicians, policymakers, and industry practitioners working towards sustainability practices that contribute to sustainable growth and development in the shipping industry.

Similar content being viewed by others

Sustainability in Organizations: Bhutan’s Perspective

Preventing and developmental factors of sustainability in healthcare organisations from the perspective of decision makers: an exploratory factor analysis

Factors affecting the outcome of corporate sustainability policy: a review paper

Avoid common mistakes on your manuscript.

1 Introduction

Sustainability has increasingly been considered one of society and industry’s most significant focus areas [ 1 ] along with regulatory bodies in recent times, although only partially new [ 2 ]. Sustainable development is generally quoted as the one that “satisfies the needs and wants of the present generation” simultaneously without compromising the future generation’s needs and aspirations” [ 3 ]. Moreover, the challenge is balancing sustainability and economic growth [ 4 ]. This prerequisite has led organizations to look beyond mere economic performance and build social and environmentally friendly business models by integrating sustainability principles and practices [ 5 , 6 , 7 ] targeting competitive advantages [ 8 , 9 ]. Against this backdrop, Lun et al. [ 3 ] highlighted the importance of shipping ports in a country’s sustainable growth and development in the long run as they generate employment along with the export–import trade. It is widely accepted that port-led economic growth and development have been the backbone of many developed and developing countries. For instance, the Indian maritime sector, which is one of the most promising, emerging, developing, dynamic markets in the world, has received a facelift with one of the significant mega-initiatives of the government, “ Sagarmala ”, which is focused on “port-led economic development” [ 10 ]. 95% of India’s overall goods trade volume is through shipping ports, contributing about 14% of the GDP [ 11 ]. The impact of the subdued port performance is reflected in a country’s economic development [ 12 ]. Shipping ports are vital nodes that link other modes of transportation in global trade and are considered strategic assets demanding significant attention in maritime and transportation research and practice [ 4 , 13 , 14 ].

Lee et al. [ 15 ] flagged the concern of less attention given to sustainability in the shipping, port, and maritime industries, unlike the aviation and road transport sectors. Many studies [ 14 , 16 , 17 , 18 , 19 ] have emphasized sustainability as one of the most crucial aspects influencing the competitiveness and long-term sustenance of shipping ports; however, it is not fully incorporated into a strategic decision that demands a long-term view when deciding on port development and management. Further, coastal lines are densely populated, leading to higher levels of economic activity and rapid urbanization; however, they face the consequences that come along as the byproduct of development in the form of environmental, economic, and social concerns [ 14 , 20 ]. Some of the substantial problems in the ports discussed in many studies [ 14 , 18 , 19 , 21 , 22 , 23 , 24 , 25 ] include the depletion of the marine ecosystem and biodiversity due to dredging and reclamation works, water pollution due to ballast operations, oil spillage during ship anchoring and cargo operations, wastewater spillage from ships, air pollution due to various pollutants and particulate matters, dust and smoke pollution by a heavy vehicle, climate change effects, increased energy consumption for operations and the cost of energy, uncertainty in future economic returns of investment in the port, employment and diversity of jobs, employee productivity, displacement of local community along with impact on their livelihood, loss of agricultural land, steep increase in cost of living and land revenue rate due to swift urbanization, industrial and special economic zone development, inclusivity of community in developmental projects, safety and security in the port vicinity, social working environment, trade union interference and many more. These challenges, complemented by the growing importance and focus on the sustainable development of shipping ports, have led to a surge in research publications on sustainability-related topics that concentrate on environmental, economic, and social aspects [ 26 , 27 ].

A literature review gives insight into the focus of extant studies related to port sustainability, which is more on quantifying and measuring various dimensions of sustainability against benchmarks and developing indexes for multiple sizes of sustainability. However, qualitative studies need to be conducted to understand the extent to which sustainability measures are adopted and the interaction between the measures, directing towards empirical data-driven studies [ 27 , 28 ]. Alamoush et al. [ 2 ], in their port sustainability framework development study, found that only 16 percent of the articles published were empirical and were based on questionnaire surveys and personal interview data. At the same time, the majority, 40 percent, was conceptual and theoretical review, and the remaining was equally distributed among simulation and case studies. Empirical data-driven research on sustainability-related topics and port performance will be critical to the growing body of knowledge [ 28 ]. Further, the empirical studies on port sustainability [ 2 , 27 , 29 , 30 , 31 , 32 , 33 ] have adopted various indicators of sustainability and criteria for sustainability evaluation based on available scales from studies not directly related to port, rather context-adjusted for port industry. Moreover, despite the increasing number of empirical studies on sustainability in the port sector, in line with the claim made by Ayre et al. [ 34 ] on the hardly ever reporting of content validation by researchers, the existence of any content validation process adopted in any studies and the validated instrument so developed to measure sustainability and sustainability practices objectively for shipping ports is not traceable in the extant literature, especially for exponentially developing economy like India. The study of Ashrafi et al. [ 17 ], which pointed out their pilot study for validation, only assessed the perception of port sustainability in the US and Canada through an online survey to identify the primary factors and challenges in adopting and implementing sustainability strategies. However, the instrument needed to be more generic in capturing the overall sustainability barriers and influencing factors and needed a specific macro-level assessment of the three dimensions of sustainability.

There are studies [ 35 , 36 , 37 ] that discuss the importance of content validity to determine whether the measurement item used in the tool and the extent to which the tool is satisfactorily representing and addressing the domain of interest along with its relevance when measured. Thus, the need for precise, validated measurement tools for sustainability practices at shipping ports indicates a critical knowledge gap in the existing literature. It should also be noted that most seaport-related studies in the scholarly database concentrated on specific geographical areas in Europe [ 32 , 38 ] but not on the leading and growing economies like India. Another concern is that although sustainability is a widely discussed topic, there still needs to be a single universally acceptable and established definition for sustainability [ 39 , 40 , 41 ]. According to Maletič et al. [ 40 ], even though many have attempted to define and measure sustainability, there is an ongoing debate in the literature [ 42 ] on the existence of multiple ways to measure sustainability practices. Therefore, there is a vital need to have clarity and substantial justification on the dimensions and indicators that define the sustainability construct and standardize the assessment of sustainability to a greater extent, especially for seaports. Further to developing policies and schemes for sustainability, the implementations are essential for guaranteed sustainable development [ 28 , 43 ] and measuring the extent to which the port has focused on various sustainability practices can be a tool to assess the efforts towards sustainable development of the shipping port. Alamoush et al. [ 2 ] also pointed out their primary observation on the lack of study linking the port sustainability actions with the three sustainability dimensions represented by the three sustainability practices.

Considering this existing crucial gap and challenges discussed above, the novel contribution of this study is a validated instrument for assessing the sustainability practices followed at shipping ports covering the dimensions of sustainability. The measuring instrument can act as a guideline for seaport administrators and stakeholders to evaluate sustainability in shipping ports and develop seaport sustainability policy for sustainable maritime growth and sustainable development, specifically for an empirical and objective evaluation of sustainability practices adopted in shipping ports in India. Given this compelling necessity to have a content and construct validated instrument for sustainability assessment and strategy development, specifically in the context of Indian seaports, this study aims to explore, design, and develop an instrument for objectively assessing sustainability practices in shipping ports through a well-established content validation process. To achieve the aim of the study, the objectives are:

To identify the comprehensive list of dimensions of sustainability practice for shipping ports through an extensive literature survey

To validate the content of the measurement tool using globally accepted content validation indices viz Content Validity Index, Kohens Kappa coefficient, and Content Validity Ratio

To ascertain the factor structure of the measurement model using Confirmatory Factor Analysis

To estimate the relationship and contribution of three dimensions of sustainability practices to the higher-order sustainability practices construct

The study is structured into the following three major sections. The following section explores the theoretical foundations of sustainability construct and the related conceptual framework that shapes shipping ports’ sustainability dimensions. The following section covers the research methodology for achieving the study’s objectives. It outlines the steps followed in item identification through literature review, instrument development, and instrument assessment based on globally accepted indices and measurement model structure evaluation for validity and reliability using confirmatory factor analysis leading to Partial Least Square-Structural Equation Modeling (PLS-SEM) methods for relationship estimation and prediction of the relationship among the variables. Finally, the results are critically discussed, with the findings of the study highlighting the implications leading to a conclusion along with future research directions.

2 Theoretical background

Although certain sustainability practices are compelled by regulatory compliance, organizations are fortified to adopt and engage voluntarily and proactively in other sustainability practices to meet the needs of the broader society within which they operate [ 44 ]. Extant studies on sustainability [ 7 , 14 , 16 , 17 , 18 , 45 , 46 ] have discussed the need to integrate sustainability efforts into organizational goals, processes, and initiatives and link them with organizational strategy, without which the efforts are likely to fail. The goal for any firm is to secure a competitive advantage over its competitors, create wealth, capture the highest possible market share, and add value to the stakeholders while maintaining a balance between sustainability and economic growth [ 4 ]. Studies [ 16 , 47 ] have opined sustainability to be one of the most crucial aspects that influence the competitiveness of ports. Moreover, to achieve and sustain competitive advantage, organizations have been increasingly grappling with sustainability practices [ 8 ]. Simpson et al. [ 48 ] define practices as “the customary, habitual, or expected procedure or way of doing something. “The practices focused on sustainability could differ from industry to industry, and the shipping industry could concentrate on various practices, and relevant systems would be in place to support sustainable growth and development. Kang et al. [ 31 ] highlight the best practices that embrace sustainability and suggest many practices related to operations, resource optimization, safety and security, finance, risk, infrastructure upgradation, stakeholder management, environmental management systems, and the Port’s eco-friendly and social work environment.

Discussions in prior studies indicate mixed responses regarding the definition of sustainability. There is no universally acceptable definition [ 39 , 42 , 49 , 50 ], but a more generic definition emphasizes sustainability as the set of business strategies, policies, and associated practices or activities where the requirement of the present is satisfied without impacting the requirements of the future in the best interests of the port and related stakeholders. Different schools of thought have a general opinion that sustainability encompasses the three significant dimensions popularly termed as the triple bottom line (TBL) dimensions of “economic, social, and environmental practices,” which comprehend the broad framework of sustainable development [ 39 , 49 , 50 , 51 ]. Elkington [ 51 , 52 ] introduced the triple-bottom-line approach (TBL) incorporating these interrelated three dimensions—“environmental sustainability, economic sustainability, and social sustainability,” advocating organizations to adopt the TBL approach for long-term success [ 53 , 54 , 55 ], rather than short-term success focusing only on the economic dimension. The TBL aspects are considered the critical dimensions of sustainability [ 56 ]. Environmental dimensions concentrate on policies, initiatives, and practices that promote environmental management. In contrast, economic dimensions focus on policies, initiatives, and practices related to investments, economic benefits, and returns from those investments [ 57 ]. Social sustainability focuses on policies, initiatives, and practices that promote the overall improvement of society at large, including all other stakeholders [ 58 ]. Bansal [ 59 ] asserted that the three pillars, i.e., environmental integrity, economic prosperity, and social equity, should intersect for sustainability. Alamoush et al. [ 2 ] further related these dimensions of TBL to the planet, profit, and people as synonyms for environment, economic, and social sustainability.

In the context of ports-related studies, various environmental, economic, and social dimensions were adopted to assess the sustainability of the ports using different methodologies [ 2 , 27 ]. Oh et al. [ 29 ] adopted the importance-performance analysis technique to evaluate the sustainability of South Korean ports using 27 vital measures of the sustainability of ports adapted from the findings and discussions of previous research and found that those measures are essential from a port sustainability point of view. Their study classified the indicators of port sustainability in the three dimensions of sustainability as opined in the TBL concept. In contrast to this empirical quantitative approach, Vejvar et al. [ 33 ] adopted a case-study-based approach to study the institutional forces that compel organizations to adopt sustainability practices. However, they adopted open-ended questions to probe the sustainability practices adopted in the selected shipping ports. They performed a cross-case analysis to make the study more generalizable and increase the validity of the findings [ 32 ]. A thematic analysis of the sustainability performance of seaports was conducted, followed by semi-structured interviews. Later, a fuzzy analytical hierarchy process was applied to compute the weight for each port sustainability performance indicator. Their study also categorized the indicators into three dimensions of sustainability performance, namely social, environmental, and economic sustainability performance practices. Another multi-dimensional framework of sustainability practices was empirically tested by Maletič et al. [ 40 ], and they defined sustainability exploitation and exploration as two different sustainability practices. According to them, sustainability exploitation practices aim at incremental improvement in organizational processes, and sustainability exploration challenges current practices with innovative concepts in developing competencies and capabilities for sustainability. However, they also acknowledged the suitability of more objective measures, such as the TBL practices for sustainability studies. Sustainability practices aid organizations in developing opportunities while managing the three dimensions of organizational processes—economic, environmental, and social aspects in value creation over the long term [ 51 ]. In that definition given, profitability is the focus of economic sustainability, protection and concern towards the environment is the focus of environmental sustainability [ 60 , 61 ], and social sustainability focuses on sustained relations with all the stakeholders, including suppliers, customers, employees, and the community as well [ 62 ].

Regarding developing an index related to sustainability, Laxe et al. [ 43 ] developed the “Port Sustainability Synthetic Index” covering economic and environmental indicators using a sample of 16 ports in Spain. Molavi et al. [ 25 ] developed a novel framework for the smart port index for achieving sustainability using key performance indicators (KPI) that can assist in strategy development and policy framing. Their study indicated several sub-domains of environmental domains in the smart port index study, along with other domains such as operations, energy, safety, and security. However, their study mentioned environment-related quantitative KPIs and other domains that can be used to evaluate the smart port index. Still, it did not mention economic and social, although the sub-domain can be related to economic and social dimensions. In contrast, Stanković et al. [ 63 ] developed a novel composite index for assessing the sustainability of seaports covering environmental, economic, and social dimensions through its indicators based on the secondary data available in the Eurostat and the Organization for Economic Co-operation and Development database. However, the environmental dimension captured only air pollution particulate matter value as the only indicator. Their study also mentioned the limitations of not covering many indicators, including social inclusion and waste management, due to the unavailability of the database. These limitations of quantitative data available in secondary databases for index inclusion are also challenging. The data collection across ports is not yet standardized, and the diverse type of cargo handled in ports makes the index not universally adaptable. Mori et al. [ 64 ] had the same opinion about avoiding a synthesized composite index due to the chances of offset in evaluation [ 65 ]. Although indices have benefits, standard data availability limitations for computing indexes are another added concern that limits index adoption for benchmarking and assessment, thus making sustainability index adoption with caveats.

Therefore, following the justifications and proven theoretical foundations discussed above, this study is grounded on sustainability theory orchestrated by the TBL view, which incorporates the three interrelated dimensions of sustainability—“environmental sustainability, economic sustainability, and social sustainability” and the relevant sustainability practices focused on shipping ports. Therefore, based on previous studies on sustainability and sustainability practices, this study considers sustainability constructs, namely environmental sustainability practices (EnvSP), economic sustainability (EcoSP), and social sustainability practices (SocSP). The indicators thus identified would be used as the measurement scales to empirically measure through survey instruments and objectively evaluate sustainability practices adopted in shipping ports in India.

3 Methodology

The authors adopted the content validation process prescribed by Barbosa et al. [ 35 ]. The process starts with item identification based on an extensive literature review, instrument assessment by subject matter experts, and instrument evaluation with suitable content validation indices. This was followed by assessing the validity and reliability of the hypothesized model to confirm the theory established in the literature using Confirmatory Factor Analysis (CFA) [ 66 ]. CFA is the most widely adopted statistical technique that helps to determine the underlying structure among a set of latent variables and confirm the reliability of how well the scale measures the proposed concept. Hair et al. [ 67 ] elucidated on CFA as a technique to assess the contribution of each scale of item on the latent variable, which later can be incorporated into the estimation of the relationships in the structural model along with accounting for associated measurement error using the variance-based Partial Least Squares-Structural Equation Modelling (PLS-SEM) framework. Explaining the relationship between the exogenous and endogenous variables and predicting the variation in the relationship is the primary focus of PLS-SEM. The five-stage procedure adopted in the study is shown in the flow chart Fig.  1 below.

Content validation process for the study (Author’s own)

3.1 Stage #1. Item identification and face validation

In the first stage, an extensive review of relevant articles related to port studies was performed to compile a comprehensive list of items related to the three dimensions of sustainability viz environmental, economic, and sustainability practices, with the help of a relevant keyword search in the Scopus database in the context of shipping ports. Multiple iterations with different combinations of keywords were performed to see the diversity of articles that can be traced in the scholarly database and the final set of keywords as [(“sustainability”) OR (“sustainability practices”)] AND [(“shipping ports”) OR (“maritime ports”) OR (“container ports”)] AND (scale OR items OR measurement OR indicators OR SEM) were adopted in article identification and followed by screening of articles for item identification and compilation of the list of items related to sustainability practices. The final set of related articles was critically reviewed to identify the relevant items for sustainability practices at shipping ports. Following the recommendation of Boateng et al. [ 68 ], face validation of the instrument was conducted with review and inputs from two senior academicians and experts with theoretical and practical knowledge of sustainability practices.

3.2 Stage #2. Instrument assessment

A subject matter expert panel selection followed this in the second phase of the content validation process to perform instrument assessment. Typically, experts evaluate the content validity, and for that, the recommended minimum number of experts is three and can go up to a maximum of 10 [ 68 , 69 , 70 ]. Following this study’s prescribed expert number requirement, a panel comprising six experts from academic and Port industry backgrounds assessed and validated items. Barbosa and Cansino [ 35 ] claim no unique formula or approach for selecting an expert panel exists. However, it points out the need for a heterogenous panel to mitigate the risk of biases in the validation. Therefore, the study included subject experts from the port industry and academicians with experience in port-related research studies.

The relevance of the items identified through literature review from various sources and the essentiality of these items are supposed to be assessed and content validated through the instrument assessment by the panel of experts. For content validation and evaluation, the Content Validity Index (CVI), Cohens Kappa coefficient, and Lawshe’s Content Validity Ratio (CVR) were adopted as they are the most widely adopted content validation tools for quantifying the opinions of experts [ 69 , 71 , 72 , 73 , 74 ]. The items were assessed for relevance on a 4-point Likert scale. The 4-point Likert scale for relevance captured the response as “1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, and 4 = very relevant” for every item in the measurement instrument. Further, the items were assessed on a 3-point Likert scale to capture the extent of essentiality. Moreover, the 3-point Likert scale for essentiality captured the response as “1 = not essential; 2 = useful, but not essential; and 3 = essential”. Further, an additional comments column for each item was also provided to add feedback and remarks by the expert against each item.

3.3 Stage #3. Content validation using CVI, Cohen’s Kappa, and CVR index

Following the recommendation of [ 75 ], the validity of the instrument content was assessed using CVI, Cohen’s Kappa coefficient, and CVR indices. CVI is a straightforward computation of the agreement among the panelists and can be computed at both the individual item level and the overall scales [ 71 , 72 , 73 , 76 ]. Accordingly, I-CVI is the validity index for each item of the constructs of the study, whereas S-CVI is for the overall scale, which is calculated as the average of I-CVI. I-CVI can be determined as the ratio of several panelists’ ratings on a scale of 3 and above for each measurement item and the total panelists evaluating the relevance. Along similar lines, S-CVI can be computed using the number of measurement items in the assessment tool with a rating of 3 and above for each measurement item. To complement and increase the strength of assessment of relevance through CVI, Barbosa and Cansino [ 35 ] highlight the benefit of Cohen’s Kappa coefficient for evaluation of content validation with due consideration of the degree of agreement on the measurement item beyond certain chance along with the associated probability of inflated scales of agreement merely due to chance agreement. The formula to compute Cohen’s Kappa coefficient is as follows:

where the total number of experts is denoted as N, and it indicates the total number of subject matter experts indicating “essential,” P c is the probability of chance agreement and computed as:

According to Lawshe (1975), the CVR index can be computed using the formula:

The formulas described above were entered in a spreadsheet for the computation of CVR, Cohen’s Kappa coefficient, and CVI based on the rating given by the experts for the items identified. The scale of relevance and necessity of items marked by each expert was recorded and coded into the spreadsheet to facilitate the computation of indices for every item and the entire scale.

3.4 Stage #4. Reliability and validity using confirmatory factor analysis

3.4.1 sampling and data collection.

The content-validated questionnaire instrument was administered online through Microsoft Forms as well as offline to port employees working at the mid and senior management (Officer Category) level employees of various significant ports located in both the western and eastern coastal belts of India for data collection to test the validity and reliability of the measurement model. The instrument captured the respondents’ demographics and perceptions of how much the Port focuses on the three pillars of sustainability practices adopted in their respective ports. Many authors have opined the choice of sample size determination in business management and social science based on G-power [ 77 , 78 , 79 , 80 ]. As per the calculation, for an effect size at a medium level, implied as 0.15, a 5% significance level, and a power level of eighty percent, the recommended minimum sample size was 166. Further, Hair et al. [ 67 ] have outlined the guideline for the minimum sample size for a model structure with less than or equal to seven constructs as 150. However, to avert any possible statistical loss, the rationalized sample size was determined to be 20 percent over 166, thereby establishing the sample size required for the study to be 200. The respondents had to indicate their level of agreement on each item on a Likert scale of 1 to 5, 1 indicating “Strongly Disagree” and five indicating “Strongly Agree”. The data collection activity was carried out between January and December 2023 until the required sample usable data was received for further analysis.

3.4.2 Reliability and validity of the measurement model

Confirmatory factor analysis (CFA) was performed to check the factor structure confirmation of the sustainability practices dimensions using the sample data collected with IBM AMOS. In contrast to measurement error, reliability is the indicator of the “degree to which the observed variable measures the true value and is error-free [ 66 ]. It is also an “assessment of the degree of consistency between multiple measurements of a variable and the set of variables being measured.” Model fit, reliability, and construct validity indices were assessed based on the recommendations by [ 81 , 82 ]. Construct reliabilities were evaluated using Cronbach’s alpha, Composite reliability, and Average Variance Extracted (AVE) values. Construct validity was assessed using convergent validity and discriminant validity measures.

3.5 Stage #5. Structural equation modeling

SEM methodology facilitates the indirect measurement of unobserved latent variables with the measurement of indicator items for the variables in the model structure [ 82 ]. SEM methodology assesses how the latent variables in the model are related to one another and accounts for any errors in the measurement of the observed variables. Therefore, we adopted the PLS-SEM technique to estimate the relationship between the three dimensions of sustainability and their contribution to the overall sustainability construct through different item indicators for each of the dimensions of sustainability. Further, as per the recommendation of Hair et al. [ 83 ], the variance-based PLS-SEM framework is more suitable for our study because the sample size is comparatively less, and the normal distribution assumption is not significant for our study due to the innate nature of the items measuring the dimensions of sustainability. To specify the model parameters and estimate the relationship between the higher and lower-order constructs, we used SmartPLS [ 84 ], the most popular tool for PLS-SEM.

In the first stage of the analysis, items were identified based on an extensive literature review and face validation, instrument assessment by subject matter experts later, and instrument evaluation with suitable content validation indices. This was followed by assessing the validity and reliability of the hypothesized model to confirm the theory using CFA, leading to path relationship assessment using PLS-SEM methodology.

4.1 Item identification and face validation

The literature review compiled a comprehensive and exhaustive initial list comprising 48 items as indicators of sustainability practices adopted in shipping ports and the source (refer to Appendix 1 ). The initial list of items identified comprised 17 items as indicators of Environmental sustainability practices, 19 as indicators of environmental sustainability practices, and 12 as indicators of social sustainability practices adopted in shipping ports. The initial draft of the measuring instrument was subjected to face validation. The inputs from the two senior academicians and experts who carried out face validation were incorporated, which included necessary corrections such as the elimination of ambiguous terms, the inclusion of other indicators that were not included in the initial list, rephrasing of the sentences in the instrument for a better understanding of the context of the study along with the final formatting of the layout [ 68 ]. After incorporating the corrections of face validation along with their source, the items were compiled in the measurement instrument for content validation in the next stage.

4.2 Instrument assessment and content validation using CVI, Kappa, and CVR index

In the second stage, six selected subject matter experts conducted the content validation of the face-validated instrument. “Content validity is a subjective approach that evaluates the extent to which the content described through scale measures certain factors of study interest. Content validation evaluates whether the items in the questionnaire instrument are clear, readable, and relevant to the study context [ 85 ]. The relevance of the items, as well as the essentiality of these items, are supposed to be assessed and validated through the instrument assessment by the panel of subject matter experts. The assessment tool of the study instruments was administered to the six subject experts who were impaneled. Table 1 summarizes the profile of the experts who participated in validating the questionnaire items.

CVI and Kappa coefficients were calculated to assess the relevance of the items, and CVR was calculated to determine the items’ essentiality for the study context [ 74 ]. The responses of the experts on the item’s relevancy and essentiality were coded to spreadsheets for computation of CVI, Kappa coefficient, and CVR value as per the respective formulas [ 69 , 71 , 72 , 73 , 74 ]. The results after the computation of CVI, Kappa coefficient, and CVR are consolidated in Appendix 2 .

CVI indicates the proportion of experts who agreed on the tool and the measurement of the items for a given construct by considering ratings of 1 and 2 as invalid, whereas 3 and 4 are valid contents and consistent with the study conceptual framework [ 74 ]. Adopting the cut-offs suggested in previous established studies [ 69 , 74 ], items with a CVI of at least 0.84 were accepted in the validation. The validation tool indicated S-CVI as 0.86 and satisfies the minimum requirement of 0.80 per Shrotryia et al. [ 86 ] for an instrument to be considered content valid. Along with that, the Cohens Kappa coefficient was also computed with a cut-off of 0.74 to avoid any errors due to chance agreement by the expert panel [ 71 , 72 , 73 , 75 , 76 ]. According to Lawshe’s benchmark CVR index, for a panel size of 6, the cut-off CVR value prescribed is 0.99. It indicates agreement among the panel judges on the item’s necessity in the study questionnaire. Based on these inclusion criteria of CVI, Kappa, and CVR, the most essential and relevant shortlisted items and the final questionnaire were administered for construct validation. The finalized instrument for measuring the extent of the adoption of sustainability practices in shipping ports is shown in Appendix 3 .

4.3 Construct validity and reliability using confirmatory factor analysis and PLS-SEM

Empirical studies attempt to validate and justify the research framework developed with the help of primary data collected from respondents through a questionnaire instrument. Since the analysis solely depends on the data collected through the instrument and the data collected are not accurate measurements of factors of interest but observations of the respondent’s perceptions, the questionnaire should be subjected to validation and reliability checks [ 85 ]. The validation and reliability checking procedures aim to measure and address the measurement error caused by the difference in the actual scores measures from the measured or observed scores [ 66 ]. Validity exemplifies the extent to which the collected data represents the study’s primary purpose, in other words, “measuring what it proposes to measure. The content-validated measurement instrument was administered to port employees of Officer and above designation across various significant ports in India for data collection. Table 2 shows the demographic profiles of the samples who gave the responses to the questions in the instrument administered.

The goodness-of-fit indices were evaluated for the reflective measurement model considering the recommendation [ 81 , 82 ]. The model fit indexes for the hypothesized model were acceptable considering the benchmark recommended values [ 66 , 87 ]. The results [ \( \chi^{2} /\) df was 1.6, Goodness-of-fit index (GFI), Tucker Lewis Index (TLI) and Comparative Fit Index (CFI) > 0.9, Standardized Root Mean Square Residual (SRMR) = 0.053, and Root Mean Square Error of approximation (RMSEA) = 0.055] indicated acceptable model fit as per the recommendations. The standardized factor loading, construct validity, and reliability values are shown below in Table  3 .

Although Hair et al. [ 67 , p. 152–153] suggest a minimum factor loading benchmark value of 0.7 for statistical significance in general, it is also meant to consider 0.50 or above as practically significant in addition to another guideline recommending statistical significance of greater than 0.40-factor loading for a sample size of 200. Further, as per the recommendation of Chin et al. [ 88 ] and considering the practical significance of the items having more than 0.6 loadings, we believe all items with a factor loading above 0.60 are acceptable in the model structure. Therefore, all 26 items are retained in the measurement instrument. Construct reliabilities were assessed using Cronbach’s alpha and Composite reliability measures between 0.85 and 0.90, respectively. Following the reference guidelines by Hair et al. [ 89 ], the measures indicate good and acceptable internal consistency, thereby establishing the scale’s reliability in measuring the construct.

Construct validity was evaluated using convergent and discriminant validity measures except for the EcoSP construct; the other two constructs, viz. EnvSP and SocSP had AVE above the minimum benchmark of 0.50, whereas EcoSP was very close at 0.49. It can be approximated to 0.5, which is correct at the acceptable benchmark for estimating the convergent validity of the measurement model [ 90 ]. There are recommendations that marginal shortfall in AVE is adequate when Cronbach’s alpha and composite reliability are higher than 0.60 [ 89 , 90 ]. These results indicate the acceptable reliability of the scale for measuring sustainability practices in ports.

Hair et al. [ 89 ] emphasize the two established measures of discriminant validity in a model, viz., the Fornell–Larcker criterion and the Heterotrait-monotrait (HTMT) ratio. In the Fornell–Larcker criterion approach, the inter-construct correlations that measure the shared variance between latent variables are compared with the square root of average variance extracted values of the construct. The square root of AVE of the specific construct under consideration is expected to be greater than the particular construct’s highest inter-construct correlation, which signifies the shared variance with other constructs of the model under study. The square root of the AVE of all the constructs was compared with the correlation measures for every build. It was found to be greater than the respective correlation values of the construct under consideration, thereby ascertaining the discriminant validity of the construct. In the HTMT ratio approach, the estimated correlations measured are also termed unattenuated correlation, and the value of unattenuated correlation close to 1 implies an absence of discriminant validity. The benchmark value for the HTMT ratio is 0.90, and any measures above this threshold imply the absence of discriminant validity of the constructs [ 91 , 92 ]. All the measures of discriminant validity assessment indicated HTMT ratio values to be less than 0.9, thus satisfying the discriminant validity requirement of the measurement scale.

Variance-inflation-factor (VIF) was checked for the possibility of multi-collinearity issues [ 89 , 91 , 93 , 94 ]. Multi-collinearity was ruled out as all the VIF values were less than three. The above results support the reliability and validity of the sustainability constructs as collective indicators of the three dimensions of sustainability viz economic, environmental, and social sustainability, and confirm the relationship. Further, the bootstrapping procedure was run to test the significance of the path. The standardized path coefficient values, T-statistics, and p-values shown in Table  4 explain the variance of the three dimensions of the sustainability practice construct. The p-values (< 0.05) indicate that all the structural model relationships are statistically significant.

5 Discussion and implications of the study

The authors followed the systematic procedure of compiling a comprehensive list of related items for the three sustainability practices constructs through an extensive literature review followed by face validation and content validation to assess the relevance and essentiality of the items in the context of shipping ports in India. Empirical studies attempt to validate and justify the research framework developed with the help of primary data collected from respondents through a questionnaire instrument. Since the analysis solely depends on the data collected through the instrument and the data collected are not accurate measurements of factors of interest but observations of the respondent’s perceptions, the questionnaire should be subjected to validation and reliability checks [ 85 ]. The content-validated instrument was subjected to empirical evaluation with sample data collected and using the CFA technique to ascertain the reliability and validity of the model.

Specifically, the results indicate that the subject matter experts have prioritized essential and relevant items in the contemporary business environment, giving nearly equal weightage and importance to all three dimensions of sustainability practices: environmental, economic, and social. Among the items validated, the expert panel had the minor agreement for relevance and necessity on foreign direct investment and funding items, which postulates the shipping ports in India are primarily funded by the government as the minor and significant ports that comprise most of the ports controlled and administered by state and central government respectively. The same reason can be attributed to the low relevance of job security in the context of Indian shipping ports. Further, the items related to odor and smoke also received low relevance as they indicate the low degree of industrial development in shipping ports in India. Although shown as relevant, cold-ironing power sources for vessels on the berth also received a low degree of agreement for necessity. Even recognizing requirements and supporting the community also received little agreement for necessity. However, the remarks provided by the panelist highlight that these focus areas are essentially part of corporate social responsibility, and there is no necessity to assess this separately.

Content validation evaluates whether the items in the questionnaire instrument are clear, readable, and relevant to the study context [ 85 ]. After face and content-validation of the instrument, the finalized list comprised eight items as indicators of EnvSPs, ten as indicators of EcoSPs, and eight as indicators of SocSPs adopted in shipping ports. Thus, the content-validated items for the questionnaire instrument comprised 26 items for measuring the constructs of the study, which is closer to the number of items. Oh et al. [ 29 ] had adopted in the sustainability of ports study. Their study adopted the importance-performance analysis technique to evaluate the sustainability of South Korean ports using 27 vital measures of the sustainability of ports adapted from the findings and discussions of previous research and found that those measures are essential from a port sustainability point of view. Their study classified the indicators of port sustainability in the three dimensions of sustainability as opined in the TBL concept. Along similar lines, Narasimha et al. [ 32 ] conducted a thematic analysis of the sustainability performance of seaports followed by semi-structured interviews. They later applied a fuzzy analytical hierarchy process to compute the weight for each port sustainability performance indicator. Their study categorized the indicators into three dimensions of sustainability performance, namely social, environmental, and economic sustainability performance practices. Therefore, it can be interpreted from the results that these content-validated items are reflective indicators of the sustainability practice constructs and collectively constitute latent variables for empirical studies, confirming that the measurement model reflects the construct validity.

Very Specifically, this study supports the well-established “Tripple Bottom Line” (TBL) theory of sustainability coined by Elkington [ 52 ] that these validated sustainability practice-related items in the measuring instrument adequately represent the seaport domain, and the instrument can be used for measuring the constructs through empirical studies. Even the Sustainable Development Goals (SDG) of the United Nations Development Programme (UNDP) also talks about integrated sustainable development by balancing the three pillars of sustainability: environmental, economic, and social. Chang and Kuo [ 95 ] advise organizations to look at short- and long-term sustainable practices for short-term earnings and safeguard the environment and social integrity simultaneously. Thus, at the strategic level, the TBL practices are the higher-order constructs of sustainability practices, focusing on the long term [ 96 ]. Therefore, the findings of this study contribute to the extant body of literature knowledge by providing empirical evidence on the practical and statistical relationship between the environmental, economic, and social sustainability-related practices of the sustainability construct in the TBL theory-based framework applied for shipping ports.

Yadav et al. [ 97 ] also emphasized the availability of several environmental management systems (EMS) for achieving environmental sustainability. They also recommended introducing methods promoting green culture, supporting green behavior, and improving employee commitment to achieving environmental sustainability. The social dimension of sustainability primarily focuses on facilitating and providing equitable opportunities and the well-being of the port employees and other stakeholders, including the local community, driven by the policies and practices of the port authority. Alamoush et al. [ 2 ] equated economic sustainability to generating revenue and monetary gains and considered economic sustainability to be one of the primary drivers of the other two dimensions—environmental and social sustainability. Further, the results from the PLS-SEM analysis indicate that the most significant contribution towards the overall sustainability of the port is from the economic sustainability dimension of sustainability. Like the findings of Alamoush et al. [ 2 ], the financial investments in the port are the drivers of environmental and social sustainability. Poulsen et al. [ 98 ] proved with facts and figures that air quality was improved even with an increase in cargo throughput, mainly driven by the financial investments in air quality control systems in many ports across Europe.

The improvement in air quality around the port vicinity contributes to environmental sustainability. In addition, it also contributes to social sustainability as the community and the port surroundings, including the ecosystems and the natural habitat for birds and animals, experience better living conditions around the port vicinity. This affirms the indirect benefits achieved in environmental and social dimensions by implementing economic sustainability-related strategies and policies. Our findings also emphasize the need for an integrative approach to achieving sustainability of ports, and it can be achieved only when all three dimensions intersect and contribute to complement each other for overall sustainable development.

This study contributes with both novel theoretical and practical implications. Firstly, the study provides a comprehensive list of items about the indicators of sustainability practices in shipping ports, which are available in published scholarly articles and from domain experts working in the port industry. Secondly, as the first of its kind in the seaport sector, the study adopted a scientific content validation approach of indices and procedures to assess the relevance and essentiality of items in the context of shipping ports and contemporary sustainability practices focused on shipping ports. Our study validated an instrument for assessing the sustainability practices in shipping ports, which is a significant step in formulating policies and developing strategies focusing on the sustainable development of ports. The validated instrument can be adapted to determine the extent of adoption of sustainability practices and drive the necessary implementation through policy centered around the sustainability of shipping ports. The instrument can be a guideline for practitioners, policymakers, and researchers focusing on the sustainable development of shipping ports through environmental, economic, and social sustainability practices. Ports authorities can embrace the validated instrument to assess their level of adoption and focus on these sustainability practices, which will aid in developing policies and strategies for the sustainable development of ports. Further, the Global Reporting Initiative (GRI) Standards, developed by the Global Sustainability Standards Board (GSSB) primarily for sustainability reporting, can be referred to along with our validated instrument for sustainability evaluation and reporting in compliance with the GRI standards [ 99 ]. GRI Standards assist organizations in understanding and reporting the extent to which the organization impacts sustainability and contributes to sustainable development, considering the interests of all the stakeholders, including investors, policymakers, capital markets, and civil society, thus making the organization transparent and responsible for sustainability. Sector-specific standards have been developed, of which ports are part of Group 3, which comprises various Transport, infrastructure, and tourism-related sectors. However, it is not readily available for shipping ports but can be developed and customized by the port authorities. To do so, the findings of the validated instrument of our study can be considered as a guide in assessing and preparing the sustainability report as per the applicable GRI standards.

Further, sustainability assessment should not be considered a one-time activity in the port. Instead, the port authorities should have strategies and policies to track the trends and changes taking place to the extent of adopting sustainability practices in the port and their impact on sustainable development. Each individual port must do it through its team/department or personnel responsible for the sustainability assessment and policy implementation in the port, and it also must be a continuous activity at regular intervals, maybe once in 3 months or 6 months, depending on the policy and management decision. Thus, the longitudinal assessment, which keeps track of the various aspects of sustainability, will help the port evaluate the effectiveness of sustainability interventions implemented at shipping ports.

6 Limitation and scope for future work

Although the study achieved its objectives of a novel contribution of a content-validated sustainability measurement instrument for assessing sustainability practices in seaports, there were a few limitations, and there is also further scope for advancing the study in the future. The keywords used in the literature search were confined to published articles in the Scopus database. Future work can expand the search in other scholarly databases and increase the items’ relevance to measuring shipping ports’ sustainability practices. The study was limited to government-controlled significant ports on India’s east and west coasts. Due to permission and access-related challenges, the data collection did not cover the privately managed ports. The items of the study are generalized concerning a shipping port, and further research can consider further refinement specific to the type of cargo handled in the Port or confined to the terminal instead of a generalizable study irrespective of the kind of cargo being handled. The applications of digital technology and automation using Artificial Intelligence and Machine Learning, along with big data and blockchain technology, could be explored to assess their impact on sustainable port management and development. A different methodological approach can be adopted like the study of Yadav et al. [ 97 ], where the “multi-criteria-decision making (MCDM)” approach was used to identify the enablers of sustainability along with the determination of its intensity using “Robust-Best–Worst-method” (RBWM). Their analysis identified economic and environmental-related enablers as the high-intensity enablers of sustainability that organizations can focus on. Other stakeholders, such as customers, port users, government agencies linked with the port operations, and the local community, were not part of the panel for the validation process. In future studies, these other stakeholders can also be considered in the panel so that every aspect is covered in the evaluation. The items were based on a 5-point Likert scale in this study to capture only the perception of port employees on the sustainability practices adopted in the port. A suitable triangulation method and case studies can also be used to analyze the qualitative aspects of adopting sustainability practices in the port.

7 Conclusion

The study validated an instrument for assessing the sustainability practices in shipping ports, which is a significant step in formulating strategies focusing on the sustainable development of ports. The instrument can be a guideline for practitioners, policymakers, and researchers focusing on the sustainable development of shipping ports through environmental, economic, and social sustainability practices. The study prepared a comprehensive list comprising relevant items identified through a thorough literature review of articles published in the Scopus database. After face validation, the measurement tool was administered to six subject matter experts who evaluated its relevance and essentiality in measuring sustainability practices in shipping ports. The content validity was assessed using the most widely used and adopted indices: CVI, Cohen’s Kappa’s coefficient, and CVR. CVI and Cohen’s Kappa’s coefficient are the indices for assessing the relevance of the items in measuring sustainability practices, and CVR is the index for determining the essentiality of the items in measuring sustainability practices in shipping ports. Further, this study contributes to the extant body of literature by providing evidence on the empirical relationship between the environmental, economic, and social sustainability-related practices of the sustainability construct in the TBL theory-based framework applied for shipping ports.

Data availability

The data for analysis in the study was based on survey data collected through a questionnaire instrument administered on Likert scales, both online and offline modes of data collection. The data collection period was between December 2022 and Dec 2023. The instrument had a declaration mentioning maintaining the privacy of the participants and therefore, the data cannot be made public to protect study participant privacy. The primary data collected in the study are not publicly accessible but are available from the corresponding author upon reasonable request.

Meixell MJ, Luoma P. Stakeholder pressure in sustainable supply chain management: a systematic review. Int J Phys Distrib Logist Manag. 2015;45:69–89. https://doi.org/10.1108/IJPDLM-05-2013-0155 .

Article   Google Scholar  

Alamoush AS, Ballini F, Ölçer AI. Revisiting port sustainability as a foundation for the implementation of the United Nations Sustainable Development Goals (UN SDGs). J Shipp Trade. 2021;6(1):1–40. https://doi.org/10.1186/S41072-021-00101-6 .

Dyllick T, Hockerts K. Beyond the business case for corporate sustainability. Bus Strateg Environ. 2002;11(2):130–41. https://doi.org/10.1002/bse.323 .

Lun YHV, Lai K, Wong CWY, Cheng TCE. Green shipping management. Cham: Springer International Publishing; 2016. https://doi.org/10.1007/978-3-319-26482-0 .

Book   Google Scholar  

Porter ME, Van Der Linde C. Green and competitive: ending the stalemate. In: Corporate environmental responsibility. 2017. p. 47–60. https://doi.org/10.1016/0024-6301(95)99997-e .

Russo MV, Fouts PA. A resource-based perspective on corporate environmental performance and profitability. Acad Manag J. 1997;40(3):534–59. https://doi.org/10.2307/257052 .

Roszkowska-Menkes M. Porter and Kramer’s (2006) “shared value.” In: Encyclopedia of sustainable management. Cham: Springer International Publishing; 2021. p. 1–6. https://doi.org/10.1007/978-3-030-02006-4_393-1 .

Chapter   Google Scholar  

Zhu Q, Sarkis J. Relationships between operational practices and performance among early adopters of green supply chain management practices in Chinese manufacturing enterprises. J Oper Manag. 2004;22(3):265–89. https://doi.org/10.1016/j.jom.2004.01.005 .

Hong J, Zhang Y, Ding M. Sustainable supply chain management practices, supply chain dynamic capabilities, and enterprise performance. J Clean Prod. 2018;172:3508–19. https://doi.org/10.1016/J.JCLEPRO.2017.06.093 .

Ministry of Ports Shipping and Waterways. Maritime India vision 2030. Sagarmala; 2021.

Pradhan RP, Rathi C, Gupta S. Sagarmala & India’s maritime big push approach: seaports as India’s geo-economic gateways & neighborhood maritime lessons. J Indian Ocean Reg. 2022;18(3):209–29. https://doi.org/10.1080/19480881.2022.2114195 .

Mantry S, Ghatak RR. Comparing and contrasting competitiveness of major Indian and select international ports. Int J Res Finance Mark. 2017;7(5):1–19.

Google Scholar  

Song DW, Panayides PM. Global supply chain and port/terminal: integration and competitiveness. In: Maritime policy and management. London: Taylor & Francis; 2008. p. 73–87. https://doi.org/10.1080/03088830701848953 .

Yap WY, Lam JSL. 80 million-twenty-foot-equivalent-unit container port? Sustainability issues in port and coastal development. Ocean Coast Manag. 2013;71:13–25. https://doi.org/10.1016/j.ocecoaman.2012.10.011 .

Lee PTW, Kwon OK, Ruan X. Sustainability challenges in maritime transport and logistics industry and its way ahead. Sustainability. 2019;11(5):1331. https://doi.org/10.3390/SU11051331 .

Dragović B, Tzannatos E, Park NK. Simulation modelling in ports and container terminals: literature overview and analysis by research field, application area and tool. Flex Serv Manuf J. 2017;29(1):4–34. https://doi.org/10.1007/s10696-016-9239-5 .

Ashrafi M, Acciaro M, Walker TR, Magnan GM, Adams M. Corporate sustainability in Canadian and US maritime ports. J Clean Prod. 2019;220:386–97. https://doi.org/10.1016/j.jclepro.2019.02.098 .

Peris-Mora E, Orejas JMD, Subirats A, Ibáñez S, Alvarez P. Development of a system of indicators for sustainable port management. Mar Pollut Bull. 2005;50(12):1649–60. https://doi.org/10.1016/j.marpolbul.2005.06.048 .

Article   CAS   Google Scholar  

Ashrafi M, Walker TR, Magnan GM, Adams M, Acciaro M. A review of corporate sustainability drivers in maritime ports: a multi-stakeholder perspective. Marit Policy Manag. 2020;47(8):1027–44. https://doi.org/10.1080/03088839.2020.1736354 .

Stanković JJ, Marjanović I, Papathanasiou J, Drezgić S. Social, economic and environmental sustainability of port regions: MCDM approach in composite index creation. J Mar Sci Eng. 2021;9(1):74. https://doi.org/10.3390/JMSE9010074 .

Dinwoodie J, Tuck S, Knowles H, Benhin J, Sansom M. Sustainable development of maritime operations in ports. Bus Strateg Environ. 2011. https://doi.org/10.1002/bse.718 .

Ports primer: 7.1 environmental impacts | US EPA. https://www.epa.gov/community-port-collaboration/ports-primer-71-environmental-impacts . Accessed Apr 28 2024.

Notteboom T, Pallis A, Rodrigue J-P. Port economics, management and policy. Port Econ Manag Policy. 2021. https://doi.org/10.4324/9780429318184 .

Notteboom T, van der Lugt L, van Saase N, Sel S, Neyens K. The role of seaports in green supply chain management: initiatives, attitudes, and perspectives in Rotterdam, Antwerp, North Sea Port, and Zeebrugge. Sustainability. 2020;12(4):1688. https://doi.org/10.3390/su12041688 .

Molavi A, Lim GJ, Race B. A framework for building a smart port and smart port index. Int J Sustain Transp. 2020;14(9):686–700. https://doi.org/10.1080/15568318.2019.1610919 .

Wu Q, He Q, Duan Y. Explicating dynamic capabilities for corporate sustainability. EuroMed J Bus. 2013;8(3):255–72. https://doi.org/10.1108/EMJB-05-2013-0025 .

Argyriou I, Daras T, Tsoutsos T. Challenging a sustainable port. A case study of Souda port, Chania, Crete. Case Stud Transp Policy. 2022;10(4):2125–37. https://doi.org/10.1016/J.CSTP.2022.09.007 .

Bjerkan KY, Seter H. Reviewing tools and technologies for sustainable ports: does research enable decision making in ports? Transp Res D Transp Environ. 2019;72:243–60. https://doi.org/10.1016/j.trd.2019.05.003 .

Oh H, Lee S-W, Seo Y-J. The evaluation of seaport sustainability: the case of South Korea. Ocean Coast Manag. 2018;161:50–6. https://doi.org/10.1016/j.ocecoaman.2018.04.028 .

Lu CS, Shang KC, Lin CC. Examining sustainability performance at ports: port managers’ perspectives on developing sustainable supply chains. Marit Policy Manag. 2016;43(8):909–27. https://doi.org/10.1080/03088839.2016.1199918 .

Kang D, Kim S. Conceptual model development of sustainability practices: the case of port operations for collaboration and governance. Sustainability. 2017;9(12):2333. https://doi.org/10.3390/su9122333 .

Narasimha PT, Jena PR, Majhi R. Sustainability performance assessment framework for major seaports in India. Int J Sustain Dev Plan. 2022;17(2):693–704. https://doi.org/10.18280/ijsdp.170235 .

Vejvar M, Lai K, Lo CKY, Fürst EWM. Strategic responses to institutional forces pressuring sustainability practice adoption: case-based evidence from inland port operations. Transp Res D Transp Environ. 2018;61:274–88. https://doi.org/10.1016/j.trd.2017.08.014 .

Ayre C, Scally AJ. Critical values for Lawshe’s content validity ratio: revisiting the original methods of calculation. Meas Eval Couns Dev. 2014;47(1):79–86. https://doi.org/10.1177/0748175613513808 .

Barbosa MW, Cansino JM. A water footprint management construct in agri-food supply chains: a content validity analysis. Sustainability. 2022;14(9):4928. https://doi.org/10.3390/su14094928 .

Waltz CF, Strickland OL, Lenz ER. Measurement in nursing and health. Research. 2016. https://doi.org/10.1891/9780826170620 .

Ibiyemi A, Mohd Adnan Y, Daud MN, Olanrele S, Jogunola A. A content validity study of the test of valuers’ support for capturing sustainability in the valuation process in Nigeria. Pac Rim Prop Res J. 2019;25(3):177–93. https://doi.org/10.1080/14445921.2019.1703700 .

Diniz NV, Cunha DR, de Santana Porte M, Oliveira CBM, de Freitas Fernandes F. A bibliometric analysis of sustainable development goals in the maritime industry and port sector. Reg Stud Mar Sci. 2024;69: 103319. https://doi.org/10.1016/j.rsma.2023.103319 .

Amui LBL, Jabbour CJC, de Sousa Jabbour ABL, Kannan D. Sustainability as a dynamic organizational capability: a systematic review and a future agenda toward a sustainable transition. J Clean Prod. 2017;142:308–22. https://doi.org/10.1016/j.jclepro.2016.07.103 .

Maletič M, Maletič D, Gomišček B. The impact of sustainability exploration and sustainability exploitation practices on the organisational performance: a cross-country comparison. J Clean Prod. 2016. https://doi.org/10.1016/j.jclepro.2016.02.132 .

Berns M, Hopkins MS, Townend A, Khayat Z, Balagopal B, Reeves M. The business of sustainability: what it means to managers now. MIT Sloan Manag Rev. 2009;51(1).

Montiel I, Delgado-Ceballos J. Defining and measuring corporate sustainability. Organ Environ. 2014;27(2):113–39. https://doi.org/10.1177/1086026614526413 .

Laxe FG, Bermúdez FM, Palmero FM, Novo-Corti I. Assessment of port sustainability through synthetic indexes. Application to the Spanish case. Mar Pollut Bull. 2017;119(1):220–5. https://doi.org/10.1016/j.marpolbul.2017.03.064 .

Torugsa NA, O’Donohue W, Hecker R. Proactive CSR: an empirical analysis of the role of its economic, social and environmental dimensions on the association between capabilities and performance. J Bus Ethics. 2013. https://doi.org/10.1007/s10551-012-1405-4 .

Lauring J, Thomsen C. Collective ideals and practices in sustainable development: managing corporate identity. Corp Soc Responsib Environ Manag. 2009;16(1):38–47. https://doi.org/10.1002/csr.181 .

Hallstedt SI, Thompson AW, Lindahl P. Key elements for implementing a strategic sustainability perspective in the product innovation process. J Clean Prod. 2013;51:277–88. https://doi.org/10.1016/J.JCLEPRO.2013.01.043 .

Parola F, Risitano M, Ferretti M, Panetti E. The drivers of port competitiveness: a critical review. Transp Rev. 2017;37(1):116–38. https://doi.org/10.1080/01441647.2016.1231232 .

Simpson J, Weiner E, Durkin P. The Oxford English dictionary today. Trans Philol Soc. 2004;102(3):335–81. https://doi.org/10.1111/j.0079-1636.2004.00140.x .

Ruggerio CA. Sustainability and sustainable development: a review of principles and definitions. Sci Total Environ. 2021;786: 147481. https://doi.org/10.1016/J.SCITOTENV.2021.147481 .

Moore JE, Mascarenhas A, Bain J, Straus SE. Developing a comprehensive definition of sustainability. Implement Sci. 2017;12(1):1–8. https://doi.org/10.1186/S13012-017-0637-1/TABLES/3 .

Elkington J. Partnerships from cannibals with forks: the triple bottom line of 21st-century business. Environ Qual Manag. 1998;8(1):37–51. https://doi.org/10.1002/tqem.3310080106 .

Elkington J. Tripple bottom line. In: Cannibals with forks. Oxford: Capstone; 1997.

Waddock SA, Graves SB. The corporate social performance-financial performance link. Strateg Manag J. 1997;18(4):303–19. https://doi.org/10.1002/(SICI)1097-0266(199704)18:4%3c303::AID-SMJ869%3e3.0.CO;2-G .

Sharma S, Vredenburg H. Proactive corporate environmental strategy and the development of competitively valuable organizational capabilities. Strateg Manag J. 1998;19(8):729–53. https://doi.org/10.1002/(sici)1097-0266(199808)19:8%3c729::aid-smj967%3e3.3.co;2-w .

Carroll AB, Shabana KM. The business case for corporate social responsibility: a review of concepts, research and practice. Int J Manag Rev. 2010;12(1):85–105. https://doi.org/10.1111/j.1468-2370.2009.00275.x .

Beske P. Dynamic capabilities and sustainable supply chain management. Int J Phys Distrib Logist Manag. 2012;42(4):372–87. https://doi.org/10.1108/09600031211231344 .

Lam JSL, Li KX. Green port marketing for sustainable growth and development. Transp Policy. 2019;84:73–81. https://doi.org/10.1016/j.tranpol.2019.04.011 .

Olakitan Atanda J. Developing a social sustainability assessment framework. Sustain Cities Soc. 2019;44:237–52. https://doi.org/10.1016/j.scs.2018.09.023 .

Bansal P. Evolving sustainably: a longitudinal study of corporate sustainable development. Strateg Manag J. 2005;26(3):197–218. https://doi.org/10.1002/smj.441 .

Carter CR, Liane Easton P. Sustainable supply chain management: evolution and future directions. Int J Phys Distrib Logist Manag. 2011;41(1):46–62. https://doi.org/10.1108/09600031111101420 .

Janic M. Sustainable transport in the European Union: a review of the past research and future ideas. Transp Rev. 2006;26(1):81–104. https://doi.org/10.1080/01441640500178908 .

Steurer R, Langer ME, Konrad A, Martinuzzi A. Corporations, stakeholders and sustainable development I: a theoretical exploration of business-society relations. J Bus Ethics. 2005;61(3):263–81. https://doi.org/10.1007/s10551-005-7054-0 .

Stanković JJ, Marjanović IM, Papathanasiou J, Drezgić SD. Marine science and engineering social, economic and environmental sustainability of port regions: MCDM approach in composite index creation. J Mar Sci Eng. 2021. https://doi.org/10.3390/jmse9010074 .

Mori K, Christodoulou A. Review of sustainability indices and indicators: towards a new city sustainability index (CSI). Environ Impact Assess Rev. 2012;32(1):94–106. https://doi.org/10.1016/J.EIAR.2011.06.001 .

Mayer AL. Strengths and weaknesses of common sustainability indices for multidimensional systems. Environ Int. 2007. https://doi.org/10.1016/j.envint.2007.09.004 .

Hair J, Black W, Babin B, Anderson R. Multivariate data analysis: a global perspective. In: Multivariate data analysis: a global perspective, vol. 7. Upper Saddle River: Pearson Education; 2010.

Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate data analysis. Hampshire: Cengage Learning; 2019.

Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL. Best practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Health. 2018;6:149. https://doi.org/10.3389/fpubh.2018.00149 .

Elangovan N, Sundaravel E. Method of preparing a document for survey instrument validation by experts. MethodsX. 2021;8: 101326. https://doi.org/10.1016/J.MEX.2021.101326 .

Papadas KK, Avlonitis GJ, Carrigan M. Green marketing orientation: conceptualization, scale development and validation. J Bus Res. 2017;80:236–46. https://doi.org/10.1016/J.JBUSRES.2017.05.024 .

Polit DF, Beck CT. The content validity index: are you sure you know what’s being reported? Critique and recommendations. Res Nurs Health. 2008;31(4):489–97.

Polit DF, Beck CT, Owen SV. Focus on research methods: Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health. 2007;30(4):459–67. https://doi.org/10.1002/nur.20199 .

Zamanzadeh V, Ghahramanian A, Rassouli M, Abbaszadeh A, Alavi-Majd H, Nikanfar A-R. Design and implementation content validity study: development of an instrument for measuring patient-centered communication. J Caring Sci. 2015;4(2):165. https://doi.org/10.15171/jcs.2015.017 .

de Souza AC, Alexandre NMC, Guirardello EDB, de Souza AC, Alexandre NMC, Guirardello EDB. Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade. Epidemiologia e Serviços de Saúde. 2017;26(3):649–59. https://doi.org/10.5123/S1679-49742017000300022 .

Rodrigues IB, Adachi JD, Beattie KA, MacDermid JC. Development and validation of a new tool to measure the facilitators, barriers and preferences to exercise in people with osteoporosis. BMC Musculoskelet Disord. 2017;18(1):540. https://doi.org/10.1186/s12891-017-1914-5 .

Bobos P, Pouliopoulou DVS, Harriss A, Sadi J, Rushton A, MacDermid JC. A systematic review and meta-analysis of measurement properties of objective structured clinical examinations used in physical therapy licensure and a structured review of licensure practices in countries with well-developed regulation systems. PLoS ONE. 2021;16(8): e0255696. https://doi.org/10.1371/journal.pone.0255696 .

Hair JF, Hult GTM, Ringle CM, Sarstedt M, Thiele KO. Mirror, mirror on the wall: a comparative evaluation of composite-based structural equation modeling methods. J Acad Mark Sci. 2017;45(5):616–32. https://doi.org/10.1007/s11747-017-0517-x .

Cohen J. A power primer. In: Methodological issues and strategies in clinical research. 4th ed. Washington: American Psychological Association; 2016. p. 279–84. https://doi.org/10.1037/14805-018 .

Roldán JL, Sánchez-Franco MJ. Variance-based structural equation modeling. In: Research methodologies, innovations and philosophies in software systems engineering and information systems. Pennsylvania: IGI Global; 2012. p. 193–221. https://doi.org/10.4018/978-1-4666-0179-6.ch010 .

Faul F, Erdfelder E, Buchner A, Lang A-G. Statistical power analyses using G*Power 3.1: tests for correlation and regression analyses. Behav Res Methods. 2009;41(4):1149–60. https://doi.org/10.3758/BRM.41.4.1149 .

Goodboy AK, Kline RB. Statistical and practical concerns with published communication research featuring structural equation modeling. Commun Res Rep. 2017;34(1):68–77. https://doi.org/10.1080/08824096.2016.1214121 .

Crawford JA, Kelder J-A. Do we measure leadership effectively? Articulating and evaluating scale development psychometrics for best practice. Leadersh Q. 2019;30(1):133–44. https://doi.org/10.1016/j.leaqua.2018.07.001 .

Sarstedt M, Hair JF, Cheah JH, Becker JM, Ringle CM. How to specify, estimate, and validate higher-order constructs in PLS-SEM. Australas Mark J. 2019;27(3):197–211. https://doi.org/10.1016/J.AUSMJ.2019.05.003 .

Ringle CM, Wende S, Becker J-M. SmartPLS 4. http://www.smartpls.com .

Malhotra S. Study of features of mobile trading apps: a silver lining of pandemic. J Global Inf Bus Strateg. 2020. https://doi.org/10.5958/2582-6115.2020.00009.0 .

Shrotryia VK, Dhanda U. Content validity of assessment instrument for employee engagement. SAGE Open. 2019;9(1):2158244018821751. https://doi.org/10.1177/2158244018821751 .

Bagozzi RP, Yi Y. On the evaluation of structural equation models. J Acad Mark Sci. 1988;16(1):74–94. https://doi.org/10.1007/BF02723327 .

Chin WW, Gopal A, Salisbury WD. Advancing the theory of adaptive structuration: the development of a scale to measure faithfulness of appropriation. Inf Syst Res. 1997;8(4):342–67. https://doi.org/10.1287/isre.8.4.342 .

Hair JF, Hult Jr GTM, Ringle CM, Sarstedt M. A primer on partial least squares structural equations modeling (PLS-SEM). J Tour Res. 2021;6(2).

Fornell C, Larcker DF. Evaluating structural equation models with unobservable variables and measurement error. J Mark Res. 1981;18(1):39–50. https://doi.org/10.1177/002224378101800104 .

Henseler J, Ringle CM, Sarstedt M. A new criterion for assessing discriminant validity in variance-based structural equation modeling. J Acad Mark Sci. 2015;43(1):115–35. https://doi.org/10.1007/s11747-014-0403-8 .

Franke G, Sarstedt M. Heuristics versus statistics in discriminant validity testing: a comparison of four procedures. Internet Res. 2019;29(3):430–47. https://doi.org/10.1108/IntR-12-2017-0515 .

Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model. 1999;6(1):1–55. https://doi.org/10.1080/10705519909540118 .

Becker JM, Ringle CM, Sarstedt M, Völckner F. How collinearity affects mixture regression results. Mark Lett. 2015;26(4):643–59. https://doi.org/10.1007/s11002-014-9299-9 .

Chang DS, Kuo LCR. The effects of sustainable development on firms’ financial performance—an empirical approach. Sustain Dev. 2008;16(6):365–80. https://doi.org/10.1002/sd.351 .

Ogunbiyi O, Oladapo A, Goulding J. An empirical study of the impact of lean construction techniques on sustainable construction in the UK. Constr Innov. 2014;14(1):88–107. https://doi.org/10.1108/CI-08-2012-0045 .

Yadav G, Kumar A, Luthra S, Garza-Reyes JA, Kumar V, Batista L. A framework to achieve sustainability in manufacturing organisations of developing economies using industry 4.0 technologies’ enablers. Comput Ind. 2020;122: 103280. https://doi.org/10.1016/j.compind.2020.103280 .

Poulsen RT, Ponte S, Sornn-Friese H. Environmental upgrading in global value chains: the potential and limitations of ports in the greening of maritime transport. Geoforum. 2018;89:83–95. https://doi.org/10.1016/J.GEOFORUM.2018.01.011 .

GRI -Standards. https://www.globalreporting.org/standards/ . Accessed 09 May 2024.

Lu C-S, Shang K-C, Lin C-C. Identifying crucial sustainability assessment criteria for container seaports. Marit Bus Rev. 2016;1(2):90–106. https://doi.org/10.1108/MABR-05-2016-0009 .

Download references

Acknowledgements

We acknowledge the contribution of the expert panel for their reviews and feedback that enabled us to optimize the items in the instrument.

Open access funding provided by Manipal Academy of Higher Education, Manipal. This study has not received any funding from any institutions or agencies.

Author information

Authors and affiliations.

Department of Commerce, Manipal Academy of Higher Education, Manipal, 576104, India

Department of Humanities and Management, Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal, 576104, India

Yogesh P. Pai

T A Pai Management Institute, Yelahanka, Govindapura, Bengaluru, 560064, Karnataka, India

Parthesh Shanbhag

You can also search for this author in PubMed   Google Scholar

Contributions

All the authors contributed to the manuscript equally K.L conceptualized the study and executed data collection All the authors jointly performed data analysis and authored the manuscript All authors reviewed the manuscript before submitting.

Corresponding author

Correspondence to Yogesh P. Pai .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix 1. Construct-wise list of items and source

Appendix 2. Results of content validity

• i-CVI indicates item CVI, Pc the probability of a chance occurrence, and K—the kappa statistic

Appendix 3. Instrument for data collection

4.1 section a—demographic profile.

4.2 Section B—practices related to port

Please indicate the extent to which you agree on statements related to your port on a scale of 1–5.

1—strongly disagree, 2—disagree, 3—neutral, 4—agree, 5—strongly agree.

If you are unaware of the Port’s practices, you may choose “3-Neutral.”

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Kishore, L., Pai, Y.P. & Shanbhag, P. Reliability and validity assessment of instrument to measure sustainability practices at shipping ports in India. Discov Sustain 5 , 236 (2024). https://doi.org/10.1007/s43621-024-00395-z

Download citation

Received : 27 January 2024

Accepted : 02 August 2024

Published : 03 September 2024

DOI : https://doi.org/10.1007/s43621-024-00395-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Shipping ports
• Sustainability practices
• Measurement instrument
• Content Validity Index
• Cohen’s Kappa Index
• Content validity ratio
• Confirmatory factor analysis
• Structural equation model
• Find a journal
• Publish with us
• Track your research