Community Members as Reviewers of Medical Journal Manuscripts

The process of publishing a scientific paper in a medical journal involves an author(s) submitting a manuscript (a written paper about a research topic or study) to the journal editor. The editor then assigns the manuscript for review by scientific reviewers or subject matter experts—such as physicians, researchers, or other scientists—in the research area that the manuscript is based on. Journal editors use scientific reviews to make decisions about whether to accept or reject a manuscript for publication.

The scientific review process does not typically include community members (e.g., patients or caregivers), even though they may be involved in the research process and may have valuable insights and contributions that can improve the quality of the research findings. An NIMHD-funded study determined that community members can review medical manuscripts and provide useful feedback to journal editors if they are trained, supervised, and compensated.

Findings from medical research, which help researchers and physicians learn more about diseases and conditions to improve health and health care, are usually published in medical journals. Community members may be involved in several phases of a research process, which indicates that they have knowledge and experience that can be used to improve understanding of research. In addition, community members may possess other types of knowledge (such as personal experiences as patients or caregivers, community perspectives, cultural beliefs) that can enhance the ways that research findings are communicated and shared to reach a wider audience. Yet, medical journal editors do not include community members in the scientific review process due to concerns that they lack basic knowledge about research issues and methods, lack understanding of scientific language or data analyses, have unrealistic expectations of medical research, or lack objectivity. To determine whether community members can review medical manuscripts and provide useful feedback, several researchers collaborated to conduct a type of research study known as randomized controlled trial.

Randomized controlled trials are research studies that evaluate the effectiveness of a new approach or intervention. In this study, researchers evaluated the effectiveness of community members reviewing medical manuscripts. The researchers recruited 59 eligible community members, but some did not meet all the study requirements, and some dropped out of the study; a total of 16 community members participated until the end of the study.

The researchers recruited individuals who were at least 18 years of age, had at least a high school education, were proficient in reading and writing in English, and had access to a computer. To enhance community members’ ability to provide useful reviews, those recruited were also required to have a personal experience as a patient or caregiver with at least one of the following common medical conditions: ischemic heart disease, congestive heart failure, stroke, hypertension, diabetes, cancer, chronic obstructive pulmonary disease, cirrhosis, Alzheimer’s disease, and kidney disease. However, individuals who previously worked as health care providers, health care support staff, or researchers were excluded, because the researchers did not want a community perspective that would be influenced by working professionally in health care or research.

Selected individuals were asked to answer questions about a research topic and those who correctly answered at least four questions were invited to a 6-week training on a wide range of research topics; they were also given several opportunities to practice reviewing manuscripts. Only individuals who completed the training and able to write an effective review by the end of the training were invited to participate in the study as community reviewers.

The researchers conducted this study in collaboration with two medical journals: Annals of Internal Medicine and Annals of Family Medicine. The researchers asked the staff of these journals to send them all full-length original research manuscripts that their editors planned to send for scientific review. Out of these, the researchers selected 578 manuscripts, which excluded those that community reviewers were less likely to provide useful comments about (including manuscripts that had a statistical focus, were about medical practice or medical education, or addressed a rare health condition). The researchers then assigned these manuscripts randomly and equally between two groups: scientific review only (or the control group) and community and scientific reviews (or the intervention group). All the manuscripts were sent to scientific reviewers for review. After the scientific review, the intervention group manuscripts only were sent to community members for additional review. In other words, scientific reviewers reviewed both control and intervention group manuscripts, while community members reviewed only intervention group manuscripts. All comments provided in the reviews by scientific and community reviewers were submitted to the journal editors.

To determine the effectiveness of community members reviewing medical manuscripts—

1. The researchers examined all the reviews and categorized their content by two major themes, namely, study attributes and viewpoints. Study attributes are all content areas or topics in the manuscripts that reviewers commented on. Viewpoints are reviewers’ perspectives on the research described in the manuscripts. The researchers also compared the content of community and scientific reviews.
2. The researchers asked the editors to numerically rate the scientific and community reviews, and then provide feedback on the usefulness of the content of community reviews.
3. The researchers examined how many community reviewers’ comments the manuscripts’ authors accepted and integrated in papers that were published.

Compared to scientific reviewers, community reviewers were more likely to address several study attributes, especially study design, participant selection, and importance of research. Scientific reviewers were more likely to address study attributes related to data analysis, interpretation of findings, quality of writing, and provide confidential comments for the editor.

Editors rated community reviews to be nearly as useful as scientific reviews. They rated community and scientific reviews an average of 3.1 and 3.3, respectively, on a 5-point scale where a higher score indicated a more useful review. Editors stated that the most useful comments in the community reviews were the viewpoints. Community members provided viewpoints that the manuscripts’ authors had not previously considered as well specific suggestions for improving the manuscripts. Community viewpoints included comments on the racial and ethnic diversity of research participants described in the manuscripts, how research findings could be used by patients and communities, cultural considerations and social context, and relevance of research topics to patients and communities.

The published papers also showed that authors accepted and integrated many community reviewers’ comments. At total of 186 community reviewer comments were integrated into 64 published papers. Out of these comments, viewpoint themes were present more often in the papers (66) reviewed by community members compared to those by scientific reviewers (54).

The researchers concluded that, with training, supervision, and compensation, community members can review manuscripts submitted to medical journals and provide useful feedback to editors. They recommended that medical journals broaden their understanding of who can be an expert in the scientific review process to include trained community members, because they can provide valuable and complementary feedback to scientific reviews.

Citation

Huml, A. M., Albert, J. M., Beltran, J. M., Berg, K. A., Collins C. C., Hood, E. N., Nelson, L. C., Pezynski, A. T., Stange, K. C., & Seghal, A. R. (2022). Community members as reviewers of medical journal manuscripts: A randomized controlled trial. Journal of General Internal Medicine, 1-9. https://doi.org/10.1007/s11606-022-07802-z

Page updated May 9, 2023