Use of guidelines, checklists, frameworks, and recommendations in behavioral intervention preliminary studies and associations with reporting comprehensiveness: a scoping bibliometric review

Background Guidelines, checklists, frameworks, and recommendations (GCFRs) related to preliminary studies serve as essential resources to assist behavioral intervention researchers in reporting findings from preliminary studies, but their impact on preliminary study reporting comprehensiveness is unknown. The purpose of this study was to conduct a scoping bibliometric review of recently published preliminary behavioral-focused intervention studies to (1) examine the prevalence of GCFR usage and (2) determine the associations between GCFR usage and reporting feasibility-related characteristics. Methods A systematic search was conducted for preliminary studies of behavioral-focused interventions published between 2018 and 2020. Studies were limited to the top 25 journals publishing behavioral-focused interventions, text mined to identify usage of GCFRs, and categorized as either not citing GCFRs or citing ≥ 2 GCFRs (Citers). A random sample of non-Citers was text mined to identify studies which cited other preliminary studies that cited GCFRs (Indirect Citers) and those that did not (Never Citers). The presence/absence of feasibility-related characteristics was compared between Citers, Indirect Citers, and Never Citers via univariate logistic regression. Results Studies (n = 4143) were identified, and 1316 were text mined to identify GCFR usage (n = 167 Citers). A random sample of 200 studies not citing a GCFR were selected and categorized into Indirect Citers (n = 71) and Never Citers (n = 129). Compared to Never Citers, Citers had higher odds of reporting retention, acceptability, adverse events, compliance, cost, data collection feasibility, and treatment fidelity (ORrange = 2.62–14.15, p < 0.005). Citers also had higher odds of mentioning feasibility in purpose statements, providing progression criteria, framing feasibility as the primary outcome, and mentioning feasibility in conclusions (ORrange = 6.31–17.04, p < 0.005) and lower odds of mentioning efficacy in purpose statements, testing for efficacy, mentioning efficacy in conclusions, and suggesting future testing (ORrange = 0.13–0.54, p < 0.05). Indirect Citers had higher odds of reporting acceptability and treatment fidelity (ORrange = 2.12–2.39, p < 0.05) but lower odds of testing for efficacy (OR = 0.36, p < 0.05) compared to Never Citers. Conclusion The citation of GCFRs is associated with greater reporting of feasibility-related characteristics in preliminary studies of behavioral-focused interventions. Researchers are encouraged to use and cite literature that provides guidance on design, implementation, analysis, and reporting to improve the comprehensiveness of reporting for preliminary studies. Supplementary Information The online version contains supplementary material available at 10.1186/s40814-023-01389-w.


Background
Early-stage, preliminary studies (i.e., pilot/feasibility) are the foundation of larger-scale behavioral interventions.Preliminary studies provide critical evidence regarding trial feasibility -recruitment and retention of participants and measurement of outcomes; intervention feasibility -participant enjoyment/acceptability, attendance/ dosage, and missing or needed intervention components; and preliminary efficacy -whether changes are observed in primary or secondary outcomes to determine if an investment in a larger-scale, well-powered intervention is warranted [1,2].Particularly for preliminary studies, the capturing and comprehensive reporting of both trial and intervention feasibility are critical procedures needed to understand if the research can be appropriately conducted and whether the scale-up to a larger, more well-powered trial is worth pursuing.Furthermore, the thorough and transparent reporting of this information in published manuscripts may aid other researchers planning to conduct similar studies in learning how to design and implement successful preliminary studies of behavioral interventions.
Published guidelines, checklists, frameworks, and recommendations (GCFRs) are an essential resource for researchers conducting behavioral interventions to aid in the collection and reporting of intervention studies.For example, the Consolidated Standards of Reporting Trials (CONSORT) statement [3], the Template for Intervention Description and Replication (TIDieR) checklist [4,5], and the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) statement [6,7], each provide important guidance for key elements needing to appear in published clinical trials.Evidence suggests the adoption of these guidelines has resulted in an overall improvement in reporting of clinical trials [5,7], but additional work should be undertaken to understand how preliminary study-specific guidelines impact the reporting of the smaller, foundational studies, which are conducted prior to the larger-scale clinical trial.
For preliminary studies, there are multiple resources describing what should be collected and reported.Collectively, these resources describe high-quality preliminary studies as those which primarily focus on feasibility and thoroughly report all aspects of feasibility-related characteristics, study design, and implementation.These include the CONSORT extension to randomized pilot and feasibility trials [8] and recommendations from Pearson et al. [9] and Bowen et al. [10] for feasibility testing.Whether the use of such GCFRs leads to the comprehensive reporting of preliminary studies, however, is unknown.Evidence from a historical scoping review of 600 obesity-related preliminary studies published between 1982 and 2020 [11] suggests citing common GCFRs (e.g., CONSORT extension to pilot and feasibility studies) is positively associated with a more thorough reporting of feasibility indicators and higher preliminary study reporting quality.Guidance on preliminary studies, however, has only recently been published (within the past 20 years), and the usage and impact of these publications on recently published preliminary study reporting and comprehensiveness are not well understood.Additionally, there have not been any investigations into how GCFR utilization impacts the comprehensiveness of preliminary study reporting from a broader range of behavioral-focused intervention topics, not just those targeting obesity.The purpose of this study is to conduct a scoping bibliometric review of recently published (2018-2021) behavioral-focused preliminary intervention studies to identify the application of GCFRs and their association with the reporting of feasibility indicators and other feasibility-related study characteristics.

Methods
This scoping bibliometric review was conducted and reported according to the Preferred Reporting Items of Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines [12].

Identification of guidelines, checklists, frameworks, and recommendations
An iterative and systematic search was conducted to identify preliminary study-related guidelines, checklists, frameworks, and recommendations.Known GCFRs were used as starting points.These included the CONSORT Extension for Pilot and Feasibility Studies [8], Medical Research Council guidance [13], and publications such as Bowen et al. [10], Pearson et al. [9], and Eldridge et al. [14].Backward citation tracking was used to identify additional GCFRs, which involved searching reference lists of identified GCFRs to locate additional relevant publications.This iterative approach was conducted until saturation was reached, and no new GCFRs could be identified.This approach has been used in previous literature reviews to identify and consolidate theoretical approaches in implementation research [15] and guidance on complex population health interventions [16].Additionally, two separate samples of over 600 pilot studies from other reviews [11,17] were used to identify GCFRs via backward citation tracking.Finally, the EQUATOR Network [18] was searched for any additional GCFRs related to preliminary studies.In total, we identified 152 GCFRs which we classified into 10 domains related to preliminary studies: adaptations, definitions of pilot and feasibility studies, design and interpretation, feasibility, implementation, intervention development, progression criteria, reporting, sample size, and scaleup.The Supplementary file contains a full list of these publications.

Preliminary study search strategy
A systematic literature search was conducted in Pub-Med/MEDLINE, Embase/Elsevier, EBSCOhost, and Web of Science online databases for preliminary behavioral intervention studies published from January 1, 2018, to December 31, 2021.This time frame was chosen to highlight the most recently published behavioral intervention preliminary studies and to adequately capture usage of the most recently published GCFRs.The initial search strategy consisted only of the keywords as follows: "pilot, " "feasibility, " "preliminary, " "proof of concept, " and "vanguard" paired with the keyword "intervention" as found in the title and/or abstract.The following additional filters were applied to each search: English language, human species, articles only, peer-reviewed journals.A full search strategy is provided in the Supplementary File.

Eligibility criteria and sampling strategy
Before the initial screening, the full list of articles was purposively sampled only from journals that served as likely outlets for published behavioral interventions.This was done to systematically reduce the large number of articles needed to be screened and focus on studies that were likely behavioral interventions.The full list of included journals can be found in the Supplementary file.
All preliminary studies published between 2018 and 2021 that delivered a behavioral intervention were considered for inclusion in this review.Similar to previous reviews [11,17], behavioral interventions were defined as "interventions that target actions which lead to improvements in health indicators, separate from mechanistic, laboratory, pharmacological, feeding/dietary supplementation, and medical device or surgical procedure studies, " and preliminary studies were defined as "those studies which are conducted separately from and prior to a large-scale trial and are designed to test the feasibility of an intervention and/or provide evidence of preliminary effects before scale-up." Articles were excluded if they employed a nonbehavioral intervention, were not a preliminary study, were only observational/cross-sectional in nature, or were presented only qualitative data.

Screening process
Database search results were electronically downloaded as a RIS file and converted to an XML file, using EndNote X9 Reference Manager (Philadelphia, PA, USA), and uploaded to Microsoft Excel for review.The total number of articles downloaded from database search results was cross-checked with the total number of articles in the RIS and XML file to ensure all articles were converted to each file format.The XML file contained study information such as year, author, title, abstract, and the journal in which each article was published.Using the journal information, an article count for each journal was created, and the list of journals was sorted by article count.Journals that were not likely to have published a behavioral intervention were excluded from the list and their associated articles not considered for title and abstract screening.Articles published in the 25 journals with the largest article count were considered for screening.It is worth noting that all 25 journals endorsed the use of reporting guidelines when appropriate for the study type.Title and abstract screening were completed by three reviewers (C.D. P., L. V., A. B.) to identify remaining references that met the eligibility criteria.Disagreements were resolved by having a third member of the research team (M.B.) review the reference and make a final decision.Full-text PDFs were retrieved for references that passed the initial title and abstract screening process and were reviewed in duplicate by three members (C.D. P., L. V., A. B.) of the research team.Disagreements were resolved by having a third member of the research team (MWB) independently review the full-text manuscript and make a final decision.

Study coding and categorization
The full list of studies that remained after the screening process was uploaded to NVivo 12 Plus Qualitative Data Analysis software (QSR International, 2021) as PDF files.To determine GCFR usage within each preliminary study, reference lists within each PDF were text mined in duplicate by two members of the research team (C.D. P. and L. V.) using the title of each GCFR identified from the search previously described.The presence or absence of a GCFR was documented for each study and each GCFR.An Excel matrix indicating the presence (coded as 1) or absence (coded as 0) of each GCFR for each preliminary study was exported from NVivo, and a sum total of cited GCFRs was created in a new column of the spreadsheet.Studies were initially categorized based on this sum total as "citers" (citing two or more GCFRs) and "Non-Citers" (citing no GCFRs).A copy of each file from both reviewers (C.D. P. and L. V.) was merged and cross-checked to ensure there were no discrepancies in data extraction and coding.
Because the non-Citers category had a large number of articles (n = 826), a random sample of 200 (~25%) were chosen for final inclusion in the review.All articles coded as non-Citers were assigned a unique, random identification number with STATA's "rannum" command, sorted numerically, and the first 200 studies were included.After randomization, the non-Citers were further categorized as Never Citers, or Indirect Citers, using backward citation tracking and are defined below.Reference lists of the non-Citers were searched for (1) a protocol or preliminary study of the same intervention conducted by the same author, (2) a protocol or preliminary study of a different intervention conducted by the same author, or (3) a protocol or preliminary study conducted by different authors.If any of these were found in the non-Citers' reference lists, the reference list of the cited study was searched for the presence/absence of GCFRs in the same way as the original sample.If a GCFR was present in the cited study's reference list, the original Non-Citer was categorized as an Indirect Citer.If GCFRs were not present in any of the cited studies' reference lists, the original Non-Citer was categorized as a Never Citer.

Descriptive characteristics
Study-and participant-level characteristics were extracted by four members of the research team (C.D. P., L. V., A. B., and J. P.) and coded in an Excel spreadsheet.These included publication year, location of study (country), baseline sample size, participant age, intervention treatment length, study design, trial registration information, and type of funding.An initial training set of 25 articles were coded in duplicate and cross-checked for consistency before the remaining articles were divided amongst the extraction team and extracted individually.

Reporting of feasibility indicators
The presence/absence of nine feasibility indicators was cataloged for each included preliminary study.Feasibility indicators included recruitment, retention, participant acceptability (satisfaction), adverse events, attendance, compliance, cost, data collection feasibility, and treatment fidelity.The list of relevant feasibility indicators was established in a previous review [11] and was based on definitions from the NIH and other peer-reviewed sources [19].The identification of feasibility indicators was done by utilizing a combination of text mining and manual search procedures.Text mining procedures were conducted in NVivo 12 Plus Qualitative Data Analysis software (QSR International, 2021) by two members of the research team (C.D. P. and L. V.) and consisted of full-text searches with keywords related to feasibility outcomes.Full-text PDFs were also manually searched by CDP and LV to ensure text mining procedures identified all possible reported feasibility indicators.The presence or absence of each feasibility indicator was coded as 1 = present or 0 = absent for each included preliminary study.Similar to procedures used to extract descriptive characteristics, an initial training set of 25 articles were coded in duplicate by C. D. P. and L. V. and crosschecked for consistency before the remaining articles were divided amongst the extraction team and extracted/ coded individually.Additional spot-checking was independently performed by a third member of the research team (M.B.), prior to analyses, to ensure data was accurately extracted.Examples of keywords used in the text mining procedures and the operational definitions of each feasibility indicator are provided in the Supplementary file.

Feasibility-related characteristics
The presence or absence of several feasibility-related characteristics was manually extracted.These included whether (1) the title included feasibility-related language (had "pilot, " "feasibility, " or similar language in the title), (2) the purpose statement mentioned feasibility and/or efficacy, (3) progression criteria were provided, (4) feasibility was framed as the primary outcome, (5) the conclusion mentioned feasibility and/or efficacy, (6) caution was advised regarding interpretations of preliminary efficacy, and (7) future testing was suggested.The presence or absence of each of these characteristics was similarly coded as 1 = present or 0 = absent.These feasibilityrelated characteristics, in addition to the reporting of feasibility indicators, were of interest because they are also important considerations unique to preliminary studies according to much of the guiding literature (8)(9)(10).

Statistical analysis
Descriptive statistics were compared between Citers and the full sample (the total number of articles which constituted the larger sampling pool from which Never Citers, Indirect Citers, and Citers came from) as well as between Never Citers, Indirect Citers, and Citers using Kruskal-Wallis tests and chi-square tests when appropriate.Tests conducted between Citers and the full sample were conducted to ensure the quasi-random sampling procedure used to select studies did not produce any systematic differences between groups.Univariate logistic regression models were employed to assess differences in feasibility-related characteristics and in the reporting of feasibility indicators between Never Citers, Indirect Citers, and Citers.The presence or absence of feasibility-related characteristics and feasibility indicators was treated as the binary dependent variable, while two dummy variables identifying Indirect Citers and Citers (with Never Citers as the reference category) were independent variables.Because we did not consider the original CONSORT [3] a true preliminary study-related GCFR, we accounted for its presence by including its use as a covariate in all univariate models (coded as 0 = not cited, 1 = cited).Additionally, for four of the feasibilityrelated characteristics (mentioning feasibility in the conclusion, mentioning efficacy in the conclusion, advising caution regarding efficacy, suggesting future testing), we did not include protocols in the sample used for the logistic regression models, as these aspects of feasibility would not have been reported in protocol studies.All analyses had an alpha level of p < 0.05 and were carried out using STATA v17.0 statistical software package (College Station, TX, USA).

Search results
Results of the screening and randomization procedures are communicated in the study flow diagram (Fig. 1).A total of 33,840 records were identified from the database search.Limiting records to only the 25 health behavior journals with the largest article counts reduced the number of records to 4143.Abstract screening further reduced this sample to 1657.After assessing articles for eligibility, a total of 341 were excluded, and 1316 articles were included in the full sample The full sample of 1316 articles was then text mined to identify the usage of GCFRs within their references.A total of 826 articles cited zero GCFRs, and 167 cited two or more.A random sample of 200 articles from the 826 which cited zero GCFRs were further categorized into Never Citers (n = 129) and Indirect Citers (n = 71) using the methodology previously described.Thus, a total of 367 articles were included in this bibliometric review.

Discussion
This was a scoping bibliometric review of 367 behavioral-focused preliminary intervention studies published between 2018 and 2020.We examined the usage of guidelines, checklists, frameworks, and recommendations (GCFRs) related to preliminary studies and determined associations between GCFR usage and the reporting of feasibility indicators and feasibility-related characteristics.Citing two or more GCFRs was associated with reporting a greater number of feasibility indicators and framing the study findings from a feasibility perspective.These data demonstrate the use of GCFR, as inferred from citations within reference lists, and have a clear positive impact on the overall comprehensiveness of the information presented.This information, in turn, should lead to greater transparency in reporting and more informed decisions regarding the viability of a behavioral intervention in a larger-scale trial.
To the authors' knowledge, this is the first review to document and analyze the usage of GCFRs in preliminary health behavior preliminary intervention studies.Literature on reporting, design and interpretation, and defining pilot and feasibility studies made up the majority of citations, and the three most cited GCFRs were the Medical Research Council (MRC) Guidance for Developing and Evaluating Complex Interventions [13,20], the CONSORT Extension for Pilot and Feasibility Studies [8], and the Template for Intervention Description and Replication (TIDieR) [21].According to the Web of Science (accessed April 14, 2022), the MRC guidance has been cited over 4000 times, the CON-SORT extension over 500 times, and TIDieR over 3000 times, although preliminary studies do not account for all of these citations.It is not surprising though that these are the top cited GCFRs in our sample of preliminary studies as each of them are widely supported in the behavioral sciences, funding agencies.These GCFRs can also be found on the EQUATOR network [18], which makes them easily accessible to a larger audience of researchers developing intervention studies.Less common were GCFRs related to adaptations and scale-up, with the least commonly cited GCFR category being progression criteria.This might explain the very low presence of progression criteria in our included sample of preliminary studies, which has been found in other reviews as well [22,35].The low usage of GCFRs related to scale-up is interesting, especially since the majority of preliminary studies are presumably designed for future scale-up.Several frameworks, including the National Institutes of Health Stage Model [36] and the Obesity-Related Behavioral Intervention Trials (ORBIT) model [37], outline the sequential and iterative processes for developing large-scale interventions, and specific attention is devoted to preliminary studies as a foundational piece of this scale-up process.
Many of the GCFRs highlighted in this review can provide useful guidance on the scale-up process.
The reporting of feasibility indicators and feasibilityrelated characteristics in the sample of preliminary studies included in this review is similar to that of other reviews of health behavior intervention studies that are not preliminary studies [11,35,38].Most studies did not report feasibility indicators such as treatment fidelity, data collection feasibility, cost, participant compliance and attendance, and adverse events.Overall, most studies reported two or fewer feasibility indicators, which were typically recruitment and retention of participants.While trial-related feasibility (i.e., recruitment and retention) is important to assess, intervention-related feasibility indicators (treatment fidelity, the ability to collect data on participants, etc.) are equally important to measure and report to highlight the potential viability of an intervention and/or what aspects of the intervention need to be addressed and altered to ensure the intervention is successfully scaled up in the future.While most studies did mention feasibility (or use similar language) in the title and purpose statement, over half mentioned efficacy in the purpose statement and conducted statistical testing for efficacy.It is wellestablished researchers should exercise caution when conducting efficacy testing within preliminary intervention studies and should not depend on these estimates to inform the design of a larger-scale trial, which would likely be conducted with hypothesis testing in mind [39][40][41].In fact, many of the GCFRs identified for this review caution against the use and interpretation of preliminary study effect sizes.This could explain why the use of GCFRs was associated with lower odds of many efficacy-related study characteristics in our sample.There are also inherent issues (both type I and type II error) with using preliminary study effect sizes for sample size estimation for a larger-scale trial, although we did not identify whether this was being done in our sample of preliminary studies.It is worth noting that just over half of the included preliminary studies did advise caution regarding interpretations of efficacy, and most studies framed feasibility indicators as the primary outcomes of interest.This is a promising finding and shows that many authors of recently published preliminary studies are [1] acknowledging the pitfalls of using preliminary studies to demonstrate the efficacy of an intervention and [2] prioritizing feasibility indicators as the main outcome of interest.
While this study is not the first to explore reporting quality in preliminary studies [11,35,38], it is the first to investigate associations between the usage of GCFRs and preliminary study reporting.Some studies have broadly explored the impact of guidelines (mostly CONSORT and TIDieR) on study reporting quality, not necessarily in preliminary studies exclusively, and the results of these studies are heterogeneous [42][43][44].There are also several other examples of these types of reviews for a variety of disciplines and study designs [45][46][47][48][49], but most use somewhat ambiguous proxies for the usage of guidelines.In other words, most reviews have assessed reporting quality before and after the publication of a guideline, with the assumption that the presence of the published guideline might influence reporting quality.Our study utilized text mining to identify preliminary studies that cited GCFRs, using the presence of a GCFR citation as the indicator for usage and found that usage significantly and positively associated with the reporting of most feasibility indicators and feasibility-related characteristics.These findings provide compelling support for the use of GCFRs as a tool to improve the reporting quality of preliminary intervention studies.
Not only does the usage of GCFRs (via citation) associate with better reporting, our results in the Indirect Citer group, show citing a different preliminary study that cited a GCFR associated with increased odds of reporting acceptability and treatment fidelity as well as decreased odds of conducting statistical testing for efficacy.These results demonstrate a possible diffusive nature of the information published in GCFRs, whereby authors use other published preliminary studies (which cited a GCFR) as a model for their work.Thus, the true beneficial impact of GCFRs related to preliminary studies may go beyond just improving the reporting quality of studies which cite them.

Strengths and limitations
This review included a large sample (N = 367) of preliminary intervention studies published between 2018 and 2020, capturing some of the most recently published health behavior preliminary studies in the literature.Preliminary studies were not excluded based on location, design, or health behavior topic, which means results are generalizable to a large audience of health behavior researchers who conduct preliminary intervention studies.However, several limitations need to be considered.Included studies came from a sample 25 journals publishing the largest number of preliminary studies.Sampling from a wider variety of journals may have produced different results.Because the number of studies not citing any GCFR (Non-Citers) was so large, we opted to randomly sample 200 of them.Although random sampling is supposed to be an unbiased approach to sampling and no differences between the subsample and larger sample were found in our study, there is a possibility of the presence of sample selection bias.Limitations regarding the coding of studies are also important to mention.First, the presence/absence of GCFR usage and the reporting of feasibility indicators and feasibility-related characteristics were identified via a combination of text mining and manual approaches.It is possible that some items may have either not been identified or improperly coded due to human error, although several quality control checks were put in place to avoid these issues.While we used an extensive list of GCFRs, it is possible that some GCFRs were missed and therefore not searched within our sample, which could influence results by miscoding studies that did cite a GCFR which we did not include in our search.Finally, we recognize that what is reported in a study is not completely identical to the actual conduct of the study.The failure to cite a GCFR or report a feasibility indicator does not mean authors of a study had not considered preliminary study guidance or measured feasibility indicators during their preliminary intervention study.

Conclusions
Preliminary studies provide an ideal opportunity to improve multiple facets of an intervention study.Each of these improvements has the potential to enhance the overall rigor and reproducibility of the intervention when delivered at scale.Tools to aid researchers in the development and implementation of preliminary studies exist, namely in the form of published guidelines, checklists, frameworks, and recommendations, but our review indicates that many are never cited.Results from this review provide evidence that the use of GCFRs (via citation) is associated with more thorough reporting of feasibilityrelated characteristics in behavioral-focused preliminary studies.Researchers should utilize available literature that provides guidance on various aspects of preliminary study design, implementation, analysis, and reporting to improve the comprehensiveness and reporting of their preliminary studies, inform efficient scale-up to largerscale trials, and avoid unnecessary research waste.

Table 4
also summarizes the presence of feasibilityrelated characteristics for all studies included in the

Table 1
Characteristics of included preliminary studies (N = 367)

Table 2
List of GCFR topics with citation counts for the full sample and those in the Citers category

Table 3
List of the top 10 most cited GCFRs

Table 4
Feasibility-related characteristics of included preliminary studies (N = 367) a Protocols were not included in this analysis.See Table1for total number of protocols in each citation category

Table 5
Summary of logistic regression analysis for reporting feasibility indicators and other related characteristics in studies considered Never Citers, Indirect Citers, and Citers (N = 367)