This article has Open Peer Review reports available.
Risk of venous thromboembolism after total hip and knee replacement in older adults with comorbidity and co-occurring comorbidities in the Nationwide Inpatient Sample (2003-2006)
- Alok Kapoor1Email author,
- Alan J Labonte†2,
- Michael R Winter†2,
- Jodi B Segal†3,
- Rebecca A Silliman†4, 5,
- Jeffrey N Katz†6, 7,
- Elena Losina†7, 8, 9 and
- Dan Berlowitz†10, 11
© Kapoor et al; licensee BioMed Central Ltd. 2010
Received: 10 April 2010
Accepted: 17 September 2010
Published: 17 September 2010
Venous thromboembolism is a common, fatal, and costly injury which complicates major surgery in older adults. The American College of Chest Physicians recommends high potency prophylaxis regimens for individuals undergoing total hip or knee replacement (THR or TKR), but surgeons are reluctant to prescribe them due to fear of excess bleeding. Identifying a high risk cohort such as older adults with comorbidities and co-occurring comorbidities who might benefit most from high potency prophylaxis would improve how we currently perform preoperative assessment.
Using the Nationwide Inpatient Sample, we identified older adults who underwent THR or TKR in the U.S. between 2003 and 2006. Our outcome was VTE, including any pulmonary embolus or deep venous thrombosis. We performed multivariate logistic regression analyses to assess the effects of comorbidities on VTE occurrence. Comorbidities under consideration included coronary artery disease, congestive heart failure (CHF), chronic obstructive pulmonary disease (COPD), diabetes, and cerebrovascular disease. We also examined the impact of co-occurring comorbidities on VTE rates.
CHF increased odds of VTE in both the THR cohort (OR = 3.08 95% CI 2.05-4.65) and TKR cohort (OR = 2.47 95% CI 1.95-3.14). COPD led to a 50% increase in odds in the TKR cohort (OR = 1.49 95% CI 1.31-1.70). The data did not support synergistic effect of co-occurring comorbidities with respect to VTE occurrence.
Older adults with CHF undergoing THR or TKR and with COPD undergoing TKR are at increased risk of VTE. If confirmed in other datasets, these older adults may benefit from higher potency prophylaxis.
Medical injury, or harm associated with a therapeutic or diagnostic intervention , commonly complicates major surgery . Age alone confers a small effect on postoperative injury but medical comorbidities and co-occurring comorbidities contribute substantially. Co-occurring comorbidities, that is, 2 comorbidities which frequently occur together, are part of a larger phenomenon of multiple morbidity .
Venous thromboembolism (VTE) is a common, often fatal, and costly injury which complicates major surgery in older adults. VTE is particularly common following total hip and knee replacement (THR and TKR), with venographic rates of up to 60% without prophylaxis . Newer, more potent prophylaxis regimens including the synthetic pentasacchride fondaparinux and twice daily dosing of enoxaparin, a low molecular weight heparin, offer the ability to significantly reduce the risk of VTE. Their wide spread adoption has been slow, however, given the increased risk of bleeding . The ability to identify a high risk cohort among older adults undergoing THR and TKR, who would potentially benefit from high potency prophylaxis, would be an improvement in the way clinicians currently conduct preoperative assessments.
Comorbidities such as congestive heart failure (CHF) and chronic obstructive pulmonary disease (COPD) have been associated with increased VTE risk in some studies [6–8] but not in others [9, 10]. The association with CHF may relate to blood flow stasis which is part of Virchow's triad  of alteration in blood flow (stasis), endothelial injury, and alterations in blood constituents (hypercoagulable factors). Similarly the association of COPD may be explained through an immobility or stasis mechanism. Newer evidence, in both surgical [6, 7, 9, 10] and non-surgical settings , suggests that atherosclerotic conditions such as coronary artery disease (CAD) and cerebrovascular disease (CVD) are also associated with an increased risk of VTE. In general, these studies had a small number of observations and/or did not focus on discrete surgical procedures, making it difficult to draw conclusions about modest but clinically important effects on VTE of comorbidities for individual surgeries. The combined effect of comorbidities, i.e., co-occurring comorbidities, has been incompletely evaluated.
In this investigation, we assessed the Nationwide Inpatient Sample to determine the risk of VTE in older adults with prevalent comorbidities and co-occurring comorbidities undergoing THR or TKR. We hypothesized that CHF, CAD, CVD, and COPD would predict increased risk of VTE. We also proposed that the joint effect of co-occurring comorbidities -- composed of comorbidities with distinct pathogenesis, such as CAD and COPD -- would equal the sum of the component effects or possibly exceed their sum, indicating a positive interaction.
Data Sources and Study Sample
We used the Nationwide Inpatient Sample (NIS) from 2003-2006 for our analysis . This work involves the use of publicly available, archival information abstracted from patient medical records and was approved by the Institutional Review Board at the Boston University Medical Center in Boston, Massachusetts. The NIS contains information from hospital inpatient stays collected at the state-level and then assembled by the Agency for Healthcare Research and Quality (AHRQ) as part of the Healthcare Utilization Project (HCUP). NIS includes primary and secondary diagnoses, procedures, admission and discharge status, patient demographics (gender, age, race, median income, and residence zip code), expected payment source, total charges, length of stay, and hospital characteristics (ownership, size, and teaching status). We identified a cohort of patients (age 65 or older) who underwent primary THR or TKR identified through ICD-9-CM procedure codes 81.51 and 81.54. We restricted our analysis to elective, primary THR and TKR because we felt that non-elective surgery and revision joint surgery represented a distinct population with different risks for VTE. We therefore excluded surgical cases conducted in the setting of fracture, pelvic or thigh infection, or removal of prior prosthesis, internal fixation device, implant or graft consistent with earlier work [14, 15] conducted by members of our team.
Outcomes and Exposures
We determined VTE to have occurred if any one of eight ICD-9-CM DVT codes, three PE codes, and two non-specific VTE codes were present at discharge (Additional file 1).
We included specific comorbidities based on evidence of associations with VTE documented in the literature or a known biological link to the outcomes These included CAD, CHF, COPD, CVD, and diabetes. Comorbidities and presence of obesity were coded as by Elixhauser et al. (Additional file 1). Multiple publications [17–19] have supported the use of the Elixhauser comorbidity coding algorithms rather than the earlier Charlson comorbidity index, including those looking at discrete surgical procedures [20, 21]. The accuracy of coding of the comorbidities has been validated in multiple publications [22, 23]. We also examined co-occurring comorbidities from the above mentioned comorbidities if they had a prevalence of greater than 2% (given that exposures occurring less frequently than 2% of the time would be infeasible to study because of sample size constraints).
Consistent with previous work [14, 15, 24], we evaluated the effects of potentially confounding factors, i.e., race, insurance status, hospital surgical volume, obesity, bi-laterality (two primary procedures during the same surgery), chronic kidney disease, and hypercoagulable state (including cancer and genetic predisposition), by adding them, one at a time, to the crude model (with comorbidity exposure, age group, and sex). If the addition of a given factor did not change the odds ratios for the comorbidity exposure by more than 10%, it was deleted from the final model.
We calculated descriptive statistics and cross-tabular frequencies for each outcome and comorbidity to determine the shape of distributions, the extent of missing data, and the presence of small frequencies of the discrete comorbidity levels. We examined comorbidities for statistical interactions with each other. Specifically, we measured whether prevalent combinations of comorbidities (as defined above) produced statistically significant associations with VTE in multivariate models. Because we found these to be present in the TKR cohort, for the ease of interpretability, we report the associations between co-occurring comorbidities and outcomes using a categorical comorbidity variable for both the THR and TKR cohorts. We created a ten level categorical variable with a separate, mutually exclusive, level for each of nine comorbidities or co-occurring comorbidities (with 1 additional for all other combinations of two or more comorbidities). While this limited the population size for each comorbidity group, it allowed us to compare the risk of VTE for groups of older adults with single or co-occurring comorbidities against a common reference group of older adults without any of the nine comorbidities.
Finally, we built multivariate logistic regression models to assess the independent effects of comorbidities. In our analysis we retained age and sex in all models. For remaining potential confounders, we retained any variable that changed the effect estimate for any comorbidity by more than 10%.
Frequency counts (and row percentages) for selected variables and stratified by surgical procedure.
Total Hip Replacement
Total Knee Replacement
Cerebrovascular Disease (CVD) alone
Coronary Artery Disease (CAD) alone
Congestive Heart Failure (CHF) alone
Chronic Obstructive Pulmonary Disease (COPD) alone
CAD and CHF alone
CAD and COPD alone
CAD and Diabetes alone
COPD and Diabetes alone
All other 2+ combinations
None of the above comorbidities
Median Length of Stay (days)
Frequency counts for selected variables stratified by age and surgical procedure.
65-69 years of age
70-74 years of age
75-79 years of age
≥80 years of age
65-69 years of age
70-74 years of age
75-79 years of age
≥80 years of age
CAD and CHF alone
CAD and COPD alone
CAD and Diabetes alone
COPD and Diabetes alone
All other 2+ combinations
None of the above comorbidities
Odds ratios (and 95% CIs) for exposures, adjusted† for covariates and stratified by surgical procedure.
CAD and CHF vs.
CAD & COPD vs.
Diabetes & CAD vs.
Diabetes & COPD vs.
All other combinations
Age 70-74 vs. Age 65-69
Age 75-79 vs. Age 65-69
Age ≥80 vs. Age 65-69
We did not find any positive interactions between comorbidities in our analysis. The combination of CAD and CHF alone was associated with a twofold increase in odds in both the THR and TKR cohort, an effect which was smaller than for CHF alone. The combination of diabetes and CAD was associated with a 39% decrease in the odds of VTE only in the TKR cohort, an effect which was smaller, i.e., more protective, than either comorbidity alone. (Table 3) The six potential confounding factors that we tested were deleted from the final model for lack of confounding.
We comprehensively examined the association of comorbidities and co-occurring comorbidities and VTE in a unique population of older adults undergoing primary total hip and knee replacement, high risk surgeries for VTE. We found that the rates of VTE captured in administrative data of older adults for the period immediately following THR and TKR to be low at 0.8-1.2%. Having CHF substantially increased the odds of VTE after THR or TKR and having COPD somewhat increased the odds of VTE after TKR. Co-occurring comorbidities did not increase the risk of VTE beyond their individual effects.
Comparison of our results with previous studies is limited by differences in outcomes measured and populations studied. Gangireddy et al.  conducted one of the largest studies to date using data from the Veterans Affairs National Surgical Quality Improvement Program (NSQIP), which included veterans undergoing nine different surgeries, including THR, between 1996 and 2001. After controlling for multiple preoperative and postoperative clinical variables, a multivariate analysis with 76,771 individuals showed that CHF and COPD were not associated with increased rates of VTE. This study supported our findings of the association between diabetes and a slightly lower rate of VTE (OR = 0.75).
We also detected a 29% reduction in the risk of VTE in patients undergoing THR with CVD. This result was not statistically significant nor did we detect an association in the knee population. There has been little evidence, however, regarding the relationship between CVD and postoperative VTE. Prior work  has suggested a common inflammatory pathway but this has not been evaluated extensively in the postoperative setting. In our study the association was not present for both knee and hip cohorts and was not statistically significant. We plan to re-assess this relationship in our future work.
Kikura et al.  examined 21,903 Japanese patients of multiple ages and multiple surgery types and found that history of acute myocardial infarction (AMI) was significantly related (OR = 7.7 95% CI 1.7-34.7) to the development of postoperative thrombotic events (including repeat AMI). Although we did collect information about the history of AMI in particular, we did not find an association with CAD in general in our analysis. In a cohort of 269 post-menopausal women undergoing THR and TKR, Jaffer et al.  found a trend towards CHF predicting more postoperative VTE events (OR = 5.50 95% CI 0.94-43.3) but a trend towards COPD predicting fewer VTE events (OR = 0.42 95% CI 0.07-1.98).
The association of CHF and VTE may relate to blood flow stasis as discussed earlier. Alternatively, CHF may indicate a degree of immobility that was not measured in the data we analyzed. Other comorbidities may also contribute to the development of postoperative VTE but their effects may have been attenuated by a selection bias. Surgeons may select only the healthiest subset of older adults with comorbidities for surgery. The absence of positive interactions between frequently co-occurring comorbidities (especially CAD and CHF) also suggests a potential source of a selection bias. Older adults with co-occurring comorbidities deemed to be suitable surgical candidates are presumably healthier in other ways than other older adults with the same comorbidities.
In the case of COPD, we only detected an increase in risk for older adults undergoing knee surgery. This could be explained by the generally weak predictor effect of COPD on VTE or it could be related to the inherent differences between hip and knee surgery. Postoperative mobility may be significantly less for hip surgery and the effect of immobility in this group may dwarf other predictors such as COPD. Future work should examine the interaction between mobility and surgery type in data where this information is available.
The association of CHF and COPD with postoperative VTE has important implications. Although the American College of Chest Physicians currently recommends high potency prophylaxis such as fondaparinux or LMWH for all individuals undergoing THR and TKR , surgeons are reluctant to prescribe these regimens, fearing bleeding complications. Identification of a high risk subset among a group of older adults already at increased risk for VTE based on the surgery planned would be an important improvement in the way we currently perform preoperative assessment. In addition to the use of high potency prophylaxis, surgeons may also use the risk information to incorporate other practices, such as regional anesthesia, mechanical prophylaxis devices, or stockings, aimed at lowering VTE rates.
There are limitations to the work we presented. Due to the nature of the NIS administrative data we have limited ability to capture VTE. A recent study , suggests that administrative data capture only 58% of VTE events. There is no evidence, however, to suggest that the events indentified are differentially being diagnosed in individuals with CHF or other comorbidities. In addition, we did not have access to medication information including prophylaxis agent. A recent study by Cohen et al. in 2008  indicated that in the United States, only 48% of medical patients are receiving the recommended ACCP prophylaxis and only 71% of surgical patients are receiving prophylaxis . If comorbidities prompted physicians to prescribe more potent prophylaxis in older adults with CHF or other comorbidities, however, the effects we observed would represent an underestimate of the true effect. A recent survey  suggests that orthopedic surgeons vary their prescribing patterns less than 10% of the time when evaluating a patient with cardiopulmonary disease. Future work should examine the relationship between comorbidities and VTE while controlling for prophylaxis agents in data where medication information is available.
We did not have information about events which took place after hospitalization. Given that the median time for development of DVT is 17 days for THR and 7 days for TKR  and the median length of stay was 3 or 4 days for each surgery in our analysis, the associations we present may not reflect the experience of older adults who develop injury in the post discharge period. Controlling for length of stay would not disentangle the relation between these comorbidities and VTE and we, therefore, did not control for it in our analysis. Length of stay may very well be a surrogate for immobility and stasis which are on the causal pathway of VTE development. Alternatively, increased length of stay may also be associated with VTE because of added time needed to achieve therapeutic levels of warfarin. Even though NIS data does not allow for measurement of the 30 or 90 day incidence of VTE, we believe that post discharge rates of VTE events will be similarly disparate in individuals with compared to those without comorbidities. In the future we plan to confirm these associations in data where this information is available.
Administrative data are susceptible to upcoding where medical coders assign a diagnosis that may have only been considered but not proven. We did not have information about those VTE events that were present on admission compared to those that occurred during hospitalization. We plan to conduct further validation studies in other databases where pre-existing diagnosis modifiers are available. We cannot firmly establish causality between comorbidities and VTE using the data available to us. Comorbidities may be linked to other processes such as increased operative time or difficulty weaning from a ventilator after surgery. Future research with datasets containing these clinical variables may clarify the exact causal pathway. Lastly, although in our analysis we controlled for the presence of several factors that might increase the risk of VTE, we did not have data on the smoking status of individuals, which might be related to both the exposure and outcome.
CHF is strongly associated with an increased risk of VTE after THR and TKR; COPD is associated with an increased risk of VTE after TKR. The absence of risk associated with other comorbidities and co-occurring comorbidities may be explained by the selection of healthier older adults for these surgeries or by some of the limiting factors described above. If these findings are confirmed in other datasets, higher risk older adults with comorbidities could potentially benefit from more aggressive preventive interventions including high potency pharmacologic prophylaxis.
This work was supported by a generous pilot grant from the Boston University School of Medicine Department of Medicine Pilot Grant Program. This work involves the use of publicly available, archival information abstracted from patient medical records and was approved by the Institutional Review Board at the Boston University Medical Center in Boston, Massachusetts.
- Patient Safety and Quality: Not adjusting for pre-existing health problems may have exaggerated the number of deaths due to medical injury. [http://www.ahrq.gov/research/feb07/0207RA5.htm]
- Zhan C, Miller MR: Excess length of stay, charges, and mortality attributable to medical injuries during hospitalization. JAMA. 2003, 290 (14): 1868-1874. 10.1001/jama.290.14.1868.View ArticlePubMedGoogle Scholar
- Boyd CM, Weiss CO, Halter J, Han KC, Ershler WB, Fried LP: Framework for evaluating disease severity measures in older adults with comorbidity. Journals of Gerontology Series A Biological Sciences & Medical Sciences. 2007, 62 (3): 286-295.View ArticleGoogle Scholar
- Geerts WH, Pineo GF, Heit JA, Bergqvist D, Lassen MR, Colwell CW, Ray JG: Prevention of venous thromboembolism: the Eighth ACCP Conference on Antithrombotic and Thrombolytic Therapy. Chest. 2008, 133 (3 Suppl): 381S-453. 10.1378/chest.08-0656.View ArticlePubMedGoogle Scholar
- American Academy of Orthopaedic Surgeons Clinical Guideline on Prevention of Symptomatic Pulmonary Embolism in Patients Undergoing Total Hip or Knee Arthroplasty. [http://www.aaos.org/Research/guidelines/PE_guideline.pdf]
- Kikura M, Takada T, Sato S: Preexisting morbidity as an independent risk factor for perioperative acute thromboembolism syndrome. Arch Surg. 2005, 140 (12): 1210-1217. 10.1001/archsurg.140.12.1210. discussion 1218View ArticlePubMedGoogle Scholar
- Jaffer AK, Barsoum WK, Krebs V, Hurbanek JG, Morra N, Brotman DJ: Duration of anesthesia and venous thromboembolism after hip and knee arthroplasty. Mayo Clin Proc. 2005, 80 (6): 732-738. 10.4065/80.6.732.View ArticlePubMedGoogle Scholar
- Fimognari FL, Scarlata S, Conte ME, Incalzi RA, Fimognari FL, Scarlata S, Conte ME, Incalzi RA: Mechanisms of atherothrombosis in chronic obstructive pulmonary disease. Int J Chron Obstruct Pulmon Dis. 2008, 3 (1): 89-96.PubMedPubMed CentralGoogle Scholar
- Gangireddy C, Rectenwald JR, Upchurch GR, Wakefield TW, Khuri S, Henderson WG, Henke PK: Risk factors and clinical impact of postoperative symptomatic venous thromboembolism. J Vasc Surg. 2007, 45 (2): 335-341. 10.1016/j.jvs.2006.10.034. discussion 341-332View ArticlePubMedGoogle Scholar
- Schiff RL, Kahn SR, Shrier I, Strulovitch C, Hammouda W, Cohen E, Zukor D: Identifying orthopedic patients at high risk for venous thromboembolism despite thromboprophylaxis. Chest. 2005, 128 (5): 3364-3371. 10.1378/chest.128.5.3364.View ArticlePubMedGoogle Scholar
- Overview of the causes of venous thrombosis. [http://www.uptodate.com]
- Prandoni P, Bilora F, Marchiori A, Bernardi E, Petrobelli F, Lensing AW, Prins MH, Girolami A, Prandoni P, Bilora F, et al: An association between atherosclerosis and venous thrombosis. New England Journal of Medicine. 2003, 348 (15): 1435-1441. 10.1056/NEJMoa022157.View ArticlePubMedGoogle Scholar
- Overview of the Nationwide Inpatient Sample (NIS). [http://www.hcup-us.ahrq.gov/nisoverview.jsp]
- Katz JN, Barrett J, Mahomed NN, Baron JA, Wright RJ, Losina E: Association between hospital and surgeon procedure volume and the outcomes of total knee replacement. Journal of Bone & Joint Surgery - American Volume. 2004, 86-A (9): 1909-1916.Google Scholar
- Katz JN, Losina E, Barrett J, Phillips CB, Mahomed NN, Lew RA, Guadagnoli E, Harris WH, Poss R, Baron JA: Association between hospital and surgeon procedure volume and outcomes of total hip replacement in the United States medicare population. Journal of Bone & Joint Surgery American. 2001, 83 (11): 1622-1629.Google Scholar
- Elixhauser A, Steiner C, Harris DR, Coffey RM: Comorbidity measures for use with administrative data. Med Care. 1998, 36 (1): 8-27. 10.1097/00005650-199801000-00004.View ArticlePubMedGoogle Scholar
- Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi J-C, Saunders LD, Beck CA, Feasby TE, Ghali WA: Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005, 43 (11): 1130-1139. 10.1097/01.mlr.0000182534.19832.83.View ArticlePubMedGoogle Scholar
- Dominick KL, Dudley TK, Coffman CJ, Bosworth HB: Comparison of three comorbidity measures for predicting health service use in patients with osteoarthritis. Arthritis & Rheumatism. 2005, 53 (5): 666-672. 10.1002/art.21440.View ArticleGoogle Scholar
- Southern DA, Quan H, Ghali WA: Comparison of the Elixhauser and Charlson/Deyo methods of comorbidity measurement in administrative data. Med Care. 2004, 42 (4): 355-360. 10.1097/01.mlr.0000118861.56848.ee.View ArticlePubMedGoogle Scholar
- Livingston EH: Development of bariatric surgery-specific risk assessment tool. Surg. 2007, 3 (1): 14-20. discussion 20Google Scholar
- Fry DE, Pine M, Jordan HS, Elixhauser A, Hoaglin DC, Jones B, Warner D, Meimban R: Combining administrative and clinical data to stratify surgical risk. Ann Surg. 2007, 246 (5): 875-885. 10.1097/SLA.0b013e3180cc2e7a.View ArticlePubMedGoogle Scholar
- Birman-Deych E, Waterman AD, Yan Y, Nilasena DS, Radford MJ, Gage BF: Accuracy of ICD-9-CM Codes for Identifying Cardiovascular and Stroke Risk Factors. Med Care. 2005, 43 (5): 480-485. 10.1097/01.mlr.0000160417.39497.a9.View ArticlePubMedGoogle Scholar
- Lee DS, Donovan L, Austin PC, Gong Y, Liu PP, Rouleau JL, Tu JV, Lee DS, Donovan L, Austin PC, et al: Comparison of coding of heart failure and comorbidities in administrative and clinical data for use in outcomes research. Med Care. 2005, 43 (2): 182-188. 10.1097/00005650-200502000-00012.View ArticlePubMedGoogle Scholar
- Romano PS, Mull HJ, Rivard PE, Zhao S, Henderson WG, Loveland S, Tsilimingras D, Christiansen CL, Rosen AK, Romano PS, et al: Validity of selected AHRQ patient safety indicators based on VA National Surgical Quality Improvement Program data. Health Serv Res. 2009, 44 (1): 182-204. 10.1111/j.1475-6773.2008.00905.x.View ArticlePubMedPubMed CentralGoogle Scholar
- Cohen AT, Tapson VF, Bergmann J-F, Goldhaber SZ, Kakkar AK, Deslandes B, Huang W, Zayaruzny M, Emery L, Anderson FA: Venous thromboembolism risk and prophylaxis in the acute hospital care setting (ENDORSE study): a multinational cross-sectional study. Lancet. 2008, 371 (9610): 387-394. 10.1016/S0140-6736(08)60202-0.View ArticlePubMedGoogle Scholar
- Markel DC, York S, Liston MJ, Flynn JC, Barnes CL, Davis CM: Venous Thromboembolism: Management by American Association of Hip and Knee Surgeons. The Journal of Arthroplasty. 2009, 25 (1): 3-9.e2. 10.1016/j.arth.2009.07.021.View ArticlePubMedGoogle Scholar
- White RH, Romano PS, Zhou H, Rodrigo J, Bargar W: Incidence and Time Course of Thromboembolic Outcomes Following Total Hip or Knee Arthroplasty. Arch Intern Med. 1998, 158 (14): 1525-1531. 10.1001/archinte.158.14.1525.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://0-www.biomedcentral.com.brum.beds.ac.uk/1471-2318/10/63/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.