Skip to main content
Free access
Research Article
24 May 2019

Agreement between manual and automatic analyses of home sleep examinations in pediatric obstructive sleep apnea

Abstract

Aim: Home sleep testing is becoming more common in the diagnostics of pediatric obstructive sleep apnea. The aim was to examine the agreement between manual and automatic analyses of home sleep examinations in children and adolescents. Materials & methods: Sleep examinations recorded with a type 3 sleep monitor (Nox T3) were analyzed manually by a registered polysomnographic technologist and automatically with Noxturnal version 5.1. Results: 51sleep examinations on children and adolescents with a median age of 13.6 years were included. The median manual apnea–hypopnea index (AHI) was 2.7 (range 0.2 to 28.2), while the median automatic AHI was 11.9 (range 4.2 to 45.6; p < 0.001). Conclusion: The agreement between manual and automatic analyses was poor. The AHI was consistently overestimated by automatic analysis.
Obstructive sleep apnea (OSA) in childhood is associated with behavioral and neurocognitive dysfunctions [1,2], cardiovascular morbidities [3–5] and reduced quality of life [6,7]. The prevalence of OSA in the general pediatric population is estimated to be 1–5% [8]. Overweight is one of the known risk factors for OSA, and the reported prevalence of OSA in children and adolescents with overweight/obesity ranges from 19 to 61% [9–11].
OSA is diagnosed based on an overnight sleep examination, and in-laboratory polysomnography (PSG) is the gold standard for diagnosing OSA [12]. However, home sleep testing is becoming more common in the diagnostics of OSA. The American Academy of Sleep Medicine has now approved the use of type 3 portable sleep monitors as an alternative to PSG in adults [13]. In children, these monitors are not yet officially approved for diagnosing OSA; however, portable sleep monitoring is the realistic first choice in many sleep clinics because of the inaccessibility and great costs of PSG.
It is recommended that home sleep examinations are manually analyzed by a registered practitioner in sleep medicine [13]. As an alternative to manual analysis, most portable devices have a software program that can analyze data automatically. Especially designed algorithms for the analysis of pediatric sleep examinations have also been developed. However, knowledge about the agreement between pediatric manual analysis and pediatric automatic analysis is lacking. Therefore, the aim of this study was to compare the results from manual analysis of home sleep examinations with automatic analysis in children and adolescents using a type 3 portable sleep monitor (Nox T3).

Materials & methods

The participants

The study population was recruited from an ongoing research project concerning the effect of weight loss in children and adolescents with OSA and overweight/obesity. We included children and adolescents who were referred to a follow-up consultation in the sleep clinic at Zealand University Hospital, Køge, Denmark, in the period April 2016 to December 2016. All participants in this study had been diagnosed with OSA (defined as an AHI ≥ 2) using portable sleep monitoring (Nox T3) 6–8 months prior to inclusion. The inclusion criteria were age <18 years and a night sleep examination containing ≥4 h of sleep time in acceptable signal quality (overall signal quality ≥90%). During the consultations, demographic data and anthropometric measurements were obtained, and the participants and their parents were meticulously instructed in the use of the portable sleep monitor (Nox T3). The BMI standard deviation score was estimated using the LMS method (LMS refers to three smooth age-specific curves called L [lambda], M [mu] and S [sigma] [14]), based on Danish references [15].

Home sleep examinations

Home sleep examinations were performed using the Nox T3 device (Nox Medical, Reykjavík, Iceland), which is a type 3 portable sleep monitor previously validated in adults [16,17]. It measures airflow via a nasal cannula; respiratory effort via thoracic and abdominal belts; body position and activity via an integrated accelerometer; and pulse and oxygen saturation via an oximeter.

Manual & automatic analysis

The sleep examinations were all manually analyzed by the same registered polysomnographic technologist. Apneas and hypopneas were manually scored according to the pediatric respiratory rules defined by the American Academy of Sleep Medicine [18]. Apneas were identified if there was a ≥90% drop in airflow for at least the duration of two breaths. Apneas associated with the presence of respiratory effort throughout the entire period of the event were classified as obstructive apneas. Apneas associated with absent respiratory effort during one portion of the event and the presence of inspiratory effort in another portion were classified as mixed apneas. Hypopneas were identified if there was a ≥30% drop in airflow for at least the duration of two breaths associated with a ≥3% oxygen desaturation. The apnea–hypopnea index (AHI) was determined as the sum of the average number of obstructive apneas per hour of sleep (obstructive apnea index [OAI]), the average number of mixed apneas per hour of sleep (mixed apnea index [MAI]) and the average number of hypopneas per hour of sleep (hypopnea index [HI]) [12]. Central apneas, defined as apneas with absent respiratory effort [18], were excluded from the AHI calculation [12].
The automatic analysis was performed using the Pediatric Respiratory Cannula Flow algorithm in the Noxturnal software version 5.1 (Nox Medical). The criteria for apneas, hypopneas and AHI were the same as those used for manual analysis. The start and stop times in the automatic analysis were manually regulated, so that they corresponded to the start and stop times used in the manual analysis.
The OSA severity was categorized based on AHI. Mild OSA was defined as 2 ≤ AHI < 5, moderate OSA was defined as 5 ≤ AHI < 10 and severe OSA was defined as AHI ≥ 10. An AHI < 2 was defined as normal.

Statistical methods

Data were analyzed using SAS Enterprise Guide version 7.1. The primary outcome variable was the AHI derived from manual and automatic analyses. First, the agreement between manual and automatic AHI was assessed using Wilcoxon signed-rank test and a scatter plot with a line of identity. Second, a linear regression model with a 95% prediction interval was performed based on log-transformed AHI data to investigate if the manual AHI could be predicted from the automatic AHI. This model was chosen after exclusion of the Bland–Altman method due to unevenly distributed differences between automatic and manual AHI even after log-transformation. In order to show the possible impact of automatic analysis on clinical decision-making, a table describing the classification of OSA severity by automatic analysis compared with manual analysis was created.
The agreement between manual and automatic analyses of OAI and HI was investigated using the Wilcoxon signed-rank test and scatter plots. After log-transformation, the HI data were suitable for description in a Bland–Altman plot with calculation of average bias, using a paired t-test, and limits of agreement (mean ± 2 × standard deviation). A Wilcoxon signed-rank test was performed to investigate the agreement between manual and automatic analyses of MAI; however, MAI only accounted for a minimal part of both automatic and manual AHI and was therefore not further analyzed.
The sample size was calculated based on a paired t-test with alpha 0.05 and power 0.8, an AHI effect size of 1 event/hour and an expected standard deviation of the difference in the AHI of 1.5 events/hour. Under these assumptions, at least 17 individuals were required. The p-values reported are two-tailed and alpha was set at 0.05 for statistical significance.

Results

A total of 51 participants aged less than 18 years with an acceptable home sleep examination were included (Figure 1). 23 participants (45.1%) were boys and 28 (54.9%) were girls. The median age was 13.6 years (range 7.9–17.8). The median BMI standard deviation score was 2.68 (range 1.04–4.22) corresponding to a median BMI percentile of 99.6 (range 85.0–100.0). 30 participants (58.8%) were diagnosed with OSA by manual analysis.
Figure 1. Flowchart describing the inclusion and exclusion of participants.

The primary outcome variable: AHI

There was a statistically significant difference in AHI between manual and automatic analyses (p < 0.001) (Table 1). The median manual AHI was 2.7 (range 0.2 to 28.2) and the median automatic AHI was 11.9 (range 4.2 to 45.6). In the scatter plot of automatic AHI versus manual AHI (Figure 2A), all points are located above the line of identity indicating a consistent overestimation of AHI by automatic analysis.
Table 1. Comparison of manually and automatically analyzed sleep parameters in 51 children and adolescents.
Sleep parameterManually analyzedAutomatically analyzedp-value
AHI (events/hour)2.7 (0.2–28.2)11.9 (4.2–45.6)<0.001
OAI (events/hour)0.1 (0.0–16.0)9.2 (2.5–35.3)<0.001
HI (events/hour)2.3 (0.2–22.8)2.4 (0.30–16.4)0.59
MAI (events/hour)0.0 (0.0–0.8)0.0 (0.0–0.5)0.30
Data are presented with medians and ranges.
Bold values indicate p < 0.05.
AHI: Apnea–hypopnea index; HI: Hypopnea index; MAI: Mixed apnea index; OAI: Obstructive apnea index.
Figure 2. Scatter plots of automatically versus manually analyzed respiratory events in 51 children and adolescents.
Figure 2A illustrates the AHI. Figure 2B illustrates the OAI. Figure 2C illustrates the HI.
AHI: Apnea–hypopnea index; HI: Hypopnea index; OAI: Obstructive apnea index.
Figure 3 shows the linear regression model based on log-transformed AHI. The root-mean-square error was 0.78 corresponding to a 95% prediction interval of 0.21–4.70 (i.e., the manual AHI could be between 21 and 470% of the AHI predicted by the model).
Figure 3. Linear regression model based on log-transformed apnea–hypopnea index (AHI).

Misclassification of OSA severity

Table 2 shows the classification of OSA severity by automatic analysis compared with manual analysis. The overall misclassification rate of the OSA severity by automatic analysis was 84.3%. All participants with normal AHI were misclassified as having mild, moderate or severe OSA by automatic analysis. Likewise, all participants with mild OSA were misclassified and had either moderate or severe OSA by automatic analysis. A third of the participants with moderate OSA were correctly classified while the remaining were misclassified as having severe OSA by automatic analysis. All participants with severe OSA were correctly classified by automatic analysis.
Table 2. Classification of obstructive sleep apnea severity by automatic analysis compared with manual analysis in 51 children and adolescents.
  Automatic analysis
  Normal AHIMild OSAModerate OSASevere OSATotal
Manual analysisNormal AHI02 (9.5%)9 (42.9%)10 (47.6%)21
 Mild OSA005 (27.8%)13 (72.2%)18
 Moderate OSA002 (33.3%)4 (66.7%)6
 Severe OSA0006 (100%)6
 Total02163351
Data are presented with frequency and row percent.
Gray areas represent complete agreement between automatic and manual analyses.
Normal AHI: AHI < 2, Mild OSA: 2 ≤ AHI < 5, Moderate OSA: 5 ≤ AHI < 10, Severe OSA: AHI ≥ 10.
AHI: Apnea–hypopnea index; OSA: Obstructive sleep apnea.

The additional variables: OAI, HI & MAI

There was a statistically significant difference between manual OAI and automatic OAI (p < 0.001; Table 1).
The scatter plot of automatic OAI versus manual OAI (Figure 2B) shows no linear relation between the two variables. All data points are located above the line of identity indicating a consistent overestimation of OAI by the automatic analysis.
No statistically significant difference between manual HI and automatic HI was found (p = 0.59; Table 1). In the scatter plot of automatic versus manual HI (Figure 2C), the data points are located around the identity line indicating some level of agreement between automatic and manual HI. Figure 4 shows the Bland–Altman plot of the log-transformed HI data with average bias and limits of agreement. The average bias was 0.05% (95% CI: -14–17%). The antilog values of the lower and upper limits of agreement were 0.33 and 3.05, respectively, meaning that the automatic HI may differ from the manual HI by 67% below to 205% above.
Figure 4. Bland-Altman plot of log-transformed hypopnea index (HI) data.
The difference between manual and automatic MAI was not statistically significant (p = 0.30; Table 1). MAI only accounted for a minimal part of both the manual and automatic AHI, 0.71 and 0.12%, respectively.

Discussion

Our results showed that the AHI was consistently overestimated by automatic analysis. The overestimation of the AHI resulted in misclassification of the OSA severity in 84.3% of the cases. The linear regression model (Figure 3) demonstrated wide prediction intervals, meaning that the model is not suitable for use in a clinical setting as a method for estimating AHI from the automatic analysis.
The difference between manual and automatic HI was not statistically significant, and the scatter plot (Figure 2C) suggested agreement between the two. A possible agreement between manual and automatic HI was further supported by the small average bias of 0.05% found using the Bland–Altman method. However, the limits of agreement were very wide indicating poor agreement between the two measurements on an individual basis.
Regarding OAI, we found a statistically significant difference between manual and automatic OAI and the scatter plot (Figure 2B) showed a clear overestimation of the OAI by automatic analysis. Therefore, our findings indicate that the overestimation of the OAI is the main explanation for the overall overestimation of the AHI by automatic analysis.
In adults, a number of studies have investigated the agreement between manual and automatic analyses for type 3 portable sleep monitors [16,17,19–24]. Both Cairns et al. [16] and Xu et al. [17] used the same device as in our study (Nox T3) and found good agreement between manual and automatic analyses. Most of the other mentioned studies found that automatic analysis underestimated the AHI [19–22]. This is in contrast to our findings; however, it is not possible to directly compare the results from the studies in adults with our study in children and adolescents. It would have been interesting to compare our findings with studies in children, but searching for literature, we did not find any studies investigating the agreement between manual and automatic analyses using type 3 portable sleep monitors in children. More research in this area would be valuable.
The major difference between the adult and the pediatric criteria for detecting apneas and hypopneas is the duration requirements of respiratory events. In children, an apnea or a hypopnea should last at least the duration of two breaths, while it should last at least 10 s in adults [18]. We speculate whether a software program would be better at recognizing the duration of 10 s than it would be at recognizing the duration of two breaths. If this were the case, it would add uncertainty to all respiratory events recognized by the pediatric automatic analysis possibly contributing to the overestimation of respiratory events. However, this does not explain why the obstructive apneas were more difficult to detect compared with the hypopneas, since the duration criteria are the same for both the obstructive apneas and the hypopneas.
It should be kept in mind that the accuracy requirements for a pediatric automatic analysis are higher than for an adult automatic analysis. With an AHI ≥ 2 as the criterion for detecting OSA in children even small differences in the AHI would be clinically relevant. Therefore, a very precise AHI calculation is necessary before a pediatric automatic analysis can be approved, while less precision can be tolerated in an automatic analysis designed for adults. An overestimation of OSA severity as seen in our study could lead to overdiagnosis and as a consequence improper treatment of patients and an unnecessary waste of health system resources.
The same experienced sleep technician analyzed all sleep examinations. However, it might have been valuable if more than one sleep technician had manually analyzed all sleep examinations because it would have given us the opportunity to compare the automatic analysis with more than one manual analysis. This is a limitation to our study. However, we chose a pragmatic approach, and it was not part of the clinical routine to have more than one sleep technician analyzing the same sleep examination.
Our study population had a high BMI status and all participants had a history of OSA. These characteristics separate our study population from the general pediatric population and might limit the extent to which the findings can be extrapolated to the general pediatric population. However, 41.2% of the participants had a normal AHI. Therefore, it seems fair to say that we have tested the automatic analysis in both ends of the AHI spectrum.
We did not compare our findings with PSG-derived sleep examinations. We wanted to evaluate the automatic analysis in a daily pragmatic way. Conducting a similar study with the inclusion of PSG monitoring would be both costlier and more inconvenient for the participants and it was considered beyond the scope of this study. When using type 3 portable sleep monitors, it is a limitation that information about arousals is not obtained. This implies a risk of underestimating the number of hypopneas. Furthermore, the determination of the total sleep time is less accurate compared with PSG.
The possible advantages of an automatic scoring method in practice could be considerable, both economically because of the reduction in technician time but also in terms of quality. We did not have data to investigate the cost–effectiveness. This could be an aspect of future investigations if automatic methods become more accurate.

Conclusion

In this study, we found that the agreement between the pediatric manual analysis and the tested pediatric automatic analysis of home sleep examinations was poor. The automatic analysis consistently overestimated the AHI leading to a high rate of misclassification of OSA severity. Our findings imply that improvements in especially the automatic detection of the OAI are needed before the tested automatic analysis can be implemented in the clinic.
Summary points
The aim of this study was to examine the agreement between manual and automatic analyses of home sleep examinations in children and adolescents using a type 3 portable device (Nox T3).
Fifty-one home sleep examinations on children and adolescents with a median age of 13.6 were included.
The sleep examinations were analyzed manually by a registered polysomnographic technologist and automatically with Noxturnal version 5.1.
The agreement between the manually analyzed apnea–hypopnea index (AHI) and the automatically analyzed AHI was poor.
The automatic analysis consistently overestimated the AHI.
The overall misclassification rate of obstructive sleep apnea severity by automatic analysis was 84.3%.
A small average bias (0.05%) was found comparing the manual and automatic hypopnea index, while the obstructive apnea index was consistently overestimated by automatic analysis.
Our findings indicate that improvements in especially the automatic detection of the obstructive apnea index are needed before the tested automatic analysis can be implemented in the clinic.

Acknowledgments

The authors thank the staff in the Sleep Clinic, Zealand University Hospital Køge, for the support.

Financial & competing interests disclosure

The study received funding from Region Zealand and Region Zealand Health Scientific Research Foundation (grant nos. 15-000342 and 180886). The study was supported by a grant from Toyota-Fonden, Denmark (grant no. KJ/BG8866H) and from ResMed Maribo (2700 € study related funding). The funding sources were not involved in the conduct of the study or in the collection, analysis and interpretation of the data. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.
No writing assistance was utilized in the production of this manuscript.

Author contributions

M Ørntoft and IG Andersen contributed equally to the design and implementation of the research, to the analysis of the results and to the writing of the manuscript. P Homøe supervised the project, discussed the results and critically revised and approved the manuscript.

Ethical conduct of research

The authors state that they have obtained appropriate institutional review board approval or have followed the principles outlined in the Declaration of Helsinki for all human or animal experimental investigations. The study was approved by the Ethics Committee of Region Zealand, Denmark (ID no. SJ-404) and by the Danish Data Protection Agency (ID no. REG-111-2014). Informed consent was obtained from all individual participants included in the study. Informed and written consent was also obtained from the parents of all individual participants.

Data sharing statement

The authors certify that this manuscript reports original clinical trial data (NCT02463201). Danish data law prohibits data sharing of person related data but on request data can be shared without person identifiable data.

References

Papers of special note have been highlighted as: • of interest; •• of considerable interest
1.
Gozal D. Obstructive sleep apnea in children: implications for the developing central nervous system. Semin. Pediatr. Neurol. 15(2), 100–106 (2008).
2.
Hunter SJ, Gozal D, Smith DL, Philby MF, Kaylegian J, Kheirandish-Gozal L. Effect of sleep-disordered breathing severity on cognitive performance measures in a large community cohort of young school-aged children. Am. J. Respir. Crit. Care Med. 194(6), 739–747 (2016).
3.
Marcus CL, Greene MG, Carroll JL. Blood pressure in children with obstructive sleep apnea. Am. J. Respir. Crit. Care Med. 157(4), 1098–1103 (1998).
4.
Kohyama J, Ohinata JS, Hasegawa T. Blood pressure in sleep disordered breathing. Arch. Dis. Child. 88(2), 139–142 (2003).
5.
Amin RS, Kimball TR, Kalra M et al. Left ventricular function in children with sleep-disordered breathing. Am. J. Cardiol. 95(6), 801–804 (2005).
6.
Rosen CL, Palermo TM, Larkin EK, Redline S. Health-related quality of life and sleep-disordered breathing in children. Sleep 25(6), 657–666 (2002).
7.
Franco RA, Rosenfeld RM, Rao M. First place--resident clinical science award 1999. Quality of life for children with obstructive sleep apnea. Otolaryngol. Head. Neck Surg. 123(1 Pt 1), 9–16 (2000).
8.
Marcus CL, Brooks LJ, Ward SD et al. Diagnosis and management of childhood obstructive sleep apnea syndrome. Pediatrics 130(3), 714–755 (2012).
9.
Verhulst SL, Schrauwen N, Haentjens D et al. Sleep-disordered breathing in overweight and obese children and adolescents: Prevalence, characteristics and the role of fat distribution. Arch. Dis. Child. 92(3), 205–208 (2007).
10.
Andersen IG, Holm J-C, Homøe P. Obstructive sleep apnea in children and adolescents with and without obesity. Eur. Arch. Oto-Rhino-Laryngology. 1–8 (2019). (Epub ahead of print).
11.
Verhulst SL, Franckx H, Van Gaal L, De Backer W, Desager K. The effect of weight loss on sleep-disordered breathing in obese teenagers. Obesity 17(6), 1178–1183 (2009).
12.
American Academy of Sleep Medicine. Obstructive sleep apnea disorders – pediatric obstructive sleep apnea. In: Sateia M (Ed.). International Classification of Sleep Disorders (3rd Edition). American Academy of Sleep Medicine, IL, USA, 63–68 (2014).
13.
Collop NA, Anderson WM, Boehlecke B et al. Clinical guidelines for the use of unattended portable monitors in the diagnosis of obstructive sleep apnea in adult patients. Portable Monitoring Task Force of the American Academy of Sleep Medicine. J. Clin. Sleep Med. 3(7), 737–747 (2007).
14.
Cole TJ, Green PJ. Smoothing reference centile curves: the LMS method and penalized likelihood. Stat. Med. 11(10), 1305–1319 (1992).
15.
Nysom K, Mølgaard C, Hutchings B, Michaelsen KF. Body mass index of 0 to 45-y-old Danes: reference values and comparison with published European reference values. Int. J. Obes. 25(2), 177–184 (2001).
•• Investigates the agreement between manual and automatic analyses for type 3 portable sleep monitors in adults. The Nox T3™ device is used.
16.
Cairns A, Wickwire E, Schaefer E, Nyanjom D. A pilot validation study for the NOX T3TM portable monitor for the detection of OSA. Sleep Breath. 18(3), 1–6 (2014).
•• Investigates the agreement between manual and automatic analyses for type 3 portable sleep monitors in adults. The Nox T3™ device is used.
17.
Xu L, Han F, Keenan BT et al. Validation of the Nox-T3 portable monitor for diagnosis of obstructive sleep apnea in chinese adults. J. Clin. Sleep Med. 13(5), 675–683 (2017).
• Investigates the agreement between manual and automatic analyses for type 3 portable sleep monitors in adults.
18.
Berry RB, Brooks R, Gamaldo C et al. Respiratory rules – part two: rules for children. In: Berry RB, Brooks R, Gamaldo C et al. (Eds). The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Specifications (Version 2.1). American Academy of Sleep Medicine, IL, USA, 47–49 (2014).
• Investigates the agreement between manual and automatic analyses for type 3 portable sleep monitors in adults.
19.
Aurora RN, Swartz R, Punjabi NM. Misclassification of OSA severity with automated scoring of home sleep recordings. Chest 147(3), 719–727 (2015).
• Investigates the agreement between manual and automatic analyses for type 3 portable sleep monitors in adults.
20.
Masa JF, Corral J, Pereira R et al. Effectiveness of sequential automatic-manual home respiratory polygraphy scoring. Eur. Respir. J. 41(4), 879–887 (2013).
• Investigates the agreement between manual and automatic analyses for type 3 portable sleep monitors in adults.
21.
Bridevaux P-O, Fitting J-W, Fellrath J-M, Aubert J-D. Inter-observer agreement on apnoea hypopnoea index using portable monitoring of respiratory parameters. Swiss Med. Wkly. 137(43–44), 602–607 (2007).
• Investigates the agreement between manual and automatic analysis for type 3 portable sleep monitors in adults.
22.
Fietze I, Glos M, Röttig J, Witt C. Automated analysis of data is inferior to visual analysis of ambulatory sleep apnea monitoring. Respiration 69(3), 235–241 (2002).
• Investigates the agreement between manual and automatic analysis for type 3 portable sleep monitors in adults.
23.
Verse T, Junge-Hülsing B, Pirsig W, Zimmermann E. First results of a prospective study validating the method of ambulatory polysomnography using the POLY-MESAM unit. Sleep Breath. 2, 56–64 (1997).
24.
Ruiz-Lopez FJ, Fertuindez-Sudrez B, Guardiola-Martinez J, Vergara-Lahuerta I, Latour-Perez J, Lorenzo-Cruz M. Quality control of the ambulatory polygraphy using automatic analysis. Chest 135, 194–200 (2009).