Teacher Assistant-Based Knowledge Distillation Extracting Multi-level Features on Single Channel Sleep EEG

International Joint Conference on Artificial Intelligence (IJCAI) 2023

Heng Liang1,  Yucheng Liu1,   Haichao Wang2,  Ziyu Jia1* 
1Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China, 2Tsinghua-Berkeley Shenzhen Institute, Shenzhen, China
Code | Video | Paper | Poster
Teacher Assistant-Based Knowledge Distillation Extracting Multi-level Features on Single Channel Sleep EEG (SleepKD)

Fig. 1 Framework

SleepKD includes three modules, which are the multi-level knowledge distillation module (Lepoch and Lseq), the teacher assistant module, and other knowledge distillation module (Lsoft and Lhard). The multi-level knowledge distillation module transfers knowledge in the features from the sleep epochs and sleep sequences. The teacher assistant module is designed to bridge the gap between the teacher and student network.

Experiment Result
Table. 1 Performance of SleepKD on DeepSleepNet and SalientSleepNet.
We evaluate SleepKD in different aspects. Table 1 presents that the student models achieve 74.68% and 71.78% on the compression ratio while the reduction of the accuracy are less than 1%. These data reveal that our framework is able to compress the model the most with the least cost of accuracy. Therefore, the performance in different aspects gets a significant improvement.
As shown in Table 2, we perform several experiments with SleepKD and baseline methods on SalientSleepNet and DeepSleepNet, which are classic models with a CNN framework and a hybrid framework based on CNN and RNN, respectively. SleepKD achieves the SOTA knowledge distillation results.
Table. 2 Comparison results.
Talk
Reference
        @inproceedings{DBLP:conf/ijcai/LiangLWJ23,
            author       = {Heng Liang and
                            Yucheng Liu and
                            Haichao Wang and
                            Ziyu Jia},
            title        = {Teacher Assistant-Based Knowledge Distillation Extracting Multi-level
                            Features on Single Channel Sleep {EEG}},
            booktitle    = {Proceedings of the Thirty-Second International Joint Conference on
                            Artificial Intelligence, {IJCAI} 2023, 19th-25th August 2023, Macao,
                            SAR, China},
            pages        = {3948--3956},
            publisher    = {ijcai.org},
            year         = {2023},
            url          = {https://doi.org/10.24963/ijcai.2023/439},
            doi          = {10.24963/ijcai.2023/439},
            timestamp    = {Mon, 14 Aug 2023 16:05:12 +0200},
            biburl       = {https://dblp.org/rec/conf/ijcai/LiangLWJ23.bib},
            bibsource    = {dblp computer science bibliography, https://dblp.org}
          }