中文版 | English
Title

基于 Ecoflex 的柔性光电式传感器及手势识别研究

Alternative Title
Flexible photoelectric sensor based on Ecoflex and gesture recognition research
Author
Name pinyin
YE Chaoxiang
School number
12032269
Degree
硕士
Discipline
0856 材料与化工
Subject category of dissertation
0856 材料与化工
Supervisor
易正琨
Mentor unit
中国科学院深圳先进技术研究院
Publication Years
2022-05-11
Submission date
2022-06-25
University
南方科技大学
Place of Publication
深圳
Abstract
  随着人机交互技术的发展,手势识别广泛应用于虚拟现实、医疗、服务、工业等领域。手势不仅在游戏娱乐中可以为玩家带来真实的体验感,还能够提高人们的生活质量,为聋哑人带来福音。目前最主要的手势识别方法有两类,一是基于视觉可穿戴设备的手势识别,缺点是视觉在实际应用中容易受到照明条件和视野遮挡等外界因素的影响,很难实现高精度的手势识别。二是基于触觉可穿戴设备的手势识别,虽然使用触觉传感能够避免外界因素的干扰,但是现有的触觉传感器通常只能识别一维空间的手势运动,难以完成复杂的手势识别任务。
  为了解决以上问题,本文设计了基于触觉可穿戴设备和深度学习的手势识别系统,首先设计并制作了基于 Ecoflex 的光电式传感器及其可穿戴设备,基于光强调制的原理,使用不同位置分布的光敏元件赋予传感器感受二维空间力学刺激的能力,实现了全方向弯曲识别。接着设计并搭建了硬件电路系统和信号采集系统,完成了美国手势语言数据集的构建,该数据集包含 24 个静态的英文字母手势,一共 36000 帧数据,并进行了数据预处理,为触觉手势识别领域贡献了宝贵的数据资源。最后在 TASLD 数据集上
使用传统机器学习算法和深度学习算法验证了所提出的手势识别系统的识别性能,并基于正则化策略,提出了两种适用于触觉手势识别任务的深度学习算法,与基准方法相比具有更好的识别准确率和鲁棒性,实现了对复杂手势任务的高精度识别。
Keywords
Language
Chinese
Training classes
独立培养
Enrollment Year
2020
Year of Degree Awarded
2022-05
References List

[1] 王苏振.基于深度学习的手势识别技术研究[D].杭州:浙江大学,2019.
[2] Hsien-Ting Chang, Jen-Yuan Chang. Sensor Glove Based on Novel Inertial Sensor Fusion Control Algorithm for 3-D Real-Time Hand Gestures Measurements[J]. IEEE Transactions on Industrial Electronics, 2020, 67(1):658-666.
[3] Wang Y T, Ma H P. Real-Time Continuous Gesture Recognition with Wireless Wearable IMU Sensors[C]//2018 IEEE 20th International Conference on e -Health Networking, Applications and Services (Healthcom). IEEE, 2018.
[4] Jian W , Lu S , Jafari R . A Wearable System for Recognizing American Sign Language in Real-Time Using IMU and Surface EMG Sensors[J]. IEEE Journal of Biomedical and Health Informatics, 2016, 20(5):1281-1290.
[5] Lederman S J, Klatzky R L. Extracting object properties through haptic exploration[J]. Acta Psychol, 1993, 84(1):29-40.
[6] Wen F, Sun Z, He T, et al. Machine learning glove using self‐powered conductive superhydrophobic triboelectric textile for gesture recognition in VR/AR applications[J]. Advanced Science, 2020, 7(14): 2000261.
[7] Zhu M, Sun Z, Zhang Z, et al. Haptic-feedback smart glove as a creative human machine interface (HMI) for virtual/augmented reality applications[J]. Science Advances, 2020, 6(19): eaaz8693.
[8] Zhou Z, Chen K, Li X, et al. Sign-to-speech translation using machine-learning assisted stretchable sensor arrays[J]. Nature Electronics, 2020, 3(9): 571 -578.
[9] Yuan Y, Liu Y, Barner K. Tactile gesture recognition for people with disabilities[C]//Proceedings.(ICASSP'05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005. IEEE, 2005, 5: v/461-v/464 Vol. 5.
[10] Huang J , Yang X , Yu J, et al. A Universal and Arbitrary Tactile Interactive System Based on Self-powered Optical Communication[J]. Nano Energy, 2019, 69:104419.
[11] Freeman E, Brewster S, Lantz V. Towards in-air gesture control of household appliances with limited displays[C]//IFIP Conference on Human-Computer Interaction. Springer, Cham, 2015: 611-615.
[12] Fruchard B, Lecolinet E, Chapuis O. Side-Crossing Menus: Enabling Large Sets of Gestures for Small Surfaces[J]. Proceedings of the ACM on Human-Computer Interaction, 2020, 4(ISS): 1-19.
[13] Pusch A, Noël F. Augmented reality for operator training on industrial workplaces-Comparing the Microsoft hololens vs. small and big screen tactile devices[C]//IFIP International Conference on Product Lifecycle Management. Springer, Cham, 2019: 3-13.
[14] de Gea Fernández J, Mronga D, Günther M, et al. Multimodal sensor-based wholebody control for human–robot collaboration in industrial settings[J]. Robotics and Autonomous Systems, 2017, 94: 102-119.
[15] 刘宇昕.智能传感器的应用与发展趋势展望[J].数码设计(下),2019, 000(002):206.
[16] Lipomi D J, Vosgueritchian M, Tee B C K, et al. Skin-like pressure and strain sensors based on transparent elastic films of carbon nanotubes[J]. Nature Nanotechnology, 2011, 6(12): 788-792.
[17] Wang X, Dong L, Zhang H, et al. Recent progress in electronic skin[J]. Advanced Science, 2015, 2(10): 1500169.
[18] Yang J C, Mun J, Kwon S Y, et al. Electronic skin: recent progress and future prospects for skin ‐ attachable devices for health monitoring, robotics, and prosthetics[J]. Advanced Materials, 2019, 31(48): 1904765.
[19] 任 海 兵 , 祝 远 新 , 徐 光 , 等 . 基 于 视 觉 手 势 识 别 的 研 究 — 综 述 [J]. 电子学报,2000,28(2):118-121.
[20] Luo N, Dai W, Li C, et al. Flexible piezoresistive sensor patch enabling ultralow power cuffless blood pressure measurement[J]. Advanced Functional Materials, 2016, 26(8): 1178-1187.
[21] Zhu S E, Krishna Ghatkesar M, Zhang C, et al. Graphene based piezoresistive pressure sensor[J]. Applied Physics Letters, 2013, 102(16): 161904.
[22] Sundaram S, Kellnhofer P, Li Y, et al. Learning the signatures of the human grasp using a scalable tactile glove[J]. Nature, 2019, 569(7758): 698 -702.
[23] Jiang S, Li L, Xu H, et al. Stretchable e-skin patch for gesture recognition on the back of the hand[J]. IEEE Transactions on Industrial Electronics, 2019, 67(1): 647-657.
[24] Esposito D, Andreozzi E, Gargiulo G D, et al. A piezoresistive array armband with reduced number of sensors for hand gesture recognition[J]. Frontiers in Neurorobotics, 2020: 114.
[25] Yao S, Zhu Y. Wearable multifunctional sensors using printed stretchable conductors made of silver nanowires[J]. Nanoscale, 2014, 6(4): 2345-2352.
[26] Wong R D P, Posner J D, Santos V J. Flexible microfluidic normal force sensor skin for tactile feedback[J]. Sensors and Actuators A: Physical, 2012, 179: 62-69.
[27] Shahmiri F, Dietz P H. Sharc: A geometric technique for multi-bend/shape sensing[C]//Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 2020: 1-12.
[28] Nie B, Li R, Cao J, et al. Flexible transparent iontronic film for interfacial capacitive pressure sensing[J]. Advanced Materials, 2015, 27(39): 6055 -6062.
[29] Nie B, Li R, Brandt J D, et al. Microfluidic tactile sensors for three -dimensional contact force measurements[J]. Lab on a Chip, 2014, 14(22): 4344 -4353.
[30] Li T, Luo H, Qin L, et al. Flexible capacitive tactile sensor based on micropatterned dielectric layer[J]. Small, 2016, 12(36): 5042-5048.
[31] Nie B, Li R, Brandt J D, et al. Iontronic microdroplet array for flexible ultrasensitive tactile sensing[J]. Lab on a Chip, 2014, 14(6): 1107-1116.
[32] Li R, Nie B, Zhai C, et al. Telemedical wearable sensing platform for management of chronic venous disorder[J]. Annals of Biomedical Engineering, 2016, 44(7): 2282-2291.
[33] 李森.离电柔性压力传感技术及其应用研究[D].安徽:中国科学技术大学,2020.
[34] Zhu Z, Li R, Pan T. EIS: a wearable device for epidermal pressure sensing[C]//2018 IEEE Haptics Symposium (HAPTICS). IEEE, 2018: 1-6.
[35] Park K I, Son J H, Hwang G T, et al. Highly‐efficient, flexible piezoelectric PZT thin film nanogenerator on plastic substrates[J]. Advanced Materials, 2014, 26(16): 2514-2520.
[36] Dagdeviren C, Su Y, Joe P, et al. Conformable amplified lead zirconate titanate sensors with enhanced piezoelectric response for cutaneous pressure monitoring[J]. Nature Communications, 2014, 5(1): 1-10.
[37] Li C, Wu P M, Lee S, et al. Flexible dome and bump shape piezoelectric tactile sensors using PVDF-TrFE copolymer[J]. Journal of Microelectromechanical Systems, 2008, 17(2): 334-341.
[38] Syu M H, Guan Y J, Lo W C, et al. Biomimetic and porous nanofiber-based hybrid sensor for multifunctional pressure sensing and human gesture identification via deep learning method[J]. Nano Energy, 2020, 76: 105029.
[39] Gao C, Long Z, Zhong T, et al. A self-powered intelligent glove for real-time human-machine gesture interaction based on piezoelectric effect of T-ZnO/PVDF film[J]. Journal of Physics D: Applied Physics, 2022, 55(19): 194004.
[40] Yan L, Mi Y, Lu Y, et al. Weaved piezoresistive triboelectric nanogenerator for human motion monitoring and gesture recognition[J]. Nano Energy, 2022, 96: 107135.
[41] Zhao H, O’Brien K, Li S, et al. Optoelectronically innervated soft prosthetic hand via stretchable optical waveguides[J]. Science Robotics, 2016, 1(1): eaai7529.
[42] Huang J, Zhou W, Li H, et al. Attention-based 3D-CNNs for large-vocabulary sign language recognition[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 29(9): 2822-2832.
[43] Min Y, Zhang Y, Chai X, et al. An efficient pointlstm for point clouds based gesture recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Visio n and Pattern Recognition. 2020: 5761-5770.
[44] Molchanov P, Yang X, Gupta S, et al. Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 4207-4215.
[45] Rodolà E, Cosmo L, Litany O, et al. SHREC'17: Deformable shape retrieval with missing parts[C]//10th Eurographics Workshop on 3D Object Retrieval, 3DOR 2017. Eurographics Association, 2017: 85-94.
[46] Wang H, Chai X, Chen X. A novel sign language recognition framework using hierarchical Grassmann covariance matrix[J]. IEEE Transactions on Multimedia, 2019, 21(11): 2806-2814.
[47] Huang X, Wang Q, Zang S, et al. Tracing the motion of finger joints for gesture recognition via sewing RGO-coated fibers onto a textile glove[J]. IEEE Sensors Journal, 2019, 19(20): 9504-9511.
[48] Liu G Y, Kong D Y, Hu S G, et al. Smart electronic skin having gesture recognition function by LSTM neural network[J]. Applied Physics Letters, 2018, 113(8): 084102.
[49] Yuan G, Liu X, Yan Q, et al. Hand gesture recognition using deep feature fusion network based on wearable sensors[J]. IEEE Sensors Journal, 2020, 21(1): 539 -547.
[50] Garcia-Garcia A, Zapata-Impata B S, Orts-Escolano S, et al. Tactilegcn: A graph convolutional network for predicting grasp stability with tactile sensors[C]//2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 2019: 1 -8.
[51] Gao R, Taunyazov T, Lin Z, et al. Supervised autoencoder joint learning on heterogeneous tactile sensory data: Improving material classification performance[C]//2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020: 10907-10913.
[52] Ryu S C, Quek Z F, Renaud P, et al. An optical actuation system and curvature sensor for a MR-compatible active needle[C]//2012 IEEE International Conference on Robotics and Automation. IEEE, 2012: 1589-1594.
[53] Dobrzynski M K, Halasz I, Pericet-Camara R, et al. Contactless deflection sensing of concave and convex shapes assisted by soft mirrors[C]//2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2012: 4810-4815.
[54] Polygerinos P, Ataollahi A, Schaeffter T, et al. MRI-compatible intensity-modulated force sensor for cardiac catheterization procedures[J]. IEEE Transactions on Biomedical Engineering, 2010, 58(3): 721-726.
[55] Polygerinos P, Seneviratne L D, Razavi R, et al. Triaxial catheter-tip force sensor for MRI-guided cardiac procedures[J]. IEEE/ASME Transactions on Mechatronics, 2012, 18(1): 386-396.
[56] Fainman Y, Lee L, Psaltis D, et al. Optofluidics: Fundamentals, Devices, and Applications[M]. McGraw Hill Professional, 2009.
[57] Pegan J D, Ho A Y, Bachman M, et al. Flexible shrink-induced high surface area electrodes for electrochemiluminescent sensing[J]. Lab on a Chip, 2013, 13(21): 4205-4209.
[58] Li Z, Cheng L, Song Q. An ultra-stretchable and highly sensitive photoelectric effect-based strain sensor: implementation and applications[J]. IEEE Sensors Journal, 2020, 21(4): 4365-4376.
[59] Kashi S, Gupta R K, Baum T, et al. Dielectric properties and electromagnetic interference shielding effectiveness of graphene-based biodegradable nanocomposites[J]. Materials & Design, 2016, 109: 68-78.
[60] Danilov V, Dölle C, Ott M, et al. Plasma treatment of polydimethylsiloxane thin films studied by infrared reflection absorption spectroscopy[J]. The 29th International Conference on Phenomena in Ionized Gases, 2009.
[61] Wu C Y, Liao W H, Tung Y C. Integrated ionic liquid-based electrofluidic circuits for pressure sensing within polydimethylsiloxane microfluidic systems[J]. Lab on a Chip, 2011, 11(10): 1740-1746.
[62] Liu T, Sen P, Kim C J. Characterization of nontoxic liquid -metal alloy galinstan for applications in microdevices[J]. Journal of Microelectromechanical Systems, 2011, 21(2): 443-450.
[63] Polygerinos P, Ataollahi A, Schaeffter T, et al. MRI-compatible intensity-modulated force sensor for cardiac catheterization procedures[J]. IEEE Transactions on biomedical engineering, 2010, 58(3): 721-726.
[64] Ozioko O, Dahiya R. Smart tactile gloves for haptic interaction, communication, and rehabilitation[J]. Advanced Intelligent Systems, 2022, 4(2): 2100091.
[65] Al-Handarish Y, Omisore O M, Chen J, et al. A Hybrid Microstructure Piezoresistive Sensor with Machine Learning Approach for Gesture Recognition[J]. Applied Sciences, 2021, 11(16): 7264.
[66] Yan L, Mi Y, Lu Y, et al. Weaved piezoresistive triboelectric nanogenerator for human motion monitoring and gesture recognition[J]. Nano Energy, 2022, 96: 107135.
[67] Pugeault N, Bowden R. Spelling it out: Real-time ASL fingerspelling recognition[C]//2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops). IEEE, 2011: 1114-1119.
[68] Van der Maaten L, Hinton G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9(11).
[69] Le L, Patterson A, White M. Supervised autoencoders: Improving generalization performance with unsupervised regularizers[J]. Advances in Neural Information Processing Systems, 2018, 31.

Academic Degree Assessment Sub committee
中国科学院深圳理工大学(筹)联合培养
Domestic book classification number
TP391.4
Data Source
人工提交
Document TypeThesis
Identifierhttp://kc.sustech.edu.cn/handle/2SGJ60CL/342776
DepartmentShenzhen Institute of Advanced Technology Chinese Academy of Sciences
Recommended Citation
GB/T 7714
叶超翔. 基于 Ecoflex 的柔性光电式传感器及手势识别研究[D]. 深圳. 南方科技大学,2022.
Files in This Item:
File Name/Size DocType Version Access License
12032269-叶超翔-中国科学院深圳(4282KB) Restricted Access--Fulltext Requests
Related Services
Fulltext link
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Export to Excel
Export to Csv
Altmetrics Score
Google Scholar
Similar articles in Google Scholar
[叶超翔]'s Articles
Baidu Scholar
Similar articles in Baidu Scholar
[叶超翔]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[叶超翔]'s Articles
Terms of Use
No data!
Social Bookmark/Share
No comment.

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.