A context-enhanced sentence representation learning method for close domains with topic modeling

Li, Shuangyin1; Chen, Weiwei2; Zhang, Yu3; Zhao, Gansen1; Pan, Rong2; Huang, Zhenhua1; Tang, Yong1

2022-08-01

10.1016/j.ins.2022.05.113

INFORMATION SCIENCES   Impact Factor & Quartile

0020-0255

1872-6291

Sentence representation approaches have been widely used and proven to be effective in many text modeling tasks and downstream applications. Many recent proposals are avail-able on learning sentence representations based on deep neural frameworks. However, these methods are pre-trained in open domains and depend on the availability of large-scale data for model fitting. As a result, they may fail in some special scenarios, where data are sparse and embedding interpretations are required, such as legal, medical, or technical fields. In this paper, we present an unsupervised learning method to exploit representa-tions of sentences for some closed domains via topic modeling. We reformulate the infer-ence process of the sentences with the corresponding contextual sentences and the associated words, and propose an effective context-enhanced process called the bi-Directional Context-enhanced Sentence Representation Learning (bi-DCSR). This method takes advantage of the semantic distributions of the nearby contextual sentences and the associated words to form a context-enhanced sentence representation. To support the bi-DCSR, we develop a novel Bayesian topic model to embed sentences and words into the same latent interpretable topic space called the Hybrid Priors Topic Model (HPTM). Based on the defined topic space by the HPTM, the bi-DCSR method learns the embedding of a sentence by the two-directional contextual sentences and the words in it, which allows us to efficiently learn high-quality sentence representations in such closed domains. In addition to an open-domain dataset from Wikipedia, our method is validated using three closed-domain datasets from legal cases, electronic medical records, and technical reports. Our experiments indicate that the HPTM significantly outperforms on language modeling and topic coherence, compared with the existing topic models. Meanwhile, the bi-DCSR method does not only outperform the state-of-the-art unsupervised learning methods on closed domain sentence classification tasks, but also yields competitive performance com-pared to these established approaches on the open domain. Additionally, the visualizations of the semantics of sentences and words demonstrate the interpretable capacity of our model.(c) 2022 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Sentence representations learning Closed domains Bayesian sentence embedding Bi-directional context-enhanced Semantic interpretability Topic modeling

SCI ; EI

English

Others

National Natural Science Foundation of China[62006083] ; GuangZhou Basic and Applied Basic Research Foundation[202102020654] ; Applied Basic Research Fund of Guangdong Province[2019B1515120085]

Computer Science

Computer Science, Information Systems

WOS:000817892200011

ELSEVIER SCIENCE INC

20222412213143

Embeddings ; Medical computing ; Modeling languages ; Unsupervised learning

Biomedical Engineering:461.1 ; Artificial Intelligence:723.4 ; Computer Applications:723.5

COMPUTER SCIENCE

Web of Science

1.South China Normal Univ, Sch Comp Sci, Guangzhou, Guangdong, Peoples R China
2.Sun Yat sen Univ, Sch Data & Comp Sci, Guangzhou, Guangdong, Peoples R China
3.Southern Univ Sci & Technol, Dept Comp Sci & Engn, Shenzhen, Guangdong, Peoples R China

Li, Shuangyin,Chen, Weiwei,Zhang, Yu,et al. A context-enhanced sentence representation learning method for close domains with topic modeling[J]. INFORMATION SCIENCES,2022,607:186-210.

Li, Shuangyin.,Chen, Weiwei.,Zhang, Yu.,Zhao, Gansen.,Pan, Rong.,...&Tang, Yong.(2022).A context-enhanced sentence representation learning method for close domains with topic modeling.INFORMATION SCIENCES,607,186-210.

Li, Shuangyin,et al."A context-enhanced sentence representation learning method for close domains with topic modeling".INFORMATION SCIENCES 607(2022):186-210.