Deakin University
Browse
- No file added yet -

Vector representation based on a supervised codebook for Nepali documents classification

Download (4.39 MB)
Version 3 2024-06-19, 02:05
Version 2 2024-06-05, 11:56
Version 1 2021-03-27, 12:28
journal contribution
posted on 2024-06-19, 02:05 authored by C Sitaula, A Basnet, Sunil AryalSunil Aryal
Document representation with outlier tokens exacerbates the classification performance due to the uncertain orientation of such tokens. Most existing document representation methods in different languages including Nepali mostly ignore the strategies to filter them out from documents before learning their representations. In this article, we propose a novel document representation method based on a supervised codebook to represent the Nepali documents, where our codebook contains only semantic tokens without outliers. Our codebook is domain-specific as it is based on tokens in a given corpus that have higher similarities with the class labels in the corpus. Our method adopts a simple yet prominent representation method for each word, called probability-based word embedding. To show the efficacy of our method, we evaluate its performance in the document classification task using Support Vector Machine and validate against widely used document representation methods such as Bag of Words, Latent Dirichlet allocation, Long Short-Term Memory, Word2Vec, Bidirectional Encoder Representations from Transformers and so on, using four Nepali text datasets (we denote them shortly as A1, A2, A3 and A4). The experimental results show that our method produces state-of-the-art classification performance (77.46% accuracy on A1, 67.53% accuracy on A2, 80.54% accuracy on A3 and 89.58% accuracy on A4) compared to the widely used existing document representation methods. It yields the best classification accuracy on three datasets (A1, A2 and A3) and a comparable accuracy on the fourth dataset (A4). Furthermore, we introduce the largest Nepali document dataset (A4), called NepaliLinguistic dataset, to the linguistic community.

History

Journal

PeerJ Computer Science

Volume

7

Article number

ARTN e412

Pagination

1-18

Location

United States

Open access

  • Yes

ISSN

2376-5992

eISSN

2376-5992

Language

English

Publication classification

C1 Refereed article in a scholarly journal

Publisher

PEERJ INC