Introduction
Text classification is a core task of natural language processing and
has been used in many real-world applications, such as spam
detection1 and opinion mining2.
Transduction learning3 is a special text
classification method that uses both labelled and unlabeled samples in
the training process. A graph neural network (GNN) is an effective
transduction learning method4,5 and is widely used in
text classification applications. This method constructs a graph to
model the relationship between documents. Nodes in the figure represent
text units such as text or documents, while edges are constructed based
on semantic similarity between nodes. Therefore, a GNN can be used to
learn and classify nodes in the figure. The advantages of this method
for classification are as follows: (1) The representation of each node
depends not only on itself but also on its neighbors, endowing the
representation of nodes certain context information; (2) During
training, the model spreads the influence of supervision labels in
training and test cases through graphs and edges. Even data with no
labels help to represent the learning process, yielding a higher
performance.
However, the use of GNN for text classification has the following
problems: (1) The method based on GNN does not regard the text as a
sequence but as a set of co-occurring words. In the task of text
classification, the word order relationship in the sentence plays a
crucial role in the final classification result; (2) The traditional GNN
does not make full use of edge features. Only 0 and 1 are used between
nodes to ascertain whether there is a connection, namely, the
connectivity feature; however, the edge features of the graph often have
rich semantic information, such as the type of connection between nodes,
connection strength, etc ., and should be represented as
continuous vector features rather than binary variables.
Recent studies have shown that large-scale pre-training models are
effective for various natural language processing tasks, especially text
classification tasks6,7. The pre-training model takes
the unsupervised corpus as the training object and can learn the rich
text semantics implied in the language. However, the methods used for
transducing text classification tasks prior to 20204,5,13-17 did not consider the use of pre-trained
models. It was not until 2021 when Lin et al . proposed
BERTGCN26, which combines BERT and GCN and
demonstrates the effectiveness of pre-trained models in transductive
learning.