3.1

TF-IDF

A weighting method that is frequently used in information retrieval and information exploration is term frequency inverse document frequency. A statistical technique called TF-IDF can be used to determine how important a word is to a document inside a corpus or group of documents. A word’s significance rises in direct proportion to how frequently it appears in a text, but it falls in direct proportion to how frequently it appears in a corpus.

The word “term frequency” (TF) denotes how frequently an entry appears in the text. The TF formula is used to express this number, which is typically normalized (typically word frequency divided by the total number of words in the article) to avoid bias against long documents (the same word may occur more frequently in a long document than in a short one, regardless of whether the word is important or not):

whereas, n_i,j represents the number of times the term t_i appears in the document d_j, and TF_i,j represents the frequency of the term t_i in the document d_j.

3.2

SVM

A Support Vector Machine (SVM) is a binary classification model. Its basic model is a linear classifier with maximum intervals defined in the feature space, which makes it different from a perceptron; the SVM also includes kernel techniques, which makes it essentially a non-linear classifier.The learning strategy of the SVM is interval maximisation, which can be formalised as a problem for solving convex quadratic programming, which is also equivalent to the minimisation of the regularised hinge loss function.The learning algorithm of the SVM is an optimisation algorithm for solving convex quadratic programming.

3.3

TextCNN

Textcnn model is a model that uses convolutional neural network to deal with NLP problem, which is proposed in the article “Convolutional natural networks for sense classification” proposed by Yoon Kim Compared with the traditional RNN/LSTM models in NLP, CNN can extract important features more efficiently, which occupy an important position in classification. The architecture of textCNN is shown in Figure 2.

Figure 2.

The architecture of textCNN model.

A phrase can be thought of as a series of words first. The sequence is n words long, each word is represented by a vector x_i, and k is the dimension that each word carries. Therefore, the phrase is expressed as:

So x_{i: i+j} is a left closed and right closed interval.

The width of the convolution kernel and the dimension of the word embedding are the same since textCNN uses a onedimensional convolution. The number of words used in each window is represented by the convolution kernel’s height h. Convolution kernel, then w ∈ R^hk.

The outcome of the convolution operation is, for each sliding window result (scalar) c_i

whereb ∈ R, and f is a nonlinear function, such as tanh or ReLU.

4.1

Baseline

In this experiment, we use real data to test our predictions in real-world applications. In 2021, we collected the content records of telecom calls in 13 cities in a month. They are recorded as 20 dimensions, namely id, order number, question, reply, handler, handling date, type of question, etc. From these features we extract effective data information. We trained our model with a total of 7W pieces of data, and then tested it on 49 categories of data.

We use ranking metrics such as precision recall fi-score to check whether our classifier can accurately identify these 49 categories. We are comparing the accuracy generated by linear, randomforest, and MLP as our baselines.

As shown in Table 1, in our comparative experiment, we can see that the SVM model has achieved the best classification results. Our SVM model has achieved 82% precision, 83% recall and 77% F1 score, while the worst is the performance of randomforest model, which has only 70% precision, 70% recall and F1 score.

Table 1.

Baseline comparison.

According to the results of baseline, we can know that our feature engineering is still good, but there is still room for further optimization.

4.2

TextCNN versus baseline

We set the max feature of the embedding layer to 20000, the word vector size of the convolution layer to 64, and the pool size of max poolingid_ to 2. We set three dense layers. After a non-linear variation of the dense layers, we extract the association between these features and finally map them to the output space for classification. The structure of the model is shown in Figure 3. We used 10,000 real data for testing, including 8,000 training sets, 1,000 validation sets, and 1,000 test sets. The precision and loss curves of the model are shown in Figure 4. The model achieved the best in the 8th round. The effect, the precision reached 91%.

Figure 3.

The architecture of our telecom textCNN predictor model.

Figure 4.

The accuracy and loss curves of our telecom textCNN predictor model.