(1) Text input

The FastText model inputs a batch of documents, each consisting of a sequence of lexical indexes, and the output layer generates probability values indicating the likelihood that the document will belong to a category. The word and the set of word vectors are projected onto the hidden layer, which is ultimately projected onto the tag. Almost all network inputs rely on simple linear combinations, but they cannot fit all functions, so nonlinear activation functions are used here. The FastText model principle is not much different from CBOW in Word2Vec^{9, 10}. The only difference is that the former targets multiple words and word vectors, while the latter targets the context of the word. The whole process is actually word segmentation, deleting redundant words and invalid numbers and so on.

(2) Vector features of generated words

In FastText model, the low-latitude vector is related to each word. As a word bag-based classification model, FastText captures the context word order relationship in the document text through N-gram features to deepen the understanding of the meaning of the document. In order to solve the problem of large vocabulary and difficult calculation, FastText cleverly adopts the Hash bucket method, which hashes all the features captured by N-Grams into the bucket, and n-Grams in the same bucket share a word vector.

Word2vec allocates a vector to each word in the constructed word set, thus constituting the word vector. Using this process, we can discard its meaning characteristics, for example, “tower” and “towers”, Most of them are the same. In other words, they look the same, but understood from traditional word2vec, because they have their own ids, the underlying implications will be ignored. A way to solve this problem is to use n-grams as a word, such as “tower”, to treat n as three, so all trigrams are: ““to”, “tow”, “owe,” “wer er >”, ””, the symbols “<” on behalf of the word formation in front of the composition, “ >” on behalf of root formation. Therefore, trigram was used instead of “pole and tower” for description.

(3) Generate classification model

The negative logarithmic likelihood function of the classification model can be expressed by equation (4):

In the above equation, x_n is the independent variable, and the multiplication with A is equivalent to extracting the desired word from the thesaurus, and the obtained result is multiplied by A coefficient B. Finally, the classification result can be easily obtained by combining the corresponding word vector and using the characteristics of linear classifier, which is called softmax function.

If O(kh) is represented as time complexity, k is represented as the number of all categories and H stands for the scope of the entire text. In the process of operation, the probability value of all words should be obtained through softmax function, and Max value should be sought on this basis. Due to the large amount of input, the whole process is relatively complex and time-consuming. In order to optimize the training time, hierarchical Softmax method is adopted, and hierarchical Softmax is combined with Huffman algorithm. It is a kind of tree structure, each leaf node represents a word of corpus, a word corresponds to a code value, each coding sequence can be mapped to each sequence of events, used to solve from hidden layer to output layer softmax function the problem of large computation, effectively reduce the classification model to predict the number of labels, makes one category prediction probability calculation, Computing costs have dropped significantly. Compared with other multi-class logarithmic models, hierarchical Softmax functions can greatly reduce the complexity of model training and the time-consuming operation process. According to Huffman’s principle of putting the most important features first, words with high frequency in corpus are located closer to the root node than words with low frequency, which reduces path dependence and optimizes operation efficiency.

(4) Negative sampling rule

The negative sampling algorithm considers the positive path, that is, the final output target words should be captured and retained by the model, and the wrong output words should also be selected as negative samples to optimize model parameters. Because different words are interfered by many factors, it is impossible to determine in which range they will appear. Therefore, the influence of this part is studied, and the weighted value of each word is calculated to ensure that the high-frequency words will be selected as negative samples with a greater probability. In the process of model parameter updating, part of corpus is randomly selected for training, which saves the training time. For known positive samples(context (w), w), to maximize the g(w) = ∏_{u∈{w}∪NEG(w)} p(u | context (w)), The conditions are as follows:

w is the background word of the positive sample, context (w) is the center word of the positive sample, represents the probability of predicting the center word w when the context is context (w). is the sum of vectors of each word, and T represents the rank of transformation; θ^и is the vector given by this word, the untrained variable; new u ∈ w ∪ NEG(w); L^w is the label of the word w, that is, the label of the positive sample is 1 and the label of the negative sample is 0.

Therefore, the final loss function is L = log ∏_w∈C g(w) = Σ_w∈C log g(w), under the rules of negative samples are:

Through negative sampling, the central word of the word order can be extracted and replaced by other words, so as to increase the number of negative samples in the corpus. In this way, the frequency of positive samples can be increased and the frequency of negative samples can be reduced. In the training process, the weight of the hidden layer of the network changes constantly, but the negative sampling only changes the weight of the low. This idea can reduce the difficulty of reducing the computational complexity of gradient.