8 results about "Text corpus" patented technology

An incremental self-learning based dialogue apparatus for dialogue knowledge includes a dialogue processing unit configured to determine a intention of a user utterance by using a knowledge base and perform processing or a response suitable for the user intention, a dialogue establishment unit configured to automatically learn a user intention stored in a intention annotated learning corpus, store information about the learned user intention in the knowledge base, and edit and manage the knowledge base and the intention annotated learning corpus, and a self-knowledge augmentation unit configured to store a log of a dialogue performed by the dialogue processing unit, detect and classify an error in the stored dialogue log, automatically tag a user intention for the detected and classified error, and store the tagged user intention in the intention annotated learning corpus.

The invention provides a multilayer retrieval method based on word segmentation. The method comprises the steps of obtaining retrieval basic split words of a retrieval text; combining the retrieval basic split words to form retrieval combined split words of the retrieval text; using the retrieval basic split words and the retrieval combined split words of the retrieval text to search for text datain a corpus to obtain a retrieval result of the retrieval text.

Embodiments of the invention disclose a text corpus processing method and apparatus, a storage medium and an electronic device. The method comprises the steps of obtaining a text corpus; performing word segmentation processing on the text corpus to obtain a word sequence corresponding to the text corpus; information extraction processing is conducted on the word sequence, feature information and weight information corresponding to each word in the word sequence are obtained, and the weight information is determined according to the semantic importance degree and the position importance degree of the words in the word sequence; performing Hash mapping on the feature information corresponding to each word to obtain coded information corresponding to each word; according to the coding information corresponding to each word and the corresponding weight information, obtaining weighted coding information corresponding to each word; and carrying out fusion operation on the coding information corresponding to each word to obtain text information corresponding to the text corpus. According to the method and the device, the text corpus deduplication speed and deduplication accuracy can be improved.

The invention relates to an artificial intelligence technology, and provides a word vector generation method and device, computer equipment and a storage medium, and the method comprises the steps: training a word in a corpus through a preset model, and obtaining an initialization vector of the word in the corpus; the candidate words of the to-be-constructed word vector words generated based on the initialization vectors of the words are sorted through the preset sorting model to obtain the positive correlation set and the negative correlation set of the to-be-constructed word vector words, and the candidate words are sorted, so that the number of annotations of the words is reduced, the training efficiency of the model is improved, and the training efficiency of the model is improved. The positive correlation set and negative correlation set samples are obtained through the preset sorting model, the quality of the samples is improved, the high-quality samples are sent into the contrast learning model for training, and the generation quality of the word vector of the word vector word to be constructed is improved.

The invention discloses an online translation model selection method for statistical machine translation, which includes two stages of training and translation. The training stage includes: collecting bilingual parallel corpora, and dividing the bilingual parallel corpus into different sub-corpora according to types; The corpus trains the candidate translation model; builds an index for the sub-corpus, and obtains the corpus index file; the translation stage includes: inputting the text to be translated, and retrieving sentences similar to the sentences in the text to be translated from the corpus index file; The candidate translation model corresponding to the sub-corpus where the sentence is located, selects the final translation model from all the candidate translation models; translates the input text to be translated according to the final translation model, and obtains the final translation result. The invention well solves the problem that the statistical machine translation system cannot adapt to input texts in different fields, and can effectively improve the translation quality of the statistical machine translation system.