Neural network visual dialogue model and method based on KR product fusion multi-modal information

The invention discloses a neural network visual dialogue model and method based on KR product fusion multi-modal information. The model comprises a modal feature extraction module, a different modal information fusion module and a candidate answer prediction module. The modal feature extraction module extracts features of question texts and features of historical information through an LSTM network, extracts entity features of pictures by using a Faster R-CNN network, and extracts visual features related to questions by using an attention mechanism; the different modal information fusion module captures feature information in different modals by using a later fusion method, captures associated information among the different modals through a feature fusion method based on a KR product, and fuses the information in the modals and the information among the modals; and the candidate answer prediction module performs answer prediction by using a fusion vector fusing intra-modal information and inter-modal information, so that related answers can be found out more accurately. According to the method, the current situation that in a traditional visual dialogue model, associated information among different modals is insufficiently captured through later fusion is overcome.