Structured Knowledge Modeling, Extraction and Localization from Images

[0025] overview

[0026] Techniques and systems are described that support knowledge extraction from images to generate descriptive summaries of images that can then be used to support image searches, automatically generate captions and metadata for images, and various other uses. A descriptive summary may describe, for example, the quality of the image as a whole as well as attributes, objects, and mutual interactions of objects within the image as described further below. Thus, while examples involving image searching are described below, these techniques are equally applicable to a variety of other examples, such as automated structured image tagging, caption generation, and the like.

[0027] First obtain training data to use machine learning to train a model to generate structured image representations. Techniques are described herein in which training data is obtained using images and associated text (e.g., captions for images that include any text describing the scene c