Data storage method and device, computer equipment and storage medium

[0023] figure 1 It is a flow chart of a data storage method provided by Embodiment 1 of the present invention. This embodiment is applicable to the situation of extracting and storing data from different sources. The method can be executed by a data storage device, which can be implemented by software and / or hardware, and generally integrated in computer equipment.

[0024] Such as figure 1 As shown, the technical solution of the embodiment of the present invention specifically includes the following steps:

[0025] S110. Acquire source data through a data source connector, and perform format conversion on the source data through a data converter.

[0026] Wherein, the data source connector connects to different data sources. Optionally, the data source connector connects to the data source through a data source connector plug-in that matches the data source; the data source includes a structured data source and / or unstructured data sources. The source data is the data th...

[0036] Figure 2a It is a flow chart of a data storage method provided by Embodiment 2 of the present invention. On the basis of the above embodiments, the embodiment of the present invention includes the process of converting the format of the source data, the process of generating the index file of the source data, and storing the index The process of the file has been further specified, and the process of processing the data in the database with MapReduce and the process of reading the data have been added.

[0037] Correspondingly, such as Figure 2a As shown, the technical solution of the embodiment of the present invention specifically includes the following steps:

[0038] S210. Acquire source data through a data source connector.

[0039] In the embodiment of the present invention, a data source connector and a data converter are deployed in a docker container. The docker container is an open source application container engine that allows developers to package appl...

[0083] image 3 It is a schematic structural diagram of a data storage device in Embodiment 3 of the present invention. The device includes a source data format conversion module 310, a source data storage module 320, and an index file storage module 330, wherein:

[0084] A source data format conversion module 310, configured to obtain source data through a data source connector, and perform format conversion on the source data through a data converter; wherein, the data source connector establishes connections with at least two data sources in advance;

[0085] The source data storage module 320 is used to save the source data after the format conversion into the database through the data storage if it is determined that the source data after the format conversion meets the data storage conditions;

[0086] The index file storage module 330 is configured to generate an index file of source data through the index generation unit, and store the index file into the index storag...