Data acquisition method

A data collection and data source technology, applied in the field of big data, can solve problems such as low collection efficiency and inability to access the data source website normally, and achieve the effect of improving collection efficiency

Pending Publication Date: 2020-10-09
BEIJING DINGTAI ZHIYUAN TECH CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

If the visit frequency of the data source website is too high, the anti-crawling mechanism of the website will be triggered, resulting in the inability to

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0017] The present invention will be specifically introduced below with reference to specific embodiments.

[0018] An embodiment of the present invention provides a data collection method, which includes the following steps:

[0019] S101, using the Q-Learning algorithm to establish a Q table for each data source website respectively, wherein the Q table takes the best access frequency of each data source website to be accessed as a field, and takes the URL corresponding to each data source website as a primary key;

[0020] S102, visit the corresponding data source website according to the best visit frequency in the Q table.

[0021] Optionally, the calculation process of the optimal access frequency of each data source website to be accessed includes:

[0022] Use different access frequencies to visit the data source website, until the data source website cannot be accessed normally;

[0023] Extract the access frequency that can normally access the data source website, and

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data acquisition method, and relates to the technical field of big data. The method comprises the steps: establishing a Q table for each data source website by utilizing a Q-Learning algorithm, visiting the corresponding data source website according to the optimal access frequency in the Q table, and updating the optimal access frequency of each data source website by adopting a time difference method, thereby improving the collection efficiency.

Description

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Owner BEIJING DINGTAI ZHIYUAN TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products