The invention relates to a method for detecting crossing of a temporary fence on a construction site based on visual analysis. The method comprises the following steps: 1, constructing a detector; 2,acquiring a camera picture of a construction site in real time, calling a detector, continuing the step 2 N seconds later if the fence is not detected, continuing the step 2 if the constructor is notdetected, and otherwise, entering the step 3; 3, initializing the constructors detected in the step 2 into a tracking target queue; 4, predicting the state of the tracking target queue; reading a real-time video frame f of the camera, and calling a detector to obtain a position d of the construction personnel in the video frame f; 5, matching the tracking result with the detection result; 6, detecting a fence position in the video frame f, judging whether an intersection point exists between the forming section and the motion trail of the construction personnel, and if so, giving an alarm; and7, rejecting the unmatched queue in the step 5, adding the unmatched d into the tracking target queue, and returning to the step 4. According to the method, intelligent construction site violation behavior detection is realized.