Method and system for inpainting forgery detection in a video image are disclosed according to some aspects of the present invention. The video image is divided into a plurality of image frames and each image frame is further divided into pixel blocks. Each image frame is segmented to identify one or more objects in the image frames. A Hessian Matrix corresponding to each of the pixel blocks of the segmented image frames is computed and correlation is computed between pixel blocks of neighboring frames. The region of inpainting forgery is detected in a video frame based on the correlation.