US 7613322 B2
A method of video surveillance may include performing on input video at least one of the operations selected from the group consisting of motion detection and change detection, recording a motion pattern based on a result of said at least one of the operations, and analyzing the motion pattern to detect periodic motion in the video. A video surveillance apparatus may include a change detector, a motion detector, and/or a combination motion/change detector, a pattern analyzer, and a periodic motion detector.
1. A method of video surveillance comprising:
receiving input video comprising a plurality of input video frames;
performing on the plurality of input video frames at least one of motion detection or change detection to obtain at least one foreground mask having a plurality of foreground pixels and a time stamp;
generating at least one motion block within an area of interest of the input video based on the obtained at least one foreground mask;
analyzing the generated at least one motion block to detect periodic motion of an object within the area of interest; and
generating at least one alert if periodic motion is detected,
wherein said generating at least one motion block comprises:
dividing one of the at least one foreground mask having a plurality of foreground pixels and one time stamp to obtain at least one block having a search window;
determining a number of foreground pixels present in the search window of the obtained at least one block;
comparing the number of foreground pixels present in the search window of the obtained at least one block with a threshold value; and
generating at least one motion block having the time stamp of the foreground mask from which said at least one block is obtained, if the threshold value has been surpassed,
wherein said analyzing the generated at least one motion block further comprises:
determining a period of time for which the generated at least one motion block has had its number of foreground pixels exceed said threshold value;
comparing said period of time with a time threshold;
if the period of time is larger than said time threshold, storing the pixel coordinates and time stamp of the generated at least one motion block in a motion block array for further analysis;
obtaining the pixel coordinates and time stamp of one motion block within the area of interest from the motion block array;
comparing the obtained time stamp to the time stamp(s) of at least one other motion block in the motion block array to determine if the differences between the obtained time stamp and the time stamp(s) of at least one other motion block are larger than a difference threshold;
defining a periodic motion counter for each set of pixel coordinates; and
incrementing the periodic motion counter for the obtained pixel coordinates corresponding to the same motion block as that of the obtained time stamp if the differences between the obtained time stamp and the time stamp(s) of at least one other motion block are larger than the difference threshold.
2. The method according to
comparing each periodic motion counter with a motion counter threshold to determine if the periodic motion counter exceeds the motion counter threshold,
wherein periodic motion is detected if the periodic motion counter exceeds the motion counter threshold.
3. A computer-readable medium containing instructions that, when executed on a computer system, cause the computer system to implement the method according to
4. A video-based surveillance system comprising:
a computer system; and
the computer-readable medium according to
5. The video surveillance system according to
6. Application-specific hardware designed to implement the method according to
7. The application-specific hardware according to
8. A method of multi-grabbing detection comprising the method of video surveillance according to
detecting said periodic motion as multi-grabbing.
9. Video surveillance apparatus comprising:
a component selected from the group consisting of: a change detector, a motion detector, and a combination motion/change detector; wherein said component is to:
receive input video comprising a plurality of input video frames, each having a time stamp;
process the plurality of input video frames; and
output motion data comprising at least one foreground mask having a plurality of foreground pixels and one time stamp;
a motion pattern analyzer to:
receive said motion data comprising at least one foreground mask having a plurality of foreground pixels and one time stamp;
generate at least one motion block based on the received motion data; and
output motion pattern data comprising a motion block array; and a periodic motion detector to:
receive the motion pattern data comprising a motion block array;
detect periodic motion of an object within an area of interest based on the motion pattern data comprising a motion block array; and
generate at least one alert if periodic motion is detected,
wherein the motion block array comprises an array of pixel coordinates and corresponding time stamps of the at least one motion block generated by the motion pattern analyzer,
wherein said periodic motion detector is arranged to compare differences between pairs of said time stamps from the motion block array to a time threshold and to maintain and update a counter corresponding to each set of pixel coordinates based on said comparisons.
10. The video surveillance apparatus according to
a motion block detector to receive the motion data comprising at least one foreground mask having a plurality of foreground pixels and one time stamp and to detect one or more motion blocks based on the motion data; and
a motion temporal change recorder to check the persistence of each motion block.
11. The video surveillance apparatus according to
12. A multi-grabbing detection apparatus comprising the video surveillance apparatus according to
Embodiments of the invention generally relate to surveillance systems. Some specific embodiments of the invention relate to a video-based surveillance system that can be used, for example, in retail-based surveillance areas, to detect shoplifting.
Some state-of-the-art intelligent video surveillance (IVS) system can perform content analysis on frames generated by surveillance cameras. Based on user-defined rules or policies, IVS systems can automatically detect potential threats by detecting, tracking and analyzing the targets in the scene. One significant constraint of the system is that the targets have to be isolated in the camera views. Existing IVS systems have great difficulty in tracking individual targets in a crowd situation, mainly due to target occlusions. For the same reason, the types of targets or the behaviors of the targets that a conventional IVS system can distinguish are also limited.
In many situations, security needs demand much greater capabilities from an IVS. One example is the detection of shoplifting. Theft from stores, including employee and vendor theft, costs retailers many billions of dollars per year. Independent retail studies have estimated that theft from retail stores costs the American public between 20 and 30 billion dollars per year. Depending on the type of retail store, retail inventory shrinkage ranges from 0.5%-6% of gross sales, with the average falling around 1.75%. Whole retail store chains have gone out of business due to their inability to control retail theft losses. Although most stores have video surveillance cameras installed, most of them just serve as forensic tape providers. Intelligent real-time theft detection capability is highly desired but is not available.
Certain suspicious activities by customers in retail stores could be indicators of potential theft. Examples of these include when a person takes and replaces a single item from a store shelf multiple times, and when a person takes more than the typical number of items from a shelf. Shoplifting rates can be greatly reduced if an IVS system can detect such suspicious activities and alert store management in real time.
Another exemplary scenario in which multi-grabbing may occur is in a gaming milieu. A casino, for example, may be concerned about monitoring when gaming participants reach onto gaming tables. Multi-grabbing in this scenario may reflect cheating and/or stealing. In particular, most casinos have rules applying to various gaming tables that require that no player reach beyond a certain threshold, into the domain of the dealer. This is to prevent players from reaching in and grabbing a stack of chips or cash. An algorithm to detect the motion of a player (that is, hand and arm) crossing this (generally invisible) threshold in a particular direction and then returning, while ignoring the opposite motion (e.g., by the dealer) would thus be useful to the casino owner.
Embodiments of the present invention are directed to algorithms and system modules for an IVS system to automatically detect “multi-grabbing” behavior of human targets by detecting and analyzing the periodic motion of target or part of the target.
An exemplary embodiment of the invention may be directed to a “multi-grabbing” detection algorithm comprising performing motion/change detection of an input surveillance video; recording a motion pattern; and detecting a “multi-grabbing” event by analyzing the motion pattern. A method for implementing such an embodiment may comprise performing on input video at least one of the operations selected from the group consisting of motion detection and change detection; recording a motion pattern based on a result of said at least one of the operations; and analyzing the motion pattern to detect periodic motion in the video.
A further embodiment of the invention may comprise an apparatus including a component selected from the group consisting of: a change detector, a motion detector, and a combination motion/change detector; wherein said component is to receive input video and to output motion data; a motion pattern analyzer to receive said motion data and to output motion pattern data; and a periodic motion detector to receive said motion pattern data.
The invention may also be embodied in the form(s) of hardware, software, firmware, and/or combinations thereof. Furthermore, such hardware may take the form of programmable and/or application-specific hardware, for example, one or more microprocessors, special-purpose processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.
The invention may also be embodied in the form of software embodied in a computer-readable medium and/or in a system containing such a computer-readable medium.
The following definitions are applicable throughout this disclosure, including in the above.
A “video” refers to motion pictures represented in analog and/or digital form. Examples of video include: television, movies, image sequences from a video camera or other observer, and computer-generated image sequences.
A “frame” refers to a particular image or other discrete unit within a video.
An “object” refers to an item of interest in a video. Examples of an object include: a person, a vehicle, an animal, and a physical subject.
A “target” refers to the computer's model of an object. The target is derived from the image processing, and there is a one to one correspondence between targets and objects.
A “block” refers to a group of pixels.
“Foreground” refers to the area in a frame having meaningful change over time. For example, a walking person may be meaningful to a user, and should thus be considered as foreground. In contrast, some types of moving areas are not meaningful and should be considered as background, such as water waves, tree leaves blowing, sun glittering, etc. Other areas, that do not contain motion, may be considered as background.
A “MOTION block” refers to a video block in which there are at least some predetermined number of foreground pixels present.
A “computer” refers to any apparatus that is capable of accepting a structured input, processing the structured input according to prescribed rules, and producing results of the processing as output. The computer can include, for example, any apparatus that accepts data, processes the data in accordance with one or more stored software programs, generates results, and typically includes input, output, storage, arithmetic, logic, and control units. Examples of a computer include: a computer; a general purpose computer; a supercomputer; a mainframe; a super mini-computer; a mini-computer; a workstation; a micro-computer; a server; an interactive television; a web appliance; a telecommunications device with internet access; a hybrid combination of a computer and an interactive television; a portable computer; a personal digital assistant (PDA); a portable telephone; and application-specific hardware to emulate a computer and/or software. A computer can be stationary or portable. A computer can have a single processor or multiple processors, which can operate in parallel and/or not in parallel. A computer also refers to two or more computers connected together via a network for transmitting or receiving information between the computers. An example of such a computer includes a distributed computer system for processing information via computers linked by a network.
A “computer-readable medium” may refer to any storage device used for storing data accessible by a computer. Examples of a computer-readable medium include: a magnetic hard disk; a floppy disk; an optical disk, such as a CD-ROM and a DVD; a magnetic tape; a memory chip; and a carrier wave used to carry computer-readable electronic data, such as those used in transmitting and receiving e-mail or in accessing a network.
“Software” refers to prescribed rules to operate a computer. Examples of software include: software; code segments; instructions; software programs; computer programs; and programmed logic.
A “computer system” refers to a system having a computer, where the computer comprises a computer-readable medium embodying software to operate the computer.
A “network” refers to a number of computers and associated devices that are connected by communication facilities. A network involves permanent connections such as cables or temporary connections such as those made through telephone, wireless, or other communication links. Examples of a network include: an internet, such as the Internet; an intranet; a local area network (LAN); a wide area network (WAN); and a combination of networks, such as an internet and an intranet.
An “information storage device” refers to an article of manufacture used to store information. An information storage device has different forms, for example, paper form and electronic form. In paper form, the information storage device includes paper printed with the information. In electronic form, the information storage device includes a computer-readable medium storing the information as software, for example, as data.
Specific embodiments of the invention will now be described in further detail in conjunction with the attached drawings, in which:
Various embodiments of the invention will now be presented in detail.
Module 304, may go through the foreground masks in output 308 and may generate motion block masks. Then, based on a history of each motion block, module 304 may decide if the motion block information should be stored in an array 310, where the motion block information may include x and y coordinates of the motion block, as well as a current time stamp. Module 306, may be used to perform periodic motion event detection based on the motion block array, as well as on user-defined rules, which may include, but which are not limited to, such rules as area of interest, time window, and/or a minimum number of incidences of a periodic motion to indicate that a user should pay attention.
The second module 404 in the exemplary embodiment of
The embodiments and examples discussed herein should be understood to be non-limiting examples.
The invention is described in detail with respect to preferred embodiments, and it will now be apparent from the foregoing to those skilled in the art that changes and modifications may be made without departing from the invention in its broader aspects, and the invention, therefore, as defined in the claims is intended to cover all such changes and modifications as fall within the true spirit of the invention.