Video Sentiveillance SDK
Persons or vehicles recognition and tracking for video surveillance systems
SentiVeillance SDK is designed for developing software that performs biometric face identification, detects moving pedestrians or vehicles or other objects and performs automatic license plate recognition using live video streams from digital surveillance cameras.
The SDK is used for passive identification – when passers-by do not make any efforts to be recognized. List of possible uses includes law enforcement, security, attendance control, visitor counting, traffic monitoring and other commercial applications.
-
Biometric face recognition – based on deep neural networks and
provides these capabilities for surveillance systems:
- Multiple face detection, features extraction and template matching with the internal database in real time.
- Facial identification reliability enables using large watchlist databases.
- Face tracking is performed in all successive frames from the video source until they disappear from camera field of view. The face tracking algorithm uses dynamic face and motion prediction models that make it robust to occlusions like other objects or even other faces. The algorithm is able to continue tracking a face even when it re-appears after being fully covered by occlusions (like walls, furniture, posters etc). Also, if the pedestrian tracking algorithm tracks a person without their face visible, and later their face is recognized, the whole tracking history will be associated with the person's identity, as well as the further non-face tracking.
- Gender classification (optional) for each person in the frame.
- Age determination (optional) for each person in the frame.
- Face mask, smile, open-mouth, closed-eyes, glasses, dark-glasses, beard and mustache attributes detection (configurable).
-
Vehicle or human detection, classification and movement
tracking – performs object detection of moving and static objects
in the scene, their classification and tracking until they disappear.
These features are available for surveillance systems:
- Object classification. SentiVeillance allows to perform object classification, locations and tracking based on its type. Over 20 classes are recognized: pedestrian, bus, truck, sedan, SUV, etc.
- Color estimation. The algorithm returns most likely color estimation for cars and pedestrians. The estimated color values are: red, orange, yellow, green, blue, silver, white, black, brown, grey.
- Make and model estimation for vehicles. A neural networks-based performs the estimation of the tracked vehicle's make (brand) and model.
- Clothing analysis. When a pedestrian is detected, their appearance can be evaluated for the presence of different clothes types. Over 20 clothes and headwear classes are recognized, like shirts, trousers, skirts or coats, as well as certain national clothings. Also, this algorithm can estimate if the person is partially or totally naked.
- Movement direction estimation. The algorithm estimates vehicle or pedestrian movement direction as angle relative to camera orientation.
- Tolerance to object visibility. The detection algorithm works with partially visible objects and from great distance.
- Combination with facial recognition and ALPR. If a pedestrian was tracked without their face visible, and later the person is recognized by the facial recognition algorithm, the whole tracking history will be associated with the person's identity, as well as the further non-face tracking. Vehicle tracking can be updated in the similar way with the help of the ALPR algorithm.
-
Automated license plate recognition (ALPR) – once a vehicle has
been detected, SentiVeillance ALPR algorithm detects and reads the license
plate:
- Traffic data processing. SentiVeillance algorithms can simultaneously read vehicle license plates from multiple moving vehicles.
- Tolerance to camera position. Depending on camera resolution, the ALPR algorithm can read license plates from longer distance and higher angle.
- Preventing cheating with replaced license plates. Integrators can use vehicle recognition and ALPR modalities together for making software logic which checks if recognized license plate corresponds other registration data, like vehicle color or type, and not being spoofed or moved from another vehicle.
SentiVeillance 8.0 SDK is based on the SentiVeillance 8.0 technology that is specially designed for integrating biometric facial recognition into video surveillance systems. Face templates created with SentiVeillance SDK are fully compatible with VeriLook SDK and MegaMatcher SDK multi-biometric technology.
SentiVeillance 8.0 SDK includes SentiVeillance component, which can process multiple video streams on the same PC or server. The SentiVeillance component supports three modalities: biometric facial recognition, vehicle/human classification and automatic license plate recognition. Also the SDK includes Device Manager library for Microsoft Windows and Linux that allows to perform simultaneous capture from multiple cameras.
| Components | Microsoft Windows (64 bit only) | Linux (64 bit only) |
|---|---|---|
| SentiVeillance component's Face modality | license for 1 video stream | |
| SentiVeillance component's VH modality | license for 1 video stream | |
| SentiVeillance component's ALPR modality | license for 1 video stream | |
| Device Manager library | + | + |
| Programming samples | ||
| C++ | + | + |
| C# | + | |
| Visual Basic .NET | + | |
| Programming tutorials | ||
| C | + | + |
| C# | + | |
| Java | + | + |
| Documentation | ||
| SentiVeillance 8.0 SDK documentation | + | |
Device Manager Library
Device Manager library works under Microsoft Windows and Linux and provides functionality for simultaneous capture from multiple cameras. The library supports a range of high-resolution digital surveillance cameras and other cameras that provide DirectShow interface for Windows platform, or GStreamer interface for Linux platform.
The list of supported cameras is available below in the system requirements.
The Device Manager includes a plug-in framework that allows integrators to write plug-ins to support their cameras using the provided API. The SentiVeillance SDK documentation contains the detailed information and samples.
A video file can be also used as a data source for SentiVeillance. The input from the file is processed as coming from a virtual camera, thus the video is processed in real-time.
The Face modality from the SentiVeillance component performs real-time detection, biometric identification and tracking of all detected faces:
SentiVeillance algorithms allows to perform these advanced actions with the detected faces:
VH modality can be used together with the Face modality to perform further tracking of the identified persons even when their faces no longer visible, as well as associating previous tracking history with the recognized person. Also, the VH modality can be used to obtain additional data about person's clothes.
Programming samples from the SDK show how to use this modality.
A license for using the SentiVeillance Face modality on one video stream is included with SentiVeillance 8.0 SDK. Additional licenses or upgrade for the number of processed video streams in the existing licenses can be purchased any time by SentiVeillance 8.0 SDK customers.
Basic Usage Recommendations and Constraints
Face recognition accuracy of SentiVeillance heavily depends on the quality of a face image in a frame. There are some basic recommendations and constraints when using face recognition applications based on SentiVeillance SDK.
- Several images during enrollment are recommended for better facial template quality which results in improvement of recognition accuracy and reliability.
- Additional enrollments may be needed when facial hair style changes, especially when beard or mustache is grown or shaved off.
- head roll (tilt) – ±15 degrees;
-
head pitch (nod) – ±15 degrees from frontal position.
- The head pitch tolerance can be increased up to ±25 degrees if several views of the same face that covered different pitch angles were used during enrollment.
-
head yaw (bobble) – ±90 degrees from frontal position (default value).
- Several views of the same face can be enrolled to the database to cover the whole ±90 degrees yaw range from frontal position.
Technical Specification
Below are provided specifications for SentiVeillance facial recognition modality.
4 % of the frame's larger side (at least 32 pixels) is the minimal recommended distance between eyes for a face on video stream or image to perform reliable face tracking and template extraction. The speeds of face tracking, template extraction and matching against a watchlist database are dependent on actual size of a face in a frame, not on the size of the whole frame.
The performance specifications are provided for Intel Core i7-4771 processor, running at 3.5 GHz clock rate, and 1920 x 1080 pixels videos.
| SentiVeillance 8.0 biometric face recognition algorithm technical specifications | |
|---|---|
| Frame rate when tracking up to 5 faces | More than 25 frames per second |
| Face watch-list database matching time (1) | Less than 0.5 second |
| Single face record size in a template (kilobytes) | 0.5 or 5.0 (configurable) |
| Maximum face watch-list database size | Limited by amount of free RAM |
(1) up to 20,000 face records in the database; larger database yelds slower response time. Note that each person may be represented by several records in the database with different appearance variations, different capture angles etc.
The Vehicle-Human (VH) modality from the SentiVeillance component performs real-time vehicle or human detection, classification and movement tracking. SentiVeillance algorithms allows to perform these actions with the detected objects:
In addition, data about the detected objects can be used in combination with other SentiVeillance modalities:
Programming samples from the SDK show how to use this modality.
A license for using the SentiVeillance VH modality on one video stream is included with SentiVeillance 8.0 SDK. Additional licenses or upgrade for the number of processed video streams in the existing licenses can be purchased any time by SentiVeillance 8.0 SDK customers.
The ALPR modality from the SentiVeillance component performs real-time vehicle license plate (number plate) detection and recognition. The deep neural networks based algorithms feature:
VH modality can be used together with the ALPR modality, especially in these cases:
Programming samples from the SDK show how to use this modality.
A license for using the SentiVeillance ALPR modality on one video stream is included with SentiVeillance 8.0 SDK. Additional licenses or upgrade for the number of processed video streams in the existing licenses can be purchased any time by SentiVeillance 8.0 SDK customers.
Reliability Tests
We present the testing results to show SentiVeillance 8.0 ALPR algorithm reliability on the car images taken in different conditions. Images from the platesmania.com website were used for testing. The images were grouped into three groups:
<100 pixels
100-200 pixels
>200 pixels
Two experiments were performed with images from each group:
- Experiment 1 – the OCR was set to interpret 1 (one) as I (letter I) and 0 (zero) as O (letter O).
- Experiment 2 – the OCR had no specific rules on letters and numbers interpreteation.
| SentiVeillance 8.0 ALPR algorithm reliability testing | |||||
|---|---|---|---|---|---|
| < 100 pixels | 100-200 pixels | > 200 pixels | |||
| Image count | 584 | 3351 | 4203 | ||
| License plate width (pixels) | less than 100 | 100-200 | more than 200 | ||
| Experiment 1 recognition ratio | 88.7 % | 96.1 % | 99.1 % | ||
| Experiment 2 recognition ratio | 77.9 % | 86.6 % | 91.2 % | ||
The requirements below are provided for a single computational node (PC or server) used to run the SentiVeillance algorithms. Multiple nodes can be connected together via network to support larger surveillance systems.
- 3 GHz or better processor with 6 cores is recommended.
- AVX2 support is highly recommended. Processors that do not support AVX2 will still run the SentiVeillance algorithms, but in a mode, which will not provide the specified performance. Most modern processors support this instruction set, but please check if a particular processor model supports it.
- a GPU is required if a SentiVeillance component with any of the supported modalities is run to process more than 2 video streams simultaneously.
- NVIDIA GeForce GTX 1080 GPU or better is recommended to process up to 10 video streams on a node.
- Several GPUs can be used on the same machine to process larger number of video streams.
- at least 6 GB of VRAM is recommended.
- Compute Capability 3.5 or better should be supported by the GPU.
- CUDA 11.2 toolkit or newer is required
- cuDNN 7.5 library is required.
- Microsoft Windows 7 / 8 / 10 / Server 2008 / Server 2008 R2 / Server 2012, 64-bit.
- Microsoft .NET framework 4.5 or newer (for .NET components usage).
- Microsoft DirectX 9.0 or later.
-
One of following development environments for application development:
- Microsoft Visual Studio 2012 or newer (for application development under C/C++, C#, Visual Basic .Net)
- Java SE JDK 8 or newer
- Ubuntu 20.04 OS
- glibc 2.17 or newer
- GStreamer 1.10.x or newer with gst-vaapi plugins installed for hardware accelerated video decoding
- libgudev-1.0 219 or newer
- wxWidgets 3.0.0 or newer libs and dev packages (to build and run SDK samples and applications based on them)
- Java SE JDK 8 or newer (for application development with Java)
-
Any IP camera, that supports RTSP (Real Time
Streaming Protocol):
- Only RTP over UDP is supported.
- VLC framework can be optionally used for reading video streams.
- H.264/MPEG-4 AVC or Motion JPEG should be used for encoding the video stream.
-
Any high-resolution digital camera that is accessible using:
- DirectShow or Windows Media or Media Foundation interfaces for Microsoft Windows platform;
- GStreamer interface for Linux platform.
- Mobotix M16 Thermal camera is supported for systems that need thermal imaging. SentiVeillance SDK includes a specific programming sample for processing video streams from this camera.
- Any other device support can be added by customers using the provided Device Manager Plug-in Framework. Please refer to the SentiVeillance 8.0 SDK documentation for the detailed information.
Below are provided specifications for SentiVeillance facial recognition modality.
4 % of the frame's larger side (at least 32 pixels) is the minimal recommended distance between eyes for a face on video stream or image to perform reliable face tracking and template extraction. The speeds of face tracking, template extraction and matching against a watchlist database are dependent on actual size of a face in a frame, not on the size of the whole frame.
- head roll (tilt) – ±15 degrees from frontal position;
- head pitch (nod) – ±15 degrees from frontal position;
- head yaw (bobble) – ±45 degrees from frontal position.
The performance specifications are provided for Intel Core i7-4771 processor, running at 3.5 GHz clock rate, and 1920 x 1080 pixels videos.
| SentiVeillance 8.0 biometric face recognition algorithm technical specifications | |
|---|---|
| Frame rate when tracking up to 5 faces | More than 25 frames per second |
| Face watch-list database matching time (1) | Less than 0.5 second |
| Single face record size in a template (kilobytes) | 0.5 or 5.0 (configurable) |
| Maximum face watch-list database size | Limited by amount of free RAM |
(1) up to 20,000 face records in the database; larger database yelds slower response time. Note that each person may be represented by several records in the database with different appearance variations, different capture angles etc.
Product Development
- A license to use SentiVeillance component with Face modality on one video stream.
- A license to use SentiVeillance component with ALPR modality on one video stream.
- A license to use SentiVeillance component with Vehicle/Human modality on one video stream.
Each license has a specified number of processed video streams.
Integrators can obtain additional licenses or upgrade the number of processed video streams in their existing licenses if processing videos from more cameras or on more computers required for the development process.
License Activation Options
The components are copy-protected. The following license activation options are available:
- Activating single computer licenses – An installation license for a SentiVeillance component will be activated for use on a particular computer. The number of available licenses in the license manager will be decreased by the number of activated licenses.
- Managing single computer licenses via a LAN or the Internet – The license manager allows the management of installation licenses for SentiVeillance components across multiple computers or mobile/embedded devices in a LAN or over the Internet. The number of managed licenses is limited by the number of licenses in the license manager. No license activation is required and the license quantity is not decreased. Once issued, the license is assigned to a specific computer or device on the network.
- Using license manager as a dongle – A volume license manager containing at least one license for a SentiVeillance component may be used as a dongle, allowing the SentiVeillance component to run on the particular computer where the dongle is attached.
Additional SentiVeillance component licenses for the license manager may be purchased at any time.
Licenses Validity
All SDK and component licenses are perpetual and do not have expiration. There are no annual fee or any other fees except license purchasing fee. It is possible to move licenses from one computer or device to another. Neurotechnology provides a way to renew the license if the computer undergoes changes due to technical maintenance.