SentiSight is intended for developers who want to use computer vision-based object recognition in their applications. Through manual or fully automatic object learning it enables searching for learned objects in images from almost any camera, webcam, still picture or live video in an easy, yet versatile, way.
Available as a software development kit that provides for the development of object recognition systems for Microsoft Windows or Linux platforms.

The SentiSight 3.4 technology is able to perform fully automatic or manual object learning and has these capabilities for advanced visual-based object learning and recognition:

• Accurate Object Detection

The SentiSight algorithm is able to find out:

• whether a particular object is present in a scene;
• where the object is located within the scene;
• how many instances of the object occur in the scene.
• Two Algorithms for Object Recognition

Depending on the object type, one of these algorithms (or both) may be used for successful recognition:

• The blob-based algorithm uses small details of an object as distinctive features that are extracted into an object model and are used later to recognize the object. This algorithm offers performance but is not suitable for solid-colored, reflecting or transparent (glass, etc.) objects.
• Shape recognition

The shape-based algorithm is useful for objects which do not have any distinctive details but have stable external edges (boundaries) and/or internal edges. This algorithm performs at slower speeds but allows for the recognition of most objects not identified by the blob-based algorithm.

• Color Usage Mode

The blob-based and shape-based algorithms may be configured to detect object colors and use this information for improving recognition accuracy. This mode enables SentiSight-based applications to distinguish similar objects that only differ in color.

• Object’s Image Quality Determination

A quality threshold can be used during object learning to ensure that only the best quality object model will be stored into database.

• Simultaneous Multiple Object Recognition

The SentiSight Embedded is able to detect and recognize several 2D and 3D objects simultaneously.

• Object Evaluation

The algorithm makes estimates based on the region an object occupies in a scene, providing additional information about the size, orientation and scale of the recognized object.

• Fast image processing

SentiSight processes video streams in real time, making it useful for real-time applications. The algorithm is able to run several threads on multi-core processors making the recognition several times faster.

• Objects tracking mode

The SentiSight 3.4 library has a multiple-objects tracking mode for tasks that require very fast image processing during the object recognition stage. The tracking works with complex backgrounds and fast-moving objects. Tracking is initialized when an object is recognized and located, and then tracks the object until it changes somewhat in appearance, at which point tracking is reinitialized by recognition. In tracking-mode SentiSight is able to process more that 100 frames per second (320 x 240 pixels, single object in a frame).

• Reliable, innovative algorithm that is tolerant of variation in appearance, object scale, rotation and pose.
• Accurate detection, processing and tracking of objects in real-time.
• Webcams or other low cost cameras are suitable for obtaining object images.
• Available as multiplatform SDK that supports multiple programming languages.
• Reasonable prices, flexible licensing and free customer support.

SentiSight has two operation modes: learning and recognition. In learning mode, the SentiSight algorithm creates an object model by extracting object features from an image or video. In recognition mode, SentiSight finds and tracks objects with features matching those previously stored in object models.

Object Learning Process

In order to recognize an object in an image, the appearance of an object must first be catalogued. In the learning phase, SentiSight algorithms extract specific object features from a video stream or single image and save them into an object model.In many cases there is information in a video or single image beyond the features of an object you wish SentiSight to learn: a background, other objects in the room, a hand holding the target object. For SentiSight to properly learn an object, information about the exact location of the object within the image should be provided.SentiSight supports 2 methods of object learning: manual and automatic.Manual object learning is suitable for most situations. A user performs the following steps for manual object learning in the SentiSight-based application:

1. Outline an object’s shape on an image by marking the object’s corner points to build a polygon. The image can be provided from an image file, video file or live video stream.
2. Select the algorithm to use: blob-based, shape-based, or both.
3. As an option, additional images of the object may be provided, repeating Step 1 for each image. The algorithm assists the user by estimating an approximate shape for the object if the image is recognized by way of previously catalogued images. Learning the object from different sides and angles results in better recognition quality.
4. Input the learned object name (ID) into the system.

Automatic Object Learning is suitable for lightweight, movable objects. This learning procedure is based on detecting an object through the exclusion of a static background and the object’s holder (usually a hand). A fixed camera is highly recommended for this process.

A user performs the following steps for automatic object learning in the SentiSight-based application:

1. Select a background and position the camera.
2. Select a holder – an object that will be used to hold and move the object to be learned. A user’s hand can be the “holder”.
3. The “holder,” if it is not a rigid onject, should be presented to the camera in various poses and configurations so SentiSight can learn it.
4. Select the algorithm to use: blob-based, shape-based, or both.
5. After the holder has been learned, SentiSight is ready to learn the object itself. Use the holder to rotate and move the object, both closer to, and further from, the camera.
6. Input the learned object name (ID) into the system.

The automatic method requires the use of live video or separate video / image sets of background, holder and object. Other background elements may be learned together with the object if the object is hardly separable from the background. Too little disparity between object and background, if not learned together, may affect the ability of the algorithm to recognize an object’s unique qualities, possibly resulting in the object being misclassified along with other objects having the same background.

Manual object learning should be used for objects that cannot be moved or if there is no way to provide separate media of an object’s background and/or holder. Automatic learning requires less user interaction with the system, but it is not as precise as manual learning. Manual learning is suitable, generally, for a wider range of cases.

Object Recognition Process

Object recognition requires no user interaction apart from providing a video file containing the object or pointing a camera at a scene wherein the learned object is present or will appear. When an object appears in the vision field, SentiSight works at recognizing it. If the object is identified by SentiSight, the object’s name (ID) and coordinates are returned.
The SentiSight algorithm creates a model in the object learning stage, using views of the object from different sides, different 3D poses and in variable lighting conditions, improving the system’s overall recognition capability.

• Recognition of documents, stamps, labels, packaging and other items for sorting, logo masking, usage monitoring and similar applications
• Object counting and inspection for assembly lines and other industrial applications
• Augmented and extended reality applications for toys, games, devices and Web applications such as: smart toys for children that recognize cards, images, pictograms, etc.; recognition of places based on photographs; recognition of products such as beverages, foods and other consumer goods.
• Robotic vision for navigation and manipulation
• Law enforcement applications for identification, such as tattoo recognition

SentiSight 3.4 SDK is intended for developers who want to use computer vision-based object recognition in their applications. The SDK allows rapid development of computer vision-based object recognition systems using functions from the SentiSight library for Microsoft Windows or Linuxplatforms. Developers have complete control over SDK data input and output; therefore SDK functions can be used in connection with most cameras (including webcams), with any database and with any user interface.

SentiSight 3.4 SDK distribution package contains:

Components	Microsoft Windows	Linux
• SentiSight 3.4 installation license	1 single computer license
• Device manager library	+	+
Programming samples
• C++	+	+
• C#	+
• Visual Basic .NET	+
Programming tutorials
• C/C++	+	+
• C#	+
• Visual Basic .NET	+
• Java	+	+
Documentation
• SentiSight 3.4 SDK documentation	+

• PC with x86 (32-bit) or x86-64 (64-bit) processor:
  • SSE2 support is required. Processors that do not support SSE2 cannot run the SentiSight 3.4 algorithm. Please check if a particular processor model supports SSE2 instruction set.
  • SSSE3 support is recommended, as the SentiSight 3.4 algorithm provides higher performance using this instruction set. Please check if a particular processor model supports SSSE3.
  • 64-bit architecture allows to work with larger images, larger model databases and also increases general SentiSight 3.4 algorithm performance due to usage of 64-bit CPU registers.
• At least 256 MB of free RAM should be available for the SentiSight-based application. Additional RAM may be required for:
  • Applications that need to recognize objects using large database as the whole database must be loaded into RAM before recognition. The database size depends on objects quantity and number of templates saved in each object model. Each object model may be rather large due to using long videos for learning (as template learned from each frame is saved separately) and/or using multiple views for each object.
    For example, a database of 100 object models with 36 templates per model will require about 25 MB of RAM when blob recognition algorithm is used, or about 50 MB when shape recognition algorithm is used (for 320 x 240 pixels resolution).
  • Applications that need to work with high resolution videos. Higher resolution allows to extract more features from objects, thus the object model sizes will be bigger. In general the template size has about linear dependence from the image or video resolution.
    For example: a frame from 1 MegaPixel camera is about 13 times larger than a frame from a regular 320 x 240 pixels (0.08 MegaPixel) camera. If an object model learned from a sequence of 320 x 240 frames occupies 250 KB of memory, then a model learned from the same sequence of frames obtained from 1 MegaPixel camera will require about 3.4 MB of memory. Note that each video frame is processed separately and the obtained object template is saved into the model.
• Optional camera or webcam. These cameras are supported by SentiSight:
  • Any webcam or camera that is accessible using:
    • DirectShow interface for Microsoft Windows platform.
    • GStreamer interface for Linux platform.
  • Any IP camera, that supports RTSP(Real Time Streaming Protocol):
    • Only RTP over UDP is supported.
    • 264/MPEG-4 AVCor Motion JPEG should be used for encoding the video stream.

Note: Please see the supported hardware section for the camera models.

• Microsoft Windows specific requirements:
  • Microsoft Windows XP / Vista / 7 / 8 / 10 / Server 2003 / Server 2008 / Server 2008 R2 / Server 2012, 32-bit or 64-bit.
  • Microsoft .NET framework 3.5 or newer (for .NET components usage).
  • Microsoft DirectX 9.0 or later (for camera/webcam usage).
  • One of following development environments for application development:
    • Microsoft Visual Studio 2008 SP1 or newer (for application development under C/C++, C#, Visual Basic .Net)
    • Sun Java 1.6 SDK or later
  • Linux specific requirements:
    • Linux 2.6 or newer kernel (32-bit or 64-bit) is required. Linux 3.0 or newer kernel is recommended.
    • glibc 2.11.3 or newer
    • GStreamer 1.2.2 or newer with gst-plugin-base and gst-plugin-good (for face capture using camera/webcam or rtsp video)
    • libgudev-1.0 164-3 or newer (for camera usage)
    • GTK+ 2.10.x or newer libs and dev packages (to run SDK samples and applications based on them)
    • GCC-4.0.x or newer (for application development)
    • GNU Make 3.81 or newer (for application development)
    • Sun Java 1.6 SDK or later (for application development with Java)

• These models of still cameras are supported:
  • Canon EOS family still cameras (Microsoft Windows only)
  • Nikon DSLR still cameras (Microsoft Windows only; a specific camera model should support video capture and should be listed there)
• These specific models of high-resolution IP cameras are supported:
  • Axis M1114 camera (Microsoft Windows and Linux)
  • Basler BIP2-1600-25c-DN IP camera (Microsoft Windows and Linux)
  • Cisco 4500 IP camera (Microsoft Windows only)
  • CMITech EMX-30– face & iris camera (Microsoft Windows only)
  • IrisGuard IG-AD100– face & iris camera (Microsoft Windows only)
  • Mobotix S14D and DualNight M12 IP cameras (Microsoft Windows and Linux)
  • PiXORD N606 camera (Microsoft Windows and Linux)
  • Prosilica GigE Vision camera (Microsoft Windows and Linux)
  • Sony SNC-CS50 camera (Microsoft Windows and Linux)
  • VistaFA2/ VistaFA2E / VistaEY2 face & iris cameras (Microsoft Windows only)

All specifications are provided for an Intel Core i7-2600 processor running at 3.4 GHz.
These specifications are for SentiSight 3.4 blob-recognition and shape-recognition algorithms. These algorithms may be used together or separately, depending on object type.
The specifications are provided for 320 x 240 pixel images. These image area performance dependencies are valid for the same images with different resolutions:

• The blob-based algorithm has linear dependencefor object learning and linearithmic (n log n) dependence for object recognition.
• The shape-based algorithm has linearithmic (n log n) dependencefor object learning and quadratic dependence for object recognition.
• The color usagemode has linear dependence for both object learning and recognition.

The object-model size depends on how feature-rich an object is, thus varying for each object.

These conditions may alter the performance of the algorithms:

• Rotation and translation.The algorithm is generally rotation- and translation-invariant in a plane perpendicular to the camera. The algorithm is also invariant for rotations of up to 10-15 degrees beyond a plane perpendicular to the camera. Different views of an object may be added to a model to handle larger rotations.
• Resolution and scale changes.Scale (size of the object within the image) difference between an object’s model and the object itself can be up to 2-3 times. Objects should contain enough detail, and be large enough to be recognizable.
• The algorithm is robust to occlusions as big as 50 % of the object if enough unique edges remain visible.
• Lighting conditions(illumination, shadows and reflectance).
  • Planar objects will have issues with reflectance.
  • 3D objects will have issues with variable lighting conditions – consistent lighting greatly reduces potential problems.
• In general, transparent objects may be difficult to recognize.
• The algorithm recognizes only rigid/stable objects. At minimum, a significant piece of the object should be unchanging.

Object recognition algorithms may be configured to run on more than one thread of multi-core processors, allowing for an increase in object-model matching speed. The table below provides object recognition speeds as a range; the smaller number represents recognition speed using 1 thread, while the larger number represents recognition speed using 8 threads. Note that the specified processor model has 4 cores and executes 2 threads per processor core in parallel.

SentiSight 3.4 blob-based object recognition algorithm – technical specifications
	Without color usage mode	With color usage mode
Static Background Extraction/ Object mask separation	30 frames per second
Learning: Processing of single object’s frame	0.014 seconds	0.017 seconds
Learning: Generalization time (for 100 frames of object)	0.15 seconds
Recognition speed (1)	160,000 – 290,000 models per second	90,000 – 140,000 models per second

 

SentiSight 3.4 shape based object recognition algorithm – technical specifications
Static Background Extraction/ Object mask separation	30 frames per second
Learning: Processing of single object’s frame	0.215 seconds
Learning: Generalization time (for 100 frames of object)	Not applicable
Recognition speed (1)	3,500 – 8,000 models per second

When object model contains one template. The object model may contain multiple templates (usually corresponding with different viewpoints). In that case the algorithm will compare an object against all templates in the model before returning the recognition result. Also, this recognition speed is reached with sufficiently large databases (2,000 images or more); with smaller databases the recognition is slower.

The following licensing model is intended for end-user product developers. Integrators who want to develop and sell a SentiSight-based development tool (with API, programming possibilities, programming samples, etc.), must obtain permission from Neurotechnology and sign a special VAR agreement.10.1 Product Development

An integrator should obtain a SentiSight 3.4 SDK (EUR 339) to develop a product based on SentiSight technology. The SDK needs to be purchased just once and may be used by all the developers within the integrator’s company.
SentiSight 3.4 SDK includes SentiSight component. A license for an individual SentiSight component is required for each CPU that runs the component (a processor can have any number of cores).
One single computer license for the SentiSight component is included with SentiSight 3.4 SDK.
Components are copy-protected – a license is required for a component to run. License activation options are listed below on this page.
Additional component licenses may be obtained by SentiSight SDK customers as required by their development process.

Product Development

An integrator should obtain a SentiSight 3.4 SDK (EUR 339) to develop a product based on SentiSight technology. The SDK needs to be purchased just once and may be used by all the developers within the integrator’s company.
SentiSight 3.4 SDK includes SentiSight component. A license for an individual SentiSight component is required for each CPU that runs the component (a processor can have any number of cores).
One single computer license for the SentiSight component is included with SentiSight 3.4 SDK.
Components are copy-protected – A license is required for a component to run. License activation options are listed below on this page.
Additional component licenses may be obtained by SentiSight SDK customers as required by their development process.

Product Deployment

To deploy a product developed with SentiSight SDK, an integrator need obtain only the additional licenses required for the SentiSight components that will run on each CPU of their customer’s computers. The available license types for product deployment are the same as for product development.
Each SentiSight component running on a computer belonging to the integrator’s customer requires a license. License activation options are listed below on this page.
Prices for SentiSight 3.4 SDK and additional SentiSight component licenses can be found here.

Licensing Agreement

The Licensing Agreement contains all licensing terms and conditions.
Note that you unambiguously accept this agreement by placing an order using Neurotechnology online ordering service or by email or other means of communications. Please read the agreement before making an order.

Single Computer Licenses

A single computer license allows the installation and running of a SentiSight component installation on one CPU (a processor can have any number of cores). Neurotechnology provides a way to renew the license if the computer undergoes changes due to technical maintenance.
Each single computer license requires activation for a SentiSight component to run. The available activation options are listed below on this page.
Additional single computer licenses for SentiSight components may be obtained at any time by SentiSight SDK customers.

License Activation Options

Single computer licenses are supplied in three ways:

• Serial numbers are used to activate licenses for SentiSight components. The activation is done via the Internet or by email. After activation the network connection is not required for single computer license usage.
  Note: activation by serial number is not suitable for virtual environments.
• Internet activation. A special license file is stored on a computer; the license file allows to run SentiSight components on that computer after checking the license over the Internet.Internet connection should be available periodically for a short amount of time. A single computer license can be transferred to another computer by moving the license file there and waiting until the previous activation expires.
  Licenses may be stored in a volume license manager dongle. License activation using volume license manager may be performed without connection to the Internet and is suitable for virtual environments.

Volume License Manager

Volume license manager is used on site by integrators or end users to manage licenses for SentiSight components. It consists of license management software and a dongle, used to store the purchased licenses. An integrator or an end-user may use the volume license manager in the following ways:

• Activating single computer licenses – An installation license for a SentiSight component will be activated for use on a particular computer. The number of available licenses in the license manager will be decreased by the number of activated licenses.
• Managing single computer via a LAN or the Internet – The license manager allows the management of installation licenses for SentiSight components across multiple computers in a LAN or over the Internet. The number of managed licenses is limited by the number of licenses in the license manager. No license activation is required and the license quantity is not decreased. Once issued, the license is assigned to a specific computer on the network.
• Using license manager as a dongle – A volume license manager containing at least one license for a SentiSight component may be used as a dongle, allowing the SentiSight component to run on the particular computer where the dongle is attached.
  Additional SentiSight component licenses for the license manager may be purchased at any time. Neurotechnology will generate an update code and send it to you. Simply enter the code into the license manager to add the purchased licenses.

Enterprise License

The SentiSight enterprise license allows an unlimited use of SentiSight components in end-user products for a specific territory, market segment or project. Specific restrictions would be included in the licensing agreement.
The enterprise license price depends on the application size and the number of potential users of the application within the designated territory, market segment or project.

SentiSight Embedded SDK – A product for object recognition on Android smartphones, tablets and other mobile devices. Produces object models that are the same as those in the SentiSight SDK, thus can be also used for developing client-side mobile applications for systems with a server-side based on the SentiSight SDK.
SentiSculpt SDK – For development of PC-based system which perform 3D object models creation from regular images.
NVeiler Video Filter – A plug-in for the VirtualDub video processing application that is able to detect and hide faces in videos; the SentiSight algorithm is used for the face and object-tracking feature of the plug-in.
NPointer –A freeware application for gesture-based computer control that captures hand movements with a webcam. Can also accept voice commands.