Scene Me

Video analytics often mean acting on some pre-defined event producing analytics.  What if you can get hold of the underlaying analytics to process and create your own specific events, logs or visualization?

Scene Me replaces my previous ACAP MQTT Me and Radar Me (they are no longer maintained).  Scene Me is an extension of MQTT Me with the additional support of publishing analytics data. The analytics in this case are trackers, objects and scenes.   Trackers is the data of moving objects within the camera view.  Tracker data is published continuously while the video object moves across the scene.  Trackers can be used for real-time event filters detecting specific behavior or visualization/augmentation.  When the tracker loses the video object an “object” is published.  An object is a data structure that includes place of birth/death and time of birth/death.   While a tracker targets real-time processing, objects target post-processing applications that have no need to process trackers.  The third type, “scene” is a data structured that is published when a tracker fulfils a specific behavior filter during the time a tracker is active.  A scene is a filter of place-of-birth, minimum size, minimum age and minimum vertical and/or horizontal movement.  Scenes are “events”, similar to “motion detection” but having multiple properties, making it possible to detect specific behavior.  Scenes are also automatically published on the ONVIF event stream.

What about Integration?

MQTT is a publish/subscribe protocol where data is published with a topic and a payload.  MQTT clients do not connect directly to the device, they connects to a MQTT broker.  Multiple clients can subscribe to teh same topic can also publish messages.  The broker becomes the system data bus.  A client may say to the broker “send me all the messages with the topic “tracker/#”.  This will give the client all the trackers from all the cameras in a single subscription.  It does not need to connect to each end every camera.  It is a single-point-of-integration and at the same time de-coupling the publisher from the consumer.  The subscriber does not need to not have direct access to the device producing the data.

This integration requires a MQTT broker.  Installing and operating a broker is easy.  If you do not have any possibilities hosting a private mosquitto broker, you can use a public  enterprise MQTT broker service on Internet (CloudMQTT, HIVEMQ or other).

Does this integrate with my VMS?

To my knowledge there are no VMS that have a built-in MQTT client. I believe it is just a matter of time.  However, there are work-arounds to integrate MQTT messaging into a VMS depending on use-case.

Scene Me supports “scenes”.  A defined scene in Scene Me will be automatically sent as an “event” on the ONVIF event stream that most VMS support.  For this case you do not need an MQTT broker to make the integration.

Scene Me can subscribe to a user defined MQTT topic and generate an ONVIF event that a VMS can monitor.  This makes it possible for a 3:rd party MQTT client ( e.g. an Android/IOS  MQTT Dashboard) to trigger actions in the VMS, using a camera with Scene Me as a bridge/proxy.

Node-Red, an integrators dream

A VMS is a Video Management System.  Counting objects, visual augementation, controlling lights, sending alerts, optimizing building/business automation systems may not require a VMS.  What is needed is a Data Management Service that can integrate and process data from and to a VMS, other devices or other systemes.  As long as devices and systems have  some public API you can integrate.

Out of all the different ways of doing integration, including C, PHP, Python and more, I have found Node-Red to be the easiest and most dynamic tool out there.  Originally developed by IBM’s Emerging Technology Services, this web-based flow editor makes it easy to wire togheter inputs and outputs including MQTT, HTTP, TCP, DataBase, files, timers, email and more.  Knowing some basic Node/JavaScript is needed for complex logic but the vast amount of community flows and nodes allows you to do a lot without any programming skills. You can also add a user dashboard where users can view data/graphs or initiate actions to the systems. Node-Red can be hosted on a Rasberry Pi, Windows server, Linux Server or VM.  It is free of charge.  Download and play!  My own system runs a Node-Red server and a Mosquitto broker on the same machine that hosts the VMS.  Adding a Mongo DataBase completes the system.  My Node-Red integrates to my VMS, my home automation system, my SONOS (voice notifications) and my mobile phones (GPS tracking, Zones and notifications).

As an example, the following screenshot shows how Node-Red sends a Scene Me Video analytics object to a database, email, twitter, file and http at the same time.

References/Links

Documentation

The user documentation [PDF] describes the overview of Scene Me.

License

As always, this is a freeware ACAP service but license registration.  License registrations requires an “Axis MyAccount”.  Go to Axis Portal for ACAP license registation, register the serial number of your camera and select “Free application” and search for “Scene Me”.  Download and install the license key file in you Axis Device.

Supported Axis Devices

All Axis devices based on MIPS or ARM7hf with firmware 6.50 or later.  In order to utilize video analystics, the firmware must be 9.20 or later.

Note:  If you have previously installed MQTT Me, remove that ACAP before starting Scene Me.  You may also need to adjust your integration as some topics have changed.

Download Scene Me

Release notes [TXT]

 

Advertisements

Radar Me (MQTT)

Radar Me is an MQTT client for Axis Radar D2050.  It publish radar detection data to an MQTT broker to simplify integration for systems processing radar data.  Writing specific rules in e.g. Node-Red is very easy.

This integration also decouples the data consumer from the device, allowing 3:rd party consumers without the risk of compromising the device.
mqtt

The tab “Radar” provides a way validate published objects.  Leaving the page running for a while will give a heatmap of detected objects.  The web page uses a JavaScript MQTT client that can only connect to an MQTT broker using Secure WebSocket (WSS).  You need to set your broker WSS port on the page.  If your broker does not support Secure WebSocket you can use a MQTT snooper like MQTT.fx to monitor published objects.

It is also possible to filter unnecessary detections (reflections and swaying objects) to improve the system performance (bandwidth, broker and consumer).

radar

MQTT Topic:

object/<clientid>

MQTT Payload:

{
“id”: 495555,
“timestamp”: 1561666067258,
“birth”: 1561666065458,
“speed”: 0.6,
“direction”: 29.9,
“distance”: 28.2,
“angle”: 49.1,
“class”: 2,
“x”: 6273,
“y”: 5278,
“sx”: 6173,
“sy”: 5441,
“px”: 6269,
“py”: 5290,
“active”: true
}

Id:  Unique object tracking ID
Timestamp: Detection time (standard JavaScript millisecond timestamp)
Birth:  Timestamp when object was first detected.  Age = Timestamp – Birth
Speed: Object speed in meter/second
Direction:  Object direction angle in degrees -180 to 180 where 0 is direction of the radar
Distance:  Distance in meter between object and radar
Angle: The location angle of the object related to the radar direction (-180 to 180)
Class:  Object classification. 2 = Undefined, 3 = Human, 4 = Vehicle.  Note that Classification is a bit unpredictable.
x/y:  The graphical coordinates for plotting on a screen.  The coordinate system is 0-8192,0-8192 where 0,0 is the top left corner.  This simplifies plotting data on e.g. a HTML canvas.
sx/sy:  The graphical coordinates of the first detection.  Can be used to calculate the graphical travelled distance.  DeltaX = x – sx, DeltaY  = y – sy
px/py:  The coordinates of the previous published x,y.  This simplifies plotting a line-path.
Active: True while object is being tracked.  False when the object disappears, and that tracking ID will not be published any more.

Considerations:

  • You need to have an MQTT broker.  Install a local mosquito broker on Linux or Windows.  Or, try a free cloud-based broker such as CloudMQTT (Cute Cat plan with up to 5 client connections and maximum bitrate of 10/Kbps).
  • Radar Me only supports standard MQTT 3.1 (TCP port 1883) connection, not TLS.  This means that the payload will be sent in clear text.

Download

MQTT Me Cookbook

After playing around with my MQTT Me and testing various use cases I realized that I needed to extend MQTT Me with additional capabilities.  If you have installed version 1.x you need to uninstall it before installing version 2.0.  Sorry about this.  The release notes are included in the download package.

I have created a cookbok that describes how MQTT Me can be configured and used.

  1. Publishing text to a camera Text overlay
  2. Mobile Voice Notification on Camera Motion Detection
  3. Monitoring Axis Device Alive (or not)
  4. Monitoring Device CPU load and Bandwidth
  5. Publishing JPEG images and storing them with PHP
  6. Setting Device Parameter Settings with MQTT message

Documentation:

 

Download MQTT ME

MQTT integration with Axis devices

MQTT Me gives your Axis Device the ability to integrate with MQTT systems.  MQTT (Message Queuing Telemetry Transport) is an ISO standard machine-machine messaging protocol targeting IoT devices.

There are several applications/clients supporting MQTT such as Home Assistant (home automation) and mobile apps such as Zanzito and MQTT Dashboard.  There are also more professional systems that use MQTT extensively.

MQTT Me supports both publishing (upon Axis device events) and subscribe to topics to execute actions on the Axis device.

One of my usage is to get a speach notification in my phone when someone triggers motion detection in any of my cameras.  I have Zanzito on my phone that subscribes to a topic published by my cameras.  The MQTT payload from my cameras includes a string that Zanzito will do TTS (Text-To-Speach) to play on my phones speaker.  I will hear the message even if the phone is idle in my pocket.

Onther usage is having MQTT Dashboard on my phone.  The app is configured with a button that toggles a digital IO port on the Axis device.

Compared to integrating with VAPIX/ONFIV, MQTT enables a for more flexible and simplified integration for event/action solutions.  The Axis device does not need to have network access to the MQTT application as long as the device and client have access to the same MQTT broker.  You can close all the incoming ports on your firewall.  MQTT clients conntect to broker from the inside.

Since MQTT Me does not support TLS (encryption) you should not connect to a broker on the Internet.  A  local broker is safe OK.   Remote clients can always connect to the broker over TLS if you configured your broker with plain text (port 1883) and  TLS (port 8883) for the clients accessing from Internet e.g. mobil devices.

Read more in the MQTT Me manual [PDF]

Download the ZIP file containing ACAP (MIPS, ARM7 and ARM7hf)

Current version is 2.0.   Note: Version 2.0 is not compatible with version 1.X.  You need to uninstall 1.x before installing version 2.0.

Please, share any interesting MQTT integration solutions by using the comments.