MongoDB Logging for Fawkes and ROS

Autonomous mobile robots produce an astonishing amount of run-time data during their operation. Data is acquired from sensors and actuator feedback, processed to extract information, and further refined as the basis for decision making or parameter estimation. In today's robot systems, this data is typically volatile. It is generated, used, and disposed right away. However, some of this data might be useful later, for example to analyze faults or evaluate the robot's performance. A system is required to store this data as well as enable efficient and flexible querying mechanisms.

On this page we link to the relevant pieces to get started with MongoDB logging on either Fawkes or ROS. Once the data has been recorded, it can be used independently of a particular robot middleware. While the data recorded follows the format defined by the blackboard interface or topic message types respectively, access and processing of the data can be done solely relying on database features.

This page presents joint work by the Personal Robotics Lab at The Robotics Institute of the Carnegie Mellon University and the Knowledge-based Systems Group of the Computer Science Department of the RWTH Aachen University.

Publication

In the paper "A Generic Robot Database and its Application in Fault Analysis and Performance Evaluation" we describe the concepts behind the database and the recording, why MongoDB is a good choice for generic robot communication logging, and present results from test runs on HERB 2.0 and in synthetic benchmarks.

The database has been used in experiments for fault analysis and performance evaluation. While these have been only simple applications, we are already working on more sophisticated versions.

There has been a talk at ROSCon 2013 which described the basics and two specific applications of data recording. The video is embedded below.

MongoDB Logging on Fawkes or ROS

Logging on Fawkes is done using the mongodb-log plugin. Once loaded, it records all data stored in the blackboard and all images and point clouds available via shared memory through the respective Fawkes thread aspects. No further configuration is necessary, but possible, e.g. to choose a database on a remote host for logging. Please check the mongolog plugin documentation for details on how to enable MongoDB logging.

On ROS logging to MongoDB is performed by the mongodb_log package. It has been used extensively on HERB 2.0. ROS has two properties which need to be taken into account when it comes to logging. First, it supports message introspection (detecting which fields are in a message at run-time for types not necessarily known at compile-time) in Python only, and not in C++. Second, message deserialization can be quite slow in Python, especially for lists of nested types. Some numbers for this are given in the roslua documentation. More info on how to deal with these constraints and how to get MongoDB logging to run smoothly on your system anyway are presented on the mongodb_log package wiki page.

Application in fault Analysis

In the paper we exemplify the database in the context of fault analysis, in particular for fault that are caused by data errors. First, the acquisition, processing, and aggregation of data is modeled in a hierarchy, which at run-time manifests in knowledge-trees build in a bottom-up fashion. Then, after a fault has occurred, this understanding of the data is used to guide a top-down search for the fault. The overall process is sketched in the schematic below. We would like to invite interested visitors to read more about the details in the paper.

Performance

The logging systems support automated generation of performance graphs like the ones shown below using RRDtool. These graphs are useful to inspect the performance of the logging and see if modifications and improvements have the desired effect. We are now going to give some examples and briefly explain what the graphs show. The first two graphs have been created from data taken during synthetic benchmark runs. They are used to explain some of the things to keep in mind and to show the high efficiency of database logging. The following graphs have been automatically created during real-world demo runs of HERB. When setting up MongoDB logging on your robot, these graphs will be available to analyze logging performance immediately.

RoboDB RRD graph showing CPU and memory usage during a synthetic benchmark. The rosbag CPU line (orange) is almost hidden behind the accumulated C++-Logger/MongoDB line (red), showing that both approaches cause a similarly low CPU load. The C++ logger averages at 10 MB memory usage and the Python logger at about 35 MB. The Python logger also requires an order of magnitude more CPU time, as seen by the purple lines. This is a problem of ROS message parsing in Python, which is computationally expensive. A solution could be to implement C++ ROS message introspection and a generic C++ logger. Since this already exists in Fawkes it does not suffer from this problem.

I/O writing bandwidth comparing rosbag and MongoDB ROS logger throughput. The graph shows the throughput overhead of MongoDB of about 30%. Note that we recorded two topics to MongoDB, but only one using rosbag. A considerable overhead is produced by rosbag storing the type definition with each message.


RoboDB RRD graph showing op counters, in particular for insertion when logging. Commands are used to gather data for the graphing process.


RoboDB RRD graph showing database statistics of relevant item sizes.


RoboDB RRD graph showing growth in number of objects during logging.


RoboDB RRD graph showing collections, indexes, and extents growth.


RoboDB RRD graph with info about index operations.

Future Work

The project is ongoing in particular with regard to further ways to employ the robotic database and by improving performance and ease of use in fault analysis. If you are interested in this work and want to get in touch please contact Tim Niemueller

Acknowledgements

This work was partly supported by the Quality of Life Technology Center at The Robotics Institute of the Carnegie Mellon University. T. Niemueller was partly supported by the Intel Summer Fellowship 2010 and by the German National Science Foundation (DFG) with grants GL747/9-5 and LA747/18-1.