Rawlog Format

This page describes the format of binary files “.rawlog”, which store robotic datasets (see repository) and are the input of many MRPT applications for off-line processing.

1. Existing manipulation tools.

  • RawLogViewer: This powerful GUI program is very useful to quickly visualize the contents of any dataset, modify it, export and import to other formats, etc…
  • rawlog-grabber: This command-line program grabs observations from a number of sensors and put all together into a single timestamp-ordered dataset file.
  • rawlog-edit: The command-line equivalent of RawlogViewer.
  • carmen2rawlogA command-line tool to import CARMEN logs as Rawlogs.
  • rgbd_dataset2rawlog: A command-line utility to convert the TUM rgbd datasets into rawlogs.

2. FORMAT #1: A Bayesian filter-friendly file format

The purpose of a rawlog file is to reflect as accurately as possible all the data gathered by a robot as it moves through an environment, autonomously or manually guided.

Under the perspective of Bayesian SLAM methods, these data are divided in two clearly differentiated groups: actions and observations, denoted typically asuk and zk in the literature, respectively.

Hence, to ease the implementation of Bayesian methods in the MRPT, a rawlog file is divided in a sequence of actions, observations, actions, observations, … “Actions” typically include robot motor actuations (odometry), but any kind of user-defined actions can be defined as well (e.g. robot arm actuations). “Observations” include readings from the rest of robotic sensors: laser scanners, images from cameras, sonar ranges, etc.

Note that the intention of grouping several observations between two consecutive actions is to assure they are gathered approximately at the same time, although each individual observation has its own timestamp. The GUI application RawLogViewer provides several tools for visualizing and manipulating rawlog files.

2.1 Actual contents of a “.rawlog” file in this format

A rawlog file is a binary serialization of alternating objects of the classes:

3. FORMAT #2: An timestamp-ordered sequence of observations

While the previous format is really well-suited for Bayesian approaches with clearly separate steps of process action-process observation, in the case of complex datasets with many different sensors, working at different rates, and possibly without odometry (the typical ‘action’ in SLAM algorithms), it is more clear to just store datasets as an ordered list of observations.

3.1 Actual contents of a “.rawlog” file in this format

In this case, the rawlog file is a binary serialization of objects derived from the class mrpt::obs::CObservation. In this case, odometry is also stored as an observation.

The applications RawLogViewerrawlog-grabber, the class mrpt::obs::CRawlog, and many of the localization & SLAM applications, all support both formats.

4. Compression of rawlog files

All rawlog files are transparently compressed using the gzip algorithm.

The compression level is set by default to ‘minimum’ to reduce as much as possible the computational load, while still deflating file sizes by a ratio of ~3. If compatibility with old MRPT versions (<MRPT 0.6.0) is required, the files can be renamed to “.rawlog.gz”, then decompressed using standard tools.

5. Generating Rawlog files

A standalone application that grabs rawlog from a set of robotic sensors is now also included with MRPT: rawlog-grabber.

This section describes the generic method to generate rawlog files from your own source code, which is useful to transform existing datasets (e.g. in custom plain text files) into the Rawlog format, or to capture online data from robotics sensors:


6. Reading Rawlog files

6.1. Option A: Streaming from the file

This is the preferred mode of operation in general: actions and observations are read sequentially from the file, processed, then memory freed, and so on. In this way only the required objects are loaded in memory at any time, which is mandatory when managing large datasets, e.g. containing thousands of embedded images, millions of laser scans, etc.

A typical loop for loading a rawlog in this way is shown next:


6.2. Option B: Read at once

A rawlog file can be read as a whole using the class mrpt::obs::CRawlog.

Notice that this may be impractical for very large datasets (e.g. several tens of millions of entries) due to memory requirements, but for mid-sized datasets it definitively is the easiest way of loading rawlogs.


7. A note on Odometry

Notice that the representation of odometry depends on the Rawlog format being in format “#1 (Sensory frames)” or “#2 (observations only)”:

  • When using #1 (Actions & Sensory frames): Odometry readings are stored as actions (mrpt::slam::CActionRobotMovement2D), thus the increments in odometry are stored, not the absolute odometry readings.
  • When using #2 (observations only): Odometry is just treated as any other sensor, and that’s why in this case it’s stored as amrpt::obs::CObservationOdometry object, which contains the absolute odometry position read so far.