Rawlog Format

This page describes the format of binary files “.rawlog”, which store robotic datasets (see repository) and are the input of many MRPT applications for off-line processing.

Contents

1. Existing manipulation tools.
2. FORMAT #1: A Bayesian filter-friendly file format
- 2.1 Actual contents of a “.rawlog” file in this format
3. FORMAT #2: An timestamp-ordered sequence of observations
- 3.1 Actual contents of a “.rawlog” file in this format
4. Compression of rawlog files
5. Generating Rawlog files
6. Reading Rawlog files
- 6.1. Option A: Streaming from the file
- 6.2. Option B: Read at once
7. A note on Odometry

1. Existing manipulation tools.

RawLogViewer: This powerful GUI program is very useful to quickly visualize the contents of any dataset, modify it, export and import to other formats, etc…
rawlog-grabber: This command-line program grabs observations from a number of sensors and put all together into a single timestamp-ordered dataset file.
rawlog-edit: The command-line equivalent of RawlogViewer.
carmen2rawlog: A command-line tool to import CARMEN logs as Rawlogs.
rgbd_dataset2rawlog: A command-line utility to convert the TUM rgbd datasets into rawlogs.

2. FORMAT #1: A Bayesian filter-friendly file format

The purpose of a rawlog file is to reflect as accurately as possible all the data gathered by a robot as it moves through an environment, autonomously or manually guided.

Under the perspective of Bayesian SLAM methods, these data are divided in two clearly differentiated groups: actions and observations, denoted typically asu_k and z_k in the literature, respectively.

Hence, to ease the implementation of Bayesian methods in the MRPT, a rawlog file is divided in a sequence of actions, observations, actions, observations, … “Actions” typically include robot motor actuations (odometry), but any kind of user-defined actions can be defined as well (e.g. robot arm actuations). “Observations” include readings from the rest of robotic sensors: laser scanners, images from cameras, sonar ranges, etc.

Note that the intention of grouping several observations between two consecutive actions is to assure they are gathered approximately at the same time, although each individual observation has its own timestamp. The GUI application RawLogViewer provides several tools for visualizing and manipulating rawlog files.

2.1 Actual contents of a “.rawlog” file in this format

A rawlog file is a binary serialization of alternating objects of the classes:

CActionCollection, one or more actions (e.g. odometry), and
mrpt::obs::CSensoryFrame, which stores the observations.

3. FORMAT #2: An timestamp-ordered sequence of observations

While the previous format is really well-suited for Bayesian approaches with clearly separate steps of process action-process observation, in the case of complex datasets with many different sensors, working at different rates, and possibly without odometry (the typical ‘action’ in SLAM algorithms), it is more clear to just store datasets as an ordered list of observations.

3.1 Actual contents of a “.rawlog” file in this format

In this case, the rawlog file is a binary serialization of objects derived from the class mrpt::obs::CObservation. In this case, odometry is also stored as an observation.

The applications RawLogViewer, rawlog-grabber, the class mrpt::obs::CRawlog, and many of the localization & SLAM applications, all support both formats.

4. Compression of rawlog files

All rawlog files are transparently compressed using the gzip algorithm.

The compression level is set by default to ‘minimum’ to reduce as much as possible the computational load, while still deflating file sizes by a ratio of ~3. If compatibility with old MRPT versions (<MRPT 0.6.0) is required, the files can be renamed to “.rawlog.gz”, then decompressed using standard tools.

5. Generating Rawlog files

A standalone application that grabs rawlog from a set of robotic sensors is now also included with MRPT: rawlog-grabber.

This section describes the generic method to generate rawlog files from your own source code, which is useful to transform existing datasets (e.g. in custom plain text files) into the Rawlog format, or to capture online data from robotics sensors:

#include <mrpt/utils.h>
#include <mrpt/obs.h>

using namespace mrpt;
using namespace mrpt::utils;
using namespace mrpt::obs;
using namespace mrpt::poses;

int main()
{
  CFileGZOutputStream f("my_dataset.rawlog");

  while (there_is_more_data)
  {
    CActionCollection actions;
    CSensoryFrame SF;

    // Fill out the actions:
    // ----------------------------------
    CActionRobotMovement2D myAction; // For example, 2D odometry
    myAction.computeFromOdometry( ... );

    actions.insert( myAction );

    // Fill out the observations:
    // ----------------------------------
    CObservation2DRangeScanPtr myObs = CObservation2DRangeScan::Create();
    // CObservation2DRangeScanPtr myObs = CObservation2DRangeScanPtr(new CObservation2DRangeScan()); // This is an alternative to the line above

    myObs->... // Fill out the data

    SF.insert( myObs ); // memory of "myObs" will be automatically freed.

    // Save to the rawlog file:
    // --------------------------------
    f << actions << SF;
  };
  return 0;
}

#include <mrpt/utils.h>

#include <mrpt/obs.h>

using namespace mrpt;

using namespace mrpt::utils;

using namespace mrpt::obs;

using namespace mrpt::poses;

int main()

{

CFileGZOutputStream f("my_dataset.rawlog");

while (there_is_more_data)

{

CActionCollection actions;

CSensoryFrame SF;

// Fill out the actions:

// ----------------------------------

CActionRobotMovement2D myAction; // For example, 2D odometry

myAction.computeFromOdometry( ... );

actions.insert( myAction );

// Fill out the observations:

// ----------------------------------

CObservation2DRangeScanPtr myObs = CObservation2DRangeScan::Create();

// CObservation2DRangeScanPtr myObs = CObservation2DRangeScanPtr(new CObservation2DRangeScan()); // This is an alternative to the line above

myObs->... // Fill out the data

SF.insert( myObs ); // memory of "myObs" will be automatically freed.

// Save to the rawlog file:

// --------------------------------

f << actions << SF;

};

return 0;

}

6. Reading Rawlog files

6.1. Option A: Streaming from the file

This is the preferred mode of operation in general: actions and observations are read sequentially from the file, processed, then memory freed, and so on. In this way only the required objects are loaded in memory at any time, which is mandatory when managing large datasets, e.g. containing thousands of embedded images, millions of laser scans, etc.

A typical loop for loading a rawlog in this way is shown next:

CFileGZInputStream rawlogFile(filename); // "file.rawlog"
CActionCollectionPtr action;
CSensoryFramePtr observations;
CObservationPtr observation;
size_t rawlogEntry=0;
bool end = false;

// Read from the rawlog:
while ( CRawlog::getActionObservationPairOrObservation(
    rawlogFile, // Input file
    action, // Possible out var: action of a pair action/obs
    observations, // Possible out var: obs's of a pair action/obs
    observation, // Possible out var: a single obs.
    rawlogEntry // Just an I/O counter
    ) )
{
  // Process action & observations
  if (observation)
  {
    // Read a single observation from the rawlog (Format #2 rawlog file)
    ...
  }
  else
  {
    // action, observations should contain a pair of valid data (Format #1 rawlog file)
    ...
  }
};
// Smart pointers will be deleted automatically.

CFileGZInputStream rawlogFile(filename); // "file.rawlog"

CActionCollectionPtr action;

CSensoryFramePtr observations;

CObservationPtr observation;

size_t rawlogEntry=0;

bool end = false;

// Read from the rawlog:

while ( CRawlog::getActionObservationPairOrObservation(

rawlogFile, // Input file

action, // Possible out var: action of a pair action/obs

observations, // Possible out var: obs's of a pair action/obs

observation, // Possible out var: a single obs.

rawlogEntry // Just an I/O counter

) )

{

// Process action & observations

if (observation)

{

// Read a single observation from the rawlog (Format #2 rawlog file)

...

}

else

{

// action, observations should contain a pair of valid data (Format #1 rawlog file)

...

}

};

// Smart pointers will be deleted automatically.

6.2. Option B: Read at once

A rawlog file can be read as a whole using the class mrpt::obs::CRawlog.

Notice that this may be impractical for very large datasets (e.g. several tens of millions of entries) due to memory requirements, but for mid-sized datasets it definitively is the easiest way of loading rawlogs.

CRawlog dataset;
dataset.loadFromRawLogFile("file.rawlog");
cout << dataset.size() << " entries loaded." << endl;

CRawlog dataset;

dataset.loadFromRawLogFile("file.rawlog");

cout << dataset.size() << " entries loaded." << endl;

7. A note on Odometry

Notice that the representation of odometry depends on the Rawlog format being in format “#1 (Sensory frames)” or “#2 (observations only)”:

When using #1 (Actions & Sensory frames): Odometry readings are stored as actions (mrpt::slam::CActionRobotMovement2D), thus the increments in odometry are stored, not the absolute odometry readings.
When using #2 (observations only): Odometry is just treated as any other sensor, and that’s why in this case it’s stored as amrpt::obs::CObservationOdometry object, which contains the absolute odometry position read so far.