Rawlog Format
This page describes the format of binary files “.rawlog”, which store robotic datasets (see repository) and are the input of many MRPT applications for off-line processing.
Contents
1. Existing manipulation tools.
- RawLogViewer: This powerful GUI program is very useful to quickly visualize the contents of any dataset, modify it, export and import to other formats, etc…
- rawlog-grabber: This command-line program grabs observations from a number of sensors and put all together into a single timestamp-ordered dataset file.
- rawlog-edit: The command-line equivalent of RawlogViewer.
- carmen2rawlog: A command-line tool to import CARMEN logs as Rawlogs.
- rgbd_dataset2rawlog: A command-line utility to convert the TUM rgbd datasets into rawlogs.
2. FORMAT #1: A Bayesian filter-friendly file format
The purpose of a rawlog file is to reflect as accurately as possible all the data gathered by a robot as it moves through an environment, autonomously or manually guided.
Under the perspective of Bayesian SLAM methods, these data are divided in two clearly differentiated groups: actions and observations, denoted typically asuk and zk in the literature, respectively.
Hence, to ease the implementation of Bayesian methods in the MRPT, a rawlog file is divided in a sequence of actions, observations, actions, observations, … “Actions” typically include robot motor actuations (odometry), but any kind of user-defined actions can be defined as well (e.g. robot arm actuations). “Observations” include readings from the rest of robotic sensors: laser scanners, images from cameras, sonar ranges, etc.
Note that the intention of grouping several observations between two consecutive actions is to assure they are gathered approximately at the same time, although each individual observation has its own timestamp. The GUI application RawLogViewer provides several tools for visualizing and manipulating rawlog files.
2.1 Actual contents of a “.rawlog” file in this format
A rawlog file is a binary serialization of alternating objects of the classes:
- CActionCollection, one or more actions (e.g. odometry), and
- mrpt::obs::CSensoryFrame, which stores the observations.
3. FORMAT #2: An timestamp-ordered sequence of observations
While the previous format is really well-suited for Bayesian approaches with clearly separate steps of process action-process observation, in the case of complex datasets with many different sensors, working at different rates, and possibly without odometry (the typical ‘action’ in SLAM algorithms), it is more clear to just store datasets as an ordered list of observations.
3.1 Actual contents of a “.rawlog” file in this format
In this case, the rawlog file is a binary serialization of objects derived from the class mrpt::obs::CObservation. In this case, odometry is also stored as an observation.
The applications RawLogViewer, rawlog-grabber, the class mrpt::obs::CRawlog, and many of the localization & SLAM applications, all support both formats.
4. Compression of rawlog files
All rawlog files are transparently compressed using the gzip algorithm.
The compression level is set by default to ‘minimum’ to reduce as much as possible the computational load, while still deflating file sizes by a ratio of ~3. If compatibility with old MRPT versions (<MRPT 0.6.0) is required, the files can be renamed to “.rawlog.gz”, then decompressed using standard tools.
5. Generating Rawlog files
A standalone application that grabs rawlog from a set of robotic sensors is now also included with MRPT: rawlog-grabber.
This section describes the generic method to generate rawlog files from your own source code, which is useful to transform existing datasets (e.g. in custom plain text files) into the Rawlog format, or to capture online data from robotics sensors:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
#include <mrpt/utils.h> #include <mrpt/obs.h> using namespace mrpt; using namespace mrpt::utils; using namespace mrpt::obs; using namespace mrpt::poses; int main() { CFileGZOutputStream f("my_dataset.rawlog"); while (there_is_more_data) { CActionCollection actions; CSensoryFrame SF; // Fill out the actions: // ---------------------------------- CActionRobotMovement2D myAction; // For example, 2D odometry myAction.computeFromOdometry( ... ); actions.insert( myAction ); // Fill out the observations: // ---------------------------------- CObservation2DRangeScanPtr myObs = CObservation2DRangeScan::Create(); // CObservation2DRangeScanPtr myObs = CObservation2DRangeScanPtr(new CObservation2DRangeScan()); // This is an alternative to the line above myObs->... // Fill out the data SF.insert( myObs ); // memory of "myObs" will be automatically freed. // Save to the rawlog file: // -------------------------------- f << actions << SF; }; return 0; } |
6. Reading Rawlog files
6.1. Option A: Streaming from the file
This is the preferred mode of operation in general: actions and observations are read sequentially from the file, processed, then memory freed, and so on. In this way only the required objects are loaded in memory at any time, which is mandatory when managing large datasets, e.g. containing thousands of embedded images, millions of laser scans, etc.
A typical loop for loading a rawlog in this way is shown next:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
CFileGZInputStream rawlogFile(filename); // "file.rawlog" CActionCollectionPtr action; CSensoryFramePtr observations; CObservationPtr observation; size_t rawlogEntry=0; bool end = false; // Read from the rawlog: while ( CRawlog::getActionObservationPairOrObservation( rawlogFile, // Input file action, // Possible out var: action of a pair action/obs observations, // Possible out var: obs's of a pair action/obs observation, // Possible out var: a single obs. rawlogEntry // Just an I/O counter ) ) { // Process action & observations if (observation) { // Read a single observation from the rawlog (Format #2 rawlog file) ... } else { // action, observations should contain a pair of valid data (Format #1 rawlog file) ... } }; // Smart pointers will be deleted automatically. |
6.2. Option B: Read at once
A rawlog file can be read as a whole using the class mrpt::obs::CRawlog.
Notice that this may be impractical for very large datasets (e.g. several tens of millions of entries) due to memory requirements, but for mid-sized datasets it definitively is the easiest way of loading rawlogs.
1 2 3 |
CRawlog dataset; dataset.loadFromRawLogFile("file.rawlog"); cout << dataset.size() << " entries loaded." << endl; |
7. A note on Odometry
Notice that the representation of odometry depends on the Rawlog format being in format “#1 (Sensory frames)” or “#2 (observations only)”:
- When using #1 (Actions & Sensory frames): Odometry readings are stored as actions (mrpt::slam::CActionRobotMovement2D), thus the increments in odometry are stored, not the absolute odometry readings.
- When using #2 (observations only): Odometry is just treated as any other sensor, and that’s why in this case it’s stored as amrpt::obs::CObservationOdometry object, which contains the absolute odometry position read so far.