Software developer on the product here, I handled parts of the storage backend on this one and it presented some fun challenges.
Waldo already had features to stream logs and requests in the Sessions (Previously named "Live") view of the application and to save them so that they can be downloaded during automated tests but both features were heavily scoped toward their specific usage and Sessions meant that we needed to save all of theses in a way that we could keep them forever and show them as fast as possible when a replay is requested (With the ability to seek into them and advance all of that in sync with the video).
The previous storage was a JSONL text file (One log entry per line in JSON object format) but there was no way to use it as-is in a web viewer and get it to be fast-enough.
The full system log of a session can be up to 1Go of data (It's essentially logcat output in JSON format for android) that need to be filtered and accessible in small batches both by index when scrolling and by time when seeking the video.
Our current solution is based on having compressed Apache Arrow Columnar Format (IPC files) as long term storage and a server that caches filtered versions in-memory and full versions on disk when requested, along with a client that can load subsets of theses arrow files (Using apache-arrow npm package)
Software developer on the product here, I handled parts of the storage backend on this one and it presented some fun challenges.
Waldo already had features to stream logs and requests in the Sessions (Previously named "Live") view of the application and to save them so that they can be downloaded during automated tests but both features were heavily scoped toward their specific usage and Sessions meant that we needed to save all of theses in a way that we could keep them forever and show them as fast as possible when a replay is requested (With the ability to seek into them and advance all of that in sync with the video).
The previous storage was a JSONL text file (One log entry per line in JSON object format) but there was no way to use it as-is in a web viewer and get it to be fast-enough.
The full system log of a session can be up to 1Go of data (It's essentially logcat output in JSON format for android) that need to be filtered and accessible in small batches both by index when scrolling and by time when seeking the video.
Our current solution is based on having compressed Apache Arrow Columnar Format (IPC files) as long term storage and a server that caches filtered versions in-memory and full versions on disk when requested, along with a client that can load subsets of theses arrow files (Using apache-arrow npm package)