v0.1.3

A bug-fix release to improve error reporting, improve the UI, and fix some macOS-specific issues.

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

Bug fixes & improvements

• clp-s: Report exactly where parsing error occurs when parsing JSON (fixes #514). (#503)
• webui: Improve UI of links for viewing search results in context. (#515)
• core: Add support for retrieving CURL error messages, handle unexpected CURL return code on macOS, and log such codes in tests (fixes #519). (#517)
• core: Mark spdlog specialization functions as const (fixes #524). (#525)

View the full changelog for more details.

Thanks to @AVMatthews, @Bill-hbrhbr, @gibber9809, @haiqi96, @junhaoliao, @kirkrodrigues, and @LinZhihao-723 for their contributions.

v0.1.2

A release with support for viewing text search results in context by opening the file that contains them using using the log viewer. In a future release, we will also add support for viewing JSON search results in context. This release also includes a handful of bug fixes.

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

Features/Changes

Support for viewing text search results in context by opening the file that contains them using the log viewer. (#420, #424, #435, #437, #440, #442, #449, #451, #452, #458, #460, #468, #470, #472, #474, #476, #481, #485, #489, #490)

Bug-fixes

• clp-package: Add support for running package as root (fixes #500). (#464)
• clp-s: Ensure the option to print archive stats remains set when splitting archives. (#462)
• clp-s: Correctly report uncompressed size of archives when splitting archives (fixes #469). (#463)
• core-clp: Add class to encapsulate libcurl's global resource management to fix a race condition. (#461)
• core-clp: Add a more complete UTF-8 validation function to filter out binary data. (#477)
• core-clp: Fix ArchiveMetadata deserialization by deserializing previously missed field. (#488)
• webui: Optimize Meteor bundle size (fixes #498). (#499)

View the full changelog for more details.

Thanks to @Bill-hbrhbr, @gibber9809, @haiqi96, @Henry8192, @junhaoliao, @kirkrodrigues, @LinZhihao-723, and @wraymo for their contributions.

CLP v0.1.1

A release with a few new features and bug fixes.

The CLP release includes two tars:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

Features/Changes

• A new way of storing arrays found in JSON log events. (#355, #401, #408, #413, #429)
  • Try it using clp-s's --structurize-arrays option.
• clp-json no longer adds implicit wildcards (’*’) at the beginning or end of a query. (#406)
• More frequent progress updates (on the command line) for compression jobs. (#388)

Bug fixes

• clp-json: Better handling for wildcards in the middle of a token. (#407)
• Improved robustness when cancelling search jobs. (#425)
• Reduced CPU usage when queries from the webui complete. (#418)

View the full changelog for more details.

Thanks to @davemarco, @gibber9809, @haiqi96, @Henry8192, @kirkrodrigues, @LinZhihao-723, and @wraymo for their contributions.

CLP v0.1.0

A release containing several new features including a new subsystem for compressing JSON logs, a web interface, a server for aggregations, and more!

Note that the release now includes two tarballs:

• clp-json for compressing and searching JSON logs
• clp-text for compressing and searching unstructured text logs

Docs

The docs for this release are available here.

Highlights

• Added a subsystem for compressing and searching JSON logs. (#217)
• Added a web interface for searching logs and viewing compression statistics. (#250)
• Added support for ingesting CLP IR streams generated by our logging library plugins. (#152)
• Added a reducer worker to support aggregation jobs like count. (#221)
• Added support for tagging archives and filtering archives by tags and/or timestamps. (#138, #285)
• Replaced the scheduler with separate compression and search schedulers for improved robustness and performance. (#238, #240)

This release also includes many other improvements and fixes. View the full changelog for more details.

Thanks to @abvarun226, @davemarco, @davidlion, @diy1, @gibber9809, @haiqi96, @junhaoliao, @kirkrodrigues, @LinZhihao-723, @OliverSM95, @SharafMohamed, and @wraymo for their contributions!

CLP v0.0.2

A release containing all of CLP's open-source features and bug-fixes to date.

Docs

The package contains a README or you can view it here.

Highlights

This release features several usability improvements, bug-fixes, and performance improvements. Some highlights include:

• Support for compression/search using user-specified variable schemas
• A custom encoding for floating-point variables that improves performance and provides better compression
• Reduced memory usage during compression
• Search using multiple workers

Features

• Use a custom float encoding for improved performance and better compression (#38)
• Reduce memory usage during compression by keeping only the necessary parts of various data structures (including the dictionaries) in memory (#40, #47, #48)
• Separate components into different containers and allow them to be started separately (#26, #65)
• Use the host's network rather than Docker's networking to eliminate the difference between single and multi-node deployments (#65)
• Use an instance ID to name all components to allow multiple deployments on a single node (#65)
• Expose the complete config file to the user rather than a simplified one (#65)
• Don't generate a config file but instead use default settings unless modified by a config file (#65)
• Add support for search with multiple nodes (#68)
• Add initial support for compression/search using user-defined variable schemas (#74)
• Update variables schema and encoding methods to match those of the existing logging libraries (#92, #115)
• Add support for building CLP core on macOS (#105, #106, #107)
• Switch to C++ 17's from_chars for better performance (#85)

Bug Fixes

• Use the current user's UID/GID (rather than root) when reading/writing files within mounted directories (#10)
• Recombine large files split across archives in their original order (#37)
• Fixes to enable compression on macOS and within WSL (#45)
• Update celery to 5.2.2 to mitigate CVE-2021-23727 (#46)
• Resolve input paths for compression before using them in the containers (#49)
• Use the archive output directory specified in the config file (#65)
• Cancel decompression/search when the user sends a keyboard interrupt (#65)
• Wait for the database to initialize before trying to use it (#65)
• Mount input logs at a dedicated path in the container that doesn't overlap with files in the container (#65)
• Allow filtering for files which overlap (rather than are encompassed by) the specified time range (#70)
• Allow an empty logtype for messages that only include a timestamp (#76)
• Remove ad-hoc rules used on top of CLP's default variables schema (#42)
• Compress integers less than -2^63 losslessly (#85)
• Add support for compressing zstd-compressed log files (#114)

Thanks to @All-less, @davidlion, @haiqi96, @kirkrodrigues, @LinZhihao-723, @NamanGulati, @SharafMohamed, and @thepegasos for their contributions.

CLP v0.0.1

A patch release to correct some archive metadata.

Documentation

The package contains a README.md or you can view it here.

Features & Fixes

• Fix for #15: The global metadata database now stores the correct uncompressed size of each archive
• Add the table prefix option for users of the global mysql metadata database

CLP v0.0.0

CLP's first release, one of many to come!

Documentation

The package contains a README.md or you can view it here.

Features

• Parallel compression via a push-button cluster of Docker containers or through a multi-node deployment
• Decompression
• Search

Thanks to @davidlion @diy1 @jackluo923 @kirkrodrigues @NamanGulati @rishikeshdevsot @robinlee09201 @wraymo for their code and testing contributions.