Help users of t-digest upgrade

[tdunning]

The following important projects are using older t-digest versions and should probably upgrade. An asterisk marks projects that have had a pull request or notification.

3.2:

• elastic: server
• atlassian bamboo core
  datavec api
• apache drill
• apache pinot core
  ML Dataset
  spotify metrics core
  apache servicemix
  apache nutch
  apache beam sdks java extensions sketching
  kixi stats
  apache druid extensions
  deeplearning4j modelexport solr

3.1:
apache solr
Apache Solr Content Extraction Library
Apache Solr Language Identifier
Apache Solr Prometheus Exporter Package
apache mahout math
apache kylin
Kite Morphlines Metrics Scalable
Brushfire Core (from stripe)

[dblock]

I am upgrading OpenSearch, opensearch-project/OpenSearch#3634.

We have the following assertion in tests with 3.2:

        final TDigestState state = new TDigestState(100);
        Arrays.stream(values).forEach(state::add);
        assertEquals(state.centroidCount(), values.length);

This is no longer the case with 3.3. @tdunning, could you please help me understand what changed? There are more failures after this, including very different percentiles returned with this upgrade.

[tdunning]

I think that this test is flawed by having assumptions out of view.

The number of centroids was always defined as <= the number of samples inserted. That is the entire point of the sketch ... to not store everything.

What may have changed is that the limit on centroid count has been made a bit more strict.

I could comment more intelligently if I knew what was in values

[dblock]

Thanks @tdunning. I tagged you in opensearch-project/OpenSearch#3634 that has source data from basic tests, and the diff in results from the upgrade. Could I please ask you to take a look? I certainly didn't expect such different numbers from an upgrade of a dot release of t-digest.