No hits on "fitbit" when searching for "fit bit"

[hans-vereyken-arvesta]

Hi,

On our Magento 2 (2.4.6-p1) we are using the ElasticSuite (2.11.3.2) for search. Works great, except for one thing, and I can't get my head around it. When people search for "fit bit", they should get the product with the name "fitbit" as a hit, but it returns no results.
I tried creating a new analyzer "spaceless_shingle" of the "shingle" type with a "token_separator" of an empty string. I noticed the "reference" analyzer, with the "reference_shingle" token filter, doing already exactly what I want. So I tried to set the default analyzer of the "name" product attribute to reference, but still, no hits on "fitbit". I'm clearly missing something, but can't figure it out... (after every change, I did reindex the Elasticsearch indices).

Any clues where I need to look next?

Thanks in advance

[hans-vereyken-arvesta]

Ok, the example 'fitbit' vs 'fit bit' isn't a good one in our situation. Digging deeper I found a possible reason why it's not working. It's actually a Dutch website, the real examples are 'graszaad' vs 'gras zaad' (doesn't work), 'houtpellets' vs 'hout pellets' (doesn't work) and 'grasmaaier' vs 'gras maaier' (WORKS!).
So, looking in how things work I think this is what happens: For the first two examples, the TermVectors pre-analysis gives me SPELLING_TYPE_EXACT because the terms 'gras' and 'zaad' do exist as separate terms in different products, the same for 'hout' and 'pellets'. But 'maaier' doesn't exist as a standalone term, that's why it does find 'grasmaaier' when searching for 'gras maaier' and doesn't find 'graszaad' when searching for 'gras zaad'. But now, how to fix it?

[rbayet]

Hello @hans-vereyken-arvesta,

Concerning your initial issue, indeed you identified that reference_shingle filter of the reference analyzer is able to regroup two tokens ("fit bit" will have been tokenized into "fit" and "bit"). (Was the minimal interface for analysis in the Magento admin (Elasticsuite > System > Analysis) useful in that effect ?).

Concerning your issue with the Dutch language (you would have the same in German), we have yet to introduce a better support for compound words of germanic languages out of the box.

Basically, it would consist in using the Hyphenation decompounder token filter with both a quality hyphenation rules list and a good word_list_path.
Stack overflow tells[1] me that https://github.com/uschindler/german-decompounder would be good start for the german language.

That's basically the road people seem to take for the Dutch language too, see https://www.luminis.eu/blog/search-en/using-compound-words-in-elasticsearch/

Regards,

[1] https://stackoverflow.com/questions/59595689/elasticsearch-handling-german-compound-words

I use Hyphenation decompounder with the dictionary from github.com/uschindler/german-decompounder (OnlyLongestMatch = true)

[hans-vereyken-arvesta]

Thx a lot rbayet, I can see the problem clearly now. Now we need to see what is the preferable solution, go through implementing the Hyphenation decompounder token filter, or keep a list of synonyms (works great, without any development effort).