lucene - Shingles in Elasticsearch which respect punctuation -


i building address matching engine uk addresses in elasticsearch , have found shingles useful seeing issues when comes punctuation. query "4 walmley close" returning following matches:

  1. units 3 , 4, walmley chambers, 3 walmley close
  2. flat 4, walmley court, 10 walmley close
  3. co-operative retail services ltd, 4 walmley close

the true match number 3, both 1 , 2 match (falsely) both become '4 walmley' when turned shingles. tell shingle analyzer not generate shingles straddle commas. so, example 1) get:

  • units 3
  • 3 and
  • and 4
  • 4 walmley
  • walmley chambers
  • chambers 3
  • 3 walmley
  • walmley close

...when in actual fact want is....

  • units 3
  • 3 and
  • and 4
  • walmley chambers
  • 3 walmley
  • walmley close

my current settings below. have experimented swapping tokenizer standard whitespace, helps in retains commas , potentially avoid situation above (i.e. end '4, walmley' shingle in address 1 , 2) end lots of unusable shingles in index , 70 million documents need keep index size down.

as can see in index settings have have street_sym filter love able use in shingles e.g. example, in addition generating 'walmley close' have 'walmley cl' when attempted include got shingles of 'close cl' not terribly helpful!

any advice more experienced elasticsearch users hugely appreciated. have read through gormley , tong's excellent book cannot head around particular issue.

thanks in advance offered.

"analysis": {     "filter": {         "shingle": {             "type": "shingle",             "output_unigrams": false         },         "street_sym": {             "type": "synonym",             "synonyms": [                 "st => street",                 "rd => road",                 "ave => avenue",                 "ct => court",                 "ln => lane",                 "terr => terrace",                 "cir => circle",                 "hwy => highway",                 "pkwy => parkway",                 "cl => close",                 "blvd => boulevard",                 "dr => drive",                 "ste => suite",                 "wy => way",                 "tr => trail"             ]         }     },     "analyzer": {         "shingle": {             "type": "custom",             "tokenizer": "standard",             "filter": [                 "lowercase",                 "shingle"             ]         }     } }

see comment on question why solution still won't stop "4 walmley close" matching 3 of matches provided. however, possible @ least tokens want. i'm not sure it's elegant/performant solution, using pattern replace, pattern capture, , length filters on shingles seems trick:

"analysis": {     "filter": {         "shingle": {             "type": "shingle",             "output_unigrams": false         },         "street_sym": {             "type": "synonym",             "synonyms": [                 "st => street",                 "rd => road",                 "ave => avenue",                 "ct => court",                 "ln => lane",                 "terr => terrace",                 "cir => circle",                 "hwy => highway",                 "pkwy => parkway",                 "cl => close",                 "blvd => boulevard",                 "dr => drive",                 "ste => suite",                 "wy => way",                 "tr => trail"             ]         },         "no_middle_comma": {             "type": "pattern_replace",             "pattern": ".+,.+",             "replacement": ""          },         "no_trailing_comma": {             "type": "pattern_capture",             "preserve_original": false,             "patterns": [                 "(.*),"             ]         },         "not_empty": {             "type": "length",             "min": 1         }     },     "analyzer": {         "test": {             "type": "custom",             "tokenizer": "whitespace",             "filter": [                 "lowercase",                 "street_sym",                 "shingle",                 "no_middle_comma",                 "no_trailing_comma",                 "not_empty"             ]         }     } }
  • no_middle_comma: replace tokens comma in middle empty token
  • no_trailing_comma: replace tokens ending comma part before comma
  • not_empty: remove empty tokens resulting above

for example, "units 3 , 4, walmley chambers, 3 walmley cl" becomes:

{    "tokens": [       {          "token": "units 3",          "start_offset": 0,          "end_offset": 7,          "type": "shingle",          "position": 0       },       {          "token": "3 and",          "start_offset": 6,          "end_offset": 11,          "type": "shingle",          "position": 1       },       {          "token": "and 4",          "start_offset": 8,          "end_offset": 14,          "type": "shingle",          "position": 2       },       {          "token": "walmley chambers",          "start_offset": 15,          "end_offset": 32,          "type": "shingle",          "position": 4       },       {          "token": "3 walmley",          "start_offset": 33,          "end_offset": 42,          "type": "shingle",          "position": 6       },       {          "token": "walmley close",          "start_offset": 35,          "end_offset": 45,          "type": "shingle",          "position": 7       }    ] }

note synonym filter works: "walmley cl" became "walmley close".


Comments

Post a Comment

Popular posts from this blog

javascript - Google App Script ContentService downloadAsFile not working -

i have web app developed using google app script htmlservice , html form, populating excel sheet in google drive using spreadsheetapp . , 1 section calling contentservice download data excel file. function doget(e) { // read excel sheet //getappfile(); // render application html template return htmlservice.createtemplatefromfile('index').evaluate() .settitle('go smart') .setsandboxmode(htmlservice.sandboxmode.iframe); } function downloaddoublequatecsvfile() { var sheetid = propertiesservice.getscriptproperties().getproperty('sheetid'); var ss = spreadsheetapp.openbyid(sheetid).getactivesheet(); //var ss = spreadsheetapp.getactivespreadsheet().getactivesheet(); var maxcolumn = ss.getlastcolumn(); var maxrow = ss.getlastrow(); var data = ss.getrange(1, 1, maxrow, maxcolumn).getvalues(); if (data.length > 1) { var csv = ""; (var row = 0; row < data.length; row++) { (var c...
Read more

javascript - Function overwritting -

is there way overwrite function or event handling in function? using addeventlistener, can have many event handlers on element want var test=document.getelementbyid("div"); test.addeventlistener("click",function(){alert("im first");},false); test.addeventlistener("click",function(){alert("im second");},false); // alerts "im first" , after "im second" but if want overwrite function/event example based on client width example this function one() { alert("first"); } function two() { alert("second"); } window.onresize = function() { var test = document.getelementbyid("div"); if (window.innerwidth > 500) { test.onclick = 1 } else { test.onclick = two; } } is possible in javascript? in case use effective little known approach. addeventlistener can accept object property handleevent event handler. in case it's easy overwr...
Read more

php - Find a regex to take part of Email -

i have following emails need extract part from: bestellnummer xxx: 1 von xxx, 1er pack ------------- anfang der nachricht ------------- foo bar baz foo bar baz // <<<<< need text here ------------- ende der nachricht ------------- ------------- anfang der nachricht ------------- foo bar baz foo bar baz ------------- ende der nachricht ------------- there 0 unlimited occurences of ------------- anfang der nachricht ------------- ------------- ende der nachricht ------------- and i'm able extract first part regex: $re = "/------------- .*? -------------.?(.*?).?------------- .*? -------------/s"; but, i'm quite new on learning regex, i'm pretty sure there must better regex extract part (foo bar baz foo bar baz) of text between ------------- anfang der nachricht ------------- ------------- ende der nachricht ------------- as can in different languages, i'm using .? to match between hyphens. i...
Read more