Remove stop words from text

Hi,

I have to remove all Danish stop words from a lot of sentences. I use this function to do it.

Option Explicit
Function CleanStopWords(S As String, StopWords As Range)
    Dim RE As Object
    Dim SW() As String
    Dim C As Range
    Dim I As Long

ReDim SW(1 To StopWords.Count)
For I = 1 To StopWords.Count
    SW(I) = StopWords(I)
Next I

Set RE = CreateObject("vbscript.regexp")
With RE
    .Global = True
    .ignorecase = True

    'create pattern using the StopWords
    .Pattern = "\b(?:" & Join(SW, "|") & ")\b\s*"

    CleanStopWords = .Replace(S, "")
End With

End Function

The function is working but there is a small problem. Stop words containing the Danish letter 'å' are not removed. I think it has something to do with the pattern, but I can't figure out what's wrong?

Maybe someone could help me with the code.

I have attached a sample file.

Best regards
Morten

Hi Morten

Originally Posted by morsoe

The function is working but there is a small problem. Stop words containing the Danish letter 'å' are not removed. I think it has something to do with the pattern, but I can't figure out what's wrong?

If I understand your problem correctly this should not happen with all words with å. This should only happen with words starting or ending in å. For ex., hvornår, has an å in the middle and so should be no problem.

Please check.


Looking at your pattern I see you are using the \b anchor, a word boundary. This is defined as a position between a word and a non-word character. This was defined long ago and still uses the ascii letters. Remember that a regex word character was an ascii letter, a digit or an underscore, in regex: [a-zA-Z0-9_]. Some languages now allow regex's to use unicode letters instead but that's not the case with the vba implementation (yet).

I also saw that you have another non-ascii letter, ø, and you should have the same problem with it that you have with the å.

This means that in my opinion your problems lies in you using the \b. You are not correctly identifying the word boundaries when the words start or end with å or ø (there may be other non-ascii characters that you want to add to this list).


Solution: define your own word boundary, adding to the old definition of a word character the ø (H00F8) and å (H00E5), something like [a-zA-Z0-9_\u00F8\u00E5] and amend the code to take into account this new definition.

This should solve the problem. If you have problems adapting the code, post back and I'll try to do it tomorrow night (GMT).