Cfrgtkky

Although I understand there are tools such as NLTK to do this for me, however, I would like to understand how I can slice several stems within a list efficiently.

Say my list of words is;

list = ["another", "cats", "walrus", "relaxed", "annoyingly", "rest", "normal", "hopping", "classes", "wing", "feed"]

And my common stems I would like to remove may be;

stems = ["s", "es", "ed", "est", "ing", "ly"] etc

With words I do not want stemmed specified as;

noStem = ["walrus", "rest", "wing", "feed"]

I have worked out how to do it for one specific stem like "s". For example, my code would be;

for eachWord in list:

    if eachWord not in noStem:

       if eachWord[-1] == "s":

           eachWord = eachWord[:-1]



stemmedList = stemmedList + [eachWord]

I am not sure how I would apply this to all my stems in a more efficient way.

Thanks for your help and advice!

I would suggest you convert noStem to a set so that the check if eachWord not in noStem is fast. Then you can check if the word endswith any stem in stems. If it does, you can use the largest stem that matches and remove it from the word:

lst = ["another", "cats", "walrus", "relaxed", "annoyingly", "rest", "normal", "hopping", "classes", "wing", "feed"]

stems = ["s", "es", "ed", "est", "ing", "ly"]

noStem = {"walrus", "rest", "wing", "feed"}

stemmedList = 



for word in lst:

    if word in noStem or not any([word.endswith(stem) for stem in stems]):

        stemmedList.append(word)

    else:

        stem = max([s for s in stems if word.endswith(s)], key=len)

        stemmedList.append(word[:len(word) - len(stem)])



print(stemmedList)

# ['another', 'cat', 'walrus', 'relax', 'annoying', 'rest', 'normal', 'hopp', 'class', 'wing', 'feed']

It's much more complicated than this but here goes a starter code using much faster pandas module. Here it goes.

import pandas as pd

import re



word_list = ["another", "cats", "walrus", "relaxed", "annoyingly", "rest", "normal", "hopping", "classes", "wing", "feed"]



stems = ["es",  "ed", "est", "ing", "ly", "s"]



# a set for quick lookup 

noStem = set(["walrus", "rest", "wing", "feed"])



# build series

words = pd.Series(word_list)



# filter out words in noStem

words = words[words.apply(lambda x: x not in noStem)]



# compile regular explession - performance - join all stems for matching

term_matching = '|'.join(stems)

expr = re.compile(r'(.+?)({})$'.format(term_matching))



df = words.str.extract(expr, expand=True)

df.dropna(how='any', inplace=True)

df.columns = ['words', 'stems']



stemmed_list = df.words.tolist()

I hope it helps ...

I think that isn't a bad start. You just need to add the second loop to be able to work with multiple endings. You can try something like below, (You'll notice I've renamed the variable list because it's dangerous having variables shadowing built-in names)

stemmed_list = 

for word in word_list:

    if word not in noStem:

        for ending in stems:

            if word.endswith(ending):

                 word = word[:-len(ending)]

                 break   # This will prevent iterating over all endings once match is found

    stemmed_list.append(word)

Or if as per your comment you don't want to use endswith

stemmed_list = 

for word in word_list:

    if word not in noStem:

        for ending in stems:

            if word[-len(ending):] == ending:

                 word = word[:-len(ending)]

                 break   # This will prevent iterating over all endings once match is found

    stemmed_list.append(word)