Compare lists in one column to the first n elements of another column in...

pasta_paul

I'm trying to compare words present within sentences in one column, to the first n words in a wordlist (in another column), and return the words in the sentences that don't exist in that subset of the wordlist (to a new column). The "n" amount should be dynamically determined by the index of the sentence being currently compared. More specifically, n should be determined by the index position of the sentence divided by some scaling factor as...

Originally asked by qtamo on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

EDIT: I originally put in two solutions, but when I tested the code more robustly, the first one doesn't work. I think that collecting the column into a dataframe actually makes it impossible to do what you need to do here. You can only pull the current index out as an expression (Expr), and I think that means you're stuck doing everything with expressions, so no collecting. (Also I fixed a small typo in the working solution below. I think I pasted it in wrong.) The current index can be found in p.col("index"), since you put it there. You'll have to make sure it's there and starts at 1 when you read in from the CSV, but I'm assuming you've already done that. The best strat I could think of...

Full answer by concerned_citizen: https://stackoverflow.stackexchange.com/a/80014517 (CC BY-SA 4.0).

Need it done for your business? UphillTech.