Generating an n-gram dataset based on an LLM

bash_bob

I want a dataset of common n-grams and their log likelihoods. Normally I would download the Google Books Ngram Exports , but I wonder if I can generate a better dataset using a large language model. For example, this script uses llama_cpp.Llama.create_completion to find likely 3-grams starting with "welcome to": from llama_cpp import Llama # pip install llama-cpp-python llm = Llama.from_pretrained(...

Originally asked by evashort on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Good thread on this — worth a read before you pay anyone to build it.

Discussion: https://stackoverflow.com/questions/79425052/generating-an-n-gram-dataset-based-on-an-llm.

Want it set up for you? UphillTech.