How do I pass an image to DSPy for analysis?

kernel_kate

I am running qwen2.5vl:7b with Ollama and want the model to analyze an image. When running: lm = dspy.LM(model="ollama_chat/qwen2.5vl:latest", api_base="http://...") dspy.configure(lm=lm) img = dspy.Image.from_file("image.png") response = lm("Describe this image", image=img) I get TypeError: Object of type Image is not JSON serializable Following the instructions from https://www.langtrace.ai/blog/attribute-extraction-from-images-using-dspy , I...

Originally asked by 1058u4 on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

As of dspy v3.0.2, you'd create a signature and pass dspy.Image as input, instead of a data uri for the image, as so: describe = dspy.Predict(dspy.Signature("image -> scene_description")) img = dspy.Image.from_file("image.png") print(describe(image=img).scene_description) or, if using a class-based Signature , class SceneDescriptionSignature(dspy.Signature): """Describe the contents of an image in detail.""" image: dspy.Image = dspy.InputField(desc="...") scene_description: str = dspy.OutputField(desc="...") describe = dspy.Predict(SceneDescriptionSignature) img = dspy.Image.from_file("image.png") print(describe(image=img).scene_description)

Full answer by tlo8: https://stackoverflow.stackexchange.com/a/79749769 (CC BY-SA 4.0).

Need it done for your business? UphillTech.