api proposal: send images to llm #245104
Description
TPI: #244511 (1.99)
TPI: #247538 (1.100, for tool calling)
reference issue: #239976
right now, our LanguageModelChatMessage does not accept image parts in the message.
vscode/src/vscode-dts/vscode.d.ts
Lines 19747 to 19789 in a508d75
In order to support vision requests to the LLM, we need to support an additional type and format.
I propose:
vscode/src/vscode-dts/vscode.proposed.languageModelDataPart.d.ts
Lines 18 to 58 in 0237863
This adds a new LanguageModelDataPart that can be sent in the content of the LanguageModelChatMessage.
/**
* A language model response part containing arbitrary data, returned from a {@link LanguageModelChatResponse}.
*/
export class LanguageModelDataPart {
/**
* Factory function to create a `LanguageModelDataPart` for an image.
* @param data Binary image data
* @param mimeType The MIME type of the image
*/
static image(data: Uint8Array, mimeType: ChatImageMimeType): LanguageModelDataPart;
static json(value: object): LanguageModelDataPart;
static text(value: string): LanguageModelDataPart;
/**
* The mime type which determines how the data property is interpreted.
*/
mimeType: string;
/**
* The data of the part.
*/
data: Uint8Array;
/**
* Construct a generic data part with the given content.
* @param value The data of the part.
*/
constructor(data: Uint8Array, mimeType: string);
}Example usage:
const messages = [
vscode.LanguageModelChatMessage2.User([new vscode.LanguageModelDataPart({
data: imageData,
mimeType: 'image/png',
})]),
vscode.LanguageModelChatMessage2.User('Tell me about this image. Start each setence with "MEOW"'),
];
const chatResponse = await request.model.sendRequest(messages, {}, token);