GeminiWithFiles
v2.0.2
Gemini API is continuing to grow. On August 5, 2024, I largely updated GeminiWithFiles to v2.x.x. With this large update, the version is changed from v1 to v2.
If you want to use GeminiWithFiles v1.x.x, please see here.
This is a Google Apps Script library for Gemini API with files.
A new Google Apps Script library called GeminiWithFiles simplifies using Gemini, a large language model, to process unstructured data like images and PDFs. GeminiWithFiles can upload files, generate content, and create descriptions from multiple images at once. This significantly reduces workload and expands possibilities for using Gemini.
Recently, Gemini, a large language model from Google AI, has brought new possibilities to various tasks by enabling the use of unstructured data as structured data. This is particularly significant because a vast amount of information exists in unstructured formats like text documents, images, and videos.
Gemini 1.5 API, released recently, significantly expands these capabilities. It can generate content by up to 1 million tokens, a substantial increase compared to previous versions. Additionally, Gemini 1.5 can now process up to 3,000 image files, vastly exceeding the 16-image limit of Gemini 1.0. Ref
While Gemini cannot directly work with Google Drive formats like Docs, Sheets, and Slides, there are workarounds. In the current stage, PDF data can be directly processed with Gemini API. Using this, those Google Docs files are converted to PDF and used with Gemini API. Ref
This report introduces a new Google Apps Script library called "GeminiWithFiles" that simplifies this process. GeminiWithFiles allows users to easily upload files and generate content using Gemini's powerful capabilities. It also enables efficient description creation from multiple images with a single API call, significantly reducing the workload compared to processing each image individually as demonstrated in my prior report. Ref
By streamlining the process and expanding capabilities, GeminiWithFiles holds promise for various use cases across different domains. This report serves as an extended approach to the previous one, aiming to further reduce process costs and improve efficiency when working with Gemini and unstructured data.
I created this library based on the following reports.
This library GeminiWithFiles allows you to interact with Gemini, a powerful document processing and management platform, through an easy-to-use API. Here's what you can achieve with this library:
File Management:
Content Upload:
Chat History Management:
Content Generation:
Output Specification:
Specify the desired output format for the results generated by the Gemini API.
Using response_mime_type and JSON schema, the output format is controlled. Ref
In order to test this script, please do the following steps.
Please access https://makersuite.google.com/app/apikey and create your API key. At that time, please enable Generative Language API at the API console. This API key is used for this sample script.
This official document can also be seen. Ref.
Please create a standalone Google Apps Script project. Of course, this script can also be used with the container-bound script.
And, please open the script editor of the Google Apps Script project.
There are 2 patterns for using GeminiWithFiles.
If you use this library as a Google Apps Script library, please install the library to your Google Apps Script project as follows.
Create a Google Apps Script project. Or, open your Google Apps Script project.
Install this library.
1dolXnIeXKz-BH1BlwRDaKhzC2smJcGyVxMxGYhaY2kqiLa857odLXrIC
If you use this library in your own Google Apps Script project, please copy and paste the script "classGeminiWithFiles.js" into your Google Apps Script project. By this, the script can be used.
"main.js" is used for the Google Apps Script library. So, in this pattern, you are not required to use it.
This library uses the following 2 scopes.
https://www.googleapis.com/auth/script.external_requesthttps://www.googleapis.com/auth/driveIf you want to use the access token, please link the Google Cloud Platform Project to the Google Apps Script Project. And, please add the following scope.
https://www.googleapis.com/auth/generative-languageAlso, you can see the official document of Gemini API at https://ai.google.dev/api/rest.
| Methods | Description |
|---|---|
| setFileIds(fileIds, asImage = false) | Set file IDs. |
| setBlobs(blobs) | Set blobs. |
| withUploadedFilesByGenerateContent(fileList = []) | Create object for using the generateContent method. |
| uploadFiles(n = 50) | Upload files to Gemini. |
| getFileList() | Get file list in Gemini. |
| deleteFiles(names, n = 50) | Delete files from Gemini. |
| generateContent(object) | Main method. Generate content by Gemini API. |
| setFileIdsOrUrlsWithResumableUpload(object) | File over 50 MB can be uploaded to Gemini. |
When you install GeminiWithFiles as a library to your Google Apps Script project, please use the following script.
const g = GeminiWithFiles.geminiWithFiles(object);or
When you directly copy and paste the script of Class GeminiWithFiles into your Google Apps Script project, please use the following script.
const g = new GeminiWithFiles(object);The value of object is as follows.
{Object} object API key or access token for using Gemini API.
{String} object.apiKey API key.
{String} object.accessToken Access token.
{String} object.model Model. Default is "models/gemini-1.5-pro-latest".
{String} object.version Version of API. Default is "v1beta".
{Boolean} object.doCountToken Default is false. If this is true, when Gemini API is requested, the token of request is shown in the log.
{Array} object.history History for continuing chat.
{Array} object.functions If you want to give the custom functions, please use this.
{String} object.response_mime_type In the current stage, only "application/json" can be used.
{String} object.responseMimeType In the current stage, only "application/json" can be used.
{Object} object.response_schema JSON schema for controlling the output format.
{Object} object.responseSchema JSON schema for controlling the output format.
{Number} object.temperature Control the randomness of the output.
{Object} object.systemInstruction Ref: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini.
{Boolean} object.exportTotalTokens When this is true, the total tokens are exported as the result value. At that time, the generated content and the total tokens are returned as an object.
{Boolean} object.exportRawData The default value is false. When this is true, the raw data returned from Gemini API is returned.
{Object} object.toolConfig The default is null. If you want to directly give the object of "toolConfig", please use this.
{Array} object.tools The default value is null. For example, when you want to use "codeExecution", please set `tools: [{ codeExecution: {}}]`.
{PropertiesService.Properties} object.propertiesService PropertiesService.getScriptProperties()
{Boolean} object.resumableUploadAsNewUpload When you want to upload the data with the resumable upload as new upload, please set this as true. The default is false.
When you want to use response_mime_type, please give jsonSchema to generateContent method. In the current stage, only "application/json" can be used to response_mime_type.
When you want to use systemInstruction, please confirm the official document Ref.
Gemini 1.5 Flash Latest (models/gemini-1.5-flash-latest) is used as the default model. When you want to use Gemini 1.5 Pro Latest (models/gemini-1.5-pro-latest), please use it like const g = GeminiWithFiles.geminiWithFiles({ apiKey, model: "models/gemini-1.5-pro-latest" }).
In the current stage, when response_schema is used, response_mime_type: "application/json" is automatically used.
Set file IDs. The files of file IDs are uploaded to Gemini.
In this case, async/await is used in the function.
function myFunction() {
const apiKey = "###"; // Please set your API key.
const folderId = "###"; // Please set your folder ID including images.
let fileIds = [];
const files = DriveApp.getFolderById(folderId).getFiles();
while (files.hasNext()) {
const file = files.next();
fileIds.push(file.getId());
}
const g = GeminiWithFiles.geminiWithFiles({ apiKey }); // This is for installing GeminiWithFiles as a library.
// const g = new GeminiWithFiles({ apiKey }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
const res = g.setFileIds(fileIds, false).uploadFiles();
console.log(res);
}setFileIds are String[] (the file IDs on Google Drive) and the boolean, respectively. If the 2nd argument is false, the inputted files of file IDs are uploaded as raw data. If the 2nd argument is true, the inputted files of file IDs are converted to image data and are uploaded. The default of 2nd argument is false.false at this method like setFileIds(fileIds, false).Set blobs. The blobs are uploaded to Gemini.
function myFunction() {
const apiKey = "###"; // Please set your API key.
const folderId = "###"; // Please set your folder ID including images.
const blobs = [];
const files = DriveApp.getFolderById(folderId).getFiles();
while (files.hasNext()) {
blobs.push(files.next().getBlob());
}
const g = GeminiWithFiles.geminiWithFiles({ apiKey }); // This is for installing GeminiWithFiles as a library.
// const g = new GeminiWithFiles({ apiKey }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
const res = g.setBlobs(blobs).uploadFiles();
console.log(res);
}setBlobs is Blob[].Create object for using the generateContent method.
function myFunction() {
const apiKey = "###"; // Please set your API key.
const q = "###"; // Please set your question.
const g = GeminiWithFiles.geminiWithFiles({ apiKey }); // This is for installing GeminiWithFiles as a library.
// const g = new GeminiWithFiles({ apiKey }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
const fileList = g.getFileList();
const res = g
.withUploadedFilesByGenerateContent(fileList)
.generateContent({ q });
console.log(res);
}withUploadedFilesByGenerateContent has only one argument. That is the value from the getFileList method. You can see the actual values after you uploaded files.Upload files to Gemini. The files are uploaded to Gemini using the inputted file IDs or blobs.
function myFunction() {
const apiKey = "###"; // Please set your API key.
const fileIds = ["###fileId1###", "###fileId2###", , ,]; // Please set your file IDs in this array.
const g = GeminiWithFiles.geminiWithFiles({ apiKey }); // This is for installing GeminiWithFiles as a library.
// const g = new GeminiWithFiles({ apiKey }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
const res = g.setFileIds(fileIds, false).uploadFiles();
console.log(res);
}In this script, the files of fileIds are uploaded to Gemini with the raw data. If setFileIds(fileIds, false) is modified to setFileIds(fileIds, true), the files are uploaded to Gemini as images.
When you directly use Blob, you can use the following script.
function myFunction() {
const apiKey = "###"; // Please set your API key.
const fileIds = ["###fileId1###", "###fileId2###", , ,]; // Please set your file IDs in this array.
const blobs = fileIds.map(id => DriveApp.getFileById(id).getBlob());
const g = GeminiWithFiles.geminiWithFiles({ apiKey }); // This is for installing GeminiWithFiles as a library.
// const g = new GeminiWithFiles({ apiKey }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
const res = g.setBlobs(blobs).uploadFiles();
console.log(res);
}Get file list in Gemini.
function myFunction() {
const apiKey = "###"; // Please set your API key.
const g = GeminiWithFiles.geminiWithFiles({ apiKey }); // This is for installing GeminiWithFiles as a library.
// const g = new GeminiWithFiles({ apiKey }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
const res = g.getFileList();
console.log(res);
}Delete files from Gemini.
function myFunction() {
const apiKey = "###"; // Please set your API key.
const g = GeminiWithFiles.geminiWithFiles({ apiKey }); // This is for installing GeminiWithFiles as a library.
// const g = new GeminiWithFiles({ apiKey }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
const names = g.getFileList().map(({ name }) => name);
if (names.length == 0) return;
g.deleteFiles(names);
console.log(`${names.length} files were deleted.`);
}Main method. Generate content by Gemini API. More sample scripts can be seen in the following "Sample scripts" section.
function myFunction() {
const apiKey = "###"; // Please set your API key.
const g = GeminiWithFiles.geminiWithFiles({ apiKey }); // This is for installing GeminiWithFiles as a library.
// const g = new GeminiWithFiles({ apiKey }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
const res = g.generateContent({ q: "What is Google Apps Script?" });
console.log(res);
}In this script, the content is generated with the function calling.
When you want to use response_mime_type, please give jsonSchema to generateContent method as follows. In this case, by giving only JSON schema, this library can return a valid object. You can also see the detailed information about response_mime_type at my report.
function myFunction() {
const apiKey = "###"; // Please set your API key.
const g = GeminiWithFiles.geminiWithFiles({
apiKey,
response_mime_type: "application/json",
}); // This is for installing GeminiWithFiles as a library.
// const g = new GeminiWithFiles({ apiKey, response_mime_type: "application/json" }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
const jsonSchema = {
title: "5 popular cookie recipes",
description: "List 5 popular cookie recipes.",
type: "array",
items: {
type: "object",
properties: {
recipe_name: {
description: "Names of recipe.",
type: "string",
},
},
},
};
const res = g.generateContent({ jsonSchema });
console.log(res);
}When this script is run, the following result is obtained.
[
{ "recipe_name": "Chocolate Chip Cookies" },
{ "recipe_name": "Peanut Butter Cookies" },
{ "recipe_name": "Oatmeal Cookies" },
{ "recipe_name": "Sugar Cookies" },
{ "recipe_name": "Snickerdoodle Cookies" }
]This method can upload files over 50 MB.
From v2.x.x, this can be achieved. This is from Ref and Ref.
From v2.0.3, when you use this method, please include propertiesService: PropertiesService.getScriptProperties() into the initial object as follows. Because, when PropertiesService.getScriptProperties() is used in the library, the values are put into the library. When I created Ref and Ref, I supposed that the script is used by copying and pasting instead of the library. So, I included PropertiesService.getScriptProperties() in the script. But I noticed that when this is used with GeminiWithFiles, each user is required to use PropertiesService.getScriptProperties(). So, I modified this.
The sample script is as follows.
function myFunction() {
// This URL is from https://github.com/google/generative-ai-docs/blob/main/site/en/gemini-api/docs/prompting_with_media.ipynb
const url = "https://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4"; // 64,657,027 bytes
const apiKey = "###"; // Please set your API key.
const q = "Description this video.";
const g = GeminiWithFiles.geminiWithFiles({ apiKey, propertiesService: PropertiesService.getScriptProperties() }); // This is for installing GeminiWithFiles as a library.
// const g = new GeminiWithFiles({ apiKey, propertiesService: PropertiesService.getScriptProperties() }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
const fileList = g.setFileIdsOrUrlsWithResumableUpload([{ url }]).uploadFiles();
Utilities.sleep(10000); // This might be required to be used because the state of the uploaded file might not be active.
const res = g.withUploadedFilesByGenerateContent(fileList).generateContent({ q });
console.log(res);
}When this script is run, the following log can be seen at the log.
- Get metadata
- Calculate chunks
- Get location
- Download and upload data.
- Now... 1/4
- Start downloading data with 0-16777215
- Finished downloading data with 0-16777215
- Start uploading data with 0-16777215
- Finished uploading data with 0-16777215
- Upload the next chunk.
- Now... 2/4
- Start downloading data with 16777216-33554431
- Finished downloading data with 16777216-33554431
- Start uploading data with 16777216-33554431
- Finished uploading data with 16777216-33554431
- Upload the next chunk.
- Now... 3/4
- Start downloading data with 33554432-50331647
- Finished downloading data with 33554432-50331647
- Start uploading data with 33554432-50331647
- Finished uploading data with 33554432-50331647
- Upload the next chunk.
- Now... 4/4
- Start downloading data with 50331648-64657026
- Finished downloading data with 50331648-64657026
- Start uploading data with 50331648-64657026
- Finished uploading data with 50331648-64657026
- Done.
- Now, the state of the uploaded files "url@https://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4$page@1$maxPage@1" is not active. So, it will wait until it is active. Please wait for 10 seconds. Retry (1/3)
- 1 uploaded files are used with generateCotent.
- The video is a cartoon that shows a large, white rabbit in a field. The rabbit is shown waking up from a nap and then is seen eating an apple. After eating the apple, the rabbit is approached by a bird. The rabbit is scared of the bird and tries to hide from it. The bird flies away. The rabbit is seen smiling and then a squirrel flies toward the rabbit. The squirrel is startled by the rabbit and flies away. The rabbit is then seen catching another squirrel with its vine and the scene ends with a close-up of the rabbit's face.
If your file is large and the state of uploaded file has not still been "ACTIVE", please test the following script.
function myFunction() {
// This URL is from https://github.com/google/generative-ai-docs/blob/main/site/en/gemini-api/docs/prompting_with_media.ipynb
const url = "https://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4"; // 64,657,027 bytes
const apiKey = "###"; // Please set your API key.
const q = "Description this video.";
const g = GeminiWithFiles.geminiWithFiles({ apiKey, propertiesService: PropertiesService.getScriptProperties() }); // This is for installing GeminiWithFiles as a library.
// const g = new GeminiWithFiles({ apiKey, propertiesService: PropertiesService.getScriptProperties() }); // This is for directly copying and pasting Class GeminiWithFiles into your Google Apps Script project.
const fileList = g.setFileIdsOrUrlsWithResumableUpload([{ url }]).uploadFiles();
console.log(JSON.stringify(fileList));
// Please copy the value of "fileList".
}By this, the file can be uploaded. And, you can use the uploaded file after it waits enough time to change the state to "ACTIVE". The uploaded file can be used as follows.
