voxelgpt
1.0.0
Wish you could search your images or videos without writing a line of code? Want to extract insights from your data by asking in plain English? Now you can! ?
VoxelGPT is a FiftyOne Plugin that combines the power of large language models (LLMs) and large multimodal models (LMMs) with FiftyOne's computer vision query language, enabling you to filter, sort, semantically slice and ask questions about your data using natural language. It can even perform computations on your dataset for you — with approval, of course!
You can try VoxelGPT live at gpt.fiftyone.ai!
VoxelGPT is capable of handling any of the following types of queries:
When you ask VoxelGPT a question, it will interpret your intent and determine which type of query you are asking. If VoxelGPT is unsure, it will ask you to clarify.
VoxelGPT can handle the following types of queries about your dataset:
You can ask VoxelGPT to search your datasets for you. Here's some examples of things you can ask:
Under the hood, VoxelGPT interprets your query and translates it into the corresponding dataset view. VoxelGPT understands the schema of your dataset, as well as things like evaluation runs and similarity indexes.
It can also automatically inspect the contents of your dataset in order to retrieve specific entities.
VoxelGPT can answer questions about the schema of your dataset, brain runs, evaluation runs, and more. Here are some examples:
If your dataset contains one or more
fo.Detections
field(s), VoxelGPT can filter or match based on the size (relative and
absolute) of bounding boxes, and on the number of detections.
If your dataset has a
GeoLocation
field, you can run geographic queries on your dataset. VoxelGPT can perform
geocoding to go from location name (or textual description) to a (lon, lat)
pair, or a list of (lon, lat) points defining a boundary region. Here are a
few examples:
If your dataset has a
Date or DateTime field,
VoxelGPT can perform temporal queries such as:
event field reading a time of day between 8pm and
11pmVoxelGPT has access to Aggregation stages in FiftyOne, so it can perform
aggregations like count, mean, sum, std, min, max, values, and
distinct for a field or expression over the entire dataset or a view into the
dataset. Here are some examples:
cat label?confidence field in my predictions?VoxelGPT can perform computations on your dataset, such as:
Here's some examples of computational queries you can ask VoxelGPT:
If you do not want to allow VoxelGPT to run computations, set the environment variable:
export VOXELGPT_ALLOW_COMPUTATIONS=falseYou can also set the minimum dataset size at which VoxelGPT needs to ask for permission to run computations:
export VOXELGPT_APPROVAL_THRESHOLD=1000The default value is 100 samples.
VoxelGPT is not only a pair programmer; it is also an educational tool. VoxelGPT has access to the entire FiftyOne docs, as well as all of the blog posts on the Voxel51 Blog, and transcripts from videos on the Voxel51 YouTube channel. It can use all of these resources to answer FiftyOne-related questions.
Here's some examples of documentation queries you can ask VoxelGPT:
VoxelGPT will provide links to the most helpful resources across Voxel51's docs, blog, and YouTube channel. For YouTube videos, the links will point directly to the most relevant timestamp!
VoxelGPT can answer questions about the environment in which you are running FiftyOne, including:
Here's some examples of workspace queries you can ask VoxelGPT:
Finally, VoxelGPT can answer general questions about computer vision, machine learning, and data science. It can help you to understand basic concepts and learn how to overcome data quality issues.
Here's some examples of machine learning queries you can ask VoxelGPT:
If you haven't already, install FiftyOne:
pip install fiftyoneYou'll also need to provide an OpenAI API key (create one):
export OPENAI_API_KEY=XXXXXXXXFor use with your private Azure deployment, see here
If you only want to use VoxelGPT in the FiftyOne App, then you can simply install it as a plugin:
fiftyone plugins download https://github.com/voxel51/voxelgpt
fiftyone plugins requirements @voxel51/voxelgpt --installIf you want to directly use the voxelgpt module or develop the project
locally, then you'll want to clone repository:
git clone https://github.com/voxel51/voxelgpt
cd voxelgptinstall the requirements:
pip install -r requirements.txtand make the plugin available for use in the FiftyOne App by symlinking it into your plugins directory:
# Symlinks your clone of voxelgpt into your FiftyOne plugins directory
ln -s "$(pwd)" "$(fiftyone config plugins_dir)/voxelgpt"Want to add VoxelGPT to your FiftyOne Teams deployment? You can! Instructions here.
You can use VoxelGPT with your private Azure deployment by setting the following environment variables:
export OPENAI_API_TYPE=azure
export AZURE_OPENAI_ENDPOINT=<azure_endpoint>
export AZURE_OPENAI_KEY=<azure_api_key>
export AZURE_OPENAI_GPT35_DEPLOYMENT_NAME=<gpt35-deployment-name>
export AZURE_OPENAI_GPT4O_DEPLOYMENT_NAME=<gpt4o-deployment-name>
export AZURE_OPENAI_TEXT_EMBEDDING_3_LARGE_DEPLOYMENT_NAME=<embedding-deployment-name>If any of the first three environment variables is not set, VoxelGPT will default to using the OpenAI API. For the last three environment variables, if any of them is not set of the resource is not found, VoxelGPT will default to using the OpenAI API for that specific model.
You can use VoxelGPT in the FiftyOne App by loading any dataset:
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
session = fo.launch_app(dataset)and then either:
+ icon next to the Samples tab and choosing VoxelGPTFor example, try asking the following questions:
Pro tip: use the now keyword to incorporate your previous
prompts as context for your next query!
You can also run VoxelGPT as an
operator by
pressing the ~ key on your keyboard and selecting Ask VoxelGPT from the
list. This will open up a small modal where you can type in your query.
If you've installed locally, you can also directly interact with VoxelGPT via Python.
You can use ask_voxelgpt_interactive() to launch an interactive session where
you can converse with VoxelGPT via input() prompts. Navigate to the directory
where VoxelGPT is located and run:
import fiftyone as fo
import fiftyone.zoo as foz
from voxelgpt import ask_voxelgpt_interactive
dataset = foz.load_zoo_dataset("quickstart")
session = fo.launch_app(dataset)
ask_voxelgpt_interactive(dataset, session=session)Pro tip: use the now keyword to incorporate your previous
prompts as context for your next query.
As usual, you can prompt VoxelGPT with any combination of dataset, documentation, and general computer vision queries. For example, a conversation might look like:
You:
what is the difference between precision and recall?
VoxelGPT:
Precision and recall are two important metrics used to evaluate the
performance of a machine learning model in classification tasks. Precision
measures the proportion of true positive predictions among all positive
predictions made by the model. Recall, on the other hand, measures the
proportion of true positive predictions among all actual positive instances
in the dataset.
In other words, precision is the ability of the model to correctly identify
positive instances, while recall is the ability of the model to identify
all positive instances. A high precision means that the model is making
very few false positive predictions, while a high recall means that the
model is able to identify most of the positive instances in the dataset.
It is important to balance precision and recall when evaluating a model's
performance. A model with high precision but low recall may be missing
important positive instances, while a model with high recall but low
precision may be making too many false positive predictions.
In the context of the FiftyOne library, precision and recall can be
computed using the `fiftyone.utils.metrics` module, which provides
functions for computing various classification metrics. For example, the
`fiftyone.utils.metrics.binary_classification` function can be used to
compute precision and recall for binary classification tasks.
You:
does FiftyOne integrate with Label Studio?
VoxelGPT:
Yes, FiftyOne integrates with Label Studio. The integration allows you to
easily upload your data directly from FiftyOne to Label Studio for labeling.
You can get started with Label Studio through a simple pip install to get a
local server up and running. FiftyOne provides simple setup instructions
that you can use to specify the necessary account credentials and server
endpoint to use. FiftyOne provides an API to create projects, upload data,
define label schemas, and download annotations using Label Studio, all
programmatically in Python. All of the following label types are supported
for image datasets: Classification, Detections, Instance segmentations,
Polygons and polylines, Keypoints, Scalar fields, and Semantic segmentation.
Sources:
- https://docs.voxel51.com/integrations/index.html
- https://docs.voxel51.com/integrations/labelstudio.html
You:
show me predicted airplanes
VoxelGPT:
Identified potential algorithms: evaluation
No evaluation runs found.
Identified potential fields: predictions
Identified potential label classes: {'predictions': ['airplane']}
Identified potential view stages: ['match', 'filter_labels', 'match_labels', 'exclude_labels', 'filter_field']
Okay, I'm going to load dataset.filter_labels("predictions",F("label") == "airplane")
You:
now only show me the first 10 samples
VoxelGPT:
Identified potential algorithms: evaluation
No evaluation runs found.
Identified potential fields: predictions
Identified potential label classes: {'predictions': ['airplane']}
Identified potential view stages: ['match', 'limit', 'limit_labels', 'skip', 'sort_by']
Okay, I'm going to load dataset.match(F("predictions.detections.label").contains("airplane")).limit(10)
You: exit
In interactive mode, VoxelGPT automatically loads any views it creates in the App, and you can access them via your session object:
print(session.view.count("predictions.detections"))You can also use ask_voxelgpt() to prompt VoxelGPT with individual queries:
from voxelgpt import ask_voxelgpt
ask_voxelgpt("Does FiftyOne integrate with CVAT?")Yes, FiftyOne integrates with CVAT, which is an open-source image and video
annotation tool. You can upload your data directly from FiftyOne to CVAT to add or
edit labels. You can use CVAT either through the hosted server at app.cvat.ai or
through a self-hosted server. In either case, FiftyOne provides simple setup
instructions that you can use to specify the necessary account credentials and
server endpoint to use. The tight integration between FiftyOne and CVAT allows
you to curate and explore datasets in FiftyOne and then send off samples or
existing labels for annotation in CVAT with just one line of code. To use CVAT,
you must create an account on a CVAT server. By default, FiftyOne uses app.cvat.ai.
If you haven’t already, go to app.cvat.ai and create an account now. Another option
is to set up CVAT locally and then configure FiftyOne to use your self-hosted server.
A primary benefit of setting up CVAT locally is that you are limited to 10 tasks and
500MB of data with app.cvat.ai.
Sources:
- https://docs.voxel51.com/integrations/cvat.html#examples
- https://docs.voxel51.com/tutorials/cvat_annotation.html#Annotating-Datasets-with-CVAT
- https://docs.voxel51.com/tutorials/cvat_annotation.html#Setup
- https://docs.voxel51.com/integrations/index.html#fiftyone-integrations
When VoxelGPT creates a view in response to your query, it is returned:
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
view = ask_voxelgpt("show me 10 random samples", dataset)Identified potential view stages: ['match', 'limit', 'skip', 'take', 'sort_by']
Okay, I'm going to load dataset.take(10)
VoxelGPT is trained to recognize certain keywords that help it understand your intent:
| Keyword | Meaning |
|---|---|
show/display
|
Tells VoxelGPT that you want it to query your dataset and display the results |
docs/how/FiftyOne
|
Tells VoxelGPT that you want it to query the FiftyOne docs. |
now |
Use your chat history as context to interpret your next query. For example, if you ask "show me images with people" and then ask "now show me the 10 most unique ones", VoxelGPT will understand that you want to show the 10 most unique images with people |
help |
Prints a help message with usage instructions |
reset |
Resets the conversation history |
exit |
Exits interactive Python sessions |
Contributions are welcome! Check out the contributions guide for instructions.
VoxelGPT uses:
VoxelGPT provides limited support for videos, grouped datasets, and 3D media. Basic filtering, querying, and aggregations will still work, but don't expect deep insights into 3D data.
This implementation is based on a limited set of examples, so it may not generalize well to all datasets. The more specific your query, the better the results will be. If you find that the results are not what you expect, please let us know!
If you've made it this far, we'd greatly appreciate if you'd take a moment to check out FiftyOne and give us a star!
FiftyOne is an open source library for building high-quality datasets and computer vision models. It's the engine that powers this project.
Thanks for visiting! ?
If you want join a fast-growing community of engineers, researchers, and practitioners who love visual AI, join the FiftyOne Slack community!
