This is a Multimodal Search demo built with Weaviate, ImageBind/ Vertex AI and Next.js

??‍? Blogpost - Building Multimodal AI in TypeScript

Alt text

? Getting Started

First, clone the project with the command below

git clone https://github.com/weaviate-tutorials/next-multimodal-search-demo

The repository lets you do three things

Run the Next.js Web App.
Run an instance of Weaviate OR create a Weaviate Sandbox
Import images, audio and videos into your Weaviate database.

Running Weaviate (using ImageBind)

Note that the first time you run it, Docker will download ~4.8GB multi2vec-bind Weaviate module, which contains the ImageBind model.

To start the Weaviate instance, run the following command, which will use the docker-compose.yml file.

docker compose up -d

Create a Weaviate Instance (Using Vertex AI)

Create a Weaviate instance on Weaviate Cloud Services as described in this guide

? Create a `.env` file and add the following keys

your Google Vertex API key as GOOGLE_API_KEY (you can get this in your Vertex AI settings)
your Weaviate API key as WEAVIATE_ADMIN_KEY (you can get this in your Weaviate dashboard under sandbox details)
your Weaviate host URL as WEAVIATE_HOST_URL (you can get this in your Weaviate dashboard under sandbox details)

? Importing Data

Before you can import data, add any files to their respective media type in the public/ folder.

With your data in the right folder, run yarn install to install all project dependencies and to import your data into Weaviate and initialize a collection, run: