Gelaito4: A Sports Media Platform Powered by Collaborative Computer Vision and Generative AI
Introduction
- Gelaito4 is an application prototype designed to enhance the viewing experience of sports events by providing real-time player information, automatic video categorization, and personalized video recommendations.
Demo
Motivation
- When watching sports event videos, viewers often want to quickly know the names or related data of players but can only rely on vague jersey numbers or player characteristics to search online. This not only requires pausing the video, missing exciting moments, but also takes a lot of time to find information.
- Users often spend a lot of time manually categorizing and tagging video content when watching a large number of sports event videos, which is tedious and time-consuming. If the platform categorizes the videos, it requires a lot of manpower, is time-consuming and costly, and the efficiency of video categorization is low.
- Viewers want to quickly grasp the highlights of the game when watching sports event videos, but creating a game highlight requires manpower to watch the entire game, select clips, and edit them, which is time-consuming and labor-intensive.
- When users express interest in a particular topic while watching a video, recommending other similar videos has a high probability of matching the user's preferences, enhancing the viewing experience.
Features
-
Real-time Player Information Display: Viewers can click on a player in the video to see their name, tactical role, and historical data in a pop-up dialog box without having to search for it themselves.
-
Quickly Grasp Game Highlights: Using AI technology, users can quickly grasp the highlights of the game, reducing the time spent searching for videos and simplifying the tedious search process. AI can summarize the game content and mark the hotspots and highlights, saving the company the cost of hiring part-time students.
-
Automatic Categorization: Using AI technology, the system automatically categorizes and tags videos based on different categories, making it easy for users to quickly find related videos and improve the viewing experience.
-
Video Recommendations: By analyzing the user's viewing behavior, the AI recommends other similar videos based on the currently watched video, making it easier for users to discover interesting content and enhance the viewing experience.
Solution and Innovation
-
Real-time Player Information Synchronization: Using AI visual models to recognize player jersey numbers on the field, combined with web crawling technology and GenAI to process and aggregate information, generating real-time and historical data of players.
-
Video Categorization: Using OpenAI's text embedding model to convert video information into vector representations, and categorizing similar videos using the Kmeans algorithm. GenAI analyzes each category set and assigns an appropriate category name, achieving automated video library categorization.
-
Generating Video Summaries: Using GenAI models to capture video audio information, convert it into text files, and summarize the information with GenAI to generate video outlines and summary clips.
-
Generating Video Highlights: Embedding searching of the generated video summary text, comparing it with common keywords of key segments (e.g., "Goal!", "Score~"), and identifying the time points of exciting segments. Expanding the time points before and after, and using the NLTK model to trim and merge multiple expanded video clips, ensuring the clips have complete context.
Deliverables
-
Homepage Design: Users can choose videos of interest on the homepage. Clicking on a video will jump to another page with three main functions:
-
Video Playback and Real-time Player Information: Users can click on players of interest during the game, and a dialog box displaying the player's name and information will pop up immediately.
-
Highlights: The highlights of the game are displayed below the video playback, and users can click on the titles of interest to watch the highlights.
-
Recommended Videos: Based on the user's viewing content, the system recommends related videos of interest.
-
Menu Bar and Video Categorization: The homepage has a menu bar for users to browse videos by category, with categories automatically generated by AI.
How to setup
Prerequisites
- Flutter: https://docs.flutter.dev/get-started/install
Setup Project
git clone https://github.com/deeeelin/Gelaito4.git
- In project folder , run
flutter run -d chrome --web-renderer html
