Video Search with Mixpeek Multimodal Embeddings
Implement video search with the Mixpeek Multimodal Embed API and Supabase Vector.
The Mixpeek Embed API allows you to generate embeddings for various types of content, including videos and text. You can use these embeddings for:
- Text-to-Video / Video-To-Text / Video-to-Video / Text-to-Text Search
- Fine-tuning on your own video and text data
This guide demonstrates how to implement video search using Mixpeek Embed for video processing and embedding, and Supabase Vector for storing and querying embeddings.
You can find the full application code as a Python Poetry project on GitHub.
Create a new Python project with Poetry
Poetry provides packaging and dependency management for Python. If you haven't already, install poetry via pip:
_10pip install poetry
Then initialize a new project:
_10poetry new video-search
Setup Supabase project
If you haven't already, install the Supabase CLI, then initialize Supabase in the root of your newly created poetry project:
_10supabase init
Next, start your local Supabase stack:
_10supabase start
This will start up the Supabase stack locally and print out a bunch of environment details, including your local DB URL
. Make a note of that for later use.
Install the dependencies
Add the following dependencies to your project:
_10poetry add supabase mixpeek
Import the necessary dependencies
At the top of your main Python script, import the dependencies and store your environment variables:
_10from supabase import create_client, Client_10from mixpeek import Mixpeek_10import os_10_10SUPABASE_URL = os.getenv("SUPABASE_URL")_10SUPABASE_KEY = os.getenv("SUPABASE_API_KEY")_10MIXPEEK_API_KEY = os.getenv("MIXPEEK_API_KEY")
Create embeddings for your videos
Next, create a seed
method, which will create a new Supabase table, generate embeddings for your video chunks, and insert the embeddings into your database:
_46def seed():_46 # Initialize Supabase and Mixpeek clients_46 supabase: Client = create_client(SUPABASE_URL, SUPABASE_KEY)_46 mixpeek = Mixpeek(MIXPEEK_API_KEY)_46_46 # Create a table for storing video chunk embeddings_46 supabase.table("video_chunks").create({_46 "id": "text",_46 "start_time": "float8",_46 "end_time": "float8",_46 "embedding": "vector(768)",_46 "metadata": "jsonb"_46 })_46_46 # Process and embed video_46 video_url = "https://example.com/your_video.mp4"_46 processed_chunks = mixpeek.tools.video.process(_46 video_source=video_url,_46 chunk_interval=1, # 1 second intervals_46 resolution=[720, 1280]_46 )_46_46 for chunk in processed_chunks:_46 print(f"Processing video chunk: {chunk['start_time']}")_46_46 # Generate embedding using Mixpeek_46 embed_response = mixpeek.embed.video(_46 model_id="vuse-generic-v1",_46 input=chunk['base64_chunk'],_46 input_type="base64"_46 )_46_46 # Insert into Supabase_46 supabase.table("video_chunks").insert({_46 "id": f"chunk_{chunk['start_time']}",_46 "start_time": chunk["start_time"],_46 "end_time": chunk["end_time"],_46 "embedding": embed_response['embedding'],_46 "metadata": {"video_url": video_url}_46 }).execute()_46_46 print("Video processed and embeddings inserted")_46_46 # Create index for fast search performance_46 supabase.query("CREATE INDEX ON video_chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100)").execute()_46 print("Created index")
Add this method as a script in your pyproject.toml
file:
_10[tool.poetry.scripts]_10seed = "video_search.main:seed"_10search = "video_search.main:search"
After activating the virtual environment with poetry shell
, you can now run your seed script via poetry run seed
. You can inspect the generated embeddings in your local database by visiting the local Supabase dashboard at localhost:54323.
Perform a video search from a text query
With Supabase Vector, you can query your embeddings. You can use either a video clip as search input or alternatively, you can generate an embedding from a string input and use that as the query input:
_32def search():_32 # Initialize Supabase and Mixpeek clients_32 supabase: Client = create_client(SUPABASE_URL, SUPABASE_KEY)_32 mixpeek = Mixpeek(MIXPEEK_API_KEY)_32_32 # Generate embedding for text query_32 query_string = "a car chase scene"_32 text_emb = mixpeek.embed.video(_32 model_id="vuse-generic-v1",_32 input=query_string,_32 input_type="text"_32 )_32_32 # Query the collection_32 results = supabase.rpc(_32 'match_video_chunks',_32 {_32 'query_embedding': text_emb['embedding'],_32 'match_threshold': 0.8,_32 'match_count': 5_32 }_32 ).execute()_32_32 # Display the results_32 if results.data:_32 for result in results.data:_32 print(f"Matched chunk from {result['start_time']} to {result['end_time']} seconds")_32 print(f"Video URL: {result['metadata']['video_url']}")_32 print(f"Similarity: {result['similarity']}")_32 print("---")_32 else:_32 print("No matching video chunks found")
This query will return the top 5 most similar video chunks from your database.
You can now test it out by running poetry run search
, and you will be presented with the most relevant video chunks to the query "a car chase scene".
Conclusion
With just a couple of Python scripts, you are able to implement video search as well as reverse video search using Mixpeek Embed and Supabase Vector. This approach allows for powerful semantic search capabilities that can be integrated into various applications, enabling you to search through video content using both text and video queries.