The Silo Problem

Modern research does not live in a single format. A typical project might involve a stack of PDF papers, a recorded lecture from a conference, a podcast interview with a leading researcher, slides from a collaborator's presentation, and a handful of diagrams saved as images. Each of these contains valuable information, but they all live in separate silos.

If you want to find every mention of a particular concept across all of these materials, you are in trouble. You can search your PDFs for keywords, but that does not cover the lecture video. You might remember that someone discussed the topic in that podcast, but scrubbing through ninety minutes of audio to find the right segment is painful. And good luck searching the content of an image or diagram.

The result is that researchers end up with fragmented knowledge. You know the information exists somewhere in your collection, but finding it means switching between different tools, different search methods, and often just relying on memory.

Cross-modal search is the ability to search across different types of media, or modalities, using a single query. You type a text query and get back results from PDFs, video segments, audio transcripts, and images, all ranked by relevance in a single unified list.

The "cross-modal" part is what makes this different from simply searching each format separately. It is not just running a text search on PDFs and a separate search on transcripts and then combining the lists. Instead, all of these different content types are represented in the same meaning space, so the system can directly compare a text query against a video frame, an audio transcript, or a paragraph from a paper.

The Shared Meaning Space

The core idea behind cross-modal search is multimodal embeddings. In Understanding Semantic Search, we described how text embeddings convert words into numerical representations that capture meaning. Multimodal embeddings extend this idea to other formats.

A multimodal embedding model can take a piece of text, an image, or a video frame and produce the same type of numerical representation for all of them. This means a diagram of mitochondria, a paragraph describing mitochondrial function, and a video frame showing a mitochondria slide in a lecture all end up as nearby points in the same meaning space.

When you search for "mitochondrial structure," the system converts your query into this shared space and then finds the closest matches regardless of their original format. The diagram, the text passage, and the video frame all show up as results because they are all semantically close to your query.

How It Works, Step by Step

Cross-modal search requires several components working together. Here is what happens behind the scenes.

Processing Text (PDFs and Documents)

Text from PDFs is extracted and split into passages. Each passage is converted into an embedding using a multimodal model. For scanned documents, OCR first converts page images to text. Both the text and the page images themselves can be embedded.

Processing Video

Video is handled in two parallel tracks. The visual track extracts keyframes, frames selected at scene changes and regular intervals, and embeds each one as an image. The audio track extracts the audio, runs it through a speech-to-text model to produce a transcript, and then embeds the transcript passages. Speaker diarization identifies who is speaking in each segment, so you can search for what a specific speaker said.

The result is that a one-hour lecture video becomes searchable in three ways: by what is shown on screen (slides, diagrams, demonstrations), by what is said (the spoken content), and by who said it.

Processing Audio

Audio files like podcasts and interviews go through transcription and speaker diarization. The transcript is split into time-stamped segments and embedded. When you search and find a match, you get the exact timestamp so you can jump to that moment.

Processing Images

Images, whether standalone or extracted from documents, are embedded directly by the multimodal model. This means a photograph of a lab setup, a chart, or a hand-drawn diagram are all searchable by describing what they contain.

The Search

When you type a query, it is embedded into the same shared space. The system calculates similarity between your query and every embedded piece of content across all formats. Results are ranked by relevance and presented in a unified list, with each result linking back to the specific page, timestamp, or frame where it came from.

Real Research Scenarios

To make this more concrete, here are some ways cross-modal search changes how you can work with your materials.

Finding a Diagram You Vaguely Remember

You recall seeing a useful diagram about gene regulatory networks, but you cannot remember if it was in a paper, a set of slides, or shown during a recorded talk. Instead of hunting through each source separately, you search "gene regulatory network diagram" and find it immediately, whether it was a figure in a PDF, a slide captured as a video frame, or an image file.

Locating a Spoken Explanation

A colleague mentioned an interesting analogy about protein folding during a recorded seminar. You search "protein folding analogy" and the system finds the exact moment in the recording where they said it, complete with a timestamp you can click to jump to.

Gathering All Evidence on a Topic

You are writing a literature review section on "CRISPR off-target effects." A cross-modal search pulls together relevant paragraphs from multiple papers, a segment from a recorded panel discussion where researchers debated the topic, and a figure from a presentation showing off-target mutation rates. One query, all your evidence.

Searching by Visual Content

You need a specific type of chart, say a survival curve, from somewhere in your collection. You search "Kaplan-Meier survival curve" and the system finds matching figures in your PDFs and matching frames from video presentations, even if the figure caption never used the term "Kaplan-Meier."

How Scholaris Makes This Work Locally

Scholaris implements cross-modal search using a combination of AI models that run entirely on your hardware. The key components are:

Qwen3-VL for multimodal embeddings, the model that puts text, images, and video frames into the same meaning space
Parakeet (or Whisper on AMD hardware) for transcribing audio and video into searchable text
Speaker diarization to identify individual speakers in recordings
FFmpeg for extracting keyframes and audio tracks from video files

All of this processing happens when you convert a file to the SPDF format. The SPDF container stores the original content alongside all the embeddings, transcripts, and extracted frames, making everything instantly searchable.

Because everything runs locally, there are no file size limits imposed by a cloud API, no privacy concerns about uploading sensitive research, and no recurring costs. The tradeoff is that processing large video files takes real time, especially without a dedicated GPU, but once a file is processed, searching across it is nearly instant.

Current Limitations

Cross-modal search is genuinely useful, but it has boundaries worth understanding.

Visual understanding is not perfect. Multimodal models are better at recognizing common visual patterns than highly specialized ones. A general-purpose model will easily match "bar chart" but may struggle to distinguish between specific types of specialized scientific visualizations.

Audio quality matters. Transcription accuracy depends on audio clarity. A well-recorded lecture will produce excellent searchable transcripts. A noisy recording from a crowded conference room will have more errors, which affects search quality for the spoken content.

Processing time is real. Converting a one-hour video into a fully searchable SPDF takes time, roughly fifteen to twenty minutes on a modern GPU. This is a one-time cost per file, but it means cross-modal search requires some upfront investment in processing your collection.

Despite these limitations, the ability to search across all your research materials with a single query is a significant step forward from the siloed approach most researchers are used to. It does not replace careful reading, but it makes sure that when you search, nothing gets left behind because it happened to be in the wrong format.