Introducing
dvps
Advancing Multimodal Foundation Models
DVPS (Diversibus Viis Plurima Solvo, Latin for "Through diverse paths, I solve many issues") builds on the success of large language models by exploring the future of AI through multimodal foundation models.
Unlike today's systems, which learn from representations of the world via text, images, and video, these next-generation models are designed to learn across multiple input channels, including visual, auditory, linguistic, and sensory signals, to gain a grounded understanding of the physical world. This multimodal approach enables them to interpret meaning in parallel, manage complexity, and adapt to real-world scenarios where today's single-modal AI often fails.
What Makes DVPS Models Transformative?
Beyond performance benchmarks, MMFMs offer a new approach to AI development with three key advantages:
Label efficiency
The ability to learn from limited labelled data through transfer learning and few-shot adaptation.
Compute reusability
Leveraging pre-training on large-scale data to reduce the computational cost of developing downstream task models.
Engineering efficiency
Reducing the development effort and expertise required to create specialised models for each new task or domain.
Project Objectives
Developing and disseminating to the research community scientific foundations and methodology;
Releasing impactful open-source assets to the world, for developers to exploit;
Delivering concrete innovations from our use case with medical, social, and industrial benefits.
We will achieve our objective by developing toolkit named AutoDVPS that will be released as an open-source software after being used in three planned application domains (Cardiology, Geo-Intelligence, Language Communication) and tested in two surprise application domains introduced in the second part of the research project to force our methods to generalize beyond the initial assumptions, driving innovation.
Technical details
What we are building
AutoDVPS
An open-source toolkit for automated MMFM design, pre-training, fine-tuning, and modality expansion.
DVPSBench
A comprehensive benchmarking suite specifically designed to evaluate the performance, robustness, and ethical implications of MMFM.
DVPS-FM
An MMFM trained on hundreds of modalities.