We show that continuous bitstream diffusion (CoBit), originally developed for language modeling, provides a unified framework for joint and conditional multimodal generation using the same training and sampling protocol.
Across image-text, speech-text and protein sequence-structure modeling, models are trained separately from scratch, but each uses a single input projection, transformer backbone, bitwise prediction head and binary score-matching objective shared across its modalities.
Within each setting, a single checkpoint supports joint generation and the conditional directions included in training by changing which parts of the bitstream are observed.
Validation on MNIST-Sum
We first validate the approach on MNIST-Sum, a controlled image-text task pairing raw binary images with equations encoded as text, achieving near-perfect conditional accuracy and high joint consistency without learned tokenizers.
Application at Larger Scale
We then apply the same protocol at larger scale to speech-text and protein sequence-structure modeling.
Trained without text pretraining, a single speech-text model performs speech synthesis, recognition, continuation, and unconditional co-generation.
Jointly generated speech and text achieve 3.5% word error rate between the generated text and automatic transcriptions of the generated speech.
Protein Modeling
In protein modeling, a single CoBit checkpoint supports forward folding, inverse folding, and joint sequence-structure generation.
For jointly generated proteins spanning 100-500 residues, computational refolding of CoBit's sequences yields structures that agree with the generated backbones at a mean self-consistency TM-score of 0.911.
Conclusion
Together, these results show that CoBit provides a common generative framework for substantially different modalities and forms of cross-modal dependence, without introducing a domain-specific objective or sampling procedure.