Multimedia Regular Expressions
19 Jun 2006What would regular expressions for non-text look like? I think many of the quantifiers would be similar, but the notions of “character classes” and what goes in an atom would be totally different.
Two ways of specifying bits to match for audio could be score-oriented or sample-oriented. Score-oriented “classes” could match patterns consisting of particular notes, notes in particular keys, with particular duration, particular chords, parts of particular chords, and so on. These could be built up into multi-note patterns. Other notation would look for other parts of the score – tempo, all those Italian words that describe how to play the notes, etc.
Sample-oriented classes could match particular samples (or subsets of samples), certain rhythms, melodies, etc. With fancy enough signal processing, the classes could match against particular words being sung, a particular singer doing it, or slices that “sound like” some other slice (where “sound like” is implemented by some pluggable algorithm.
A text-based notation might be doable for the score-oriented classes and some of the sample-oriented classes but to fully extend the analogy the expression of the audio-regex could/should be with audio (or visual representations of the audio) as well – “Find the five bars on either side of any music that sounds like this” where this is some scoring or sample that’s either taken literally or has been approrpriately “expression-ified” (with audio processing filters?), or, alternatively, is a visual representation of a score that has been similarly transformed.
Extending this to video is an expansion of the “sample-oriented” audio expression language, but with visual idioms in addition to the auditory ones – some possible classes to match against are things such as frames with human faces in them, frames with a particular face, indoor frames, outdoor frames, frames that are predominantly a particular color, frames from the work identified on IMDB by ID XXX, and so on.