Wednesday, August 27, 2008

A Free Tutorial on Spectral Sound Processing Using Max/MSP and Jitter

Version FrancaiseWhen I discovered electronic music during the Centre Acanthes 2000/Ircam, my favorite topic was real time sound processing in frequency domain. Hans Tutschku taught the wonders of AudioSculpt in Avignon, before Benjamin Thigpen taught Max/MSP in Helsinki.

Now, the Computer Music Journal just published an article I wrote about spectral sound processing in real time and performance time (whereas real-time treatments are applied on a live sound stream, performance-time treatments are transformations of sound files that are generated during a performance). If you are interested in graphical sound synthesis, phase vocoder, and sonograms (or spectrograms), I hope you will enjoy this tutorial.

The great news is that you can download the article for free on the page of the Computer Music Journal, Issue 32, Volume 3.

Max/MSP/Jitter patches

You can readily apply the described techniques in the development environment Max/MSP/Jitter. For a hands-on approach, make sure you download the patches on my cycling74 share page (link updated 01/12/2010).

By the way, here is Petsi Pies in Cambridge, a coffee-shop in which I spent time writing this article:
petsi pies cambridge

Outline of the article

  • Spectral Sound Processing with Graphical Interaction
    • Non-Real-Time Tools
    • Real-Time Environments
  • An Extended Phase Vocoder
    • FFT Data in Matrices
    • Interacting with a Sonogram
    • Recording
    • Playback
  • Advanced Playback
    • Removing the “Frame Effect”
    • Transient Preservation
    • Segmentation
  • Graphically Based Transformations
    • Direct Transformations
    • Masks
    • Interaction of Sounds
    • Mosaicing
  • Real-Time Freeze and Beyond
    • Real-Time Freeze
    • Automatic Cross-Fade
    • Melody to Harmony

11 comments:

  1. congratulations on the article, jean-francois!

    and yay for petsi pies! i miss that place. :)

    katherine

    ReplyDelete
  2. Very nice tutorials. I tried some of your "Freeze" patches, and I love them. I have also listened to your interactive piece for violin and computer which has very nice time-stretch sound. Is the real-time time-stretch patch example also available for download or is it already included in your tutorial patches ? I am quite interested in it.

    Thanks.

    Alex

    ReplyDelete
  3. Thanks for your comment, Alex.
    The technique used for the long stretched layer in "Plex" is demonstrated in the patch "7-play-blur", without processing of amplitude (I think in the patch, there is a denoiser as example), with a blur size of 8 frames - if I recall, and with [pfft~] parameters FFT size 4096 and overlap 4. The reading speed is something like 0.016667 (the score asks to transform 10 seconds into 6 minutes). In the recording, the other layers, some simple reverb and the space make the whole thing different than what you can get with just the time-stretching, of course. Note that this patch enables to playback the time-stretched sound before the end of the recording: you need to play back the matrix "spectrum" (and get rid of the "newspectrum", only used if you want to graphically process the sonogram).

    ReplyDelete
  4. Thanks for the info.

    In fact, it is still quite difficult for me to understand your tutorial. But I will keep trying since the topics are really interesting.


    I cannot find info about "freeze-tail" patch. How is it different from your other "freeze" patches ?

    By the way, I was wondering if you also put "Melody to Harmony" patch on line.

    Currently, I am trying to build a polyphonic freezor based on your patch. I used three separate pfft subpatches with different matrix name, but it consumes 18% of my CPU (each pfft consumes 6%). Any better solutions ?

    I hope there will be a more detailed tutorial in the future for musicians.

    Thank you so much.

    Alex

    ReplyDelete
  5. I appreciate your interest! First I'll comment on the freeze-tail patch, then on "melody to harmony" in a second comment.
    The freeze-tail patch is not described in the article, but the technique is similar. Its particularity is that either it is freezing (same as freeze), or it is playing back a "sonic tail" after the sound (here, the tail is 8 frames long).
    In other words, there is only one Jitter matrix (called freeze); its content is constantly played back (the stochastic way). It is updated like follows:
    -either constant update (we record constantly the new live sound into it)-> tail function (a blur of the most recent 8 frames is played)
    -or no update-> the sound is frozen on the last 8 frames recorded.

    ReplyDelete
  6. About Melody to Harmony.
    The technique I describe in the article consists in having just one [pfft~]. The idea is to extend the "freeze" patch in a simple way: instead of replacing the matrix with the new 8 frames, replace it with a mix of the old and new frames.
    For instance, you freeze 8 frames from the spectrum of an instrument playing a C. Then, to add an E, you would put into the new matrix something like (50% of the C that is currently frozen + 50% of this new coming E). Note that this is a simple matrix operation in Jitter. This is the formulas 10 and 11 in the article (tricky because when you freeze a third frame, you want to have 33% of each).
    Formulas 12 and 13 are a variation that you should ignore in a first step.

    About Multiple [pfft~] Objects
    Whether in Max/MSP or not, traveling between time domain and spectral domain is an expensive trip. Several cases:
    -you feed the same input to several [pfft~] objects: it would probably be more efficient to make a [pfft~] with 1 input, and maybe several outputs if you need them
    -you chain [pfft~] objects (the output of the first enters the input of the next...): you would rather chain the processes inside a unique [pfft~]
    -you use several [pfft~] and sum/mix their outputs: you could make this sum inside a unique [pfft~] (that could have several inputs if you need).

    Making a Melody to Harmony tutorial is on my list.
    Best,

    ReplyDelete
  7. thank you thank you thank you for sharing this very interesting tutorial. pity some patches are for version 5 only, and I still have to upgrade.. but it's ok, basics of your artichle will keep me busy in the meantime.
    ciao

    antonio

    ReplyDelete
  8. Hello,
    Just to let you know, I did post a "Melody to Harmony" patch. It could also be called "Audio - Multi - Freeze" or as you say "polyphonic freezer". You can download it where all the other Max/MSP/Jitter examples are: http://www.cycling74.com/twiki/bin/view/Share/Jean-francoisCharles
    Enjoy, and let me know if that helps!

    ReplyDelete
  9. Thanks for your tutorials. I could not find the patches though. Apparently the web page has been removed. Would you please let me know how can I get them?

    Pouya

    ReplyDelete
  10. Hi Pouya,
    Thanks for your question.
    Indeed, the "user pages" are not yet functional on the new cycling 74 web site. In the meantime, just e-mail me, and I'll send you the patches.

    ReplyDelete
  11. Congratulations !! very good article. Hope see you again soon. Best

    Francisco

    ReplyDelete