You HAVE to have libvlc version 1.2 or later for mod_vlc to work. Both for the reason that 
libvlc version 1.2 is LGPL and because there are new api's in 1.2 that are required.

Building and usage:

I used git://git.videolan.org/vlc.git to get the latest version of libvlc. 

On debian I had to install:
sudo apt-get install libavcodec-dev libavutil-dev libavformat-dev liba52-0.7.4-dev

My configure line for libvlc was: 
./configure --prefix=/usr/ --disable-vlc --disable-lua --disable-mad --disable-swscale --disable-postproc --disable-xcb --disable-alsa

Debian unstable now has VLC 2.0 packaged. I am now testing with the packaged version of the library.

The dialplan syntax I use are one of the following:
<action application="playback" data="vlc://http://example.com/vlc.mp3"/>
<action application="playback" data="vlc://mms://example.com/mms_stream"/>
<action application="playback" data="vlc:///usr/local/freeswitch/sounds/vlc.mp3"/>

Development:

If you are having troubles, first change the value of vlc_args in mod_vlc.c. 

The documentation for the libvlc library is located at:
http://www.videolan.org/developers/vlc/doc/doxygen/html/group__libvlc.html