Order Now AdSolution Sign Up | Login » Bits on the Run Sign Up | Login »

Forums

/

Subtitles, captions from the mp4 container when streaming!

14 replies [Last post]
Reply

Hey we are now using mp4 embedded subtitles (closed captions) and the next step for us is to move on top of Wowza streaming server with this.

Now Wowza just released this:
http://www.wowza.com/forums/showthread.php?14701-New-MP4-video-on-demand-playback-support-subtitle-t...

Wrap up:

  • Streaming server is very common for VOD
  • Having JW Player to take the mp4 embedded text from there would be purely awesome!!

Just in case I'll post the code from Wowza forum here if someone is lazy clicking. Point ends here further reading is optional ;)

package com.wowza.wms.plugin.test.module;

import java.util.*;

import com.wowza.util.*;
import com.wowza.wms.module.*;
import com.wowza.wms.amf.*;
import com.wowza.wms.application.*;
import com.wowza.wms.request.*;
import com.wowza.wms.stream.*;
import com.wowza.wms.mediareader.h264.*;
import com.wowza.wms.client.*;
import com.wowza.wms.mediareader.h264.atom.*;

public class ModuleMP4AudioChannelSelector extends ModuleBase
{
public static final String PROPERTY_AUDIOINDEX = "audioindex";
public static final String PROPERTY_VIDEOINDEX = "videoindex";
public static final String PROPERTY_DATAINDEX = "dataindex";
public static final String[] PROPERTY_INDEXES = {PROPERTY_AUDIOINDEX, PROPERTY_VIDEOINDEX, PROPERTY_DATAINDEX};

class MediaReaderListener implements IMediaReaderActionNotify
{
public void onMediaReaderCreate(IMediaReader mediaReader)
{
getLogger().info("ModuleMediaReaderNotify#MediaReaderListener.onMediaReaderCreate");
}

public void onMediaReaderInit(IMediaReader mediaReader, IMediaStream stream)
{
getLogger().info("ModuleMediaReaderNotify#MediaReaderListener.onMediaReaderInit: "+stream.getName());
}

public void onMediaReaderOpen(IMediaReader mediaReader, IMediaStream stream)
{
getLogger().info("ModuleMediaReaderNotify#MediaReaderListener.onMediaReaderOpen: "+stream.getName());

while(true)
{
IClient client = stream.getClient();
if (client == null)
break;

int audioIndex = -1;
int videoIndex = -1;
int dataIndex = -1;

Integer audioIndexObj = (Integer)client.getProperties().getProperty(PROPERTY_AUDIOINDEX);
Integer videoIndexObj = (Integer)client.getProperties().getProperty(PROPERTY_VIDEOINDEX);
Integer dataIndexObj = (Integer)client.getProperties().getProperty(PROPERTY_DATAINDEX);

if (audioIndexObj != null)
audioIndex = audioIndexObj.intValue();
if (videoIndexObj != null)
videoIndex = videoIndexObj.intValue();
if (dataIndexObj != null)
dataIndex = dataIndexObj.intValue();

if (mediaReader instanceof MediaReaderH264)
{
MediaReaderH264 mediaReaderH264 = (MediaReaderH264)mediaReader;

int audioTrackCount = mediaReaderH264.getTrackCountAudio();
for(int i=0;i<audioTrackCount;i++)
{
String langStr = mediaReaderH264.getTrackLanguageAudio(i);
long trackId = mediaReaderH264.getTrackAudioTrackId(i);
QTAtomtrak trackAtom = mediaReaderH264.getTrackAudioAtom(i);
getLogger().info("  audio["+i+"]: trackId:"+trackId+" lang:"+langStr+" more:"+trackAtom.getTkhdAtom().toString());
}

int videoTrackCount = mediaReaderH264.getTrackCountVideo();
for(int i=0;i<videoTrackCount;i++)
{
long trackId = mediaReaderH264.getTrackVideoTrackId(i);
long trackWidth = mediaReaderH264.getTrackVideoWidth(i);
long trackHeight = mediaReaderH264.getTrackVideoHeight(i);
QTAtomtrak trackAtom = mediaReaderH264.getTrackVideoAtom(i);
getLogger().info("  video["+i+"]: trackId:"+trackId+" width:"+trackWidth+" height:"+trackHeight+" more:"+trackAtom.getTkhdAtom().toString());
}

int dataTrackCount = mediaReaderH264.getTrackCountData();
for(int i=0;i<dataTrackCount;i++)
{
String langStr = mediaReaderH264.getTrackLanguageData(i);
long trackId = mediaReaderH264.getTrackDataTrackId(i);
QTAtomtrak trackAtom = mediaReaderH264.getTrackDataAtom(i);
getLogger().info("  data["+i+"]: trackId:"+trackId+" lang:"+langStr+" more:"+trackAtom.getTkhdAtom().toString());
}

if (audioIndex >= 0)
{
getLogger().info("  setTrackIndexAudio: "+audioIndex);
mediaReaderH264.setTrackIndexAudio(audioIndex);
}
if (videoIndex >= 0)
{
getLogger().info("  setTrackIndexVideo: "+videoIndex);
mediaReaderH264.setTrackIndexVideo(videoIndex);
}
if (dataIndex >= 0)
{
getLogger().info("  setTrackIndexData: "+dataIndex);
mediaReaderH264.setTrackIndexData(dataIndex);
}
}

break;
}
}

public void onMediaReaderExtractMetaData(IMediaReader mediaReader, IMediaStream stream)
{
getLogger().info("ModuleMediaReaderNotify#MediaReaderListener.onMediaReaderExtractMetaData: "+stream.getName());
}

public void onMediaReaderClose(IMediaReader mediaReader, IMediaStream stream)
{
getLogger().info("ModuleMediaReaderNotify#MediaReaderListener.onMediaReaderClose: "+stream.getName());
}
}

public void onAppStart(IApplicationInstance appInstance)
{
appInstance.addMediaReaderListener(new MediaReaderListener());
}

public void play(IClient client, RequestFunction function, AMFDataList params)
{
String streamName = params.getString(PARAM1);

getLogger().info("ModuleMediaReaderNotify.play: "+streamName);

if (streamName != null)
{
int qindex = streamName.indexOf("?");
if (qindex >= 0)
{
String queryStr = streamName.substring(qindex+1);
Map<String, String> queryParams = HTTPUtils.splitQueryStr(queryStr);

for(int i=0;i<PROPERTY_INDEXES.length;i++)
{
String indexStr = PROPERTY_INDEXES[i];
if (queryParams.containsKey(indexStr))
{
int index = -1;
try
{
index = Integer.parseInt(queryParams.get(indexStr));
}
catch(Exception e)
{
}
if (index >= 0)
{
client.getProperties().setProperty(indexStr, new Integer(index));
getLogger().info("  "+indexStr+": "+index);
}
}
}
}
}

invokePrevious(client, function, params);
}

Reply

If this feature works the same as reading built-in MP4 Text Tracks, it will simply work out of the box with the captions plugin.

Reply

We, will definately test that out, haven't yet. I was looking at the actionscript source, but could not make 100% sense of how the MP4 Text Tracks were read. I'll let you know if this would work allready with newest captions plugin and if there is any problems.

Reply

OK,

The textdata is read by listening to META events. They contain either a "texttracks" object, or the event name is actually "onTextData".

Reply

Hello again, we've tried this now with Wowza Streaming Server and with it's newest version. We have not yet managed to use the embedded subtitles in MP4 files, streamed over RTMP.

Do you happen to have a test case for RTMP Streaming where text tracks are read from the file. :) Or would you recommend some program that (open source/commercial program) that could snoop/snif the data from the stream?

Here's the code we worked with:

<div id="container">Ups. Get yourself <a href="http://www.macromedia.com/go/getflashplayer">Flash player</a> to enjoy the videos.</div>
  <script type="text/javascript" src="$link.setRelative("site/jw_player/swfobject_5_8.js")"></script>
  <script type="text/javascript">
var s1 = new SWFObject('$link.setRelative("site/jw_player/jw_player_5_8.swf")','mediaplayer','640','360','9');
s1.addParam('allowfullscreen','true');
s1.addVariable('width','640');
s1.addVariable('height','360');
s1.addVariable('controlbar','over');
s1.addVariable('skin','$link.setRelative("site/---/skins/--/--.xml")');
s1.addVariable('logo.file','$link.setRelative("--/---/--_logo.png")');
s1.addVariable('logo.position','top-left');

s1.addVariable('file','mp4:$program.series.getFname()/$program.getFname()');
        s1.addVariable('streamer','rtmp://---.--/--')

s1.addVariable('plugins','captions, timeslidertooltipplugin-1');

s1.addVariable('plugins','captions');

s1.addVariable('image','$program.getImageURL()');
s1.addVariable('autostart','false');
s1.write('container');
</script>

Reply

We don't have a test for RTMP + inline captions. But you can grab one of our test videos and drop it on an RTMP server to see if it works. E.g. this one:

http://content.bitsontherun.com/videos/aytCR4cx-393434.mp4

Reply

Well now with this setup the captions work, but not multiple captions:

<script type="text/javascript" src="player/jwplayer_5_8.js"></script>

<div id="container">Loading the player ...</div>

<script type="text/javascript">
jwplayer('container').setup({
'flashplayer': 'player/jwplayer_5_8.swf',
'provider': 'rtmp',
'streamer': 'rtmp://192.168.5.165/vod',
'file': 'israel_puhuu-121-w.mp4',
'height': '360',
'width': '640',
'controlbar': 'over',
'autostart': 'false',      
'plugins':
{
'captions-2.0': {
'fontFamily': 'Arial',
'fontSize': '18',
'fontWeight': 'bold',
'label': 'Finnish',
'labels':'Eesti, Suomi'
}
},    
'skin': 'player/skins/oma/oma.zip'
});
</script>

Reply

I mean player finds the multiple captions but, doesn't know how to resolve those.

What happens exactly is this:
1. I have 3 languages in video
2. I start playback.
3. It shows the menu wit 3 langs with same labels. In this case English.
4. However it plays only the first subtitle track from mp4 which is selected from the second option in the menu. Super weird.
5.1st and 2nd lang option display nothing.

Reply

Have you tried playing the video not using RTMP? Do the captions then show up fine? If so, the RTMP server is somehow destroying the text data.

If the captions also work strangely when using HTTP, are you able to play the MP4 in QuickTime X or on an iPhone/iPad?

*) If the captions don't show up there either, there's a problem with the file. In that case I recommend you re-author the MP4 file, e.g. by starting with the audio/video and adding the captions using the free "Handbrake" tool.

*) If the captions show up fine there, the problem is likely in our plugin. In that case, can you provide me with a download link to your video, so I can test/fix?

Reply

Yes, we have tried them without RTMP. They work fine. All of the files I've tested are in production, as progressive http downloads and they work fine. All the labels and subtitles are fine.

We make the files with mp4Box, but I also now tested to make a file with Handbrake amd the issue is the same. I've acctually narrowed the issue a bit.

Here is addiotional info from testing:

1. It actually does not matter what are the languages inside. Player will always show the labels as Egnlish.
2. Player will always show the first subfile inside the mp4.
3. Strangely the "first" subtitle, seems to be selected from the second language name in the menu(which is allways english no matter what language is set in the file). As I pointed before it doesn't matter if there are more then two languages the symptoms are the same.

Here are links to one test videofile(1st made with MP4Box 2nd with HandBrake)
1. http://vod-2.tv7.fi/vod2/israel_puhuu/israel_puhuu-135-w.MP4
2. http://helsinki.tv7.fi/public/israel_puhuu-135-w-iphone.mp4

Reply

IMPORTANT!

Hey, I tested this a bit more and found actually that PS3 doesn't work with captions plugin version 2.4, but according to my tests works with version 2.0, 2.1, 2.2 and 2.3.

Reply

Sorry wrong thread was about to post to this Caption plugin not working on PS3

Reply

Tell me if your test results have any success, otherwise it might also be that Wowza 3.0.3 messes up the text data inside mp4.

Reply

Since the files work fine when delivering as progressive download, I'm afraid the FMS servers are breaking the text data. I'm not sure what can be done to fix this, but perhaps there's more info on Adobe's website here?

For progressive downloads, note your videos will play much better if your MP4 metadata is placed at the start of the files. This can be done with a simple tool like QT-faststart. Handbrake has this option too, as a flag called "Optimized for web playback".

Reply

Thanks for the tips! We will have to look on to that regarding the metadata.

We generate the files with an automated script, utilizing MP4 Box. So I'll have to look on to the MP4 Box for an option to set the metadata at the beginning.

PS. I noticed that Wowza supports flv files with multiple captions inside but for some reason I can not understan, not on the mp4 files, which happen to work on the iOS devices :/

Post new comment

  • Allowed HTML tags: <code> <blockquote> <em> <strong> <strike> <ul> <li> <ol>
  • You may post code using <code>...</code> .
  • Lines and paragraphs break automatically.
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options