Streaming and (very) inefficient bandwidth usage

27 posts | return to the Bug Reports forum | get the rss feed for this thread

Jan. 25, 2008JohnDoe

Hi,

I think I found another bug or extremely inefficient behaviour of the player.

Here's the scenario:

I'm using lighttpd http-streaming for flv videos.

1. I start streaming a video.

2. I wait until it gets downloaded. (the playback data-rate is lower than my bandwidth, so the download is always ahead of the playback, and it goes until the video is fully downloaded).

3. I press Pause.

4. Being paused, I can scrub freely from position to position, and no further downloads (http requests) are sent to server. Everything works as expected.

5a. With the ver. 3.14 of the player, I pick one point in the video while still paused and then press "Play", and here the player sends another http request and starts downloading again (from that point on and again until the end of the file). This is bad.

5b. I made a little test also with the ver. 3.12: still *while paused*, and under the same conditions (from point 1. to 4.), pressing play again DOESN'T send another http request, so apparently ver 3.12 behaves more intelligently than 3.14: it has downloaded the video and it doesn’t need to request it again.

5c. Now, holding the same scenario from points 1. to 4., but instead of pressing pause and then scrubbing, I simply try to scrub without pause, and I noticed that it always sends an http request (both ver. 3.12 and ver 3.14).


This behaviour is extremely bandwidth inefficient: the player carries on downloading until the end of the file, but whenever you need to jump to another position (even very near one), it simply THROWS AWAY all the downloaded parts and starts downloading it again f(from that point on).

As explained before, on this regard the ver. 3.14 is even worse than 3.12, since 3.12 at least reuses the downloaded parts IF YOU FORST PAUSE IT and then jump forward, while 3.14 makes http request for every press of play or scrub, whatsoever.


In general, in this context there are two strategies for rational bandwidth usage:

A. one is to put to sleep the downloading process as soon as some kind of buffer is full (this can also be done server side, and some PHP-streaming scripts do it, look for example Xmoov’s PHP: http://xmoov.com/xmoov-php/source/ (see bandwidth limiting code chunk and the usage of usleep() function). By the way, Lighttpd’s mod_flv_streaming doesn’t behave that intelligently, and could in the future have a configuration option for that kind of behaviour; but that’s another story.

B. The secont possible strategy is to carry on downloading (until the end), but then reuse the downloaded parts for scrubbing, without the need for further download as long as the required position is within the downloaded range. This is ho YouTube does it (try to jump to a position and look at the red bar).

None of this seems to be the case with the behaviour of JW FLV Media Player.

Theoretically, it seems to me that there could be a very simple fix: just pick the current 3.12 behaviour, and when an user scrubs *during* playback you internally first pause it, then jump to the position, and then start the playback again.

Cheers.

P.S. By the way, this is not the only case of 3.12 behaving better than 3.14: you can have a look at this one: http://www.jeroenwijering.com/?thread=8629

Jan. 25, 2008JaneDoe

This is the exact same behavior as all RTMP streaming from servers such as FMS and Red5.

Viewing a 5MB video can easily create total traffic of 10-15MB per view if the user scrubs much.

(As an aside, you are talking about traffic, NOT bandwidth.)

This is why I don't recommend using RTMP streaming.

Yet there are many video sites using RTMP streaming for millions of users.

Go figure...

Jan. 25, 2008JohnDoe

Hi JaneDoe,

I understand perfectly your point, so I will try to be as precise and detailed as possible in explaing that I was attempting to making a different point.

It is true that RTMP behaves in a similar way. Actually, there is some difference with well configured and managed systems, as they should stop downloading when some sort of buffer is reached, not just download the file all through the end.

As I said, this is also how some PHP-based and http-streaming solutions work (for example Xmoov's way I was mentioning.) However, that is not how http-based lighttpd streaming works, and especially that is not how for example YouTube or Goodle Video work. You can make that test: go to YouTube, get a video, and jump to some arbitrary point: it will start downloading the video, and if you have enough bandwidth it will download through the end before the playback finishes, but then if you jump to another forward position, it WILL NOT start to download again and will use client cache instead – look carefully the red progress bar to notice what I am talking about. Also, try to jump to different random positions back and forward and look how it behaves.
(BTW, under typical and statistically dominant user behaviour, THIS IS a reasonable and traffic-saving strategy and that's why many video sites are using it (I doubt that otherwise YouTube or Google would use it, being the two services very traffic-intensive and traffic-expensive services.))

Now, my point was not only that JW Player should behave that way, but that IT ACTUALLY BEHAVES that way in some circumstances, namely when: (i) you use JW player ver. 3.12 (ii) you start a video, (ii) you PAUSE IT (and wait for some download, better until the end), (this is a CRUCIAL STEP!), (iii) you scrub to a forward position (drag the slider to some point), (iv) and then un-pause (play) it again. Keep in mind that I’m always talking about a a lighttpd streaming scenario, not about progressive download. Now, under the above circumstances the version 3.12 behaves wonderfully, and that behaviour is traffic-wise and youtube-like.

But, if you use ver. 3.14 (where every “play/un-pause” button press seems to correspond to a http-request) , or if you just jump to a different forward position WITHOUT PAUSING the video first, the player will THROW AWAY useful data and will start to download it again.

I understand all the client and server side issues implied here, and that is precisely the reason why I was suggesting a hypothetical simple fix, which could make JW player behave exactly as YouTube and Google players.

If I was not clear enough and if you want, I could set up an example proof-of-concept on my server, and you can see it yourself.

I hope this can be taken into consideration for the next release, as it may look like one little and too specialised issue, but it really is of a crucial importance for massive and very traffic-intensive video services.

Cheers.

Jan. 25, 2008YouTube

YouTube and Google are using lighttpd. Look at the headers and you will see:Server: lighttpd/1.4.18

Since their videos seem to play in the JW Media Player with the same scrubbing and caching behaviors as in the Youtube/Google players (I just tested extensively with JW MP v3.14), perhaps it's something in your lighttpd configuration or maybe YouTube/Google have "tweaked" things a bit?

I'm not seeing the JW MP throw away any data, either during the download, with/without scrubbing, or after the download is complete and I then scrub throughout the timeline.

Jan. 25, 2008JohnDoe

Hi YouTube,

Are you sure YouTube and Google are using JW Media Player?

I've tried different lighttpd configurations, and I'm quite sure it is related to the player. As I explained, I manage to reproduce youtube-like behaviour with ver 3.12 and with "pause-scrub-play" procedure, but not "only-scrub-forward" procedure. So if Youtube and Google players are based on JW, my hypothesis is that they tweaked it, as it seems to me quite easy thing to do.

I'm curious: what kind of tests did you do with JW 3.14? Can you tell me where? Can I give it a try?

Thanks in advance.

(BTW, it would be useful in JW to have that colored-bar behaviour actually showing the download progress, like Google and YouTube players.)

Jan. 25, 2008YouTube

Are you sure YouTube and Google are using JW Media Player?No, they are using their own player. I don't see anywhere that I stated that they were using the JW Media Player.

The tests that I performed were simply using a playlist of YouTube/Google videos in the JW Media Player v3.14. The video files come directly from YouTube/Google lighttpd servers.

Meanwhile, I performed another experiment where I used:s1.addVariable('streamscript', 'lighttpd'); in the JW Media Player code to play the YouTube/Google videos. They exhibited the negative caching/playing behaviors that you mentioned in your first post.

So... I'm thinking, maybe you should try playing your videos from your lighttpd server WITHOUT thes1.addVariable('streamscript', 'lighttpd');code.

Jan. 25, 2008JohnDoe

Hi YouTube,

(I misunderstood your answer, indeed you didn’t state YouTube and Google were using JW; sorry about that).

Now, I tried to play my videos WITHOUT the lighttpd streamscript variable, but that switches the player from http-streaming to progressive download mode, and therefore doesn’t allow me to scrub to any point of the video, but only within the downloaded range.

I also made the test playing YouTube videos in JW player as you suggested, and that confirmed me my suspicions were correct, and that indeed JW exhibits that extremely inefficient behaviour as I explained before.

Jan. 26, 2008JohnDoe

Making some additional testing with other available players, I discovered that FLV-Scrubber (http://topfstedt.de/weblog/?page_id=208) behaves correctly and traffic-wise.

Jan. 29, 2008ScottR

JohnDoe I agree entirely with all points raised in this thread. I always thought this was a limitation of "fake streaming" but if the JW player could show what is buffered and then allow you to skip to the buffered section without restarting, that would be an amazing fix both bandwidth wise (much less usage) and speed wise (no rebuffering / server request).

Is this possible? I thought fake streaming meant JW player doesn't know how much of the video has loaded and therefore has to reload every scrub?

Jan. 30, 2008JohnDoe

Hi ScottR,

Thank you for your feedback.

Indeed, as you correctly point out, "fake streaming" doesn't at all mean no-buffering, and that can easily be seen in YouTube and Google Video (and hey, if they do it this way, they must have made their calculations well, sice you should have in mind YouTube's traffic running costs: here's a somewhat old but still striking estimation:
http://willy.boerland.com/myblog/youtube_bandwidth_usage_25_petabytes_per_month ).

Furthermore, as I explained before, some other free players, like FLV-Scrubber, supports that kind of behaviour.

I'm not an expert in Flash development, but my guess is that this is something that shoudn't be too difficut to implement (and it is of a crucial importance for any medium to large-scale deployment of JW-based httpd streaming).

Jan. 30, 2008YouTube

Indeed, as you correctly point out, "fake streaming" doesn't at all mean no-buffering, and that can easily be seen in YouTube and Google Video (and hey, if they do it this way, they must have made their calculations well, sice you should have in mind YouTube's traffic running costs:

I don't think so...

Every time that I scrub in the YouTube player, it initiates a download. After multiple scrubs, there are many get_video[n] files in the browser's cache. For a 2,201KB video, this is what is in the cache:get_video[1]     485KB
get_video[1]   1,467KB
get_video[1]     857KB
get_video[2]   1,846KB
get_video[1]   2,201KBlooks like total traffic of 6,856KB and I can see that much traffic on my traffic monitor also. That's TRIPLE the size of the video file.

Jan. 30, 2008JohnDoe

YouTube,

It depends on how and where you scrub to, and what portion has already been downloaded/buffered by the player.

I invite you to try this:

1. Go to a YouTube video
2. Put it in pause and WAIT for a while until it downloads few seconds/minutes ahead (look at the red progress bar!)
3. Then scrub to a position WITHIN the dowloaded red area.

If yuo do this, you will notice that no additional http requests nor downloading take place. (If the player hasn't yet downloaded the whole video, it will simply carry on with that.)

On the other hand, if you scrub to a position which is BEFORE the point you jumped on sometime earlier, or if you scrub to a position AFTER the buffered area (BEYOND the red progress bar), only then YouTube looses the buffer and starts downloading. Again, the red progress bar tells it all!

It is true that you can have YouTube generate more traffic that the actual length of the movies. It can be double, triple, quadruple, or just any possible multiple of it, if that's what you want.
In fact, I was not saying that YouTube is /perfectly/ optimal, but only that it is reasonably efficient with statistically predominand users behavour (people aren't scrubbing-junkies randomly jumping at random positions just for the sake of it, but they do the scrubbing for some purpose, and that gives birth to statistically more stable patterns of behaviour) and taking into account the size of Youtube videos.


However, all my previous points stand, as the JW Player doesn't do the kind of traffic-wise buffering as that of YouTube, I descrbed before (call it the red-progress-bar-way).

Stay well.

Jan. 30, 2008YouTube

Actually, YouTube is probably a poor example to use because most of their videos are short and users probably don't scrub forward of the download point too much. If you haven't watched the video once, why would you be scrubbing forward.

And once you have watched the video, the scrubbing takes place locally, out of the browser's cache.

However, many of the videos that I see user's of the JW Media Player making available, are quite long and are frequently accompanied by other material that may encourage the user to scrub forward of the download point (educational material, religious sermons, etc.). Of course, if it's a progressive download, they can't scrub forward of the download point. However, using streamscript with the PHP script or lightppd, they can scrub forward and then may later scrub back to the beginning.

The ideal situation would be RTMP supporting a "buffer full" command which would then stop the stream until the buffer was partially empty. The other almost ideal solution is to break the long video into chapters in a playlist and only request the chunk for the chapter. The video file itself is NOT split, the requests are made for only a chunk of the full file.

That is implemented in the chapters scripts in this thread: [url=http://www.jeroenwijering.com/?thread=6080#msg33286]Chapterizer - Town Meeting[/url]. There are two advantages to this method: one, only the chapter chunk is downloaded, limiting traffic unless the user selects another chapter, and two, if the user selects a previously viewed chapter, it comes out of the local cache instead of being downloaded again.

Jan. 31, 2008JohnDoe

Yes YouTube, this is an interesting and useful possibility. Thank you for pointing me at that thread, I will surely play a bit with that "chapterization script".

However, I still strongly think JW Player should support the YouTube-like behaviour.

Cheers.

Jan. 31, 2008ScottR

Totally, my analysis of the Flash code is that the lighttpd streaming was to an extent hacked in (sorry for the harsh word, your product rocks and is amazing), and just loads the full bar because that enables the user to click where they want. After all it is much easier to just send through a request every scrub instead of calculating the buffer and using it to determining how the scrub will work

Hopefully this is something that will be changed, because it will certainly make a massive difference functionality and performance wise, if users can actually accurately watch the buffer and scrub to it without a wait.

Feb. 05, 2008JeroenW

Yes, I consider this a bug too, and it'll be fixed with the next update. Like with YouTube, the next update will show exactly which part of the video is in memory.

Feb. 23, 2008JohnDoe

Hi JeroenW,

I noticed that this issue hasn't been addressed in the 3.15.

I particular the player still doesn't make any use of the buffered parts nor it shows it somehow in the download bar.

Is this issue still on the road map?

Cheers.

Feb. 23, 2008JeroenW

One issue has been addressed - if you pause and replay the movie it won't rebuffer. The other parts indeed aren't done yet. I'll add that to the 4.0 player.

Jul. 03, 2008Thomas

The inefficient bandwidth usage has not been fully corrected in version 4.0.

When I hit the pause button, I see that the buffer still gets filled up. However if I do scrub in a portion of the downloaded video it doesn't make a new request to the server, but anyway it was still being downloaded, so it didn't change much.

I also noticed that if I scrub outside of the buffer, then the previous buffer gets cleared.

Jul. 18, 2008RogeR

Yeah, I noticed the issue is still not fixed also.. JW is such a great player, but I wish this streaming issue was fixed.

Aug. 20, 2008cyber

Likewise here.

this is so bugging me as well, i've finally got an rtmp server and the buffering makes the video unbearable to watch... Does it really work like this? ... i mean i have a bufferlength set to 5, whenever i play the video it waits for 5 seconds then after it got loaded 100% it would play for 5 secs while the load % is still there decreasing until 0% then loads again, rinse and repeat. shouldn't it load continuously? It seems to only start loading again after it got back to 0%.

Load until 100%... Plays (while % decreases as well)... then when the % goes back to 0 it's only then that it starts loading again... this is quite annoying, is there any fix for this?

i've paid for the unlimited website license btw and i hope this gets fix soon as im running the player on a video-on-demand pay per minute business of mine.

Aug. 20, 2008kLink

Temporarily, you can get rid of the annoying loading icon by setting bufferlength=null in your player code.

Yeah, we all hope this gets fixed soon...

Aug. 21, 2008JeroenW

I think there's two issues here:

1) The behaviour of HTTP streaming: downloading will continue even while the video is paused and when seeking to nondownloaded parts the previous buffer gets cleared.

This is really how HTTP download works. Check Youtube, or other players, they'll do the same. It's practically impossible to have multiple downloaded parts in buffer, and the loading-while-paused is actually handy for people who have a slow connection but do want to see the HiQ video.

2) The showing of a buffericon with RTMP servers.

This is a bg and shouldn't happen. I've never encountered this though. cyber, kLink, can you give more info? Are these videos very short? What RTMP server do you use? Are the videos very HiQ?

Aug. 21, 2008kLink

@JeroenW,

I see the buffering icon quite frequently when using RTMP from my own Red5 server over a 1Gb LAN and I see it on many RTMP streams from FMS. The videos can be short (10s) or long (2hr).

I also have noticed that recent versions like v4.1.60 will give a 2032 Security Sandbox error if the filename doesn't have an extension and you don't specify a type of 'video'.

Try this RTMP stream:
streamer: 'rtmp://flv.williams.edu/streams/media',
file: 'obrecht/obrechtmp4s/wny',
type: 'video',

I'll collect some more RTMP streams that buffer frequently and post them.

Aug. 21, 2008cyber

I see, pausing will not buffer the next parts like youtube? how youtube streams is excellent, perhaps a feature like this can be implemented? it will be so helpful for users with slow connection especially since most people are more familiar with how youtube streams.

I see the buffering icon quite a lot as well, although this is only true for videos that are very big, i have the bufferlength set to 5... I don't know but im annoyed at the buffering icon's frequency:
1. Buffer Load to 100% (5 seconds)
2. Movie Plays for 5 seconds (While playing im still seeing the buffering percentage, but it is going down until 0%, maybe it's just me but whenever the percentage goes down to 0% that's the time the 5 second buffered has already reached the last second as well.)
3. After the buffered 5 second is finished it loads until 100% again and rinse and repeat... the buffer icon is there forever.

Try loading a big movie and perhaps you can recreate this. This could be answered with the feature: buffering while paused, since everyone is used to how youtube works.

btw,
JeroenW, your player is so darn excellent and great... It's a dream come true, keep up the good work! This is the only feature missing, hope it's possible =)

Sep. 11, 2008JeroenW

The 2032 security error is easily explained: the player doesn't know which filetype to expect and presumes you load a playlist. Hence, please add .flv

As to the buffering, I'm afraid this is simply how the RTMP system works. Data is loaded until the buffer is filled and then the loading stops. YouTube (and HTTP streaming) indeed continues loading even when paused. I am working on two enhancements that might fix this:

1. Dual buffering: once the initial buffer is filled, the bufferlength is set at a higher value.
2. Bandwidth switching: next to the high-quality video, you can add a Lo-Q one and switch between the both depending on your bandwidth.

Jan. 27, 2009onerob

Wasn't JohnDoe's original point that the JW Player would send a new http request even when clicking within a buffered segment of video?

Has this been fixed?

Add a reaction

You can also return to the category or try this search for related threads.

Your name *

Supported codes: submit