Intermittent starts with Amazon S3 Expiring URL's

@Kevin,

Have you read through this thread: [url=http://www.jeroenwijering.com/?thread=6625]Amazon S3 FLV files not being played[/url]

And are you using something like this:

in your player code?

I noticed the test URL that MyPoems was using wasn't valid. I guess that's because it was an expiring link. Can you give me a link to a page that generates those links so I can do some testing?

Hey, Will...actually, we have and it helped us get the videos going (btw, thanks). Unfortunately, the playback is intermittent. In other words, sometimes it cranks right up and sometimes it spins for a fraction of a second and goes right back to the play button. If you click it again, it just spins indefinitely. This happens indiscriminately. Sometimes videos work and sometimes they don't. Any thoughts?

Thanks again...

@Kevin,

Well, I pretty much did a Brain Dump™ :d on that other thread, but if you can give me some valid links, I can run a packet sniffer and see what is happening when the player makes a request.

JavaScript & Flash really trash URLs, so seeing what kind of request they are actually making to the server, then modifying the code to get them to send a valid request is the only way to overcome this kind of a problem. :|

Is there a way I can contact you privately?

Will...you can contact me at: kevin_as123-at-yahoo.com.

We are seeing a similar problem..

It seems the JW FLV Player is decoding the URLEncoded file flashvar parameters wrongly in certain cases.

Whenever the Amazon S3 generated URL has percent-sign as a part of the last "signature" parameter value (and this seems a very random occurence), it seems that it has trouble decoding these urls properly, even though the percent-signs themselves have been properly encoded before adding it the file flash-variable.

Example:

Original S3 URL:
http://s3.amazonaws.com/1EFVEK0RZ934MM4KTC82.Videotest/20051210-w50s.flv?AWSAccessKeyId=1EFVEK0RZ934...

Encoded URL passed in flash variable:
http%3A//s3.amazonaws.com/1EFVEK0RZ934MM4KTC82.Videotest/20051210-w50s.flv%3FAWSAccessKeyId%3D1EFVEK0RZ934MM4KTC82%26Expires%3D1281427868%26Signature%3DUE%252BSuGX4kY3bGRCKqLHiVPuPLNY%253D

Decoded URL requested by FLV Player detected via packet sniffing:
http://s3.amazonaws.com/1EFVEK0RZ934MM4KTC82.Videotest/20051210-w50s.flv?AWSAccessKeyId=1EFVEK0RZ934...

Clearly the first percent sign in the signature parameter gets decoded wrongly.

Can Jeroen or anyone else shed light on this?

The player applies an unescape() to every flashvars applied to it - to prevent any problems with the & = ? characters. The unescape() function of Flash is very similar to urlencoding, but unfortunately not exactly so. This is what goes wrong.

You can fix this by escaping the urlstring with a regex that does the same as Flash's escape() function, insteadof using a urlencode. It's an annoying problem, but unfortunately the Flash IDE works like this. There's probably more info at Adobe's livedoc pages on the escape() and unescape() functions.

Hi Jeroen,

Thanks for the prompt feedback.

At a first glance Adobe seems to think that their function works in a very standardized way according to their docs, which it hardly does.

I will try to investigate a bit more.

Using Amazon S3 for stream serving is probably quite interesting for many adopters of your player suite. For us it is a necessity.

Ulrik Bo Larsen

Has anyone addressed this successfully? I've used several regex's and urlencode through PHP to try to get this to work with Amazon's expiring URL, but have been unsuccessful (intermittently breaks as mentioned above). Thanks in advance.

I got authenticated S3 URLs working when using JavaScript to load and play a file by escaping the URL and specifying the media type, as someone suggested above.

However, I can't seem to get it working when using an RSS playlist of this form:

<filename>
URL
</filename>
<type>mp3</type>
<artist>Test</artist>
<title>Test</title>
</item>

I've also tried escaped/unescaped URLs and a <meta rel="type"> tag to specify the type. Nothing seems to work.
Is there anything that needs to be done that is playlist-specific? Should I use a different form of playlist?

Thanks, guys!

This thread has been very helpful for me with my experiments with S3. However I'm having the same problems as Herbert and others. That of sometimes (more than half) it works other times it hangs apparently trying to load the flash file. Here is a code snippet:

which PHP converts to:

When it doesn't play I 'view source' and copy the signedURL to a browser window and it finds the file for download so my signing script is always working.

The link for this is [url=http://www.wakewatcher.redirect.hm/football/2007/signed_test.php]Here.[/url]

(For testing I have the expiration set to an hour.)

Jeroen, My regular expression chops aren't up to speed. Any suggestions?

ps. Ignore the second window that comes up. It is an attempt to pre-fetch the likely next clip to watch. Also in this test patch the next and previous buttons aren't pointing to the right links. (I'm in the process of moving my navigation scripts out of S3 land since PHP doesn't work there.)

Hello, all. I've found a solution that has my FLVs with Signatures playing 100% of the time now.

After manually reviewing the URLs for quite a bit, I, basically, noticed that the snippet of %2B within the Signature would cause my video not to play _consistently_. I threw this bit of code (however inelegant) in:

<!-- CODE -->
while( $encVideoURL = getS3Redirect( $videoURL ) ){
if( !preg_match( '/%2B/', $encVideoURL ) ){
echo "$count <font color='green'>GOOD:</font> $encVideoURL<br />";
break;
}
$count++;
}
<!-- /CODE -->

and tested. It worked - _every time_ (counts btwn 0 - ~5k; delay maybe a sec and a half sometimes, negligible); this is what is required for my production environment.
I'm not a full-time coder (CEO, actually) and I'm sure that there are improvements that could be done with what I've provided above. I just want to put something out there for the folks (like me, Wakewatcher, etc.) that must have this working now and _everytime_ with signatures. Please comment; thanks.

Herbert.

So you are requesting up to 5,000 S3 URLs until you get one without a "+" character!

Has anyone talked to Amazon about producing S3 URLs without a "+"? Seems like they should fix this because the "+" is a VERY problematic character to use in a URL because of the historical usage of a "+" to represent a space character in URLs.

Their application is the one that is broken and needs fixing, so users don't have to do these crazy workarounds.

Yes, Jorge. It works consistently - which is what many folks needed now. Do you have an idea of a better implementation which would work today? If so, please pass it on because it would be helpful for many of us - really. Waiting until "someone" does "something" or "talk to someone" has been taking entirely too long. I agree with your assessment on speaking with Amazon re: the "+" and S3 URLs; I will shoot them a message today (now that I have something that will hold my application over); thanks.

I was asking because the original post in the other thread was almost 5 months ago. Seems like that would be plenty of time for amazon to fix their application.

If I was the customer, it would be fixed in about a week or I would find another vendor. You S3 users have way too much patience.

No, I don't have another solution as a workaround. I would never put up with such an absurd mistake in the first place.

However, if you choose this form of self-flagellation, your workaround is as good as any.

Thanks Herbert,

I'll give that a try. I'm a bit confused as to why this is positioned as an 'Amazon' problem given that the troublesome signatures are valid. Seems to me that its a player problem of not handling valid signatures correctly. (But then again I may not understand it all that well.) Also while your work around does the trick (and thanks for that) I'm wondering what impact it will have on the server. In my application I will have initially hundreds and then (hopefully) thousands hammering on the server at the same time. Looks like this loop could slow things down quite a bit. (Plus I've got a bunch of database stuff happening as well.) But thanks again.

Using a "+" in a URI is not valid given the historical use of "+" to indicate the space character in URIs. amazon should have disallowed the "+" character in their URIs to avoid this issue. So it is an amazon created/caused problem.

OK. Thanks for the info. I'm finding another issue. (Or perhaps another 'reality') For my application my average video clip size is about 15 seconds and that clip is viewed several times before going to the next clip and there is a natural 'next' video clip to view so I start downloading the next clip after the current clip has downloaded. However it seems with this signed URL that my browser doesn't know its the same next clip and so reloads it anyway. I guess that makes sense and so if I'm going to try to pre-fetch the next clip I'll have to more clever. (maybe use PHP to generate the 'next' link to match the pre-fetched instance.)

No problem, Wakewatcher. For my setup, it's working fine - customers happy for the time being (they don't care who "caused" the problem - just that it was resolved). If Amazon responds with a solution in the near term, that will be great. Happy Holidays.

I too attempted to solve this problem, but I was not able to find a way to properly escape Expiring Amazon S3 URLs that contained encoded "+" signs.

However, I think I found a better approach thanks to a co-worker's comments. Basically what I do can be summed up with this:

/recordings/123 -> (redirect) -> (Amazon S3 URL)

The 123 is the primary key of the audio recording in the database. Whenever a request comes in to this URL, I lookup the Amazon S3 path for the associated recording, generate the expiry URL, and then redirect the browser to this URL. The beauty of this is that I don't actually need to generate the expiry URL until someone requests it.. making it so I can display dozens or even hundreds of URLs on a page without the overhead of generating the URLs.

Since this URL contains on special characters that need escaping it works perfectly with the Flash Media Player. In my case I also like to check to make sure people are logged in before redirecting them, so I can make sure only authorized people get access to the audio on S3.

After some time off I'm back to working with expiry and flash files. Herbert I used your approach and I still have problems with about 6% of the files. (9 out a 150 clips) Are there any other signed Url strings that cause problems?

I had this same problem too ... the quick fix was to convert the plus sign in the signature to a %2B (I think someone already mentioned this further up in this thread)

Hmmm. I thought was was done above was to look for a %2B in the returned signature and reject it if found and request another one. What I'm finding is when I've got a non %2B url it still won't play a number of clips.

Dan Kubb - Thank you for that solution!

I'm passing a "safe" url to the flash movie that then redirects to the amazon S3 file. Works great.

Hello, Wakewatcher. I haven't visited the thread in a while; I returned and saw your message.

Since my implementation above, I have had no problems whatsoever; it works successfully 100% of the time. I like Dan Kubb's solution for my current issue - I need to dynamically generate a list of URLs (can't use encodeURIComponent with my dynamically generated XML file).

Herbert.

Dan Kubb - you "Da Man." I used your tip and it works great. Many thanks to all on the forum for tips/help; if any of you gents are in NC, give me a holler.

Herbert.

Double-encoding the url seems to work, anyone else have success with this?

hi, I already tried Dan's hint but it doesn't work...
I think I am doing something wrong
I use php function header("location: amazonURL"); to redirect to s3
is this how you do it?

Replacing % with %2525 worked for me. Looks like the + and = in the signature field has to escaped before the url is encoded and passed to the player.

I can't get it to work.
See http://media.muellers.ms/test.html

I have got a Amazon S3 Query String Authentication
URL that works with every browser/mediaplayer except JW Player the url looks like this

http://files.muellers.ms/test.mp4?AWSAccessKeyId=1WDA8W17T60NWQZR8Y82&Expires=1240650360&Signature=n...

Therefore my code looks like this:

<embed
src="http://media.muellers.ms/player/player.swf"
width="640"
height="480"
bgcolor="#"
allowscriptaccess="always"
allowfullscreen="true"
flashvars="file=http://files.muellers.ms/test.mp4?AWSAccessKeyId=1WDA8W17T60NWQZR8Y82&Expires=1240650360&Signature=nyQ1S6WRlua4Kv4iTT%2BmY2ZiOoo%3D"
/>

I was not able to encode the url or the string Signature=nyQ1S6WRlua4Kv4iTT%2BmY2ZiOoo%3D
in a way that JW Player works. It just returns "Error #2048"

Since Firefox/IE/wget curl and mplayer do not have any problem with the URL I think that JW Player is the one who causes these problems.

Any Ideas how to fix this ?

There are two things that you need to do:

1) the JW Player uses the last three characters of the file flashvar URI to determine the type of media file that it will be receiving. Since your URI does not end in a recognized media file extenstion, you need to use the type flashvar to tell the player what type of media file it will be receiving. See:  http://developer.longtailvideo.com/trac/wiki/FlashVars#Fileproperties  for the supported types,

2 the characters ( ? = & ) must be urlencoded if they appear in the file URI. Urlencoding chart here:  http://i-technica.com/whitestuff/urlencodechart.html

Wow, that was quick help and I can confirm
that it is working! Thanks a lot brokenString !

This is my perl code to use JW Player with
Amazon S3 Query String Authentication

The only change I hat to add was to url encode
the URL before passing it to the player
and I added "&type=video" which also was a pit
fall because I first thought "&type=video/mp4" was correct.

#!/usr/bin/perl

use Digest::HMAC_SHA1;
use MIME::Base64;
use URI::Escape;
use CGI qw/:standard/;

# s3 account information
my $secret_key='put your amazon s3 secret key here';
my $access_key='put your amazon s3 access key here';

my $bucket = 'files.muellers.ms';
my $key = 'test.mp4'; # no leading slashes

my $hmac = Digest::HMAC_SHA1->new($secret_key);
my $useby = time() + 60 * 60 * 24 ;

# sign string
my $string_to_sign = "GET\n\n\n$useby\n/$bucket/$key";
$hmac->add($string_to_sign);

# encode und escape string
my $digest = encode_base64($hmac->digest,'');
$digest = uri_escape($digest);

# http and html header
print header;
print start_html;

# this URL works in Browsers but unfortunately not in JW Player
my $url = "http://$bucket/$key?AWSAccessKeyId=$access_key&Expires=$useby&Signature=$digest";

# so we urlencode the URL for JW Player
$url =~ s/([^a-z0-9\/\@_\-\.:])/sprintf "%%%02X", ord($1)/egi;

print <<EOF;

<embed
src="http://media.muellers.ms/player/player.swf"
width="640"
height="480"
bgcolor="#"
allowscriptaccess="always"
allowfullscreen="true"
flashvars="file=$url&type=video"
/>

EOF

print end_html

Just want to say that following this thread has let me play S3 expiring URL's via an XPSF playlist and the loadFile function. Following Dan's post and BrokenString's advice on specifying a filetype - I came up with the following solution.

I use the redirect file as the file location and <meta rel='type'>sound</meta> in my playlists (ex: http://website.com/s3redirect.php?id=12345).

For loadFile, I append ".mp3" to the end of my redirect url (ex: http://website.com/s3redirect.php?id=12345.mp3). This tricks the player into thinking its playing just an mp3 file rather than a php page redirecting to an mp3 file. I then remove the '.mp3' before redirecting by doing this: $id = str_replace('.mp3', '', $id);

Just thought I'd share and thanks to everyone who contributed!

I don't want to do the redirect thing...

Why does
Signature=%2Bk5q4ZyJ%2BXd9Uqxk8%2FsaZ68ZUqM%3D
encoded as
%26Signature%3D%252Bk5q4ZyJ%252BXd9Uqxk8%252FsaZ68ZUqM%253D
fail?

Since 1994, browsers have converted the space character to the plus character in URLs.

Amazon should not have used any non-alphanumeric characters in the expiring URL.

I really can't believe that Amazon hasn't fixed this yet. Are they brain dead or just the "village idiots"?

Guess not...

I ended up doing the silly redirect thing, there's no need to create another file. Here's what I did:

switch($_REQUEST['action'])
{
case 'redirect':
// Generate a viewable url
$s3 = new S3(S3_ACCESS_KEY, S3_SECRET_KEY);
$authenticated_url = $s3->getAuthenticatedURL('bucket', 'file.flv', 600, false);
$authenticated_url = $authenticated_url;

header("location: $authenticated_url");
break;
}

Then you can just use view.php?action=redirect to get the file. Hope this helps.

Following the recommendations in this thread, I set up a working redirect URL and redirect.php for mp3, but with another flash player.

I am right now testing the JW player to use mp4, but I cannot get the player to receive the file from my redirect.php. I am printing the javascript code with PHP (to insert the song id in the URL), and the generated code seems to be fine:

<script type="text/javascript">
var so = new SWFObject('plugins/players/jw/player.swf','mpl','260','20','9');
so.addParam('allowscriptaccess','always');
so.addParam('allowfullscreen','true');
so.addParam('flashvars','&duration=00&file=redirect.php%3Fid%3D8.mp4&type=video&dock=false');
so.write('player');
</script>

The duration variable will also get a dynamic value to it via PHP, but not done yet; the ".mp4" extension is perhaps not necessary with JW player, but it was necessary with the other flash player - the extension is removed in the redirect.php file to provide the database table ID for the song.

The other difference to the earlier implementation is that the JW player is put in a pop-up window, but I don't see how that should affect the JW player.

Highly appreciated If anyone has a clue about this.

Regards,
Anders

Your code looks good — provided that redirect.php results in a valid redirect.

You need either the "mp4" extension or type=video to tell the player that it is receiving a video file.

If you call redirect.php in your browser, can you download the video file?

Lost,
The redirect.php is working fine in the browser. The redirect URL works fine in another flash player using mp3. I have also tried to use an absolute URL in the JW player to the redirect file, but that did not make any difference.

The only other difference compared to before is that the player is now called in a pop-up window which is connecting to Joomla "from outside" (the code connects and makes a login check to verify that the user is logged in).

Regards,
Anders

This is *not* an Amazon problem. Amazon S3 preauthenticated URL's are
correctly url-encoded. The problem is that jw player wants to also use url
encoding for an *atlernate* purpose (delimiting flashvars). If jw player wants
to use url encoding for this purpose, the only solution I see is to require two
layers of url encoding for *all* flashvars. In the following example, I have
tried every suggested encoding for the plus sign, to no avail

<param name="flashvars"
value="file=http://files.pooryorick.com/36e629f017916c9344f8ad2b35e3a5b1cd64b89fd8368a7b9dc4ee934fad4c94%3FSignature%3D0Xr1vrNllPbq/1i2xCxYiR%2BIPZo%3D%26Expires%3D1254339333%26AWSAccessKeyId%3D0NX0E89J778KNRPENR82&autostart=true&type=video" />

Don't want to do the redirect thing. Any other suggestions on how to get this
to work?

For what it's worth, we are grappling with the same issue.

I've even tried triple-encoding the characters like plus (+) and somehow when I sniff what URL the jwplayer is trying to load, it's decoded my %25252B back into a plus sign! WTF!

This is a major problem.

 
See: http://www.longtailvideo.com/support/forum/Bug-Reports/20182/Long-URLS-not-supported-

Yeah the people on this thread commenting on Amazon having the problem are wrong. The URLs contain signatures that are base64 encoded. This results in slashes, plus signs and equal signs in the signature. This is no problem, you encode these, it should not be a problem. Plus sign is encoded as %2B. This has nothing to do with a plus sign being used to represent a space, since your URL contains %2B, it does no contain a plus sign once it's encoded. When the server (Amazon) receives it, they decode it and they see a signature containing the original + sign.

The problem is in jwplayer and/or flash. At some point, it is unencoding the %2B as a plus sign, and then using that URL. This seems like something that could be fixed in the jw source, but I don't have time to figure out how that works. I'm going to resort to the 'redirect fix' even though it's not ideal.

walesa, thanks for the tip. I have tried using encodeURIComponent() and so far it doesn't seem to help. When I look at my source, the url being passed into encodeURIComponent() contains "%2B" but when I use livehttpheaders to see what URL the flash player is requesting, it contains the + sign in the URL, so it's still decoding it for some lame reason...

Did you try the code posted in: http://www.longtailvideo.com/support/forum/Bug-Reports/20182/Long-URLS-not-supported- ?

It works for me and it has a plus sign encoded as "%2B" in the URI.

Hmmm, I tried the encoding, but I did not try it using the addVariable() call like that. We have some older code that is calling swfobject.embedSWF(). I wasn't sure why they had used that call, but I remember there was some headaches with compatibility when they created this area, so I left it alone.

Maybe I will revisit and try to replace the call with the new SWFObject call instead, thanks!

 
It does not matter if you use:

a) new SWFObject() which is SWFObject v1.x

or

b) swfobject.embedSWF() which is swfobject v2.x

All swfobject does, is it creates an object element for embedding your Flash content (and it has Flash version detection and some nice utility functions via its API).

You could manually write the same object element and get the same results (minus the Flash version detection and the API).

Also, since it's JavaScript code, JavaScript's encodeURIComponent() method is available, but again, you could manually urlencode the file URI and get exactly the same results.

swfobject v2.2 is greatly improved over SWFObject v1.5, so I would recommend using v2.2, loaded from Google's CDN, but v1.5 will get the job done in simple code like this.

-improperly posted at another thread - it belongs here-

the problem cannot be solved using jwplayer at its present version. those who say they got it to work, im mystified.

the URLDecode function in jwplayer is 2 pass, when it should not be.

passing %252B should decode as %2B, NOT +. this means jw is decoding 1st pass to %2B and a second pass to +.

in a desperate attempt to URL encode 3 times (%2525252B), jw returns %252B, which is further evidence of its 2-pass decoding algorithm, which is broken.

save yourselves time and headache. switch to flowplayer. it works just fine, and only single pass URL decodes, solving this issue entirely by simply encoding the + to %2B prior to URLencoding the amazon URI.

Cheers,

Q

@quasidynamic -

You're correct; the player had a bug which prevents encoded URIs to be loaded properly. This should now be fixed in the most recent build of the player. Anyone with this issue should update their player from http://developer.longtailvideo.com/trac/browser/trunk/fl5/player.swf