byte-range requests for podcast mp3

[benbyf]

related to this topic on wireSendFile(). I've been trying to create a RSS feed for a podcast which i've done previously using PW 2.7.2 with no trouble (e.g. http://machine-ethics.net/), however sitting on the same infustructure a new site (http://www.threepointspodcast.com/) on PW 3.0.42 is having issues validating with itunes with this error from them: Can’t submit your feed. Your episodes are hosted on a server which does not support byte-range requests. Enable byte-range requests and try again. Which is obviously untrue as the other podcast works fine.

Just wondering if there's something I need to add in the template settings etc to make it work, I'm currently running the .mp3 file out using wireSendFile() and setting the type to application/mp3 in template settings but with no avail. I'm running nginx.

Any help would be appreciated.

[abdus]

It looks like you need to set some headers to enable byte range requests and check incoming requests for byte ranges that client is currently at. 

Check out this SO answer:

http://stackoverflow.com/a/157447

[benbyf]

not sure how to do that in PW. But also it's mega strange it works on my v2.7 install but not on v3.

[adrian]

1 hour ago, benbyf said:

not sure how to do that in PW. But also it's mega strange it works on my v2.7 install but not on v3.

Not sure why it's working on 2.7 v 3, but setting headers in PW is just like any PHP script - just use the example in that SO thread and I am guessing it will work.

[abdus]

If you have control over the server and you're able to install PECL module (pecl_http) on PHP, then http_send_file() function is the most straightforward way to implement partial downloads. Otherwise you have to implement it manually using a code similar to one in SO link above.

http://php.net/manual/fa/function.http-send-file.php

[benbyf]

im still puzzled why it works on the same infustructure but different versions of PW.

[Macrura]

@benbyf

just a shot in the dark, but have you considered that with the 2.7.2 version, when you submitted it, Apple was not validating the byte-range request support?

[benbyf]

21 hours ago, Macrura said:

@benbyf

just a shot in the dark, but have you considered that with the 2.7.2 version, when you submitted it, Apple was not validating the byte-range request support?

could be the case.

[benbyf]

On 15/04/2017 at 6:36 PM, abdus said:

It looks like you need to set some headers to enable byte range requests and check incoming requests for byte ranges that client is currently at. 

Check out this SO answer:

http://stackoverflow.com/a/157447

That anwser did the trick i think, i managed to make the implementation below. It must have been a new itunes requirement for byte-range headers.

$filesize = filesize($page->mp3->httpUrl);
    $offset = 0;
    $length = $filesize;

    if ( isset($_SERVER['HTTP_RANGE']) ) {
        // if the HTTP_RANGE header is set we're dealing with partial content

        $partialContent = true;

        // find the requested range
        // this might be too simplistic, apparently the client can request
        // multiple ranges, which can become pretty complex, so ignore it for now
        preg_match('/bytes=(\d+)-(\d+)?/', $_SERVER['HTTP_RANGE'], $matches);

        $offset = intval($matches[1]);
        $length = intval($matches[2]) - $offset;

    } else {
        $partialContent = false;
    }

    $file = fopen($page->mp3->httpUrl, 'r');

    // seek to the requested offset, this is 0 if it's not a partial content request
    fseek($page->mp3->httpUrl, $offset);

    $data = fread($page->mp3->httpUrl, $length);

    fclose($page->mp3->httpUrl);

    if ( $partialContent ) {
        // output the right headers for partial content

        $headers = array(
            'HTTP/1.1 206 Partial Content' => true,
            'Content-Type' => 'audio/mp3',
            'Content-Length' => $filesize,
            'Content-Disposition' => 'attachment; filename="' . $fileName . '"',
            'Content-Range' => 'bytes ' . $offset . '-' . ($offset + $length) . '/' . $filesize,
            'Accept-Ranges' => 'bytes',
        );

        // send partial file
        wireSendFile($page->mp3->filename, $options, $headers);
    }else{
        // send file
        wireSendFile($page->mp3->filename, $options);
    }

 

[abdus]

@benbyf, I am not sure if this is the whole code, but I want to point out some parts.

<?php
// you should use $page->mp3->filename to prevent unnecessary (and slower) network request
// let PHP read file from the disk directly
$file = fopen($page->mp3->httpUrl, 'r');
fseek($page->mp3->httpUrl, $offset);
$data = fread($page->mp3->httpUrl, $length);
fclose($page->mp3->httpUrl);

// also, this forces client to download whole file
wireSendFile($page->mp3->filename, $options, $headers);

// whereas sending the parts you prepared is just cheaper
print($data);

So, the current implementation just sets the headers iTunes was checking for, but it sends whole file, not the parts client requested. This might total to an unnoticeable difference for lower traffic files, but you should opt for sending only the parts client requested.
Just my 2 cents.

[benbyf]

cooooooooool, will try and amend. Your 2 cents is super useful but seems your code was a mix of your suggestions and code needing amending no? bit confusing sorry

[abdus]

Yeah, that might be confusing somewhat. To summarize the changes:

• The part starting with $file = open($page->mp3->httpUrl, 'r'); should be

  <?php
  $filePath = $page->mp3->filename;
  
  $file = fopen($filePath, 'r');
  fseek($filePath, $offset);
  $data = fread($filePath, $length);
  fclose($filePath);

  to read parts of the file client requested directly from the disk.
• While sending the file to the client, you should replace this part

  <?php
      // send partial file
      wireSendFile($page->mp3->filename, $options, $headers);
  } else {
      // send file
      wireSendFile($page->mp3->filename, $options);
  }

  with print($data); like so

  <?php
  // send partial file
      print($data);
  } else {
      // send file
      print($data);
  }

  Because you've set headers earlier, you dont need to set them again.

With these changes, hopefully, you'll have saved server from wasting some precious bandwidth.

Good luck

[benbyf]

think maybe my Headers are still wrong as it has a problem when seeking - gives me a "video or MIME type not supported" error

[abdus]

Are all podcasts in MP3 format? If 'Content-Type' => 'audio/mp3 header and mime type of podcasts do not match, you'd get the error. Try removing 'Content-Type' => 'audio/mp3' from $headers array.

In case it doesnt work, https://gist.github.com/benrolfe/5453074 claims to solve iTunes byte range problem. Changing the $file_path at line 2 to $page->mp3->filename should be enough.

[benbyf]

yep that gist solves it. I just kept the wireSendFile() if the file isn't a partial content reponse and works well. Thanks @abdus!

new games podcast with my friends www.threepointspodcast.com/

[abdus]

Glad to help, @benbyf

[Torsten Baldes]

On 18.4.2017 at 6:39 PM, benbyf said:

yep that gist solves it. I just kept the wireSendFile() if the file isn't a partial content reponse and works well. Thanks @abdus!

new games podcast with my friends www.threepointspodcast.com/

@benbyf is it possible for your to share your current implementation for downloading/sending out the mp3? i tried to follow your steps but got confused with the last gist.

thanks!

[Torsten Baldes]

@benbyf

ah, sorry. i just found out that you created the podcast profile (https://github.com/benbyford/pw-podcast/) and updated it. sorry and thanks!

[benbyf]

Just checked on: http://castfeedvalidator.com/

and still not delivering partial headers, anyone have any experience on this, I'm running PW on nginx. Here is my current php code for the file request:

<?php namespace ProcessWire;
if($page->mp3){

    $page->of(false);
    $page->counter += 1;
    $page->save();
    $page->of(true);


    //
    // adapted from GIST: https://gist.github.com/codler/3906826
    //
    $options = array(
        // boolean: halt program execution after file send
        'exit' => true,
        // boolean|null: whether file should force download (null=let content-type header decide)
        'forceDownload' => false,
        // string: filename you want the download to show on the user's computer, or blank to use existing.
        'downloadFilename' => '',
    );

    $file_path = $page->mp3->filename;

    $fp = @fopen( $file_path, 'rb' );
    $size   = filesize( $file_path );
    $length = $size;
    $start  = 0;
    $end    = $size - 1;

    header("Content-type: audio/{$page->mp3->ext}");
    header( "Accept-Ranges: 0-$length" );
    header( "Content-Length: $length" );

    // find request headers for partial content
    if ( isset($_SERVER['HTTP_RANGE']) ) {
        // if the HTTP_RANGE header is set we're dealing with partial content
        $partialContent = true;

    } else {
        $partialContent = false;
    }

    if ( $partialContent ) {

        $c_start = $start;
        $c_end   = $end;

        // Extract the range string
        list(, $range) = explode( '=', $_SERVER['HTTP_RANGE'], 2 );
        // If the range starts with an '-' we start from the beginning
        // If not, we forward the file pointer
        if ( $range{0} == '-' )
        {
        	// The n-number of the last bytes is requested
        	$c_start = $size - substr($range, 1);
        }
        else
        {
        	$range  = explode( '-', $range );
        	$c_start = $range[0];
        	$c_end   = ( ( isset( $range[1] ) && is_numeric( $range[1] ) ) ? $range[1] : $size );
        };
        // End bytes can not be larger than $end.
        $c_end = ($c_end > $end) ? $end : $c_end;
        $start  = $c_start;
        $end    = $c_end;
        $length = $end - $start + 1;
        fseek( $fp, $start );

        // Start buffered download
        $buffer = 1024 * 8;
        while ( ! feof( $fp ) && ( $p = ftell($fp) ) <= $end )
        {
        	if ( $p + $buffer > $end )
        	{
        		$buffer = $end - $p + 1;
        	};

        	set_time_limit( 0 );

        	echo fread( $fp, $buffer );

        	flush();
        };
        header( "Content-Range: bytes $start-$end/$size" );
        header( 'HTTP/1.1 206 Partial Content' );

        fclose( $fp );

    }else{
        // send file
        wireSendFile($page->mp3->filename, $options);
    }
}

 

[benbyf]

Couldnt get the tracking and partial headers to work together so ive allowed the RSS feed direct link to the file and only track when viewing on the site. This seems to work now on itunes and means users can track through the audio which was erroring before :/

[cstevensjr]

Just had a need to use this profile.  Thanks for your work on this profile.

[horst]

On 3/12/2018 at 11:21 AM, benbyf said:

Couldnt get the tracking and partial headers to work together so ive allowed the RSS feed direct link to the file and only track when viewing on the site. This seems to work now on itunes and means users can track through the audio which was erroring before ?

Here is some code I used which is working fine:

https://github.com/horst-n/LocalAudioFiles/blob/master/site-default/templates/local-audio-files_stream.php#L34-L45

https://github.com/horst-n/LocalAudioFiles/blob/master/site-default/templates/local-audio-files_stream.php#L264

 

[benbyf]

thanks @horst will check it out and update the profile.

[Noel Boss]

Maybe a bit late, but I solved this with a Module:

https://gist.github.com/noelboss/7815ff2fea96b41544672f5e1a15f1f5

Usage:

wire('modules')->get('ProcessPageView')->sendFile($page, $filename);

I remember doing it that way, since I couldn't hook WireHttp or something and I needed to look up the file using the filename (protected site, using repeaters etc.) – you could also adapt it to accept a PageFile or PageImage or a path even… Maybe a bit more straight forward. 

[benbyf]

not sure if we have our wires crossed but providing the file wasnt really the issue, other wise I would just link to the location, it was capturing statistics on file access where range-headers needed to be provided. This was mainly as itunes at teh time didnt supply any podcast data – they've changed their tune abit since but still what they track is limited and to this day my podcast still says "Not Enough Data" sooooo still super annoying.