504 Gate Timeout on Long script

[Jay D]

Hi all, 

I am building a custom module for my project that processes some data from a separate db table into useable PW fields. I have a very basic module that extends Process. I think one of the issues is I need to start to return data to the screen, but using the

return $data

 method, the script will wait until everything is completed to show the progress, that's where I get the 504 Gateway Timeout. Is there any way to start to output to the screen using this method? 

Thank you in advance, 

Jay

[Rudy]

Hi Jay,

If you are expecting a lot of data to process, perhaps you might want to paginate your result to prevent long execution. You might also want to increase your PHP max_execution_time to 300 or more (up to you).

[Jay D]

Thanks for the tips Rudy! 

The pagination idea seems like the way to go. Do you know if there is a way to call a method by API, with no header/footer admin pages being returned? I could do some AJAX to print to screen as I loop through the data. 

[Rudy]

Since you're working on a PW environment, you should probably use PW's WireArray and Paginated Array API.

[Tom.]

@Rudy are you able to provide any example on how you may paginate array to prevent long execution times? 

I have done imports before, but when the data get's too large it can take a while to load. How would you automate the pagination without also running into long execution times?