Jump to content

Prevent 504 Gateway Time-out while importing large dataset


ttttim
 Share

Recommended Posts

I have an application running which every night imports/update data from an API. Below is a sort of summary about the code:

$data = callApi($urL); //returns all the course id's 

for ($i=0; $i<count($data); $i++) {
    $id = $data[$i]->id;
  	
  	//for every id do an extra api call to get detailed information.
  	$course_data = callApi($url, $id); 
  
  	//get course page or create a new page
  
  	//update fields if data has changed
}

The API doesn't return a timestamp for changes so i can't check if there has been updates since the last import. Meaning i have to make an API call for every course to get the detailed information and have to loop through every field to see if something has changed. The code works fine but the problem is that i often get a 504 Gateway Time-out because the script is taking to long...

Is there a way to run the code in the background or is my only option to increase the max_execution_time?

Link to comment
Share on other sites

@ttttim In my experience a 504 gateway timeout comes prior to PHP max_execution_time and may not even be related to it (?). Which is to say, I think it's a server timeout setting that is probably specified somewhere in Apache or prior, rather than PHP. But my experience with 504s is mostly limited to this server we are typing on, and in the case of this server, I will get a 504 timeout on a long running process and the 504 comes from the load balancer rather than the actual node that is running the script. So when a 504 occurs here, it just breaks the connection with the browser but the script continues running till it finishes, interestingly. I have no idea if your case is similar, but one thing I can suggest in your script is that you call PHP's set_time_limit(600); within your long-running loop to reset PHP's timer on each iteration (600=10 minutes, or a different time if it suits you), otherwise you likely will be subject to PHP's max_execution_time as well. 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...