Prevent 504 Gateway Time-out while importing large dataset

[ttttim]

I have an application running which every night imports/update data from an API. Below is a sort of summary about the code:

$data = callApi($urL); //returns all the course id's 

for ($i=0; $i<count($data); $i++) {
    $id = $data[$i]->id;
  	
  	//for every id do an extra api call to get detailed information.
  	$course_data = callApi($url, $id); 
  
  	//get course page or create a new page
  
  	//update fields if data has changed
}

The API doesn't return a timestamp for changes so i can't check if there has been updates since the last import. Meaning i have to make an API call for every course to get the detailed information and have to loop through every field to see if something has changed. The code works fine but the problem is that i often get a 504 Gateway Time-out because the script is taking to long...

Is there a way to run the code in the background or is my only option to increase the max_execution_time?

[ryan]

@ttttim In my experience a 504 gateway timeout comes prior to PHP max_execution_time and may not even be related to it (?). Which is to say, I think it's a server timeout setting that is probably specified somewhere in Apache or prior, rather than PHP. But my experience with 504s is mostly limited to this server we are typing on, and in the case of this server, I will get a 504 timeout on a long running process and the 504 comes from the load balancer rather than the actual node that is running the script. So when a 504 occurs here, it just breaks the connection with the browser but the script continues running till it finishes, interestingly. I have no idea if your case is similar, but one thing I can suggest in your script is that you call PHP's set_time_limit(600); within your long-running loop to reset PHP's timer on each iteration (600=10 minutes, or a different time if it suits you), otherwise you likely will be subject to PHP's max_execution_time as well.