Flat file input/output system: thoughts?

[strandoo]

Hi folks. I wonder if anyone has built or encountered an API-like FTP system of interacting with another computer. My client wishes to exchange product information (primarily stock quantity) via small text files that they will get/put via FTP to their website. (Currently, they FTP 10 csv files every day which I parse and use to update their website, so hopefully I can build on that). I think writing the files won't be difficult, but I imagine that I'll have to somehow monitor a directory, looking for new files and/or changes to existing files. I'd instigate the action on my end, then wait for a response via ftp and respond by parsing the new file and responding accordingly.

I'm most worried about detecting the incoming files: any ideas on how this can be done? Maybe only start 'watching' after the initial action? Any other thoughts about parsing/writing/deleting the files?

I'd appreciate any thoughts, ideas, etc. - Paul

[JayGee]

1 hour ago, strandoo said:

Hi folks. I wonder if anyone has built or encountered an API-like FTP system of interacting with another computer. My client wishes to exchange product information (primarily stock quantity) via small text files that they will get/put via FTP to their website. (Currently, they FTP 10 csv files every day which I parse and use to update their website, so hopefully I can build on that). I think writing the files won't be difficult, but I imagine that I'll have to somehow monitor a directory, looking for new files and/or changes to existing files. I'd instigate the action on my end, then wait for a response via ftp and respond by parsing the new file and responding accordingly.

I'm most worried about detecting the incoming files: any ideas on how this can be done? Maybe only start 'watching' after the initial action? Any other thoughts about parsing/writing/deleting the files?

I'd appreciate any thoughts, ideas, etc. - Paul

We have this kind of setup on integrations for a number of clients. It's a bit old fashioned and clunky but still seems to persist as a common way of working particularly in certain industries, e.g. fulfilment and ecommerce price lists.

We have mainly used a couple of approaches.

1) Creating cloud based endpoints to which files can be sent with cloud functions (E.g. Google cloud) sitting behind them. So the relevant processing is triggered automatically when the data arrives to the endpoint. Not quite a full API but not traditional FTP either.

2) Time-based send-and-receive using SFTP and CRON tasks to trigger the processing by regularly checking for files in certain folders on the FTP servers.

The biggest challenges we find are more about process and safeguarding the quality of data exchange rather than technical issues. E.g. should files be overwritten on arrival at the FTP sever? What is the naming convention for the files? What is the protocol for handling missing or overlapping data - e.g. if 2 files have a row with the same ID number should data be overwritten in your database? How are people notified if data exchange fails or there is a file parse error?

 

 

[strandoo]

Thanks @Guy Incognito. Yes, now that I've dived in a bit, I can see that handling the communication may be tricky. (Logged-in sales rep initiates a 'stock query' > check if exists, creates a new DB record > writes file > waits for response > checks if new response file matches my stock query > parse file and update DB record > display result > rinse and repeat.)

The php module 'Inotify' looks promising as far as checking for directory/file changes, but I haven't gotten there yet. Thanks again.

[JayGee]

44 minutes ago, strandoo said:

Thanks @Guy Incognito. Yes, now that I've dived in a bit, I can see that handling the communication may be tricky. (Logged-in sales rep initiates a 'stock query' > check if exists, creates a new DB record > writes file > waits for response > checks if new response file matches my stock query > parse file and update DB record > display result > rinse and repeat.)

The php module 'Inotify' looks promising as far as checking for directory/file changes, but I haven't gotten there yet. Thanks again.

Definitely sounds like they would be better off with a proper API to make for a shorter round-trip - but I know it can be hard sometimes to get people to change their ways! ?