Jump to content

Good practices for a cronjob based app?

Recommended Posts

Hello everyone,
I would like to ask for some advice on what options I have to build an app I have in mind.
It will scrape images from instagram and save each of them as a PW page, then the admin can edit them and it will be displayed on the front-end.
The challenge for me is to have a cronjob, to run a scraper maybe written in Python, and populate the database using the PW API.

Is this sensible? Is it suitable to use PW for this or should I use another technology?

Link to comment
Share on other sites

5 minutes ago, michelangelo said:

Is this sensible? Is it suitable to use PW for this or should I use another technology?

You may have a specific reason for going this route, but first things first: Instagram normally prohibits this sort of use. They used to name "scraping" and "caching" specifically, but current version of their ToS just disallows "collecting information in an automated way without our express permission".

So I'd say that while you may be able to do this technically, you probably shouldn't, especially if any of the data pulled from IG is going to be displayed publicly. You can of course request permission for this, but I'd wager that there's a very low chance of that working out 🤷‍♂️

Legal aspect aside, whether PW is a sensible platform for something like this would depend a lot on the specific case. For storing, displaying and searching through structured data? Sure, why not. Though in such a case it might make more sense to write the scraper in PHP as well, so you could just bootstrap PW. Alternatively the scraper could store data locally or in a DB, and PW would then read it from there... or you could provide an API endpoint at PW side 🙂

  • Like 1
  • Thanks 1
Link to comment
Share on other sites

@teppo That's a good point about the Instagram policies, I will go over it once more.

Maybe writing the scraper in PHP is better idea than switching to a completely different stack where I have to build the backend and everything from scratch. Thanks!
Out of curiosity though: It is possible to run a python script with a cronjob, right?

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Create New...