Create pages from JSON feed

[SamC]

Hi,

I might have to create the following website with parts of it coming from an external feed. Some pages about, contact, staff profiles etc... will be standard PW pages. However, the building listings will come from a JSON feed.

What I 'think' I need to do is:

1) Get the feed, read the file, make it a JSON string - can only be accessed once every 24hrs
2) Use json_decode to create an object/or assoc array
3) Create (a) a template building-entry (b) fields for the template - either manually (because I know the required fields in advance or using info from the feed (with the PW API)
4) Add these fields to the template
5) Create pages using the data
6) Every 24hrs, the feed needs to be compared to the pages in the tree, if the building has been deleted from the feed, needs to be deleted from the tree. If a building has been changed/edited in the feed, these changes need to be reflected in the pages in the tree.

...so I basically end up with this lot in the tree:

- buildings (parent - building-index.php)
 -- building 1 (child - building-entry.php)
  --- title
  --- desc
  --- area
  --- type

 -- building 2 (child - building-entry.php)
  --- title
  --- desc
  --- area
  --- type

 -- building 3 (child - building-entry.php)
  --- title
  --- desc
  --- area
  --- type

The buildings themselves are not editable in PW, they will use a CRM to upload the details, all I get is the JSON feed. I wanted this approach then I can use PW for filtering/sorting etc. and I'm fairly comfortable with presenting the data to the end user once it's in the tree.

I guess I'm not going to progress unless I make something more complex but this worries me somewhat being more of a designer. Been brochure style sites (with maybe a news/blog section) all the way until now.

Any advice would be awesome, thanks.

[Robin S]

Sounds like you have a good handle on how the process will work, so no reason to worry I think. I've just done something very similar so I can attest that it's quite simple to do in PW.

Regarding 1 & 2: most likely you will be querying an API endpoint that will return a JSON string to you. You'll probably find the WireHTTP class useful here - that's what I used.

The getJSON() method will convert the JSON response into an associative array that contains all the data you requested. Tracy Debugger will be useful for exploring this array. Then you will use some part of the data as a unique identifier (you could use the title string but there's probably some kind of reference code that will be a better candidate) and based on whether you can find an existing page with that identifier you will either create a new page or update an existing page. Then it's simply a matter of matching items in the array to fields on your page and saving the page at the end.

You can use a lazy cron function to automatically query the API endpoint every 24 hours. To delete pages that are no longer contained in the JSON response you can collect all of the unique identifiers in the response and then use a PW selector to get pages not matching any of those identifiers. Those pages you delete.

[SamC]

@Robin S that's an awesome response, thanks. I'm going to get the Tracy debugger installed as I haven't used it yet, then test how to make pages/fields with the API in a template rather than through the admin.

[Artomultiplo]

@SamC I'm trying to do a similar thing (if not identitical) and I'm in a similar situation like the one you were in the first thread, been also more a designer myself I'm trying to digest as much php as possible but some concepts regarding the json to pages are very hard to understand.

Is it possible for you to explain a little about your final solution/workflow/code for this situation?

Thanks in advance.

[SamC]

7 hours ago, Artomultiplo said:

@SamC I'm trying to do a similar thing (if not identitical) and I'm in a similar situation like the one you were in the first thread, been also more a designer myself I'm trying to digest as much php as possible but some concepts regarding the json to pages are very hard to understand.

Is it possible for you to explain a little about your final solution/workflow/code for this situation?

Thanks in advance.

Hi @Artomultiplo 

Sorry but this site never materialised. Maybe @Robin S would be able to help you more with this.

[Artomultiplo]

Thanks @SamC, I understand.

I kind-of did it in a test environment, let me explain the full case scenario:

A local (and really small) movie theater in my area was looking for a web designer to help them out with their website and primarily their ticket management integration. They absolutely need to sell their tickets on-line.
I quickly jumped in to start the negotiations around budget, best solution, how to manage the whole thing, ecc.
During our chats I found that there was a widely used ticket management service in Italy called Wintic/Webtic, it is used in around 70% of cinemas in Italy and I knew that very probably I would have to integrate that sooner or later.

So I started calling the software house regarding their ways to integrate the whole "thing" in an existing portal.
They told me that they made available a webservice with the cinema schedule and send me a link to the JSON
I was frightened! Being this the first time I've found myself in a situation (that I was actively avoiding) where I just needed to step out of my relatively comfy zone of CMSs and tackle the issue as best as I could and in a budget friendly way. (The integration with Wordpress is just a headache to think about compared with Processwire)

Then I started to read (many times) all the threads regarding JSON import and this one really helped me out understanding the whole process.
From that thread and a couple of others I modified the script and now it looks like this: https://gist.github.com/protorob/e6050c78b4bee2dbce59c38234afa0de

In the end I came out with a functional mockup that you can find here: http://webtic.artomultiplo.net/ and accepted the job. Full of courage.

Now I have a whole lot of new questions regarding the best way to wrap it out, so expect to see me around a little more this time.
I'm planing to document the whole process in a very long blog post

Now my main "concerns" are regarding:

- Automate the create/modify/delete of the schedule comparing differences between the current JSON and a New one
- Adding extra information to the events and not losing them when the schedule re-synchronize
- Working with AJAX Load more for some parts of the website
- Handling the blog

I know I'll need a little bit of help specially with the importing script.

I'll let you know how it goes

[dragan]

Well, the first time you have to handle such situations is always a bit rough. Lots of trial and error, and trying to stay in sync, but also trying to get the best possible speed performance etc.

From what little I could see, the JSON you get is well-structured. There are unique IDs that you could use in your import/update process. There are time-stamps for performances. So, overall you should be fine (not every SaaS who claims it has its own API is actually usable).

With projects like this, it's even more important to have a solid test/dev platform. If you have a live site, and add or change features, you don't wanna do it directly first on the production site.

I'm sure we'll have suggestions and tips if you run into any problems.

3 hours ago, Artomultiplo said:

Full of courage

That's the most important thing. 

 

[SamC]

4 hours ago, Artomultiplo said:

I'm planing to document the whole process in a very long blog post

I'd be interested in reading this  

Good luck with the project, sounds exciting!

[gebeer]

I am right now working on a quite similar project. In my case I import from an XML that I get through an API and populate product pages, check for there status etc.

I can encourage you to go on the way you started. It is all pretty straight forward and nice to handle with the PW API. Your import code looks basically fine. So you shouldn't have major problems there.

I set up a process module that creates a page in the backend where admins can manually trigger imports. If you would like to go that route, there is a skeleton Hello Process module that you can use and @bernhard wrote a great article on building admin pages with process modules.
I find this approach much cleaner than doing all the logic in a template file.

Regarding your concerns

18 hours ago, Artomultiplo said:

- Automate the create/modify/delete of the schedule comparing differences between the current JSON and a New one

Use lazy cron module for automation. Each time you get a new JSON, loop through it and compare it to the single-movie pages you have and act on them (modify/delete) or create new ones.

18 hours ago, Artomultiplo said:

- Adding extra information to the events and not losing them when the schedule re-synchronize

You can use custom fields on the single-movie template to store that

18 hours ago, Artomultiplo said:

- Working with AJAX Load more for some parts of the website

Have a look at https://github.com/webcreate/infinite-ajax-scroll

18 hours ago, Artomultiplo said:

- Handling the blog

You can use kongondos Blog module  or at least get a lot of inspiration from it ;-)

 

[Artomultiplo]

Wow @gebeer, that info is really valuable!
Thank you very very much!

[3fingers]

I'm close to approach something similar for the first time.
My client has a Wordpress site where he wants to integrate a search functionality (various filters, category and so on). Of course I'm looking for a way to handle it in Processwire.
My idea is to leverage the Wp rest api and build upon the json response in Processwire.

My concern though is that the search page should live on the same domain (eg. originadomain/search) and I don't know how to handle this request to point to my Processwire installation, nor how to let Wp and Processwire live together happily on the same server.

... and what about trying to integrate my Processwire search page inside an iframe on the Wp page? Does it sound to you like a big hack?

Any advice would be very appreciated ?

Thanks.