What would be the best way to mirror json data into the Processwire DB

[h365]

Hi Guys,

So i have a big piece of Json data for a Travel - Tour 

all kinds of rates, dates, services and so on.. 

The Structure of the Json slightly changes here and there depending on the Tour. 

What would be the best way to import that data in processwire to make it easily searchable etc trough the API.. ? 

I tried Multi Level Repeater Matrixes which are good and would do the job, but then i have no big options to search trough that data.. 

Would really appreciate all input!

Thanks so much

[fbg13]

Have you considered pages?

[dragan]

What's the big picture? Where are these JSON files coming from? Do you just build a frontend with externally fetched JSON data?

Do you have to fetch / update JSON data on a regular basis?

Or do you create this JSON yourself?

[BitPoet]

9 hours ago, h365 said:

What would be the best way to import that data in processwire to make it easily searchable etc trough the API.. ? 

I tried Multi Level Repeater Matrixes which are good and would do the job, but then i have no big options to search trough that data..

If you don't need to search by specific field names nested in the matrix, you could add a concatenated (hidden) textarea field, hook into saveReady and render the matrix contents into that field. Then include that field for your search.

Or, if you want to be able to search by property names (at any depth) have at least MySQL 5.7.8, you could try out the attached module JsonSearchable.zip. It's a field type derived from Textarea that uses native JSON storage. Supposed you made a page field named "myfield" and entered a JSON of:

{
    "testdata":[
        {
            "segment":1,
            "title":"Hello World",
            "average":20,
            "max":"30",
            "min":5
        },
        {
            "segment":2,
            "title":"This is Funny",
            "average":40,
            "max":"30",
            "min":5,
            "details":{
                "name":"nobody",
                "email":"secret"
            }
        }
    ]
}

You could e.g. search for

<?php

$pages->find("myfield=30");
$pages->find("myfield.max=30");
$pages->find("myfield.details.name=nobody");

Note however that this kind of search on many large datasets can be an absolute performance killer. Numeric comparisons are not supported.

[h365]

12 hours ago, dragan said:

What's the big picture? Where are these JSON files coming from? Do you just build a frontend with externally fetched JSON data?

Do you have to fetch / update JSON data on a regular basis?

Or do you create this JSON yourself?

Hey, so basically its our inhouse system for Travel Bookings, it provides an API for all tours added into that system. With All Kinds of data, and its pretty nested as well..

Yes i have to fetch it ervery 3 hours or so to keep them up to date. 

No i dont create the json myself!

 

The Bigpicture :

So if i have imported the json data somehow in a Page or something...(with Dates, Rates .. ) then i would add some rich content.. Pictures, Descriptions to that tour .. 

After that, the tours need to be searchable from the website...eg. search.php?dateFrom=01.01.2018&dateTo=05.10.2018

 

[h365]

3 hours ago, BitPoet said:

If you don't need to search by specific field names nested in the matrix, you could add a concatenated (hidden) textarea field, hook into saveReady and render the matrix contents into that field. Then include that field for your search.

Or, if you want to be able to search by property names (at any depth) have at least MySQL 5.7.8, you could try out the attached module JsonSearchable.zip. It's a field type derived from Textarea that uses native JSON storage. Supposed you made a page field named "myfield" and entered a JSON of:

{
    "testdata":[
        {
            "segment":1,
            "title":"Hello World",
            "average":20,
            "max":"30",
            "min":5
        },
        {
            "segment":2,
            "title":"This is Funny",
            "average":40,
            "max":"30",
            "min":5,
            "details":{
                "name":"nobody",
                "email":"secret"
            }
        }
    ]
}

You could e.g. search for

<?php

$pages->find("myfield=30");
$pages->find("myfield.max=30");
$pages->find("myfield.details.name=nobody");

Note however that this kind of search on many large datasets can be an absolute performance killer. Numeric comparisons are not supported.

Thanks so much for your help! Ill give that a try, but i dont know if we are running into performance issues as the datasets can get really big..