Jump to content

What would be the best way to mirror json data into the Processwire DB


h365
 Share

Recommended Posts

Hi Guys,

So i have a big piece of Json data for a Travel - Tour 

all kinds of rates, dates, services and so on.. 

The Structure of the Json slightly changes here and there depending on the Tour. 

What would be the best way to import that data in processwire to make it easily searchable etc trough the API.. ? 

I tried Multi Level Repeater Matrixes which are good and would do the job, but then i have no big options to search trough that data.. 

Would really appreciate all input!

Thanks so much

Link to comment
Share on other sites

What's the big picture? Where are these JSON files coming from? Do you just build a frontend with externally fetched JSON data?

Do you have to fetch / update JSON data on a regular basis?

Or do you create this JSON yourself?

  • Like 2
Link to comment
Share on other sites

9 hours ago, h365 said:

What would be the best way to import that data in processwire to make it easily searchable etc trough the API.. ? 

I tried Multi Level Repeater Matrixes which are good and would do the job, but then i have no big options to search trough that data..

If you don't need to search by specific field names nested in the matrix, you could add a concatenated (hidden) textarea field, hook into saveReady and render the matrix contents into that field. Then include that field for your search.

Or, if you want to be able to search by property names (at any depth) have at least MySQL 5.7.8, you could try out the attached module JsonSearchable.zip. It's a field type derived from Textarea that uses native JSON storage. Supposed you made a page field named "myfield" and entered a JSON of:

{
    "testdata":[
        {
            "segment":1,
            "title":"Hello World",
            "average":20,
            "max":"30",
            "min":5
        },
        {
            "segment":2,
            "title":"This is Funny",
            "average":40,
            "max":"30",
            "min":5,
            "details":{
                "name":"nobody",
                "email":"secret"
            }
        }
    ]
}

You could e.g. search for

<?php

$pages->find("myfield=30");
$pages->find("myfield.max=30");
$pages->find("myfield.details.name=nobody");

Note however that this kind of search on many large datasets can be an absolute performance killer. Numeric comparisons are not supported.

  • Like 5
Link to comment
Share on other sites

12 hours ago, dragan said:

What's the big picture? Where are these JSON files coming from? Do you just build a frontend with externally fetched JSON data?

Do you have to fetch / update JSON data on a regular basis?

Or do you create this JSON yourself?

Hey, so basically its our inhouse system for Travel Bookings, it provides an API for all tours added into that system. With All Kinds of data, and its pretty nested as well..

Yes i have to fetch it ervery 3 hours or so to keep them up to date. 

No i dont create the json myself!

 

The Bigpicture :

So if i have imported the json data somehow in a Page or something...(with Dates, Rates .. ) then i would add some rich content.. Pictures, Descriptions to that tour .. 

After that, the tours need to be searchable from the website...eg. search.php?dateFrom=01.01.2018&dateTo=05.10.2018

 

Link to comment
Share on other sites

3 hours ago, BitPoet said:

If you don't need to search by specific field names nested in the matrix, you could add a concatenated (hidden) textarea field, hook into saveReady and render the matrix contents into that field. Then include that field for your search.

Or, if you want to be able to search by property names (at any depth) have at least MySQL 5.7.8, you could try out the attached module JsonSearchable.zip. It's a field type derived from Textarea that uses native JSON storage. Supposed you made a page field named "myfield" and entered a JSON of:


{
    "testdata":[
        {
            "segment":1,
            "title":"Hello World",
            "average":20,
            "max":"30",
            "min":5
        },
        {
            "segment":2,
            "title":"This is Funny",
            "average":40,
            "max":"30",
            "min":5,
            "details":{
                "name":"nobody",
                "email":"secret"
            }
        }
    ]
}

You could e.g. search for


<?php

$pages->find("myfield=30");
$pages->find("myfield.max=30");
$pages->find("myfield.details.name=nobody");

Note however that this kind of search on many large datasets can be an absolute performance killer. Numeric comparisons are not supported.

Thanks so much for your help! Ill give that a try, but i dont know if we are running into performance issues as the datasets can get really big.. 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...