server slows down: looping through repeaters

[pppws]

hey there,

 

i have a collection (parent page) of persons (child pages of 'collection'). each person has several fields. two of them are repeater fields where the person can enter their 'jobs' and 'recidencies'. i'm trying to build a list of entries which looks like:

Actor

• Peter
• Maria
• Paul
• …

Doctor

• Eva
• Julia
• William
• …

for the first 5 persons everything worked smoothly. but now that i've reached about 20 entries the server slows down and i'm wondering if my loop is somehow cluttred up.

 

<?php $langname = $user->language->title; //get current user language?>
<?php $persons = $pages->get('/collection')->children->filter("lang=$langname") // get all children for current user language ?>

<section class="profession">
    <h1>professions</h1>
    <?php foreach ($persons as $child): ?>
        <?php foreach ($child->professions as $profession): // the repeaterfield is called: professions, the field itself is profession?>
            <?php
                $profAll[] = $profession->profession // store all entries;
                $profUnique = array_unique($profAll) // only unique entries ;
                sort($profUnique) // sort the entries;
            ?>
        <?php endforeach; ?>
    <?php endforeach; ?>
    <ul style="column-count: 2;">
        <?php foreach ($profUnique as $profLetter): // loop through all professions ?>
            <li style="font-size: 2rem; list-style-type: none;" class="letter"><?= $profLetter // output one profession e.g. Actor?></li>
                <?php foreach ($persons->find("professions.profession=$profLetter")->sort('givenname') as $person): // find all persons who have the profession Actor ?>
                    <li><a class="ajax" href="<?= $person->url ?>"><?= $person->givenname // output the name of person who fits the profession  ?></a></li>
                <?php endforeach; ?>
        <?php endforeach; ?>
    </ul>
</section>

 

is there a way to make this request faster? (i'll have at least two of them on the same page)

 

[abdus]

This looks like a job for Page fields instead of Repeaters, and the module ConnectPageFields by @Robin S which makes 2-way relations possible. This way you wont have to use 4 nested loops to build your markup. 

You need a set up templates and fields like this

PAGES:
    collection/
        Peter : person
        Maria : person
        Paul : person
    professions/
        Doctor : profession
        Actor : profession

FIELDS:
    professions
        Page field,
        limited to `profession` template, and professions under /professions
        Enable create new pages

    persons
        Page field,
        limited to `person` template, and persons under /collection

Add `professions` field to `person` template, and `persons` field to `profession` template.
Then set up a synced pair from ConnectPageFields module settings
When you add a new profession to a person, it will be created under /professions

Then in your templates you can just use:

<ul class="jobs">
<?php foreach ($jobs as $j): ?>
    <li class="job">
        <ul class="job__persons">
        <?php foreach($j->persons as $p): ?>
            <li class="person">
                <?= $p->title ?>
            </li>
        <?php endforeach; ?>
        </ul>
    </li>
<?php endforeach; ?>
</ul>

and you're done.

I'm not sure how Residency system works, but if range of its values is not too wide, Page field can be replace repeaters for that too.

Also, instead of getting all pages then filtering, you can extend your  selector to include the filter, so you wont overload DB and RAM for nothing. Also I've found using IDs for speed-critical operations to be more performant. Around 2ms for id based fetches, >6ms for non-id based fetches

$persons = $pages->find("parent=$collectionId, lang=$user->language"); // $user->language is interpreted as page id for the user's language

 

[pppws]

thanks! i'll give it a try.

 

FIELDS:
    professions
        Page field,

 Page field = Page Reference?

the residencies work similiar, but have two additionals fields: city, country (both text) and current (checkbox) – can i use the same method you explained above?

 

[abdus]

Yes, that's what I meant

You can use the same method for those as well. 

[pppws]

wow that was kinda easy – at least the setup for the fields and so much easier than repeaters.

i'm kinda struggling to understand what you are doing in the template with php. $jobs is supposed to be … ?

[abdus]

$jobs = $pages->find('template=profession, persons.count>0');

Find all professions that has at least one person referenced (i.e. where at least one person has this profession)

 

[pppws]

wohoo! thank you so much! you saved my day

if you have 5 more minutes you maybe can help me with two additional things:

1. i somehow need to filter the persons:

<?php foreach($j->persons as $p): ?> // what you wrote
<?php foreach($j->persons->find(lang=$langname) as $p): ?> // what i was thinking, but isn't working

2. how can i add a checkbox to the professions to mark the "current" job?

[abdus]

To mark the current job, you need a different approach, Page field only references other pages. For this situation, I'd pick the first job as the current one, (or the last) because it makes sense if you're ordering jobs chronologically.

Oh by the way, install Autocomplete input from Code modules (it's not installed by default) and change input type of the page fields to `Multiple page selection (sortable)` > `Autocomplete`. This way you can order job entries and simulate a chronological order.

Then in your templates you can do something like this

<ul class="persons">
    <?php foreach ($persons as $p): ?>
        <li class="person">
            Name: <?= $p->title ?>
            <?php
            $jobs = $p->jobs;
            $lastJob = $jobs->pop();
            ?>
            <ul class="person__jobs">
                <li class="job job--current">
                    Current Job: <?= $lastJob->title ?>
                </li>
                <?php foreach ($jobs as $j): ?>
                    <li class="job">
                        <?= $j->title ?>
                    </li>
                <?php endforeach; ?>
            </ul>
        </li>
    <?php endforeach; ?>
</ul>

On my multi language setup there's no `lang` field, I'm not sure it it's a native field. Is it a field that you've created yourself?

[pppws]

i've already installed the autocomplete module! suddenly it's much more fun to populate the fields ;).

hmm, there might be a view persons who have more than one current job, but i'll figure something out.

the lang field is a field i created myself. the persons pages are created on the frontend via the api, so i get the current user language and store it in the `lang` field. persons which were created on the english page should only be displayed on the english collection, the german persons only on the german collection and the spanish persons on the spanish.

[abdus]

Then I'd add another page field called `currentJobs` just like `jobs` field, and when outputting you can combine both to get the markup you want. A better way would be hooking into Page::saved to copy `jobs` field value to `currentJobs` and changing `currentJobs` input field to checkboxes. After you save the page, you can use checkboxes to mark the current jobs. 

I can try to see if this would even work, but I need to eat my lunch first. 

Also, this is where Page reference field falls short. It just references and nothing more. 

If you have ProFields, you can use Table field to get a very lean and performant repeater-like experience

[pppws]

since it is a project i'm doing for university there is no money involved to buy profields.
if you find the time to help me with the hook that would be great, otherwise i'll try the other option and create a current jobs/city field.

while you're at lunch i try to figure out how to filter the persons for my language field

enjoy your meal!

[abdus]

OK, I got it working. Using an extra `currentJobs` field works. 

• Go to `jobs` field and clone it from Actions tab
• Disable `create new pages` option
• Change inputfield type to `checkboxes`
• Add field to `person` template

Edit your /site/ready.php to include this hook

wire()->addHookAfter('InputfieldPage::getSelectablePages', function (HookEvent $e) {
    /** @var Page $page */
    $field = $e->object;
    if(!$field == 'currentJobs') return;

    $page = $e->arguments(0);
    $jobs = $page->jobs;
    $e->return = $jobs;
});

Once you add jobs to a person and save the page, currentJobs field is populated with jobs, and you can pick current jobs.

[abdus]

When building your markup, to decide whether a job is current you can use WireArray::has($item) method. 

<?php namespace ProcessWire;

$persons = pages('template=person');
?>

<ul class="persons">
    <?php foreach ($persons as $person): ?>
        <li class="person">
            Name: <?= $person->title ?>
            <?php if ($person->jobs->count): ?>
                Jobs:
                <ul class="person__jobs">
                    <?php foreach ($person->jobs as $job): ?>
                        <?php $isCurrent = $person->currentJobs->has($p); ?>
                        <li class="job <?= $isCurrent ? 'job--current' : '' ?>">
                            <?= $job->title ?>
                        </li>
                    <?php endforeach; ?>
                </ul>
            <?php endif; ?>
        </li>
    <?php endforeach; ?>
</ul>

 

[pppws]

wow wow wow. you're great!

unfortunately i just remembered that those fields get populated via api on the frontend, so this current job hook won't work? i'm actually thinking of buying the profields out of my own pocket, otherwise it seems to much of a trouble. do you think profields: table will perform good with about 200 person, each with 1-5 jobs, cities and countries? 

[abdus]

2 hours ago, pppws said:

1. i somehow need to filter the persons:

<?php foreach($j->persons as $p): ?> // what you wrote
<?php foreach($j->persons->find(lang=$langname) as $p): ?> // what i was thinking, but isn't working

In your first post, there's this line

<?php $langname = $user->language->title; //get current user language?>

What type of field is it? (You can use Page field for that too ) Is it a basic text field or what?

[pppws]

1 minute ago, abdus said:

In your first post, there's this line

<?php $langname = $user->language->title; //get current user language?>

What type of field is it? (You can use Page field for that too ) Is it a basic text field or what?

jep, it's basic text!

[abdus]

15 minutes ago, pppws said:

do you think profields: table will perform good with about 200 person, each with 1-5 jobs, cities and countries? 

Page table is basically a custom table in DB, so you can add any column you want. Instead of using pages like repeaters, with table field, every record is a row in DB with the columns you define.

$page->of(false); // turn off output formatting, if necessary
$award = $page->awards->makeBlankItem(); 
$award->title = "Tallest Building";
$award->date = "2014-05-01";
$award->category = "Engineering";
$award->url = "http://di.net/tallest/building/award/";
$page->awards->add($award); 
$page->save('awards'); 

As you see using Table field is very straightforward. But one glaring problem is to get 2-way relations for getting jobs a certain person have or people that have a certain job. For that you probably need a custom module that updates a job page with the person that added a job to its table field and vice versa.

But now thinking of it you can still use page fields `jobs` and `currentJobs` and make it work. I'm not sure how you render your form on the front end, but you can render a list of jobs with a checkbox to mark as current. Then when you post your data with AJAX or HTTP request, you add all jobs to `jobs` field and filter current jobs (that user checked) and add those to `currentJobs` field.

I'm not sure if that was clear, I can rephrase it better if you want.

[pppws]

i'm not quite sure if i undestand you right: the code is belongs to the page table? and the 2-way relations problem also?
 

the persons don't own users. they enter their e-mail adress in a form, by pressing submit a page is created and the 'status' of the page is set to 'editable'. the persons gets the page link via e-mail and can start filling in the blank input fields: jobs, loctions, name etc. 

i'm just confused what's the best way to achieve this – even more because i already got what i needed, it was just slow as hell :/.

 

[abdus]

Since I had nothing to do, I went ahead, and implemented a form as a proof of concept. This works with `jobs` and `currentJobs` setup.

When you submit and check the person page

Here's the code:

<?php namespace ProcessWire;
/* @var $input WireInput */
/* @var $sanitizer Sanitizer  */
/* @var $pages Pages  */
if(input()->requestMethod('POST')) {

    $jobs = [];
    foreach ($input->post->hasJob as $jobId) {
        $jobs[] = $sanitizer->int($jobId);
    }
    $currentJobs = [];
    foreach ($input->post->currentJobs as $currId) {
        // make sure person has the current job in his jobs
        if(!in_array($currId, $jobs)) continue;
        $currentJobs[] = $sanitizer->int($currId);
    }


    $p = new Page();
    $p->template = 'person';
    $p->parent = 1076;
    $p->name = 'name-lastname';
    $p->jobs = $jobs;
    $p->currentJobs = $currentJobs;
    $p->save();

    return $this->halt();
}



$jobs = pages('template=profession');
?>
<form method="POST">
    Pick your jobs:
    <table>
        <thead>
        <tr>
            <td></td>
            <td>Job Title</td>
            <td>Current?</td>
        </tr>
        </thead>
        <tbody>
        <?php foreach ($jobs as $job): ?>
            <tr class="job">
                <td><input class="job__has" id="job<?= $job->id ?>" type="checkbox" name="hasJobs[]" value="<?= $job->id ?>"></td>
                <td><label class="job__title" for="job<?= $job->id ?>"><?= $job->title ?></label></td>
                <td><input class="job__current" disabled type="checkbox" name="currentJobs[]" value="<?= $job->id ?>"></td>
            </tr>
        <?php endforeach; ?>
        </tbody>
    </table>

    <button type="submit" name="submit" value="1">Submit</button>

</form>
<script>
    let jobs = document.querySelectorAll('.job');
    [].slice.call(jobs).forEach(job => {
        job.addEventListener('click', function (e) {
            if(!e.target.classList.contains('job__has')) return;

            if (e.target.checked) {
                job.querySelector('.job__current').removeAttribute('disabled');
            } else {
                let current = job.querySelector('.job__current');
                current.checked = false;
                current.setAttribute('disabled', 'disabled');
            }
        });
    });
</script>

 

 

[pppws]

ahhhhhhhhh. you are fantastic – but the persons need to be able to enter new professions:

thats why i thought the hook for the current pages won't work?

[abdus]

19 minutes ago, pppws said:

the code is belongs to the page table

The code is to show how to add new records to a Table field

20 minutes ago, pppws said:

2-way relations problem

Unless you develop your own module, there's not a native way to handle 2-way relations from Table <--> Page Reference field. This means you'll still have to use a code similar to the one in the first post.

23 minutes ago, pppws said:

the persons gets the page link via e-mail and can start filling in the blank input fields: jobs, loctions, name etc

 

5 minutes ago, pppws said:

Can you show the whole form?

Also, one thing to note is that the pages you create from Page field input aren't created instantaneously. They're created once you submit (to perform permission checks, sanitizations etc). This would mean page has to reload to add new jobs in the first place, and you wont be able to dynamically render/populate currentJobs field using JS. 

So it seems you will have to either instruct user to save the page, or replace repeaters with Table field, or find other ways to shave time off of your process

[pppws]

ahh!

slight misunderstanding: those are two forms.

1) is just this, which creates the new pages and sends the url to the users e-mail address (the name of the pages is the md5hash of the email and gets replaced with the 'name' input from form w), which is working fine

 

2) when the user is opening the link in the e-mail something like that shows up:

 

i will have loops for:

• givenname
• year of birth
• year of moment
• last edited
• professions
• cities
• countries

 

[pppws]

@abdus

i don't mean to bug you. but do you have any tips? if not thats also fine!

i asked a friend – who bought the profields matrix – to give it a try and the setup still slows down massively when cycling through the fields. or is it only working faster with the table fieldtype? then i would buy it now.
 

thanks again for your enormous help yesterday! i really appreciate it!

[abdus]

I'd still keep the Page fields, and use two fields to determine all jobs and current jobs a person has. What you should do is to pull up your sleeves and get down to build a select field, maybe extend @ryan's asmSelect, or more modern alternatives (chosen.js select2.js etc) that have tagging support.

In fact, I had some free time and went ahead and created a signup form that does this. I used selectize.js for selecting/creating jobs on the frontend. Options load with AJAX at page load

Once you submit the form, you get this

On the backend, I get the input, sanitize it and filter values, if a new job name is specified, I create the page, and add it to `jobs` or `currentJobs`. Here's the code in all its glory:

<?php namespace ProcessWire;
/* @var $input WireInput */
/* @var $sanitizer Sanitizer */
/* @var $pages Pages */
if (input()->requestMethod('GET') && input()->urlSegment1 === 'jobs.json') {
    // serve json for input fields
    $jobs = pages('template=profession, sort=title');
    echo wireEncodeJSON($jobs->explode(['id', 'title']));
    return $this->halt();
} elseif (input()->urlSegment1) {
    // don't allow other url segments
    throw new Wire404Exception();
} elseif (input()->requestMethod('POST')) {
    $jobs = $input->post->jobs;
    $currentJobs = $input->post->currentJobs;

    // sanitize inputs
    $saneJobs = [];
    foreach ($jobs as $j) {
        $jobId = $sanitizer->int($j);
        $jobTitle = $sanitizer->text($j);

        if ($jobId) $saneJobs[$jobId] = false; // not current job
        elseif ($jobTitle) $saneJobs[$jobTitle] = false; // not current job
    }

    // check if a job is one of current jobs
    foreach ($currentJobs as $cj) {
        $currId = $sanitizer->int($cj);
        $currTitle = $sanitizer->text($cj);

        // if person has the job, set it to current
        if (isset($saneJobs[$currId])) $saneJobs[$currId] = true; // set to current
        elseif (isset($saneJobs[$currTitle])) $saneJobs[$currTitle] = true; // set to current
    }

    // save info
    $person = $pages->get(1228);
    $person->of(false);
    $addableJobs = [];
    $addableCurrents = [];
    foreach ($saneJobs as $j => $isCurrent) {
        // $j is job id or new job title

        $job = null;

        // does job already exist?
        if (is_int($j)) {
            $job = $pages->get($j);
        } else {
            // create a new job and add it
            $job = new Page();
            $job->template = 'profession';
            $job->parent = 1027;
            $job->name = $sanitizer->pageName($j);
            $job->title = ucwords($j); // already sanitized, but capitalize it
            $job->save();
        }

        if (!$job) continue;

        $addableJobs[] = $job;
        if ($isCurrent) $addableCurrents[] = $job;
    }
    $person->jobs->add($addableJobs);
    $person->currentJobs->add($addableCurrents);
    $person->save();
}


?>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/selectize.js/0.12.4/js/standalone/selectize.js"></script>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
<link rel="stylesheet" href="https://rawgit.com/selectize/selectize.js/master/dist/css/selectize.bootstrap3.css">

<div class="flex">
    <form method="POST" class="signup-form">
        <div class="form-group">
            <label for="name">Name</label>
            <input name="name" required type="text" class="form-control" id="name" placeholder="Name Lastname">
        </div>

        <div class="form-group">
            <label for="jobs">Your Jobs</label>
            <select name="jobs[]" required id="jobs" multiple class="form-control">
                <option>Loading...</option>
            </select>
        </div>


        <button class="btn" type="submit">Save</button>
    </form>
</div>

<script>
    $jobs = $('#jobs');

    // fetch all jobs
    $.getJSON('./jobs.json', function (data) {
        initSelect(data);
    });

    function initSelect(data) {
        const makeOption = (it) => ({text: it.title, value: it.id});
        let selected = data.splice(0, 5).map(makeOption);
        window.jobs = data;

        $jobs.selectize({
            create: true,
            render: {
                item: function (data, escape) {
                    return `<div class='option'>
                                <span class="option__label">${escape(data.text)}</span>
                                <input class="option__current" type="checkbox" name="currentJobs[]" value="${escape(data.value)}">
                            </div>`;
                }
            },
            options: data.map(makeOption),
            // populate the field with the jobs person already has
            // items: data.splice(1,4).map(i => i.id)
        });
    }
</script>


<style>
    .flex {
        display: flex;
        justify-content: center;
        margin-top: 10rem;
    }

    .signup-form {
        width: 500px;
        margin: auto;
    }

    .option {
        width: 100%;
    }

    .option:after {
        content: '';
        display: table;
    }

    .option__label {
        float: left;
    }

    .option__current {
        float: right;
    }
</style>

 

[abdus]

I think it's important to sync `jobs` `currentJobs` fields such that `currentJobs` doesnt have more jobs than `jobs` field has. You can use the following hook to update `currentJobs`.

wire()->addHookBefore('Pages::saveReady', function (HookEvent $event) {
    $page = $event->arguments('page');
    if ($page->template != 'person') return;

    // make sure currentJobs items <= jobs items
    if ($page->isChanged('jobs')) {
        foreach ($page->currentJobs as $cj) {
            if (!$page->jobs->has($cj)) $page->currentJobs->remove($cj);
        }
    }
});