Help optimizing PHP memory usage

[Tyssen]

This isn't strictly a Processwire question, but it is being used on a PW site, so if anyone has any ideas about this question, I'd be most grateful.

[MoritzLost]

What exactly is your use case here? I haven't completely cracked your code yet, you want only unique combinations, right? So only one of (A + B) and (B + A)? In this case, you're looking at n^2 possible combinations (I guess that's called a Power Set, but math theory is not my strong suit). You mention 14 elements as the limit at which you run out of memory. In this case, you're looking to generate 2^14 = 16384 combinations (at least it's a step up from ALL possible combinations, which would be 14! = 87178291200 ?). Why do you need that many? Given your variable names, I assume this is for some kind of product configurator on a store page? In this case, you're not going to present 16 thousand possible combinations anyway. Couldn't you just calculate the price for a given combination on the fly?

Sorry if I'm rambling, maybe you really need this. In this case, I'd say your solution needs to be a lot more imperative and less functional. Recursive functions are kinda RAM-heavy, as far as I'm aware PHP doesn't even have tail call optimizations, so all those intermediate results are staying in RAM. Also, you're passing the $addOns and $base arrays by value, so those are duplicated for every function call. So using simple for- and while-loops instead of a recursive function might already be enough to reduce memory usage a lot.

[Tyssen]

Ideally, yes, there would only be unique combinations. But it's not been set up to filter them out yet because it's not strictly necessary. The reason for that is a little hard to explain.

This is for a site that uses Snipcart. The way Snipcart works, you click on a button which has data-attributes for price and ID (and other things) on it and it adds the item to the cart and then at the end of the process, by way of validation, Snipcart returns to your page to check that a button exists with the price and ID that you added to your order. You can either have hidden buttons in the page with the data-attributes that match the order, or you can instruct Snipcart to use a JSON file which contains all that info instead.

This is why changing prices on the fly doesn't work, because the price that you've arrived at by selecting different combinations of options, has to exist somewhere that the Snipcart validator can access. So to answer another of your other questions, yes I do need to be able to present several thousand different combinations to the Snipcart validator, not a site visitor, otherwise the order won't validate.

[Tyssen]

And in case you're wondering, Snipcart does have different price modifier data-attributes that can be used which would avoid all of this, but… and this is why I'm still having to do it this way… you can only choose one option, e.g. if you buy a shirt it might come in S, M or L and the price might go up the bigger it gets. But you're only choosing one modifier.

In my case I have a base product which can be ordered for 1, 3, 6 or 12 months (which changes the base price), but it can also have a range of optional add-ons added to it which also change the price.

[MoritzLost]

That sounds ... bad ? Are you sure there isn't a more convenient way to do this in Snipcart?

Anyway, as far as I can tell your solution is producing the correct result, so the only problem is the memory consumption. You could try to build the JSON on the fly instead of pushing everything on one big array. That is, open a file handle, encode individual lines in your recursive function and write them directly to the file. Might be a bit slower, but wouldn't require holding all combinations in memory. Though you gotta take care to have valid JSON in the end, i.e. no trailing comma for the last line and stuff like that.

[LostKobrakai]

It doesn't seem like addons also get a multiplier applied. Can you not just have the addons be "additional products" then?

Edit: To save on raw memory you can also calculate the combinations just by id, while calculating the price one object at a time. Currently you create an object for each permutation.

[LostKobrakai]

Another idea: You could generate permutations in advance from like an array [1, 2, 3, 4, 5, 6, 7, 8, …], write them each in a file per line. Then stream the file and use the numbers in each line to access the addon at index X. It's still a lot to iterate and creating those files (needs one per number of addons) is a task at raw computing power, but at least the json will probably be created quicker.

[teppo]

On 10/8/2020 at 12:00 PM, Tyssen said:

... and then at the end of the process, by way of validation, Snipcart returns to your page to check that a button exists with the price and ID that you added to your order. You can either have hidden buttons in the page with the data-attributes that match the order, or you can instruct Snipcart to use a JSON file which contains all that info instead.

This is why changing prices on the fly doesn't work, because the price that you've arrived at by selecting different combinations of options, has to exist somewhere that the Snipcart validator can access. So to answer another of your other questions, yes I do need to be able to present several thousand different combinations to the Snipcart validator, not a site visitor, otherwise the order won't validate.

It seems that whatever route you take, the memory / performance optimization approach would eventually run off the cliff. I mean... it's just not scalable. So, trying to think outside the box for a moment:

• When you say that "Snipcart returns to your page", what does that mean specifically? Does it perform a GET/POST request against the page?
• If so, is there any chance that it would provide enough information for you with that request to actually generate a smaller subset of prices used in that specific order?

This would technically shift the responsibility for the validation step from Snipcart to your site, but it also seems like something that could scale much better ?

Anyway, just throwing in ideas.

[OLSA]

Sorry if I don't understand your question well, but I just try Snipcart integration on this temporary url (PW site), and get that all works fine (ordering, validation, complete procedure). There I use url params to get that same product item (image) have different price "per frame" size. Also with that approach, product can have and other attributes (and depending on that different prices).  As example, same image in small size frame , or in large frame size have different prices. Or try to "buy" multiple products and after that check cart (and if you want, finish order with demo card data (will be displayed in the last step)).

Please can you check is that what you want to get? If it is ok I will post here that very simple solution
But, also I think there is option with PW url segments and dynamically generated page (url depends on your addons/options).
Regards.
 

[Tyssen]

@OLSA In my case, I have a base product which can have a variety of different add-ons added to it. So base price might be $50 and you might have 10 add-ons which are all different prices. So the final price could be $50 + add-on A + add-on B but it could be $50 + every permutation of 10 different numbers added to each other, e.g.

A+B
A+B+C
A+C
A+B+C+D
B+C+D
etc.

For everyone else who has replied so far, thanks for your suggestions. I'm actually looking into using https://github.com/violet-php/streaming-json-encoder as it will avoid having to load the whole data set into memory, which is similar to what @LostKobrakai suggested.

Quote

Can you not just have the addons be "additional products" then?

No, because in some cases, the add-ons are for subscription products with Stripe. If they were separate products, then each one would need to be a separate subscription which then gets messy to manage for the customer and client.

[OLSA]

32 minutes ago, Tyssen said:

In my case, I have a base product which can have a variety of different add-ons added to it. So base price might be $50 and you might have 10 add-ons which are all different prices. So the final price could be $50 + add-on A + add-on B but it could be $50 + every permutation of 10 different numbers added to each other, e.g.

Sorry @Tyssen but we do not understand each other, I still don't see a problem to get that.
Best regards.