Osclass forums

Support forums => General help => Topic started by: Aficionado on September 05, 2015, 12:47:09 am

Title: Osclass Extreme Memory Usage with Multi-Country
Post by: Aficionado on September 05, 2015, 12:47:09 am
As you can see in the image capture, i (try to) run one Osclass with 22 countries. Right now it has less than 2000 ads, so it is practically empty.

BUT i hit memory limits all the time. I increased from 256 to 392 and now to 512. The errors of short memory are when adding a listing or Editing a listing and who knows where else. I run 2 plugins, nothing special, same as my other 4 Osclass installations (not multi-country).

I'm not a programmer but I think a better strategy of countries/regions/cities should be adopted by Osclass. It is absurd to need 512mb of memory just to Edit a post. All themes i have tried.

Or maybe it is a bug somewhere ?


 

Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: Aficionado on September 05, 2015, 12:55:27 am
Here are some errors, very strange, because when i increase the allowed memory, the "needed" memory also increase (in the error). Doesn't seem right to me.

[04-Sep-2015 18:32:52 Europe/London] PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 87 bytes) in /home/oc-includes/osclass/classes/database/DBRecordsetClass.php on line 186

[04-Sep-2015 18:33:49 Europe/London] PHP Fatal error:  Allowed memory size of 411041792 bytes exhausted (tried to allocate 7864320 bytes) in /home/oc-includes/osclass/frm/Form.form.class.php on line 26

[04-Sep-2015 18:38:08 Europe/London] PHP Fatal error:  Allowed memory size of 411041792 bytes exhausted (tried to allocate 7602176 bytes) in /home/oc-includes/osclass/frm/Form.form.class.php on line 26

The above two last errors are with 392 mb. Now i'm at 512 and that is the max i can have. But with 512 of memory, with several visitors at once, it will crash again.
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: SmaRTeY on September 05, 2015, 01:07:21 am
Mmmm not good.
Do all these countries and subs get loaded in item post/edit page? (that could be a memory hog)

Regards,
Eric
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: Aficionado on September 05, 2015, 01:11:02 am
Mmmm not good.
Do all these countries and subs get loaded in item post/edit page? (that could be a memory hog)

Regards,
Eric

Eric i have no idea how Osclass works, as i said i'm not a programmer. I only know my way around php code, as needed for a webmaster.

And apart from France, Germany and Italy, the rest of countries have little data as regions/cities.

Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: frosticek on September 05, 2015, 01:53:19 am
@Aficionado
There are 2 options (I think) locations can be loaded, depending on theme:
1) countries, regions and cities are loaded when item_post.php is opened > this is very bad when there is many locations
2) countries are loaded, regions & cities are loaded via ajax on request (when you choose Germany, only regions from Germany are loaded)

Second should not cause problems with locations. It might be good to enable sql debug on your osclass to see what SQL queries were called on page load.
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: Aficionado on September 05, 2015, 04:03:54 am
@Aficionado
There are 2 options (I think) locations can be loaded, depending on theme:
1) countries, regions and cities are loaded when item_post.php is opened > this is very bad when there is many locations
2) countries are loaded, regions & cities are loaded via ajax on request (when you choose Germany, only regions from Germany are loaded)

Second should not cause problems with locations. It might be good to enable sql debug on your osclass to see what SQL queries were called on page load.

Both Modern and OsclassWizards theme seem to use (2). Not 100% sure, but i think they do.

Memory problems with multi-country exist from the day i joined Osclass users. I do remember 2+ years ago opening a github issue about the recalculations stats during cron that crashed with more than 3-4 countries. Daniel did a mod in Osclass do to that in several "pieces" and not at once.

But i see now that i'm hitting an other wall, again with memory. And i can't go beyond 512mb.
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: frosticek on September 05, 2015, 10:40:35 am
@Aficionado
That is the issue with locations that there can be a lot of them... Were you able to setup sql debug?
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: SmaRTeY on September 05, 2015, 03:25:52 pm
There is one other option, some kind of memory leak on your server either your own or when on shared hosting due to some other site.
Also it is possible system is not releasing used memory meaning a steady memory growth leading up to memory boundary, looks like a leak but actually is different.

I *think* 512 should be enough for your setup, can you have a look in your cpanel and see memory usage per application or something before it crashes?


Regards,
Eric
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: Aficionado on September 05, 2015, 05:21:22 pm
@Aficionado
That is the issue with locations that there can be a lot of them... Were you able to setup sql debug?

I did but nothing was written and no file was created. Strange, i will see this during the weekend.

Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: Aficionado on September 05, 2015, 05:28:50 pm
There is one other option, some kind of memory leak on your server either your own or when on shared hosting due to some other site.
Also it is possible system is not releasing used memory meaning a steady memory growth leading up to memory boundary, looks like a leak but actually is different.

I *think* 512 should be enough for your setup, can you have a look in your cpanel and see memory usage per application or something before it crashes?


No, that is not a hosting problem, but an Osclass problem (or a theme problem).

Of cource 512 is enough for my case, that is why i try  figure this out. The memory usage on my server is low, less than 100-150 mb and i'm on a clould linux. Cloud Linux pretty much isolates at 99% the hosting plans so they can't mess up with each other. No one can (for example) overtake the server CPU / Memory etc etc.

Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: SmaRTeY on September 05, 2015, 05:34:26 pm
Okay, so you have same issue with other theme(s) or is there a theme where you do not have this issue?
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: Aficionado on September 05, 2015, 05:38:11 pm
Okay, so you have same issue with other theme(s) or is there a theme where you do not have this issue?

I have tried Modern theme, Bender and OsclassWizards free theme. It happens in all of them. But OsclassWizards is based on Bender, so if Bender has this bug then it was inherited to others also.

Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: SmaRTeY on September 05, 2015, 05:45:22 pm
Mmmmzzz sounds like a core issue more than a theme issue, besides what already has been said I have no clue what could be bugging...
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: Aficionado on September 05, 2015, 05:53:32 pm
@Aficionado
That is the issue with locations that there can be a lot of them... Were you able to setup sql debug?

Ok, for some reason the log file had to be created (initially) manually and here it comes.
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: frosticek on September 05, 2015, 06:29:51 pm
@Aficionado
I have checked your file and found that there is more queries running for long time (8 or more seconds), there is probably relation to city table in all of them. It is very strange.
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: Aficionado on September 05, 2015, 06:44:47 pm
@Aficionado
I have checked your file and found that there is more queries running for long time (8 or more seconds), there is probably relation to city table in all of them. It is very strange.

And that is related to memory problems  ?

Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: teseo on September 05, 2015, 06:45:38 pm
Hi,

@frosticek, those are exponential numbers (9.7036361694336E-5), in fact they are very small values.

This is what bothers me:

Code: [Select]
1.6615619659424
**************************************************
SELECT pk_i_id, fk_i_region_id, s_name, fk_c_country_code, b_active, s_slug
FROM (oc_t_city)
ORDER BY s_name ASC
--------------------------------------------------

What module would need to retrieve absolutely all the cities in your database in alphabetical order? ???

@Aficionado, have you tried using another theme?

Regards
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: Aficionado on September 05, 2015, 06:51:58 pm


@Aficionado, have you tried using another theme?

Regards

Yes, Bender and Modern. Same problem. But OsclassWizards is based on Bender, so ... if Bender had this ....

I will spend this weekend to solve this, since i can't bear it anymore. If Osclass can't handle multi-countries then i will have to find something else for this case.

Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: frosticek on September 05, 2015, 07:14:09 pm
@teseo
Ah yes I missed that.
I have same query on my theme, but with filter on region ID.
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: teseo on September 05, 2015, 08:06:27 pm
...
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: teseo on September 05, 2015, 10:09:42 pm
That query is produced by osc_get_cities() helper with no region parameter. Also, it might appear as City::newInstance()->listAll();

Now, in vanilla Bender is only used in user-profile.php, and even if the user still hasn't set any location it would retrieve only the cities of the 1st country in your database.

So, you must have something like that in a plugin or in some custom modification. ???

Regards
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: Aficionado on September 05, 2015, 10:29:17 pm
That query is produced by osc_get_cities() helper with no region parameter. Also, it might appear as City::newInstance()->listAll();

Now, in vanilla Bender is only used in user-profile.php, and even if the user still hasn't set any location it would retrieve only the cities of the 1st country in your database.

So, you must have something like that in a plugin or in some custom modification. ???

Regards

I have no modifications related to that. But i have some plugins from Osclass team, like rich edit etc etc. I will disable everything and test with stock Bender and come back to post.

But i do remember same problems with Modern theme, still i will try again.

Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: Aficionado on September 06, 2015, 01:57:15 am
ok. some update on this for anyone who cares to read.

For some reason Bender i was testing was Bender 2.01 and i installed Bender 3x. With Bender 3 i see no memory errors with 256 mb.

So, without being 100% sure, i think it is theme related. I will do some more testing with other themes and see how it goes.

Editing an ad WITHIN admin/Listings there is no error whatever. So one more reason to see if it is theme related.
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: teseo on September 06, 2015, 02:21:11 am
***CORRECTED AGAIN***

It's a theme issue. Just seen a reference to Osclasswizards on your queries log.

Locate this in item-post:

Code: [Select]
ItemForm::city_select(osc_get_cities(),osc_user());
an replace with:

Code: [Select]
ItemForm::city_select(osc_get_cities(osc_user_region_id()), osc_user());
That seems to be working all right with no loading of all cities anymore.

Regards
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: Aficionado on September 06, 2015, 02:43:01 am
@Teseo

Dude i don't know HOW TO THANK YOU !!

THANKS !!!

That did solve the problem and also some strange timeouts when editing.

I will copy your fix to their forum.

Again, a big Thanks.

Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: teseo on September 06, 2015, 02:56:27 am
...
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: Aficionado on September 06, 2015, 03:02:55 am
@Teseo

Are the above fixes only for multi-countries ?

Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: SmaRTeY on September 06, 2015, 03:09:44 am
Nice find teseo! 8)

One question looking at the whole thread here, were ALL cities (from all countries) being loaded using that initial code??


Regards,
Eric
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: dev101 on September 06, 2015, 01:08:35 pm
I was completely absent from the forums past few days and I missed this exciting discussion.
Although teseo already did a great job, I would just add a different approach for solution:

Code: [Select]
ItemForm::city_select(osc_get_cities(osc_user_region_id()), osc_user());

ItemForm class is still being used (instead of UserForm in teseo's solution) which keeps things as it should.

Regards
dev101
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: teseo on September 06, 2015, 02:42:22 pm
Hi @dev101,

Yep, I was looking at the only use of osc_get_cities() -with no region parameter- in Bender vanilla, in user-profile.php and I used Userform:: inadvertedly, same result anyway.

Now I realize the solution you propose is what is recommended here:

https://doc.osclass.org/Changing_between_drop-down_and_autocomplete_for_locations

And seems good enough. If it's a non-logged user or a logged user with no location data in his/her profile yet, that would retrieve cities for the first country in the database.

My hack using that dummy array (1,2) intended to save time and skip population of a select that will be overriden later via Ajax call (once you expressly select a region), but seems you need something more there for the neutral option "Select a city" to be shown...

What is totally clear is that osc_get_cities() -with no region parameter- never should be used unless (as in the case on that user-profile.php) already a View variable "cities" has been created.

Thanks for your input, regards.
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: SmaRTeY on September 06, 2015, 03:06:35 pm
Quote
What is totally clear is that osc_get_cities() -with no region parameter- never should be used unless (as in the case on that user-profile.php) already a View variable "cities" has been created.

This memory issue was bound to be related to something like this looking at the amount of countries Aficionado is using and imagining how many cities this will fetch....
I can even find it funny to be honest though the issue itself is not funny at all! ;D

We have to thank Aficionado for putting so many countries to the test making us aware of pitfalls that otherwise might slip through.


Regards,
Eric
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: frosticek on September 07, 2015, 10:15:27 am
@dev101
Your solution will fail in case osc_user_region_id() is empty, I think for unregistered user it is still empty.
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: SmaRTeY on September 07, 2015, 11:18:55 am
I think you're right Frosticek, in that case a pre-liminary check on region and a 'default' region would solve that or similar trick teseo used where population of (dropdown) fields is put on hold till user selects location data.

Actually what the 'issue' tells us developers is that we have to be 'smarter'/'offer better quality' with regard to usability in this case related to location input and what Osclass offers us. 8)
Title: Re: Osclass Extreme Memory Usage with Multi-Country
Post by: dev101 on September 07, 2015, 12:24:17 pm
@ frosticek
Yes, you are correct (I don't allow guests), but in that case js will not act until region is selected ;)