Vineet Sawant

How to export all user emails to a CSV file?

Recommended Posts

Hello,

I've a website with over 40k registered users who sign up using their email IDs. We've a ticket booking service and we wish to alert users about certain functionality changes in the site.

Whenever I'm trying to print all the email using foreach, page is going blank due to memory issue I suppose (it takes ages to respond).

Is there any way or any module which will export all the emails to a single CSV file?

Thanks.

Share this post


Link to post
Share on other sites

I don't think there's a module for this, but it's kinda easy to circumvent timeouts. You'd build a php file that calls a reasonable number of users (1000 or so) using the pagination feature of processwire. The results are then written / appended to a file somewhere in the system. And the php then returns the next pagenumber, at best as simple json string. Now you'd just build some js, which calls the same page via ajax and iterates on as soon as the response is returned. After 40 calls you've your full csv. 

Another way would be using some SQL, to get all user id's and select the emails directly from the email fields table. 

  • Like 1

Share this post


Link to post
Share on other sites

Another option is to call $pages->uncacheAll() at a regular interval. This helps you avoid memory issues in most cases, and allows you to run the whole query at once. For a rough example, see how it's done in VersionControl.module.

If you do take this route, make sure that your maximum execution time isn't an issue here either. If your script takes a long time to run, you'll usually want to increase the max execution time using set_time_limit() first. The default value is typically 30 seconds, and a script running longer than that gets killed.

First things first, though: you mentioned that the page goes blank supposedly because of a memory issue. If you're not 100% sure of the reason for the issue, the very first thing you should do is checking why it actually happens. Do the same thing in a test environment with $config->debug = true, or see your error logs on the live site. Assuming is never a good strategy :)

  • Like 2

Share this post


Link to post
Share on other sites

This worked really great for me in a site with thousands of pages

$selector = "template=pages_template"; // as an example

while (1) {
    $p = wire('pages')->get("{$selector}, id>$id"); // get page with id bigger than previous
    if(!$id = $p->id) break; // assign current page's id to $id or break the loop if it doesn't exist

    // do stuff using $p as the current page

    wire('pages')->uncacheAll();
};

Like this you never have an array in memory, only one page at each time, and you can do all the operation in one go. Of course you would probably have to constantly write to the csv file (like, open,write,close,open,write,close,open,write,close  edit: this doesn't make any sense you can open, write, write, write,... close), but I don't think that would be a problem.

Edit: Ideally you would do this from the terminal by bootstrapping PW, to prevent overloading apache.

  • Like 3

Share this post


Link to post
Share on other sites

The code on the post above is not full proof, if the process is interrupted for any reason, you would have to start over. The ideal would be to, on each iteration, read the last line of the file, identify the user, find the next user, and write the next line.

  • Like 1

Share this post


Link to post
Share on other sites

I ended up using limit with foreach & pagination.

I also found out that start=n in the selectors breaks pagination module. It stops working. Pagination renders page numbers but after clicking the page, e.g., 3 still shows data from page 1. I guess it happens cause I'm using start=n. When I removed, it started working just fine.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By NorbertH
      I have trouble exporting fields via the buildin export. 
      For example when i export a single field  i get:
      { "bestellung_status_name": { "id": 194, "type": "FieldtypeText", "flags": 0, "name": "bestellung_status_name", "label": "Status Intern", "textformatters": [ "TextformatterEntities" ], "collapsed": 0, "minlength": 0, "maxlength": 100, "showCount": 0, "size": 0, "pattern": "[a-z\\A-Z\\(\\)]+", "showIf": "", "themeInputSize": "", "themeInputWidth": "", "themeOffset": "", "themeBorder": "", "themeColor": "", "themeBlank": "", "columnWidth": 100, "required": "", "requiredAttr": "", "requiredIf": "", "stripTags": "", "placeholder": "" } } exporting a secon single field i get :
      { "bestellung_status_name_ext": { "id": 195, "type": "FieldtypeText", "flags": 0, "name": "bestellung_status_name_ext", "label": "Status Extern", "textformatters": [ "TextformatterEntities" ], "collapsed": 0, "minlength": 0, "maxlength": 100, "showCount": 0, "size": 0, "pattern": "[a-z\\A-Z\\(\\)]+", "showIf": "", "themeInputSize": "", "themeInputWidth": "", "themeOffset": "", "themeBorder": "", "themeColor": "", "themeBlank": "", "columnWidth": 100, "required": "", "requiredAttr": "", "requiredIf": "", "stripTags": "", "placeholder": "" } } So far everything works fine .
      When i try to export both fields together  i get only an error message :
      Call to a member function getModuleInfo() on null File: .../wire/modules/Fieldtype/FieldtypeText.module:171 I added " bd($textformatter);" on line 170 to see whats wrong. so have a look at the screenshot i apended to this post.
       
      Its perfectly possible that one textformater module got removed by accident while experimenting whith some textformaters but my question is how to fix this maybe somewhere in the DB and possibly what went wrong?
      ProcessWire 3.0.120 © 2019
      Apache/2.4.25 (FreeBSD) OpenSSL/1.0.2k mod_fcgid/2.3.9
      PHP 7.1.2
           

       
      Edit: After adding
      if ($textformatter ===NULL) continue; I can export my fields , as there arent any Textformater missing in the fields , i get a perfect result. but still there is one textformater whith a NULL value.  
       
       
    • By SwimToWin
      I have a website that allows users to create their personal "website" (a page with sub-pages).
      Users shall be able to:
      Log in (frontend and/or admin), Edit "their" page(s) - I am using the "Page Edit Per User"-module (https://modules.processwire.com/modules/page-edit-per-user/) to grant access to the relevant pages Create child pages - possible? Users shall not be able to see other pages in the admin interface - "Admin Restrict Page Tree" may do the trick (https://modules.processwire.com/modules/admin-restrict-page-tree/)? Frontend editing shall be possible - I am considering "Fredi" (https://modules.processwire.com/modules/fredi/) for this. The challenge is that it takes a lot of modules and configuration.
      Is there a way to set this up that doesn't require a lot of configuration for each new user?
    • By psy
      I'm combining two PW sites into one, Site A into Site B.
      At each step, I did it bit by bit as the 'all at once' approach failed.
       
      First, I exported all the fields from Site A and imported into Site B. Any field types not supported by import/export, eg FieldtypeOptions I manually recreated. All good.
      Next I exported all the templates from Site A and imported them into Site B and copied across their associated template files. All good.
      Finally I exported the pages I needed from Site A into Site B - again, bit by bit to ensure it all went smoothly.
      From the admin side, it all looked and worked perfectly.
      Front end was a totally different story. All existing pages in Site B worked as expected. NONE of the pages imported from Site A displayed. They all ended in a redirect loop with no errors in the PW logs or Tracy Debugger.
      After some trial-and-error, I finally got it working with:
      - create a new template in Site B admin with no associated template file and just a title field
      - import the fields from the imported Site A template into the newly created template (both on Site B)
      - copy the Site A php template file into a new file that matched the new PW Site B template name and save in Site B site/templates
      I can deal with the above workaround. Just curious to know if I did something wrong or if the template import/export feature is problematic?
       
      ### Solution:
      While the export/import was a slow process, turned out the front end redirecting issue was unrelated. For reasons unknown, all templates marked as HTTPS only were the ones redirecting, ie all templates from Site A. Finally solved it by changing the $config->https to true in site/config.php
      Now the pages display correctly as https whether the template forces the issue or not.
       
    • By Noel Boss
      To create a new gitlab issue, I'd like to send an Email to the following email address using FormBuilder: 
      incoming+account/repository@incoming.gitlab.com
      unfortunately, the / in the email gets striped by the sanitizer:
      wire('sanitizer')->email(incoming+account/repository@incoming.gitlab.com'); // output: incoming+accountrepository@incoming.gitlab.com Is there any way to configure the sanitizer? Any other Ideas how to send the mail to this address?
       
      Ps.: I can not define the email format as it is defined by gitlab. The format is also kind of "common" and known as email-alias in the form of emailaccount+alias@domain.com – gmail uses it to label mails…
      https://help.wodify.com/hc/en-us/articles/234441508-How-Do-I-Create-an-Email-Alias-
    • By louisstephens
      I was attempting to export some fields from my dev branch to move them over to a live site when I got the following error:

      Has anyone experienced this before? I was thinking I could just write a script using the api to create the fields, but there are about 44 fields (2 are repeater matrix) that are all slightly unique. If anyone has experienced this, what was your work around?