Jump to content
gebeer

Image field performance with large number of images

Recommended Posts

Hello everybody,

I'd like to use the images field as an upload UI.

One image field would hold 1000+ images and I am wondering how this would perform in the backend.

I would only need the upload functionality. No need for thumbnail display, ordering etc.

Has anyone experience in using an image or file field with very large quantities of images?

Or is there an alternative upload UI for PW that stores all images to a single folder?

EDIT: I don't need to upload 1000+ images in one go.

Share this post


Link to post
Share on other sites

I think a file field could handle that in an acceptable manner (no thumbnails there), but I doubt the loading-time on page-edit will be sufficient. I'd think you'd need something paginated (maybe filterable) to control such an amount of data.

Edit: If you really need only the upload and nothing else, then I'd use a custom inputfield for fieldtype file, which doesn't load all the images at once, which is most likely the biggest issue to solve.

Share this post


Link to post
Share on other sites

So the bottleneck would be the thumbnail-display.

Couldn't we just switch that off through a hook or take the image input field module as a template and remove the part for thumbnail display?

I just realised that I can use the file field for handling the uploading of images (in the end they're files).

I will try this with a large amount of images and report back.

Share this post


Link to post
Share on other sites

The thumbnails sure are a bottleneck, but as we now both know, that's not the case for a file field. The only potential bottleneck for a file-field could be, that the inputfield will load all your files to show them, which is not only a large list, but also lots of data.

Share this post


Link to post
Share on other sites

Thanks for the hint.

I don't really need a list of files. Just the uploading part.

So I will have a look at the InputFieldFile module and see if I can deactivate the listing of files.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By MateThemes
      Hello everyone!
      I have a markup and image field question:
      I have a markup in which a gallery preview should be displayed on home page. This gallery have a special markup, 3 of 4 images have special image ratio.

      The gallery looks like this. 
      The html markup is like this:
      <div class="uk-section-muted" uk-scrollspy="target: [uk-scrollspy-class]; cls: uk-animation-slide-top-small; delay: 200;"> <div data-src="./assets/images/arrahof/home-restaurant-bg.svg" class="uk-background-norepeat uk-background-contain uk-background-top-center uk-section uk-section-large" uk-img> <div class="uk-container uk-container-small"> <div class="uk-margin-large" uk-grid> <div class="uk-width-1-1@m"> <h2 class="uk-text-center" uk-scrollspy-class>Das Angebot im ArraHof</h2> </div> </div> <div class="uk-grid-small uk-grid-margin-small" uk-grid> <div class="uk-width-expand@s"> <div class="uk-margin-remove-vertical uk-text-center" uk-scrollspy-class> <a class="el-container uk-inline-clip uk-transition-toggle uk-link-reset" href="#"> <img class="el-image" data-src="https://via.placeholder.com/610x604" data-sizes="(min-width: 610px) 610px" data-width="610" data-height="604" alt="Placeholder Image" uk-img> <div class="uk-overlay-default uk-transition-fade uk-position-cover"></div> <div class="uk-position-center uk-position-small"> <div class="uk-overlay uk-padding-large uk-transition-fade uk-margin-remove-first-child"> <h3 class="el-title uk-h4 uk-heading-divider uk-transition-slide-top-small uk-margin-top uk-margin-remove-bottom">Lorem Ipsum</h3> <div class="el-content uk-panel uk-transition-slide-bottom-small uk-margin-top">Comfort Food</div> </div> </div> </a> </div> </div> <div class="uk-width-expand@s"> <div class="uk-margin-remove-vertical uk-text-center" uk-scrollspy-class> <a class="el-container uk-inline-clip uk-transition-toggle uk-link-reset" href="#"> <img class="el-image" data-src="https://via.placeholder.com/610x604" data-sizes="(min-width: 610px) 610px" data-width="610" data-height="604" alt="Placeholder Image" uk-img> <div class="uk-overlay-default uk-transition-fade uk-position-cover"></div> <div class="uk-position-center uk-position-small"> <div class="uk-overlay uk-transition-fade uk-margin-remove-first-child"> <h3 class="el-title uk-h4 uk-heading-divider uk-transition-slide-top-small uk-margin-top uk-margin-remove-bottom">The Williams family</h3> <div class="el-content uk-panel uk-transition-slide-bottom-small uk-margin-top">Comfort Food</div> </div> </div> </a> </div> </div> </div> <div class="uk-grid-small uk-grid-margin-small" uk-grid uk-height-match="target: .uk-card; row: false"> <div class="uk-width-2-3@s"> <div class="uk-margin-remove-vertical uk-text-center" uk-scrollspy-class> <a class="el-container uk-inline-clip uk-transition-toggle uk-link-reset" href="#"> <img class="el-image" data-src="https://via.placeholder.com/610x400" data-width="610" data-height="400" alt="Placeholder Image" uk-img> <div class="uk-overlay-default uk-transition-fade uk-position-cover"></div> <div class="uk-position-center uk-position-small"> <div class="uk-overlay uk-transition-fade uk-margin-remove-first-child"> <h3 class="el-title uk-h4 uk-heading-divider uk-transition-slide-top-small uk-margin-top uk-margin-remove-bottom">The Williams family</h3> <div class="el-content uk-panel uk-transition-slide-bottom-small uk-margin-top">Comfort Food</div> </div> </div> </div> </a> </div> <div class="uk-width-expand@s"> <div class="uk-margin-remove-vertical uk-text-center" uk-scrollspy-class> <a class="el-container uk-inline-clip uk-transition-toggle uk-link-reset" href="#"> <img class="el-image" data-src="https://via.placeholder.com/610x820" data-sizes="(min-width: 610px) 610px" data-width="610" data-height="820" alt="Placeholder Image" uk-img> <div class="uk-overlay-default uk-transition-fade uk-position-cover"></div> <div class="uk-position-center uk-position-small"> <div class="uk-overlay uk-transition-fade uk-margin-remove-first-child"> <h3 class="el-title uk-h4 uk-heading-divider uk-transition-slide-top-small uk-margin-top uk-margin-remove-bottom">The Williams family</h3> <div class="el-content uk-panel uk-transition-slide-bottom-small uk-margin-top">Comfort Food</div> </div> </div> </div> </a> </div> </div> </div> <div class="uk-margin-large" uk-grid> <div class="uk-width-1-1@m"> <div class="uk-text-lead uk-width-xxlarge uk-margin-auto uk-text-center" uk-scrollspy-class>Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur.</div> <div class="uk-margin-medium uk-text-center" uk-scrollspy-class> <a class="el-content uk-button uk-button-default uk-button-large" href="#">Unterkunft</a> </div> </div> </div> </div> </div> How can I achieve when I have an image field to add the custom markup within the image field? I have no clue to get this done.
      Thank you!
    • By Mobiletrooper
      Hey Ryan, hey friends,
      we, Mobile Trooper a digital agency based in Germany, use ProcessWire for an Enterprise-grade Intranet publishing portal which is under heavy development for over 3 years now. Over the years not only the user base grew but also the platform in general. We introduced lots and lots of features thanks to ProcessWire's absurd flexibility. We came along many CMS (or CMFs for that matter) that don't even come close to ProcessWire. Closest we came across was Locomotive (Rails-based) and Pimcore (PHP based).
      So this is not your typical ProcessWire installation in terms of size.
      Currently we count:
      140 Templates (Some have 1 page, some have >6000 pages)
      313 Fields
      ~ 15k Users (For an intranet portal? That's heavy.)
      ~ 195 431 Pages (At least that's the current AUTOINCREMENT)
       
      I think we came to a point where ProcessWire isn't as scalable anymore as it used to be. Our latest research measured over 20 seconds of load time (the time PHP spent scambling the HTML together). That's unacceptable unfortunately. We've implemented common performance strategies like:
      We're running on fat machines (DB server has 32 gigs RAM, Prod Web server has 32gigs as well. Both are running on quadcores (xeons) hosted by Azure.
      We have load balancing in place, but still, a single server needs up to 20 sec to respond to a single request averaging at around about 12 sec.
      In our research we came across pages that sent over 1000 SQL queries with lots of JOINs. This is obviously needed because of PWs architecture (a field a table) but does this slow mySQL down much? For the start page we need to get somewhere around 60-80 pages, each page needs to be queried for ~12 fields to be displayed correctly, is this too much? There are many different fields involved like multiple Page-fields which hold tags, categories etc.
      We installed Profiler Pro but it does not seem to show us the real bottleneck, it just says that everything is kinda slow and sums up to the grand total we mentioned above.
      ProCache does not help us because every user is seeing something different, so we can cache some fragments but they usually measure at around 10ms. We can't spend time optimising if we can't expect an affordable benefit. Therefore we opted against ProCache and used our own module which generates these cache fragments lazily. 
      That speeds up the whole page rendering to ~7 sec, this is acceptable compared to 20sec but still ridiculously long.
      Our page consists of mainly dynamic parts changing every 2-5 minutes. It's different across multiple users based on their location, language and other preferences.
      We also have about 120 people working on the processwire backend the whole day concurrently.
       
      What do you guys think?
      Here are my questions, hopefully we can collect these in a wiki or something because I'm sure more and more people will hit that break sooner than they hoped they would:
       
      - Should we opt for optimising the database? Since >2k per request is a lot even for a mysql server, webserver cpu is basically idling at that time.
      - Do you think at this point it makes sense to use ProcessWire as a simple REST API?
      - In your experience, what fieldtypes are expensive? Page? RepeaterMatrix?
      - Ryan, what do you consider as the primary bottleneck of processwire?
      - Is the amount of fields too much? Would it be better if we would try to reuse fields as much as possible?
      - Is there an option to hook onto ProcessWires SQL builder? So we can write custom SQL for some selectors?
       
      Thanks and lots of wishes,
      Pascal from Mobile Trooper
       
       
    • By mr-fan
      What i wanna achive is a simple counter like that count up on visit (this is no problem) AND save the specific date (year/month/day) of the count...
      in the end i will be able to get visits per day/per month/per year in a nice and dirty graph.
      Just to have a way better simple counter system.
      Should i only go with a complex setup of pages like this:
      --stats (home template for pageviews)
      ----2018 (year)
      ------08 (month)
      ---------29 ->page_views   (integers on every day template)
      ---------30 ->page_views
      Or just simple use:
      --stats (home template for pageviews)
      ---->count (template) that holds simple field page_views and a date field
      or could a fieldtype like tables (one table field for every month/year or so) be also a solution?
      Or a own SQL table special for this and use it in a module? I don't have any experience on this topic...
      What i have in mind of performance sideeffects on such a thing?
      Or is there a solution that works with PW?
      I wanna go the hard way and implement something like this:
      http://stats.simplepublisher.com/
      only directly within PW and use the API to get the data...maybe create a simple module from it later i don't know if i  could set it up right from the start 😉
      this is the reason for my questions on more experienced devs
      Kind regards mr-fan
       
    • By FrancisChung
      Long but well written, detailed and informative article written by an Engineering Manager for Google Chrome about the true cost of Javascript and what you can do to alleviate some of that cost.
      Must read!

      https://medium.com/@addyosmani/the-cost-of-javascript-in-2018-7d8950fbb5d4
    • By LMD
      I'm wondering if it is possible to add a label to the description input in image fields (in the admin).
      I'm using the module Image Extra, which has labels for each input, but I'd like to add a label to the default 'description' input too. The image below illustrates this:

       
      If there is no way (no hook?) then, I suppose I could just not use the default description and add a new description input with the Image Extra module.  But I thought I'd ask in case I (or others) ever want to do this without using the module (i.e. just the one input required).
      I'm using PW 3.0.98
      Thanks.
      ---
      FYI: yes, that is a cat and not a quilt -- this is on my local dev server and I don't have the actual photos yet! She is on a quilt, so it counts... technically.
       
×
×
  • Create New...