Jump to content
hellerdruck

Filter folder structure, download, file-import in PW

Recommended Posts

Hi all

I need help with something. Situation: We have let's say 2'000 Files (Excel) that should be displayed (list with links) on a page. We'd need to filter these files by given Keywords or a tree structure or both. Now, I'm looking for a solution whereas our customer can synchronise the files from his local computer with the folder on the webserver. They will update and upload files on a daily basis. Therefore, it would need to synchronise rather than load the files manually in pages or repeaters. Maybe indexing would be an idea, too.

Are there any modules for Processwire that would help achieving this? Could anyone point me in the right direction?

Thanks in advance.

Share this post


Link to post
Share on other sites
10 hours ago, hellerdruck said:

that should be displayed (list with links) on a page

  1. Displayed in the backend?
  2. Links to whats? 
  3. Are the files pages? or stored as files in file fields in multiple or a single page?
10 hours ago, hellerdruck said:

with the folder on the webserver

Hmm. So, seems like 'loose' files stored on the system? Not ProcessWire managing the files?

10 hours ago, hellerdruck said:

Are there any modules for Processwire that would help achieving this?

I don't think so. If the files were pages, then maybe Lister or (Lister Pro). There could be others I have forgotten about.

Depending on your responses to the above questions, there could be other approaches/solutions.

Share this post


Link to post
Share on other sites

Hey @kongondo, thanks for replying.

1. no, displayed on a front-end page
2. Links to the files, the files should be downloadable by the visiting user
3. not sure how to store them best. it's a load of files, and it would be great, if they shouldn't be uploaded one by one. More like batch processing or a folder with the files on the server. I guess it would be great if PW could manage the files, but the customer does not want to go to the backend to upload the new versions of the files individually. rather some sort of folder synchronisation. if possible.

does that make sense?

Share this post


Link to post
Share on other sites
5 minutes ago, hellerdruck said:

does that make sense?

It does. So, there's clearly two main groups of issues. One, the frontend bit and two the backend.

Frontend
 

12 hours ago, hellerdruck said:

We'd need to filter these files by given Keywords or a tree structure or both.

This is for frontend as well? If yes, rather than re-invent the wheel, you could use on of the plethora of 'data tables' out there. 

Backend

I am assuming this is where you want the client to do the uploading (albeit not individually) or...

9 minutes ago, hellerdruck said:

but the customer does not want to go to the backend to upload the new versions of the files individually. rather some sort of folder synchronisation.

do you mean you want to give them some sort of FTP access? I hope not.

Bottom line is the client will need to upload the files, somewhere so best if they login, get authenticated and upload the files. The easiest thing is for them to upload a zip file. Obviously, size of file matters as well as for synchronisation, you need identical file names. You may also need to store backups of older files for a while just in case client uploaded the wrong files and everything got overwritten (basically some sort of versioning system). 

For the upload area, you could go for a custom Process module that allows ajax upload (works nicely with batch file uploads if you need that sort of thing). The batch Processing would be all PHP. Somewhere in the forums there's a module that can run tasks at set times, but not sure if that is what you want. The Process module could either have pages, for uploading and for viewing current files and maybe for doing backups (that only you can access). Or, you can have one page split into n tabs.

Pages or Not

This depends on your needs, especially the frontend ones. For instance, if you want human readable titles to be displayed in the frontend together with the files, then pages would server you well. This is because ProcessWire would sanitize the file names and you end up with lots of underscores or hyphnes in file names. On the other hand, if you manage the files yourself, you will have to write the code (but $files() can help), to track files. If you are worried about the visibility of 2000+ pages with files in the backend, you can 'hide' these under /admin/some-parent/.

Just thinking out loud here.

22 minutes ago, hellerdruck said:

sort of folder synchronisation

There are tools for that, but not ProcessWire, although you could achieve some sort of synchronisation yourself using ProcessWire based on a few simple rules.

Share this post


Link to post
Share on other sites
32 minutes ago, kongondo said:

do you mean you want to give them some sort of FTP access? I hope not.

Actually, I thought this could be an option. Include a folder structure somewhere in "templates" and have them synchronise with something like Cyberduck.

34 minutes ago, kongondo said:

This is for frontend as well? If yes, rather than re-invent the wheel, you could use on of the plethora of 'data tables' out there. 

yes. Their customers need to access this page front-end and filter by type, classes, keywords, whatever. Jquery Data Table looks good at first glance. I'll definitely check that out.

1 hour ago, kongondo said:

For the upload area, you could go for a custom Process module that allows ajax upload (works nicely with batch file uploads if you need that sort of thing). The batch Processing would be all PHP. Somewhere in the forums there's a module that can run tasks at set times, but not sure if that is what you want. The Process module could either have pages, for uploading and for viewing current files and maybe for doing backups (that only you can access). Or, you can have one page split into n tabs.

Could the Media Manager work here for the batch processing? Something else that would make things even more complicated: is it possible to index the filex by keywords in a batch upload?

 

1 hour ago, kongondo said:

There are tools for that, but not ProcessWire, although you could achieve some sort of synchronisation yourself using ProcessWire based on a few simple rules.

Could you specify this a little?

Sorry for all the questions..

Share this post


Link to post
Share on other sites
10 hours ago, hellerdruck said:

Actually, I thought this could be an option. Include a folder structure somewhere in "templates" and have them synchronise with something like Cyberduck.

OK. Won't they get confused by the interface though?

10 hours ago, hellerdruck said:

Jquery Data Table looks good at first glance. I'll definitely check that out.

Also check out Tabulator

10 hours ago, hellerdruck said:

Could the Media Manager work here for the batch processing?

Yes, in a sense, though it depends on whether you want automatic  (i.e. no user interaction, like an auto module or cron thing) versus manual (press a button and it gets synchronized.  The latter can be done, easily in Media Manager. The former would need a bit of coding, essentially calling Media Manager API via some automated process. 

Media Manager has a 'Scan' uploads mode. How it works is you upload files to a set folder in /site/MediaManager. You can upload files to that folder via FTP, or other method. Inside Media Manager. The files can be loose or zipped. If zipped, Media Manager would unzip them. The files are then presented in a list. You can then select the ones you wish to upload to the Media Manager library. Those added to the library are then deleted from the folder where they were uploaded. There are settings for what to do when Media Manager encounters duplicates: rename/replace/skip. In your case, you would go for replace (synchronization). 

10 hours ago, hellerdruck said:

is it possible to index the filex by keywords in a batch upload?

In Media Manager? Yes, but post upload (not during). Each media is a page. Say you uploaded a file called My End of Year Report.xlsx. Media Manager would create a page with that file name (sanitized) as the title. That will end up as My End of Year Report. The file name itself would be sanitized as my-end-of-year-report.xlsx. Post-upload, you can manually add keywords, e.g. via tags, or a custom field or fields (e.g. options, page reference fields, etc). 

If you want automatic indexation, that would require some custom code. If this is something you want to pursue alongside a Media Manager option, we can discuss further via PM (as a custom paid option).

10 hours ago, hellerdruck said:

Could you specify this a little?

I meant tools like rsync 😄 

  • Like 1

Share this post


Link to post
Share on other sites
6 hours ago, kongondo said:

In Media Manager? Yes, but post upload (not during). Each media is a page

Is there a demo version i can try? would love to play around and find out if it works for me..

Share this post


Link to post
Share on other sites
17 minutes ago, hellerdruck said:

Is there a demo version i can try?

There isn't, I am afraid. Now that I think about it, maybe I should set up an online demo. Otherwise, you can purchase a copy (fully refundable within 7-days' of purchase date) to test.

Share this post


Link to post
Share on other sites
25 minutes ago, kongondo said:

maybe I should set up an online demo

you should 😀

25 minutes ago, kongondo said:

you can purchase a copy (fully refundable within 7-days' of purchase date) to test.

will do that right away, thanks!

  • Like 1

Share this post


Link to post
Share on other sites
17 minutes ago, hellerdruck said:

will do that right away, thanks!

Please let me know how it goes. Post questions related to MM in its support forum here:

 

Share this post


Link to post
Share on other sites

You could also look into implementing Dropzone.js on the frontend and pass the files via ajax to a function that would loop through the files and create pages for them. It has been awhile since I last did it (over a year) and I would need to hunt down the code, but it wasn't too much of a pain to get sorted.

However, Media Manager is a solid option in the backend.

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Rodd
      Hi everyone!
      I have a website in a production environment and I want to duplicate it in a local environment. I exported the content of the website (with the 'Site Profile Exporter' module) but I cannot use it actually. I've got an issue with the database. I imported this one in MAMP then.

      I also exported the pages (with the 'ProcessPagesExportImport' module), but I cannot import it to my local website because the fields don't exist. So I created this fields, but I have this error :
      How can I use the elements that already exist and are presents in my database? How can I duplicate correctly the templates, fields and pages?
      Thanks by advance
      PS: Sorry if my english is bad
       
    • By iNoize
      Hello, need some help for an RealEstate project. It have to use the OnOffice to import the objects. 
      https://apidoc.onoffice.de/
       
    • By gebeer
      Hello all,
      wasn't sure where to put this, so it goes in General section.
      Ryan shows a hook that we can use to mirror files on demand from live server to development environment to be up to date with the files on the server without having to download complete site/assets/files folder.
      I just implemented this but had problems getting files to load from a site in development that is secured with user/password via htaccess.
      First I tried to use WireHttp setHeader method for basic authentication like this
      function mirrorFilesfromLiveServer(HookEvent $event) { $config = $event->wire('config'); $file = $event->return; if ($event->method == 'url') { // convert url to disk path $file = $config->paths->root . substr($file, strlen($config->urls->root)); } if (!file_exists($file)) { // download file from source if it doesn't exist here $src = 'http://mydomain.com/site/assets/files/'; $url = str_replace($config->paths->files, $src, $file); $http = new WireHttp(); // basic authentication $u = 'myuser'; $pw = 'mypassword'; $http->setHeader('Authorization: Basic', base64_encode("$u:$pw")); $http->download($url, $file); } } But, unfortunately this didn't work.
      So now I am using curl to do the download. My hook function now looks like this
      function mirrorFilesfromLiveServer(HookEvent $event) { $config = $event->wire('config'); $file = $event->return; if ($event->method == 'url') { // convert url to disk path $file = $config->paths->root . substr($file, strlen($config->urls->root)); } if (!file_exists($file)) { // download file from source if it doesn't exist here $src = 'http://mydomain.com/site/assets/files/'; $fp = fopen($file, 'w+'); // init file pointer $url = str_replace($config->paths->files, $src, $file); $u = 'myuser'; $pw = 'mypassword'; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_TIMEOUT, 50); // crazy high timeout just in case there are very large files curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_USERPWD, "$u:$pw"); // authentication curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC); // authentication curl_setopt($ch, CURLOPT_FILE, $fp); // give curl the file pointer so that it can write to it curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); $data = curl_exec($ch); curl_close($ch); } } Now I can load files and images from the htaccess protected development server 🙂
      If anyone knows how to get this to work with WireHttp, please let me know. Thank you.
    • By Robin S
      If you've ever needed to insert links to a large number of files within CKEditor you may have found that the standard PW link modal is a somewhat slow way to do it.
      This module provides a quicker way to insert links to files on the page being edited. You can insert a link to an individual file, or insert an unordered list of links to all files on the page with a single click.
      CKEditor Link Files
      Adds a menu to CKEditor to allow the quick insertion of links to files on the page being edited.

      Features
      Hover a menu item to see the "Description" of the corresponding file (if present). Click a menu item to insert a link to the corresponding file at the current cursor position. The filename is used as the link text. If you Alt-click a menu item the file description is used as the link text (with fallback to filename if no description entered). If text is currently selected in the editor then the selected text is used as the link text. Click "* Insert links to all files *" to insert an unordered list of links to all files on the page. Also works with the Alt-click option. Menu is built via AJAX so newly uploaded files are included in the menu without the page needing to be saved. However, descriptions are not available for newly uploaded files until the page is saved. There is an option in the module config to include files from Repeater fields in the edited page. Nested Repeater fields (files inside a Repeater inside another Repeater) are not supported. Installation
      Install the CKEditor Link Files module.
      For any CKEditor field where you want the "Insert link to file" dropdown menu to appear in the CKEditor toolbar, visit the field settings and add "LinkFilesMenu" to the "CKEditor Toolbar" settings field.
       
      http://modules.processwire.com/modules/cke-link-files/
      https://github.com/Toutouwai/CkeLinkFiles
    • By neophron
      Hi,
      I was searching in the forum about a method how to remove a file extension, like .jpg, but couldn't find anything.
      There are some php methods out in the wild, but I wanted first to get sure, that Processwire is offering a solution.
      The background is, that a clients website has a lot of images, packed in different galleries. All image files contain the name, dimensions and other stuff about the image (painting). I want to echo the image name in the image tag.
       
      thanks 
×
×
  • Create New...