Filter folder structure, download, file-import in PW

[hellerdruck]

Hi all

I need help with something. Situation: We have let's say 2'000 Files (Excel) that should be displayed (list with links) on a page. We'd need to filter these files by given Keywords or a tree structure or both. Now, I'm looking for a solution whereas our customer can synchronise the files from his local computer with the folder on the webserver. They will update and upload files on a daily basis. Therefore, it would need to synchronise rather than load the files manually in pages or repeaters. Maybe indexing would be an idea, too.

Are there any modules for Processwire that would help achieving this? Could anyone point me in the right direction?

Thanks in advance.

[kongondo]

10 hours ago, hellerdruck said:

that should be displayed (list with links) on a page

1. Displayed in the backend?
2. Links to whats? 
3. Are the files pages? or stored as files in file fields in multiple or a single page?

10 hours ago, hellerdruck said:

with the folder on the webserver

Hmm. So, seems like 'loose' files stored on the system? Not ProcessWire managing the files?

10 hours ago, hellerdruck said:

Are there any modules for Processwire that would help achieving this?

I don't think so. If the files were pages, then maybe Lister or (Lister Pro). There could be others I have forgotten about.

Depending on your responses to the above questions, there could be other approaches/solutions.

[hellerdruck]

Hey @kongondo, thanks for replying.

1. no, displayed on a front-end page
2. Links to the files, the files should be downloadable by the visiting user
3. not sure how to store them best. it's a load of files, and it would be great, if they shouldn't be uploaded one by one. More like batch processing or a folder with the files on the server. I guess it would be great if PW could manage the files, but the customer does not want to go to the backend to upload the new versions of the files individually. rather some sort of folder synchronisation. if possible.

does that make sense?

[kongondo]

5 minutes ago, hellerdruck said:

does that make sense?

It does. So, there's clearly two main groups of issues. One, the frontend bit and two the backend.

Frontend
 

12 hours ago, hellerdruck said:

We'd need to filter these files by given Keywords or a tree structure or both.

This is for frontend as well? If yes, rather than re-invent the wheel, you could use on of the plethora of 'data tables' out there. 

Backend

I am assuming this is where you want the client to do the uploading (albeit not individually) or...

9 minutes ago, hellerdruck said:

but the customer does not want to go to the backend to upload the new versions of the files individually. rather some sort of folder synchronisation.

do you mean you want to give them some sort of FTP access? I hope not.

Bottom line is the client will need to upload the files, somewhere so best if they login, get authenticated and upload the files. The easiest thing is for them to upload a zip file. Obviously, size of file matters as well as for synchronisation, you need identical file names. You may also need to store backups of older files for a while just in case client uploaded the wrong files and everything got overwritten (basically some sort of versioning system). 

For the upload area, you could go for a custom Process module that allows ajax upload (works nicely with batch file uploads if you need that sort of thing). The batch Processing would be all PHP. Somewhere in the forums there's a module that can run tasks at set times, but not sure if that is what you want. The Process module could either have pages, for uploading and for viewing current files and maybe for doing backups (that only you can access). Or, you can have one page split into n tabs.

Pages or Not

This depends on your needs, especially the frontend ones. For instance, if you want human readable titles to be displayed in the frontend together with the files, then pages would server you well. This is because ProcessWire would sanitize the file names and you end up with lots of underscores or hyphnes in file names. On the other hand, if you manage the files yourself, you will have to write the code (but $files() can help), to track files. If you are worried about the visibility of 2000+ pages with files in the backend, you can 'hide' these under /admin/some-parent/.

Just thinking out loud here.

22 minutes ago, hellerdruck said:

sort of folder synchronisation

There are tools for that, but not ProcessWire, although you could achieve some sort of synchronisation yourself using ProcessWire based on a few simple rules.

[hellerdruck]

32 minutes ago, kongondo said:

do you mean you want to give them some sort of FTP access? I hope not.

Actually, I thought this could be an option. Include a folder structure somewhere in "templates" and have them synchronise with something like Cyberduck.

34 minutes ago, kongondo said:

This is for frontend as well? If yes, rather than re-invent the wheel, you could use on of the plethora of 'data tables' out there. 

yes. Their customers need to access this page front-end and filter by type, classes, keywords, whatever. Jquery Data Table looks good at first glance. I'll definitely check that out.

1 hour ago, kongondo said:

For the upload area, you could go for a custom Process module that allows ajax upload (works nicely with batch file uploads if you need that sort of thing). The batch Processing would be all PHP. Somewhere in the forums there's a module that can run tasks at set times, but not sure if that is what you want. The Process module could either have pages, for uploading and for viewing current files and maybe for doing backups (that only you can access). Or, you can have one page split into n tabs.

Could the Media Manager work here for the batch processing? Something else that would make things even more complicated: is it possible to index the filex by keywords in a batch upload?

 

1 hour ago, kongondo said:

There are tools for that, but not ProcessWire, although you could achieve some sort of synchronisation yourself using ProcessWire based on a few simple rules.

Could you specify this a little?

Sorry for all the questions..

[kongondo]

10 hours ago, hellerdruck said:

Actually, I thought this could be an option. Include a folder structure somewhere in "templates" and have them synchronise with something like Cyberduck.

OK. Won't they get confused by the interface though?

10 hours ago, hellerdruck said:

Jquery Data Table looks good at first glance. I'll definitely check that out.

Also check out Tabulator

10 hours ago, hellerdruck said:

Could the Media Manager work here for the batch processing?

Yes, in a sense, though it depends on whether you want automatic  (i.e. no user interaction, like an auto module or cron thing) versus manual (press a button and it gets synchronized.  The latter can be done, easily in Media Manager. The former would need a bit of coding, essentially calling Media Manager API via some automated process. 

Media Manager has a 'Scan' uploads mode. How it works is you upload files to a set folder in /site/MediaManager. You can upload files to that folder via FTP, or other method. Inside Media Manager. The files can be loose or zipped. If zipped, Media Manager would unzip them. The files are then presented in a list. You can then select the ones you wish to upload to the Media Manager library. Those added to the library are then deleted from the folder where they were uploaded. There are settings for what to do when Media Manager encounters duplicates: rename/replace/skip. In your case, you would go for replace (synchronization). 

10 hours ago, hellerdruck said:

is it possible to index the filex by keywords in a batch upload?

In Media Manager? Yes, but post upload (not during). Each media is a page. Say you uploaded a file called My End of Year Report.xlsx. Media Manager would create a page with that file name (sanitized) as the title. That will end up as My End of Year Report. The file name itself would be sanitized as my-end-of-year-report.xlsx. Post-upload, you can manually add keywords, e.g. via tags, or a custom field or fields (e.g. options, page reference fields, etc). 

If you want automatic indexation, that would require some custom code. If this is something you want to pursue alongside a Media Manager option, we can discuss further via PM (as a custom paid option).

10 hours ago, hellerdruck said:

Could you specify this a little?

I meant tools like rsync ? 

[hellerdruck]

6 hours ago, kongondo said:

In Media Manager? Yes, but post upload (not during). Each media is a page

Is there a demo version i can try? would love to play around and find out if it works for me..

[kongondo]

17 minutes ago, hellerdruck said:

Is there a demo version i can try?

There isn't, I am afraid. Now that I think about it, maybe I should set up an online demo. Otherwise, you can purchase a copy (fully refundable within 7-days' of purchase date) to test.

[hellerdruck]

25 minutes ago, kongondo said:

maybe I should set up an online demo

you should ?

25 minutes ago, kongondo said:

you can purchase a copy (fully refundable within 7-days' of purchase date) to test.

will do that right away, thanks!

[kongondo]

17 minutes ago, hellerdruck said:

will do that right away, thanks!

Please let me know how it goes. Post questions related to MM in its support forum here:

 

[louisstephens]

You could also look into implementing Dropzone.js on the frontend and pass the files via ajax to a function that would loop through the files and create pages for them. It has been awhile since I last did it (over a year) and I would need to hunt down the code, but it wasn't too much of a pain to get sorted.

However, Media Manager is a solid option in the backend.