[Resolved] very many pictures to organize
we take the case is to manage a huge number of images.
Normally upload of all the pictures is the only directory (uploads/wppa)
Is it possible to organize them in subfolders?
(uploads/wppa/gallery_1; uploads/wppa/gallery_2 etc.)
THX for answer!
Is it possible to organize them in subfolders?
How many is a huge number? over 250.000 ?
@patsch27: I am going to use the plugin with about 100,000 pictures and was asking myself the same thing. So I asked the same question already a few days ago: http://wordpress.org/support/topic/images-in-subfolders?replies=4
At the moment I consider two ways to deal with it.
a) Write myself a script that runs through the gallery and creates a folder structure from the album names and album structure with hardlinks. I would run the script on a regular basis or every time after I uploaded new images. With that, I would have a folder structure that still presents the album structure in case the database crushes or I move to another plugin eventually somewhen in the future. I do not plan to do so, but you never know…
b) Alter the plugin to make it able to work with subfolders. This would be more work, but would provide me with what I really wanted. If I do so, I will contribute the code of course, expecially for opajaap if he wants to use it in any way. But as there are other things to do at the moment, this will take a while.
It’s not yet decided, maybe I will find another solution I didn’t think of until now.
I feared all the pictures in a directory. if something is messed up, I can not suss more. Now the pictures are beautifully structured in directories.
If better for Wp, SQL, and WPPA+ and for data security I understand that all images in one directory.
sorry for my bad english
Currently there is a test site running with 30.000 pics.
It runs still on a shared hosting server. The only problem is currently that searching takes too long to complete within the 60 seconds of the php configs max_execution_time.
On monday they are going to a dedicated server. I have direct contact with the webmaster. The target is to go to approx 250.000 pics.
Performance improvements was my specialism in my professional IT years, and its still a challenge for me to shift the limits as far as possible.
I am closely monitoring what is happening on that site, and i can promise you that IF having all files in one dir is a show-stopper, i will provide a solution.
Please do not attempt to change the code to use multiple dirs, i am sure you are not aware of the impact.
A few ( only a few!! ) things to keep in mind:
- Dir names = album names will fail because of (for filenames) invalid characters in albumnames
- Mess-up due to server time-out while moving photos to an other album
- A proper OS can handle many entries in a dir with less overhead than your script
- You can Export albums into a zipfile for backup purposes including metadata of albums and photos
- Why would 250.000 be a problem for a filesystem and not for a sql database table?
As stated above: we are seeking the systems limits. I will keep you informed of the findings.
As I said before, i don’t need the script to create directories by the name of the album. I will create them myself and import them in the backend. It also never needs to move pictures around in the file system. The only thing I need is the script to accept a “path” as additional field in the picture table.
Dir names = album names will fail because of (for filenames) invalid characters in albumnames
I don’t think this will be a problem, as my dir names only inlcude lower case letters, numbers and underlines_instead_or_spaces.
Mess-up due to server time-out while moving photos to an other album
This confuses me. As far as I know, moving pictures would not change anything to the current state, or would it? Because moving the pictures only changes the album ID, or am I wrong? This shouldn’t be too dangerous to do.
You can Export albums into a zipfile for backup purposes including metadata of albums and photos
This is actually a neat feature, but it needs to be done automatically. There will be about 1000 albums… Also I guess this will produce a lot of server load, which a script that creates and updates the album structure from the database with hardlinks presumably will not.
Why would 250.000 be a problem for a filesystem and not for a sql database table?
You are right. After some googling I am not asured that this huge number actually won’t be a performance problem for the filesystem. AFAIK ext4 limits to 64,000 subfolders but has no limit to files in a folder. You are not able to let apache show you the content of the dir, but as the script points to files directly and does not do a
lscommand or whatever, this will work. It’s no question of performance, it’s only a question of a neat dir and file structure on the server.
so I will surely not change the code.
Only style in CSS. PHP > Hands off
for I know too little about programming.
going to change my style and is of html code to wordpress big change.
the only one might add as an idea on the todo list.
sorry for my bad english
It’s no question of performance, it’s only a question of a neat dir and file structure on the server.
I feel it as a responsability to the users of the plugin to keep the system consistent, to make it as less vulnerable to internal errors as possible.
The content of …/uploads/wppa/ must be seen as a part of the wppa system as the 6 sql db tables are. If it were doable, i had even rather had the actual image data inside the WPPA_PHOTOS db table.
I said it before, but i can only support this sytem if everybody keeps his hand off from the db tables as well as from the content of the wppa/ subdir.
As it is no matter of performance, and, as it looks now also not a matter of limitation in the maximum number of photos in the system, i will not change the structure of the files within wppa/.
But there are more reasons why one should not want to use the wppa/ subdir as an archive or treasory of the photos in the system.
The photo files in …/uploads/wppa/ are – on the majority of installations – resized (i.e. downsized) by php routines and probably ‘damaged’ with a watermark. The resizing by php has the side effect of loosing all exif and iptc data, and of course a poorer resolution.
For backup purposes there is the export functionality, what i could change to make a zip per album instead of per session as it is now, but that is also not intented – and not really suitable – to be your treasory of photos.
I recognize the need to have a neat dir structure ( on an album basis ) for many reasons like backup, the possibility to migrate to a different system, and for updating the wppa system with e.g. larger resized images or remove/alter watermarks or the ability for visitors to download the un-downsized versions (if you allow them to).
There is already the update feature in Import, but it requires a new upload (if the files were originally deleted after successfol import ur uploaded directly). So, apart from the idea to backup your original photo files ‘in the cloud’ in a overseeable ‘neat’ structure, also from the point of view from wppa, it would be handy to have the original source image files on the server.
I therefor have the following idea, what i would be glad to implement:
The creation and maintenance of an optionally source directory, say …/uploads/wppasrc/ with album subdirs like …/uploads/wppasrc/album-13/ that will get the original uploaded or imported files as they are.
The photo table entry will have an extra unchangeable field: filename, that will be used for updating the wppa image.
The user will be responsable for the uniqueness of the names, the wppa system will not rely on completelyness or correctlyness of this ‘treasury’ directory, it is just a helpfull resource.
I think we have the best of many worlds this way, disk space should not be a problem with prices less than €100 per terabyte nowadays.
How do you think about this idea?
Sorry for the delayed answer, I was kind of busy the last days. Thanks for your suggestion, it’s not that bad actually.
I like the idea of having a separate set of pictures, so the published images are just a “derived” version in smaller size, with watermark and so on. One question though: Will this be the same for user-uploaded pictures? Or are their pictures still going directly to uploads/wppa without the external source file?
The proposed change would be a good solution for people that run their website on vservers and have plenty of space. Still, disk space is not that cheap when it comes to shared hosting solutions. If your users upload fullsize images that are scaled for the online output but still hosted in the uploads/wppasrc folder, your webspace might get full quite fast. That is a reason why the separate hosting of two versions of every picture might not be a good idea.
Also, I am back to the question of performance. I am now convinced that your filesystem has no problem with roughly 100,000 pictures in one folder if you run your own server, because the filesystem has a lot of caching capabilities that are able to cope with it. But, if you are on shared-hosting, caching of the filesystem is not exclusively made for your visitors but for the many visitors of the server you are hosted on.
I guess it’s a decision what kind of users are the target group for this plugin. If your pictures aren’t that big, you can host a 100,000 pictures on a 10GB shared-hosting account. If so, you can not upload all images twice and your server may get performance problems if all images are in one folder. Earlier you said, that searching on that site with 30,000 pictures on a shared-hosting account failed. If those sites should be able to work with this plugin I guess, the solution you suggested does not solve all problems here.
The source files system is fully optional, i.e. missing some image files or missing some album subdirectories will never produce an error. So, it is ok to ‘manually’ remove files and/or albums from the source directory.
There will also be an option to add photos (in the regular wppa system) from the source pool if you added them there manually. Kind of import from the source. I will add a switch to seperately saving the front end uploads, this is a good suggestion.
I am currently trying to find a way to use diskspace elsewhere for it (like STRATO’s hidrive where i have 100 GB storage space for only approx €10 a month) because i need this myself as a ‘backup in the cloud’.
OK Here is what I have:
Purchased a Good Theme for Photos
Hosted on GoDaddy
Started Uploading Images… my theme makes a lot of thumbs and various images (different sizes of each original)
For each Original Image I upload my theme creates about 4 additional images.
My upload/2013/06 folder blew up to 3500 images and stopped accepting uploads… delete a few and every things back to normal.
I call GoDaddy, they claim that Linux should never have a single folder with more then 1200 files.
OK so I look all over the internet for a method that would allow subfolders in my uploads that not only will work but is easy enough to implement.
I want an upload media folder say ‘Disney’ with 300+ images, then another media folder say ‘Universal’ with 300+ images.
I have ftp access and have the file manager and wordpress… I can make just about anything happen … its seems silly to have such limits placed on a folder and not be able to create separate folders quickly in wordpress media uploads.
my site: rodrigosobral.info/blog
Looking at a Product Called:
This question does not apply to wppa+
In wppa+ all photos will be in one directory, and all thumbs also in one (sub) directory.
The largest website ( still under construction ) so far has 107368 photos in 460 albums.
Its php config says:
Linux admin26738735.localhost 2.6.32-358.2.1.el6.x86_64 #1 SMP Wed Mar 13 00:26:49 UTC 2013 x86_64
- The topic ‘[Resolved] very many pictures to organize’ is closed to new replies.