Processing Photos When Using The Ftp Upload Feature

Hi,

I have been using the FTP upload option, because I found I got so many errors (91 out of 271 images in a gallery) when using the built in uploader. That solved that problem, but I see that it seems to process two photos and then has a 10 second rest. Is there a way to reduce that timeout? I have a dedicated server and was hoping that if it is just feature to manage the load on a shared server, then perhaps it would not be as important in my case.

also, is it possible for me to create all the photos that the system requires and upload them to where they need to be, as this would be dramatically faster. At the moment, I am uploading tiny galleries, like a few hundred images and it takes 20 to 30 minutes to process - in a few months I will be in a situation where I will have galleries that are 5 and 6 thousand images - and I will have 4 or 5 galleries a day to upload like this... styist could really do with some improvements in this area for event photography power users like myself :-)

Edited Sat May 15, 21 9:36 PM by Craig Stewart

Just my personal thought is it is your server speed and your internet provider upload speed and nothing really to do with sytist. If you are making different size photos to download all those need to be processed, and the largest size you are uploading can have a factor too. A couple 100 images in 20-30 minutes is very fast. There is a "Rest time" but not allowing enough time may result in errors.

Edited Sun May 16, 21 10:37 AM by Vance Birno

Hi Vance,

Upload speed is unlikely to be an issue as I am getting 150+ mbps and I have a dedicated server with 16GB of ram, and was doing this during a period with low activity on the site - but even so, I then uploaded via FTP and did the processing part separately through Sytist. I am not entirely sure how many different size photos it is producing when the processing happens so I will look into that - as I do not need many for sales purposes, at least.

I find 20-30 minutes to be _glacial_ for 220 or so images. On my older cart I create all the different sizes needed by the system myself (viewing, thumbnail and free download size) and upload them as a zip file that the system then unpacks, moves to the correct spot and writes the file paths to the Database. Once the images are sized (and for a job with 5000 images, this takes about 10 minutes) and uploaded (about 3 minutes for a 2-3GB zip files) the bit on the server takes about 2 minutes.

I am surprised that this part of Sytist is so slow, given how good the cart is in so many other respects. It is marketed to event photographers in part (school photos) so I would expect a more robust upload feature.

Bring back Dave's Uploader!!! :-)

Craig

Edited by

What internet provider and service are you using? I only get about 5mps upload, but download speed is 200 mps. Imagemagick is the program converting the files.

Edited Mon May 17, 21 6:38 AM by Vance Birno

The problem isn't with Sytist. It will be with the server.

It should be processing a lot more than 2 photos at a time processing FTP uploaded photos. And you should not have a lot of failures using the built in uploader. It works perfectly fine.

The issues will be with the server not having enough speed, CPU, ram, etc....

Sign up for a free test drive here and upload from the admin and you will see.

To answer your question, you can change the FTP process and rest time in Settings -> Photo Settings.

Edited by

Hi Tim,

I shot a hockey carnival (field hockey) this past weekend. The total images to upload is about 7500. I started doing this at 5pm yesterday. It is now 9:30 am the following day and it is only two thirds the way through - so 16.5 hours so far.

I left it overnight and discovered that I had an error page where the "Processing Photos. Do not Leave this window." page is. I hit the back button on the browser and then refresh the page and it resumes from where it stopped. This happen several times over night and actually just again while typing this sentence. SO I lost a few hours processing time due to that.

It was processing for 15 seconds with a 5 second rest, and that was doing 4 images per go. I have changed that a few minutes ago to 30 seconds with a 2 second rest and it now does 7 images per go.

*PLEASE* tell me there is better way to do this. I *LOVE* your cart - but as I am finding out, it is not set up to cater to event shooters with large volume. I must have a way to get the images up faster.

My previous cart, I had custom work done to allow me to upload a zip file containing all the image sizes that were needed in a hierarchical folder structure, with image already water marked, so the Original size, the Viewing Size, the thumbnail size and we also offered a free download version. I put the zip file in the correct place and then told the cart to unpack that and it and it made all the galleries much like Sytist does now. I am using S3, so it would be easy to put everything there. I see that the files have somewhat random prefixes which could be a problem. With my old cart, they had standard prefixes of gallery_ for the thumbnails and none of the others needed one because they were in different spots or only shared the folder with the gallery_ versions so there were no filename clashes.

I can not be the only user that would benefit from a speedier way to handle volume.

I understand that the server specs play a part in this, but I do have a dedicated Intel Xeon 12-Core 24-Thread server with 16GB of ram. But even so, I know this plays a part, which is why it makes even more sense to have a bulk "manual" processing option for customers like me. My desktop machine would race through the resizing processing and uploading to S3 would be easy. It would just need a mechanism to tell the cart where to look for them and write that info into the database.

I hope you will consider this.

Craig

Edited by

The speed all depends on 1) The server, 2) The size of the photos being processed and 3) the size of the watermarking.

There is no other way.

Edited by

Hi Tim,

If I could take that load off the server, though by watermarking them myself and creating the sizes that I need myself - this would dramatically reduce the time.

Craig

Edited by

the problem is not sytist, the hosting I use with 1 cpu and little ram from 150 € per year processes about 5000 photos every hour including the watermark. I resize them first with acdsee or faststone to the largest resolution that the site will use (in my case 900 pixels). with the technical specifications you say you have your own hosting you should do much sooner, the problem lies somewhere else.

Edited by

As part of my on-site viewing system I create 2 sized files (small and large) with watermark added plus all needed html for my local intranet web server.

I can process about 64 images per second bc of my workstation. I have a heavily built threadripper 2990x system. 16 cores, 32 threads and the software I created for the images and html generation can double the thread count to 64. 64gb of ram, all ssd (M. 2 and 2.5s and one 5tb spinner)

It is a beast.

I then upload those watermarked "thumbnails" to my website via sytist uploader with no issues. Sending the smaller images greatly increases the overall number of images you can send up.

I'll have 50k images over an extended weekend.

Just the way my system has evolved.

Edited by