Slow Uploads When Using S3 Upload Checkbox.

I'm seeing more or less the same thing others reported at various times with what seem to be slow upload when processing to s3 during upload. Speeds 3-5 seconds per file when using the s3 upload option checkmark at same time I upload and process to the site (using Sytist upload process). I did not realize that checking s3 might take all day to upload 19,404 Files, 99 Folders with a total of 12.3 GB (615.7 KB average size according to S3).  Even being slow I think on a fast server and with fast connections, it should move along a bit faster.

I read posts of people hitting server limits when uploading so I checked the server loads and everything is under or at 1% for cpu load, memory load, etc on the account so the server seems to be pretty much idling through the process. I get not wanting to overload servers most people would select or use with smaller amounts to process, but maybe there could be some optional tweaks for faster less loaded servers?

Is it faster to upload to Sytist then process later to s3 or is it going to take as long to clear files over to s3? I have limited space to hold files on the server and that is the point of using s3 for storage.

Edited by

I don't know if the info below helps anything or not.

Instead of stopping the process partway through when I could see that it was going to take a really long time, I let the process run until finished to see if it could be successful. It completed uploading the 19,404 files. The total process took about 24 hours and 30 minutes (start at aprox. 11:30AM Friday (yesterday) and finished at 12:06PM Saturday (today) and the process had a random 14 files that failed to upload. I retried each failed file and upload was successful on retry. The Sytist upload method (uses a http method?) for handling large uploads IMO is just too slow if doing large amounts of files at once.  

Maybe there should be a mention that this method may take a really long time if used with a large amount of files to upload in the documentation?

I checked the web server for errors
No web server errors exist for your domain:
No suEXEC events exist for your domain:
There were no Sytist errors in the error log

Since the process ran so long and went through the peak afternoon after school internet usage window as well as my local networks overnight peak server backups period, having issues with just 14 files over that 24 hour period was actually not bad. Other than being slow it was successful.

Edited by

Uploading via ftp is an option in sytist.

Edited by

Yes it is and that is the method I will likely recommend she use if over a thousand or so files.

Edited by

It is going to take longer when moving to S3 when upload.

The file has to upload to the server, process the image (create new sizes, watermark, etc...) then it has to move / upload all those again to S3. 

You can upload without moving S3 then move them later. They will be available sooner.

Edited by