To post a new support question, click the Post New Topic button below.
Current Version: 4.9.3 | Sytist Manual | Common  Issues | Feature Requests

Please log in or Create an account to post or reply to topics.
You will still receive notifications of replies to topics you are part of even if you do not subscribe to new topic emails.

Upload Vs S3 Vs Speed

B
59 posts
Sat Jun 04, 22 2:43 PM CST
Hi Tim,

Before my question, here is an overview of my situation. Today I have 35,000+ pictures to upload and Sunday I'll likely will have at least 75,000+ pictures to uploads. Because of capacity, files are moved to an S3 bucket upon upload.

This process is excruciatingly slow. It takes more or less 7 to 10 seconds per files. You can see that it adds up quite fast.

Now, my questions.

Do you think there could be a way to optimize this ? For example, loading the files in the bucket first and "simply" having Sytist update the location (while making the secondary files locally) ?

Regarding the secondary files (thumbnails, previews, etc), could these remain stored locally, avoiding the time & cost for transfer and access to the bucket ? These are rather small and space is less of an issue here.

Also, are ALL those secondary files required ? If we only need the original, the large and the small, we could save time by not creating the other ones.

Again, will a relatively small number of files, I have nothing to see. It works great. But with such events, people are expecting their picture within 36 to 48 hours and it simply cannot be achieved at this time.

I understand, this would require code changes and I am more than willing to help. But Is it at least feasible and/or code wise modifiable ?

I know I'm asking much. Sorry. Your system is great in many ways. It's my needs that are ... well ... giving headaches and no sleep ! hahaha !

I'm opened to any suggestions !

Warmest regards,
Benoit
M
296 posts
Mon Jun 06, 22 7:15 AM CST
Are you uploading in bulk at the end of the day? We do partial uploads throughout the day

What size images and how much are they compressed? Size matters

It may be better to upload to the site and do the transfer to S3 later - use watched folder and ftp

Mike
B
59 posts
Mon Jun 06, 22 7:38 AM CST
Thanks for the reply and suggestions Michael. But ...

We cannot upload during the day since the event is held within a 3 to 4 hours span and we are remote and outside. It wouldn't help much. ;)

Files need to be 3000 px on the longest side since the event is a Free download one and we simply do not have the time to handle each request one by one with so many participants. So I try compress the files as much as can be. And the Thumbnails, Small and Large are at the lowest acceptable size. I doubt I could get much smaller. Quality is at 75. I might try some tests to see the file size difference, but sincerely I doubt it"s size matter. Transfer of each file is actually fast. It's the hand shaking and protocol between the servers the is killing us.

That is what I am doing at this point. But even then I hit some walls. Capacity being one. I can upload a bunch but then I hit our max capacity. The upload is rather fast. That's not a problem at all. But then the Move to S3 is called. WOW ! That takes forever. And I have to wait for it to progress enough so I can upload more files. And doing too much in parallel then I hit the CPU limit on my hosting service ! hahaha !

That is why, I was brainstorming to see if ,by reducing the numbers of files that need to be moved to S3 (the mini, small, thumbnails for example), it should help a lot on the S3 phase. These files are so small that I'm confident most of them could live locally on the server without a problem. In other words, just moving the big files (Original and Large).

What a project !! haha !
T
370 posts
Mon Jun 06, 22 10:05 AM CST
If time is of the essence, which it appears to be, can't you store them local (avoiding the transfer to S3, which is what appears to be giving you grief) and give participants a deadline to download? Transfer to s3 after the rush. Have a word with your host to see what they can do for you regarding bandwidth, capacity, node limit, increasing speeds to an s3 bucket etc.

I have to say, I think your case is quite unique - uploading 100,000 images at 10inch at 300dpi where participants need them yesterday. Do they need such hi-res? Lo-res downloads free, hi-res purchasable and only uploaded after purchase (which is how I do it). Maybe temper the expectations of participants and warn them they will take a few days to upload.
Edited Mon Jun 06, 22 10:07 AM by Trailboy
B
59 posts
Mon Jun 06, 22 10:13 AM CST
Hi Trailboy,

Thanks for the feedback and I get your points. Historically, I've always provided access to the pictures within 36 to 48 hours. And all my competitors do as well. Sponsors pay big bucks for this service and expect a good visibility with downloads. But if it takes too long, downloads go down. It has to be while the event is still fresh and people still all excited.

As for storing locally, that was my hope. But it turns out my hosting company cannot sell me more space unless I go with something that is about 10x the price. I would be loosing money over a year (yes I need to keep the files up for at least 12 months).

My case is unique and not at the same time. It's the same kind of situations for any big events like big Marathons, Iron Man, etc.. We end up providing pictures for thousands and thousands of participants. So, no, I'm not that unique. Not many but not unique ;)

I am investigating moving or using another hosting service to move the second part of this project. One that would give me much more disk space. Do you have any suggestions ?

Kind regards
461 posts
Mon Jun 06, 22 11:24 AM CST
The only way to really solve your problem is your own server at your home or business location. Uploading during the event is easy, i am streaming live video and uploading pics from very remote locations. so that is another option for you. Plenty of services out there that offer hotspot cards for laptops.
Edited Mon Jun 06, 22 11:26 AM by Vance Birno
J
8 posts
Tue Jun 07, 22 7:58 AM CST
Consider dedicated hosting. I have 4TB of space for under 40euro a month.
You have to set it all up yourself, but its the cheapest option by far, and negates the use of the slow S3 bucket
7 total messages
Please log in or Create an account to post or reply to topics.
This post has been viewed 921 times
 
Loading more pages
Loading more pages

Sign up for email promotions.

Your information is safe with us and won't be shared.

Thank you for signing up!

 
©2003 - 2021 Grissett, LLC. All Rights Reserved.

By continuing to browse or by clicking Accept Cookies, you agree to the storing of cookies on your device necessary to provide you with the services available through our website.

    Accept   Privacy & Cookie Policy
Loading More Photos
Scroll To Top
Close Window
Loading
Close