S3 access incredibly slow... not able to add transfer acceleration points.

You need to be logged in to post in the forum - Log In

An active JCE Pro Subscription is required to post in the forum - Buy a Subscription

Support is currently Offline

Official support hours

Monday to Friday
09:00 - 17:00 Europe/London (BST)

Please create a new Ticket and we will get back to you as soon as we can.

#100669 S3 access incredibly slow... not able to add transfer acceleration points.

Posted in ‘Editor’

This is a public ticket

Everybody will be able to see its contents. Do not include usernames, passwords or any other sensitive information.

Latest post by Ryan on Monday, 05 August 2019 09:44 BST

Friday, 02 August 2019 06:20 BST

Wytchcat

Hey there Ryan,

I am having a couple of issues specifically with images and files on S3.

I am using a single bucket on S3 with about 1451 files... mostly pics or pdfs. On pulling up Image editor pro to add or update a pic it can take up to 2 mins to finalize the file hierarchy.

I've tried toggling settings to do things like SSL vs no SSL inquiries and considered using transfer acceleration endpoints.. but the settings won't allow that in Profile Editor.

And it doesn't speed up over time. Any suggestions?

Latest of JCE, S3 plugin and Joomla. Also running RSFirewall but haven't had an issue.

Monday, 05 August 2019 09:44 BST

Ryan

There are factors that contribute to this issue:

1. When a request is made to display the files and folders, or perform any action on them, this request is made from your browser, to your server, and then to Amazon S3. The speed at which this happens is there dependant on the speed of your internet connection, the capacity and speed of the server your site runs and, and the speed at which it is able to connect to Amazon S3.

2. S3 has a limit of 1000 items, so in this case 2 requests must be made to get all the items. Furthermore, S3 does not provide all the information in the response to generate the data needed for the display and use of the files. An additional request must be made for each item to determine its ACL parameters, so a suitable URL can be generated for the file - either a public URL with or without a canonical name, or an authenticated URL.

The S3 Filesystem plugin uses a local cache to attempt to speed up this process, but this can cause its own issues when the cache contains a lot of data.

The S3 API does have a mechanism to retrieve portions of the item list at a time, but as the PHP filesystem - what is used to get a file list from your server - does not do this, the JCE Filesystem API is not built to accommodate this.

I have been experimenting over the weekend with implementing this, and I hope to have a solution in JCE 2.8. There are still a number of issues with the S3 API that will not make this a complete solution (S3 does not provide a total number or items, so it is not possible to know how many files or folders are in a bucket or folder without retrieving the full list). Hopefully I will be able to deal with this completely when the whole filesystem is re-written in JCE 3.x

Ryan Demmer

Lead Developer / CEO / CTO

Just because you're not paranoid doesn't mean everybody isn't out to get you.