Can't check if 'fileExists' from s3 fileStorage

t.vignal · October 30, 2024, 1:55pm

Jmix 1.5
Hi I need to store my files in a sub-folder of my s3 bucket.
I follow the idea suggested here, it worked.
(I can upload, download and delete ) but AwsFileStorage.fileExists() throws an exception. It says that the file doesn’t exist but it do exist.
I tried with both:

jmix.awsfs.endpoint-url = https://bucketName.s3.amazonaws.com/clientFoo
jmix.awsfs.endpoint-url = https://s3.amazonaws.com/bucketName/clientFoo

If I don’t specify a sub-folder it works.

the error stack

Caused by: software.amazon.awssdk.services.s3.model.NoSuchKeyException: The specified key does not exist. (Service: S3, Status Code: 404, Request ID: txga8f4ba7f97ae422698a3-0067223076, Extended Request ID: txga8f4ba7f97ae422698a3-0067223076)

	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:135)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:161)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:114)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
	at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:6032)
	at io.jmix.awsfs.AwsFileStorage.fileExists(AwsFileStorage.java:291)

s.markevich · October 31, 2024, 9:07am

Hi!
Thank you for contacting Jmix support.
Could you please provide a bucket name and file name causing the exception. Or what symbols are used in bucket and file name?

t.vignal · October 31, 2024, 10:22am

actually my url looks like this: https://s3.fr-par.company.com/my-bucket/toto (dash in the bucket’s name but not in the folder’s)
The file is named README(1)(1)(1).md but I think any name would do the job.

Thanks

t.rusiaikina · November 5, 2024, 12:34pm

Hi, I was able to reproduce this problem.

jmix.awsfs.accessKey=<>
jmix.awsfs.secretAccessKey=<>
jmix.awsfs.region =us-east-1
jmix.awsfs.bucket =elasticbeanstalk-us-east-1-226388427045
jmix.awsfs.chunkSize = 5500
jmix.core.defaultFileStorage=s3

I originally filled in this data and did not specify jmix.awsfs.endpoint-url. I uploaded a few files, only then did I specify jmix.awsfs.endpoint-url. Uploaded the files again.
I got an exception when downloading old files. The new ones downloaded without error
All the files were in the repository, but their paths were different

My paths look like this:

jmix.awsfs.endpoint-url = http://s3.amazonaws.com/elasticbeanstalk-us-east-1-226388427045.s3.us-east-1.amazonaws.com/2024/11/05/testfolder/

s3://elasticbeanstalk-us-east-1-226388427045/.s3.us-east-1.amazonaws.com/2024/11/05/testfolder/ - Copy S3 URI

https://elasticbeanstalk-us-east-1-226388427045.s3.us-east-1.amazonaws.com/.s3.us-east-1.amazonaws.com/2024/11/05/testfolder/ - Copy URL

Please check if the paths to the downloaded file are correct.

t.vignal · November 5, 2024, 1:25pm

Hi, my test case was different:

set jmix.awsfs.endpoint-url to something like https://s3.fr-par.company.com/my-bucket-name/my-folder-name
start the app
upload a file. (we now have a file that exist like my-bucket-name/my-folder-name/2024/11/05/UUID.jpeg )
try to check if the file exists with AwsFileStorage.fileExists()
the last step throws the same error stack as in my first message. The file exist so true should returned by the function instead of an exception.

No need to change the url or to restart the app.