JMIX 1.x app NOT working on AWS - Communication error: Invalid JSON from server: 1|X

gorelov · March 15, 2024, 7:41am

Hi,

According to Vaadin forum topics with similar problem, as an option, you can try to change Push transport from websocket to long-polling. To do this:

Create a custom AppUI class

@Push(transport = Transport.LONG_POLLING)
public class CustomAppUI extends AppUI {
}

Register this bean

@Bean
public AppUI customAppUI() {
    return new CustomAppUI();
}

Exclude built-in AppUI class in application.properties:

jmix.core.exclude-beans = appUI

Regards,
Gleb

tomas.klems · March 18, 2024, 9:51am

@gorelov Dear Gleb,
thank you very much for this suggestion.
And yes I am able to switch to LONG POLLING:

2024-03-17 23:54:35.830 TRACE 71664 --- [nio-8080-exec-1] o.a.c.DefaultAtmosphereResourceFactory   : Adding: AtmosphereResource{
	 uuid=04d3481c-e839-4441-8986-af86ecb9401c,
	 transport=LONG_POLLING,
	 isInScope=true,
	 isResumed=false,
	 isCancelled=false,
	 isSuspended=true,
	 broadcasters=/*,
	 isClosedByClient=false,
	 isClosedByApplication=false,
	 action=Action{timeout=-1, type=SUSPEND}}

Interesting is that, I got a lot of these error (it is INFO level actually):

2024-03-17 23:49:25.977  INFO 71664 --- [nio-8080-exec-2] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:49:26.994  INFO 71664 --- [nio-8080-exec-4] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:49:29.014  INFO 71664 --- [nio-8080-exec-7] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-03-17 23:49:31.018  INFO 71664 --- [io-8080-exec-10] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:49:32.021  INFO 71664 --- [nio-8080-exec-2] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:49:36.036  INFO 71664 --- [nio-8080-exec-7] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:49:37.047  INFO 71664 --- [nio-8080-exec-9] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:49:41.049  INFO 71664 --- [nio-8080-exec-4] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:49:42.061  INFO 71664 --- [nio-8080-exec-6] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:49:46.069  INFO 71664 --- [nio-8080-exec-1] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:49:47.078  INFO 71664 --- [nio-8080-exec-3] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:49:51.081  INFO 71664 --- [nio-8080-exec-8] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:49:52.094  INFO 71664 --- [io-8080-exec-10] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:49:56.098  INFO 71664 --- [nio-8080-exec-5] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:49:57.111  INFO 71664 --- [nio-8080-exec-7] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:50:01.113  INFO 71664 --- [nio-8080-exec-2] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:50:02.149  INFO 71664 --- [nio-8080-exec-4] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:50:06.134  INFO 71664 --- [nio-8080-exec-9] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:50:07.171  INFO 71664 --- [nio-8080-exec-1] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.
2024-03-17 23:50:11.145  INFO 71664 --- [nio-8080-exec-6] com.vaadin.server.VaadinServlet          : Requested resource [/VAADIN/push] not found from filesystem or through class loader. Add widgetset and/or theme JAR to your classpath or add files to WebContent/VAADIN folder.

BUT this error is suspicious:

IllegalStateException: Session has already been set.

2024-03-17 23:55:02.306 ERROR 71664 --- [nio-8080-exec-9] com.vaadin.server.DefaultErrorHandler    : 

java.lang.IllegalStateException: Session has already been set. Old session: com.vaadin.server.VaadinSession@62bc9424 for jmixVaadinServlet. New session: com.vaadin.server.VaadinSession@16549935 for jmixVaadinServlet.
	at com.vaadin.ui.UI.setSession(UI.java:502) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.communication.UIInitHandler.getBrowserDetailsUI(UIInitHandler.java:189) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]

I will try it on AWS…

Thank you.
Regards
Tomas

tomas.klems · March 19, 2024, 3:44pm

OK, changing the ApUI bean is the way to hell…

After start start of the app:

2024-03-19 10:51:00.217 ERROR 69968 --- [nio-8080-exec-6] com.vaadin.server.DefaultErrorHandler    : 
java.lang.IllegalStateException: Session has already been set. Old session: com.vaadin.server.VaadinSession@40113ab6 for jmixVaadinServlet. New session: com.vaadin.server.VaadinSession@755c7445 for jmixVaadinServlet.
	at com.vaadin.ui.UI.setSession(UI.java:502) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.communication.UIInitHandler.getBrowserDetailsUI(UIInitHandler.java:189) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.communication.UIInitHandler.synchronizedHandleRequest(UIInitHandler.java:67) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.SynchronizedRequestHandler.handleRequest(SynchronizedRequestHandler.java:40) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.VaadinService.handleRequest(VaadinService.java:1636) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:465) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) ~[javax.servlet-api-4.0.1.jar:4.0.1]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) ~[tomcat-embed-core-9.0.83.jar:9.0.83]

2024-03-19 10:51:00.220 ERROR 69968 --- [nio-8080-exec-6] o.a.c.c.C.[.[.[/].[jmixVaadinServlet]    : Servlet.service() for servlet [jmixVaadinServlet] threw exception
java.lang.IllegalStateException: Session has already been set. Old session: com.vaadin.server.VaadinSession@40113ab6 for jmixVaadinServlet. New session: com.vaadin.server.VaadinSession@755c7445 for jmixVaadinServlet.
	at com.vaadin.ui.UI.setSession(UI.java:502) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.communication.UIInitHandler.getBrowserDetailsUI(UIInitHandler.java:189) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.communication.UIInitHandler.synchronizedHandleRequest(UIInitHandler.java:67) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.SynchronizedRequestHandler.handleRequest(SynchronizedRequestHandler.java:40) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.VaadinService.handleRequest(VaadinService.java:1636) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:465) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) ~[javax.servlet-api-4.0.1.jar:4.0.1]

2024-03-19 10:51:00.231 ERROR 69968 --- [nio-8080-exec-6] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [com.vaadin.server.ServiceException: java.lang.IllegalStateException: Session has already been set. Old session: com.vaadin.server.VaadinSession@40113ab6 for jmixVaadinServlet. New session: com.vaadin.server.VaadinSession@755c7445 for jmixVaadinServlet.] with root cause
java.lang.IllegalStateException: Session has already been set. Old session: com.vaadin.server.VaadinSession@40113ab6 for jmixVaadinServlet. New session: com.vaadin.server.VaadinSession@755c7445 for jmixVaadinServlet.
	at com.vaadin.ui.UI.setSession(UI.java:502) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.communication.UIInitHandler.getBrowserDetailsUI(UIInitHandler.java:189) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.communication.UIInitHandler.synchronizedHandleRequest(UIInitHandler.java:67) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.SynchronizedRequestHandler.handleRequest(SynchronizedRequestHandler.java:40) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.VaadinService.handleRequest(VaadinService.java:1636) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:465) ~[vaadin-server-8.14.3-4-jmix.jar:8.14.3-4-jmix]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) ~[javax.servlet-api-4.0.1.jar:4.0.1]

The App works. Just log has these errors.

The real pain is after logout.
If you click the Logout link, the app end s in deadend loop.

Mar 19 15:37:12 ip-172-31-14-147 web[5978]: 2024-03-19 15:37:12.579 ERROR 5978 --- [nio-5000-exec-7] io.jmix.ui.AppUI                         : Error handling exception
Mar 19 15:37:12 ip-172-31-14-147 web[5978]: Original exception:
Mar 19 15:37:12 ip-172-31-14-147 web[5978]: java.lang.IllegalStateException: This UI instance is already initialized (as UI id 0) and can therefore not be initialized again (as UI id 2). Please make sure you are not accidentally reusing an old UI instance.
Mar 19 15:37:12 ip-172-31-14-147 web[5978]: #011at com.vaadin.ui.UI.doInit(UI.java:730)
Mar 19 15:37:12 ip-172-31-14-147 web[5978]: #011at com.vaadin.server.communication.UIInitHandler.getBrowserDetailsUI(UIInitHandler.java:209)
Mar 19 15:37:12 ip-172-31-14-147 web[5978]: #011at com.vaadin.server.communication.UIInitHandler.synchronizedHandleRequest(UIInitHandler.java:67)
Mar 19 15:37:12 ip-172-31-14-147 web[5978]: #011at com.vaadin.server.SynchronizedRequestHandler.handleRequest(SynchronizedRequestHandler.java:40)
Mar 19 15:37:12 ip-172-31-14-147 web[5978]: #011at com.vaadin.server.VaadinService.handleRequest(VaadinService.java:1636)
Mar 19 15:37:12 ip-172-31-14-147 web[5978]: #011at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:465)
Mar 19 15:37:12 ip-172-31-14-147 web[5978]: #011at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
Mar 19 15:37:12 ip-172-31-14-147 web[5978]: #011at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209)

Mar 19 15:38:22 ip-172-31-14-147 web[5978]: 2024-03-19 15:38:22.349 ERROR 5978 --- [nio-5000-exec-5] io.jmix.ui.AppUI                         : Error handling exception
Mar 19 15:38:22 ip-172-31-14-147 web[5978]: Original exception:
Mar 19 15:38:22 ip-172-31-14-147 web[5978]: java.lang.IllegalStateException: This UI instance is already initialized (as UI id 0) and can therefore not be initialized again (as UI id 3). Please make sure you are not accidentally reusing an old UI instance.


Mar 19 15:38:22 ip-172-31-14-147 web[5978]: Exception in handlers:
Mar 19 15:38:22 ip-172-31-14-147 web[5978]: java.lang.IllegalStateException: No App is bound to the current VaadinSession

It is impossible to reload and login agaian. Only server restart helps.

tomas.klems · March 20, 2024, 10:05am

No difference.
This “AppUI customAppUI()” does not work.
IllegalStateException: Session has already been set. Old session:…

krivopustov · March 22, 2024, 10:09am

@tomas.klems - could you prepare a simplest test application and deployment instructions to reproduce the problem?
Also, if possible, please test this example application with the Quick Cloud Deployment (it uses Apache HTTP Server as a reverse proxy).

tomas.klems · March 26, 2024, 10:22pm

Dear @krivopustov, here is the update:

The problem is the same with Jmix 2.2, not only 1.5
Sample project attached, but there is nothing special. It works on local. On AWS it fails randomly.
The project are deployed here and you an experience the problem yourself (just wait for it).

http://awstestone.eu-central-1.elasticbeanstalk.com/?debug
http://awstesttwo.eu-central-1.elasticbeanstalk.com/

I will keep them here for few days to let you test…

There is one more news: In JMix 2.2, there is HTTP 403 after websocket 101 “error”.

Thank you.

PS: as I wrote before, it is

AWS Elastic Beanstalk Java PaaS, Corretto 11 resp. 17, 64bit, Amazon Linux 2023, Postgres DB, t4g.small.
Single instance, no balancer, no firewall, no ssl.
There is standard EC2 instance NGINX proxy 80:5000

Regards,
Tomas.

tomas.klems · March 26, 2024, 10:38pm

In JMix 2.2 there are also ERRORs in the server log:

After logout:

Mar 26 22:32:52 ip-172-31-30-39 web[28045]: 2024-03-26T22:32:52.920Z ERROR 28045 --- [sphere-Shared-1] c.v.f.s.c.PushAtmosphereHandler          : Exception in push connection
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: java.io.IOException: Connection remotely closed for 867aaffc-74f6-4ff8-9398-224b9597e376
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at org.atmosphere.websocket.WebSocket.write(WebSocket.java:237) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at org.atmosphere.websocket.WebSocket.write(WebSocket.java:227) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at org.atmosphere.websocket.WebSocket.write(WebSocket.java:48) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at org.atmosphere.cpr.AtmosphereResponseImpl$Stream.write(AtmosphereResponseImpl.java:955) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at org.atmosphere.handler.AbstractReflectorAtmosphereHandler.onStateChange(AbstractReflectorAtmosphereHandler.java:154) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at com.vaadin.flow.server.communication.PushAtmosphereHandler.onStateChange(PushAtmosphereHandler.java:54) ~[flow-server-24.3.3.jar!/:24.3.3]
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at org.atmosphere.cpr.DefaultBroadcaster.invokeOnStateChange(DefaultBroadcaster.java:1036) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at org.atmosphere.cpr.DefaultBroadcaster.prepareInvokeOnStateChange(DefaultBroadcaster.java:1056) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at org.atmosphere.cpr.DefaultBroadcaster.executeAsyncWrite(DefaultBroadcaster.java:870) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at org.atmosphere.cpr.DefaultBroadcaster$2.run(DefaultBroadcaster.java:477) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[na:na]
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[na:na]
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
Mar 26 22:32:52 ip-172-31-30-39 web[28045]: #011at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

And this error:

Mar 26 22:51:16 ip-172-31-30-39 web[41299]: 2024-03-26T22:51:16.179Z DEBUG 41299 --- [io-5000-exec-10] o.a.websocket.DefaultWebSocketProcessor  : Delaying closing operation for firefox and resource 8294ef86-9ddb-4959-9111-bd3384ba4e0f
Mar 26 22:51:48 ip-172-31-30-39 web[41299]: 2024-03-26T22:51:48.440Z DEBUG 41299 --- [nio-5000-exec-7] o.a.websocket.DefaultWebSocketProcessor  : Delaying closing operation for firefox and resource 8294ef86-9ddb-4959-9111-bd3384ba4e0f
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: 2024-03-26T22:52:40.471Z DEBUG 41299 --- [nio-5000-exec-5] o.a.websocket.DefaultWebSocketProcessor  : Delaying closing operation for firefox and resource f6f0246f-2677-4f07-8163-fa1a101542a6
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: 2024-03-26T22:52:40.501Z ERROR 41299 --- [sphere-Shared-0] c.v.f.s.c.PushAtmosphereHandler          : Exception in push connection
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: java.io.IOException: Connection remotely closed for f6f0246f-2677-4f07-8163-fa1a101542a6
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at org.atmosphere.websocket.WebSocket.write(WebSocket.java:237) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at org.atmosphere.websocket.WebSocket.write(WebSocket.java:227) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at org.atmosphere.websocket.WebSocket.write(WebSocket.java:48) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at org.atmosphere.cpr.AtmosphereResponseImpl$Stream.write(AtmosphereResponseImpl.java:955) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at org.atmosphere.handler.AbstractReflectorAtmosphereHandler.onStateChange(AbstractReflectorAtmosphereHandler.java:154) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at com.vaadin.flow.server.communication.PushAtmosphereHandler.onStateChange(PushAtmosphereHandler.java:54) ~[flow-server-24.3.3.jar!/:24.3.3]
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at org.atmosphere.cpr.DefaultBroadcaster.invokeOnStateChange(DefaultBroadcaster.java:1036) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at org.atmosphere.cpr.DefaultBroadcaster.prepareInvokeOnStateChange(DefaultBroadcaster.java:1056) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at org.atmosphere.cpr.DefaultBroadcaster.executeAsyncWrite(DefaultBroadcaster.java:870) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at org.atmosphere.cpr.DefaultBroadcaster$2.run(DefaultBroadcaster.java:477) ~[atmosphere-runtime-3.0.3.slf4jvaadin2.jar!/:3.0.3.slf4jvaadin2]
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[na:na]
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[na:na]
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
Mar 26 22:52:40 ip-172-31-30-39 web[41299]: #011at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: 2024-03-26T22:53:41.045Z DEBUG 41299 --- [nio-5000-exec-4] o.atmosphere.container.JSR356Endpoint    : Problem in web socket session
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: java.io.EOFException: null
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1296) ~[tomcat-embed-core-10.1.18.jar!/:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1184) ~[tomcat-embed-core-10.1.18.jar!/:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at org.apache.tomcat.websocket.server.WsFrameServer.onDataAvailable(WsFrameServer.java:74) ~[tomcat-embed-websocket-10.1.18.jar!/:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at org.apache.tomcat.websocket.server.WsFrameServer.doOnDataAvailable(WsFrameServer.java:184) ~[tomcat-embed-websocket-10.1.18.jar!/:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at org.apache.tomcat.websocket.server.WsFrameServer.notifyDataAvailable(WsFrameServer.java:164) ~[tomcat-embed-websocket-10.1.18.jar!/:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:152) ~[tomcat-embed-websocket-10.1.18.jar!/:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:60) ~[tomcat-embed-core-10.1.18.jar!/:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:57) ~[tomcat-embed-core-10.1.18.jar!/:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896) ~[tomcat-embed-core-10.1.18.jar!/:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744) ~[tomcat-embed-core-10.1.18.jar!/:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-10.1.18.jar!/:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-10.1.18.jar!/:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-10.1.18.jar!/:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-10.1.18.jar!/:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: #011at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
Mar 26 22:53:41 ip-172-31-30-39 web[41299]: 2024-03-26T22:53:41.046Z DEBUG 41299 --- [nio-5000-exec-4] o.a.websocket.DefaultWebSocketProcessor  : Delaying closing operation for firefox and resource f7acc60b-f0ce-4717-9895-324fea322814

tomas.klems · March 27, 2024, 3:31pm

The error is much more frequent in AWSTestOne than in AwsTestTwo.
So I guess something is better in JMix 2.2. … the new VAADIN, newer Atmosphere,…etc.

klaus · March 27, 2024, 7:51pm

So I also had this problem and I also did a lot of investigation and trial and error.

For the application load balancer I use a HTTP client keep alive duration: 3600 seconds. Very long I know. Somehow it seems to help. Also a Connection idle timeout: 900 seconds.

For the nginx proxy configuration I use the following:

            proxy_connect_timeout 600s;
            proxy_read_timeout  900s;
            proxy_send_timeout  900s;

            proxy_set_header    Connection          $connection_upgrade;
            proxy_set_header    Upgrade             $http_upgrade;
            proxy_set_header    Host                $host;
            proxy_set_header    X-Real-IP           $remote_addr;
            proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
            proxy_redirect      off;

I didn’t run into this problem now for quite some time, so it looks like this helps. Also I use newest Jmix (almost) 2.1.3

krivopustov · March 28, 2024, 12:42pm

Hi @tomas.klems

I’ve been testing access to your 2 apps for a few hours (by opening them in the browser and periodically refreshing Users and opening details). I got the aforementioned error only once in the Classic UI app, and never in Flow UI.
My colleague did the same and got similar results - one error in several hours, but in Flow UI, and with unstable internet connection.

I don’t think these errors are acceptable, but I’m afraid we can do little in this situation. The problem is somewhere deep in Vaadin.

So I would suggest trying recommendations by @klaus or if it doesn’t help to change your deployment from Elastic Beanstalk to bare metal EC2.

Regards,
Konstantin

mario · March 28, 2024, 4:51pm

Hi,

BTW I also back then saw problems with CUBA and elastic beanstalk. If I recall correctly on ECS ist was gone as well (Beanstalk is the only service adding Nginx in front).

Cheers
Mario

tomas.klems · March 28, 2024, 5:20pm

OK, thank you @klaus and @krivopustov.
So my understanding so far is:

I am not alone, it happened to other guys too. So my design/implementation is not “the guilty party”.
It happens only in AWS deployment, no problems in local dev environment.
It does not depend on browser or client OS. It is totally random. I tested AS, FF, GC on macOS and Windows.
The app logs show no error, the nginx logs show no error. The error is only in client in the browser reported by VAADIN.
It affects Jmix1 and 2, where Jmix 2 has much less incidents. But it has been reported in CUBA and pure Vaadin apps too.
It is probably caused by the Vaadin and/or Atmoshpere library.
The AWS Elasctic Loadbalancer is not the suspect, because it happens even without the ELB.
The main suspect is local Nginx reverse proxy on EC2 Linux VM instance.
In Jmix 2 it is possible to tweak timeouts to minimise the occurrences. But it is with no guarantee.
In Jmix 1, no solution. Sometimes it is OK for several hours, sometimes it happens every few minutes. No workaround helped.
The only strange things from logging - before UI freezes - are these from browser JS console:

vaadinPush.js?v=8.14.3-4-jmix:1 Websocket closed, reason: Connection was closed abnormally (that is, with no close frame being sent). - wasClean: false

10:59:51.389 Websocket closed, reason: 
Connection was closed abnormally (that is, with no close frame being sent). - 
wasClean: false vaadinPush.js:1:40443

IMHO going to bare metal does make any difference.
Going to EC2 instance without nginx reverse proxy might help. Agree.
But we want the reverse proxy to be there. It is a could deployment pattern, I want the app to be HA in an autoscaling group. The nginx is there for a reason. Elastic Beanstalk is not a special EC2 instance. It is only a Cloudformation script which spins up an EC2 instance with nginx preconfigured.

So Konstatin @krivopustov, what can I do with Vaadin? I can report thatg to Vaadin, but I am not paying for Vaadin and their support. Bu I gues you are the customer of Vaadin. How could I proceed?

Thank you.
Regrds,
Tomas

tomas.klems · March 28, 2024, 5:28pm

OK, thank you Mario. I got it. Get rid of the nginx…
It is not easy to switch to containers at the moment…
Regards,
Tomas

krivopustov · March 29, 2024, 7:16am

An executable JAR or containerized application with custom configured Nginx as a reverse proxy should work fine on bare EC2. At least we don’t have reports of the opposite.

tomas.klems · April 10, 2024, 7:49am

I am quite sure, that it is all the same, the Beanstalk or AKS(container).
Beanstalk has pure EC2 under the hood. Nothing special. I can fully control it.
The problem is there ONLY with a proxy.

So is there a bug in NGINX? I doubt it.

The problem might be in Atmosphere/Vaadin.
Or in Jmix.
IMHO it is related to

WARN org.atmosphere.util.IOUtils: More than one Servlet Mapping defined. WebSocket may not work org.apache.catalina.core.ApplicationServletRegistration@df3dcd3

Which causes a problem in combination with a proxy.

tomas.klems · April 10, 2024, 8:04am

I managed to fix (hotfix) the issue using the massive timeouts as suggested by

I have never seen this error since this.

FYI, For AWS EB, the change of nginx config is lost after restart/redeploy/…
To make it permanent, use AWS EB .platfom extension.

For EB deployment, make a zip, which will contain

uber jar
.platform/nginx/conf.d/elasticbeanstalk/timeout.conf

with this content: timeout.conf

    proxy_connect_timeout    600s;
    proxy_read_timeout    900s;
    proxy_send_timeout   900s;

It is also good idea to create a Gradle task to reate that zip.

build.gradle:

...
.....
.......
tasks.register('createPlatformDir', Copy) {
    into('build/.platform')
    from('src/main/elasticbeanstalk/.platform')
}
tasks.register('createEbExtensionsDir', Copy) {
    into('build/.ebextensions')
    from('src/main/elasticbeanstalk/.ebextensions')
}

tasks.register('bundleForElasticBeanstalk', Zip) {
    dependsOn bootJar
    dependsOn createPlatformDir
    dependsOn createEbExtensionsDir
    from('build/libs') {
        include '*.jar'
    }
    into('.platform') {
        from('build/.platform')
    }
    into('.ebextensions') {
        from('build/.ebextensions')
    }
    archiveBaseName.set("aws-eb-${project.name}")
    destinationDirectory.set(file('build/distributions'))
}

Good luck!

mbucan · April 10, 2024, 8:28am

Nice work Tomas!
Were you then able to deploy your application as jar to AWS EB, using AWS Load balancer?

tomas.klems · April 10, 2024, 8:38am

Well yes, it is in PROD, behind AWS Loadbalancer, EC2 instance in a autoscaling group…
No errors so far.
Regards,
Tomas

qilin · April 10, 2024, 10:23am

The issue can also be reproduced when deploy the Jmix app in a ECS instance behand the AWS LB.

We are using Jmix v1.5.5, and deploy the app into an ECS instance as docker container, we also configured a AWS LB for high availability. The error appears from time to time.

tomas.klems · April 10, 2024, 10:30am

Thank you @qilin!
This is a confirmation, that suggested solution “using ECS instead of EB” does not make any difference.
The error is there on EB as well as on ESC.

Also my finding is, that AWS Application Loadbalancer is not important.
I got the error with and without it too.

Regards,
Tomas