JMIX 1.x app NOT working on AWS - Communication error: Invalid JSON from server: 1|X

[krivopustov (Konstantin Krivopustov)]

Hi @tomas.klems

I’ve been testing access to your 2 apps for a few hours (by opening them in the browser and periodically refreshing Users and opening details). I got the aforementioned error only once in the Classic UI app, and never in Flow UI.
My colleague did the same and got similar results - one error in several hours, but in Flow UI, and with unstable internet connection.

I don’t think these errors are acceptable, but I’m afraid we can do little in this situation. The problem is somewhere deep in Vaadin.

So I would suggest trying recommendations by @klaus or if it doesn’t help to change your deployment from Elastic Beanstalk to bare metal EC2.

Regards,
Konstantin

[mario (Mario David)]

Hi,

BTW I also back then saw problems with CUBA and elastic beanstalk. If I recall correctly on ECS ist was gone as well (Beanstalk is the only service adding Nginx in front).

Cheers
Mario

[tomas.klems (Tomáš Klemš)]

OK, thank you @klaus and @krivopustov.
So my understanding so far is:

1. I am not alone, it happened to other guys too. So my design/implementation is not “the guilty party”.
2. It happens only in AWS deployment, no problems in local dev environment.
3. It does not depend on browser or client OS. It is totally random. I tested AS, FF, GC on macOS and Windows.
4. The app logs show no error, the nginx logs show no error. The error is only in client in the browser reported by VAADIN.
5. It affects Jmix1 and 2, where Jmix 2 has much less incidents. But it has been reported in CUBA and pure Vaadin apps too.
6. It is probably caused by the Vaadin and/or Atmoshpere library.
7. The AWS Elasctic Loadbalancer is not the suspect, because it happens even without the ELB.
8. The main suspect is local Nginx reverse proxy on EC2 Linux VM instance.
9. In Jmix 2 it is possible to tweak timeouts to minimise the occurrences. But it is with no guarantee.
10. In Jmix 1, no solution. Sometimes it is OK for several hours, sometimes it happens every few minutes. No workaround helped.
11. The only strange things from logging - before UI freezes - are these from browser JS console:

vaadinPush.js?v=8.14.3-4-jmix:1 Websocket closed, reason: Connection was closed abnormally (that is, with no close frame being sent). - wasClean: false

10:59:51.389 Websocket closed, reason: 
Connection was closed abnormally (that is, with no close frame being sent). - 
wasClean: false vaadinPush.js:1:40443

IMHO going to bare metal does make any difference.
Going to EC2 instance without nginx reverse proxy might help. Agree.
But we want the reverse proxy to be there. It is a could deployment pattern, I want the app to be HA in an autoscaling group. The nginx is there for a reason. Elastic Beanstalk is not a special EC2 instance. It is only a Cloudformation script which spins up an EC2 instance with nginx preconfigured.

So Konstatin @krivopustov, what can I do with Vaadin? I can report thatg to Vaadin, but I am not paying for Vaadin and their support. Bu I gues you are the customer of Vaadin. How could I proceed?

Thank you.
Regrds,
Tomas

[tomas.klems (Tomáš Klemš)]

OK, thank you Mario. I got it. Get rid of the nginx…
It is not easy to switch to containers at the moment…
Regards,
Tomas

[krivopustov (Konstantin Krivopustov)]

@tomas.klems

Going to EC2 instance without nginx reverse proxy might help. Agree.
But we want the reverse proxy to be there.

An executable JAR or containerized application with custom configured Nginx as a reverse proxy should work fine on bare EC2. At least we don’t have reports of the opposite.

[tomas.klems (Tomáš Klemš)]

I am quite sure, that it is all the same, the Beanstalk or AKS(container).
Beanstalk has pure EC2 under the hood. Nothing special. I can fully control it.
The problem is there ONLY with a proxy.

So is there a bug in NGINX? I doubt it.

The problem might be in Atmosphere/Vaadin.
Or in Jmix.
IMHO it is related to

WARN org.atmosphere.util.IOUtils: More than one Servlet Mapping defined. WebSocket may not work org.apache.catalina.core.ApplicationServletRegistration@df3dcd3

Which causes a problem in combination with a proxy.

[tomas.klems (Tomáš Klemš)]

I managed to fix (hotfix) the issue using the massive timeouts as suggested by

JMIX 1.x app NOT working on AWS - Communication error: Invalid JSON from server: 1|X

So I also had this problem and I also did a lot of investigation and trial and error. For the application load balancer I use a HTTP client keep alive duration: 3600 seconds. Very long I know. Somehow it seems to help. Also a Connection idle timeout: 900 seconds. For the nginx proxy configuration I use the following: proxy_connect_timeout 600s; proxy_read_timeout 900s; proxy_send_timeout 900s; proxy_set_header Connection $connecti…

I have never seen this error since this.

FYI, For AWS EB, the change of nginx config is lost after restart/redeploy/…
To make it permanent, use AWS EB .platfom extension.

For EB deployment, make a zip, which will contain

• uber jar
• .platform/nginx/conf.d/elasticbeanstalk/timeout.conf

with this content: timeout.conf

    proxy_connect_timeout    600s;
    proxy_read_timeout    900s;
    proxy_send_timeout   900s;

It is also good idea to create a Gradle task to reate that zip.

build.gradle:

...
.....
.......
tasks.register('createPlatformDir', Copy) {
    into('build/.platform')
    from('src/main/elasticbeanstalk/.platform')
}
tasks.register('createEbExtensionsDir', Copy) {
    into('build/.ebextensions')
    from('src/main/elasticbeanstalk/.ebextensions')
}

tasks.register('bundleForElasticBeanstalk', Zip) {
    dependsOn bootJar
    dependsOn createPlatformDir
    dependsOn createEbExtensionsDir
    from('build/libs') {
        include '*.jar'
    }
    into('.platform') {
        from('build/.platform')
    }
    into('.ebextensions') {
        from('build/.ebextensions')
    }
    archiveBaseName.set("aws-eb-${project.name}")
    destinationDirectory.set(file('build/distributions'))
}

Good luck!

[mbucan (Mladen Bucan)]

Nice work Tomas!
Were you then able to deploy your application as jar to AWS EB, using AWS Load balancer?

[tomas.klems (Tomáš Klemš)]

Well yes, it is in PROD, behind AWS Loadbalancer, EC2 instance in a autoscaling group…
No errors so far.
Regards,
Tomas

[qilin (Lin Qi)]

The issue can also be reproduced when deploy the Jmix app in a ECS instance behand the AWS LB.

We are using Jmix v1.5.5, and deploy the app into an ECS instance as docker container, we also configured a AWS LB for high availability. The error appears from time to time.

[tomas.klems (Tomáš Klemš)]

Thank you @qilin!
This is a confirmation, that suggested solution “using ECS instead of EB” does not make any difference.
The error is there on EB as well as on ESC.

Also my finding is, that AWS Application Loadbalancer is not important.
I got the error with and without it too.

Regards,
Tomas