Schedulers get triggered by multiple servers on HA Environment Setup

Support
#1

Hello, Jmix Team.

We are using the HA setup(Load balancer) for our application, which consists of two servers (Server 1 and Server 2) with a shared database. We set up schedulers that can be triggered on any of the servers. But the scheduler is sometimes triggered by both servers. How can we prevent it from being triggered from both servers?

#3

Hello Yogesh.

Please privide additional info.

  1. Do you have a case when some job fires on both servers at the same time according to its schedule?
  2. How frequently does this job fire? And what is expected execution duration?
  3. Did you try @DisallowConcurrentExecution annotation on your job class?
  4. What quartz-specific application properties do you use in your application?

Regards,
Ivan

#4

Hi Ivan,
Here are the answers to your questions:-

1.Do you have a case when some job fires on both servers at the same time according to its schedule?

Yes, we do encounter scenarios where the job fires simultaneously on both servers, with only a few milliseconds of difference between them.

2.How frequently does this job fire? And what is the expected execution duration?

The job is scheduled to fire at a specific time,i our case its 10:00:00, but it executes on both servers almost simultaneously, with only a few milliseconds of difference between them.On Server 1, it fires at 10:00:00.128, and on Server 2, it fires at 10:00:00.075. The duration of execution for this job is within acceptable limits.

3.Did you try @DisallowConcurrentExecution annotation on your job class?

No, we haven’t utilized the @DisallowConcurrentExecution annotation in our job class.

4.What quartz-specific application properties do you use in your application?

We haven’t configured any quartz-specific application properties in our application.

Kindly suggest something to prevent it from being triggered from both servers.

#5

Hi Puneet.

This can happen because scheduler is not properly configured for cluster environment.
Please take a look at this - Configuration Reference.

The key points:

  • org.quartz.scheduler.instanceId = AUTO
  • org.quartz.jobStore.isClustered = true

Also take a look at @DisallowConcurrentExecution - this annotation prevents multiple parallel execution of the same Job (in case of overlapping trigger execution in general).
Note: It affects the single Job (Quartz JobDetails) using annotated job class. If you have multiple Jobs using the same job class - they are treated as independent Jobs (just with the same logic) and can be executed at the same time.

#6

Aftermath Analysis:

  1. Upon further examination, discovered that, rather than the real usable Scheduler/Job class (com.company.api_test.jobs.WhatsappSingleJob), there was one Whatsapp Scheduler/Job configured before whose class (com.company.api_test.jobs.WhatsappJob) we’re not taking into account.
  2. That Scheduler/Job class (com.company.api_test.jobs.WhatsappJob) was subsequently deleted from the code during the code cleanup and enhancement period (19 March 2024) because we aren’t utilizing it. Afterwards, on March 29, 2024, this patch was installed on the Varthana server.
  3. Despite the fact that the Scheduler/Job class was previously set up in Varthana’s system, it’s possible that the incorrect class was chosen; the desired configuration was for the usable Scheduler/Job class (com.company.api_test.jobs.WhatsappSingleJob).
  4. It was later discovered that some additional schedulers were not displayed on the portal or user interface since the class had been removed from the code level but the configuration was still there in Varthana’s database. Although it states that the Scheduler and Job were previously available and scheduled, the operation team may have set up them anew if they were not visible on the portal or user interface.

Example:
Attaching the recorded video ([all-jobs-not-displaying.webm]) , how presence of configuration in the db which is not available in the application code impacts displaying scheduler configurations on the portal/ui.
Steps in the recorded video

  1. Started the application, configured com.company.api_test.jobs.WhatsappJob class that is present to be configured at that time (named: com.company.api_test.jobs.WhatsappJob), total : 6 Schedulers/Jobs are displaying.
  2. Removed the com.company.api_test.jobs.WhatsappJob class as it was not in use
  3. Re-started the application, no ui error, logging exception in the system, however only 3 Scheduler/Job are displaying now
  4. Removed the unused configuration also from db, now total 5 Scheduler/Job are displaying as expected.


#7

Hi jmix Team

Waiting for your response
artefact_urls.txt (399 Bytes)

#8

Hi @yogesh.shinde
Thank you for reporting the problem.
We’ll investigate it and come back to you.

Regards,
Konstantin

#9

Hi Yogesh,

  1. Job info is not deleted by Jmix code from database automatically to avoid unexcpected data loss. For the moment only few system job configurations of some addons like webdav has such special functionality that is done via Quartz API and connected with their own configuration needs. Having records without job classes should not be an issue as jobs without classes can not be executed by Quartz.

  2. In older versions jobs without classes may not be shown in User Inferface. The best way to solve this is to return or recreate job class with the same FQDN and delete jobs from UI first. Deleting from Database is not recommended as there can be some connected records like in trigger tables. But in case you have no any other option you should note that unique Quartz job key is composed from job name(title) and job group fields.

  3. Newer versions of Jmix (starting from 2.3) will display jobs without classes as “Invalid” (see attached) and will allow to delete them from User Interface so consider upgrading;), you can check
    the details of this feature with the following task: Ability to remove Quartz jobs with invalid class name · Issue #1669 · jmix-framework/jmix · GitHub

quartz_deleted_job_class