Updating data through a job using unconstrained data manager sometimes failing

Support
#1

Some data becomes updated in an in-memory datastore by retrieving data by a rest/remote API:

  • once on application start up listening an ApplicationReadyEvent
  • periodically in a job

Both hooks are using the same method in a springboot component which uses the UnconstrainedDataManager.

The datastore update triggered by the application event always works fine. The job triggered update process works on some days, on others it complains about unset authentication:

Success (e.g. yesterday):

2023-05-22 05:17:00.203  INFO [k-app,292d1b33b34e7699,292d1b33b34e7699] 867 --- [quartzScheduler_Worker-10] c.c.app.kapp.job.UpdateRemoteDataJob  : Fetched remote data. Updating transient store...
2023-05-22 05:17:00.847  INFO [k-app,292d1b33b34e7699,292d1b33b34e7699] 867 --- [quartzScheduler_Worker-10] c.c.app.kapp.job.UpdateRemoteDataJob  : Updating transient store successfully.

Failure (e.g. today):

2023-05-23 05:17:00.181  INFO [k-app,fe220a45bdbe5d80,fe220a45bdbe5d80] 867 --- [quartzScheduler_Worker-2] c.c.app.kapp.job.UpdateRemoteDataJob  : Fetched remote data. Updating transient store...
2023-05-23 05:17:00.243 ERROR [k-app,fe220a45bdbe5d80,fe220a45bdbe5d80] 867 --- [quartzScheduler_Worker-2] c.c.addon.ctl.service.RemoteDataService   : Could not update datastore. Authentication is not set. Use SystemAuthenticator in non-user requests like schedulers or asynchronous calls.

java.lang.IllegalStateException: Authentication is not set. Use SystemAuthenticator in non-user requests like schedulers or asynchronous calls.
        at io.jmix.core.security.impl.CurrentAuthenticationImpl.getAuthentication(CurrentAuthenticationImpl.java:47) ~[jmix-core-1.4.4.jar!/:na]
        at io.jmix.security.impl.constraint.AuthenticationPolicyStore.extractFromAuthenticationByScope(AuthenticationPolicyStore.java:123) ~[jmix-security-1.4.4.jar!/:na]
        at io.jmix.security.impl.constraint.AuthenticationPolicyStore.getEntityResourcePolicies(AuthenticationPolicyStore.java:69) ~[jmix-security-1.4.4.jar!/:na]
        at io.jmix.security.impl.constraint.SecureOperationsImpl.isEntityOperationPermitted(SecureOperationsImpl.java:57) ~[jmix-security-1.4.4.jar!/:na]
        at io.jmix.security.impl.constraint.SecureOperationsImpl.isEntityCreatePermitted(SecureOperationsImpl.java:36) ~[jmix-security-1.4.4.jar!/:na]
        at io.jmix.securityui.constraint.UiEntityConstraint.applyTo(UiEntityConstraint.java:52) ~[jmix-security-ui-1.4.4.jar!/:na]
        at io.jmix.securityui.constraint.UiEntityConstraint.applyTo(UiEntityConstraint.java:28) ~[jmix-security-ui-1.4.4.jar!/:na]
        at io.jmix.core.AccessManager.lambda$applyConstraints$2(AccessManager.java:77) ~[jmix-core-1.4.4.jar!/:na]
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[na:na]
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[na:na]
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) ~[na:na]
        at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[na:na]
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[na:na]
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[na:na]
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:na]
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[na:na]
        at io.jmix.core.AccessManager.applyConstraints(AccessManager.java:76) ~[jmix-core-1.4.4.jar!/:na]
        at io.jmix.core.AccessManager.applyRegisteredConstraints(AccessManager.java:86) ~[jmix-core-1.4.4.jar!/:na]
        at io.jmix.ui.action.list.ViewAction.isPermitted(ViewAction.java:257) ~[jmix-ui-1.4.4.jar!/:na]
        at io.jmix.ui.action.BaseAction.refreshState(BaseAction.java:148) ~[jmix-ui-1.4.4.jar!/:na]
        at io.jmix.ui.component.impl.AbstractTable.refreshActionsState(AbstractTable.java:1315) ~[jmix-ui-1.4.4.jar!/:na]
        at io.jmix.ui.component.impl.AbstractTable.tableSourcePropertyValueChanged(AbstractTable.java:1756) ~[jmix-ui-1.4.4.jar!/:na]
        at io.jmix.ui.component.table.TableDataContainer.datasourceValueChanged(TableDataContainer.java:301) ~[jmix-ui-1.4.4.jar!/:na]
        at io.jmix.core.common.event.EventHub.publish(EventHub.java:170) ~[jmix-core-1.4.4.jar!/:na]
        at io.jmix.ui.component.data.table.ContainerTableItems.containerItemPropertyChanged(ContainerTableItems.java:79) ~[jmix-ui-1.4.4.jar!/:na]
        at io.jmix.core.common.event.EventHub.publish(EventHub.java:170) ~[jmix-core-1.4.4.jar!/:na]
        at io.jmix.ui.model.impl.InstanceContainerImpl.itemPropertyChanged(InstanceContainerImpl.java:182) ~[jmix-ui-1.4.4.jar!/:na]
        at io.jmix.ui.model.impl.CollectionContainerImpl.itemPropertyChanged(CollectionContainerImpl.java:231) ~[jmix-ui-1.4.4.jar!/:na]
        at io.jmix.core.entity.BaseEntityEntry.firePropertyChanged(BaseEntityEntry.java:198) ~[jmix-core-1.4.4.jar!/:na]
        at io.jmix.core.impl.EntityInternals.fireListeners(EntityInternals.java:90) ~[jmix-core-1.4.4.jar!/:na]
...
        at c.c.addon.ctl.service.RemoteDataService.updateDataStore(RemoteDataService.java:800) ~[ctl-1.4.5.jar!/:1.4.5]
        at com.company.app.kapp.job.UpdateRemoteDataJob.execute(UpdateRemoteDataJob.java:50) ~[classes!/:2.4.2]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz-2.3.2.jar!/:na]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) ~[quartz-2.3.2.jar!/:na]

2023-05-23 05:17:00.243 ERROR [kapp,fe220a45bdbe5d80,fe220a45bdbe5d80] 867 --- [quartzScheduler_Worker-2] c.c.app.kapp.job.UpdateRemoteDataJob  : Updating transient store failed
  • Do I need to set system authentication explicitly even when using an unconstrained data manager?
  • Why does it work on one day but does not on another?
  • I wonder, why user interaction related code is involved in a scheduler triggered thread.
#2

Looks like your in-memory data store contains entity instances linked to UI components.
It shouldn’t happen if you are using DataContext and its commitDelegate (or saveDelegate in Flow UI).
Could you explain how do you save data to your store?

#3

@krivopustov

The datastore model and the datastore controller are implemented in a seperate add-on each. Both are not depending on UI code artifacts (a.f.a.i.k.):

The current behavior occurs in Jmix 1.4.4.

Jmix and Springboot dependencies:

  • addon-mdl:
    implementation(group: "org.springframework.boot", name: "spring-boot-autoconfigure")
    implementation(group: "io.jmix.core", name: "jmix-core")
    implementation(group: "io.jmix.data", name: "jmix-eclipselink")
    implementation(group: "io.jmix.translations", name: "jmix-translations-de")
  • addon-mdl-starter:
    api(project(":addon-mdl"))

    implementation(group: "org.springframework.boot", name: "spring-boot-autoconfigure")
    implementation(group: "io.jmix.core", name: "jmix-core-starter")
    implementation(group: "io.jmix.data", name: "jmix-eclipselink-starter")
  • addon-ctl:
implementation(project(":addon-mdl"))

    annotationProcessor(group: "org.springframework.boot", name: "spring-boot-configuration-processor", version: "2.7.10")

    implementation(group: "io.jmix.core", name: "jmix-core")
    implementation(group: "io.jmix.data", name: "jmix-eclipselink")
    implementation(group: "io.jmix.translations", name: "jmix-translations-de")
  • addon-ctl-starter:
    implementation(group: "org.springframework.boot", name: "spring-boot-autoconfigure")
    implementation(group: "io.jmix.core", name: "jmix-core-starter")
    implementation(group: "io.jmix.data", name: "jmix-eclipselink-starter")

    testImplementation(project(":addon-mdl"))

RemoteJobService excerpt / pseudo code:

/***/
        final SaveContext saveContext = new SaveContext();
        saveContext.setDiscardSaved(true);

        /** methods, loops **/
            OrgUnitMember entity = unconstrainedDataManager.create(OrgUnitMember.class);
            entity.setAvatar(new URL(...));
            // entity.setters

            saveContext.saving(entity)


        unconstrainedDataManager.save(saveContext);
/***/

The job execution does not fail on each execution. Sometimes it works, sometimes it does not.

If I understood you right, it fails if an opened screen shows a UI component with such entities. Yes, the OrgUnitMember has a URL property containing an avatar image which is displayed in screens. So the job fails, if any screen shows such a component, right?

My goal is to update those entities in a background task/job - not in any UI context.

The avatar use case (image UI component) does not need to be updated right then.

#5

I assume you are talking about something like this, right?

Then the question is how do you save entities to this data store?
If you use a commitDelegate/saveDelegate of DataContext, the saved instances are cleaned from listeners added by UI components. But the error you get shows that some instances in the store have references to UI components, that is they weren’t properly cleaned. It also explains the fact that the error is seemingly random - it’s produced only by instances saved from the UI.

#6

@krivopustov I’m sorry for responding late.

I assume you are talking about something like this , right?

Exactly, the mentioned entity is a DTO implemented in a in-memory (non-JPA) custom data store which is fed by a REST API.

These DTO entities are saved by the mentioned job only. All application screens perform read-only operations on this DTO.

The avatar image is displayed by an image component. But the DTO entity is not defined the screen descriptor data section, but programmatically set instead. To determine the changed URL the DTA is looked up simply using dataManager.load(OrgUnitMember.class)....

That’s probably not the designed way to embed that image in a screen, is it?

Well, I easily could add system authentication to the job but I wonder why it is necessary when committing by using the unconstrained data manager.

#7

I was wrong that links to UI components can remain only in entity instances saved to the data store. In fact, any instance loaded from the store and shown on a screen will get UI listeners.

In order to isolate instances in the in-memory data store from the UI, you should clone them when returning from load() methods:

@Nullable
@Override
public Object load(LoadContext context) {
    Map<Object, Object> instances = entities.get(context.getEntityMetaClass().getJavaClass());
    return instances == null ? null : cloneInstance(instances.get(context.getId()));
}

@Override
public List<Object> loadList(LoadContext context) {
    Map<Object, Object> instances = entities.get(context.getEntityMetaClass().getJavaClass());
    return instances == null ? Collections.emptyList() : cloneInstances(instances.values());
}

@Nullable
private Object cloneInstance(@Nullable Object instance) {
    return instance == null ? null : metadataTools.deepCopy(instance);
}

private List<Object> cloneInstances(Collection<Object> instances) {
    return instances.stream().map(this::cloneInstance).collect(Collectors.toList());
}

You can clone instances using MetadataTools.deepCopy() as shown above or by re-serializing with StandardSerialization as in DataContextImpl, see jmix/DataContextImpl.java at ec96da00ed4cc1ea9ec0897086353ad9963a8b16 · jmix-framework/jmix · GitHub

1 Like