Missing childField on association

t.vignal · May 23, 2023, 8:07am

Hi,
I have a JOINED inheritance between ParentEntity and ChildEntity. secondly I have a AnotherEntity wich have a association on ParentEntity.(see code below) On top of that I have a service and a repository associated with each entity.

MY PROBLEM IS:
when I fetch AnotherEntity through the repository, the parentEntityAssociation is fetched BUT not entirely. childField and all child specific fields are not fetched. The object is correctly identified as ChildEntity but only ParentEntity’s fields are filled.

am I doing it wrong ? I can’t find any annotation or whatever to make it execute the jointure on childEntity table.

the context is more or less the same as here: Data Modelling: Entity Inheritance

@JmixEntity
@Table(name = "ADDON_PARENT_ENTITY")
@Entity(name = "addon_parentEntity")
@Inheritance(strategy = InheritanceType.JOINED)
@DiscriminatorColumn(name = "FOO")
open class ParentEntity {
    @Column(name = "PARENT_FIELD")
    var parentField: String? = null
}

@JmixEntity
@Table(name = "ADDON_CHILD_ENTITY")
@Entity(name = "addon_childEntity")
@PrimaryKeyJoinColumn(name = "ID", referencedColumnName = "ID")
@DiscriminatorValue("L")
open class ChildEntity: ParentEntity {
    @Column(name = "CHILD_FIELD")
    var childField: String? = null
}

@JmixEntity
@Table(name = "ADDON_ANOTHER_ENTITY", indexes = [
    Index(name = "IDX_ADDON_ANOTHER_ENTITY_ADDON_PARENT_ENTITY", columnList = "ADDON_PARENT_ENTITY_ID")
])
@Entity(name = "ADDON_ANOTHER_ENTITY")
class AnotherEntity {

    @NotNull
    @OnDeleteInverse(DeletePolicy.CASCADE)
    @JoinColumn(name = "ADDON_PARENT_ENTITY_ID", nullable = false)
    @ManyToOne(fetch = FetchType.LAZY, optional = false)
    var parentEntityAssociation: ParentEntity? = null
}

b.vallettedosia · May 25, 2023, 5:41pm

I think Jmix / EclipseLink is only halfway here, the real concrete class is loaded with only the parent fields …

In InheritancePolicy.selectOneRowUsingDefaultMultipleTableSubclassRead()
a first query to determine concrete class is executed
then a second query on the concrete class but only with parent fields !!!

after that in ObjectLevelReadQuery.buildObject()
the concrete class is instancied but with the parent descriptor.

So I think Jmix have to handle this case, because have a not fully loaded object can be root of many problems.

The only alternative I found:
On each entity with a ParentEntity field, add a entity event on loading to reset correctly the field half loaded by the framework

b.vallettedosia · July 5, 2023, 9:27am

As I explained, a alternative solution exists to catch the bug

@EventListener
fun onParentEntityLoading(event: EntityLoadingEvent<ParentEntity>) {
    val t = event.entity.parentField
    if (t != null) {
        event.entity.parentField = dataManager.load(t::class.java).id(t.id).one()
    }
}

But with this code, each ParentEntity loading execute a query,
so in case of list of ParentEntity, there is the N+1 queries problem.

Is there a better solution ?

b.vallettedosia · July 24, 2023, 2:52pm

Nobody ?

b.vallettedosia · February 19, 2024, 9:56am

This problem is really annoying and the current fix is really bad (too much queries, save problems because reload…)

I created 2 projects to explain it (jmix 1 and jmix 2 same bug).

The datamodel is an abstract BaseThirdParty entity with two concret entity LegalPerson and NaturalPerson. There is also a Case which has a owner of type BaseThirdParty and a list of Inspection with an agent of type NaturalPerson.

I generated screens (Entity list and detail views) without modification and a second screen for Case and Inspection where I selected owner and agent.

When I navigate on the application, I can go on

BaseThirdParty browser and detail
LegalPerson browser and detail
NaturalPerson browser and detail
Inspection browser and detail
Inspection + agent browser and detail
Case browser and detail

but I get an error on

Case + owner browser and detail


java.lang.IllegalStateException: Cannot get unfetched attribute [firstname] from detached object com.company.sample2.entity.NaturalPerson-4af5c473-8a0e-259b-36df-2c93fd5bc871 [detached].
	at org.eclipse.persistence.internal.queries.EntityFetchGroup.onUnfetchedAttribute(EntityFetchGroup.java:100) ~[org.eclipse.persistence.core-4.0.1-15-jmix.jar:na]
	at io.jmix.eclipselink.impl.JmixEntityFetchGroup.onUnfetchedAttribute(JmixEntityFetchGroup.java:78) ~[jmix-eclipselink-2.1.3.jar:na]
	at org.eclipse.persistence.internal.jpa.EntityManagerImpl.processUnfetchedAttribute(EntityManagerImpl.java:2996) ~[org.eclipse.persistence.jpa-4.0.1-15-jmix.jar:na]
	at com.company.sample2.entity.BaseThirdParty._persistence_checkFetched(BaseThirdParty.kt) ~[main/:na]
	at com.company.sample2.entity.NaturalPerson._persistence_get_firstname(NaturalPerson.kt) ~[main/:na]
	at com.company.sample2.entity.NaturalPerson.getFirstname(NaturalPerson.kt:15) ~[main/:na]
	at io.jmix.core.entity.BaseEntityEntry.getAttributeValue(BaseEntityEntry.java:85) ~[jmix-core-2.1.3.jar:na]
	at io.jmix.core.entity.EntityValues.getValue(EntityValues.java:100) ~[jmix-core-2.1.3.jar:na]
	at io.jmix.core.impl.InstanceNameProviderImpl.getInstanceName(InstanceNameProviderImpl.java:147) ~[jmix-core-2.1.3.jar:na]
	at io.jmix.core.MetadataTools.getInstanceName(MetadataTools.java:213) ~[jmix-core-2.1.3.jar:na]
	at io.jmix.core.MetadataTools.format(MetadataTools.java:156) ~[jmix-core-2.1.3.jar:na]
	at io.jmix.flowui.data.provider.StringPresentationValueProvider.apply(StringPresentationValueProvider.java:40) ~[jmix-flowui-2.1.3.jar:na]
	at io.jmix.flowui.data.provider.StringPresentationValueProvider.apply(StringPresentationValueProvider.java:26) ~[jmix-flowui-2.1.3.jar:na]
	at com.vaadin.flow.component.grid.Grid.applyValueProvider(Grid.java:1785) ~[vaadin-grid-flow-24.1.12.jar:na]
	at com.vaadin.flow.component.grid.Grid.lambda$addColumn$63838bef$1(Grid.java:1773) ~[vaadin-grid-flow-24.1.12.jar:na]
	at com.vaadin.flow.component.grid.ColumnPathRenderer$SingleValueProviderRendering.lambda$getDataGenerator$96b60bbc$1(ColumnPathRenderer.java:78) ~[vaadin-grid-flow-24.1.12.jar:na]
	at com.vaadin.flow.data.provider.CompositeDataGenerator.lambda$generateData$0(CompositeDataGenerator.java:47) ~[flow-data-24.1.14.jar:24.1.14]
	at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]

I think the problem is in query.

2024-02-19T10:49:09.035+01:00 DEBUG 23076 --- [nio-8080-exec-5] eclipselink.logging.sql                  : <t 1135008119, conn 1735013723> 
SELECT LIMIT ? ? 
  t3.ID AS a1, t3.CREATED_BY AS a2, t3.CREATED_DATE AS a3, t3.DELETED_BY AS a4, t3.DELETED_DATE AS a5, t3.LAST_MODIFIED_BY AS a6, t3.LAST_MODIFIED_DATE AS a7, t3.REFERENCE AS a8, t3.VERSION AS a9, t3.OWNER_ID AS a10, 
  t0.ID AS a11, t0.DTYPE AS a12, t0.CREATED_BY AS a13, t0.CREATED_DATE AS a14, t0.DELETED_BY AS a15, t0.DELETED_DATE AS a16, t0.FULLNAME AS a17, t0.LAST_MODIFIED_BY AS a18, t0.LAST_MODIFIED_DATE AS a19, t0.VERSION AS a20 
FROM CASE_ t3 LEFT OUTER JOIN 
(
  BASETHIRDPARTY t0 LEFT OUTER JOIN LEGAL_PERSON t1 ON (t1.ID = t0.ID) 
  LEFT OUTER JOIN NATURAL_PERSON t2 ON (t2.ID = t0.ID)) ON (t0.ID = t3.OWNER_ID) 
  WHERE (t3.DELETED_DATE IS NULL
)
	bind => [0, 50]

I think that FROM is OK but the problem in the SELECT there is no field from LEGAL_PERSON t1 or NATURAL_PERSON t2 !

t.vignal · February 19, 2024, 1:45pm

@gaslov

krivopustov · March 4, 2024, 11:03am

Hi @t.vignal @b.vallettedosia

Thank you for the detailed description and the test projects.
We’ll investigate the problem and come back to you in the next few weeks.

Regards,
Konstantin

taimanov · March 12, 2024, 3:50pm

Hello @t.vignal @b.vallettedosia,

This is a known issue.
As a workaround, I can suggest next solution to get rid of N+1 and save problems:

load each subtype by a separate query with required fetch plan
merge loaded entities into the DataContext of the screen

E.g. for jmix-1 project it will look like:

@Component
class BaseThirdPartyEntityLoader {

   @Autowired
   private lateinit var dataManager: DataManager

   fun reloadCaseOwners(entities: List<Case>, fetchPlanName: String, dataContext: DataContext) {
       val naturalPersonIds: MutableList<UUID> = LinkedList()
       val legalPersonIds: MutableList<UUID> = LinkedList()
       val caseByOwner: MutableMap<UUID, Case> = HashMap()

       for (case in entities) {
           if (case.owner == null) continue

           if (case.owner is NaturalPerson) naturalPersonIds.add(case.owner!!.id!!)
           if (case.owner is LegalPerson) legalPersonIds.add(case.owner!!.id!!)

           caseByOwner.put(case.owner!!.id!!, case)
       }

       val naturalPersons: List<NaturalPerson> = dataManager.load(NaturalPerson::class.java)
               .ids((naturalPersonIds as Collection<*>?)!!)
               .fetchPlan(fetchPlanName)
               .list()

       val legalPersons: List<LegalPerson> = dataManager.load(LegalPerson::class.java)
               .ids((legalPersonIds as Collection<*>?)!!)
               .fetchPlan(fetchPlanName)
               .list()

       for (person in naturalPersons) caseByOwner[person.id!!]?.owner = person
       for (person in legalPersons) caseByOwner[person.id!!]?.owner = person

       dataContext.merge(naturalPersons)
       dataContext.merge(legalPersons)
   }
}

//...
class CaseBrowse3 : StandardLookup<Case>() {
    @Autowired
    private lateinit var dataManager: DataManager
    @Autowired
    private lateinit var baseThirdPartyEntityLoader: BaseThirdPartyEntityLoader


    @Subscribe(id = "casesDl", target = Target.DATA_LOADER)
    private fun onCasesDlPostLoad(event: CollectionLoader.PostLoadEvent<Case>) {
         baseThirdPartyEntityLoader.reloadCaseOwners(event.loadedEntities, FetchPlan.BASE, screenData.dataContext)
    }

}

<window xmlns="http://jmix.io/schema/ui/window"
        xmlns:c="http://jmix.io/schema/ui/jpql-condition"
        caption="msg://caseBrowse3.caption"
        focusComponent="casesTable">
    <data readOnly="true">
        <collection id="casesDc"
                    class="com.company.sample1.entity.Case">
            <fetchPlan extends="_base">
                <property name="owner" fetchPlan="_instance_name"/>
            </fetchPlan>
            <!-- ... -->
        </collection>
    </data>
    <!-- ... -->
</window>

Please, see the project with this fix:
jmix1_wa.zip (122.7 KB)

Regards,
Dmitry

t.vignal · March 14, 2024, 3:19pm

Hi, thank you for you answer !

I implemented the workaroud in my project, it works well in the screens. In the other hand the problem is still present when fetching data through the reportRunner. In fact, the ReportRunnerImpl.createReportDocumentInternal doesn’t use the InstanceLoaderImpl that I fixed (see below). So I want to override PrototypesLoader to introduce my fix after data loading, do you see any probable issue ?
The goal is to fix the issue everywhere it could occur.
Do you see any other methods that would need some personalization ? Is there another data loader class that I forgot to mention ?(InstanceLoaderImpl, CollectionLoaderImpl, PrototypesLoader)
Or maybe override UnconstrainedDataManagerImpl.loadList (and load) would be a more direct solution…

Best regards,
Thibaud

FYI:
Instead of implementing it in each screens manually I have overridden the InstanceLoaderImpl and DataComponents (I still need to do the CollectionLoaderImpl). I also added protection against reload loops. I plan to generify the reloading mecanism so it can fix automatically any property deeper in the data graph.

class InstanceLoaderFix<E>: InstanceLoaderImpl<E>() {
    @Autowired
    private lateinit var baseThirdPartyEntityLoader: BaseThirdPartyEntityLoader

    init {
        addPostLoadListener(this::doFix)
    }

    private fun doFix(event: InstanceLoader.PostLoadEvent<E>) {
        val entityToFix =
            when (event.loadedEntity) {
                is BaseInspectionForm -> (event.loadedEntity as BaseInspectionForm).parentFieldInspection
                is Inspection -> event.loadedEntity
                else -> null
            } as Inspection?
        if (entityToFix != null)
            baseThirdPartyEntityLoader.reloadInspectionRequestor(listOf(entityToFix), FetchPlan.BASE, dataContext)
    }
}

@Primary
@Component("ui_DataComponentsFix")
class DataComponentsFix: DataComponents(), Logging {

    override fun <E : Any?> createInstanceLoader(): InstanceLoader<E> =
        InstanceLoaderFix<E>()
            .also { autowire(it) }
}

taimanov · March 15, 2024, 5:47pm

Hello @t.vignal,

Instead of implementing it in each screens manually I have overridden the InstanceLoaderImpl and DataComponents

Yes, your implementation looks more appropriate for project-wide solution.

If you want to get rid of such problem everywhere it could occur it is much better to override UnconstrainedDataManagerImpl.loadList,load because even after overriding all beans it is still at least lazy loading wrappers which difficult to replace without overriding unnecessary code (e.g. io.jmix.eclipselink.impl.lazyloading.CollectionValuePropertyHolder).
They are also using UnconstrainedDataManagerImpl.loadList under the hood.

Unfortunately I cannot guarantee absence of any problems without implementing and mindful testing this solution, but also I am not seeing any problems in case of careful implementation.

It makes sense to wrap entities reloading in try-finally block with thread local variable to guarantee absence of any loops (like here).

Regards,
Dmitry

t.vignal · April 19, 2024, 9:27am

hi,
thanks to your advice I could solve my problem.I implemented a complete workaround for this issue by overloading DataManagerImpl.

class DataManagerFixed : DataManagerImpl() {
    @Autowired
    private lateinit var inheritanceBugfixService: InheritanceBugfixService

    override fun <E : Any?> load(context: LoadContext<E>): E? {
        if (context.fetchPlan != null) {
            context.fetchPlan =
                inheritanceBugfixService.replaceAbstractFetchPlanIfNeeded(context.fetchPlan!!,
                context.entityMetaClass.getJavaClass<Any>())
        }
        return super.load(context)
            .also { if (it is BaseEntity) inheritanceBugfixService.applyInheritanceFix(listOf(it), context.fetchPlan) }
    }
...

TheapplyInheritanceFix is the core of the workaround. It explore the data-graph by following the FetchPlan and recursively reloads the needed entities.

Entities that need to be reloaded implement an interface so I know if it needs a reload or not.

/**
 * Has to be implemented by all entity with @Inheritance
 */
interface ReloadableEntity {
    val id: UUID?
    var hasBeenReloaded: Boolean
}

If you are curious about the implementation, I can give you more infos about it.