{"id":24738045,"url":"https://github.com/datanucleus/datanucleus-core","last_synced_at":"2025-05-15T20:03:22.033Z","repository":{"id":12402441,"uuid":"15055689","full_name":"datanucleus/datanucleus-core","owner":"datanucleus","description":"DataNucleus core persistence support - the basis for anything in DataNucleus","archived":false,"fork":false,"pushed_at":"2025-04-25T07:08:09.000Z","size":12061,"stargazers_count":123,"open_issues_count":39,"forks_count":75,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-25T08:23:56.260Z","etag":null,"topics":["jdo","jpa","persistence"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datanucleus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-12-09T18:40:29.000Z","updated_at":"2025-04-25T07:08:12.000Z","dependencies_parsed_at":"2023-02-12T02:18:20.066Z","dependency_job_id":"b31fb031-c440-43cf-9d40-514f7f5ad4d0","html_url":"https://github.com/datanucleus/datanucleus-core","commit_stats":{"total_commits":2097,"total_committers":23,"mean_commits":91.17391304347827,"dds":0.05865522174535054,"last_synced_commit":"725824a05db208316e86b7e58e4555ce54395052"},"previous_names":[],"tags_count":103,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datanucleus%2Fdatanucleus-core","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datanucleus%2Fdatanucleus-core/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datanucleus%2Fdatanucleus-core/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datanucleus%2Fdatanucleus-core/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datanucleus","download_url":"https://codeload.github.com/datanucleus/datanucleus-core/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254414493,"owners_count":22067271,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["jdo","jpa","persistence"],"created_at":"2025-01-27T22:34:59.188Z","updated_at":"2025-05-15T20:03:19.612Z","avatar_url":"https://github.com/datanucleus.png","language":"Java","funding_links":[],"categories":["数据库开发"],"sub_categories":[],"readme":"# datanucleus-core\n\nDataNucleus core persistence support - the basis for anything in DataNucleus.\n\nThis is built using Maven, by executing `mvn clean install` which installs the built jar in your local Maven repository.\n\n## KeyFacts\n\n__License__ : Apache 2 licensed  \n__Issue Tracker__ : http://github.com/datanucleus/datanucleus-core/issues  \n__Javadocs__ : [6.0](http://www.datanucleus.org/javadocs/core/6.0/), [5.2](http://www.datanucleus.org/javadocs/core/5.2/), [5.1](http://www.datanucleus.org/javadocs/core/5.1/), [5.0](http://www.datanucleus.org/javadocs/core/5.0/), [4.1](http://www.datanucleus.org/javadocs/core/4.1/), [4.0](http://www.datanucleus.org/javadocs/core/4.0/)  \n__Download__ : [Maven Central](https://repo1.maven.org/maven2/org/datanucleus/datanucleus-core)  \n__Dependencies__ : See file [pom.xml](pom.xml)  \n__Support__ : [DataNucleus Support Page](http://www.datanucleus.org/support.html)  \n\n----  \n\n## Persistence Process\nThe primary classes involved in the persistence process are\n\n* *PersistenceNucleusContext* - maps across to a PMF/EMF, and provides access to the StoreManager and ExecutionContext(s) (PersistenceNucleusContextImpl). \n* *ExecutionContext* - maps across to a PM/EM, and handles the transaction (ExecutionContextImpl)  \n* *StateManager* - manages access to a persistent object (StateManagerImpl)  \n* *StoreManager* - manages access to the datastore (see the datastore plugins, e.g RDBMSStoreManager)  \n* *MetaDataManager* - manages the metadata for the class(es), so how it is persisted  \n\n\n### Persistence : Retrieve of Objects\n\n    MyClass myObj = (MyClass)pm.getObjectById(id);\n    myObj.getSomeSet().add(newVal);\n\n* calls wrapper (see _org.datanucleus.store.types.wrappers.XXX_ or _org.datanucleus.store.types.wrappers.backed.XXX_)\n* if optimistic txns then queues up til flush/commit\n* otherwise will call backing store for the wrapper (RDBMS) which updates the DB, or will mark the field as dirty (non-RDBMS) and the field is sent to the datastore at the next convenient place.\n\n\n    Query q = pm.newQuery(\"SELECT FROM \" + MyClass.class.getName());\n    List\u003cMyClass\u003e results = (List\u003cMyClass\u003e)q.execute();\n\n* Makes use of QueryManager to create an internal Query object (wrapped by a JDO/JPA Query object). This may be something like org.datanucleus.store.rdbms.query.JDOQLQuery specific to \nthe datastore.\n* The query is compiled generically. This involves converting each component of the query (filter, ordering, grouping, result etc) into Node trees, and then converting that into Expression trees. \nThis is then stored in a QueryCompilation, and can be cached.\n* The query is then converted into a datastore-specific compilation. In the case of RDBMS this will be an RDBMSCompilation, and will be an SQL string (and associated parameter/result lookups).\n* The query is executed in the datastore and/or in-memory. The in-memory evaluator is in datanucleus-core under org.datanucleus.query.evaluator.memory. \nThe execution process will return a QueryResult (which is a List).\n* Operations on the QueryResult such as \"iterator()\" will result in lazy loading of results from the underlying ResultSet (in the case of RDBMS)\n\n\n\n### Persistence : Pessimistic Transactions\n\nAll persist, remove, field update calls go to the datastore straight away. \nFlush() doesn't have the same significance here as it does for optimistic, except in that it will queue \"update\" requests until there are more than say 3 objects waiting.\nThis means that multiple setters can be called on a single object and we get one UPDATE statement.\n\n\n#### persist\nCalls ExecutionContext.persistObject which calls EC.persistObjectWork.  \nCreates a StateManager (StateManagerImpl - SM). Adds the object to EC.dirtySMs.  \nCalls SM.makePersistent which calls SM.internalMakePersistent which will pass the persist through to the datastore plugin.  \nCalls PersistenceHandler.insertObject, which will do any necessary cascade persist (coming back through EC.persistObjectInternal, EC.indirectDirtySMs).  \n\n\n#### remove\nCalls ExecutionContext.deleteObject, which calls ExecutionContext.deleteObjectWork.  \nThis will add the object to EC.dirtySMs.  \nCalls SM.deletePersistent.  \nCalls SM.internalDeletePersistent which will pass the delete through to the datastore plugin.  \nCalls PersistenceHandler.deleteObject, which will do any necessary cascade delete (coming back through EC.deleteObjectInternal, EC.indirectDirtySMs).  \n\n\n#### update field\nCalls SM.setXXXField which calls SM.updateField and, in turn, EC.makeDirty.  \nThe update is then queued internally until EC.flushInternal is triggered (e.g 3 changes waiting).  \n\n\n#### Collection.add\nCalls SCO wrapper.add which will add the element locally.  \nIf a backing store is present (RDBMS) then passes it through to the backingStore.add().  \n\n\n#### Collection.remove/clear\nCalls SCO wrapper.remove/clear which will add the element locally.  \nIf a backing store is present (RDBMS) then passes it through to the backingStore.remove()/clear().  \nIf no backing store is present and cascade delete is true then does the cascade delete, via EC.deleteObjectInternal.  \n\n\n### Persistence : Optimistic Transactions\n\nAll persist, remove, field update calls are queued.\nFlush() processes all remove/add/updates that have been queued.\nCall ExecutionContext.getOperationQueue() to see the operations that are queued up waiting to flush.\n\n\n#### persist\nCalls ExecutionContext.persistObject which calls EC.persistObjectWork.  \nCreates a StateManager (StateManagerImpl - SM). Adds the object to EC.dirtySMs.  \nCalls SM.makePersistent. Uses PersistFieldManager to process all reachable objects.  \n\n\n#### remove\nCalls ExecutionContext.deleteObject, which calls ExecutionContext.deleteObjectWork.  \nCreates a StateManager as required. Adds the object to EC.dirtySMs.  \nCalls SM.deletePersistent. Uses DeleteFieldManager to process all reachable objects.\n\n\n#### update field\nCalls SM.setXXXField which calls SM.updateField and, in turn, EC.makeDirty.  \nThe update is then queued internally until EC.flushInternal is triggered.  \n\n\n#### Collection.add\nCalls SCO wrapper.add which will add the element locally.  \nAdds a queued operation to the queue for addition of this element.  \n\n\n#### Collection.remove/clear\nCalls SCO wrapper.remove/clear which will add the element locally.  \nAdds a queued operation to the queue for removal of this element.  \n\n----  \n\n### Flush Process\n\nWhen a set of mutating operations are required to be flushed (e.g transaction commit) the _FlushProcess_ for the *StoreManager*\nis executed. At the start of the flush process we have a set of primary objects that were directly modified by the user and passed in to calls,\nas well as a set of secondary objects that were connected to primary objects by relationships, and were also modified. A \"modification\" could mean\ninsert, update, delete. \n\nAn RDBMS uses a _org.datanucleus.flush.FlushOrdered_\n[[Javadoc]](http://www.datanucleus.org/javadocs/core/latest/org/datanucleus/flush/FlushOrdered.html).\nOther datastores typically use a _org.datanucleus.flush.FlushNonReferential_\n[[Javadoc]](http://www.datanucleus.org/javadocs/core/latest/org/datanucleus/flush/FlushNonReferential.html).\n\n----  \n\n### MetaData Process\n\nThe _MetaDataManager_ is responsible for loading and providing access to the metadata for all persistable classes. \nMetaData can come from Java annotations, XML metadata files, or via the JDO MetaData API.\n\nEach class is represented via a _org.datanucleus.metadata.ClassMetaData_\n[[Javadoc]](http://www.datanucleus.org/javadocs/core/latest/org/datanucleus/metadata/ClassMetaData.html).\nThis in turn has a Collection of \n_org.datanucleus.metadata.FieldMetaData_\n[[Javadoc]](http://www.datanucleus.org/javadocs/core/latest/org/datanucleus/metadata/FieldMetaData.html) andy/or\n_org.datanucleus.metadata.PropertyMetaData_\n[[Javadoc]](http://www.datanucleus.org/javadocs/core/latest/org/datanucleus/metadata/PropertyMetaData.html)\ndepending whether the metadata is specified on a field or on a getter/setter method.\nFields/properties are numbered alphabetically, with the _absolute_ field number starting at the root class in an inheritance tree\nand the _relative_ field number starting in the current class.\n\n----\n\n### Query Process\n\nDataNucleus provides a _generic_ query processing engine. It provides for compilation of __string-based query languages__. \nAdditionally it allows _in-memory evaluation_ of these queries. This is very useful when providing support for new datastores which either\ndon't have a native query language and so the only alternative is for DataNucleus to evaluate the queries, or where it will take some time \nto map the compiled query to the equivalent query in the native language of the datastore.\n\n#### Query : Input Processing\n\nWhen a user invokes a query, using the JDO/JPA APIs, they are providing either\n\n* A single-string query made up of keywords and clauses\n* A query object that has the clauses specified directly\n\nThe first step is to convert these two forms into the constituent clauses. It is assumed that a string-based query is of the form\n\n\tSELECT {resultClause} FROM {fromClause} WHERE {filterClause}\n\tGROUP BY {groupingClause} HAVING {havingClause}\n\tORDER BY {orderClause}]]\u003e\u003c/source\u003e\n\nThe two primary supported query languages have helper classes to provide this migration from the _single-string query form_ into the individual clauses. \nThese can be found in _org.datanucleus.store.query.JDOQLSingleStringParser_\n[[Javadoc]](http://www.datanucleus.org/javadocs/core/latest/org/datanucleus/store/query/JDOQLSingleStringParser.html)\nand _org.datanucleus.store.query.JPQLSingleStringParser_\n[[Javadoc]](http://www.datanucleus.org/javadocs/core/latest/org/datanucleus/store/query/JPQLSingleStringParser.html).\n\n#### Query : Compilation\n\nSo we have a series of clauses and we want to compile them. So what does this mean? Well, in simple terms, we are going to convert the individual clauses \nfrom above into expression tree(s) so that they can be evaluated. The end result of a compilation is a _org.datanucleus.store.query.compiler.QueryCompilation_\n[[Javadoc]](http://www.datanucleus.org/javadocs/core/latest/org/datanucleus/store/query/compiler/QueryCompilation.html).\n\nSo if you think about a typical query you may have\n\n\tSELECT field1, field2 FROM MyClass\n\nThis has 2 result expressions - field1, and field2 (where they are each a \"PrimaryExpression\" meaning a representation of a field).\nThe query compilation of a particular clauses has 2 stages\n\n1. Compilation into a Node tree, with operations between the nodes\n2. Compilation of the Node tree into an Expression tree of supported expressions\n\nand compilation is performed by a JavaQueryCompiler, so look at _org.datanucleus.store.query.compiler.JDOQLCompiler_\n[[Javadoc]](http://www.datanucleus.org/javadocs/core/latest/org/datanucleus/store/query/compiler/JDOQLCompiler.html)\nand _org.datanucleus.store.query.compiler.JPQLCompiler_\n[[Javadoc]](http://www.datanucleus.org/javadocs/core/latest/org/datanucleus/store/query/compiler/JPQLCompiler.html).\nThese each have a Parser that performs the extraction of the different components of the clauses and generation of the Node tree. \nOnce a Node tree is generated it can then be converted into the compiled Expression tree; this is handled inside the JavaQueryCompiler.\n\nThe other part of a query compilation is the _org.datanucleus.store.query.compiler.SymbolTable_\n[[Javadoc]](http://www.datanucleus.org/javadocs/core/latest/org/datanucleus/store/query/compiler/SymbolTable.html)\nwhich is a lookup table (map) of identifiers and their value. So, for example, an input parameter will have a name, so has an entry in \nthe table, and its value is stored there. This is then used during evaluation.\n\n#### Query : Evaluation In-datastore\n\nIntuitively it is more efficient to evaluate a query within the datastore since it means that fewer actual result objects need \ninstantiating in order to determine the result objects. To evaluate a compiled query in the datastore there needs to be a compiler \nfor taking the generic expression compilation and converting it into a native query. Additionally it should be noted that you aren't \nforced to evaluate the whole of the query in the datastore, maybe just the filter clause. This would be done where the datastore \nnative language maybe only provides a limited amount of query capabilities. For example with db4o we evaluated the _filter_ and \n_ordering_ in the datastore, using their SODA query language. The remaining clauses can be evaluated on the resultant objects \n_in-memory_ (see below). Obviously for a datastore like RDBMS it should be possible to evaluate the whole query in-datastore.\n\n#### Query : Evaluation In-memory\n\nEvaluation of queries in-memory assumes that we have a series of \"candidate\" objects. These are either user-input to the query itself, \nor retrieved from the datastore. We then use the in-memory evaluator _org.datanucleus.store.query.inmemory.InMemoryExpressionEvaluator_\n[[Javadoc]](http://www.datanucleus.org/javadocs/core/latest/org/datanucleus/store/query/inmemory/InMemoryExpressionEvaluator.html).\nThis takes in each candidate object one-by-one and evaluates whichever of the query clauses are desired to be evaluated. \nFor example we could just evaluate the filter clause. Evaluation makes use of the values of the fields of the candidate objects \n(and related objects) and uses the SymbolTable for values of parameters etc. Where a candidate fails a particular clause \nin the filter then it is excluded from the results.\n\n#### Query : Results\n\nThere are two primary ways to return results to the user.\n\n* Instantiate all into memory and return a (java.util.)List. This is the simplest, but obviously can impact on memory footprint.\n* Return a wrapper to a List, and intercept calls so that you can load objects as they are accessed. This is more complex, \nbut has the advantage of not imposing a large footprint on the application.\n\nTo make use of the second route, consider extending the class _org.datanucleus.store.query.AbstractQueryResult_ and implement the key methods.\nAlso, for the iterator, you can extend _org.datanucleus.store.query.AbstractQueryResultIterator_.\n\n----  \n\n\n### Types : Second-Class Objects\n\nWhen a persistable class is persisted and has a field of a (mutable) second-class type (Collection, Map, Date, etc) then DataNucleus needs to know when the\nuser calls operations on it to change the contents of the object. To do this, at the first reference to the field once enlisted in a transaction, DataNucleus \nwill replace the field value with a _proxy wrapper_ wrapping the real object. This has no effect for the user in that the field is still castable to \nthe same type as they had in that field, but all operations are intercepted.\n\n\n### Types : Container fields and caching of Values\n\nBy default when a container field is replaced by a second-class object (SCO) wrapper it will be enabled to cache the values in that field. This means that once \nthe values are loaded in that field there will be no need to make any call to the datastore unless changing the container. This gives significant speed-up \nwhen compared to relaying all calls via the datastore. You can change to \u003cb\u003enot\u003c/b\u003e use caching by setting either\n\n* Globally for the PersistenceManagerFactory - this is controlled by setting the persistence property \n__org.datanucleus.cache.collections__. Set it to false to pass through to the datastore.\n* For the specific Collection/Map - add a MetaData \u0026lt;collection\u0026gt; or \u0026lt;map\u0026gt; extension \n_cache_ setting it to false to pass through to the datastore.\n\nThis is implemented in a typical SCO proxy wrapper by using the SCOUtils method _useContainerCache()_ which determines if caching is required, and by having a \nmethod _load()_ on all proxy wrapper container classes.\n\n\n#### Types : Container fields and Lazy Loading\n\nJDO and JPA provide mechanisms for specifying whether fields are loaded lazily (when required) or whether they are loaded eagerly (when the object is first met). \nDataNucleus follows these specifications but also allows the user to override the lazy loading for a SCO container. For example if a collection field was marked \nas being part of the default fetch group it should be loaded eagerly which means that when the owning object is instantiated the collection is loaded up too. \nIf the user overrides the lazy loading for that field in that situation to make it lazy, DataNucleus will instantiate the owning object and instantiate the \ncollection but leave it marked as \"to be loaded\" and the elements will be loaded up when needed. You can change the lazy loading setting via\n\n* Globally for the PMF/EMF - this is controlled by setting the persistence property __org.datanucleus.cache.collections.lazy__. \nSet it to true to use lazy loading, and set it to false to load the elements when the collection/map is initialised.\n* For the specific Collection/Map - add a MetaData \u0026lt;collection\u0026gt; or \u0026lt;map\u0026gt; extension __cache-lazy-loading__. \nSet it to true to use lazy loading, and false to load once at initialisation.\n\n\n#### Types : SCO fields and Queuing operations\n\nWhen DataNucleus is using an optimistic transaction it attempts to delay all datastore operations until _commit_ is called on the transaction or _flush_ is \ncalled on the PersistenceManager/EntityManager. This implies a change to operation of SCO proxy wrappers in that they must __queue__ up all mutating operations \n(add, clear, remove etc) until such a time as they need to be sent to the datastore. The ExecutionContext has the queue for this purpose.\n\nAll code for the queued operations are stored under _org.datanucleus.flush_.\n\n\n#### Types : Simple SCO interceptors\n\nThere are actually two sets of SCO wrappers in DataNucleus. The first set provide lazy loading, queueing, etc and have a \"backing store\" where the operations\ncan be fed through to the datastore as they are made (for RDBMS). The second set are simple wrappers that intercept operations and mark the field as dirty in \nthe StateManager. This second set are for use with all (non-RDBMS) datastores that don't utilise backing stores and just want to know when the field is dirty\nand hence should be written.\n\nAll code for the backed SCO wrappers are stored under _org.datanucleus.store.types.wrappers.backed_.\nAll code for the simple SCO wrappers are stored under _org.datanucleus.store.types.wrappers_.\n\n----  \n\n### Schema\n\n#### MultiTenancy\n\nThe handling for multi-tenancy code is present in each of the store plugins but is controlled from \n\n* __org.datanucleus.metadata.AbstractClassMetaData.getMultitenancyMetaData__ : returns details of any multi-tenancy discriminator null if the class does not need it).\n* __org.datanucleus.ExecutionContext.getTenantId__ : returns the tenant id to use for multi-tenancy (for any write operations, and optionally any read operations)\n* __org.datanucleus.PersistenceNucleusContext.getTenantReadIds__ : returns the tenant ids to use in any read operations (overriding the tenant id above when specified).\n\nThe metadata of the class defines whether it has a tenancy discriminator (i.e you have to explicitly add the metadata to get a discriminator). \n\n\n### CDI Integration\n\nDataNucleus allows use of CDI injected resources into attribute converter classes (JDO and JPA) as well as JPA lifecycle listeners.\nThe basis for this is the specification of the persistence property *datanucleus.cdi.bean.manager*. \nIf this is set then when creating an instance of the objected with injected resources, we call \n\n    CDIHandler.createObjectWithInjectedDependencies(cls);\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatanucleus%2Fdatanucleus-core","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatanucleus%2Fdatanucleus-core","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatanucleus%2Fdatanucleus-core/lists"}