MoReq2010 A Look Into The Specification

The first part of this blog series gave an introduction to the new debate about RM initiated by the release of the MoReq2010 specification. This part will be a bit more technical and give an overview about the spec, the ideas and how they were realized. A blog like this can only scratch the surface of such a complex spec. For a full understanding you will have to read the spec, perhaps not even all the 520 pages but some parts: Section 1.4 is an excellent introduction. Each service chapter starts with an overview about the purpose and scope followed by a more detailed list of requirements. The function definitions and data type definitions coming then are more reference material.

And the spec is really quite readable. There are lots of explanations; they start with explaining concepts, ideas and illustrating diagrams before going into the details. In this regard the committee has done an excellent job. One of the most discussed aspects of Moreq2010 is the attempt to achieve better modularity. How does this work and what is different in MoReq2010 compared to similar specifications?

The MoReq2010 core (and this is the only part available yet) specifies a set of services and methods each compliant repository must implement. Services are the foundation how different modules can exchange records or use functionality in other modules. This is the path to go away from having one monolithic application doing everything in a central repository. An example would be an "in-place RM" setup where the records system is outside of the business system. The business creates and stores the data and a separate system does the classification and disposal of those records.

To share data between systems MoReq2010 defines an XML schema and a mandatory requirement for a system to be able to export data according to this schema. The standard has detailed requirements which data have to be exported guaranteeing that all aspects of a record are preserved. For example this includes the history (events) and the user identification. The import capability is an optional feature. Keep in mind that for long retention periods like 75 years, multiple export/import cycles due to system changes are very likely.

Some services usually will exist only once in a deployment and are shared between the different modules. Others may occur in multiple incarnations. User management for example in many cases will be a shared service. Classification on the other side may be implemented in different schemas each one being a service.
The following sections give an overview about the services MoReq2010 defines including their key characteristics and functionality. For a complete description please refer to the specification.

1.    User-Group-Service:

Manages users and groups, often may be done within or with the help of the corporate directory, but enforcing certain restrictions:

• Users and groups must have a unique id.
• Unique ids are never destroyed even if the user/group no longer exists.
• Any time even after delete it must be reproducible what groups existed and which user was in which group (provide a report for any historical point in time).
• Users and groups have a defined set of mandatory metadata

2.    Model-Role-Service

Provides the means to manage access control in a repository. Roles contain a list of functions that can be executed and are assigned to user or groups in Access Control Lists (ACLs). ACLs are attached to records or can be inherited from a parent-child relationship (an aggregation classification). Implementation is optional, but if not available it must be proven that an equally powerful mechanism exists. Roles are divided in administrative and non-administrative roles that have different inheritance behavior. ACLs can include or exclude inherited roles. A System of Records must be able to tell what functions a specific user can use.

3.    Classification Service

Classifications are used to associate a business context to a record. They are used to determine the disposal schedule. Every record has a class and only one class. A repository may support different classification schemes and then each scheme will get its own service.  Classifications can be inherited. The informational model contains a more detailed description of a hierarchical classification scheme.

4.    Record Service

The Record Service manages aggregation of records (in many repositories known as folders). A specialty in MoReq2010 is that an aggregation either contains other aggregations or records but not both. Aggregations are used to manage inheritance of metadata, ACLs and classifications. The maximum number of nesting levels of aggregations can be limited and an aggregation can be closed (means cannot add more entities). Records can be duplicated in multiple aggregations and a record can have sub-components called parts (for example an HTML page plus pictures). A repository must support moving records from one aggregation to another.

5.    Model Metadata Service

The Model Metadata Service deals with the definition of metadata sets for records. Again implementation is optional but must be replaced with equivalent functionality if not implemented). MoReq2010 differentiates between system metadata (part of the standard and always mandatory) and context metadata (customer specific). Metadata definitions are grouped into templates and types. Each type has a unique type identifier and each entity in a system has exactly one type. Metadata definitions can have additional constraints like (optional, mandatory, or a maximum number of allowed values). They can also indicate if their values must be preserved when an entity is destroyed. Datatypes  indicate if they are textual and have a language identifier. The standard does not define specific data types (like Date, String, Integer, …) but refers to the XML specification instead.

6.    Disposal Scheduling  Service

As the name indicates, this service is responsible for managing dispositions of records. MoReq2010 distinguishes between deletion and destruction. A disposal process never erases a record completely, but instead destroys it. Destruction means that certain information is still preserved (for example that the record existed, its id, when it was destructed and why, etc.). This is called a residual object then. When retention periods expire, disposal runs are scheduled and these trigger disposal actions. An action must not necessarily be a destroy operation it also can be a transfer operation or some kind of approval workflow. Retention periods have a start date and an end date. Start dates are triggered by event. Such a trigger event exists in many flavors. It can be a metadata change, a change in the parent aggregation, a certain date and so on. Each record always has only one disposal schedule, but this may change over time. An aggregation is destroyed when the last active record it contains is destroyed. A repository must send alerts when disposal actions are not carried out.

7.    Disposal Holding Service

Holds are a well known concept in records management. A hold is a legal or administrative order to interrupt the disposal process in MoReq2010. A hold always must explicitly be removed ("lifted"). The Disposal Holding Service contains the methods to manage holds.

8.    Searching-Reporting Service

Not hard to guess to guess this service defines the methods for reporting and searching. Searching can be done by metadata or (optionally) as text search in the content. The standard defines a lot of criteria for search and an interesting option to search all metadata text fields simultaneously.

• search by entity type, events, any(!) system or context metadata
• find records by metadata values
• combinations to do complex searches

Searches must reflect access control and search results must be paginated and returned in an ordered list. The standard does not contain a syntax for a query language.
A repository must be able to generate reports (detailed or summary) but makes no assumption about the output format.

9.    Export Service

The export service is used to transfer data out of a system into an XML file (or a set of files). Reasons for exporting data can be transfer, migration or replication of data. The XML schema for the data format is not yet specified. Some use cases may prefer or require a partial export of records but this is not further specified and is optional. The service guarantees to preserve sufficient level of detail when moving data between systems (for example includes all events of a record and referred entities like users). Importing data is an optional feature. The service is not only used to transfer records, but also other entities like classes, users, roles, metadata definitions and templates.

Conclusion

MoReq2010 is definitely more than just another records management standard. The service based approach; the focus on modularity and the premise "keep it simple" are all new ways of specifying requirements for records management. But it is also obvious that it is not the silver bullet for everything. There are parts in the standard that will make assumptions about elements being deep in the core of a repository. If a specific vendor model does not fit into this approach it will be hard to implement conformance without breaking compatibility. Other features are not trivial to implement if they have to scale to millions and billions of records in really large systems.

There is also some caution needed to not make expectations that are unrealistic to achieve in the current state. MoReq2010 defines an abstract model with services and methods. It does not define any mechanism how to implement the services or how to access these methods. If this is a Java interface, a web service or something completely different is left open. This means there won't be a test script that you can run against any repository for validation. There won't be a way that you can simply plug different modules together and they can talk to each other. MoReq2010 is not like CMIS and I wonder if this is in scope for a future extension. It does not even define concrete list of input and output parameters for the methods. You won't get a generic client that can browse the pending disposal schedules, list the holds or do a federated query in multiple repositories.  This looks a bit like having stopped on half of their way. The XML schema is also not yet available meaning that you can't even begin an implementation at the moment. I fear that this will lead to a lot of frustration when people start looking closer at the standard. But of course this does not mean that there is something fundamentally wrong with it. Many things can be specified on top or added later, but you have to know what you will get at the moment. The committee did an ambitious step. The specification has triggered a new debate about Records Management. Do not underestimate what has happened in the first few weeks after release.
And there are other standards that can fill the gap partially. Have you ever looked at CMIS? Do some of the concepts and terms from above sound familiar to you? Types, folders, ACLs? Is it conflict or a perfect match? A non-relevant replication of things defined elsewhere? I think at least a topic interesting enough to think about in another article.

What else can we do? There is one thing coming into my mind that really would help getting adoption. RM systems are a usually a big investment. It would be helpful to have an implementation to play with. To test, to see how it behaves to check, if and how such a model can be integrated to your system. Ideally something that you can adapt and change according to your needs. Not for production use but for everything else. Something that fills the gaps of the standard, not as a replacement, but as something that turns it into a runnable piece of code, provides a test system and the necessary tools. What am I talking about? Yes, an open source implementation. Ideally it would not come from a vendor but from a neutral organization. Illusion? May be, but also not impossible. There is a community already in ECM, perhaps MoReq2010 will get some attraction…

Part I: MoReq2010 And The New RM Debate

Part III: CMIS and Moreq2010: Good Match Or Just Overlap? http://blogs.opentext.com/vca/blog/1.11.623/article/1.26.954/2011/7/26