If you think BIRT Data Objects (BDOs) are just for ad hoc queries, we have news for you: BDOs are much more than simply another semantic metadata layer. Unlike conventional metadata, which is used by many business intelligence and analysis tools to provide data access for non-technicians, BIRT Data Objects have capabilities that address many metadata shortcomings and enhance the performance of data-driven applications.
What is Semantic Metadata, and How Does It Help?
Data in most large companies lives in many systems, is diverse and complex, and requires highly skilled technical staff to acquire and make sense of it. To overcome this complexity, IT specialists create a semantic metadata layer to bridge the gap between the typical business user and the data. Semantic metadata abstracts things like SQL queries and business logic and transforms obscure database columns names to simple, easy-to-understand terms.
Semantic metadata is an attempt to present data to the user in the friendliest possible way for autonomous report authoring. If effectively set up and maintained, a metadata layer can reduce or eliminate the need for IT intervention in the case of ad hoc queries and reports created by business users. With this promise, it’s easy to see why the metadata concept is ubiquitous in the current crop of business analysis tools.
What’s Wrong with Semantic metadata?
But a semantic metadata layer for reporting is not the holy grail of end-user empowerment that it purports to be. There are drawbacks to the concept, especially with those tools that use an Online Analytical Processing (OLAP) or “data cube” approach:
- The semantic metadata layer can be inflexible. When changes are made to the underlying data, remodeling and re-acquiring the source data is often required to inherit updated information in the database.
- Data cubes take forever to model, and can take just as long to generate, resulting in giant multi-purpose cubes that provide dubious performance for end users.
- Data cubes typically pre-aggregate data on defined hierarchies (also known as drill paths). Pre-aggregated hierarchies box in the user to the drill paths that the cube modeler has defined, stifling true data exploration. Users should be able to roll up or down the data in any path they choose. And the step of pre-aggregating adds loads of time and overhead to generating the cube, as each and every calculation must be performed and the result loaded to the cube in the proper sequence.
- Many tools require the semantic metadata layer before any reporting or visualizations can be created, which can actually limit productivity as users wait for modeling tasks to be completed in order to do simple things like see last quarter’s sales figures.
Drawbacks aside, the concept of a simplified data access layer, and its benefits to the technical and business worker in an organization, are obvious, and in fact should be applied to applications as a whole, not just to empower the occasional ad hoc report.
A Better Option: BIRT Data Objects
BIRT, and specifically BIRT Data Objects (BDOs), help realize the promise of the semantic metadata layer by getting around these challenges and creating a data access mechanism strong enough to power an entire application. BIRT Data Objects offer unique advantages to application developers, providing a combination of features for accessing, federating (joining) and delivering data from multiple data sources to millions of users efficiently.
Prior to BIRT Data Objects, no single technology could meet the combination of requirements for real-time operational access to source data, data federation, and modeling of the data for delivery of analytic information into dashboards and visualizations. With BIRT Data Objects every requirement is met, and is all developed natively in the BIRT Designer environment.
A unified data layer provides tremendous productivity benefits for developers who no longer need to map business requirements to separate technologies and efforts, such as disposable “just for the ad hoc user” applications. In addition to ad hoc report authoring in BIRT Report Studio, BDOs provide efficient easy access to data for dashboards, interactive reports and crosstabs, maps, custom visualizations, scorecards, spreadsheets, you name it. This ain’t your daddy’s semantic metadata layer!
BDOs include:
- Performance optimizations for real-time direct data source access
- Push-down on joins, filters, sorts, etc. for faster and more efficient processing
- Query trimming on tables and columns. This is important if you join multiple databases, because you only connect to the database that has the data you need right now
- Re-synchronizing of semantic metadata to changes in the underlying database(s)
- Grouping, organizing, and custom ordering of fields
- Columnar backend, in-memory processing, enabling super-fast performance for analytic operations and calculations
- Dozens of direct connectors to data sources of all kinds
- Ability to cache for efficiency
- Are designed in Actuate BIRT Designer Pro, the same integrated environment where all other BIRT application components are built
This last point is important; it’s the secret sauce to a BDO’s ability to power an entire application. BIRT developers assemble components in a built-in sandbox, leveraging BDOs for data access, and crafting the various aspects of the user experience. Built-in UX aspects include setting behavior like how dashboard charts drill to reports, how maps drill to dashboards, how certain data fields are visible to only certain users, and stuff like that. All of this is done in one designer: BIRT.
A completed app is published as a whole to BIRT iHub, where it’s instantly available on the web to real users. The portability of an application, complete with data thanks to BDOs, enables sharing and collaboration. I know because my own team uses these tools to share and collaborate on demo BIRT apps, then efficiently deploy them to the field sales engineers with one click.
Better Processing, Better Performance
How do BDOs enable better processing of Big Data? The iHub BDO data architecture implements a dictionary that enables smaller representation of large-size values, greatly reducing the overall volume of the original data. Further, data is compressed using the Run Length Encoding (RLE) technique. The compression ratio achieved through RLE is heavily dependent on the degree to which data values are repeated, but generally results in an extremely high compression ratio.
Additionally, BIRT iHub delivers significant performance gains through a number of techniques that maximize the amount of data stored in memory. The data remains compressed in memory, and queries are executed directly on compressed data. Only the necessary columns are loaded for each query, and columns can be reused across queries. Also, query components (such as calculations) can be reused in different queries. The queries are executed directly on compressed data, further increasing the volume of data that can be retained in memory. The data is materialized much faster with BDOs than with more traditional semantic metadata cubes. Only the columns that are used are materialized, and the data representation is column based.
Some (But Not All) BDO Benefits:
- Powers more than ad hoc queries. BDOs can be a cornerstone of an entire application deployed on the BIRT iHub
- Makes joining disparate data easy. You can include nosql, sql, csv, POJO, excel, etc, in the same BDO. Try doing that, easily at least, in SQL
- Nested BDOs can mix and match cached data with real time data as business need require
- Columnar, in-memory database has quick response times against massive amounts of data
- No need to pre-define and pre-aggregate cubes like an OLAP tool, resulting in faster analysis and drill in any direction
- Provides incremental updates. No need to refresh the entire data model just to capture new data
- Did I mention you can power entire applications with a BDO? It’s not just for ad hoc reporting!
BDO-backed, data-driven applications can range from personalized portals to apps powering wearable devices like Nike Fuelband. Designing compelling data-driven experiences – especially for customer-facing apps that have both a large number of users and large data sets – remains an art and a science. Check out this free whitepaper on the “5 Best Practices for Designing Data-Driven Apps” and put your BDOs to work.
(Editor: All mentions of metadata have been clarified to read as “semantic” metadata)
