Monthly Archives: December 2014

MDS Versions, Flags and Subscriptions

Leave a reply

Recently I collaborated with Andrew Milner in a scenario in MDS SQL 2012 to provide the client the ability to with between multiple Versions and Flags via Subscription Views. Vishal Singh did an excellent job in implementing this scenario in MDS.

Versions and Flags

MDS Model defines the structure of the Data and Version defines the content of the data. In Master Data Services, multiple versions of the master data can be created within a model. Versions created in model can be in sequence fashion (one version after another) or in simultaneous fashion (multiple versions in parallel). This way we ensure we have a dedicated/consistent version of our master data for downstream consumption while at the same time having another version for adding/deleting/modifying members or for testing our model data, without compromising consistency in master data availability.

For example: We may want to have one version of the data marked as “current”, and another set marked as “proposed” using Version Flags. When the proposed version is confirmed as valid and clean, we can move that from proposed to current. So the version marked as “current” will always be the “Golden Copy of the Data”

When to Use Versions

Use versions to:

  • Prevent users from making changes while all data validates successfully against business rules.
  • Lock down a model for use by subscribing systems.

Version Flags

When a version is ready for users or for a subscribing system, we can set a flag to identify the version. We can move flag from version to version as required. Flags help user and downstream applications identify which version of a model to use.

Scenario:

Create Model and 3 Versions (one in Commit Status and other two in Open Status).

An initial version is created by default when a model is created. In order to create two other versions in Open status, commit default version and create two versions from the committed version. Name two versions as PROD and TEST and map it to two different Flags PROD and TEST respectively. Data inserted in two different versions are mutually exclusive meaning they are not related to each other. Data modified in one of the version will not be reflected in another version. Subscription views created based on Version will also have data related to that Version only. Flags help user and downstream applications identify which version of a model to use.

For example: Created a sample Model called “Employee” and created three versions VERSION_1 (committed status), MDS (open status) and SSIS (open status). In the same way created two flags MDS and SSIS mapped to MDS and SSIS versions respectively. Data present in different versions (MDS and SSIS) are not related and thus not in synch.

In Employee entity there is one employee which works in two different technologies MDS and SSIS as highlighted below. Employee (Employee ID: IW003) works in both SSIS and MDS technology but there is a difference in their name in different versions.

mds1

mds2

Records highlighted above are common in MDS and SSIS versions of Employee Entity with only difference in their Name (EmployeeID: IW003) . Subscription view created on the MDS (linked to MDS flag) and SSIS (linked to SSIS flag) versions will have different set of data. Please find below the data present in subscription-view:

mds2sv

Downstream application will fetch data based on the subscription views which they are consuming. If downstream application is consuming MDS version (mapped to MDS flag) will have different data and one consuming SSIS version (mapped to SSIS flag) will have different of data.

Architecture in MDS Acme

We have used below logic in MDS Acme:

mds3arch

In MDS Acme we have two different data sources Production and Test. Data coming from different data sources are getting inserted in the same model (Sample) and same entities. Two versions V1 and V2 are created pointing to Production and Test data sources. Versions V1 and V2 are mapped to Flags F1 and F2 respectively. Subscription views created are linked to Version flags. Downstream applications are consuming data based on the subscription view published. When a new Version needs to be created called V3 which is copy of version V2 below steps need to be done:

  1. Validate and commit old version V2 as in this example.
  2. Copy the old version and create a new version called V3 from V2
  3. Remove the flag F1 from the old version V2
  4. Map flag F1 to newly created version V3

As the subscription views are created based on the Version Flags and not on Version. So we don’t have to change the subscription views for the newly created version. Downstream application will now be able to fetch data from the new versions instead of old versions.

Organic Data Quality vs Machine Data Quality

Leave a reply

In reading a few recent blog post on centralized data quality vs. distributed data quality have provoked me to offer another point of view.

Organic distributed data quality provides todays awareness  for all current source analysis and problem resolution and they are accomplished manually by individuals in most cases.

In many cases when  management or leadership(Folks worried about their jobs) are presented with any type of organized machine based data quality results that can easily be viewed, understood and is permanently available, the usual result is to kill or discredit the messenger.

Automated “Data Quality Scoring” (Profiling + Metadata Mart) brings them from a general “awareness”  to a realized state “awarement”.

Awarement  is the established form of awareness. Once one has accomplished their sense of awareness they have come to terms with awarement.

It’s one thing to know that alcohol can get you drunk, it quite another to be aware that you are drunk when you are drunk.

metadata scales

Perception = Perception

Awarement = Reality

Departmentral  DQ = Awareness

vs

Centralized DQ = Awarement

Share the love… of Data Quality–  A distributed approach to Data Quality may result in better data outcomes

Reference Data Management Implementation: Four Key Considerations You May Be Overlooking

Leave a reply

Great article in discussing Reference Data Management or RDM and it’s relation to MDM. Also important to remember that this is an excellent starting point in organically evolving the underlying processes in support of MDM and a Data Governance organization.

InfoTrellis blog

In recent years reference data management (RDM) has slowly crept into the forefront of business decision-makers’ consciousnesses, making its way steadily upwards in priority within corporate goals and initiatives. Organizations are suddenly seeing the benefits of investing in RDM, attention grabbed by potential paybacks like smoother interoperability among various functions of the organization and centralized ownership and accountability in creating trustworthy data.

Before we dive into talking about approaches for implementation, I want to look at the potential significance of RDM in an enterprise. When the market is inclined towards concepts like Data Integration, MDM and Business Intelligence tend to hog the spotlight. For these particular corporate initiatives, the primary focus of data is key business information like customers, products or suppliers. It is equally important, however, to appreciate the fact that reference data plays a major role in organizing and comprehending all these key pieces of business data.

Whenever…

View original post 846 more words

Informatica Cloud MDM for Salesforce (formerly Data Scout) Review

Leave a reply

 

Tactically improving Data Quality and incrementally achieving Data Governance and metadata management  is a natural path and MDM  is the center of that strategy. See Gartner Group’s Applying Data Mart and Data Warehousing Concepts to Metadata Management

 

pyramid

In Metadata Mart the Road to Data Governance or Guerilla Data Governance I outline this approach

I’ve just completed a Data Governance Assessment and review of Informatica CLOUD MDM(formerly Data Scout for Salesforce) with my colleague and excellent Solution Architect Baliji Kkarade .  The client in this case is interested in implementing Informatica CLOUD MDM in Salesforce , as a tactical approach to improving Data Quality and incrementally improving Data Governance . I’d like to aknowledge the incredible insight I gained from Balaji Kharade in this effort.

In general and product is positioned to provide a transactional MDM within Salesforce. We will cover the steps for implementation and some back ground on Fuzzy Matching or de-duplication.

We will walk thru the steps for setting up the tool.

  1. Cloud MDM Settings
  2. Cloud MDM Profile
  3. Adding Cloud related Information to Page Layout
  4. Synchronization Settings
  5. Data Cleansing
  6. Fuzzy Matching and Segments
  7. External Data Sources
  8. Consolidation and Enrichment
  9. Limitations

 

This post assumes familiarity with the Saleforce architecture.

 

1. Cloud MDM Settings

  • Cloud MDM master on/off switch is configured using this setting and other settings like extracting the legal form and domain , overriding Salesforce Account information using Master bean after Match and Merge in Cloud MDM, and Standardizing Country.
  • In some cases, you may wish to turn off Cloud MDM after you have installed and configured it.
  • For example, if you wish to bring in a new set of data without creating beans. To achieve this, you need to switch Cloud MDM off.

mdmsettings

2. Cloud MDM Profile:

  • When Cloud MDM is installed, a default profile is given to all users. In order for your user to get access to all the features of Cloud MDM, you must configure an admin or super user profile. When you implement Cloud MDM, you can use profiles to assign MDM functionality to Salesforce user profiles.
  • Users can have Permissions to Create/Update/Merge/consolidate Account, Contact and Leads, Create/Ignore duplicate Account, Contact and Leads, View consolidated information and create/Edit Hierarchy information

mdmsettings2

3. Add Cloud Related Information to Page Layout:

Helps to add MDM related components like Consolidated view, Find duplicates, MDM related fields like Synchronize, Legal forms, ISO country ,duplicate Account section, Related beans and Master Beans etc .

mdmpageinfos

mdmpageinfos2

4. Synchronization Settings:

  • This setting helps in Synchronizing/Mapping the Salesforce Attributes to Cloud MDM stage Area.
  • We can map Standard fields and 10 custom fields. These standard and custom fields help us in configuring segment settings and match strategy in cloud MDM.
  • Sync job helps creating beans and Master beans in cloud MDM stage Area.

mdmaccountsync

5. Data Cleansing:

Data cleansing ensures the data is in a consistent format. Consistent data improves the quality of reporting and also improves matching results and the accuracy of duplicate detection.

Legal Form :

Legal form normalization is the process of extracting the legal form from the company norm and populating the legal form field with normalized data.

For example, We can configure the legal form field to contain normalized data for business entity designations such as Limited, Ltd., and L.T.D. We can add legal forms to the list available already after profoiling our data set.

Domain Normalization :

We can enable Cloud MDM to populate the domain field with a domain extracted from the website field. Cloud MDM uses the domain field during fuzzy matching.

For example, if a user enters http://www.acme.com/products or www. acme.com in the website field, Cloud MDM can populate the domain field with acme.com. normalized domain ensures domain field consistency and improves match results.

mdmdsatacleansing

6. Fuzzy Matching and Segments:

Segment :

The segment field in the master bean record contains a matching segment. The matching segment is a string of characters that Cloud MDM uses to filter records before it performs fuzzy matching.

To improve fuzzy match performance, Cloud MDM performs an exact match on the matching segments to eliminate records that are unlikely to match. Cloud MDM then performs fuzzy matching on the remaining records. This can be basically creating Categories and Groups in Advanced Fuzzy Matching or “Blocking Indexes” Record Linkage. This will be created for all the Accounts once the Sync between Salesforce and MDM is done. It is also generated for external beans.

Fuzzy matching :

Fuzzy matching can match strings that are not exactly the same but have similar characteristics and similar patterns.

One example of a Fuzzy Matching algorithm is LevenShtein, the original Fuzzy algorithm Levenshtein  Distance or Edit Distinceinvented in 1965

Levenshtein:

Counts the number of incorrect characters, insertions and deletions.

Returns:

(maxLen – mistakes) / maxLen

Levenshtein is a good algorithm for catching keyboarding errors

mdmFuzzy1

Matching is a two step process that determines a match score between two records. First, Cloud MDM performs an exact match on the matching segments to exclude records that are unlikely to have matches. Then, Cloud MDM performs a fuzzy match on the remaining records to calculate a match score between pairs of records. If the match score of the two records achieves the match score threshold, Cloud MDM considers the two records a match.

mdmsegment settingsmdmpageinfos2 mdmpageinfos

7. External Data Source:

We have external data in a system, such as SAP or Oracle EBS. We wish to load this data directly into beans, so we can take some of the information (SIC Code, No of Employees.) from the SAP record, and retain some information (e.g. Company Name) from the Salesforce record.

This setting allows us to configure the external Data source and defining the trust/priority score i.e. which value will win over the other during Consolidation and Enrichment process.

 

7ext

8. Consolidation and Enrichment.

Consolidation :

The consolidated view allows us to look at all beans associated with a master bean. In order to use this view, we must configure the fields that will display in the list of associated beans, as well as the account address information. This is done by configuring field sets.

Enrichment:

This setting allows us to over write the value from the Master bean to the Salesforce Org Account based on the trust/priority score provided during the configuration of the external Data source. We can use the override account option in cloud MDM settings to prevent the automatic override of the Salesforce Org Account accordingly.

8ent

 

9. Limitations.

 

There are two primary limitations:

 

  1. Custom Fields are limited to 10 and only 6 can be used in syncing.
  2. High volume matching from External Source is completed in a “Pre Match” process, which is basically accessing the “Master Bean” externally and developing ETL Process with another tool.

 

Metadata Mart the Road to Data Governance

2 Replies

 or Guerilla Data Governance

Implementing a Metadata Mart the Road to Data Governance best viewed in PRESENTATION Mode, there is animation.

iradg

This presentation has narrative, play in presentation mode with sound on.

https://onedrive.live.com/redir?resid=E7059817CA5789D0!10028&authkey=!AE6eWl-gzh2mq_M&ithint=file%2cpptx

Self Service Semantic BI (Business Intelligence) Concept

Leave a reply

metadatamarttripleWe want to build an Enterprise Analytical capability by integrating the concepts for building a Metadata Mart with the facilities for the Semantic Web

 

    • Metadata Mart Source
      • (Metadata Mart as is) Source Profiling(Column, Domain & Relationship)
    • +
      • (Metadata Mart Plus Vocabulary(Metadata Vocabulary)) Stored as Triples(subject-predicate-object) (SSIS Text Mining)
    • +
      • (Metadata Mart Plus)Create Metadata Vocabulary following RDFa applied to Metadata Mart Triple(SSIS Text Mining+ Fuzzy (SPARGL maybe))
    • +
    • Bridge to RDFa – JSON-LD via Schema.org
      • Master data Vocabulary with lineage (Metadata Vocabulary + Master Vocabulary) mapped to  MetaContent Statements)) based on person.schema.org
      • Creates link to legacy data in data warehouse
      • +RDFa applied to web pages
      • +JSON-LD applied to
      • + any Triples from any source
  • Semantic Self Service BI
    • Metadata Mart Source + Bridge to RDFa

 

 

 

 

I have spent some time in this for quite a while now and I believe there is a quite a bit of merit in approaching the collection of domain data and column profile data, in regards to the meta-data mart, and organize them in a triple’s fashion

 

The basis for JSON-LD and  RDFa is the collection of data as a triple. Delving into said deeper

 

I believe with the proper mapping for the object reference and deriving of the appropriate predicates in the collection of the value we could gain some of the same benefits as well as bringing the web data being “collected, there by linking to source data.

Consider the following excerpt regarding Vocabularies derived from MetaContent via Metadata Structure

“Metadata structures[edit]

Metadata (metacontent), or more correctly, the vocabularies used to assemble metadata (metacontent) statements, are typically structured according to a standardized concept using a well-defined metadata scheme, including: metadata standards and metadata models. Tools such as controlled vocabularies, taxonomies, thesauri, data dictionaries, and metadata registries can be used to apply further standardization to the metadata. Structural metadata commonality is also of paramount importance in data model development and in database design.

Metadata syntax[edit]

Metadata (metacontent) syntax refers to the rules created to structure the fields or elements of metadata (metacontent).[11] A single metadata scheme may be expressed in a number of different markup or programming languages, each of which requires a different syntax. For example, Dublin Core may be expressed in plain text, HTML, XML, and RDF.[12]

A common example of (guide) metacontent is the bibliographic classification, the subject, the Dewey Decimal class number. There is always an implied statement in any “classification” of some object. To classify an object as, for example, Dewey class number 514 (Topology) (i.e. books having the number 514 on their spine) the implied statement is: “<book><subject heading><514>. This is a subject-predicate-object triple, or more importantly, a class-attribute-value triple. The first two elements of the triple (class, attribute) are pieces of some structural metadata having a defined semantic. The third element is a value, preferably from some controlled vocabulary, some reference (master) data. The combination of the metadata and master data elements results in a statement which is a metacontent statement i.e. “metacontent = metadata + master data”. All these elements can be thought of as “vocabulary”. Both metadata and master data are vocabularies which can be assembled into metacontent statements. “

 

The MetadataMart serve as the source for both metadata vocabulary and MDM for the Master Data Vocabulary.

For the Master Data Vocabulary consider schema.org which defines most of the schemas we need. Consider the following schema.org Persons Properties of Objects and Predicates:

 

Thing > Person

A person (alive, dead, undead, or fictional).

Property Expected Type Description
Properties from Person
additionalName Text An additional name for a Person, can be used for a middle name.
address PostalAddress Physical address of the item.

The key is to link source data in the Enterprise via a Business Vocabulary from MDM to the Source Data Metadata Vocabulary from a Metadata Mart to conform the triples collected internally and externally.

 

In essence information from the web applications can be integrated with the dimensional metadata mart, MDM Model and existing Data Warehouses providing lineage for selected raw data from web to Enterprise conformed Dimensions that have gone thru Data Quality processes.

 

Please let me know your thoughts.