Tag Archives: MDM

The $450 Billion Paradox: 5 Impactful Truths About the Agentic AI Revolution

Leave a reply

The enterprise technology landscape is currently defined by a staggering strategic chasm. On one hand, Capgemini estimates that Agentic AI could generate $450 billion in economic value over the next three years. On the other, Gartner forecasts that 40% of these projects will be canceled by 2027. This is not merely a contradiction; it is a high-stakes gamble on the future of work.

We are moving beyond the era of “query-based assistants”—Generative AI that merely synthesizes information—to a world of “autonomous systems” that proactively execute multi-step processes. Gartner further projects that by 2028, 15% of day-to-day work decisions will be made autonomously by these agents. For the C-suite, the challenge is no longer adoption, but avoiding the trap of building a sophisticated workforce of agents on a foundation of crumbling business logic.

1. Why 40% of Projects are Headed for the Scrapyard

The high failure rate predicted for Agentic AI is not a failure of the technology itself, but a failure of operational redesign. Many organizations are making the fatal error of layering autonomous agents onto broken manual processes, expecting the AI to “fix” the underlying chaos.

“Over 40% of agentic AI projects will be canceled by the end of 2027… Rising costs, unclear business value, and inadequate risk controls are the culprits.” — Gartner

Strategic failure typically occurs when leadership fails to separate execution from accountability. Agents can execute, but the accountability framework must be redesigned to handle autonomous actions. Furthermore, we are seeing a massive wave of “agent-washing,” where vendors relabel basic API integrations or rigid chatbots as “agentic” to capture market hype. True Agentic AI requires the capacity to reason, plan, and adapt—capabilities that demand a fundamental overhaul of how work is orchestrated, not just a new software layer.

2. From “Answering” to “Doing”—The Dawn of the Action-Oriented Workforce

The fundamental shift in this revolution is the move from passive information retrieval to active task execution. While standard GenAI is limited to content generation, Agentic AI functions as a “decision engine” that selects and calls tools, uses memory, and executes multi-turn plans to achieve outcomes end-to-end.

Siemens captures this architectural distinction precisely:

“We are moving from query-based assistants that respond to user requests, to autonomous agents that proactively execute processes under the coordination of an orchestrator.”

Comparison: Passive GenAI vs. Active Agentic AI

  • GenAI (Passive): Retrieves a knowledge base article explaining the steps for a user to perform a password reset.
  • Agentic AI (Active): Authenticates the user via MFA, accesses the Identity Access Management (IAM) system, resets the credentials, and closes the support ticket autonomously.

3. The “Agentic Advantage” Across 8 Key Industries

Approximately 70% of current deployments are concentrated in high-coordination industries where work moves across disparate systems and departments.

  • Banking & Wealth Management: Unlike traditional automation that follows “if-then” logic, agents use probabilistic reasoning to handle fraud investigations. They build case narratives and recommend dispositions, adapting as new transaction data surfaces. This is under intense scrutiny: UK banking regulators are actively monitoring the “speed of autonomy” to prevent cascading errors from destabilizing financial systems.
  • Insurance: In claims triage, agents move beyond rigid templates to analyze photos and forms, calculating settlements for low-complexity claims. The advantage over traditional automation is the ability to handle multi-step adaptation—if a document is missing, the agent doesn’t simply “fail”; it proactively contacts the claimant to retrieve it.
  • Retail & eCommerce: Agents manage “Post-Purchase Orchestration,” autonomously offering remediation like expedited shipping or refunds based on real-time logistics delays.
  • Manufacturing: Systems diagnose machine issues from sensor data and propose corrective maintenance windows to minimize shopfloor disruptions.
  • Healthcare: Agents automate prior authorization by validating requests against clinical guidelines and assembling documentation packets, reducing administrative cycles from days to minutes.
  • Logistics & Supply Chain: Agents monitor for exceptions, such as customs holds, and autonomously retrieve and submit missing documentation to keep goods moving.
  • Legal & Professional Services: Automation of client intake and matter management, including preliminary conflict checks and engagement letter drafting.
  • Energy & Utilities: Agents coordinate outage responses by correlating telemetry with network topology and proposing crew dispatch options based on skill and proximity.

4. The Identity Pivot: Managing “Non-Deterministic” Digital Employees

As agents gain the autonomy to modify records and initiate transactions, they must be governed as Non-Human Identities (NHIs), not simple service accounts. The core risk is Non-Deterministic Behavior: because agents are probabilistic, they can chain tool invocations in ways developers never anticipated.

This introduces a shift from “Output Risk” (incorrect text) to “Action Risk” (unauthorized transactions or data deletions). To mitigate this, organizations must adopt:

  • Least Privilege by Default: Ensuring agents inherit only the specific permissions necessary for a task, often mirroring the user they assist to prevent privilege escalation.
  • Just-in-Time (JIT) Access: Granting permissions only for the duration of a specific execution, eliminating “standing” privileges that could be exploited.
  • Identity as the Control Plane: Treating agents as first-class identities allows for complete audit trails of reasoning, tool calls, and actions—making “autonomous” no longer mean “unaccountable.”

5. Governance Must Become as Autonomous as the Agents It Controls

Static, rule-based governance is failing to keep pace with distributed data. Governance must transition to an “adaptive,” always-on system that monitors metadata in real-time to detect anomalies and enforce policies as data flows.

“More than 25% of organizations estimate they lose over $5 million annually because of poor data quality.” — Forrester

To protect the business, organizations must implement a Human-in-the-Loop (HITL) framework. For high-stakes decisions—such as large financial transfers, medical approvals, or deleting production data—the agentic system must pause for a human reviewer. This ensures that while the agent handles the coordination and “toil,” the human maintains authority over the intent and final consequence.

Conclusion: The Future is an “Agentic Mesh”

The end state for the modern enterprise is the Agentic Mesh—a coordination fabric that acts as the organization’s “nervous system.” As enterprises deploy dozens of disparate agents, the Mesh prevents “agentic chaos” where different systems optimize for conflicting KPIs (e.g., one agent cutting costs while another inadvertently damages customer satisfaction).

The competitive edge will not go to those who simply install new software, but to those who redesign their business logic to support this hybrid workforce. As you evaluate your current AI roadmap, you must ask one provocative question:

“Is your organization building a coordinated workforce of agents, or just a new, more expensive layer of technical debt?”

The 2026 Pivot: Why the Age of AI Autonomy Just Killed Traditional Data Governance

Leave a reply

1. The Hook: The Death of the Experimental Era

For years, enterprise AI has lived in a protected sandbox. It was the era of the “pilot,” a time defined by low-stakes experimentation and “innovation at any cost.” But as we enter 2026, that era is officially dead. The transition to autonomous, agent-driven systems has hit a hard ceiling: the realization that innovation without control is a structural liability.

The “data chaos” that once served as mere operational friction has mutated into a fundamental threat to the business. Organizations are discovering that the velocity of their AI is capped by the integrity of their data foundations. We have shifted from a post-GDPR world of reactive compliance to a high-stakes environment where Accountability is the only currency that matters.

This transformation is driven by a convergence of maturing technologies and a heavy-handed regulatory reality. Enterprises are no longer asking if they canbuild it; they are asking if they can prove its origin, quality, and safety. In 2026, the competitive edge belongs to those who stopped chasing “more data” and started building a governed foundation for the age of autonomy.

2. Governance is No Longer a Burden—It’s the Engine

August 2026 marks the first major enforcement cycle of the EU AI Act, and the shockwaves are being felt globally. Under Article 10, high-risk AI systems must meet rigorous quality criteria for training, validation, and testing datasets. Governance has evolved from a “reactive defense” tax into a “proactive competitive edge.”

A crucial strategic shift within Article 10 is the newly “legalized” use of sensitive data for the sake of fairness. Paragraph 5 allows providers to process special categories of personal data strictly for bias detection and correction, provided they meet stringent safeguards. This marks a pivot toward using governance as a tool for engineering social and technical trust.

To manage this, enterprises are establishing AI Governance Officers and adopting frameworks like ISO/IEC 42001 and the NIST AI RMF. These roles oversee model inventories and risk assessments, ensuring that intelligence is not just powerful, but sustainable and audit-ready.

“True intelligence must be portable, open, and sovereign—because your ability to move, scale, and adapt is what determines your competitive edge.” — Brett Sheppard

3. The Unstructured Data Goldmine: From Messy Files to Vector Reality

While 90% of enterprise data is unstructured—think images, video, and billions of PDFs—less than 1% was utilized for GenAI just two years ago. In 2026, the goldmine is finally open. The key has been the rise of Unstructured Data Integration (UDI) and Unstructured Data Governance (UDG).

This isn’t just about file storage; it’s about making legacy documents “agent-ready.” UDI pipelines now automate text chunking, embedding generation, and vectorization, allowing messy inputs to be ingested directly into vector databases. This enables Retrieval-Augmented Generation (RAG) at a scale that was previously impossible.

By unlocking these assets, companies are powering a new wave of Agentic AI capable of real-time risk detection and sophisticated document analysis. The goal is no longer just “search”—it is the conversion of raw organizational knowledge into actionable intelligence.

4. The Great Rapprochement: The Hybrid “Meshy Fabric”

The architectural civil war between Data Fabric and Data Mesh has ended in a hybrid marriage. Organizations that fell into the “velocity trap”—focusing on decentralization (Mesh) without automated infrastructure (Fabric)—found themselves buried in inconsistency. The most successful 2026 enterprises use a Data Fabric to automate intelligence while using a Data Mesh to enforce domain-led ownership.

Architectural Pivot

Data Fabric (Automation Layer)

Data Mesh (People/Process)

Strategic Driver

Unifying distributed systems via active metadata.

Managing data as a product with domain accountability.

Implementation

Technology-centric; automated integration.

Organizational-centric; domain-owned governance.

Key Enabler

Augmented data catalogs and AI-driven mapping.

Self-serve platforms and federated standards.

This “meshy fabric” ensures that the Data Fabricprovides the intelligent connective tissue, while the Data Mesh ensures the human domain experts are accountable for the quality of the data products being fed into AI agents.

5. Synthetic Data: The “Privacy-First” Training Hack

The “Privacy Paradox”—the friction between the need for massive datasets and the legal mandates of the GDPR—has been bypassed via Privacy Enhancing Technology (PET). Synthetic data, which mirrors the statistical patterns of real-world datasets without copying individual identities, has moved into the mainstream.

Beyond privacy, synthetic data is now a primary tool for bias mitigation. It allows developers to fill “data gaps” and create “edge cases” that real-world datasets often ignore. In sectors like healthcare and finance, this mimics the statistical properties required for high-utility models without the risk of re-identification or regulatory exposure.

“Synthetic data can be defined as data that has been generated from real data and that has the same statistical properties as the real data.” — Dr. Khaled El Emam

6. “Agent-Ready” Data and the Science of Model Provenance

As AI evolves toward Agentic AI—systems that act autonomously in procurement or IT operations—the demand for Accountability has reached a fever pitch. For an agent to execute a contract, it must have “agent-ready” data: information that is traceable, high-quality, and context-rich.

Simultaneously, the industry is moving from heuristic fingerprinting to mathematical proof. Using the Model Provenance Set (MPS), a sequential test-and-exclusion procedure, organizations can now achieve a provable asymptotic guarantee of a model’s lineage.

This isn’t just a tool; it’s a statistical proof. It allows enterprises to detect unauthorized reuse and protect intellectual property by identifying related models in complex derivation chains. In 2026, you don’t just “verify” a model; you prove its provenance.

7. Sovereignty is the New Architecture

Cloud strategy has shifted from a matter of IT efficiency to a compliance and risk management obligation. Driven by the EU Data Act, organizations are pivoting toward Sovereign Multicloud Architectures. This isn’t just about local hosting; it’s about the legal mandate of “fair cloud switching” and “vendor neutrality.”

The EU Data Act has fundamentally changed data sharing by mandating new rights for data access and portability. This has forced a mass redesign of data-sharing processes and vendor contracts. In 2026, the question of “where your data sits” is a matter of sovereignty.

Public sector and finance leaders are leading this charge, moving critical workloads to certified sovereign environments. They recognize that in the age of autonomous AI, control over the underlying infrastructure is the only way to mitigate the risk of vendor lock-in and geopolitical friction.

8. Conclusion: The Trust Dividend

The digital economy of the next decade is being built on the foundations we lay today. By 2026, the convergence of Governance, Sovereignty, and Automation has created a “Trust Dividend.” Those who invested in making their data agent-ready and audit-proof are now scaling autonomous systems with a level of confidence their competitors can’t match.

As we look toward an increasingly autonomous future, the question for every technical leader has shifted:

Is your data estate merely a collection of assets, or is it a governed foundation ready for the age of autonomy?

Why Your Writing Workflow Needs an AI Upgrade: Lessons from a Technical Insider

Leave a reply

Deploying agentic workflows is no longer a luxury for the modern creator; it is the baseline for survival in a field that moves faster than most can read. As a Senior Technical Content Strategist, I focus on systems that actually perform. I’m Ira Warren Whiteside, and my perspective on AI and Agentic AI isn’t theoretical—it’s built into my daily architecture. This shift toward high-efficiency workflows became a necessity during a recent recovery period. While my throat was healing from extreme weight lee (loss), I had to ensure my output remained high-fidelity without the luxury of manual, exhaustive research sessions.

The challenge is the “Creator’s Dilemma”: how to manage research-heavy technical projects while staying at the cutting edge of a relentless industry. The solution lies in treating AI not as a ghostwriter, but as a sophisticated research and synthesis layer that bridges the gap between deep technical archives and publication-ready insights.

1. Speed as a Competitive Advantage

In a technical ecosystem, speed is the ultimate competitive advantage. NotebookLM serves as a powerful catalyst for this, functioning as a specialized engine for rapid synthesis. By offloading the heavy lifting of initial research and document correlation, the platform allows a strategist to bypass the friction of manual data sorting.

Reducing the time spent on manual synthesis shifts the focus where it belongs: on high-level strategy and technical exploration. When you aren’t bogged down in the mechanics of organization, you are free to find the narrative within the data. As my recent workflow proves, this approach:

“speeds up research… saves time… excellent creators workflow.”

2. Turning Your Archives into a Discovery Engine

Generic AI models provide generic results. To produce truly authoritative content, you must mine your own intellectual property. This workflow uses the tool as a mirror, bringing out new discoveries based specifically on my own writings, ideas, and targeted prompts. It creates a closed-loop feedback system where past logic informs future innovation.

This is far more valuable than a standard LLM query; it ensures the output is grounded in a unique perspective rather than a homogenized dataset. It allows the creator to see patterns in their own thinking that might otherwise remain buried in thousands of lines of documentation.

  • Exploration through Variety: The system produces a wide variety of outputs—from summaries to deep-dive briefings—enabling a more comprehensive exploration of complex technical topics.

3. Bridging the Gap: From AI to RDBMS

For a Technical Insider, a workflow must handle more than just prose. It must integrate seamlessly with structured engineering data. My process bridges the gap between creative synthesis and the world of RDBMS STATISTICS, T-SQL SCRIPTS, AND SERVICES FROM METADATA MECHANICS.

This isn’t just about storing scripts; it’s about using AI to interpret technical metadata. It’s the ability to turn a raw T-SQL execution plan or a complex database schema into a high-level architectural narrative. By processing these technical artifacts through an intelligent workflow, I can generate documentation and insights that are as functionally accurate as they are readable.

METADATA MECHANICS represents the intersection of structured data and narrative strategy. This “clean aesthetic” in data management allows me to move from raw database statistics to polished technical blogging without losing the underlying technical rigor.

4. Grounding Insights in Reality

The primary risk of AI-integrated writing is the “hallucination”—the confident assertion of a technical falsehood. In technical blogging, credibility is the only currency that matters. This workflow mitigates that risk by ensuring that “references are included” for every generated insight.

Direct citations back to the source context are the essential antidote to AI errors. When writing about complex RDBMS behaviors or specific T-SQL implementations, having a clickable path back to the source material ensures that every claim is verified. This grounding transforms an AI tool from a creative assistant into a reliable technical partner.

The Future of the Intelligent Workflow

Integrating a tech-focused AI workflow allows a creator to explore and keep up with new technology while maintaining a rigorous publishing cadence. By leveraging these agentic systems, we move beyond simple content creation and into the realm of intellectual discovery.

As you evaluate your own technical output, ask yourself: how are you integrating your own METADATA MECHANICS into your creative process? The goal is to move past the manual synthesis bottleneck and begin gaining deeper, data-driven insights from the archives you’ve already built.

Thank You,

Ira Warren Whiteside

414.239-1266

Beyond the Hype: 5 Surprising Realities of the Modern LLM Frontier

Leave a reply

1. Introduction: The Unseen Mechanics of the AI Revolution

Large Language Models (LLMs) have successfully transitioned from laboratory curiosities to ubiquitous enterprise tools. To the casual observer, the progress looks like a linear march toward increasingly “smarter” chatbots. However, the technical reality is far more nuanced. Behind the curtain of viral interfaces, the most impactful breakthroughs are no longer just about increasing parameter counts or ingestion volume. As a Research Strategist, I observe that the real frontier has shifted toward “unseen mechanics”—the sophisticated methods researchers use to steer, optimize, and ground these models to transform them from unpredictable black boxes into high-precision, reliable instruments.

2. The Operational Safety Gap: Why Your Agent “Enters the Wrong Chat”

A critical challenge for enterprise deployment is “operational safety.” While global discourse often focuses on preventing generic harms (e.g., assisting in illegal acts), operational safety addresses a model’s ability to remain faithful to its intended purpose. Recent research, specifically the OffTopicEvalbenchmark, reveals a startling reality: LLMs are prone to “entering the wrong chat.”

When tasked with a professional role—such as an AI bank teller—models frequently fail to refuse out-of-domain (OOD) queries, straying into discussions about poetry or travel advice. The data shows that even top-tier models struggle; Llama-3 and Gemma collapsed to accuracy levels of 23.84% and 39.53% respectively in agentic scenarios. Even GPT-4 plateaus in the 62–73% range. Interestingly, the benchmark identifies Mistral (24B) at 79.96% and Qwen-3 (235B) at 77.77% as the current leaders in operational reliability.

To suppress these failures without the overhead of retraining, researchers are utilizing prompt-based steering. Techniques like Query Grounding (Q-ground) provide consistent gains of up to 23%, while System-Prompt Grounding (P-ground) delivered a massive 41% boost to Llama-3.3 (70B).

“To suppress these failures, we propose prompt-based steering methods: query grounding (Q-ground) and system-prompt grounding (P-ground), which substantially improve OOD refusal. Q-ground provides consistent gains of up to 23%, while P-ground delivers even larger boosts.”

3. Surgical Alignment: Steering the “Brain” Without Retraining

A major obstacle in fine-tuning is the “superposition” problem: LLM neurons are semantically entangled, often responding to multiple unrelated factors. This makes standard fine-tuning messy, as adjusting one behavior (like bias) often accidentally degrades linguistic fluency.

The Sparse Representation Steering (SRS)framework offers a “surgical” alternative. Using Sparse Autoencoders (SAEs), SRS projects dense activations (n) into a significantly higher-dimensional sparse feature space (m>n). This allows researchers to disentangle activations into millions of monosemantic features. To identify exactly which features to “turn up or down,” SRS utilizes bidirectional KL divergencebetween contrastive prompt distributions to quantify per-feature sensitivity.

This level of precision, often characterized by the L0 norm (the number of non-zero elements), allows developers to modulate specific attributes like truthfulness or safety at inference time with minimal side effects on overall quality.

“Due to the semantically entangled nature of LLM’s representation, where even minor interventions may inadvertently influence unrelated semantics, existing representation engineering methods still suffer from… content quality degradation.”

4. The 20% Rule: Efficiency via the “Heavy Hitter Oracle”

Deploying LLMs at scale is hindered by the KV Cache bottleneck. Because the cache scales linearly with sequence length, long conversations eventually overwhelm GPU memory. However, the Heavy Hitter Oracle (H2O) discovery has revealed a counter-intuitive efficiency: LLMs only need a fraction of their “memory” to maintain performance.

Researchers found that a small portion of tokens—Heavy Hitters (H2)—contribute the vast majority of value to attention scores. These tokens correlate with frequent co-occurrences in the text. By formulating KV Cache eviction as a dynamic submodular problem, the H2O framework retains only the most critical 20% of tokens. This results in up to a 29x improvement in throughput. This breakthrough democratizes AI, allowing massive models to run on smaller, cheaper hardware while retaining full contextual awareness.

5. The “Tool-Maker” Evolution: From Passive Solvers to Software Engineers

We are witnessing a fundamental shift from LLMs as “Tool Users” to LLMs as “Tool Makers” (LATM). Frameworks like LATM and CREATOR allow models to recognize when their inherent capabilities are insufficient—such as for complex symbolic logic—and respond by writing their own reusable Python functions.

This enables a cost-effective “division of labor.” An expensive, high-reasoning model (like GPT-4) acts as the Tool Maker, crafting a sophisticated utility function. A lightweight, cheaper model then acts as the Tool User, applying that function to thousands of requests. This allows models to solve problems they were never originally trained for by essentially creating their own specialized software on the fly.

6. The Semantic Shift: Moving Beyond the “Library Card Catalog”

Search technology is evolving from traditional Lexical Search to Semantic Search, fundamentally changing how information is retrieved.

  • Lexical Search acts like a literal “card catalog.” It relies on exact keyword matching. Searching for “affordable electric vehicles” might miss a document about a “Tesla Model 3” if those specific words are absent.
  • Semantic Search functions like a “knowledgeable librarian.” Using Dense Embeddings and Natural Language Processing (NLP), it maps queries into a vector space where similar concepts are mathematically grouped. It understands that “budget” and “affordable” are conceptually linked.

By leveraging Vector Databases (such as Milvus or Qdrant), modern systems now utilize a Hybridapproach. This combines the literal precision and speed of lexical search with the deep conceptual “brain” of semantic search, ensuring that intent is captured even when language is misaligned.

7. Conclusion: The Dawn of the “Interpretable” Era

The advancements moving through the AI frontier—from sparse steering and heavy-hitter optimization to autonomous tool-making—signal the end of the “black box” era. We are entering a phase where LLMs are becoming modular, efficient, and, most importantly, interpretable. By moving toward surgical control over internal representations, we move closer to systems we can truly understand and govern.

As we look forward, a vital question remains for the industry: Does the future of AI rely on building ever-larger models, or is the true path to intelligence found in making our control over them more modular and precise?

Practical artificial intelligence and it’s application to traditional business intelligence

Leave a reply
AI + BI

There are many books and photos that talk about the so-called revolution of artificial intelligence or AI I appreciate their enthusiasm however, I don’t think that is really being recognized that much of what they have invented i can be used in reality to greatly, reduce the time and effort to creat the kind of things we’ve had a creat for decades And they are essential to the creation of AI, which is predicated on learning and contact

One of the things that we will address is, there is a proclivity to use much language much of it is not very semantic. Today folks are describing architecture that already have descriptions. We have a mix of marketing terms and creative terms that mean the same thing, but they use words from the past. It’s causing confusion.

Just one example, maybe the term dimensionis one thing it meant to me thing 10 years ago come today it’s being used in a different context, in AI is it’s dimension or parameter or a feature There are many people of many years in a IT language is important it’s semantic

Another is the term similarity . It is used completely different in AI versus traditional, fuzzy matching. true the concept is the same but the Technical use Is different

There is no doubt of the benefit of what’s been created through the use of neural networks and transformers tthat hey can have tremendous positive impact on delivering business intelligence with the aid of artificial intelligence, machine, learning, and Deep Learning subsequently

I have been deeply involved in business intelligence, data quality, data profiling, and MDM and Data Governance for several decades.

I would like to take you on a journey and be able to help you exploit all these capabilities today and yesterday we are experience in the evolution of what we’re doing it is not a revolution. It is an evolution. if anything I hope to help achieve a basic understanding and terminalnology used in information architecture and various techniques that we have that will help , frankly nobody has a corner on the best approach it has all been done before at the logical level I want to be a part of helping us leverage, reuse, and apply what we were doing for decades, to what is now being introduced in the last several years you have to judge among the three goals, cheaper faster, better or we can guarantee is cheaper and faster. It’s up to you to make it better not necessarily the technology.

More to come . . .p

AI LLM New Project Roles including Prompt Engineering and Role AI Scientist and integrated data and information

Leave a reply

I’d like to offer some advice on transitioning skills and knowledge skills and knowledge who worked hard to retainto include some of the new AI and NLM developments it’s actually less impactful, and better than you may think

Well, I dressed up in a little bit for now let’s talk about prompt engineers You most likely have currenb SME’s or expert on your current data and requirements

First, do you notice I used the term AI scientist Instead of data scientist , A data scientist Currently is actually a AI model scientist and they will help you apply. We are concerned here with how a lot of folks have opinions heuristic no not necessarily fact-based already we are going to suggest some techniques, and provide some mentoring to explore this important factor in AI is proper training we specialize in providing techniques and mentoring in separate information which is Formulated and an opinion and facts or Data, which, cannot change

There is a series of steps involved in preparing for the use of Data inAI and Chat in the form of LLM models. This is not much different and you may have most of the information already gathered in order to properly design the requirements for your model we would collect the phone.it is important to realize the steps are critical, for you have confidence in your models putput which will be your result of integrating, your Word documents, your Presentations,, spreadsheets, and, of course your actual data.

We wiKeaton, Billy, and modeling of information words versus modeling and requirements for data preparation . There is a difference that is extremely important and in line with what you’ve been doing.I know that Data preparation is not glamorous, but in my 20+ years you’ll get nowhere without proper data preparation you can’t AI it you can’t impute it you need to discuss requirements with people and write them down and then execute it The AI will make the legwork, faster. but in the end you’ll have to review it otherwiseit otherwise you may end up needlessly retracing your steps based on Improper preparation I know that Data preparation is not glamorous, but in my 20+ years you’ll get nowhere without proper data preparation you can AI it you can’t imputed you need to discuss requirements with people and write them down and then execute a I will do it. Faster time is the legwork, but in the end you’ll have to review with Stuck you may end up needlessly retracing your steps based on.I know that Data preparation is not glamorous, but in my 20+ years you’ll get nowhere without proper data preparation you can AI it you can’t imputed you need to discuss requirements with people and write them down and then execute a I will do it. Faster time is the legwork, but in the end you’ll have to review with Stuck you may end up needlessly retracing your steps based on. improper preparation. This can be at Floydd by phone, the proper steps.

1. Word document

2. Presentations

3. Spreadsheets

4.Data reports

5. Data quality report for AI preparation

6.Internet

7.Other Sources (Network,Internet or Local)

We have suggested tools/techniques/open source.and suggestions for each of these. Don’t let that bother you, however, is important with today’s capabilities in AI integrate your words your thoughts, your abstraction, and your actual data together in order for you. They’re trustworthy results from your AI.

We will be providing a separate post on each of these and then finally how they come together our point is that the what you’ve been doing to understand and form requires for tradition BI can be reutilized and extend it for AI

Evolving Data Into Information through Lineage

Leave a reply


Information lineage through data DNA

 

Ira Warren Whiteside         

 

ORIGINS
COMMON SENSE

We in IT  have complicated and diluted the concept and  process of analyzing data and business metrics  incredibly in the last few decades. We seem to be focusing on the word data.         

“There is a subtle difference between data and information.”

Information vs data

There is a subtle difference between data and information. Data are the facts or details from which information is derived. Individual pieces of data are rarely useful alone. For data to become information, data needs to be put into context.

Examples of Data and Information

The history of temperature readings all over the world for the past 100 years is data.

If this data is organized and analyzed to find that global temperature is rising, then that is information.

The number of visitors to a website by country is an example of data.

Finding out that traffic from the U.S. is increasing while that from Australia is decreasing is meaningful information.

Often data is required to back up a claim or conclusion (information) derived or deduced from it.

For example, before a drug is approved by the FDA, the manufacturer must conduct clinical trials and present a lot of data to demonstrate that the drug is safe.

“Misleading” Data

Because data needs to be interpreted and analyzed, it is quite possible — indeed, very probable — that it will be interpreted incorrectly. When this leads to erroneous conclusions, it is said that the data are misleading. Often this is the result of incomplete data or a lack of context.

For example, your investment in a mutual fund may be up by 5% and you may conclude that the fund managers are doing a great job. However, this could be misleading if the major stock market indices are up by 12%. In this case, the fund has underperformed the market significantly.

 

Comparison chart

Synthesis: the combining of the constituent elements of separate material or abstract entities into a single or unified entity ( opposed to analysis, ) the separating of any material or abstract entity into its constituent elements.

Synthesis         

 Data into Information is dominant in terms of data movement and replication, in essence data logistics.

 

Lineage is the key.

 

 And with the simple action of linking data file metadata names to a businesses glossary  or terms, Will result in deeply insightful and informative business insight and analysis.

“Analysis the separating of any material or abstract entity into its constituent elements”

 In order for a business manager for analysis you need to be able to start the analysis at a understandable business terminology.

 And then provide the manager with the ability to decompose or break apart the result.

They are three essential set of capabilities and associated techniquestechniques for analysis and lineage.

 

  1. Data profiling and domain analysis as well as fuzzy matching components available on my blog https://irawarrenwhiteside.com/2014/04/13/creating-a-metadata-mart-via-tsql/
  2. Meta-data driven creation of a meta-data mart through code generation techniques, implemented.

 

 

Underlining each of these capabilities is a set of refined, developed and proven code says for accomplishing these basic fundamental task.

 

One case study

 I have been in this business over 45 years and I’d like to offer one example of the power of the concept of a meta-data mart and lineage as it regards to business insight.

 

A lineage, information and data story for BCBS

I was called on Thursday and told to attend a meeting on Friday between our companies leadership and the new Chief Analytics Officer. He was prototypical of the new IT a “new school” IT Director.

 I had been  introduced via LinkedIn to this director a week earlier as he had followed one of my blogs on metadata marts and lineage.

 

After a brief introduction, our leadership began to speak and the director immediately held up his hand he said “Please don’t say anything right now the profiling you provided me is at the kindergarten level and you are dishonest”

 

 The project was a 20 week $900,000 effort and we were in week 10.

 The company has desired to do a proof of concept and better understand the use of the informatics a tool  DQ as well as direction for a data governance program.

 

 To date what had been accomplished was in a cumulation of hours of effort in billing that has not resulted in any tangible deliverable.

 

The project had focused on the implementation and functionally of the popular vendor tool, canned data profiling results and not providing information to the business.

 

 The director commented on my blog post and asked if we could achieve that at his company, I of course said yes.

 Immediately I proposed we use the methodology that would allow us to focus on a  tops down process of understanding critical business metrics and a bottoms up process of linking data to business terms.

 My basic premise was that unless your deliverable from a data quality project can provide you business insight from the top down it is of little value. In essence you’ll spend $900,000 to tell  a business executive they have dirty data. At which point he will say to you “so what’s new”.

 The next step was to use the business terminology glossary that existed in informatica  metadata manager and map  those terms to source data columns and source systems, not an extremely difficult exercise. However this is the critical step in providing a business manager the understanding and context of data statistics.

 The next step, was the crucial step in which we made a slight modification to the IDQ tool and allowed the storing of the profiling results into a meta-data mart and the association of a business dimension from the business glossary the reporting statistics.

 We were able to populate my predefined metadata mart dimensional model by using the tool the company and already purchased.

 Lastly by using a dimensional model we were able to allow the business to apply their current reporting tool.

 

 Upon realizing the issues they faced in their business metrics, they accelerated the data governance program and canceled the data lake until a future date.

 

 

 

Now for the results. 

 Within six weeks we provided an executive dashboard based on a meta-data mart that allowed the business to reassess their plans involving governance and a data lake.

 Here are some of the results of their ability to analyze their basic data statistics but mapped to their business terminology.

 

  •  6000 in properly form SS cents
  • 35,000 dependence of subscribers over 35 years old
  • Thousands of charges to PPO plans out of the counties they were restricted to.
  •  There were mysterious double counts in patient eligibility counts, managers were now able to drill into those accounts by source system and find that a simple Syncsort utility had been used improperly and duplicated records.

 

                    

                               

                                                 

 

 

 

 

 

 

         

 

                                        

 

         

 

 

 

 

 

 

 

 

J

Navigating the Information Value Chain

2 Replies

Data Governance

Navigating the Information Value Chain

Demystifying the path forward

The challenge for businesses is to seek answers to questions, they do this with Metrics (KPI’s) and know the relationships of the data, organized by logical categories(dimensions) that make up the result or answer to the question. This is what constitutes the Information Value Chain

Navigation

Let’s assume that you have a business problem, a business question that needs answers and you need to know the details of the data related to the business question.

Information Value Chain

Information Value Chain

 

Information Value Chain

• Business is based on Concepts.

• People thinks in terms of Concepts.

• Concepts come from Knowledge.

• Knowledge comes from Information.

• Information comes from Formulas.

• Formulas determine Information relationships based on quantities.

• Quantities come from Data.

• Data physically exist.

In today’s fast-paced high-tech business world this basic navigation (drill thru) business concept is fundamental and seems to be overlooked, in the zeal to embrace modern technology

In our quest to embrace fresh technological capabilities, a business must realize you can only truly discover new insights when you can validate them against your business model or your businesses Information Value Chain, that is currently creating your information or results.

Today data needs to be deciphered into information in order to apply formulas to determine relationships and validate concepts, in real time.

We are inundated with technical innovations and concepts it’s important to note that business is driving these changes not necessarily technology

Business is constantly striving for a better insights, better information and increased automation as well as the lower cost while doing these things several of these were examined and John Thuma’s‘ latest article

Historically though these changes were few and far between however innovation in hardware storage(technology) as well as software and compute innovations have led to a rapid unveiling of newer concepts as well as new technologies

Demystifying the path forward.

In this article we’re going to review the basic principles of information governance required for a business measure their performance. As well as explore some of the connections to some of these new technological concepts for lowering cost

To a large degree I think we’re going to find that why we do things has not changed significantly it’s just how, we now have different ways to do them.

It’s important while embracing new technology to keep in mind that some of the basic concepts, ideas, goals on how to properly structure and run a business have not changed even though many more insights and much more information and data is now available.

My point is in the implementing these technological advances could be worthless to the business and maybe even destructive, unless they are associated with a actual set of Business Information Goals(Measurements KPI’s) and they are linked directly with understandable Business deliverables.

And moreover prior to even considering or engaging a data scienkce or attempt data mining you should organize your datasets capturing the relationships and apply a “scoring” or “ranking” process and be able to relate them to your business information model or Information Value Chain, with the concept of quality applied real time.

The foundation for a business to navigate their Information Value Chain is an underlying Information Architecture. An Information Architecture typically, involves a model or concept of information that is used and applied to activities which require explicit details of complex information systems.

Subsequently a data management and databases are required, they form the foundation of your Information Value Chain, to bring this back to the Business Goal. Let’s take a quick look at the difference between relational database technology and graph technology as a part of emerging big data capabilities.

However, considering the timeframe for database technology evolution, has is introduced a cultural aspect of implementing new technology changes, basically resistance to change. Business that are running there current operations with technology and people form the 80s and 90s have a different perception of a solution then folks from the 2000s.

Therefore, in this case regarding a technical solution “perception is not reality”, awarement is. Business need to find ways to bridge the knowledge gap and increase awarement that simply embracing new technology will not fundamentally change the why a business is operates , however it will affect how.

Relational databases were introduced in 1970, and graph database technology was introduced in the mid to 2000

There are many topics included in the current Big Data concept to analyze, however the foundation is the Information Architecture, and the databases utilized to implement it.

There were some other advancements in database technology in between also however let’s focus on these two

 

History

1970

In a 1970s relational database, Based on mathematical Set theory, you could pre-define the relationship of tabular (tables) , implement them in a hardened structure, then query them by manually joining the tables thru physically naming attributes and gain much better insight than previous database technology however if you needed a new relationship it would require manual effort and then migration of old to new , In addition your answer it was only good as the hard coding query created

Early Relationship Conepts

2020

In mid-2000’s the graph database was introduced , based on graph theory, that defines the relationships as tuples containing nodes and edges. Graphs represent things and relationships events describes connections between things, which makes it an ideal fit for a navigating relationship. Unlike conventional table-oriented databases, graph databases (for example Neo4J, Neptune) represent entities and relationships between them. New relationships can be discovered and added easily and without migration, basically much less manual effort.

Nodes and Edges

Graphs are made up of ‘nodes’ and ‘edges’. A node represents a ‘thing’ and an edge represents a connection between two ‘things’. The ‘thing’ in question might be a tangible object, such as an instance of an article, or a concept such as a subject area. A node can have properties (e.g. title, publication date). An edge can have a type, for example to indicate what kind of relationship the edge represents.

Current Relationship Concepts

Takeaway.

The takeaway there are many spokes on the cultural wheel, in a business today, encompassing business acumen, technology acumen and information relationships and raw data knowledge and while they are all equally critical to success, the absolute critical step is that the logical business model defined as the Information Value Chain is maintained and enhanced.

It is a given that all business desire to lower cost and gain insight into information, it is imperative that a business maintain and improve their ability to provide accurate information that can be audited and traceable and navigate the Information Value Chain,Data Science can only be achieved after a business fully understand their existing Information Architecture and strive to maintain it.

Note as I stated above an Information Architecture is not your Enterprise Architecture. Information architecture is the structural design of shared information environments; the art and science of organizing and labelling websites, intranets, online communities and software to support usability and findability; and an emerging community of practice focused on bringing principles of design, architecture and information science to the digital landscape. Typically, it involves a model or concept of information that is used and applied to activities which require explicit details of complex information systems.

Ancient Relationship Concepts

In essence, a business needs a Rosetta stone in order translate past, current and future results.

In future articles we’re going to explore and dive into how these new technologies can be utilized and more importantly how they relate to Business outcomes

Chocolate cake, MDM, data quality, machine learning and creating the information value chain’

Leave a reply

The primary take away from this article will be that you don’t start your Machine Learning project, MDM , Data Quality or Analytical project with “data” analysis, you start with the end in mind, the business objective in mind. We don’t need to analyze data to know what it is, it’s like oil or water or sand or flour.

Unless we have a business purpose to use these things, we don’t need to analyze them to know what they are. Then because they are only ingredients to whatever we’re trying to make. And what makes them important is to what degree they are part of the recipe , how they are associated

Business Objective: Make Desert

Business Questions: The consensus is Chocolate Cake , how do we make it?

Business Metrics: Baked Chocolate Cake

Metric Decomposition: What are the ingredients and portions?

2/3 cup butter, softened

1-2/3 cups sugar

3 large eggs

2 cups all-purpose flour

2/3 cup baking cocoa

1-1/4 teaspoons baking soda

1 teaspoon salt

1-1/3 cups milk

Confectioners’ sugar or favorite frosting

So here is the point you don’t start to figure out what you’re going to have for dessert by analyzing the quality of the ingredients. It’s not important until you put them in the context of what you’re making and how they relate in essence, or how the ingredients are linked or they are chained together.

In relation to my example of desert and a chocolate cake, an example could be, that you only have one cup of sugar, the eggs could’ve set out on the counter all day, the flour could be coconut flour , etc. etc. you make your judgment on whether not to make the cake on the basis of analyzing all the ingredients in the context of what you want to, which is a chocolate cake made with possibly warm eggs, cocunut flour and only one cup of sugar.

Again belaboring this you don’t start you project by looking at a single entity column or piece of data, until you know what you’re going to use it for in the context of meeting your business objectives.

Applying this to the area of machine learning, data quality and/or MDM lets take an example as follows:

Business Objective: Determine Operating Income

Business Questions: How much do we make, what does it cost us.

Business. Metrics: Operating income = gross income – operating expenses – depreciation – amortization.

Metric Decomposition: What do I need to determine a Operating income?

Gross Income = Sales Amount from Sales Table, Product, Address

Operating Expense = Cost from Expense Table, Department, Vendor

Etc…

Dimensions to Analyze for quality.

Product

Address

Department

Vendor

You may think these are the ingredients for our chocolate cake in regards to business and operating income however we’re missing one key component, the portions or relationship, in business, this would mean the association,hierarchy or drill path that the business will follow when asking a question such as why is our operating income low?

For instance the CEO might first ask what area of the country are we making the least amount of money?

After that the CEO may ask well in that part of the country, what product is making the least amount of money and who manages it, what about the parts suppliers?

Product => Address => Department => Vendor

Product => Department => Vendor => Address

Many times these hierarchies, drill downs, associations or relationships are based on various legal transaction of related data elements the company requires either between their customers and or vendors.

The point here is we need to know the relationships , dependencies and associations that are required for each business legal transaction we’re going to have to build in order to link these elements directly to the metrics that are required for determining operating income, and subsequently answering questions about it.

No matter the project, whether we are preparing for developing a machine learning model, building an MDM application or providing an analytical application if we cannot provide these elements and their associations to a metric , we will not have answered the key business questions and will most likely fail.

The need to resolve the relationships is what drives the need for data quality which is really a way of understanding what you need to do to standardize your data. Because the only way to create the relationships is with standards and mappings between entities.

The key is mastering and linking relationships or associations required for answering business questions, it is certainly not just mastering “data” with out context.

We need MASTER DATA RELATIONSHIP MANAGEMENT

not

MASTER DATA MANAGEMENT.

So final thoughts are the key to making the chocolate cake is understanding the relationships and the relative importance of the data/ingredients to each other not the individual quality of each ingredient.

This also affects the workflow, Many inexperienced MDM Data architects do not realize that these associations form the basis for the fact tables in the analytical area. These associations will be the primary path(work flow) the data stewards will follow in performing maintenance , the stewards will be guided based on these associations to maintain the surrounding dimensions/master entities. Unfortunately instead some architects will focus on the technology and not the business. Virtually all MDM tools are model driven APIs and rely on these relationships(hierarchies) to generate work flow and maintenance screen generation. Many inexperienced architects focus on MVP(Minimal Viable Product), or technical short term deliverable and are quickly called to task due to the fact the incurred cost for the business is not lowered as well as the final product(Chocolate Cake) is delayed and will now cost more.

Unless the specifics of questionable quality in a specific entity or table or understood in the context of the greater business question and association it cannot be excluded are included.

An excellent resource for understanding this context can we found by following: John Owens

Final , final thoughts, there is an emphasis on creating the MVP(Minimal Viable Product) in projects today, my take is in the real world you need to deliver the chocolate cake, simply delivering the cake with no frosting will not do,in reality the client wants to “have their cake and eat it too”.

Note:

Operating Income is a synonym for earnings before interest and taxes (EBIT) and is also referred to as “operating profit” or “recurring profit.” Operating income is calculated as: Operating income = gross income – operating expenses – depreciation – amortization.