How your customer or organization can lean the mission critical data?

Summary

First this blog post will highlight the symptoms we might hear about in Big Data, or any EDW of any size.  A performance problem, sure we can let the issue fall into performance maintenance.  Although the source of the problem is a symptom or a clue.

A physical table duplicate implies a shadow process in place

• A non-standard or non-authorized person using the data outside the sanctioned process.  

A physical table enables the represented data to be collected or acquired and written to a custom report or presented in a custom (in house) transactional application. 

• We might even hear about these problems on the network with burst or high congestion during batch processing and a denial of service attack.

Yes, the physical table is the center of the data processing or curation world.  The box "processing" and the way vertical and horizontal processes work within or across technology stacks.

In many database engines, the natural processing power leverages a few concepts which most business people would simply assume are common sense. 

For example duplicate indexes in a parallel processing appliance for example. 

• The duplicate tables forces the appliance to work harder, confusing the even distribution across many nodes to balance the workload.  
• The failure will be in too many CPU's, spikes in the IO and may fail fault tolerance.  
• A perfect place for a DDOS attack to make it's way into your organization.  

The entire technology stack becomes far less reliable and fails to meet the specifications promised by the vendor.  

a symptom of shadow IT

• Duplicate indices - high power and degraded performance

• a problem with the duplication of physical tables 
• the table may be renamed and data structured in the same way as the source physical table
• the table may be named slightly different yet structured in the same way
• a symptom of an offline data algorithm writing back to an operational application 
• I've seen this most often in CRM or marketing application

Person Centered Design

In the case of duplicate indices we must be concerned as the quality into an operational process may force re-work and has the highest probability of impacting the customer and external stakeholders.

"why would I use this application, the information isn't used by anyone else"
"why bother, have you seen how many customer records I'd have to go through, it's faster to just enter my customer and information"  

• Employees may re-key rather than try to find things in an application when each day the application is flushed with backend data algorithms.  
• Especially when the application is controlled and uncontrolled obviously. 
• Worst case I've seen, users entered the information 7 times between the acceptance of the customer or win of an opportunity to commission crediting. 

What may be threatening your customer experience?

• Common issues requiring maintenance 

• a large amount of null values 
• a large amount of data type changes which never get changed
• column decimal format error
• data format error
•  a table with zero values

Worst Case Scenario

• By the time the order booked, 7 different contractors entered into another application at different times for a different stakeholder within the company. 
• Only when I was allowed to sit in a bull pen with the other roles within the company did I learn of the way we speak, and frequency of the events we process will either prompt fight to get it in or flight push it back to get it away.  
• "the purpose and the motivations for each audience are highly dependent on access to the information and frequency of transactions processed."
• Assignment of specialized resources who only deal with the larger slower transactions significantly improves the situation.  
• The systems are not setup to support the larger slower deals.    
• A high transaction process will be the types most people are comfortable with the volume makes them the stakeholder to please.  
• everything else must work around the volume stakeholders
• The low transaction high dollar deals are immediately non-standard and avoidance is the favored response.  
• Null values
• Very few applications designed to support the customer experience allow a user to enter a null in the process workflow (sanctioned)
• A backend alogorithm isn't subject to the rule
• trust is lost by all stakeholders internally as Sales must be doing something wrong.
• Sales - not all sales and many times only a subset of sales is doing this.  
• A null on the country forces a touch by all stakeholders in every process
• A null on the state or zip code - same effect
• A decimal error forces manual touches on pricing or cost which are both illegal unless you hide them in adjustments.  
• A date error is typically a back door to trick a rebate or commission or performance on a business unit during any given period.  
• in many cases, the forced date error will prompt a static date to be applied indicating no change when the actions in fact are performing a change showing another field to allow the illegal behavior.
• Many large temp files are another indicator of the same bad behaviors. 

FACT 
When my career shifted from accounting or implementing ERP systems working with finance;  I held a role supporting direct business models for our foreign headquartered companies business and professional group in the Western Region.

• Western Region - Highest sales region with Northern western region generally higher than Southern due to the number of Global accounts headquartered in Silicon Valley.  
• Biggest customers - No two customers are the same
• Most demanding - forming relationships gets a team beyond this high touch part of direct business. 
• Customer negotiated terms-No two legal agreements are the same.  YOU CANNOT AND MUST NOT try to force a standard contract on these customers.  
• You will violate the terms of the master agreement or not be working in good faith
• Use a transformation to a service work order model 

When I went to work for the smallest reseller who had just won the sole source contract for all of the Western region as a system integrator for an innovative technology "Video Conferencing". I supported the highest selling team with the hardest to please customers.

As time moved on, I was asked to manage all Video Conference deals to ensure the smooth internal process which the other women were not willing to adjust to support.  

The same challenges were faced that we had while working as the assigned product manager  for the roll out in the Western Region rather than from New Jersey, as me and the new Director for our region were an awesome team.

Later when I made my way through a number of smaller or startups, the rule remained true  in each case, a large account is in all parts of the world.  We know where these customers are located.  A few people in the company have access and make decisions on these accounts.

If you segment your organization first by the large accounts, again Sub-Account 30 in this example.

The segment 30 is the slow lane, high dollars but slow to travel.  

The best way to manage the 30 business model is in their own applications, feeding to ERP at the time of the opportunity being validated.

• Customer commits-push 14 data fields to all downstream forms in ERP as the header and constraint people must remain in to meet the customer expectations.   
• On the use case where 7 entries were made-you just reduced the cost by 6 contractors and removed the need to re-key.  
• We know re-key is waste-imagine the savings you have from holding all stakeholders in the process accountable to their roles based on the date of push.   
• You have enabled the customers experience
• you reduced the inbound data collection 
• only new values enriched by the process will update the original table in a different physical table associated with the opportunity.  
• You enabled commission crediting 
• You enables supply chain
• You enabled customer service
• You enabled manufacturing or strategic alliances
• You improved data quality 
• You've met the SOX requirements
• You've met the ISO requirements
•  You are agile and ready to grow in all circumstances.  
• You are governing the organization and have excellent operational efficiency
•  Now, big data isn't so big anymore is it?