{"id":11317,"date":"2017-06-05T16:39:14","date_gmt":"2017-06-05T11:09:14","guid":{"rendered":"https:\/\/cigniti.com\/blog\/?p=11317"},"modified":"2017-06-05T18:12:24","modified_gmt":"2017-06-05T12:42:24","slug":"emerging-trends-etl-big-data-and-beyond","status":"publish","type":"post","link":"https:\/\/www.cigniti.com\/blog\/emerging-trends-etl-big-data-and-beyond\/","title":{"rendered":"Emerging Trends of ETL &#8211; Big Data and Beyond"},"content":{"rendered":"<p>Amidst the analysis of driving voluminous data, and the analytics challenges, there are concerns about whether the conventional process of extract, transform and load (ETL) is applicable.<\/p>\n<p>ETL tools quickly \u201cintrude\u201d across Mobile apps and Web applications as they can access data very efficiently. Eventually, ETL applications will accumulate industry standards and gain power.<\/p>\n<p>Let\u2019s discuss practically something rather new &#8211; that offers an approach to easily build sensible, adaptable data models that dynamize your data warehouse:\u00a0<strong>The Data Vault!<\/strong><\/p>\n<p>Enterprise Data Warehouse (EDW) systems intent to sustain an authentic Business Intelligence (BI) for the data-driven enterprise.\u00a0Companies must acknowledge critical metrics which are deep-rooted in this significant and dynamic data.<\/p>\n<p><strong>Challenges that wreck ETL with traditional data modelling<\/strong><\/p>\n<p>Following is a list of top 5 challenges that ETL faces due to traditional data modelling:<\/p>\n<ul>\n<li>Issue #1: When upstream business flows or when business rules of EDW\/BI systems adapt <strong>Change<\/strong>.<\/li>\n<li>Issue #2: Analysis of business data in terms of Volume, Velocity, Variety, Visualization, Veracity and Value (6Vs) \u2013 <strong>Big Data<\/strong> with technical realities to affirm competitive edge.<\/li>\n<li>Issue #3: To design, create, deliver, and sustain vigorous, accessible large powerful storage EDW\/BI systems for intelligent adoption has become <strong>Complex<\/strong>.<\/li>\n<li>Issue #4: To tailor the data to meet the needs of the business with its core <strong>Business Domain<\/strong> value, without failing in giving simple solutions for the definite needs of the business.<\/li>\n<li>Issue #5: Lack of <strong>Flexibility<\/strong> to adopt new unpredicted and unplanned sources with or without the impact of upstream process.<\/li>\n<\/ul>\n<p>Now it\u2019s time to emerge with much focused, all time solution for all the potential challenges depicted above.<\/p>\n<p><strong>Data Vault<\/strong> is a methodology with Hybrid Data modelling.<\/p>\n<p>As per <strong>Dan Linstedt<\/strong> it is detailed as follows:<\/p>\n<p>\u201cA detail oriented, historical tracking and uniquely linked set of normalized tables that support one or more functional areas of business.\u00a0It is a hybrid approach\u00a0encompassing the best of breed between 3NF and Star Schemas.\u00a0The design is flexible, scalable, consistent and adaptable to the needs of the enterprise.\u201d<\/p>\n<p>It\u2019s elegant, easy, and simple to execute. It is established on a set of too many structures, and with auditable rules. By exploiting the Data Vault principles your project will undoubtedly gratify auditability, scalability, and flexibility.<\/p>\n<p>The following stipulated standards will help you to build a Data Vault:<\/p>\n<ol>\n<li>\u00a0Step 1: Determine the Business Keys, Hubs<\/li>\n<li>\u00a0Step 2: Confirm the relationships between the Business Keys, Links<\/li>\n<li>\u00a0Step 3: Specify the description around the Business Keys, Satellites<\/li>\n<li>\u00a0Step 4: Connect Standalone factors like Time dimension attributes and code\/descriptions for decoding in Data\u00a0 Marts<\/li>\n<li>\u00a0Step 5: Integrate for query optimization, append performance tables like Bridge tables and Point-In-Tim (PD) structure model tables.<\/li>\n<\/ol>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-11318\" src=\"https:\/\/cigniti.com\/blog\/wp-content\/uploads\/Data-Vault.png\" alt=\"Data Vault\" width=\"648\" height=\"458\" srcset=\"https:\/\/www.cigniti.com\/blog\/wp-content\/uploads\/Data-Vault.png 648w, https:\/\/www.cigniti.com\/blog\/wp-content\/uploads\/Data-Vault-500x353.png 500w, https:\/\/www.cigniti.com\/blog\/wp-content\/uploads\/Data-Vault-600x424.png 600w\" sizes=\"(max-width: 648px) 100vw, 648px\" \/><\/p>\n<p><strong>Building a Data Vault is simple as you go<\/strong>; ultimately it will mutate the conventional methods generally used in Enterprise Integration Architectures. The model is built in a way that it can be efficiently extended whenever required.<\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-11319\" src=\"https:\/\/cigniti.com\/blog\/wp-content\/uploads\/Data-vault-2.png\" alt=\"\" width=\"813\" height=\"327\" srcset=\"https:\/\/www.cigniti.com\/blog\/wp-content\/uploads\/Data-vault-2.png 813w, https:\/\/www.cigniti.com\/blog\/wp-content\/uploads\/Data-vault-2-500x201.png 500w, https:\/\/www.cigniti.com\/blog\/wp-content\/uploads\/Data-vault-2-768x309.png 768w, https:\/\/www.cigniti.com\/blog\/wp-content\/uploads\/Data-vault-2-600x241.png 600w\" sizes=\"(max-width: 813px) 100vw, 813px\" \/><\/p>\n<p><strong>Data Vault Modelling + Architecture + Methodology provides solutions for the challenges depicted above.<\/strong><\/p>\n<p><em>\u201cBusiness Agility is the ability to improve from Continuous Change.\u201d<\/em><\/p>\n<p><strong>Let\u2019s see how Data Vault can adapt the \u201cChange\u201d.<\/strong><\/p>\n<p>With the partying of business keys (as they are static) in Data Vault and the associations between the business keys from their descriptive attributes, Data Vault can address the problem of change in the environment.<\/p>\n<p>Crafting these keys as the structural backbone of a data warehouse, all the associated data can be organized around them.\u00a0These Hubs (business keys), Links (associations), and SAT (descriptive attributes) yield an extremely adaptable data structure while sustaining an immense degree of data integrity.\u00a0Specific Links are like synapses (vectors in the opposite direction).\u00a0They can be created or dropped whenever the business relationships are bound to change automatically by transforming the data model as needed without any impact to the existing data structures.<\/p>\n<p><strong>Let\u2019s see how Data Vault engulfs the ETL challenge of Big Data.<\/strong><\/p>\n<p>Data Vault blends consistent integration of Big Data technologies along with modelling, methodology, architecture, and outstanding practices. With the adoption of very large voluminous data, data can easily be blended into a Data Vault data model to incorporate adopting products like Hadoop, MongoDB, and various other NoSQL varieties.\u00a0Eradicating the cleansing specifications of a Star Schema design, the Data Vault triumphs over huge data sets by reducing exhaustion and sustaining correlated insertions which impact the potential of Big Data systems.<\/p>\n<p><strong>Data Vault also decodes the challenge of complexity through Simplification. Let\u2019s see how.<\/strong><\/p>\n<p>Designing a competent and dynamic Data Vault model can be done instantaneously once you know the core of the 3 table types: Hub, Satellite and Link.\u00a0Determining the business keys\u00a0and specifying the Hubs is invariably the perfect thing to kick-off.\u00a0\u2018Hub-Satellites\u2019 simulate source table columns that can change and certainly Links connect them.\u00a0It is also feasible to have Link-Satellite tables.<\/p>\n<p>Once the Data Vault data model is done, the next uphill task is to build the Data Integration process through ETL (i.e. to populate data into target systems from source systems). So, with the Data Vaults design, you can connect the data-driven enterprise and enable data integration.<\/p>\n<p>ETL, with its simplified development process, decreases the total cost of Open platform. ETL can certainly be used to populate and maintain a robust EDW system built upon a Data Vault model.<\/p>\n<p>This can be achieved through any prominent ETL tools available in the market.<\/p>\n<p><strong>Overcoming the challenge of understanding Business domain using Data Vault:<\/strong><\/p>\n<p>The Data Vault typically specifies the outlook\/values of an Enterprise that analyze and details the business domain and relationships bounded within the vicinity.\u00a0Accomplishing business rules must ensue before populating a Star Schema.\u00a0Through Data Vault you can drive the business rules to the downstream, after EDW incorporation\/ingestion.\u00a0Another Data Vault philosophy is that any data is significant, even if it is irrelevant. The theory of the Data Vault is to ingest source data of any type (good, bad).<\/p>\n<p>This data model is designed eminently to resolve and meet the absolute needs of latest and present-day\u2019s EDW\/BI systems.<\/p>\n<p><strong>Data Vault is Flexible enough to adopt new unpredicted and unplanned sources without impacting the existing data model.<\/strong><\/p>\n<p>The Data Vault methodology is based on SEI\/CMMI Level 5 processes and practices, and comprises of various components constituting with outstanding features of Six Sigma, Total Quality Management (TQM) and SDLC (Agile).\u00a0Data Vault projects have short and considerable release cycles usually adopting the repeatable, defined, manageable, consistent and optimized projects expected at CMMI Level 5.\u00a0While adding new data sources, business keys which are alike and new Hubs-Satellites-Links can be added and then can be linked to the existing Data Vault structures without any impact with underlying model.<\/p>\n<p><strong>Testing a Data Vault \u2013 ETL\/Data Warehouse pursuit<\/strong><\/p>\n<p>Unlike in non-Data Vault ETL programs, general testing strategy best suits for any Data Vault adopted programs. However, by using raw Data Vault loads we can moderate transformations to a minimum level in the entire ETL process through \u201cPermissible Load Errors\u201d.<\/p>\n<p>ETL\/Data Warehouse Testing should emphasize on:<\/p>\n<ul>\n<li>Data quality of Source Systems<\/li>\n<li>Data Integration<\/li>\n<li>Performance\/Scalability\/Upgrade issues of BI Reports<\/li>\n<\/ul>\n<p>Following are the prominent 5 proposals to execute tests for a Data Vault \u2013 ETL\/DWH project to adhere the above baseline pointers:<\/p>\n<ol>\n<li>Design a small &#8211; static test database derived from the actual data to run the tests quickly and expected results can be identified in the earlier stage.<\/li>\n<li>Early execution of system testing to ensure the connecting boxes in ETLs interface.<\/li>\n<li>Using Test Automation tools to:<\/li>\n<\/ol>\n<ul style=\"margin-left: 20px;\">\n<li>Set up the test environment<\/li>\n<li>Restore a Virtual environment<\/li>\n<li>Update static data to validate complex data validations<\/li>\n<li>Execute the ETLs, capture logs and validate Bad\/Rejected\/Valid data flows<\/li>\n<\/ul>\n<ol start=\"4\">\n<li>Engage Business users while creating and deriving the real data to a small test database to ensure the data profiling and data quality.<\/li>\n<li>Simulate the test environment as is of the Production environment to cut down the cost issues.<\/li>\n<\/ol>\n<p><strong>To summarize<\/strong>\u00a0\u2013 exploiting various innovative methodologies to visualize the business trends coupled with substantial evidence will do wonders in ETL-Big Data engagements.<\/p>\n<p>Although it is important to discuss on the ETL trends to cop-up the Challenges, it is not enough to stop here. We need to extend and reflect upon how we can develop automation solutions to create Test Data for any ETL requirement using Component Libraries, RowGen (tool).<\/p>\n<p>Cigniti technologies has helped many clients with ETL\/Data warehousing testing that produced effective results of quick test cycles and achieved no defect leak to production and significantly met the production go timelines. For any of your ETL\/Data Warehousing Testing requirements, Cigniti\u2019s testing experts are available to help you out with your projects.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Amidst the analysis of driving voluminous data, and the analytics challenges, there are concerns about whether the conventional process of extract, transform and load (ETL) is applicable. ETL tools quickly \u201cintrude\u201d across Mobile apps and Web applications as they can access data very efficiently. Eventually, ETL applications will accumulate industry standards and gain power. Let\u2019s [&hellip;]<\/p>\n","protected":false},"author":36,"featured_media":11328,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[478,203],"tags":[200,1862,728,1861,1079,1863,20],"ppma_author":[3760],"authors":[{"term_id":3760,"user_id":36,"is_guest":0,"slug":"sree-lakshmi","display_name":"Sree Lakshmi - Project Lead Delivery","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/e345edbf026654f957977c5c52715d9a?s=96&d=mm&r=g","user_url":"","last_name":"Lakshmi","first_name":"Sree","job_title":"","description":"Sree Lakshmi's career spans 11+ years of extensive experience in ETL. Her expertise spreads across Big data Analytics, Visualization, Predictive Analytics, &amp; Data Sciences. She is well versed with business environments requiring powerful Lean methodologies, and is very passionate about learning new trends &amp; providing strategic solutions through analytics. An MCA graduate, Sree Lakshmi supports Cigniti's well-crafted success in Aviation, Insurance, BFSI, Healthcare, Retail and Technology capabilities."}],"_links":{"self":[{"href":"https:\/\/www.cigniti.com\/blog\/wp-json\/wp\/v2\/posts\/11317"}],"collection":[{"href":"https:\/\/www.cigniti.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cigniti.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cigniti.com\/blog\/wp-json\/wp\/v2\/users\/36"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cigniti.com\/blog\/wp-json\/wp\/v2\/comments?post=11317"}],"version-history":[{"count":0,"href":"https:\/\/www.cigniti.com\/blog\/wp-json\/wp\/v2\/posts\/11317\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.cigniti.com\/blog\/wp-json\/wp\/v2\/media\/11328"}],"wp:attachment":[{"href":"https:\/\/www.cigniti.com\/blog\/wp-json\/wp\/v2\/media?parent=11317"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cigniti.com\/blog\/wp-json\/wp\/v2\/categories?post=11317"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cigniti.com\/blog\/wp-json\/wp\/v2\/tags?post=11317"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.cigniti.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=11317"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}