oaxino Posted April 12, 2023 Report Share Posted April 12, 2023 System Design For Big Data PipelinesPublished 4/2023MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHzLanguage: English | Size: 1.82 GB | Duration: 6h 33mAnalyze, Design and Build scalable, resilient and cost-effective Big Data pipelines with a methodical processWhat you'll learnLearn about the building blocks of a big data pipeline, their functions and challengesAdapt an end-to-end methodical approach to designing a big data pipelineExplore techniques to ensure overall scaling of a big data pipelineStudy design patterns for building blocks, their advantages, shortcomings, applications and available technologiesFocus additionally on Infrastructure, Operations and Security for Big Data deploymentsExercise the learnings in the course with a Batch and Realtime use case studyRequirementsBig Data Technology ConceptsFamiliarity with Big Data Technologies like Apache Spark, Apache Kafka and NoSQLDevelopment / Deployment Experience with Big Data Technologies and PipelinesSoftware Design and Development Experience including Cloud & MicroservicesDescriptionBig data technologies have been growing exponentially over the past few years and have penetrated into every domain and industry in software development. It has become a core skill for a software engineer. Robust and effective big data pipelines are needed to support the growing volume of data and applications in the big data world. These pipelines have become business critical and help increase revenues and reduce cost.Do quality big data pipelines happen by magic? High quality designs that are scalable, reliable and cost effective are needed to build and maintain these pipelines.How do you build an end-to-end big data pipeline that leverages big data technologies and practices effectively to solve business problems? How do you integrate them in a scalable and reliable manner? How do you deploy, secure and operate them? How do you look at the overall forest and not just the individual trees? This course focuses on this skill gap.What are the topics covered in this course?We start off by discussing the building blocks of big data pipelines, their functions and challenges.We introduce a structured design process for building big data pipelines.We then discuss individual building blocks, focusing on the design patterns available, their advantages, shortcomings, use cases and available technologies.We recommend several best practices across the course.We finally implement two use cases for illustration on how to apply the learnings in the course to a real world problem. One is a batch use case and another is a real time use case.OverviewSection 1: Introduction & ExpectationsLecture 1 Need for Quality Pipeline DesignLecture 2 Course Coverage and Pre-requisitesLecture 3 Cloud Serverless TechnologiesSection 2: Building Blocks for Big Data PipelinesLecture 4 The Big Data Pipeline NetworkLecture 5 Data Acquisition BlocksLecture 6 Data Transport BlocksLecture 7 Data Processing BlocksLecture 8 Data Storage BlocksLecture 9 Data Serving BlocksLecture 10 Data Pipeline InfrastructureLecture 11 Data Pipeline OperationsSection 3: System Design ProcessLecture 12 System Design Process OverviewLecture 13 Analyze Functional RequirementsLecture 14 Analyze Pipeline InputLecture 15 Analyze Non-functional RequirementsLecture 16 Draw a Pipeline FlowchartLecture 17 Create a Skeleton DesignLecture 18 Analyze ScalingLecture 19 Select TechnologiesLecture 20 Design Infrastructure and OperationsLecture 21 Develop a Test StrategySection 4: Scalable Pipelines - Design PrinciplesLecture 22 Batch vs Realtime PipelinesLecture 23 Distributed ArchitecturesLecture 24 Microservices based ArchitecturesLecture 25 Batch Pipelines - Best PracticesLecture 26 Realtime Pipelines - Best PracticesLecture 27 Performance Benchmarking for Big Data PipelinesSection 5: Data Acquisition DesignLecture 28 File Transfer PatternLecture 29 Extraction Client PatternLecture 30 Ingestion API PatternLecture 31 Pub Sub Acquisition PatternLecture 32 Data Acquisition Design PracticesSection 6: Data Transport DesignLecture 33 Extract Load PatternLecture 34 Request Response PatternLecture 35 Event Streaming PatternLecture 36 Data Transport Design PracticesSection 7: Data Processing & Transformation DesignLecture 37 Data Processing PatternsLecture 38 Distributed Processing with Big DataLecture 39 Batch Processing Design Practices - Part 1Lecture 40 Batch Processing Design Practices - Part 2Lecture 41 Stream Processing Design PracticesLecture 42 Batch vs Realtime ProcessingLecture 43 Input and Output Considerations for ProcessingLecture 44 Processing Engine TechnologiesSection 8: Storage DesignLecture 45 Distributed File System PatternLecture 46 Relational Database PatternLecture 47 Document Database PatternLecture 48 Columnar Database PatternLecture 49 Graph Database PatternLecture 50 Distributed Cache PatternLecture 51 Data Storage Design Practices - 1Lecture 52 Data Storage Design Practices - 2Section 9: Serving DesignLecture 53 Query Interface PatternLecture 54 Serving API PatternLecture 55 Push Client PatternLecture 56 Publish Subscribe PatternLecture 57 Data Serving Design PracticesSection 10: Infrastructure and DeploymentsLecture 58 Infrastructure TechnologiesLecture 59 Microservices DeploymentsLecture 60 Processing Jobs DeploymentsLecture 61 Databases and Queues DeploymentsLecture 62 Geographical DistributionSection 11: SecurityLecture 63 Pipeline Security by DesignLecture 64 Secure External InterfacesLecture 65 Secure Data StorageLecture 66 Privacy ConsiderationsLecture 67 Multi-Tenancy ConsiderationsSection 12: ServiceabilityLecture 68 Elements of ServiceabilityLecture 69 Monitoring PipelinesLecture 70 Data to MonitorLecture 71 Pipeline TroubleshootingSection 13: Use Case I : Customer Journey Analytics (CJA)Lecture 72 Problem Definition for CJALecture 73 Study CJA Functional RequirementsLecture 74 Analyze CJA Input DataLecture 75 Study CJA Non-Functional RequirementsLecture 76 Study CJA Pipeline FlowchartLecture 77 Create CJA Skeleton DesignLecture 78 Analyze CJA ScalingLecture 79 Select Technologies for CJALecture 80 Design Infrastructure and Operations for CJASection 14: Use Case II : Suspicious Login Alerting (SLA)Lecture 81 Problem Definition for SLALecture 82 Study SLA Functional RequirementsLecture 83 Analyze SLA Input DataLecture 84 Study SLA Non-Functional RequirementsLecture 85 Draw SLA Pipeline FlowchartLecture 86 Create SLA Skeleton DesignLecture 87 Analyze SLA ScalingLecture 88 Select SLA TechnologiesLecture 89 Define SLA Infrastructure and OperationsSection 15: ConclusionLecture 90 Thank YouBig Data Pipeline Designers & Architects,Big Data Developers looking to move into Design/Architecture roles,Software Architects looking to gain Big Data ExperienceDownload linkrapidgator.net:https://rapidgator.net/file/d2376b5c4930fec0376290ca49e0c0b8/cpavz.System.Design.For.Big.Data.Pipelines.part1.rar.htmlhttps://rapidgator.net/file/d71000592e6cf9ecd32fe14485039e99/cpavz.System.Design.For.Big.Data.Pipelines.part2.rar.htmluploadgig.com:https://uploadgig.com/file/download/b8c8312ee3886eA7/cpavz.System.Design.For.Big.Data.Pipelines.part1.rarhttps://uploadgig.com/file/download/b210a48fbfc2fD08/cpavz.System.Design.For.Big.Data.Pipelines.part2.rarnitroflare.com:https://nitroflare.com/view/A13AA54CCA54F72/cpavz.System.Design.For.Big.Data.Pipelines.part1.rarhttps://nitroflare.com/view/C5A2B6802887A8E/cpavz.System.Design.For.Big.Data.Pipelines.part2.rar1dl.net:https://1dl.net/eshzgaxlmvmp/cpavz.System.Design.For.Big.Data.Pipelines.part1.rarhttps://1dl.net/shtanpe6wdx3/cpavz.System.Design.For.Big.Data.Pipelines.part2.rar Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now