I hate mocking
Download as pptx, pdf1 like332 views
Steve Loughran expresses dislike for mocking in tests because mock code reflects assumptions rather than reality. Any changes to the real code can break the tests, leading to false positives. Test failures are often "fixed" by editing the test or mock code, which could hide real problems. He proposes avoiding mock tests and instead adding functional tests against real infrastructure with fault injection for integration testing.
1 of 8
Download to read offline
Ad
Recommended
Python: the coolest is yet to come
Python: the coolest is yet to comePablo Enfedaque
?
The document provides an overview of new features in Python discussed at the PDI DevCon 2013, including the super function, yield from statement, exception chaining, single-dispatch generic functions, and virtual environments. It includes code examples and references to related PEPs that introduce these features. The author, Pablo Enfedaque, emphasizes practical implementations alongside theoretical explanations.Unit testing with mock libs
Unit testing with mock libsValentin Kolesnikov
?
PowerMock is an open source Java framework that allows mocking of static methods, final classes, and private methods. It extends EasyMock and JUnit to enable these additional types of mocking. The document provides details on how to add PowerMock dependencies to Maven and Ant projects, examples of unit testing with and without mocks, and techniques like capturing arguments and emulating code in mock objects. It also discusses skipping static initialization and the F.I.R.S.T. properties that unit tests should have.Implicit and Explicit waits in Selenium WebDriwer, how to.
Implicit and Explicit waits in Selenium WebDriwer, how to.Yaroslav Pernerovsky
?
This document discusses the proper use of implicit and explicit waits in Selenium. It notes that implicit waits should be used to wait for element presence, while explicit waits should not be used for this. Implicit wait timeouts should be set lower than explicit wait timeouts. Using implicit and explicit waits together requires care to avoid conflicts between the timeouts. The document provides examples of how to set up waits correctly and what exceptions may occur in different scenarios. It also briefly mentions alternatives to WebDriver waits like using Selenide.WebDriver Waits
WebDriver WaitsYaroslav Pernerovsky
?
This document discusses different types of waits that can be used in Selenium tests: implicit waits, explicit waits, and timeouts. Implicit waits set a global wait time for all findElement calls, while explicit waits allow waiting for specific conditions. It is recommended to use explicit waits over implicit waits and to set implicit timeouts lower than explicit waits. Care must be taken when waiting for elements to not be present.Automation puzzlers
Automation puzzlersYaroslav Pernerovsky
?
This document contains a series of test automation questions and answers related to Selenium and Selenide. It discusses behaviors like opening URLs, handling exceptions, implicit vs explicit waits, element locators, and more. The questions are multiple choice and include the correct answers. Examples of code are provided to demonstrate the concepts covered by each question.Martin Anderson - threads v actors
Martin Anderson - threads v actorsbloodredsun
?
This lightning talk discusses replacing threads with actors for making concurrent service calls in web applications. It describes how actors provide an alternative concurrency model that can be more resource efficient than threads. The talk shows how to implement a master/worker pattern using Akka actors to dispatch work to remote services in parallel from a servlet, collecting the responses to render the page. Key advantages of the actor model include easier reasoning about concurrency and built-in supervision to restart failed actors.Developed your first Xamarin.Forms Application
Developed your first Xamarin.Forms ApplicationCheah Eng Soon
?
This document provides instructions for building a mobile app with Xamarin.Forms that allows accessing location data, making phone calls and sending SMS, and integrating with the HockeyApp crash reporting service. It covers creating a basic Xamarin.Forms project, adding layouts, controls and event handlers, integrating plugins for geolocation, messaging, and crash reporting, and deploying to iOS, Android and UWP.Testing with Kotlin
Testing with KotlinRobert Fletcher
?
This document summarizes techniques for writing iterative and parameterized tests in Kotlin using the Spock and Spek testing frameworks. It discusses generating test data, tokenizing test names, nested and tabular iteration, mocking with Mockito, test class hierarchies (TCKs), and proposes diagrammed assertions to visualize test failures. The final section announces an upcoming Bay Area Kotlin User Group meetup on Netflix engineering tools and Kotlin Spring Cloud extensions.The Ring programming language version 1.2 book - Part 28 of 84
The Ring programming language version 1.2 book - Part 28 of 84Mahmoud Samir Fayed
?
This document describes using the Ring programming language to build web applications using CGI (Common Gateway Interface). It provides instructions on configuring the Apache web server to support Ring CGI scripts. It then demonstrates a simple "Hello World" CGI program in Ring. Further sections explain how to use the Web Library module to simplify Ring web development. Features of the Web Library like generating HTML, handling HTTP GET/POST, and templates are overviewed. An example HTTP GET form is provided to illustrate collecting user input.Easy Button
Easy ButtonAdam Dale
?
This document contains code for a Java GUI application called EasyGUI that allows users to generate automation results for order processing. It contains classes like EasyGUI, EasyButton, and BatchEngine that create a GUI, handle button clicks, and execute automation batches. When the submit button is clicked, it validates the order ID, runs batches to get conversation IDs, account numbers, and CSIXML files, and displays the results. It uses GridBagLayout to arrange components on the main panel and stores selection values in variables like orderID and accountNumber to pass to batch processing methods.Android App Development - 07 Threading
Android App Development - 07 ThreadingDiego Grancini
?
The document discusses Android threading and processes. It explains that by default all components of an application run in the same process and thread, but components can be defined to run in separate processes. It describes how the system manages processes and process importance levels. It also discusses how to handle long operations to avoid blocking the UI thread, including using AsyncTask and Loaders to run operations asynchronously.JS and patterns
JS and patternsDavid Rodenas
?
The document discusses the Model-View-Controller (MVC) design pattern in JavaScript, emphasizing the principles of high cohesion and low coupling. It illustrates the roles of models, views, and controllers through examples, highlighting best practices and common pitfalls in implementing this pattern. The author cautions against overcomplicating designs while encouraging the effective reuse of components.VisualStudio2012-WhatsNew-TechEd_v3-9
VisualStudio2012-WhatsNew-TechEd_v3-9SSW
?
Microsoft is replacing its Metro UI style naming with "Modern UI" to reduce confusion with other Metro trademarks. The new Modern UI style will be the design language for Windows 8 and Windows Phone. It aims to create a consistent experience across devices with its focus on touch, tiles, typography, motion and content before chrome. Microsoft is pushing the Modern UI style with all its first-party apps and also making guidelines and tools available for third-party developers.¶ÅÔöÇ¿-¹ó±ô±ð³æ3×é¼þÉúÃüÖÜÆÚ
¶ÅÔöÇ¿-¹ó±ô±ð³æ3×é¼þÉúÃüÖÜÆÚÔöÇ¿ ¶Å
?
This document discusses the Adobe Flex component lifecycle, which includes three main phases - birth, life, and death.
The birth phase involves construction, configuration, attachment, and initialization of a component. Configuration assigns property values before attachment and initialization. Attachment adds a component to the display list.
The life phase revolves around the invalidation/validation process, where properties and sizing are invalidated and then validated through commitProperties(), measure(), and updateDisplayList() calls. Interaction occurs through event handling in this phase.
The death phase covers detachment, where a component is removed from the display list, and garbage collection, where detached components with no references are freed from memory. The document stressesTesting in android
Testing in androidjtrindade
?
The document discusses testing in Android applications. It recommends moving as much logic as possible to the Java Virtual Machine (JVM) to allow for easier unit testing. This includes business logic, models, and network code using Retrofit. It also suggests using Espresso for UI testing on Android and monkeyrunner for basic OS interaction testing. The document acknowledges testing on Android can be painful due to speed and fragmentation issues, and proposes compromises like helping quality assurance engineers with tools like the Bee library.ASP.NET MVC Internals
ASP.NET MVC InternalsVitaly Baum
?
The document discusses ASP.NET MVC internals including routing, controllers, views, and view engines. It explains how requests come to controllers through routing and routing handlers. It also describes how controllers execute actions and return results, and how view engines render views using view engines and templates. The document provides examples of customizing routing handlers, controllers, dependency injection, and view engines in ASP.NET MVC.Intro to Retrofit 2 and RxJava2
Intro to Retrofit 2 and RxJava2Fabio Collini
?
This document provides an introduction to Retrofit 2 and RxJava 2. It discusses how Retrofit allows turning a REST API into a Java interface and how RxJava implements reactive programming with observable sequences. It demonstrates making synchronous and asynchronous HTTP requests with Retrofit and handling the "callback hell" problem with RxJava. Operators like map, flatMap, zip and concatMap are shown to transform and combine observables. The document emphasizes that RxJava contains over 400 methods and discusses additional RxJava concepts like error handling and threading.Easy REST APIs with Jersey and RestyGWT
Easy REST APIs with Jersey and RestyGWTDavid Chandler
?
The document provides an overview of using RestyGWT with Jersey to create REST APIs, highlighting its simplicity, reduced boilerplate, and enhanced testing capabilities. It includes code examples for creating service interfaces, managing CRUD operations, handling authentication, and implementing error handling. Additionally, it discusses configuration requirements such as dependencies in POM and web.xml files for Jersey integration.Reliability and Reslience
Reliability and ReslienceDonald Belcham
?
The document discusses various techniques for building reliable systems including tuples, null objects, circuit breakers, and retries. Tuples can return multiple values without specifying a type. Null objects avoid null checks by providing a default object. Circuit breakers monitor dependencies and fail fast when they are unavailable, preventing cascading failures. Retries can help recover from transient errors but need limits to avoid denial of service attacks.Testing with Containers
Testing with ContainersRomain Baugue
?
The document contains unit tests for a Go application utilizing SQL mock for database interactions, including functions to update stats and poll data. It defines various service configurations in a Docker Compose file and includes an example of a test case using the mysqltest package for managing a MySQL database. Additionally, there are set-up procedures and expectations for handling errors and verifying outputs in test scenarios.Android architecture component - FbCircleDev Yogyakarta Indonesia
Android architecture component - FbCircleDev Yogyakarta IndonesiaPratama Nur Wijaya
?
The document discusses Android Architecture Components (AAC). It describes AAC as a library, guidelines and standards that aim to standardize architecture and reduce boilerplate code. It discusses key components of AAC like Lifecycles, LiveData, ViewModel and Room that help address issues like lifecycle handling, data persistence and offline support. It provides code examples to demonstrate how these components can be used to build more robust Android applications that properly handle lifecycles and data.Hadoop Vectored IO
Hadoop Vectored IOSteve Loughran
?
Steve Loughran discusses the benefits of Hadoop's vectored I/O, which significantly speeds up data retrieval from cloud storage by optimizing read operations for columnar formats like ORC and Parquet. The new API allows for parallel data fetching, decreasing query runtimes in environments like S3 by up to 70%. Vectored I/O will be included in Hadoop version 3.3.5, and efforts are underway to support it in ORC and Parquet libraries.The age of rename() is over
The age of rename() is overSteve Loughran
?
The document discusses how storage models need to evolve as the underlying technologies change. Object stores like S3 provide scale and high availability but lack semantics and performance of file systems. Non-volatile memory also challenges current models. The POSIX file system metaphor is ill-suited for object stores and NVM. SQL provides an alternative that abstracts away the underlying complexities, leaving just object-relational mapping and transaction isolation to address. The document examines renaming operations, asynchronous I/O, and persistent in-memory data structures as examples of areas where new models may be needed.What does Rename Do: (detailed version)
What does Rename Do: (detailed version)Steve Loughran
?
The document discusses the challenges and methodologies related to state persistence and recovery in Hadoop and cloud storage systems, particularly focusing on the `rename()` function and its implications for efficient data management. It highlights the need for a more streamlined approach to handle state in distributed environments and the potential shifts towards new data storage paradigms, including non-volatile memory. Furthermore, it raises questions about the future of file and data handling APIs in light of changing technology landscapes.Put is the new rename: San Jose Summit Edition
Put is the new rename: San Jose Summit EditionSteve Loughran
?
The document discusses the evolution of cloud storage in the context of Hadoop and various cloud services, focusing on the importance of object stores like S3, Azure Data Lake, and Google Cloud Storage for analytics applications. It highlights advancements such as S3Guard for metadata consistency, Azure's adaptive seek algorithm, and the introduction of SQL querying capabilities on S3. The presentation concludes with a comparison of the security features and usability of different cloud storage services, emphasizing the growing complexity and offerings in cloud infrastructure.@Dissidentbot: dissent will be automated!
@Dissidentbot: dissent will be automated!Steve Loughran
?
This document outlines the development history of the Dissident bot from its creation in January 2017 to June 2018. It discusses improvements made over time including adding conversation mode, a TODO item to develop a Chomsky-Type-1 Grammar AI, and fixing a bug where conversation mode would spam the bot's username. It also provides details on the bot's configuration settings and methods used to detect spam, bots, and politicans spreading misinformation.PUT is the new rename()
PUT is the new rename()Steve Loughran
?
The document discusses the significance of cloud storage for analytic applications, detailing various object storage solutions such as Azure WASB, Azure Data Lake, and Amazon S3. It highlights features like S3Guard for metadata consistency, the challenges of using CSV files, and introduces S3 Select for issuing SQL queries directly on S3 data. The document concludes that while all cloud infrastructures offer object stores, AWS S3 is the least suited as a destination, despite available solutions.Extreme Programming Deployed
Extreme Programming DeployedSteve Loughran
?
This document discusses the principles and practices of Extreme Programming (XP), an agile software development process. It describes XP as an intense, test-centric programming process focused on projects with high rates of change. Key practices include pair programming, test-driven development, planning with user stories and tasks, doing the simplest thing that could work, and refactoring code aggressively. Problems may include short-term "hill-climbing" solutions and risks of fundamental design errors. The document provides additional resources on XP and notes that the day's session will involve practicing XP techniques through pair programming.Testing
TestingSteve Loughran
?
The document discusses the challenges and methodologies involved in software testing, particularly emphasizing the importance of test-driven development (TDD) and formal methods for proving code correctness. It covers various testing frameworks and strategies, such as JUnit for unit testing and approaches for system and performance testing. Additionally, it highlights issues related to interop testing, dependency management, and the significance of having tests for any feature or standard.What does rename() do?
What does rename() do?Steve Loughran
?
The document discusses the function of 'rename()' in data storage and state management, emphasizing its importance for safely persisting and recovering application states. It explores storage definitions and contrasts different file systems, such as HDFS and Amazon S3, highlighting their varying capabilities for tasks like renaming and committing changes. The presentation also touches on future trends in storage technology, including non-volatile memory and the challenges that arise with data consistency and application state management.More Related Content
What's hot (13)
The Ring programming language version 1.2 book - Part 28 of 84
The Ring programming language version 1.2 book - Part 28 of 84Mahmoud Samir Fayed
?
This document describes using the Ring programming language to build web applications using CGI (Common Gateway Interface). It provides instructions on configuring the Apache web server to support Ring CGI scripts. It then demonstrates a simple "Hello World" CGI program in Ring. Further sections explain how to use the Web Library module to simplify Ring web development. Features of the Web Library like generating HTML, handling HTTP GET/POST, and templates are overviewed. An example HTTP GET form is provided to illustrate collecting user input.Easy Button
Easy ButtonAdam Dale
?
This document contains code for a Java GUI application called EasyGUI that allows users to generate automation results for order processing. It contains classes like EasyGUI, EasyButton, and BatchEngine that create a GUI, handle button clicks, and execute automation batches. When the submit button is clicked, it validates the order ID, runs batches to get conversation IDs, account numbers, and CSIXML files, and displays the results. It uses GridBagLayout to arrange components on the main panel and stores selection values in variables like orderID and accountNumber to pass to batch processing methods.Android App Development - 07 Threading
Android App Development - 07 ThreadingDiego Grancini
?
The document discusses Android threading and processes. It explains that by default all components of an application run in the same process and thread, but components can be defined to run in separate processes. It describes how the system manages processes and process importance levels. It also discusses how to handle long operations to avoid blocking the UI thread, including using AsyncTask and Loaders to run operations asynchronously.JS and patterns
JS and patternsDavid Rodenas
?
The document discusses the Model-View-Controller (MVC) design pattern in JavaScript, emphasizing the principles of high cohesion and low coupling. It illustrates the roles of models, views, and controllers through examples, highlighting best practices and common pitfalls in implementing this pattern. The author cautions against overcomplicating designs while encouraging the effective reuse of components.VisualStudio2012-WhatsNew-TechEd_v3-9
VisualStudio2012-WhatsNew-TechEd_v3-9SSW
?
Microsoft is replacing its Metro UI style naming with "Modern UI" to reduce confusion with other Metro trademarks. The new Modern UI style will be the design language for Windows 8 and Windows Phone. It aims to create a consistent experience across devices with its focus on touch, tiles, typography, motion and content before chrome. Microsoft is pushing the Modern UI style with all its first-party apps and also making guidelines and tools available for third-party developers.¶ÅÔöÇ¿-¹ó±ô±ð³æ3×é¼þÉúÃüÖÜÆÚ
¶ÅÔöÇ¿-¹ó±ô±ð³æ3×é¼þÉúÃüÖÜÆÚÔöÇ¿ ¶Å
?
This document discusses the Adobe Flex component lifecycle, which includes three main phases - birth, life, and death.
The birth phase involves construction, configuration, attachment, and initialization of a component. Configuration assigns property values before attachment and initialization. Attachment adds a component to the display list.
The life phase revolves around the invalidation/validation process, where properties and sizing are invalidated and then validated through commitProperties(), measure(), and updateDisplayList() calls. Interaction occurs through event handling in this phase.
The death phase covers detachment, where a component is removed from the display list, and garbage collection, where detached components with no references are freed from memory. The document stressesTesting in android
Testing in androidjtrindade
?
The document discusses testing in Android applications. It recommends moving as much logic as possible to the Java Virtual Machine (JVM) to allow for easier unit testing. This includes business logic, models, and network code using Retrofit. It also suggests using Espresso for UI testing on Android and monkeyrunner for basic OS interaction testing. The document acknowledges testing on Android can be painful due to speed and fragmentation issues, and proposes compromises like helping quality assurance engineers with tools like the Bee library.ASP.NET MVC Internals
ASP.NET MVC InternalsVitaly Baum
?
The document discusses ASP.NET MVC internals including routing, controllers, views, and view engines. It explains how requests come to controllers through routing and routing handlers. It also describes how controllers execute actions and return results, and how view engines render views using view engines and templates. The document provides examples of customizing routing handlers, controllers, dependency injection, and view engines in ASP.NET MVC.Intro to Retrofit 2 and RxJava2
Intro to Retrofit 2 and RxJava2Fabio Collini
?
This document provides an introduction to Retrofit 2 and RxJava 2. It discusses how Retrofit allows turning a REST API into a Java interface and how RxJava implements reactive programming with observable sequences. It demonstrates making synchronous and asynchronous HTTP requests with Retrofit and handling the "callback hell" problem with RxJava. Operators like map, flatMap, zip and concatMap are shown to transform and combine observables. The document emphasizes that RxJava contains over 400 methods and discusses additional RxJava concepts like error handling and threading.Easy REST APIs with Jersey and RestyGWT
Easy REST APIs with Jersey and RestyGWTDavid Chandler
?
The document provides an overview of using RestyGWT with Jersey to create REST APIs, highlighting its simplicity, reduced boilerplate, and enhanced testing capabilities. It includes code examples for creating service interfaces, managing CRUD operations, handling authentication, and implementing error handling. Additionally, it discusses configuration requirements such as dependencies in POM and web.xml files for Jersey integration.Reliability and Reslience
Reliability and ReslienceDonald Belcham
?
The document discusses various techniques for building reliable systems including tuples, null objects, circuit breakers, and retries. Tuples can return multiple values without specifying a type. Null objects avoid null checks by providing a default object. Circuit breakers monitor dependencies and fail fast when they are unavailable, preventing cascading failures. Retries can help recover from transient errors but need limits to avoid denial of service attacks.Testing with Containers
Testing with ContainersRomain Baugue
?
The document contains unit tests for a Go application utilizing SQL mock for database interactions, including functions to update stats and poll data. It defines various service configurations in a Docker Compose file and includes an example of a test case using the mysqltest package for managing a MySQL database. Additionally, there are set-up procedures and expectations for handling errors and verifying outputs in test scenarios.Android architecture component - FbCircleDev Yogyakarta Indonesia
Android architecture component - FbCircleDev Yogyakarta IndonesiaPratama Nur Wijaya
?
The document discusses Android Architecture Components (AAC). It describes AAC as a library, guidelines and standards that aim to standardize architecture and reduce boilerplate code. It discusses key components of AAC like Lifecycles, LiveData, ViewModel and Room that help address issues like lifecycle handling, data persistence and offline support. It provides code examples to demonstrate how these components can be used to build more robust Android applications that properly handle lifecycles and data.More from Steve Loughran (20)
Hadoop Vectored IO
Hadoop Vectored IOSteve Loughran
?
Steve Loughran discusses the benefits of Hadoop's vectored I/O, which significantly speeds up data retrieval from cloud storage by optimizing read operations for columnar formats like ORC and Parquet. The new API allows for parallel data fetching, decreasing query runtimes in environments like S3 by up to 70%. Vectored I/O will be included in Hadoop version 3.3.5, and efforts are underway to support it in ORC and Parquet libraries.The age of rename() is over
The age of rename() is overSteve Loughran
?
The document discusses how storage models need to evolve as the underlying technologies change. Object stores like S3 provide scale and high availability but lack semantics and performance of file systems. Non-volatile memory also challenges current models. The POSIX file system metaphor is ill-suited for object stores and NVM. SQL provides an alternative that abstracts away the underlying complexities, leaving just object-relational mapping and transaction isolation to address. The document examines renaming operations, asynchronous I/O, and persistent in-memory data structures as examples of areas where new models may be needed.What does Rename Do: (detailed version)
What does Rename Do: (detailed version)Steve Loughran
?
The document discusses the challenges and methodologies related to state persistence and recovery in Hadoop and cloud storage systems, particularly focusing on the `rename()` function and its implications for efficient data management. It highlights the need for a more streamlined approach to handle state in distributed environments and the potential shifts towards new data storage paradigms, including non-volatile memory. Furthermore, it raises questions about the future of file and data handling APIs in light of changing technology landscapes.Put is the new rename: San Jose Summit Edition
Put is the new rename: San Jose Summit EditionSteve Loughran
?
The document discusses the evolution of cloud storage in the context of Hadoop and various cloud services, focusing on the importance of object stores like S3, Azure Data Lake, and Google Cloud Storage for analytics applications. It highlights advancements such as S3Guard for metadata consistency, Azure's adaptive seek algorithm, and the introduction of SQL querying capabilities on S3. The presentation concludes with a comparison of the security features and usability of different cloud storage services, emphasizing the growing complexity and offerings in cloud infrastructure.@Dissidentbot: dissent will be automated!
@Dissidentbot: dissent will be automated!Steve Loughran
?
This document outlines the development history of the Dissident bot from its creation in January 2017 to June 2018. It discusses improvements made over time including adding conversation mode, a TODO item to develop a Chomsky-Type-1 Grammar AI, and fixing a bug where conversation mode would spam the bot's username. It also provides details on the bot's configuration settings and methods used to detect spam, bots, and politicans spreading misinformation.PUT is the new rename()
PUT is the new rename()Steve Loughran
?
The document discusses the significance of cloud storage for analytic applications, detailing various object storage solutions such as Azure WASB, Azure Data Lake, and Amazon S3. It highlights features like S3Guard for metadata consistency, the challenges of using CSV files, and introduces S3 Select for issuing SQL queries directly on S3 data. The document concludes that while all cloud infrastructures offer object stores, AWS S3 is the least suited as a destination, despite available solutions.Extreme Programming Deployed
Extreme Programming DeployedSteve Loughran
?
This document discusses the principles and practices of Extreme Programming (XP), an agile software development process. It describes XP as an intense, test-centric programming process focused on projects with high rates of change. Key practices include pair programming, test-driven development, planning with user stories and tasks, doing the simplest thing that could work, and refactoring code aggressively. Problems may include short-term "hill-climbing" solutions and risks of fundamental design errors. The document provides additional resources on XP and notes that the day's session will involve practicing XP techniques through pair programming.Testing
TestingSteve Loughran
?
The document discusses the challenges and methodologies involved in software testing, particularly emphasizing the importance of test-driven development (TDD) and formal methods for proving code correctness. It covers various testing frameworks and strategies, such as JUnit for unit testing and approaches for system and performance testing. Additionally, it highlights issues related to interop testing, dependency management, and the significance of having tests for any feature or standard.What does rename() do?
What does rename() do?Steve Loughran
?
The document discusses the function of 'rename()' in data storage and state management, emphasizing its importance for safely persisting and recovering application states. It explores storage definitions and contrasts different file systems, such as HDFS and Amazon S3, highlighting their varying capabilities for tasks like renaming and committing changes. The presentation also touches on future trends in storage technology, including non-volatile memory and the challenges that arise with data consistency and application state management.Dancing Elephants: Working with Object Storage in Apache Spark and Hive
Dancing Elephants: Working with Object Storage in Apache Spark and HiveSteve Loughran
?
The document discusses the integration of object storage with Apache Spark and Hive, emphasizing its significance for agile cloud applications and data management. It highlights the evolution of cloud storage, the challenges posed by eventual consistency, and performance issues related to S3. Additionally, it addresses the improvements in Hadoop's performance on Azure and the implementation of various optimizations for better handling of object storage in data processing.Apache Spark and Object Stores ¡ªfor London Spark User Group
Apache Spark and Object Stores ¡ªfor London Spark User GroupSteve Loughran
?
The document discusses the integration of Apache Spark with object stores, emphasizing their importance as persistent storage solutions for cloud applications. It outlines technical considerations such as classpath configuration, credentials management, and performance challenges associated with object stores like Amazon S3 and Azure Blob Storage. Additionally, it highlights best practices for working with data, including using Spark SQL for querying and managing data formats like Parquet and ORC.Spark Summit East 2017: Apache spark and object stores
Spark Summit East 2017: Apache spark and object storesSteve Loughran
?
The document discusses the integration of Apache Spark with object stores, focusing on the necessary configurations, challenges, and performance optimization strategies. It highlights the importance of properly setting the classpath, managing credentials securely, and addressing challenges related to commitment and performance in data processing. Additionally, it introduces topics like S3Guard for metadata consistency and server-side encryption for data security.Hadoop, Hive, Spark and Object Stores
Hadoop, Hive, Spark and Object StoresSteve Loughran
?
This document discusses optimizing Apache Hadoop's performance on cloud platforms, specifically focusing on enhancements for data storage and retrieval in object stores like AWS S3 and Azure Blob Storage. It covers best practices for using Hadoop with various data formats, performance benchmarks, and the improvements in read/write operations. Additionally, it highlights the commitment challenges in distributed file systems and presents strategies for efficient data processing.Apache Spark and Object Stores
Apache Spark and Object StoresSteve Loughran
?
This document discusses using Apache Spark with object stores like Amazon S3 and Microsoft Azure Blob Storage. It covers challenges around classpath configuration, credentials, code examples, and performance commitments when using these storage systems. Key points include using Hadoop connectors like S3A and WASB, configuring credentials through properties or environment variables, and tuning Spark for object store performance and consistency.Household INFOSEC in a Post-Sony Era
Household INFOSEC in a Post-Sony EraSteve Loughran
?
This document discusses household information security risks in the post-Sony era. It identifies key risks like data integrity, privacy, and availability issues. It provides examples of vulnerabilities across different devices and platforms like LG TVs, iPads, iPhones, and PS4s. It also discusses vulnerabilities in software like Firefox, Chrome, Internet Explorer, Flash, and SparkContext. It recommends approaches to address these risks like using containers for isolation, validating packages with PGP to ensure authentication, and enabling audit logs.Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 edition
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 editionSteve Loughran
?
The document discusses the integration of Kerberos authentication in modern Hadoop clusters, emphasizing its role in securing access to services through various ticket and token mechanisms. It explains how user authentication and access control are managed via principals and delegation tokens, as well as the underlying architecture necessary for secure communication. Additionally, it touches on challenges and error handling associated with utilizing Kerberos in Hadoop environments.Hadoop and Kerberos: the Madness Beyond the Gate
Hadoop and Kerberos: the Madness Beyond the GateSteve Loughran
?
The document provides a comprehensive overview of integrating Kerberos with Hadoop security, detailing key concepts such as authentication, service principals, and delegation tokens. It discusses the architectural elements necessary for securing Hadoop clusters using Kerberos, user group information, and the importance of proper configuration. Additionally, it addresses error handling and debugging techniques related to Kerberos in the context of Hadoop applications.ºÝºÝߣr: Applications on YARN
ºÝºÝߣr: Applications on YARNSteve Loughran
?
This document discusses Apache ºÝºÝߣr, which allows applications to be deployed and managed on Apache Hadoop YARN. ºÝºÝߣr uses an Application Master, agents, and scripts to deploy applications defined in an XML package. The Application Master keeps applications in a desired state across YARN containers and handles lifecycle commands like start, stop, and scaling. ºÝºÝߣr integrates with Apache Ambari for graphical management and configuration of applications on YARN.YARN Services
YARN ServicesSteve Loughran
?
This document discusses YARN services in Hadoop, which allow long-lived applications to run within a Hadoop cluster. YARN (Yet Another Resource Negotiator) provides an operating system-like platform for data processing by allowing various applications to share cluster resources. The document outlines features for long-lived services in YARN, including log aggregation, Kerberos token renewal, and service registration/discovery. It also discusses how Hadoop 2.6 and later versions implement these features to enable long-running applications that can withstand failures.Datacentre stack
Datacentre stackSteve Loughran
?
This document appears to be a list of terms related to computer science and data systems including HDFS, YARN, Kernighan, Cerf, Lamport, Codd, Knuth and SQL. It references people, technologies and concepts but provides no additional context or explanation.Ad
Recently uploaded (20)
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
CodeCleaner: Mitigating Data Contamination for LLM Benchmarkingarabelatso
?
The pdf-version of "CodeCleaner: Mitigating Data Contamination for LLM Benchmarking" presented in Internetware 2025.Best Practice for LLM Serving in the Cloud
Best Practice for LLM Serving in the CloudAlluxio, Inc.
?
Alluxio Webinar
June 17, 2025
For more Alluxio Events: https://www.alluxio.io/events/
Speaker:
Nilesh Agarwal (Co-founder & CTO @ Inferless)
Nilesh Agarwal, co-founder & CTO at Inferless, shares insights on accelerating LLM inference in the cloud using Alluxio, tackling key bottlenecks like slow model weight loading from S3 and lengthy container startup time. Inferless uses Alluxio as a three-tier cache system that dramatically cuts model load time by 10x.Introduction to Agile Frameworks for Product Managers.pdf
Introduction to Agile Frameworks for Product Managers.pdfAli Vahed
?
As a Product Manager, having a solid understanding of Agile frameworks is essential, whether you are joining an established team or helping shape a new one.
Agile is not just a delivery method; it is a mind set that directly impacts how products are built, iterated, and brought to market. Knowing the strengths and trade-offs of each framework enables you to collaborate more effectively with teams, foster alignment, and drive meaningful outcomes.
Simply put, Agile fluency helps you lead with clarity in fast-paced, ever-changing environments.
The next few slides are simple ¡®STUDY CARDS¡¯ to help you learn or refresh your understanding of a few popular Agile frameworks.Heat Treatment Process Automation in India
Heat Treatment Process Automation in IndiaReckers Mechatronics
?
Automate your heat treatment processes for superior precision, consistency, and cost savings. Explore solutions for furnaces, quench systems. Heat treatment is a critical manufacturing process that alters the microstructure and properties of materials, typically metals, to achieve desired characteristics such as hardness, strength, ductility, and wear resistance. Test Case Design Techniques ¨C Practical Examples & Best Practices in Software...
Test Case Design Techniques ¨C Practical Examples & Best Practices in Software...Muhammad Fahad Bashir
?
This presentation was part of the SQA & PM Bootcamp, where I served as a trainer. It focuses on effective test case design techniques, blending theoretical knowledge with practical application, especially useful for manual testers and QA beginners.
Video Lecture Recording : https://www.facebook.com/share/v/1Z44DiXN5v/
What¡¯s Covered:
? Definition and Purpose of Test Case Design
? Importance of Structured Test Design
? Test Case Design Techniques Overview:
? Black Box Techniques
? White Box Techniques
? Experience-Based Techniques
? Manual Test Case Design in Detail
? Real-World Example: Login Page Test Case
? Black Box Methods Explained with Examples:
Equivalence Partitioning (EP)
Boundary Value Analysis (BVA)
Difference Between EP & BVA
? Decision Table Testing
? Experience-Based Testing: Error Guessing, Exploratory Testing
? White Box Techniques (Introductory Overview)
? Much More to Help You Build Strong Manual Testing Skills
This resource is especially helpful for students, aspiring QA professionals, and junior testers looking to master test design fundamentals with clarity and practical insight.Folding Cheat Sheet # 9 - List Unfolding ?????? as the Computational Dual of ...
Folding Cheat Sheet # 9 - List Unfolding ?????? as the Computational Dual of ...Philip Schwarz
?
For newer version see https://fpilluminated.org/deck/264
In this deck we look at the following:
* How unfolding lists is the computational dual of folding lists
* Different variants of the function for unfolding lists
* How they relate to the iterate functionWhich Hiring Management Tools Offer the Best ROI?
Which Hiring Management Tools Offer the Best ROI?HireME
?
In today's cost-focused world, companies must assess not just how they hire, but which hiring management tools deliver the best value for money. Learn more in PDF!Digital Transformation: Automating the Placement of Medical Interns
Digital Transformation: Automating the Placement of Medical InternsSafe Software
?
Discover how the Health Service Executive (HSE) in Ireland has leveraged FME to implement an automated solution for its ¡°Order of Merit¡± process, which assigns medical interns to their preferred placements based on applications and academic performance. This presentation will showcase how FME validates applicant data, including placement preferences and academic results, ensuring both accuracy and eligibility. It will also explore how FME automates the matching process, efficiently assigning placements to interns and distributing offer letters. Beyond this, FME actively monitors responses and dynamically reallocates placements in line with the order of merit when offers are accepted or declined. The solution also features a self-service portal (FME Flow Gallery App), enabling administrators to manage the entire process seamlessly, make adjustments, generate reports, and maintain audit logs. Additionally, the system upholds strict data security and governance, utilising FME to encrypt data both at rest and in transit.MOVIE RECOMMENDATION SYSTEM, UDUMULA GOPI REDDY, Y24MC13085.pptx
MOVIE RECOMMENDATION SYSTEM, UDUMULA GOPI REDDY, Y24MC13085.pptxMaharshi Mallela
?
Movie recommendation system is a software application or algorithm designed to suggest movies to users based on their preferences, viewing history, or other relevant factors. The primary goal of such a system is to enhance user experience by providing personalized and relevant movie suggestions.Decipher SEO Solutions for your startup needs.
Decipher SEO Solutions for your startup needs.mathai2
?
A solution deck that gives you an idea of how you can use Decipher SEO to target keywords, build authority and generate high ranking content.
With features like images to product you can create a E-commerce pipeline that is optimized to help your store rank.
With integrations with shopify, woocommerce and wordpress theres a seamless way get your content to your website or storefront.
View more at decipherseo.comOpen Source Software Development Methods
Open Source Software Development MethodsVICTOR MAESTRE RAMIREZ
?
Open Source Software Development MethodsThe Anti-Masterclass Live - Peak of Data & AI 2025
The Anti-Masterclass Live - Peak of Data & AI 2025Safe Software
?
Even the most experienced FME users still make mistakes, and that includes us. And that's okay because making mistakes makes you better. Come join and learn from us as we run down the most common ones so you can lift yourself to a higher level of FME workflow development. Based on decades of experience using and teaching FME, our tips can save you a lot of time, frustration and bad data.On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?
On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?Hassan Abid
?
As mobile hardware becomes more powerful, the promise of running advanced AI directly on-device is closer than ever. With Google¡¯s latest on-device model Gemini Nano, accessible through the new ML Kit GenAI APIs and AI Edge SDK, alongside the open-source Gemma-3n models, developers can now integrate lightweight, multimodal intelligence that works even without an internet connection. But does this mean we no longer need cloud-based AI? This session explores the practical trade-offs between on-device and cloud AI for mobile apps.Simplify Task, Team, and Project Management with Orangescrum Work
Simplify Task, Team, and Project Management with Orangescrum WorkOrangescrum
?
Streamline project workflows, team collaboration, and time tracking with orangescrum work, your all-in-one tool for smarter and faster project execution.arctitecture application system design os dsa
arctitecture application system design os dsaza241967
?
wow amazing see and enjoy for this the best pdf slideshow you could see random shit is what i do best Zoneranker¡¯s Digital marketing solutions
Zoneranker¡¯s Digital marketing solutionsreenashriee
?
Zoneranker offers expert digital marketing services tailored for businesses in Theni. From SEO and PPC to social media and content marketing, we help you grow online. Partner with us to boost visibility, leads, and sales.
Streamlining CI/CD with FME Flow: A Practical Guide
Streamlining CI/CD with FME Flow: A Practical GuideSafe Software
?
Join us as we explore how to deploy a fault-tolerant FME Flow installation using FME Flow¡¯s IaC templates alongside Terraform, AWS, and GitHub in a CI/CD workflow. This session will also cover how to leverage FME¡¯s CI/CD capabilities for FME Flow upgrades and FME object migration between different environments, ensuring seamless automation and scalability.IObit Driver Booster Pro 12 Crack Latest Version Download
IObit Driver Booster Pro 12 Crack Latest Version Downloadpcprocore
?
?????:???? ???? & ????? ???? ?????? ??? ????? https://pcprocore.com/ ??
IObit Driver Booster Pro is the solution. It automatically downloads and updates drivers with just one click, avoiding hardware failures, system instability, and security vulnerabilities.How Automation in Claims Handling Streamlined Operations
How Automation in Claims Handling Streamlined OperationsInsurance Tech Services
?
Claims processing automation reduces manual work, cuts errors, and speeds up resolutions. Using AI, RPA, and OCR, it enables faster data handling and smoother workflows¡ªboosting efficiency and customer satisfaction. Visit https://www.damcogroup.com/insurance/claims-processing-automation for more details! Emvigo Capability Deck 2025: Accelerating Innovation Through Intelligent Soft...
Emvigo Capability Deck 2025: Accelerating Innovation Through Intelligent Soft...Emvigo Technologies
?
Welcome to Emvigo¡¯s Capability Deck for 2025 ¨C a comprehensive overview of how we help businesses launch, scale, and sustain digital success through cutting-edge technology solutions.
With 13+ years of experience and a presence across the UK, UAE, and India, Emvigo is an ISO-certified software company trusted by leading global organizations including the NHS, Verra, London Business School, and George Washington University.
? What We Do
We specialize in:
Rapid MVP development (go from idea to launch in just 4 weeks)
Custom software development
Gen AI applications and AI-as-a-Service (AIaaS)
Enterprise cloud architecture
DevOps, CI/CD, and infrastructure automation
QA and test automation
E-commerce platforms and performance optimization
Digital marketing and analytics integrations
? AI at the Core
Our delivery model is infused with AI ¨C from workflow optimization to proactive risk management. We leverage GenAI, NLP, and ML to make your software smarter, faster, and more secure.
? Impact in Numbers
500+ successful projects delivered
200+ web and mobile apps launched
42-month average client engagement
30+ active projects at any time
? Industries We Serve
Fintech
Healthcare
Education (E-Learning)
Sustainability & Compliance
Real Estate
Energy
Customer Experience platforms
?? Flexible Engagement Models
Whether you're looking for dedicated teams, fixed-cost projects, or time-and-materials models, we deliver with agility and transparency.
? Why Clients Choose Emvigo
? AI-accelerated delivery
? Strong focus on long-term partnerships
? Highly rated on Clutch and Google Reviews
? Proactive and adaptable to change
? Fully GDPR-compliant and security-conscious
? Visit: emvigotech.com
? Contact: sales@emvigotech.com
? Ready to scale your tech with confidence? Let¡¯s build the future, together.Test Case Design Techniques ¨C Practical Examples & Best Practices in Software...
Test Case Design Techniques ¨C Practical Examples & Best Practices in Software...Muhammad Fahad Bashir
?
Emvigo Capability Deck 2025: Accelerating Innovation Through Intelligent Soft...
Emvigo Capability Deck 2025: Accelerating Innovation Through Intelligent Soft...Emvigo Technologies
?
Ad
I hate mocking
- 5. when(mockClient .initiateMultipartUpload(any(InitiateMultipartUploadRequest.class))) .thenAnswer(new Answer<InitiateMultipartUploadResult>() { @Override public InitiateMultipartUploadResult answer( InvocationOnMock invocation) throws Throwable { LOG.debug("initiateMultipartUpload for {}", mockClient); synchronized (lock) { if (results.requests.size() == errors.failOnInit) { if (errors.recover) errors.failOnInit(-1); throw new AmazonClientException("Fail on init " + results.requests.size()); } String uploadId = UUID.randomUUID().toString(); results.requests.put(uploadId, invocation.getArgumentAt( 0, InitiateMultipartUploadRequest.class)); results.uploads.add(uploadId); return newResult(results.requests.get(uploadId), uploadId); } } });
- 6. ? Mock code reflects our assumptions about system behaviour ? Doesn't verify that the caller would work against real service ? Any change in real code breaks the tests: false positives ? Test failures often "fixed" by editing test/mock ... which could hide real problems Why bother? It's high-maintenance suffering
- 8. Proposed ? Avoid mock tests ? Add functional tests against real infrastructure ? Embed fault injection ability into functional code ? Use for integration testing