�ݺ�ߣ

Editor's Notes

• #3: I am Javier Moscardo, from Spain, QE during 1 year in Spotify. Developer background Working in a feature team, owners of Search feature among others With Devs, QA and TA in the same team. The team is a small startup inside the company. We are independent and autonomous, and we still aligned with other teams in the company.
• #4: Here the question is, how can we be better at testing? I am going to describe the journey we did from 3 to 5 stars, in this journey I am going to say what have we been doing to be better at testing To rise the quality of our product, focus on the test automation, timeline of the journey It is not only bug detection, it is about bug prevention One thing to achieve this is to have more tests at different levels And that’s what this talk is about We are going to have some time afterwards for Q&A
• #5: (present the problem) Our application in the App store was 3 stars Version control system Crashes Poor code quality
• #6: Why we tried to solve this problem? We are a fast-pace company, with a fast growing product. Affecting a big number of people, 75M users Number of songs: Over 30 million Number of playlists: Over 1.5 billion Available in 58 markets A lot of code changes - > translate to lines of code You can use this as a source since it is publicly available: https://press.spotify.com/se/information/
• #7: Every release we need to do regression testing as you can see There are a lot of areas and combinations to cover. We were doing manual test every 3 weeks
• #8: We tried our best and we failed, D’OH! We need to do a lot of manual testing for regression. This is a bottleneck, it can’t be done manually in every release. Some code has been changed, maybe new bugs are in the product.. What happen with the rest of the features? To rise the quality of the user experience, we decide to write more Automated tests, The product was becoming bigger, more features to test we realized that we need to engaged the entire team to achieve the mission of automate some of this manual tests Going into more technical detail
• #9: We are creating our own test interface for our mobile client. Mention instead of Java tests they are Acceptance or end to end tests (fix the diagram) Why? The idea of having the Test API inside the app is because we have access to the client state. We want to have our own asserts, enable us to write much better assertions. And this can’t be done with other mobile frameworks. For example, There are other options like Appium. In this case the mobile automation framework is situated outside the App. It has some good points, like you don’t need to recompile the client for a TA change. However, we are not reinventing the wheel, we are giving the TA’s the option to use another language for the tests. This API is not included in the app when we build it for Production. It is only included when we build the app for Testing/debug mode Who is written what? We need our developers to do this. Test API TA Acceptance tests And because the implementation of the acceptance test is born from a collaboration between the TA and the Client Developer This is done before saying the Feature is complete. It is manually tested and if possible this manual test is automated When? So the moment to write this code is when we are creating the feature. We are creating the test interface at the same time that the TA is writing the Java test, so we know what we need to implement. Having the Test interface inside the App give us more control about the assertions that we can do in the tests, Some people are writing the acceptance tests when the feature is built. That can be a mistake. An Acceptance test is also code. It’s another task of the development process. We are applying Agile methodologies. We have a story sliced into tasks, and one of them is the TA Api, other is to write the acceptance test. You have the logic fresh in your mind, the discussions about the feature and what to test. The sooner that you can find a bug the better, and here it is about bug prevention Maslow hierarchy, is the idea that you need to satisfy your more basic needs first before higher levels. We have unit tests, is the base of our pyramid. We are trying to move to a higher level to the top of it.
• #10: Ask how many are using MBT in the model slide. Then tell. We are using Model Based Testing in combination with Graphwalker (open source tool) to cover corner cases and bigger areas. Zoom in in the clickagemediaobjetc this model is truly dependent to the client
• #11: this model is really dependent of the client I want you to focus on the zoom ok first we create the model for a client and after that we wanted to implemented it for other client. We need to repeat the same model AGAIN!�� here you have an example of Search. This model was one of the first models created: We have 2 mobile clients, Android and iOS and 1 more client, desktop. We need to create and maintain the 3 models for the same functionality.
• #12: This is how it looks now, and the kind of model that we are creating. In this model the actions are client independent. We can share the same model between different clients The idea is to take the action as a status change. not as a click, tap or other client related action.
• #13: PlayTrack This is a state-model, cheaper to maintain, to add or remove a vertex This is created with the effort of the QA and the TA because testers knows the business logic and the requirements. The implementation is done by the TA, and the Test interface by the Developer. The details of the implementation are done in the Java test, so it can be click a button, tap a key,.. Remember, these models are created between the QA and the TA. MBT is a tool to provide Manual testers a way out of the boring regression work. They will have more room to do better and deeper exploratory testing and cover corner cases.
• #14: Say 9 months Boom, 4 stars!
• #15: We talked about writing tests, but then what? When are they running? Before merging new code into master another developer needs to do a code review and approve the changes. We also run the unit tests. We have pre-merge tests. This is compiling and building the entire client. And running the tests that verify the basic features are working. These are non-MBT. We run BVT (Build Verification Tests) every time we build the client, the average time is every 15 mins. The complex test, MBT, are running independently of the PR every 2-4h in another cloud machines. We are using teamcity to execute all this tasks.
• #16: Where is a good question We have a QA Lab with all kind of mobile devices connected and accessible from our cloud machines where the tests are running. For iOS mostly we are using simulators so we have a battery of apple minis in the rack Explain the video playback issues. With different android versions and brand devices like Samsung, LG, Sony. mention a number of devices, close to 90
• #17: Now we are combining manual regression testing with checking the Test reports. We have created our own report service. Here a QA can go to the dashboard and verify the different acceptance tests before approving a feature for a release. This is a complement to the manual regression testing. I realized Quality visibility is important. Sometimes we still having flaky tests, so we need to double check with a manual test. This is happening every 2 weeks, before it was 3 weeks. Here we made a good improvement about quality
• #18: Say 18 months Finally, 5 stars! QA is Quality Assistance not assurance. it is also Quality Ambassador in the team. We reduce the release cycle to 2 weeks One single repo for the client code Acceptance tests in the same client code repo More Acceptance tests Continuous Delivery
• #19: So I want to finish by talking about… In which direction are we moving. One TA against 3 clients devs. the TA can be a bottleneck The developers are written the end to end tests. We, the TA are coaching them. It is Java, so Android developers have it easier, iOS devs can need more support. On Android we stay on 4.5 rating from 5. Also on iOS. The title is rounded
• #20: So in closing… Less bugs means more development speed, when you are focus on a feature, you are faster updating the code, try to avoid switching context, it is expensive to fix it later when it is already in production and you are working on another feature Automate the manual tests than can be automated. These tools provide your testers more time will find corner cases bugs. It is not only bug detection, it is about bug prevention
• #22: say Questions? Don’t rush