Testing sync engine
0 likes356 views
This document discusses testing strategies for the iOS Sync Engine. It provides overall test stats, including that unit tests make up 80% of tests and integration tests make up 20%. The main testing strategy is to test against a fake backend called MockTransportSession, which provides advantages like speed, flexibility and independence over using stubs. MockTransportSession simulates the real backend by making fake requests, managing an internal database, and responding to requests. Future plans include splitting tests into smaller frameworks to reduce runtime, running tests on pull requests, and using record/replay to keep MockTransportSession in sync with the real backend.
More Related Content
What's hot (20)
Similar to Testing sync engine (20)
![Static Code Analysis PHP[tek] 2023](https://cdn.slidesharecdn.com/ss_thumbnails/staticcodeanalysisphptek2023-230517122556-f35bc9a2-thumbnail.jpg?width=560&fit=bounds)
Testing sync engine
- 2. Testing (iOS) Sync Engine
- 3. Agenda Overall tests stats Our test strategies MockTransportSession Workflow Problems Future plans
- 5. 3356 tests on OSX Overall tests stats
- 6. 3356 tests on OSX 3415 tests on iOS (+ apns tests) Overall tests stats
- 7. 3356 tests on OSX 3415 tests on iOS (+ apns tests) ~ 7.5 minutes to build and run all the tests on one platform (build ~ 40s, tests ~ 7 min) Overall tests stats
- 8. 3356 tests on OSX 3415 tests on iOS (+ apns tests) ~ 7.5 minutes to build and run all the tests on one platform (build ~ 40s, tests ~ 7 min) Overall tests stats
- 9. Unit tests ~ 80% Overall tests stats
- 10. Unit tests ~ 80% Integration tests ~ 20% Overall tests stats
- 11. Unit tests ~ 80% Integration tests ~ 20% Performance tests - 18 Overall tests stats
- 12. Unit tests ~ 80% Integration tests ~ 20% Performance tests - 18 Tests of tests Overall tests stats
- 13. Unit tests ~ 80% Integration tests ~ 20% Performance tests - 18 Tests of tests Sorry, no UI tests Overall tests stats
- 15. What we DONT do: Our test strategies
- 16. What we DONT do: test against http:// (real backend) test using http requests stubs (like OHHTTPStubs) Our test strategies
- 17. What we DONT do: test against http:// (real backend) test using http requests stubs (like OHHTTPStubs) Why? Stubs are not that flexible. Our test strategies
- 18. What we DONT do: test against http:// (real backend) test using http requests stubs (like OHHTTPStubs) Why? Stubs are not that flexible. Its still slow. Our test strategies
- 19. What we really do: Test against fake backend Our test strategies
- 20. What we really do: Test against fake backend Advantages Fast Flexible Independent Testable Our test strategies
- 21. What we really do: Test against fake backend Advantages Fast Flexible Independent Testable Disadvantages Need to write (and test!) code Need to be in sync with real backend (can be improved) Our test strategies
- 26. WTF is THAT? ZMTransportSession - is a delegate of a delegate of NSURLSession MockTransportSession
- 27. WTF is THAT? ZMTransportSession - is a delegate of a delegate of NSURLSession Sends http requests to scheduler MockTransportSession
- 28. WTF is THAT? ZMTransportSession - is a delegate of a delegate of NSURLSession Sends http requests to scheduler Handles authentication MockTransportSession
- 29. WTF is THAT? ZMTransportSession - is a delegate of a delegate of NSURLSession Sends http requests to scheduler Handles authentication Opens/closes web socket connection MockTransportSession
- 30. WTF is THAT? MockTransportSession - is a mock of ZMTransportSession MockTransportSession
- 31. WTF is THAT? MockTransportSession - is a mock of ZMTransportSession MockTransportSession - acts as a fake backend MockTransportSession
- 32. WTF is THAT? Make fake requests MockTransportSession
- 33. WTF is THAT? Make fake requests Records requests and responds to them MockTransportSession
- 34. WTF is THAT? Make fake requests Records requests and responds to them Manages internal database MockTransportSession
- 35. WTF is THAT? Make fake requests Records requests and responds to them Manages internal database Can simulate remote changes. MockTransportSession
- 36. WTF is THAT? Make fake requests Records requests and responds to them Manages internal database Can simulate remote changes. Can send push channel (web socket) events MockTransportSession
- 37. Before each test we insert objects in internal database (in memory) MockTransportSession
- 38. Before each test we insert objects in internal database (in memory) Test code make request using MockTransportSession MockTransportSession
- 39. Before each test we insert objects in internal database (in memory) Test code make request using MockTransportSession It records and process incoming request (filtering by path and lots of ifs) MockTransportSession
- 40. Before each test we insert objects in internal database (in memory) Test code make request using MockTransportSession It records and process incoming request (filtering by path and lots of ifs) Performs some CRUD action on database depending on request (add message, create conversation, fetch conversation) MockTransportSession
- 41. Before each test we insert objects in internal database (in memory) Test code make request using MockTransportSession It records and process incoming request (filtering by path and lots of ifs) Performs some CRUD action on database depending on request (add message, create conversation, fetch conversation) Creates and sends response. Optionally can delegate it to other object (i.e. test case). MockTransportSession
- 42. Workflow
- 43. Finished feature - run all the tests Workflow
- 44. Finished feature - run all the tests Create pull request Workflow
- 45. Finished feature - run all the tests Create pull request Review, merge Workflow
- 46. Finished feature - run all the tests Create pull request Review, merge Release script runs all tests locally Workflow
- 47. Finished feature - run all the tests Create pull request Review, merge Release script runs all tests locally Pushes to remote branches (master/develop) Workflow
- 48. Finished feature - run all the tests Create pull request Review, merge Release script runs all tests locally Pushes to remote branches (master/develop) Xcode bots make integration on each push to master/ develop Workflow
- 49. Finished feature - run all the tests Create pull request Review, merge Release script runs all tests locally Pushes to remote branches (master/develop) Xcode bots make integration on each push to master/ develop Each bot for each platform runs all the tests Workflow
- 50. Finished feature - run all the tests Create pull request Review, merge Release script runs all tests locally Pushes to remote branches (master/develop) Xcode bots make integration on each push to master/ develop Each bot for each platform runs all the tests Bots archive and upload binary to the cloud Workflow
- 51. Problems
- 52. Solved: Tests isolation (mostly database state) Problems
- 53. Solved: Tests isolation (mostly database state) Handling asynchronous code Problems
- 54. Solved: Tests isolation (mostly database state) Handling asynchronous code Not yet solved: Problems
- 55. Solved: Tests isolation (mostly database state) Handling asynchronous code Not yet solved: Reduce tests running time U dont like me? =( Problems
- 56. Solved: Tests isolation (mostly database state) Handling asynchronous code Not yet solved: Reduce tests running time Sync with real backend Problems
- 57. Solved: Tests isolation (mostly database state) Handling asynchronous code Not yet solved: Reduce tests running time Sync with real backend Flaky tests Problems
- 58. Future plans
- 59. Split to smaller frameworks - to improve run time Future plans
- 60. Split to smaller frameworks - to improve run time Setup Buildasaur - to run tests on Github pull requests Future plans
- 61. Split to smaller frameworks - to improve run time Setup Buildasaur - to run tests on Github pull requests Record/replay tests for MockTransportSession - to keep it in sync Future plans
- 62. Thats it!