Eco-friendly Linux kernel development
0 likes18 views
Eco-friendly Linux kernel development focuses on minimizing energy consumption during continuous integration and delivery (CI/CD). The presenter from Canonical discusses how building minimal kernels with KernelCraft and deploying them in virtual machines can significantly reduce the time and power required for kernel testing compared to building full custom kernels. Experimental results show KernelCraft can cut kernel build time by over 80% and energy usage by over 90% for the same test. While challenges remain around coverage and service testing, efforts like virtualizing the build environment and migrating to more efficient tools can improve the sustainability of kernel development at large scale.
![¡ñ
Lines of code:
$ find ( -name '*.[chS]' -o -name '*.rs' ) -exec wc -l {} ; | awk 'BEGIN{sum=0}{sum+=$1}END{print sum}'
32,190,298
¡ñ
Amount of patches applied last week:
$ git log --oneline v6.3-rc4..v6.3-rc5 | wc -l
244
¡ñ
Lines of code changed last week:
$ git diff --stat v6.3-rc4..v6.3-rc5 | tail -1
201 files changed, 1811 insertions(+), 913 deletions(-)
Linux kernel is big](https://image.slidesharecdn.com/eco-friendly-kernel-testing-230415124225-7e009af3/85/Eco-friendly-Linux-kernel-development-4-320.jpg)
Eco-friendly Linux kernel development
- 1. Eco-friendly Linux kernel development Minimizing energy consumption during CI/CD Andrea Righi - Kernel Engineer at Canonical Email: andrea.righi@canonical.com 5th OSPM summit ¨C Ancona - 2023
- 2. Agenda ¡ñ Overview ¡ñ Cost of CI/CD in the kernel ¡ñ Use KernelCraft for fast and efficient CI/CD ¡ñ Results ¡ñ Demo ¡ñ Q&A
- 3. Overview
- 4. ¡ñ Lines of code: $ find ( -name '*.[chS]' -o -name '*.rs' ) -exec wc -l {} ; | awk 'BEGIN{sum=0}{sum+=$1}END{print sum}' 32,190,298 ¡ñ Amount of patches applied last week: $ git log --oneline v6.3-rc4..v6.3-rc5 | wc -l 244 ¡ñ Lines of code changed last week: $ git diff --stat v6.3-rc4..v6.3-rc5 | tail -1 201 files changed, 1811 insertions(+), 913 deletions(-) Linux kernel is big
- 5. Kernel development is difficult ¡ñ How can I reproduce a bug? ¡ñ How long does it take to build the kernel on my machine? ¡ñ How do I test this fix? ¡ñ Does my change cause problems on another arch? ¡ñ Does my change introduce a vulnerability? ¡ñ ?
- 6. It¡¯s not easy being a maintainer, either ¡ñ End users expect Linux distros to do proper testing ¡ñ Volume is massive
- 7. Cost of CI/CD in the kernel
- 8. Linux CI projects ¡ñ kselftests ¡ñ Syzbot ¡ñ 0-Day ¡ñ KernelCI ¡ñ CKI ¡ñ LKFT ¡ñ Fuego ¡ñ ...
- 9. Kernel CI workflow ¡ñ Tests triggered periodically or by events (git push, patch on the ML, ...) ¡ñ Create a source artifact ¡ñ Build a fully-featured general-purpose kernel... ? ¡ñ Create packages for the new kernel (distro) ¡ñ Deploy a test system (VM or bare metal) ¡ñ Run tests ¡ñ Collect results ¡ñ Decommission a system
- 10. The problem of custom kernels ¡ñ Ubuntu has a lot of custom kernels ¡ñ Custom changes need to be tested (more than upstream changes) ¡ñ Lots of kernels, systems and devices
- 11. Scalability ¡ñ This approach doesn¡¯t scale very well ¡ñ More hardware! (expensive and not very energy efficient...)
- 13. Proposed solution (in a nutshell) ¡ñ Build minimal kernel to test individual features ¡ñ Deploy a single golden system ¡ñ Virtualize it among multiple VMs (in a CoW way) ¡ñ Run tests in VM
- 14. Pros and cons ¡ñ Pros ¡ñ A minimal kernel still allows to run tests effectively ¡ñ More efficient in terms of time and energy ¡ñ Cons ¡ñ Doesn¡¯t work to test specific hardware ¡ñ Kernel config coverage can be challenging ¡ñ Identify the minimum subset of configs required to run the test(s)
- 15. KernelCraft
- 16. What is KernelCraft (kc)? ¡ñ Build a kernel from source ¡ñ Run it inside a virtualized snapshot of your live system ¡ñ Based on virtme (by Andrew Lutomirski) and qemu / kvm
- 17. How does it work? ¡ñ Generate a minimal kernel .config ¡ñ Build the kernel from source in a couple of minutes ¡ñ Run it inside a VM that is a snapshot of your live system (safe) ¡ñ Host fs is exported to guest via 9p fs ¡ñ Local changes are handled by overlayfs / tmpfs ¡ñ Output is captured and can be fed to other tools
- 18. Results
- 19. Test ¡ñ Hardware ¡ñ Dell XPS 13 ¡ñ Test ¡ñ Build kernel and run `uname -r` (avg of 10 kernel builds) ¡ñ Metrics: ¡ñ Elapsed time (sec) ¡ñ Total power consumption (Joule) ¡ñ Average power consumption (Watt)
- 20. Result: time (sec) KernelCraft Ubuntu 0 500 1000 1500 2000 2500 3000 3500 4000 Time (sec)
- 21. Result: total power consumption (Joule) KernelCraft Ubuntu 0 20000 40000 60000 80000 100000 120000 Power (J)
- 22. Result: average power (Watt) KernelCraft Ubuntu 0 5 10 15 20 25 30 35 Avg Power (W)
- 23. Result (summary) ¡ñ v6.2-rc7 using kc ¡ñ Time=371.749s, power=10,854.843 J (avg=29.19W) ¡ñ 2.6 kcal =~ half leaf of lettuce ¡ñ v6.2-rc7 w/ Ubuntu config using make ¡ñ Time=3481.965s, power=104,504.228 J (avg=30.013W) ¡ñ 25 kcal =~ half hard-boiled egg
- 24. Demo
- 25. commit 154e07c164859fc90bf4e8143f2f6c1af9f3a35e Author: Andrea Righi <andrea.righi@canonical.com> Date: Thu Mar 30 11:54:42 2023 +0200 l2tp: generate correct module alias strings Commit 65b32f801bfb ("uapi: move IPPROTO_L2TP to in.h") moved the definition of IPPROTO_L2TP from a define to an enum, but since __stringify doesn't work properly with enums, we ended up breaking the modalias strings for the l2tp modules: $ modinfo l2tp_ip l2tp_ip6 | grep alias alias: net-pf-2-proto-IPPROTO_L2TP alias: net-pf-2-proto-2-type-IPPROTO_L2TP alias: net-pf-10-proto-IPPROTO_L2TP alias: net-pf-10-proto-2-type-IPPROTO_L2TP Use the resolved number directly in MODULE_ALIAS_*() macros (as we already do with SOCK_DGRAM) to fix the alias strings: $ modinfo l2tp_ip l2tp_ip6 | grep alias alias: net-pf-2-proto-115 alias: net-pf-2-proto-115-type-2 alias: net-pf-10-proto-115 alias: net-pf-10-proto-115-type-2 Moreover, fix the ordering of the parameters passed to MODULE_ALIAS_NET_PF_PROTO_TYPE() by switching proto and type. Fixes: 65b32f801bfb ("uapi: move IPPROTO_L2TP to in.h") Link: https://lore.kernel.org/lkml/ZCQt7hmodtUaBlCP@righiandr-XPS-13-7390 Signed-off-by: Guillaume Nault <gnault@redhat.com> Signed-off-by: Andrea Righi <andrea.righi@canonical.com> Reviewed-by: Wojciech Drewek <wojciech.drewek@intel.com> Tested-by: Wojciech Drewek <wojciech.drewek@intel.com> Signed-off-by: David S. Miller <davem@davemloft.net>
- 26. What¡¯s next?
- 27. Open issues ¡ñ Test coverage: ¡ñ Minimal kernel doesn¡¯t allow to test everything ¡ñ We can¡¯t easily perform tests that require to start services via systemd ¡ñ fs performance over 9p is still not ideal ¡ñ Nice improvement with: ¡ñ 00833408bb16 ("Merge tag '9p-for-6.1' of https://github.com/martinetd/linux") ¡ñ virtme is almost an abandoned project ¡ñ We are considering to fork and create a virtme-ng project
- 28. References
- 29. References ¡ñ KernelCraft https://github.com/arighi/kernelcraft ¡ñ Virtme https://github.com/arighi/virtme https://github.com/amluto/virtme ¡ñ All the benefits of CI/CD but ¡°Greener¡± https://www.perforce.com/blog/kw/green-software-development-ci-cd ¡ñ Usage, Costs and Benefits of Continuous Integration in Open-Source Projects http://cope.eecs.oregonstate.edu/papers/OpenSourceCIUsage.pdf
- 30. Questions?