A Decade of Comment Quality Assessment: A Systematic Literature Review
0 likes26 views
The document summarizes the key findings of a systematic literature review on comment quality assessment over the past decade (2011-2020). The review identified 21 quality attributes studied, with consistency and completeness being most frequent. Manual assessment was the most used technique, while deep learning was under-explored. Code comments and Java programs were most analyzed. Nearly half the studies lacked data/tool availability for replication. Future work directions included relating attributes to developer roles and automatic measurement.
![/**
* A class representing a window on the screen.
*
* For example:
* <pre>
* Window win = new Window(parent);
* win.show();
* </pre>
*
* @author Sami Shaio
* @version 1.13, 06/08/06
* @see java.awt.BaseWindow
*/
class Window extends BaseWindow{
..
}
class OneHotCategorical(Distribution):
r"""
Creates a one-hot categorical distribution parameterized by :attr:`probs` or
:attr:`logits`.
Samples are one-hot coded vectors of size ``probs.size(-1)``.
.. note:: The `probs` argument must be non-negative, finite and have a non-zero sum,
and it will be normalized to sum to 1 along the last dimension. :attr:`probs`
will return this normalized value.
The `logits` argument will be interpreted as unnormalized log probabilities
and can therefore be any real number. It will likewise be normalized so that
the resulting probabilities sum to 1 along the last dimension. :attr:`logits`
will return this normalized value.
See also: :func:`torch.distributions.Categorical` for specifications of
:attr:`probs` and :attr:`logits`.
Example::
>>> m = OneHotCategorical(torch.tensor([ 0.25, 0.25, 0.25, 0.25 ]))
>>> m.sample() # equal probability of 0, 1, 2, 3
tensor([ 0., 0., 0., 1.])
Args:
probs (Tensor): event probabilities
logits (Tensor): event log probabilities (unnormalized)
"""
Smalltalk class comment
26
Comments of multi-languages](https://image.slidesharecdn.com/decade-of-comment-quality-assessment-231027140812-8ec8f576/85/A-Decade-of-Comment-Quality-Assessment-A-Systematic-Literature-Review-26-320.jpg)
![Full_Stack_Dule_1.NNNNNNNNNNNNNNNNNNNNNNNNNNNNNN[1].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/fullstackdevelopment-module11-240529132159-e5033368-thumbnail.jpg?width=560&fit=bounds)
More Related Content
Similar to A Decade of Comment Quality Assessment: A Systematic Literature Review (20)
More from Pooja Rani (7)
Recently uploaded (20)
A Decade of Comment Quality Assessment: A Systematic Literature Review
- 1. A Decade of Comment Quality Assessment: A Systematic Literature Review Pooja Rani, Arianna Blasi, Natallia Stulova, Sebastiano Panichella, Alessandra Gorla, Oscar Nierstrasz Journal First University of Zurich, Switzerland ESEM 2023
- 2. 2 How software developers spend their time? Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S. (2017). Measuring program comprehension: A large-scale fi eld study with professionals. IEEE Transactions on Software Engineering. Comprehension 58% Navigation 24% Others 13% Edition 5%
- 3. 3 Comprehension 58% Navigation 24% Others 13% Edition 5% Code comments /** * A class representing a window on the screen. * * For example: * <pre> * Window win = new Window(parent); * win.show(); * </pre> * * @author Sami Shaio * @version 1.13, 06/08/06 * @see java.awt.BaseWindow * @see java.awt.Button */ class Window extends BaseWindow { ¡ }
- 4. 4 /** * A class representing a window on the screen. * * For example: * <pre> * Window win = new Window(parent); * win.show(); * </pre> * * @author Sami Shaio * @version 1.13, 06/08/06 * @see java.awt.BaseWindow * @see java.awt.Button */ class Window extends BaseWindow { ¡ } Code comments
- 5. 5 /** * A class representing a window on the screen. * * For example: * <pre> * Window win = new Window(parent); * win.show(); * </pre> * * @author Sami Shaio * @version 1.13, 06/08/06 * @see java.awt.BaseWindow * @see java.awt.Button */ class Window extends BaseWindow { ¡ } Code comments
- 6. 6 /** * A class representing a window on the screen. * * For example: * <pre> * Window win = new Window(parent); * win.show(); * </pre> * * @author Sami Shaio * @version 1.13, 06/08/06 * @see java.awt.BaseWindow * @see java.awt.Button */ class Window extends BaseWindow { ¡ } Code comments Trustworthy form of documentation - McMillan et al. 2010
- 7. 7 High-quality comments support developers in various activities /** * A class representing a window on the screen. * * For example: * <pre> * Window win = new Window(parent); * win.show(); * </pre> * * @author Sami Shaio * @version 1.13, 06/08/06 * @see java.awt.BaseWindow * @see java.awt.Button */ class Window extends BaseWindow { ¡ } Code comments Trustworthy form of documentation - McMillan et al. 2010 - Dekel et al. 2009
- 8. 8 /** * A class representing a window on the screen. * * For example: * <pre> * Window win = new Window(parent); * win.show(); * </pre> * * @author Sami Shaio * @version 1.13, 06/08/06 * @see java.awt.BaseWindow * @see java.awt.Button */ class Window extends BaseWindow { ¡ } Is this a high- quality comment? correct? consistent? enough? Other quality attributes? Code comments
- 9. 9 Gillies et al. 2011 Quality is a multi-dimensional concept
- 10. 10 How do researchers measure comment quality?
- 11. Systematic literature review 10 years timeline (2011-2020) 195 software engineering venues 332 proceedings 2353 candidate papers 47 relevant papers 11 How do researchers measure comment quality?
- 12. 12 21 quality attributes Quality attributes
- 13. 13 Consistency Completeness Accuracy Readability Up-to-date-ness Content relevance Maintainability Spelling and grammar Quality attributes Quality attributes 23 12 21 quality attributes
- 14. 14 Some are frequently considered Consistency Completeness Accuracy Readability Up-to-date-ness Content relevance Maintainability Spelling and grammar Quality attributes Quality attributes
- 15. 15 Some are rarely considered Consistency Completeness Accuracy Readability Up-to-date-ness Content relevance Maintainability Spelling and grammar Conciseness Usability Correctness Traceability Accessibility Coherence Format Information organization Understandability Documentation technoogy Internatioalization Author-related Quality attributes Quality attributes
- 16. 16 Techniques Consistency Completeness Accuracy Readability Up-to-date-ness Content relevance Maintainability Spelling and grammar Conciseness Usability Correctness Traceability Accessibility Coherence Format Information organization Understandability Documentation technoogy Internatioalization Author-related Quality attributes
- 17. 17 Techniques Manual assessment Consistency ML-based Heuristic-based Completeness Accuracy Metric-based Static analysis NLP Readability Up-to-date-ness Content relevance DNN-based Maintainability Spelling and grammar Conciseness First order logic Usability Correctness Traceability Accessibility Coherence Format Information organization Understandability Empirically fNIRS Documentation technoogy Internatioalization Author-related Techniques Quality attributes
- 18. 18 Techniques Manual assessment Consistency ML-based Heuristic-based Completeness Accuracy Metric-based Static analysis NLP Readability Up-to-date-ness Content relevance DNN-based Maintainability Spelling and grammar Conciseness First order logic Usability Correctness Traceability Accessibility Coherence Format Information organization Understandability Empirically fNIRS Documentation technoogy Internatioalization Author-related Techniques Quality attributes
- 19. Manual assessment Consistency ML-based Heuristic-based Completeness Accuracy Metric-based Static analysis NLP Readability Up-to-date-ness Content relevance DNN-based Maintainability Spelling and grammar Conciseness First order logic Usability Correctness Traceability Accessibility Coherence Format Information organization Understandability Empirically fNIRS Documentation technoogy Internatioalization Author-related Techniques Quality attributes 19 Techniques Manual assessment is still the most frequent technique to measure quality attributes
- 20. Manual assessment Consistency ML-based Heuristic-based Completeness Accuracy Metric-based Static analysis NLP Readability Up-to-date-ness Content relevance DNN-based Maintainability Spelling and grammar Conciseness First order logic Usability Correctness Traceability Accessibility Coherence Format Information organization Understandability Empirically fNIRS Documentation technoogy Internatioalization Author-related Techniques Quality attributes 20 Techniques Deep learning-based techniques have not been extensively explored for comment analysis
- 21. Manual assessment Consistency ML-based Heuristic-based Completeness Accuracy Metric-based Static analysis NLP Readability Up-to-date-ness Content relevance DNN-based Maintainability Spelling and grammar Conciseness First order logic Usability Correctness Traceability Accessibility Coherence Format Information organization Understandability Empirically fNIRS Documentation technoogy Internatioalization Author-related Techniques Quality attributes 21 Which of these quality attributes are important to which developers? Do code reviewers care more for accurate comments than complete comments? When? How to automatically measure these quality attributes? Can LLMs help writing better comments?
- 22. 22 Is comments worth the efforts?
- 23. 23 /** * A class representing a window on the screen. * * For example: * <pre> * Window win = new Window(parent); * win.show(); * </pre> * * @author Sami Shaio * @version 1.13, 06/08/06 * @see java.awt.BaseWindow * @see java.awt.Button */ class Window extends BaseWindow { ¡ } Task Design Implementation Maintenance Summary Usage Pointer Information Quality attributes Accessibility Readability Author-related Completeness Consistency Correctness Audience User Developer Future work
- 24. 24 Replication: https:// zenodo.org/record/ 4729054 Paper pooja-rani-01985b68/ A Decade of Comment Quality Assessment: A Systematic Literature Review
- 25. More Curious?
- 26. /** * A class representing a window on the screen. * * For example: * <pre> * Window win = new Window(parent); * win.show(); * </pre> * * @author Sami Shaio * @version 1.13, 06/08/06 * @see java.awt.BaseWindow */ class Window extends BaseWindow{ .. } class OneHotCategorical(Distribution): r""" Creates a one-hot categorical distribution parameterized by :attr:`probs` or :attr:`logits`. Samples are one-hot coded vectors of size ``probs.size(-1)``. .. note:: The `probs` argument must be non-negative, finite and have a non-zero sum, and it will be normalized to sum to 1 along the last dimension. :attr:`probs` will return this normalized value. The `logits` argument will be interpreted as unnormalized log probabilities and can therefore be any real number. It will likewise be normalized so that the resulting probabilities sum to 1 along the last dimension. :attr:`logits` will return this normalized value. See also: :func:`torch.distributions.Categorical` for specifications of :attr:`probs` and :attr:`logits`. Example:: >>> m = OneHotCategorical(torch.tensor([ 0.25, 0.25, 0.25, 0.25 ])) >>> m.sample() # equal probability of 0, 1, 2, 3 tensor([ 0., 0., 0., 1.]) Args: probs (Tensor): event probabilities logits (Tensor): event log probabilities (unnormalized) """ Smalltalk class comment 26 Comments of multi-languages
- 29. 29 7 6 8 9 10 1 2 3 4 5 Methodology 11 47 papers
- 30. 47 papers over years 30 0 2 4 6 8 10 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 # Papers
- 31. 31 consistency, completeness Quality attributes Techniques Comment types method comments, inline comments heuristic-based, machine learning-based Dimensions analyzed Data availability tool, dataset
- 32. 32 /** * A class representing a window on the screen. * * For example: * <pre> * Window win = new Window(parent); * win.show(); * </pre> * * @author Sami Shaio * @version 1.13, 06/08/06 * @see java.awt.BaseWindow * @see java.awt.Button */ class Window extends BaseWindow { ¡ } Code comment metrics Consistency: Overlap between the terms used in comment and code. Higher value of CIC with higher readability of that code. Readability: Flesch reading-ease test
- 33. 33 Nearly 50% of the studies still lack in the replicability dimension, as their respective dataset or tool is often not publicly accessible. Data availability
- 34. 34 48% 19% 8% 4% 4% 2% 2% 6% 6% 2% 6% 4% 2% 4% 4% 2% 2% 8% 4% 2% 2% Code comments Method comments API documentation License comments Inline comments TODO Comments Software documentation Deprecation Comments 0% 20% 40% 60% 80% Java Python C++ C C# Other Comment types
- 35. 35 48% 19% 8% 4% 4% 2% 2% 6% 6% 2% 6% 4% 2% 4% 4% 2% 2% 8% 4% 2% 2% Code comments Method comments API documentation License comments Inline comments TODO Comments Software documentation Deprecation Comments 0% 20% 40% 60% 80% Java Python C++ C C# Other 52% of the studies Comment types
- 36. 36 48% 19% 8% 4% 4% 2% 2% 6% 6% 2% 6% 4% 2% 4% 4% 2% 2% 8% 4% 2% 2% Code comments Method comments API documentation License comments Inline comments TODO Comments Software documentation Deprecation Comments 0% 20% 40% 60% 80% Java Python C++ C C# Other 48% of the studies Comment types
- 37. 37 Comment types 48% 19% 8% 4% 4% 2% 2% 6% 6% 2% 6% 4% 2% 4% 4% 2% 2% 8% 4% 2% 2% Code comments Method comments API documentation License comments Inline comments TODO Comments Software documentation Deprecation Comments 0% 20% 40% 60% 80% Java Python C++ C C# Other % of studies