際際滷

際際滷Share a Scribd company logo
The Art and Science of
Grammar Coverage Analysis

Dominique Boucher
Nu Echo Inc.
dominique.boucher@nuecho.com
                         SpeechTEK 2009
                         New York, USA
The grammar development process
Enter / get initial           Write initial
set of sentences           grammar to cover
 in coverage set              utterances


                            Run coverage
                                tests



   Generate
                      no      Problems?       yes       Fix grammar
   sentences



                                                                Objective:
Enrich coverage                                 To obtain:
      set                                       (a) A complete coverage set
                                                (b) A grammar that covers the coverage set
                                                    and produces the correct semantic
                                                    result
The importance of coverage analysis


  Design and       ? Ensure grammars conform to their specification
 development       ? Semantic tags testing




 Maintenance       ? Grammars evolve over the life of an application
                   ? Provides an effective tool for testing that a grammar
and optimization     isn¨t accidentally broken by a grammar change



  Conversion       ? Ensure proper conversion between grammar
                     formats
   projects
Grammar coverage challenges

?   Provide exhaustive coverage of all sentence patterns´
?   ´ with the smallest possible set of sentences.
    Otherwise:
    C Analysis of generated sentences will be time-consuming; and
    C Errors will go undetected
Two complementary techniques




                       #2
      #1
                    Sentence
   Sentence
                    patterns
   Generation
                   Exploration
Technique #1
Sentence generation

?   Tools usually provided by the ASR engine SDK
    C Operate on the source or compiled grammar




?   Commonly used generation strategies
    C Exhaustive generation
    C Generation of a fixed number of random sentences
Technical difficulties

?   Some grammars generate an infinite number of
    sentences
    C Exhaustive generation not possible
?   Semantic tags not all tested
    C Errors may remain undetected until application run time
?   All interesting cases may not be covered by the
    generated sentences
?   Uninteresting patterns are generated over and over
Sentence generation best practices

?   Avoid generating redundant sentences
    C Powerful sentence generation tools makes this possible
    C Too many sentences increases the risk of errors going
      undetected
?   Carefully examine generated sentences
    C If a sentence doesn¨t look right, it probably shouldn¨t be in
      there (although, to be sure, look at the parse tree)
?   Make sure the coverage test is as complete as possible
    C Should include all semantic tags and all sentence patterns
    C Full coverage is best (whenever possible)
Sentence generation revisited

?   Individual rule
    configuration
?   More effective strategies
    C Tags coverage
    C All grammar paths
    C Pick from @examples
    C Use fixed sentence
?   Generation can be
    started from any set of
    sentence patterns
Sentence generation tool




       Interactive Demo
Technique #2
Exploring sentence patterns

?   Interactive expansion of
    grammar rules
?   Derive sentence patterns
    C Useful to generate
      sentences for a specific
      pattern
?   Derive complete sentences
    C Can be debugged, etc.
?   Ideal to understand the
    structure of a grammar
Sentence explorer tool




       Interactive Demo
Three typical problems´
´ and how to fix them!




       Interactive Demo

More Related Content

The Art and Science of Grammar Coverage Analysis

  • 1. The Art and Science of Grammar Coverage Analysis Dominique Boucher Nu Echo Inc. dominique.boucher@nuecho.com SpeechTEK 2009 New York, USA
  • 2. The grammar development process Enter / get initial Write initial set of sentences grammar to cover in coverage set utterances Run coverage tests Generate no Problems? yes Fix grammar sentences Objective: Enrich coverage To obtain: set (a) A complete coverage set (b) A grammar that covers the coverage set and produces the correct semantic result
  • 3. The importance of coverage analysis Design and ? Ensure grammars conform to their specification development ? Semantic tags testing Maintenance ? Grammars evolve over the life of an application ? Provides an effective tool for testing that a grammar and optimization isn¨t accidentally broken by a grammar change Conversion ? Ensure proper conversion between grammar formats projects
  • 4. Grammar coverage challenges ? Provide exhaustive coverage of all sentence patterns´ ? ´ with the smallest possible set of sentences. Otherwise: C Analysis of generated sentences will be time-consuming; and C Errors will go undetected
  • 5. Two complementary techniques #2 #1 Sentence Sentence patterns Generation Exploration
  • 6. Technique #1 Sentence generation ? Tools usually provided by the ASR engine SDK C Operate on the source or compiled grammar ? Commonly used generation strategies C Exhaustive generation C Generation of a fixed number of random sentences
  • 7. Technical difficulties ? Some grammars generate an infinite number of sentences C Exhaustive generation not possible ? Semantic tags not all tested C Errors may remain undetected until application run time ? All interesting cases may not be covered by the generated sentences ? Uninteresting patterns are generated over and over
  • 8. Sentence generation best practices ? Avoid generating redundant sentences C Powerful sentence generation tools makes this possible C Too many sentences increases the risk of errors going undetected ? Carefully examine generated sentences C If a sentence doesn¨t look right, it probably shouldn¨t be in there (although, to be sure, look at the parse tree) ? Make sure the coverage test is as complete as possible C Should include all semantic tags and all sentence patterns C Full coverage is best (whenever possible)
  • 9. Sentence generation revisited ? Individual rule configuration ? More effective strategies C Tags coverage C All grammar paths C Pick from @examples C Use fixed sentence ? Generation can be started from any set of sentence patterns
  • 10. Sentence generation tool Interactive Demo
  • 11. Technique #2 Exploring sentence patterns ? Interactive expansion of grammar rules ? Derive sentence patterns C Useful to generate sentences for a specific pattern ? Derive complete sentences C Can be debugged, etc. ? Ideal to understand the structure of a grammar
  • 12. Sentence explorer tool Interactive Demo
  • 13. Three typical problems´ ´ and how to fix them! Interactive Demo