Property-based Testing of Attribute Grammars - Artifact

Property-Based Testing of Attribute Grammars

Authors

• José Nuno Macedo; Universidade do Minho; jose.n.macedo@inesctec.pt
• Marcos Viera; Universidad de la República; mviera@fing.edu.uy
• João Saraiva; Universidade do Minho; saraiva@di.uminho.pt

Associated Paper Information

• Title: Property-Based Testing of Attribute Grammars
• Abstract: Software testing is an integral part of modern software development. Testing frameworks are part of the toolset of any software language allowing programmers to test their programs in order to detect bugs. Unfortunately, there is no work on testing in attribute grammars. In this paper we combine the powerful property-based testing technique with the attribute grammar formalism. In such property-based attribute grammars, properties are defined on attribute instances. Properties are tested on large sets of randomly generated (abstract syntax) trees by evaluating their attributes. We present an implementation that relies on strategies to express property-based attribute grammars. Strategies are tree-based recursion patterns that are used to encode logic quantifiers defining the properties.

Documentation

The provided .zip file is a subset of the online repository available at: https://bitbucket.org/zenunomacedo/ztrategic/src/380e4fa48ef7b986c3640c8bab6503a6c7d32df8/

Structure

• stack.yaml
• • Stack configuration file
• package.yaml
• • Stack configuration file
• Dockerfile
• • Docker configuration
• docker-compose.yaml
• • Docker-compose configuration - facilitates Docker usage
• Library/
• • Contains the strategic programming library used to define quantifiers
• AGMaker/
• • Contains the Template Haskell code to automatically derive Attribute Grammar boilerplate code
• Examples/
• • Contains the base data types, attribute grammars and properties for the RepMin and Let examples. Notably:
  • • Examples/RepMin/RepMinProperty.hs - contains properties defined for the RepMin language.
    • Examples/Let/LetProperty.hs - contains properties defined for the Let language.
• Scopes/
• • Contains all code needed for the Haskell example. Notably:
  • • Scopes/HaskellProperty.hs - contains properties defined for the Haskell language.

Artifact Evaluation Environment

All artifacts are Haskell code expected to run in any system. The code was successfully executed in both Windows 11 and Ubuntu 22.04. This project requires several Haskell dependencies. For this, a Stack configuration was set up so that the dependencies are downloaded automatically. Alternatively: both docker and docker-compose can be used to run this artifact.

Requirements

• Either Stack > 2.0.0 - a project manager for Haskell. Note: Package managers such as apt might only contain old versions of Stack - the preferred way to install Stack is through Ghcup: https://www.haskell.org/ghcup/
• Or Docker - tested with version 27.5.1. Usage of docker-compose is recommended but not necessary.

Kick-the-Tires - Docker

For the local Stack installation instructions, see below

This artifact is set up for a one-time run, and it does not persist all cached Stack information. This means that subsequent usages of the artifact might entail long configuration times which would be avoided in a more proper development environment.

Steps to Perform a Quick Test

1. build docker image: Follow either of these steps
2. • docker-compose: sudo docker-compose build
   • docker: sudo docker build -t haskell-dev .
3. Run docker image: Follow either of these steps
4. • docker-compose: sudo docker-compose run haskell
   • docker: sudo docker run -it -v "$PWD":/sle -w /sle haskell-dev
5. Wait. First, Docker needs to unpack the Haskell image in step 1. Then, Stack needs to prepare the necessary project dependencies in step 2.
6. (Optional) Consider opening the following file in a code editor: Examples/RepMin/RepMinProperty.hs
7. Inside the interpreter, run the following command: repMinProperty
8. This should run all RepMin properties automatically. The user should see several occurrences of names of properties (starting with prop_) followed by +++ OK, passed 100 tests..
9. GHCi can be closed with the :quit command.

Full Evaluation - Docker

This artifact focuses on property-based testing of attribute grammars for RepMin, Let and Haskell. The steps above execute all properties for RepMin; the steps for Let and Haskell are similar, but there are some caveats:

Let

1. (Optional) Consider opening the following file in a code editor: Examples/Let/LetProperty.hs
2. Inside the interpreter opened before using docker, run the following command: letProperty
3. This should run all Let properties automatically. The user should see several occurrences of names of properties (starting with prop_), mostly followed by +++ OK, passed 100 tests.. Some properties might fail due to taking too long to execute. We welcome expert Haskell developers to study why the properties fail, what does that mean in their context, and how the code/properties could be improved accordingly.
4. These properties are using a standard Let generator. To swap to the circular Let generator, open file Examples/Let/LetGenerator.hs, comment line 13 and uncomment line 14. If needed, reload the file in GHCi with :reload. Then, run the properties again with letProperty.

Haskell

1. (Optional) Consider opening the following file in a code editor: Scopes/HaskellProperty.hs
2. Because we do not own a generator of Haskell programs, we are not able to use QuickCheck "properly" with Haskell. Still, we can run on concrete examples inside the interpreter opened before using docker:
3. • exRemove
   • • This is a string representing a very small valid Haskell program.
   • computeDeadDecl exRemove
   • • Function computeDeadDecl takes a string representing a program, and returns a pair, where the first element is the same program with dead code removed, and the second element is a list of removed dead names.
   • quickCheck (prop_noDeadCode (parse exRemove))
   • • We quickCheck the property prop_noDeadCode on the AST produced by parsing exRemove.
   • Repeat the previous steps with exRemoveBU instead of exRemove, or with a custom string representing another Haskell program.
4. The prop_noDeadCode property should fail in both cases as it considers main to be a dead function as it is not used anywhere - this property should be fixed to ignore the main name, just as computeDeadDecl does.
5. Since the computeDeadDecl function takes only 1 program, to test it, we merged the HaLex library into a single file in Scopes/GeneratedHalex.hs. As such, we can build a Block DSL representation of it (as described in the paper, Section 5):
6. • halex_block
7. (Extra) We can generate the same module without dead names, and/or compute the dead names. Let us just compute the dead names in the module. NOTE: This code is extremely inefficient. We ran this code on a compiled executable with all optimization flags turned on, and executable ran for almost 1 hour. We strongly suggest not to run this on the interpreter, and consider instead compiling the file.
8. • halex_deadCode

Kick-the-Tires - local Stack installation

For the Docker instructions, see above

To use these artifacts, it is only needed to install Stack and use stack ghci to open the standard Haskell interpreter. Stack will read the provided configuration files and automatically download and prepare all required dependencies.

Steps to Perform a Quick Test

1. Install Ghcup
2. • instructions available here: https://www.haskell.org/ghcup/
3. Use ghcup to install stack - ghcup install stack 3.3.1
4. • Most recent versions should work fine, but 3.3.1 is verified to work
5. Set active stack version - ghcup set stack 3.3.1
6. Run the following command: stack ghci Examples/RepMin/RepMinProperty.hs
7. • This will open RepMinProperty module with the Haskell interpreter. Stack might take a long time downloading and preparing dependencies for the first run.
8. Inside the interpreter, run the following command: runTests
9. This should run all RepMin properties automatically. The user should see several occurrences of names of properties (starting with prop_) followed by +++ OK, passed 100 tests..
10. GHCi can be closed with the :quit command.

Troubleshooting:

• If Stack fails to build the project, ensure that you have a recent version of Stack installed.
• The files produced by Stack are placed inside a .stack-work folder. If the setup is not working properly, delete this folder and try again.

Full Evaluation - local Stack installation

This artifact focuses on property-based testing of attribute grammars for RepMin, Let and Haskell. The steps to execute RepMin are provided above; the steps for Let and Haskell are similar, but there are some caveats:

Let

1. Run the following command: stack ghci Examples/Let/LetProperty.hs
2. • This will open LetProperty module with the Haskell interpreter. Stack might take a long time downloading and preparing dependencies for the first run.
3. Inside the interpreter, run the following command: runTests
4. This should run all Let properties automatically. The user should see several occurrences of names of properties (starting with prop_), mostly followed by +++ OK, passed 100 tests.. Some properties might fail due to taking too long to execute. We welcome expert Haskell developers to study why the properties fail, what does that mean in their context, and how the code/properties could be improved accordingly.
5. These properties are using a standard Let generator. To swap to the circular Let generator, open file Examples/Let/LetGenerator.hs, comment line 13 and uncomment line 14. If needed, reload the file in GHCi with :reload. Then, run the properties again with runTests.

Haskell

1. Run the following command: stack ghci Scopes/HaskellProperty.hs
2. Because we do not own a generator of Haskell programs, we are not able to use QuickCheck "properly" with Haskell. Still, we can run on concrete examples:
3. • exRemove
   • • This is a string representing a very small valid Haskell program.
   • computeDeadDecl exRemove
   • • Function computeDeadDecl takes a string representing a program, and returns a pair, where the first element is the same program with dead code removed, and the second element is a list of removed dead names.
   • quickCheck (prop_noDeadCode (parse exRemove))
   • • We quickCheck the property prop_noDeadCode on the AST produced by parsing exRemove.
   • Repeat the previous steps with exRemoveBU instead of exRemove, or with a custom string representing another Haskell program.
4. The prop_noDeadCode property should fail in both cases as it considers main to be a dead function as it is not used anywhere - this property should be fixed to ignore the main name, just as computeDeadDecl does.
5. Since the computeDeadDecl function takes only 1 program, to test it, we merged the HaLex library into a single file in Scopes/GeneratedHalex.hs. As such, we can build a Block DSL representation of it (as described in the paper, Section 5):
6. • halex_block
7. (Extra) We can generate the same module without dead names, and/or compute the dead names. Let us just compute the dead names in the module. NOTE: This code is extremely inefficient. We ran this code on a compiled executable with all optimization flags turned on, and executable ran for almost 1 hour. We strongly suggest not to run this on the interpreter, and consider instead compiling the file.
8. • halex_deadCode