{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Translating NL (English) to (Javascript) Property-based Tests\n", "\n", "This notebook replicates experiments setting up our paper.\n", "\n", "First, some imports and a hack to allow us to use the full width of the browser window for notebook content (parse trees can be quite wide)." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "from nltk.ccg import chart, lexicon" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from IPython.display import display, HTML\n", "# This allows the cells to take up most of the screen width on wider monitors; derivations can be quite wide.\n", "display(HTML(data=\"\"\"\n", "\n", "\"\"\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Lexicon setup\n", "\n", "First we define the lexicon, associating both grammatical types and semantics with each word, in some cases multiple types and semantics.\n", "\n", "Note that the NLTK CCG parser for semantics does *not* parse lambda expressions according to standard practice in PL. The standard assumption in PL is that after the dot separating arguments from the body, the scope of the body extends as far \"to the right\" as possible.\n", "NLTK instead stops parsing the body early, so \"\\x y.x and y\" is parsed as \"(\\x y.x) and y\" rather than \"\\x y.(x and y)\", which is why there are so many parentheses below." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "## Lexicon syntax notes:\n", "# Can use capital variables to bind predicates (though no currying, can only apply a lowercase constant or an uppercase predicate, not the result of another predicate)\n", "# No . in words, so 10.1 can't be written here\n", "lex_gen_check = lexicon.fromstring('''\n", " :- S, CN, ADJ, NP, VAR, DET, PP, AQUAL\n", " \n", " Quant :: (S/(S\\\\NP))/CN[Gen]\n", " \n", " than => PP[Than]/NP {\\\\x.x}\n", " to => PP[To]/NP {\\\\x.x}\n", " by => PP[By]/NP {\\\\x.x}\n", " greater => ADJ/PP[Than] {\\\\x y.lessthan(x,y)}\n", " less => ADJ/PP[Than] {\\\\x y.lessthan(y,x)}\n", " equal => ADJ/PP[To] {\\\\x y.equals(x,y)}\n", " divisible => ADJ/PP[By] {\\\\d n.divisibleby(n,d)}\n", " and => (ADJ/,.ADJ)\\,.ADJ {\\\\P Q x.(P(x) and Q(x))}\n", " and => ((ADJ/NP)/,.(ADJ/NP))\\\\,.(ADJ/NP) {\\\\P Q n x.(apply(P(n),x) and apply(Q(n),x))}\n", " and => ((ADJ\\\\,.(ADJ/NP))/,.NP)\\\\,.NP {\\\\a b P x.(apply(P(a),x) and apply(P(b),x))}\n", " or => (ADJ/,.ADJ)\\,.ADJ {\\\\P Q x.(P(x) or Q(x))}\n", " or => ((ADJ/NP)/,.(ADJ/NP))\\\\,.(ADJ/NP) {\\\\P Q n x.(apply(P(n),x) or apply(Q(n),x))}\n", " not => ADJ/ADJ {\\\\P x.not(P(x))}\n", " zero => NP {0}\n", " 1 => NP {1}\n", " 5 => NP {5}\n", " 3 => NP {3}\n", " 10 => NP {10}\n", " every => Quant {\\\\n p.foreach(n,p)}\n", " any => Quant {\\\\n p.foreach(n,p)}\n", " integer => CN[Gen] {gen_integers}\n", " number => CN[Chk] {isnumeric}\n", " number => CN[Gen] {gen_integers}\n", " float => CN[Gen] {gen_floats}\n", " a => DET {a}\n", " is => (S\\\\NP[Gen])/ADJ {\\\\A n.A(n)}\n", " is => ((S\\\\NP[Gen])/CN[Chk])/DET {\\\\d A n.A(n)}\n", " even => ADJ {\\\\x.even(x)}\n", " yes => S {yes}\n", " but => S/S {\\\\x.x}\n", "\n", " that => (CN[Gen]/(S\\\\NP[Gen]))\\\\CN[Gen] {\\\\P n.filter(P,n)}\n", " any => (((S/(S\\\\NP))/ADJ)/CN[Gen]) {\\\\n P p.foreach(filter(n,P),p)}\n", " for => ((((S/ADJ)/CN[Gen])/ADJ)/CN[Gen])/AQUAL {\\\\q g p F C.foreach(filter(g,p),\\\\x.C(F(x)))}\n", " for => ((((S/,.(S\\\\NP[Gen]))/,.CN[Gen])/,.ADJ)/,.CN[Gen])/,.AQUAL {\\\\q g p F C.foreach(filter(g,p),\\\\x.C(F(x)))}\n", "\n", " returns => ADJ/NP[Chk] {\\\\P x.P(x)}\n", " returns => (S\\\\NP[Gen])/NP[Chk] {\\\\P x.P(x)}\n", " any => AQUAL {any}\n", " exception => CN[Chk] {isexception}\n", " an => DET {an}\n", " throws => (S\\\\NP[Gen])/,.CN[Chk]/,.DET {\\\\d c n.checkthrows(c,\\\\u.n)} \n", " \n", " Fizz => NP[Chk] {\\\\x.(x=\"Fizz\")}\n", " Buzz => NP[Chk] {\\\\x.(x=\"Buzz\")}\n", " FizzBuzz => NP[Chk] {\\\\x.(x=\"FizzBuzz\")}\n", "\n", " passing => ADJ {\\\\x.passing(x)}\n", " passing => CN[Chk] {passing}\n", " fizzbuzz => CN[Chk] {fizzbuzz}\n", " \n", " positive => CN[Gen]/CN[Gen] {\\\\x.(filter(x,(\\\\u.(lessthan(0,u)))))}\n", "''', include_semantics=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Parsing\n", "\n", "The backwards type raise rule in NLTK has a bug that breaks semantics any time it \n", "applies to a category whose semantics are functions (e.g., ADJ), which results in\n", "NLTK breaking the semantics of \n", " and : (ADJ/ADJ)\\ADJ \n", "in \"P and Q\" if it first combines on the left (using composition),\n", "then applies ``` 0):\n", " print(\"\\nParses for: \"+t)\n", " chart.printCCGDerivation(parses[t][0])\n", " print(\"^^Parses for: \"+t+\" (\"+str(len(parses[t])-1)+\" additional parses)\")\n", " else:\n", " print(\"*************************\")\n", " noparse.append(t)\n", " print(\"Sentences without parses:\")\n", " for n in noparse:\n", " print(\"\\t\"+n)\n", " print(\"==================================\")\n", " print(\"Summary statistics on parse multiplicity:\")\n", " for t in tests:\n", " print(\"[\"+str(len(parses[t]))+\"] parses for: \"+t)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Kicking the Tires" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "([Tree((, '>'), [Tree((, '>T'), [Tree((, 'Leaf'), [Tree(, ['1'])])]), Tree((, '>'), [Tree((, 'Leaf'), [Tree(, ['is'])]), Tree((, '>'), [Tree((, 'Leaf'), [Tree(, ['less'])]), Tree((, '>'), [Tree((, 'Leaf'), [Tree(, ['than'])]), Tree((, 'Leaf'), [Tree(, ['5'])])])])])]),\n", " Tree((, '>'), [Tree((, '>T'), [Tree((, 'Leaf'), [Tree(, ['1'])])]), Tree((, '>'), [Tree((, 'Leaf'), [Tree(, ['is'])]), Tree((, '>'), [Tree((, '>B'), [Tree((, 'Leaf'), [Tree(, ['less'])]), Tree((, 'Leaf'), [Tree(, ['than'])])]), Tree((, 'Leaf'), [Tree(, ['5'])])])])]),\n", " Tree((, '>'), [Tree((, '>T'), [Tree((, 'Leaf'), [Tree(, ['1'])])]), Tree((, '>'), [Tree((, '>B'), [Tree((, 'Leaf'), [Tree(, ['is'])]), Tree((, 'Leaf'), [Tree(, ['less'])])]), Tree((, '>'), [Tree((, 'Leaf'), [Tree(, ['than'])]), Tree((, 'Leaf'), [Tree(, ['5'])])])])]),\n", " Tree((, '>'), [Tree((, '>T'), [Tree((, 'Leaf'), [Tree(, ['1'])])]), Tree((, '>'), [Tree((, '>B'), [Tree((, 'Leaf'), [Tree(, ['is'])]), Tree((, '>B'), [Tree((, 'Leaf'), [Tree(, ['less'])]), Tree((, 'Leaf'), [Tree(, ['than'])])])]), Tree((, 'Leaf'), [Tree(, ['5'])])])]),\n", " Tree((, '>'), [Tree((, '>T'), [Tree((, 'Leaf'), [Tree(, ['1'])])]), Tree((, '>'), [Tree((, '>B'), [Tree((, '>B'), [Tree((, 'Leaf'), [Tree(, ['is'])]), Tree((, 'Leaf'), [Tree(, ['less'])])]), Tree((, 'Leaf'), [Tree(, ['than'])])]), Tree((, 'Leaf'), [Tree(, ['5'])])])]),\n", " Tree((, '>'), [Tree((, '>B'), [Tree((, '>T'), [Tree((, 'Leaf'), [Tree(, ['1'])])]), Tree((, 'Leaf'), [Tree(, ['is'])])]), Tree((, '>'), [Tree((, 'Leaf'), [Tree(, ['less'])]), Tree((, '>'), [Tree((, 'Leaf'), [Tree(, ['than'])]), Tree((, 'Leaf'), [Tree(, ['5'])])])])]),\n", " Tree((, '>'), [Tree((, '>B'), [Tree((, '>T'), [Tree((, 'Leaf'), [Tree(, ['1'])])]), Tree((, 'Leaf'), [Tree(, ['is'])])]), Tree((, '>'), [Tree((, '>B'), [Tree((, 'Leaf'), [Tree(, ['less'])]), Tree((, 'Leaf'), [Tree(, ['than'])])]), Tree((, 'Leaf'), [Tree(, ['5'])])])]),\n", " Tree((, '>'), [Tree((, '>B'), [Tree((, '>T'), [Tree((, 'Leaf'), [Tree(, ['1'])])]), Tree((, '>B'), [Tree((, 'Leaf'), [Tree(, ['is'])]), Tree((, 'Leaf'), [Tree(, ['less'])])])]), Tree((, '>'), [Tree((, 'Leaf'), [Tree(, ['than'])]), Tree((, 'Leaf'), [Tree(, ['5'])])])]),\n", " Tree((, '>'), [Tree((, '>B'), [Tree((, '>B'), [Tree((, '>T'), [Tree((, 'Leaf'), [Tree(, ['1'])])]), Tree((, 'Leaf'), [Tree(, ['is'])])]), Tree((, 'Leaf'), [Tree(, ['less'])])]), Tree((, '>'), [Tree((, 'Leaf'), [Tree(, ['than'])]), Tree((, 'Leaf'), [Tree(, ['5'])])])]),\n", " Tree((, '>'), [Tree((, '>B'), [Tree((, '>T'), [Tree((, 'Leaf'), [Tree(, ['1'])])]), Tree((, '>B'), [Tree((, 'Leaf'), [Tree(, ['is'])]), Tree((, '>B'), [Tree((, 'Leaf'), [Tree(, ['less'])]), Tree((, 'Leaf'), [Tree(, ['than'])])])])]), Tree((, 'Leaf'), [Tree(, ['5'])])]),\n", " Tree((, '>'), [Tree((, '>B'), [Tree((, '>T'), [Tree((, 'Leaf'), [Tree(, ['1'])])]), Tree((, '>B'), [Tree((, '>B'), [Tree((, 'Leaf'), [Tree(, ['is'])]), Tree((, 'Leaf'), [Tree(, ['less'])])]), Tree((, 'Leaf'), [Tree(, ['than'])])])]), Tree((, 'Leaf'), [Tree(, ['5'])])]),\n", " Tree((, '>'), [Tree((, '>B'), [Tree((, '>B'), [Tree((, '>T'), [Tree((, 'Leaf'), [Tree(, ['1'])])]), Tree((, 'Leaf'), [Tree(, ['is'])])]), Tree((, '>B'), [Tree((, 'Leaf'), [Tree(, ['less'])]), Tree((, 'Leaf'), [Tree(, ['than'])])])]), Tree((, 'Leaf'), [Tree(, ['5'])])]),\n", " Tree((, '>'), [Tree((, '>B'), [Tree((, '>B'), [Tree((, '>T'), [Tree((, 'Leaf'), [Tree(, ['1'])])]), Tree((, '>B'), [Tree((, 'Leaf'), [Tree(, ['is'])]), Tree((, 'Leaf'), [Tree(, ['less'])])])]), Tree((, 'Leaf'), [Tree(, ['than'])])]), Tree((, 'Leaf'), [Tree(, ['5'])])]),\n", " Tree((, '>'), [Tree((, '>B'), [Tree((, '>B'), [Tree((, '>B'), [Tree((, '>T'), [Tree((, 'Leaf'), [Tree(, ['1'])])]), Tree((, 'Leaf'), [Tree(, ['is'])])]), Tree((, 'Leaf'), [Tree(, ['less'])])]), Tree((, 'Leaf'), [Tree(, ['than'])])]), Tree((, 'Leaf'), [Tree(, ['5'])])])],\n", " {: 14},\n", " {: 1})" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "processSentence(\"1 is less than 5\")" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "splitparser = chart.CCGChartParser(lex_gen_check, default_rules_without_nltk_bug)\n", "ps = list(splitparser.parse(\"1 is less than 5\".split()))\n", "s = sem(ps[0])" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", " Check for logical equivalence.\n", " Pass the expression (self <-> other) to the theorem prover.\n", " If the prover says it is valid, then the self and other are equal.\n", "\n", " :param other: an ``Expression`` to check equality against\n", " :param prover: a ``nltk.inference.api.Prover``\n", " \n" ] } ], "source": [ "dir(s)\n", "print(s.equiv.__doc__)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'lessthan(1,5)'" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "str(s)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Equivalence Checking\n", "\n", "NLTK has built-in support for checking equivalence of the logical forms generated by the CCG parsing process, however it relies on the very old Prover9 tool.\n", "\n", "If you don't have this in the right place, you'll get an exception from NLTK:\n", "```\n", "LookupError: \n", "\n", "===========================================================================\n", "NLTK was unable to find the prover9 file!\n", "Use software specific configuration paramaters or set the PROVER9 environment variable.\n", "\n", " Searched in:\n", " - /usr/local/bin/prover9\n", " - /usr/local/bin/prover9/bin\n", " - /usr/local/bin\n", " - /usr/bin\n", " - /usr/local/prover9\n", " - /usr/local/share/prover9\n", "\n", " For more information on prover9, see:\n", " \n", "===========================================================================\n", "```\n", "The source can be obtained from the link above, and after fixing one compilation error, the resulting ```prover9``` binary can be placed in any of the directories suggested by the error message. Unfortunately, those locations are hard-coded, and NLTK does not use the PATH environment variable when locating Prover9. (This mostly works as expected on Linux, but on MacOS the final copy of the binaries to a bin directory fails, and you have to copy the binary from ```/provers.src/prover9```." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# If prover9 is installed correctly, this will be true\n", "s.equiv(sem(ps[1]))" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "s2 = sem(list(splitparser.parse(\"1 is even\".split()))[0])" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "False" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s.equiv(s2)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Parses for: 1 is even\n", " 1 is even\n", " NP {1} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} ADJ {\\x.even(x)}\n", "-------->T\n", "(S/(S\\NP)) {\\F.F(1)}\n", " --------------------------------------------------->\n", " (S\\NP['Gen']) {\\n.even(n)}\n", "----------------------------------------------------------->\n", " S {even(1)}\n", "^^Parses for: 1 is even (1 additional parses)\n", "\n", "Parses for: every number is even\n", " every number is even\n", " ((S/(S\\NP))/CN['Gen']) {\\n p.foreach(n,p)} CN['Gen'] {gen_integers} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} ADJ {\\x.even(x)}\n", "---------------------------------------------------------------------->\n", " (S/(S\\NP)) {\\p.foreach(gen_integers,p)}\n", " --------------------------------------------------->\n", " (S\\NP['Gen']) {\\n.even(n)}\n", "------------------------------------------------------------------------------------------------------------------------->\n", " S {foreach(gen_integers,\\n.even(n))}\n", "^^Parses for: every number is even (1 additional parses)\n", "\n", "Parses for: every number is a number\n", " every number is a number\n", " ((S/(S\\NP))/CN['Gen']) {\\n p.foreach(n,p)} CN['Gen'] {gen_integers} (((S\\NP['Gen'])/CN['Chk'])/DET) {\\d A n.A(n)} DET {a} CN['Chk'] {isnumeric}\n", "---------------------------------------------------------------------->\n", " (S/(S\\NP)) {\\p.foreach(gen_integers,p)}\n", " -------------------------------------------------------->\n", " ((S\\NP['Gen'])/CN['Chk']) {\\A n.A(n)}\n", " ------------------------------------------------------------------------------->\n", " (S\\NP['Gen']) {\\n.isnumeric(n)}\n", "----------------------------------------------------------------------------------------------------------------------------------------------------->\n", " S {foreach(gen_integers,\\n.isnumeric(n))}\n", "^^Parses for: every number is a number (1 additional parses)\n", "\n", "Parses for: 1 is less than 5\n", " 1 is less than 5\n", " NP {1} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} (ADJ/PP['Than']) {\\x y.lessthan(y,x)} (PP['Than']/NP) {\\x.x} NP {5}\n", "-------->T\n", "(S/(S\\NP)) {\\F.F(1)}\n", " -------------------------------->\n", " PP['Than'] {5}\n", " ----------------------------------------------------------------------->\n", " ADJ {\\y.lessthan(y,5)}\n", " -------------------------------------------------------------------------------------------------------->\n", " (S\\NP['Gen']) {\\n.lessthan(n,5)}\n", "---------------------------------------------------------------------------------------------------------------->\n", " S {lessthan(1,5)}\n", "^^Parses for: 1 is less than 5 (13 additional parses)\n", "\n", "Parses for: 1 is equal to 5\n", " 1 is equal to 5\n", " NP {1} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} (ADJ/PP['To']) {\\x y.equals(x,y)} (PP['To']/NP) {\\x.x} NP {5}\n", "-------->T\n", "(S/(S\\NP)) {\\F.F(1)}\n", " ------------------------------>\n", " PP['To'] {5}\n", " ----------------------------------------------------------------->\n", " ADJ {\\y.equals(5,y)}\n", " -------------------------------------------------------------------------------------------------->\n", " (S\\NP['Gen']) {\\n.equals(5,n)}\n", "---------------------------------------------------------------------------------------------------------->\n", " S {equals(5,1)}\n", "^^Parses for: 1 is equal to 5 (13 additional parses)\n", "\n", "Parses for: 1 is less than or equal to 5\n", " 1 is less than or equal to 5\n", " NP {1} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} (ADJ/PP['Than']) {\\x y.lessthan(y,x)} (PP['Than']/NP) {\\x.x} (((ADJ/NP)/,.(ADJ/NP))\\,.(ADJ/NP)) {\\P Q n x.(apply(P(n),x) | apply(Q(n),x))} (ADJ/PP['To']) {\\x y.equals(x,y)} (PP['To']/NP) {\\x.x} NP {5}\n", "-------->T\n", "(S/(S\\NP)) {\\F.F(1)}\n", " --------------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.lessthan(y,x)}\n", " ----------------------------------------------------------------------------------------------------------------------------------------------<\n", " ((ADJ/NP)/,.(ADJ/NP)) {\\Q n x.(apply(\\y.lessthan(y,n),x) | apply(Q(n),x))}\n", " --------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.equals(x,y)}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (ADJ/NP) {\\n x.(apply(\\y.lessthan(y,n),x) | apply(\\y.equals(n,y),x))}\n", " --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(apply(\\y.lessthan(y,5),x) | apply(\\y.equals(5,y),x))}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " (S\\NP['Gen']) {\\n.(apply(\\y.lessthan(y,5),n) | apply(\\y.equals(5,y),n))}\n", "-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " S {(apply(\\y.lessthan(y,5),1) | apply(\\y.equals(5,y),1))}\n", "^^Parses for: 1 is less than or equal to 5 (4 additional parses)\n", "\n", "Parses for: 10 is passing\n", " 10 is passing\n", " NP {10} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} ADJ {\\x.passing(x)}\n", "--------->T\n", "(S/(S\\NP)) {\\F.F(10)}\n", " ------------------------------------------------------>\n", " (S\\NP['Gen']) {\\n.passing(n)}\n", "--------------------------------------------------------------->\n", " S {passing(10)}\n", "^^Parses for: 10 is passing (1 additional parses)\n", "\n", "Parses for: 1 is not passing\n", " 1 is not passing\n", " NP {1} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} (ADJ/ADJ) {\\P x.-P(x)} ADJ {\\x.passing(x)}\n", "-------->T\n", "(S/(S\\NP)) {\\F.F(1)}\n", " --------------------------------------------->\n", " ADJ {\\x.-passing(x)}\n", " ------------------------------------------------------------------------------>\n", " (S\\NP['Gen']) {\\n.-passing(n)}\n", "-------------------------------------------------------------------------------------->\n", " S {-passing(1)}\n", "^^Parses for: 1 is not passing (4 additional parses)\n", "\n", "Parses for: any float that is greater than or equal to 1 and less than 5 is not passing\n", " any float that is greater than or equal to 1 and less than 5 is not passing\n", " ((S/(S\\NP))/CN['Gen']) {\\n p.foreach(n,p)} CN['Gen'] {gen_floats} ((CN['Gen']/(S\\NP['Gen']))\\CN['Gen']) {\\P n.filter(P,n)} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} (ADJ/PP['Than']) {\\x y.lessthan(x,y)} (PP['Than']/NP) {\\x.x} (((ADJ/NP)/,.(ADJ/NP))\\,.(ADJ/NP)) {\\P Q n x.(apply(P(n),x) | apply(Q(n),x))} (ADJ/PP['To']) {\\x y.equals(x,y)} (PP['To']/NP) {\\x.x} NP {1} ((ADJ/,.ADJ)\\,.ADJ) {\\P Q x.(P(x) & Q(x))} (ADJ/PP['Than']) {\\x y.lessthan(y,x)} (PP['Than']/NP) {\\x.x} NP {5} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} (ADJ/ADJ) {\\P x.-P(x)} ADJ {\\x.passing(x)}\n", " ----------------------------------------------------------------------------------<\n", " (CN['Gen']/(S\\NP['Gen'])) {\\n.filter(gen_floats,n)}\n", " --------------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.lessthan(x,y)}\n", " ----------------------------------------------------------------------------------------------------------------------------------------------<\n", " ((ADJ/NP)/,.(ADJ/NP)) {\\Q n x.(apply(\\y.lessthan(n,y),x) | apply(Q(n),x))}\n", " --------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.equals(x,y)}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (ADJ/NP) {\\n x.(apply(\\y.lessthan(n,y),x) | apply(\\y.equals(n,y),x))}\n", " --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(apply(\\y.lessthan(1,y),x) | apply(\\y.equals(1,y),x))}\n", " -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<\n", " (ADJ/,.ADJ) {\\Q x.((apply(\\y.lessthan(1,y),x) | apply(\\y.equals(1,y),x)) & Q(x))}\n", " -------------------------------->\n", " PP['Than'] {5}\n", " ----------------------------------------------------------------------->\n", " ADJ {\\y.lessthan(y,5)}\n", " ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.((apply(\\y.lessthan(1,y),x) | apply(\\y.equals(1,y),x)) & lessthan(x,5))}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (S\\NP['Gen']) {\\n.((apply(\\y.lessthan(1,y),n) | apply(\\y.equals(1,y),n)) & lessthan(n,5))}\n", " ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " CN['Gen'] {filter(gen_floats,\\n.((apply(\\y.lessthan(1,y),n) | apply(\\y.equals(1,y),n)) & lessthan(n,5)))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (S/(S\\NP)) {\\p.foreach(filter(gen_floats,\\n.((apply(\\y.lessthan(1,y),n) | apply(\\y.equals(1,y),n)) & lessthan(n,5))),p)}\n", " --------------------------------------------->\n", " ADJ {\\x.-passing(x)}\n", " ------------------------------------------------------------------------------>\n", " (S\\NP['Gen']) {\\n.-passing(n)}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " S {foreach(filter(gen_floats,\\n.((apply(\\y.lessthan(1,y),n) | apply(\\y.equals(1,y),n)) & lessthan(n,5))),\\n.-passing(n))}\n", "^^Parses for: any float that is greater than or equal to 1 and less than 5 is not passing (99 additional parses)\n", "\n", "Parses for: any float is not passing\n", " any float is not passing\n", " ((S/(S\\NP))/CN['Gen']) {\\n p.foreach(n,p)} CN['Gen'] {gen_floats} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} (ADJ/ADJ) {\\P x.-P(x)} ADJ {\\x.passing(x)}\n", "-------------------------------------------------------------------->\n", " (S/(S\\NP)) {\\p.foreach(gen_floats,p)}\n", " --------------------------------------------->\n", " ADJ {\\x.-passing(x)}\n", " ------------------------------------------------------------------------------>\n", " (S\\NP['Gen']) {\\n.-passing(n)}\n", "-------------------------------------------------------------------------------------------------------------------------------------------------->\n", " S {foreach(gen_floats,\\n.-passing(n))}\n", "^^Parses for: any float is not passing (4 additional parses)\n", "\n", "Parses for: any float less than 5 is not passing\n", " any float less than 5 is not passing\n", " (((S/(S\\NP))/ADJ)/CN['Gen']) {\\n P p.foreach(filter(n,P),p)} CN['Gen'] {gen_floats} (ADJ/PP['Than']) {\\x y.lessthan(y,x)} (PP['Than']/NP) {\\x.x} NP {5} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} (ADJ/ADJ) {\\P x.-P(x)} ADJ {\\x.passing(x)}\n", "-------------------------------------------------------------------------------------->\n", " ((S/(S\\NP))/ADJ) {\\P p.foreach(filter(gen_floats,P),p)}\n", " -------------------------------->\n", " PP['Than'] {5}\n", " ----------------------------------------------------------------------->\n", " ADJ {\\y.lessthan(y,5)}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (S/(S\\NP)) {\\p.foreach(filter(gen_floats,\\y.lessthan(y,5)),p)}\n", " --------------------------------------------->\n", " ADJ {\\x.-passing(x)}\n", " ------------------------------------------------------------------------------>\n", " (S\\NP['Gen']) {\\n.-passing(n)}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " S {foreach(filter(gen_floats,\\y.lessthan(y,5)),\\n.-passing(n))}\n", "^^Parses for: any float less than 5 is not passing (24 additional parses)\n", "\n", "Parses for: any float that is passing is greater than or equal to 5 or less than or equal to 10\n", " any float that is passing is greater than or equal to 5 or less than or equal to 10\n", " ((S/(S\\NP))/CN['Gen']) {\\n p.foreach(n,p)} CN['Gen'] {gen_floats} ((CN['Gen']/(S\\NP['Gen']))\\CN['Gen']) {\\P n.filter(P,n)} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} ADJ {\\x.passing(x)} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} (ADJ/PP['Than']) {\\x y.lessthan(x,y)} (PP['Than']/NP) {\\x.x} (((ADJ/NP)/,.(ADJ/NP))\\,.(ADJ/NP)) {\\P Q n x.(apply(P(n),x) | apply(Q(n),x))} (ADJ/PP['To']) {\\x y.equals(x,y)} (PP['To']/NP) {\\x.x} NP {5} ((ADJ/,.ADJ)\\,.ADJ) {\\P Q x.(P(x) | Q(x))} (ADJ/PP['Than']) {\\x y.lessthan(y,x)} (PP['Than']/NP) {\\x.x} (((ADJ/NP)/,.(ADJ/NP))\\,.(ADJ/NP)) {\\P Q n x.(apply(P(n),x) | apply(Q(n),x))} (ADJ/PP['To']) {\\x y.equals(x,y)} (PP['To']/NP) {\\x.x} NP {10}\n", " ----------------------------------------------------------------------------------<\n", " (CN['Gen']/(S\\NP['Gen'])) {\\n.filter(gen_floats,n)}\n", " ------------------------------------------------------>\n", " (S\\NP['Gen']) {\\n.passing(n)}\n", " ---------------------------------------------------------------------------------------------------------------------------------------->\n", " CN['Gen'] {filter(gen_floats,\\n.passing(n))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " (S/(S\\NP)) {\\p.foreach(filter(gen_floats,\\n.passing(n)),p)}\n", " --------------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.lessthan(x,y)}\n", " ----------------------------------------------------------------------------------------------------------------------------------------------<\n", " ((ADJ/NP)/,.(ADJ/NP)) {\\Q n x.(apply(\\y.lessthan(n,y),x) | apply(Q(n),x))}\n", " --------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.equals(x,y)}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (ADJ/NP) {\\n x.(apply(\\y.lessthan(n,y),x) | apply(\\y.equals(n,y),x))}\n", " --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(apply(\\y.lessthan(5,y),x) | apply(\\y.equals(5,y),x))}\n", " -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<\n", " (ADJ/,.ADJ) {\\Q x.(apply(\\y.lessthan(5,y),x) | apply(\\y.equals(5,y),x) | Q(x))}\n", " --------------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.lessthan(y,x)}\n", " ----------------------------------------------------------------------------------------------------------------------------------------------<\n", " ((ADJ/NP)/,.(ADJ/NP)) {\\Q n x.(apply(\\y.lessthan(y,n),x) | apply(Q(n),x))}\n", " --------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.equals(x,y)}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (ADJ/NP) {\\n x.(apply(\\y.lessthan(y,n),x) | apply(\\y.equals(n,y),x))}\n", " ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(apply(\\y.lessthan(y,10),x) | apply(\\y.equals(10,y),x))}\n", " --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(apply(\\y.lessthan(5,y),x) | apply(\\y.equals(5,y),x) | apply(\\y.lessthan(y,10),x) | apply(\\y.equals(10,y),x))}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " (S\\NP['Gen']) {\\n.(apply(\\y.lessthan(5,y),n) | apply(\\y.equals(5,y),n) | apply(\\y.lessthan(y,10),n) | apply(\\y.equals(10,y),n))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " S {foreach(filter(gen_floats,\\n.passing(n)),\\n.(apply(\\y.lessthan(5,y),n) | apply(\\y.equals(5,y),n) | apply(\\y.lessthan(y,10),n) | apply(\\y.equals(10,y),n)))}\n", "^^Parses for: any float that is passing is greater than or equal to 5 or less than or equal to 10 (19 additional parses)\n", "\n", "Parses for: 1 is an exception\n", " 1 is an exception\n", " NP {1} (((S\\NP['Gen'])/CN['Chk'])/DET) {\\d A n.A(n)} DET {an} CN['Chk'] {isexception}\n", "-------->T\n", "(S/(S\\NP)) {\\F.F(1)}\n", " --------------------------------------------------------->\n", " ((S\\NP['Gen'])/CN['Chk']) {\\A n.A(n)}\n", " ---------------------------------------------------------------------------------->\n", " (S\\NP['Gen']) {\\n.isexception(n)}\n", "------------------------------------------------------------------------------------------>\n", " S {isexception(1)}\n", "^^Parses for: 1 is an exception (1 additional parses)\n", "\n", "Parses for: 1 throws an exception\n", " 1 throws an exception\n", " NP {1} (((S\\NP['Gen'])/,.CN['Chk'])/,.DET) {\\d c n.checkthrows(c,\\u.n)} DET {an} CN['Chk'] {isexception}\n", "-------->T\n", "(S/(S\\NP)) {\\F.F(1)}\n", " ---------------------------------------------------------------------------->\n", " ((S\\NP['Gen'])/,.CN['Chk']) {\\c n.checkthrows(c,\\u.n)}\n", " ----------------------------------------------------------------------------------------------------->\n", " (S\\NP['Gen']) {\\n.checkthrows(isexception,\\u.n)}\n", "------------------------------------------------------------------------------------------------------------->\n", " S {checkthrows(isexception,\\u.1)}\n", "^^Parses for: 1 throws an exception (0 additional parses)\n", "Sentences without parses:\n", "==================================\n", "Summary statistics on parse multiplicity:\n", "[2] parses for: 1 is even\n", "[2] parses for: every number is even\n", "[2] parses for: every number is a number\n", "[14] parses for: 1 is less than 5\n", "[14] parses for: 1 is equal to 5\n", "[5] parses for: 1 is less than or equal to 5\n", "[2] parses for: 10 is passing\n", "[5] parses for: 1 is not passing\n", "[100] parses for: any float that is greater than or equal to 1 and less than 5 is not passing\n", "[5] parses for: any float is not passing\n", "[25] parses for: any float less than 5 is not passing\n", "[20] parses for: any float that is passing is greater than or equal to 5 or less than or equal to 10\n", "[2] parses for: 1 is an exception\n", "[1] parses for: 1 throws an exception\n" ] } ], "source": [ "tests = [\n", " \"1 is even\",\n", " \"every number is even\",\n", " \"every number is a number\",\n", " \"1 is less than 5\",\n", " \"1 is equal to 5\",\n", " \"1 is less than or equal to 5\",\n", " \"10 is passing\",\n", " \"1 is not passing\",\n", " \"any float that is greater than or equal to 1 and less than 5 is not passing\",\n", " \"any float is not passing\",\n", " \"any float less than 5 is not passing\",\n", " \"any float that is passing is greater than or equal to 5 or less than or equal to 10\",\n", " \"1 is an exception\",\n", " \"1 throws an exception\"\n", "]\n", "trySentences(tests)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## STTP Experiments" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Parses for: any float greater than or equal to 1 and less than 5 is not passing\n", " any float greater than or equal to 1 and less than 5 is not passing\n", " (((S/(S\\NP))/ADJ)/CN['Gen']) {\\n P p.foreach(filter(n,P),p)} CN['Gen'] {gen_floats} (ADJ/PP['Than']) {\\x y.lessthan(x,y)} (PP['Than']/NP) {\\x.x} (((ADJ/NP)/,.(ADJ/NP))\\,.(ADJ/NP)) {\\P Q n x.(apply(P(n),x) | apply(Q(n),x))} (ADJ/PP['To']) {\\x y.equals(x,y)} (PP['To']/NP) {\\x.x} NP {1} ((ADJ/,.ADJ)\\,.ADJ) {\\P Q x.(P(x) & Q(x))} (ADJ/PP['Than']) {\\x y.lessthan(y,x)} (PP['Than']/NP) {\\x.x} NP {5} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} (ADJ/ADJ) {\\P x.-P(x)} ADJ {\\x.passing(x)}\n", "-------------------------------------------------------------------------------------->\n", " ((S/(S\\NP))/ADJ) {\\P p.foreach(filter(gen_floats,P),p)}\n", " --------------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.lessthan(x,y)}\n", " ----------------------------------------------------------------------------------------------------------------------------------------------<\n", " ((ADJ/NP)/,.(ADJ/NP)) {\\Q n x.(apply(\\y.lessthan(n,y),x) | apply(Q(n),x))}\n", " --------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.equals(x,y)}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (ADJ/NP) {\\n x.(apply(\\y.lessthan(n,y),x) | apply(\\y.equals(n,y),x))}\n", " --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(apply(\\y.lessthan(1,y),x) | apply(\\y.equals(1,y),x))}\n", " -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<\n", " (ADJ/,.ADJ) {\\Q x.((apply(\\y.lessthan(1,y),x) | apply(\\y.equals(1,y),x)) & Q(x))}\n", " -------------------------------->\n", " PP['Than'] {5}\n", " ----------------------------------------------------------------------->\n", " ADJ {\\y.lessthan(y,5)}\n", " ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.((apply(\\y.lessthan(1,y),x) | apply(\\y.equals(1,y),x)) & lessthan(x,5))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " (S/(S\\NP)) {\\p.foreach(filter(gen_floats,\\x.((apply(\\y.lessthan(1,y),x) | apply(\\y.equals(1,y),x)) & lessthan(x,5))),p)}\n", " --------------------------------------------->\n", " ADJ {\\x.-passing(x)}\n", " ------------------------------------------------------------------------------>\n", " (S\\NP['Gen']) {\\n.-passing(n)}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " S {foreach(filter(gen_floats,\\x.((apply(\\y.lessthan(1,y),x) | apply(\\y.equals(1,y),x)) & lessthan(x,5))),\\n.-passing(n))}\n", "^^Parses for: any float greater than or equal to 1 and less than 5 is not passing (19 additional parses)\n", "\n", "Parses for: any float greater than or equal to 5 and less than or equal to 10 is passing\n", " any float greater than or equal to 5 and less than or equal to 10 is passing\n", " (((S/(S\\NP))/ADJ)/CN['Gen']) {\\n P p.foreach(filter(n,P),p)} CN['Gen'] {gen_floats} (ADJ/PP['Than']) {\\x y.lessthan(x,y)} (PP['Than']/NP) {\\x.x} (((ADJ/NP)/,.(ADJ/NP))\\,.(ADJ/NP)) {\\P Q n x.(apply(P(n),x) | apply(Q(n),x))} (ADJ/PP['To']) {\\x y.equals(x,y)} (PP['To']/NP) {\\x.x} NP {5} ((ADJ/,.ADJ)\\,.ADJ) {\\P Q x.(P(x) & Q(x))} (ADJ/PP['Than']) {\\x y.lessthan(y,x)} (PP['Than']/NP) {\\x.x} (((ADJ/NP)/,.(ADJ/NP))\\,.(ADJ/NP)) {\\P Q n x.(apply(P(n),x) | apply(Q(n),x))} (ADJ/PP['To']) {\\x y.equals(x,y)} (PP['To']/NP) {\\x.x} NP {10} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} ADJ {\\x.passing(x)}\n", "-------------------------------------------------------------------------------------->\n", " ((S/(S\\NP))/ADJ) {\\P p.foreach(filter(gen_floats,P),p)}\n", " --------------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.lessthan(x,y)}\n", " ----------------------------------------------------------------------------------------------------------------------------------------------<\n", " ((ADJ/NP)/,.(ADJ/NP)) {\\Q n x.(apply(\\y.lessthan(n,y),x) | apply(Q(n),x))}\n", " --------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.equals(x,y)}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (ADJ/NP) {\\n x.(apply(\\y.lessthan(n,y),x) | apply(\\y.equals(n,y),x))}\n", " --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(apply(\\y.lessthan(5,y),x) | apply(\\y.equals(5,y),x))}\n", " -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<\n", " (ADJ/,.ADJ) {\\Q x.((apply(\\y.lessthan(5,y),x) | apply(\\y.equals(5,y),x)) & Q(x))}\n", " --------------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.lessthan(y,x)}\n", " ----------------------------------------------------------------------------------------------------------------------------------------------<\n", " ((ADJ/NP)/,.(ADJ/NP)) {\\Q n x.(apply(\\y.lessthan(y,n),x) | apply(Q(n),x))}\n", " --------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.equals(x,y)}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (ADJ/NP) {\\n x.(apply(\\y.lessthan(y,n),x) | apply(\\y.equals(n,y),x))}\n", " ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(apply(\\y.lessthan(y,10),x) | apply(\\y.equals(10,y),x))}\n", " --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.((apply(\\y.lessthan(5,y),x) | apply(\\y.equals(5,y),x)) & (apply(\\y.lessthan(y,10),x) | apply(\\y.equals(10,y),x)))}\n", "----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (S/(S\\NP)) {\\p.foreach(filter(gen_floats,\\x.((apply(\\y.lessthan(5,y),x) | apply(\\y.equals(5,y),x)) & (apply(\\y.lessthan(y,10),x) | apply(\\y.equals(10,y),x)))),p)}\n", " ------------------------------------------------------>\n", " (S\\NP['Gen']) {\\n.passing(n)}\n", "----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " S {foreach(filter(gen_floats,\\x.((apply(\\y.lessthan(5,y),x) | apply(\\y.equals(5,y),x)) & (apply(\\y.lessthan(y,10),x) | apply(\\y.equals(10,y),x)))),\\n.passing(n))}\n", "^^Parses for: any float greater than or equal to 5 and less than or equal to 10 is passing (3 additional parses)\n", "\n", "Parses for: for any float less than 1 or greater than 10 passing throws an exception\n", " for any float less than 1 or greater than 10 passing throws an exception\n", " (((((S/,.(S\\NP['Gen']))/,.CN['Gen'])/,.ADJ)/,.CN['Gen'])/,.AQUAL) {\\q g p F C.foreach(filter(g,p),\\x.C(F(x)))} AQUAL {any} CN['Gen'] {gen_floats} (ADJ/PP['Than']) {\\x y.lessthan(y,x)} (PP['Than']/NP) {\\x.x} NP {1} ((ADJ/,.ADJ)\\,.ADJ) {\\P Q x.(P(x) | Q(x))} (ADJ/PP['Than']) {\\x y.lessthan(x,y)} (PP['Than']/NP) {\\x.x} NP {10} CN['Gen'] {passing} (((S\\NP['Gen'])/,.CN['Chk'])/,.DET) {\\d c n.checkthrows(c,\\u.n)} DET {an} CN['Chk'] {isexception}\n", "----------------------------------------------------------------------------------------------------------------------------->\n", " ((((S/,.(S\\NP['Gen']))/,.CN['Gen'])/,.ADJ)/,.CN['Gen']) {\\g p F C.foreach(filter(g,p),\\x.C(F(x)))}\n", "----------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (((S/,.(S\\NP['Gen']))/,.CN['Gen'])/,.ADJ) {\\p F C.foreach(filter(gen_floats,p),\\x.C(F(x)))}\n", " -------------------------------->\n", " PP['Than'] {1}\n", " ----------------------------------------------------------------------->\n", " ADJ {\\y.lessthan(y,1)}\n", " -------------------------------------------------------------------------------------------------------------------<\n", " (ADJ/,.ADJ) {\\Q x.(lessthan(x,1) | Q(x))}\n", " --------------------------------->\n", " PP['Than'] {10}\n", " ------------------------------------------------------------------------>\n", " ADJ {\\y.lessthan(10,y)}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(lessthan(x,1) | lessthan(10,x))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " ((S/,.(S\\NP['Gen']))/,.CN['Gen']) {\\F C.foreach(filter(gen_floats,\\x.(lessthan(x,1) | lessthan(10,x))),\\x.C(F(x)))}\n", "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (S/,.(S\\NP['Gen'])) {\\C.foreach(filter(gen_floats,\\x.(lessthan(x,1) | lessthan(10,x))),\\x.C(passing(x)))}\n", " ---------------------------------------------------------------------------->\n", " ((S\\NP['Gen'])/,.CN['Chk']) {\\c n.checkthrows(c,\\u.n)}\n", " ----------------------------------------------------------------------------------------------------->\n", " (S\\NP['Gen']) {\\n.checkthrows(isexception,\\u.n)}\n", "-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " S {foreach(filter(gen_floats,\\x.(lessthan(x,1) | lessthan(10,x))),\\x.checkthrows(isexception,\\u.passing(x)))}\n", "^^Parses for: for any float less than 1 or greater than 10 passing throws an exception (5 additional parses)\n", "\n", "Parses for: for any number divisible by 3 and not divisible by 5 fizzbuzz returns Fizz\n", " for any number divisible by 3 and not divisible by 5 fizzbuzz returns Fizz\n", " (((((S/ADJ)/CN['Gen'])/ADJ)/CN['Gen'])/AQUAL) {\\q g p F C.foreach(filter(g,p),\\x.C(F(x)))} AQUAL {any} CN['Gen'] {gen_integers} (ADJ/PP['By']) {\\d n.divisibleby(n,d)} (PP['By']/NP) {\\x.x} NP {3} ((ADJ/,.ADJ)\\,.ADJ) {\\P Q x.(P(x) & Q(x))} (ADJ/ADJ) {\\P x.-P(x)} (ADJ/PP['By']) {\\d n.divisibleby(n,d)} (PP['By']/NP) {\\x.x} NP {5} CN['Gen'] {fizzbuzz} (ADJ/NP['Chk']) {\\P x.P(x)} NP['Chk'] {\\x.(x = \"Fizz\")}\n", "--------------------------------------------------------------------------------------------------------->\n", " ((((S/ADJ)/CN['Gen'])/ADJ)/CN['Gen']) {\\g p F C.foreach(filter(g,p),\\x.C(F(x)))}\n", "----------------------------------------------------------------------------------------------------------------------------------->\n", " (((S/ADJ)/CN['Gen'])/ADJ) {\\p F C.foreach(filter(gen_integers,p),\\x.C(F(x)))}\n", " ------------------------------>\n", " PP['By'] {3}\n", " ---------------------------------------------------------------------->\n", " ADJ {\\n.divisibleby(n,3)}\n", " ------------------------------------------------------------------------------------------------------------------<\n", " (ADJ/,.ADJ) {\\Q x.(divisibleby(x,3) & Q(x))}\n", " ------------------------------>\n", " PP['By'] {5}\n", " ---------------------------------------------------------------------->\n", " ADJ {\\n.divisibleby(n,5)}\n", " ---------------------------------------------------------------------------------------------->\n", " ADJ {\\x.-divisibleby(x,5)}\n", " ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(divisibleby(x,3) & -divisibleby(x,5))}\n", "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ((S/ADJ)/CN['Gen']) {\\F C.foreach(filter(gen_integers,\\x.(divisibleby(x,3) & -divisibleby(x,5))),\\x.C(F(x)))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (S/ADJ) {\\C.foreach(filter(gen_integers,\\x.(divisibleby(x,3) & -divisibleby(x,5))),\\x.C(fizzbuzz(x)))}\n", " ---------------------------------------------------------->\n", " ADJ {\\x.(x = \"Fizz\")}\n", "----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " S {foreach(filter(gen_integers,\\x.(divisibleby(x,3) & -divisibleby(x,5))),\\x.(fizzbuzz(x) = \"Fizz\"))}\n", "^^Parses for: for any number divisible by 3 and not divisible by 5 fizzbuzz returns Fizz (44 additional parses)\n", "\n", "Parses for: for any number divisible by 5 and not divisible by 3 fizzbuzz returns Buzz\n", " for any number divisible by 5 and not divisible by 3 fizzbuzz returns Buzz\n", " (((((S/ADJ)/CN['Gen'])/ADJ)/CN['Gen'])/AQUAL) {\\q g p F C.foreach(filter(g,p),\\x.C(F(x)))} AQUAL {any} CN['Gen'] {gen_integers} (ADJ/PP['By']) {\\d n.divisibleby(n,d)} (PP['By']/NP) {\\x.x} NP {5} ((ADJ/,.ADJ)\\,.ADJ) {\\P Q x.(P(x) & Q(x))} (ADJ/ADJ) {\\P x.-P(x)} (ADJ/PP['By']) {\\d n.divisibleby(n,d)} (PP['By']/NP) {\\x.x} NP {3} CN['Gen'] {fizzbuzz} (ADJ/NP['Chk']) {\\P x.P(x)} NP['Chk'] {\\x.(x = \"Buzz\")}\n", "--------------------------------------------------------------------------------------------------------->\n", " ((((S/ADJ)/CN['Gen'])/ADJ)/CN['Gen']) {\\g p F C.foreach(filter(g,p),\\x.C(F(x)))}\n", "----------------------------------------------------------------------------------------------------------------------------------->\n", " (((S/ADJ)/CN['Gen'])/ADJ) {\\p F C.foreach(filter(gen_integers,p),\\x.C(F(x)))}\n", " ------------------------------>\n", " PP['By'] {5}\n", " ---------------------------------------------------------------------->\n", " ADJ {\\n.divisibleby(n,5)}\n", " ------------------------------------------------------------------------------------------------------------------<\n", " (ADJ/,.ADJ) {\\Q x.(divisibleby(x,5) & Q(x))}\n", " ------------------------------>\n", " PP['By'] {3}\n", " ---------------------------------------------------------------------->\n", " ADJ {\\n.divisibleby(n,3)}\n", " ---------------------------------------------------------------------------------------------->\n", " ADJ {\\x.-divisibleby(x,3)}\n", " ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(divisibleby(x,5) & -divisibleby(x,3))}\n", "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ((S/ADJ)/CN['Gen']) {\\F C.foreach(filter(gen_integers,\\x.(divisibleby(x,5) & -divisibleby(x,3))),\\x.C(F(x)))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (S/ADJ) {\\C.foreach(filter(gen_integers,\\x.(divisibleby(x,5) & -divisibleby(x,3))),\\x.C(fizzbuzz(x)))}\n", " ---------------------------------------------------------->\n", " ADJ {\\x.(x = \"Buzz\")}\n", "----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " S {foreach(filter(gen_integers,\\x.(divisibleby(x,5) & -divisibleby(x,3))),\\x.(fizzbuzz(x) = \"Buzz\"))}\n", "^^Parses for: for any number divisible by 5 and not divisible by 3 fizzbuzz returns Buzz (44 additional parses)\n", "\n", "Parses for: for any number divisible by 5 and 3 fizzbuzz returns FizzBuzz\n", " for any number divisible by 5 and 3 fizzbuzz returns FizzBuzz\n", " (((((S/ADJ)/CN['Gen'])/ADJ)/CN['Gen'])/AQUAL) {\\q g p F C.foreach(filter(g,p),\\x.C(F(x)))} AQUAL {any} CN['Gen'] {gen_integers} (ADJ/PP['By']) {\\d n.divisibleby(n,d)} (PP['By']/NP) {\\x.x} NP {5} (((ADJ\\,.(ADJ/NP))/,.NP)\\,.NP) {\\a b P x.(apply(P(a),x) & apply(P(b),x))} NP {3} CN['Gen'] {fizzbuzz} (ADJ/NP['Chk']) {\\P x.P(x)} NP['Chk'] {\\x.(x = \"FizzBuzz\")}\n", "--------------------------------------------------------------------------------------------------------->\n", " ((((S/ADJ)/CN['Gen'])/ADJ)/CN['Gen']) {\\g p F C.foreach(filter(g,p),\\x.C(F(x)))}\n", "----------------------------------------------------------------------------------------------------------------------------------->\n", " (((S/ADJ)/CN['Gen'])/ADJ) {\\p F C.foreach(filter(gen_integers,p),\\x.C(F(x)))}\n", " -------------------------------------------------------------->B\n", " (ADJ/NP) {\\x n.divisibleby(n,x)}\n", " -----------------------------------------------------------------------------------<\n", " ((ADJ\\,.(ADJ/NP))/,.NP) {\\b P x.(apply(P(5),x) & apply(P(b),x))}\n", " ------------------------------------------------------------------------------------------->\n", " (ADJ\\,.(ADJ/NP)) {\\P x.(apply(P(5),x) & apply(P(3),x))}\n", " ---------------------------------------------------------------------------------------------------------------------------------------------------------<\n", " ADJ {\\x.(apply(\\n.divisibleby(n,5),x) & apply(\\n.divisibleby(n,3),x))}\n", "-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ((S/ADJ)/CN['Gen']) {\\F C.foreach(filter(gen_integers,\\x.(apply(\\n.divisibleby(n,5),x) & apply(\\n.divisibleby(n,3),x))),\\x.C(F(x)))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " (S/ADJ) {\\C.foreach(filter(gen_integers,\\x.(apply(\\n.divisibleby(n,5),x) & apply(\\n.divisibleby(n,3),x))),\\x.C(fizzbuzz(x)))}\n", " -------------------------------------------------------------->\n", " ADJ {\\x.(x = \"FizzBuzz\")}\n", "-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " S {foreach(filter(gen_integers,\\x.(apply(\\n.divisibleby(n,5),x) & apply(\\n.divisibleby(n,3),x))),\\x.(fizzbuzz(x) = \"FizzBuzz\"))}\n", "^^Parses for: for any number divisible by 5 and 3 fizzbuzz returns FizzBuzz (2 additional parses)\n", "\n", "Parses for: for any number less than or equal to zero fizzbuzz throws an exception\n", " for any number less than or equal to zero fizzbuzz throws an exception\n", " (((((S/,.(S\\NP['Gen']))/,.CN['Gen'])/,.ADJ)/,.CN['Gen'])/,.AQUAL) {\\q g p F C.foreach(filter(g,p),\\x.C(F(x)))} AQUAL {any} CN['Gen'] {gen_integers} (ADJ/PP['Than']) {\\x y.lessthan(y,x)} (PP['Than']/NP) {\\x.x} (((ADJ/NP)/,.(ADJ/NP))\\,.(ADJ/NP)) {\\P Q n x.(apply(P(n),x) | apply(Q(n),x))} (ADJ/PP['To']) {\\x y.equals(x,y)} (PP['To']/NP) {\\x.x} NP {0} CN['Gen'] {fizzbuzz} (((S\\NP['Gen'])/,.CN['Chk'])/,.DET) {\\d c n.checkthrows(c,\\u.n)} DET {an} CN['Chk'] {isexception}\n", "----------------------------------------------------------------------------------------------------------------------------->\n", " ((((S/,.(S\\NP['Gen']))/,.CN['Gen'])/,.ADJ)/,.CN['Gen']) {\\g p F C.foreach(filter(g,p),\\x.C(F(x)))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (((S/,.(S\\NP['Gen']))/,.CN['Gen'])/,.ADJ) {\\p F C.foreach(filter(gen_integers,p),\\x.C(F(x)))}\n", " --------------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.lessthan(y,x)}\n", " ----------------------------------------------------------------------------------------------------------------------------------------------<\n", " ((ADJ/NP)/,.(ADJ/NP)) {\\Q n x.(apply(\\y.lessthan(y,n),x) | apply(Q(n),x))}\n", " --------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.equals(x,y)}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (ADJ/NP) {\\n x.(apply(\\y.lessthan(y,n),x) | apply(\\y.equals(n,y),x))}\n", " --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(apply(\\y.lessthan(y,0),x) | apply(\\y.equals(0,y),x))}\n", "---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ((S/,.(S\\NP['Gen']))/,.CN['Gen']) {\\F C.foreach(filter(gen_integers,\\x.(apply(\\y.lessthan(y,0),x) | apply(\\y.equals(0,y),x))),\\x.C(F(x)))}\n", "-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (S/,.(S\\NP['Gen'])) {\\C.foreach(filter(gen_integers,\\x.(apply(\\y.lessthan(y,0),x) | apply(\\y.equals(0,y),x))),\\x.C(fizzbuzz(x)))}\n", " ---------------------------------------------------------------------------->\n", " ((S\\NP['Gen'])/,.CN['Chk']) {\\c n.checkthrows(c,\\u.n)}\n", " ----------------------------------------------------------------------------------------------------->\n", " (S\\NP['Gen']) {\\n.checkthrows(isexception,\\u.n)}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " S {foreach(filter(gen_integers,\\x.(apply(\\y.lessthan(y,0),x) | apply(\\y.equals(0,y),x))),\\x.checkthrows(isexception,\\u.fizzbuzz(x)))}\n", "^^Parses for: for any number less than or equal to zero fizzbuzz throws an exception (0 additional parses)\n", "Sentences without parses:\n", "==================================\n", "Summary statistics on parse multiplicity:\n", "[20] parses for: any float greater than or equal to 1 and less than 5 is not passing\n", "[4] parses for: any float greater than or equal to 5 and less than or equal to 10 is passing\n", "[6] parses for: for any float less than 1 or greater than 10 passing throws an exception\n", "[45] parses for: for any number divisible by 3 and not divisible by 5 fizzbuzz returns Fizz\n", "[45] parses for: for any number divisible by 5 and not divisible by 3 fizzbuzz returns Buzz\n", "[3] parses for: for any number divisible by 5 and 3 fizzbuzz returns FizzBuzz\n", "[1] parses for: for any number less than or equal to zero fizzbuzz throws an exception\n" ] } ], "source": [ "sttp = [\n", " # Evaluation tests, as close as possible to word-for-word from STTP (removing parentheticals),\n", " # Original: For all floats, ranging from 1 (inclusive) to 5.0 (exclusive), the program should return false.\n", " \"any float greater than or equal to 1 and less than 5 is not passing\",\n", " # Original: For all floats, ranging from 5 (inclusive) to 10 (inclusive), the program should return true.\n", " \"any float greater than or equal to 5 and less than or equal to 10 is passing\",\n", " # Original: For all invalid grades (which we define as any number below 0.9 or greater than 10.1), the program must throw an exception.\n", " \"for any float less than 1 or greater than 10 passing throws an exception\",\n", " # Original: For all numbers divisible by 3, and not divisible by 5, the program returns \"Fizz\"\n", " \"for any number divisible by 3 and not divisible by 5 fizzbuzz returns Fizz\",\n", " # Original: For all numbers divisible by 5 (and not divisible by 3), the program returns \"Buzz\"\n", " \"for any number divisible by 5 and not divisible by 3 fizzbuzz returns Buzz\",\n", " # Original: For all numbers divisible by 3 and 5, the program returns \"FizzBuzz\"\n", " \"for any number divisible by 5 and 3 fizzbuzz returns FizzBuzz\",\n", " # Original: The program throws an exception for all numbers that are zero or smaller\n", " \"for any number less than or equal to zero fizzbuzz throws an exception\"\n", "]\n", "trySentences(sttp)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Checking how many semantically unique logical forms the many parse trees yield.\n", "The equivalence check uses a theorem prover (via NLTK), so these are truly logical equivalence classes.\n", "\n", "This same loop also picks out a string logical form for each sentence." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1 semantically unique logical forms for [any float greater than or equal to 1 and less than 5 is not passing]: {: 1}\n", "1 semantically unique logical forms for [any float greater than or equal to 5 and less than or equal to 10 is passing]: {: 1}\n", "1 semantically unique logical forms for [for any float less than 1 or greater than 10 passing throws an exception]: {: 1}\n", "1 semantically unique logical forms for [for any number divisible by 3 and not divisible by 5 fizzbuzz returns Fizz]: {: 1}\n", "1 semantically unique logical forms for [for any number divisible by 5 and not divisible by 3 fizzbuzz returns Buzz]: {: 1}\n", "1 semantically unique logical forms for [for any number divisible by 5 and 3 fizzbuzz returns FizzBuzz]: {: 1}\n", "1 semantically unique logical forms for [for any number less than or equal to zero fizzbuzz throws an exception]: {: 1}\n", "\n", "Logical forms in order:\n", "foreach(filter(gen_floats,\\x.((apply(\\y.lessthan(1,y),x) | apply(\\y.equals(1,y),x)) & lessthan(x,5))),\\n.-passing(n))\n", "foreach(filter(gen_floats,\\x.((apply(\\y.lessthan(5,y),x) | apply(\\y.equals(5,y),x)) & (apply(\\y.lessthan(y,10),x) | apply(\\y.equals(10,y),x)))),\\n.passing(n))\n", "foreach(filter(gen_floats,\\x.(lessthan(x,1) | lessthan(10,x))),\\x.checkthrows(isexception,\\u.passing(x)))\n", "foreach(filter(gen_integers,\\x.(divisibleby(x,3) & -divisibleby(x,5))),\\x.(fizzbuzz(x) = \"Fizz\"))\n", "foreach(filter(gen_integers,\\x.(divisibleby(x,5) & -divisibleby(x,3))),\\x.(fizzbuzz(x) = \"Buzz\"))\n", "foreach(filter(gen_integers,\\x.(apply(\\n.divisibleby(n,5),x) & apply(\\n.divisibleby(n,3),x))),\\x.(fizzbuzz(x) = \"FizzBuzz\"))\n", "foreach(filter(gen_integers,\\x.(apply(\\y.lessthan(y,0),x) | apply(\\y.equals(0,y),x))),\\x.checkthrows(isexception,\\u.fizzbuzz(x)))\n" ] } ], "source": [ "sttp_logical_forms = []\n", "for spec in sttp:\n", " (parses,syntactic,semantic) = processSentence(spec)\n", " print(str(len(semantic))+\" semantically unique logical forms for [\"+spec+\"]: \"+str(semantic))\n", " sttp_logical_forms.append(str(list(semantic.keys())[0]))\n", "print()\n", "print(\"Logical forms in order:\")\n", "for lf in sttp_logical_forms:\n", " print(lf)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And then we'll hammer those logical forms into valid Javascript code." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "const fc = require('fast-check');\n", "\n", "function filter(g,p) { return g.filter(p); }\n", "function lessthan(x,y) { return x < y; }\n", "function equals(x,y) { return x == y; }\n", "function apply(f,a) { return f(a); }\n", "function checkthrows(p,f) {\n", " try {\n", " f();\n", " } catch (error) {\n", " if (p(error))\n", " return true;\n", " }\n", " return false;\n", "}\n", "function isexception(x) { return x instanceof Error; }\n", "\n", "\n", "function passing(x) {\n", " if (x < 1 || x > 10) {\n", " throw new Error(\"out of range\");\n", " }\n", " return x >= 5;\n", "}\n", "function fizzbuzz(x) {\n", " if (x < 0) {\n", " throw new Error(\"out of range\");\n", " }\n", " var div3 = x % 3 == 0;\n", " var div5 = x % 5 == 0;\n", " if (div3 && div5) {\n", " return \"FizzBuzz\";\n", " } else if (div3) {\n", " return \"Fizz\";\n", " } else if (div5) {\n", " return \"Buzz\";\n", " } else {\n", " return \"Nope\";\n", " }\n", "}\n", "\n", "fc.assert(fc.property(filter(fc.float({next:true}),x => ((apply(y => lessthan(1,y),x) || apply(y => equals(1,y),x)) && lessthan(x,5))),n => !passing(n)))\n", "fc.assert(fc.property(filter(fc.float({next:true}),x => ((apply(y => lessthan(5,y),x) || apply(y => equals(5,y),x)) && (apply(y => lessthan(y,10),x) || apply(y => equals(10,y),x)))),n => passing(n)))\n", "fc.assert(fc.property(filter(fc.float({next:true}),x => (lessthan(x,1) || lessthan(10,x))),x => checkthrows(isexception,u => passing(x))))\n", "fc.assert(fc.property(filter(fc.integer(),x => (divisibleby(x,3) && !divisibleby(x,5))),x => (fizzbuzz(x) == \"Fizz\")))\n", "fc.assert(fc.property(filter(fc.integer(),x => (divisibleby(x,5) && !divisibleby(x,3))),x => (fizzbuzz(x) == \"Buzz\")))\n", "fc.assert(fc.property(filter(fc.integer(),x => (apply(n => divisibleby(n,5),x) && apply(n => divisibleby(n,3),x))),x => (fizzbuzz(x) == \"FizzBuzz\")))\n", "fc.assert(fc.property(filter(fc.integer(),x => (apply(y => lessthan(y,0),x) || apply(y => equals(0,y),x))),x => checkthrows(isexception,u => fizzbuzz(x))))\n" ] } ], "source": [ "def to_js(s):\n", " s = s.replace(\"foreach\",\"fc.property\")\n", " s = s.replace(\"gen_floats\",\"fc.float({next:true})\") # bizarrely, they default to only [0,1) unless passing this flag\n", " s = s.replace(\"gen_integers\",\"fc.integer()\")\n", " s = s.replace(\"=\",\"==\") # Must come before introducing lambdas with =>\n", " s = s.replace(\"\\\\x.\",\"x => \")\n", " s = s.replace(\"\\\\y.\",\"y => \")\n", " s = s.replace(\"\\\\u.\",\"u => \")\n", " s = s.replace(\"\\\\z6.\",\"z6 => \")\n", " s = s.replace(\"\\\\n.\",\"n => \")\n", " s = s.replace(\"\\\\x n.\",\"x => n => \")\n", " s = s.replace(\"|\",\"||\")\n", " s = s.replace(\"&\",\"&&\")\n", " s = s.replace(\"-\",\"!\")\n", " s = s.replace(\"\\\\a b P x.\",\"a => b => P => x => \")\n", " return s\n", "js_prefix = '''\n", "const fc = require('fast-check');\n", "\n", "function filter(g,p) { return g.filter(p); }\n", "function lessthan(x,y) { return x < y; }\n", "function equals(x,y) { return x == y; }\n", "function apply(f,a) { return f(a); }\n", "function checkthrows(p,f) {\n", " try {\n", " f();\n", " } catch (error) {\n", " if (p(error))\n", " return true;\n", " }\n", " return false;\n", "}\n", "function isexception(x) { return x instanceof Error; }\n", "'''\n", "sttp_impls = '''\n", "function passing(x) {\n", " if (x < 1 || x > 10) {\n", " throw new Error(\"out of range\");\n", " }\n", " return x >= 5;\n", "}\n", "function fizzbuzz(x) {\n", " if (x < 0) {\n", " throw new Error(\"out of range\");\n", " }\n", " var div3 = x % 3 == 0;\n", " var div5 = x % 5 == 0;\n", " if (div3 && div5) {\n", " return \"FizzBuzz\";\n", " } else if (div3) {\n", " return \"Fizz\";\n", " } else if (div5) {\n", " return \"Buzz\";\n", " } else {\n", " return \"Nope\";\n", " }\n", "}\n", "'''\n", "print(js_prefix)\n", "print(sttp_impls)\n", "for lf in sttp_logical_forms:\n", " print(\"fc.assert(\"+to_js(lf)+\")\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stats and simple demo of generalization" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "29\n" ] } ], "source": [ "sttp_words = {}\n", "for s in sttp:\n", " for w in s.split():\n", " sttp_words[w] = 1\n", "print(len(sttp_words)) # includes the numbers and the stand-ins for string constants" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Parses for: any float that is passing is greater than or equal to 5 and less than or equal to 10\n", " any float that is passing is greater than or equal to 5 and less than or equal to 10\n", " ((S/(S\\NP))/CN['Gen']) {\\n p.foreach(n,p)} CN['Gen'] {gen_floats} ((CN['Gen']/(S\\NP['Gen']))\\CN['Gen']) {\\P n.filter(P,n)} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} ADJ {\\x.passing(x)} ((S\\NP['Gen'])/ADJ) {\\A n.A(n)} (ADJ/PP['Than']) {\\x y.lessthan(x,y)} (PP['Than']/NP) {\\x.x} (((ADJ/NP)/,.(ADJ/NP))\\,.(ADJ/NP)) {\\P Q n x.(apply(P(n),x) | apply(Q(n),x))} (ADJ/PP['To']) {\\x y.equals(x,y)} (PP['To']/NP) {\\x.x} NP {5} ((ADJ/,.ADJ)\\,.ADJ) {\\P Q x.(P(x) & Q(x))} (ADJ/PP['Than']) {\\x y.lessthan(y,x)} (PP['Than']/NP) {\\x.x} (((ADJ/NP)/,.(ADJ/NP))\\,.(ADJ/NP)) {\\P Q n x.(apply(P(n),x) | apply(Q(n),x))} (ADJ/PP['To']) {\\x y.equals(x,y)} (PP['To']/NP) {\\x.x} NP {10}\n", " ----------------------------------------------------------------------------------<\n", " (CN['Gen']/(S\\NP['Gen'])) {\\n.filter(gen_floats,n)}\n", " ------------------------------------------------------>\n", " (S\\NP['Gen']) {\\n.passing(n)}\n", " ---------------------------------------------------------------------------------------------------------------------------------------->\n", " CN['Gen'] {filter(gen_floats,\\n.passing(n))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " (S/(S\\NP)) {\\p.foreach(filter(gen_floats,\\n.passing(n)),p)}\n", " --------------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.lessthan(x,y)}\n", " ----------------------------------------------------------------------------------------------------------------------------------------------<\n", " ((ADJ/NP)/,.(ADJ/NP)) {\\Q n x.(apply(\\y.lessthan(n,y),x) | apply(Q(n),x))}\n", " --------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.equals(x,y)}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (ADJ/NP) {\\n x.(apply(\\y.lessthan(n,y),x) | apply(\\y.equals(n,y),x))}\n", " --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(apply(\\y.lessthan(5,y),x) | apply(\\y.equals(5,y),x))}\n", " -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<\n", " (ADJ/,.ADJ) {\\Q x.((apply(\\y.lessthan(5,y),x) | apply(\\y.equals(5,y),x)) & Q(x))}\n", " --------------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.lessthan(y,x)}\n", " ----------------------------------------------------------------------------------------------------------------------------------------------<\n", " ((ADJ/NP)/,.(ADJ/NP)) {\\Q n x.(apply(\\y.lessthan(y,n),x) | apply(Q(n),x))}\n", " --------------------------------------------------------->B\n", " (ADJ/NP) {\\x y.equals(x,y)}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (ADJ/NP) {\\n x.(apply(\\y.lessthan(y,n),x) | apply(\\y.equals(n,y),x))}\n", " ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(apply(\\y.lessthan(y,10),x) | apply(\\y.equals(10,y),x))}\n", " --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.((apply(\\y.lessthan(5,y),x) | apply(\\y.equals(5,y),x)) & (apply(\\y.lessthan(y,10),x) | apply(\\y.equals(10,y),x)))}\n", " ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " (S\\NP['Gen']) {\\n.((apply(\\y.lessthan(5,y),n) | apply(\\y.equals(5,y),n)) & (apply(\\y.lessthan(y,10),n) | apply(\\y.equals(10,y),n)))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " S {foreach(filter(gen_floats,\\n.passing(n)),\\n.((apply(\\y.lessthan(5,y),n) | apply(\\y.equals(5,y),n)) & (apply(\\y.lessthan(y,10),n) | apply(\\y.equals(10,y),n))))}\n", "^^Parses for: any float that is passing is greater than or equal to 5 and less than or equal to 10 (19 additional parses)\n", "Sentences without parses:\n", "==================================\n", "Summary statistics on parse multiplicity:\n", "[20] parses for: any float that is passing is greater than or equal to 5 and less than or equal to 10\n" ] } ], "source": [ "# stregthening example\n", "trySentences([\"any float that is passing is greater than or equal to 5 and less than or equal to 10\"])" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{: 1}" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Checking semantic ambiguity for sentence above\n", "# The final/third element of the (0-indexed) tuple is the set of semantically distinguishable logical forms\n", "processSentence(\"any float that is passing is greater than or equal to 5 and less than or equal to 10\")[2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Correcting Nuance in STTP Test Descriptions" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Parses for: for any positive number divisible by 3 and not divisible by 5 fizzbuzz returns Fizz\n", " for any positive number divisible by 3 and not divisible by 5 fizzbuzz returns Fizz\n", " (((((S/ADJ)/CN['Gen'])/ADJ)/CN['Gen'])/AQUAL) {\\q g p F C.foreach(filter(g,p),\\x.C(F(x)))} AQUAL {any} (CN['Gen']/CN['Gen']) {\\x.filter(x,\\u.lessthan(0,u))} CN['Gen'] {gen_integers} (ADJ/PP['By']) {\\d n.divisibleby(n,d)} (PP['By']/NP) {\\x.x} NP {3} ((ADJ/,.ADJ)\\,.ADJ) {\\P Q x.(P(x) & Q(x))} (ADJ/ADJ) {\\P x.-P(x)} (ADJ/PP['By']) {\\d n.divisibleby(n,d)} (PP['By']/NP) {\\x.x} NP {5} CN['Gen'] {fizzbuzz} (ADJ/NP['Chk']) {\\P x.P(x)} NP['Chk'] {\\x.(x = \"Fizz\")}\n", "--------------------------------------------------------------------------------------------------------->\n", " ((((S/ADJ)/CN['Gen'])/ADJ)/CN['Gen']) {\\g p F C.foreach(filter(g,p),\\x.C(F(x)))}\n", " --------------------------------------------------------------------------------->\n", " CN['Gen'] {filter(gen_integers,\\u.lessthan(0,u))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " (((S/ADJ)/CN['Gen'])/ADJ) {\\p F C.foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),p),\\x.C(F(x)))}\n", " ------------------------------>\n", " PP['By'] {3}\n", " ---------------------------------------------------------------------->\n", " ADJ {\\n.divisibleby(n,3)}\n", " ------------------------------------------------------------------------------------------------------------------<\n", " (ADJ/,.ADJ) {\\Q x.(divisibleby(x,3) & Q(x))}\n", " ------------------------------>\n", " PP['By'] {5}\n", " ---------------------------------------------------------------------->\n", " ADJ {\\n.divisibleby(n,5)}\n", " ---------------------------------------------------------------------------------------------->\n", " ADJ {\\x.-divisibleby(x,5)}\n", " ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(divisibleby(x,3) & -divisibleby(x,5))}\n", "---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ((S/ADJ)/CN['Gen']) {\\F C.foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(divisibleby(x,3) & -divisibleby(x,5))),\\x.C(F(x)))}\n", "-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (S/ADJ) {\\C.foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(divisibleby(x,3) & -divisibleby(x,5))),\\x.C(fizzbuzz(x)))}\n", " ---------------------------------------------------------->\n", " ADJ {\\x.(x = \"Fizz\")}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " S {foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(divisibleby(x,3) & -divisibleby(x,5))),\\x.(fizzbuzz(x) = \"Fizz\"))}\n", "^^Parses for: for any positive number divisible by 3 and not divisible by 5 fizzbuzz returns Fizz (74 additional parses)\n", "\n", "Parses for: for any positive number divisible by 5 and not divisible by 3 fizzbuzz returns Buzz\n", " for any positive number divisible by 5 and not divisible by 3 fizzbuzz returns Buzz\n", " (((((S/ADJ)/CN['Gen'])/ADJ)/CN['Gen'])/AQUAL) {\\q g p F C.foreach(filter(g,p),\\x.C(F(x)))} AQUAL {any} (CN['Gen']/CN['Gen']) {\\x.filter(x,\\u.lessthan(0,u))} CN['Gen'] {gen_integers} (ADJ/PP['By']) {\\d n.divisibleby(n,d)} (PP['By']/NP) {\\x.x} NP {5} ((ADJ/,.ADJ)\\,.ADJ) {\\P Q x.(P(x) & Q(x))} (ADJ/ADJ) {\\P x.-P(x)} (ADJ/PP['By']) {\\d n.divisibleby(n,d)} (PP['By']/NP) {\\x.x} NP {3} CN['Gen'] {fizzbuzz} (ADJ/NP['Chk']) {\\P x.P(x)} NP['Chk'] {\\x.(x = \"Buzz\")}\n", "--------------------------------------------------------------------------------------------------------->\n", " ((((S/ADJ)/CN['Gen'])/ADJ)/CN['Gen']) {\\g p F C.foreach(filter(g,p),\\x.C(F(x)))}\n", " --------------------------------------------------------------------------------->\n", " CN['Gen'] {filter(gen_integers,\\u.lessthan(0,u))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " (((S/ADJ)/CN['Gen'])/ADJ) {\\p F C.foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),p),\\x.C(F(x)))}\n", " ------------------------------>\n", " PP['By'] {5}\n", " ---------------------------------------------------------------------->\n", " ADJ {\\n.divisibleby(n,5)}\n", " ------------------------------------------------------------------------------------------------------------------<\n", " (ADJ/,.ADJ) {\\Q x.(divisibleby(x,5) & Q(x))}\n", " ------------------------------>\n", " PP['By'] {3}\n", " ---------------------------------------------------------------------->\n", " ADJ {\\n.divisibleby(n,3)}\n", " ---------------------------------------------------------------------------------------------->\n", " ADJ {\\x.-divisibleby(x,3)}\n", " ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ADJ {\\x.(divisibleby(x,5) & -divisibleby(x,3))}\n", "---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ((S/ADJ)/CN['Gen']) {\\F C.foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(divisibleby(x,5) & -divisibleby(x,3))),\\x.C(F(x)))}\n", "-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (S/ADJ) {\\C.foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(divisibleby(x,5) & -divisibleby(x,3))),\\x.C(fizzbuzz(x)))}\n", " ---------------------------------------------------------->\n", " ADJ {\\x.(x = \"Buzz\")}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " S {foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(divisibleby(x,5) & -divisibleby(x,3))),\\x.(fizzbuzz(x) = \"Buzz\"))}\n", "^^Parses for: for any positive number divisible by 5 and not divisible by 3 fizzbuzz returns Buzz (74 additional parses)\n", "\n", "Parses for: for any positive number divisible by 5 and 3 fizzbuzz returns FizzBuzz\n", " for any positive number divisible by 5 and 3 fizzbuzz returns FizzBuzz\n", " (((((S/ADJ)/CN['Gen'])/ADJ)/CN['Gen'])/AQUAL) {\\q g p F C.foreach(filter(g,p),\\x.C(F(x)))} AQUAL {any} (CN['Gen']/CN['Gen']) {\\x.filter(x,\\u.lessthan(0,u))} CN['Gen'] {gen_integers} (ADJ/PP['By']) {\\d n.divisibleby(n,d)} (PP['By']/NP) {\\x.x} NP {5} (((ADJ\\,.(ADJ/NP))/,.NP)\\,.NP) {\\a b P x.(apply(P(a),x) & apply(P(b),x))} NP {3} CN['Gen'] {fizzbuzz} (ADJ/NP['Chk']) {\\P x.P(x)} NP['Chk'] {\\x.(x = \"FizzBuzz\")}\n", "--------------------------------------------------------------------------------------------------------->\n", " ((((S/ADJ)/CN['Gen'])/ADJ)/CN['Gen']) {\\g p F C.foreach(filter(g,p),\\x.C(F(x)))}\n", " --------------------------------------------------------------------------------->\n", " CN['Gen'] {filter(gen_integers,\\u.lessthan(0,u))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>\n", " (((S/ADJ)/CN['Gen'])/ADJ) {\\p F C.foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),p),\\x.C(F(x)))}\n", " -------------------------------------------------------------->B\n", " (ADJ/NP) {\\x n.divisibleby(n,x)}\n", " -----------------------------------------------------------------------------------<\n", " ((ADJ\\,.(ADJ/NP))/,.NP) {\\b P x.(apply(P(5),x) & apply(P(b),x))}\n", " ------------------------------------------------------------------------------------------->\n", " (ADJ\\,.(ADJ/NP)) {\\P x.(apply(P(5),x) & apply(P(3),x))}\n", " ---------------------------------------------------------------------------------------------------------------------------------------------------------<\n", " ADJ {\\x.(apply(\\n.divisibleby(n,5),x) & apply(\\n.divisibleby(n,3),x))}\n", "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " ((S/ADJ)/CN['Gen']) {\\F C.foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(apply(\\n.divisibleby(n,5),x) & apply(\\n.divisibleby(n,3),x))),\\x.C(F(x)))}\n", "------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " (S/ADJ) {\\C.foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(apply(\\n.divisibleby(n,5),x) & apply(\\n.divisibleby(n,3),x))),\\x.C(fizzbuzz(x)))}\n", " -------------------------------------------------------------->\n", " ADJ {\\x.(x = \"FizzBuzz\")}\n", "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->\n", " S {foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(apply(\\n.divisibleby(n,5),x) & apply(\\n.divisibleby(n,3),x))),\\x.(fizzbuzz(x) = \"FizzBuzz\"))}\n", "^^Parses for: for any positive number divisible by 5 and 3 fizzbuzz returns FizzBuzz (4 additional parses)\n", "Sentences without parses:\n", "==================================\n", "Summary statistics on parse multiplicity:\n", "[75] parses for: for any positive number divisible by 3 and not divisible by 5 fizzbuzz returns Fizz\n", "[75] parses for: for any positive number divisible by 5 and not divisible by 3 fizzbuzz returns Buzz\n", "[5] parses for: for any positive number divisible by 5 and 3 fizzbuzz returns FizzBuzz\n" ] } ], "source": [ "# These are corrections/clarifications to properties 4-6 which generate passing tests with the correct restriction to positive integers\n", "sttp_corrections = [\n", " \"for any positive number divisible by 3 and not divisible by 5 fizzbuzz returns Fizz\",\n", " \"for any positive number divisible by 5 and not divisible by 3 fizzbuzz returns Buzz\",\n", " \"for any positive number divisible by 5 and 3 fizzbuzz returns FizzBuzz\"\n", "]\n", "trySentences(sttp_corrections)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Error processing: for any positive number divisible by 3 and not divisible by 5 fizzbuzz returns Fizz\n", "(FATAL)\n", "%%ERROR: A term cannot be constructed from the marked string:\n", "\n", "\n", " (foreach(filter(filter(%%START ERROR%%gen_integers,\\u.%%END ERROR%%\n", "\n", "Fatal error: sread_term error\n", "Error processing: for any positive number divisible by 5 and not divisible by 3 fizzbuzz returns Buzz\n", "(FATAL)\n", "%%ERROR: A term cannot be constructed from the marked string:\n", "\n", "\n", " (foreach(filter(filter(%%START ERROR%%gen_integers,\\u.%%END ERROR%%\n", "\n", "Fatal error: sread_term error\n", "Error processing: for any positive number divisible by 5 and 3 fizzbuzz returns FizzBuzz\n", "(FATAL)\n", "%%ERROR: A term cannot be constructed from the marked string:\n", "\n", "\n", " (foreach(filter(filter(%%START ERROR%%gen_integers,\\u.%%END ERROR%%\n", "\n", "Fatal error: sread_term error\n" ] } ], "source": [ "# This is here for the record, though something about the semantics of \"positive\" trips over a bug in Prover9\n", "for spec in sttp_corrections:\n", " try:\n", " (parses,syntactic,semantic) = processSentence(spec)\n", " print(str(len(semantic))+\" semantically unique logical forms for [\"+spec+\"]: \"+str(semantic))\n", " except BaseException as error:\n", " print(\"Error processing: \"+spec)\n", " print(error)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2 syntactically unique logical forms for [for any positive number divisible by 3 and not divisible by 5 fizzbuzz returns Fizz]: \n", " > foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(divisibleby(x,3) & -divisibleby(x,5))),\\x.(fizzbuzz(x) = \"Fizz\"))\n", " > foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(divisibleby(x,3) & -divisibleby(x,5))),\\z24.(fizzbuzz(z24) = \"Fizz\"))\n", "2 syntactically unique logical forms for [for any positive number divisible by 5 and not divisible by 3 fizzbuzz returns Buzz]: \n", " > foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(divisibleby(x,5) & -divisibleby(x,3))),\\x.(fizzbuzz(x) = \"Buzz\"))\n", " > foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(divisibleby(x,5) & -divisibleby(x,3))),\\z27.(fizzbuzz(z27) = \"Buzz\"))\n", "2 syntactically unique logical forms for [for any positive number divisible by 5 and 3 fizzbuzz returns FizzBuzz]: \n", " > foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(apply(\\n.divisibleby(n,5),x) & apply(\\n.divisibleby(n,3),x))),\\x.(fizzbuzz(x) = \"FizzBuzz\"))\n", " > foreach(filter(filter(gen_integers,\\u.lessthan(0,u)),\\x.(apply(\\n.divisibleby(n,5),x) & apply(\\n.divisibleby(n,3),x))),\\z30.(fizzbuzz(z30) = \"FizzBuzz\"))\n" ] } ], "source": [ "# As backup, we manually inspect the syntactically-distinct semantics, which tend to be equivalent modulo alpha-renaming of bound variables, so are easy to check\n", "for spec in sttp_corrections:\n", " (parses,syntactic,semantic) = processSentence(spec, checkSemantics=False)\n", " print(str(len(syntactic))+\" syntactically unique logical forms for [\"+spec+\"]: \")\n", " for stx in syntactic:\n", " print(\" > \"+str(stx))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Additional Corpus Searches\n", "\n", "We were able to locate 19 additional pairs of English descriptions of PBTs with matching PBTs.\n", "\n", "Notice that these examples are drawn from introductory tests, and there is overlap (Testing with F# includes fizzbuzz testing as does STTP, both Testing with F# and Real World Haskell test a sort function that operates on lists.\n", "\n", "Also consulted were two textbooks which covered property-based testing, but managed to do so without explicitly stating the purpose of any examples:\n", "\n", "- Programming Scala (O'Reilly)\n", "- Functional Programming in Scala (Manning)\n", "\n", "### Testing with F# (Packt)\n", "1. Reversing a list twice will result in the original list\n", "2. The first item should always be the smallest in the result\n", "3. The last item should always be the largest in the result\n", "4. The result should always be a permutation of the input\n", "5. The result should always be ordered\n", "6. The sorted list is a permutation of the original list\n", "7. The sorted list is actually ordered\n", "8. Sorting twice is the same as sorting once \n", "9. It returns the fizz string value for every number divisible by 3, the buzz string value for every number divisible by 5, and the fizzbuzz string value for every number divisible by both 3 and 5\n", " - Note this repeats problems with conflicting overlapping specifications of FizzBuzz as the STTP examples\n", "\n", "While the other texts here are widely available, these examples came from an online supplement to the book and are not present in e.g. electronic versions provided by library subscriptions. The publisher has removed the link to this supplement from the product page, but they still host the file publicly: https://static.packt-cdn.com/downloads/1232OS_Chapter_11.pdf\n", "\n", "### Testing in Scala (O'Reilly)\n", "1. A sum is greater than its parts\n", "2. Sums are associative\n", "3. An album can be created using a year from 1900 to 3000\n", "\n", "### Real World Haskell (O'Reilly)\n", "1. The first element in a sorted list should always be the smallest element of the input list\n", "2. If the list is nonempty, then the first element of the sorted list is the minimum\n", "3. The output should be ordered\n", "4. The output should be a permutation of the input\n", "5. The last sorted element should be the largest element\n", "6. If we find the smallest element of two different lists, that should be the first element if we append and sort those lists\n", "7. Appending or prepending the empty list onto a second list should leave the second list unchanged" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" } }, "nbformat": 4, "nbformat_minor": 4 }