Gradescope Python Utilities

Installing

Make sure you have pip installed (eg. on Debian/Ubuntu, apt-get install python-pip).

Then, run pip install gradescope-utils

Packages

• Autograder Utilities

Changelog

See the Releases page.

Releasing new versions

Follow https://packaging.python.org/tutorials/packaging-projects/, but in brief:

1. Bump the version in setup.py
2. Draft a release on https://github.com/gradescope/gradescope-utils/releases
3. This can take care of tagging for you. Otherwise, tag the commit: git tag vX.Y.Z COMMIT_SHA
4. Make sure that the setup.py version matches the release/tag version
5. GitHub releases can help auto-generate release notes from merged PRs. Edit these as necessary.
6. Publish the release on GitHub. GitHub Actions will build and publish a new release when a version is tagged (e.g. when a GitHub release is published).

Support

Contact us at [email protected] if you need help with these packages.

Issues

Figured out how to pass parameters from settings

opened on 2023-03-29 14:47:40 by AndrewQuijano

Hello, I figured out how to pass parameters to the JSONTestRunner from settings.py file in Django. I was able to test it locally with no issues.

I am wondering, when would anyone else be able to download these updates? Furthermore, I figure this could be useful for anyone teaching courses on either developing Django applications or in my case, use a buggy Django implementation for a CTF.

Human-readable name of test case does not appear in results.json

opened on 2021-05-06 19:29:54 by bryceadam1

Description Naming test cases with a new-line character on the first line causes the name of the function to be placed in the results.json file instead of the human-readable name in the docstring of the testcase.

To Reproduce Create a testcase and name it with a docstring that has a new-line character on the first line. This file is called tests.py.

```python import unittest from gradescope_utils.autograder_utils.decorators import weight, number

class TestAutograder(unittest.TestCase): @weight(1.0) @number("1") def test_prob1(self): """ A human-readable name. """ pass ```

Running the above with the JSONTestRunner provided in gradescope_utils produces the following results.json:

json { "tests": [ { "name": "test_prob1 (tests.TestAutograder)", "score": 1.0, "max_score": 1.0, "number": "1" }, ], "leaderboard": [], "execution_time": "0.00", "score": 10.0 }

Expected behavior Should have produced the following results.json file: json { "tests": [ { "name": "A human-readable name.", "score": 1.0, "max_score": 1.0, "number": "1" }, ], "leaderboard": [], "execution_time": "0.00", "score": 10.0 }

Additional context Removing the new-line character seems to fix the issue. I.e. the following produces the expected result. ```python import unittest from gradescope_utils.autograder_utils.decorators import weight, number

class TestAutograder(unittest.TestCase): @weight(1.0) @number("1") def test_prob1(self): """A human-readable name.""" pass ``` This also seems to only be the case inside of Gradescope's docker containers used for actually grading student submissions. This did not happen when I ran the code on my macbook.

Better Error Messages (easy version)

opened on 2020-12-04 20:39:56 by habermanUIUC

When a student submits code that breaks a test but it's not due to an assertion error, the feedback is either confusing or misleading.

For example, if a student has a simple KeyError mistake def my_code(dataframe): return dataframe['SalesPerson'] # assume SalesPerson is not a valid key

The test will fail but with the message: "Test Failed: 'SalesPerson' That really offers no help

Describe the solution you'd like An easy solution is to provide the class of the exception or error:

"Test Failed: <class 'KeyError'> 'SalesPerson'\n"

The fix is simply in json_test_runner.py (buildResult) line 77

``` if err: if err[0] is AssertionError: output += "Test Failed: {0}\n".format(err[1]) else: output += "Test Failed: {0} {1}\n".format(err[0], err[1])

note that isinstance will NOT work here

```

Describe alternatives you've considered A similar pull request

18

However with that solution you get the following:

"Test Failed: 'SalesPerson'\nhashtable_class_helper.pxi:pandas._libs.hashtable.PyObjectHashTable.get_item line 1627: \n"

In this context, the backtrace is not helpful to the student. Perhaps a config option?

Also, in that pull request it should not use isinstance() to test the error class. (tested in python3.6).

Additional context

Provide stack backtrace for errors in student code

opened on 2020-09-05 00:20:07 by marie-r

When student code generates exceptions, e.g. IndexError, tests fail without any indication of the root cause, making it difficult for students to debug their code. This change to the JSON test runner provides context by outputting the last stack frame (easy to change for last N frames).

Help prevent reverse engineering of tests

opened on 2020-02-18 21:24:52 by habermanUIUC

One of the issues with Python testing, is that it's easy for students to read directories and figure out what files to potentially read. Although one could argue that a student who has figured this out probably should deserve to pass the tests. However, this kind of information then spreads.

There's many ways we mitigate this but all of them involve a lot of jumping through hoops (dynamically renaming the tests, removing the test files after they are loaded but before the students code is loaded), etc (which only work if in a sandboxed environment).

A simple and effective way to prevent this would be to allow the initialization of JSONTestRunner to take a parameter(s) to limit how much stdout and stderr is written out in the results. That's the most likely way for students to get information back. If both stdout and stderr are truncated to a specified maximum amount, then it allows both the framework to print relevant messages, but also keeps students from reading the test files.

Another option would be allow tests cases to write to a logfile that is then sent back to stdout first and anything else truncated.

Support Python 3.4's subTest feature

opened on 2018-02-15 20:03:03 by ibrahima

https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests

Makes it easier to script parametrized tests.

Releases

v0.5.0 2022-10-17 21:57:53

What's Changed

• Add MIT License by @ibrahima in https://github.com/gradescope/gradescope-utils/pull/26
• Convert the value passed to @number to a string by @trombonehero in https://github.com/gradescope/gradescope-utils/pull/20
• Hide error messages by @peterstory in https://github.com/gradescope/gradescope-utils/pull/25
• Use "status" prop and other misc improvements by @isaacl in https://github.com/gradescope/gradescope-utils/pull/27
• Make test failure prefix msg configurable by @isaacl in https://github.com/gradescope/gradescope-utils/pull/29

New Contributors

• @trombonehero made their first contribution in https://github.com/gradescope/gradescope-utils/pull/20
• @peterstory made their first contribution in https://github.com/gradescope/gradescope-utils/pull/25
• @isaacl made their first contribution in https://github.com/gradescope/gradescope-utils/pull/27

Full Changelog: https://github.com/gradescope/gradescope-utils/compare/v0.4.0...v0.5.0

Add post_processor callback to JSONTestRunner 2020-06-11 17:58:15

Adds a callback function to JSONTestRunner which can be used to modify the autograder results at the end of the test suite but before they are written to disk.

Usage Example:

```python def post_process(results): results["score"] = 9001

if name == 'main': suite = unittest.defaultTestLoader.discover('tests')

JSONTestRunner(visibility='visible', post_processor=post_process).run(suite)

```

https://github.com/gradescope/gradescope-utils/pull/16

Add `@number` decorator 2020-01-14 00:41:20

The @number decorator allows setting the "number" field on tests, which is used to sort the tests when displayed on Gradescope. See the format specification for more information.

Allow tests to give partial credit 2018-09-26 23:59:25

Adds a new decorator, @partial_credit, which allows setting the final score directly for cases where you want to give partial credit.

When using it, you don't need to use the @weight decorator (you pass in the test weight to @partial_credit).

Allow check_submitted_files to take in base directory 2018-09-25 22:28:39

https://github.com/gradescope/gradescope-utils/pull/8

Visibility options for JSONTestRunner 2018-08-04 00:19:17

JSONTestRunner now accepts options for setting the top-level visibility and stdout_visibility fields (see the docs for more details).