Add foundation for automatic refactoring tool

[MWR27]

This tool takes advantage of the warnings thrown in the Pygrate runtime. Built off of lib2to3, it refactors the Python 2 source code according to given warnings. The tool is a Python module called pygrate2 and is run as such:

pygrate2 [options] source dest

The tool runs source and writes the refactored files into the directory dest. source can either be a single .py file or a directory. If source is a directory, then the tool assumes source is a module.

After the execution(s) of source is complete, the warnings are collected and passed to the interactive tool. For each warning, the tool will show the user the refactoring and prompt the user to apply or skip it. After processing every warning, it will apply the desired fixes. Here is an example:

/testpkg/div.py:2: classic int division
Could safely refactor:
-  5 / divisor
+  classic_div(5, divisor)
Apply refactoring? [y/n]
> y
/testpkg/div.py:3: classic int division
Could safely refactor:
- 8 / 3
+ classic_div(8, 3)
Apply refactoring? [y/n]
> n
Skipped warnings:
/testpkg/div.py:3: DeprecationWarning: classic int division
--- /testpkg/div.py     (original)
+++ /testpkg/div.py     (refactored)
@@ -1,5 +1,8 @@
 divisor = 2
-print 5 / divisor
+print classic_div(5, divisor)
 8 / 3

+
+def classic_div(dividend, divisor):
+    return dividend // divisor if isinstance(dividend, int) and isinstance(divisor, int) else dividend / divisor

Technical Details

Expanding on lib2to3, the tool takes a set of fixers and applies them to the source code. As shown above, fixers are only applied to the actual warnings thrown. The tool is passed a dictionary that maps each warning to a fixer.

All dynamic fixers are located in pygrate2/fixes/ and extend SemanticFix, which is located in pygrate2/semfixer_base.py. The tool currently has one fixer called FixSemDiv (in fix_sem_div.py) to handle classic division. This fixer simply replaces any instance of integer division (provided by the warnings) with a helper function called classic_div that preserves the behavior of Python 2 division. Helper functions are defined in pygrate2/pygrate_helpers.py. Any fixer inheriting SemanticFix contains a class constant called HELPER_FUNCTION to store name of the helper function associated with the fixer, if there is one. When a fixer's transform function is called, it must set its Boolean value self.wrote_helper to indicate whether the helper function was included in the transformation. This is necessary for the tool to determine what helper functions should be written to the output. If source is a single file, the helper functions will be appended to the end of the file. If source is a directory, pygrate_helpers.py is created at the top-level directory, containing the necessary functions, and import statements are inserted at the top of files that use helpers.

To map a warning to a fixer, simply add an entry to warning_fixers in pygrate2/main.py, where the key is the warning message and the value is the name of the fixer file without the extension. Currently, there is only one entry:

warning_fixers = {'classic int division': 'fix_sem_div'}

Note that all fixer files must reside in pygrate2/fixes/, and the file name must follow the lib2to3 convention, in snake case matching with the fixer class's Pascal case (e.g. FixSemDiv and fix_sem_div.py).

Since this tool depends on warnings in runtime, it is likely that a single run of source does not catch every 2-to-3 incompatibility. There is an option --argv, which takes in a text file. source is run for each line in the file, and each line will be passed as the command-line arguments for the program. This makes it easier for test coverage if the program takes input.

Current limitations

• Cannot handle detected incompatibilities spanning multiple lines.
• Cannot use the base lib2to3 fixers, which can be useful for purely syntactical changes.
• Does not filter any warnings from Python's core runtime, which should be ignored.
• No testing framework to verify accuracy in refactoring.