Ad
Testing
Frameworks

Basic idea for the new cw-2.py's timeout function to have an extra argument that prevents running the function right away.

pass
Testing

This tries to address the problem observed in this Kata which tries to wrap the user function in the tests.

def replicate(times, num):
    if times <= 0: return []
    return [num] + replicate(times-1, num)
Testing
Frameworks

More testing on expect_error. By the nature of try..except statement, it is possible to specify multiple exception types to expect.

Testing
Frameworks

This Kumite is a showcase for further possible improvements of Python testing framework cw-2.py, which has been going through major changes. You can see already implemented features here (on Preview site).

Proposal

Improved expect_error(), idea by Blind4Basics

As-is:

  • Test if the function throws an error. It doesn't matter which exception it exactly throws.

To-be:

  • Test if the function throws an error, which is an instance of the given Exception class.
# BaseException >> Exception
def f1(): raise Exception()

# BaseException >> Exception >> ArithmeticError >> ZeroDivisionError
def f2(): return 1 // 0

# BaseException >> Exception >> LookupError >> KeyError
def f3(): return {}[1]

def f0(): pass
Testing
Frameworks

This kumite illustrates some of the new features of the Python test framework (a.k.a. cw-2.py) through a simple kata format.

Example Task

Compute the sum of two integers.

Constraints

-10**9 <= a, b <= 10**9

Example Solutions

  1. add: Correct solution
  2. add1: Incorrect solution
  3. add2: Solution that timeouts
  4. add3: Solution that cheats
  5. add4: Solution that throws error (see how the error is formatted; files and lines are shown properly now)

You can uncomment each line at the bottom of Code section to see how the tests combat the invalid solutions in different ways.

Extra Things in the code section

  1. You can use and print Unicode characters without error!

Example Test Fixture

  1. @Test.describe(name)
  2. @Test.it(name)
  3. @Test.timeout(sec)
  4. Test.assert_equals and Test.assert_not_equals are non-blocking by default
  5. Test.assert_not_equals can now prevent cheating with magic methods

For more complete list of features and guide on how to write a test fixture, consult this page on GitHub Wiki.

Code
Diff
  • # Correct solution
    def add(a, b): return a + b
    
    # Incorrect solution
    def add1(a, b): return a | b
    
    # Solution that timeouts
    def add2(a, b):
        while a: a, b = a-1, b+1
        return b
    
    # Solution that cheats
    def add3(a, b):
        class X(object):
            def __eq__(self, other): return True
            def __ne__(self, other): return True
            def __gt__(self, other): return True
            def __lt__(self, other): return True
            def __ge__(self, other): return True
            def __le__(self, other): return True
        return X()
    
    # Solution that throws error
    def add4(a, b):
        if a > 10: raise NotImplementedError()
        return a + b
        
    #add = add1
    #add = add2
    #add = add3
    #add = add4
    
    # Other things to demonstrate
    print('가나다')
    • # Correct solution
    • def add(a, b): return a + b
    • # Incorrect solution
    • def add1(a, b): return a | b
    • # Solution that timeouts
    • def add2(a, b):
    • while a: a, b = a-1, b+1
    • return b
    • # Solution that cheats
    • def add3(a, b):
    • class X(object):
    • def __eq__(self, other): return True
    • def __ne__(self, other): return True
    • def __gt__(self, other): return True
    • def __lt__(self, other): return True
    • def __ge__(self, other): return True
    • def __le__(self, other): return True
    • return X()
    • # Solution that throws error
    • def add4(a, b):
    • if a > 10: raise NotImplementedError()
    • return a + b
    • #add = add1
    • #add = add2
    • #add = add3
    • #add = add3
    • #add = add4
    • # Other things to demonstrate
    • print('가나다')
Testing
Frameworks

Please go to same kumite on the preview site to run these tests correctly :)

Example Task

Compute the sum of two integers.

Constraints

-10**9 <= a, b <= 10**9

Example Solutions

  1. add: Correct solution
  2. add1: Incorrect solution
  3. add2: Solution that timeouts
  4. add3: Solution that cheats

You can uncomment each line at the bottom of Code section to see how the tests combat the invalid solutions in different ways.

Example Test Fixture

  1. @Test.describe(name)
  2. @Test.it(name)
  3. @Test.timeout(sec)
  4. Test.assert_equals and Test.assert_not_equals are non-blocking by default
  5. Test.assert_not_equals can now prevent cheating with magic methods

For more complete list of features and guide on how to write a test fixture, consult this page on GitHub Wiki.

# Correct solution
def add(a, b): return a + b

# Incorrect solution
def add1(a, b): return a | b

# Solution that timeouts
def add2(a, b):
    while a: a, b = a-1, b+1
    return b

# Solution that cheats
def add3(a, b):
    class X(object):
        def __eq__(self, other): return True
        def __ne__(self, other): return True
        def __gt__(self, other): return True
        def __lt__(self, other): return True
        def __ge__(self, other): return True
        def __le__(self, other): return True
    return X()
    
#add = add1
#add = add2
#add = add3
Testing

Outline

This is a proposal for CodeWars test framework for Python to improve it in many ways (and be more consistent with other languages' frameworks).

Unlike my previous Kumite, this one is designed to completely replace cw-2.py in the runner repo.

Changes / Improvements

Individual Testing / Logging Functions

  • Make testing functions non-blocking
  • Change the expression inside assert_not_equals so that it can prevent operator injection hacks
  • Provide a way to log and test Unicode strings without Unicode-related error
  • Provide a utility for timeout
  • Provide pass, fail and assert_approx_equals

Describe / It

  • Build the decorator version of describe and it, so the text fixture may look like this (and the decorator itself runs the code, so it doesn't need a separate runner function):
@describe('describe-text')
def describe1():
    @it('it-text', before=f1, after=f2)
    def it1():
        # some test function
    @it('it-text')
    def it2():
        # some test function
  • Properly close describe and it blocks in the test output
  • Print the running times of describe and it blocks
  • Provide before and after for describe and it

Changelog

v1.1

  • Replace the timeout utility with decorator version.

v1.2

  • Make the whole code (including fixture examples) compatible with Python 2 using six.
Code
Diff
  • from __future__ import print_function
    import re, six
    range = six.moves.range
    
    
    class AssertException(Exception):
        pass
    
    
    '''Fix the dreaded Unicode Error Trap'''
    def uni_print(*args, **kwargs):
        from sys import stdout
        sep = kwargs.get('sep', ' ')
        end = kwargs.get('end', '\n')
        file = kwargs.get('file', stdout)
        
        def _replace(c):
            if ord(c) >= 128: return u'&#{};'.format(ord(c))
            return c
        def _escape(s):
            escaped = ''.join(_replace(c) for c in six.text_type(s))
            escaped = re.sub(r'\\u([\da-f]{4})', lambda m: '&#{};'.format(int(m.group(1), 16)), escaped)
            escaped = re.sub(r'\\U([\da-f]{8})', lambda m: '&#{};'.format(int(m.group(1), 16)), escaped)
            return escaped
            
        six.print_(*map(_escape, args), sep=_escape(sep), end=_escape(end), file=file)
    
    
    def format_message(message):
        def _replace(c):
            if ord(c) >= 65536: return r'\U' + hex(ord(c))[2:].zfill(8)
            if ord(c) >= 128: return r'\u' + hex(ord(c))[2:].zfill(4)
            return c
        def _escape(s): return ''.join(_replace(c) for c in s)
        return _escape(message.replace("\n", "<:LF:>"))
    
    
    def display(type, message, label="", mode=""):
        print("\n<{0}:{1}:{2}>{3}".format(type.upper(), mode.upper(), label, format_message(message)))
    
    
    def expect(passed=None, message=None, allow_raise=False):
        if passed:
            display('PASSED', 'Test Passed')
        else:
            message = message or "Value is not what was expected"
            display('FAILED', message)
            if allow_raise:
                raise AssertException(message)
    
    
    def assert_equals(actual, expected, message=None, allow_raise=False):
        equals_msg = "{0} should equal {1}".format(repr(actual), repr(expected))
        if message is None:
            message = equals_msg
        else:
            message += ": " + equals_msg
    
        expect(actual == expected, message, allow_raise)
    
    
    def assert_not_equals(actual, expected, message=None, allow_raise=False):
        equals_msg = "{0} should not equal {1}".format(repr(actual), repr(expected))
        if message is None:
            message = equals_msg
        else:
            message += ": " + equals_msg
    
        expect(not (actual == expected), message, allow_raise)
    
    
    def expect_error(message, function):
        passed = False
        try:
            function()
        except:
            passed = True
        expect(passed, message)
    
    
    def pass_(): expect(True)
    def fail(message): expect(False, message)
    
    
    def assert_approx_equals(actual, expected, margin=1e-9, message=None, allow_raise=False):
        equals_msg = "{0} should be close to {1} with absolute or relative margin of {2}".format(
            repr(actual), repr(expected), repr(margin))
        if message is None: message = equals_msg
        else: message += ": " + equals_msg
        div = max(abs(actual), abs(expected), 1)
        expect(abs((actual - expected) / div) < margin, message, allow_raise)
    
    
    '''
    Usage:
    @describe('describe text')
    def describe1():
        @it('it text')
        def it1():
            # some test cases...
    '''
    def _timed_block_factory(opening_text):
        from timeit import default_timer as timer
        from traceback import format_exception
        from sys import exc_info
        
        def _timed_block_decorator(s, before=None, after=None):
            display(opening_text, s)
            def wrapper(func):
                if callable(before): before()
                time = timer()
                try: func()
                except:
                    fail('Unexpected exception raised')
                    tb_str = ''.join(format_exception(*exc_info()))
                    display('ERROR', tb_str)
                display('COMPLETEDIN', '{:.2f}'.format((timer() - time) * 1000))
                if callable(after): after()
            return wrapper
        return _timed_block_decorator
    
    describe = _timed_block_factory('DESCRIBE')
    it = _timed_block_factory('IT')
    
    
    '''
    Timeout utility
    Usage:
    @timeout(sec)
    def some_tests():
        any code block...
    Note: Timeout value can be a float.
    '''
    def timeout(sec):
        def wrapper(func):
            from multiprocessing import Process
            process = Process(target=func)
            process.start()
            process.join(sec)
            if process.is_alive():
                fail('Exceeded time limit of {:.3f} seconds'.format(sec))
                process.terminate()
                process.join()
        return wrapper
    
    
    '''Old-style Fixture'''
    describe('Old-style Describe')
    it('Old-style It')
    assert_equals(0, 0)
    assert_equals(0, 1)
    print('<COMPLETEDIN::>')
    it('Old-style It 2')
    assert_equals('a', 'a')
    assert_equals('a', 'b')
    print('<COMPLETEDIN::>')
    print('<COMPLETEDIN::>')
    
    
    '''Sample Fixture #1'''
    @describe('Sample Fixture #1')
    def sample_describe_1():
        @it('Sample Testcase #1-1')
        def sample_it_1():
            assert_equals(0, 0)
            assert_equals(0, 1)
            assert_not_equals(0, 2)
            pass_()
            fail('This should fail')
        @it('Sample Testcase #1-2')
        def sample_it_2():
            expect_error('ZeroDivisionError', lambda: 0 / 0)
            assert_equals(0, 0 / 0)
            assert_equals(1, 1, 'This is not run due to exception')
        @it('Sample Testcase #1-3')
        def sample_it_3():
            assert_equals('abc', 'abc')
            # source code doesn't support utf-8 chars, but you can at least log and test unicode
            assert_equals(u'\uac00 \ub098 \ub2e4', u'\uac00 \ub098 \ub2e4')
            uni_print(1, 'a', u'\uac00 \ub098 \ub2e4', [2, 'b', u'\uac00'])
            assert_equals(u'\uac00 \ub098 \ub2e4', 'a b c')
    
    
    '''Sample Fixture #2: Featuring Before and After'''
    @describe('Sample Fixture #2')
    def sample_describe_2():
        a = {0}
        def before():
            a.add(len(a))
        @it('Sample Testcase #2-1', before=before, after=before)
        def sample_it_1():
            assert_equals(a, {0, 1})
        @it('Sample Testcase #2-2')
        def sample_it_2():
            assert_equals(a, {0, 1, 2})
    
    
    '''Sample Fixture #3: Featuring Timeout'''
    @describe('Sample Fixture #3')
    def sample_describe_3():
        @it('Sample Testcase #3-1')
        def sample_it_1():
            @timeout(0.01)
            def count():
                for _ in range(100): pass
                pass_()
        @it('Sample Testcase #3-2')
        def sample_it_2():
            @timeout(0.01)
            def count():
                for _ in range(10**10): pass
                pass_()
    
    
    '''Sample Fixture #4: Featuring assert_approx_equals'''
    @describe('Sample Fixture #4')
    def sample_describe_4():
        @it('Sample Testcase #4-1')
        def sample_it_1():
            assert_approx_equals(1, 1 + 1e-10, 1e-9)
            assert_approx_equals(1, 1 + 1e-7, 1e-9)
            assert_approx_equals(-1, -1 - 1e-10, 1e-9)
            assert_approx_equals(-1, -1 - 1e-7, 1e-9)
        @it('Sample Testcase #4-2')
        def sample_it_2():
            assert_approx_equals(0, 1e-10, 1e-9)
            assert_approx_equals(0, 1e-7, 1e-9)
            assert_approx_equals(0, -1e-10, 1e-9)
            assert_approx_equals(0, -1e-7, 1e-9)
    
    • #from __future__ import print_function
    • from __future__ import print_function
    • import re, six
    • range = six.moves.range
    • class AssertException(Exception):
    • pass
    • '''Fix the dreaded Unicode Error Trap'''
    • _print = print
    • def print(*args, sep=' ', end='
    • '):
    • from io import StringIO
    • def _escape(s): return s.encode('ascii', 'xmlcharrefreplace').decode('ascii')
    • sio = StringIO()
    • _print(*args, sep=sep, end=end, file=sio)
    • _print(_escape(sio.getvalue()))
    • sio.close()
    • def uni_print(*args, **kwargs):
    • from sys import stdout
    • sep = kwargs.get('sep', ' ')
    • end = kwargs.get('end', '
    • ')
    • file = kwargs.get('file', stdout)
    • def _replace(c):
    • if ord(c) >= 128: return u'&#{};'.format(ord(c))
    • return c
    • def _escape(s):
    • escaped = ''.join(_replace(c) for c in six.text_type(s))
    • escaped = re.sub(r'\\u([\da-f]{4})', lambda m: '&#{};'.format(int(m.group(1), 16)), escaped)
    • escaped = re.sub(r'\\U([\da-f]{8})', lambda m: '&#{};'.format(int(m.group(1), 16)), escaped)
    • return escaped
    • six.print_(*map(_escape, args), sep=_escape(sep), end=_escape(end), file=file)
    • def format_message(message):
    • return message.replace("
    • ", "<:LF:>")
    • def _replace(c):
    • if ord(c) >= 65536: return r'\U' + hex(ord(c))[2:].zfill(8)
    • if ord(c) >= 128: return r'\u' + hex(ord(c))[2:].zfill(4)
    • return c
    • def _escape(s): return ''.join(_replace(c) for c in s)
    • return _escape(message.replace("
    • ", "<:LF:>"))
    • def display(type, message, label="", mode=""):
    • print("\n<{0}:{1}:{2}>{3}".format(type.upper(), mode.upper(), label, format_message(message)))
    • def expect(passed=None, message=None, allow_raise=False):
    • if passed: print("\n<PASSED::>Test Passed")
    • if passed:
    • display('PASSED', 'Test Passed')
    • else:
    • message = message or "Value is not what was expected"
    • print("\n<FAILED::>{0}".format(message))
    • if allow_raise: raise AssertException(message)
    • display('FAILED', message)
    • if allow_raise:
    • raise AssertException(message)
    • '''Fix the blocking asserts to non-blocking'''
    • def assert_equals(actual, expected, message=None, allow_raise=False):
    • equals_msg = "{0} should equal {1}".format(repr(actual), repr(expected))
    • if message is None: message = equals_msg
    • else: message += ": " + equals_msg
    • if message is None:
    • message = equals_msg
    • else:
    • message += ": " + equals_msg
    • expect(actual == expected, message, allow_raise)
    • '''
    • Fix the blocking asserts to non-blocking
    • Also change the expected formula from `actual != expected` to `not (actual == expected)`
    • so that using this assertion can prevent the `==` / `!=` injection hack
    • '''
    • def assert_not_equals(actual, expected, message=None, allow_raise=False):
    • equals_msg = "{0} should not equal {1}".format(repr(actual), repr(expected))
    • if message is None: message = equals_msg
    • else: message += ": " + equals_msg
    • if message is None:
    • message = equals_msg
    • else:
    • message += ": " + equals_msg
    • expect(not (actual == expected), message, allow_raise)
    • def expect_error(message, function):
    • passed = False
    • try: function()
    • except: passed = True
    • try:
    • function()
    • except:
    • passed = True
    • expect(passed, message)
    • '''Additional test functions: pass, fail, and assert_approx_equals'''
    • def pass_(): expect(True)
    • def fail(message): expect(False, message)
    • def assert_approx_equals(actual, expected, margin=1e-9, message=None, allow_raise=False):
    • equals_msg = "{0} should be close to {1} with absolute or relative margin of {2}".format(
    • repr(actual), repr(expected), repr(margin))
    • if message is None: message = equals_msg
    • else: message += ": " + equals_msg
    • div = max(abs(actual), abs(expected), 1)
    • expect(abs((actual - expected) / div) < margin, message, allow_raise)
    • def display(type, message, label="", mode=""):
    • print("\n<{0}:{1}:{2}>{3}".format(type.upper(), mode.upper(), label, format_message(message)))
    • '''
    • Modern-Style Describe & It
    • Usage:
    • @describe('describe text')
    • def describe1():
    • @it('it text')
    • def it1():
    • # some test cases...
    • '''
    • def _timed_block_factory(opening_text):
    • from timeit import default_timer as timer
    • from traceback import format_exception
    • from sys import exc_info
    • def _timed_block_decorator(s, before=None, after=None):
    • print('<{}::>{}'.format(opening_text, s))
    • display(opening_text, s)
    • def wrapper(func):
    • if callable(before): before()
    • time = timer()
    • try: func()
    • except:
    • fail('Unexpected exception raised')
    • tb_str = ''.join(format_exception(*exc_info())).replace('\n', '<:LF:>')
    • print('<ERROR::>' + tb_str)
    • print('<COMPLETEDIN::>{}'.format(round((timer() - time) * 1000, 2)))
    • tb_str = ''.join(format_exception(*exc_info()))
    • display('ERROR', tb_str)
    • display('COMPLETEDIN', '{:.2f}'.format((timer() - time) * 1000))
    • if callable(after): after()
    • return wrapper
    • return _timed_block_decorator
    • describe = _timed_block_factory('DESCRIBE')
    • it = _timed_block_factory('IT')
    • '''
    • Timeout utility
    • Usage:
    • with run_with_timeout(func, tuple_of_args, timeout_in_seconds) as run:
    • Test.assert_equals(run.get(), expected_value)
    • @timeout(sec)
    • def some_tests():
    • any code block...
    • Note: Timeout value can be a float.
    • '''
    • class run_with_timeout(object):
    • def __init__(self, func, inputs, sec):
    • from multiprocessing import Process, Queue
    • def timeout_wrapper(func, inputs, q):
    • q.put(func(*inputs))
    • self.sec = sec
    • self.q = Queue()
    • self.p = Process(target=timeout_wrapper, args=(func, inputs, self.q))
    • self.result = None
    • def __enter__(self):
    • self.p.start()
    • return self
    • def get(self):
    • if self.result is None: self.result = self.q.get(timeout=self.sec)
    • return self.result
    • def __exit__(self, typ, val, traceback):
    • self.q.close()
    • self.p.terminate()
    • self.p.join()
    • if traceback: fail('Exceeded time limit of {:.3f} seconds'.format(self.sec))
    • return True
    • def timeout(sec):
    • def wrapper(func):
    • from multiprocessing import Process
    • process = Process(target=func)
    • process.start()
    • process.join(sec)
    • if process.is_alive():
    • fail('Exceeded time limit of {:.3f} seconds'.format(sec))
    • process.terminate()
    • process.join()
    • return wrapper
    • '''Old-style Fixture'''
    • describe('Old-style Describe')
    • it('Old-style It')
    • assert_equals(0, 0)
    • assert_equals(0, 1)
    • print('<COMPLETEDIN::>')
    • it('Old-style It 2')
    • assert_equals('a', 'a')
    • assert_equals('a', 'b')
    • print('<COMPLETEDIN::>')
    • print('<COMPLETEDIN::>')
    • '''Sample Fixture #1'''
    • @describe('Sample Fixture #1')
    • def sample_describe_1():
    • @it('Sample Testcase #1-1')
    • def sample_it_1():
    • assert_equals(0, 0)
    • assert_equals(0, 1)
    • assert_not_equals(0, 2)
    • pass_()
    • fail('This should fail')
    • @it('Sample Testcase #1-2')
    • def sample_it_2():
    • expect_error('ZeroDivisionError', lambda: 0 / 0)
    • assert_equals(0, 0 / 0)
    • assert_equals(1, 1, 'This is not run due to exception')
    • @it('Sample Testcase #1-3')
    • def sample_it_3():
    • assert_equals('abc', 'abc')
    • # source code doesn't support utf-8 chars, but you can at least log and test unicode
    • assert_equals('\uac00 \ub098 \ub2e4', '\uac00 \ub098 \ub2e4')
    • print('\uac00 \ub098 \ub2e4')
    • assert_equals('\uac00 \ub098 \ub2e4', 'a b c')
    • assert_equals(u'\uac00 \ub098 \ub2e4', u'\uac00 \ub098 \ub2e4')
    • uni_print(1, 'a', u'\uac00 \ub098 \ub2e4', [2, 'b', u'\uac00'])
    • assert_equals(u'\uac00 \ub098 \ub2e4', 'a b c')
    • '''Sample Fixture #2: Featuring Before and After'''
    • @describe('Sample Fixture #2')
    • def sample_describe_2():
    • a = {0}
    • def before():
    • a.add(len(a))
    • @it('Sample Testcase #2-1', before=before, after=before)
    • def sample_it_1():
    • assert_equals(a, {0, 1})
    • @it('Sample Testcase #2-2')
    • def sample_it_2():
    • assert_equals(a, {0, 1, 2})
    • '''Sample Fixture #3: Featuring Timeout'''
    • @describe('Sample Fixture #3')
    • def sample_describe_3():
    • def wait_count(n):
    • for _ in range(n): pass
    • return n
    • @it('Sample Testcase #3-1')
    • def sample_it_1():
    • with run_with_timeout(wait_count, (100,), 0.01) as run:
    • assert_equals(run.get(), 100)
    • @timeout(0.01)
    • def count():
    • for _ in range(100): pass
    • pass_()
    • @it('Sample Testcase #3-2')
    • def sample_it_2():
    • with run_with_timeout(wait_count, (10 ** 10,), 0.01) as run:
    • assert_equals(run.get(), 10 ** 10)
    • @timeout(0.01)
    • def count():
    • for _ in range(10**10): pass
    • pass_()
    • '''Sample Fixture #4: Featuring assert_approx_equals'''
    • @describe('Sample Fixture #4')
    • def sample_describe_4():
    • @it('Sample Testcase #4-1')
    • def sample_it_1():
    • assert_approx_equals(1, 1 + 1e-10, 1e-9)
    • assert_approx_equals(1, 1 + 1e-7, 1e-9)
    • assert_approx_equals(-1, -1 - 1e-10, 1e-9)
    • assert_approx_equals(-1, -1 - 1e-7, 1e-9)
    • @it('Sample Testcase #4-2')
    • def sample_it_2():
    • assert_approx_equals(0, 1e-10, 1e-9)
    • assert_approx_equals(0, 1e-7, 1e-9)
    • assert_approx_equals(0, -1e-10, 1e-9)
    • assert_approx_equals(0, -1e-7, 1e-9)
Testing

Outline

This is a proposal for CodeWars test framework for Python to improve it in many ways (and be more consistent with other languages' frameworks).

Unlike my previous Kumite, this one is designed to completely replace cw-2.py in the runner repo.

Changes / Improvements

Individual Testing / Logging Functions

  • Make testing functions non-blocking
  • Change the expression inside assert_not_equals so that it can prevent operator injection hacks
  • Provide a way to log and test Unicode strings without Unicode-related error
  • Provide a utility for timeout
  • Provide pass, fail and assert_approx_equals

Describe / It

  • Build the decorator version of describe and it, so the text fixture may look like this (and the decorator itself runs the code, so it doesn't need a separate runner function):
@describe('describe-text')
def describe1():
    @it('it-text', before=f1, after=f2)
    def it1():
        # some test function
    @it('it-text')
    def it2():
        # some test function
  • Properly close describe and it blocks in the test output
  • Print the running times of describe and it blocks
  • Provide before and after for describe and it
#from __future__ import print_function

class AssertException(Exception):
    pass


'''Fix the dreaded Unicode Error Trap'''
_print = print
def print(*args, sep=' ', end='\n'):
    from io import StringIO
    def _escape(s): return s.encode('ascii', 'xmlcharrefreplace').decode('ascii')
    sio = StringIO()
    _print(*args, sep=sep, end=end, file=sio)
    _print(_escape(sio.getvalue()))
    sio.close()


def format_message(message):
    return message.replace("\n", "<:LF:>")


def expect(passed=None, message=None, allow_raise=False):
    if passed: print("\n<PASSED::>Test Passed")
    else:
        message = message or "Value is not what was expected"
        print("\n<FAILED::>{0}".format(message))
        if allow_raise: raise AssertException(message)


'''Fix the blocking asserts to non-blocking'''
def assert_equals(actual, expected, message=None, allow_raise=False):
    equals_msg = "{0} should equal {1}".format(repr(actual), repr(expected))
    if message is None: message = equals_msg
    else: message += ": " + equals_msg
    expect(actual == expected, message, allow_raise)


'''
Fix the blocking asserts to non-blocking
Also change the expected formula from `actual != expected` to `not (actual == expected)`
so that using this assertion can prevent the `==` / `!=` injection hack
'''
def assert_not_equals(actual, expected, message=None, allow_raise=False):
    equals_msg = "{0} should not equal {1}".format(repr(actual), repr(expected))
    if message is None: message = equals_msg
    else: message += ": " + equals_msg
    expect(not (actual == expected), message, allow_raise)


def expect_error(message, function):
    passed = False
    try: function()
    except: passed = True
    expect(passed, message)


'''Additional test functions: pass, fail, and assert_approx_equals'''
def pass_(): expect(True)
def fail(message): expect(False, message)


def assert_approx_equals(actual, expected, margin=1e-9, message=None, allow_raise=False):
    equals_msg = "{0} should be close to {1} with absolute or relative margin of {2}".format(
        repr(actual), repr(expected), repr(margin))
    if message is None: message = equals_msg
    else: message += ": " + equals_msg
    div = max(abs(actual), abs(expected), 1)
    expect(abs((actual - expected) / div) < margin, message, allow_raise)


def display(type, message, label="", mode=""):
    print("\n<{0}:{1}:{2}>{3}".format(type.upper(), mode.upper(), label, format_message(message)))


'''
Modern-Style Describe & It
Usage:
@describe('describe text')
def describe1():
    @it('it text')
    def it1():
        # some test cases...
'''
def _timed_block_factory(opening_text):
    from timeit import default_timer as timer
    from traceback import format_exception
    from sys import exc_info
    
    def _timed_block_decorator(s, before=None, after=None):
        print('<{}::>{}'.format(opening_text, s))
        def wrapper(func):
            if callable(before): before()
            time = timer()
            try: func()
            except:
                fail('Unexpected exception raised')
                tb_str = ''.join(format_exception(*exc_info())).replace('\n', '<:LF:>')
                print('<ERROR::>' + tb_str)
            print('<COMPLETEDIN::>{}'.format(round((timer() - time) * 1000, 2)))
            if callable(after): after()
        return wrapper
    return _timed_block_decorator

describe = _timed_block_factory('DESCRIBE')
it = _timed_block_factory('IT')


'''
Timeout utility
Usage:
with run_with_timeout(func, tuple_of_args, timeout_in_seconds) as run:
    Test.assert_equals(run.get(), expected_value)
Note: Timeout value can be a float.
'''
class run_with_timeout(object):
    def __init__(self, func, inputs, sec):
        from multiprocessing import Process, Queue
        def timeout_wrapper(func, inputs, q):
            q.put(func(*inputs))
        self.sec = sec
        self.q = Queue()
        self.p = Process(target=timeout_wrapper, args=(func, inputs, self.q))
        self.result = None
    def __enter__(self):
        self.p.start()
        return self
    def get(self):
        if self.result is None: self.result = self.q.get(timeout=self.sec)
        return self.result
    def __exit__(self, typ, val, traceback):
        self.q.close()
        self.p.terminate()
        self.p.join()
        if traceback: fail('Exceeded time limit of {:.3f} seconds'.format(self.sec))
        return True


'''Old-style Fixture'''
describe('Old-style Describe')
it('Old-style It')
assert_equals(0, 0)
assert_equals(0, 1)
print('<COMPLETEDIN::>')
it('Old-style It 2')
assert_equals('a', 'a')
assert_equals('a', 'b')
print('<COMPLETEDIN::>')
print('<COMPLETEDIN::>')


'''Sample Fixture #1'''
@describe('Sample Fixture #1')
def sample_describe_1():
    @it('Sample Testcase #1-1')
    def sample_it_1():
        assert_equals(0, 0)
        assert_equals(0, 1)
        assert_not_equals(0, 2)
        pass_()
        fail('This should fail')
    @it('Sample Testcase #1-2')
    def sample_it_2():
        expect_error('ZeroDivisionError', lambda: 0 / 0)
        assert_equals(0, 0 / 0)
        assert_equals(1, 1, 'This is not run due to exception')
    @it('Sample Testcase #1-3')
    def sample_it_3():
        assert_equals('abc', 'abc')
        # source code doesn't support utf-8 chars, but you can at least log and test unicode
        assert_equals('\uac00 \ub098 \ub2e4', '\uac00 \ub098 \ub2e4')
        print('\uac00 \ub098 \ub2e4')
        assert_equals('\uac00 \ub098 \ub2e4', 'a b c')


'''Sample Fixture #2: Featuring Before and After'''
@describe('Sample Fixture #2')
def sample_describe_2():
    a = {0}
    def before():
        a.add(len(a))
    @it('Sample Testcase #2-1', before=before, after=before)
    def sample_it_1():
        assert_equals(a, {0, 1})
    @it('Sample Testcase #2-2')
    def sample_it_2():
        assert_equals(a, {0, 1, 2})


'''Sample Fixture #3: Featuring Timeout'''
@describe('Sample Fixture #3')
def sample_describe_3():
    def wait_count(n):
        for _ in range(n): pass
        return n
    @it('Sample Testcase #3-1')
    def sample_it_1():
        with run_with_timeout(wait_count, (100,), 0.01) as run:
            assert_equals(run.get(), 100)
    @it('Sample Testcase #3-2')
    def sample_it_2():
        with run_with_timeout(wait_count, (10 ** 10,), 0.01) as run:
            assert_equals(run.get(), 10 ** 10)


'''Sample Fixture #4: Featuring assert_approx_equals'''
@describe('Sample Fixture #4')
def sample_describe_4():
    @it('Sample Testcase #4-1')
    def sample_it_1():
        assert_approx_equals(1, 1 + 1e-10, 1e-9)
        assert_approx_equals(1, 1 + 1e-7, 1e-9)
        assert_approx_equals(-1, -1 - 1e-10, 1e-9)
        assert_approx_equals(-1, -1 - 1e-7, 1e-9)
    @it('Sample Testcase #4-2')
    def sample_it_2():
        assert_approx_equals(0, 1e-10, 1e-9)
        assert_approx_equals(0, 1e-7, 1e-9)
        assert_approx_equals(0, -1e-10, 1e-9)
        assert_approx_equals(0, -1e-7, 1e-9)
Testing

This shows a piece of test code in Python that is improved over the default cw-2 (and consistent with other languages' test frameworks) in the following aspects:

Initial release (v1.0)

  • Utilize the with-block to support proper indentation of describe and it blocks in the code
  • Properly close describe and it blocks in the test output
  • Print the running times of describe and it blocks
  • Make testing functions non-blocking

v1.1

  • Provide a way to log Unicode strings for users (does not work for test framework)

v1.2

  • Provide before and after for describe and it
  • Provide a utility for timeout
  • Version 1: Function version
  • Version 2: Context manager version (looks cleaner)

Run the tests and see how the test output looks like.

Testing

This shows a piece of test code in Python that is improved over the default cw-2 (and consistent with other languages' test frameworks) in the following aspects:

Initial release (v1.0)

  • Utilize the with-block to support proper indentation of describe and it blocks in the code
  • Properly close describe and it blocks in the test output
  • Print the running times of describe and it blocks
  • Make testing functions non-blocking

v1.1

  • Provide a way to log Unicode strings for users (does not work for test framework)

Run the tests and see how the test output looks like.

def fibonacci(n):
    if n <= 1: return 1
    if n > 35: raise RuntimeError('Too large input')
    return fibonacci(n-1) + fibonacci(n-2)