Decorators With Arguments in Python

True Story Follows

So I’m on the grind, straight tearing up the keyboard. You’d think I was playing Street Fighter II on my laptop using Ken and fighting against Guile in front of a 2 dimensional F-16 with some random dudes fist pumping, but no, I was typing into vim.

Anyway, in my conscious effort for code to be as clean as possible and ensure that a function does exactly one thing and no more, I encountered a case where I could either break this rule or avoid redundant network calls. Specifically, I had a function that validated input (with Django forms) and a separate function that actually used the input. In this case, the input was an address, and I was making a network call via an API to turn that into lat lon coordinates.

I could certainly just bend my self-imposed rule a little bit and make the function in question mutate a value to store the latitude and longitude, but this isn’t straightforward, and now the operator of the said function would need to know somehow that this mutation is occuring. Whenever I’m the operator reading code like this, the rate of “wtf’s per minute” spikes briefly.

So to get to the point, I could basically keep myself and everyone else happy by writing code in the style I wanted to and maintain efficiency by avoiding the network call using memoization. The standard way to memoize functions in python is just with a simple decorator, but a super simple caching mechanism can also create memory pressure and incur a trade-off that leads to a whole new set of problems.

So to make a long story short, I wanted to make a memoization decorator that took a parameter that set a cache size. And in order to do that, I needed to do a little bit of playing around to understand how decorators in python with parameters work.

The Basic Python Decorator

Here’s the most basic syntax for a Python Decorator:

def pass_thru(func_to_decorate):
    def new_func(*original_args, **original_kwargs):
        print "Function has been decorated.  Congratulations."
        # Do whatever else you want here
        return func_to_decorate(*original_args, **original_kwargs)
    return new_func


@pass_thru
def print_args(*args):
    for arg in args:
        print arg


print_args(1, 2, 3)

This sort of syntax is really the equivalent of:

def pass_thru(func_to_decorate):
    def new_func(*original_args, **original_kwargs):
        print "Function has been decorated.  Congratulations."
        # Do whatever else you want here
        return func_to_decorate(*original_args, **original_kwargs)
    return new_func


# Notice nothing here now
def print_args(*args):
    for arg in args:
        print arg


# Notice the change here
pass_thru(print_args)(1, 2, 3)

Output

Function has been decorated.  Congratulations.
1
2
3

Function Decorators with Arguments

I can also do some other crazy stuff to return a function that will take a function to decorate.

def decorator(arg1, arg2):

    def real_decorator(function):

        def wrapper(*args, **kwargs):
            print "Congratulations.  You decorated a function that does something with %s and %s" % (arg1, arg2)
            function(*args, **kwargs)
        return wrapper

    return real_decorator


@decorator("arg1", "arg2")
def print_args(*args):
    for arg in args:
        print arg

And this sort of syntax is equivalent to:

def decorator(arg1, arg2):

    def real_decorator(function):

        def wrapper(*args, **kwargs):
            print "Congratulations.  You decorated a function that does something with %s and %s" % (arg1, arg2)
            function(*args, **kwargs)
        return wrapper

    return real_decorator


# No more decorator here
def print_args(*args):
    for arg in args:
        print arg


# getting crazy down here
decorator("arg1", "arg2")(print_args)(1, 2, 3)

Output

Congratulations.  You decorated a function that does something with arg1 and arg2
1
2
3

Class Based Decorators

If you want to maintain some sort of state and/or just make your code more confusing, you can also use class based decorators. So here’s an example:

class ClassBasedDecorator(object):

    def __init__(self, func_to_decorate):
        print "INIT ClassBasedDecorator"
        self.func_to_decorate = func_to_decorate

    def __call__(self, *args, **kwargs):
        print "CALL ClassBasedDecorator"
        return self.func_to_decorate(*args, **kwargs)


@ClassBasedDecorator
def print_moar_args(*args):
    for arg in args:
        print arg


print_moar_args(1, 2, 3)
print_moar_args(1, 2, 3)

Output

INIT ClassBasedDecorator
CALL ClassBasedDecorator
1
2
3
CALL ClassBasedDecorator
1
2
3

Notice how __call__ is executed each time the function is executed.

Class Based Decorator with Arguments

Now if we wanted to add arguments to the decorator the structure of the class changes, you’ll note that the function to decorate is now a parameter with the call method.

class ClassBasedDecoratorWithParams(object):

    def __init__(self, arg1, arg2):
        print "INIT ClassBasedDecoratorWithParams"
        print arg1
        print arg2

    def __call__(self, fn, *args, **kwargs):
        print "CALL ClassBasedDecoratorWithParams"

        def new_func(*args, **kwargs):
            print "Function has been decorated.  Congratulations."
            return fn(*args, **kwargs)
        return new_func


@ClassBasedDecoratorWithParams("arg1", "arg2")
def print_args_again(*args):
    for arg in args:
        print arg


print_args_again(1, 2, 3)
print_args_again(1, 2, 3)

Output

INIT ClassBasedDecoratorWithParams
arg1
arg2
CALL ClassBasedDecoratorWithParams
Function has been decorated.  Congratulations.
1
2
3
Function has been decorated.  Congratulations.
1
2
3

You can also see now that __call__ is only called once.

My Memoization Decorator

And finally, we come to the reason for making an effort to understanding decorators: a simple class to memoize the results of a function. For clarity’s sake, I’m simply making it so that the results of a function with certain arguments are cached so that redundant calls don’t execute potentially costly code.

from collections import deque


class Memoized(object):

    def __init__(self, cache_size=100):
        self.cache_size = cache_size
        self.call_args_queue = deque()
        self.call_args_to_result = {}

    def __call__(self, fn, *args, **kwargs):

        def new_func(*args, **kwargs):
            memoization_key = self._convert_call_arguments_to_hash(args, kwargs)
            if memoization_key not in self.call_args_to_result:
                result = fn(*args, **kwargs)
                self._update_cache_key_with_value(memoization_key, result)
                self._evict_cache_if_necessary()
            return self.call_args_to_result[memoization_key]

        return new_func

    def _update_cache_key_with_value(self, key, value):
        self.call_args_to_result[key] = value
        self.call_args_queue.append(key)

    def _evict_cache_if_necessary(self):
        if len(self.call_args_queue) > self.cache_size:
            oldest_key = self.call_args_queue.popleft()
            del self.call_args_to_result[oldest_key]

    @staticmethod
    def _convert_call_arguments_to_hash(args, kwargs):
        return hash(str(args) + str(kwargs))


@Memoized(cache_size=5)
def get_not_so_random_number_with_max(max_value):
    import random
    return random.random() * max_value

Output

Let’s say I ran this:

print get_not_so_random_number_with_max(1)
print get_not_so_random_number_with_max(1)
print get_not_so_random_number_with_max(2)
print get_not_so_random_number_with_max(2)
print get_not_so_random_number_with_max(3)
print get_not_so_random_number_with_max(3)
print get_not_so_random_number_with_max(4)
print get_not_so_random_number_with_max(4)
print get_not_so_random_number_with_max(5)
print get_not_so_random_number_with_max(5)
print get_not_so_random_number_with_max(6)
print get_not_so_random_number_with_max(6)
print get_not_so_random_number_with_max(1)

If this works properly, a random number will not be generated a second time with a particular argument because of the cached result. But since I passed a cache size of 5, then after 5 function calls with different arguments, cache evictions will free up the memoized results and the function will be run again. So the actual output:

0.622446623043
0.622446623043
1.40971776532
1.40971776532
1.56628678698
1.56628678698
2.21064925087
2.21064925087
2.80447028359
2.80447028359
4.05414581689
4.05414581689
0.0251415894698

Sweet, it works (The last call with an argument of 1 does not match the initial two function calls).

To do just a bit more code analysis, a typical problem you might run into with memoizing results in Python is two-fold:

Since you can make function calls to python using any combination of arguments and keyword arguments (if keyword arguments are allowable for a function), it might be difficult to find a way to map all of the possible argument permutations to the same result. I dealt with this just by not mapping them to the same result at all. This shouldn’t be a problem because code is likely to use the same combination of arguments (between arguments and keyword arguments) anyway, and this won’t create problems with memory since we’ve already limited how large the cache can grow.

The other issue is that you can only hash immutable objects. So if a function parameter happens to be a list or a set or dictionary or anything else that can’t be hashed, you need to account for that. I handled this just by casting everything as a string and hashing that value.

The End

  • Pingback: Python decorators - Where do parameters come from? - BlogoSfera()

  • zofy

    Great!

  • Rodolfo Torres Jaime

    I read 10 articles and questions on stackoverflow and you were the only one who answered my question

    # getting crazy down here
    decorator(“arg1”, “arg2”)(print_args)(1, 2, 3)

    wanted to know what the syntax of a decorator with arguments looked like
    Thank you very much!

  • Rishikesh Agrawani

    Excellent Article about Python decorators.