Understanding Python variables as pointers

TRUE STORY FOLLOWS:

I was writing some code, and everyone was counting on me, much like the scene depicted below as Wolverine faced insurmountable challenges in writing some powerful and architecturally sound code in the movie Swordfish.

7.-Swordfish

All was going well, and I was obviously getting high-fives all over the place, but I unfortunately reached a stumbling block that took me several hours to figure out what was going on.  If only I’d been more familiar with what was going on under the hood with Python variables, I would’ve saved myself a few hours.

The Heart of the Problem

At its core, the specific part of the problem was basically as follows:

case1

 

In the class depicted above, it appears fairly intuitive to the programmer of what the intent is here.   Make a copy of the dictionary so as to not modify the original values, then manipulate and use some of the values in the dictionary that’s inside of the original one.  However, those familiar with python can likely catch the problem if you examine it closer.  The copy of the original dictionary is just copying the references to the values and not the actual values themselves.  Therefore, when the copied dictionary’s sub-dictionary is modified, it’s really the original dictionary that’s still being modified as well.   As I said, this tripped me up for several hours.

BUT DON’T JUDGE ME YET!

This problem in isolation might be easily solved, but the plot thickens!  What really threw off my game was that there were, in essence, two separate instances of SomeClass(), yet the function do_some_stuff () called from the first object would propagate problems to the second object, which created other weird symptoms that I had to examine.  Here’s some additional context:

case3

 

So, if  I create two instances of SubClass() and then call change_things() on only one of those classes, will it affect my_dict_obj in the other object?  The answer is yes.  Here’s some quick proof in case you need it:

case4

Output:

case4_1

The dictionaries inside the classes are actually statically defined, so whatever object is instantiated  as SomeClass() will have a dictionary that references the same place in memory.  So, that’s pretty much the entire story.  This specific problem was solved just by doing a deepcopy() of the dictionary in question, and then of course everyone gave me high-fives all over the place.  However, this prompted me to write a few more code samples to get a better understanding of what was going on under the hood with Python.   I come from a C++ background myself, so things made a lot more sense to me with python when I examined the memory addresses of all of the referenced objects in the program.  I had a few take-aways myself by running through this exercise, so I’d hope that by sharing it with the internets, I can make the world a better place.

 

So first of all, you can take a look at some other well explained examples at Stack Overflow for what python variables are. All python references are, in effect, C pointers under the hood.  For the rest of my examples in python, let me just define a quick function that we’ll reuse throughout:

__mem_address

 

Quite simply, this returns the memory address of the referenced object that’s passed into the function.

So for starters, here’s some basically identical behavior (from a programmer’s perspective) in C++ and python and their respective outputs:

001 003
002 004

You’ll notice that in python, an integer object is created (in this case to represent the number 5), and subsequent variables set to that value are actually references to the same object, and thus both variables reference the same place in memory.  In C++, however, memory is allocated for the specific variable to the size of an integer, and the address in memory takes on the value 5.

 

Just to take it one step further, let’s take the same actions in python but make the variables lose their scope:

next_case

 

Output:

next_case_output

Same thing happens.  This would help explain why integer variables that go out of scope still do not free up the memory that you’d intuitively expect.

So what probably most closely resembles what’s going on underneath the hood is something like this:

015 007
016 008

It makes sense to think of python variables as pointers.  But, all of these pointers point to an object.  So here’s another C++ to python example dealing with arrays and lists, respectively:

019 017
020 018

If you commonly think of C++ arrays as pointers, then you know that an array itself and the first index of the array point to the same place in memory.  In python with list objects, this is not so.  The list is an object with a bunch of self-contained behavior.  Let me just illustrate this one step further:

 

23 21
24 22

 

I used some random numbers just to make the example more intuitive.  You’ll notice that with the C-style array, the memory addresses are sequential and have a difference of 4 bytes (the size of an int).  In python, for a list at least, you’ll notice that the indices have no correlation to the memory address of the referenced item, but they do appear to correlate to the values themselves (this is why I didn’t use sequential numbers; you would have seen an unrelated correlation to the list indices).

 

…But I digress

To the point that I originally brought up in the context of the problem that I had, here’s some straightforward and intuitive behavior:

33

Output:

34

 

The some_class object has a dictionary defined within, and each instantiated object has a unique memory address, and the dictionary within also contains a unique memory address.  This is what you’d expect.

Now if you make what might be considered a small change in some contexts, you get entirely different behavior.

35

Output:

36

 

The same dictionary in this instance is defined statically.  Therefore each instantiated object still references the same dictionary.  This is useful to know both to deter unwanted behavior, but this can also be used to your advantage as in the case below:

37

 

Output:

38

 

You’ll notice that the dictionary contained within the class can be referenced both from an instantiated object and from the class object itself.  Having a better understanding of this at the outset of my initial debugging in the problem described above would have vastly expedited the process.

Now, onto the core problem that I described.  Consider this:

39

Output:

40

 

When we create a copy of a dictionary, we’re creating a copy of the key, and a copy of the value.  But in this the case, the value is really a pointer.  So a copy of the pointer is made, but what the pointer references has not changed.  So now, this unwanted behavior is produced:

41

Output:

42

 

The second dictionary still points to an item in the first dictionary.  So whether we manipulate the sub dictionary from either dict_obj or dict_obj2, the values in both dictionaries will change.  To fix this, we can just create a deepcopy:

43

Output:

44

 

That concludes what was relevant to my specific problem addressed, but while I was at this, I just created a few quick examples of some other common problems that might surface as a result of referencing the same objects in memory.

Here’s a quick function with some straightforward and intuitive behavior:

47

 

Output:

48

 

As you make a function call to some_func, the default paramer includes an immutable type, and since no parameters are passed to the function, its behavior is as you’d expect.

With immutable types, operations are not done directly on the object, but instead, those operations will create and reference a new object entirely.   For example:

49

Output:

50

 

The += operation in this case does not actually amend anything to the initial string.  Instead, a new object is created in memory that concatenates both the initial string and the value on the right side of the operation.

The example below is in most cases agreeably a no-no, in which counter-intuitive behavior is being invited:

 

45

 

Output:

46

 

Here we follow the same process as the previous example.  Get a value from a function which has a default parameter, modify the returned value, and then call the original function again.  This time, however, the default parameter is a mutable type, and the operation against the value later on does not create a new object.   The default_param parameter which references a place in memory is not redefined upon the function call, which is behavior that someone might expect.  Instead, it just establishes a reference to an object in memory, and that object is later manipulated.

Since Steve Ballmer announced his retirement yesterday, I’ll close with the thoughts conveyed in this video:

  • TOm

    While I don’t understand any of the stuff you’re writing about, I loved the video at the end.

    Dig the new blog Scott – keep it up!

  • Richard Howell-Peak

    I had a similar issue a while back writing a video game. The whole self.variable thing tripped me up for a few hours as well! I like the extra depth you’ve gone into here though really shedding a lot more light on how python works “under the hood”, thanks for taking the time to do that.