reload and work flow suggestions

[Peter Cacioppi]

This is an idea brought over from another post.

When I write Python code I generally have 2 or 3 windows open simultaneously.

1) An editor for the actual code.
2) The interactive interpreter.
3) An editor for the unit tests. (Sometimes skipped for quick one-off scripts)

My work flow tends to involve using 2 to debug the issues that come up with1 and 3. I'll write some new code in 1, play around with it in 2, then solidify the tests in 3. Or a test in 3 fails and I dig around with it using 2..

My problem is that I tend to use reload() quite a bit. I want to call functions and construct objects that are inside the guts of 1 and pass them arguments that are stored as variables in 2. If I restart my session for 2 I lose these variables (iPython does mitigate the pain here somewhat). Hence, Ireload() modules into 2 when they are changed.

I use ipdb a lot in 2. I usually don't feel comfortable with virgin code ora debug fix that hasn't been stepped through with the debugger.

Is there something wrong with this work flow? I understand most python experts avoid reload(). So what are they doing that I'm not? I love the abilityof Python to quickly let you dive deep into your code and set up a difficult case with 2, it's hard to imagine giving this up, and it's hard to imagine using it without reload().

Thanks for any tips.

 

[Terry Reedy]

This is an idea brought over from another post.

When I write Python code I generally have 2 or 3 windows open simultaneously.

1) An editor for the actual code.
2) The interactive interpreter.
3) An editor for the unit tests. (Sometimes skipped for quick one-off scripts)

At present, I use an Idle Shell and two editor windows.

My work flow tends to involve using 2 to debug the issues that come up with 1 and 3. I'll write some new code in 1, play around with it in 2, then solidify the tests in 3. Or a test in 3 fails and I dig around with it using 2.

F5 in an editor window runs the code, puts the results in Shell, and
leaves the objects loaded for interactive exploration in Shell.

My problem is that I tend to use reload() quite a bit. I want to call functions and construct objects that are inside the guts of 1 and pass them arguments that are stored as variables in 2. If I restart my session for 2 I lose these variables (iPython does mitigate the pain here somewhat). Hence, I reload() modules into 2 when they are changed.

Since I work with files small enough to reload by rerunning, I have no
use for reload().

I use ipdb a lot in 2. I usually don't feel comfortable with virgin code or a debug fix that hasn't been stepped through with the debugger.

I trust tests, formal and informal, and use occasional print to debug.

Is there something wrong with this work flow? I understand most python experts avoid reload(). So what are they doing that I'm not? I love the ability of Python to quickly let you dive deep into your code and set up a difficult case with 2, it's hard to imagine giving this up, and it's hard to imagine using it without reload().

Since 'experts do not use Idle', I must not be one ;-). I will let the
real 'experts' explain what they do.

 

[Westley Martínez]

This is an idea brought over from another post.



When I write Python code I generally have 2 or 3 windows open simultaneously.



1) An editor for the actual code.

2) The interactive interpreter.

3) An editor for the unit tests. (Sometimes skipped for quick one-off scripts)



My work flow tends to involve using 2 to debug the issues that come up with 1 and 3. I'll write some new code in 1, play around with it in 2, then solidify the tests in 3. Or a test in 3 fails and I dig around with it using2.



My problem is that I tend to use reload() quite a bit. I want to call functions and construct objects that are inside the guts of 1 and pass them arguments that are stored as variables in 2. If I restart my session for 2 I lose these variables (iPython does mitigate the pain here somewhat). Hence,I reload() modules into 2 when they are changed.



I use ipdb a lot in 2. I usually don't feel comfortable with virgin code or a debug fix that hasn't been stepped through with the debugger.



Is there something wrong with this work flow? I understand most python experts avoid reload(). So what are they doing that I'm not? I love the ability of Python to quickly let you dive deep into your code and set up a difficult case with 2, it's hard to imagine giving this up, and it's hard to imagine using it without reload().



Thanks for any tips.

In short, there's nothing wrong with any workflow so long as it works.

 

[rusi]

This is an idea brought over from another post.

When I write Python code I generally have 2 or 3 windows open simultaneously.


1) An editor for the actual code.
2) The interactive interpreter.
3) An editor for the unit tests. (Sometimes skipped for quick one-off scripts)

My work flow tends to involve using 2 to debug the issues that come up with 1 and 3. I'll write some new code in 1, play around with it in 2, then solidify the tests in 3. Or a test in 3 fails and I dig around with it using 2.



My problem is that I tend to use reload() quite a bit. I want to call
functions and construct objects that are inside the guts of 1 and pass them
arguments that are stored as variables in 2. If I restart my session for 2 I
lose these variables (iPython does mitigate the pain here somewhat). Hence, I
reload() modules into 2 when they are changed.

I (tend to) use emacs.

Now before this becomes an editor-war, I need to say that its hardly a question of editor and mostly a question of workflow. For example I know some core emacs devs who use vi-mode in emacs and as the joke goes:

Emacs is an excellent operating system; it only lacks a good editor.

About workflow: Its important to get that

1. editor-inside-interpreter -- eg !vi in ipython
2. interpreter-and-editor side-by-side -- your workflow
3. interpreter-inside-editor -- natural for emacs and IDLE (I think)
are significantly different

And one important difference is that for 3. the burden of reload goes from python to the editor -- one hotkey and the buffer is reloaded into the interpreter.

 

[Peter Cacioppi]

This is an idea brought over from another post.



When I write Python code I generally have 2 or 3 windows open simultaneously.



1) An editor for the actual code.

2) The interactive interpreter.

3) An editor for the unit tests. (Sometimes skipped for quick one-off scripts)



My work flow tends to involve using 2 to debug the issues that come up with 1 and 3. I'll write some new code in 1, play around with it in 2, then solidify the tests in 3. Or a test in 3 fails and I dig around with it using2.



My problem is that I tend to use reload() quite a bit. I want to call functions and construct objects that are inside the guts of 1 and pass them arguments that are stored as variables in 2. If I restart my session for 2 I lose these variables (iPython does mitigate the pain here somewhat). Hence,I reload() modules into 2 when they are changed.



I use ipdb a lot in 2. I usually don't feel comfortable with virgin code or a debug fix that hasn't been stepped through with the debugger.



Is there something wrong with this work flow? I understand most python experts avoid reload(). So what are they doing that I'm not? I love the ability of Python to quickly let you dive deep into your code and set up a difficult case with 2, it's hard to imagine giving this up, and it's hard to imagine using it without reload().



Thanks for any tips.

Thanks for the tips guys.

One thing re: editors and interactive environments. I'm not a huge emacs fan (ducking) and I really like iPython.

In order to avoid calling reload, I've changed my workflow to the following..

1) The "real" code
2) The unit tests
3) A scratch script

Then, I launch iPython, which can intellisense launch 3 easily. Then I makewhatever changes I need to 1-3 to make a baby step forward, close iPython,and repeat.

This seems to preserve the easy-to-dive-into, play-around-with-it fun that is Python without forcing me to call reload(). I suspect I'm now a bit moreclose to what the "experts" are doing.

Thanks again, I love kibbutzing about work-flow, toys, etc.

 

[rusi]

One thing re: editors and interactive environments. I'm not a huge emacs fan
(ducking) and I really like iPython.

Heh! Yeah we are an endangered species
G enerally
N ot
U sed
E ditor for
M iddle
A ged
C omputer
S cientists

In order to avoid calling reload, I've changed my workflow to the following.


1) The "real" code
2) The unit tests
3) A scratch script



Then, I launch iPython, which can intellisense launch 3 easily. Then I make
whatever changes I need to 1-3 to make a baby step forward, close iPython,
and repeat.

Hardly looks very ergonomic to me

 

[Oscar Benjamin]

Hardly looks very ergonomic to me

I'm not quite sure what's meant by intellisense here but using ipython
with a scratch script works very well for me.

Here's how a typical session works for writing a sqrt_floor() function
that works for all positive integers:

In [1]: import math

In [2]: math.sqrt(16)
Out[2]: 4.0

In [3]: math.sqrt(15)
Out[3]: 3.872983346207417

In [4]: int(math.sqrt(15))
Out[4]: 3

In [5]: x = 10 ** 20 + 1

In [6]: int(math.sqrt(x ** 2)) == x
Out[6]: False

In [7]: edit tmp.py

[At this point vim opens, I do some working on paper and write the
following in vim:

import math

def sqrt_floor(y):
x = int(y)
while x ** 2 <= y < (x+1) ** 2:
x = (x + y // x) // 2
return x
]

done. Executing edited code...

In [8]: sqrt_floor(100)
Out[8]: 100

In [9]: sqrt_floor(101)
Out[9]: 101

[Okay whoops! that's not what I wanted... Need to invert the condition
in the while loop!]

In [10]: edit tmp.py
done. Executing edited code...

In [11]: edit tmp.py
done. Executing edited code...

In [12]: sqrt_floor(101)
Out[12]: 10

In [13]: sqrt_floor(100)
Out[13]: 10

In [14]: x = 10 ** 20 + 1

In [15]: sqrt_floor(x ** 2) == x
Out[15]: True

[Okay that's better]

In [16]: edit tmp.py

[Now I'll test against a load of integers so the file looks like

import math

def sqrt_floor(y):
x = int(y)
while not (x ** 2 <= y < (x+1) ** 2):
x = (x + y // x) // 2
return x


for y in range(10 ** 6):
x = sqrt_floor(y)
assert x ** 2 <= y < (x+1) ** 2
]
done. Executing edited code...

[All tests have passed but it took a little while so let's time this function]

In [17]: timeit sqrt(10 ** 10)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<snip traceback>
NameError: global name 'sqrt' is not defined

In [18]: from math import sqrt

In [19]: timeit sqrt(10 ** 10)
1000000 loops, best of 3: 393 ns per loop

In [20]: timeit sqrt_floor(10 ** 10)
10000 loops, best of 3: 52.4 us per loop

In [21]: edit tmp.py

[Since sqrt is 100x faster we can use it internally to get an initial guess..
Now the file looks like:

import math

def sqrt_floor(y):
x = int(math.sqrt(y))
while not (x ** 2 <= y < (x+1) ** 2):
x = (x + y // x) // 2
return x


for y in range(10 ** 6):
x = sqrt_floor(y)
assert x ** 2 <= y < (x+1) ** 2
]
done. Executing edited code...

In [22]: timeit sqrt_floor(10 ** 10)
100000 loops, best of 3: 4.58 us per loop

[Great. That's 10x faster than before and probably as good as it gets
for speed but this optimisation has problems...]

In [23]: sqrt_floor(10 ** 1000)
---------------------------------------------------------------------------
OverflowError Traceback (most recent call last)
<ipython-input-23-378bf805089d> in <module>()
----> 1 sqrt_floor(10 ** 1000)

Q:\tmp.py in sqrt_floor(y)
2
3 def sqrt_floor(y):
----> 4 x = int(math.sqrt(y))
5 while not (x ** 2 <= y < (x+1) ** 2):
6 x = (x + y // x) // 2

OverflowError: long int too large to convert to float

In [24]: edit tmp.py

[Now the file looks like

import math

def sqrt_floor(y):
try:
x = int(math.sqrt(y))
except OverflowError:
x = y
while not (x ** 2 <= y < (x+1) ** 2):
x = (x + y // x) // 2
return x


for y in range(10 ** 6):
x = sqrt_floor(y)
assert x ** 2 <= y < (x+1) ** 2
]

done. Executing edited code...
[ This reruns the tests in the file; no AssertionError so all good ]

In [25]: sqrt_floor(10 ** 1000)
Out[25]: 100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000L

In [26]: timeit sqrt(10 ** 10)
1000000 loops, best of 3: 381 ns per loop

And now we're basically done. Recheck the proof, think about how to
handle negative numbers, write a docstring and move on.


Oscar

 

[Peter Cacioppi]

This is an idea brought over from another post.



When I write Python code I generally have 2 or 3 windows open simultaneously.



1) An editor for the actual code.

2) The interactive interpreter.

3) An editor for the unit tests. (Sometimes skipped for quick one-off scripts)



My work flow tends to involve using 2 to debug the issues that come up with 1 and 3. I'll write some new code in 1, play around with it in 2, then solidify the tests in 3. Or a test in 3 fails and I dig around with it using2.



My problem is that I tend to use reload() quite a bit. I want to call functions and construct objects that are inside the guts of 1 and pass them arguments that are stored as variables in 2. If I restart my session for 2 I lose these variables (iPython does mitigate the pain here somewhat). Hence,I reload() modules into 2 when they are changed.



I use ipdb a lot in 2. I usually don't feel comfortable with virgin code or a debug fix that hasn't been stepped through with the debugger.



Is there something wrong with this work flow? I understand most python experts avoid reload(). So what are they doing that I'm not? I love the ability of Python to quickly let you dive deep into your code and set up a difficult case with 2, it's hard to imagine giving this up, and it's hard to imagine using it without reload().



Thanks for any tips.

So when I say I exploit the intellisense of iPython, it's simply this.

I have a "working on X" script that has some handy variables and runs whatever is currently of interest in the functional code or the units tests. It even launches into the ipdb debugger, if needed.

The "working on X" is kept in the scripts directory of iPython. The older "working on Y" scripts are renamed to "not working on Y", so that there is only one script in that particular directory that starts with "wo".

When iPython launches, I just type run wo [tab] and it completes the file name. So the whole "proxy for reload" process is actually pretty quick. For me it's faster than what I used to do with reload(), plus the "working on" script keeps a nice record of where I was in a debugging/experiment process.. iPython makes it easy to create history dumps which can then be selectively copied over to the "working on" if I forget what was useful.

At any rate, it sounds like emacs plays very well with python and iPython, not disparaging the utility of emacs kung fu.

But if you want to use a different editor (I use idle for editing), and also use iPython for interactivity (which is very slick), this workflow works nice for me and might be helpful to you.

Cheers