Development tools and practices for Pythonistas

S

snorble

I'm not a Pythonista, but I aspire to be.

My current tools:

Python, gvim, OS file system

My current practices:

When I write a Python app, I have several unorganized scripts in a
directory (usually with several named test1.py, test2.py, etc., from
random ideas I have tested), and maybe a todo.txt file. Then I hack
away, adding features in a semi-random order. Then I get busy with
other things. Maybe one week I spend 20 hours on development. The next
week, no time on development. A few weeks later when I have some time,
I'm excited to get back to making progress, only to find that I have
to spend 30-60 minutes figuring out where I left off. The code is
usually out of sync with todo.txt. I see people who release new
versions and bug fixes, so I sometimes will create a new directory and
continue working from that copy, because it seems like the thing to
do. But if I ever made something worth releasing, and got a request
like, "I have problems with the 2.0 version. Can you send me the old
1.1 version?" I'd be like, "uhhh... let me hunt through my files by
hand and get back to you in a month". I'm thinking I can do a lot
better than this.

I am aware of tools like version control systems, bug trackers, and
things like these, but I'm not really sure if I need them, or how to
use them properly. I think having some organization to all of this
would help me to make more consistent progress, and spend less time
bringing myself up to speed after some time off.

I really like the idea of having a list of features, and tackling
those features one at a time. I read about people who do this, and
each new features gets a new minor version number. It sounds very
organized and clean. But I'm not really sure of the best way to
achieve this. Mainly I think I just need some recommendations to help
create a good mental map of what needs to happen, and mapping jargon
to concepts. Like, "each feature gets its own directory". Or with a
version control tool, I don't know if a feature maps to a branch, or a
commit?

I appreciate any advice or guidance anyone has to offer.
 
R

rusi

I am aware of tools like version control systems, bug trackers, and
things like these, but I'm not really sure if I need them,

You either dont want version control....
But if I ever made something worth releasing, and got a request
like, "I have problems with the 2.0 version. Can you send me the old
1.1 version?" I'd be like, "uhhh... let me hunt through my files by
hand and get back to you in a month". I'm thinking I can do a lot
better than this.

Or you do!

or how to use them properly.

So the best advice would be: Forget python (for a while) and study one
(modern distributed) version control system:
http://en.wikipedia.org/wiki/Distributed_revision_control

From Joel Spolsky's http://joelonsoftware.com/items/2010/03/17.html

With distributed version control, the distributed part is actually not
the most interesting part.
The interesting part is that these systems think in terms of changes,
not in terms of versions.

bug-trackers are a bit of overkill for a solo developer. However...
My current tools:
Python, gvim, OS file system
Hand in hand with a DVCS, you need to add a testing framework.
 
M

Martin P. Hellwig

On 26/04/2011 14:39, snorble wrote:
<cut explanation>
I would strongly advice to get familiar with:
- Lint tools (like PyLint)
- Refactoring
- Source Control Systems (like Mercurial Hg)
- Unit Testing with Code Coverage

Followed by either writing your own toolset that integrates all of the
above or start learning an IDE that has that stuff built-in (my personal
preference is the latter with my current IDE being PyDev (Eclipse)).

Yes you will be less productive for a couple of weeks, but I promise you
that once you know the above you win that time back very shortly, or if
you are as chaotic as me, within a week :).
 
T

Thomas Rachel

Am 26.04.2011 16:39, schrieb snorble:
When I write a Python app, I have several unorganized scripts in a
directory (usually with several named test1.py, test2.py, etc., from
random ideas I have tested), and maybe a todo.txt file. Then I hack
away, adding features in a semi-random order. Then I get busy with
other things. Maybe one week I spend 20 hours on development. The next
week, no time on development. A few weeks later when I have some time,
I'm excited to get back to making progress, only to find that I have
to spend 30-60 minutes figuring out where I left off. The code is
usually out of sync with todo.txt.

That happens...

I see people who release new
versions and bug fixes, so I sometimes will create a new directory and
continue working from that copy, because it seems like the thing to
do. But if I ever made something worth releasing, and got a request
like, "I have problems with the 2.0 version. Can you send me the old
1.1 version?" I'd be like, "uhhh... let me hunt through my files by
hand and get back to you in a month". I'm thinking I can do a lot
better than this.

That is another subject (IMO), and you are right: you can do a very lot
better, using the right tools.

I am aware of tools like version control systems, bug trackers, and
things like these, but I'm not really sure if I need them, or how to
use them properly.

I have been using several VCS for about 5 or 6 years now, and I can only
tell you that, once started using them, you will wonder how you ever got
along without them.

I think having some organization to all of this
would help me to make more consistent progress, and spend less time
bringing myself up to speed after some time off.

I don't see how these tools will help to get up to date the way you
describe it - but all other issues are well coped with using a VCS. I
personally started with cvs (don't do that!), then worked with svn (do
that only if you really need that), then got working with hg
(Mercurial), which is a very good thing.


Say, you have a certain development progress, and you add a new feature.
Then you do all the changes, including increasing the version number.
The changes you just have done are one changeset which you commit,
providing a good commit message. If you have a version which you ship
out, you give it a tag. In this way, you can easily change from 2.0 you
are working on to 1.5 requested by the customer.


Thomas
 
J

Jean-Michel Pichavant

snorble said:
I'm not a Pythonista, but I aspire to be.

My current tools:

Python, gvim, OS file system

My current practices:

When I write a Python app, I have several unorganized scripts in a
directory (usually with several named test1.py, test2.py, etc., from
random ideas I have tested), and maybe a todo.txt file. Then I hack
away, adding features in a semi-random order. Then I get busy with
other things. Maybe one week I spend 20 hours on development. The next
week, no time on development. A few weeks later when I have some time,
I'm excited to get back to making progress, only to find that I have
to spend 30-60 minutes figuring out where I left off. The code is
usually out of sync with todo.txt. I see people who release new
versions and bug fixes, so I sometimes will create a new directory and
continue working from that copy, because it seems like the thing to
do. But if I ever made something worth releasing, and got a request
like, "I have problems with the 2.0 version. Can you send me the old
1.1 version?" I'd be like, "uhhh... let me hunt through my files by
hand and get back to you in a month". I'm thinking I can do a lot
better than this.

I am aware of tools like version control systems, bug trackers, and
things like these, but I'm not really sure if I need them, or how to
use them properly. I think having some organization to all of this
would help me to make more consistent progress, and spend less time
bringing myself up to speed after some time off.

I really like the idea of having a list of features, and tackling
those features one at a time. I read about people who do this, and
each new features gets a new minor version number. It sounds very
organized and clean. But I'm not really sure of the best way to
achieve this. Mainly I think I just need some recommendations to help
create a good mental map of what needs to happen, and mapping jargon
to concepts. Like, "each feature gets its own directory". Or with a
version control tool, I don't know if a feature maps to a branch, or a
commit?

I appreciate any advice or guidance anyone has to offer.
You can have a look at SVN and bugzilla, they are free SCM & bug tracker
applications.
Make sure it's worth the pain though, these tools are not that easy to
administrate (the usage is pretty simple).

JM
 
C

CM

I'm not a Pythonista, but I aspire to be.

My current tools:

Python, gvim, OS file system

My current practices:

When I write a Python app, I have several unorganized scripts in a
directory (usually with several named test1.py, test2.py, etc., from
random ideas I have tested), and maybe a todo.txt file.

First, give your files meaningful names. test1.py, test2.py...no
wonder you have to spend an hour just figuring out where you left
off. Imagine instead if these were modules called
DiskAccessUtilities.py and UserPreferencesPanel.py. You should also
use highly descriptive names in naming classes and functions, too,
with functions being verbs and don't be afraid of long(ish) function
names like AddNewCustomerToDatabase()
Then I hack away, adding features in a semi-random order.

I'd spend an hour with your TODO list and break it into priority
categories, like High, Med, Low, and It Would Be Nice, and then take
them strictly in that order. I'd also date the issues so you have a
sense for how long something has been waiting to be fixed.
Then I get busy with other things. Maybe one week I spend 20 hours > on development. The next week, no time on development. A few
weeks later when I have some time, I'm excited to get back to making
progress, only to find that I have to spend 30-60 minutes figuring out
where I left off.

I would try to not stop in the middle of creating a new feature or
fixing a bug, but try to finish it out before taking a week off. This
way, when you come back, you can just tackle something from the High
Priority stack and not have to figure out where you were.
The code is usually out of sync with todo.txt.

That's just a matter of being disciplined. When I fix a bug, I simply
cut the bug from the TODO part to the DONE part of the .txt file,
nearly every time. It requires no effort compared to actually fixing
the bug, yet it feels satisfying to get that text moved.
would help me to make more consistent progress, and spend less time
bringing myself up to speed after some time off.

Are you documenting your code? That can help (I need to get better
about that as well). Also, are things broken down into modules that
are self-contained? That also can help much. Is the TODO.txt always
open while you are working? It should be. Lastly, keeping some kind
of notebook or even Post-Its or a bulletin board over your desk with
notes as to what's next and where to hunt in your code to get at it
should help. Imagine if you take two weeks off, come back, want to
work on the project, and you find this note on your bulletin board:
"In the CustomersPanel.py module, add support for a wxSearchCtrl
(search bar) that searches the Former_Customers table in the main
database..." Now you are ready to jump right in!
I really like the idea of having a list of features, and tackling
those features one at a time. I read about people who do this, and
each new features gets a new minor version number.

Is that true? I'm under the impression that projects do a bunch of
changes (bug fixes, and new features) and then release a new version
that has a decent amount of changes. I don't think people want tons
of versions of most projects around, but that each release should have
an appreciable amount of good changes.
to concepts. Like, "each feature gets its own directory".

I guess it depends on your project, but that sounds needlessly complex
and way too tough with a VCS. I'd say just don't go there.

Once you use a VCS you will probably settle into a better pattern, but
things like good naming, documenting, notes, prioritizing features/
bugs, and roadmaps don't magically go away. Software development
takes long range thinking and some organizational discipline.

Che
 
C

CM

I guess it depends on your project, but that sounds needlessly complex
and way too tough with a VCS.  I'd say just don't go there.

(Whoops, I meant way too tough *without* a VCS, not with)
 
A

Algis Kabaila

Am 26.04.2011 16:39, schrieb snorble:
I don't see how these tools will help to get up to date the
way you describe it - but all other issues are well coped
with using a VCS. I personally started with cvs (don't do
that!), then worked with svn (do that only if you really
need that), then got working with hg (Mercurial), which is a
very good thing.

Thomas

Thomas, have you tried bzr (Bazaar) and if so do you consider hg
(Mercurial) better?

And why is it better? (bzr is widely used in ubuntu, which is
my favourite distro at present).

TIA,

OldAl.
 
A

Algis Kabaila

(Whoops, I meant way too tough *without* a VCS, not with)

And read your own emails *before* sending them :)

Actually, CM has given some very good advice! As I am probably
the oldest person on this list, so my penny's worth is that some
going over old stuff is good - learning is repetitious and
memory is not going to get better with age, so learn to live
with it (it being the frailty of memory...)

OldAl.
 
C

Chris Angelico

When I write a Python app, I have several unorganized scripts in a
directory (usually with several named test1.py, test2.py, etc., from
random ideas I have tested), and maybe a todo.txt file. ... The code is
usually out of sync with todo.txt. I see people who release new
versions and bug fixes, so I sometimes will create a new directory and
continue working from that copy, because it seems like the thing to
do. But if I ever made something worth releasing, and got a request
like, "I have problems with the 2.0 version. Can you send me the old
1.1 version?"

As other people have said, version control is very handy. I use git
myself, but imho the choice of _which_ VCS you use is far less
important than the choice of _whether_ to use one.

As to the todo file - I tend to keep only vague ideas in a separate
file. Any todo that can be logically associated with a code file or,
especially, a particular piece of code, goes in that source file:

def function(parms):
# TODO: This should check if Foo matches Bar and shortcut the computation
...

I have a very strict format: T, O, D, O, literal ASCII, always
uppercase. Makes it easy to grep (and I try to avoid "todo" in
lower-case, which means I can use a case-insensitive search if I
choose).

Additionally, if there's any task that will require checking of
multiple parts of the code, I'll create a keyword for it. For
instance, if I'm considering adding a local cache to an application to
reduce database traffic, I might do this:

//TODO CACHE: This will need to update the cache
....
//TODO CACHE: Read from cache instead
....
//TODO CACHE: Would this affect the cache?
.... etc

The benefits of having the comments right beside the code cannot be
underestimated. Comments are far less likely to get out of sync if
they stare you in the face while you're changing the code - this is
why doxygen and friends are so useful.

Ultimately, it all comes down to discipline, and how important the
project is to you. At work, I have a lot of disciplines; we have a
wiki where stuff gets documented, we have source control, we have
daily backups (as well), etc, etc, etc. For little home projects, it's
not usually worth the effort. Take your pick, where do you want to go?

Chris Angelico
 
D

Dan Stromberg

You can have a look at SVN and bugzilla, they are free SCM & bug tracker
applications.
Make sure it's worth the pain though, these tools are not that easy to
administrate (the usage is pretty simple).

http://trac.edgewall.org/ is purportedly pretty easy to set up - I've
only used it, not set it up. Trac gives you SVN and an issue tracker.
It has plugins for other source control systems.
 
T

Thomas Rachel

Am 26.04.2011 20:42, schrieb Algis Kabaila:
Thomas, have you tried bzr (Bazaar) and if so do you consider hg
(Mercurial) better?

I have played around with bzr, but afterwards more with hg which gave me
a better beeling (don't know why)...

Thomas
 
A

Algis Kabaila

True enough. But the modern crop of first-tier VCSen –
Bazaar, Git, Mercurial – are the ones to choose from.
Anoyone recommending a VCS tool that has poor merging
support (such as Subversion or, heaven help us, CVS) is
doing the newcomer a disservice.
All golden advice! Two things were not mentioned:

1. Work on an existing project that is based on Python. The
"Launchpad" of ubuntu has a great environment for a project.
There are many existing projects in which to participate. Even
Bazaar's GUI version is programmed in Python with PyQt for the
GUI proper. You will get a lot out of participation in an
existing project, probably a lot more than you input.
Ultimately it is how much you input that is of of most benefit
to you yourself.

2. I failed to see questions and suggestions of platform to work
from. Your experience, interests, and computer platform will
determine what is useful to you and what is less useful.

3. Finally, it is important to have a project about which you
can master enough enthusiasm to persist. "What project "
depends very much on item 2.

May the favourable wind fill your sails,

OldAl.
 
T

Tim Chase

Thomas, have you tried bzr (Bazaar) and if so do you consider hg
(Mercurial) better?

And why is it better? (bzr is widely used in ubuntu, which is
my favourite distro at present).

Each of the main 3 (bzr, hg, git) have advantages and
disadvantages. As Ben (and others?) mentions, it's best to learn
one of these instead of starting with something like Subversion
or worse (CVS or worse, *shudder* MS Visual SourceSafe)


Bazaar (bzr)
============
launchpad.net popular for hosting
Pros:
- some Ubuntu interactions (such as launchpad) easier
- a rigorous focus on correctness
- written in Python (with a small optional bit of C)
- easy-to-use interface (CVS-ish)
- good cross-platform support

Cons:
- was slow, though I understand they've worked on improving this

Protocols:
- custom/smart protocol
- http
- sftp
- ftp
- rsync (via plugin)


Mercurial (hg)
==============
BitBucket is popular for hosting
Pros:
- speedy
- written in Python (with a small optional bit of C)
- easy-to-use interface (CVS-ish)
- fairly compact repositories
- EXCELLENT documentation via online book
- chosen by Python as the repository of choice
- good cross-platform support

Cons:
- no biggies that I've found

Protocols:
- http
- ssh


Git (git)
=========
GitHub is popular for hosting
Pros:
- a *lot* of popular projects use it (Linux kernel)
- fast
- fairly compact repositories
- good documentation (though somewhat scattered)

Cons:
- interface diverges from the "CVS standards"
- (was?) not native on
- repositories require periodic maintenance using git gc
- Win32 support is/was a little clunky
- interface was under tumultuous change for a while (though it
seems to have stabilized now)

Protocols:
- custom/smart protocol
- http
- sftp
- ftp


So that said, I've become a Mercurial user because the interface
was close to SVN which I used previously, and it was speedy on my
older machines. If bzr has come up to comparable speed, I'd be
game to probe it again. I just don't care for git's command-line
UI, but that's a personal preference thing (just like I prefer
vi/vim over emacs, but acknowledge there are lots of smart folks
on the other side, too).

-tkc

For at least hg vs. git, see
http://stackoverflow.com/questions/1598759/git-and-mercurial-compare-and-contrast
 
R

rusi

Each of the main 3 (bzr, hg, git) have advantages and
disadvantages.  As Ben (and others?) mentions, it's best to learn
one of these instead of starting with something like Subversion
or worse (CVS or worse, *shudder* MS Visual SourceSafe)

<pros and cons of bzr, git, mercurial snipped>

The distributed revision control page on wikipedia (bottom)
http://en.wikipedia.org/wiki/Distributed_revision_control
in addition to these, mentions fossil -- something I had not heard of
till now.


Its claims seem to match the OPs lightweight requirements more closely
than any other:

(from above link)
---------------------------
Fossil is cross-platform; its source code compiles on Linux, Mac OS X
and Microsoft Windows. It is not only capable of distributed version
control like Git and Mercurial but also supports distributed bug
tracking, a distributed wiki and a distributed blog mechanism all in a
single integrated package. With its built-in and easy-to-use web
interface, Fossil simplifies project tracking and promotes situational
awareness. A user may simply type "fossil ui" from within any check-
out and Fossil automatically opens the user's web browser in a page
that gives detailed history and status information on that project.
 
A

alex23

rusi said:
What's the facts? Anyone with any experiences on this?

No experience, but I'm rather torn over Fossil. On the one hand, it
feels like NIH writ large; on the other hand, it's a DVCS with Trac-
like features in a standalone executable less than 1MB in size...by
the author of sqlite.
 
G

Gour-Gadadhara Dasa

I'm not a Pythonista, but I aspire to be.

My current tools:

Python, gvim, OS file system

I'm also starting with Python after abandoning idea to use D for our
desktop GUI application.

We plan to use Python + Qt along with Cython extensions and wrapping
external C library.

I'm interested which Python mode can you recommend for Emacs:

a) python.el

b) python-mode.el or

c) 'new' python.el (https://github.com/fgallina/python.el)

considering our needs above and desire to use IPython running
Emacs-23.2 on FreeBSD?


Sincerely,
Gour

--
“In the material world, conceptions of good and bad are
all mental speculations…†(Sri Caitanya Mahaprabhu)

http://atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (FreeBSD)

iQIcBAEBAgAGBQJNt7o5AAoJELhxXPVStcgQIKEQAMhLBrGE9sNxjJ7GFHlgzEZr
cSVFEkqImoiLxs2gjooTGKa+tSdyJ18JtamFsXVBJiSjO2kVFksQPpg8xle919Qc
2wH2tqH2GFYDKx4tntw+Pa4nPWon4FOLWKErbMRJVej7VMxB07ALFuh+GEgp4hUq
LjpaTp8dTWKdvb2zw8jlwktRvkY9nEVWfRAU9E8wbKCQiN3Ztmmv+tGh3auMAFw+
4lc2Z6j6VePD022eXZMlIXTv00zkspwTNznVyxvny0nUEYgyCNui/EmYJi0gIdsv
9gntmd8cygvsWifxwqcq4u6qcx3inVZ05g96KnofFycl7SYD2wdzXNBVk323BvHE
7k20p5+ZFP335ZAB4SOlmkugLjiXgqMkfroFEo2VU7x0/VFvF60sFtowjfg7eghc
AEwGAaEbwEoPwV0R3ZvlqNVi4726WVk8KbFVL+lOZLw7V7XpIDxO+pp9QUdqkZwd
9zoFvdIi1RhZvkgThsX+7HRuWZ2p9smA7iwgKOHO1mb3odgumIdCQOUja7dWN7To
9SvRQW2oP5wMFj/4x3om8fE29kwrJWUrzgTgX7XUgb7AzGUDr3GX9hMNEIbkdaka
zEdxMCFtT9X1g8UWJst/CKCf9iR+S56IlIpkByldguDQ1ctnKXkCh2qSiqyTm6Wj
XA6iIXakdEf4qO0kCW6B
=5NO3
-----END PGP SIGNATURE-----
 
G

Gour-Gadadhara Dasa

Thats what I use.

Upon hearing there is some bug in 23.2 branches with this mode, I've
switched to 'emacs-devel' port and will start with this mode as well.

Thanks.


Sincerely,
Gour

--
“In the material world, conceptions of good and bad are
all mental speculations…†(Sri Caitanya Mahaprabhu)

http://atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (FreeBSD)

iQIcBAEBAgAGBQJNt89hAAoJELhxXPVStcgQjNoP/0q6haNLG422NYYq48VAhz5B
HdCo2+xPwedJ204Ii07IGOgRt2jCpRkT87bVcDtSbW2s0pcsitcXmpgxRfB9rkfV
J2MWUIWFD+1dB865LDP76fYVXWdQXaHhjGkw4cb19nCZYGeyM7Bq++iG9yWZH/Ns
8YyAiAFDaNfuUFHBoQmLwiln3ZQ03kqCKJ4jwsfqbyRy6jBj7VFCiTF5NJ3HCZcy
y1+i1lm3V2+LHvgmCwmC9oXnLcCsPMfpHyQL08CiDcoV05ZfOQuZWY60Sl4UsO7N
kzcVI1++DZJZPBehMhZ2nOlic2kKZ1n+BRe++juT8Dzzf9QrI3mDso9xo7nsFoVY
tVLh18a7YuLapdFb2+PQvKehedQVTas7hmPLiNTUzDQRJntw5c0Jst7bAxcsg/HQ
I1GI7PxaYN/emzC6SwPwgHoXD+kD1Q/Tl5H4Tljm3oYRHCGxP+vtfkxekQvsJxWF
oIvIcK8FNOSNr4CLIgJ5aUScsBap0/FGXDkvsvbxOusoW7vQBEw6LTJxpVXr9+eC
Gums1hjyUWqY07teTDcCsidz4Y3SQObQT/lBciA1kCb2nRBSXmqdne9BLQLqGliJ
KtCXYrcLAvbMtf7ycPPk+8b0XDUSgQ0kEcA7uMS14GEnHz59QNWt0yoNMJPATaz1
pt6HRJUn/0ljyQLsT50f
=vofk
-----END PGP SIGNATURE-----
 
J

Jean-Michel Pichavant

Ben said:
Mercurial – are the ones to choose from. Anoyone recommending a VCS tool
that has poor merging support (such as Subversion or, heaven help us,
CVS) is doing the newcomer a disservice.

True enough. But the modern crop of first-tier VCSen – Bazaar, Git,
For a single user, there would be no merge issue. And svn is very simple
to use.
That would not be a such bad advice for a beginner with VCS systems.

JM
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top