|Thus Spake Dave Brueck On the now historical date of Wed, 31 Dec 2003
09:47:31 -0700|
From what I've seen the market is definitely growing, but if you're
hunting job
listings for an all-Python job the pickings are pretty slim.
*sigh* I know. I didn't ask my question here without first searching
around to see what was being asked for. It was not encouraging.
The easiest
way to use Python in your job will probably be if you introduce it to
the company yourself and then use it as the opportunities arise, but not
try to force it - if it's not the right tool for a particular job then
using it anyway will hurt its acceptance in cases when it _is_ the right
tool.
An easy entry point is on small tasks that don't _appear_ to be that
important, such as automating your build/release process, writing test
harnesses, and any other internal dev/QA utilities. From there it's an
easy migration to tools used in production but aren't outward (customer)
facing, such as tools that process log files and load them into the
database or generate reports, or monitoring scripts for production
systems - behind-the-scenes stuff. Write a few utilities to make the
lives of the IT / Operations staff easier, and they'll defend Python to
the death.
I have a funny story about this type of python advocacy. Forgive me if I
bellyache for a while. I think this is the first time I can describe this
situation to people who can understand where I'm coming from. I'm sure a
few of you have gone through this too.
I picked up python while at my last development job. We processed medical
insurance claims, which come in a scadzillion different formats. The one
thing that all these formats have in common is that they're designed to be
mainframe friendly. That means fixed width fields in lines of uniform
length. The basic process was to get the file, archive it, do some basic
validity checking and log some information to a database before
concatenating everything into a big file for ftp'ing to the mainframe. It
was a batch process, so essentially a set of daemons watched a directory
for new files, grabbed them, performed some transformation, then dumped it
to another directory for the next daemon to process. I was sold on Python
the moment I learned how to slice a string.
I started simple.
"Hey guys, you all used to be cobol programmers. I found this new
language that's really easy to learn and can handle records almost like
cobol does, and it will talk to our database!"
"That's nice Sam, but we're a C++ shop." (Side Note: The only object in
our code was a crude dictionary class I slipped into the all C code base a
few months after starting work there. My entire team was baffled by the
linked list inside of it, but eventually they just accepted that it did
what they wanted it to do.)
The only person I can get to listen to me is my old boss, Jason, the team
leader of the computer operators. He's voraciously curious about all
things computer-related and we had a ball exploring the language. We
start scripting all kinds of things, big and small. I don't think I've
ever had so much fun learning a language.
So, in an effort just to learn python and apply it to a domain I know
well, I develop some classes that allow me to parse our files by almost
directly transliterating the file definition formats into some simple
calls. Essentially the classes allowed you to retrieve a byte range in a
string via a name, and defined legal variations of those slices. In a
nutshell, I implemented cobol data structures in Python. This took me a
grand total of a week. The next week I implemented a set of tools that
could identify what format a particular file was if I had already
transcribed the structure. The week after, I had the class able to build
the standard format reference sheet documentation and a complete set of
regression files for error checking. On the fourth week, I implemented a
simple class to pull data out of our logging database and check that it\0
matched a file that was supposed to have been processed. I show this to
Jason, and we both declare "There is a god, and his name is Guido,
pronounced with a 'Ch.'" We figured that all we'd have to do is wait for
the something to go wrong, use the new tools to diagnose the problem
faster than we ever could before, and then when asked how it was solved so
fast, they'll be eager to see our new toy.
Well, that opportunity came several times. Each time noone had time to
look at any of the scripts I had written. Each time I was told never to
use any of it in a production system. Nevermind that I wasn't trying to
put python code into production. I was just using it to identify where
our existing code was failing. It went on this way for about a month and
a half.
Meanwhile, several of the mainframe cobol programmers, including their
team lead, said that they want to learn some PC programming. I love to
teach, especially programming, so I thought "Excellent! I've got some
tools that work in a paradigm they already know and we can ease them into
the PC world with it." We all start meeting once a week to talk about
python. They loved it. All of them agreed that this is the first PC
programming experience they'd had that made sense. (I also learned a
*lot* about mainframes from that.)
Then an odd opportunity came up. We had just been awarded a big new
contract. We were taking it over from a company that had very different
ideas than us on how to process claims, so there were about a hundred or
so new formats to process. Each one had to be redocumented and a
regression suite developed for it. About a week into the affair, I go
visit Charles, our lead QA man, and he remarks that it's horribly tedious
documenting and editing each of the test files by hand. That's when I
find out that they've been building their regression suites by hand with a
hex editor for years. It dawns on me. I've developed exactly the sort of
tool these guys need. I tell Charles that I built a tool that might be
able to help him, but he'll have to help me test it and make sure it does
what he wants it to do. This was a bald-faced lie. I knew it did exactly
what he wanted it to do. I had tested it against the files he sent me for
phase 1 QA. I just wanted an excuse to get him to browse the code with
me. That night, we stayed late to hash things out. The first thing I did
is open a format definition module. His reaction: "That looks almost
exactly like we document it! If I had that, I'll I'd have to do is cut
and paste parts of it into an excel." So, I double click the icon labeled
"documenter" It prompts me for a format name, I type one in. An excel
icon appears on my desktop. I double-click it. Charles stutters:
"D..did it just make that? From the synopsis? That takes me an hour to
type up!" (the beauty of csv's) Charles and I spent a couple of weeks
double and triple checking that the regression suite is exactly as it
should be. We didn't change a single lick of the code. The QA people
still had to verify that the regression sweet matched the expected output,
but now they didn't have to edit it in hex, and they only had to write the
documents once to have it exported to the other formats they wanted. Two
weeks later, QA people started to show up at our python meetings.
Somewhere between six months and a year passed. It seemed that I was able
to get everyone but the people who would most benefit to look at python.
For reasons completely unrelated to the job, I had to move. They replaced
me with a perl programmer (who's apparently a pretty good coder.) In the
absence of my close working relationship with Charles, the QA guy, it was
decided that they needed a liason between the pc team and the QA team.
Jason, the first person to take an interest in my python craze, was
promoted into that position. My contacts there tell me that they're in
the process of switching to being a python shop. Even the perl guy
converted.
It seems that every time my old team floundered with a problem, someone
would mention that I had written a program to check for errors of type
Foo. They'd dig it up. I guess what I did wrong in being a python
advocate was thatI didn't stop being right. They didn't notice the
difference those tools made until I wasn't around to use them. I think
python might possibly become *the* killer app for that company. It
handles everything they do so naturally, especially once I got that
library there to abstract the data. *sigh* I just wish I had been around
to enjoy things running smoothly. I don't think I've ever felt like a
language let me work with a data format so naturally.
Thank you for putting up with that very loquacious rant. It's been pent
up a long time and I feel better I feel better thinking that someone,
somewhere probably understands how I feel.
You know. I never thought about this before. Maybe I could find a niche
reformatting old cobol records and systems. Is there any demand for that
sort of thing? While a mainframe can be an absolutely wonderful tool if
it's the type of thing you require (nothing beats the data integrity of a
well run mainframe) they're expensive. I imagine that a lot of companies
are looking to cut the expense of maintaining big iron. I can't say that
I understand mainframes and cobol, but I do understand how they like to
talk about data. I spent a long time making sure a mainframe got what it
wanted.
Another entry point is through web development, e.g. using Zope or
Webware or Quixote first for an intranet, and later for actual
production websites. In all cases it seems to be a matter of the new
technology getting a foothold, an opportunity where it can prove its
usefulness. Having a meeting and deciding e.g. "will we use Java or
Python?" is IMO the worst possible route: you can argue forever on
theoretical problems that may arise, whereas most critics are silenced
when faced with a real, working, and reliable example that's already in
use.
I am definitely looking at web development. That's the other thing I did
at that job. I wrote cgi in c to get our database to talk to the web.
While developing cgi from scratch in c is not an experience I'd like to
repeat, I enjoyed developing it. It was nice to build a system from the
ground up and grin to myself about the uptime.
If you're introducting Python (or any new technology, for that matter),
it seems to be easiest in really big companies or really small companies
(YMMV as I'm basing this on only 3 or 4 data points for which I've had
first-hand experience
) where the initial number of people affected
is small. In a big company development is likely divided into teams for
different areas of functionality or different products, so using Python
locally isn't likely to ruffle many feathers. In a medium-sized company
I've seen a single product with sub-teams, and higher resistance to do
anything different (sadly, at one such company there was enough
agreement to prototype a new system in Python, but even though the
prototype was robust, fast, and _fully-functioning_, too many developers
refused to see it as anything but a prototype, and insisted on
reimplementing it in Java. The result was slow, flaky, and was never
completed. Grrr.).
I think people tend to over-estimate the the value of compile-time type
checking. That's a big minus in the heads of a lot of people. I know it
was one of the hardest things for me to get over. Then I realized that I
can check the type and structure of a data-item in Python any time I
please as often as I please. I can make it as stringent as I like rather
than convincing a compiler that I want less strict typing. Of course, I
may be wrong. I haven't really coded anything where interactivity is an
issue. The fact that python was not a big name language also made a
difference in it's consideration for the company I used to work for.
Turnover was high on my team, and they were always wondering "who are we
going to get to replace you when you're gone." I don't blame them. I'm
very glad they picked up someone with enough culture to have learned perl.
I think he came into the situation with a lot of good mental models for
the type of processing they do. I understand that he's the one
maintaining my cgi scripts and database, and probably does a good job of
it.
The most success I've had with Python is in a small company where there
is _way_ too much work to get done. Here there are fewer people to cry
foul when you do something different, a much greater emphasis on results
(so any tool that boosts productivity is favored), and generally fewer
layers in the decision-making hierarchy. That's the current situation
I'm in, doing server, web, database, and Win32 client development all in
Python (with very occasional work in C/C++). We didn't have a big
meeting to set a mandate that we're an all-Python company, but over time
the projects in Python flourished because the initial versions got done
so quickly, bugs have been few with fixes coming quickly, the cost of
adding new features has been low, and the developers have been able to
take on additional projects. The non-Python projects have gained a
reputation of being slow-moving, complex resource gluts (in terms of
developers needed) and have fallen by the wayside.
I can't count the times I've hashed out a bit of pseudo-code on a napkin,
gotten it home and realized that I almost wrote out the entire program in
python. If I have three weeks to accomplish a task, I would rather spend
one week coding and two weeks testing than two weeks coding and one week
testing. Especially considering that with python, it's easier to develop
a reusable testing suite. Okay. Now I think I've finally crossed the
line where I'm just preaching to the choir.
It's funny to sit in on a sales call because now even our sales guys
"brag" that Python-based technology is part of our competitive advantage
as if it's our secret sauce or something, whereas initially it was more
"Shh, lest people find our we're using some weird scripting language".
I imagine it's funny. It was odd to see who became allied with me and who
didn't. In my naivety, I thought the hard-core code monkeys would be the
first to recognize how much time and energy python could save them.
Instead, it was the normals who saw it. I remember when Charles and I
decided to see if tech-support could make use of any of the scripts. It
saved them a whole step of searching a six-inch thick document for what
might betray one offensive character in a 3 meg file. I showed them how
modify the messages thrown and they even went through and put page numbers
into each error message so that they didn't have to bother with the index.
Though, I think the most wonderful part about doing all this was that,
after having helped all these people in "other" departments, I had an army
of people eager to help me. The few times when I did royally screw up,
people were not only eager to help clean up the mess I made, they had the
time to help. That, if nothing else, is what I learned about life in the
corporate world.
Well, I'll be surprised if anyone has read this far into my rant. Sorry,
I guess I just had a lot of pent-up frustration to get out. And, well.
Ranting is not exactly a novelty on usenet, now is it?
As far as my original question goes.
I'm at a stable place in my life. I'm doing work in the first field I
ever truly loved: teaching math. I have about six months to a year to
prepare for where I'm going from here. I've run linux at home for around
4 years. While I'm pretty comfortable in it, I need to focus on learning
what it takes to run a mission-critical server well. Running your home
'puter on a 1337 05 is a far cry from a maintaining a system for others.
I've also begun poking around the internals of python with the intent of
becoming more competent with it and comfortable with the python-c api. I'd
love to end up doing sysadmin/web dev/python dev work somewhere. I've
also started mapping out where I might be able to help the community
since, *gasp*, I finally have spare time. Any advice how to get there
from here is most welcome.
Sam Walters
