a defense of ad hoc software development

Discussion in 'Perl Misc' started by ccc31807, Jan 14, 2010.

  1. ccc31807

    ccc31807 Guest

    On page 229 of Paul Graham's 'ANSI Common Lisp' he writes:

    If programming were an entirely mechanical process -- a matter of
    simply translating specifications into code -- it would be reasonable
    to do everything in a single step. But programming is never like that.
    No matter how precise the specifications, programming always involves
    a certain amount of exploration -- usually a lot more than anyone had
    It might seem that if the specifications wee good, programming
    would simply be a matter of translating them into code. This is a
    widespread misconception. Programming necessarily involves
    exploration, because specifications are necessarily vague. If they
    weren't vague, they wouldn't be specifications.
    In other fields, it may be desirable for specifications to be as
    precise as possible. If you're asking for a piece of metal to be cut
    into a certain shape, it's probably best to say exactly what you want.
    But this rule does not extend to software, because programs and
    specifications are made out of the same thing: text. You can't write
    specifications that say exactly what you want. If the specification
    were that precise, then they would be the program.

    In a footnote, Graham writes:

    The difference between specifications and programs is a
    difference in degree, not a difference in kind. Once we realize this,
    it seems strange to require that one write specifications for a
    program before beginning to implement it. If the program has to be
    written in a low-level language, then it would be reasonable to
    require that it be described in high-level terms first. But as the
    programming language becomes more abstract, the need for
    specifications begins to evaporate. Or rather, the implementation and
    the specifications can become the same thing.
    If the high-level language is going to be re-implemented in a
    lower-level language, it starts to look even more like specifications.
    What Section 13l.7 is saying, in other words, is that the
    specifications for C programs could be written in Lisp.

    In my SwE classes, we spent a lot of time looking at process and
    processes, including MIL STD 498 and IEEE Std 830-1984, etc. My
    professors were both ex-military, one who spent his career in the USAF
    developing software and writing Ada, and both were firmly in the
    'heavy' camp (as opposed to the lightweight/agile camp).

    In my own job, which requires writing software for lay users, all I
    ever get is abbreviated English language requests, and I have learned
    better than to ask users for their requirements because they have no
    idea what requirements are. (As a joke, I have sent a couple a copy of
    IEEE Std 830-1984 and told them that I needed something like that, but
    the joke went over like a lead balloon -- not funny at all.) Of
    necessity I have learned to expect to write a number of versions of
    the same script before the user accepts it.

    I understand that Graham isn't talking about requirements, and to many
    people specifications and requirements amount to the same thing. I
    also understand the necessity for planning. However, the Graham quote
    seems to me a reasonable articulation for ad hoc development. (It's
    something I wanted to say to jue in particular but couldn't find the


    ccc31807, Jan 14, 2010
    1. Advertisements

  2. ccc31807

    Marc Girod Guest

    Paul Graham is a genious. He speaks Gold.
    I loved his book /On LISP/.
    Planning is useful, but plans are worthless.

    Marc Girod, Jan 14, 2010
    1. Advertisements

  3. ccc31807 wrote:
    Yeah, Paul Graham is a total b.s. artist.
    Besides a prolific author of internet screeds
    he wrote some successful software a long time ago.
    Oh, and I do not deny he is a lisp expert.
    But, yeah, PG is full of shit and his bazillion
    bullshit essays all have the same annoying
    common theme "I'm the best! Be like me!".
    Richard McBeef, Jan 14, 2010
  4. ccc31807

    fortunatus Guest

    Nobody expects non-programmers to be able to generate programmers's

    Programmers generate requirements for themselves based on less formal
    Q&A with the customer.

    No matter what, a programmer works to specific requirements that are
    supposed to match the user's expectations and/or needs. The
    requirements might be in the programmer's head, and they might be
    developed by evolutionary process throughout the code writing, but
    they exist. If there is more than one programmer on a team then there
    is a need to codify requirements into documents to some degree.

    I think it is not feasible to say that requriements can be so detailed
    as to be directly executable. In teamwork it is useful to communicate
    when there remains a large degree of ambiguity, and allow team members
    to contribute to resolutions.

    So you can use Lisp to capture/express requirements, but it won't be a
    complete solution to the problem before you break the work up among
    team members. Yet at that stage it would normally be named a
    requirements document.

    My point is that the work needs to be broken up and distributed among
    team members before ambiguity is resolved. At that point there should
    be some requirements written down for team members to start working
    from. Therefore the requirements, at that stage, even if captured in
    Lisp or other executable format, /must/ have too many holes to really
    solve the whole problem, or else the team is not dividing the workload
    fortunatus, Jan 14, 2010
  5. No, he's pretty good and a very nice guy, too. No b.s., unless by b.s.
    you mean anything you disagree with. That is my definition, too!

    Kenneth Tilton, Jan 15, 2010
  6. That may be true, but what he says in this particular article is
    correct. Read the book "Coders at Work", and you'll learn that most of
    the great programmers think similarly. Programming simply cannot be
    mechanized. Writing specifications precise enough to be validated
    mechanically *is* programming.
    Barry Margolin, Jan 15, 2010
  7. Pascal J. Bourguignon, Jan 15, 2010
  8. ccc31807

    Marc Girod Guest

    Unfortunately, many people do.
    This is bullshit from the start and
    results in more bullshit. How this
    bullshit is dealt with is an important
    part of everyday life nowadays.
    Formality is another interesting way
    to glide astray from reality.
    Formality doesn't bind to precision,
    or to correctness.

    Formal semantics are a myth, precisely
    one which fights the reality of code
    What makes code semantics special, and
    thus more interesting, is that code is
    meant to be run, and not only to be read.

    That is, contrarily to any other kind
    of representation, code is not semantics
    about something else: it is embodied,
    operational, semantics.

    Now, the problem is one of communications
    (whether Q&A with customers or else.)
    Until there is a program, their existence
    doesn't imply 'essence', i.e may not be
    consistent of even feasible (no proof of
    And even then.
    Ambiguities get resolved while merging the
    contributions (aka integration).

    Marc Girod, Jan 15, 2010
  9. but it isn't unreasonable to expect them to be able to articulate
    their own requirements. Do they want a word processor or an ABS?
    various people have though (wrongly I think). The TDD people sometimes
    claim the test is an executable requirement.
    no. requirements are a communication with an end user. Only the most
    sophisticated of end users could be epected to read lisp.
    Nick Keighley, Jan 15, 2010
  10. ccc31807

    Tamas K Papp Guest

    Reasonableness is not the right concept here.

    There is a market for programmers, and some employers/customers want
    programmers who can write software without being given super-specific
    formal requirements. Programmers who can cater to this market by
    making an effort to understand the problem domain may enjoy an
    advantage. The market will decide. If you don't like the market
    outcome, you can complain that it is "unreasonable", but no one will

    Tamas K Papp, Jan 15, 2010
  11. ccc31807

    ccc31807 Guest

    I don't know the difference between 'requirements' and 'specification'
    but I assume that the requirements state what the software will do,
    and the specification states how the software will do it.

    People like to poke fun at UML use cases and use case diagrams, me
    included. However, use cases can describe in the language of the
    problem domain the functional requirements (if not the non-functional
    requirements as well), e.g.
    - customer presents the goods to the cashier
    - cashier determines the price of the goods
    - cashier calculates the total price including tax
    - cashier informs customer of the total price
    - customer tenders the price in cash, check, or plastic
    - cashier accepts customer's tender
    - cashier gives customer a receipt for the goods.
    It may be more accurate to say that the programmer generates the
    design based on (1) functional requirements furnished by the client,
    and (2) an explicit or tacit specification based on those
    I've been taken to task (on c.l.p.m.) for saying that many times I
    just start writing code without doing any analysis or design.
    Obviously, for a larger project, analysis and design is essential. I
    think that Graham's point is that analysis, design, and implementation
    can be integrated with more efficiency and less work. Obviously, you
    can't build a sizable application this way, but just as obviously any
    sizable application necessarily contains problems that can only be
    solved by experimentation.

    ccc31807, Jan 15, 2010
  12. ccc31807

    Tim Bradshaw Guest

    I was going to say "and there are these programs called compilers which
    turn them into executable programs" but that's not quite right. A
    specification could just give a way of testing whether something meets
    it, without giving a way for that thing to be constructed, while a
    program has to give a way to construct that thing. So perhaps a
    specification is not programming: it's just something which is as hard
    as programming.
    Tim Bradshaw, Jan 15, 2010
  13. ccc31807

    MarkHaniford Guest

    Programming is already mechanized. The problem is with having
    "business analysts" spending 10x as much time "writing out the specs"
    to be mechanized than a "normal" programmer. Or on another note, your
    CFO could spend hours a day cleaning toilets at the office too.
    MarkHaniford, Jan 15, 2010
  14. ccc31807

    Tim X Guest

    Analysis is critical because that is how, you as the programmer, get to
    understand the domain problem you will be working in. Design is also
    important, but it should not be carved in stone.

    I think PGs main point is that it is often during the implementation
    that you gain that deeper understanding of the problem that provides the
    insight for better solutions and you want to avoid any specification
    that is so detailed that you cannot adopt those better solutions because
    they would not fit with the specification.

    I've worked on projects in the past where the specification was so
    detailed, it included all the names of procedures/functions, the
    arguments they would accept and the values they would return. As a
    developer on that project, all you were expected to do was write those
    functions and procedures and at the end, it would all fall together. It
    was an uninspiring, horrible boring project and it was pretty much a

    I think what PG is talking about is close to what the Agile (God I hate
    that term - bloody weasel word. Lets face it, who the hell doesn't want
    to be 'agile'!) camp has been proposing. Use user stories to get an
    understanding of what is required and then use test driven development
    to not only verify the code and protect against regressions after
    refactoring, but also to verify you as the programmer understand the
    customer/owner's requirements. This is why tests should be developed in
    conjunction with the product owner or representatitve/expert from the
    client side.

    Requirements and specifications are all about communication. They are
    there to ensure that everyone knows what the widget is and what the
    widget needs to be able to do. The old waterfall methodology tried to
    make it like dam and bridge building - eveyrthing would be specified
    down to the last bolt and nail. Management would know exactly how much
    it would cost and exactly how long it would take to build (at least,
    they felt reassured they did). This approach
    essentially failed. It failed because it was unable to take into account
    that programming isn't like dam and bridge building. We don't have the
    hundreds of years experience or the relatively well understood laws,
    such as the laws of physics, that engineering in the physical world has.
    Programming has more art than science. The solution often comes with
    understanding and imagination, which is acquired as you attempt to solve
    the problem and will seldom become obvious through analysis alone.

    A major point that PG was trying to make is that your development
    environment can assist in this process by making it easier to experiment
    and explore the problem domain. He argues that lisp is one of the best
    languages for doing this and it is why he believes a single lisp
    developer can be more productive than, lets say, a developer using C.
    This theme is also carried through in his other writings. For example,
    he argues that the success of viaweb was because it used lisp and they
    were able to add features faster than their competition. In fact, he
    says that their competition often thought he had spies in their team
    because they would release a new feature and viaweb would have
    incorporated it by the end of that day.

    Tim X, Jan 16, 2010
  15. ccc31807

    MarkHaniford Guest

    Tim, I might reply to your response in detail later on, but I think
    you made some good points. "Agile" has become a religion of its own,
    and the whole mission of sitting down with your "customers" and
    rapidly banging out mockups has been lost. It's the rapid-feedback
    that gave death to waterfall (communication).

    Not responding to you Tim, but on another note, nobody in their right
    mind would hire anybody that writes Paul Graham type code. It's the
    classic, "what happens when Billy gets hit by a bus".
    MarkHaniford, Jan 16, 2010
  16. There is one exception where very little experimentation may be
    needed, namely implementing formal protocol specifications, such as
    RFC822. It would be reasonable for a high-level API to be written
    directly from the specs to implement each little piece of RFC822 as
    a callable routine. Thus there might be a function/method to create
    an empty RFC822 object, another for each type of header line to add
    to the object, another to verify that all the required lines are
    present, and then one to pass the completed object to some
    receiver. Likewise there might be a parser for RFC822 which returns
    an object showing each of the fields that was present, and then
    accessors for each field separately.

    The rest of the application would be as you quote, experimental
    development, but this one module which does the RFC822 API
    implementation would be from the specs rather than very
    I'm less familiar with military specifications, but I would guess
    these are like InterNet specs, where you might code an API module
    for each set of specs, coding directly from the specs, then write
    the rest of the application in a more experimental style.
    It is sometimes desireable to quickly write non-functional UI which
    shows the user how it "looks", and get their approval of that,
    before starting the real code under the UI.
    Robert Maas, http://tinyurl.com/uh3t, Jan 16, 2010
  17. If by "programmer" you mean "somebody who is capable of designing
    new software and implementing them", and if by "market" you mean
    somebody wiling to hire, I don't believe you. I'm a "programmer"
    per that defintion, and I challenge you to find even one person or
    company willing to hire for what I do.
    I would prefer to brainstorm with the employer to work out our
    common idea of what they want that I believe is
    possible/reasonable to implement.
    How deeply must I understand the problem domain?? Can you give an
    example of a currently open requirement, the problem domain it
    involves, and some idea how deep I would need to understand it
    before I even talk to the employer, and how much the employer can
    explain to me to bring me up to the needed level of understanding?
    Robert Maas, http://tinyurl.com/uh3t, Jan 16, 2010
  18. Are you saying that software designer and customer/employer should
    talk to the point where they think they agree what needs to be
    implemented, then software designer quickly makes a mockup, and
    shows it to the customer/employer to "try" to see whether it's what
    c/e wanted or not, and if not then another round of mockup, until
    c/e is satisfied, then do same for next requirment, etc. until the
    whole application is mocked-up to the satisfaction of the c/e, then
    the s.d. can integrate everything into an application? If so, I
    agree, and I just wish I could find some c/e to hire me so that I
    can try it with a real live c/e for the first time ever.

    Note that for Web-server applications, PHP may be the best language
    for rapid prototyping/mockingup, better even then Lisp because it
    better integrates the HTML with the algorithms, with not much loss
    of programming power compared to Lisp. Thus the time saved in not
    needing to do any work integrating HTML with algorithms more than
    makes up for the time spent figuring out how to do an algorithm in
    PHP which was trivial in Lisp. On the other hand, for any really
    serious new algorithms, I'd prefer to develop in Lisp, and not even
    bother to interface to HTML until after the algorithm is working as
    wanted by the c/e.
    Robert Maas, http://tinyurl.com/uh3t, Jan 16, 2010
  19. In theory. But in practice, you still need to bump against other
    implementations to learn how to treat the cases that are not covered
    by the protocol. There are RFC that are specifically describing the
    "best practices", which means that there are a lot of implementations
    implementing "worst practices" or "good-enough practices".

    It also depend on the specific protocol (and its specification) some
    protocol specifications may lead to more variability and more "fun".
    Pascal J. Bourguignon, Jan 16, 2010
  20. ccc31807

    Tamas K Papp Guest

    That's your problem. Clearly there are employed programmers. And you
    are a very special case, so your job market experience is not indicative
    of anything.
    Whatever. I am not a job search agency. (BTW, these "I challenge you"-
    type utterances must be a clear winner in job interviews).
    These things are project specific, and are usually decided between you
    and your employer/customer. Eg if you are writing a library for
    numerical applications, most employers would prefer that you have a
    clear understanding of floating-point arithmetic (eg on the level of
    "What Every Scientist Should Know ...").

    But the more you know, the less your customer will have to explain.
    And the flipside is that if you don't know anything, sometimes
    introducing you to the problem domain is prohibitively costly, so
    people become their own programmers (which happens in science very

    Tamas K Papp, Jan 16, 2010
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.