[ANN] git peer-to-peer bittorrent experiment: first milestone reached

Discussion in 'Python' started by Luke Kenneth Casson Leighton, Sep 1, 2010.

  1. http://gitorious.org/python-libbittorrent/pybtlib

    this is to let people know that a first milestone has been reached in
    an experiment to combine git with a file-sharing protocol, thus making
    it possible to use git for truly distributed software development and
    other file-revision-management operations (such as transparently
    turning git-configured ikiwiki and moinmoin wikis into peer-to-peer

    the milestone reached is to transfer git commit "pack objects", as if
    they were ordinary files, over a bittorrent network, and have them
    "unpacked" at the far end. the significance of being able to transfer
    git commit pack objects is that this is the core of the "git fetch"

    the core of this experiment comprises a python-based VFS layer,
    providing alternatives to os.listdir, os.path.exists, open and so on -
    sufficient to make an interesting experiment itself by combining that
    VFS layer with e.g. python-fuse.

    the bittornado library, also available at the above URL, has been
    modified to take a VFS module as an argument to all operations, such
    that it would be conceivable to share maildir mailboxes, mailing list
    archives, .tar.gz archives, .deb and .rpm archives and so on, as if
    they were files and directories within a file-sharing network.

    as the core code has only existed for under three days, and is only
    400 lines long, there are rough edges:

    * all existing commit objects are unpacked at startup time and are
    stored in-memory (!). this is done so as to avoid significant
    modification of the bittorrent library, which will be required.
    * all transferred commit objects are again stored in-memory before
    being unpacked. so, killing the client will lose all transfers
    received up to that point.

    on the roadmap:

    * make things efficient! requires modification of the bittornado library.
    * create some documentation!
    * explore how to make git use this code as a new URI type so that it
    will be possible to just do "git pull"
    * explore how to use PGP/GPG to sign commits(?) or perhaps just
    tags(?) in order to allow commits to be pulled only from trusted
    * share all branches and tags as well as just refs/heads/*
    * make "git push" re-create the .torrent (make_torrent.py) and work out
    how to notify seeders of a new HEAD (name the torrent after the HEAD ref,
    and just create a new one rather than delete the old?)

    so there is quite a bit to do, with the priority being on making a new
    URI type and a new "git-remote-{URI}" command, so that this becomes
    actually useable rather than just an experiment, and the project can
    be self-hosting as a truly distributed peer-to-peer development

    if anyone would like to assist, you only have to ask and (ironically)
    i will happily grant access to the gitorious-hosted repository.

    if anyone would like to sponsor this project, that would be very
    timely, as if i don't get some money soon i will be unable to pay for
    food and rent.

    Luke Kenneth Casson Leighton, Sep 1, 2010
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John O'Conner
    Jan 30, 2004
  2. Sudheer
    Roedy Green
    Apr 1, 2004
  3. Guy Noir

    Re: peer to peer messaging

    Guy Noir, May 26, 2005, in forum: Java
    Owen Jacobson
    May 31, 2005
  4. Matt Gessner
  5. Replies:

Share This Page