Establishing a p2p connection in python

Discussion in 'Python' started by gs, Oct 8, 2004.

  1. gs

    gs Guest

    Hi!

    This is my first time posting to a newsgroup so please be gentle to me
    :)

    Introduction to my problem:

    I'm studying at the university and a lot of friends and I use MSN to
    communicate. The problem is that MSN file sending/receiving
    capabilities are terribly slow and has no resume option. We're also on
    different subnets and can not access eachoters computers through smb
    shares, nor can we have local ftp-servers.

    My programming background:

    I've been taking a few classes in JAVA, enough to get familiar with
    sockets, threads and the most common sorting algortithms and
    datastructures (binary trees, stacks etc). I'm also very comfortable
    programming PHP on a webdeveloper basis.
    I recently started out with Python and i immediatly fell in love with
    it, and now I want to "master" it! :) Alas, my problem is also of
    educational nature.

    Setup:
    We're about 30 friends who want the ability to share files.
    We're spread out on about 3 different translated external IPs.
    I have access to a public IP that has the ability to run python (but
    not enough bandwith nor space to host an FTP).
    I have limited programming experience.

    Proposed solution:
    I thought that one approach could be to write a small server for my
    public server with the sole purpose of keeping track of connected
    users and establishing p2p connections on demand.

    Problems:
    1) I've written a threaded server that stores information about
    connected servers in a "container class". This class holds information
    about username, a unique ID, the connecting adress and the actual
    socket. I have not been able to find any information online on how i
    could go about to connect two sockets. It would seem like the socket
    is already "occupied" since it is connected to the server. Could one
    duplicate, or otherwise get a dedicated socket for sending binary
    data, to one connected socket from another? A few pointers and/or tips
    would be greatly appreciated!

    2) I've also failed to find python specific information on how abouts
    one would do file resuming. I would think that you, somehow, use
    md5-checksums to check file status and somehow skip the first part of
    the datastream. Also, pointers and maybe an explanation in (short)
    pseudo-code would be much appreciated.

    3) Is this too big of a project for a novice programmer such as
    myself? I do want to go through with this since it's a genuine
    problem, but i do not know if i'm ready. I'm not a "quitter" per say,
    but i'm afraid doing something TOO advanced in the beginning might be
    overwhelming :)

    Thanks for reading,
    gs
    gs, Oct 8, 2004
    #1
    1. Advertising

  2. gs <> wrote:
    ...
    > We're about 30 friends who want the ability to share files.


    You could look into BitTorrent, a popular p2p program which, I believe,
    is implemented in Python. Maybe, if not usable as-is, it might be made
    to fit your needs with some minor modifications...?


    Alex
    Alex Martelli, Oct 8, 2004
    #2
    1. Advertising

  3. gs

    Jeremy Jones Guest

    gs wrote:

    >Hi!
    >
    >This is my first time posting to a newsgroup so please be gentle to me
    >:)
    >
    >
    >

    Welcome. You have little to fear from this crowd.

    >Introduction to my problem:
    >
    >
    >

    <snip>
    I think that your problem can be solved with Twisted. That is _a_
    solution. Not the only, maybe not the best for everything you're
    wanting to do (which it seems that learning a lot is important to you),
    but _a_ solution. It's got a bit of a learning curve to wrap your head
    around how everything fits together, but it's a powerful framework.
    I've used it less than I plan to, but I've got some interest in how it
    works.

    >Problems:
    >1) I've written a threaded server that stores information about
    >connected servers in a "container class". This class holds information
    >about username, a unique ID, the connecting adress and the actual
    >socket. I have not been able to find any information online on how i
    >could go about to connect two sockets. It would seem like the socket
    >is already "occupied" since it is connected to the server. Could one
    >duplicate, or otherwise get a dedicated socket for sending binary
    >data, to one connected socket from another? A few pointers and/or tips
    >would be greatly appreciated!
    >
    >

    Are you talking about your server binding two different ports? If you
    are using the Python standard SocketServer, you should be able to just
    spin off a thread for each server you start and tell it to
    serve_forever() (or what have you). If you're using Twisted, I think
    all you would need to do is factory.listenTCP(<port number>, <factory>),
    then reactor.run(). I'll leave getting data between the different ports
    you've bound as an exercise for the reader.

    >2) I've also failed to find python specific information on how abouts
    >one would do file resuming. I would think that you, somehow, use
    >md5-checksums to check file status and somehow skip the first part of
    >the datastream. Also, pointers and maybe an explanation in (short)
    >pseudo-code would be much appreciated.
    >
    >

    Can't help you here, sorry.

    >3) Is this too big of a project for a novice programmer such as
    >myself? I do want to go through with this since it's a genuine
    >problem, but i do not know if i'm ready. I'm not a "quitter" per say,
    >but i'm afraid doing something TOO advanced in the beginning might be
    >overwhelming :)
    >
    >

    I don't think this is too big at all. But I do think you'll want to
    break it down a little bit and set smaller goals for yourself. Try to
    first get the smallest fairly usable thing working. Then you can start
    adding functionality from there.

    >Thanks for reading,
    >gs
    >
    >

    HTH,


    Jeremy Jones
    Jeremy Jones, Oct 8, 2004
    #3
  4. gs

    Darren Kirby Guest

    quoth the gs:
    > Hi!
    >
    > This is my first time posting to a newsgroup so please be gentle to me
    >
    > :)
    >
    > Introduction to my problem:
    >
    > I'm studying at the university and a lot of friends and I use MSN to
    > communicate. The problem is that MSN file sending/receiving
    > capabilities are terribly slow and has no resume option. We're also on
    > different subnets and can not access eachoters computers through smb
    > shares, nor can we have local ftp-servers.
    >
    > My programming background:
    >
    > I've been taking a few classes in JAVA, enough to get familiar with
    > sockets, threads and the most common sorting algortithms and
    > datastructures (binary trees, stacks etc). I'm also very comfortable
    > programming PHP on a webdeveloper basis.
    > I recently started out with Python and i immediatly fell in love with
    > it, and now I want to "master" it! :) Alas, my problem is also of
    > educational nature.
    >
    > Setup:
    > We're about 30 friends who want the ability to share files.
    > We're spread out on about 3 different translated external IPs.
    > I have access to a public IP that has the ability to run python (but
    > not enough bandwith nor space to host an FTP).
    > I have limited programming experience.
    >
    > Proposed solution:
    > I thought that one approach could be to write a small server for my
    > public server with the sole purpose of keeping track of connected
    > users and establishing p2p connections on demand.
    >
    > Problems:
    > 1) I've written a threaded server that stores information about
    > connected servers in a "container class". This class holds information
    > about username, a unique ID, the connecting adress and the actual
    > socket. I have not been able to find any information online on how i
    > could go about to connect two sockets. It would seem like the socket
    > is already "occupied" since it is connected to the server. Could one
    > duplicate, or otherwise get a dedicated socket for sending binary
    > data, to one connected socket from another? A few pointers and/or tips
    > would be greatly appreciated!
    >
    > 2) I've also failed to find python specific information on how abouts
    > one would do file resuming. I would think that you, somehow, use
    > md5-checksums to check file status and somehow skip the first part of
    > the datastream. Also, pointers and maybe an explanation in (short)
    > pseudo-code would be much appreciated.
    >
    > 3) Is this too big of a project for a novice programmer such as
    > myself? I do want to go through with this since it's a genuine
    > problem, but i do not know if i'm ready. I'm not a "quitter" per say,
    > but i'm afraid doing something TOO advanced in the beginning might be
    > overwhelming :)
    >
    > Thanks for reading,
    > gs


    I can suggest taking a look at nicotine, which is a full fledged p2p app
    written in python. I'm not saying it's your solution, but you may get some
    ideas about how to implement what you want by reading the source code, as it
    does support file resuming.

    -d

    --
    Part of the problem since 1976
    http://badcomputer.no-ip.com
    Get my public key from
    http://keyserver.linux.it/pks/lookup?op=index&search=bulliver
    "...the number of UNIX installations has grown to 10, with more expected..."
    - Dennis Ritchie and Ken Thompson, June 1972

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.2.4 (GNU/Linux)

    iD8DBQBBZrDxOrzWcOwL7mwRAu9zAJ996iG8jAuNzdnVYCFTlc6TrAc/GgCffScs
    FZK6DINDLa4YmWAHrc2OIo4=
    =tR8D
    -----END PGP SIGNATURE-----
    Darren Kirby, Oct 8, 2004
    #4
  5. gs

    Ville Vainio Guest

    >>>>> "gs" == gs <> writes:

    gs> Proposed solution: I thought that one approach could be to
    gs> write a small server for my public server with the sole
    gs> purpose of keeping track of connected users and establishing
    gs> p2p connections on demand.

    One lightweight approach (assuming that a web server is available)
    would be to put up a cgi script that adds the ip:port information to
    onlineusers.txt. Then the clients can just use urllib to notify the
    script when they become online, and also get the list of connected
    clients by just retrieving onlineusers.txt. Of course onlineusers.txt
    could contain the lists of shared files for each client as well...

    If you use unique user ids, you can keep the file from growing up too
    much. If not, you could clean up dead clients by trying to connect all
    the specified addresses.

    gs> 1) I've written a threaded server that stores information
    gs> about connected servers in a "container class". This class
    gs> holds information about username, a unique ID, the connecting
    gs> adress and the actual socket. I have not been able to find any
    gs> information online on how i could go about to connect two
    gs> sockets. It would seem like the socket is already "occupied"
    gs> since it is connected to the server. Could one duplicate, or
    gs> otherwise get a dedicated socket for sending binary data, to
    gs> one connected socket from another? A few pointers and/or tips
    gs> would be greatly appreciated!

    If you went on to "connect" the two sockets, you would just
    instantiate a thread that forwards the data, and would use more server
    bandwidth than an FTP server would have used.

    The idea in p2p is to make clients connect to each other. The task of
    the server is to tell the clients about other clients so they can
    connect to each other.


    gs> 2) I've also failed to find python specific information on how
    gs> abouts one would do file resuming. I would think that you,
    gs> somehow, use md5-checksums to check file status and somehow
    gs> skip the first part of the datastream. Also, pointers and
    gs> maybe an explanation in (short) pseudo-code would be much
    gs> appreciated.

    It's trivial since you are going to implement a custom protocol
    anyway. Make the "get file" command that you pass to the file provider
    such that in addition to the file name you pass the starting offset
    within the file.


    gs> 3) Is this too big of a project for a novice programmer such
    gs> as myself? I do want to go through with this since it's a
    gs> genuine problem, but i do not know if i'm ready. I'm not a
    gs> "quitter" per say, but i'm afraid doing something TOO advanced
    gs> in the beginning might be overwhelming :)

    No, it's a great first project. Go ahead, I believe you will find it
    surprisingly easy, educational and something for which Python is a
    perfect fit.

    It'll be a fun project to expand as well, to provide chunked file
    transfer etc. You could also explore the alternative of making it a
    bunch of wrapper scripts that just invoke bittorrent...

    --
    Ville Vainio http://tinyurl.com/2prnb
    Ville Vainio, Oct 8, 2004
    #5
  6. gs

    gs Guest

    (Alex Martelli) wrote in message news:<1glc4eg.zvju3nsu0g6N%>...
    > gs <> wrote:
    > ...
    > > We're about 30 friends who want the ability to share files.

    >
    > You could look into BitTorrent, a popular p2p program which, I believe,
    > is implemented in Python. Maybe, if not usable as-is, it might be made
    > to fit your needs with some minor modifications...?
    >
    >
    > Alex


    Ok, that sounds very interesting, Alex. BitTorrent seems to be able to
    handle my needs at the moment, althoug i'd eventually want to write
    something myself :) Thanks for the pointer!

    Off to read up on this and try to persuade my isp to let me add
    ..torrent apptypes :)

    sincerely,
    gs
    gs, Oct 8, 2004
    #6
  7. gs

    Ville Vainio Guest

    >>>>> "gs" == gs <> writes:

    gs> Off to read up on this and try to persuade my isp to let me
    gs> add .torrent apptypes :)

    What does your ISP have to do with this? .torrents are just normal
    files.

    --
    Ville Vainio http://tinyurl.com/2prnb
    Ville Vainio, Oct 8, 2004
    #7
  8. gs

    gs Guest

    Ville Vainio <> wrote in message news:<>...
    > >>>>> "gs" == gs <> writes:

    >
    > gs> Proposed solution: I thought that one approach could be to
    > gs> write a small server for my public server with the sole
    > gs> purpose of keeping track of connected users and establishing
    > gs> p2p connections on demand.
    >
    > One lightweight approach (assuming that a web server is available)
    > would be to put up a cgi script that adds the ip:port information to
    > onlineusers.txt. Then the clients can just use urllib to notify the
    > script when they become online, and also get the list of connected
    > clients by just retrieving onlineusers.txt. Of course onlineusers.txt
    > could contain the lists of shared files for each client as well...
    >
    > If you use unique user ids, you can keep the file from growing up too
    > much. If not, you could clean up dead clients by trying to connect all
    > the specified addresses.
    >
    > gs> 1) I've written a threaded server that stores information
    > gs> about connected servers in a "container class". This class
    > gs> h


    olds information about username, a unique ID, the connecting
    > gs> adress and the actual socket. I have not been able to find any
    > gs> information online on how i could go about to connect two
    > gs> sockets. It would seem like the socket is already "occupied"
    > gs> since it is connected to the server. Could one duplicate, or
    > gs> otherwise get a dedicated socket for sending binary data, to
    > gs> one connected socket from another? A few pointers and/or tips
    > gs> would be greatly appreciated!
    >
    > If you went on to "connect" the two sockets, you would just
    > instantiate a thread that forwards the data, and would use more server
    > bandwidth than an FTP server would have used.
    >
    > The idea in p2p is to make clients connect to each other. The task of
    > the server is to tell the clients about other clients so they can
    > connect to each other.
    >
    >
    > gs> 2) I've also failed to find python specific information on how
    > gs> abouts one would do file resuming. I would think that you,
    > gs> somehow, use md5-checksums to check file status and somehow
    > gs> skip the first part of the datastream. Also, pointers and
    > gs> maybe an explanation in (short) pseudo-code would be much
    > gs> appreciated.
    >
    > It's trivial since you are going to implement a custom protocol
    > anyway. Make the "get file" command that you pass to the file provider
    > such that in addition to the file name you pass the starting offset
    > within the file.
    >
    >
    > gs> 3) Is this too big of a project for a novice programmer such
    > gs> as myself? I do want to go through with this since it's a
    > gs> genuine problem, but i do not know if i'm ready. I'm not a
    > gs> "quitter" per say, but i'm afraid doing something TOO advanced
    > gs> in the beginning might be overwhelming :)
    >
    > No, it's a great first project. Go ahead, I believe you will find it
    > surprisingly easy, educational and something for which Python is a
    > perfect fit.
    >
    > It'll be a fun project to expand as well, to provide chunked file
    > transfer etc. You could also explore the alternative of making it a
    > bunch of wrapper scripts that just invoke bittorrent...


    Thanks all of you for the nice feedback. I'm spending all of my
    sparetime just reading up on all technlogies and suggestions! :)

    cheers,
    gs
    gs, Oct 11, 2004
    #8
  9. (gs) wrote in message news:<>...
    > Hi!
    >
    > This is my first time posting to a newsgroup so please be gentle to me
    > :)
    >
    > Introduction to my problem:
    >
    > I'm studying at the university and a lot of friends and I use MSN to
    > communicate. The problem is that MSN file sending/receiving
    > capabilities are terribly slow and has no resume option. We're also on
    > different subnets and can not access eachoters computers through smb
    > shares, nor can we have local ftp-servers.
    >

    <snip>

    You might look into the Waste p2p software. It is a p2p program that
    uses RSA encryption and requires manual addition of nodes, I believe.
    So you can use it to create a private p2p network. It might work for
    you, or at least give you some ideas in terms of security features and
    such.
    Carl Scharenberg, Oct 11, 2004
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Chad W. Taylor
    Replies:
    1
    Views:
    2,961
    iksrazal
    Jun 26, 2004
  2. =?Utf-8?B?dGhvbQ==?=

    error while establishing a connection to the server

    =?Utf-8?B?dGhvbQ==?=, Jul 25, 2006, in forum: ASP .Net
    Replies:
    3
    Views:
    12,451
    =?Utf-8?B?dGhvbQ==?=
    Jul 25, 2006
  3. Tor Inge Rislaa
    Replies:
    0
    Views:
    694
    Tor Inge Rislaa
    Sep 27, 2006
  4. relisoft
    Replies:
    1
    Views:
    384
    red floyd
    Jul 11, 2006
  5. gnewsgroup
    Replies:
    0
    Views:
    311
    gnewsgroup
    Jan 2, 2008
Loading...

Share This Page