Re: How suitable is Python to write system utilities?

Discussion in 'Python' started by Alice Bevan–McGregor, Jan 6, 2011.

  1. On 2011-01-06 01:35:58 -0800, Rohit Coder said:

    > Is Python suitable to write low-level system utilities like Defrag,
    > Malware Removal Tools and Drivers?


    Yes and no.

    Python does include libraries (and has available third-party libraries)
    to interface with external low-level libraries of every kind, has
    Python-native third-party libraries to do things like examine ELF
    object files / executables, manipulate raw IP datagrams, etc, and it is
    possible to access glib (and other C libraries or even Windows DLLs)
    directly from within Python without creating wrapper interfaces.

    While you -can- do all of these things, the question becomes do you
    really -want to-?

    I've implemented server monitoring suites, FUSE filesystem drivers, and
    other strange low-level things in Python. This doesn't mean I'm
    getting the best "bang for the buck" when it comes to performance or
    overhead; I'm trading these things for the ease of prototyping,
    debugging, and the ability to utilize other high-level interfaces. My
    FUSE driver won't be as performant as it would have been had I written
    it in C. My server monitoring suite consumes more RAM than an
    equivalent solution in C. When it comes down to it, if you want it
    done efficiently, use C. ;)

    (As an aside, you -can- create hideous frankenstein monsters by using
    compiled CPython modules with glue code between the driver API, e.g.
    FUSE, and your CPython driver implementation; but in that case you're
    adding the overhead of Python for no gain whatsoever.)

    For anything system critical, Python might not be the best choice.
    Malware removal tools are themselves the target of malware (e.g. virii
    attempting to disable scanners and removal tools), and utilizing Python
    adds (IMHO) too many points of failure.

    Also, file fragmentation is a non-issue on all modern filesystems
    (ext3/4, reiser, ntfs, hfs+, etc.) as they perform live-system
    defragmentation to varying degrees. I have never seen a production
    server of mine (utilizing reiserfs) go above 11% fragmentation
    (non-contiguous extant allocations), and even that resolved itself
    within a few hours of active use.

    Note that non-contiguous extants are a distinct problem, reducing file
    read performance substantially, thus why filesystem drivers generally
    handle solving this problem by themselves. The other forms of
    fragmentation (free space fragmentation and file scattering /
    related-file fragmentation) substantially less so. Certain filesystems
    have features designed to avoid the latter (e.g. squashfs for ordering
    files on bootable CDs) and the former becomes more of an issue as you
    attempt to allocate extremely large contiguous files. (Becoming worse
    as free space is exhausted as more and more files of greater size need
    to be shuffled around the disk platter in order to free up a contiguous
    run of extants.)

    Hope this helps,

    - Alice.
     
    Alice Bevan–McGregor, Jan 6, 2011
    #1
    1. Advertising

  2. Alice Bevan–McGregor

    J.O. Aho Guest

    Alice Bevan–McGregor wrote:
    > On 2011-01-06 01:35:58 -0800, Rohit Coder said:
    >
    >> Is Python suitable to write low-level system utilities like Defrag,
    >> Malware Removal Tools and Drivers?

    >
    > Yes and no.
    >
    > Also, file fragmentation is a non-issue on all modern filesystems
    > (ext3/4, reiser, ntfs, hfs+, etc.) as they perform live-system
    > defragmentation to varying degrees.


    According to microsoft documentation, the recommendation is to run
    defragmentation on ntfs on a regular bases. There seems to come some
    improvement on the mft fragmentation, but still it feels long behind the
    linux/unix file systems.

    Do you have any recent documentation on ntfs that shows it has the capability
    to defragmentate itself other than mft?



    --

    //Aho
     
    J.O. Aho, Jan 6, 2011
    #2
    1. Advertising

  3. Alice Bevan–McGregor

    Roy Smith Guest

    In article <>,
    "J.O. Aho" <> wrote:

    > According to microsoft documentation, the recommendation is to run
    > defragmentation on ntfs on a regular bases. There seems to come some
    > improvement on the mft fragmentation, but still it feels long behind the
    > linux/unix file systems.
    >
    > Do you have any recent documentation on ntfs that shows it has the capability
    > to defragmentate itself other than mft?


    This is the best defragmenter for a windows file system is this:

    http://www.ubuntu.com/desktop/get-ubuntu/download
     
    Roy Smith, Jan 6, 2011
    #3
  4. Alice Bevan–McGregor

    J.O. Aho Guest

    Roy Smith wrote:
    > In article <>,
    > "J.O. Aho" <> wrote:
    >
    >> According to microsoft documentation, the recommendation is to run
    >> defragmentation on ntfs on a regular bases. There seems to come some
    >> improvement on the mft fragmentation, but still it feels long behind the
    >> linux/unix file systems.
    >>
    >> Do you have any recent documentation on ntfs that shows it has the capability
    >> to defragmentate itself other than mft?

    >
    > This is the best defragmenter for a windows file system is this:
    >
    > http://www.ubuntu,com/desktop/get-ubuntu/download


    It depends on your taste, I favour to be able to customize quite a lot of my
    installation and those rather use a meta-distribution.

    As SourceMage ( www.sourcemage.org ) or Gentoo ( www.gentoo.org ).

    But your reply don't point at a ms-documentation about auto defragmentation of
    a file system.

    --

    //Aho
     
    J.O. Aho, Jan 6, 2011
    #4
  5. Alice Bevan–McGregor

    Roy Smith Guest

    In article <>,
    "J.O. Aho" <> wrote:

    > Roy Smith wrote:
    > > In article <>,
    > > "J.O. Aho" <> wrote:
    > >
    > >> According to microsoft documentation, the recommendation is to run
    > >> defragmentation on ntfs on a regular bases. There seems to come some
    > >> improvement on the mft fragmentation, but still it feels long behind the
    > >> linux/unix file systems.
    > >>
    > >> Do you have any recent documentation on ntfs that shows it has the
    > >> capability
    > >> to defragmentate itself other than mft?

    > >
    > > This is the best defragmenter for a windows file system is this:
    > >
    > > http://www.ubuntu,com/desktop/get-ubuntu/download

    >
    > It depends on your taste, I favour to be able to customize quite a lot of my
    > installation and those rather use a meta-distribution.
    >
    > As SourceMage ( www.sourcemage.org ) or Gentoo ( www.gentoo.org ).
    >
    > But your reply don't point at a ms-documentation about auto defragmentation
    > of
    > a file system.


    I think you missed the point :)
     
    Roy Smith, Jan 6, 2011
    #5
  6. Alice Bevan–McGregor

    David Boddie Guest

    On Thursday 06 January 2011 12:08, Alice Bevan?McGregor wrote:

    > Python does include libraries (and has available third-party libraries)
    > to interface with external low-level libraries of every kind, has
    > Python-native third-party libraries to do things like examine ELF
    > object files / executables, manipulate raw IP datagrams, etc, and it is
    > possible to access glib (and other C libraries or even Windows DLLs)
    > directly from within Python without creating wrapper interfaces.


    Just out of interest, which module/package are you using to examine ELF
    files?

    David
     
    David Boddie, Jan 6, 2011
    #6
  7. On 2011-01-06 06:38:24 -0800, David Boddie said:

    > Just out of interest, which module/package are you using to examine ELF files?


    http://pypi.python.org/pypi/elffile

    - Alice.
     
    Alice Bevan–McGregor, Jan 6, 2011
    #7
  8. list from FTP server to a text file

    Hi,

    I am trying to create a list in a txt file from an ftp server. The
    following code is retrieving the list of the files but could not able to
    write in a text file. Any help is highly appreciated.

    Thanks



    ****************************
    import os
    import time
    from ftplib import FTP
    ftp = FTP("*.org","","") # connect to host, default port
    ftp.login()
    ftp.cwd("/pub/remotefolder/")
    ftp.retrlines('NLST')
    ******************************
     
    Ahmed, Shakir, Jan 6, 2011
    #8
  9. RE: list from FTP server to a text file

    -----Original Message-----
    From: python-list-bounces+shahmed=
    [mailto:python-list-bounces+shahmed=] On Behalf Of
    Dan M
    Sent: Thursday, January 06, 2011 11:06 AM
    To:
    Subject: Re: list from FTP server to a text file

    On Thu, 06 Jan 2011 10:51:42 -0500, Ahmed, Shakir wrote:

    > Hi,
    >
    > I am trying to create a list in a txt file from an ftp server. The
    > following code is retrieving the list of the files but could not able

    to
    > write in a text file. Any help is highly appreciated.
    >
    > Thanks
    >
    >
    >
    > ****************************
    > import os
    > import time
    > from ftplib import FTP
    > ftp = FTP("*.org","","") # connect to host, default port ftp.login()
    > ftp.cwd("/pub/remotefolder/")
    > ftp.retrlines('NLST')
    > ******************************


    WARNING: I am a newbie! Expect more pythonic ways to do this in other
    replies

    from ftplib import FTP
    ftp = FTP("host", "user", "pass")
    ftp.cwd("/pub/myfolder")
    files = ftp.nlst(".")
    f = open('files.txt', 'w')
    for file in files:
    f.write('%s\n' % (file,))
    f.close()

    --

    It worked
    Thanks,
    shk
     
    Ahmed, Shakir, Jan 6, 2011
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hai Nguyen
    Replies:
    1
    Views:
    4,566
    Natty Gur
    Jan 13, 2004
  2. JKV
    Replies:
    0
    Views:
    440
  3. JKV
    Replies:
    4
    Views:
    6,053
    P.Hill
    Feb 14, 2006
  4. SeSe
    Replies:
    2
    Views:
    400
    Uche Ogbuji
    Jan 2, 2005
  5. Tony Johansson
    Replies:
    0
    Views:
    283
    Tony Johansson
    Feb 24, 2014
Loading...

Share This Page