[Newbie] Is there any method to urlretrieve to a file the html source

Discussion in 'Python' started by Aldo Ceccarelli, Feb 25, 2008.

  1. Hello All,
    I apologize for posting a question on this but I could not find a
    complete answer after reading and searching so far:)

    My problem is that I'd like to store the html source of a certain web
    url but when I try via urllib / urllib2 reads or urlretrieve I get
    only a part of the contents (ist type is text/html).

    Metainformation associated with the url show that it has a:

    Transfer-Encoding: chunked

    and this seems the reason why I can get only a part of it.

    Can anybody address me to a possible solution?

    Many thanks in advance!
    WKR Aldo
     
    Aldo Ceccarelli, Feb 25, 2008
    #1
    1. Advertising

  2. Aldo Ceccarelli

    Thinker Guest

    Re: [Newbie] Is there any method to urlretrieve to a file the html

    Aldo Ceccarelli wrote:
    > Hello All,
    > I apologize for posting a question on this but I could not find a
    > complete answer after reading and searching so far:)
    >
    > My problem is that I'd like to store the html source of a certain web
    > url but when I try via urllib / urllib2 reads or urlretrieve I get
    > only a part of the contents (ist type is text/html).
    >
    > Metainformation associated with the url show that it has a:
    >
    > Transfer-Encoding: chunked
    >
    > and this seems the reason why I can get only a part of it.
    >

    Please show your code, here. urllib doesn't always return full content
    in one time. I guess it is the problem.
     
    Thinker, Feb 25, 2008
    #2
    1. Advertising

  3. Re: Is there any method to urlretrieve to a file the html source from

    Thank you All for your suggestions:

    I could finally assess that

    besides partial read from chunked url - by-passed thanks to a socket
    approach:

    f.i. this was good for me: http://python.about.com/od/networkingwithpython/ss/beg_web_client_9.htm

    I have still two kind of problems that quit this post's aims (original
    url is redirected and has password protections too, I will go for
    these now)

    Again SUPER thanks to You All:) Aldo
     
    Aldo Ceccarelli, Feb 25, 2008
    #3
  4. Aldo Ceccarelli

    7stud Guest

    Re: Is there any method to urlretrieve to a file the html source from

    On Feb 25, 8:25 am, Aldo Ceccarelli <> wrote:
    > Thank you All for your suggestions:
    >
    > I could finally assess that
    >
    > besides partial read from chunked url - by-passed thanks to a socket
    > approach:
    >
    > f.i. this was good for me:http://python.about.com/od/networkingwithpython/ss/beg_web_client_9.htm
    >
    > I have still two kind of problems that quit this post's aims (original
    > url is redirected and has password protections too, I will go for
    > these now)
    >
    > Again SUPER thanks to You All:) Aldo


    Don't use that code--it's merely a bare bones example. Post the url
    that is giving you problems.
     
    7stud, Feb 25, 2008
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Cloud Burst
    Replies:
    11
    Views:
    1,123
  2. HP
    Replies:
    2
    Views:
    399
    Kevin Cazabon
    Jul 31, 2003
  3. Sam Sungshik Kong

    urllib.urlretrieve error

    Sam Sungshik Kong, May 23, 2004, in forum: Python
    Replies:
    2
    Views:
    625
    Sam Sungshik Kong
    May 24, 2004
  4. Sven

    urlretrieve get file name

    Sven, Nov 9, 2006, in forum: Python
    Replies:
    6
    Views:
    527
  5. Даниил Рыжков

    urllib, urlretrieve method, how to get headers?

    Даниил Рыжков, Jul 1, 2011, in forum: Python
    Replies:
    6
    Views:
    1,097
    Даниил Рыжков
    Jul 2, 2011
Loading...

Share This Page