LWP::UserAgent question

Discussion in 'Perl Misc' started by a, Aug 14, 2006.

  1. a

    a Guest

    Hi
    I would like to use LWP::UserAgent to login the web site and process the web
    content. Then I should use $ua -> credentials($netloc, $realm, $uname,
    $pass)
    What is $netloc, $realm, $uname, and $pass?
    Can someone post an example to demonstrate?
    Thanks
    a, Aug 14, 2006
    #1
    1. Advertising

  2. a

    John Bokma Guest

    Re: LWP::UserAgent question--MULTIPOSTED

    wrote:

    Last warning: next time I report this annoying piece of garbage as
    Usenet abuse. I am sure that running bots, especially the piece of
    crap you are using, are a ToS violation of Giganews.

    You can't fix an issue by causing a bigger one.


    --
    John Bokma Freelance software developer
    &
    Experienced Perl programmer:
    http://castleamber.com/
    John Bokma, Aug 14, 2006
    #2
    1. Advertising

  3. a

    John Bokma Guest

    "a" <> wrote:

    > Hi
    > I would like to use LWP::UserAgent to login the web site and process
    > the web content. Then I should use $ua -> credentials($netloc, $realm,
    > $uname, $pass)
    > What is $netloc, $realm, $uname, and $pass?
    > Can someone post an example to demonstrate?


    First, don't multipost.

    Second, if it's basic authentication, you can use the example given under
    "ACCESS TO PROTECTED DOCUMENTS" in lwpcook.


    Examples of the use of credentials is under HTTP Authentication of lwptut.

    "

    $browser->credentials(
    'servername:portnumber',
    'realm-name',
    'username' => 'password'
    );


    "

    Note that the realm-name is displayed when you get the pop up window that
    allows you to log in.


    (type on the command line: perldoc lwptut )

    --
    John Bokma Freelance software developer
    &
    Experienced Perl programmer: http://castleamber.com/
    John Bokma, Aug 14, 2006
    #3
  4. Re: LWP::UserAgent question--MULTIPOSTED

    wrote:
    > "a" <> wrote:
    >>> [ snip and ignore MULTIPOSTED message ]


    I think this multiposting bot is a great idea.
    However I would suggest to tune it down a bit. Just make the text
    informative instead of accusing, remove those heavy borders, etc.
    A 3 to 4 line message with a link to more information is really all that is
    needed. No need for those KB and KB of heavy shots.

    jue
    Jürgen Exner, Aug 14, 2006
    #4
  5. Re: LWP::UserAgent question--MULTIPOSTED

    <> wrote:
    > "a" <> wrote:
    >>> [ snip and ignore MULTIPOSTED message ]



    > Questions or comments are welcome #



    Please undeploy it forthwith.


    > # Q-Why am I doing this? A--For a better usenet.



    It isn't working. This "cure" is worse than the disease.


    > # Some folks try to
    > # discourage job postings; some discourage off-topic posts. I try #

    ^
    ^
    > # to discourage multiposts



    No _you_ don't.

    Your _machine_ does.

    Not the same thing.


    > # If you don't wish to be bothered with these auto-generated #
    > # responses, please killfile the scanner. #



    If you don't stop this right quick, expect more than the scanner
    to be killfiled...


    > # But I choose to run this #
    > # scanner anonymously



    Bad choice.

    It hurts the credibility of your message so much as to make the
    postings worthless.



    ( I will note however that multiposting was the primary reason that
    I stopped participating in the beginners mailing list, way back
    when.
    )

    --
    Tad McClellan SGML consulting
    Perl programming
    Fort Worth, Texas
    Tad McClellan, Aug 14, 2006
    #5
  6. a

    John Bokma Guest

    Re: LWP::UserAgent question--MULTIPOSTED

    "Jürgen Exner" <> wrote:

    > wrote:
    >> "a" <> wrote:
    >>>> [ snip and ignore MULTIPOSTED message ]

    >
    > I think this multiposting bot is a great idea.


    Me no, and I have reported it for what I see it is right now: Usenet
    abuse.

    --
    John Bokma Freelance software developer
    &
    Experienced Perl programmer: http://castleamber.com/
    John Bokma, Aug 14, 2006
    #6
  7. a

    John Bokma Guest

    Re: LWP::UserAgent question--MULTIPOSTED

    Tad McClellan <> wrote:

    > If you don't stop this right quick, expect more than the scanner
    > to be killfiled...


    Too late :)

    --
    John Bokma Freelance software developer
    &
    Experienced Perl programmer: http://castleamber.com/
    John Bokma, Aug 14, 2006
    #7
  8. a

    Dr.Ruud Guest

    Re: LWP::UserAgent question--MULTIPOSTED

    schreef:
    > "a" <> wrote:


    >> [ snip and ignore MULTIPOSTED message ]

    >
    > **********************************************************************
    > ********** PLEASE DO NOT RESPOND TO THIS THREAD ***********
    > **********************************************************************
    >
    > This message has been multiposted as indicated by these message IDs:
    > <JoPDg.386093$Mn5.194189@pd7tw3no>
    > <GbPDg.382930$iF6.158066@pd7tw2no>



    s/</<news:/g

    and you should put them both in the References: header field (without
    the "news:" of course).

    --
    Affijn, Ruud

    "Gewoon is een tijger."
    Dr.Ruud, Aug 14, 2006
    #8
  9. a

    DJ Stunks Guest

    Re: LWP::UserAgent question--MULTIPOSTED

    John Bokma wrote:
    > "Jürgen Exner" <> wrote:
    >
    > > wrote:
    > >> "a" <> wrote:
    > >>>> [ snip and ignore MULTIPOSTED message ]

    > >
    > > I think this multiposting bot is a great idea.

    >
    > Me no, and I have reported it for what I see it is right now: Usenet
    > abuse.


    What happened to your "last warning", Bokma? Itchy reporting finger?
    Perhaps David can hook you up with his source for his bot and you could
    modify it to create your own bot reporting bot. The downside, however,
    is that, while more efficient, this approach would almost certainly
    take away from what appears to be a very enjoyable pastime for you.

    - Jake "Free Xah" Peavy
    DJ Stunks, Aug 14, 2006
    #9
  10. a

    John Bokma Guest

    Re: LWP::UserAgent question--MULTIPOSTED

    "DJ Stunks" <> wrote:

    > John Bokma wrote:
    >> "Jürgen Exner" <> wrote:
    >>
    >> > wrote:
    >> >> "a" <> wrote:
    >> >>>> [ snip and ignore MULTIPOSTED message ]
    >> >
    >> > I think this multiposting bot is a great idea.

    >>
    >> Me no, and I have reported it for what I see it is right now: Usenet
    >> abuse.

    >
    > What happened to your "last warning", Bokma? Itchy reporting finger?


    I decided to let the bot owner's Usenet provider decide ;-)

    > - Jake "Free Xah" Peavy


    Ha ha ha. If you're real (which I doubt), give Xah some momey for his
    process against DreamHost. It will increase the joy tenfold :-D.

    --
    John Bokma Freelance software developer
    &
    Experienced Perl programmer: http://castleamber.com/
    John Bokma, Aug 14, 2006
    #10
  11. a

    Guest

    Re: LWP::UserAgent question--MULTIPOSTED

    John Bokma wrote:
    > Last warning: next time I report this annoying piece of garbage as
    > Usenet abuse.


    > I am sure that running bots, especially the piece of
    > crap you are using, are a ToS violation of Giganews.


    I do not believe it is a ToS violation; GigaNews' terms may be found
    here:
    http://www.giganews.com/legal/aup.html

    > You can't fix an issue by causing a bigger one.


    Prehaps you will be kind enough to explain your objections. I have, on
    numerious occasions (along with many others) flagged multiposted
    messages (manually). I don't recall that anyone has ever been critical
    of this practice, and such notifications are common courtesy in
    professional newsgroups.

    Now I write a bot to do this and several people complain (but don't
    really say why). I thought folks like us write Perl scripts to automate
    repetitive manual tasks.

    Is it the fact that it's a bot that bothers you? That would seem an
    odd objection for a programmer. Or do you not like the content of the
    auto-messages? I'm not an English major - I would be happy to consider
    content edits. But I don't want to simply furnish a link - I've seen
    lots of links given to lots of OP's (Posting Guidelines, etc) which
    seem to be ignored. Prehaps a direct reply such as this would be more
    effective.

    Or do you think my code is crap? I would be happy to post it for peer
    review.

    Regarding the length of the message: There is a lot of "introductory"
    info there which was designed to be temporary (as the message says). I
    have removed this text, and the message is now down to a much more
    resonable size (about 60 lines).

    If you don't like seeing the messages, you may killfile the scanner
    (just as you may killfile the Faq-O-Matic bot or the Posting Guidelines
    bot). If you think I'm a jerk for doing this then you may killfile me
    as well. But, first, I hope you would at least give me a chance to
    understand and address any concerns you have.

    --
    David Filmer (http://DavidFilmer.com)
    , Aug 14, 2006
    #11
  12. Re: LWP::UserAgent question--MULTIPOSTED

    John Bokma wrote:
    > wrote:
    >
    > Last warning: next time I report this annoying piece of garbage as
    > Usenet abuse. I am sure that running bots, especially the piece of
    > crap you are using, are a ToS violation of Giganews.
    >
    > You can't fix an issue by causing a bigger one.


    Tell that to Bush. ;-)


    John
    --
    use Perl;
    program
    fulfillment
    John W. Krahn, Aug 14, 2006
    #12
  13. a

    John Bokma Guest

    Re: LWP::UserAgent question--MULTIPOSTED

    "John W. Krahn" <> wrote:

    > John Bokma wrote:
    >> wrote:
    >>
    >> Last warning: next time I report this annoying piece of garbage as
    >> Usenet abuse. I am sure that running bots, especially the piece of
    >> crap you are using, are a ToS violation of Giganews.
    >>
    >> You can't fix an issue by causing a bigger one.

    >
    > Tell that to Bush. ;-)


    :-D

    --
    John Bokma Freelance software developer
    &
    Experienced Perl programmer: http://castleamber.com/
    John Bokma, Aug 14, 2006
    #13
  14. a

    John Bokma Guest

    Re: LWP::UserAgent question--MULTIPOSTED

    wrote:

    > If you don't like seeing the messages, you may killfile the scanner
    > (just as you may killfile the Faq-O-Matic bot or the Posting Guidelines
    > bot). If you think I'm a jerk for doing this then you may killfile me
    > as well. But, first, I hope you would at least give me a chance to
    > understand and address any concerns you have.


    Netiquette shouldn't be enforced by posting bots. Moreover, a bot with
    such an impact (several messages a day I have seen so far) should probably
    be voted for (or against).

    And no, I am not going to add every bot to a kill file that some
    programmer thinks is nifty to run on Usenet. As a programmer I appreciate
    bots, but I appreciate the human touch more.

    --
    John Bokma Freelance software developer
    &
    Experienced Perl programmer: http://castleamber.com/
    John Bokma, Aug 14, 2006
    #14
  15. a

    Guest

    Re: LWP::UserAgent question--MULTIPOSTED

    John Bokma wrote:

    > Netiquette shouldn't be enforced by posting bots.


    Hmmm. Is that the essence of your objection?

    I agree that it is very difficult in most instances and thus should not
    even be attempted. Natural language is difficult to parse. I might
    TRY to write a job-listing bot (and save Mr. Adler some trouble) but I
    don't think it would be very effective, and I would never even try.

    But, if someone could write an AI module that could reliably parse
    natural language for meaning and intent, it would change everything.
    Then, I COULD parse-and-flag job-posting messages (and I would). But I
    can't do that (so I don't).

    But it's easy to detect a multipost with a program (without AI). It is
    fully possible to eliminate false positives. A few false-negatives may
    slip through (where the OP tweaks the content) and that's to be
    expected. The majority of multiposts CAN be identified (and, in fact, I
    cannot recall a single multipost which was not a simple cut-and-paste -
    I did go back and research the multiposts that I'vemanually flagged and
    every single one of them would have been flagged by my bot).

    > Moreover, a bot with such an impact (several messages a day I have seen
    > so far) should probably be voted for (or against).


    Where are you seeing such volume??? The scanner was deployed last
    Thursday, and as of now (wee hours of Monday morning in my $TZ), I've
    only observed two messages get flagged (and my tracking database agrees
    with my observation). And the first message was an improperly flagged
    crosspost because I had not un-commented a line during testing (grrr).
    Had the bug been fixed initially, only one message so far would have
    been flagged. That doesn't strike me as a high-impact process - one
    message in 3.5 days.

    But I am not opposed to the idea of a vote, as you suggest. I wish to
    be a good usenet citizen in whatever group I participate in. That's
    why I strive to adhere to the Posting Guidelines, and often recommend
    that others do so as well. If the community at large doesn't like this
    bot then I will certainly shut it down (without anyone needing to
    resort to demands or threats). If folks think the bot is OK but the
    message sucks, I can change it (and, note, I have removed most of the
    "introductory text" but no message has yet been flagged under the new
    parameters with the shorter reply-message). I'm a reasonable guy; I
    can adapt to whatever the community consensus happens to be, and will
    gladly do so.

    --
    David Filmer (http://DavidFilmer.com)
    , Aug 14, 2006
    #15
  16. Re: LWP::UserAgent question--MULTIPOSTED

    Tad McClellan wrote:
    > <> wrote:
    >># Q-Why am I doing this? A--For a better usenet.

    >
    > It isn't working.


    How is it not working? We know that guidelines alone don't work...

    > This "cure" is worse than the disease.


    Are manual objections better? If so, in what way?

    >># Some folks try to
    >># discourage job postings; some discourage off-topic posts. I try #
    >># to discourage multiposts

    >
    > No _you_ don't.
    >
    > Your _machine_ does.
    >
    > Not the same thing.


    What significance would that have?

    Personally I support David's initiative.

    --
    Gunnar Hjalmarsson
    Email: http://www.gunnar.cc/cgi-bin/contact.pl
    Gunnar Hjalmarsson, Aug 14, 2006
    #16
  17. "a" <> writes:

    > Hi
    > I would like to use LWP::UserAgent to login the web site and process the web
    > content. Then I should use $ua -> credentials($netloc, $realm, $uname,
    > $pass)
    > What is $netloc, $realm, $uname, and $pass?
    > Can someone post an example to demonstrate?


    Have a look at "perldoc lwpcook". There's an example of using this method in
    there.

    sherm--

    --
    Web Hosting by West Virginians, for West Virginians: http://wv-www.net
    Cocoa programming in Perl: http://camelbones.sourceforge.net
    Sherm Pendley, Aug 14, 2006
    #17
  18. Re: LWP::UserAgent question--MULTIPOSTED

    writes:

    > **********************************************************************
    > ********** PLEASE DO NOT RESPOND TO THIS THREAD ***********
    > **********************************************************************


    Grow up. I intend to reply to each and every thread in which your little
    whine-bot posts. Self-appointed "net nannies" are *far* more annoying than
    any amount of cross- and or multi-posting.

    sherm--

    --
    Web Hosting by West Virginians, for West Virginians: http://wv-www.net
    Cocoa programming in Perl: http://camelbones.sourceforge.net
    Sherm Pendley, Aug 14, 2006
    #18
  19. a

    John Bokma Guest

    Re: LWP::UserAgent question--MULTIPOSTED

    wrote:

    > John Bokma wrote:
    >
    >> Netiquette shouldn't be enforced by posting bots.

    >
    > Hmmm. Is that the essence of your objection?


    Ok, a short reply (hah). From what I have seen so far, your bot replies to
    each multiposted message (correct me if I am wrong), in each group.
    Meaning if someone multipost in 3 groups what we might get is:

    1 post per group (3 in total)
    1 reply by your bot per post (3 in total)
    0 or more possible replies per post by people not being aware of the
    multipost. [1]
    0 or more angry replies by people who discovered the multipost. [1]
    0 or more people replying to your bot replies (complaints, asking what it
    is.)

    Does it stop multiposting: no
    Does it add a lot of noise to the group: I am afraid that the answer is
    yes.

    Personally I can live with 2 kinds of bots: info bots that start a thread
    (post a FAQ, postings statistics, etc), and cancel bots. What you have
    written I am tempted to call a stalker bot.

    If you just wanted to teach people not to multipost, carefully reread what
    your bot spits out and try to wear the shoes of the offender. Do you
    really think that 140+ (IIRC) lines with big black boxes (####) is
    something anyone really is going to read without being offended?

    What was wrong with:

    Don't post the same message to several groups (multipost), see
    http://www.cs.tut.fi/~jkorpela/usenet/xpost.html.


    [1] people might miss your bot because of several reasons, one might be
    because they kill filed your bot.

    --
    John Bokma Freelance software developer
    &
    Experienced Perl programmer: http://castleamber.com/
    John Bokma, Aug 14, 2006
    #19
  20. a

    John Bokma Guest

    Re: LWP::UserAgent question--MULTIPOSTED

    Sherm Pendley <> wrote:

    > writes:
    >
    >> **********************************************************************
    >> ********** PLEASE DO NOT RESPOND TO THIS THREAD
    >> ***********
    >> **********************************************************************

    >
    > Grow up. I intend to reply to each and every thread in which your
    > little whine-bot posts. Self-appointed "net nannies" are *far* more
    > annoying than any amount of cross- and or multi-posting.


    Like I just wrote. This bot is just going to add its own noise, and is
    going to attract noise by people who don't like it. Instead of 2 postings
    in 2 groups we now get 2 postings + 2 "don't multipost" post, followed by
    0 or more comments on the bot :-(

    --
    John Bokma Freelance software developer
    &
    Experienced Perl programmer: http://castleamber.com/
    John Bokma, Aug 14, 2006
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. La Jesus
    Replies:
    9
    Views:
    1,299
    Gunnar Hjalmarsson
    Oct 27, 2003
  2. Chandra

    Problem with LWP::USERAGENT

    Chandra, Feb 25, 2004, in forum: Perl
    Replies:
    0
    Views:
    491
    Chandra
    Feb 25, 2004
  3. Vinay Gupta
    Replies:
    2
    Views:
    1,098
    Michael Schlenker
    Jul 27, 2004
  4. Playker
    Replies:
    0
    Views:
    500
    Playker
    Feb 24, 2005
  5. Replies:
    13
    Views:
    2,721
    Arne Vajhøj
    Mar 18, 2008
Loading...

Share This Page