Gateway is malfunctioning

Discussion in 'Ruby' started by Trans, Dec 1, 2006.

  1. Trans

    Trans Guest

    Someone recently mentioned missing posts. I just did a comparsion of
    comp.lang.ruby and ruby-talk-google and indeed the usenet is missing
    posts.

    James you want to have a look at the gateway?

    T.
     
    Trans, Dec 1, 2006
    #1
    1. Advertising

  2. Trans wrote:
    > Someone recently mentioned missing posts. I just did a comparsion of
    > comp.lang.ruby and ruby-talk-google and indeed the usenet is missing
    > posts.
    >
    > James you want to have a look at the gateway?
    >
    > T.
    >
    >

    I think someone mentioned earlier that the missing posts were mostly
    those with HTML in them. Why the HTML posts are filtered out I do not know.
     
    Michael W. Ryder, Dec 1, 2006
    #2
    1. Advertising

  3. Trans

    Damphyr Guest

    Michael W. Ryder wrote:
    > Trans wrote:
    >> Someone recently mentioned missing posts. I just did a comparsion of
    >> comp.lang.ruby and ruby-talk-google and indeed the usenet is missing
    >> posts.
    >> James you want to have a look at the gateway?
    >>
    >> T.
    >>
    >>

    > I think someone mentioned earlier that the missing posts were mostly
    > those with HTML in them. Why the HTML posts are filtered out I do not
    > know.

    Not true. I had some missing posts (a couple misdated and a couple not)
    and I only send text.
    Cheers,
    V.-

    --
    http://www.braveworld.net/riva
     
    Damphyr, Dec 1, 2006
    #3
  4. Trans

    Guest

    On Fri, 1 Dec 2006, Trans wrote:

    > Someone recently mentioned missing posts. I just did a comparsion of
    > comp.lang.ruby and ruby-talk-google and indeed the usenet is missing posts.
    >
    > James you want to have a look at the gateway?


    as far as i can tell it always is - or at least has been for weeks.

    can we write a script to compare on a regular basis? a monitor?

    are you in charge of this james?

    cheers.

    -a
    --
    if you want others to be happy, practice compassion.
    if you want to be happy, practice compassion. -- the dalai lama
     
    , Dec 1, 2006
    #4
  5. On Nov 30, 2006, at 9:22 PM, Trans wrote:

    > Someone recently mentioned missing posts. I just did a comparsion of
    > comp.lang.ruby and ruby-talk-google and indeed the usenet is missing
    > posts.


    Here's the information I have on this currently:

    * I rewrote the entire Gateway this last week, so anything before
    that is history. The news-to-email side is already deployed. My
    email-to-news code is complete, but not deployed. (Our SysAdmin will
    get to it soon now.) If you raise issues, please make sure they are
    recent.
    * I now have very detailed logs on everything the Gateway does, so
    please be specific. For example, tell me which messages are missing,
    not just that they are.
    * Our Usenet host does not allow us to post HTML emails. This is not
    changing. (I'm not bothered by this because I do not believe anyone
    should be sending HTML email to Ruby Talk.)
    * Our Gateway is an NNTP <-> email Gateway. There has been at least
    one instance of a Usenet post using ancient header formatting
    predating NNTP. These messages are not supported.

    > James you want to have a look at the gateway?


    I will address individual issues, as long as they are aware of all
    four points above.

    James Edward Gray II
     
    James Edward Gray II, Dec 1, 2006
    #5
  6. On Dec 1, 2006, at 8:29 AM, wrote:

    > are you in charge of this james?


    I rewrote and now run the Ruby Gateway, yes.

    James Edward Gray II
     
    James Edward Gray II, Dec 1, 2006
    #6
  7. On Dec 1, 2006, at 4:06 AM, Damphyr wrote:

    > Not true. I had some missing posts (a couple misdated and a couple
    > not) and I only send text.


    Your posts appeared on one side of the Gateway (the Ruby Talk mailing
    list or the Usenet group comp.lang.ruby), but did not reach the other
    side? If so, please lead me to the posts on the side where they did
    appear.

    James Edward Gray II
     
    James Edward Gray II, Dec 1, 2006
    #7
  8. Trans

    Guest

    On Sat, 2 Dec 2006, James Edward Gray II wrote:

    > * I rewrote the entire Gateway this last week, so anything before that is
    > history. The news-to-email side is already deployed. My email-to-news code
    > is complete, but not deployed. (Our SysAdmin will get to it soon now.) If
    > you raise issues, please make sure they are recent.


    mine aren't - so ignore for now.

    > * I now have very detailed logs on everything the Gateway does, so please be
    > specific. For example, tell me which messages are missing, not just that
    > they are.


    can't we automate this? monitor/scraper?


    > * Our Usenet host does not allow us to post HTML emails. This is not
    > changing. (I'm not bothered by this because I do not believe anyone should
    > be sending HTML email to Ruby Talk.)


    harp:~ > echo '<html><body><b>__html_2_text?__</b></body></html>' | links -dump -stdin
    __html_2_text?__

    ??

    > * Our Gateway is an NNTP <-> email Gateway. There has been at least one
    > instance of a Usenet post using ancient header formatting predating NNTP.
    > These messages are not supported.


    check.


    > I will address individual issues, as long as they are aware of all four
    > points above.


    so... i have this rash... ;-)

    -a
    --
    if you want others to be happy, practice compassion.
    if you want to be happy, practice compassion. -- the dalai lama
     
    , Dec 1, 2006
    #8
  9. On Dec 1, 2006, at 9:39 AM, wrote:

    >> * I now have very detailed logs on everything the Gateway does, so
    >> please be specific. For example, tell me which messages are
    >> missing, not just that they are.

    >
    > can't we automate this? monitor/scraper?


    Probably not too meaningfully. Every Usenet host uses a different
    system for numbering posts, so the numbers wouldn't be much help to
    the outside world.

    Besides, now that I understand the Gateway code intimately, I'm
    hoping to get it robust enough to make problems rare.

    >> * Our Usenet host does not allow us to post HTML emails. This is
    >> not changing. (I'm not bothered by this because I do not believe
    >> anyone should be sending HTML email to Ruby Talk.)

    >
    > harp:~ > echo '<html><body><b>__html_2_text?__</b></body></html>'
    > | links -dump -stdin
    > __html_2_text?__
    >
    > ??


    I would much rather Ruby Talk rejected HTML emails. ;)

    I think they are pure evil, so it will be hard to convince me to
    spend much effort on them.

    >> I will address individual issues, as long as they are aware of all
    >> four points above.

    >
    > so... i have this rash... ;-)


    <dies laughing>

    OK... Gateway issues!

    James Edward Gray II
     
    James Edward Gray II, Dec 1, 2006
    #9
  10. Trans

    Trans Guest

    James Edward Gray II wrote:
    > On Nov 30, 2006, at 9:22 PM, Trans wrote:
    >
    > > Someone recently mentioned missing posts. I just did a comparsion of
    > > comp.lang.ruby and ruby-talk-google and indeed the usenet is missing
    > > posts.

    >
    > Here's the information I have on this currently:
    >
    > * I rewrote the entire Gateway this last week, so anything before
    > that is history. The news-to-email side is already deployed. My
    > email-to-news code is complete, but not deployed. (Our SysAdmin will
    > get to it soon now.) If you raise issues, please make sure they are
    > recent.
    > * I now have very detailed logs on everything the Gateway does, so
    > please be specific. For example, tell me which messages are missing,
    > not just that they are.
    > * Our Usenet host does not allow us to post HTML emails. This is not
    > changing. (I'm not bothered by this because I do not believe anyone
    > should be sending HTML email to Ruby Talk.)
    > * Our Gateway is an NNTP <-> email Gateway. There has been at least
    > one instance of a Usenet post using ancient header formatting
    > predating NNTP. These messages are not supported.


    Have a look at recent topic: Wisdom of including Rakefile in releases.
    The 2nd post from Jason Roelofs appears to be missing.

    T.
     
    Trans, Dec 1, 2006
    #10
  11. On 01.12.2006 17:05, Trans wrote:
    > James Edward Gray II wrote:
    >> On Nov 30, 2006, at 9:22 PM, Trans wrote:
    >>
    >>> Someone recently mentioned missing posts. I just did a comparsion of
    >>> comp.lang.ruby and ruby-talk-google and indeed the usenet is missing
    >>> posts.

    >> Here's the information I have on this currently:
    >>
    >> * I rewrote the entire Gateway this last week, so anything before
    >> that is history. The news-to-email side is already deployed. My
    >> email-to-news code is complete, but not deployed. (Our SysAdmin will
    >> get to it soon now.) If you raise issues, please make sure they are
    >> recent.
    >> * I now have very detailed logs on everything the Gateway does, so
    >> please be specific. For example, tell me which messages are missing,
    >> not just that they are.
    >> * Our Usenet host does not allow us to post HTML emails. This is not
    >> changing. (I'm not bothered by this because I do not believe anyone
    >> should be sending HTML email to Ruby Talk.)
    >> * Our Gateway is an NNTP <-> email Gateway. There has been at least
    >> one instance of a Usenet post using ancient header formatting
    >> predating NNTP. These messages are not supported.

    >
    > Have a look at recent topic: Wisdom of including Rakefile in releases.
    > The 2nd post from Jason Roelofs appears to be missing.


    I do not see *any* posting of Jason on the news side and only a single
    posting on the email side:

    <quote>
    Rakefiles allow users to run the tests for themselves, to understand how a
    library is put together, etc. The only thing that can go wrong is that the
    stuff doesn't work (unless you're grabbing very untrustworthy packages, but
    assumptions have to be made somewhere). Not to mention, having the Rakefile
    there allows other people to properly make changes to a library if / when
    they need to.

    I don't see anything bad with including said Rakefiles. Do you have any
    specific worries?
    </quote>

    Cheers

    robert
     
    Robert Klemme, Dec 1, 2006
    #11
  12. Andreas Schwarz, Dec 1, 2006
    #12
  13. On Dec 1, 2006, at 10:35 AM, Robert Klemme wrote:

    > On 01.12.2006 17:05, Trans wrote:
    >>
    >> Have a look at recent topic: Wisdom of including Rakefile in
    >> releases.
    >> The 2nd post from Jason Roelofs appears to be missing.


    I have not yet received this message from Trans. I do find it in the
    mailing list archives though and I see it on comp.lang.ruby. I'll
    assume my host blocked it, ruby talk is hiccuping, or something
    because the Gateway logs show it was forwarded.

    > I do not see *any* posting of Jason on the news side and only a
    > single posting on the email side:


    Right. I'll assume we are talking about this message:

    http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/227459

    The log says:

    <> - Re: Wisdom of including Rakefile in releases
    11/30/06-15:21:43: mail2news | nntp.post() result = 441 437 HTML post

    Translation: Our NNTP host rejected the HTML post.

    James Edward Gray II
     
    James Edward Gray II, Dec 1, 2006
    #13
  14. On 01.12.2006 18:30, James Edward Gray II wrote:
    > On Dec 1, 2006, at 10:35 AM, Robert Klemme wrote:
    >
    >> On 01.12.2006 17:05, Trans wrote:
    >>>
    >>> Have a look at recent topic: Wisdom of including Rakefile in releases.
    >>> The 2nd post from Jason Roelofs appears to be missing.

    >
    > I have not yet received this message from Trans. I do find it in the
    > mailing list archives though and I see it on comp.lang.ruby. I'll
    > assume my host blocked it, ruby talk is hiccuping, or something because
    > the Gateway logs show it was forwarded.
    >
    >> I do not see *any* posting of Jason on the news side and only a single
    >> posting on the email side:

    >
    > Right. I'll assume we are talking about this message:
    >
    > http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/227459
    >
    > The log says:
    >
    > <> - Re: Wisdom of including Rakefile in releases
    > 11/30/06-15:21:43: mail2news | nntp.post() result = 441 437 HTML post
    >
    > Translation: Our NNTP host rejected the HTML post.


    I cannot see that it is a HTML posting - in fact, I do not see any HTML
    in that posting. Is it possible that the gateway rejects multipart
    messages? James, if you need the source (or what reached my gmail box)
    let me know and I forward it to you.

    Kind regards

    robert
     
    Robert Klemme, Dec 1, 2006
    #14
  15. On Dec 1, 2006, at 4:20 PM, Robert Klemme wrote:

    > On 01.12.2006 18:30, James Edward Gray II wrote:
    >> On Dec 1, 2006, at 10:35 AM, Robert Klemme wrote:
    >>> I do not see *any* posting of Jason on the news side and only a
    >>> single posting on the email side:

    >> Right. I'll assume we are talking about this message:
    >> http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/227459
    >> The log says:
    >> <> - Re: Wisdom of including Rakefile in
    >> releases
    >> 11/30/06-15:21:43: mail2news | nntp.post() result = 441 437 HTML
    >> post
    >> Translation: Our NNTP host rejected the HTML post.

    >
    > I cannot see that it is a HTML posting - in fact, I do not see any
    > HTML in that posting. Is it possible that the gateway rejects
    > multipart messages? James, if you need the source (or what reached
    > my gmail box) let me know and I forward it to you.


    Fred, can you comment on this? The link above shows pretty much how
    the rejected message was formatted.

    James Edward Gray II
     
    James Edward Gray II, Dec 1, 2006
    #15
  16. Trans

    F. Senault Guest

    Le 1 décembre 2006 à 23:30, James Edward Gray II a écrit :

    > On Dec 1, 2006, at 4:20 PM, Robert Klemme wrote:


    >>> http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/227459


    >> I cannot see that it is a HTML posting - in fact, I do not see any
    >> HTML in that posting. Is it possible that the gateway rejects
    >> multipart messages? James, if you need the source (or what reached
    >> my gmail box) let me know and I forward it to you.


    I'd like to see the whole raw message, headers included (my email is
    valid, please attach the whole message, (g)zipped, or it will be bounced
    by my _personal_ filters).

    > Fred, can you comment on this? The link above shows pretty much how
    > the rejected message was formatted.


    Are you sure the mailing-list archive software doesn't filter out HTML ?
    A multipart message with only one part sounds quite unlikely to me.

    James, maybe the gateway should save the rejected messages somewhere, if
    only temporarily, don't you think ?

    Anyway, I'm using cleanfeed on the server, an omnipresent filtering
    package. It's usually quire reliable, even if it has to use some
    complicated heuristics from time to time (since MIME isn't widely used
    on Usenet). It's a quite standard version, with a bit of tweaking, but
    not in that domain.

    Why am I telling you all of this, you ask ? Well, even if it is a false
    positive, if I tune my filters to let the message pass, it will anyway
    have a really bad distribution, being rejected by most of the hosts that
    run the same soft.

    > James Edward Gray II


    Fred
    --
    Sometimes I sleep, sometimes it's not for days The people I meet
    always go their separate ways Sometimes you tell the day By the
    bottle that you drink And times when you're all alone all you do is
    think (Bon Jovi, Wanted Dead or Alive)
     
    F. Senault, Dec 2, 2006
    #16
  17. Trans

    F. Senault Guest

    Le 2 décembre 2006 à 11:16, F. Senault a écrit :

    > Le 1 décembre 2006 à 23:30, James Edward Gray II a écrit :
    >
    >> On Dec 1, 2006, at 4:20 PM, Robert Klemme wrote:

    >
    >>>> http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/227459

    >
    >>> I cannot see that it is a HTML posting - in fact, I do not see any
    >>> HTML in that posting. Is it possible that the gateway rejects
    >>> multipart messages? James, if you need the source (or what reached
    >>> my gmail box) let me know and I forward it to you.

    >
    > I'd like to see the whole raw message, headers included (my email is
    > valid, please attach the whole message, (g)zipped, or it will be bounced
    > by my _personal_ filters).


    Ok, Robert sent me what seemed to be the full message, and it's in fact
    a message declared with a MIME content-type of multipart/alternative but
    only one part (text/plain). This is quite absurd, and I'd suspect that
    something, somewhere, decided to strip away the HTML part but leave the
    MIME declaration intact.

    And, of course, cleanfeed declared the message as HTML on the basis of
    the headers...

    Fred
    (Not sure of my English on this one...)
    --
    Broken bottles under children's feets Bodies strewn across a dead end
    street But I won't heed the battle call It puts my back up, puts
    my back up against the wall
    (U2, Sunday Bloody Sunday)
     
    F. Senault, Dec 2, 2006
    #17
  18. On Dec 2, 2006, at 5:50 AM, F. Senault wrote:

    > Le 2 d=E9cembre 2006 =E0 11:16, F. Senault a =E9crit :
    >
    >> Le 1 d=E9cembre 2006 =E0 23:30, James Edward Gray II a =E9crit :
    >>
    >>> On Dec 1, 2006, at 4:20 PM, Robert Klemme wrote:

    >>
    >>>>> http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/227459

    >>
    >>>> I cannot see that it is a HTML posting - in fact, I do not see any
    >>>> HTML in that posting. Is it possible that the gateway rejects
    >>>> multipart messages? James, if you need the source (or what reached
    >>>> my gmail box) let me know and I forward it to you.

    >>
    >> I'd like to see the whole raw message, headers included (my email is
    >> valid, please attach the whole message, (g)zipped, or it will be =20
    >> bounced
    >> by my _personal_ filters).

    >
    > Ok, Robert sent me what seemed to be the full message, and it's in =20
    > fact
    > a message declared with a MIME content-type of multipart/=20
    > alternative but
    > only one part (text/plain). This is quite absurd, and I'd suspect =20
    > that
    > something, somewhere, decided to strip away the HTML part but leave =20=


    > the
    > MIME declaration intact.


    Forgive my weak knowledge of email types here, but if someone sent a =20
    short message to the list with an attached Ruby script, would it be a =20=

    multipart message? If so, that would be rejected by the NNTP host?

    James
     
    James Edward Gray II, Dec 2, 2006
    #18
  19. Trans

    Ross Bamford Guest

    On Sun, 2006-12-03 at 01:10 +0900, James Edward Gray II wrote:
    > On Dec 2, 2006, at 5:50 AM, F. Senault wrote:
    >=20
    > > Le 2 d=C3=A9cembre 2006 =C3=A0 11:16, F. Senault a =C3=A9crit :
    > >
    > >> Le 1 d=C3=A9cembre 2006 =C3=A0 23:30, James Edward Gray II a =C3=A9cri=

    t :
    > >>
    > >>> On Dec 1, 2006, at 4:20 PM, Robert Klemme wrote:
    > >>
    > >>>>> http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/227459
    > >>
    > >>>> I cannot see that it is a HTML posting - in fact, I do not see any
    > >>>> HTML in that posting. Is it possible that the gateway rejects
    > >>>> multipart messages? James, if you need the source (or what reached
    > >>>> my gmail box) let me know and I forward it to you.
    > >>
    > >> I'd like to see the whole raw message, headers included (my email is
    > >> valid, please attach the whole message, (g)zipped, or it will be =20
    > >> bounced
    > >> by my _personal_ filters).

    > >
    > > Ok, Robert sent me what seemed to be the full message, and it's in =20
    > > fact
    > > a message declared with a MIME content-type of multipart/=20
    > > alternative but
    > > only one part (text/plain). This is quite absurd, and I'd suspect =20
    > > that
    > > something, somewhere, decided to strip away the HTML part but leave =20
    > > the
    > > MIME declaration intact.

    >=20
    > Forgive my weak knowledge of email types here, but if someone sent a =20
    > short message to the list with an attached Ruby script, would it be a =20
    > multipart message? If so, that would be rejected by the NNTP host?


    I have a feeling this is what is, or at least was, happening - I wasn't
    going to say anything (since it was prior to December first) but I
    noticed that posts with attachments never made it across.=20

    ruby-talk 226884, for example, doesn't appear to have made it to the
    newsgroup.

    --=20
    Ross Bamford -
     
    Ross Bamford, Dec 2, 2006
    #19
  20. Trans

    F. Senault Guest

    Le 2 décembre 2006 à 17:10, James Edward Gray II a écrit :

    > On Dec 2, 2006, at 5:50 AM, F. Senault wrote:
    >
    >> Le 2 décembre 2006 à 11:16, F. Senault a écrit :


    >>> I'd like to see the whole raw message, headers included (my email is
    >>> valid, please attach the whole message, (g)zipped, or it will be
    >>> bounced by my _personal_ filters).

    >>
    >> Ok, Robert sent me what seemed to be the full message, and it's in
    >> fact a message declared with a MIME content-type of multipart/
    >> alternative but only one part (text/plain). This is quite absurd,
    >> and I'd suspect that something, somewhere, decided to strip away
    >> the HTML part but leave the MIME declaration intact.

    >
    > Forgive my weak knowledge of email types here, but if someone sent a
    > short message to the list with an attached Ruby script, would it be a
    > multipart message? If so, that would be rejected by the NNTP host?


    Yes, but it won't be multipart/alternative (which means different
    representations of the same data, nearly universally HTML + plain text),
    more likely multipart/mixed (simply different parts).

    For instance, with a quick search, I found this one :

    http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/227642

    Which has been gated correctly, as it has :

    Content-Type: multipart/mixed; boundary="sm4nu43k4a2Rpi4c"

    (But it lacks the MIME-Version header, which is, AFAICR, mandatory, if
    quite useless...)

    > James


    Fred
    --
    Do you know how far this has gone ? Just how damaged have I become ?
    When I think I can overcome It runs even deeper
    Everything that matters is gone All the hands of hope have withdrawn
    Could you try to help me hang on ? (Nine Inch Nails, Even Deeper)
     
    F. Senault, Dec 2, 2006
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John
    Replies:
    0
    Views:
    331
  2. Sameer
    Replies:
    3
    Views:
    392
  3. John
    Replies:
    0
    Views:
    1,059
  4. John
    Replies:
    0
    Views:
    1,065
  5. vimal
    Replies:
    8
    Views:
    106
    beegee
    Jul 25, 2008
Loading...

Share This Page