FPGA to solve the two most annoying problems on usenet - SuggestionsWelcome

Discussion in 'VHDL' started by Charles Xavier, Jun 11, 2008.

  1. As you all know, downloading files from usenet leaves you with two
    sets of files.. The rar files from what you're downloading and the
    par2 files for incomplete file repair.

    If anyone has attempted to download anything in the 8GB range, you'll
    find that well.. if you're missing enough parts of the file, the par2
    recovery can be a painful, painful process taking up to three hours in
    some cases.

    I'm sick of it.

    So, here's the idea. Use a FPGA to do the reed-solomon decoding to
    accelerate the PAR2 repair/recovery process. The system should utilize
    a USB connection to pipe data directly from the disk to the FPGA that
    will do the offboard processing of the data. The data transfer should
    be controlled by an application on the computer.

    Second Problem..

    The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
    Converting from x.264 to h.264 could be done offboard on an FPGA
    because it takes for-ever to complete on my system (8 hours). This
    should have the same premise as the previous issue, minus using a x.
    264 decoding core and possibly directly converting it to h.264 or
    doing a decompression-recompression..

    SPECS: The development system i'm using is the XILINX ML-505 board
    with the Virtex 5 chip. This is a open-source project being done for
    fun and learning btw.

    Suggestions / Comments / Complaints?
    Charles Xavier, Jun 11, 2008
    #1
    1. Advertising

  2. Charles Xavier

    Atilla Filiz Guest

    Re: FPGA to solve the two most annoying problems on usenet -SuggestionsWelcome

    Seems like a really nice idea. Please notify your progress.
    Atilla Filiz, Jun 11, 2008
    #2
    1. Advertising

  3. Charles Xavier

    Jon Beniston Guest

    Re: FPGA to solve the two most annoying problems on usenet -Suggestions Welcome


    > The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
    > Converting from x.264 to h.264 could be done offboard on an FPGA
    > because it takes for-ever to complete on my system (8 hours). This
    > should have the same premise as the previous issue, minus using a x.
    > 264 decoding core and possibly directly converting it to h.264 or
    > doing a decompression-recompression..


    There are chips that will do this already. Here's one:

    http://www.aspex-semi.com/q/product_features.shtml

    Cheers,
    Jon
    Jon Beniston, Jun 11, 2008
    #3
  4. Re: FPGA to solve the two most annoying problems on usenet -Suggestions Welcome

    On Jun 11, 11:24 am, Jon Beniston <> wrote:
    > > The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
    > > Converting from x.264 to h.264 could be done offboard on an FPGA
    > > because it takes for-ever to complete on my system (8 hours). This
    > > should have the same premise as the previous issue, minus using a x.
    > > 264 decoding core and possibly directly converting it to h.264 or
    > > doing a decompression-recompression..

    >
    > There are chips that will do this already. Here's one:
    >
    > http://www.aspex-semi.com/q/product_features.shtml
    >
    > Cheers,
    > Jon


    Thanks for the reply. Aspects of this design can certainly be used and
    incorporated into this design, but as you know.. Their product isn't
    free/open source. As this is also a learning project, I think it would
    be better to build it from the ground up (using some pre-existing open
    source IP if possible).
    Charles Xavier, Jun 11, 2008
    #4
  5. Charles Xavier

    PFC Guest

    Re: FPGA to solve the two most annoying problems on usenet - Suggestions Welcome


    > So, here's the idea. Use a FPGA to do the reed-solomon decoding to
    > accelerate the PAR2 repair/recovery process. The system should utilize
    > a USB connection to pipe data directly from the disk to the FPGA that
    > will do the offboard processing of the data. The data transfer should
    > be controlled by an application on the computer.


    Why USB ? Gigabit Ethernet would be much faster ;)
    Don't tell me a Virtex5 can't handle a measly 100 MB/s ;)

    And... is it a PCI express slot that I see there ?
    That's even better !

    > Second Problem..
    >
    > The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
    > Converting from x.264 to h.264 could be done offboard on an FPGA
    > because it takes for-ever to complete on my system (8 hours). This
    > should have the same premise as the previous issue, minus using a x.
    > 264 decoding core and possibly directly converting it to h.264 or
    > doing a decompression-recompression..


    I was under the impression that x264 was h264 in disguise. Perhaps
    tweaking the file format would be enough ?

    Also you could recode the thing in a much simpler codec like MPEG2 or
    something, who cares if it takes 10x the size, just stream it in realtime
    ;)

    Stick a DVI port on your FPGA !

    > SPECS: The development system i'm using is the XILINX ML-505 board
    > with the Virtex 5 chip. This is a open-source project being done for
    > fun and learning btw.
    PFC, Jun 11, 2008
    #5
  6. Re: FPGA to solve the two most annoying problems on usenet -Suggestions Welcome

    On Jun 13, 5:46 pm, Eric Smith <> wrote:
    > Charles Xavier wrote:
    > > If anyone has attempted to download anything in the 8GB range, you'll
    > > find that well.. if you're missing enough parts of the file, the par2
    > > recovery can be a painful, painful process taking up to three hours in
    > > some cases.
    > > The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
    > > Converting from x.264 to h.264 could be done offboard on an FPGA

    >
    > I have a hard time believing that those are the "two most annoying problems
    > on usenet".  I've been using Usenet since 1984, and I've never personally
    > encountered either problem.
    >
    > I've never tried to download an 8GB file from Usenet, and don't have any
    > clue why anyone would want to do such a thing.  There are much better
    > ways to distributed 8GB files.  But if I *did* want to download an 8GB
    > file from Usenet, I can't imagine that waiting three hours for a
    > reconstruction of missing pieces would really bother me that much.
    > There's never been a single thing that I've needed from Usenet so
    > urgently that it couldn't have waited three more hours.
    >
    > I watch a fair number of movies, but I've never had any problem with
    > x.264.  It sounds like you just need better player software.  That has
    > nothing to do with Usenet.
    >
    > Since it doesn't appear that there is any real problem here, it also
    > doesn't appear that there is any need for an FPGA-based "solution".


    Thank you for sharing. Let your next post be in contribution to the
    project.

    If it doesn't apply to you and you don't see the need, why even reply?
    Charles Xavier, Jun 14, 2008
    #6
  7. Charles Xavier

    Symon Guest

    Re: FPGA to solve the two most annoying problems on usenet - Suggestions Welcome

    Charles Xavier wrote:
    > On Jun 13, 5:46 pm, Eric Smith <> wrote:
    >> Charles Xavier wrote:
    >>> If anyone has attempted to download anything in the 8GB range,
    >>> you'll find that well.. if you're missing enough parts of the file,
    >>> the par2 recovery can be a painful, painful process taking up to
    >>> three hours in some cases.
    >>> The XBOX 360 doesn't play x.264 and all the good movies are in
    >>> x.264. Converting from x.264 to h.264 could be done offboard on an
    >>> FPGA

    >>
    >> I have a hard time believing that those are the "two most annoying
    >> problems on usenet". I've been using Usenet since 1984, and I've
    >> never personally encountered either problem.
    >>
    >>
    >> Since it doesn't appear that there is any real problem here, it also
    >> doesn't appear that there is any need for an FPGA-based "solution".

    >
    > Thank you for sharing. Let your next post be in contribution to the
    > project.
    >
    > If it doesn't apply to you and you don't see the need, why even reply?


    Hi Charles,
    Maybe to share the benefit of his experience to help you avoid wasting your
    time on a solution without a problem?
    Good luck, Syms.
    Symon, Jun 14, 2008
    #7
  8. Re: FPGA to solve the two most annoying problems on usenet -Suggestions Welcome

    On Jun 14, 12:39 pm, "Symon" <> wrote:
    > Charles Xavier wrote:
    > > On Jun 13, 5:46 pm, Eric Smith <> wrote:
    > >> Charles Xavier wrote:
    > >>> If anyone has attempted to download anything in the 8GB range,
    > >>> you'll find that well.. if you're missing enough parts of the file,
    > >>> the par2 recovery can be a painful, painful process taking up to
    > >>> three hours in some cases.
    > >>> The XBOX 360 doesn't play x.264 and all the good movies are in
    > >>> x.264. Converting from x.264 to h.264 could be done offboard on an
    > >>> FPGA

    >
    > >> I have a hard time believing that those are the "two most annoying
    > >> problems on usenet". I've been using Usenet since 1984, and I've
    > >> never personally encountered either problem.

    >
    > >> Since it doesn't appear that there is any real problem here, it also
    > >> doesn't appear that there is any need for an FPGA-based "solution".

    >
    > > Thank you for sharing. Let your next post be in contribution to the
    > > project.

    >
    > > If it doesn't apply to you and you don't see the need, why even reply?

    >
    > Hi Charles,
    > Maybe to share the benefit of his experience to help you avoid wasting your
    > time on a solution without a problem?
    > Good luck, Syms.


    Engineers can often be of a single track mind, so let me restate. This
    is a problem that I am having. I could certainly get a faster
    computer, or try to finesse the algorithms for the decoding, but I
    want to learn about this specific algorithm implementation on an FPGA,
    so I'm using the FPGA to solve this problem (that is a problem for
    me).

    To me, this is a practical solution and an excellent gateway for
    myself and others who want to get into FPGA-based hardware
    acceleration of software algorithms. You give me a better way that
    solves a practical problem for _me_ and I'll take it.

    This reminds of the idiots at In-N-Out Burger who created the drive-
    through fast food restaurant (they claim). It doesn't affect the
    geriatric who only wants to eat inside because they can't imagine
    driving while eating. But apparently it turned out to be a good idea.

    Responding to this type of feedback is the real waste of time.
    Charles Xavier, Jun 16, 2008
    #8
  9. Re: FPGA to solve the two most annoying problems on usenet - Suggestions Welcome

    On 2008-06-11, Charles Xavier <> wrote:
    > The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
    > Converting from x.264 to h.264 could be done offboard on an FPGA
    > because it takes for-ever to complete on my system (8 hours). This
    > should have the same premise as the previous issue, minus using a x.
    > 264 decoding core and possibly directly converting it to h.264 or
    > doing a decompression-recompression..


    I'm not sure what the problem with x.264 is, but lets just say that
    it is far from trivial to write a H.264 decoder/encoder. The decoder
    is especially tricky as you need to be able to parse all parts of the
    standard whereas you can select only the parts you care about for the
    encoder.

    If you are an expert on video codecs and a newcomer to FPGAs, this
    would probably be a quite ambitiouis project to start with. If you
    are not an expert on video codecs, you will have to spend a lot of
    time to understand H.264 before you can even get started on the FPGA
    parts.

    This is written from personal experience by the way, I have written
    a grayscale H.264 decoder in C to get an understanding of how the
    standard works. Lets just say that it took way more time than I wanted
    to...

    In summary: I don't recommend a H.264 codec as a first FPGA project.

    The Reed-Solomon application seems like a much more reasonable project.
    You will still have a lot of things to figure out like how you are going
    to communicate with your FPGA board, but the problem itself is much more
    limited.

    /Andreas
    Andreas Ehliar, Jun 16, 2008
    #9
  10. Charles Xavier

    Aiken Guest

    Re: FPGA to solve the two most annoying problems on usenet -Suggestions Welcome

    I have an idea simular or not to your mind..(but I still
    thinking....hope you are now starting ..your project).
    my project is that

    FILE COMPRESSION (zip or other)
    read files from SD card, then compress it, and write a new zipped file
    back to SD card.

    it will not include the PC to do it.

    On Jun 11, 9:50 am, Charles Xavier <> wrote:
    > As you all know, downloading files from usenet leaves you with two
    > sets of files.. The rar files from what you're downloading and the
    > par2 files for incomplete file repair.
    >
    > If anyone has attempted to download anything in the 8GB range, you'll
    > find that well.. if you're missing enough parts of the file, the par2
    > recovery can be a painful, painful process taking up to three hours in
    > some cases.
    >
    > I'm sick of it.
    >
    > So, here's the idea. Use a FPGA to do the reed-solomon decoding to
    > accelerate the PAR2 repair/recovery process. The system should utilize
    > a USB connection to pipe data directly from the disk to the FPGA that
    > will do the offboard processing of the data. The data transfer should
    > be controlled by an application on the computer.
    >
    > Second Problem..
    >
    > The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
    > Converting from x.264 to h.264 could be done offboard on an FPGA
    > because it takes for-ever to complete on my system (8 hours). This
    > should have the same premise as the previous issue, minus using a x.
    > 264 decoding core and possibly directly converting it to h.264 or
    > doing a decompression-recompression..
    >
    > SPECS: The development system i'm using is the XILINX ML-505 board
    > with the Virtex 5 chip. This is a open-source project being done for
    > fun and learning btw.
    >
    > Suggestions / Comments / Complaints?
    Aiken, Jun 16, 2008
    #10
  11. Re: FPGA to solve the two most annoying problems on usenet - SuggestionsWelcome

    Charles Xavier wrote:

    > Responding to this type of feedback is the real waste of time.


    "Suggestions / Comments / Complaints?"
    Mike Treseler, Jun 16, 2008
    #11
  12. Charles Xavier

    Arash Partow Guest

    Re: FPGA to solve the two most annoying problems on usenet -Suggestions Welcome

    Consider using rateless codes, reed-solomon and ldpc based codes do
    provide erasure correction capabilities, however rateless codes such
    those based on LT class of codes (tornado etc) ,or more generally
    digital fountain codes, will have better performance due to their near
    linear decoding complexity as opposed to say reed-solomon erasure only
    decoding complexity which is around O(n^2) for naive implementations.



    Arash Partow
    __________________________________________________
    Be one who knows what they don't know,
    Instead of being one who knows not what they don't know,
    Thinking they know everything about all things.
    http://www.partow.net


    On Jun 11, 11:50 pm, Charles Xavier <> wrote:
    > As you all know, downloading files from usenet leaves you with two
    > sets of files.. The rar files from what you're downloading and the
    > par2 files for incomplete file repair.
    >
    > If anyone has attempted to download anything in the 8GB range, you'll
    > find that well.. if you're missing enough parts of the file, the par2
    > recovery can be a painful, painful process taking up to three hours in
    > some cases.
    >
    > I'm sick of it.
    >
    > So, here's the idea. Use a FPGA to do thereed-solomondecoding to
    > accelerate the PAR2 repair/recovery process. The system should utilize
    > a USB connection to pipe data directly from the disk to the FPGA that
    > will do the offboard processing of the data. The data transfer should
    > be controlled by an application on the computer.
    >
    > Second Problem..
    >
    > The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
    > Converting from x.264 to h.264 could be done offboard on an FPGA
    > because it takes for-ever to complete on my system (8 hours). This
    > should have the same premise as the previous issue, minus using a x.
    > 264 decoding core and possibly directly converting it to h.264 or
    > doing a decompression-recompression..
    >
    > SPECS: The development system i'm using is the XILINX ML-505 board
    > with the Virtex 5 chip. This is a open-source project being done for
    > fun and learning btw.
    >
    > Suggestions / Comments / Complaints?
    Arash Partow, Jun 18, 2008
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Michael Hopkins

    Most annoying aspects of C++

    Michael Hopkins, Jun 15, 2006, in forum: C++
    Replies:
    52
    Views:
    1,680
    Kevin Handy
    Jun 22, 2006
  2. Vikram
    Replies:
    0
    Views:
    812
    Vikram
    Jul 24, 2008
  3. TheRightInfo

    FPGA BOARD FOR NEWBIE TO FPGA

    TheRightInfo, Feb 2, 2011, in forum: VHDL
    Replies:
    1
    Views:
    1,340
    Tricky
    Feb 2, 2011
  4. Oliver Mattos

    Re: FPGA BOARD FOR NEWBIE TO FPGA

    Oliver Mattos, Feb 2, 2011, in forum: VHDL
    Replies:
    0
    Views:
    1,129
    Oliver Mattos
    Feb 2, 2011
  5. Replies:
    3
    Views:
    85
Loading...

Share This Page