How to design a file system to store uploaded files via ASP.NET

Discussion in 'ASP .Net' started by RedHair, Feb 12, 2006.

  1. RedHair

    RedHair Guest

    I'd like to set up a file system for the ASP.NET 2.0 application
    to store user-uploaded files, since the members are more than
    100,000 people, the basic requirements are as below:

    (1) The file system is separate with front-end web site
    (2) Need to re-size user-uploaded image file to same size
    (3) Need to rename file name to avoid duplicate name
    (4) How to design directory and file structure to increase disk I/O
    performance
    (5) How to scale the file system by adding more HD or file server
    (6) What kind of file server? Window 2003 or Linux

    Any suggestion? thanks in advance.
     
    RedHair, Feb 12, 2006
    #1
    1. Advertising

  2. RedHair

    Alec MacLean Guest

    RedHair,

    (1) I would recommend that you don't permit direct access to the filesystem.
    Perhaps consider using a database (e.g. SQL Server) to store the files
    instead. If you wanted direct file system access, you would need to
    configure special permissions to allow the web app security the correct
    context for writing to the web server disks. This can pose a big security
    risk, no matter which OS is used.

    Using SQL Server datatype "image" (sometimes referred to as BLOB, from
    Binary Large OBject), you can have up to a 2Gb file in each record, so
    there's plenty of storage potential.

    (2 / 3 / 4) Using a DB would also remove problems 2, 3 and 4. For (3) you
    would use a unique key (primary key) for each file record created in the
    storage table. This means you can have duplicate file names if necessary
    without conflict. You would need to consider adding a supplemental column
    to the table to hold some other value that would allow users to distinguish
    the files if same name used. This could be the UserID/name of the person
    uploading, or the date-time stamp of the upload, or use both for example.

    For point 2, do you mean a file that is a photo, or any type of file such as
    a document/spreadsheet?

    (5) If you use a hot-swap capable RAID based server or NAS, you should be
    able to add more / larger disks and grow your RAID partition into the new
    space.

    (1 / 6) Since you're posting in the dotnet forums, we can probably assume
    you're at least considering to use .NET to create this app, so you'll
    probably be wanting a Windows host server. Windows Server 2003 Web Edition
    is perfect for this as it is much cheaper than Standard (or Enterprise)
    versions and you don't need to buy separate licences. Also, this can host
    an MSDE version of SQL server if required to keep infrastructure costs down
    to bare minimum, though with your stated user audience, you should probably
    consider a separate dedicated DB server. (E.g. SQL Server with CPU
    licence(s) for Internet use).

    Hope that helps.

    Al


    "RedHair" <> wrote in message
    news:...
    > I'd like to set up a file system for the ASP.NET 2.0 application
    > to store user-uploaded files, since the members are more than
    > 100,000 people, the basic requirements are as below:
    >
    > (1) The file system is separate with front-end web site
    > (2) Need to re-size user-uploaded image file to same size
    > (3) Need to rename file name to avoid duplicate name
    > (4) How to design directory and file structure to increase disk I/O
    > performance
    > (5) How to scale the file system by adding more HD or file server
    > (6) What kind of file server? Window 2003 or Linux
    >
    > Any suggestion? thanks in advance.
    >
     
    Alec MacLean, Feb 12, 2006
    #2
    1. Advertising

  3. RedHair

    RedHair Guest

    Thanks for your detailed reply.
    Use db instead of file system is the original idea, but many articles
    mentioned
    that read/write binary from db costs the performance.
    Besides, it is possible to put db generated image file into the cache either
    in server or client side?

    "Alec MacLean" <> ¼¶¼g©ó¶l¥ó·s»D:e9vN$...
    > RedHair,
    >
    > (1) I would recommend that you don't permit direct access to the
    > filesystem. Perhaps consider using a database (e.g. SQL Server) to store
    > the files instead. If you wanted direct file system access, you would
    > need to configure special permissions to allow the web app security the
    > correct context for writing to the web server disks. This can pose a big
    > security risk, no matter which OS is used.
    >
    > Using SQL Server datatype "image" (sometimes referred to as BLOB, from
    > Binary Large OBject), you can have up to a 2Gb file in each record, so
    > there's plenty of storage potential.
    >
    > (2 / 3 / 4) Using a DB would also remove problems 2, 3 and 4. For (3) you
    > would use a unique key (primary key) for each file record created in the
    > storage table. This means you can have duplicate file names if necessary
    > without conflict. You would need to consider adding a supplemental column
    > to the table to hold some other value that would allow users to
    > distinguish the files if same name used. This could be the UserID/name of
    > the person uploading, or the date-time stamp of the upload, or use both
    > for example.
    >
    > For point 2, do you mean a file that is a photo, or any type of file such
    > as a document/spreadsheet?
    >
    > (5) If you use a hot-swap capable RAID based server or NAS, you should be
    > able to add more / larger disks and grow your RAID partition into the new
    > space.
    >
    > (1 / 6) Since you're posting in the dotnet forums, we can probably assume
    > you're at least considering to use .NET to create this app, so you'll
    > probably be wanting a Windows host server. Windows Server 2003 Web
    > Edition is perfect for this as it is much cheaper than Standard (or
    > Enterprise) versions and you don't need to buy separate licences. Also,
    > this can host an MSDE version of SQL server if required to keep
    > infrastructure costs down to bare minimum, though with your stated user
    > audience, you should probably consider a separate dedicated DB server.
    > (E.g. SQL Server with CPU licence(s) for Internet use).
    >
    > Hope that helps.
    >
    > Al
    >
    >
    > "RedHair" <> wrote in message
    > news:...
    >> I'd like to set up a file system for the ASP.NET 2.0 application
    >> to store user-uploaded files, since the members are more than
    >> 100,000 people, the basic requirements are as below:
    >>
    >> (1) The file system is separate with front-end web site
    >> (2) Need to re-size user-uploaded image file to same size
    >> (3) Need to rename file name to avoid duplicate name
    >> (4) How to design directory and file structure to increase disk I/O
    >> performance
    >> (5) How to scale the file system by adding more HD or file server
    >> (6) What kind of file server? Window 2003 or Linux
    >>
    >> Any suggestion? thanks in advance.
    >>

    >
    >
     
    RedHair, Feb 13, 2006
    #3
  4. RedHair

    Alec MacLean Guest

    I've not read that deeply into the performance issues myself, not having had
    to produce an app of this type (though one is planned). I'll need to do
    some further reading myself, it seems!

    I have been working on an app that uploads photo images (HR system, staff
    photo) with no discernable speed issues, but this is a Win-forms app used on
    local LAN, so direct comparison is probably not a fair one.

    It seems logical that there would be a performance hit while the binary file
    is "translated" into the stream required to up/down-load, though I don't
    know how much impact that has. Probably varies a lot with hardware being
    used as well the number of user hits per second/minute. You'd need to weigh
    the pro's and cons of the available approaches (between File I/O and DB I/O)
    to judge which will best suit your current project's needs.

    I've not used caching myself as of this time, but I wouldn't see a technical
    barrier with caching at either end. I'm more than happy to be corrected /
    illuminated on this though!

    Al


    "RedHair" <> wrote in message
    news:%...
    > Thanks for your detailed reply.
    > Use db instead of file system is the original idea, but many articles
    > mentioned
    > that read/write binary from db costs the performance.
    > Besides, it is possible to put db generated image file into the cache
    > either
    > in server or client side?
    >
     
    Alec MacLean, Feb 13, 2006
    #4
  5. RedHair

    RedHair Guest

    Thanks!
    "Alec MacLean" <> ¼¶¼g©ó¶l¥ó·s»D:...
    > I've not read that deeply into the performance issues myself, not having
    > had to produce an app of this type (though one is planned). I'll need to
    > do some further reading myself, it seems!
    >
    > I have been working on an app that uploads photo images (HR system, staff
    > photo) with no discernable speed issues, but this is a Win-forms app used
    > on local LAN, so direct comparison is probably not a fair one.
    >
    > It seems logical that there would be a performance hit while the binary
    > file is "translated" into the stream required to up/down-load, though I
    > don't know how much impact that has. Probably varies a lot with hardware
    > being used as well the number of user hits per second/minute. You'd need
    > to weigh the pro's and cons of the available approaches (between File I/O
    > and DB I/O) to judge which will best suit your current project's needs.
    >
    > I've not used caching myself as of this time, but I wouldn't see a
    > technical barrier with caching at either end. I'm more than happy to be
    > corrected / illuminated on this though!
    >
    > Al
    >
    >
    > "RedHair" <> wrote in message
    > news:%...
    >> Thanks for your detailed reply.
    >> Use db instead of file system is the original idea, but many articles
    >> mentioned
    >> that read/write binary from db costs the performance.
    >> Besides, it is possible to put db generated image file into the cache
    >> either
    >> in server or client side?
    >>

    >
    >
     
    RedHair, Feb 21, 2006
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Matt G
    Replies:
    1
    Views:
    1,157
    Deepak Kumar Vasudevan
    Aug 22, 2003
  2. Arsen V.
    Replies:
    9
    Views:
    10,175
    Steve C. Orr [MVP, MCSD]
    Feb 11, 2005
  3. darrel
    Replies:
    1
    Views:
    363
    darrel
    Jun 10, 2005
  4. =?Utf-8?B?Q2hyaXM=?=

    scan files uploaded via asp.net

    =?Utf-8?B?Q2hyaXM=?=, Jun 17, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    489
    Lau Lei Cheong
    Jun 17, 2005
  5. Replies:
    5
    Views:
    5,924
    Arne Vajhøj
    Aug 28, 2008
Loading...

Share This Page