Data files in the App_Data Folder

[wwwmike]

I have an asp.net 2.0 application where I move about 200,000 text files
daily in and out.

When using VS 2005 to debug my application everything works find if
only a few files are in the folder. (The folder is locaed within the
project file). Yesterday I did a stress test were I placed 200,000
files into this folder. As a result I took me about 15 mintues to start
VS 2005 and I was not able to start debugging since the asp.net
compiler somehow kills itself.

I tried to exclude the folder from the project ,like in VS2003,but this
function is not available and probably did not the trick.

Then I moved all files into the app_data folder and everything worked
fine. Unfortunatelly it takes now ages when I try to use the SEARCH
function (over the entire project)


Questions:

1) If the data files are placed directly into the project ( not in the
app_data folder) and new files are always added and deleted, does
asp.net recompile these text files. My asp.worker process is constantly
reccycling (60 times a day).


2) Is the app_data folder the right place to move all these files
(shared hosting)

3) Would it be better if I move out the files of the root directory
(would have to ask my provider to set the asp.net access rights)?

Thanks for helping

Michael

 

[Mark Rae]

I have an asp.net 2.0 application where I move about 200,000 text files
daily in and out.
Wow!

1) If the data files are placed directly into the project ( not in the
app_data folder) and new files are always added and deleted, does
asp.net recompile these text files. My asp.worker process is constantly
reccycling (60 times a day).

Yes, I think so. As I understand it, the App_Data folder is specifically
designed to house file-based databases e.g. SQL Server 2005 Express, Access
etc...

2) Is the app_data folder the right place to move all these files
(shared hosting)

IMO no, for precisely the reasons you describe.

3) Would it be better if I move out the files of the root directory

IMO yes.

(would have to ask my provider to set the asp.net access rights)?

Almost certainly, though it will obviously depend on your ISP.

AAMOI, since you say that these files are text files, have you considered
storing their content in a database rather than as individual files?
Assuming the files themselves are fairly small, I would say that that would
almost guarantee to increase the performance of your website by many
times...

 

[wwwmike]

I dont understand, why is my application running well when I use the
app_data folder , but you say its wrong?

 

[Mark Rae]

I dont understand, why is my application running well when I use the
app_data folder , but you say its wrong?

????? I didn't say it was "wrong" ???

You said that it takes fifteen minutes to do a search, and I suggested that
this would improve dramatically if you were to store the text in a database
in stead of 200,000 individual text files...

 

[wwwmike]

Yes, you are right on this point. Storing the text files in the DB
would solve ALL problems. Unfortunatelly I am limited up to 1GB (shared
hosting).

The thing I dont understand is, when I moved the files into the
app_data directory, I could start the (debug) right away. However when
the files were in a directory right inside the project it took about 5
minutes to compile and endless to start the service.

What is causing this?

 

[Mark Rae]

Yes, you are right on this point. Storing the text files in the DB
would solve ALL problems. Unfortunatelly I am limited up to 1GB (shared
hosting).

Sounds like you're making the classic mistake of letting IT drive your
business instead of the other way round...

If your business would benefit from storing the text files in a database,
but your current ISP doesn't offer you that functionality, then it's time to
find an ISP who does...

The thing I dont understand is, when I moved the files into the
app_data directory, I could start the (debug) right away. However when
the files were in a directory right inside the project it took about 5
minutes to compile and endless to start the service.

What is causing this?

Like I said, App_Data is a "special" folder designed to hold database files.
The asp_net user account needs read and write access on this folder if it
needs to update the contents of the database or if it contains an Access
database (because of the locking .ldb file) , and its contents are not
served by the web server. See MSDN for further details:
http://msdn2.microsoft.com/en-us/library/445z2s49.aspx