Time consuming requests in .NET

[Adrian Dragomirescu]

Hello,

I have the following problem:

In my ASPX page is a button called "Import". The action of this button is to
import a set of data in my database. I have to import big datasets and this
opperation is time consuming (aproximativelly 3-4 hours).
After an hour my page display the error:

"The page cannot be displayed
The page you are looking for is currently unavailable. The Web site might be
experiencing technical difficulties, or you may need to adjust your browser
settings.
.................
................
Cannot find server or DNS Error
Internet Explorer "


But behind the scene my import is still working even i received this error
(i executed statement like "select count(*) from myImportedTable" and the
total number of imported rows still raise even my browser is dispaying this
error).

I modified the following timeout setting to hugh values to hope fix this
problem but without any result

1. in my web.config file
<authentication mode="Forms">
<forms name=".HRPAUTH" loginUrl="login.aspx" protection="All"
timeout="900"></forms>
</authentication>

<sessionState cookieless="false" timeout="1200" mode="InProc"
stateConnectionString="tcpip=127.0.0.1:42424" sqlConnectionString="data
source=127.0.0.1;user id=sa;password=" />

2. in my machine.config file
<processModel
enable="true"
timeout="Infinite"
idleTimeout="Infinite"
shutdownTimeout="0:00:05"
requestLimit="Infinite"
requestQueueLimit="5000"
restartQueueLimit="10"
memoryLimit="100"
webGarden="false"
cpuMask="0xffffffff"
userName="machine"
password="AutoGenerate"
logLevel="Errors"
clientConnectedCheck="0:00:05"
comAuthenticationLevel="Connect"
comImpersonationLevel="Impersonate"
responseDeadlockInterval="10:10:00"
maxWorkerThreads="25"
maxIoThreads="25"
/>

Your help is appreciated!
Thanks

 

[Janaka]

I think your going about this the wrong way. Instead of trying to get the
page's timeout to keep up with the time-intensive operation you should try
to get the page to initially start the process and then have a method to
check the current status of your import. Each time you come to the page
afterwards you will see a progress update using this method call. That way
you could also see how far the import has gotten to, which can be useful
also if you run into errors halfway through an import.

 

[Adrian Dragomirescu]

Thanx for your response Janaka. How can i implement this behavior, to start
the process of import and the same time to receive the postback of my page
(the response from the server)? Can you help me?

 

[Janaka]

Well it depends on what your actual application is going to do. Can you
describe the import process that is currently in place.

 

[Adrian Dragomirescu]

The event of my import button looks like

Private Sub buttonImport_ServerClick(ByVal sender As System.Object,
ByVal e As System.EventArgs) Handles buttonImport.ServerClick
Dim oImportData As ImportData
Try
oImportData = New ImportData(some parameters)
oImportData.StartImport()
Catch ex As Exception
' Do something
Finally
' Do something
End Try
End Sub

So from your explanation i understand that i have to do some modifications
in my code to do StartImport an asyncronous method (i dont know how i can
make this). After that i have to preserve the oImportData object across
multiple requests of this page (probably at Application level) and to use it
to check the status of import process on subsequent request of the import
page.

 

[Janaka]

I don't think it needs to be as complicated as it sounds. Without knowing
more about how your import actually works here's 2 examples of how you could
do something like this.

Let's take a hypothetical function that needs to send out 60,000 emails to a
subscription list

1. Trigger the task to run and then check the level remaining
The StartImport() call could create all the emails and then place these in a
directory for SMTP to pickup. Thus there's no more work to be done on the
page. Then you can use a method like CheckEmails() to see how many are left
to send in the pickup directory.

2. Setup your data into a manageable amount and then recursively process it
In this case StartImport() creates entries in a SQL Table for each email to
send out, with a boolean flag set to false (for whether they have been
sent). Have a separate page to then send the mails out accordingly and
update the database setting the sent flag to true. However this may still
timeout. What I would do then is set the page to process a reasonable
amount without timing out (say 2500) and then refresh/redirect back to
itself to start the process off again.

If its possible I'd go with example1 but its likely you'll need to do
something more like the latter. Hope this gives you a few ideas now.

Janaka

 

[Adrian Dragomirescu]

Thanx for your time janaka. Even though your workaround does not fix my
problem i appreciate your responses.

 

[Janaka]

ok, sorry I couldn't be of any more help. I've found with problems such as
this that instead of trying to get it to prevent the page timing out, you
need to rethink the functionality and design. I'd like to chat some more
but I'm off now to enjoy the weekend!
Take care,

Janaka

 

[Shawn Mehaffie]

Why do you need this application to be an import. Web applications are not
really a great platform to create "Batch Type" jobs that take a long time to
run. Why don't you write this as a fat clinet applications and then
schedule this to run using some sort so scheduling software. We usre Aracan
and have been very happy with the product and it is very reasonalby priced.

If this must be a web application then try using the web.config entry below.

<!-- HTTPRuntime
This section sets the httpRuntime settings of the application.
-->
<httpRuntime executionTimeout="150" maxRequestLength="4096"
useFullyQualifiedRedirectUrl="false" minFreeThreads="8"
minLocalRequestFreeThreads="4" appRequestQueueLimit="100"/>

Thanks,

S. Shawn Mehaffie
(e-mail address removed)

 

[Shawn Mehaffie]

Why do you need this application to be an import. Web applications are not
really a great platform to create "Batch Type" jobs that take a long time to
run. Why don't you write this as a fat clinet applications and then
schedule this to run using some sort so scheduling software. We usre Aracan
and have been very happy with the product and it is very reasonalby priced.

If this must be a web application then try using the web.config entry below.

<!-- HTTPRuntime
This section sets the httpRuntime settings of the application.
-->
<httpRuntime executionTimeout="150" maxRequestLength="4096"
useFullyQualifiedRedirectUrl="false" minFreeThreads="8"
minLocalRequestFreeThreads="4" appRequestQueueLimit="100"/>

Thanks,

S. Shawn Mehaffie
(e-mail address removed)