Huge arrays - speed problems?

[Fabian]

Are there any speed issues in javascript with having really large
arrays? I know if the array gets large enough, bandwidth and download
time can be an issue, but does it take inordinate amounts of time to
access a really large array?

 

[Fabian]

Fabian hu kiteb:

Are there any speed issues in javascript with having really large
arrays? I know if the array gets large enough, bandwidth and download
time can be an issue, but does it take inordinate amounts of time to
access a really large array?

Specifically, inordiante amounts of processor/disk access time.

 

[Lee]

Fabian said:

Are there any speed issues in javascript with having really large
arrays? I know if the array gets large enough, bandwidth and download
time can be an issue, but does it take inordinate amounts of time to
access a really large array?

That depends on what you consider to be "really large".

I wrote a simple script that creates an array of a given size,
loops through the array setting each element equal to 1,
<notes how long this has taken>
then loops through the array, adding up all those 1's, and
<notes how long this has taken>

I ran this in Netscape 7.1 and IE 6, with arrays of size
10,000, 100,000 and 1,000,000 elements, recording the time
in seconds to initialize the array, and then to access each
element (totalling them). All of the times were near zero
except at 1 million elements. That took Netscape 3 seconds
to initialize and 2 seconds to access, and IE 20 seconds(!)
to initialize but also 2 seconds to access all elements.

This was on a relatively slow processor.

 

[Fabian]

Lee hu kiteb:

Fabian said:

That depends on what you consider to be "really large".

I wrote a simple script that creates an array of a given size,
loops through the array setting each element equal to 1,
<notes how long this has taken>
then loops through the array, adding up all those 1's, and
<notes how long this has taken>

I ran this in Netscape 7.1 and IE 6, with arrays of size
10,000, 100,000 and 1,000,000 elements, recording the time
in seconds to initialize the array, and then to access each
element (totalling them). All of the times were near zero
except at 1 million elements. That took Netscape 3 seconds
to initialize and 2 seconds to access, and IE 20 seconds(!)
to initialize but also 2 seconds to access all elements.

The maximum array size I was envisioning was going to be 10,000 by
20,000. I originally envisioned single character text, but there is no
functional reason they couldn't be nunbers, and numbers probably work
better in js anyway. Nice to know I am unlikely to hit any speed
barriers here.

 

[rf]

Lee hu kiteb:


The maximum array size I was envisioning was going to be 10,000 by
20,000. I originally envisioned single character text, but there is no
functional reason they couldn't be nunbers, and numbers probably work
better in js anyway. Nice to know I am unlikely to hit any speed
barriers here.

GAK. 10,000 by 20,000. Hmmm lets see, even if this were a character array
that would be <calculates /> a 190+ megabyte array. Any computer with less
than half a gig of memory would grind to a halt in the paging subsystem.

What *are* you trying to do?

Cheers
Richard.

 

[Fabian]

rf hu kiteb:

GAK. 10,000 by 20,000. Hmmm lets see, even if this were a character
array that would be <calculates /> a 190+ megabyte array. Any
computer with less than half a gig of memory would grind to a halt in
the paging subsystem.

What *are* you trying to do?

That was the maximum *theoretical* size. I would be somewhat surprised
if i ever actually used an array that size. Basically, I was thinking of
virtually dividing the world's surace into 3 mile squares, then creating
a pseudo-virtual reality.

Oh well.

Lets see. 100k^0.5 is 316. I guess I can do something with a grid that
size.

 

[Richard Cornford]

GAK. 10,000 by 20,000. Hmmm lets see, even if this were a
character array that would be <calculates /> a 190+ megabyte
array. Any computer with less than half a gig of memory
would grind to a halt in the paging subsystem.

<snip>

Even that seems an optimistic calculation as JavaScript has no character
data type, and if it did it would be a two byte Unicode character.

I would think that storing a single character would require JavaScript
to use a string value, probably implemented as a minimum of a 32 bit
unsigned integer describing the length plus the string date (2 bytes
minimum for a char), plus another 4 bytes so the Array could point to
the string.

And storing the character code as a number wouldn't help much either as
JavaScript would use a 64 bit double precision float for that.

Richard.

 

[Grant Wagner]

<snip>

Even that seems an optimistic calculation as JavaScript has no character
data type, and if it did it would be a two byte Unicode character.

I would think that storing a single character would require JavaScript
to use a string value, probably implemented as a minimum of a 32 bit
unsigned integer describing the length plus the string date (2 bytes
minimum for a char), plus another 4 bytes so the Array could point to
the string.

And storing the character code as a number wouldn't help much either as
JavaScript would use a 64 bit double precision float for that.

Richard.

Not to mention that even to the most frugal representation to send the data
to the browser would consist of something like:

var a=[1,2,3,4,5...];

Even for a 10,000 member array consisting of a sequence of numbers (which of
course you wouldn't send to the browser anyway, since you could generate it
locally), you'd be looking at nearly 50,000 bytes of data sent to construct
the array:

9 x 1 digit numbers + 89 x 2 digit numbers + 899 x 3 digit numbers + 8999 x
4 digit numbers + 10000 x comma

If you start looking at sending an array of say, 10,000 5 digit product
codes, you're now looking at over 60K worth of data to download a browser
over an average 3K/s dialup line, so 20 seconds just to finish loading the
page.

Unless you have something really astonishing to sell or say, few people
would wait that long.

--
| Grant Wagner <[email protected]>

* Client-side Javascript and Netscape 4 DOM Reference available at:
*
http://devedge.netscape.com/library/manuals/2000/javascript/1.3/reference/frames.html

* Internet Explorer DOM Reference available at:
*
http://msdn.microsoft.com/workshop/author/dhtml/reference/dhtml_reference_entry.asp

* Netscape 6/7 DOM Reference available at:
* http://www.mozilla.org/docs/dom/domref/
* Tips for upgrading JavaScript for Netscape 7 / Mozilla
* http://www.mozilla.org/docs/web-developer/upgrade_2.html

 

[keyur shah]

Doesnt make much difference... Is your array in MBs or GBs wont be..
so no need to have any worry for large arrays... we have bandwidth these
days... dsl/cable/t1... why should one have speed issues.



Keyur Shah
Verizon Communications
732-423-0745

 

[Dr John Stockton]

JRS: In article <[email protected]>, seen in

Doesnt make much difference... Is your array in MBs or GBs wont be..
so no need to have any worry for large arrays... we have bandwidth these
days... dsl/cable/t1... why should one have speed issues.



Keyur Shah
Verizon Communications
732-423-0745

That's rather a limited point of view; perhaps you are new to the
business? BTW, I don't think that number works from most telephones. A
communications firm should know about communication.

(A) Many users still have dial-up POTS connections, sometimes pay-by-
the-minute.

(B) Some users have portables, with low-bandwidth radio links.

(C) Since Javascript is a more-or-less general-purpose language, one can
perfectly use arrays whose contents have been generated by local
computation, and have not been transmitted in detail over the Internet.


Note : the OP was specifically referring to accessing large arrays.