python to enable javascript , tried selinium, ghost, pyQt4 already

[Jaiprakash Singh]

hi,

can you please suggest me some method for study so that i can scrap a site having JavaScript behind it


i have tried selenium, ghost, pyQt4, but it is slow and as a am working with thread it sinks my ram memory very fast.

 

[Denis McMahon]

can you please suggest me some method for study so that i can
scrap a site having JavaScript behind it

Please expand upon the requirement, are you trying to:

a) replace server side javascript with server side python, or
b) replace client side javascript with server side python, or
c) replace client side javascript with client side python, or
d) something else?

(c) is not possible (you can't guarantee that all clients will have
python, or that there will be a mechanism for calling it from your
webpages), (b) doesn't make a lot of sense (you'll be trading cpu in the
client for cpu in the server + network bandwidth and latency).

 

[Chris Angelico]

hi,

can you please suggest me some method for study so that i can scrap a site having JavaScript behind it


i have tried selenium, ghost, pyQt4, but it is slow and as a am working with thread it sinks my ram memory very fast.

Do you mean "scrape"? You're trying to retrieve the displayed contents
of a web page that uses JavaScript? If so, that's basically impossible
without actually executing the JS code, which means largely
replicating the web browser.

ChrisA

 

[Denis McMahon]

Do you mean "scrape"? You're trying to retrieve the displayed contents
of a web page that uses JavaScript? If so, that's basically impossible
without actually executing the JS code, which means largely replicating
the web browser.

Oh, you think he meant scrape? I thought he was trying to scrap (as in
throw away / replace) an old javascript heavy website with something
using python instead.

 

[Chris Angelico]

Oh, you think he meant scrape? I thought he was trying to scrap (as in
throw away / replace) an old javascript heavy website with something
using python instead.

I thought so too at first, but since we had another recent case of
someone confusing the two words, and since "scrape" would make sense
in this context, I figured it'd be worth asking the question.

ChrisA

 

[Giorgos Tzampanakis]

hi,

can you please suggest me some method for study so that i can
scrap a site having JavaScript behind it


i have tried selenium, ghost, pyQt4, but it is slow and as a am
working with thread it sinks my ram memory very fast.

I have tried selenium in the past and I remember it working reasonably
well. I am afraid you can't get around the slowness since you have to have
a web browser running.