Is it possible to pass CSV Reader Object As Argument to anotherPython File ???

[bansi]

I have following two python scripts
-namelookupWrapper.py
-namelookup.py


The namelookupWrapper.py takes input of "memberId", "memberName" from
CLI and has following code snippet

idf = sys.argv[1]
namef = sys.argv[2]
real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py"
r = csv.reader(sys.stdin)
os.execv(python_executable, [ python_executable, real_script ] +
sys.argv[1:] )



Wondering how would i pass csv reader object "r" as an argument using
os.execv() to another python script i.e. namelookup.py

 

[Emile van Sebille]

On 1/26/2011 7:51 AM bansi said...

I have following two python scripts
-namelookupWrapper.py
-namelookup.py


The namelookupWrapper.py takes input of "memberId", "memberName" from
CLI and has following code snippet

idf = sys.argv[1]
namef = sys.argv[2]
real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py"
r = csv.reader(sys.stdin)
os.execv(python_executable, [ python_executable, real_script ] +
sys.argv[1:] )



Wondering how would i pass csv reader object "r" as an argument using
os.execv() to another python script i.e. namelookup.py

I suspect you're on the wrong path. You probably want to import
namelookup within namelooupWrapper to use the functions it defines.

Consider:

[root@fcfw2 src]# cat > test1.py

def say(what): print what

[root@fcfw2 src]# cat > test2.py

#!/usr/local/bin/python
import sys
from test1 import say
say(sys.argv[1])

[root@fcfw2 src]# chmod a+x test2.py

[root@fcfw2 src]# ./test2.py hello
hello


HTH,

Emile

 

[bansi]

On 1/26/2011 7:51 AM bansi said...

I have following two python scripts
-namelookupWrapper.py
-namelookup.py

The namelookupWrapper.py takes input of "memberId", "memberName" from
CLI and has following code snippet

idf = sys.argv[1]
namef = sys.argv[2]
real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py"
r = csv.reader(sys.stdin)
os.execv(python_executable, [ python_executable, real_script ] +
sys.argv[1:] )

Wondering how would i pass csv reader object "r" as an argument using
os.execv() to another python script i.e. namelookup.py

I suspect you're on the wrong path.  You probably want to import
namelookup within namelooupWrapper to use the functions it defines.

Consider:

[root@fcfw2 src]# cat > test1.py

def say(what): print what

[root@fcfw2 src]# cat > test2.py

#!/usr/local/bin/python
import sys
from test1 import say
say(sys.argv[1])

[root@fcfw2 src]# chmod a+x test2.py

[root@fcfw2 src]# ./test2.py hello
hello

HTH,

Emile- Hide quoted text -

- Show quoted text -

Emile,
Thanks for quick response. I am not sure if "import
namelookup within namelooupWrapper" helps because they are two
independent scripts which has to be executed in sequence.
First namelookupWrapper.py running under Python 2.6 accept arguments
from stdin and uses csv reader object to read it i.e.
r=csv.reader(sys.stdin)

And then it has to pass csv reader object to another python script
namelookup.py running under Python 2.7 because it uses pyodbc to
connect to database and iterates thru reader object

Any better ideas/suggestions will be greatly appreciated

 

[Chris Rebert]

I have following two python scripts
-namelookupWrapper.py
-namelookup.py


The namelookupWrapper.py takes input of "memberId", "memberName" from
CLI and has following code snippet

idf = sys.argv[1]
namef = sys.argv[2]
real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py"
r = csv.reader(sys.stdin)
os.execv(python_executable, [ python_executable, real_script ] +
sys.argv[1:] )

Wondering how would i pass csv reader object "r" as an argument using
os.execv() to another python script i.e. namelookup.py

It's not possible to pass Python objects between processes in such a
manner. Given that "independent" scripts can't directly take objects
as input anyway, I doubt the two scripts are truly independent from
each other. I would therefore concur with van Sebille that you should
just rewrite them so that one script imports from the other rather
than spawning the other. It should not be too hard to port the Python
2.6 script to Python 2.7 (or vice-versa if necessary).

Cheers,
Chris

 

[bansi]

I have following two python scripts
-namelookupWrapper.py
-namelookup.py

The namelookupWrapper.py takes input of "memberId", "memberName" from
CLI and has following code snippet

idf = sys.argv[1]
namef = sys.argv[2]
real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py"
r = csv.reader(sys.stdin)
os.execv(python_executable, [ python_executable, real_script ] +
sys.argv[1:] )

Wondering how would i pass csv reader object "r" as an argument using
os.execv() to another python script i.e. namelookup.py

It's not possible to pass Python objects between processes in such a
manner. Given that "independent" scripts can't directly take objects
as input anyway, I doubt the two scripts are truly independent from
each other. I would therefore concur with van Sebille that you should
just rewrite them so that one script imports from the other rather
than spawning the other. It should not be too hard to port the Python
2.6 script to Python 2.7 (or vice-versa if necessary).

Cheers,
Chris
--http://blog.rebertia.com- Hide quoted text -

- Show quoted text -

Thanks Chris. Sorry for mis-communicating, the two python scripts are
dependant in a way that
namelookupWrapper.py needs to pass csv record object to another
python script

If thats not possible then please let me know how to do the workaround
i didnt understood the import thing and not sure if it helps in my
case

Here are the details
namelookupwrapper.py - takes input from stdin. Using csv reader object
i iterate thru the input which looks like as shown below

[MemberId, MemberName]
[123, ]
[456, ]
[989, ]


Now i have another script i.e. namelookup.py running under Python 2.7
using pyodbc to retrieve Member Names from database for a given Member
Id in namelooupWrapper.py

So please let me know how to accomplish this

 

[bansi]

Why have you structured them that way, though? What constraint is
keeping you from doing the work in a single process, where the CSV
reader object can be shared?


The problem as you've described it so far is best solved by having a
single process accessing the CSV reader object in memory. If that
doesn't suit your use case, you'll need to explain why not.

--
 \       “To have the choice between proprietary software packages, is |
  `\      being able to choose your master. Freedom means not having a |
_o__)                        master.” —Richard M. Stallman, 2007-05-16 |
Ben Finney

Thanks Ben for quick response. The constraint is in using third party
tool called Splunk which has in-built Python 2.6 which doesnt support
pyodbc on Windows 64 bit OS. Hence i have to install Python 2.7 for
pyodbc.
That means namelookupwrapper.py acts as a wrapper running under
Splunk environment taking input from Splunk via stdin and storing it
in csv reader object and then needs to call actual script
namelookup.py under Python 2.7 for making connection to database

Hope i have clarified a bit

 

[Ethan Furman]

> First namelookupWrapper.py running under Python 2.6 accept arguments
> from stdin and uses csv reader object to read it i.e.
> r=csv.reader(sys.stdin)
>
> And then it has to pass csv reader object to another python script
> namelookup.py running under Python 2.7 because it uses pyodbc to
> connect to database and iterates thru reader object

Why have you structured them that way, though? What constraint is
keeping you from doing the work in a single process, where the CSV
reader object can be shared?


The problem as you've described it so far is best solved by having a
single process accessing the CSV reader object in memory. If that
doesn't suit your use case, you'll need to explain why not.

In other words, why can't you use Python 2.7 to accept input and
generate a csv.reader?

~Ethan~

 

[bansi]

bansi wrote:

 > First namelookupWrapper.py running under Python 2.6 accept arguments
 > from stdin and uses csv reader object to read it i.e.
 > r=csv.reader(sys.stdin)
 >
 > And then it has to pass csv reader object to another python script
 > namelookup.py running under Python 2.7 because it uses pyodbc to
 > connect to database and iterates thru reader object









In other words, why can't you use Python 2.7 to accept input and
generate a csv.reader?

~Ethan~- Hide quoted text -

- Show quoted text -

Ethan,
The python script takes the input from Splunk (http://www.splunk.com/
base/Documentation/) which supports only Python 2.6
So the real constraint is Splunk supports only Python 2.6 .

As you know Python 2.6 doesnt support or doesnt have pyodbc install
for Windows 64 bit OS
So i installed Python 2.7 and thereafter pyodbc install for Windows 64
bit OS for Python 2.7

 

[MRAB]

Ethan,
The python script takes the input from Splunk (http://www.splunk.com/
base/Documentation/) which supports only Python 2.6
So the real constraint is Splunk supports only Python 2.6 .

As you know Python 2.6 doesnt support or doesnt have pyodbc install
for Windows 64 bit OS
So i installed Python 2.7 and thereafter pyodbc install for Windows 64
bit OS for Python 2.7

Have you actually tried Splunk with Python 2.7? It might not work with
versions which are earlier than Python 2.6, but that doesn't
necessarily mean that it won't work with versions of Python 2 which are
later than Python 2.6 (unless the documentation says that it must be
Python 2.6).

 

[bansi]

Have you actually tried Splunk with Python 2.7? It might not work with
versions which are earlier than Python 2.6, but that doesn't
necessarily mean that it won't work with versions of Python 2 which are
later than Python 2.6 (unless the documentation says that it must be
Python 2.6).- Hide quoted text -

- Show quoted text -

Splunk's latest version 4.1.6 doesn't support Python 2.7
I tried the import trick but it didnt work because the real script
which runs under Python 2.7 has import pyodbc so it results in
following error

c:\Splunk\etc\apps\search\bin>splunk cmd python namelookupWrapper.py
memberId memberName < memberInput.csv
Traceback (most recent call last):
File "namelookupWrapper.py", line 3, in <module>
import namelookup
File "c:\Splunk\etc\apps\search\bin\namelookup.py", line 7, in
<module>
import pyodbc
ImportError: DLL load failed: The specified module could not be found.

Please let me know if i am missing something on import. If so please
provide me with an example

 

[bansi]

Splunk's latest version 4.1.6 doesn't support Python 2.7
I tried the import trick but it didnt work because the real script
which runs under Python 2.7 has import pyodbc so it results in
following error

c:\Splunk\etc\apps\search\bin>splunk cmd python namelookupWrapper.py
memberId memberName < memberInput.csv
Traceback (most recent call last):
  File "namelookupWrapper.py", line 3, in <module>
    import namelookup
  File "c:\Splunk\etc\apps\search\bin\namelookup.py", line 7, in
<module>
    import pyodbc
ImportError: DLL load failed: The specified module could not be found.

Please let me know if i am missing something on import. If so please
provide me with an example- Hide quoted text -

- Show quoted text -

Here are some more details from my earlier posting. Please click the
below link

http://answers.splunk.com/questions/11145/its-getting-mysterious-to-make-the-lookup-script-work

 

[Benjamin Kaplan]

Here are some more details from my earlier posting. Please click the
below link

http://answers.splunk.com/questions/11145/its-getting-mysterious-to-make-the-lookup-script-work

Have you tried downloading the source for PyODBC and compiling it
yourself? All you need to do is python setup.py install. My guess
would be that it works just fine on 64-bit Python 2.6, they just never
released a re-compiled version of it for that platform.

 

[bansi]

Have you tried downloading the source for PyODBC and compiling it
yourself? All you need to do is python setup.py install. My guess
would be that it works just fine on 64-bit Python 2.6, they just never
released a re-compiled version of it for that platform.- Hide quoted text -

- Show quoted text -

Thanks Benjamin. Please point me to the website from where i can
download pyodbc for Windows 64 bit OS under Python 2.6 and
installation instructions

 

[Benjamin Kaplan]

Thanks Benjamin. Please point me to the website from where i can
download pyodbc for Windows 64 bit OS under Python 2.6 and
installation instructions
--

You don't download it for 64-bit Windows with Python 2.6. You download
the source code from the website and make the Python 2.6, 64-bit
Windows version yourself.

Download the source zip file and extract it. Then, open up the command
prompt and use the "cd" command to change directories to that source
folder. For instance, if the source code has been extracted to
C:\pyodbc-2.1.8\, you type in "cd C:\pyodbc-2.1.8" and press enter.

After that, you just run the build script which is already in there:

C:\Python26\python26.exe setup.py install

You'll need to have Visual C++ 2008 (not 2010) installed for this to
work. You can get it for free from
http://www.microsoft.com/express/Downloads/ if you don't already have
it.

 

[bansi]

You don't download it for 64-bit Windows with Python 2.6. You download
the source code from the website and make the Python 2.6, 64-bit
Windows version yourself.

Download the source zip file and extract it. Then, open up the command
prompt and use the "cd" command to change directories to that source
folder. For instance, if the source code has been extracted to
C:\pyodbc-2.1.8\, you type in "cd C:\pyodbc-2.1.8" and press enter.

After that, you just run the build script which is already in there:

C:\Python26\python26.exe setup.py install

You'll need to have Visual C++ 2008 (not 2010) installed for this to
work. You can get it for free fromhttp://www.microsoft.com/express/Downloads/if you don't already have
it.




- Show quoted text -- Hide quoted text -

- Show quoted text -

Thanks Benjamin. Wondering why i need to Visual C++ 2008 . What it has
to do with Python?
Isn't it possible to implement your suggestion without installing
Visual C++ 2008 .

 

[rusi]

Thanks Benjamin. Wondering why i need to Visual C++ 2008 . What it has
to do with Python?
Isn't it possible to implement your suggestion without installing
Visual C++ 2008 .

http://code.google.com/p/pyodbc/wiki/Building#Windows

 

[rusi]

http://code.google.com/p/pyodbc/wiki/Building#Windows

Well... This is what the official site says...
On second thoughts I wonder: Would it not be possible to compile python
+pyodbc from source and use gcc/ming for that? Someone who knows more
of the windows build process may say something about the issues
involved.