lock problem

[Ritesh Raj Sarraf]

Hi,

I think there is some lock problem.
Let me show the code first

import os
import sys
import string
import tempfile

import threading
import Queue

from time import sleep

_hostname = None
_username = None
_password = None
_ldapDataFile = None

if _hostname is None or _username is None or _password is None or _ldapDataFile
is None:
sys.stderr.write("Please first set the credentials properly.\n")
sys.exit(1)

_ldapPreCommand = 'ldapsearch -LLL -z 0 "(Sn='
_ldapPostCommand = '*)" -h ' + _hostname + ' -x -D "' + _username + '" -b
Cn=users,DC=hq,DC=domain,DC=com' + " -w " + _password
NumberOfThreads = 5


#char = 's'
#ldapCommand = _ldapPreCommand + char + _ldapPostCommand
#x = os.system(ldapCommand)
#if x == 1024:
# print "I'm in the exception"
#print x
#sys.exit(1)

ldap_attributes =
['dn:', 'cn:', 'sn:', 'l:', 'st:', 'title:', 'description:', 'postalCode:', 'telephoneNumber:', 'facsimileTelephoneNumber:',
'givenName:', 'mail:', 'homePhone:', 'mobile:', 'pager:']

try:
writeFile = open(_ldapDataFile, 'w')
except IOError:
sys.stderr.write("Couldn't open file %s to write.\n" % (writeFile) )
sys.exit(1)

#print "Writing data to %s\n" % (temp_file_name)

def RecordFetcher(char):
(temp_file_fd, temp_file_name) = tempfile.mkstemp()
os.environ['__kabc_ldap'] = temp_file_name
sleep(5) #Let's not thrash the exchange server ;-)
ldapCommand = _ldapPreCommand + char + _ldapPostCommand
if os.system(ldapCommand + "> $__kabc_ldap") != 0:
sys.stderr.write("Couldn't execute the command %s\n" %
(ldapCommand) )
sys.exit(1)
#temp_file_name = "/tmp/tmpnhYrps"
try:
readFile = open(temp_file_name, 'r')
except IOError:
sys.stderr.write("Couldn't open file %s to read.\n" % (readFile) )
sys.exit(1)

for record in readFile.readlines():
if record.startswith(' '): # Remove the junk
pass
record = string.rstrip(record, "\n")
for attrib in ldap_attributes:
if record.startswith(attrib):
try:
FileLock.acquire(True)

if ldap_attributes[0] == attrib: #This attribute is
common/mandatory in all records, so we can rely on it
writeFile.write("\n")

writeFile.write(record)
writeFile.write("\n")

finally:
writeFile.flush()
FileLock.release()
break
readFile.close()
os.remove(temp_file_name)
#writeFile.write("\n")


def run(request, response, func=RecordFetcher):
while 1:
item = request.get()
if item is None:
break
(char, subChar) = item

response.put(func(char+subChar) )

# Start from here
requestQueue = Queue.Queue()
responseQueue = Queue.Queue()

FileLock = threading.Lock()

thread_pool = [
threading.Thread(
target=run,
args=(requestQueue, responseQueue)
)
for i in range(NumberOfThreads)
]

for t in thread_pool: t.start()

for char in string.lowercase:
# I know this is ugly. Too many cycles
# But ldapsearch or exchange is restricting, the query max result limit is
1000
for subChar in string.lowercase:
requestQueue.put( (char, subChar) )

for t in thread_pool: requestQueue.put(None)

for t in thread_pool: t.join()

writeFile.close()


=====

Now as per the above code, "aa" is the first string which will be executed in
Thread-1. In my query to the ldap server, I am getting a record which matches
the "aa" string. I've verified it by putting a breakpoint and checking the
value.

The problem is that when I run the program manually, I don't get the data from
the first thread i.e. of the string "aa".

I'm not sure if there's something wrong in the code mentioned above or is it
really a lock problem.

Can somebody please help about where I'm doing any mistake ?

Thanks,
Ritesh
--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
"Stealing logic from one person is plagiarism, stealing from many is research."
"The great are those who achieve the impossible, the petty are those who
cannot - rrs"

 

[Leo Kislov]

[snip]

os.environ['__kabc_ldap'] = temp_file_name
[snip]

Now as per the above code, "aa" is the first string which will be executed in
Thread-1. In my query to the ldap server, I am getting a record which matches
the "aa" string. I've verified it by putting a breakpoint and checking the
value.

The problem is that when I run the program manually, I don't get the data from
the first thread i.e. of the string "aa".

I'm not sure if there's something wrong in the code mentioned above or is it
really a lock problem.

Can somebody please help about where I'm doing any mistake ?

You're changing environmental variable __kabc_ldap that is shared
between your threads. Environment is not designed for that kind of
usage, it was designed for settings. Either use an option to set
output file or just redirect stdout. If the interface of ldapsearch is
so lame that it requires environmental variable use env to set the
variable: "env __kabc_ldap=/tmp/wrjhdsf ldapsearch ..."

-- Leo

 

[Ritesh Raj Sarraf]

You're changing environmental variable __kabc_ldap that is shared
between your threads. Environment is not designed for that kind of
usage, it was designed for settings. Either use an option to set
output file or just redirect stdout. If the interface of ldapsearch is
so lame that it requires environmental variable use env to set the
variable: "env __kabc_ldap=/tmp/wrjhdsf ldapsearch ..."

The environment variable is set with temp_file_name which gets the name from
tempfile.mkstemp(), which is run in every thread. So I don't think the
environment variable is going to be the same.

Ritesh
--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
"Stealing logic from one person is plagiarism, stealing from many is research."
"The great are those who achieve the impossible, the petty are those who
cannot - rrs"

 

[Leo Kislov]

The environment variable is set with temp_file_name which gets the name from
tempfile.mkstemp(), which is run in every thread. So I don't think the
environment variable is going to be the same.

But you miss the fact that there is only one environment per process.

-- Leo

 

[Ritesh Raj Sarraf]

But you miss the fact that there is only one environment per process.

Maybe there's a confusion.
The environment variable that I'm setting has noting to do with ldapsearch. I
use the environment variable as a filename to which ldapsearch can redirect its
output. And that I do is because the output can be huge and useless.
Then I do some pattern matching on that file and filter my data and then delete
it.

If you think I still am missing something important, request you to describe it.

Thanks,
Ritesh
--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
"Stealing logic from one person is plagiarism, stealing from many is research."
"The great are those who achieve the impossible, the petty are those who
cannot - rrs"

 

[Ritesh Raj Sarraf]

But you miss the fact that there is only one environment per process.

Oh!! I think I get your point.

There'd be only one __kabc_ldap environment variable to which all the threads
would be overwriting.

Hmmm!! Yes, you're correct. Thanks for pointing it out.

Ritesh
--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
"Stealing logic from one person is plagiarism, stealing from many is research."
"The great are those who achieve the impossible, the petty are those who
cannot - rrs"

 

[Gabriel Genellina]

The environment variable is set with temp_file_name which gets the name
from
tempfile.mkstemp(), which is run in every thread. So I don't think the
environment variable is going to be the same.

But the environment is global for all threads.
I don't know how ldapsearch works, but can't you pass it an additional
argument, instead of setting an environment variable? From the long
commandline you're building, I bet it has a suitable option.
Or, instead of os.system, use subprocess.Popen, wich lets you specify the
new environment of the child process.

 

[Gabriel Genellina]

Maybe there's a confusion.
The environment variable that I'm setting has noting to do with
ldapsearch. I
use the environment variable as a filename to which ldapsearch can
redirect its
output. And that I do is because the output can be huge and useless.
Then I do some pattern matching on that file and filter my data and then
delete
it.

If you think I still am missing something important, request you to
describe it.

Then, why do you use an environment variable? Just redirect the output to
the desired file name (when you are making the command line to be executed
by os.system)

 

[Leo Kislov]

Maybe there's a confusion.
The environment variable that I'm setting has noting to do with ldapsearch. I
use the environment variable as a filename to which ldapsearch can redirect its
output. And that I do is because the output can be huge and useless.
Then I do some pattern matching on that file and filter my data and then delete
it.

If you think I still am missing something important, request you to describe it.

Imagine this timeline:

<thread1> os.environ['__kabc_ldap'] = '/tmp/tmp1'
<thread1 suspended, thread2 starts to run>
<thread2> os.environ['__kabc_ldap'] = '/tmp/tmp2'
<thread2> launch ldapsearch (output goes to '/tmp/tmp2')
<thread2 suspended, thread1 starts to run>
<thread1> launch ldapsearch (output goes to '/tmp/tmp2' over output
from ldapsearch launched from thread1)

Seems like that's what is happening to your program.

-- Leo