Rake Environment variables gone missing

[Herman Schultz]

I posted this about six months ago but the form seems to have gone
missing.

Unfortunately, I have been sent back to work on the project from hell
and am running into the same issue.

The previous posts didn't produce much help as no one seems to be able
to produce the problem but me.

I am running on windows-xp machine with ruby 1.8.6 (2008-08-11
patchlevel 287) [i386-mswin32] and rake, version 0.8.1.

Here's the issue. I read in a windows batch file that has a bunch of
environment variable in it that are used in a hornets nest of Devstudio
C++ projects. I want to parse the variables out and pass them to ENV so
I can invoke incredibuild from rake.

Here's the problem code.

File.foreach env_batch_file do |line|
if /^\s*SET\s*([^\s=]+)\s*=\s*(.*)$/i =~ line
env_count = ENV.size()
key = $1
value = resolve_variables($2)

key.strip!
value.strip!
already_has_key = ENV.has_key?(key)

ENV[key] = value.dup
env_count = env_count + 1

if( env_count != ENV.size())
if ( already_has_key )
verbose("Variable already set.", 5)
else
verbose("ENV Hash whacked." , 1)
end
end
end


In this particular manifestation of the problem it happens when the ENV
size grows to 360. Here's some output:

360:
In Value = %IIE_SOURCE%\cen_xml_support\build\windows\static\release
Rep = F:\source\iie\trunk
Val = %IIE_SOURCE%\cen_xml_support\build\windows\static\release
Val = F:\source\iie\trunk\cen_xml_support\build\windows\static\release
No quotes.
Return ->
F:\source\iie\trunk\cen_xml_support\build\windows\static\release
360:ENV[CEN_XML_SUPPORT_REL_LIB_LOC] =
F:\source\iie\trunk\cen_xml_support\build\windows\static\release
ENV Hash whacked. 360:361 -> CEN_XML_SUPPORT_REL_LIB_LOC
360

I should note that in the very next attempt the size of ENV goes to 27!
So something is going seriously wrong here.

Any thoughts as to what specifically is failing.

 

[t3ch.dude]

I posted this about six months ago but the form seems to have gone
missing.

Unfortunately, I have been sent back to work on the project from hell
and am running into the same issue.

The previous posts didn't produce much help as no one seems to be able
to produce the problem but me.

I am running on windows-xp machine with ruby 1.8.6 (2008-08-11
patchlevel 287) [i386-mswin32] and rake, version 0.8.1.

Here's the issue.  I read in a windows batch file that has a bunch of
environment variable in it that are used in a hornets nest of Devstudio
C++ projects.  I want to parse the variables out and pass them to ENV so
I can invoke incredibuild from rake.

Here's the problem code.

 File.foreach env_batch_file do |line|
   if /^\s*SET\s*([^\s=]+)\s*=\s*(.*)$/i =~ line
      env_count = ENV.size()
      key = $1
      value = resolve_variables($2)

      key.strip!
      value.strip!
      already_has_key = ENV.has_key?(key)

      ENV[key] = value.dup
      env_count = env_count + 1

      if( env_count != ENV.size())
         if ( already_has_key )
              verbose("Variable already set.", 5)
           else
              verbose("ENV Hash whacked." , 1)
         end
      end
   end

In this particular manifestation of the problem it happens when the ENV
size grows to 360.  Here's some output:

360:
In Value = %IIE_SOURCE%\cen_xml_support\build\windows\static\release
Rep = F:\source\iie\trunk
Val = %IIE_SOURCE%\cen_xml_support\build\windows\static\release
Val = F:\source\iie\trunk\cen_xml_support\build\windows\static\release
No quotes.
Return ->
F:\source\iie\trunk\cen_xml_support\build\windows\static\release
360:ENV[CEN_XML_SUPPORT_REL_LIB_LOC] =
F:\source\iie\trunk\cen_xml_support\build\windows\static\release
ENV Hash whacked. 360:361 -> CEN_XML_SUPPORT_REL_LIB_LOC
360

I should note that in the very next attempt the size of ENV goes to 27!
So something is going seriously wrong here.

Any thoughts as to what specifically is failing.

Herman,

I found one reference (http://blogs.msdn.com/oldnewthing/archive/
2006/07/06/657868.aspx ) indicating that the Windows environment is
limited to 64K, or 32Kcharacters in size. I'll bet you hit that limit.

It probably broke for you where it worked for others because by
resolving variables, you're increasing the size of the values.

-T3ch.dude

 

[Herman Schultz]

patchlevel 287) [i386-mswin32] and rake, version 0.8.1.
� � � env_count = ENV.size()
� � � if( env_count != ENV.size())

ENV Hash whacked. 360:361 -> CEN_XML_SUPPORT_REL_LIB_LOC
360

I should note that in the very next attempt the size of ENV goes to 27!
So something is going seriously wrong here.

Any thoughts as to what specifically is failing.

Herman,

I found one reference (http://blogs.msdn.com/oldnewthing/archive/
2006/07/06/657868.aspx ) indicating that the Windows environment is
limited to 64K, or 32Kcharacters in size. I'll bet you hit that limit.

It probably broke for you where it worked for others because by
resolving variables, you're increasing the size of the values.

-T3ch.dude

Interesting. Doesn't give me much hope though. I suppose I could try
it without resolving all variables. That means I'll have to split the
environment between automation variables and devstudio variables. For
example to shared library paths are used by both the automation system
and devstudio. I'll need those resolved at some point for the
automation system.

hmmm. Don't think I'll get to it today. But I will try soon.


Thanks.

 

[Aldric Giacomoni]

Here's the issue. I read in a windows batch file that has a bunch of
environment variable in it that are used in a hornets nest of Devstudio
C++ projects. I want to parse the variables out and pass them to ENV so
I can invoke incredibuild from rake.

Herman,
Could you convince me that you're not doing any kind of useless steps,
here? How about just doing the parsing from within Ruby/Rake ?

 

[Herman Schultz]

Herman,
Could you convince me that you're not doing any kind of useless steps,
here? How about just doing the parsing from within Ruby/Rake ?

Well the problem is that the vcproj files are all based on environment
variables. The defines, the compiler options, the include paths, and
the paths to the libraries for linking.


So you have things like
...
<Tool
Name="VCCLCompilerTool"
AdditionalOptions="$(GQT_DRIVER_DEB_DEFINES)"
...
in the environment batch file it looks like this

SET GQT_REL_DEFINES=%COMMON_DEFINES% %COMMON_REL_DEFINES% /D
XERCES_TMPLSINC %COMMON_LIBRARY_DEFINES%

The project was setup like this by the team in such a way that you have
to run the batch script before you open the vcproj file. If you just
double click it from an explorer window, for example, it won't work.

Likewise if you're automating the build, you need to have the
environment resolved before you build. So if you want to shell out from
rake to invoke the build you need to have that environment set.

In the passed, I used to chain the calling the environment script to the
call to incredibuild.

Something like

result = system( "setEnvironment.bat;{incredibuild statement}" )

There were issues with that as I remember, so I wrote some ruby code to
parse the environment batch file and set the environment. It works fine
on smaller projects but this one is gianormous.

Finally, once I had all the environment variables, I could insure that
the build verification tests and the publishing scripts were all pulling
the same files as the build system had used. I like that; single point
of failure and all.

So if I am going to try to slide under the limit ( if that indeed is my
problem ) I'll need to identify those variables which devstudio needs to
have resolved vs. what variables I can read from rake at run time.

thanks again for the help. I did try today, but I wasn't able to get
under the limit. I'll need a sharper knife.

 

[t3ch.dude]

So if I am going to try to slide under the limit ( if that indeed is my
problem ) I'll need to identify those variables which devstudio needs to
have resolved vs. what variables I can read from rake at run time.

thanks again for the help.  I did try today, but I wasn't able to get
under the limit.  I'll need a sharper knife.

You know, since you're testing the size of the ENV hash, you can test
the sum of the sizes of the values and report it out, just to see if
that is in fact the problem. I'd hate to have sent you down the wrong
path... always best to test!

-T3ch.dude

 

[Herman Schultz]

You know, since you're testing the size of the ENV hash, you can test
the sum of the sizes of the values and report it out, just to see if
that is in fact the problem. I'd hate to have sent you down the wrong
path... always best to test!

-T3ch.dude

No question it's the size.

I wrote a little ruby script to fill up the environment and it the env
gets whacked as the size approaches 32k. It does seem like there is a
little wriggle room depending on how big the string that I am adding
over and over again is, it fails at different sizes.

require 'utilities.rb'

i = 0
s = "X"

100.times { s += "X"}

#puts s

def env_size()
result = 0
ENV.each do | key, value |
result += value.length();
end
return result
end

j = 0
whacked = false
while ( ! whacked && j < 100000)

env_count = ENV.size()
verbose( ENV.size().to_s() + ":", 5)
key = j.to_s()
value = s;

already_has_key = ENV.has_key?(key)
ENV[key] = value.dup

verbose(ENV.size().to_s()+":ENV[" + key + "] = " + value, 11)
env_count = env_count + 1

if( env_count != ENV.size())
if ( already_has_key )
verbose("Variable already set.", 5)
else
verbose("ENV Hash whacked. " + ENV.size().to_s() + ":" +
env_count.to_s() + " -> " + key, 1)
whacked = true
end
end
verbose( ENV.size().to_s(), 5)
verbose( "Nmbr of Env entries = " + ENV.size().to_s(), 1)
verbose( "Size of Environment = " + env_size().to_s(), 1)

j += 1
end


Well ok, thanks for the help. I guess I have to find a work around.