OSX -flat_namespace issues show up in hitimes, nokogiri andamalgalite

[Jeremy Hinegardner]

I'm not sure if it is the problem, but it appears that because ruby is compiled
with -flat_namespace on Mac OS X that unexpected behavior can occur if an
extension links against CoreServices. Here's a fairly indepth example, using
ruby 1.8.6. The exact same thing happens on 1.9.1.

% ./bin/ruby -v
ruby 1.8.6 (2009-08-04 patchlevel 383) [i686-darwin9.8.0]

% ./bin/ruby -rrbconfig -e 'puts Config::CONFIG["LDSHARED"]'
cc -dynamic -bundle -undefined suppress -flat_namespace

% ./bin/ruby -rubygems -e "require 'hitimes'" -e "require 'nokogiri'" -e 'puts "libxml version : #{Nokogiri::LIBXML_PARSER_VERSION}"'
WARNING: Nokogiri was built against LibXML version 2.7.2, but has dynamically loaded 2.6.16
HI. You're using libxml2 version 2.6.16 which is over 4 years old and has
plenty of bugs. We suggest that for maximum HTML/XML parsing pleasure, you
upgrade your version of libxml2 and re-install nokogiri. If you like using
libxml2 version 2.6.16, but don't like this warning, please define the constant
I_KNOW_I_AM_USING_AN_OLD_AND_BUGGY_VERSION_OF_LIBXML2 before requring nokogiri.

libxml version : 20616

Change the order of the requires and all is good:

% ./bin/ruby -rubygems -e "require 'nokogiri'" -e "require 'hitimes'" -e 'puts "libxml version : #{Nokogiri::LIBXML_PARSER_VERSION}"'
libxml version : 20702

A similar effect happens when using the Amalgalite gem, that case complains
about the SQLite version. Both hitimes and amalgalite are my gems, that's how
I found this situation, but it could be exposed by someone elses gems too.

Why does this happen?

The answer is, hitimes, on OSX links against '-framework CoreServices' in order
to get access to the 'AbsoluteToNanoseconds()' and 'UpTime()' functions.

When hitimes loads and therefore the CoreServices framework loads, CoreServices
brings along with it the system libxml and the sytem libsqlite3. We can see
this by setting DYLD_PRINT_LIBRARIES=1:

% DYLD_PRINT_LIBRARIES=1 ./bin/ruby -rubygems -e "require 'hitimes'" -e "require 'nokogiri'" -e 'puts "libxml version : #{Nokogiri::LIBXML_PARSER_VERSION}"'
dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/./bin/ruby
dyld: loaded: /usr/lib/libSystem.B.dylib
dyld: loaded: /usr/lib/libobjc.A.dylib
dyld: loaded: /usr/lib/libgcc_s.1.dylib
dyld: loaded: /usr/lib/system/libmathCommon.A.dylib
dyld: loaded: /usr/lib/libauto.dylib
dyld: loaded: /usr/lib/libstdc++.6.dylib
dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/thread.bundle
dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/etc.bundle
dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/stringio.bundle
dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/syck.bundle
dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/gems/1.8/gems/hitimes-1.0.4/lib/hitimes/1.8/hitimes_ext.bundle
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
dyld: loaded: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
dyld: loaded: /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
dyld: loaded: /usr/lib/libicucore.A.dylib
dyld: loaded: /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
dyld: loaded: /usr/lib/libbsm.dylib
dyld: loaded: /usr/lib/libz.1.dylib
dyld: loaded: /System/Library/Frameworks/Security.framework/Versions/A/Security
dyld: loaded: /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
--> dyld: loaded: /usr/lib/libsqlite3.0.dylib
dyld: loaded: /usr/lib/libresolv.9.dylib
--> dyld: loaded: /usr/lib/libxml2.2.dylib
dyld: loaded: /usr/lib/libxslt.1.dylib
dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/gems/1.8/gems/nokogiri-1.3.3/lib/nokogiri/nokogiri.bundle
dyld: loaded: /opt/local/lib/libexslt.0.dylib
dyld: loaded: /opt/local/lib/libxslt.1.dylib
--> dyld: loaded: /opt/local/lib/libxml2.2.dylib
dyld: loaded: /opt/local/lib/libz.1.dylib
dyld: loaded: /opt/local/lib/libiconv.2.dylib
WARNING: Nokogiri was built against LibXML version 2.7.2, but has dynamically loaded 2.6.16
HI. You're using libxml2 version 2.6.16 which is over 4 years old and has
plenty of bugs. We suggest that for maximum HTML/XML parsing pleasure, you
upgrade your version of libxml2 and re-install nokogiri. If you like using
libxml2 version 2.6.16, but don't like this warning, please define the constant
I_KNOW_I_AM_USING_AN_OLD_AND_BUGGY_VERSION_OF_LIBXML2 before requring nokogiri.

dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/racc/cparse.bundle
dyld: loaded: /Users/jeremy/pkgs/ruby-flat_namespace/lib/ruby/1.8/i686-darwin9.8.0/strscan.bundle
libxml version : 20616

You can see that the system libxml2, which is version 20616 is loaded first and
then the version 20702 that we want to use, is loaded second.

So it appears that the -flat_namespace compiler paramter when compiling ruby,
and the binary extensions,is shooting us in the foot. Lets make sure. So
editing the distributed 'configure' script line 16991 from

darwin*) : ${LDSHARED='cc -dynamic -bundle -undefined suppress -flat_namespace'}
to
darwin*) : ${LDSHARED='cc -dynamic -bundle -undefined dynamic_lookup'}

we have to change -undefined too or else we'll get :

ld: can't use -undefined warning or suppress with -twolevel_namespace

Now we have a new ruby that uses the two_level namespace on OS X

% ./bin/ruby -v -rrbconfig -e 'puts Config::CONFIG["LDSHARED"]'
ruby 1.8.6 (2009-08-04 patchlevel 383) [i686-darwin9.8.0]
cc -dynamic -bundle -undefined dynamic_lookup

And when we run the same thing that before blew up before all is well.

% DYLD_PRINT_LIBRARIES=1 ./bin/ruby -rubygems -e "require 'hitimes'" -e "require 'nokogiri'" -e 'puts "libxml version : #{Nokogiri::LIBXML_PARSER_VERSION}"'
dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/./bin/ruby
dyld: loaded: /usr/lib/libSystem.B.dylib
dyld: loaded: /usr/lib/libobjc.A.dylib
dyld: loaded: /usr/lib/libgcc_s.1.dylib
dyld: loaded: /usr/lib/system/libmathCommon.A.dylib
dyld: loaded: /usr/lib/libauto.dylib
dyld: loaded: /usr/lib/libstdc++.6.dylib
dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/thread.bundle
dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/etc.bundle
dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/stringio.bundle
dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/syck.bundle
dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/gems/1.8/gems/hitimes-1.0.4/lib/hitimes/1.8/hitimes_ext.bundle
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
dyld: loaded: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
dyld: loaded: /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
dyld: loaded: /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
dyld: loaded: /usr/lib/libicucore.A.dylib
dyld: loaded: /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
dyld: loaded: /usr/lib/libbsm.dylib
dyld: loaded: /usr/lib/libz.1.dylib
dyld: loaded: /System/Library/Frameworks/Security.framework/Versions/A/Security
dyld: loaded: /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
--> dyld: loaded: /usr/lib/libsqlite3.0.dylib
dyld: loaded: /usr/lib/libresolv.9.dylib
--> dyld: loaded: /usr/lib/libxml2.2.dylib
dyld: loaded: /usr/lib/libxslt.1.dylib
dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/gems/1.8/gems/nokogiri-1.3.3/lib/nokogiri/nokogiri.bundle
dyld: loaded: /opt/local/lib/libexslt.0.dylib
dyld: loaded: /opt/local/lib/libxslt.1.dylib
--> dyld: loaded: /opt/local/lib/libxml2.2.dylib
dyld: loaded: /opt/local/lib/libz.1.dylib
dyld: loaded: /opt/local/lib/libiconv.2.dylib
dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/racc/cparse.bundle
dyld: loaded: /Users/jeremy/pkgs/ruby-nonflat/lib/ruby/1.8/i686-darwin9.8.0/strscan.bundle
libxml version : 20702

This time around even though the 20616 version is loaded first, and the correct one,
20702, is what is used by the extension.

I guess my question is, what is the proper way to resolve this:

1) should the ./configure script that is distributed with ruby be altered to use
the two level namespaces?

2) or should nokogiri and amalgalite and possibly others, warn the user when the
library they are uses is not the one the extension expects?

Thoughts?

enjoy,

-jeremy

 

[Michal Suchanek]

Hello

Twolevel namespace is used for quite some time in fink. It 's one of
the remaining build system adjustments in recent ruby.

Until now there was nothing I could point at that breaks with flat
namespace but twolevel namespace is the recommended linking on OS X
since some version like 10.2 or 10.3 and it does not seem to break
anything.

Note the -Wl prefix which allows building extensions that use libtool.

Thanks

Michal

--- configure.in 2007-04-18 12:49:43.000000000 +0200
+++ configure.in 2007-04-19 14:23:04.000000000 +0200
@@ -1052,7 +1052,7 @@
rhapsody*) : ${LDSHARED='cc -dynamic -bundle -undefined suppress'}
: ${LDFLAGS=""}
rb_cv_dlopen=yes ;;
- darwin*) : ${LDSHARED='cc -dynamic -bundle -undefined
suppress -flat_namespace'}
+ darwin*) : ${LDSHARED='cc -dynamic -bundle
-Wl,-undefined,dynamic_lookup -Wl,-multiply_defined,sup
press'}
: ${LDFLAGS=""}
: ${LIBPATHENV=DYLD_LIBRARY_PATH}
rb_cv_dlopen=yes ;;

 

[Michal Suchanek]

Hello

Twolevel namespace is used for quite some time in fink. It 's one of
the remaining build system adjustments in recent ruby.

Until now there was nothing I could point at that breaks with flat
namespace but twolevel namespace is the recommended linking on OS X
since some version like 10.2 or 10.3 and it does not seem to break
anything.

Note the -Wl prefix which allows building extensions that use libtool.

Thanks

Michal

--- configure.in =C2=A0 =C2=A0 =C2=A0 =C2=A02007-04-18 12:49:43.000000000= +0200
+++ configure.in =C2=A0 =C2=A0 =C2=A0 =C2=A02007-04-19 14:23:04.000000000= +0200
@@ -1052,7 +1052,7 @@
=C2=A0 =C2=A0 =C2=A0 =C2=A0rhapsody*) =C2=A0 =C2=A0 =C2=A0: ${LDSHARED=3D=

'cc -dynamic -bundle -undefined suppress'}

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0: ${LDFLAGS=3D""}
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=

=A0 =C2=A0rb_cv_dlopen=3Dyes ;;

- =C2=A0 =C2=A0 =C2=A0 darwin*) =C2=A0 =C2=A0 =C2=A0 =C2=A0: ${LDSHARED=

=3D'cc -dynamic -bundle -undefined

suppress -flat_namespace'}
+ =C2=A0 =C2=A0 =C2=A0 darwin*) =C2=A0 =C2=A0 =C2=A0 =C2=A0: ${LDSHARED= =3D'cc -dynamic -bundle
-Wl,-undefined,dynamic_lookup -Wl,-multiply_defined,sup
press'}
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0: ${LDFLAGS=3D""}
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=

=A0 =C2=A0: ${LIBPATHENV=3DDYLD_LIBRARY_PATH}

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=

=A0 =C2=A0rb_cv_dlopen=3Dyes ;;
It should get fixed in 1.8.8 hopefully.

It was rejected for earlier versions for compatibility reasons.

See http://redmine.ruby-lang.org/issues/show/1991

Thanks

Michal

 

[Jeremy Hinegardner]

It should get fixed in 1.8.8 hopefully.

It was rejected for earlier versions for compatibility reasons.

See http://redmine.ruby-lang.org/issues/show/1991

Yup, I saw the thread on ruby-core. thanks.

-jeremy