Dynamically loading a static C library (compiler suggestion?)

N

Nickolai Leschov

Hello all,

I am programming an embedded controller that has a 'C' library for using
its system functions (I/O, timers, all the specific devices). The
supplied library has .LIB and .H files.

How can I dynamically load a LIB file and access all its functions?
Surely someone has solved similar task?

My intention is to use a Forth system for programming the controller,
calling library functions from it.

Regards,
Nickolai Leschov
 
F

Frank Buss

Nickolai said:
I am programming an embedded controller that has a 'C' library for using
its system functions (I/O, timers, all the specific devices). The
supplied library has .LIB and .H files.

How can I dynamically load a LIB file and access all its functions?
Surely someone has solved similar task?

My intention is to use a Forth system for programming the controller,
calling library functions from it.

Usually you can't load a lib file, it is used to link to C programs (or to
make it easier to use DLLs for Windows). But you could use something like
http://ficl.sourceforge.net/ and write C wrappers for all the library
functions you need.
 
N

Nickolai Leschov

Frank said:
Usually you can't load a lib file, it is used to link to C programs (or to
make it easier to use DLLs for Windows).

I see, loading a lib (or obj) file is something you normally not do when
building a 'C' program. It is not the usual approach, but is it
possible? How?
> http://ficl.sourceforge.net/ and write C wrappers for all the library
> functions you need.

Thanks, I'll examine this closer.
Will it compile to and run on i186 ?
 
J

jacob navia

Nickolai said:
Hello all,

I am programming an embedded controller that has a 'C' library for using
its system functions (I/O, timers, all the specific devices). The
supplied library has .LIB and .H files.

How can I dynamically load a LIB file and access all its functions?
Surely someone has solved similar task?

My intention is to use a Forth system for programming the controller,
calling library functions from it.

Regards,
Nickolai Leschov

It is easy:

You extract all the object files, then look the relocations
inside. If you can resolve all the symbols then you
do the relocations within a malloced memory buffer and
jump into it.

lcc-win proposes a software that does that under the windows system.

For your system it would need to be adapted a bit, depending on your
object format
 
R

Richard Bos

Nickolai Leschov said:
I am programming an embedded controller that has a 'C' library for using
its system functions (I/O, timers, all the specific devices). The
supplied library has .LIB and .H files.

How can I dynamically load a LIB file and access all its functions?

That depends (at least in C; don't know about Forth) completely on the
system you're using.

Richard
 
R

Richard Bos

jacob navia said:
It is easy:
*Snigger*

You extract all the object files, then look the relocations
inside. If you can resolve all the symbols then you
do the relocations within a malloced memory buffer and
jump into it.

*Snigger*

And how do you propose to do that...
lcc-win proposes a software that does that under the windows system.

....on an embedded controller?

As usual, jacob, you don't realise how deep the waters are in which
you're dangling your toes.

Richard
 
J

jacob navia

Richard said:
*Snigger*

And how do you propose to do that...


...on an embedded controller?

As usual, jacob, you don't realise how deep the waters are in which
you're dangling your toes.

Richard

I have written that software... Of course since it is from me and I am
by definition an ignorant, I must have written it in a dream.

Most embedded controllers do not invent new object code formats but
use an existing one. COFF is highly used, and object files under
windows are COFF.

You are of course not an ignorant. This is highly visible in the
depth of your arguments

* Snigger *

All you manage to say :)
 
D

Doug

*Snigger*

And how do you propose to do that...


...on an embedded controller?

Presumably using SDK tools that come with the controller. Or write
your own.

Most of the dev we do for embedded devices isn't done 'on the device'
anyway. The images are built on standard development machines then
transferred to the device. What Jacob describes is perfectly possible
within this development environment.

I don't see what embedded controllers have to do with this anyway.
The OP didn't mention them. I assume you brought them up just to be
difficult.
As usual, jacob, you don't realise how deep the waters are in which
you're dangling your toes.

No. As usual, we're seeing another pointless attack on Jacob. I
suspect, as usual, his answer has gone over your head.

I saw nothing but a useful answer from Jacob, I'm afraid. As usual.
This time it was possibly even on-topic.
 
B

Bart C

Nickolai Leschov said:
Hello all,

I am programming an embedded controller that has a 'C' library for using
its system functions (I/O, timers, all the specific devices). The supplied
library has .LIB and .H files.

How can I dynamically load a LIB file and access all its functions? Surely
someone has solved similar task?

My intention is to use a Forth system for programming the controller,
calling library functions from it.

Let me try and answer this.

You have a microprocessor with integrated devices, for which you have a
library of useful functions. This library is in .LIB format and designed for
use with a C compiler for that processor together with a Linker that
produces a binary image that can be transfered somehow into the processor's
memory.

So presumably you have such tools available and you are supposed to program
it in C. But you want to use Forth.

Well, I suppose you could write some code in C, linked to the LIB, and
create a mechanism so that your Forth language can call it (some sort jump
table in memory). Sounds fiddly though. Another possibility is just to
program the devices directly in Forth, if that's not too much work. Just
need the datasheet.

But perhaps best to ask in comp.arch.embedded and specify the exact device
number.

Bart
 
A

astrobe

Well, I suppose you could write some code in C, linked to the LIB, and
create a mechanism so that your Forth language can call it (some sort jump
table in memory). Sounds fiddly though. Another possibility is just to
program the devices directly in Forth, if that's not too much work. Just
need the datasheet.

Or, one can write the Forth system in assembly language, hence
prodincing an object file, and statically link it with the C library.

Nickolai, you should give us more information on your system: Is the
Forth system written in C, assembly, or something else? Does the
system run an OS?

Amicalement,
Astrobe
 
B

Brad Eckert

Usually you can't load a lib file, it is used to link to C programs (or to
make it easier to use DLLs for Windows). But you could use something likehttp://ficl.sourceforge.net/and write C wrappers for all the library
functions you need.

You could also take a look at F-: http://sourceforge.net/projects/fminus

It's made for small embedded systems and uses C as its assembly
language. Your Forth program gets converted into a C source file
containing a VM and the Forth converted into bytecode executable by
the VM. C can call Forth functions using VM(VM_FnName). Forth can call
C by placing the C code in CODE words.

Brad
 
J

Jack Klein

[snip]
I don't see what embedded controllers have to do with this anyway.
The OP didn't mention them. I assume you brought them up just to be
difficult.

You quoted the OP's mention of them.

Besides, the mechanics of libraries and dynamic loading are completely
off-topic in comp.lang.c, they are 100% implementation and 0%
language.

--
Jack Klein
Home: http://JK-Technology.Com
FAQs for
comp.lang.c http://c-faq.com/
comp.lang.c++ http://www.parashift.com/c++-faq-lite/
alt.comp.lang.learn.c-c++
http://www.club.cc.cmu.edu/~ajo/docs/FAQ-acllc.html
 
S

Stephen Pelc

I am programming an embedded controller that has a 'C' library for using
its system functions (I/O, timers, all the specific devices). The
supplied library has .LIB and .H files.

How can I dynamically load a LIB file and access all its functions?
Surely someone has solved similar task?

My intention is to use a Forth system for programming the controller,
calling library functions from it.

Last time we did this, it was a bit of a nightmare! The most
difficult part was making sure that the C library was properly
initialised. My recommendations would be:

C Construction:
1) Make a C program whose main() returns.
2) Make a function table of the functions you wish to use.
Store the base address of the function table at a known
location in RAM. This will form the core of your Forth
interface.

Forth construction:
1) Produce a wrapper that provides the Forth interface
to the C functions.
2) Provide a Forth interface for the functions in the
table.

Run-time:
2) Make the Forth call the C entry point and hope that main()
returns to you. You may have to place games with the crt0.s
to achieve this. The point is that much of the C initailisation
occurs before main() is run, so you have to start at the entry
point. All this stuff is compiler dependent.
2) From the known location call into the C using the Forth
wrapper.

If you want dynamic loading, then you'll have to provide a
suitable loader, but the priciples are the same.

More generic solutions are based on the "don't do that" rule.
1) Use a Forth cross compiler
or
2) Use a Forth written in C.

Stephen


--
Stephen Pelc, (e-mail address removed)
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads
 
F

Frank Buss

Nickolai said:
I see, loading a lib (or obj) file is something you normally not do when
building a 'C' program. It is not the usual approach, but is it
possible? How?

As others have written, it is possible, but difficult. You have to parse
the object file format, do the initialization etc. But maybe it is possible
with some other ideas mentioned in this thread, e.g. writing just a main
functions in a C program, linking all library functions and passing the
function pointers to the Forth system. Depends a bit on the platform and
Forth system how easy this will be.
Thanks, I'll examine this closer.
Will it compile to and run on i186 ?

Yes, it compiles on nearly everything with a standard C compiler and
sufficient memory. I have just compiled it on a WindowsCE ARM embedded
system with some minor fixes needed for Visual C++ .NET 2005. But it is not
as fast as native Forth systems.
 
N

Nickolai Leschov

Hello again,

First, I want to thank all of you for rushing out to help me, though I
supplied so little clues. I really wanted not to bother you with
unneeded details that could possibly lead the discussion to the wrong
path, but in fact said not enough to be understood.

So, let me clarify the situation:

I am programming a microcontroller, ICPDAS i-8411 to be exact.
(http://www.icpdas.com/products/PAC/i-8000/i-8411-i-8811.htm)
Microcontroller not in the sense of "Embedded CPU" but "a small box". A
box has slots: one for CPU board and several more for I/O boards. From
the programmer's point of view microcontroller is a PC. It has:
- i186-compatible CPU; 512K RAM
- 512K flash-based disk with simple filesystem
- DOS-compatible OS in boot sector
- some specific devices

Manufacturer suggests using one of the following compilers:
- Borland C++ 3.1~5.02
- MSC 6.0
- MSVC (before version 1.52)
- TC 2.01 (*available for free)
- TC++ 1.01 (*available for free)
and supplies a system library for their controller for use with one of
the above compilers. Library doesn't have source and comes as .lib and
..h files. I also have some of the above compilers .

So, the proposed design cycle looks like this:
I write my program in 'C' and link against the manufacturer's library
with one of recommended compilers, on Windows PC (maybe even in a DOS
box). Then I transfer binary file to microcontroller using
manufacturer's terminal program. I run it. Repeat until satisfied with
result.

There are 2 major problems for me:

1. Proposed compilers are all old cruft.
I would like to use more up-to-date tools for programming my embedded
PC. I want something that is at least either (a) non-proprietary or (b)
modern, or both. The best I saw is Borland C++ 5.02 (in the modern
department, but still it could use some improvement) And how do you
obtain license for the compilers _that_ old? Will Borland or Microsoft
grant me license for them tomorrow? (or even today) Will DOS compilers
run on my computers tomorrow? (or even today)

2. No debugging.
It doesn't need explaining, I guess. Debugging is good.

So I started to think about using a Forth system on this controller. I
can choose a free as in freedom Forth for DOS, put it on controller's
disk and if I find a way to use system library from it, I'll be fine.
With a little modification, I hope, I will have a license-free
development environment with debugging. I'll also have the bonus of the
compiler on the target, just in case.

I want to use the supplied library because as opposed to microcontroller
as "Embedded CPU", I don't have the datasheet for all my hardware. I
don't get schematics, it's not practiced in the case of microcontrollers
in the sense of "small box". Instead, I get system library that should
fit a lineup of controllers.

So, I thought that maybe I should dynamically load the system library
and gain acess to its functions. From what I gather LIB is a collection
of OBJ files in the OMF format, developed by Microsoft, but later
standardized. I found a bunch of information here, on the 'D' community
forum: http://www.dsource.org/forums/viewtopic.php?t=959
Seems that if no one else, developers of exotic languages confront this
problem?

Either that, or I could use a better development environment for 16-bit
DOS. (But the debugging is out of the question, as I understand)
 
S

santosh

Nickolai said:
Hello again,

First, I want to thank all of you for rushing out to help me, though I
supplied so little clues. I really wanted not to bother you with
unneeded details that could possibly lead the discussion to the wrong
path, but in fact said not enough to be understood.

So, let me clarify the situation:

I am programming a microcontroller, ICPDAS i-8411 to be exact.
(http://www.icpdas.com/products/PAC/i-8000/i-8411-i-8811.htm)
Microcontroller not in the sense of "Embedded CPU" but "a small box".
A box has slots: one for CPU board and several more for I/O boards.
From the programmer's point of view microcontroller is a PC. It has:
- i186-compatible CPU; 512K RAM
- 512K flash-based disk with simple filesystem
- DOS-compatible OS in boot sector
- some specific devices

Manufacturer suggests using one of the following compilers:
- Borland C++ 3.1~5.02
- MSC 6.0
- MSVC (before version 1.52)
- TC 2.01 (*available for free)
- TC++ 1.01 (*available for free)
and supplies a system library for their controller for use with one of
the above compilers. Library doesn't have source and comes as .lib and
.h files. I also have some of the above compilers .

So, the proposed design cycle looks like this:
I write my program in 'C' and link against the manufacturer's library
with one of recommended compilers, on Windows PC (maybe even in a DOS
box). Then I transfer binary file to microcontroller using
manufacturer's terminal program. I run it. Repeat until satisfied with
result.

There are 2 major problems for me:

1. Proposed compilers are all old cruft.
I would like to use more up-to-date tools for programming my embedded
PC. I want something that is at least either (a) non-proprietary or
(b) modern, or both. The best I saw is Borland C++ 5.02 (in the modern
department, but still it could use some improvement) And how do you
obtain license for the compilers _that_ old? Will Borland or Microsoft
grant me license for them tomorrow? (or even today) Will DOS compilers
run on my computers tomorrow? (or even today)

DOS programs will continue to run (with minor problems?) for some time
yet.

As for the compilers, there is nothing wrong with working software. Not
all systems have the same unnaturally accelerated life cycle as the PC.
If they work for your systems and the manufacturer recommends them,
then I don't see the problem. In any case, have you tried something
like GCC or the Green Hills compiler?
2. No debugging.
It doesn't need explaining, I guess. Debugging is good.

Debugging embedded programs is rather tricky. Does your microcontroller
have an emulation program? Can you debug through a serial port?

Regarding dynamically loading your library file, depending on it's exact
format it might be difficult to impossible. Judging by the file name
extension, the manufacturer apparently intends the library to be used
statically linked. Try contacting your manufacturer for other options.
 
C

cr88192

Nickolai Leschov said:
Hello all,

I am programming an embedded controller that has a 'C' library for using
its system functions (I/O, timers, all the specific devices). The supplied
library has .LIB and .H files.

How can I dynamically load a LIB file and access all its functions? Surely
someone has solved similar task?

My intention is to use a Forth system for programming the controller,
calling library functions from it.

a few misc comments (have read rest of the thread):
yeah, you have a 'fun' situation...


I have done dynamic loading of static libraries, of course, I wrote my own
compiler framework, and do more than a few little hacks (try to get an app
to start rewriting and relinking itself at runtime, and one will see a
little of this).


now, probably, any way you can link against this image "statically", it
preferred (for example, if you can link the library with your forth
framework/program, that is idea).


also possible:
at least a few DOS-era compilers had mechanisms for creating DLL-like
libraries. this may be a possibility, but one will need to investigate this.

but, assuming dynamic linking:
now, the big fun in this case, is not the loading/relocation, but the MZ-EXE
file format.
for example, the format lacks a symbol table (I think, but I have hardly
really investigated the MZ-EXE format).

in good old PE-COFF (windows), we have a symbol table (symbols may be
stripped, but we can optionally leave them in). I guess it depends a lot on
the compiler/linker being used. it is likely that the compiler and linker
may have some means of including debugging information in the compiled EXE.

this may well be the best bet to look into.

look into leaving debugging info in the EXE, look into locating and
deciphering this info, and then look into writing code to use this data for
linking the lib against the existing image.


another possibility:
one can write their own linker, and manually include the needed info in a
usable form (possibly, going further, for example, including
proxies/jump-tables and other features to aid dynamic linking).


or, if one is "really" lucky, maybe they will have a compiler that does some
of this for them.


as for the other issue, the longest and hardest route is this:
write your own compiler framework.
 
N

Nickolai Leschov

santosh said:
DOS programs will continue to run (with minor problems?) for some time
yet.

They will do for now. But I'm not satisfied with this situation, thank you.
If they work for your systems and the manufacturer recommends them,
then I don't see the problem.
Programming-wise, I think manufacturer is not very competent. I believe
that manufacturer can produce hardware and software that _works_ for a
price that I'm happy with, but I don't trust them in anything that's
related to programming. I use their library because it provides
interface to their hardware, but I'm not very fond of it.
In any case, have you tried something
like GCC or the Green Hills compiler?
I'll look at them, thanks.
Debugging embedded programs is rather tricky. Does your microcontroller
have an emulation program? Can you debug through a serial port?
No, my microcontroller doesn't have an emulation program. Besides,
emulation won't buy me much compared to real debugging. I can use DOS
emulation, but it's of little use plus it's not integrated into IDE.

Debugging through a serial port is also not available, as far as I know.
This is what I wanted to accomplish with Forth.
Regarding dynamically loading your library file, depending on it's exact
format it might be difficult to impossible. Judging by the file name
extension, the manufacturer apparently intends the library to be used
statically linked. Try contacting your manufacturer for other options.
As far as I know it's OMF object file format. Is OMF loading hard? Who
did it?
 
B

Bruce McFarling

also possible:
at least a few DOS-era compilers had mechanisms for creating DLL-like
libraries. this may be a possibility, but one will need to investigate this.
but, assuming dynamic linking:
now, the big fun in this case, is not the loading/relocation, but the MZ-EXE
file format.
for example, the format lacks a symbol table (I think, but I have hardly
really investigated the MZ-EXE format).

The .LIB format is a collection of relocatable object modules, which
have a hashed symbol table and a name header in each module.

Name header:
vvv QUOTE vvv

82H LHEADR--LIBRARY MODULE HEADER RECORD
========================================

Description
-----------

This record is very similar to the THEADR record. It is used to
indicate the name of a module within a library file (which has an
internal organization different from that of an object module).

History
-------

This record type was defined in the original Intel specification with
the same format but with a different purpose, so its use for libraries
should be considered a Microsoft extension.

Record Format
-------------

1 2 1 <-String Length-> 1
82 Record String Name String Checksum
Length Length


NOTE: In LINK, THEADR, and LHEADR records are handled identically.
See Appendix 2 for a complete description of Microsoft's library
file format.

^^^ UNQUOTE ^^^

Symbol Table Hashing:

vvv QUOTE vvv
Dictionary
----------

The remaining blocks in the library compose the dictionary. The number
of blocks in the dictionary is given in the library header. Note that
there should always be a prime number of blocks in the dictionary.

The dictionary is a hashed index to the library. Public symbols are
essentially hashed twice, though in fact, both hash indexes are
produced simultaneously. The first hash index, the block index, is
used to determine a block within the dictionary in which to place the
symbol. The second hash index, the bucket index, is used to choose a
bucket within the block for the symbol. Blocks always have 37 buckets;
they are the first 37 bytes of each block. If a bucket is full, it
contains a nonzero value that points to the text of the symbol. To
actually find the symbol, take the bucket value, multiply it by two,
and use the resulting number as a byte offset from the beginning of
the block.

Collisions (that is, two or more distinct symbols hashing to the same
block and bucket in the dictionary) are resolved by a technique known
as linear open addressing. At the same time the hash indexes are
produced, two hash deltas are also produced. If a symbol collides with
a symbol already installed in the dictionary, the librarian attempts
to find an empty bucket for it by adding the bucket delta to the
bucket index and using the result mod 37 as a new bucket index. If
this new bucket index points to a bucket that is empty, the librarian
will install the symbol in that bucket. If the bucket is not empty,
the delta is applied repeatedly until an empty bucket is found or
until it is determined that there are no empty buckets on the block.
If the latter is the case, the block delta is added to the block
index, and the result mod the number of blocks in the dictionary is
used as a new block index. With the new block index and the original
bucket index, the sequence is repeated until an empty bucket on some
block is found.

The number of blocks and the number of buckets are prime so that no
matter what values of hash indexes and deltas are produced for a
symbol, in the worst case, all possible block-bucket combinations will
be tried. Once a free block-bucket pair has been found for a symbol,
the pair and information concerning its place of definition must be
installed. Since a bucket is a single byte pointing into a 512-byte
block, the bucket can give at best a word offset within that block.
Thus, symbol entries within a dictionary must start on word
boundaries. Since bytes 0 through 36 of each dictionary block make up
the hash table, the first symbol entry will begin at byte 38
(decimal).

^^^ UNQUOTE ^^^

.... but I would not be surprised if in reading the symbol table, it
should be possible to read it and then sort it yourself.
 
C

cr88192

Bruce McFarling said:
The .LIB format is a collection of relocatable object modules, which
have a hashed symbol table and a name header in each module.

yes, you have the symbol table, for the LIB.
but, here is what you don't have:
the symbol table for the *EXE*.

as such, you need the table for the EXE, to be able to properly link the LIB
into the program.

after all, what if the LIB should do something, like, say, call 'malloc'...
LIBs are very rarely completely self-contained.


so, that is why I didn't mention LIB here, after all, we allready have all
the info we need to link this, we just don't know anything about what it is
being linked to (the EXE), and we need this info...

now, in PE-COFF (Windows), and ELF (Linux, ...) we have symbol tables by
default (unless the developer strips them).

but, MZ-EXE, does not. it has relocs, but no symbol table, and that is a
problem (this is why we need either debugger info, or the linker to embed a
symbol table in some possibly non-standard way). additionally, the symbol
table could be a seperate file (the app reads this file, and figures how it
maps to the loaded program, and uses this to link the lib to the program).

but, here is another problem:
linkers often like ommiting unused object files and libraries (no links to
it exist, so it is left out of the executable image). this is a problem if
the lib being loaded needs one of these such symbols. one then needs access
to any libraries used in linking the app, so that potentially any unresolved
symbols can be dealt with by also linking in the needed objects (one may
need a customized linker, that hopefully does not link in multiple versions
of the same object file, but otherwise includes all relevant objects).

another possibility, is if one has a list of functions/variables that they
feel need be included, they can include dummy references to all these
functions/variables (I have done his before, when I was faced with
similar...).

now, combined with something like DOS, and the problem is harder (DOS has a
cramped memory space, and it is debatable if sufficient space exists to link
in the needed libraries...).


<snip>
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top