Design of a TextBuffer and support of istream

[Vincent R]

Hello,

I am working on a dynamic library written in c++ and that is used to
tokenize the buffer coming from Visual Studio IDE (but it could also be
from eclipse, ...).
This library is composed of a ProjectManager class that maps a unique
id with a TextBuffer class that is used to store source code text.
So each time some source code is loaded inside the ide, the
ProjectManager associates its buffer with a TextBuffer.

It means that everytime I modify source code inside the IDE the dynamic
lib is called and TextBuffer is updated (insertLine, RemoveLine, ...).

To design this TextBuffer class I have 2 options :

1) Store the source code buffer inside a std::string and maintain
information between the index of lines and the corresponding position
inside the std::string.

2) Store the source code buffer as a vector<std::string>, it simplifies
a bit the problem because in this case we have a direct mapping between
a line index and the representation inside TextBuffer.

I think I will try to implement the second choice (except if you tell
me that it's a bad idea) but I have a problem because this buffer will
be parsed by a lexer that requires the TextBuffer object to implement
istream :

// Abstract interface representing the source of program text.
class ProgramSource : public GC {
public:
ProgramSource(StringRef path) : filePath_(path) {
lineOffsets_.push_back(0);
}

virtual ~ProgramSource() {}

/** Opens the file and returns an input stream. */
virtual std::tistream & open() = 0;

/** Closes the input stream. */
virtual void close() = 0;

/** Read a segment from the stream (for error reporting) */
virtual bool readLineAt(uint32_t start, std::string & result) = 0;

/** Returns true if the stream is good for reading. */
virtual bool isValid() const = 0;

/** If this source file is contained inside another file, then return
the program
source object that represents the container; Otherwise return
NULL. */
virtual ProgramSource * container() const { return NULL; }

/** Mark a line break at the specified offset */
void newLine(uint32_t offset) {
lineOffsets_.push_back(offset);
}

/** Return the pointer to the program source. Overloaded for testing
purposes.
*/
virtual ProgramSource * get() { return this; }

/** Calculate the token position for the given source location. */
virtual TokenPosition tokenPosition(const SourceLocation & loc);

// Overrides

void trace() const {}

// Casting

protected:
llvm::SmallString<128> filePath_; // Path to the file
std::vector<uint32_t> lineOffsets_; // The start offset of each
line
};


class TextBuffer : public ProgramSource {
private:
vector<std::string> _lines;

//
//std::istringstream stream;

public:
TextBuffer(const char * src)
: objj:rogramSource("")
, stream(src)
{
}

...
TextLine& insertLine(int index, const TCHAR* szLine,int lineLength =
-1);
...
std::istream & open() { return stream; }
void close() {};
bool readLineAt(uint32_t lineIndex, std::string & result);
bool isValid() const { return true; }
void dump() const {}
};

And I don't know how I can implement this std::istream & open() method.
Should TextBuffer inherits from public std::streambuf ?



While I am it, if someone has already implemnted a textbuffer class
that allow to insert/remove lines and that allow to access buffer
easily either by line or by position please let me know.

 

[Nobody]

And I don't know how I can implement this std::istream & open() method.
Should TextBuffer inherits from public std::streambuf ?

How often are you going to be using this method? Unless it's very often,
construct a string containing the TextBuffer's contents, then construct an
istringstream from the string.

 

[Marcel Müller]

To design this TextBuffer class I have 2 options :

1) Store the source code buffer inside a std::string and maintain
information between the index of lines and the corresponding position
inside the std::string.

2) Store the source code buffer as a vector<std::string>, it simplifies
a bit the problem because in this case we have a direct mapping between
a line index and the representation inside TextBuffer.

I think I will try to implement the second choice (except if you tell me
that it's a bad idea)

I would strongly recommend to use the second choice. Simply because the
update frequency could be rather high. E.g. if every time the user
presses a key a large file needs to be updated in memory, including your
line index, then this could significantly degrade performance. Some
people like code files with 100kB and more. Generated code from OR
mappers could also get quite large.

but I have a problem because this buffer will be
parsed by a lexer that requires the TextBuffer object to implement
istream :

You do not need to implement an istream. std::istream will do well.
What you need to do is implement a custom streambuf. This is needed for
the istream constructor.

And I don't know how I can implement this std::istream & open() method.
Should TextBuffer inherits from public std::streambuf ?

There is no need to /inherit/ from streambuf. Composition is sufficient.

While I am it, if someone has already implemnted a textbuffer class that
allow to insert/remove lines and that allow to access buffer easily
either by line or by position please let me know.

Probably someone has done so. But whether the license and the interface
matches are other questions.


Marcel