Real Time Data Feed Design Questions

[TazaTek]

I'm designing a stock charting program in C++ and wanted to get some
advice / direction from others that have more experience than I do.

I'm 1st tackling the problem of the incoming datafeed (via TCP from a
provider or in-house sim pumper) that is then routed to various charts
(and other consumers of the data).

I am initially thinking that an Observer/Subject model would work for
this, but I didn't know if there were any "gotchas" to this approach,
or if it's even the right tool for the job.

Speed is a major design consideration, so a push model is likely
favored over a pull model.

Are there any other designs or considerations that will allow for a
low latency, high throughput data processor like this?

Are there any examples out there that anyone knows of. I've found
some generalized Observer patterns, but none of them address
specifically a real-time data feed.

Thanks

Matt

 

[tonydee]

I'm designing a stock charting program in C++ and wanted to get some
advice / direction from others that have more experience than I do.

I'm 1st tackling the problem of the incoming datafeed (via TCP from a
provider or in-house sim pumper) that is then routed to various charts
(and other consumers of the data).

I am initially thinking that an Observer/Subject model would work for
this, but I didn't know if there were any "gotchas" to this approach,
or if it's even the right tool for the job.

Speed is a major design consideration, so a push model is likely
favored over a pull model.

Are there any other designs or considerations that will allow for a
low latency, high throughput data processor like this?

Are there any examples out there that anyone knows of.  I've found
some generalized Observer patterns, but none of them address
specifically a real-time data feed.

As Victor observes, this is a bit too vague to allow much feedback.
One important consideration is how each consumer of the data will
select/find the portions relevant to them (e.g. will they want one or
a few securities, or everything). Do you need to keep historical
data, intra-day or across days (on disk)? You'll probably want a per-
security "header" with current values, possibly an intra-day tick-
stream hanging off that. The high performance option is putting that
in shared memory. Picking an existing in-memory time-series database
product would give you a head start.

(If the data's coming in over TCP, it's probably not a particularly
high performance or volume feed. You may not need to go the whole hog
to handle it....)

Cheers,
Tony