Graph Dates and Values

B

brianrpsgt1

I am trying to plot dates and values on a graph using matplotlib.
Below is the code. I can run this and it works great, until I get to
about 2000 rows from the DB. Things really start to slow down. I
have successfully plotted up to 5000 rows from the DB, but it is very
slow. I am attempting to plot values for a day, which would be equal
to 84600 records. Is there a more efficient may to accomplish this?

import os
import psycopg2
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import pylab
import dateutil
from datetime import datetime

conn = psycopg2.connect("dbname='db' user='user' password='somepass'
host='localhost'")

spo2_cur = conn.cursor()
sql = ("""SELECT System_Time, val1 FROM data WHERE DATE(System_Time)
='2009-01-07' AND Inst_Num='12345';""")

data_cur.execute(sql)

value_data = data_cur.fetchall()

data_cur.close()
conn.close()

num_rows = len(value_data)
print "There are",num_rows,"rows in the database"


datesFmt = mdates.DateFormatter('%H:%M')

plt.figure(figsize=(14,3))

for s in value_data:

dates = mdates.date2num([s[0]])

plt.plot([dates],[s[1]], 'bo', ms=6)

plt.ylim(60,100)
plt.axhline(y=90, linewidth=2, color='r')

ax1= plt.gca()
ax1.xaxis.set_major_formatter(datesFmt)

for label in ax1.xaxis.get_ticklabels():
label.set_color('black')
label.set_rotation(45)
label.set_fontsize(8)

for label in ax1.yaxis.get_ticklabels():
label.set_fontsize(8)

plt.show()
plt.close()


Any help would be great!

Thanks

B
 
G

Gabriel Genellina

I am trying to plot dates and values on a graph using matplotlib.
Below is the code. I can run this and it works great, until I get to
about 2000 rows from the DB. Things really start to slow down. I
have successfully plotted up to 5000 rows from the DB, but it is very
slow. I am attempting to plot values for a day, which would be equal
to 84600 records. Is there a more efficient may to accomplish this?
Click to expand...

(isn't it 86400?)
for s in value_data:
dates = mdates.date2num([s[0]])
plt.plot([dates],[s[1]], 'bo', ms=6)
Click to expand...

Without looking at the matplotlib docs, the above [] suggests that both
date2num and plt.plot take a list of values to act upon, and you're
feeding one point at a time. Probably you end up creating one series per
point (instead of a single series with many points). I guess something
like this should work:

x, y = zip(*value_data) # "transpose"
dates = mdates.date2num(x)
plt.plot(dates, y, 'bo', ms=6)

(totally untested)
 
B

brianrpsgt1

I am trying to plot dates and values on a graph using matplotlib.
Below is the code.  I can run this and it works great, until I get to
about 2000 rows from the DB.  Things really start to slow down.  I
have successfully plotted up to 5000 rows from the DB, but it is very
slow.  I am attempting to plot values for a day, which would be equal
to 84600 records.  Is there a more efficient may to accomplish this?
Click to expand...

(isn't it 86400?)
for s in value_data:
    dates = mdates.date2num([s[0]])
    plt.plot([dates],[s[1]], 'bo', ms=6)
Click to expand...

Without looking at the matplotlib docs, the above [] suggests that both  
date2num and plt.plot take a list of values to act upon, and you're  
feeding one point at a time. Probably you end up creating one series per  
point (instead of a single series with many points). I guess something  
like this should work:

x, y = zip(*value_data) # "transpose"
dates = mdates.date2num(x)
plt.plot(dates, y, 'bo', ms=6)

(totally untested)
Click to expand...

Gabriel ::

Thanks for the notes. That is exactly what I thought the problem
was. Here is an update. I put a limit to 100 on the SQL Query to
test. When I run your code, I get the data returned, however, I get
the same return equal to the limit I set. In other words, when I run
with a limit of 100, I get the same result 100 times. Which would
mean that when I try to run a whole day (86400 :) - it was late!), I
am getting the same result 86400 times and then it is tyring to plot
that.

Output below:

[ 733414.06489583 733414.06490741 733414.06491898 733414.06493056
733414.06494213 733414.0649537 733414.06496528 733414.06497685
733414.06498843 733414.065 733414.06501157 733414.06502315
733414.06503472 733414.0650463 733414.06505787 733414.06506944
733414.06508102 733414.06509259 733414.06510417 733414.06511574
733414.06512731 733414.06513889 733414.06515046 733414.06516204
733414.06517361 733414.06518519 733414.06519676 733414.06520833
733414.06521991 733414.06523148 733414.06524306 733414.06525463
733414.0652662 733414.06527778 733414.06528935 733414.06530093
733414.0653125 733414.06532407 733414.06533565 733414.06534722
733414.0653588 733414.06537037 733414.06538194 733414.06539352
733414.06540509 733414.06541667 733414.06542824 733414.06543981
733414.06545139 733414.06546296 733414.06547454 733414.06548611
733414.06549769 733414.06550926 733414.06552083 733414.06553241
733414.06554398 733414.06555556 733414.06556713 733414.0655787
733414.06559028 733414.06560185 733414.06561343 733414.065625
733414.06563657 733414.06564815 733414.06565972 733414.0656713
733414.06568287 733414.06569444 733414.06570602 733414.06571759
733414.06572917 733414.06574074 733414.06575231 733414.06576389
733414.06577546 733414.06578704 733414.06579861 733414.06581019
733414.06582176 733414.06583333 733414.06584491 733414.06585648
733414.06586806 733414.06587963 733414.0658912 733414.06590278
733414.06591435 733414.06592593 733414.0659375 733414.06594907
733414.06596065 733414.06597222 733414.0659838 733414.06599537
733414.06600694 733414.06601852 733414.06603009 733414.06604167]
(95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95,
95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95,
95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95,
95, 95, 95, 95, 95, 95, 94, 94, 94, 94, 94, 94, 94, 95, 95, 95, 95,
95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95,
95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 94)

If I run this code:

for s in value_data:
x = mdates.date2num([s[0]])
y = [s[1]]

print [x, y]

The results returned are the following:

There are 100 rows in the database
[ 733414.06489583] [95]
[ 733414.06490741] [95]
[ 733414.06491898] [95]
[ 733414.06493056] [95]
[ 733414.06494213] [95]
[ 733414.0649537] [95]
[ 733414.06496528] [95]
[ 733414.06497685] [95]
[ 733414.06498843] [95]
[ 733414.065] [95]
[ 733414.06501157] [95]
[ 733414.06502315] [95]
[ 733414.06503472] [95]
[ 733414.0650463] [95]
[ 733414.06505787] [95]
[ 733414.06506944] [95]
[ 733414.06508102] [95]
[ 733414.06509259] [95]
[ 733414.06510417] [95]
[ 733414.06511574] [95]
[ 733414.06512731] [95]
[ 733414.06513889] [95]
[ 733414.06515046] [95]
[ 733414.06516204] [95]
[ 733414.06517361] [95]
[ 733414.06518519] [95]
[ 733414.06519676] [95]
[ 733414.06520833] [95]
[ 733414.06521991] [95]
[ 733414.06523148] [95]
[ 733414.06524306] [95]
[ 733414.06525463] [95]
[ 733414.0652662] [95]
[ 733414.06527778] [95]
[ 733414.06528935] [95]
[ 733414.06530093] [95]
[ 733414.0653125] [95]
[ 733414.06532407] [95]
[ 733414.06533565] [95]
[ 733414.06534722] [95]
[ 733414.0653588] [95]
[ 733414.06537037] [95]
[ 733414.06538194] [95]
[ 733414.06539352] [95]
[ 733414.06540509] [95]
[ 733414.06541667] [95]
[ 733414.06542824] [95]
[ 733414.06543981] [95]
[ 733414.06545139] [95]
[ 733414.06546296] [95]
[ 733414.06547454] [95]
[ 733414.06548611] [95]
[ 733414.06549769] [95]
[ 733414.06550926] [95]
[ 733414.06552083] [95]
[ 733414.06553241] [95]
[ 733414.06554398] [95]
[ 733414.06555556] [94]
[ 733414.06556713] [94]
[ 733414.0655787] [94]
[ 733414.06559028] [94]
[ 733414.06560185] [94]
[ 733414.06561343] [94]
[ 733414.065625] [94]
[ 733414.06563657] [95]
[ 733414.06564815] [95]
[ 733414.06565972] [95]
[ 733414.0656713] [95]
[ 733414.06568287] [95]
[ 733414.06569444] [95]
[ 733414.06570602] [95]
[ 733414.06571759] [95]
[ 733414.06572917] [95]
[ 733414.06574074] [95]
[ 733414.06575231] [95]
[ 733414.06576389] [95]
[ 733414.06577546] [95]
[ 733414.06578704] [95]
[ 733414.06579861] [95]
[ 733414.06581019] [95]
[ 733414.06582176] [95]
[ 733414.06583333] [95]
[ 733414.06584491] [95]
[ 733414.06585648] [95]
[ 733414.06586806] [95]
[ 733414.06587963] [95]
[ 733414.0658912] [95]
[ 733414.06590278] [95]
[ 733414.06591435] [95]
[ 733414.06592593] [95]
[ 733414.0659375] [95]
[ 733414.06594907] [95]
[ 733414.06596065] [95]
[ 733414.06597222] [95]
[ 733414.0659838] [95]
[ 733414.06599537] [95]
[ 733414.06600694] [95]
[ 733414.06601852] [95]
[ 733414.06603009] [95]
[ 733414.06604167] [94]

Like you stated above, I am sending one point each time, instead of a
list with many values. I think I am on the right track, but still
looking to get that last step worked out.

B
 
G

Gabriel Genellina

En Tue said:
I am trying to plot dates and values on a graph using matplotlib.
Below is the code.  I can run this and it works great, until I get to
about 2000 rows from the DB.  Things really start to slow down.  I
have successfully plotted up to 5000 rows from the DB, but it is very
slow.  I am attempting to plot values for a day, which would be equal
to 84600 records.  Is there a more efficient may to accomplish this?
Click to expand...
Without looking at the matplotlib docs, the above [] suggests that both
date2num and plt.plot take a list of values to act upon, and you're  
feeding one point at a time. Probably you end up creating one series
per  
point (instead of a single series with many points). I guess something  
like this should work:

x, y = zip(*value_data) # "transpose"
dates = mdates.date2num(x)
plt.plot(dates, y, 'bo', ms=6)
Click to expand...

Thanks for the notes. That is exactly what I thought the problem
was. Here is an update. I put a limit to 100 on the SQL Query to
test. When I run your code, I get the data returned, however, I get
the same return equal to the limit I set. In other words, when I run
with a limit of 100, I get the same result 100 times. Which would
mean that when I try to run a whole day (86400 :) - it was late!), I
am getting the same result 86400 times and then it is tyring to plot
that.

Output below:

[ 733414.06489583 733414.06490741 733414.06491898 733414.06493056 ...
733414.06600694 733414.06601852 733414.06603009 733414.06604167]
(95, 95, 95, 95, ... 95, 95, 95, 94)

If I run this code:

for s in value_data:
x = mdates.date2num([s[0]])
y = [s[1]]
print [x, y]

The results returned are the following:

There are 100 rows in the database
[ 733414.06489583] [95]
[ 733414.06490741] [95]
[ 733414.06491898] [95]
[ 733414.06493056] [95] ...
[ 733414.06600694] [95]
[ 733414.06601852] [95]
[ 733414.06603009] [95]
[ 733414.06604167] [94]
Click to expand...

Well, both look the same values to me... what's wrong? Why do you say "the
same results 100 times".

Oh, the code fragment I posted is suposed to *replace* the original for
loop. Don't put it inside a loop.
 
B

brianrpsgt1

En Tue, 10 Mar 2009 13:32:10 -0200, brianrpsgt1 <[email protected]>  
escribió:




En Tue, 10 Mar 2009 05:08:41 -0200, brianrpsgt1 <[email protected]>  
escribió:
I am trying to plot dates and values on a graph using matplotlib.
Below is the code.  I can run this and it works great, until I get to
about 2000 rows from the DB.  Things really start to slow down.  I
have successfully plotted up to 5000 rows from the DB, but it is very
slow.  I am attempting to plot values for a day, which would be equal
to 84600 records.  Is there a more efficient may to accomplish this?
Without looking at the matplotlib docs, the above [] suggests that both  
date2num and plt.plot take a list of values to act upon, and you're  
feeding one point at a time. Probably you end up creating one series  
per  
point (instead of a single series with many points). I guess something  
like this should work:
x, y = zip(*value_data) # "transpose"
dates = mdates.date2num(x)
plt.plot(dates, y, 'bo', ms=6)
Click to expand...
Click to expand...
Thanks for the notes.  That is exactly what I thought the problem
was.  Here is an update.  I put a limit to 100 on the SQL Query to
test.  When I run your code, I get the data returned, however, I get
the same return equal to the limit I set.  In other words, when I run
with a limit of 100, I get the same result 100 times.  Which would
mean that when I try to run a whole day (86400 :) - it was late!), I
am getting the same result 86400 times and then it is tyring to plot
that.
Click to expand...
Output below:
Click to expand...
[ 733414.06489583  733414.06490741  733414.06491898  733414.06493056 ...
  733414.06600694  733414.06601852  733414.06603009  733414.06604167]
(95, 95, 95, 95, ...  95, 95, 95, 94)
Click to expand...
If I run this code:
Click to expand...
for s in value_data:
    x = mdates.date2num([s[0]])
    y = [s[1]]
    print [x, y]
Click to expand...
The results returned are the following:
Click to expand...
There are 100 rows in the database
[ 733414.06489583] [95]
[ 733414.06490741] [95]
[ 733414.06491898] [95]
[ 733414.06493056] [95] ...
[ 733414.06600694] [95]
[ 733414.06601852] [95]
[ 733414.06603009] [95]
[ 733414.06604167] [94]
Click to expand...

Well, both look the same values to me... what's wrong? Why do you say "the  
same results 100 times".

Oh, the code fragment I posted is suposed to *replace* the original for  
loop. Don't put it inside a loop.
Click to expand...

Gabriel ::

Thank you very much!!! That was it, I had it in the loop. I works
great now!! Graphs are coming up right away.

B
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Trying to build a SARIMAX model to forecast the S&P500 trend 0
Trouble with prediction code, for the life of me I can't figure out why it isnt running properly. Help would be appreciated. 0
Drawing shaded area depending on distance with latitude and altitude coordinate 2

Members online

Total: 24 (members: 2, guests: 22)
Robots: 366

Forum statistics

Threads
473,770
Messages
2,569,583
Members
45,072
Latest member
trafficcone

Latest Threads

Top