how to read mixed from multiple csv file


J

Jia Hu

Hi,

My 20 csv files has string header, and first two columns are string (e.g.,
1999-01-02, 01:00:00) among the 50 columns. Other columns store numerical
values (int, or float)

I need to do data analysis for these data. For example, extract the each
month data from each of the cvs files (each csv file stores 1 year data)
and there are 20 year data.

in addition, I want to store the data in disk so that I can retrieve data
quickly, just like save and load in Matlab.

Currently, I use structured array

data = []
i = 0
for s in range(1991, 2011):
fileName = folder +_{_sY}0101_{_sY}1231_725300.csv".format(_sY=s)
data.append(np.genfromtxt(fileName, delimiter=",", dtype=None,
names=True))
i += 1

np.save("alldata", data)

====

However, when I load data "np.load("alldata.npy")", it is becomes 0-d array
which is different from original one.


My question is that

(1) How to store or save the data?
(2) as you can see, I use list to store all the 20 ndarrays, I do not feel
it is a good way. Is there any suggestion for the data structure I should
use?

Thanks in advance.
 
Ad

Advertisements


Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top