I have data stored in comma delimited txt files. One of the columns represents a datetime.
I need to load each column into separate numpy arrays (and decode the date into a python datetime object).
What is the fastest way to do this (in terms of run time)?
NB. the files are several hundred MB of data and currently take several minutes to load in.
15,3,0,2003-01-01 00:00:00,12.2 15,4.5,0,2003-01-01 00:00:00,13.7 15,6,0,2003-01-01 00:00:00,18.4 15,7.5,0,2003-01-01 00:00:00,17.9 15,9,0,2003-01-01 00:00:00,17.7 15,10.5,0,2003-01-01 00:00:00,16.3 15,12,0,2003-01-01 00:00:00,17.2
Here is my current code (it works, but is slow):
import csv import datetime import time import numpy a= b= c= d= timestmp= myfile = open('mydata.txt',"r") # Read in the data csv_reader = csv.reader(myfile) for row in csv_reader: a.append(row) b.append(row) c.append(row) timestmp.append(row) d.append(row) a = numpy.array(a) b = numpy.array(b) c = numpy.array(c) d = numpy.array(d) # Convert Time string list into list of Python datetime objects times =  time_format = "%Y-%m-%d %H:%M:%S" for i in xrange(len(timestmp)): times.append(datetime.datetime.fromtimestamp(time.mktime(time.strptime(timestmp[i], time_format))))
Is there a more efficient way to do this?
Any help is very much appreciated -thanks!
(edit: In the end the bottleneck turned out to be with the datetime conversion, and not reading the file as I originally assumed.)