Let's play with Amedas data-Part 1
As a study of neural networks, I would like to handle some kind of data. Therefore, I decided to play with the data of AMeDAS, which seems to be easily available. It seems that the data can be downloaded from the following URL.
https://www.jma.go.jp/jma/menu/menureport.html
Meteorology → Download past weather data
So, it seems that you can get the appropriate data at the appropriate point. It seems that the amount of data that can be downloaded at one time is limited, so set it appropriately and download it. This time, I used the data from Matsumoto City, Nagano Prefecture, which I often jog.
The selected item is ... ■ Temperature ■ Wind direction / speed ■ Weather Three of. After selecting the appropriate period and clicking the "Download CSV file" button, the download was successful.
When I open it, it looks like a strange format.
The header is included, but it may be a little broken. In order to analyze with Tensorflow, it is necessary to import data into Python, but this time I will use Pandas. Then, it seems that there are restrictions on the CSV file. Probably ...
- The first line is the header
- The amount of data (columns) separated by commas is the same for each row.
Is it around? I can't use it as it is, and since the wind direction is Japanese, I decided to convert it to a numerical value.
I feel that it is the royal road to use Script such as Perl for this kind of thing, but since the environment I am using is Windows, I will write a program easily with VBS. How about something like the following? (Is it okay to publish this kind of thing ?? ... Those who use it are at their own risk)
transCSV.vbs
if WScript.Arguments.Count = 0 then
WScript.echo("too few arguments.")
WScript.Quit(-1)
end if
Set objFso = CreateObject("Scripting.FileSystemObject")
FileName = WScript.Arguments(0)
if objFso.FileExists(FileName) = 0 then
WScript.echo("filename err")
WScript.Quit(-1)
end if
tmpFileName = split(FileName,".")
outFileName = tmpFileName(0) & "_out." & tmpFileName(1)
Set objReadFile = objFso.OpenTextFile(FileName , 1, False)
Set objWriteFile = objFso.OpenTextFile(outFileName , 2, True)
If Err.Number > 0 Then
WScript.Echo "Open Error"
Else
' write header
objWriteFile.WriteLine "year,month,day,hour,temp,wind,angle,weather,"
line_cnt = 0
Do Until objReadFile.AtEndOfStream
line_str = objReadFile.ReadLine
If line_cnt > 5 then
line_info = split(line_str,",")
yearValue = split(split(line_info(0)," ")(0),"/")(0)
monthValue = split(split(line_info(0)," ")(0),"/")(1)
dayValue = split(split(line_info(0)," ")(0),"/")(2)
hourValue = split(split(line_info(0)," ")(1),":")(0)
temp = line_info(1)
wind = line_info(4)
strAngle = line_info(6)
weather = line_info(9)
If Not(len(temp)=0 or len(wind)=0 or len(strAngle)=0 or len(weather)=0) then
If strcomp(strAngle,"North") = 0 then
angle = 0
ElseIf strcomp(strAngle,"North-northwest") = 0 then
angle = 1
ElseIf strcomp(strAngle,"North-northwest") = 0 then
angle = 2
ElseIf strcomp(strAngle,"Northwest") = 0 then
angle = 3
ElseIf strcomp(strAngle,"West-northwest") = 0 then
angle = 4
ElseIf strcomp(strAngle,"West") = 0 then
angle = 5
ElseIf strcomp(strAngle,"West-southwest") = 0 then
angle = 6
ElseIf strcomp(strAngle,"Southwest") = 0 then
angle = 7
ElseIf strcomp(strAngle,"South-southwest") = 0 then
angle = 8
ElseIf strcomp(strAngle,"South") = 0 then
angle = 9
ElseIf strcomp(strAngle,"South-southeast") = 0 then
angle = 10
ElseIf strcomp(strAngle,"Southeast") = 0 then
angle = 11
ElseIf strcomp(strAngle,"East-southeast") = 0 then
angle = 12
ElseIf strcomp(strAngle,"east") = 0 then
angle = 13
ElseIf strcomp(strAngle,"East-northeast") = 0 then
angle = 14
ElseIf strcomp(strAngle,"Northeast") = 0 then
angle = 15
ElseIf strcomp(strAngle,"Northeast") = 0 then
angle = 16
End If
line_out_str = yearValue
line_out_str = line_out_str & "," & monthValue
line_out_str = line_out_str & "," & dayValue
line_out_str = line_out_str & "," & hourValue
line_out_str = line_out_str & "," & temp
line_out_str = line_out_str & "," & wind
line_out_str = line_out_str & "," & angle
line_out_str = line_out_str & "," & weather
objWriteFile.WriteLine line_out_str
End If
End If
line_cnt = line_cnt + 1
Loop
End If
objReadFile.Close
objWriteFile.Close
Set objReadFile = Nothing
Set objFso = Nothing
WScript.Echo FileName & " > " & outFileName & " transport end!"
To do this, run the following from the command line:
cscript transCSV.vbs data.csv
Then, data_out.csv was created in the same folder. (Display halfway)
data_out.csv
hour,temp,wind,angle,weather,
1,-3.0,1.8,0,2
2,-3.2,1.7,3,2
3,-4.1,1.1,0,2
4,-4.6,1.1,9,2
5,-4.6,1.0,12,2
6,-4.5,0.6,14,2
7,-5.1,1.0,0,2
8,-4.2,0.7,16,2
9,-2.7,0.5,13,2
Now you're ready to get it in Pandas. With reference to other samples, it seems that it can be imported with the following Python code.
import pandas as pd
import numpy as np
import tensorflow as tf
# deta making???
csv_input = pd.read_csv(filepath_or_buffer="data_out.csv", encoding="ms932", sep=",")
#Number of input items (number of lines)*The number of columns) will be returned.
print(csv_input.size)
#Returns the DataFrame object extracted only for the specified column.
x = np.array(csv_input[["hour","day","month"]])
y = np.array(csv_input[["wind"]])
So how do you play from here ??? First of all, I will try regression analysis using a neural network (predicting y from x) ... but it has become a little longer, so I will carry it over to the next time. I haven't had much time lately, but I plan to write more somewhere ... I'll do my best.
Recommended Posts