[JAVA] Try to get the road surface condition using big data of road surface management

Currently, as a "demonstration experiment in the sophistication of road surface management," big data that collects data acquired by inexpensive sensors attached to public transportation vehicles such as buses and taxis and aggregates them is open to the public.

** Road surface management big data open data contest ** http://micrms.force.com/apis

This time, we made it possible to check the road surface condition on the map using this data.

** Road surface condition Viewer ** http://needtec.sakura.ne.jp/road_mgt/road_mgt.html

GitHub https://github.com/mima3/road_mgt

路面3.png

Demonstration experiment API for advanced road surface management

The API specifications can be downloaded from the following. http://micrms.force.com/developer

The data can be roughly divided into the following three. -Road surface measurement data Data measured by a machine called i-DRIMS. Acceleration, angular velocity, position data ・ Road surface condition data Road surface condition data analyzed from road surface measurement data ・ Road surface specification data Road management information set in a short section for each local government

Normally, you will use road surface condition data and road surface specification data.

How to get road surface condition data

To get the road surface condition, GET to the following URL.

http://roadmgt.herokuapp.com/api/v1/datapoints?data_type=road-surface

The result is the following XML.

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" xmlns:rm="http://roadmgt.herokuapp.com/vocab/rm#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
  <rdf:Description rdf:about="http://roadmgt.herokuapp.com/api/v1/datapoints/road-surface_282">
    <rm:iri>2.65</rm:iri>
    <rm:step>0.0</rm:step>
    <rm:patching_num>0</rm:patching_num>
    <rm:municipality_id>1</rm:municipality_id>
    <rm:too_slow_fast_flag>0</rm:too_slow_fast_flag>
    <rm:subsidence_and_puddle>0.0</rm:subsidence_and_puddle>
    <rm:section_id>-1</rm:section_id>
    <rm:speed>37.55711977</rm:speed>
    <rdf:type>http://roadmgt.herokuapp.com/vocab/rm#road-surface</rdf:type>
    <geo:alt>0.0</geo:alt>
    <rm:rms>21.14314346</rm:rms>
    <geo:long>140.1434769</geo:long>
    <rm:rutting_amount>3.75</rm:rutting_amount>
    <rm:pothole_num>0</rm:pothole_num>
    <rm:speed_fluctuation_flag>0</rm:speed_fluctuation_flag>
    <rdfs:label>road-surface_282</rdfs:label>
    <rm:cracking_rate>5.3</rm:cracking_rate>
    <geo:lat>35.61753776</geo:lat>
    <rm:analysis_timestamp>2014-12-30 17:00:02.0</rm:analysis_timestamp>
    <rm:distance>220</rm:distance>
  </rdf:Description>
  <rdf:Description rdf:about="http://roadmgt.herokuapp.com/api/v1/datapoints/road-surface_294">
    <rm:speed_fluctuation_flag>1</rm:speed_fluctuation_flag>
    <rm:patching_num>0</rm:patching_num>
    <rm:iri>2.5</rm:iri>
    <rm:step>0.0</rm:step>
    <rm:too_slow_fast_flag>0</rm:too_slow_fast_flag>
    <rm:subsidence_and_puddle>0.0</rm:subsidence_and_puddle>
    <rm:section_id>-1</rm:section_id>
    <rm:municipality_id>1</rm:municipality_id>
    <rm:distance>340</rm:distance>
    <geo:alt>0.0</geo:alt>
    <rm:cracking_rate>7.5</rm:cracking_rate>
    <geo:long>140.1464737</geo:long>
    <rdf:type>http://roadmgt.herokuapp.com/vocab/rm#road-surface</rdf:type>
    <rm:rms>26.23188158</rm:rms>
    <geo:lat>35.61759593</geo:lat>
    <rm:pothole_num>0</rm:pothole_num>
    <rm:speed>42.15859739</rm:speed>
    <rm:analysis_timestamp>2014-12-30 17:00:02.0</rm:analysis_timestamp>
    <rm:rutting_amount>5.3</rm:rutting_amount>
    <rdfs:label>road-surface_294</rdfs:label>
  </rdf:Description>
Abbreviation
</rdf:RDF>

How to get all the data of the road surface condition

The maximum number of acquisitions when executing the API is 300 by default. To get the subsequent data, you need to use offset as follows.

(1) First get the beginning http://roadmgt.herokuapp.com/api/v1/datapoints?data_type=road-surface&limit=300&offset=0

(2) If the data can be acquired normally, change the offset and execute it. http://roadmgt.herokuapp.com/api/v1/datapoints?data_type=road-surface&limit=300&offset=300

(3) Repeat this until the following response is obtained.

404 notFound

Road surface condition parameters

The road surface condition can be filtered by specifying parameters. The parameters that can be specified are as follows.

name Mold Description
municipality xsd:int Municipal ID
It seems to be the ID of the local government, but I don't understand the meaning because there is no explanation like the right.
Contains 1 and 11. Perhaps 1 is Chiba City and 11 is Toyonaka City.
section xsd:int Interval ID. Perhaps the section of the road is given a unique ID internally, but I'm not sure.-There is 1 or something.
date_start xsd:date Analysis date. Error if not set with end specification(YYYY-MM-DD)
date_end xsd:date Analysis date. Error if not set with start specification(YYYY-MM-DD)
lat_start xsd:double Latitude start. Error if not set with end specification
lat_end xsd:double Latitude end. Error if not set with start specification
lon_start xsd:double Longitude start. Error if not set with end specification
lon_end xsd:double Longitude end. Error if not set with start specification

Execution example with longitude and latitude specified:

http://roadmgt.herokuapp.com/api/v1/datapoints?data_type=road-surface&lat_start=34.66802946&lon_start=135.362425&lat_end=35.67436512&lon_end=140.2823427&offset=300&limit=300

Caution If you specify the parameters introduced here and parameters other than offset and limit, an error will occur. This means that code like the one below will result in an error.

$.ajax({
  type : 'GET',
  url : 'http://roadmgt.herokuapp.com/api/v1/datapoints',
  cache: false //Avoiding the cache will result in an error.
  data : {
    data_type : 'road-surface',
    lat_start  : lat_start,
    lon_start : lon_start,
    lat_end  : lat_end,
    lon_end  : lon_end,
    offset : offset,
    limit : limit
  },
  success : function (data) {
  },
  'error' :  function(xhr, textStatus, error) {
  }
});

This involves the risk that the content will not change even if the data is updated in the IE system.

In case of cahce = false, by specifying a unique temporary value for the parameter, it is recognized as another request and avoids the use of cache, but this is not available.

http://stackoverflow.com/questions/4303829/how-to-prevent-a-jquery-ajax-request-from-caching-in-internet-explorer

Direct target specification

You can get the target directly by specifying the URI described in rdf: about.

Execution example: http://roadmgt.herokuapp.com/api/v1/datapoints/road-surface_282

Acquisition result:

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" xmlns:rm="http://roadmgt.herokuapp.com/vocab/rm#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
<rdf:Description rdf:about="http://roadmgt.herokuapp.com/api/v1/datapoints/road-surface_282">
<rm:iri>2.65</rm:iri>
<rm:step>0.0</rm:step>
<rm:patching_num>0</rm:patching_num>
<rm:municipality_id>1</rm:municipality_id>
<rm:too_slow_fast_flag>0</rm:too_slow_fast_flag>
<rm:subsidence_and_puddle>0.0</rm:subsidence_and_puddle>
<rm:section_id>-1</rm:section_id>
<rm:speed>37.55711977</rm:speed>
<rdf:type>http://roadmgt.herokuapp.com/vocab/rm#road-surface</rdf:type>
<geo:alt>0.0</geo:alt>
<rm:rms>21.14314346</rm:rms>
<geo:long>140.1434769</geo:long>
<rm:rutting_amount>3.75</rm:rutting_amount>
<rm:pothole_num>0</rm:pothole_num>
<rm:speed_fluctuation_flag>0</rm:speed_fluctuation_flag>
<rdfs:label>road-surface_282</rdfs:label>
<rm:cracking_rate>5.3</rm:cracking_rate>
<geo:lat>35.61753776</geo:lat>
<rm:analysis_timestamp>2014-12-30 17:00:02.0</rm:analysis_timestamp>
<rm:distance>220</rm:distance>
</rdf:Description>
</rdf:RDF>

By specifying the property name after the target name, you can get only the property. At this time, it is necessary to change ":" to "_". For example, if you want to get "rm: step", specify "rm_step".

Execution example: http://roadmgt.herokuapp.com/api/v1/datapoints/road-surface_282/rm_step

Acquisition result

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" xmlns:rm="http://roadmgt.herokuapp.com/vocab/rm#">
<rdf:Description rdf:about="http://roadmgt.herokuapp.com/api/v1/datapoints/road-surface_282">
<rm:step>0.0</rm:step>
</rdf:Description>
</rdf:RDF>

Description of the response

The main properties are explained below.

Property name Mold Description
rm:analysis_timestamp Analysis date and time xsd:dateTime 「2015-01-14 09:01:It is in the format of "57".
rm:iri IRI xsd:double International Roughness Index Objectively flatness (ride comfort) of pavement.Scale to evaluate
http://www.kandoken.jp/huroku/130913_2_kouenshiryo.pdf
rm:pothole_num Number of potholes xsd:int Number of holes created by the depression of the pavement surface of the road
rm:patching_num Number of patches xsd:int Number of first aid measures taken for pavement damage
http://www.mlit.go.jp/road/ir/ir-council/pdf/roadstock05.pdf
rm:cracking_rate Crack rate xsd:double http://www.town.oki.fukuoka.jp/file/gyousei/nyuusatsu/FILE_346_45.pdf
rm:rutting_amount Rutting depth xsd:double http://www.fuzita.org/cod/rut_.html
rm:step Step xsd:double
rm:subsidence_and_puddle Subsidence / puddle xsd:double
geo:lat latitude xsd:double
geo:long longitude xsd:double
geo:alt Altitude xsd:double

How to get road surface data

It is basically the same as the road surface condition. GET to the following URL.

http://roadmgt.herokuapp.com/api/v1/datapoints?data_type=road-master&lat_start=34.66802946&lon_start=135.362425&lat_end=35.67436512&lon_end=140.2823427&offset=300&limit=300

Currently, only the following red line data exists. 路面.png

As you can see by expanding it, it seems that data exists for each lane.

路面2.png

Used in programming language

Python Here, we will describe how to get the road surface condition in Python.

road_mgt.py


# -*- coding: utf-8 -*-
import urllib
import urllib2
from lxml import etree
import csv
from collections import defaultdict


def get_road_surface(offset, limit):
    """
Road surface condition data
    """
    url = ('http://roadmgt.herokuapp.com/api/v1/datapoints?data_type=road-surface&offset=%d&limit=%d' % (offset, limit))

    req = urllib2.Request(url)
    opener = urllib2.build_opener()
    conn = opener.open(req)
    cont = conn.read()
    parser = etree.XMLParser(recover=True)
    root = etree.fromstring(cont, parser)
    namespaces = {
        'geo' : 'http://www.w3.org/2003/01/geo/wgs84_pos#',
        'rdf' : 'http://www.w3.org/1999/02/22-rdf-syntax-ns#', 
        'rdfs' : 'http://www.w3.org/2000/01/rdf-schema#', 
        'rm' : 'http://roadmgt.herokuapp.com/vocab/rm#'
    }

    row = []
    datas = {}
    value_tags = root.xpath('//rdf:Description', namespaces=namespaces)
    value_tags = root.xpath('//rdf:Description', namespaces=namespaces)
    for value_tag in value_tags:
        label = value_tag.find('rdfs:label', namespaces).text
        step = value_tag.find('rm:step', namespaces).text
        alt = value_tag.find('geo:alt', namespaces).text
        analysis_timestamp = value_tag.find('rm:analysis_timestamp', namespaces).text
        rutting_amount = value_tag.find('rm:rutting_amount', namespaces).text
        municipality_id = value_tag.find('rm:municipality_id', namespaces).text
        speed_fluctuation_flag = value_tag.find('rm:speed_fluctuation_flag', namespaces).text
        section_id = value_tag.find('rm:section_id', namespaces).text
        distance = value_tag.find('rm:distance', namespaces).text
        long = value_tag.find('geo:long', namespaces).text
        iri = value_tag.find('rm:iri', namespaces).text
        cracking_rate = value_tag.find('rm:cracking_rate', namespaces).text
        pothole_num = value_tag.find('rm:pothole_num', namespaces).text
        subsidence_and_puddle = value_tag.find('rm:subsidence_and_puddle', namespaces).text
        speed = value_tag.find('rm:speed', namespaces).text
        rms = value_tag.find('rm:rms', namespaces).text
        lat = value_tag.find('geo:lat', namespaces).text
        too_slow_fast_flag = value_tag.find('rm:too_slow_fast_flag', namespaces).text
        patching_num = value_tag.find('rm:patching_num', namespaces).text

        row.append({
            'label' : label,
            'step' : step,
            'alt' : alt,
            'analysis_timestamp' : analysis_timestamp,
            'rutting_amount' : rutting_amount,
            'municipality_id' : municipality_id,
            'speed_fluctuation_flag' : speed_fluctuation_flag,
            'section_id' : section_id,
            'distance' : distance,
            'long' : long,
            'iri' : iri,
            'cracking_rate' : cracking_rate,
            'pothole_num' : pothole_num,
            'subsidence_and_puddle' : subsidence_and_puddle,
            'speed' : speed,
            'rms' : rms,
            'lat' : lat,
            'too_slow_fast_flag' : too_slow_fast_flag,
            'patching_num' :patching_num
        })
    return row

def get_meas_locale(offset, limit):
    """
Position data at the time of road surface measurement
    """
    url = ('http://roadmgt.herokuapp.com/api/v1/datapoints?data_type=meas-locale&offset=%d&limit=%d' % (offset, limit))

    req = urllib2.Request(url)
    opener = urllib2.build_opener()
    conn = opener.open(req)
    cont = conn.read()
    parser = etree.XMLParser(recover=True)
    root = etree.fromstring(cont, parser)

    namespaces = {
        'geo' : 'http://www.w3.org/2003/01/geo/wgs84_pos#',
        'rdf' : 'http://www.w3.org/1999/02/22-rdf-syntax-ns#', 
        'rdfs' : 'http://www.w3.org/2000/01/rdf-schema#', 
        'rm' : 'http://roadmgt.herokuapp.com/vocab/rm#'
    }

    row = []
    datas = {}
    value_tags = root.xpath('//rdf:Description', namespaces=namespaces)
    for value_tag in value_tags:
        label = value_tag.find('rdfs:label', namespaces).text
        gpstimestamp = value_tag.find('rm:gpstimestamp', namespaces).text
        course = value_tag.find('rm:course', namespaces).text
        measurement_start_timestamp = value_tag.find('rm:measurement_start_timestamp', namespaces).text
        mounting_direction = value_tag.find('rm:mounting_direction', namespaces).text
        car_model = value_tag.find('rm:car_model', namespaces).text
        long = value_tag.find('geo:long', namespaces).text
        lat = value_tag.find('geo:lat', namespaces).text
        alt = value_tag.find('geo:alt', namespaces).text
        model_number = value_tag.find('rm:model_number', namespaces).text
        car_number = value_tag.find('rm:car_number', namespaces).text
        speed = value_tag.find('rm:speed', namespaces).text
        vertical_accuracy = value_tag.find('rm:vertical_accuracy', namespaces).text
        measurement_timestamp = value_tag.find('rm:measurement_timestamp', namespaces).text
        horizontal_accuracy = value_tag.find('rm:horizontal_accuracy', namespaces).text
        row.append({
            'label' : label,
            'gpstimestamp' : gpstimestamp,
            'course' : course,
            'measurement_start_timestamp' : measurement_start_timestamp,
            'mounting_direction' : mounting_direction,
            'car_model' : car_model,
            'long' : long,
            'lat' : lat,
            'alt' : alt,
            'model_number' : model_number,
            'car_number' : car_number,
            'speed' : speed,
            'vertical_accuracy' : vertical_accuracy,
            'measurement_timestamp' : measurement_timestamp,
            'horizontal_accuracy' : horizontal_accuracy
        })
    return row

def get_road_surface_all():
    ret = []
    limit = 300
    offset = 0
    try:
        while True:
            print ('get_road_surface_all %d' % offset)
            r = get_road_surface(offset, limit)
            ret.extend(r)
            offset += limit
    except urllib2.HTTPError, ex:
        if ex.code == 404:
            return ret
        else:
            raise

You can get all road conditions by using get_road_surface_all (). As a program, I just execute the API with urllib and parse the response with lxml.

JavaScript Here, we will describe how to get the road surface condition with JavaScript.

  /**
   *Execute API for road surface information
   */
  function getDataByRange(data_type, lat_start, lon_start, lat_end, lon_end, callback) {
    var offset = 0;
    var limit = 300;
    var obj = {};

    function loadRoadOffset(lat_start, lon_start, lat_end, lon_end, offset, limit, obj, cb) {

      console.log(offset, limit);
      $.ajax({
        type : 'GET',
        url : 'http://roadmgt.herokuapp.com/api/v1/datapoints',
        cache: true , //,An error will occur if cache is set to false. Perhaps instead of ignoring weird parameters, they're flipping
        data : {
          data_type : data_type,
          lat_start  : lat_start,
          lon_start : lon_start,
          lat_end  : lat_end,
          lon_end  : lon_end,
          offset : offset,
          limit : limit
        },
        success : function (data) {
          console.log(data);
          var root = data.documentElement;
          var attrs = root.attributes;
          var records = root.childNodes;
          for(var i=0; i<records.length; i++){
            if(records[i].nodeName.match(/rdf:Description/i)){
              var s = records[i].attributes["rdf:about"].value;
              var props = records[i].childNodes;
              for(var j=0; j<props.length; j++){
                if(props[j].nodeType == 1){
                  var p = props[j].nodeName;
                  var o = props[j].textContent;
                  if (!obj[s]) {
                    obj[s] = {};
                  }
                  if (obj[s][p]) {
                    if (!Array.isArray(obj[s][p])) {
                      var tmp = arys[s][p];
                      obj[s][p] = [];
                      obj[s][p].push(tmp);
                    }
                    obj[s][p].push(o);
                  } else {
                    obj[s][p] = o;
                  }
                }
              }
            }
          }
          loadRoadOffset(lat_start, lon_start, lat_end, lon_end, offset + limit, limit, obj, cb);
        },
        'error' :  function(xhr, textStatus, error) {
          console.log(xhr);
          var err = xhr.responseText;
          if (err == '404 notFound') {
            cb(null , obj);
          } else {
            cb(err , null);
          }
        }
      });
    }
    loadRoadOffset(lat_start, lon_start, lat_end, lon_end, offset, limit, obj, function(err, res) {
      callback(err, res);
    });
  }

This function runs as follows:

getDataByRange('road-master', surfaceRange.lat_min, surfaceRange.long_min, surfaceRange.lat_max, surfaceRange.long_max, function(err, data) {
      console.log('loadRoad', err, data);
});

This process gets all the road specification data in the specified range.

Summary

You can get the road surface condition by executing the API of "Demonstration experiment in advanced road surface management".

This makes it possible to visualize the deterioration status of roads.

Recommended Posts

Try to get the road surface condition using big data of road surface management
Try to get the contents of Word with Golang
Try to get the function list of Python> os package
Try to extract the features of the sensor data with CNN
Try to get statistics using e-Stat
I tried to get the index of the list using the enumerate function
Get the column list & data list of CASTable
Try to image the elevation data of the Geographical Survey Institute with Python
Try using n to downgrade the version of Node.js you have installed
Try to predict the value of the water level gauge by machine learning using the open data of Data City Sabae
Try using django-import-export to add csv data to django
Until you try to let DNN learn the truth of the image using Colab
How to get article data using Qiita API
I want to get League of Legends data ③
I want to get League of Legends data ②
Try using the collections module (ChainMap) of python3
An introduction to data analysis using Python-To increase the number of video views-
Try to display the railway data of national land numerical information in 3D
To get the path of the currently running python.exe
I want to get League of Legends data ①
Try to simulate the movement of the solar system
Check the status of your data using pandas_profiling
Scraping the winning data of Numbers using Docker
[Introduction to Python] How to get the index of data with a for statement
I want to get custom data attributes of html as elements using Python Selenium
Try to create a battle record table with matplotlib from the data of "Schedule-kun"
Give latitude and longitude point sequence data and try to identify the road from OpenStreetMap data
How to get the number of digits in Python
Try to solve the problems / problems of "Matrix Programmer" (Chapter 1)
Big data analysis using the data flow control framework Luigi
Try to estimate the number of likes on Twitter
Get to know the feelings of gradient boosting trees
Write data to KINTONE using the Python requests module
I tried using the API of the salmon data project
Try to decipher the login data stored in Firefox
Script to get the expiration date of the SSL certificate
The story of using circleci to build manylinux wheels
The road to Pythonista
The road to Djangoist
[Question] How to get data of textarea data in real time using Python web framework bottle
How to get only the data you need from a structured data set using a versatile method
Try scraping the data of COVID-19 in Tokyo with Python
I tried to get the location information of Odakyu Bus
Get data using Ministry of Internal Affairs and Communications API
Try to evaluate the performance of machine learning / classification model
I want to get the operation information of yahoo route
Try to improve the accuracy of Twitter like number estimation
Try to solve the problems / problems of "Matrix Programmer" (Chapter 0 Functions)
Try to automate the operation of network devices with Python
View the contents of the queue using the RabbitMQ Management Web API
The story of copying data from S3 to Google's TeamDrive
Try to get data while port forwarding to RDS with anaconda.
[Python] I tried collecting data using the API of wikipedia
Try to model a multimodal distribution using the EM algorithm
Keras I want to get the output of any layer !!
How to get an overview of your data in Pandas
[Introduction to Python] How to get data with the listdir function
To get the name of the primitive etc. generated immediately before
Get the source of the page to load infinitely with python.
I tried to get data from AS / 400 quickly using pypyodbc
I sent the data of Raspberry Pi to GCP (free)