How to access the Datastore from the outside

I access the Datastore in a GAE / py environment, but sometimes I want to access it from GCE or a local environment.

I think there are two ways.

--GAE's Remote API (remote_api_shell.py)

There seems to be another approach. ..

--Google Cloud Datastore RPC API Wrapper Library --GAE Task Queue

Such.

Use Remote API

Accessing App Engine with Remote API A standard feature is the Remote API. Put python sdk in gcloud sdk? And I think the path is in remote_api_shell.py. Further on GAE's app.yaml

app.yaml


application: PROJECT_ID
version: 1
runtime: python27
api_version: 1

##Add here
builtins:
- remote_api: on

Enable remote_api for builtins as in.

When deploy is completed in that state,

$ remote_api_shell.py PROJECT_ID

By doing so, you will be able to directly touch the remote datastore etc. from the local python environment. If you can import google.appengine.ext.ndb in your local environment, you can also import the ndb model and access it in the same way as on App Engine, which is convenient.

from google.appengine.ext import ndb

class MyModel(ndb.Model):
    number = ndb.IntegerProperty(default=42)
    text = ndb.StringProperty()

#You can get remote data just like you get on GAE.
models = MyModel.query().fetch()

However, note that it is quite slow. Looking at the GAE log to find out the cause, スクリーンショット 2016-08-09 17.57.32.png

There is an http request to / _ah / remote_api. Probably because there are many http requests for each query, the overhead becomes large and the processing is heavy.

Use the Cloud Datastore API.

The Google Cloud Datastore API released this year is likely to solve performance issues. Google Cloud Datastore API new beta significantly improves performance

It is easy to understand if you do it while looking at this article. Getting started with the Google Cloud Datastore API

In situations where the gcloud command is enabled

$ pip install gcloud

Then you can install gcloud-python.

If gcloud auth is enabled, you can import and use datastore related libraries from python scripts or consoles.

from gcloud import datastore

client = datastore.Client(PROJECT_ID)
#Then you can get the iterator with query.
task = cli.query(kind='Task').fetch()

Check the tutorial and the following github page for how to use it. gcloud-python

First of all, the performance is better than the remote api. Also, it seems that the iterator also communicates and fetches every time the cursor comes there with delayed fetch, probably because the session is still pasted in the datastore. Therefore, when trying to get a lot of information like Remote API, it is less likely to get stuck.

important point

Please note that the version is still in beta, so specifications may change. In addition, the environment must be certified by gcloud. Furthermore, since there is no ORMapper like ndb, it is a little difficult to edit, and it may affect the GAE code.

However, it is convenient when accessing from the GCE environment, so you should definitely use this API from now on. In the official release, ndb etc. may be available.

Google Cloud Datastore RPC API Wrapper Library

The Datastore RPC API python wrapper library is officially available. GoogleCloudPlatform/google-cloud-datastore :googledatastore

For the time being, it seems that maintenance has been made finer following the update of the RPC API.

However, it seems that I can not recommend it very much because it requires authenticated credentials and the interface seems a little old.

Task Queue

If you use Push Queue etc. well, you can rewrite the data store information with an external trigger. However, I do not feel that I can access it because I have prepared tasks in advance and there is a time limit for tasks.

Recommended Posts

How to access the Datastore from the outside
How to operate Linux from the outside Procedure
How to access wikipedia from python
How to access RDS from Lambda (python)
How to operate Linux from the console
How to measure line speed from the terminal
[Python] How to remove duplicate values from the list
How to create an article from the command line
How to "cache enabled" access to FPGA memory from Linux
How to use the generator
How to instantly launch Jupyter Notebook from the terminal
How to access the global variable of the imported module
How to post a ticket from the Shogun API
How to use the decorator
How to increase the axis
How to start the program
How to calculate the amount of calculation learned from ABC134-D
How to log in automatically like 1Password from the CLI
How to do the initial setup from Django project creation
How to run the Export function of GCP Datastore automatically
How to hit NAPALM from the Web (NetDevOpsSec reality solution)
How to calculate the autocorrelation coefficient
How to use the zip function
How to use the optparse module
How to use SWIG from waf
How to read the SNLI dataset
How to get the Python version
How to overwrite the output to the console
How to convert from .mgz to .nii.gz
How to use the ConfigParser module
How to get followers and followers from python using the Mastodon API
[EC2] How to install and download chromedriver from the command line
How to extract the desired character string from a line 4 commands
How to resolve the error from toimage (from PIL.Image import fromarray as toimage)
How to display the progress bar (tqdm)
How to use the Spark ML pipeline
How to access environment variables in Python
How to easily convert format from Markdown
How to check the version of Django
How to solve the bin packing problem
How to update Google Sheets from Python
How to set the server time to Japanese time
How to manually update the AMP cache
How to utilize multi-core from multiple languages
[Linux] How to use the echo command
How to use the Linux grep command
Three ways to access the __main__ namespace
How to get colored output to the console
Access Google Calendar from the iOS app
How to create a repository from media
How to use the IPython debugger (ipdb)
From the introduction of GoogleCloudPlatform Natural Language API to how to use it
How to check local GAE from iPhone browser in the same LAN
Learning record (4th day) #How to get the absolute path from the relative path
Settings that allow IPython Notebook to be accessed from outside the local
How to build an application from the cloud using the Django web framework
How to pass values to JavaScript variables directly from the [Django] template tag
How to calculate the volatility of a brand
How to use the C library in Python
How to find the area of the Voronoi diagram
How to use MkDocs for the first time