I tried to make an image classification BOT by combining TensorFlow Lite and LINE Messaging API

As you study how to use machine learning libraries such as TensorFlow, you will think "I want to make something that works!". This time, I would like to create an image classification BOT by combining TensorFlow Lite, LINE Messaging API, and Heroku.

Why use Heroku

There are various choices such as AWS and GCP for the deployment destination of the created application. Heroku comes with a free tier that makes it relatively easy to get started. The free tier has restrictions such as the number of disguise environments that can be started and the usage time limit per month. It is recommended as an environment for deploying the sample APP.

Reasons to use TensorFlow Lite

This article uses Heroku's free tier as the deployment destination. In the free tier, the storage capacity is limited to about 300MB. Therefore, we use TensorFlow Lite, which can reduce the capacity of the model.

Reasons to use the LINE Messaging API

I think there are various apps that incorporate machine learning, such as Android and iOS smartphone apps and WEB apps. TensorFlow has libraries for Android and iOS environments, Of course, knowledge of application development is required to use them. You can also use TensorFlow.js to create apps in your browser or Node.js.

The app created this time is to "classify what is reflected in the image sent from the user". If you want to create it as a smartphone app, you need to develop both the smartphone app and the backend. I decided to use the LINE Messaging API because it is not an app that runs in a browser and I want to develop an app with a minimum of code.

Try to create a parrot return bot

I would like to make a simple parrot return BOT to check the operation of the BOT. I would like to briefly explain the operation flow of the LINE Messaging API.

  1. The user sends a message to the LINE platform.
  2. The LINE platform sends an HTTP POST request containing the event object to the BOT server.
  3. Verification is performed on the BOT server to confirm that the request was sent from the LINE platform.
  4. After clearing the verification, you will be able to acquire the content (text message, image message, etc.) sent by the user.
  5. Perform the necessary processing for the user request and send the message.

You may find it difficult to implement just by looking at this flow, but it is relatively easy to implement using the LINE Messaging API SDK. What actually needs to be discontinued is the "processing of messages sent by users" part.

Please check the official document for the sample code of LINE Messaging API SDK and Echolalia BOT. This time I'm using Python, but there are also SDKs for various languages ​​such as JAVA, PHP, GO, and Node.js. LINE Messaging API SDK https://github.com/line/line-bot-sdk-python

The created Echolalia BOT can be used from LINE by deploying it on Heroku. How to deploy to Heroku will be another long one, so I'd like to write another article.

Implementation of image classification BOT

The image classification BOT created this time can be implemented by applying the parrot return BOT. As a part to actually implement,

  1. Get the image sent by the user.
  2. Label what is shown in the image with the image classification model.
  3. Send the label as a text message to the user.

The implemented code is below.


#Model loading
model = Interpreter(model_path='Model path')
model.allocate_tensors()

#Get model input and output shapes
input_details = model.get_input_details()[0]
output_details = model.get_output_details()[0]

@handler.add(MessageEvent, message=ImageMessage)
def handle_image(event):

    #Get the content sent by the user
    image_url = f'https://api.line.me/v2/bot/message/{event.message.id}/content/'
    header = {
        "Content-Type": "application/json",
        "Authorization": "Bearer " + YOUR_CHANNEL_ACCESS_TOKEN
    }
    res = requests.get(image_url, headers=header)

    #Image preprocessing
    image = Image.open(BytesIO(res.content))
    input_data = process_image(input_details['shape'], image)

    #Predict the label of what is in the image
    model.set_tensor(input_details["index"], input_data)
    model.invoke()
    label = model.tensor(output_details["index"])().argmax()

    #Send the image label to the user by text message.
    line_bot_api.reply_message(
            event.reply_token,
            TextSendMessage(text=f'maybe...{label}'))

It can be operated as an image classification BOT by replacing the following part of the parrot return BOT.

@handler.add(MessageEvent, message=TextMessage)
def handle_message(event):
    line_bot_api.reply_message(
        event.reply_token,
        TextSendMessage(text=event.message.text))

sample: Screenshot_20210112-130019_2.png Screenshot_20210112-125609_2.png

Recommended Posts

I tried to make an image classification BOT by combining TensorFlow Lite and LINE Messaging API
I tried to make a simple image recognition API with Fast API and Tensorflow
How to make an artificial intelligence LINE bot with Flask + LINE Messaging API
I tried to get an image by scraping
I tried to make "Sakurai-san" a LINE BOT with API Gateway + Lambda
I made an image classification model and tried to move it on mobile
I tried to make an image similarity function with Python + OpenCV
I tried using Twitter api and Line api
I tried to make a Web API
I tried moving the image to the specified folder by right-clicking and left-clicking
I tried to make a ○ ✕ game using TensorFlow
I tried to extract a line art from an image with Deep Learning
I tried to make the weather forecast on the official line by referring to the weather forecast bot of "Dialogue system made with python".
I tried to make an automatic character dialogue generator by N floor Markov chain
I tried to implement Grad-CAM with keras and tensorflow
I tried to make an OCR application with PySimpleGUI
I tried to find an alternating series with tensorflow
I tried to verify the yin and yang classification of Hololive members by machine learning
I tried to make a bot that randomly acquires Wikipedia articles and tweets once a day
I tried to combine Discord Bot and face recognition-for LT-
I tried to make GUI tic-tac-toe with Python and Tkinter
I made a Chatbot using LINE Messaging API and Python
I tried to make it possible to automatically send an email just by double-clicking the [Python] icon
I made a Line bot that guesses the gender and age of a person from an image
I tried to make my own high school girl BOT with Rinna style with LINE BOT (Python & Heroku)
I tried to make an activity that collectively sets location information
I tried to make a periodical process with Selenium and Python
I tried to implement sentence classification by Self Attention with PyTorch
I tried to make an analysis base of 5 patterns in 3 years
[Python] Simple Japanese ⇒ I tried to make an English translation tool
[Machine learning] I tried to do something like passing an image
I made a Chatbot using LINE Messaging API and Python (2) ~ Server ~
LINE BOT (Messaging API) development with API Gateway and Lambda (Python) [Part 2]
I tried to create Quip API
I tried to touch Tesla's API
I tried to make it possible to automatically send an email just by double-clicking the [GAS / Python] icon
I tried to transform the face image using sparse_image_warp of TensorFlow Addons
I tried to summarize until I quit the bank and became an engineer
I tried to make an open / close sensor (Twitter cooperation) with TWE-Lite-2525A
Challenge image classification by TensorFlow2 + Keras 5 ~ Observe images that fail to classify ~
Challenge image classification by TensorFlow2 + Keras 7-Understanding layer types and activation functions-
I tried to make Kana's handwriting recognition Part 2/3 Data creation and learning
I tried to verify and analyze the acceleration of Python by Cython
I tried to classify Oba Hana and Emiri Otani by deep learning
I tried to make PyTorch model API in Azure environment using TorchServe
I tried to implement Autoencoder with TensorFlow
I tried to visualize AutoEncoder with TensorFlow
I tried to classify text using TensorFlow
I tried to touch the COTOHA API
I want to make an automation program!
I tried combining Fabric, Cuisine and Jinja2
I tried to make a periodical process with CentOS7, Selenium, Python and Chrome
I made a LINE BOT that returns a terrorist image using the Flickr API
[LINE Messaging API] I want to send a message from the program to everyone's LINE
Try to make BOT by linking spreadsheet and Slack with python 2/2 (python + gspread + slackbot)
I tried to automate internal operations with Docker, Python and Twitter API + bonus
Try to make BOT by linking spreadsheet and Slack with python 1/2 (python + gspread + slackbot)
I tried to make a castle search API with Elasticsearch + Sudachi + Go + echo
Python + chatwork + google extension = How to make an easy and funny chat BOT
I tried to make a translation BOT that works on Discord using googletrans
I tried to process and transform the image and expand the data for machine learning