A beginner tried coloring line art with chainer. I was able to do it.

Deplaning is tai2an, which is mainly in charge of work at P-FN, which is said to be commoditized and soldering has higher added value. At the Amazon Picking Challenge, which was broadcast nationwide on NHK, I made a sticky hand and a slide full of Robocon feeling. However, I wanted to do some deep learning, so I started studying chainer a month or two ago. I wanted to color the line art because it was a big deal, so I tried various things.

Line art coloring is supervised learning, so you need a dataset of line art and colored images (preferably in large quantities). This time, OpenCV is used to properly extract line art from color images.

Extraction example 6c55da4e7961024012e78b38db048f53ce8e3a31.jpg→ 6c55da4e7961024012e78b38db048f53ce8e3a31.jpg

Collect color images to create a line art and you have a dataset. (I'm using about 600,000 this time)

Regarding the form of the network, I used a network called U-net, which is used by merging the output of the layer when convolving at the beginning and deconvolving at the end. It seems to be suitable when referring to a line art and coloring it. (U-net is also used in px2px) For the time being, I tried to learn to minimize the square of the error from the correct answer. I understand that the network definition itself is generally OK if the output of the layer and the input of the next layer are matched, but it is a bit difficult to understand because there was not much way to create a dataset that you define yourself. I thought it was.

Here is the result of studying overnight and feeding test data ↓

スクリーンショット 2016-12-25 15.36.09.png

Well, I can feel the feeling of the neural network, "I can understand the skin color somehow, but I don't know anything else, I can't guess the color of the hair or clothes of the illustration character." .. ..

That's where the Adversary Net comes in. Known as Ad Auntie. Auntie is a guy who learns the difference between a real image and a colored image of a neural network and stingy. So, if you put out only skin-colored sepia-like images, you will learn in one shot and you will end up eating bad things.

However, if the ad woman is too strong, the colored side with the bent navel will be blurred, so be careful.

2e3a61e3da383137ae334afe52c1efacd7249a86.jpg2e3a61e3da383137ae334afe52c1efacd7249a86.jpg2e3a61e3da383137ae334afe52c1efacd7249a86.jpg

With this kind of feeling, the colors are surely attached, but it becomes an art system as if it was a line art. (Well, this may be an art path like this,

If you turn the learning while paying attention to the balance with the difference component from the original image. .. ..

スクリーンショット 2016-12-25 17.08.51.jpg

It's getting colored! Huh

It ’s even better! I also learned a network that reduces the 512x512px line art, paints 128x128px in the first stage, and colors 512x512px in the second stage (the network shape is almost the same in the second stage, but the input is 4ch and it is new. Retrain. This is no ad woman)

The result is here ↓

スクリーンショット 2016-12-25 15.11.38.jpg

(I borrowed this line art published by @ lio8644.)

It's okay.

スクリーンショット 2016-12-25 15.56.10.jpg

It's not bad.

As long as I use the extracted line art for testing, it looks pretty good, but what about the actual line art?

I borrowed a line art from a tag of pixiv's painting system and tested it. (Basically all are CNN, so even if the aspect ratio changes a little, it can be handled.

Denden

スクリーンショット 2016-12-25 15.03.39.jpg

great.

スクリーンショット 2016-12-25 14.58.21.jpg

It became a colorful monster, but there are some like this.

スクリーンショット 2016-12-25 17.29.48.jpg

Feeling safely summarized

By the way, after all there is something like wanting you to paint this color here, right? That's why I changed the input of the first stage to 4ch so that I can give additional hints.

Tsumari

スクリーンショット 2016-12-25 17.49.07.jpg

You can set requirements such as brown hair, sweater light blue, and skirt navy blue.

If you give a rough impression that you want it to look like this, it will be pretty good.

スクリーンショット 2016-12-25 14.57.25.jpg スクリーンショット 2016-12-25 15.03.03.jpg

(If the line art is good, I feel like it's beautiful if I paint it, but w

It is also possible to give a lot of hints in quite detail. (It's a little difficult to understand, but I'm hitting a lot of hints

スクリーンショット 2016-12-25 14.53.10.jpg

Merry Christmas! !!

With this, I'm also promoted from a craftsman to a drawing craftsman!

So, I think that the automatic coloring of line art and the coloring with hints have been done considerably.

I don't think I can beat the proper painting of the painters, but I think it's convenient to paint roughly. It would be convenient if it was faster to color the manga roughly than to paste the tone. (This neural network is strong for skin color ... Do you know what you want to say?)

As a side note, there are still some weaknesses.

For example, since you are learning Adversary Net and hints at the same time, the ripple effect when giving hints may become unstable.

スクリーンショット 2016-12-25 16.35.04.jpg

↑ I wanted only the swimsuit to be painted in a different color, but the colors of the other parts have also changed significantly. If you use it only as an easy coloring tool, it may be more stable to learn with hints only.

Also, since the color is painted after reducing it once, even if the line art is too thick / thin, there are cases where the line is crushed or skipped and it does not work, or even if you instruct fine painting with hints In some cases, it will not be reflected.

It would be cool if one NN could handle all the details, but it seems necessary to make adjustments for each application when using it as a tool.

Line art borrower http://www.pixiv.net/member_illust.php?mode=medium&illust_id=31274285 http://www.pixiv.net/member_illust.php?mode=manga&illust_id=43369404 http://www.pixiv.net/member_illust.php?mode=medium&illust_id=56689287 http://www.pixiv.net/member_illust.php?mode=medium&illust_id=40487409 http://www.pixiv.net/member_illust.php?mode=medium&illust_id=10552795 https://twitter.com/lio8644

By the way, the network structure this time is the same for both the first and second stages, and it looks like this

unet.py


    class UNET(chainer.Chain):
        def __init__(self):
            super(UNET, self).__init__(
                c0 = L.Convolution2D(4, 32, 3, 1, 1),
                c1 = L.Convolution2D(32, 64, 4, 2, 1),
                c2 = L.Convolution2D(64, 64, 3, 1, 1),
                c3 = L.Convolution2D(64, 128, 4, 2, 1),
                c4 = L.Convolution2D(128, 128, 3, 1, 1),
                c5 = L.Convolution2D(128, 256, 4, 2, 1),
                c6 = L.Convolution2D(256, 256, 3, 1, 1),
                c7 = L.Convolution2D(256, 512, 4, 2, 1),
                c8 = L.Convolution2D(512, 512, 3, 1, 1),

                dc8 = L.Deconvolution2D(1024, 512, 4, 2, 1),
                dc7 = L.Convolution2D(512, 256, 3, 1, 1),
                dc6 = L.Deconvolution2D(512, 256, 4, 2, 1),
                dc5 = L.Convolution2D(256, 128, 3, 1, 1),
                dc4 = L.Deconvolution2D(256, 128, 4, 2, 1),
                dc3 = L.Convolution2D(128, 64, 3, 1, 1),
                dc2 = L.Deconvolution2D(128, 64, 4, 2, 1),
                dc1 = L.Convolution2D(64, 32, 3, 1, 1),
                dc0 = L.Convolution2D(64, 3, 3, 1, 1),

                bnc0 = L.BatchNormalization(32),
                bnc1 = L.BatchNormalization(64),
                bnc2 = L.BatchNormalization(64),
                bnc3 = L.BatchNormalization(128),
                bnc4 = L.BatchNormalization(128),
                bnc5 = L.BatchNormalization(256),
                bnc6 = L.BatchNormalization(256),
                bnc7 = L.BatchNormalization(512),
                bnc8 = L.BatchNormalization(512),

                bnd8 = L.BatchNormalization(512),
                bnd7 = L.BatchNormalization(256),
                bnd6 = L.BatchNormalization(256),
                bnd5 = L.BatchNormalization(128),
                bnd4 = L.BatchNormalization(128),
                bnd3 = L.BatchNormalization(64),
                bnd2 = L.BatchNormalization(64),
                bnd1 = L.BatchNormalization(32)
        )

    def calc(self,x, test = False):
        e0 = F.relu(self.bnc0(self.c0(x), test=test))
        e1 = F.relu(self.bnc1(self.c1(e0), test=test))
        e2 = F.relu(self.bnc2(self.c2(e1), test=test))
        e3 = F.relu(self.bnc3(self.c3(e2), test=test))
        e4 = F.relu(self.bnc4(self.c4(e3), test=test))
        e5 = F.relu(self.bnc5(self.c5(e4), test=test))
        e6 = F.relu(self.bnc6(self.c6(e5), test=test))
        e7 = F.relu(self.bnc7(self.c7(e6), test=test))
        e8 = F.relu(self.bnc8(self.c8(e7), test=test))

        d8 = F.relu(self.bnd8(self.dc8(F.concat([e7, e8])), test=test))
        d7 = F.relu(self.bnd7(self.dc7(d8), test=test))
        d6 = F.relu(self.bnd6(self.dc6(F.concat([e6, d7])), test=test))
        d5 = F.relu(self.bnd5(self.dc5(d6), test=test))
        d4 = F.relu(self.bnd4(self.dc4(F.concat([e4, d5])), test=test))
        d3 = F.relu(self.bnd3(self.dc3(d4), test=test))
        d2 = F.relu(self.bnd2(self.dc2(F.concat([e2, d3])), test=test))
        d1 = F.relu(self.bnd1(self.dc1(d2), test=test))
        d0 = self.dc0(F.concat([e0, d1]))

        return d0

Ad Auntie

adv.py


class DIS(chainer.Chain):
    def __init__(self):
        super(DIS, self).__init__(
                c1 = L.Convolution2D(3, 32, 4, 2, 1),
                c2 = L.Convolution2D(32, 32, 3, 1, 1),
                c3 = L.Convolution2D(32, 64, 4, 2, 1),
                c4 = L.Convolution2D(64, 64, 3, 1, 1),
                c5 = L.Convolution2D(64, 128, 4, 2, 1),
                c6 = L.Convolution2D(128, 128, 3, 1, 1),
                c7 = L.Convolution2D(128, 256, 4, 2, 1),
                l8l = L.Linear(None, 2, wscale=0.02*math.sqrt(8*8*256)),

                bnc1 = L.BatchNormalization(32),
                bnc2 = L.BatchNormalization(32),
                bnc3 = L.BatchNormalization(64),
                bnc4 = L.BatchNormalization(64),
                bnc5 = L.BatchNormalization(128),
                bnc6 = L.BatchNormalization(128),
                bnc7 = L.BatchNormalization(256),
        )

    def calc(self,x, test = False):
        h = F.relu(self.bnc1(self.c1(x), test=test))
        h = F.relu(self.bnc2(self.c2(h), test=test))
        h = F.relu(self.bnc3(self.c3(h), test=test))
        h = F.relu(self.bnc4(self.c4(h), test=test))
        h = F.relu(self.bnc5(self.c5(h), test=test))
        h = F.relu(self.bnc6(self.c6(h), test=test))
        h = F.relu(self.bnc7(self.c7(h), test=test))
        return  self.l8l(h)

Recommended Posts

A beginner tried coloring line art with chainer. I was able to do it.
I tried to extract a line art from an image with Deep Learning
Since there was a doppelganger, I tried to distinguish it with artificial intelligence (laugh) (Part 2)
There was a doppelganger, so I tried to distinguish it with artificial intelligence (laughs) (Part 1)
A memorandum when I tried to get it automatically with selenium
I tried to make a calculator with Tkinter so I will write it
I tried to make "Sakurai-san" a LINE BOT with API Gateway + Lambda
When I tried to make a VPC with AWS CDK but couldn't make it
When I tried to create a virtual environment with Python, it didn't work
I was able to mock AWS-Batch with python, moto, so I will leave it
In IPython, when I tried to see the value, it was a generator, so I came up with it when I was frustrated.
I tried to learn the sin function with chainer
I tried to create a table only with Django
I tried to draw a route map with Python
I tried to automatically generate a password with Python3
I was able to repeat it in Python: lambda
When I tried to do socket communication with Raspberry Pi, the protocol was different
It was a little difficult to do flask with the docker version of nginx-unit
AtCoder Beginner Contest 177 Problem C I tried to find out why it was wrong
A Python beginner made a chat bot, so I tried to summarize how to make it.
Image processing with Python (I tried binarizing it into a mosaic art of 0 and 1)
I tried to make creative art with AI! I programmed a novelty! (Paper: Creative Adversarial Network)
I tried to implement a volume moving average with Quantx
[IOS] GIF animation with Pythonista3. I was addicted to it.
I was able to implement web app authentication with flask-login
I tried to automatically create a report with Markov chain
I tried to notify the train delay information with LINE Notify
I tried to solve a combination optimization problem with Qiskit
I tried to get started with Hy ・ Define a class
I tried to implement ListNet of rank learning with Chainer
I tried to sort a random FizzBuzz column with bubble sort.
A python beginner tried to intern at an IT company
I tried to divide with a deep learning language model
When I tried to install PIL and matplotlib in a virtualenv environment, I was addicted to it.
When I tried the AtCoder Beginner Contest, it was a terrible result, so I look back
I made a server with Python socket and ssl and tried to access it from a browser
[5th] I tried to make a certain authenticator-like tool with python
[2nd] I tried to make a certain authenticator-like tool with python
[Python] A memo that I tried to get started with asyncio
I tried to create a list of prime numbers with python
I want to do a full text search with elasticsearch + python
I tried to make a periodical process with Selenium and Python
I tried to summarize what was output with Qiita with Word cloud
A note I was addicted to when creating a table with SQLAlchemy
I tried to make a 2channel post notification application with Python
I tried to create Bulls and Cows with a shell program
I tried to make a todo application using bottle with python
[4th] I tried to make a certain authenticator-like tool with python
Homework was a pain, so I do management accounting with Python
I want to do ○○ with Pandas
[1st] I tried to make a certain authenticator-like tool with python
I tried to make a strange quote for Jojo with LSTM
I tried to make a mechanism of exclusive control with Go
Even if I converted jpg to png, I managed to do it because the transparency was not tampered with
I tried to introduce a serverless chatbot linked with Rakuten API to Teams
Python: I tried to make a flat / flat_map just right with a generator
I want to do it with Python lambda Django, but I will stop
I tried to communicate with a remote server by Socket communication with Python.
I tried to implement a blockchain that actually works with about 170 lines
When I tried to run Python, it was skipped to the Microsoft Store
I tried to create a program to convert hexadecimal numbers to decimal numbers with python