Ruby Scraping-Move Selenium Headless with VPS.

Development environment

ruby2.7.1 centos7

procedure

① Let's add Gem

gem install gem install selenium-webdriver

② Install Google Chrome

install yum install google-chrome-stable Check the version google-chrome -version

③ Install Google Chrome

Download the same version of Chrome Driver as google-chrome from http://chromedriver.chromium.org/downloads

Below is a command from Google Chrome version 83.0.4103.39 wget https://chromedriver.storage.googleapis.com/index.html?path=83.0.4103.39/

Deployment unzip chromedriver_linux64.zip Move sudo mv chromedriver ~/usr/local/bin/ Change permissions sudo chmod 755 /usr/local/bin/chromedriver

Ready

④ Sample code using Selenium

scraping.rb

def scraping
  url = 'https://google.com/' #URL you want to open
  options = Selenium::WebDriver::Chrome::Options.new　#New option
  options.add_argument('--headless') #Headless option added
  options.add_argument('--disable-gpu') #Disable GPU
  options.add_argument('--window-size=4000,4000') #Maximize screen
  driver = Selenium::WebDriver.for :chrome, options: options　#New driver to reflect options
#Scraping code as you like
.
.
.
.
end

--headless and --disable-gpu added for headless processing for Linux The addition of the option --window-size = 4000,4000 is written because scraping can be done only in the part displayed on the screen.

• If you want to display GUI on linux, you need to install virtual display separately *

* For those who create automatic Bitcoin trading tools (because there are many these days)

Selenium is great for automating sites that don't have an API. You don't need Selenium for automatic Bitcoin trading. I think it is better to make it using Bitflyer's API as shown below. http://benzenetarou.hatenablog.com/entry/bitcoin/automatic_trade/1