Introduce of the parallel distributed Crawler with scraping Dynamic HTML
?
1 like?1,799 views
This document contains code for a web scraping application. It uses Sinatra to create an API endpoint that stores scraped URL and data. It also includes a Greasemonkey user script that posts scraped data using AJAX calls and redirects to the processed URL returned from the API. The user script scrapes Google Image search pages and sends the data to the Sinatra API for storage and processing.
![require 'rubygems'
require 'sinatra'
post '/' do
url = params[:url]
data = params[:data]
store(url, data)
next_url = process(url)
next_url
end](https://image.slidesharecdn.com/sprk02-091214080915-phpapp01/85/Introduce-of-the-parallel-distributed-Crawler-with-scraping-Dynamic-HTML-19-320.jpg)
Introduce of the parallel distributed Crawler with scraping Dynamic HTML
- 19. require 'rubygems' require 'sinatra' post '/' do url = params[:url] data = params[:data] store(url, data) next_url = process(url) next_url end
- 20. // ==UserScript== // @name greasi_scraper // @namespace http://libelabo.jp/ // @include http://images.google.co.jp/* // @require http://ajax.googleapis.com/ajax/libs/jquery/1.3.1/jquery.min.js // ==/UserScript== function postData(data) { var postData = $.param({url: location.href, data: JSON.stringify(data)}); GM_xmlhttpRequest({ method: "POST", url: "http://libelabo.jp/greasi/", headers: {'Content-type':'application/x-www-form-urlencoded'}, data: postData, onload: function(xhr){ location.href = xhr.responseText } }); }