{"id":131,"date":"2021-04-29T10:32:03","date_gmt":"2021-04-29T10:32:03","guid":{"rendered":"https:\/\/explicitly.me\/?p=131"},"modified":"2024-03-21T11:40:50","modified_gmt":"2024-03-21T11:40:50","slug":"serp-scraping-for-fun-and-profit","status":"publish","type":"post","link":"https:\/\/explicitly.me\/serp-scraping-for-fun-and-profit\/","title":{"rendered":"Serp Scraping For Fun and Profit"},"content":{"rendered":"\n<p>So a little while ago I pointed out that Facebook is running a massive&nbsp;Grayhat Strategy to Rank for Longtail&nbsp;in the SERPs, essentially carrying out&nbsp;<a href=\"\/serp-sniffing-a-long-tail-keyword-strategy\">SERP Sniffing to an insanely large scale<\/a>, with the view of potentially building a community driven&nbsp;<a href=\"\/content-farms\">Content Farm<\/a>. Some really interesting questions popped to mind:<\/p>\n\n\n\n<ul>\n<li>How many keywords are they ranking for?<\/li>\n\n\n\n<li>How well are they ranking?<\/li>\n\n\n\n<li>What sort of traffic volumes would they get?<\/li>\n\n\n\n<li>What would the commercial value of that traffic be?<\/li>\n<\/ul>\n\n\n\n<p>Not all of these are easy to answer, and may or may not be of interest. So I expanded my questions with:<\/p>\n\n\n\n<ul>\n<li>How can I check their Rankings?<\/li>\n<\/ul>\n\n\n\n<p>So I had a rough answer.&nbsp;<strong>Scrape and Rank<\/strong>. In reality, this would not be possible without the Dark Art of Scraping Google Search Results.<\/p>\n\n\n\n<p>Which kind of lead me to think, this is a pretty simple, yet good method of referencing rankings. However scraping need not be bad. Like any SEO technique, it can be used for both,good and evil.<\/p>\n\n\n\n<ul>\n<li>How can I use that technique for blackhat purposes?<\/li>\n\n\n\n<li>How can I use that technique for whitehat purposes?<\/li>\n<\/ul>\n\n\n\n<p>So I have figured a way to answer SOME of my questions. I then asked myself:<\/p>\n\n\n\n<p><strong>Can I create a simple and rough version of my technique so that ANYONE can use and verify?<\/strong><\/p>\n\n\n\n<p>The answer was, yes, to an extent. The recipe would include:<\/p>\n\n\n\n<ol>\n<li>A ScrapingScript,<\/li>\n\n\n\n<li>A Rank Checker,<\/li>\n\n\n\n<li>A Spreadsheet<\/li>\n\n\n\n<li>A little time.<\/li>\n<\/ol>\n\n\n\n<p><strong>Summary:<\/strong>&nbsp;In the post below I demonstrate<\/p>\n\n\n\n<ul>\n<li>The potential extent to which Facebook is gaming the SERPs with crappy content<\/li>\n\n\n\n<li>The ability to use Scraping for both white and black SEO techniques<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">&nbsp;<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1227\" title=\"Scraping Assbook\" src=\"https:\/\/web.archive.org\/web\/20120920080503im_\/https:\/\/explicitly.me\/wp-content\/uploads\/2011\/03\/Scraping-Assbook.jpg\" alt=\"Scraping Assbook\" width=\"511\" height=\"100\"><canvas width=\"66\" height=\"19\"><\/canvas><canvas width=\"28\" height=\"19\"><\/canvas><\/h4>\n\n\n\n<ul>\n<li>Scrape results for:&nbsp;&nbsp;<a href=\"https:\/\/www.google.com\/search?hl=en&amp;q=site%3Afacebook.com%2F+%22Community+Pages+are+not+affiliated+with%2C+or+endorsed+by%2C+anyone+associated+with+the+topic\" target=\"_blank\" rel=\"nofollow noopener\">http:\/\/www.google.com\/search?hl=en&amp;q=site%3Afacebook.com%2F+%22Community+Pages+are+not+affiliated+with%2C+or+endorsed+by%2C+anyone+associated+with+the+topic<\/a>&nbsp;This query ideally finds all community pages indexed from the Facebook domain.<\/li>\n\n\n\n<li>Capture the title tags: e.g&nbsp; \u201cAssociation football | Facebook\u201d.<\/li>\n\n\n\n<li>Export to excel, and edit field to delete all references to \u201cFacebook\u201d to get a spreadsheet of keywords to check.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">&nbsp;<\/h4>\n\n\n\n<ul>\n<li>Grab the keywords and Rank Check for google .com<\/li>\n\n\n\n<li>Compile and sort by everything in positions:\n<ul>\n<li>Pos 1<\/li>\n\n\n\n<li>Pos 2-5<\/li>\n\n\n\n<li>Pos 6-10<\/li>\n\n\n\n<li>Page 2<\/li>\n\n\n\n<li>Pos 21-50<\/li>\n\n\n\n<li>Pos 51-100<\/li>\n\n\n\n<li>Pos 101-200<\/li>\n\n\n\n<li>Not ranked<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Sort into pie chart, and analyse.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">&nbsp;<\/h3>\n\n\n\n<p>I scraped the first 600 results of the query, with the help of the awesome&nbsp;<a href=\"https:\/\/twitter.com\/RichardShove\" target=\"_blank\" rel=\"nofollow noopener\">Richard Shove<\/a>. &nbsp;Well I had to strip 8 keywords for non eligibility \/ repeat.&nbsp; Also, when ranking, I only considered ranks up to the first 200 results. And I define a good rank to average rank as anything in the top 50.<\/p>\n\n\n\n<p>What did I find? The ranking data makes good reading, even for such a small dataset:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"467\" height=\"289\" src=\"https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Google-SERP-Rankings-for-Facebook.jpeg\" alt=\"\" class=\"wp-image-173\" srcset=\"https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Google-SERP-Rankings-for-Facebook.jpeg 467w, https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Google-SERP-Rankings-for-Facebook-300x186.jpeg 300w\" sizes=\"(max-width: 467px) 100vw, 467px\" \/><\/figure>\n\n\n\n<p>The data shows that of my keywords selected, only&nbsp;<strong>25% weren\u2019t&nbsp;<\/strong>ranked in the top 200 positions. Of the same data set,&nbsp;<strong>17% ranked on page 1 of the SERPs<\/strong>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">&nbsp;<\/h3>\n\n\n\n<p>Now lets look at that in theoretical terms. The query I used to generate this list indicates\u00a0<strong>119,000,000<\/strong>\u00a0results found.(That is\u00a0<strong>One hundred and Nineteen MILLION<\/strong>\u00a0results!!!)<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"800\" height=\"94\" src=\"https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Site-Query-Equals-119-000-000-Results-222.jpeg\" alt=\"\" class=\"wp-image-172\" srcset=\"https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Site-Query-Equals-119-000-000-Results-222.jpeg 800w, https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Site-Query-Equals-119-000-000-Results-222-300x35.jpeg 300w, https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Site-Query-Equals-119-000-000-Results-222-768x90.jpeg 768w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/><\/figure>\n\n\n\n<p>This is nowhere near the real number indexed.\u00a0 Why do I say that? Well you will get different indexed results depending on how quick google serves up the data to you\u2026 So refreshing the search gives me a\u00a0<strong>different number indexed<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"812\" height=\"105\" src=\"https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Site-Query-Equals-219-000-000-Results.jpeg\" alt=\"\" class=\"wp-image-171\" srcset=\"https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Site-Query-Equals-219-000-000-Results.jpeg 812w, https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Site-Query-Equals-219-000-000-Results-300x39.jpeg 300w, https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Site-Query-Equals-219-000-000-Results-768x99.jpeg 768w\" sizes=\"(max-width: 812px) 100vw, 812px\" \/><\/figure>\n\n\n\n<p>Lets just say for simplicities sake we have 100,000,000 pages indexed. If the data set above forms a working sample, then you expect&nbsp;<strong>75%<\/strong>&nbsp;of these results ranking for the keywords necessary.&nbsp;<strong>17%&nbsp;<\/strong>of the same&nbsp; dataset ranks in page one.<\/p>\n\n\n\n<p>BIG picture? If (and note I say&nbsp;<strong>IF<\/strong>) my hypothesis holds, then these community pages that rank on page one equal in the region of 17,000,000. That is&nbsp;<strong>Seventeen MILLION<\/strong>&nbsp;folks. What traffic wold you hope to generate with 17 Million results on page 1 in google?<\/p>\n\n\n\n<p>Not finished yet. The three most common variations on these pages title tags include:<\/p>\n\n\n\n<ul>\n<li>Community<\/li>\n\n\n\n<li>Interest<\/li>\n\n\n\n<li>Topic<\/li>\n<\/ul>\n\n\n\n<p>Now cross reference any keywords that have these extentions added, in order to make a mid range keyphrase. I can only guess at the volume of page one rankings for those keyphrase variations.<\/p>\n\n\n\n<p>At this juncture, I must point out that my friend&nbsp;Branko&nbsp;who is an awesome&nbsp;SEO Scientist, highlighted the fact that my methodology&nbsp; needs verification:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\">\n<p>If I was you, I would take 2 more random sets of 600 queries and see if the distribution is similar. That way you can get an idea of how much fluctuations you have and how solid your data is. If you want to take it to the next level, there are statistical tests that can tell you whether your sample is representative of the population.<\/p>\n<\/blockquote>\n\n\n\n<p>Now I was too lazy to do that to be honest. However I am sure those of you who are better at analysing and manipulating sata may be tempted. If so, let me know!<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">&nbsp;<\/h3>\n\n\n\n<p>Now this is a massive massive invasion of the SERPs. And if I had the full X Million KW list to hand, I would love to have dug through it. But even my unscientific approach indicated something we British would call \u201ccorkers\u201d .\u00a0<a href=\"https:\/\/www.seomoz.org\/ugc\/coke-pissed-off-danny-sullivan-twitter-insights\" target=\"_blank\" rel=\"nofollow noopener\">Danny Sullivan taught me a neat trick years ago<\/a>, of comparing a high volume KW against another to get an indication of popularity. My list indicates that Facebook ranks well for \u201cSinging\u201d. Now lets use Danny\u2019s technique:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"601\" height=\"350\" src=\"https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Singing-vs-Viagra.jpeg\" alt=\"\" class=\"wp-image-170\" srcset=\"https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Singing-vs-Viagra.jpeg 601w, https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Singing-vs-Viagra-300x175.jpeg 300w\" sizes=\"(max-width: 601px) 100vw, 601px\" \/><\/figure>\n\n\n\n<p>How cool is that?&nbsp;<a href=\"https:\/\/www.google.com\/insights\/search\/#q=%22Singing%22%2C%20%2C%22viagra%22&amp;date=today%2012-m&amp;cmpt=q\" target=\"_blank\" rel=\"nofollow noopener\">Singing gets more searches than Viagra<\/a>. and here is a scraper page auto built by Facebook ranking for it. I include this because it indicates the potential volume of traffic that can be had by a large scale site that gets involved in Gray SEO techniques.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">White Hat Scraping<\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"585\" height=\"100\" src=\"https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Whitehat-Scraping.jpeg\" alt=\"\" class=\"wp-image-169\" srcset=\"https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Whitehat-Scraping.jpeg 585w, https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Whitehat-Scraping-300x51.jpeg 300w\" sizes=\"(max-width: 585px) 100vw, 585px\" \/><\/figure>\n\n\n\n<p>When working on large scale SEO projects, for example Ecommerce SEO, competitive analysis is key in successful Long to Mid range strategy. Often we as SEOs tend to hone in on the \u201cmoney\u201d words and forget that the long tail not only exists, but is highly profitable.<\/p>\n\n\n\n<p>I am not saying that we DONT target long tail keywords, but I dont think we competitively analyse this data.<\/p>\n\n\n\n<p>So how do you use SERP Scraping for comptitive analysis? Do I have to really tell you?&nbsp;<img decoding=\"async\" class=\"wp-smiley\" src=\"https:\/\/web.archive.org\/web\/20120920080503im_\/https:\/\/explicitly.me\/wp-includes\/images\/smilies\/icon_razz.gif\" alt=\":P\"><\/p>\n\n\n\n<ul>\n<li>First off, scrape all the pages indexed for the competitor in question.<\/li>\n\n\n\n<li>Second, most common ecommerce SEO set ups use keyword splitters in the title tag. (They could be pipes, commas, arrows etc etc.) So use that knowledge to pull off keywords from your scraped data.<\/li>\n\n\n\n<li>Third, run a rank check on the full list.<\/li>\n\n\n\n<li>Four, compare against your own data. Where are the gaps?<\/li>\n\n\n\n<li>Profit.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Blackhat Scraping<\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"585\" height=\"100\" src=\"https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Blackhat-Scraping.jpeg\" alt=\"\" class=\"wp-image-168\" srcset=\"https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Blackhat-Scraping.jpeg 585w, https:\/\/explicitly.me\/wp-content\/uploads\/2024\/03\/Blackhat-Scraping-300x51.jpeg 300w\" sizes=\"(max-width: 585px) 100vw, 585px\" \/><\/figure>\n\n\n\n<p>Again, this is quite a simple use of that SERP rank data.<\/p>\n\n\n\n<ul>\n<li>Find a number of large sized sites with average or poor SEO<\/li>\n\n\n\n<li>Scrape all the pages indexed.<\/li>\n\n\n\n<li>You can use title tags, headings, etc to compile keywords from your scraped data.<\/li>\n\n\n\n<li>Third, run a rank check on the full list.<\/li>\n\n\n\n<li>Four, sort the data. Start with one word, two word, and continue sequencing till you get to 4-5 word combinations.<\/li>\n\n\n\n<li>Now you have an insanely large keyword list with rankings \u2013 sorted into&nbsp; keyword tails.<\/li>\n\n\n\n<li>I would then further sort these into \u201ccommon\u201d themes, e.g all car and auto related words \/ phrases together\u2026<\/li>\n\n\n\n<li>Use an&nbsp;<a href=\"http:\/\/bit.ly\/WPRobotAggragation\" target=\"_blank\" rel=\"nofollow noopener\">autoblog tool such as WPRobot<\/a>&nbsp;to create thematic microsites which automatically pump out content based around your keyword sets. Add links where necessary.<\/li>\n\n\n\n<li>Profit?!<\/li>\n<\/ul>\n\n\n\n<p>A serious Blackhat would probably know how to use Scraped SERPs for much much more than what I suggest above. However, I have chosen to demonstrate the lowest common denominator&nbsp;<img decoding=\"async\" class=\"wp-smiley\" src=\"https:\/\/web.archive.org\/web\/20120920080503im_\/https:\/\/explicitly.me\/wp-includes\/images\/smilies\/icon_smile.gif\" alt=\":)\">&nbsp;&nbsp; After all, this is a similar technique to the&nbsp;<a href=\"https:\/\/www.seobook.com\/black-hat-seo-case-study\" target=\"_blank\" rel=\"nofollow noopener\">one I would say Mahalo used<\/a>\u2026<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">&nbsp;<\/h3>\n\n\n\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" src=\"https:\/\/web.archive.org\/web\/20120920080503im_\/https:\/\/explicitly.me\/wp-content\/uploads\/2011\/03\/Free-Scraper.jpg\" alt=\"Free Scraper\" class=\"wp-image-1221\" title=\"Free Scraper\"\/><\/figure>\n\n\n\n<p>So there are a bunch of tools you can use to scrape, but to show you how easy it can be, see the link to the&nbsp;<a href=\"https:\/\/web.archive.org\/web\/20120920080503\/https:\/\/explicitly.me\/wp-content\/uploads\/2011\/03\/Google-Scraping-Script.txt\">Google Scraping Script<\/a>. (A big thank you to&nbsp;<a href=\"http:\/\/www.wpdoctors.co.uk\/\" target=\"_blank\" rel=\"noopener\">Dan Harrison of WordPress Doctors<\/a>&nbsp;and&nbsp;William Vicary of Semto).<\/p>\n\n\n\n<p>In case you think this is the only way to do this,&nbsp; here is another variant of the&nbsp;<a href=\"https:\/\/explicitly.me\/wp-content\/uploads\/2011\/03\/Google-Scraping-Script-2.txt\">scraping script<\/a>&nbsp;(thanks to Yousaf Sekander of&nbsp;&nbsp;Elevate Local).<\/p>\n\n\n\n<p>This is just a limited version of the scraper and will take ages to pull out industrial strength data. If you are looking for something much more robust, try:<\/p>\n\n\n\n<ul>\n<li>Scrapebox<\/li>\n\n\n\n<li>80 Legs<\/li>\n\n\n\n<li>Scrapy<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>So a little while ago I pointed out that Facebook is running a massive&nbsp;Grayhat Strategy to Rank for Longtail&nbsp;in the SERPs, essentially carrying out&nbsp;SERP Sniffing to an insanely large scale, with the view of potentially building a community driven&nbsp;Content Farm. Some really interesting questions popped to mind: Not all of these are easy to answer, [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":159,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"_links":{"self":[{"href":"https:\/\/explicitly.me\/wp-json\/wp\/v2\/posts\/131"}],"collection":[{"href":"https:\/\/explicitly.me\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/explicitly.me\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/explicitly.me\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/explicitly.me\/wp-json\/wp\/v2\/comments?post=131"}],"version-history":[{"count":7,"href":"https:\/\/explicitly.me\/wp-json\/wp\/v2\/posts\/131\/revisions"}],"predecessor-version":[{"id":174,"href":"https:\/\/explicitly.me\/wp-json\/wp\/v2\/posts\/131\/revisions\/174"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/explicitly.me\/wp-json\/wp\/v2\/media\/159"}],"wp:attachment":[{"href":"https:\/\/explicitly.me\/wp-json\/wp\/v2\/media?parent=131"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/explicitly.me\/wp-json\/wp\/v2\/categories?post=131"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/explicitly.me\/wp-json\/wp\/v2\/tags?post=131"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}