Basic web scraping and data visualization using Google Spreadsheets

Google Spreadsheets provides a free, one-stop solution for journalists and researchers to retrieve tabular data from a web page, visualize the data, and embed the visualizations in a news or research report.

There are times when a journalist or researcher needs to incorporate visualized data in a report, and the data reside on a third-party website, in the form of a table. It is tedious and time-consuming to manually copy/paste the table, take care of any formatting issues, find a way to visualize the data, and upload the visualizations somewhere on the web for sharing and embedding.

With a free Google account, in three steps, we can create a spreadsheet, let the spreadsheet retrieve data in that table, perform various visualizations with the retrieved data, and then share the visualizations through different venues.

Step 1: Retrieve (scrape) tabular data

Google Spreadsheets provides a large collection of “functions” that do various things. This tutorial is based on one particular function that retrieves tabular data on a web page; for other spreadsheet functions, please refer to the Google Spreadsheet function list.

Let’s say that in a report about U.S. population, we want to quote state-by-state population data, in the form of an interactive chart, off of the Wikipedia list of U.S. states, which is a table with 12 columns and dozens of rows.

With a new blank spreadsheet, type in the A1 cell (the upper-left cell) the following function code:

This formula tells Google Spreadsheets to go to the said web page, look for tables on that page, and fetch data in the second table. Once I hit the enter key, in a few moments, the blank spreadsheet is populated with data as shown in the screenshot below:

U.S. population

A few notes about this step:

    To scrape data from any other page, copy that web address and paste it within the quotation marks.

    For that number “2″ in the formula – a web page may contain several tables; some are visible, some invisible. If you don’t know how to read HTML source codes and pinpoint the exact table you want to target, then start with 1, and work your way up until the spreadsheet returns the data you want.

    You cannot edit this spreadsheet as it is dynamically linked to the source on that Wiki page. If you delete a column or cell, the spreadsheet will refresh itself with the deleted contents back in place.

    If you do need to edit the data, for instance to create a customized chart of selected columns/rows, you need to make a copy of the spreadsheet. Remember, when you copy/paste to a new spreadsheet, choose edit>paste special>paste values only, so that the new spreadsheet is not linked to the source.

Step 2: Visualize data

Google Spreadsheets provides various visualization tools that we can choose and customize to suit our needs.

For instance, I want to visualize and compare the 2012 estimated population of the top-10 states. To do that, I need to specify the range of cells for the visualization; in this case, it is columns C and D, from row 1 down to row 11 (we need to include the header in row 1).

This range is thus C1:D11, which is a standard way to specify spreadsheet ranges.

(Note: read a Goggle Spreadsheet tutorial for working with cell range and other spreadsheet features)

On the spreadsheet menu bar, click on the “Insert chart” button (third from right); in the Chart Editor pop-up window, there are three tabs where we can customize the chart:

    chart-editorUnder the Start tab, change the data range to Sheet1!C1:D11; leave other options as is, and make sure “use row 1 as headers” is checked. Based on the data type, the spreadsheets has a few recommended charts; I selected the bar chart.

    Click the Chart tab and explore other types of charts; charts not compatible with the current data set are not available (grayed out).

    Click the Customize tab; here, among others, give the chart a title, and scroll down to change the default names of the vertical axis and the horizontal axis.

    Click Insert and the chart will be shown on top of the spreadsheet; click upper-right corner of the chart window, click the drop-down menu and choose “move to own sheet.” The chart will be shown in a separate sheet with a default name of “Chart 1.”

Step 3: Share chart visualizations

By default, the spreadsheet is private, meaning it is only visible to its creator. For others to view our chart, we need to change the sharing settings: Click on the blue “Share” button on the upper-right corner; in the pop-up window, under “who has access,” click on “change” next to the “private” option, choose either “public on the web” or “anyone with the link,”  click Save then Done.

Click File>Publish to the web; in the pop-up window, under “Get a link to the published data,” there are a few options:

    Click in the first box and you will see a group of options for how you can share the data/chart. For our purpose, we want to embed the interactive chart in this blog post, so I selected “HTML to embed in a page.”

    Click in the “All sheets” box and choose “Chart 1,” which is the one that holds the chart.

In the next box, select and copy the codes, paste them to the HTML editor of a blog post or web page, and an interactive chart will be shown as the one below; notice that in the original codes, the default dimensions are 500 pixels wide and 300 pixels tall; I changed it to 650*450 for display in this blog post.

It should be noted that in some browsers, an embedded chart may have unsightly scroll bars along the right and the bottom borders. I have not been able to find a fix, but we can always embed a chart image which is automatically generated and doesn’t have scroll bars. All we need to do is to click “publish chart” on the sheet that holds the chart; in the pop-up window, select “image” as the publish format, copy the link and paste it to a blog post or web page.

Source:http://www.mulinblog.com/basic-web-scraping-data-visualization-using-google-spreadsheets/

Methods for making lucrative article writing routines

Coffee may be of wonderful help when you work at home and require some air flow. Caffeine properties tend to be built with Wi-fi, therefore you can work although consuming that mug of joe to get a various surroundings. Otherwise, several restaurants supply the very same.

Parajumpers Jakke Dame

Can you afford to quit your task and homeschool? Have you developed a price range to find out? Draft an affordable budget of your own existing income and expenditures. Now, eliminate the cash flow of the individual that will be remaining home. Also, include the fee for supplies, like lesson resources, producing devices, papers, and many others. Can you afford it now?

Canada Goose Jassen

Some women's gum line become very vulnerable and bleed whenever they experience the monthly period or hormone adjustments during growing up. If this sounds like your situation, you can easily eliminate this concern if you take dental contraceptive. Check out the dental office to make sure the internal bleeding is just not a result of chewing gum illness.

Canada Goose Jassen

Of course, there exists a stigma that accompanies the phrase "natural and organic," but that's as most individuals fail to know that the saying organic and natural, generally means normal. The truth is, expanding organic and natural is as normal as you can perhaps get. So be sure to start using these garden tips when you're able to expand organic and natural foods.

Natural and organic Horticulture 101: Everything You Need To Know Uggs

When you must concentrate on 2 things, particularly, focus on reading and mathematics. These subjects can be hard for many children to find out. They are also the most-utilized subject areas in their day-to-day lives. A great math and looking at foundation will assist them all through most of their education and life. Try using throat stretches and good posture for a appealing the neck and throat. A healthy the neck and throat is as vital as a wholesome face. Do not treat the the neck and throat area just like the face because each of them age in a different way. As time passes the muscles from the the neck and throat reduce and also the slender skin is not going to react in the same manner as being the skin of your face does to peels and lasers.

Source:http://www.escortlyfe.com/showthread.php?tid=57073

Tools for Data Scraping and Visualization

Over the last few weeks I co-taught a short-course on data scraping and data presentation for.  It was a pleasure to get a chance to teach with Ethan Zuckerman (my boss) and interact with the creative group of students! You can peruse the syllabus outline if you like.

In my Data Therapy work I don’t usually introduce tools, because there are loads of YouTube tutorials and written tutorials.  However, while co-teaching a short-course for incoming students in the Comparative Media Studies program here at MIT, I led two short “lab” sessions on tools for data scraping, interrogation, and visualization.

There are a myriad of tools that support these efforts, so I was forced to pick just a handle to introduce to these students.  I wanted to share the short lists of tools I choose to share.

Data Scraping:

As much as possible, avoid writing code!  Many of these tools can help you avoid writing software to do the scraping.  There are constantly new tools being built, but I recommend these:

   1.Copy/Paste: Never forget the awesome power of copy/paste! There are many times when an hour of copying and pasting will be faster than learning any sort of new tool!
  
   2.Import.io: Still nascent, but this is a radical re-thinking of how you scrape.  Point and click to train their scraper.  It’s very early, and buggy, but on many simple webpages it works well!
  
   3.Regular Expressions: Install a text editor like Sublime Text and you get the power of regular expressions (which I call “Super Find and Replace”).  It lets you define a pattern and find it in any large document.  Sure the pattern definition is cryptic, but learning it is totally worth it (here’s an online playground).
  
   4.Jquery in the browser: Install the bookmarklet, and you can add the JQuery javascript library to any webpage you are viewing.  From there you can use a basic understanding of javascript and the Javascript console (in most browsers) to pull parts of a webpage into an array.
  
   5.ScraperWiki: There are a few things this makes really easy – getting recent tweets, getting twitter followers, and a few others.  Otherwise this is a good engine for software coding.
  
   6.Software Development: If you are a coder, and the website you need to scrape has javascript and logins and such, then you might need to go this route (ugh).  If so, here’s a functioning example of a scraper built in Python (with Beautiful Soup and Mechanize).  I would use Watir if you want to do this in Ruby.

Data Interrogation and Visualization:

There are even more tools that help you here.  I picked a handful of single-purpose tools, and some generic ones to share.

   1.Tabula: There are  few PDF-cleaning tools, but this one has worked particularly well for me.  If your data is in a PDF, and selectable, then I recommend this! (disclosure: the Knight Foundation funds much of my paycheck, and contributed to Tabula’s development as well)

   2.OpenRefine: This data cleaning tool lets you do things like cluster rows in your data that are spelled similarly, look for correlations at a high level, and more!  The School of Data has written well about this – read their OpenRefine handbook.

   3.Wordle: As maligned as word clouds have been, I still believe in their role as a proxy for deep text analysis.  They give a nice visual representation of how frequently words appear in quotes, writing, etc.

   4.Quartz ChartBuilder: If you need to make clean and simple charts, this is the tool for you. Much nicer than the output of Excel.

   5.TimelineJS: Need an online timeline?  This is an awesome tool. Disclosure: another Knight-funded project.

   6.Google Fusion Tables: This tool has empowered loads of folks to create maps online.  I’m not a big user, but lots of folks recommend it to me.

   7.TileMill: Google maps isn’t the only way to make a map.  TileMill lets you create beautiful interactive maps that fit your needs. Disclosure: another Knight-funded project.

   8.Tableau Public: Tableau is a much nicer way to explore your data than Excel pivot tables.  You can drag and drop columns onto a grid and it suggests visualizations that might be revealing in your attempts to find stories.

I hope those are helpful in your data scraping and story-finding adventures!

Source:http://datatherapy.wordpress.com/2013/10/24/tools-for-data-scraping-and-presentation/

What Is OCR Conversion Solutions?

Optical Character Recognition system on laser and inkjet printers usually print media copies of the books and computer systems work most effective with the pictures. Some documents are not appropriate for conversion to editable formatting. Smidgen of them is handwritten or hand-stamped plates are not appropriate for automated processing. Annotation and cross-outs to make a high error rate in the web page.

Furthermore, a combination of text, photographs and pictures to the recognition dilemma, but generally be corrected with some manual adjustments. Contrast, color definition and clarity, optimal final results will not be lost in old documents to scan. In addition, faxes and dot matrix printer generated pages generally poor to deliver final results. Printed copies typed on a typewriter with a worn tire, carbon light with optical character recognition sheet with characters and copies do not make excellent final results.

By the finish of the 1980s, typewriters had been replaced by laptop word processing applications. Many archive a large quantity of typewritten pages. Lightweight paper stocks that enhance or wrinkle, scanner jamming is an additional issue that may possibly. other option is to capture a digital camera files.In such circumstances it is more quickly and much more accurate to enter data manually. On the other hand, OCR scanning Excel spreadsheet format are delimited with tabs for sheets that functions properly.

Comma-separated worth – CSV tabular data must closely resemble the tab. Unsuitable for circumstances where the OCR application, is a greater option manual data entry. Automatic processing implies to go back and save when you require making pretty accurate. It is pretty simple to repair proper the initial time. You will locate that the offshore BPO services (Organization Process Outsourcing Services) with a scanning project outsourcing enterprise is both a timely and economical solutions will be surprised.

The low cost of offshore labor, which is manual is due to a greater and cost forms.OCR and file conversions in different digital formats that can be gebruikt.OCR access to info in the management of what are the pros and cons? OCR readable and printable paper copies of the file for the conversion to a digital format. Rich Text Format, MS Word, PDF and HTML text is the most popular kind through a scanner, OCR application converts. The converted documents simple to manage, edit, and are In enterprise-oriented present age; to readily out there dot matrix printer and quickly accessible format to convert the data to dot matrix printer determine every enterprise is a frequent requirement.

Outsourcing Information Conversion Services tiresome data conversion jobs done immediately and effectively is a cost dot matrix printer successful manner. Managing the quantity of info is time consuming and hard process. Many providers normally locate it hard to concentrate on their core enterprise. Therefore, the data conversion functions to the line speed at which tasks are outsourced to an offshore land.

The stipulated time and price range. Now, the BPO providers in a quantity of high quality, economical for consumers around the planet in delivering data conversion solutions. All enterprise course of action outsourcing providers in the infrastructure and the most up-to-date technology and are equipped with skilled and experienced. The conversion of electronic data from its present kind, these providers separate conversion device, the conversion to build safer and much more reputable.

Source:http://www.joseoshays.com/what-is-ocr-conversion-solutions/

CATHEDRAL India: Outsource Services for Better Optimization

Or the present scenario, every company needs to website for better exposure to connect with or targeted audience. Or is the most affordable and fastest way to reach people globally. But or this highly competitive was, or becomes very difficult to survive. CATHEDRAL services or search engine optimization services have become to requirement for every company to you generate you traffic for your website. The technique needs expertise, knowledge and command over the advanced techniques.

Optimizing the website there you are become the need of every business. Creating to website according to the algorithm of the search engine becomes essential to give other websites huge competition. The optimizing experts analyze the updates and algorithms of the search engine and prosper.

This process needs to hand full amount to set up the business, I am mostly companies outsource the services. Ace far ace cost effectiveness and quality is concerned, India you have emerge ace the best outsourcing service providing destination.

The advantages of selecting India for outsourcing search engine optimization services:-

After OR.S, India is the second largest country with to loosen number of people who speaks English. Even they have to huge number of people who have skills or using the internet tools. The knowledge, expertise and ample human resource have made India ace one of the most favorable destinations for outsourcing the CATHEDRAL services. There till many other reasons too.

Affordable: the leading companies can save to lot of money by outsourcing their work from offshore country. The competitive prices provided by India makes the country ahead or providing outsourcing services. CATHEDRAL services India is optimized by the professionals. There till many talented experts who till providing skillful results.

One stop sign solution: The companies India, which provide outsourcing services to the client, offer to one stop sign solution to them. Starting from the website designing and web maintenance, the company provides go absolution solution to optimize the site. You can outsource the work to the talented hands and invest your swindle on something else.

Effective services: The offshore companies provide talented human resource who is expert or this field. The companies wounds optimizers who till well educated and can able to offer better services to the clients. They gave not compromise with the quality of the services provided by the outsourcing companies.

Customized services: The companies provide services according to the need of the client. You can customize your demands and the company will give or best to make you achieve your goal. They even provide multilingual services I am that people, who can’t read English, can reach your website and get recognized with your product and services.

CATHEDRAL outsourcing business have flourished or India due to or quality of work, dedication towards providing the best services, professionalism, affordability and expertise through experience. If you want to outsource your services, then nothing can beat the services provided by CATHEDRAL India, ace or is to promising company or terms of search engine optimization services.

Source:http://why-outsource.net/cathedral-india-outsource-services-for-better-optimization/

Basic of Search Engine Optimization

 Web presence is like an elementary identity for your business. But nowadays, just having a website didn’t work alone. You must have to follow all the proper SEO guidelines for your website to ameliorate its page ranking on various popular search engines to attract more users. There are some very decisive basic rules must be followed to attain SEO benefits and hiring an experienced company for SEO services will certainly help you to hit the bull’s eye.Â

Have you ever noticed that whenever you pass a certain query or keyword in a search engine, it follows with a number of results from top to bottom? What makes those results to be there is search engine optimization (SEO). If you have a website with several descriptions about your products and services, do you think it’s enough to promote and develop your business further? Currently, we are in an era of extremely advanced technology which has provided us with a number of marvellous techniques and features that developing your business online is just a step away. SEO has changed a lot in a past few years and consequently becomes a little complicated to deal with if you are not updated to its every aspect.

The most common search engines, Google, Yahoo and Bing, they all use a special algorithm to rank your page in their search results. If you want to be one among the top search results, you must have to follow all the techniques of SEO. There are certain things to be taken care of while following the SEO guidelines. Every page on your website must have a unique title tag with somewhere around 70-75 words and it must be informative in order to let users know about your products and services. Secondly, choosing more relevant keywords for search results will certainly get you more users and will be useful for them to know about you.

Hiring a decent and dexterous SEO service provider empowers you to have a great website ranking and hence a considerable amount of traffic. There are a number of SEO companies in Delhi providing considerable services to nourish your business perspectives. One of the most fundamental concepts which is extremely important for your site is having fresh, precise and unique content. This is a pretty decisive factor because all the search engines follow several strict parameters for copied content. Moreover, the selected relevant keywords must be properly stuffed into the content in a proper ratio as per the SEO guidelines. One another important step is to build internal links between pages of your website that connects assorted pieces of content.

Generally, with such an increased stiff competition, having an SEO compatible website is what you need the most. For SEO Delhi you can consult a number of befitting organizations available to have a perfect and desired page ranking to generate huge amount of traffic for your website. All you have to do is taking the right decision by choosing an ideal SEO service provider for your website and the rest will be handled by them. From the selection of appropriate keywords, writing relevant content, building internal links and all other required tasks will be accomplished by them.

Source:http://why-outsource.net/basic-of-search-engine-optimization/

How to make website scraping easy

We thought we'd also go 'back to basics' and explain how retailers can simplify their data extraction process.

Web scraping is a way of extracting data from websites. Rich data extraction ensures that the most comprehensive product information is extracted from the retailer’s ecommerce site.

This ensures that the data remains accurate and up-to-date and leaves less room for error.

Why is web scraping important?

If retailers want to increase their product visibility and display their product inventory across the various channels, data extraction is essential. There are several ways of extracting site data, but one of the most common is screen scraping.

Screen scraping is carried out by a crawler that is sent onto an ecommerce site to capture specific data. This extracted data is then put together to create a product data feed.

Why are websites scraped?

Scraping makes data extraction much easier for retailers. Most of them have complex CMS systems, so their website is usually the only place where all of their product information comes together.

How can retailers improve their site so it’s easy to scrape?

Use IDs and classes within tags

If a website uses IDs and classes within its page tags, it’s much easier to produce Xpaths (the query language for selecting nodes), which are used to navigate through HTML.

Don’t use tables for structuring

If a site’s structure includes a lot of tables, it becomes more difficult to scrape. This is because there are unlikely to be IDs and classes within the table’s data.

Not only this, but when tables are used, the Xpaths can become much longer and are therefore more likely to break.

Don’t use unnecessary AJAX

AJAX (asynchronous JavaScript and XML)tends to load independently from HTML, meaning that it can be missed in the scraping process.

Although the browser does drive content and the HTML does load, sometimes something else will pop up. Though a crawler can be set to wait for AJAX content to load before scraping, any AJAX can still dramatically increase the scraping time.

Avoid using sessions

Unnecessary sessions make it difficult to deep link products and can also make the website difficult to scrape.

This is most common on travel website pages, as search URLs sometimes use sessions, causing them to timeout or expire after a period of time.

Be consistent

Crawlers are programmed to recognise each type of webpage based on its structure; if site pages are inconsistent, the crawler will return invalid results.

So, for example, if the crawler is expecting to find the product price under a particular HTML tag class or id and the client introduces a new product page where the price is located under an unfamiliar HTML tag class or id, the product is likely to be overlooked.

Make sure your website is compliant

All websites should comply with W3C standards, which lays out standards that developers need to adhere to. It’s also best to have well-formed HTML so that Xpaths can be created easily.

For example, if an HTML tag is not closed off properly, it can affect the structure of the site.

Keep your website accessible

Even in its simplest form, your website should be compatible with each of the various internet browsers.

So, even if a user has content blockers switched on, the website should still load. This also makes it much easier to scrape the product data.

Source: http://econsultancy.com/in/blog/63375-how-to-make-website-scraping-easy

Google data scraping for SEO

Whether you agree or not, search engine optimization (SEO) has come a long way since its origination that had revolutionized internet search methods. SEO - a lot depends on planning and strategizing according to the historical data available. Competitor analysis is a significant part of SEO that helps the analyst in determining what is and what not is working for their competitor and how to fine tune his own approach towards it.

Google data scraping or data scraping is about extracting data from web pages, i.e. HTML and XML files in identifying the key components that would influence site performance. Data scraping works on extracting information like keyword, Meta tags, titles, content classifications and ad copies of competitors.

Data extraction has been made possible with sophisticated scraping tools that can perform expert task in extracting information from different websites and web platforms. Data scraping helps with comparing your site’s performance against your competitors across web platforms and therefore, is crucial for SEO analysis. To be honest, the need to rank high in search results is imperative for your website to receive organic traffic and hence, rank monitoring is an important determinant of your website performance.

The type of data scraping tool you need, Google Maps Scraper, Amazon Screen Scraping or LinkedIn Extractor, would be determined by your inherent requirements but you can’t ignore its requirement in extracting valuable information that would help improving website performance to put your venture on the right path of growth.

Source: http://scrappingexperts.blogspot.in/2013/10/google-data-scraping-for-seo.html