' WEB SCRAPING " - DNA - The Latest Ancestry Computer Technology :: FamilyTreeCircles.com Genealogy
<< Previous - Next >>

' WEB SCRAPING " - DNA - The Latest Ancestry Computer Technology

Journal by edmondsallan

Hello to all
I thought you may like a look at the latest technology being put together for ancestry and other subjects > It took me a while to understand it all . Now I have a briefing on what is in the future for our ancestry workings- Some of it is mind boggling on what they are trying to achieve
Web scraping
Web scraping (also called Web harvesting or Web data extraction) is a computer software technique of extracting information from websites. Usually, such software programs simulate human exploration of the Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding certain full-fledged Web browsers, such as the Internet Explorer (IE) and the Mozilla Web browser. Web scraping is closely related to Web indexing, which indexes Web content using a bot and is a universal technique adopted by most search engines. In contrast, Web scraping focuses more on the transformation of unstructured Web content, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to Web automation, which simulates human Web browsing using computer software. Uses of Web scraping include online price comparison, weather data monitoring, website change detection, Web research, Web content mashup and Web data integration.

* 1 Techniques for Web scraping
* 2 Legal issues
* 3 Technical measures to stop bots
* 4 Tools
* 5 See also
* 6 Notes
* 7 References
* 8 External links

Techniques for Web scraping

Web scraping is the process of automatically collecting Web information. Web scraping is a field with active developments sharing a common goal with the semantic Web vision, an ambitious initiative that still requires breakthroughs in text processing, semantic understanding, artificial intelligence and human-computer interactions. Web scraping, instead, favors practical solutions based on existing technologies that are often entirely ad hoc. Therefore, there are different levels of automations that existing Web-scraping technologies can provide:

* Human copy-and-paste: Sometimes even the best Web-scraping technology cannot replace a humans manual examination and copy-and-paste, and sometimes this may be the only workable solution when the websites for scraping explicitly set up barriers to prevent machine automation.

( note ) --> here is the Technical side - I am still struggling with it . How do you keep control of your own website ??? (from me- Edmondsallan )

* Text grepping and regular expression matching: A simple yet powerful approach to extract information from Web pages can be based on the UNIX grep command or regular expression matching facilities of programming languages (for instance Perl or Python).
* HTTP programming: Static and dynamic Web pages can be retrieved by posting HTTP requests to the remote Web server using socket programming.
* DOM parsing: By embedding a full-fledged Web browser, such as the Internet Explorer or the Mozilla Web browser control, programs can retrieve the dynamic contents generated by client side scripts. These Web browser controls also parse Web pages into a DOM tree, based on which programs can retrieve parts of the Web pages.
* HTML parsers: Some semi-structured data query languages, such as XQuery and the hyper-text query language (HTQL), can be used to parse HTML pages and to retrieve and transform Web content.
* Web-scraping software: There are many Web-scraping software tools available that can be used to customize Web-scraping solutions. These software may attempt to automatically recognize the data structure of a page or provide a Web recording interface that removes the necessity to manually write Web-scraping code, or some scripting functions that can be used to extract and transform Web content, and database interfaces that can store the scraped data in local databases.[1]
* Vertical aggregation platforms: There are several companies that have developed vertical specific harvesting platforms. These platforms create and monitor a multitude of bots for specific verticals with no man-in-the-loop, and no work related to a specific target site. The preparation involves establishing the knowledgebase for the entire vertical and then the platform creates the bots automatically. The platforms robustness is measured by the quality of the information it retrieves (usually number of fields) and its scalability (how quick it can scale up to hundreds or thousands of sites). This scalability is mostly used to target the Long Tail of sites that common aggregators find complicated or too labor intensive to harvest content from.[2]
* Semantic annotation recognizing: The Web pages may embrace metadata or semantic markups/annotations which can be made use of to locate specific data snippets. If the annotations are embedded in the pages, as Microformat does, this technique can be viewed as a special case of DOM parsing. In another case, the annotations, organized into a semantic layer,[3] are stored and managed separately from the Web pages, so the Web scrapers can retrieve data schema and instructions from this layer before scraping the pages.

Legal issues

Web scraping may be against the terms of use of some websites. The enforceability of these terms is unclear.[4] While outright duplication of original expression will in many cases be illegal, in the United States the courts ruled in Feist Publications v. Rural Telephone Service that duplication of facts is allowable. U.S. courts have acknowledged that users of "scrapers" or "robots" may be held liable for committing trespass to chattels,[5][6] which involves a computer system itself being considered personal property upon which the user of a scraper is trespassing. The best known of these cases, eBay, Inc. v. Bidder's Edge, Inc., resulted in an injunction ordering Bidder's Edge to stop data mining from the eBay Web site. This case involved automatic placing of bids, known as Auction Sniping. However, to succeed on a claim of trespass to chattels, the plaintiff must demonstrate that the defendant intentionally and without authorization interfered with the plaintiff's possessory interest in the computer system and that the defendant's unauthorized use caused damage to the plaintiff. Not all cases of web spidering brought before the courts have been considered trespass to chattels.[7]

One of the first major tests of screen scraping involved American Airlines, and a firm called FareChase.[8] AA successfully obtained an injunction from a Texas trial court, stopping FareChase from selling software that enables users to compare webfares if it also searches AA's website. The airline argued that FareChase's websearch software trespassed on AA's servers when it collected the publicly available data. The injunction was appealed in 2003.

Southwest Airlines has also challenged screen-scraping practices, and has involved both FareChase and another firm, Outtask, in a legal claim.[9] Southwest Airlines charged that the screen-scraping is Illegal since it is an example of "Computer Fraud and Abuse" and has led to "Damage and Loss" and "Unauthorized Access" of Southwest's site. It also constitutes "Interference with Business Relations", "Trespass" and "Harmful Access by Computer". They also claimed that screen-scraping constitutes what is legally known as Misappropriation and Unjust Enrichment, and is also a breach of the website's user agreement. Outtask denied all these claims, and claimed that the prevailing law in this case should be US Copyright law, and that under copyright, the pieces of information being scraped would not be subject to copyright protection.

Both these cases have yet to go to the Supreme Court.

Although these are early scraping decisions, and the theories of liability are not uniform, it is difficult to ignore a pattern emerging that the courts are prepared to protect proprietary content on commercial Web sites from uses which are undesirable to the owners of such sites. However, the degree of protection for such content is not settled, and will depend on the type of access made by the scraper, the amount of information accessed and copied, the degree to which the access adversely affects the Web site owners system and the types and manner of prohibitions on such conduct.[10]

While the law in this area becomes more settled, entities contemplating using scraping programs to access a public Web site should also consider whether such action is authorized by reviewing the terms of use and other terms or notices posted on or made available through the site. In the latest ruling in the Cvent Inc v. Eventbrite Inc. In the United States district court for the eastern district of Virginia, the court ruled that the terms of use should be brought to the users' attention In order for a Browse wrap contract/license to be enforced.[11] In the plaintiff's website during the period of this trial the terms of use link is displayed among all the links of the site, at the bottom of the page as most sites on the internet. This ruling contradicts the Irish ruling described below. The court also rejected the plaintiff's argument that the browsewrap restrictions were enforceable in view of Virginia's adoption of the Uniform Computer Information Transactions Act (UCITA)--a uniform law that many believed was in favor on common browsewrap contracting practices.[12]

Outside of the United States, in a February, 2006 ruling, the Danish Maritime and Commercial Court (Copenhagen) found systematic crawling, indexing and deep linking by portal site ofir.dk of real estate site Home.dk not to conflict with Danish law or the database directive of the European Union.[13] In February, 2010, the Irish High Court in the case of Ryanair Limited v Billigfluege.de GmbH. This case established a precedent by acknowledging that the Terms of Use on Ryanairs website, including the prohibitions contained in those Terms of Use against screen scraping. The case is currently under appeal in the Supreme Court.[14] In Australia, the Spam Act 2003 outlaws some forms of web harvesting, although this only applies to email addresses.[15][16][17]
Q- from me -edmondsallan
Note --> How does one keep control ( or is it open to all ???? )

Technical measures to stop bots

The administrator of a website can use various measures to stop or slow a bot. Some techniques include:

* If the application is well behaved, adding entries to robots.txt will be adhered to. Google and other well-behaved bots can be stopped this way.
* Blocking an IP address. This will also block all browsing from that address.
* Sometimes bots declare who they are. Well behaved ones do (for example 'googlebot'). They can be blocked on that basis. Unfortunately, malicious bots may declare they are a normal browser.
* Bots can be blocked by excess traffic monitoring.
* Bots can be blocked with tools to verify that it is a real person accessing the site, like a CAPTCHA.
* Commercial anti-bot services: Several companies, such as Pramana, SiteBlackBox and Sentor, offer anti-bot services for websites. A few Web Application Firewalls have limited bot detection capabilities as well.
* Locating bots with a honeypot or other method to identify the IP addresses of automated crawlers.
* Sometimes bots can be blocked with carefully crafted Javascript code.
* Using CSS sprites to display such data as phone numbers or email addresses.[18]


* Apache Camel
* Automation Anywhere
* Data Toolbar
* Outwit Hub
* Firebug
* Greasemonkey
* HtmlUnit
* HTTrack
* iMacros
* iRobotSoft
* JSoup
* SimpleTest
* Methabot
* nokogiri
* Piggy Bank
* watin
* Wget
* WSO2 Mashup Server
* yahoo pipes
* Yahoo! query language (yql)

See also

* Corpus linguistics
* Data scraping
* Mashup (web application hybrid)
* opensocial
* Scraper site
* Screen scraping
* Spamdexing
* Text corpus
* Web crawling

source and References

* Schrenk, Michael (2007). Webbots, Spiders, and Screen Scrapers. No Starch Press. ISBN 978-1-59327-120-6. http://books.google.com/books?id=5vFSAAAAMAAJ.

External links

* The Future of Web Sites = Web Services (with a section on web scraping)
* Open Social Web 3.0 Directory Compare and review web scraping companies

Retrieved from "http://en.wikipedia.org/wiki/Web_scraping"
Categories: World Wide Web | Spamming

This is part of my research into the future that I try to research every day and am trying very hard to understand and learn Its like restarting primer sdhool again.
I am also following at the moment one of the oldest known Nomad trails in the world of yesterday .From- Persia to
Australia which took 50,000 years in time -Fascinating - their stops and changes

- Regards -edmondsallan

Surnames: DNA
Viewed: 4280 times
by edmondsallan Profile | Research | Contact | Subscribe | Block this user
on 2011-03-20 18:18:03

edmondsallan , from auckland .nz , has been a Family Tree Circles member since Aug 2010. is researching the following names: CLAYTON, EDMONDS.

Do you know someone who can help? Share this:


Register or Sign in to comment on this journal.