{"moduleinfo":{"card_count":[{"count_phone":1,"count":1}],"search_count":[{"count_phone":4,"count":4}]},"card":[{"des":"阿里技术人对外发布原创技术内容的最大平台;社区覆盖了云计算、大数据、人工智能、IoT、云原生、数据库、微服务、安全、开发与运维9大技术领域。","link1":"https://developer.aliyun.com/group/?spm=a2c6h.12883283.1377930.25.7287201c9RKTCi&groupType=other","link":"https://developer.aliyun.com/","icon":"https://img.alicdn.com/tfs/TB1TlXBEkT2gK0jSZPcXXcKkpXa-200-200.png","btn2":"开发者藏经阁","tip":"打通开发者成长路径，学习中心 。全线阿里云技术大牛公开课，立即查看","btn1":"技术与产品技术圈","link2":"https://developer.aliyun.com/topic/ebook?spm=a2c6h.12883283.1362932.15.7287201c9RKTCi","title":"阿里云开发者社区"}],"search":[{"txt":"学习中心","link":"https://developer.aliyun.com/learning?spm=a2c6h.13788135.1364563.41.299f5f24exe3IS"},{"txt":"技能测试中心 ","link":"https://developer.aliyun.com/exam?spm=a2c6h.13716002.1364563.42.6cac18a3JWCM5U"},{"txt":"开发者云 ","link":"https://developer.aliyun.com/adc/?spm=a2c6h.13716002.1364563.59.6b0818a3DV0vzN"},{"txt":"在线编程 ","link":"https://developer.aliyun.com/coding?spm=5176.13257455.1364563.57.701e7facHvqi5r"},{"txt":"学习中心 ","link":"https://developer.aliyun.com/learning?spm=a2c6h.12883283.1364563.41.5f1f201c5CLDCC"},{"txt":"高校计划 ","link":"https://developer.aliyun.com/adc/college/?spm=a2c6h.13716002.1364563.58.6cac18a3JWCM5U"}],"countinfo":{"search":{"length_pc":0,"length":0},"card":{"length_pc":0,"length":0}}}
{"$env":{"JSON":{}},"$page":{"env":"production"},"$context":{"moduleinfo":{"card_count":[{"count_phone":1,"count":1}],"search_count":[{"count_phone":4,"count":4}]},"card":[{"des":"阿里技术人对外发布原创技术内容的最大平台;社区覆盖了云计算、大数据、人工智能、IoT、云原生、数据库、微服务、安全、开发与运维9大技术领域。","link1":"https://developer.aliyun.com/group/?spm=a2c6h.12883283.1377930.25.7287201c9RKTCi&groupType=other","link":"https://developer.aliyun.com/","icon":"https://img.alicdn.com/tfs/TB1TlXBEkT2gK0jSZPcXXcKkpXa-200-200.png","btn2":"开发者藏经阁","tip":"打通开发者成长路径，学习中心 。全线阿里云技术大牛公开课，立即查看","btn1":"技术与产品技术圈","link2":"https://developer.aliyun.com/topic/ebook?spm=a2c6h.12883283.1362932.15.7287201c9RKTCi","title":"阿里云开发者社区"}],"search":[{"txt":"学习中心","link":"https://developer.aliyun.com/learning?spm=a2c6h.13788135.1364563.41.299f5f24exe3IS"},{"txt":"技能测试中心 ","link":"https://developer.aliyun.com/exam?spm=a2c6h.13716002.1364563.42.6cac18a3JWCM5U"},{"txt":"开发者云 ","link":"https://developer.aliyun.com/adc/?spm=a2c6h.13716002.1364563.59.6b0818a3DV0vzN"},{"txt":"在线编程 ","link":"https://developer.aliyun.com/coding?spm=5176.13257455.1364563.57.701e7facHvqi5r"},{"txt":"学习中心 ","link":"https://developer.aliyun.com/learning?spm=a2c6h.12883283.1364563.41.5f1f201c5CLDCC"},{"txt":"高校计划 ","link":"https://developer.aliyun.com/adc/college/?spm=a2c6h.13716002.1364563.58.6cac18a3JWCM5U"}],"countinfo":{"search":{"length_pc":0,"length":0},"card":{"length_pc":0,"length":0}}}}

ABSTRACT
The main purpose of this project is to design subject-oriented web crawler process, which require to meet different performance and related to the various details of the targeted web crawler and application in detail.
Search engine is a tool to help people retrieve information. However, these general search engines also have some limitations. Users in different fields and backgrounds tend to have different purposes and needs, and the results returned by general search engines contain a large number of web pages that users don't care about. In order to solve this problem, it is of great significance for a flexible crawler.
Web crawler application of intelligent self construction technology, with the different themes of the site, you can automatically analyze the structure of URL, and cancel duplicate part. Web crawler use multi-threading technology, so that the crawler has a more powerful ability to grab. Setting connection and reading time of the network crawler is to avoid unlimited waiting. In order to adapt to the different needs, the web crawler can base on the preset themes to realize to filch the specific topics. What’s more, we should study the principle of the web crawler ,realize the relevant functions of reptiles, save the stolen data to the database after cleaning and in late achieve the visual display.
Keywords:Web crawler,Directional climb,multi-threading,mongodb

基于Python的网络爬虫摘要随着计算机技术的不断发展，新的编程语言层出不穷，Python，Html正是其中的佼佼者。相比较早期普及的高级语言(Java,C语言)等，Python有着更加实用的模块和库，虽然牺牲了底层性，但却更加方便用于开发小型项目。此外，Html也已经被普遍用于网站前端，标记语言的特性结合CSS丰富了网页内容和形式，某种意义上也促进了更加人性化的电子商务系统的发展。本文的网络爬虫正式基于Python语言编写的，通过对Html抓取加工将数据可视化，以监测逐渐从线下转移到线上，并随着电子商务发展越发简单普遍难以追踪的非法野生动物贸易。本文首先对计算器语言的发展，尤其是Python和Html的优势、基础概念以及性能进行介绍。最后着重介绍可以实现检测网络上非法野生动物贸易的爬虫的设计和实现。程序主要包括三个模块:URL解析，Html抓取，本地输出。Python的开发环境和工具分别是OS X和PyChram CE，主要调用了Python中的Urllib2，beautifulsoup模块。最终的程序可以实现对指定网站，指定关键词抓取指定内容，保存内容到本地，以便于追踪和检测。关键字:Python，HTML，爬虫，非法野生动物贸易Python-Based "Illegal wildlife trade" SpiderElectronics & Information Technology Program 11-1Haozhi ZhuSupervisorRui ZhaoAbstractWith the continuous development of computer technology, the new programming language after another, Python, Html is the one of the best. Compared to the early popularity of high-level language (Java, C language), etc., Python has a more practical modules and libraries, although at the expense of the underlying property, but it is more convenient for the development of small-scale projects. In addition, Html has also been widely used web front end, feature rich CSS markup language binding of web content and form, in a sense also promoted the development of a more user-friendly e-commerce system. In this paper, based on the official web crawler written in Python, Html crawling through the data visualization process to monitor the gradual shift from offline to online, and with the development of electronic commerce more simple universal untraceable illegal wildlife trade.Firstly, the development of the calculator language, especially Python and Html advantages, basic concepts and performance are introduced. The last focuses on the design and implementation can be achieved illegal wildlife trade on the reptile detection network. Program includes three modules: URL parsing, Html capture, data visualization output. Python development environment and tools are OS X and PyChram CE, mainly call in Python Urllib2, re and os module.The

北京林业大学本科毕业论文(设计)
PAGE \* MERGEFORMATI
基于Python的网络爬虫
摘要
随着计算机技术的不断发展，新的编程语言层出不穷，Python，Html正是其中的佼佼者。相比较早期普及的高级语言(Java,C语言)等，Python有着更加实用的模块和库，虽然牺牲了底层性，但却更加方便用于开发小型项目。此外，Html也已经被普遍用于网站前端，标记语言的特性结合CSS丰富了网页内容和形式，某种意义上也促进了更加人性化的电子商务系统的发展。本文的网络爬虫正式基于Python语言编写的，通过对Html抓取加工将数据可视化，以监测逐渐从线下转移到线上，并随着电子商务发展越发简单普遍难以追踪的非法野生动物贸易。
本文首先对计算器语言的发展，尤其是Python和Html的优势、基础概念以及性能进行介绍。最后着重介绍可以实现检测网络上非法野生动物贸易的爬虫的设计和实现。程序主要包括三个模块:URL解析，Html抓取，本地输出。Python的开发环境和工具分别是OS X和PyChram CE，主要调用了Python中的Urllib2，beautifulsoup模块。
最终的程序可以实现对指定网站，指定关键词抓取指定内容，保存内容到本地，以便于追踪和检测。
关键字:Python，HTML，爬虫，非法野生动物贸易
Python-Based "Illegal wildlife trade" Spider
Electronics & Information Technology Program 11-1
Haozhi Zhu
SupervisorRui Zhao
Abstract
With the continuous development of computer technology, the new programming language after another, Python, Html is the one of the best. Compared to the early popularity of high-level language (Java, C language), etc., Python has a more practical modules and libraries, although at the expense of the underlying property, but it is more convenient for the development of small-scale projects. In addition, Html has also been widely used web front end, feature rich CSS markup language binding of web content and form, in a sense also promoted the development of a more user-friendly e-commerce system. In this paper, based on the official web crawler written in Python, Html crawling through the data visualization process to monitor the gradual shift from offline to online, and with the development of electronic commerce more simple universal untraceable illegal wildlife trade.
Firstly, the development of the calculator language, especially Python and Html advantages, basic concepts and performance are introduced. The last focuses on the design and implementation can be achieved illegal wildlife trade on the reptile detection network. Program includes three modules: URL parsing, Html capture, data visualization output. Python development environment and tools are OS X and PyChr