{"componentChunkName":"component---src-templates-blog-detail-tsx","path":"/blog/2019-07-23-search-engine","result":{"data":{"currentBlog":{"id":"5e1c5104-bbcb-5059-8f38-36b6503f66f8","frontmatter":{"thumbnail":"https://img.serverlesscloud.cn/2020414/1586878208210-search.jpg","authors":["Anycodes"],"categories":["user-stories"],"date":"2019-07-23T00:00:00.000Z","title":"如何优雅地给搜索引擎去广告！","description":"这真的不是针对友商……","authorslink":["https://www.zhihu.com/people/liuyu-43-97"],"translators":null,"translatorslink":null,"tags":["SEO","Serverless"],"keywords":"Serverless 应用服务端,Serverless 应用,Serverless Web 后端","outdated":null},"wordCount":{"words":238,"sentences":54,"paragraphs":54},"fileAbsolutePath":"/opt/build/repo/content/blog/2019-07-23-search-engine.md","fields":{"slug":"/blog/2019-07-23-search-engine/","keywords":["go","python","serverless","云函数","title","http","在线","serverlesscloud"]},"html":"

众所周知，搜索引擎会有广告主付费推广的内容，这些付费推广常常会干扰到我们的信息获取。例如，搜索「在线编程」这个词条：

\n

\"img\"

\n

一页屏幕上就有 5 个搜索结果是推广内容，就问你怕不怕！

\n

同时，其中大部分命中的关键词是「编程」，而非「在线编程」，也会带给我们干扰以及很不友好的体验。那么我们是否可以通过云函数 SCF 来做一个简单的工具，屏蔽掉这些广告呢？

\n

当然可以！本教程将会通过 SCF+COS 来实现一个简单的网页搜索小工具，至于为啥叫「搜索小工具」而不是叫「去广告搜索小工具」，留个悬念，待会儿再说。

\n\n \n 复制代码\n \n \n
SCF：Serverless Cloud Function，云函数\nCOS：Cloud Object Storage，对象存储
\n

▎寻找广告特点

\n

在对列表页代码研究过程中，我们发现，非广告页代码有：

\n

\"img\"

\n

而推广内容，则没有这个内容。

\n

所以，我们可以通过正则化，或者 XPATH 等操作来对目标内容进行识别。

\n

▎本地实验

\n

一、本地爬虫编写

\n

对页面分析，我们可以得到这样一组参数：

\n\n \n 复制代码\n \n \n
ie=utf-8\npn=10\nwd=在线编程
\n

这组参数中，ie 是控制编码格式，pn 是翻页（偏移量），wd 是搜索词汇，对代码编写如下：

\n\n \n 复制代码\n \n \n
import urllib.request\nimport re\npage = 1\nargs_pn = (page-1) * 10\nurl = "http://www.baidu.com/s?ie=utf-8&pn=" + str(args_pn) + "&wd=" + urllib.request.quote("在线编程")\nreq_attr = urllib.request.urlopen(url)\nresult_list = re.findall("data-tools='(.*?)'",req_attr.read().decode("utf-8"))\nfor eve in result_list:\n    print(eve)
\n

运行结果：

\n\n \n 复制代码\n \n \n
{"title":"代码在线运行 - 在线工具","url":"http://www.baidu.com/link?url=2rtPQM1Yb08uBDUY61IkU3Apr7xkDiP2_zsnZH00HyMsViwBfFR9LJAiXcjqD_EK"}\n{"title":"南邮编程在线","url":"http://www.baidu.com/link?url=f_VCURb1ZNe7nVgW3G7IiBBvKDGaeht8SB3hK93jEEddFAtpJC2SGzBoFTs1BLWd"}\n{"title":"在线编程 - 编程中国","url":"http://www.baidu.com/link?url=hnYcaMNhRhf5FZfV1vlE5SGS6GjUls4fluKKW52mMIAHwXJd1wey2mOdmtb13ldf"}\n{"title":"一个简单的在线集成编程工具 - Anycodes - 随时随地有创意,随时...","url":"http://www.baidu.com/link?url=zMnTtD6cBS_3XT21FMSviS_mzF0T2daHAui6_XC7LAscng2KPt064eej0JqPgfBX"}\n{"title":"非常好用的在线编程网站 - yimisiyang - CSDN博客","url":"http://www.baidu.com/link?url=57aywD0Q6WTnl7XKbIHuE7R_FT8mrDBMv8kdi_tomry-X1SrzMnjNVuobcIXjUVnR_ZQrRZQrTHv5uSgfZ-GdR5ud1WK9aOaAKTP-KhFr4e"}\n{"title":"十个在线编程网站 _360doc个人图书馆","url":"http://www.baidu.com/link?url=ouVP_VD3EkdRLL-X2bjZNJTdFbVIJgt0jrZL3II0hxj5yT3aTAo56tUA0hUfrCbiZKWcmBNTLKcTiwA7q1bxQJwxdZnKEUzChwQ_nefSJK3"}\n{"title":"[转]在线编程刷题网站 - gravely的专栏 - CSDN博客","url":"http://www.baidu.com/link?url=gPgyV9kX2IdYqfifVCn-22OPGWIZqBb3oGfRbl0b_LVRr0MmdgiXoaACk-vRZu6f45q3qsG4X3Zs2wexiex-0K"}\n{"title":"c在线编译器,c语言在线解释器,在线编程网站","url":"http://www.baidu.com/link?url=ETluxwW57CgsUIBR7-BKj0bLA65WJr2eipQ5rJOF-1AB3ymP4AGViYXC7AUCrUcW"}\n{"title":"8个国外最流行的在线编程练习网站","url":"http://www.baidu.com/link?url=3o0SRtHkYsI2gUjmjsks7dYk00VoljXmVhnkv-CskKh9aiaTAfyDPNa25f4Mj-7AcLrX9eiaObQVChuX5eG0BfWOcE1U-PGsVFWpCFLKSzq"}\n{"title":"在线编程_百度百科","url":"http://www.baidu.com/link?url=0uxpDuoQDb_Amx9_6n9PqOpJr10tcRrbU0x-hshhnQNx2mVHyCiDJFsqthe-cEUQNv0AO0KFDYmCZP970sW8mVG02xwha-cNalwKUAogkmpwOQVy5xf9lqKdFROOMqBQ"}
\n

可以看到，这个结果是第一页的搜索结果，他包括了：

\n

\"img\"

\n

首页结果，一对一匹配，并且没有上下的广告内容。证明我们本搜索结果是靠谱的。

\n

二、搜索页面制作

\n

以一个非常简单的基础页面作为示例：

\n\n\n\n \n 新搜索 - Powered by Dfounderliu！\n\n\n\n
\n 请输入要搜索的内容：
\n \n
\n
\n
    \n
  1. 结果1
    2. \n
  2. 结果2
    3. \n
\n
\n\n`, `84651185694124200000`)\"\n >\n \n 复制代码\n \n \n
<!DOCTYPE html>\n<html lang="en">\n<head>\n    <meta charset="UTF-8">\n    <title>新搜索 - Powered by Dfounderliu！</title>\n</head>\n<body>\n\n<form>\n    请输入要搜索的内容：<input type="text"><br>\n    <input type="submit">\n</form>\n<div id="result">\n    <ol>\n        <li>结果1</li>\n        <li>结果2</li>\n    </ol>\n</div>\n</body>\n</html>
\n

效果大概这个样子：

\n

\"img\"

\n

▎发布服务

\n

一、发布 SCF 服务

\n\n \n 复制代码\n \n \n
# -*- coding: utf8 -*-\nimport json\nimport urllib.request\nimport re\ndef main_handler(event, context):\n    print(event["pathParameters"]["kw"])\n    print(event["pathParameters"]["pn"])\n    page = int(event["pathParameters"]["pn"])\n    args_pn = (page-1) * 10\n    url = "http://www.baidu.com/s?ie=utf-8&pn=" + str(args_pn) + "&wd=" + event["pathParameters"]["kw"]\n    req_attr = urllib.request.urlopen(url)\n    result_list = re.findall("data-tools='(.*?)'",req_attr.read().decode("utf-8"))\n    result = ""\n    for eve in result_list:\n        temp_json = json.loads(eve)\n        temp_result = '<li><a href="%s">%s</a></li>'%(temp_json["url"],temp_json["title"])\n        result = result + temp_result\n    return result
\n

二、API 网关配置

\n

\"img\"

\n

三、API 测试

\n

\"img\"

\n

四、静态页面

\n

接下来，对 HTML 页面进行修改，并且上传到 COS。

\n

页面修改为：

\n\n\n\n \n 新搜索 - Powered by Dfounderliu！\n \n \n\n\n

在线搜索工具

\n
\n
\n 请输入要搜索的内容：
\n 请输入要跳转的页面：

\n   |  \n
\n
\n
    \n
\n
\n\n`, `82868987263387190000`)\"\n >\n \n 复制代码\n \n \n
<!DOCTYPE html>\n<html lang="en">\n<head>\n    <meta charset="UTF-8">\n    <title>新搜索 - Powered by Dfounderliu！</title>\n    <script src="https://cdn.staticfile.org/jquery/1.10.2/jquery.min.js"></script>\n    <script>\n        var UTFTranslate = {\n            Change: function (pValue) {\n                return pValue.replace(/[^\\u0000-\\u00FF]/g, function ($0) {\n                    return escape($0).replace(/(%u)(\\w{4})/gi, "&#x$2;")\n                });\n            },\n            ReChange: function (pValue) {\n                return unescape(pValue.replace(/&#x/g, '%u').replace(/\\\\u/g, '%u').replace(/;/g, ''));\n            }\n        };\n\n        function searchKw() {\n            var kw = document.getElementById("kw").value;\n            var pn = document.getElementById("pn").value;\n            $.get("http://service-rnwcbtai-1256773370.gz.apigw.tencentcs.com/release/list/" + kw + "/" + pn, function (data, status) {\n                document.getElementById("result").innerHTML = UTFTranslate.ReChange(data)\n            });\n        }\n    </script>\n</head>\n<body>\n<center><h1>在线搜索工具</h1></center>\n<hr>\n<center>\n    请输入要搜索的内容：<input type="text" id="kw"><br>\n    请输入要跳转的页面：<input type="text" id="pn"><br><br>\n    <button type="reset">重置</button> &nbsp;&nbsp;|&nbsp;&nbsp;<button onclick="searchKw()">搜索</button>\n</center>\n<div>\n    <ol id="result">\n    </ol>\n</div>\n</body>\n</html>
\n

在对象存储中新建存储桶，并上传刚才的 html 文件：

\n

\"img\"

\n

修改权限为公有读，私有写：

\n

\"img\"

\n

基础配置，开启静态网站：

\n

\"img\"

\n

五、访问域名测试

\n

\"img\"

\n

▎总结

\n

至此，我们便完成了一个基本的去广告搜索引擎。

\n

有的同学可能就疑惑了：

\n
    \n
  1. 本分享主要目的何在？
    2. \n
  2. 这个例子有什么价值？
    3. \n
\n

关于上面两个问题，简单回复：

\n

本分享主要想告诉大家，有了 SCF 和 COS 以及 CDB 等产品。作为用户，可以暂时不用 CVM 了。静态网页，我们可以放在 COS 中，后端的处理放在 SCF，数据存储到 CDB，然后请求页面的时候，COS 为我们保证了基本的访问功能和足够的带宽，SCF 为我们提供了弹性伸缩的功能，让用户无需考虑流量多少，是否要对 CVM 扩容等。

\n

这样能让我们将更多的精力放到 Coding 上。试想，如果传统的一个 Python 程序员想要做这样一个网页，他需要哪些知识？需要 Django/flask 等基本框架知识，需要一些 Web 端的运维，例如怎么配置环境，怎么配置 Nginx 等操作。现在呢？不需要这些框架知识，你只需要做一个简单的 Html，Ajax 通信。然后，后端就是我们平常的一个爬虫，也不需要很多复杂的配置文件。这不就是一种进步么！

\n

第二点，这个教程表面上就是一个云函数 SCF 运行一个爬虫，接入了 COS 的静态页面，让我们可以不用 CVM 也能做出一个网页。但实际上，这个 demo 意义重大。首先，这么一个简单的页面，就可以完成目标页面预览功能，当鼠标移动到对应的 URL 上面，会浮动出目标网页的样子和关键词命中的位置，这算不算搜索引擎中的一种创新？

\n

此外，除了去除搜索引擎的广告，还可以继续拓展。我们可将 Google、360、bing 等多家搜索数据进行整合，然后统一显示出来，这也是一种方便。或者，我们整合程序员常去的社区论坛，编写多个爬虫，便能实现程序员专用搜索。当然，这只是本例子的一个应用场景。这个 Demo 还是希望能给大家一个思路：

\n

原来做网站可以不用 CVM 了，直接用 COS+SCF+CDB 就能搞定了啊！

\n
\n
\n

传送门：

\n\n
\n

欢迎访问：Serverless 中文网，您可以在 最佳实践 里体验更多关于 Serverless 应用的开发！

","tableOfContents":""},"previousBlog":{"id":"e8e2574a-3de9-5b3c-bb8a-75e26713caa6","frontmatter":{"thumbnail":"https://img.serverlesscloud.cn/20191227/1577408769674-v2-3635e3b55ee1fda578ea953ba927da13_1200x500.jpg","authors":["Alfred Huang"],"categories":["guides-and-tutorials"],"date":"2019-08-01T00:00:00.000Z","title":"Serverless 基本概念入门","description":"从行业趋势看，Serverless 是云计算必经的一场革命","authorslink":["https://zhuanlan.zhihu.com/ServerlessGo"],"translators":null,"translatorslink":null,"tags":["云计算","Serverless"],"keywords":"Serverless 基本概念,Serverless 入门,Serverless 基础","outdated":null},"wordCount":{"words":407,"sentences":75,"paragraphs":75},"fileAbsolutePath":"/opt/build/repo/content/blog/2019-08-01-serverless-basic-concept.md","fields":{"slug":"/blog/2019-08-01-serverless-basic-concept/","keywords":["serverless","无服务器","云函数","云原生","Serverless","架构","函数","应用","产品","业务","无需","服务器","运维"]}},"nextBlog":{"id":"344dc497-fd61-5441-9106-d593870bad6a","frontmatter":{"thumbnail":"https://img.serverlesscloud.cn/2020414/1586850670017-%E5%B0%81%E9%9D%A2%E5%9B%BE%20%283%29.png","authors":["Anycodes"],"categories":["user-stories"],"date":"2019-07-15T00:00:00.000Z","title":"网站监控脚本的实现","description":"本文将分享如何通过腾讯云云函数 SCF 开发一个简单的脚本，进行网站监控","authorslink":["https://www.zhihu.com/people/liuyu-43-97"],"translators":null,"translatorslink":null,"tags":["监控脚本","Serverless"],"keywords":"Serverless 应用服务端,Serverless 应用,Serverless Web 后端","outdated":null},"wordCount":{"words":157,"sentences":54,"paragraphs":53},"fileAbsolutePath":"/opt/build/repo/content/blog/2019-07-15-website-monitor.md","fields":{"slug":"/blog/2019-07-15-website-monitor/","keywords":["python","serverless","无服务器","无服务器云函数","云函数","电信","联通","江苏","多线","广东","final","耗时"]}},"recommendBlogs":{"edges":[{"node":{"id":"4300b21c-7209-5256-86ff-0d38e3daec9b","frontmatter":{"thumbnail":"https://main.qcloudimg.com/raw/14f1c8eed372e76c1b139703b2f6d0fa.jpg","authors":["KieranMcCarthy"],"categories":["user-stories","engineering-culture"],"date":"2018-01-09T00:00:00.000Z","title":"我是如何在四年时间里，从厨师转行为 Serverless 应用开发者","description":"我是厨师出身，现在成为了一名 Serverless 应用开发者。","authorslink":["https://serverless.com/author/kieranmccarthy/"],"translators":["Aceyclee"],"translatorslink":["https://www.zhihu.com/people/Aceyclee"],"tags":["应用开发","Serverless"],"keywords":"Serverless 应用开发,Serverless 管理,厨师转行为 Serverless 应用开发者","outdated":null},"wordCount":{"words":285,"sentences":38,"paragraphs":36},"fileAbsolutePath":"/opt/build/repo/content/blog/2018-01-09-from-chef-to-serverless-developer-in-4-years.md","fields":{"slug":"/blog/2018-01-09-from-chef-to-serverless-developer-in-4-years/","keywords":["无服务器","无服务器开发","云函数","学习","Serverless","构建","Framework","开发者","服务器","应用","学位","简历"]}}},{"node":{"id":"713a0563-4bf9-5721-bacb-3b4ef609fe4a","frontmatter":{"thumbnail":"https://s3-us-west-2.amazonaws.com/assets.blog.serverless.com/camp-fire/camp-fire-housing-thumb.jpg","authors":["EricWyne"],"categories":["guides-and-tutorials","user-stories"],"date":"2018-12-05T00:00:00.000Z","title":"Serverless Twitter 机器人帮助为坎普山火受灾者安置住房","description":"加利福尼亚州的坎普山火致使数千人流离失所，为此，我构建了一个简单的 Serverless Twitter 机器人来帮助将受灾者安置在临时住房！","authorslink":["https://serverless.com/author/ericwyne/"],"translators":["Aceyclee"],"translatorslink":["zhihu.com/people/Aceyclee"],"tags":null,"keywords":null,"outdated":null},"wordCount":{"words":157,"sentences":26,"paragraphs":26},"fileAbsolutePath":"/opt/build/repo/content/blog/2018-12-05-serverless-twitter-camp-fire.md","fields":{"slug":"/blog/2018-12-05-serverless-twitter-camp-fire/","keywords":["serverless","无服务器","云函数","Serverless","org","住房","Twitter","函数","受灾","机器人","山火"]}}},{"node":{"id":"98602143-b837-5f50-a24f-3b1ec76044d7","frontmatter":{"thumbnail":"https://s3-us-west-2.amazonaws.com/assets.blog.serverless.com/sqquid/sqquid-serverless-thumb.jpg","authors":["RonPeled"],"categories":["user-stories"],"date":"2018-12-17T00:00:00.000Z","title":"SQQUID：100% 无服务器初创公司","description":"SQQUID 将 AWS Lambda 和无服务器框架用于其核心产品和营销网站。我们来看看一个完全无服务器的初创公司是怎样的。","authorslink":null,"translators":null,"translatorslink":null,"tags":null,"keywords":null,"outdated":null},"wordCount":{"words":266,"sentences":42,"paragraphs":42},"fileAbsolutePath":"/opt/build/repo/content/blog/2018-12-17-sqquid-one-hundred-percent-serverless.md","fields":{"slug":"/blog/2018-12-17-sqquid-one-hundred-percent-serverless/","keywords":["go","serverless","无服务器","无服务器架构","服务器","架构","Lambda","集成","FaaS","串行","系统"]}}},{"node":{"id":"29dc2e58-d2ba-56f9-aee1-d21b0bc62e0e","frontmatter":{"thumbnail":"https://s3-us-west-2.amazonaws.com/assets.blog.serverless.com/ao-com-story/ao-serverless-thumbnail.png","authors":["NickGottlieb"],"categories":["user-stories"],"date":"2019-04-24T00:00:00.000Z","title":"AO.com：逐渐转向无服务器优先","description":"AO.com 的 SCV 团队率先尝试无服务器服务。折服于无服务器框架的快速周转时间和低维护成本，整个团队逐渐转向无服务器优先。","authorslink":null,"translators":null,"translatorslink":null,"tags":null,"keywords":null,"outdated":null},"wordCount":{"words":236,"sentences":42,"paragraphs":35},"fileAbsolutePath":"/opt/build/repo/content/blog/2019-04-24-ao-serverless-first.md","fields":{"slug":"/blog/2019-04-24-ao-serverless-first/","keywords":["serverless","无服务器","服务器","团队","Lambda","功能","构建"]}}},{"node":{"id":"752d08d1-387a-5bde-acf3-98141baab294","frontmatter":{"thumbnail":"https://img.serverlesscloud.cn/2020414/1586871710979-%E5%85%AC%E5%85%B1%E7%94%A8.png","authors":["Anycodes"],"categories":["user-stories"],"date":"2019-06-20T00:00:00.000Z","title":"如何用 Serverless 为 Python 云函数打包依赖","description":"在使用无服务器云函数SCF时通常会遇到导入第三方库的问题，很多小伙伴比较头疼是：应该如何打包进去？这里，推荐几个不错的方法。","authorslink":["https://zhuanlan.zhihu.com/ServerlessGo"],"translators":null,"translatorslink":null,"tags":["云函数","Serverless"],"keywords":"Serverless,Serverless应用,无服务器云函数","outdated":null},"wordCount":{"words":81,"sentences":43,"paragraphs":43},"fileAbsolutePath":"/opt/build/repo/content/blog/2019-06-20-for-python-cloud-functions.md","fields":{"slug":"/blog/2019-06-20-for-python-cloud-functions/","keywords":["java","serverless","无服务器","无服务器云函数","云函数","serverlesscloud","安装","serverless","pillowtest"]}}},{"node":{"id":"2dc78814-9d77-555b-a1bb-ad202c8ec2d1","frontmatter":{"thumbnail":"https://s3-us-west-2.amazonaws.com/assets.blog.serverless.com/cloudforecast/thumbnail.png","authors":["FrancoisLagier"],"categories":["user-stories"],"date":"2019-08-07T00:00:00.000Z","title":"Serverless：初创企业的理想选择？（CloudForecast 案例分析）","description":"CloudForecast 是 2018 年成立的一家独立初创企业，本文将介绍他们决定选择 Serverless 的原因。","authorslink":["https://serverless.com/author/francoislagier/"],"translators":["Aceyclee"],"translatorslink":["zhihu.com/people/Aceyclee"],"tags":null,"keywords":null,"outdated":null},"wordCount":{"words":211,"sentences":29,"paragraphs":29},"fileAbsolutePath":"/opt/build/repo/content/blog/2019-08-07-serverless-for-startups.md","fields":{"slug":"/blog/2019-08-07-serverless-for-startups/","keywords":["serverless","云函数","serverless","函数","Serverless","utm","Framework","blog","CloudForecast","cloudforecast"]}}},{"node":{"id":"97450b07-658b-5207-8216-1c7b9b51b115","frontmatter":{"thumbnail":"https://img.serverlesscloud.cn/2020114/1578988490344-v2-8b2cd2c5275aa2c5a3c5083a148a7a9f_1200x500.jpg","authors":["Anycodes"],"categories":["user-stories"],"date":"2019-09-01T00:00:00.000Z","title":"如何通过 Serverless 与自然语言处理，让搜索引擎「看」到你的博客","description":"Serverless 与自然语言处理结合的一个小应用","authorslink":["https://www.zhihu.com/people/liuyu-43-97"],"translators":null,"translatorslink":null,"tags":["个人博客","serverless"],"keywords":"Serverless 自然语言处理","outdated":null},"wordCount":{"words":106,"sentences":34,"paragraphs":34},"fileAbsolutePath":"/opt/build/repo/content/blog/2019-09-01-search-engine-blog.md","fields":{"slug":"/blog/2019-09-01-search-engine-blog/","keywords":["serverless","云函数","keywords","serverlesscloud","summary"]}}},{"node":{"id":"ae4fd2f8-515c-5aec-b584-38427ef33f7e","frontmatter":{"thumbnail":"https://img.serverlesscloud.cn/2020114/1578989800047-part-00492-780.jpg","authors":["Anycodes"],"categories":["guides-and-tutorials","user-stories"],"date":"2019-09-16T00:00:00.000Z","title":"突破传统 OJ 瓶颈，「判题姬」接入云函数","description":"通过 Serverless 实现在线编程","authorslink":["https://www.zhihu.com/people/liuyu-43-97"],"translators":null,"translatorslink":null,"tags":["在线编程","云函数"],"keywords":"Serverless 在线编程,Serverless OJ","outdated":null},"wordCount":{"words":169,"sentences":30,"paragraphs":30},"fileAbsolutePath":"/opt/build/repo/content/blog/2019-09-16-online-Judge.md","fields":{"slug":"/blog/2019-09-16-online-Judge/","keywords":["python","serverless","云函数","代码","函数","serverless"]}}}],"totalCount":64}},"pageContext":{"isCreatedByStatefulCreatePages":false,"blogId":"5e1c5104-bbcb-5059-8f38-36b6503f66f8","previousBlogId":"e8e2574a-3de9-5b3c-bb8a-75e26713caa6","nextBlogId":"344dc497-fd61-5441-9106-d593870bad6a","categories":["user-stories"]}}}