{"componentChunkName":"component---src-templates-best-practice-detail-tsx","path":"/best-practice/2020-05-08-wordcount","result":{"data":{"currentBlog":{"id":"249242aa-c81b-5ff8-9553-90e8792af73e","frontmatter":{"thumbnail":"https://img.serverlesscloud.cn/2020511/1589207417923-ZalNtxgQAC_small.jpg","authors":["Anycodes"],"categories":["best-practice"],"date":"2020-05-08T00:00:00.000Z","title":"云函数 SCF 与对象存储实现 WordCount","description":"本文将通过 MapReduce 模型实现一个简单的 WordCount 算法，区别于传统使用 Hadoop 等大数据框架，本文使用的是对象存储 COS 与云函数 SCF 的结合","authorslink":["https://zhuanlan.zhihu.com/ServerlessGo"],"translators":null,"translatorslink":null,"tags":["Serverless","WordCount"],"keywords":"Serverless 多环境配置,Serverless 管理环境,Serverless配置方案","outdated":true},"wordCount":{"words":181,"sentences":28,"paragraphs":28},"fileAbsolutePath":"/opt/build/repo/content/best-practice/2020-05-08-wordcount.md","fields":{"slug":"/best-practice/2020-05-08-wordcount/","keywords":["python","serverless","云函数","file","cos","bucket","client","event"]},"html":"

MapReduce 在维基百科中的解释如下：

\n
\n

MapReduce 是 Google 提出的一个软件架构，用于大规模数据集（大于 1TB）的并行运算。概念「Map（映射）」和「Reduce（归纳）」，及他们的主要思想，都是从函数式编程语言借来的，还有从矢量编程语言借来的特性。

\n
\n

通过这段描述，我们知道，MapReduce 是面向大数据并行处理的计算模型、框架和平台，在传统学习中，通常会在 Hadoop 等分布式框架下进行 MapReduce 相关工作，随着云计算的逐渐发展，各个云厂商也都先后推出了在线的 MapReduce 业务。

\n

本文将尝试通过 MapReduce 模型实现一个简单的 WordCount 算法，区别于传统使用 Hadoop 等大数据框架，本文使用对象存储 COS 与云函数 SCF 来实现。

\n

理论基础

\n

在开始之前，我们根据 MapReduce 的要求，先绘制一个简单的流程图:

\n

\n

在这个结构中，我们需要 2 个云函数分别作 Mapper 和 Reducer；以及 3 个对象存储的存储桶，分别作为输入的存储桶、中间临时缓存存储桶和结果存储桶。在实例前，由于我们的函数即将部署在广州区，因此在广州区建立 3 个存储桶：

\n\n \n 复制代码\n \n \n
对象存储1\tap-guangzhou\tsrcmr\n对象存储2\tap-guangzhou\tmiddlestagebucket\n对象存储3\tap-guangzhou\tdestcmr
\n

为了让整个 Mapper 和 Reducer 逻辑更加清晰，在开始之前先对传统的 WordCount 结构进行改造，使其更加适合云函数，同时合理分配\nMapper 和 Reducer 的工作：

\n

\n

功能实现

\n

编写 Mapper 相关逻辑，代码如下：

\n\n \n 复制代码\n \n \n
# -*- coding: utf8 -*-\nimport datetime\nfrom qcloud_cos_v5 import CosConfig\nfrom qcloud_cos_v5 import CosS3Client\nfrom qcloud_cos_v5 import CosServiceError\nimport re\nimport os\nimport sys\nimport logging\nlogging.basicConfig(level=logging.INFO, stream=sys.stdout)\nlogger = logging.getLogger()\nlogger.setLevel(level=logging.INFO)\nregion = u'ap-guangzhou'  # 根据实际情况，修改地域\nmiddle_stage_bucket = 'middlestagebucket'  # 根据实际情况，修改bucket名\ndef delete_file_folder(src):\n    if os.path.isfile(src):\n        try:\n            os.remove(src)\n        except:\n            pass\n    elif os.path.isdir(src):\n        for item in os.listdir(src):\n            itemsrc = os.path.join(src, item)\n            delete_file_folder(itemsrc)\n        try:\n            os.rmdir(src)\n        except:\n            pass\ndef download_file(cos_client, bucket, key, download_path):\n    logger.info(\"Get from [%s] to download file [%s]\" % (bucket, key))\n    try:\n        response = cos_client.get_object(Bucket=bucket, Key=key, )\n        response['Body'].get_stream_to_file(download_path)\n    except CosServiceError as e:\n        print(e.get_error_code())\n        print(e.get_error_msg())\n        return -1\n    return 0\ndef upload_file(cos_client, bucket, key, local_file_path):\n    logger.info(\"Start to upload file to cos\")\n    try:\n        response = cos_client.put_object_from_local_file(\n            Bucket=bucket,\n            LocalFilePath=local_file_path,\n            Key='{}'.format(key))\n    except CosServiceError as e:\n        print(e.get_error_code())\n        print(e.get_error_msg())\n        return -1\n    logger.info(\"Upload data map file [%s] Success\" % key)\n    return 0\ndef do_mapping(cos_client, bucket, key, middle_stage_bucket, middle_file_key):\n    src_file_path = u'/tmp/' + key.split('/')[-1]\n    middle_file_path = u'/tmp/' + u'mapped_' + key.split('/')[-1]\n    download_ret = download_file(cos_client, bucket, key, src_file_path)  # download src file\n    if download_ret == 0:\n        inputfile = open(src_file_path, 'r')  # open local /tmp file\n        mapfile = open(middle_file_path, 'w')  # open a new file write stream\n        for line in inputfile:\n            line = re.sub('[^a-zA-Z0-9]', ' ', line)  # replace non-alphabetic/number characters\n            words = line.split()\n            for word in words:\n                mapfile.write('%s\\t%s' % (word, 1))  # count for 1\n                mapfile.write('\\n')\n        inputfile.close()\n        mapfile.close()\n        upload_ret = upload_file(cos_client, middle_stage_bucket, middle_file_key,\n                                 middle_file_path)  # upload the file's each word\n        delete_file_folder(src_file_path)\n        delete_file_folder(middle_file_path)\n        return upload_ret\n    else:\n        return -1\ndef map_caller(event, context, cos_client):\n    appid = event['Records'][0]['cos']['cosBucket']['appid']\n    bucket = event['Records'][0]['cos']['cosBucket']['name'] + '-' + appid\n    key = event['Records'][0]['cos']['cosObject']['key']\n    key = key.replace('/' + str(appid) + '/' + event['Records'][0]['cos']['cosBucket']['name'] + '/', '', 1)\n    logger.info(\"Key is \" + key)\n    middle_bucket = middle_stage_bucket + '-' + appid\n    middle_file_key = '/' + 'middle_' + key.split('/')[-1]\n    return do_mapping(cos_client, bucket, key, middle_bucket, middle_file_key)\ndef main_handler(event, context):\n    logger.info(\"start main handler\")\n    if \"Records\" not in event.keys():\n        return {\"errorMsg\": \"event is not come from cos\"}\n    secret_id = \"\"\n    secret_key = \"\"\n    config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, )\n    cos_client = CosS3Client(config)\n    start_time = datetime.datetime.now()\n    res = map_caller(event, context, cos_client)\n    end_time = datetime.datetime.now()\n    print(\"data mapping duration: \" + str((end_time - start_time).microseconds / 1000) + \"ms\")\n    if res == 0:\n        return \"Data mapping SUCCESS\"\n    else:\n        return \"Data mapping FAILED\"
\n

同样的方法，建立 reducer.py 文件，编写 Reducer 逻辑，代码如下：

\n\n \n 复制代码\n \n \n
# -*- coding: utf8 -*-\nfrom qcloud_cos_v5 import CosConfig\nfrom qcloud_cos_v5 import CosS3Client\nfrom qcloud_cos_v5 import CosServiceError\nfrom operator import itemgetter\nimport os\nimport sys\nimport datetime\nimport logging\nregion = u'ap-guangzhou'  # 根据实际情况，修改地域\nresult_bucket = u'destmr'  # 根据实际情况，修改bucket名\nlogging.basicConfig(level=logging.INFO, stream=sys.stdout)\nlogger = logging.getLogger()\nlogger.setLevel(level=logging.INFO)\ndef delete_file_folder(src):\n    if os.path.isfile(src):\n        try:\n            os.remove(src)\n        except:\n            pass\n    elif os.path.isdir(src):\n        for item in os.listdir(src):\n            itemsrc = os.path.join(src, item)\n            delete_file_folder(itemsrc)\n        try:\n            os.rmdir(src)\n        except:\n            pass\ndef download_file(cos_client, bucket, key, download_path):\n    logger.info(\"Get from [%s] to download file [%s]\" % (bucket, key))\n    try:\n        response = cos_client.get_object(Bucket=bucket, Key=key, )\n        response['Body'].get_stream_to_file(download_path)\n    except CosServiceError as e:\n        print(e.get_error_code())\n        print(e.get_error_msg())\n        return -1\n    return 0\ndef upload_file(cos_client, bucket, key, local_file_path):\n    logger.info(\"Start to upload file to cos\")\n    try:\n        response = cos_client.put_object_from_local_file(\n            Bucket=bucket,\n            LocalFilePath=local_file_path,\n            Key='{}'.format(key))\n    except CosServiceError as e:\n        print(e.get_error_code())\n        print(e.get_error_msg())\n        return -1\n    logger.info(\"Upload data map file [%s] Success\" % key)\n    return 0\ndef qcloud_reducer(cos_client, bucket, key, result_bucket, result_key):\n    word2count = {}\n    src_file_path = u'/tmp/' + key.split('/')[-1]\n    result_file_path = u'/tmp/' + u'result_' + key.split('/')[-1]\n    download_ret = download_file(cos_client, bucket, key, src_file_path)\n    if download_ret == 0:\n        map_file = open(src_file_path, 'r')\n        result_file = open(result_file_path, 'w')\n        for line in map_file:\n            line = line.strip()\n            word, count = line.split('\\t', 1)\n            try:\n                count = int(count)\n                word2count[word] = word2count.get(word, 0) + count\n            except ValueError:\n                logger.error(\"error value: %s, current line: %s\" % (ValueError, line))\n                continue\n        map_file.close()\n        delete_file_folder(src_file_path)\n    sorted_word2count = sorted(word2count.items(), key=itemgetter(1))[::-1]\n    for wordcount in sorted_word2count:\n        res = '%s\\t%s' % (wordcount[0], wordcount[1])\n        result_file.write(res)\n        result_file.write('\\n')\n    result_file.close()\n    upload_ret = upload_file(cos_client, result_bucket, result_key, result_file_path)\n    delete_file_folder(result_file_path)\n    return upload_ret\ndef reduce_caller(event, context, cos_client):\n    appid = event['Records'][0]['cos']['cosBucket']['appid']\n    bucket = event['Records'][0]['cos']['cosBucket']['name'] + '-' + appid\n    key = event['Records'][0]['cos']['cosObject']['key']\n    key = key.replace('/' + str(appid) + '/' + event['Records'][0]['cos']['cosBucket']['name'] + '/', '', 1)\n    logger.info(\"Key is \" + key)\n    res_bucket = result_bucket + '-' + appid\n    result_key = '/' + 'result_' + key.split('/')[-1]\n    return qcloud_reducer(cos_client, bucket, key, res_bucket, result_key)\ndef main_handler(event, context):\n    logger.info(\"start main handler\")\n    if \"Records\" not in event.keys():\n        return {\"errorMsg\": \"event is not come from cos\"}\n    secret_id = \"SecretId\"\n    secret_key = \"SecretKey\"\n    config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, )\n    cos_client = CosS3Client(config)\n    start_time = datetime.datetime.now()\n    res = reduce_caller(event, context, cos_client)\n    end_time = datetime.datetime.now()\n    print(\"data reducing duration: \" + str((end_time - start_time).microseconds / 1000) + \"ms\")\n    if res == 0:\n        return \"Data reducing SUCCESS\"\n    else:\n        return \"Data reducing FAILED\"
\n

部署与测试

\n

遵循 Serverless Framework 的 yaml 规范，编写 serveerless.yaml:

\n\n \n 复制代码\n \n \n
WordCountMapper:\n  component: \"@serverless/tencent-scf\"\n  inputs:\n    name: mapper\n    codeUri: ./code\n    handler: index.main_handler\n    runtime: Python3.6\n    region: ap-guangzhou\n    description: 网站监控\n    memorySize: 64\n    timeout: 20\n    events:\n      - cos:\n          name: srcmr-1256773370.cos.ap-guangzhou.myqcloud.com\n          parameters:\n            bucket: srcmr-1256773370.cos.ap-guangzhou.myqcloud.com\n            filter:\n              prefix: ''\n              suffix: ''\n            events: cos:ObjectCreated:*\n            enable: true\n\nWordCountReducer:\n  component: \"@serverless/tencent-scf\"\n  inputs:\n    name: reducer\n    codeUri: ./code\n    handler: index.main_handler\n    runtime: Python3.6\n    region: ap-guangzhou\n    description: 网站监控\n    memorySize: 64\n    timeout: 20\n    events:\n      - cos:\n          name: middlestagebucket-1256773370.cos.ap-guangzhou.myqcloud.com\n          parameters:\n            bucket: middlestagebucket-1256773370.cos.ap-guangzhou.myqcloud.com\n            filter:\n              prefix: ''\n              suffix: ''\n            events: cos:ObjectCreated:*\n            enable: true
\n

完成之后，通过 sls --debug 指令进行部署。部署成功之后，进行基本的测试：

\n
    \n
  1. 准备一个英文文档:\n
    2. \n
  2. 登录腾讯云后台，打开我们最初建立的存储桶：srcmr，并上传该文件;
    3. \n
  3. 上传成功之后，稍等片刻即可看到 Reducer 程序已经在 Mapper 执行之后，产出日志:\n
    4. \n
\n

此时，我们打开结果存储桶，查看结果:\n

\n

现在，我们就完成了简单的词频统计功能。

\n

总结

\n

Serverless 架构相是适用于大数据处理的。在腾讯云官网，我们也可以看到其关于数据 ETL 处理的场景描述：

\n
\n

一些数据处理系统中，常常需要周期性/计划性地处理庞大的数据量。例如：证券公司每 12 小时统计一次该时段的交易情况并整理出该时段交易量 top 5，每天处理一遍秒杀网站的交易流日志获取因售罄而导致的错误从而分析商品热度和趋势等。云函数近乎无限扩容的能力可以使您轻松地进行大容量数据的计算。我们利用云函数可以对源数据并发执行多个 mapper 和 reducer 函数，在短时间内完成工作；相比传统的工作方式，使用云函数更能避免资源的闲置浪费从而节省资金。

\n
\n

本实例中，有一键部署多个函数的操作。在实际生产中，每个项目都不会是单个函数单打独斗的，而是多个函数组合应用，形成一个 Service 体系，所以一键部署多个函数就显得尤为重要。通过本实例，希望读者可以对 Serverless 架构的应用场景有更多的了解，并且能有所启发，将云函数和不同触发器进行组合，应用在自身业务中。

\n
\n
\n
\n
\n

传送门：

\n\n
\n

欢迎访问：Serverless 中文网，您可以在 最佳实践 里体验更多关于 Serverless 应用的开发！

","tableOfContents":""},"previousBlog":{"id":"2ed40490-e6db-51f2-a0a0-8fa52c3488d7","frontmatter":{"thumbnail":"https://img.serverlesscloud.cn/2020511/1589207418781-ZalNtxgQAC_small.jpg","authors":["Anycodes"],"categories":["best-practice"],"date":"2020-05-09T00:00:00.000Z","title":"Serverless 与 Websocket 的聊天工具","description":"传统业务实现 Websocket 并不难，然而函数计算基本上都是事件驱动，不支持长链接操作。如果将函数计算与 API 网关结合，是否可以有 Websocket 的实现方案呢？","authorslink":["https://zhuanlan.zhihu.com/ServerlessGo"],"translators":null,"translatorslink":null,"tags":["Serverless","Websocket"],"keywords":"Serverless 多环境配置,Serverless 管理环境,Serverless配置方案","outdated":true},"wordCount":{"words":263,"sentences":43,"paragraphs":43},"fileAbsolutePath":"/opt/build/repo/content/best-practice/2020-05-09-serverless-websocket.md","fields":{"slug":"/best-practice/2020-05-09-serverless-websocket/","keywords":["java","python","serverless","函数计算","云函数","secret","网关","item","客户端","Conf","bucket","retmsg","os"]}},"nextBlog":{"id":"d1450693-7418-5532-8bd4-4fe42f275bdf","frontmatter":{"thumbnail":"https://img.serverlesscloud.cn/2020512/1589274622959-J2kmfdLCPpX436WYvs0D.jpg","authors":["Anycodes"],"categories":["best-practice"],"date":"2020-05-05T00:00:00.000Z","title":"Serverless 与人工智能实现微信公众号的智能服务","description":"一般情况下，想给公众号增加更多的功能，需要我们在服务器上搭建后台服务。那么在 Serverless 架构下，能否更简单地实现相同的后台服务呢？","authorslink":["https://zhuanlan.zhihu.com/ServerlessGo"],"translators":null,"translatorslink":null,"tags":["Serverless","人工智能"],"keywords":"Serverless 多环境配置,Serverless 管理环境,Serverless配置方案","outdated":true},"wordCount":{"words":519,"sentences":119,"paragraphs":119},"fileAbsolutePath":"/opt/build/repo/content/best-practice/2020-05-05-ai-wechat.md","fields":{"slug":"/best-practice/2020-05-05-ai-wechat/","keywords":["python","serverless","函数计算","函数计算服务","event","json","media","time"]}}},"pageContext":{"isCreatedByStatefulCreatePages":false,"blogId":"249242aa-c81b-5ff8-9553-90e8792af73e","previousBlogId":"2ed40490-e6db-51f2-a0a0-8fa52c3488d7","nextBlogId":"d1450693-7418-5532-8bd4-4fe42f275bdf"}}}